Live Audio Feed Server Build

Posted: October 11, 2014 in Uncategorized

20141007_231018

So, here’s the jist of our current build: We’re bringing in the live audio from the surrounding counties using a GRE PRO-2096 digital trunk tracking scanner which is tuned to Fire/EMS/Police Dispatch and TAC/OPS channels for Wake and Johnston County’s, with priority set to Johnston’s VHF paging system, at the moment, due to me running their calls for photos. I have one TRS output from the receiver’s external speaker jack going to a set of amplified desktop speakers and the other (coming off of a Y-Splitter) is going into the sound card of my production server. I finally made the switch over from the development server earlier yesterday, with a few upgrades along the way.

So using some internal software magic (mostly reliant on Icecast2 and DarkIce), it “rebroadcasts” the direct audio input from the scanner over the Internet from my server using HTTP port 8000, with the sound output in *.ogg (Ogg Vorbis) format. By utilizing Vorbis in that way, you can listen in via any device connected to the net (via ethernet, WiFi, 3G, 4G, whatever you have) that has any web browser by simply going to the URL for the feed and clicking the “play” button.

I also took a few hours and wrote an Android app that connects directly to the server and plays the audio over the phone/device. There’s a latency of about 5-20 seconds but I’m working on improving this. I also decided to use a free dynamic DNS solution from “Afraid”, so there’s no need for a static IP address from my ISP or the registration of a domain, both of which costs money. This was key in getting the app to work because the IP changes after the modem is rebooted… so I’d have to write a new app to reflect the changing IP addresses and redistribute them to the users. The DynDNS allows me to use a single URL that is “tracked” across any IP address that my ISP leases me.

I coded in a few key features like stopping/starting the feed, stopping the audio automatically when another notification/ringer goes off (ie, during phone calls) and then immediately restarting it once you’re done; among a few other things that work in the background.

I’m also in the process of writing some software for the server that allows me to log in and control the receiver from my console. This will allow me to modify the channels and trunked talkgroups, lock them out as needed, set priorities, add/delete frequencies, etc from any device that can use SSH (my phone, laptops, etc). I’ve got all of that connected up with a homebrewed RS-232->USB converter and M-M TRS connector feeding into the PC/IF interface on the receiver and back to the server via the USB converter. I’m writing the drivers from scratch right now but I really don’t know if or how it will work. I’m hopeful but I’m not going to stress over it.

So that’s about it, really. Essentially, I had ended up moving my servers about 40 feet to the other end of the house and dropped new lines into my closet and into the new rack/cabinet and then brought it all back online. Surprisingly, it only took an hour or so to complete the physical move and get the services back running to capacity. I’ve also now got the primary VOIP system online too, which I need in place to run to the “home phone” with a POTS number.

Later this month, I’ve got a microwave relay operating on 2.412GHz (in the amateur radio 13cm band) going into a hacked WRT-54G router that’s amplified up to 60 watts ERP from the directional transmitting antenna and then a separate omnidirectional antenna for receiving the downlink. All said, she has an effective range of about 35-40 miles outbound under most atmospheric conditions, which is truly incredible considering that your typical WiFi router only gets out (reliably) to 200 feet or so. Once I get all of it set up, it’ll be used for emergency backup VOIP comms, VHF voice pager alerts, VHF alphanumeric paging (512 baud POCSAG) and of course, a direct messaging system for sending text, data, files, etc to other users with active nodes. That system is essentially just digital email over the air; just without the Internet’s infrastructure being utilized to any degree. It is all digital, point-to-point communications using a “mesh” network of active nodes (microwave relays). I’ve got two of the servers up for all that, including Asterisk PBX for the phone system and patches. Currently, I just need to get the hardware connected and up in the air. My good friend Zack and I are hard at work with this one!

So yeah… that’s been my time recently in nutshell. I’m hoping to get an interview for 911 dispatching in the next couple weeks, since I passed the CritiCall exam with flying colors. I believe I should know something by next week but we’ll see. Aside from all that, I’m just using my time as wisely as possible… nerding it out! Well, that and spending time with my wonderful girlfriend.

With all of that out of the way, I have an idea to bounce around and an issue to share, that’s now solved.

The dev server was online all week and provided the feed with no problems to all the clients. It remained responsive to Pingdoms requests on port 8000, so I never received a notification that it had actually “gone down” (well, “froze”).

The dev machine only has 512mb of RAM and an old but reliable P4 CPU. For some reason, during the middle of the night, the processor and memory usage had both jumped to 100%; effectively bringing the system to it’s knees. It took a solid 4 minutes just to get an SSH connection open and to run htop to see what was going on. The problem lies in Icecast2, which had ate up all the memory on it’s own (going from 78mb of *total* system usage with everything running, to using the full 512mb and smashing the P4 with 100% usage for hours at a time). I’m going to dig into the logs later on to see what exactly happened and hopefully I can find the exact culprit and fix it, just so it won’t happen again.

I’ve moved everything over to the main production server now with 6GB of RAM and a single dual-core CPU @ 3.20GHz, after upgrading the OS (all of my servers are currently running Debian, minus one development VM server loaded with BSD) and a couple simple pieces of hardware. I really believe we should never have another problem regarding performance. This build should support at least 50 simultaneous connections with zero hesitation and much less latency on the client’s side, which I’m still going to tweak some configs for performance, anyway. So we’re good there!

So my next idea revolves around a feature for the Android app. I’m considering implementing a “record button” just under Play/Stop. This would allow clients to record the audio directly to their SD card. This way, if you start to hear some interesting traffic, you can record it with a simple click. I can probably get it to encode well enough that 1 minute of audio would be about 750kb, maybe less. It shouldn’t be too difficult to code in, I don’t think. Does anyone have any further suggestions or comments here?

I’ve also considered allowing the app to continuously record the last 3 minutes of audio but I’m worried about degradation of the SD card due to constant writing over the same blocks. But I figured this would allow it to function similar to a “dash cam” where it records a set amount of a data and once you need to save the last __ minutes/hours, you press a button and the file is saved and moved/renamed and the application continues to record in the background as usual but on different blocks. I’m not sure how I feel about this, yet.

Anyway, that’s everything in a nutshell. If anyone has any questions or comments, feel free to leave a comment or contact me directly. I’d be happy to discuss this with anyone that is interested. That said, I’m planning on writing a tutorial covering how to build your own audio feed server, just in case anyone wants to build their own.

Take it easy and until next time, I’ll see ya at the big one.

-560 clear.

Screenshot_2014-10-07-19-40-09

Leave a comment