[RDD] More dead air

James Harrison james at talkunafraid.co.uk
Mon May 21 20:28:37 EDT 2012


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

+1 for Nagios here - try http://status.insanityradio.com/ (username
'guest') if you want to see what we've got set up (nothing specific to
Rivendell in there yet).

We also run http://www.vanheusden.com/java/CoffeeSaint/ on a PC in the
main office so if anything goes wrong, it's typically a "Hi, monitor is
showing X broken" call rather than "No audio" which helps massively.
And of course it'll email you on faults anyway, and there's various
Android/iPhone clients. http://status.insanityradio.com/dash/nagios.php
lives on one of my displays here just to assure me everything is okay
;-)

Nagios also has the advantage of loads of plugins - you can get
performance data monitoring stuff which will draw you graphs of disk
usage etc as measured, which are always handy for spotting trends and
predictors for failure.

Plus it's trivial to extend -
http://status.insanityradio.com/nagios/cgi-bin/extinfo.cgi?type=2&host=katie&service=DSPXmini-FM+SE+Processor
for instance is our audio processor for our FM broadcast, which
monitors all the various levels and ability to connect (we wrote the
plugin to interface with the proprietary UDP interface from scratch).
We also have this in Graphite, which is great for graphing metrics -
try http://bit.ly/KYNytl for a graph of our processor's levels over the
day. Great for spotting shows that over/undermodulate. Likewise we can
check our RDS encoder over the serial-telnet bridge we use to talk to
it -
http://status.insanityradio.com/nagios/cgi-bin/extinfo.cgi?type=2&host=katie&service=RDS+Encoder
for example.

I'm going to be setting up a new Nagios installation for a temporary
event setup of Rivendell; I'll share any Rivendell-specific plugins I
whip up in the course of doing so.

Back slightly more on-topic - is VNC required to be to that machine? If
you have another machine couldn't you VNC into that to rdlogedit, or
are you remotely running shows using rdairplay and VNC? The VNC to
rdlogedit with autorefresh on the rdairplay logs/service approach has
the advantage that if it is VNC, you don't break the playout machine...

Cheers,
James Harrison

On 22 May 2012 00:37:31, Christopher Cmolik (RIT Student) wrote:
> You may want to set up something like Nagios or Zabbix (much easier
> IMHO) to monitor this kind of stuff, stick it on a nice web UI, and
> send you pages when mem usage gets too high.
>
> Some of our "CATASTROPHIC FAILURE" scenarios include:
>  * RDAirplay not running on the "Playout" boxes
>  * Apache, darkice, or icecast not running on our streaming server
>  * MySQL not running
>  * MySQL not taking connections
>  * MySQL taking up too much memory
>  * Disk space too low on our audio share
>
> On Mon, May 21, 2012 at 3:16 PM, Rob Landry <41001140 at interpring.com
> <mailto:41001140 at interpring.com>> wrote:
>
>
>     Once again I got a "we're off the air" call, couldn't VNC into the
>     box,
>     and connected via ssh to find Xorg eating 80% of the system memory
>     and the
>     load displayed by "w" was up over 3.
>
>     There are two Rivendell machines I shall call Tweedledum and
>     Tweedledee;
>     they are identical in every way: same hardware, same software, same
>     schedule. Tweedledum is the one that has had the problem both times;
>     Tweedledee just keeps on going like the rabbit in the old Ennergizer
>     commercials.
>
>     I am going to guess that x11vnc is responsible; I never had the
>     problem
>     when I was running vino-server. Tweedledum gets a lot more VNC
>     connections
>     than Tweedledee, and sometimes the screen will freeze during a VNC
>     session. Could x11vnc be buggy on Debian Squeeze?
>
>     I've killed x11vnc for now and gone back to vino-server. I also
>     set up a
>     cron job to notify me if Xorg eats more than 5% of memory; maybe I can
>     catch this thing in its early stages.
>
>
>     Rob
>
>     _______________________________________________
>     Rivendell-dev mailing list
>     Rivendell-dev at lists.rivendellaudio.org
>     <mailto:Rivendell-dev at lists.rivendellaudio.org>
>     http://lists.rivendellaudio.org/mailman/listinfo/rivendell-dev
>
>
>
>
> --
>
> Chris Cmolik
> RIT Information Technology '13
> <clc5688 at rit.edu <mailto:clc5688 at rit.edu>>
>
>
>
>
> _______________________________________________
> Rivendell-dev mailing list
> Rivendell-dev at lists.rivendellaudio.org
> http://lists.rivendellaudio.org/mailman/listinfo/rivendell-dev
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+63bUACgkQ22kkGnnJQAwBbgCgrrz0u0+9XjbPN7y4rY4Y+F2O
Z20AoIZd++jDTUkHxTKfF6fDvw9BTRvX
=fX+5
-----END PGP SIGNATURE-----



More information about the Rivendell-dev mailing list