Monitor Everything

Okay, is anyone tired of my server admin tips yet? Yes? Too bad.

Monitor everything… Put as much info at your finger tips as easily as possible. Put that info in a place where you will always be looking at it for some reason. For example, I made a vBulletin plug-in that monitors 4 memcached servers (including a latency test it runs) as well as 10 blade servers. This shows every time I’m in the admin of the forum (which is a lot), so I can’t help but to not see it.

I wrote a little daemon that runs on my servers that can quickly report back whatever info I want (time, disk RAID status, server load, MySQL replication status, etc.)

The more info you have in one place (especially when you run a bunch of servers), the easier it is to see anything wrong. For example, I had an issue with a web server serving requests slow one day… it ultimately ended up not being a problem with the web server, but the memcached server it was using. The latency test was showing ~2,000 ms latency (2 seconds) vs. the normal 0.5 ms (1/2000 of a second).

And be proactive about monitoring stuff… don’t wait until something bad happens to start doing it! Then it’s too late.

Can I go to bed now?

5 thoughts on “Monitor Everything”

  1. Very cool. Have you tried Fusion Charts? I’ve been using the standard charts for basic stats…but I have my eye on the real time capability of the instrumentation suite. Not really needed, but I like looking at pretty graphs…

    -Patrick

  2. Pretty cool ;).

    Here’s one of my stupid Qs 😀 ; Looking at the chart, I saw “Pointer 64”. Based on what I *think* I know, that means the almighty DP servers are running on 64-bit servers?
    Feel free to call me an idiot. heck! thats what i call myself :D.

  3. Don’t forget about the Sister Monitor that you have running…When you arn’t nerding out in front of your computer, guaranteed that the Sister Backup Monitor is. Granted, the sister backup can’t trouble shoot the problem beyond a phone call to say, “Hey dude, your website done be broke and shit…”. But hey, knowledge is half the battle! 🙂

  4. Os this where to ask?

    I share a DSL conection with neighbors and they love to use their “ares” musci downloader. You can imagine that each time they decide to open it i loose any bandwith avalaible and can not get connected to internet. No matter if I limit the bandwith in their software, anyway it practically “sucks” all the bandwith.

    The router is a 2wire 1070 in a 1024kb conection.

    Was getting into the firewall settings but can’t get how to do it to limit their computer and avoid them to suck all the bandwith…

    Any suggestion will be apreciatte.

    Thanks.

    G.

Leave a Reply

Your email address will not be published. Required fields are marked *