Because 2.4.52 version of Apache seems to have a bug that causes it to occasionally stall, and because earlier versions have a root exploit that makes returning to them unfeasible, I wrote an automatic recovery script to recover the server in the event of a failure.

     What it does is automatically test the web server for a proper response every five minutes.  If it fails to respond, it attempts to restart apache, waits a minute to give it time to spool up, then retests.

     If the second test is good it logs the failure so I can keep track of how often it is failing.  If the second test fails, it turns my speaker volume up to 100%, and I have 100-watts per channel so that’s 200 watts worth of noise and then sends the Star Trek Next Generation Red Alert sound effect to the speakers.  I should hear that pretty much no matter what if I’m here, and also sends an e-mail so if I’m out I will be notified via my tablet or laptop.

     Hopefully this will prevent long outages like that which occurred last night when Apache decided to silently hang.