Today’s Outage

     I had to make some changes to our router configuration this afternoon.

     The way the software in our router works, you go into a configuration mode, make all the changes you want to make, then commit or save those changes at which point they become active.

     I did this, hit the SAVE button.  It said, “Save Failed” and then crashed.

     I had to drive down to the co-location facility and reconfigure the router to bring it back online.

     In order to do this I had to change the IP addresses on one of my machines to in order to communicate with the factory default address of the router and then to reconfigure it.

     That all went well and I had the router back online by 3pm, but when I went to change the IP address back on the machine I used to configure the router, the

Crash and Burn

Crash ‘N Burn, No Return

graphical tools failed and screwed up it’s configuration past the point where it could be fixed by the graphical client.

     I was not familiar with where Ubuntu keeps all of it’s net stuff, but found the files and got that machine restored to health as well.  Most things were online by 3PM mail service, web service, and some of the shell servers.  This particular box hosts a number of shell servers and so some were down until about 4:30pm when I got it completely restored.