File Server Outage

[Nanook <nanook@eskimo.com>]

     This morning around 8:15AM, Eric attempted to reboot our main file server
to resolve a problem with networked file system communications.

     The machine did not reboot cleanly and instead hung on a file system error
that required manual intervention.

     Of coarse this happened during rush hour making the trip to the
co-location facility a slow painful one.  Four hours sleep didn't help.

     After I arrived the keyboard I was using died during the manual file
system check.  This required replacing the keyboard, rebooting, and starting
over.

     One of the file systems that had an error was a large one with 800,000+
files on it and the error in question was a duplicate block error which
requires additional phases to fix.  This would have been a long file system
check even without having to start over with a replacement keyboard.

     The errors are fixed, no data was lost, and everything is back to normal
now except that when I went to remove the monitor the cable broke off so I've
got a monitor to replace as well but that is not service impacting.

     Everything was restored to service at 10:31AM Pacific.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
 Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
   Knowledgable human assistance, not telephone trees or script readers.
 See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.