Host Problems

[Robert Dinse <nanook@eskimo.com>]

     Problems this afternoon starting at about noon; were all related to human
error.

     Aaron was ill today; Catherine came in to cover for him.  I managed to get
about an hour sleep between 9am and 10am this morning so I really was not and
am not in a state to cover.

     She saw "tombstones" on the perfmeter for eskimo, this is an indication
that rcp.statd isn't responding, may be a dead machine, but may also be a load
spike.

     She rebooted eskimo; eskimo failed to come up because it couldn't mount
file systems from mx1.  She rebooted again, this time without a graceful halt.
It resulted in file system corruption which had to be manually fixed.

     After that was fixed; eskimo would still not mount files from mx1 which
has the mail spool; and it happens that that is the first nfs mounted file
system it tries to mount so it basically stopped all the others.

     Web servers and other machines are dependent upon eskimo for web files;
mail is dependent upon mx1 to access the spool.  So this really broke a lot of
things.

     Aaron had been working on trying to get knfsd operational on mx1.  We
need it because it supports file system locking and the regular nfsd doesn't
and this causes problems with some mailers.

     He wasn't successful but when he put the old daemons back, he put some
non-functional daemons back instead of the ones that were working.  The
functional daemons appear to have been deleted.

     I had to restore these from tape which is what took so long to get that
machine operational again.

     Everything appears to be back to normal, Eric is here and I'm going to
attempt to get some sleep.