News / Other Servers...

[Robert Dinse <nanook@eskimo.com>]

     Earlier tonight I had to take the news server down because the
/var/spool/news file system got corrupted and the system decided it was out of
space even though it was only at 78%. 

     We had a windstorm here with lots of power glitches and I suspect that's
what initially messed up the file system.

     It was on the third pass of fsck because the damage couldn't be repaired
in one pass.  The first pass was the automatic fsck, found an inconsistancy
that needed a manual run.  So I ran it manually, which found duplicate blocks
and then restarted.

     At 22:50 PST (10:50PM) we lost power for about 15 seconds and all the
machines rebooted.  So in addition to having to spend a lot of time bringing
all of the machines up (which is a real pain because of cross-dependencies
between the machines), I also had to restart the fsck of eskinews. 

     One other issue, the SSL server on commerce requires a pass phrase to
start.  I thought I had the correct pass phrase but I do not, Aaron does as he
set it up but I am unable to reach him by phone so it is down right now for
that reason. 

     We also had some problems putting new users in tonight and Catherine left
without giving me the details, as a result we will need to address those
problems tomorrow.