The eskimo.com shell server is down again.  This time it’s not entirely crashed but in a state where the kernel is getting CPU and it responds to pings but user space programs are not.  This is an old 4.1.4 SunOS bug that happens only on multiple CPU machines.

     I will be out of here for a while between post office run and a run down to the co-location facility.  If I don’t answer please leave a message and I will call you upon my return.

Centos7 is back up.

     CutyCapt is installed on all the shell servers except ‘eskimo.com’.  Note that on the Redhat based systems it is uppercase C’s, but on the Debian based machines, it is all lower case.

     Centos7 is down, well it is out of space on a partition preventing logins. I am working to resolve.

     The eskimo.com server crashed again so I’ve got to make another trip to the co-lo.

     The old SunOS 4.1.4 shell server, eskimo.com, is up.  It will be even slower than normal as I am writing backups.  That machine hasn’t been backed up since 2013 owing to the fact that any updates for SunOS have long since been discontinued so it is essentially static.  User data is stored on an entirely separate server that is backed up weekly.  None the less I like to back it up once in a while just to make sure there are no unreadable sectors on the disk and nothing has gotten badly corrupted.

     The scheduled maintenance has been completed.  All servers are back up except for ‘eskimo.com‘ shell server.  I am headed off to the co-location facility to address that.

     The old SunOS 4.1.4 eskimo.com shell server is down.  I started writing backups and it froze.  I’ve been up 22 hours and am not up to the drive to the co-location facility tonight so it is going to have to stay down until tomorrow afternoon.

     Debian also is taking a long time to write an image.  It may or may not be down all night.  The other machines, Mint, Fedora, OpenSuse, and Scientific7 likewise are going to be rescheduled for maintenance tomorrow, either during the day if nobody is logged in and active, or tomorrow evening if they get busy.

     Both shellx and scientific shell servers are completed and available for use.

     Maintenance is completed on the physical hosts iglulik and virtual, so there will not be any further global interruptions.

     Maintenance is completed on ftp, www, mail, and uucp, so there will not be any further interruption to those services.

     Maintenance is completed on Centos7 and Ubuntu, so those shell servers will not experience further interruption.

     Maintenance is still in progress on Scientific, Shellx, OpenSuse, Mint, Debian, and Fedora.  These servers will still have 20-25 minute downtimes to be imaged.

     There are multiple compression jobs in progress on both physical hosts, this will cause them to be somewhat slower than normal for the next 24 hours or so.

     Shellx had some problems that will require an additional boot to correct and then a downtime of approximately 25 minutes to image the fixed machine.