The machine with the mail spool completed the upgrade up to the point of reboot, but it doesn’t appear to be coming back up so I may have to go to the co-lo and physically boot it.  It might be stuck in the boot process, may not have shut down completely, hard to say at this time.

     The machine with the home directories got stuck at a point in the upgrade process and I may be forced to interrupt it and finish by hand (which is a painful process).

     I am performing release upgrades on the file servers hosting /home and /mail partitions.  This will require a reboot of pretty much everything at the completion.

     I am doing this because in the recent event where everything Intel went down, the machines with 15.10 recovered on their own, those with 15.04 required manual intervention.  So I am upgrading the remaining machines on Ubuntu 15.04 to 15.10.

     Yesterday a customer had a problem with web mail that was reproducible on browsers other than Firefox.

     I found that the version of SquirrelMail we had was not compatible with the version of PHP and upgraded it to the latest and greatest.  This resolved the issue for this customer.

     If you notice anything else strange with web mail please let me know.

     Fedora is back online.  What got hurt was the Ethernet hardware address. Not really sure how that happened, but one digit changed and no longer matched the conf file so it wouldn’t bring up the network as it couldn’t find the device.

     Ubuntu is back up.

     Fedora will take a little longer as it involves copying a 150GB disk image.

     Fedora is trashed, something is broken with the virtual disk image.  I am restoring from backups.  This will take some time, perhaps an hour.

     Ubuntu did not mount all the NFS partitions correctly, in particular home.  I am working on correcting that.

     Everything else appears to be operational.

     Tonight, around 7:30 our router and most of our servers crashed.  The router rebooted, only one of the Intel servers came back up, the Sparc servers all survived the event.

     I suspect a power hit, but I tried to call the people at Isomedia tech support and only got voice mail with a full mailbox so I was not able to find out what happened tonight.

     I rebooted and brought all the servers back online.  I did discover some configuration errors with nsswitch.conf on one server and fstab on another that prevented them from fully coming back on their own and corrected those.

     The outage today was the result of IsoFusion (formerly IsoMedia) working on the flaky jack in our cabinet.  They moved it and it went out altogether.  Turned out there was a bad jack AND a bad cable.  Both have been replaced and the connection is now solid.

     The Ubuntu update is still in progress.

     It is taking longer than it should have first because it wanted more memory than it had, and second because in my attempt to fix that I actually read the manual for virsh, and it said that the default unit was kilobytes and not thinking I thought ah 4096kb = 4gb, actually 4096mb = 4gb so I think you can probably deduce what happened.

     So I exploded it trying to make modern Linux run in 4MB, (ain’t gonna happen) and had to run dpkg –configure -a, and that blew up because of some missing dependencies, which have been resolved so it’s running now but going to take a while.

     A reboot will still be necessary when it is all done and in the meantime there are probably some things not working.

     I am taking debian.eskimo.com down for about an hour to move it from one host to another less occupied host.  The only person currently logged in has been idle for 25 hours.  If you need a Debian based host in the meantime, Mint and Ubuntu are presently available.