Server Failed

     After updates, our server hosting the /home directories failed to boot.  It was a combination of updates gone bad and operator error.  The operator error involved a typo in the /etc/fstab file and then when it would not remount the root partition / read-write, even though it was clean on an fsck and I could mount it manually, I assumed it was an issue with systemd (and there have been many).  So in desperation I switched to upstart, but it deleted a large number of necessary software packages in the process and I ended up with a machine I could not even boot into single user mode.

     So I brought the machine back home, made myself a server rescue disk, booted from that, got things cleaned up enough to boot, spent another eight hours or so installing missing software and finally got it working again.

     Total downtime was from about 12:30AM – 4:15PM on November 20th.