OpenSuse is back up and running on a different host.  Available for use.

     Debian is moved to a stable host and available for use.

     Centos7 has been moved off of the unstable host and is available for use.

     Fedora is moved to another host and back up and running and available for use.

     One host machine has become unstable. This machine hosts Debian, Fedora, OpenSuse, and Scientific7.

     I recently upgraded this machines hardware from an i7-2600 to an i7-6700k processor, and also recently upgraded from Ubuntu 15.04 to Ubuntu 15.10.

     Either the software or hardware upgrade may have introduced this instability.  However, it did not manifest until the operating system upgrade which also was interrupted so some things may not have installed properly.

     I am going to move all the guests to another machine, bring this machine back and re-install Ubuntu 15.10 from a fresh install, and then stress test it to make sure it is stable or repair any hardware problems that may show up during stress testing.

     There will be down time for each of these guest machines of about an hour to move them.  And I can’t get to this until later in the day as I have a doctors appointment to follow-up on a minor surgery and infection this morning.

     If you need a modern Redhat machine, I suggest using Centos7 in the meantime,  I would suggest Ubuntu or Mint as alternatives to Debian in the meantime.

     I upgraded the Apache web server to version 2.4.17.  This is mostly a bugs and security release.  Hopefully it fixes more bugs than it introduces.  I also took the opportunity to update the apr and apr-utils libraries since it’s been a while.

      If you get a message like this, DO NOT reply with your username and password, it is a phishing scam attempting to steal your account.

     Instead, bounce the message to spamtrap@eskimo.com.

From: Help Desk
Subject: Scheduled Maintenance & Upgrade

Your account is in the process of being upgraded to a newest
Windows-based servers and an enhanced online email interface inline with
internet infrastructure Maintenance. The new servers will provide better
anti-spam and anti-virus functions, along with IMAP Support for mobile devices
to enhance your usage.

To ensure that your account is not disrupted but active during and after this
upgrade, you are required to kindly confirm your account by stating the
details below:

* Domain\user name:
* Password:

     There was a problem with mail that was related to the upgrade and the issue with NFSv4 not recovering properly.  The client mail server, mail.eskimo.com, did not properly remount users home directories after the file server was brought back online.  This was corrected around 7:30AM Pacific Time.

     One server took over half an hour to boot, the second 45 minutes later still didn’t come up so I drove down to the co-lo facility and booted it.  It was running but just hadn’t completed the start-up.

     I’ve discovered several problems.  The new ntp daemon that “fixed” the security problems appears broken, at least systemd tries to start it unsuccessfully half a dozen times before it takes.  Each time involving a somewhat lengthy timeout.  I am guessing this may have to do with the reachability of servers but I really don’t know at this point.

     There is either a problem with my NIS configuration or ypbind doesn’t work correctly.  The behavior is not well documented in the manual pages.  It tells you what valid entries are but not what order you use them in.

     I have multiple servers configured as:

domain eskimo.com

ypserver (ip address)

ypserver (ip address)

     But rather than try each in order, it seems to try one and if that fails it gives up.  There seems to be no effective way to specify multiple servers except for broadcast and that has serious security issues.

     Then NFSv4, when an NFS server goes away and then comes back, NFS should recover automatically but it does only about 75% of the time.  In one instance, I had to restart the nfsd service on the server to get things to mount again.

     All these issues combined made for a lot of hair pulling.

     The machine with the mail spool took a long time to boot but did come back up and is now up and running with the new release which is Ubuntu 15.10.

     The machine that houses /home directories is still in progress.  I had to interrupt and re-start the upgrade which can end up with a lot of dpkg –configure -a incarnations before everything is finally current.  Not good to interrupt but had no choice as it hung during the os-prober.