I broke scientific7.eskimo.com today.  It is in the process of being restored from backups.  Please use centos7.eskimo.com, which is based upon the same RHEL7 code base, in the interim.

     Reboots to make active security patches have been completed.  Sorry it took me somewhat longer than anticipated owing to I had a couple of machines that were stubborn and did not want to go down for reboot.

     I will be rebooting all of the Intel based machines between 1:30AM – 2:00AM November 7th to make active several security updates that patch serious issues with openssl, systemd, and DNS.

     Downtime for any given machine should not exceed about 15 minutes.  Most will be less.

      I did not expect last night to be a maintenance night but it had other ideas.  Around 3AM name service for our own hosts failed although our name servers were still resolving external hosts fine.

     The issue was caused by an update that replaced the named apparmor profile on the master name server.  We use a hidden master (a master which is not reachable from the outside world) for security reasons.  I use a file system layout for named that is different from the default Ubuntu layout.  I had to modify the apparmor profile accordingly.  When the update replaced my modified profile, it resulted in named not being able to read some of it’s configuration files and then failing to resolve local host names as a result.

     This has been corrected.

     Clock speed back to 4.3 Ghz, everything tested.  Sleep time now.

     Sorry, need to do one more reboot of the machine that serves home directories to fix a clock speed issue.

     All of the physical hosts have now been upgraded to Ubuntu 18.10.  There were a couple of issues, the upgrade replaced /etc/defaults/nis and turned the NIS servers off.  It took me several hours to find and correct that.

     Second problem was that some of the old kvm machine types, including the one that our web server used, were no longer supported on the i7-6850 platform.  Finally found that and changed the virtual machines to currently supported machine types which actually improve security.

     I did not expect to use the entire midnight – 6AM time frame but as it happened got everything back online at 5:50AM so almost went over.  If you encounter any problems please use Support->Tickets to generate a ticket.  Thank you.

     Since we recently upgraded the mail servers, it comes with a newer postfix that supports more rules aimed at stopping spam and forgery and I’ve added some of those.  Hopefully they will not break any legitimate mail, but if you do get reports of mail bounced, if you can get copies of the bounced messages to me with full headers, this will be very helpful in diagnosing any problems that may result.  Hopefully this will not happen, but if it does please know I need to have full headers to diagnose this kind of problem.

     Mint is restored to service with an additional 100GB Disk.

     I have to take Mint down for a short period to increase available disk space as it has grown to where there is no space left in the root partition.