I did not expect last night to be a maintenance night but it had other ideas.  Around 3AM name service for our own hosts failed although our name servers were still resolving external hosts fine.

     The issue was caused by an update that replaced the named apparmor profile on the master name server.  We use a hidden master (a master which is not reachable from the outside world) for security reasons.  I use a file system layout for named that is different from the default Ubuntu layout.  I had to modify the apparmor profile accordingly.  When the update replaced my modified profile, it resulted in named not being able to read some of it’s configuration files and then failing to resolve local host names as a result.

     This has been corrected.

     Clock speed back to 4.3 Ghz, everything tested.  Sleep time now.

     Sorry, need to do one more reboot of the machine that serves home directories to fix a clock speed issue.

     All of the physical hosts have now been upgraded to Ubuntu 18.10.  There were a couple of issues, the upgrade replaced /etc/defaults/nis and turned the NIS servers off.  It took me several hours to find and correct that.

     Second problem was that some of the old kvm machine types, including the one that our web server used, were no longer supported on the i7-6850 platform.  Finally found that and changed the virtual machines to currently supported machine types which actually improve security.

     I did not expect to use the entire midnight – 6AM time frame but as it happened got everything back online at 5:50AM so almost went over.  If you encounter any problems please use Support->Tickets to generate a ticket.  Thank you.

     Since we recently upgraded the mail servers, it comes with a newer postfix that supports more rules aimed at stopping spam and forgery and I’ve added some of those.  Hopefully they will not break any legitimate mail, but if you do get reports of mail bounced, if you can get copies of the bounced messages to me with full headers, this will be very helpful in diagnosing any problems that may result.  Hopefully this will not happen, but if it does please know I need to have full headers to diagnose this kind of problem.

     Mint is restored to service with an additional 100GB Disk.

     I have to take Mint down for a short period to increase available disk space as it has grown to where there is no space left in the root partition.

     I’ve discovered a situation in which it is possible to forge the from address with some mail clients.

     The person wishing to forge the e-mail provides a fake “From: ” header but a real address in the envelope.  Because postfix only checks the envelope it does not prevent these kind of forgeries if the e-mail client displays the last From: line in the mail.

     I will be upgrading the physical host machines to Ubuntu 18.10 early Saturday morning.  The upgrade process is rather long and grueling and during at least part of it breaks NIS.  However, these machines are all NIS slaves and I will time the upgrades such that only one will be affected at any given time so all of the client machines should be okay.  However, all will go down several times during the upgrade process,  Normally Linux upgrades require only a single reboot however to work around bugs in the upgrade process regarding NIS, it will be necessary to de-install / re-install nis and reboot possibly more than once.  So there may be multiple outages between 12:01 – 6:00 AM on Saturday the 27th.  This upgrade is important as it fixed the issue of NFS not mounting reliably at least with all the machines with modern kernels.  It will not fix the old CentOS 6 machine that everyone loves however.

     After the upgrade caused NIS issues, I’ve discovered some users are locked out of access to web and ftp services here.  If you’re reading this you are probably okay but if you’ve got a weather station or something else automatically doing periodic logins to ftp, you may be blocked.  If this is the case please initiate a support ticket with your IP address or e-mail support with your IP address and indicate that you can not reach the web and/or ftp server and I will manually remove the block for you.