We continued to experience database problems. I found that the upgrade had erased a change I made to the systemd script to increase the system limit on file descriptors. MySQL was running out of descriptors. I will probably update again and re-apply my fixes to the script that it will no doubt erase again but I am going to let it run with this fix for a while to make sure it is table first.
At 12:49 AM the client mail server which provides SMTP, IMAP, and POP3 for clients and web mail, crashed.
This is a virtual machine sitting on a physical host. The physical host was still running clean. This is the first time this particular machine has crashed since it was loaded in 2013, the software is stable. However, CentOS 6 did release a kernel upgrade in spite of the age of the software with absolutely no notes as to what they changed / fixed. I have applied this upgrade.
MySQL has also been somewhat unstable on the web server since an upgrade several days ago. I have downgraded MySQL to 5.7.15 which is the version prior to the recent upgrade which introduced instability.
Replacing the 100-base-T switch with a Gigabit switch went smoothly. We are now back to full speed.
There will be a hopefully brief maintenance outage during this time frame as the 100mbit/switch that was temporarily put in service when the gigabit switch failed Monday morning is being replaced with a new gigabit switch.
If all goes well this should only take a few seconds, however, the last time we did a couple of the machines failed to recognize the speed change without a reboot. Hopefully this will not be necessary this time.
The outage this morning from 1:20AM until 5:10AM was caused by a failed switch. The gigabit switch has temporarily been replaced with a 100mbit switch that I had on hand. Some services relating to mail took somewhat longer to restore owing to the machine not recognizing the speed switch without a reboot and then the reboot loaded a new kernel that apparently lacked NFS support.
An update replaced PHP 7.0 with PHP 7.1 which Squirrelmail does not work with yet. I have reverted the PHP version to 7.0 to restore Squirrelmail to service.