I will be rebooting a number of servers to upgrade the kernels to a 4.18.0 kernel.  After the issues with NFS file locking caused severe operational problems with the client mail server today I decided to try the new 4.18.0 kernels on a few machines and it appears to be running significantly cleaner.  Several of the other Linux distributions we have available have already upgraded to 4.18.0 and we’ve experienced no NFS troubles on them.  Presently the web servers are now running 4.18.0.  Since that represents the heaviest load of all the servers and stresses the kernel the most, if it runs clean on the web servers it pretty much indicates it will likely run clean on everything else.

     Reboots of the mail server and a restoration from backup did not fix performance issues.

     While there was both a bot-net brute force password guessing attack in progress, which fail2ban found and locked out (over 1200 IPs involved) and an attempt to use the machines name server as an amplifier in a denial of service attack against a third party, (it won’t work because it is rate limited), neither of these represented a significant load or amount of traffic.

     I rebooted the NFS server that holds the mail spool and this fixed the issue.  I do not know what was wrong with it as I was able to access files on the spool directory prior to the boot but it had something to do with file locking as everything was waiting on a lock to clear which never did.

     We are having problems with our client mail server that provides imap, pop3 and smtp server.  Processes show waiting on disk but disk I/O is not busy.  It is not a hardware issue as far as I can tell because the machine is a virtual machine and other machines on the same hardware are fine.

     We are under going brute force password attacks and fail2ban is working to lock out the offending IPs but the botnet is large so there are many, over 1200 so far.

     To test the possibility that a recent software update broke something, I am restoring the machine image from a previous backup and will reboot into it shortly.

     OpenSuse.eskimo.com is going to go down for an upgrade to Leap 15.  The online upgrade is not working owing to some repositories not responding so going to have to do a fresh install and that’s a tedious process that’s going to take a while to get everything working again.

     Actually this is being delayed by an unspecified amount of time by the fact that the OpenSuse website is broken.

     In response to a customer request, imapfilter has been installed on all Linux based shell servers.

     Backups of all virtual private servers has been completed.

     I will be taking virtual private servers down for about 15-20 minutes each to image the machines (a form of backup that makes a copy of the machine state).

     If you are doing anything critical please e-mail nanook@eskimo.com right away.

     System reboots are completed.  NFS and NIS remounts and binds verified.  Everything is operational.  New microcode for i7-6850k still broken but I’ve gathered information Canonical needed so hopefully this got us closer to a fix.

     I will be rebooting servers early Saturday morning, shortly after midnight. The server which houses home directories will be rebooted possibly more than once.

     Most of the servers will be rebooted to load a new kernel.

     The server housing the home directories will also be testing a new microcode package from Ubuntu to see if it has corrected a problem with the 6850k multiplier register.  If the package performs properly there will be only one reboot of that machine, if not a second reboot to back out the new code will be required.

     The shell server fedora.eskimo.com has been upgraded to Fedora 28.  Gnome Flash back and KDE Plasma no longer exist with this distribution so are not available on X2Go, however Mate, LXDE, LXQT, and XFCE still work.