The mail spool directory server has been rebooted back to a 4.4 kernel.  The more aggressive caching on the 4.8 kernel appears to have not played well with pine and imap so I’ve put it back to a 4.4 kernel for now.

     Tonight’s server reboot will also take mint, debian, and scientific out of service briefly.  This should be a relatively brief outage of these shell servers and the mail subsystem lasting approximately ten minutes.  Also, there will still be DNS servers and incoming mail servers available so mail will be queued during this interval, it’s delivery will only be delayed.

     OpenSuse is still broken, I have not been able to determine yet why it will not authenticate NIS accounts.

     The other shell servers will remain in service.

     Somewhere around 11pm, could be slightly later I can’t say for sure, there will be a brief interruption of mail services as I reboot the server holding the mail spool back into a 4.4 Linux kernel.

     The 4.8 kernel that was supposed to improve efficiency (and did, the web server is measurably faster) did break something with respect to pine and imap.  Both of these get unexpected change to the spool file errors.  I do not know if mandatory locks are not working or exactly what the problem is just yet so for now backing out the kernel upgrade on the machine that holds the mail spool.

     Someone has been attacking our web server all day using a botnet, trying to find user credentials by bashing guesses using xmlrpc authentication attempts.

     If you are running a WordPress site and do not need pingback or other xmlrpc capabilities, I strongly recommend installing a plugin that disables xmlrpc.  You can find one by typing that into the new plugin search box in WordPress.

     I also strongly suggest installing wp-fail2ban if you have a WordPress site.  This will cause your site to log these attempts.  We have software on our server that will then use this to disable connections from the IP addresses probing for passwords.

     In addition to the host machines, I updated a number of guests also to the 4.8 kernel.  The main incentive for doing this was improvements in NFS handling.  In the case of the older shellx and scientific servers, they were running on a super ancient 2.6 kernel because none of the 3.x kernels would boot on those machines but the 4.8.12 kernel seems to boot and run fine.

     PHP has been upgraded on shellx.eskimo.com from php5.3 to php7.0 so that it is the same major point release as used on our web server.

     I encourage people to use ubuntu.eskimo.com for web development as it has the same exact version of everything as our web server has however many people continue to use shellx so I am endeavoring where possible to upgrade the tools on this server to match.

     I am planning on doing some maintenance at the co-lo facility tonight that will involve rebooting three of the host servers.  These are the machines that have your /home directories and /var/spool/mail mail spools, as well as various virtual machines.

     The last time I did this we were down for several hours.  I have come to a good understanding of what caused the issues last time so they can be avoided this time.  I expect a downtime of about 1/2 hour for the server that hosts /home directories and about 10-15 minutes for the others.

     The reason for these reboots is to load a 4.8 Linux kernel.  There was substantial work done to NFS in version 4.8 that improves performance by correcting a few sections of slow critical code and by using more aggressive caching.  Since our whole service is heavily dependent upon NFS to mount file systems remotely from one machine to another, this should improve the overall performance of our network, Mail, Web, shell serves, all should run faster after this reboot.