4.9 Kernel on Mail Server

     I am trialing a 4.9 Linux kernel on our mail server.  The 4.8 kernel was supposed to have some improvements in NFS but broke something critical to IMAP and Pine so we reverted to the old 2.6 kernel.  The 4.9 kernel is purported to have fixed some NFS issues in 4.8 and also has numerous improvements for the Skylake processor which happens to be the CPUs we are using in our servers.  It purports to address issues with the graphics driver in Skylake and in the reset code (which has been a problem, to reboot the physical servers we have to physically power them down and back up with the current 4.8 kernels).

     If you experience an operation failed in an attempt to fetch mail or any strangeness in pine please notify support right away, e-mail support@eskimo.com or call 206-812-0051, if you don’t get me live leave a message.

 

Web Server Restored to Service

     The web server was restored to service at approximately 2:45 AM.

     Now an operating system upgrade is in progress.  It will function most of the time during this upgrade but there may be some brief interruptions as it unpacks and configures new components.

 

Web Server Maintenance

     I am taking the web server down for about twenty minutes to image the machine prior to an upgrade from Ubuntu 16.04.1 LTS to Ubuntu 16.10.  I need to back it up in it’s current state just in case anything goes irreparably wrong with the upgrade (it’s been known to happen).

Botnet Attack

     We had a botnet attack against our web server.  This attack was attempting to enumerate user ID’s on WordPress sites hosted her.  It generated more than 512 simultaneous requests significantly slowing the web server and overloading the MySQL database.

     I have put filters in place blocking the IP space in question and normal service has been restored for everyone but the attackers.

Mail Server Reboot Completed

     The mail spool directory server has been rebooted back to a 4.4 kernel.  The more aggressive caching on the 4.8 kernel appears to have not played well with pine and imap so I’ve put it back to a 4.4 kernel for now.

Tonight’s Server Reboot

     Tonight’s server reboot will also take mint, debian, and scientific out of service briefly.  This should be a relatively brief outage of these shell servers and the mail subsystem lasting approximately ten minutes.  Also, there will still be DNS servers and incoming mail servers available so mail will be queued during this interval, it’s delivery will only be delayed.

     OpenSuse is still broken, I have not been able to determine yet why it will not authenticate NIS accounts.

     The other shell servers will remain in service.

Maintenance Outage Mail 11pm Tonight

     Somewhere around 11pm, could be slightly later I can’t say for sure, there will be a brief interruption of mail services as I reboot the server holding the mail spool back into a 4.4 Linux kernel.

     The 4.8 kernel that was supposed to improve efficiency (and did, the web server is measurably faster) did break something with respect to pine and imap.  Both of these get unexpected change to the spool file errors.  I do not know if mandatory locks are not working or exactly what the problem is just yet so for now backing out the kernel upgrade on the machine that holds the mail spool.

Web Server Attack

     Someone has been attacking our web server all day using a botnet, trying to find user credentials by bashing guesses using xmlrpc authentication attempts.

     If you are running a WordPress site and do not need pingback or other xmlrpc capabilities, I strongly recommend installing a plugin that disables xmlrpc.  You can find one by typing that into the new plugin search box in WordPress.

     I also strongly suggest installing wp-fail2ban if you have a WordPress site.  This will cause your site to log these attempts.  We have software on our server that will then use this to disable connections from the IP addresses probing for passwords.

Kernel Upgrades Completed

     In addition to the host machines, I updated a number of guests also to the 4.8 kernel.  The main incentive for doing this was improvements in NFS handling.  In the case of the older shellx and scientific servers, they were running on a super ancient 2.6 kernel because none of the 3.x kernels would boot on those machines but the 4.8.12 kernel seems to boot and run fine.