When I changed the mail server IP address, I neglected to change the firewall rules for imap and pop3.  This is fixed now.  If you have further issues please let me know.

IP address changes continued

     I am going to be making some more IP address changes tonight, these will mainly impact many of the shell servers and mail server briefly, although the IPs of these servers won’t be changed, the physical host they are on will which requires their reboots as the host is rebooted.

This Morning’s Outage

     This morning’s outage was caused by a kernel soft CPU lockup on the server that serves the home directories and also one virtual private server.  Because it is a physical host, not a virtual machine, I had to drive to the co-location to power cycle it.

     This is caused by a race condition in the kernel when two or more CPUs attempt to access a resource that has not implemented proper locking.

     I have made some changes to the system configuration that should result in an automatic panic and reboot should this occur again in the future.

Reboots 11PM-4AM Tonight

     I will be rebooting various machines not to perform kernel upgrades but instead to make changes to IP addresses.  In theory rebooting shouldn’t be necessary but Leonard Poettering has so screwed up Linux with systemd that simply taking the network out of service and returning it to service with the new settings no longer works.


     I changed the IP address of mail and changed it’s IP in DNS, however, I neglected to update /etc/hosts on all the machines and that overrides DNS.  This caused mail to file from shell servers as well as webmail.  This has been corrected.

IP Address Changes

     I am working on changing the IP addressing scheme for machines at Eskimo.  This will involve some reboots of NFS clients when IP server addresses change.  The mail server address will be changing from to  The name will remain “”.  The old shell server will become “”.

WordPress Eskimo Home Page

An automatic update of WordPress broke the way it is displaying our home page, in particular I had a lot of inline CSS that it is for some reason not properly grocking now. So the home page will be a bit ugly for a day or two, my apologies.

Kernel Upgrade Status and Manjaro

     The issues that remained with Centos7 and Scientific7 with the existing 6.0.x kernels were not expected to be fixed until 6.2 according to the developers, however, 6.0.12 appears to have fixed them as both machines booted cleanly after the upgrade last night.

     Also, I adjusted the clock on ice from 4.3 Ghz to 4.2 Ghz as it was ever so slightly unstable at 4.3 Ghz.

     Lastly, Manjaro is restored, but your existing ssh keys will not work as this is a fresh install so you will need to delete and create / propagate new keys.  There is much software not yet installed, so if something you need is missing please send a note to and I will prioritize.