The outage of virtual private servers was caused by a bug in the 6.0.12 kernel (the stability issues I mentioned earlier).  It has been responsible for locking up four different machines in the last two weeks and was scheduled to be upgrade this Friday, however, since I had already put the new kernel on the machine but not booted into it, the virtual private servers are now all upgraded.  Hopefully 6.0.15 will have resolved the issue but we will not know until some time passes.

      I had gotten confused about the day Christmas was on and had incorrectly announced kernel updates on Christmas Eve.  Actually they will be done on Friday evening the 23rd between 11pm-11:30pm providing the roads are still semi-passable.

     I will be upgrading to 6.0.15 from 6.0.12.

     Barring large amounts of snow on the roads, I will be doing a kernel upgrade this Christmas Eve starting at 11pm.  Not really the thing I want to be doing on Christmas Eve but the present kernel has shown some instability that lead to one machine locking up.

     This will impact all of Eskimo North’s paid and free services, including virtual private servers, shell servers, web hosting, e-mail, and our fediverse instances, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.

     With the exception of Yacy which always takes longer because it rebuilds the index at start-up, the rest of the services should be not down for more than about ten minutes unless a server fails to shut down properly.

     When I changed the mail server IP address, I neglected to change the firewall rules for imap and pop3.  This is fixed now.  If you have further issues please let me know.

     I am going to be making some more IP address changes tonight, these will mainly impact many of the shell servers and mail server briefly, although the IPs of these servers won’t be changed, the physical host they are on will which requires their reboots as the host is rebooted.

     This morning’s outage was caused by a kernel soft CPU lockup on the server that serves the home directories and also one virtual private server.  Because it is a physical host, not a virtual machine, I had to drive to the co-location to power cycle it.

     This is caused by a race condition in the kernel when two or more CPUs attempt to access a resource that has not implemented proper locking.

     I have made some changes to the system configuration that should result in an automatic panic and reboot should this occur again in the future.

     I will be rebooting various machines not to perform kernel upgrades but instead to make changes to IP addresses.  In theory rebooting shouldn’t be necessary but Leonard Poettering has so screwed up Linux with systemd that simply taking the network out of service and returning it to service with the new settings no longer works.

     The router software upgrade completed successfully.

     I changed the IP address of mail and changed it’s IP in DNS, however, I neglected to update /etc/hosts on all the machines and that overrides DNS.  This caused mail to file from shell servers as well as webmail.  This has been corrected.

     I am working on changing the IP addressing scheme for machines at Eskimo.  This will involve some reboots of NFS clients when IP server addresses change.  The mail server address will be changing from 204.122.16.222 to 204.122.16.14.  The name will remain “mail.eskimo.com”.  The old eskimo.com shell server will become “sunos.eskimo.com”.