Maintenance on Centos-Stream and MxLinux is complete.  These are back in service.

     Taking the Manjaro shell server down for 1/2 hour approximately for imaging.

     Taking the MxLinux shell server down for about 1/2 hour for maintenance.

     Going to be taking the CentosStream shell server down for about 1/2 hour to image it.

     Kernel upgrade for virtual private servers 1-8 and the host machine, “ice”, have been completed.  All are now running 5.10.33.  Hopefully stability will be better.

     We have been having some stability issues on the machine that serves private virtual servers vps1-vps8 since kernel 5.8 where there were some major changes that significantly improved load and responsiveness and hence the unwillingness to revert to 5.4.  I have filed a bug report on various issues that have shown up in crash dumps with the kernel Bugzilla but so far they have not directly addressed them.

     This morning, this server spontaneously rebooted, it did not print a crash dump, only a watchdog timer timeout.  But since last Friday the kernel team has kicked out a new release that addresses an issue with memory allocation that under certain circumstances could cause already allocated memory to be re-allocated and they’ve made many fixes to KVM code which is the hypervisor we are using to serve private virtual servers so could be relevant.

     I am going to do a kernel upgrade on just this server at 11PM tonight.  This will cause a brief interruption of all virtual private servers lasting 5-10 minutes.  Since no NFS/NIS mounts are involved, I do not expect it to take as long as normal updates do.

     Maintenance is completed and everything is back up and running.  We had some issues with both the mail server and the web server not automatically mounting all disk file systems and swap partitions so I had to go around to each machine and check these things.

     https://friendica.eskimo.com/, https://hubzilla.eskimo.com, and https://nextcloud.eskimo.com/ are all operational.

     E-mail and all web hosting is operational.

     Shell servers are all operational.

     We will be performing another round of kernel updates on April 22, 2021 starting at 11pm.  Also at this time we’ll be taking mail down for some additional time in order to increase the amount of RAM.  This should only take a few additional minutes.  The logs are indicating some memory pressure during peak times, not enough to result in swap usage but enough to reclaim buffers early which hurts performance.

     All of our hosting, shell, and email servers will be affected between 11PM and approximately 11:30PM though the individual downtime for any one of them should be ten minutes or less.

     Also affected are our fediverse sites https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/.

     I made the announcement some time ago that I was going to move nextcloud.  This is because in it’s old location it’s .htaccess conflicted with WordPress .htaccess when it came to well-known web addresses.  To resolve this, I gave NextCloud it’s own hostname.

     The OLD URL was: https://www.eskimo.com/nextcloud/

     The NEW URL is: https://nextcloud.eskimo.com/

     Please update your devices and any links to reflect the NEW URL.  Some devices may not properly follow the redirect.

     I’ve also updated to version 20.0.9 which is the current stable release.  I am also installing and configuring a bunch of new applications including connectors to other fediverse social media.

     Kernel upgrades completed without any major incidents at 11:38 PM.  Everything was mostly up by 11:15 PM, the remaining time was just checking NFS mounts and such.

     https://friendica.eskimo.com/ and https://hubzilla.eskimo.com/ are available.