Just a few hours later they came out with an update to ypbind and yp-tools that made them compatible with the newer libraries so Manjaro is again available.

     The morons at Manjaro have broken ypbind by making it and the libraries it depends upon incompatible versions.  I thought perhaps something just went wrong with updates so I restored from an old backup and then updated again and it brought it right back to an unusable state.

     So until the folks at Manjaro fix their repository to have a compatible version of ypbind and the libraries it depends upon, Manjaro will be unavailable.

     A Linux upgrade changed openssl in such a way that the new libraries were incompatible with the existing compilation of apache2.  I had to recompile apache2 to get it to listen to the SSL port again.  Sorry, Ubuntu does not bother to notify us when they make crushing changes to system libraries.

     This briefly affected all of Eskimo North’s SSL and HTTP/2 web services including webmail, friendica, hubzilla, and nextcloud.

     Kernel upgrades completed, Mail system and all shell servers and virtual private servers are available.  NFS and NIS connectivity has been verified on all machines.

     Tonight’s Kernel Upgrade will be brief, starting at 11PM and should be concluded by 11:15PM as the big server with home directories, Mint, Debian, Ubuntu, and the Web Server are all done because all got sick under the previous kernel and were upgraded Tuesday.

     So Web service and those three shell servers will not be interrupted.  Mail service, the other shell servers, and private virtual servers will be interrupted but it should be less than 15 minutes downtime in all.

     We have a new (old) payment option.  In addition to mail-in payments and credit and debit cards, we can now process electronic checks.

     For those who would like to learn / experiment with the bleeding edge features of C and C++, gcc 11.2 is now available on the Ubuntu shell server.  To use it make sure that /usr/local/bin is first in your path before /usr/bin.

     Because Ubuntu just locked up and because this time I got an indication it was an issue with the hypervisor and physical host, I am going to be upgrading the following machine kernels early this evening around 11:30PM:

     Iglulik (has NFS /home and /misc directories), ftp, debian, ubuntu, and mint.  These will all be down briefly, the web server longer than the others.

     Web services including all our virtual hosts, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and all webmail services as well as our own website, https://www.eskikmo.com/ will be affected.

     We’ll be doing kernel upgrades of all machines this coming Friday starting at 11pm.  Interruption to most services should be relatively brief, however, web based services may take approximately 1/2 hour to restore.  This is because I’ll be reverting the physical host to a 5.13.19 kernel (presently the only host on 5.14) as 5.14 did not fix the ethernet issue AND has proven to be less than stable.  Because a different driver for the flash card is required for 5.14 to 5.13 I can not use the dkms mechanism to compile this driver and because to compile it fresh the machine needs to be running the kernel upon which the driver is running, I will need to first reboot the physical host to 5.13, (it will be the 5.13.19 EOL kernel) and then recompile the drivers, then bring up the web server.  From previous experience this process requires around 1/2 hour.

     Web service includes all our virtual hosts, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and all webmail services as well as our own website, https://www.eskikmo.com/.

     I plan on doing some kernel upgrades over the evening but probably not the physical hosts or the lesser used virtual machines.  There is some instabilities in the existing kernel that is causing CPU lock-ups.  Mostly these are transient and resolved by watchdog timers but one did hang our web server the other day.

     I do not have access to parking at the co-location facility this weekend because they are re-stripping the parking lot so I’ll save updates of physical hosts for next weekend.

     The reboots of the virtual machines only take a few seconds so interruptions will be brief.  I’ve already done the web server so it will not be impacted, mainly shell and mail servers.

     Also I plan to take some of the private virtual servers down for between 15-60 minutes depending upon their size to image them (a form of backup).

     Apache 2.4.49 was released today and had many important security fixes and also some fixes to help it gracefully shutdown which makes it do so much faster.  This will result in a much shorter interruption during log rotates at midnight.

     We have installed 2.4.49.  Pingdom tests reveal essentially unchanged performance loading our WordPress site in 279ms.