My apologies to those who called and tried to let me know about the web outage.  My Comcast telephone service was apparently also out because I was home and in the office at the time and the phone did not ring.  I did not get notifications of your recorded messages until today.  I’ve tested the phones today, they are working again.  When it rains…

     Sorry for the downtime in web service today.  It appears suexec which is used to set the web server permissions for a given users files lost it’s setuid attribute yesterday.  Not sure how but the update time on the file coincided with when I did system updates and the Apache Makefile for 2.4.49 now installs it without this necessary bit set so I suspect it relates to an update and a rebuild that was necessary to accommodate library changes.  At any rate, I rebuilt it just to be sure someone hadn’t modified it hoping I would then set the bit again and I set the immutable bit which should prevent Ubuntu’s package manage from messing with it again.

     Just a few hours later they came out with an update to ypbind and yp-tools that made them compatible with the newer libraries so Manjaro is again available.

     The morons at Manjaro have broken ypbind by making it and the libraries it depends upon incompatible versions.  I thought perhaps something just went wrong with updates so I restored from an old backup and then updated again and it brought it right back to an unusable state.

     So until the folks at Manjaro fix their repository to have a compatible version of ypbind and the libraries it depends upon, Manjaro will be unavailable.

     A Linux upgrade changed openssl in such a way that the new libraries were incompatible with the existing compilation of apache2.  I had to recompile apache2 to get it to listen to the SSL port again.  Sorry, Ubuntu does not bother to notify us when they make crushing changes to system libraries.

     This briefly affected all of Eskimo North’s SSL and HTTP/2 web services including webmail, friendica, hubzilla, and nextcloud.

     Kernel upgrades completed, Mail system and all shell servers and virtual private servers are available.  NFS and NIS connectivity has been verified on all machines.

     Tonight’s Kernel Upgrade will be brief, starting at 11PM and should be concluded by 11:15PM as the big server with home directories, Mint, Debian, Ubuntu, and the Web Server are all done because all got sick under the previous kernel and were upgraded Tuesday.

     So Web service and those three shell servers will not be interrupted.  Mail service, the other shell servers, and private virtual servers will be interrupted but it should be less than 15 minutes downtime in all.

     We have a new (old) payment option.  In addition to mail-in payments and credit and debit cards, we can now process electronic checks.

     For those who would like to learn / experiment with the bleeding edge features of C and C++, gcc 11.2 is now available on the Ubuntu shell server.  To use it make sure that /usr/local/bin is first in your path before /usr/bin.

     Because Ubuntu just locked up and because this time I got an indication it was an issue with the hypervisor and physical host, I am going to be upgrading the following machine kernels early this evening around 11:30PM:

     Iglulik (has NFS /home and /misc directories), ftp, debian, ubuntu, and mint.  These will all be down briefly, the web server longer than the others.

     Web services including all our virtual hosts, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and all webmail services as well as our own website, https://www.eskikmo.com/ will be affected.

     We’ll be doing kernel upgrades of all machines this coming Friday starting at 11pm.  Interruption to most services should be relatively brief, however, web based services may take approximately 1/2 hour to restore.  This is because I’ll be reverting the physical host to a 5.13.19 kernel (presently the only host on 5.14) as 5.14 did not fix the ethernet issue AND has proven to be less than stable.  Because a different driver for the flash card is required for 5.14 to 5.13 I can not use the dkms mechanism to compile this driver and because to compile it fresh the machine needs to be running the kernel upon which the driver is running, I will need to first reboot the physical host to 5.13, (it will be the 5.13.19 EOL kernel) and then recompile the drivers, then bring up the web server.  From previous experience this process requires around 1/2 hour.

     Web service includes all our virtual hosts, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and all webmail services as well as our own website, https://www.eskikmo.com/.