A Linux upgrade changed openssl in such a way that the new libraries were incompatible with the existing compilation of apache2.  I had to recompile apache2 to get it to listen to the SSL port again.  Sorry, Ubuntu does not bother to notify us when they make crushing changes to system libraries.

     This briefly affected all of Eskimo North’s SSL and HTTP/2 web services including webmail, friendica, hubzilla, and nextcloud.

     Kernel upgrades completed, Mail system and all shell servers and virtual private servers are available.  NFS and NIS connectivity has been verified on all machines.

     Tonight’s Kernel Upgrade will be brief, starting at 11PM and should be concluded by 11:15PM as the big server with home directories, Mint, Debian, Ubuntu, and the Web Server are all done because all got sick under the previous kernel and were upgraded Tuesday.

     So Web service and those three shell servers will not be interrupted.  Mail service, the other shell servers, and private virtual servers will be interrupted but it should be less than 15 minutes downtime in all.

     We have a new (old) payment option.  In addition to mail-in payments and credit and debit cards, we can now process electronic checks.

     For those who would like to learn / experiment with the bleeding edge features of C and C++, gcc 11.2 is now available on the Ubuntu shell server.  To use it make sure that /usr/local/bin is first in your path before /usr/bin.

     Because Ubuntu just locked up and because this time I got an indication it was an issue with the hypervisor and physical host, I am going to be upgrading the following machine kernels early this evening around 11:30PM:

     Iglulik (has NFS /home and /misc directories), ftp, debian, ubuntu, and mint.  These will all be down briefly, the web server longer than the others.

     Web services including all our virtual hosts, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and all webmail services as well as our own website, https://www.eskikmo.com/ will be affected.

     We’ll be doing kernel upgrades of all machines this coming Friday starting at 11pm.  Interruption to most services should be relatively brief, however, web based services may take approximately 1/2 hour to restore.  This is because I’ll be reverting the physical host to a 5.13.19 kernel (presently the only host on 5.14) as 5.14 did not fix the ethernet issue AND has proven to be less than stable.  Because a different driver for the flash card is required for 5.14 to 5.13 I can not use the dkms mechanism to compile this driver and because to compile it fresh the machine needs to be running the kernel upon which the driver is running, I will need to first reboot the physical host to 5.13, (it will be the 5.13.19 EOL kernel) and then recompile the drivers, then bring up the web server.  From previous experience this process requires around 1/2 hour.

     Web service includes all our virtual hosts, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and all webmail services as well as our own website, https://www.eskikmo.com/.

     I plan on doing some kernel upgrades over the evening but probably not the physical hosts or the lesser used virtual machines.  There is some instabilities in the existing kernel that is causing CPU lock-ups.  Mostly these are transient and resolved by watchdog timers but one did hang our web server the other day.

     I do not have access to parking at the co-location facility this weekend because they are re-stripping the parking lot so I’ll save updates of physical hosts for next weekend.

     The reboots of the virtual machines only take a few seconds so interruptions will be brief.  I’ve already done the web server so it will not be impacted, mainly shell and mail servers.

     Also I plan to take some of the private virtual servers down for between 15-60 minutes depending upon their size to image them (a form of backup).

     Apache 2.4.49 was released today and had many important security fixes and also some fixes to help it gracefully shutdown which makes it do so much faster.  This will result in a much shorter interruption during log rotates at midnight.

     We have installed 2.4.49.  Pingdom tests reveal essentially unchanged performance loading our WordPress site in 279ms.

     With respect to Eskimo North’s we services, I found that the issue that hung our web server has existed since May of this year so I’m not real optimistic the next kernel upgrade will resolve it, however, I found a kernel variable I set that will cause the system to panic and reboot if it occurs again so at least it won’t result in hours long outages.

     This note covers several topics, web server crash, kernel upgrades, and 56k dial service.

     The web server crashed today shortly after 9AM but nobody called until 1PM and it was in a mode where it was still consuming CPU and appeared up to the hypervisor, but not talking to the network so I remained unaware of it until I received that call.

     This affected all Eskimo North web services including webmail, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/ and virtual domain hosting.

     Also, a small aside, unlike our previous attempts at social media, friendica has been largely successful and has acquired an active user base so far of 566 users and is growing at a daily rate of 1-2%.  It is part of a federated network of almost four million users and sports some fairly interesting discussions.

     This is the second time it has crashed since the last kernel upgrade but the first time it just locked without logging anything, this time I did get good logging of what happened.  I will be doing an upgrade on the kernel of this machine tonight in hopes of addressing this but I am not 100% sure if the issue is with the virtual host.

    So the next major kernel upgrade affecting all servers will be on September 24th between 11PM and 12AM.  I will be reverting Iglulik, the machine that hosts ftp, www, ubuntu, debian, and mint to the latest 5.13 kernel since the 5.14 appears flawed and did not fix the Intel E1000 chipset issues they said it would.or the physical host.  Tonight’s upgrade will only be of the virtual host.

     The physical host has not shown any errors however this kernel had been stable on the virtual machine until I upgraded the kernel on the physical machine.  Confusing I know.  But the bottom line is this, it is safe for me to boot a virtual machine remotely because I can control it remotely even in a crashed state, a physical machine I can’t, so the latter will need to wait until Friday the 24th of September (they are re-striping the parking lot this weekend so no parking at the co-lo when I  usually do these).

     So the next major kernel upgrade affecting all servers will be on September 24th between 11PM and 12AM.  I will be reverting Iglulik, the machine that hosts ftp, www, ubuntu, debian, and mint to the latest 5.13 kernel since the 5.14 appears flawed and did not fix the Intel E1000 chipset issues they said it would.

     Then the last note is I do not anticipate 56k dial service continuing after the end of 2023, this per my one remaining wholesale provider that only has large      So the next major kernel upgrade affecting all servers will be on September 24th between 11PM and 12AM.  I will be reverting Issumataq, the machine that hosts ftp, www, ubuntu, debian, and mint to the latest 5.13 kernel since the 5.14 appears flawed and did not fix the Intel E1000 chipset issues they said it would.contracts lasting through that date.  This could change if  one of their larger customers renews their contract but at present this is the plan.