I plan on doing some kernel upgrades over the evening but probably not the physical hosts or the lesser used virtual machines.  There is some instabilities in the existing kernel that is causing CPU lock-ups.  Mostly these are transient and resolved by watchdog timers but one did hang our web server the other day.

     I do not have access to parking at the co-location facility this weekend because they are re-stripping the parking lot so I’ll save updates of physical hosts for next weekend.

     The reboots of the virtual machines only take a few seconds so interruptions will be brief.  I’ve already done the web server so it will not be impacted, mainly shell and mail servers.

     Also I plan to take some of the private virtual servers down for between 15-60 minutes depending upon their size to image them (a form of backup).

     Apache 2.4.49 was released today and had many important security fixes and also some fixes to help it gracefully shutdown which makes it do so much faster.  This will result in a much shorter interruption during log rotates at midnight.

     We have installed 2.4.49.  Pingdom tests reveal essentially unchanged performance loading our WordPress site in 279ms.

     With respect to Eskimo North’s we services, I found that the issue that hung our web server has existed since May of this year so I’m not real optimistic the next kernel upgrade will resolve it, however, I found a kernel variable I set that will cause the system to panic and reboot if it occurs again so at least it won’t result in hours long outages.

     This note covers several topics, web server crash, kernel upgrades, and 56k dial service.

     The web server crashed today shortly after 9AM but nobody called until 1PM and it was in a mode where it was still consuming CPU and appeared up to the hypervisor, but not talking to the network so I remained unaware of it until I received that call.

     This affected all Eskimo North web services including webmail, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/ and virtual domain hosting.

     Also, a small aside, unlike our previous attempts at social media, friendica has been largely successful and has acquired an active user base so far of 566 users and is growing at a daily rate of 1-2%.  It is part of a federated network of almost four million users and sports some fairly interesting discussions.

     This is the second time it has crashed since the last kernel upgrade but the first time it just locked without logging anything, this time I did get good logging of what happened.  I will be doing an upgrade on the kernel of this machine tonight in hopes of addressing this but I am not 100% sure if the issue is with the virtual host.

    So the next major kernel upgrade affecting all servers will be on September 24th between 11PM and 12AM.  I will be reverting Iglulik, the machine that hosts ftp, www, ubuntu, debian, and mint to the latest 5.13 kernel since the 5.14 appears flawed and did not fix the Intel E1000 chipset issues they said it would.or the physical host.  Tonight’s upgrade will only be of the virtual host.

     The physical host has not shown any errors however this kernel had been stable on the virtual machine until I upgraded the kernel on the physical machine.  Confusing I know.  But the bottom line is this, it is safe for me to boot a virtual machine remotely because I can control it remotely even in a crashed state, a physical machine I can’t, so the latter will need to wait until Friday the 24th of September (they are re-striping the parking lot this weekend so no parking at the co-lo when I  usually do these).

     So the next major kernel upgrade affecting all servers will be on September 24th between 11PM and 12AM.  I will be reverting Iglulik, the machine that hosts ftp, www, ubuntu, debian, and mint to the latest 5.13 kernel since the 5.14 appears flawed and did not fix the Intel E1000 chipset issues they said it would.

     Then the last note is I do not anticipate 56k dial service continuing after the end of 2023, this per my one remaining wholesale provider that only has large      So the next major kernel upgrade affecting all servers will be on September 24th between 11PM and 12AM.  I will be reverting Issumataq, the machine that hosts ftp, www, ubuntu, debian, and mint to the latest 5.13 kernel since the 5.14 appears flawed and did not fix the Intel E1000 chipset issues they said it would.contracts lasting through that date.  This could change if  one of their larger customers renews their contract but at present this is the plan.

     Eskimo.net was unavailable to the Internet part of Thursday and Friday (Sept 9th-10th) owing to an error I made on the master DNS server.  It didn’t show up right away until the cache expired on the secondaries.  It is corrected now.

     Web server crashed at 22:10 PM Pacific Time today (Sept 9th).  Fortunately I tried to access it right at that time so it was back up after 4 minutes.  Unfortunately, it did not provide any information in the logs that gives a clue as the the cause.  It just hard hung and would not respond until forcibly booted.

     The Usenet News service I have been using to provide news for our shell servers seems to have gone TU without warning.  They sent me no notice but just disappeared from the Web.  They were a US based firm but the domain name is now owned by an outfit in England.  There is no website, phone numbers disconnected, no e-mail addresses working, no way to find out what happened.

     I am looking for a new provider but in the meantime I apologize but no Usenet News available until and unless I can find a replacement.

     If anyone knows of a potential replacement please contact support@eskimo.com or via the Fediverse nanook@friendica.eskimo.com.

     Unfortunately the 5.14.0 kernel did not contain the new E1000 driver advertised in advance, still the broken 2012 vintage broken driver.  So I’m cancelling the kernel upgrade I had planned for this Friday since nothing is to be gained from it.  I do see there is a new 2021 driver on Intel’s website, so I guess I’ll need to figure out how to integrate that into the kernel.

     I successfully got the iomemory drivers working under 5.14.0 on the server where it was important (the one with the flash card and Intel ethernet).  I’ve turned on hardware offloading and so far it has run cleanly.  Assuming it is still clean tomorrow I plan on upgrading the remaining servers to the 5.14.0 kernel tomorrow.  This will involves some downtime for all services between 11pm – Midnight.

     It was a bit of a battle, for some reason the automatic DKMS installation and update-initramfs failed so I had to do everything by hand, but by hand it all worked.  I can’t find any explanation for the failure codes and it did not fail when I did the same steps the makefile did manually.  Gotta love stuff like this, kind of beginning to feel a lot like Windows.

     This will affect ALL services including https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/.  E-mail coming in won’t be lost but will be delayed during this time.  Outgoing will be refused connection during this time.

     I have determined what I need to do to get the iomemory drivers (for flash card on main server with NFS home directories, web server, and flash drive) with the 5.14.0 kernel.  So I am going to be taking things down in about 15 minutes, between about 12:15 AM and 1:15 AM or so September 2nd.  Everything will be unavailable for a short while during reboots.