Eskimo.net was unavailable to the Internet part of Thursday and Friday (Sept 9th-10th) owing to an error I made on the master DNS server.  It didn’t show up right away until the cache expired on the secondaries.  It is corrected now.

     Web server crashed at 22:10 PM Pacific Time today (Sept 9th).  Fortunately I tried to access it right at that time so it was back up after 4 minutes.  Unfortunately, it did not provide any information in the logs that gives a clue as the the cause.  It just hard hung and would not respond until forcibly booted.

     The Usenet News service I have been using to provide news for our shell servers seems to have gone TU without warning.  They sent me no notice but just disappeared from the Web.  They were a US based firm but the domain name is now owned by an outfit in England.  There is no website, phone numbers disconnected, no e-mail addresses working, no way to find out what happened.

     I am looking for a new provider but in the meantime I apologize but no Usenet News available until and unless I can find a replacement.

     If anyone knows of a potential replacement please contact support@eskimo.com or via the Fediverse nanook@friendica.eskimo.com.

     Unfortunately the 5.14.0 kernel did not contain the new E1000 driver advertised in advance, still the broken 2012 vintage broken driver.  So I’m cancelling the kernel upgrade I had planned for this Friday since nothing is to be gained from it.  I do see there is a new 2021 driver on Intel’s website, so I guess I’ll need to figure out how to integrate that into the kernel.

     I successfully got the iomemory drivers working under 5.14.0 on the server where it was important (the one with the flash card and Intel ethernet).  I’ve turned on hardware offloading and so far it has run cleanly.  Assuming it is still clean tomorrow I plan on upgrading the remaining servers to the 5.14.0 kernel tomorrow.  This will involves some downtime for all services between 11pm – Midnight.

     It was a bit of a battle, for some reason the automatic DKMS installation and update-initramfs failed so I had to do everything by hand, but by hand it all worked.  I can’t find any explanation for the failure codes and it did not fail when I did the same steps the makefile did manually.  Gotta love stuff like this, kind of beginning to feel a lot like Windows.

     This will affect ALL services including https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/.  E-mail coming in won’t be lost but will be delayed during this time.  Outgoing will be refused connection during this time.

     I have determined what I need to do to get the iomemory drivers (for flash card on main server with NFS home directories, web server, and flash drive) with the 5.14.0 kernel.  So I am going to be taking things down in about 15 minutes, between about 12:15 AM and 1:15 AM or so September 2nd.  Everything will be unavailable for a short while during reboots.

     The server maintenance is completed for the night.  The new drivers would not compile against 5.14.0 so we are back to 5.13.12 for now.

      I am going to be booting one of the physical servers (Iglulik) into a new 5.14 kernel tonight and this will take the web server, Mint, Debian, and Ubuntu shell servers down for a short interval, approximately ten minutes, EXCEPT the web server.

     The web server will be down longer because the upgrade to 5.14 requires rebuilding the iomemory drivers for the flash drive used for our MariaDB database.

     This will mean also that https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/.

     Other web based services such as Squirrelmail will also be unavailable during this time frame.  Home directories for all shell servers will be temporarily unavailable during the reboot but should be available within about five minutes.  You can access e-mail via the shell servers or via IMAP/SMTP while work on the web server is under way.

     5.14 has a new copyless driver for the Intel E1000 Ethernet chipset used on this server and it is my hope that this driver will fix the hardware offloading issues with the existing driver that force us to disable hardware offloading.

     I am receiving complaints from people unable to connect to servers that they used to be able to using putty on versions of Windows prior to Windows 10.

     The reason you are unable to connect is that as exploits are found in various encryption schemes, those schemes are retired in favor of newer more secure encryption algorithms.

     All versions of Windows prior to Windows 10, with the exception of Windows 8.1, are past end of life.  If you are using one of these, you will not be receiving updates that update security issues like encryption vulnerabilities and thus you will not be able to connect with your old encryption methods.

     Putty is an aftermarket ssh client for older version of Windows.  If you have Windows 10, you do not need putty as it comes with openssh, you just need to install it as one of Windows features (it is not installed by default).

     If you are using 8.1, you should upgrade your putty installation to the July 21, 2021 release.  If you are using an older version of Windows you can try this as well but older versions may not have the necessary libraries for it to work properly.

     System kernel upgrades are completed.  All services are back online.

     Debian came up first time properly so I ended up not doing any systemd debugging today.  Whether this was a fluke or they fixed something with upgrades since last kernel upgrade I do not know.  If it messed up again I will debug at that time.