Kernel upgrades completed by 12:30, various services that systemd failed to start properly all corrected by 12:46.  Everything is back in service at Eskimo.

     I am planning a kernel upgrade tonight Saturday May 13th starting at 11PM.

     This will affect all of Eskimo North’s services: web hosting, shell accounts, e-mail, virtual private servers, and our free Fediverse services https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.

     Provided all servers boot correctly no service with the exception of yacy should be down for more than about ten minutes.  Yacy takes up to 45 minutes to re-index it’s database upon startup.

     Also, I will be slow for a day or so to get payment receipts out.  Processing them requires cut-n-paste and the middle button (necessary for pasting) of my Logitech G 203 mouse has bitten the dust.  I must say I am deeply saddened as I’ve never had a Logitech anything die on me before, I thought they were invincible, even inputting cans of Diet Coke into the keyboard has only resulted in temporary non-functioning until it dried.  I really can’t complain though, it has served me loyally for many years and a new one is on order.

     I am going to take the web server database offline briefly later this evening to make a backup before trying an experiment with Nextcloud.  In the past I have gotten the external_authentication module to work for versions it was not designed simply by altering the definitions file associated with it.  I am going to see if I can get NextCloud HUB 4 to work this way.  By backing everything prior, I will have a way to return to NextCloud Hub 3 if it does not work.  I will do this after midnight so as not to interrupt much normal use.

     I’ve installed a new application for your use on the Ubuntu shell server if you access it visually via x2go, rdp, vnc, or via the website.  This new application is called LibreWolf and is under the Internet menu on Gnome or Mate Desktops.

     LibreWolf is an independent fork of Firefox with enhanced security and privacy.

Try expanding a terminal, preferably one that handles ansi colors, to full screen and then in ubuntu type, “cmatrix”.

     I found the Manjaro XFCE spin installer works, including with manual partitioning.  Sad Mate doesn’t but not surprised.  It will probably take 4-5 hours to get everything configured but should be working again later this evening.

     Manjaro may be down for a while, possibly until the next distro comes out.  The existing installation was hurt by installing gcc-13.1 which corrupted some of the library files.

     I restored from backups but the Manjaro update system won’t let you update a system that is too old, so that didn’t work.

     I tried to install from the old ISO, that didn’t work, for the same reason.

     I then downloaded the CURRENT Mate distro, and installed from that.  It would not work if I manually partitioned.  So I let Mate partition the way it wanted to.

     After installing I tried to login and it would not accept my password.  So I booted with the install disk again and tried to mount the root partition but it said file system was not recognized.

     I’m trying some different desktop spins now on the off chance that only the Mate spin is broken but if if it’s all of them then it will be down until a non-broken ISO becomes available.

     Manjaro failed to boot with the new kernel and I chased the issue to my recent installation of gcc-13.1 on this machine.  It changed some system libraries in a way that broke them.

     I am restoring from backup, then will bring the machine up to date, then will install the new kernel, all others are complete.

     We will be doing a kernel upgrade of all servers tonight starting at 11pm.  If all goes well it should conclude by 11:30, if not and we have to make a trip to the co-location facility, then 12:30.  No individual service with the exception of yacy should be down for more than about ten minutes.

     Igloo, one of the physical hosts that among other things hosts the mail spool, crashed earlier this afternoon.

     I was compiling a new version of GCC (13.1) using eight cores and it just got too hot.  I stress test these machines thoroughly when I initially turn them up but over time CPUs suffer something known as chip rot, essentially even though electrons are 1/2000th the mass of protons, when electrons move one way atoms occasionally move the other ever so slightly and over time this creep causes features to be less well defined and maximum stable CPU speeds become slightly less.  I have already down clocked slightly ice and iglulik, this machine is the only one still at 4.4 GHz, I will need to bump it down to 4.3 or 4.2 the next time I am at the co-lo.

     In the meantime I am completing the compilations on this machine using only four cores so it generates less heat.

     I am also working on a new 10 core / 20 thread server using an i9-10900x that will have 256GB of RAM to handle some of the more CPU and memory intensive workload.  Unlike the other machines which have two memory channels, this one has four.  It will also have everything, even the root partition, on RAID.