Manjaro failed to boot with the new kernel and I chased the issue to my recent installation of gcc-13.1 on this machine.  It changed some system libraries in a way that broke them.

     I am restoring from backup, then will bring the machine up to date, then will install the new kernel, all others are complete.

     We will be doing a kernel upgrade of all servers tonight starting at 11pm.  If all goes well it should conclude by 11:30, if not and we have to make a trip to the co-location facility, then 12:30.  No individual service with the exception of yacy should be down for more than about ten minutes.

     Igloo, one of the physical hosts that among other things hosts the mail spool, crashed earlier this afternoon.

     I was compiling a new version of GCC (13.1) using eight cores and it just got too hot.  I stress test these machines thoroughly when I initially turn them up but over time CPUs suffer something known as chip rot, essentially even though electrons are 1/2000th the mass of protons, when electrons move one way atoms occasionally move the other ever so slightly and over time this creep causes features to be less well defined and maximum stable CPU speeds become slightly less.  I have already down clocked slightly ice and iglulik, this machine is the only one still at 4.4 GHz, I will need to bump it down to 4.3 or 4.2 the next time I am at the co-lo.

     In the meantime I am completing the compilations on this machine using only four cores so it generates less heat.

     I am also working on a new 10 core / 20 thread server using an i9-10900x that will have 256GB of RAM to handle some of the more CPU and memory intensive workload.  Unlike the other machines which have two memory channels, this one has four.  It will also have everything, even the root partition, on RAID.

I thought this was pretty cool: Eskimo-Aleut languages

     Kernel upgrades are completed.  Sorry it took longer to get some of the virtual private servers up than normal.  One of the physical servers failed to come back up into the normal run level, it booted but systemd hung (which unfortunately it sometimes does though this is not frequent on the physical servers).

     I will be performing a kernel upgrade tonight starting at 11pm, expecting to finish by 11:30pm.  Expected downtime for most services less than ten minutes, yacy.eskimo.com may take as much as 45 minutes owing to the rebuilding of the database after reboot.

     This upgrade will affect all of Eskimo North’s services, shell accounts, UUCP, e-mail, web hosting, mail lists, and virtual private servers.

     It will also affect our free services, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.

     The fax machine has been repaired so you can send me faxes now if that is your preferred method of communicating.

     Kernel upgrades are all completed except for Manjaro (did not have time to finish compiling), it will be done later this morning.

     Will be attempting an upgrade to kernel 6.1.22 tonight starting at 11pm.  Expect to be finished at least with the reboots by 11:30pm and should have all NIS and NFS dependencies restored by midnight if not sooner.  Interruptions of individual services should not exceed about ten minutes with the exception of the Yacy search engine, https://yacy.eskimo.com/, this because Yacy rebuilds it’s database after each reboot and this process takes in excess of half an hour.

     This will affect all Eskimo North services including virtual private servers, shell servers, e-mail, web hosting, and our free services including https://friendica.eskimo.com, https://hubzilla.eskimo.com, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.

     The 6.1.22 kernel has some minor fixes relative to the 6.1.20 kernel we are presently running so I do not expect any significant operational changes.

     Kernel upgrades are completed.  Ran into a few minor software issues on some machines but all are resolved now.