In about 1-1/2 hours I will be going to the co-location facility to extract a broken server.  This should not be service impacting except that it is in a physical stack that will require disturbing other machines and whenever that happens there is always to potential for a power cord to fall out or something similar.

     CORRECTION:

     Kernel Updates will be Friday May 20th, NOT tonight May 19th.

We will be doing kernel updates on all systems Friday May 20th starting at 11pm Pacific Daylight Time (-0700 GMT).  I expect interruptions of individual services not to be more than about ten minutes and it should be completed by 12PM (midnight).

This will impact all of Eskimo North’s services including:

     https://www.eskimo.com/

     https://nextcloud.eskimo.com/

     https://friendica.eskimo.com/

     https://hubzilla.eskimo.com/

This kernel upgrade is a point release that fixes some bugs.  It does not change capabilities or interfaces so should have no impact except to prevent crashes and processes hanging.  We haven’t experienced either of these with the current kernel but others have.

     Someone here, and sorry my memory does not remember who, contacted me a while back with respect to Nicholas Heer, if you could please contact me again I would very much appreciate it.

     All of the Ubuntu based physical hosts are now upgraded to Jammy 22.04.

     As far as I can tell, everything seems to be up and running.  Did need to fix some things with apparmor breaking my name server which has some configurations they didn’t expect, and various broken NFS mount points, and random services and servers that didn’t start, but I believer everything is good now save for radius running in simplex at the moment because one server died hard (won’t even post).  Working on a fix for that.

     Kernel upgrades are completed.  All NFS and NIS mounts checked and all machines checked to be sure all services are running.

     A Distribution upgrade of the physical hosts from Ubuntu 20.04 to 22.04 is in progress.  In several hours this will also require rebooting everything again and fixing some NFS configuration files which I have found this upgrade breaks.

     This will affect all ‘eskimo.com’ services since these are the physical machines that everything runs on.

     I ordinarily do kernel upgrades on Friday so I have a weekend to recover if something goes wrong before the heavier business usage during the week, however, because my car is in the shop I am going to do them this Sunday evening starting at 11pm.  That day because my wife has it off so if something locks up and I need to go to the co-location facility I can use her car.

     If the kernel upgrade goes well, I may attempt to upgrade some of the physical servers to 22.04 now that I have the NFS issues with 22.04 resolved.

     This will affect all of Eskimo’s Services, various services will be down for as long as 10 minutes but not all at the same time.

     This will affect all sites hosted here including private virtual servers, shared web hosting, Linux shell services, e-mail, and our social media sites, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/, as well as our own site https://www.eskimo.com/ but the interruptions should in general be ten minutes or less.

     For anyone maintaining a hetrogeneous network utilizing Linux NFS servers, when I upgraded from 20.04 to 22.04 on our mail server which exports the mail spool to various machines, most using version 4.2 over TCP NFS but some using version 3 UDP and one using version 2 UDP, it broke all of our machines mounting using version 2 or version 3 NFS even though rpcinfo -p showed these protocols exported.

     It turns out that what changed is capabilities and defaults.  All three NFS versions are supported, but all now DEFAULT to TCP only.  This makes ZERO sense as UDP was the only protocol available in the NFS version 2 and version 3 specification.

     However, it is configurable and now everything is back in service and I can move forward with updating the rest of our machines that are 20.04 to 22.04, however I am going to hold off on that until I get my car back from the shop as that such upgrades often result in disasters leaving a machine unbootable and requiring in person attention.

     I wrote zero since when I meant zero sense.

     Not sure why but I have a real problem with homonyms, since and sense, to, too, and two, etc.  I know the correct usage but when I write, my stream of consciousness is in a verbal form and the translation to to text more or less automatic and not taking homonyms into account, so unless I actually proof read, which I do not do often enough, this results.

     I will be doing kernel upgrades over the next week or so but not all at once as per usual.  I will be doing the virtual machines which are most exposed to the external Internet first, and holding off on the physical servers until my car returns from the shop (it ate it’s alternate, an unfortunately not infrequent occurrence for this model.

     The virtual machines I can always reboot remotely but the physical machines I need to be there in person if they lock up during a boot.