Kernel Updates Friday May 20th at 11PM Pacific Daylight Time (-0700 GMT).

We will be doing kernel updates on all systems Friday May 20th starting at 11pm Pacific Daylight Time (-0700 GMT).  I expect interruptions of individual services not to be more than about ten minutes and it should be completed by 12PM (midnight).

This will impact all of Eskimo North’s services including:

     https://www.eskimo.com/

     https://nextcloud.eskimo.com/

     https://friendica.eskimo.com/

     https://hubzilla.eskimo.com/

This kernel upgrade is a point release that fixes some bugs.  It does not change capabilities or interfaces so should have no impact except to prevent crashes and processes hanging.  We haven’t experienced either of these with the current kernel but others have.

Distro Updates Completed

     All of the Ubuntu based physical hosts are now upgraded to Jammy 22.04.

     As far as I can tell, everything seems to be up and running.  Did need to fix some things with apparmor breaking my name server which has some configurations they didn’t expect, and various broken NFS mount points, and random services and servers that didn’t start, but I believer everything is good now save for radius running in simplex at the moment because one server died hard (won’t even post).  Working on a fix for that.

Eskimo.Com Kernel Upgrades Complete – Distro Upgrades in Progress

     Kernel upgrades are completed.  All NFS and NIS mounts checked and all machines checked to be sure all services are running.

     A Distribution upgrade of the physical hosts from Ubuntu 20.04 to 22.04 is in progress.  In several hours this will also require rebooting everything again and fixing some NFS configuration files which I have found this upgrade breaks.

     This will affect all ‘eskimo.com’ services since these are the physical machines that everything runs on.

Kernel Upgrades Sunday May 14th at 11PM PDT (GMT -0700)

     I ordinarily do kernel upgrades on Friday so I have a weekend to recover if something goes wrong before the heavier business usage during the week, however, because my car is in the shop I am going to do them this Sunday evening starting at 11pm.  That day because my wife has it off so if something locks up and I need to go to the co-location facility I can use her car.

     If the kernel upgrade goes well, I may attempt to upgrade some of the physical servers to 22.04 now that I have the NFS issues with 22.04 resolved.

     This will affect all of Eskimo’s Services, various services will be down for as long as 10 minutes but not all at the same time.

     This will affect all sites hosted here including private virtual servers, shared web hosting, Linux shell services, e-mail, and our social media sites, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/, as well as our own site https://www.eskimo.com/ but the interruptions should in general be ten minutes or less.

 

NFS Kernel Server – Linux Ubuntu 22.04

     For anyone maintaining a hetrogeneous network utilizing Linux NFS servers, when I upgraded from 20.04 to 22.04 on our mail server which exports the mail spool to various machines, most using version 4.2 over TCP NFS but some using version 3 UDP and one using version 2 UDP, it broke all of our machines mounting using version 2 or version 3 NFS even though rpcinfo -p showed these protocols exported.

     It turns out that what changed is capabilities and defaults.  All three NFS versions are supported, but all now DEFAULT to TCP only.  This makes ZERO sense as UDP was the only protocol available in the NFS version 2 and version 3 specification.

     However, it is configurable and now everything is back in service and I can move forward with updating the rest of our machines that are 20.04 to 22.04, however I am going to hold off on that until I get my car back from the shop as that such upgrades often result in disasters leaving a machine unbootable and requiring in person attention.

22.04 NFS

     I wrote zero since when I meant zero sense.

     Not sure why but I have a real problem with homonyms, since and sense, to, too, and two, etc.  I know the correct usage but when I write, my stream of consciousness is in a verbal form and the translation to to text more or less automatic and not taking homonyms into account, so unless I actually proof read, which I do not do often enough, this results.

Kernel Upgrades

     I will be doing kernel upgrades over the next week or so but not all at once as per usual.  I will be doing the virtual machines which are most exposed to the external Internet first, and holding off on the physical servers until my car returns from the shop (it ate it’s alternate, an unfortunately not infrequent occurrence for this model.

     The virtual machines I can always reboot remotely but the physical machines I need to be there in person if they lock up during a boot.

Network Latency / Packet Loss

     Our router at the co-lo facility is running short of CPU during peak traffic times in the evenings resulting in greater latency and some packet loss.

     I am currently researching a replacement which will have approximately 6x as much CPU to accommodate this traffic and future growth.

     The current router is an Edgerouter Lite which has two 500 Mhz MIPS CPUs, the replacement I’m looking at is a Ubiquiti Dream Machine, which has four 1.7 Ghz cores.  The latter is designed for an enterprise environment.

     The Dream Machine comes in a lot of different flavors and I’m not familiar with all the terminology used in the specifications so I’m waiting on Ubiquiti support to answer a few questions before placing an order.

Mail

     All of our mail servers have now been upgraded to 22.04, and one of the things I was unaware of before upgrading to 22.04 is that postfix, the mail transport agent that we use, has changed a bunch of defaults.  This may cause some problems and I am reviewing all the settings to make sure that where the defaults have changed, we explicitly declare the settings we want rather than relying on the now incorrect defaults.

     I’ve also loosened somewhat the fail2ban restrictions on the incoming servers.  Since the primary function of fail2ban is to stop brute force password guessing attacks, and people do not authenticate to the incoming servers, it is not necessary that they be as strict.  We had some instances where outlook servers were banned for repeatedly trying to deliver mail to non-existent addresses.  This happens when spammers use very unclean old address lists.