Kernel Upgrades Sunday May 14th at 11PM PDT (GMT -0700)

     I ordinarily do kernel upgrades on Friday so I have a weekend to recover if something goes wrong before the heavier business usage during the week, however, because my car is in the shop I am going to do them this Sunday evening starting at 11pm.  That day because my wife has it off so if something locks up and I need to go to the co-location facility I can use her car.

     If the kernel upgrade goes well, I may attempt to upgrade some of the physical servers to 22.04 now that I have the NFS issues with 22.04 resolved.

     This will affect all of Eskimo’s Services, various services will be down for as long as 10 minutes but not all at the same time.

     This will affect all sites hosted here including private virtual servers, shared web hosting, Linux shell services, e-mail, and our social media sites, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/, as well as our own site https://www.eskimo.com/ but the interruptions should in general be ten minutes or less.

 

NFS Kernel Server – Linux Ubuntu 22.04

     For anyone maintaining a hetrogeneous network utilizing Linux NFS servers, when I upgraded from 20.04 to 22.04 on our mail server which exports the mail spool to various machines, most using version 4.2 over TCP NFS but some using version 3 UDP and one using version 2 UDP, it broke all of our machines mounting using version 2 or version 3 NFS even though rpcinfo -p showed these protocols exported.

     It turns out that what changed is capabilities and defaults.  All three NFS versions are supported, but all now DEFAULT to TCP only.  This makes ZERO sense as UDP was the only protocol available in the NFS version 2 and version 3 specification.

     However, it is configurable and now everything is back in service and I can move forward with updating the rest of our machines that are 20.04 to 22.04, however I am going to hold off on that until I get my car back from the shop as that such upgrades often result in disasters leaving a machine unbootable and requiring in person attention.

22.04 NFS

     I wrote zero since when I meant zero sense.

     Not sure why but I have a real problem with homonyms, since and sense, to, too, and two, etc.  I know the correct usage but when I write, my stream of consciousness is in a verbal form and the translation to to text more or less automatic and not taking homonyms into account, so unless I actually proof read, which I do not do often enough, this results.

Kernel Upgrades

     I will be doing kernel upgrades over the next week or so but not all at once as per usual.  I will be doing the virtual machines which are most exposed to the external Internet first, and holding off on the physical servers until my car returns from the shop (it ate it’s alternate, an unfortunately not infrequent occurrence for this model.

     The virtual machines I can always reboot remotely but the physical machines I need to be there in person if they lock up during a boot.

Network Latency / Packet Loss

     Our router at the co-lo facility is running short of CPU during peak traffic times in the evenings resulting in greater latency and some packet loss.

     I am currently researching a replacement which will have approximately 6x as much CPU to accommodate this traffic and future growth.

     The current router is an Edgerouter Lite which has two 500 Mhz MIPS CPUs, the replacement I’m looking at is a Ubiquiti Dream Machine, which has four 1.7 Ghz cores.  The latter is designed for an enterprise environment.

     The Dream Machine comes in a lot of different flavors and I’m not familiar with all the terminology used in the specifications so I’m waiting on Ubiquiti support to answer a few questions before placing an order.

Mail

     All of our mail servers have now been upgraded to 22.04, and one of the things I was unaware of before upgrading to 22.04 is that postfix, the mail transport agent that we use, has changed a bunch of defaults.  This may cause some problems and I am reviewing all the settings to make sure that where the defaults have changed, we explicitly declare the settings we want rather than relying on the now incorrect defaults.

     I’ve also loosened somewhat the fail2ban restrictions on the incoming servers.  Since the primary function of fail2ban is to stop brute force password guessing attacks, and people do not authenticate to the incoming servers, it is not necessary that they be as strict.  We had some instances where outlook servers were banned for repeatedly trying to deliver mail to non-existent addresses.  This happens when spammers use very unclean old address lists.

Mail Problems

     A distribution upgrade failed on mail.eskimo.com leaving the machine in an unbootable state where grub failed to install properly.

     An attempt to install grub2 manually results in can not write EFI variables to NvRAM.  I’ve not encountered this on a virtual machine before and not being successful at finding info on how to fix.  The easiest thing may be to just rebuild the virtual machine from scratch and then restore all the files.  I have a few other things I’m going to try first but before I do I’m making a current backup of the mail spool to make sure we do not lose any mail.  This may take several hours.

Web Server Upgraded

     Finally got the web server upgraded.  Don’t really know what went wrong last night, pretty much repeated the same procedure but this time after I had finished the compile environment still worked so I was able to recompile apache2 and have it use the newer openssl 3.0.2.

     So all of eskimo.com including friendica.eskimo.com, hubzilla.eskimo.com, and nextcloud.eskimo.com, and all the hosted websites are now running under Ubuntu 22.04.

Happy Mother’s Day + Maintenance Notice

     I wish to wish all of you who are Mother’s a very happy Mother’s day!

     My mother has been gone from this planet for about a decade now.  She was the wisest woman I’ve ever known in my life.  Lived a hard life but threw her best effort at it.  She is with the Lord now I am sure, she was a good woman.

     Today I also tested negative for Covid so in spite of being 63, diabetic, with high blood pressure and overweight it didn’t succeed in killing me!  Well now I have natural immunity without having had a clot-shot.  I know there are still people who believe in that thing but I would suggest strongly you avail yourself to some of the 75,000 pages of data Pfiezer was forced to release.  Strangely, I have had some folks be angry at me for not taking the clot-shot but I’ve not endangered anyone as I completely self-isolated for the duration of the disease.

     And on a final note, I will be taking the web server down again tonight for a while making one more attempt to upgrade.  I actually had it working briefly under 22.04 last night but with the wrong version of openssl, and trying to correct that resulted in a chain reaction of deleted software owing to new dependencies in 22.04 that are going to require some work-arounds, so I had to revert to the previous backup.

     End of life for the OS on the web server happens this July so push is coming close to shove.  After that there are no security updates and the web server is perhaps the most exposed server in terms of the total number of services it makes available to exploit so it is of particular importance to keep current.

     This will affect all web sites hosted under Eskimo North shared hosting plans as well as https://friendica.eskimo.com, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://www.eskimo.com.

     The plan is tonight to get it into the same state where it is on 22.04 but still running the old openssl, then write backups, then work on correcting that which unfortunately is going to also require fixing much of my build environment which upgrading to 22.04 also broke.