Kernel upgrades went smoothly.  My oldest son Carl, was visiting so I didn’t post this earlier, but only had two machines not boot up properly, mxlinux and the web server, and a reboot was all it too for them.  ALL NIS bindings and NFS mounts happened correctly the first time around so it feels like these aspects of Linux are finally maturing to the point where the are reliable.

     I am going to be upgrading kernels on most of our servers starting at 11PM.  If things go smoothly we should be done by 11:30PM however the last round we had a couple of machines that had a corrupted Grub configuration that had to be fixed so it took a bit longer.

     Hubzilla has caused various grief to our web server but I’ve finally chased things down to a beta worker module that was very broken.  Installed the stable version and now it seems to be behaving.

     I messed up during last night’s backup of the web server so need to take it down again tonight, starting about 12:15AM (Mon March 8th) for about 45 minutes to make another backup.

     I will be moving Nextcloud from https://www.eskimo.com/nextcloud/ to https://nextcloud.eskimo.com/ to resolve a conflict between it and WordPress that prevents some of it’s features from working properly.

     I do not yet know the exact time and date this will happen.

     On Sunday 3/7/22, I am going to take the web server down for approximately an hour to image it in it’s new configuration.

     I managed to successfully install hardware and software.  Only drawback is that the ioFusion2 drive in some way conflicted with one of my NIC cards, so I had to move the ethernet back to the Intel controller.  The E1000 chipset drivers don’t work right with hardware offloading in Linux so it is necessary to disable it which puts more burdon on the CPU.  But I doubt it will make a measurable difference, between hosts I am seeing .1ms ping time.

     So drivers worked properly with the 5.10.20 kernel I am now running, and the utility software all installed and functioned properly, so the database is now operating off of it.  It reduced our WordPress home page load times from 240-290ms region to 207ms.  Database which used to max at 400TPS has successfully handled bursts of 2923 TPS.

     Database speed issues may or may not be resolved tomorrow with maintenance.

     The preferred way of using this drive is to invoke a feature called Atomic Writes.  What that does is it bypasses cache on a write and writes directly to flash memory eliminating the need for the database to do a double write while still insuring data integrity.

     The challenge is that drivers have not been officially supported for these drives since Sans Disk bought them and that was kernel release 3.6.x and we’re on 5.10.x.  However, a volunteer effort has brought the drivers forward for kernel 5.10.x but aren’t well tested.  And while I’ve located the drive code I still have not found the fusion I/O file system tools yet.

     Tomorrow the hardware issues will be addressed but the software may or may not.

     The Fusion I/O Flash drive that Amazon first told me would arrive on March 5th, then updated and said it will arrive between March 5th and March 12th, arrived today.

     I will be installing it on Friday evening.  During this time most services will be unavailable for probably about 1/2 hour starting around 11PM.  When completed, this should resolve the database bottleneck.

     I had looked at Oracle Linux as a possible replacement for CentOS down the road, however, Oracle is too broken.  First, x2go will not work because of an incomplete perl environment, and second, it is tied to 3.10 kernel which is not safe on modern Intel CPUs and lacks many features of modern kernels.  As a result we will not be offering an Oracle Linux shell server.