Reboots are completed.  All machines up and verified proper NIS and NFS mounts and all subsystems started successfully.  Two machines are still on 5.16.14, those are the UUCP machine and manjaro, and this because I’m having difficulties getting 5.16.17 to build properly on those boxes.  I have a bug report opened on bugzilla regarding failed compilation on these machines.

     I am going to be rebooting all the machines to upgrade the kernels tonight starting at 11PM.  I expect all to be done by 11:30PM.  Interruption of individual services should be brief.

     Per a customers request, tmux is now installed on all shell servers.

     All disks are now good, no more errors, and loads on all machines now are satisfactory, switching to virtio drivers on all the virtual hosts made a HUGE difference.  Not hard to get results like below with STATIC HTML content, but find another site that loads PHP based WordPress site in this time frame!

     Doing some benchmarking on virtual machines, I discovered on fast drives that virtio drivers are about 10% faster than emulated SATA drives for the SAME hardware.

     I had erroneously assumed SATA would be faster because it supported hdparm read-ahead where virtio does not support this parameters, but it turns out that the Linux kernel has demand read-ahead that detects sequential reads and automatically reads ahead and prefills buffers when it detects this. That nullifies this advantage and thus the reduction in interrupts and system calls of virtio provides better performance.

     As a consequence, I am going to take the web, mail, and uucp servers down briefly around 11pm tonight to change it’s virtual disks from SATA to virtio.

     The brief interruption of web services at 5PM Pacific Time today March 20th was for the purpose of increasing the memory allocated to the web server from 64GB to 80GB to provide a bit more overhead for load spikes.

     Ok, running good now that I switched the emulation so going to take down around 11pm’ish to backup as it is.  Also going to take Ubuntu down to move to a different physical server.  Ubuntu and the web server are the two heaviest loads so doesn’t make good sense to have them on the same physical machine.

     Regarding the web server, I discovered the Ethernet device was set to an emulated device, but by switching to virtio, I was able to get the interrupt load down substantially, so going to watch it run for a while and make a decision later with regards to whether or not to move to physical hardware now.

     Instead of backing up the existing www configuration, I am going to attempt to move the web server from a virtual server to a physical server.  The reason for this is that 5.16 kernel is not handling interrupts as efficiently as previous kernels did but previous kernels have an exploit known as a “dirty pipe” exploit that make running them hazardous.

     So by moving the web server to a physical machine I’ll more than half the number of ethernet interface related interrupts the CPU has to handle and those are the majority.  The reason for this is that the emulated ethernet in the virtual machine has no hardware offloading so it has to generate an interrupt for every packet, and the CPU has to process both the interrupts for the emulated ethernet and the interrupts for the real ethernet though the latter are fewer because it does have hardware offloading.  By eliminating the emulated ethernet, I will cut the ethernet generated interrupt traffic by more than half.

     The work I set out to do last night is completed except the final configuration was not what I had initially intended, however, I was not able to boot from flash owing to the BIOS of the machine only recognizing SATA flash drives and not PCIe drives which is unfortunate since all the SATA slots are full and the PCIe provides superior performance.  But I was able to use it as a data drive so the database is moved off of IO/fusion drive to it, and the entire web server is on it.  The defective disk has been replaced and there have been no more disk errors.

     Starting around 11PM tonight, I plan on taking the web server down for about an hour to backup the new configuration.  This will impact all eskimo.com web services including nextcloud and our social media sites friendica and hubzilla.

     I plan to take Ubuntu down for about an hour to move it to another physical host for load balancing purposes.