Today’s web outage or near outage (the server was there just in swap hell so not very responsive), was caused by an experimental virtual machine that is running rocky10 going apeshit and consuming all available CPU cycles and memory.

      It was an attempt to hack NIS into a Redhat based distro, but I think instead I am going to turn my attention towards implementing LDAP for user info distribution and Kerberos for authentication because they are both more secure and widely supported in modern operating systems and web applications.

Linux Kernel 6.18.3 had a bug that caused crashes every 2-3 hours followed by a hang.

So having exhausted 6.18.x I tried the pre-release of 6.19-rc3, the developers did not document the fact that it made substantial changes to the RAID super block, but it did. Since all of the storage on this machine is in the form of RAID arrays, this resulted in none of them being assemblable and the machine being unbootable. It said that I needed mdadm-1.45 or later, and 1.45 was the very most recent version.

I drove home, compiled 1.45, wrote to a USB thumb drive and a rescue file system on another USB, drove down, booted off the rescue, installed mdadm 1.45 but it also claimed the superblock was not usable. So much for it needing 1.45, NO version of mdadm worked.

I ended up rebuilding all the raid metadata keeping the file systems, this worked and everything is back up and we are back on 6.17.9 in spite of it not being completely stable it is less crippled than current 6.18 releases or