Found cause of mounting issues was disk controller host was changed from Virtio to SATA, I did this intentionally on Mail but not on Debian so not sure how it got changed, but changing it back corrected the swap mounting issue however the Volatile file and directory creation issue re-emerged even with older systemd, so upgraded everything back to current and will continue to troubleshoot.

     With an older systemd, Debian did not fail at that same spot, but there does seem to be a UUID issue with one of the partitions so will require further reboot after some tweaking.

     Debian didn’t exactly crash, when it booted it did not mount tmpfs on /tmp, this is an in memory file system used for temporary files for speed and efficiency.

     Many hours later it mounted it and this broke systemd that needs access to some of those files.

     I do not know what is causing this delayed mounting of partitions but I had a similar problem with mail when I booted it.  I suspected the newer kernel so I booted off the older kernel, same issue.  Then I installed 5.12.2, the most current, still the same.

     After about four attempts it succeeded (same with mail) and is up and running but until I identify this problem reboots are going to be an issue.  This kind of smells like a Poettering systemd issue but, Googling, I can find no other reports of a similar issue.

     I will continue to investigate but for now the server is back up.

     The kernel upgrades went mostly smooth except for the client mail server.  Had issues with systemd timing out before it mounted all the disk partitions.  This is a virtual machine configured to use a physical partition on a RAID10 device on this host computer and using virtio disk.  This has not been a problem in the past but did not want to work today.  Changing the disk emulation to SATA made it work so apparently there is some ugliness with the virtio devices in the newest kernel although it was not a problem with any of the virtual machines that used a file rather than a partition for their virtual disk.

     At this point the virtual private servers and host computer is now running 5.12.2, this because 5.10.x has not been stable on that machine.  The rest are running 5.10.35.

     https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/ and all other Eskimo North services are back online and operational.  With the exception of client mail access pretty much everything else was online by 11:30PM, but it took slightly longer than that to bring the mail server back online.

     We will be performing a kernel upgrade tonight between 11pm and midnight.  If all goes well it should be concluded by 11:30pm.

     Going over the change log, I do not expect it to fix a particular bug that is affecting a couple of our virtual private servers but it does address a bug in the Intel Ethernet controller that has been problematic and quite a few other small bugs.

Debian, Centos8, and vps1 Maintenance Completed

     I am taking vps1.eskimo.net down for approximately 1/2 hour to image.

     Taking Debian shell server down for about 45-60 minutes to image.

     Please use one of the other debian based servers in the interim, such as ubuntu, mint, julinux, zorin, or mxlinux.

     Taking centos8 shell server down for approximately 1/2 hour for imaging.

     I will be taking the Zorin shell server down for approximately 1/2 hour for imaging.