Debian is back up, with home directories and mail spool this time.

     I made an error in editing the /etc/fstab to fix the swap partition and edited out the NFS mounts for home directory and mail spool.  I’ve fixed that but I’m making a backup of the fixed machine before I bring it back into service.

     Debian is restored to service and available.

     I am planning on rebooting mail client server around midnight tonight in order to increase memory allocation because there were still a few failures resulting from being unable to allocate enough memory.

     Debian will be back up in about an hour.  I identified the problem as a bug in systemd-sysusers which causes it to hang if NIS is used with containers and avahi is not present.

      Avahi is not present because I do not want it “discovering” potentially insecure machines on the network and prefer the machine knows only about what I tell it.

      Since NIS is not optional and snap is, I’ve removed snap and that made Debian boot properly.  I am now making a backup to capture the current working configuration.  That will conclude in about an hour at which point the machine will be returned to service.

     Even after restoring from backup, debian still has problems, please use one of the other Debian based servers such as Ubuntu, Mint, Julinux, or Zorin while I sort this out.

     If there is an application on Debian that you need and is not present on one of these other servers, please e-mail support@eskimo.com or submit a ticket with https://www.eskimo.com/support/osTicket/.

     There will be multiple interruptions to Debian while I troubleshoot this issue further.

     Seem to be going down one rabbit hole after another.  I think somehow the system image for Debian has gotten severely corrupted, so I am restoring it from backups made on May 3rd, then will apply updates and bring it current.

     I expect Debian to be operational around 1700 Pacific Daylight Time.

     Found cause of mounting issues was disk controller host was changed from Virtio to SATA, I did this intentionally on Mail but not on Debian so not sure how it got changed, but changing it back corrected the swap mounting issue however the Volatile file and directory creation issue re-emerged even with older systemd, so upgraded everything back to current and will continue to troubleshoot.

     With an older systemd, Debian did not fail at that same spot, but there does seem to be a UUID issue with one of the partitions so will require further reboot after some tweaking.

     Debian didn’t exactly crash, when it booted it did not mount tmpfs on /tmp, this is an in memory file system used for temporary files for speed and efficiency.

     Many hours later it mounted it and this broke systemd that needs access to some of those files.

     I do not know what is causing this delayed mounting of partitions but I had a similar problem with mail when I booted it.  I suspected the newer kernel so I booted off the older kernel, same issue.  Then I installed 5.12.2, the most current, still the same.

     After about four attempts it succeeded (same with mail) and is up and running but until I identify this problem reboots are going to be an issue.  This kind of smells like a Poettering systemd issue but, Googling, I can find no other reports of a similar issue.

     I will continue to investigate but for now the server is back up.