Because at some point I got the proposed archive enabled on Ubuntu, there are incorrect versions of a lot of software installed on Ubuntu making upgrading impossible.  I have to de-install all this stuff, upgrade, and then re-install.  So Ubuntu is going to be marginally functional while this is in progress.  I hope that I can complete this over the weekend.  In the meantime if something you need is among that removed, please use Debian, Mint, Zorin, or MxLinux, and if what you need is not present on them please use the ticket system to request what you need be installed.

     I am backing out the partial Ubuntu upgrade and restoring from backups tonight and will attempt again tomorrow.

     I am going to take the shell server, ubuntu.eskimo.com, and the web server www.eskimo.com down for maintenance tonight between 10pm-11pm to make backups prior to upgrading from Ubuntu 20.04 Focal to Ubuntu 22.04 Jammy.

     After the backups I will begin upgrade.  I do not anticipate long downtime for Ubuntu for the actual upgrade, only a reboot, however the webserver will take someone long as the change from OpenSSL 1.1.1 to OpenSSL 3.0.1 will require recompiling httpd.

     This will affect https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, and https://nextcloud.eskimo.com/ as well as all virtual domains hosted with us.  I do not expect this to take more than 10-15 minutes for the actual upgrade as apache should continue to run until I reboot.

     I have to ask that you not mail payments until further notice as TIna, Raymond, and myself are all down with Covid-19.

     We are all doing alright so far, staying in our heads and not going into our lungs, good oxygen saturation status, mild fevers.  Pretty much the symptoms of your average head cold thus far.  Tina and I have been sick since Friday, Raymond just got sick today.

     We are pretty sure we know who exposed us to this and it was a triple jabbed moron that works with Tina that got it after being exposed to his sister but continued to walk around the store with it for a week anyway.

     Kernel upgrades are completed, all machines are operational.

     I will be doing another kernel upgrade this Friday between 11pm-midnight requiring reboots of all machines.

     As I feared, there were bugs introduced in NFS, particular an issue with delayed requests in NFSD causing the daemon to die (nfs server daemon).  While I have not encountered this yet it is a documented problem so going to upgrade to fix.

     Also some issues where introduced into KVM/QEMU used for hosting virtual machines.  Again, I haven’t experienced these but it is a race condition and so just a matter of time.

     With respect to Nextcloud, still no fix from the developers.  I’m going to attempt a re-install while keeping the existing database.

     I apologize if Nextcloud is not working for you at the moment.  I attempted an upgrade yesterday and it failed resulting in file integrity check failing for a bunch of files.

     I opened a ticket for this and found out many others are experiencing the same thing for this release, but as of yet there is no fix.

     I intent to take the mail and web servers down tonight at 11pm to make backup images.  This is just a way to quickly restore the mail server or web server if something catastrophic happens to their file systems.  The mail server should take about 1/2 hour and the web server probably similar.  Don’t know exactly as I have not made an image backup since moving it to flash storage.

     Tonight’s kernel upgrade went smoother than I had anticipated.  I had anticipated issues based upon the fact that we were moving to a new major release and I knew that NFS code got somewhat re-worked again.  This has always been a problem in Linux, not sure why they can’t get a 1995 vintage protocol correct but it’s always been difficult.  But this time NFS worked flawlessly.  No duplicate ID’s in fscache either.  NFS properly mounted on all machines.

     NIS bound on all but one.

     A few machines had some start up issues unrelated to kernel, in particular I had removed hddtemp from one because it is a virtual machine and there is no temp sensor for hddtemp to read but it did not remove the systemd service unit.  Argh.  But anyway that was quick and easy to correct.

     So as far as I know everything appears to be functional.  The only machine that failed was one that I had not put into service yet.  Something in the EFI settings got messed and it thinks it’s out of space on the boot device but it is not.  But since it’s not yet in service it is of no consequence.

     Someone or some thing halted Ubuntu this evening at just after 8pm.  I do not know what, could not locate anything in the logs.

     At this point I’ve removed public access to systemctl, removed a number of graphical reboot/shutdown tools, and performed an early upgrade of the kernel on this machine in case it is a kernel bug.

     If you happen to notice the machine goes aware right after you do something, please let me know what that something is.  E-mail nanook@eskimo.com.  Thank you.