I got authentication working again by taking the version 3.0.0-beta meant for Nextcloud 24, editing the xml file that described it to include support for Nextcloud 25, tried it, and it SEEMS to work.

     Nextcloud 25.0.1 came out, but the app user_external used to authenticate Linux/Unix users hasn’t been ported to 25.x yet, thus if you use a system login for access, until this is corrected there is no way to authenticate.

     I have filed a bug report on Github but if history repeats, it will be dutifully ignored, time will tell.

     I am in the process of upgrading Nextcloud 24.0.7 to 25.0.1, this likely will take awhile as it involves major changes in database schemas, etc.

     However, one plus, I discovered part of the reason Nextcloud has been so slow.  I’ve been using memcache and I’ve been using PHP 8.0, however memcache is configured to use aPCU and aPCU module was not enabled in PHP 8.0 so this broke memcache.

     For some reason previous versions did not complain about this configuration issue, but now that this version did, it is corrected.

     I apologize for the interruption to web services today.

     For reasons unknown, because there were no new dependencies that should have pulled the Ubuntu version of Apache2 in, Ubuntu installed their version of Apache2 which does not have all the necessary capabilities our locally compiled version has and broke our web services.

     I had to uninstall theirs and recompile and install ours, a process which took around fifteen minutes.

     All services are fully operational.

     At 12:54 kernel upgrades are completed.  Sorry this took so long but one physical host failed to boot properly.  The Nvidia graphics card did not initialize properly and systemd brought the machine up into single user mode but not multi-user mode so I could not access from here.  I had to drive to the Co-Location facility which is 22 miles away and there was construction work at the I-5 / I-90 interface that made that take longer than it should have.

     I am still checking NIS/NFS mounts, but all the basic subsystems are up and running.

     I’ve tested 6.07 and 6.08 and both seemed to have resolved the issue with squashfs, so will be doing kernel upgrades.  Provided they haven’t introduced new bugs, this should eliminate all bugs of any operational consequence.  There still is an issue with startup of centos7 and scientific7 but that only generates an error message and is of no operational consequence and, according to developers, that bug will be addressed in 6.2, so still a ways off.

     This will affect off of Eskimo North’s services including our public services https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/ as well as our own website, all the sites we host and virtual private servers.

     Downtime for any one service should not exceed about ten minutes except for yacy which takes around 40 minutes to rebuild it’s database after a reboot.  This operation should be completed by midnight.

     The FTP server is restored and somewhat better secured.

     I do not know what exploit they used because all of the known exploits for wu-ftpd I had fixed, so this is one not known, however, it would appear they only had anonymous user permissions as nothing outside of the ftp directory was disturbed.  Since the server mounts the ftp directory off of another file server via NFS, I have chattr +i the files and directories they should not be allowed to change on the host machine.  Since chattr does not work across NFS there is no way for them to change it even if they were to somehow get root access so this should largely secure the server.  I am going to create a apparmor profile for it just as an additional security measure.

     Someone apparently found an exploit that allowed them to really trash the public directory of our ftp server.  Consequently, anonymous access is extremely restricted until I’ve been able to restore the directories from backups, modify some file permissions and create an apparmor profile to limit potential damage in the future.

     6.0.6 did not fix either of the outstanding bugs affecting the current kernel on our servers, therefore I will NOT be doing a kernel upgrade this Friday.