Backups of all virtual private servers has been completed.
I will be taking virtual private servers down for about 15-20 minutes each to image the machines (a form of backup that makes a copy of the machine state).
If you are doing anything critical please e-mail firstname.lastname@example.org right away.
System reboots are completed. NFS and NIS remounts and binds verified. Everything is operational. New microcode for i7-6850k still broken but I’ve gathered information Canonical needed so hopefully this got us closer to a fix.
I will be rebooting servers early Saturday morning, shortly after midnight. The server which houses home directories will be rebooted possibly more than once.
Most of the servers will be rebooted to load a new kernel.
The server housing the home directories will also be testing a new microcode package from Ubuntu to see if it has corrected a problem with the 6850k multiplier register. If the package performs properly there will be only one reboot of that machine, if not a second reboot to back out the new code will be required.
The shell server fedora.eskimo.com has been upgraded to Fedora 28. Gnome Flash back and KDE Plasma no longer exist with this distribution so are not available on X2Go, however Mate, LXDE, LXQT, and XFCE still work.
Nextcloud has been updated to Version 14, however the database was badly broken. If you had files there, login, then let me know and I should be able to add them back into the database. You will have to reset any application passwords and the like.
NextCloud is updated to version 14, tonnes of apps now but the upgrade blew up and I had to do a mostly fresh install. If you used your system login here, that should work but initially you won’t see your files. If you created a login then you will need to re-register. Once you have logged in let me know and I’ll attempt to add your existing files to the database. If you re-register be sure the login you use is the same as your old login.
I updated some out of date applications on Nextcloud and after I finished it would no longer provide a login, just a blank white page. I am attempting to diagnose and repair.
On the mail server this morning we were hit with a brute force password guessing attack by a large botnet. For some unknown reason fail2ban died and then the bots swamped the dovecot imap server to the point of totally wedging it.
Fail2ban is restarted and has locked out most of the botnet IP addresses so that the mail server is no longer choked and can again function.
We are having problems with the mail subsystem. I do not yet know the cause but am working to determine it. Service on some hosts besides mail may be interrupted briefly during the troubleshooting process.