There will be an outage of approximately 3 minutes Feb 17th close to noon in order to load a new kernel.
Maintenance Completed – Minor Problems
Last night’s reboots did not go entirely cleanly.
It took three reboots of Isumataq, which hosts home directories and a number of virtual machines, for it to come up cleanly. The first two times it did not properly start networking and manually attempting to start it did not work. This is not a new problem but one that has been ongoing since 16.04 LTS which introduced systemd which unfortunately while it sped up booting considerably, introduced quite a few bugs, all of which have not been entirely squashed yet.
Debian NFS did not sync properly when the mail server returned to service and so to restore proper functionality I had to reboot that tonight.
Ubuntu stopped talking to the network even though the kernel was still up and operational and I could get into the machine through the virtual machine manager. I rebooted it to fix.
The current state of meltdown and spectre vulnerability fixes on our network is that meltdown is fixed on all of our hosts both physical and virtual, spectre both variant 1 and variant 2 is fixed on all of our physical hosts and ALL ubuntu based hosts. Spectre variant 1 is fixed on almost all of our virtual hosts but spectre variant 2 is fixed only on ubuntu based hosts.
People ask me why I have moved and am continuing to move our services off of CentOS and to Ubuntu, this is an excellent example of why. To the best of my knowledge, ubuntu is the ONLY Linux distribution that has fixed meltdown and BOTH variants of spectre. They are usually first out of the gate with exploit fixes.
Maintenance Feb 14th 1:30AM – 3:30 AM
I am going to be rebooting servers tonight to load new kernels which fix meltdown and spectre vulnerabilities. Then I will be taking mail and web down for about 25 minutes each to image the machines so that if it becomes necessary to restore them in the future, it will not be restored to a vulnerable state.
Web Server Maintenance
I will be taking the web server down for approximately twenty minutes after midnight to image again with missing SSL certificates in place.
Maintenance Complete
Maintenance on OpenSuse has been completed. Everything has been restored to service.
Maintenance Completed vps1
Maintenance has been completed on vps1. It has been restored to service.
Maintenance Completed on Scientific and Vps4
Maintenance has been completed on the shell server Scientific and on the virtual private server vps4. Both have been restored to service.
Centos7 Maintenance Completed
Maintenance has been completed on Centos7. It has been returned to service.
vps1 and vps4 Maintenance
I will be taking vps1 an vps4 down for about twenty minutes each in order to image the machines for possible future restoration.
Centos7, OpenSuse, and Scientific shell servers down for maintenance
The shell servers Centos7, OpenSuse, and Scientific are down to be imaged. This will take about an hour in total.