I am done doing server maintenance for the night. I kept Ubuntu down longer than expected in order to troubleshoot and fix an issue with the greeter. I had to uninstall and reinstall a lot of things to get it fixed. Hopefully did not break something else in the process.
I am going to be taking mx1, mx2, ubuntu and ftp/www down for about twenty minutes each, one at a time, starting around midnight, to image the machines in preparation for an upgrade to Ubuntu 17.10 on the 19th.
Server reboots all done. Unfortunately, it did not fix the ethernet udp offload code.
I will be rebooting the host machines (which in turn will reboot all the servers) to load a new kernel that fixes some issue with UDP offloading on the ethernet port that was introduced with the last kernel upgrade. Outages should be brief (approx 10 minutes each machine).
The apache web server has been upgraded to 2.4.28. This upgrade primarily fixes a problem where by it was possible to obtain random memory data under certain circumstances. Unfortunately, Apache caching still seems to occasional serve the wrong page.
Maintenance work for the night is completed. We’ll will have individual servers go down tomorrow evening and Saturday evening for about 15-30 minutes each in order to back up those servers in preparation for upgrade to Ubuntu 17.10.
This will affect mx1, mx2, ftp, ubuntu, and several other machines that are new and not yet fully in service.
I will be rebooting servers in about an hour to upgrade kernels and test a fix to systemd. If the fix works this will be a short interruption of about the minutes. If not it may take approximately 45 minutes.
I want to serve as much content out of memory as possible because you can’t get faster than that. Google’s pagespeed module did not play nice with suexec or phpexec so I had to take that back out. Memcached also occasionally corrupted pages, so got rid of that. Now what I’ve done is created an in memory file system and used apache file caching. Please notify firstname.lastname@example.org if you see any bad behavior like corrupted css files.
Sorry for the somewhat flakey operation of the web server today.
I tried to implement two different server optimizations, Apache’s caching in concert with memcached. This mostly worked but occasionally corrupted pages so I had to back it out.
Then I tried to run mod_pagespeed, a Google Apache optimization module, but it did not peacefully co-exist with provisions we use to run dynamic code under the user id of the user that owns that code resulting in all sorts of random permission errors. So ultimately I had to back that one out as well.
The server reboots have been completed. No further interruptions are anticipated.