I am planning on upgrading Debian from Bullseye 11 to Bookworm 12 tonight. This will require a reboot at the end of the upgrade process.
No Service Interruption Tonight
After a bit of experimentation, I found having the CPU voltage follow the CPU frequency resulted in about 40 watts of idle power savings but occasionally caused CPU data errors. The latter is not acceptable so that part of the plan has been eliminated. I will be installing the new server tonight though it will take time to migrate functionality to it, but it will not require a service interruption.
Maintenance Work July 1st 10pm-2am
I am planning on doing some major maintenance work tomorrow. First will be the installation of our newest server. This should not disturb existing services but there is always the potential for operator error.
The second thing I will be doing, and this WILL result in downtime for various services, is to take the various physical servers down for some time to do some BIOS tuning. The idea is to switch from a fixed CPU voltage to a variable voltage that changes with clock speed so that during times of low load on a given machine, power consumption and heat production will be less.
CPUs require more voltage to be stable at higher clock frequencies. All of our modern machines change clock rate with load up to some defined maximum, but presently are using a fixed CPU voltage which is suitable to the highest load. This change will have them change their core voltage with clock frequency as necessary for the frequency they are operating at during any given time. This will require some benchmarking and load testing to optimize.
This will not change the peak capabilities of the machines significantly. It may allow them to clock slightly higher than normal for brief intervals if hit with a sudden load when idle and cool because of the thermal mass of the CPU cooler, but mostly it will affect only the low load and idle conditions.
Russian Hacking
You may have heard on the news items about Russians hacking into various government and business organizations, The Department of Energy for example.
Here, I have seen an approximately 3x increase in hacking activity in the last month, the majority of it in the form of e-mail phishing scams, and a non-trivial percentage from Russian hosts. This does not necessarily mean the hackers are Russians, they could be any hacker that has gained access to the Russian servers. If you have all of your life savings drained from your bank account, it really does not matter, so to prevent that please follow these instructions carefully:
If you receive an e-mail allegedly from us or your bank or other financial institution requiring you to click on a link and provide authentication information for any reason, DO NOT DO IT. Instead, if you THINK the mail may be real, then go to that institutions website directly, NOT by clicking on the link in the e-mail (which may take you some place other than advertised) but designed to LOOK like the target site, and check directly with that site. When you reach the target site, make sure your web browser has the lock symbol indicating an encrypted site AND that the domain is correct.
Mail, Web, New Server
Web and SSL Mail were temporarily interrupted earlier this evening as I replaced the SSL certificate for *.eskimo.com that was due to expire on July 6th.
Finally got the new server to boot consistently off of RAID. Main issue was EFI system disk needed to be a physical direct device and not RAID or a logical device of any sort. This makes sense as it is shared between multiple operating systems or at least can be. Other than that, also two BIOSes from Asus were bad, on the third try got one that mostly works, I say mostly because it still is not entirely without flaws, like when two boot devices share the same UUID it will only show the first one, but otherwise it’s working. So will be installing this at the co-lo shortly and then start moving applications to it.
Mail Trouble
Mail is fixed now.
Kernel Upgrades Completed
Kernel upgrades completed successfully. All services, nfs mounts, nis bindings checked and verified.
Kernel Upgrade Tonight 11pm-12pm
I will be performing a kernel upgrade requiring reboot of all of our servers starting at 11PM. If all goes well we should finish by 11:30PM with boots and midnight by checks to make sure all services properly started, NIS bindings and NFS mounts properly completed, etc.
This will be to kernel 6.1.33. Barring the release of some substantial performance gain, I plan to stick with 6.1.x long term kernel release until at least the next LTR release. To date this has been the best performing long term release kernel we have experienced.
This one will be compiled somewhat differently, a kernel upgrade failed on our newest server before I could put it online because the nvram mod did not load, so now I am compiling the /dev/nvram support into the kernel rather than separately as a module to avoid the potential for a future recurrence.
This will affect both our paid services such as virtual private servers, web hosting, e-mail, and linux shell accounts, as well as our free services, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/, and https://yacy.eskimo.com/.
Most services should be down for less than ten minutes save for yacy which takes about 40 minutes to rebuild an in memory database after reboot.
New Server
I’ve got the new server physically together. It posts, boots, and everything tests out good. Now I’ve got to load an OS in order to find optimal settings for performance.
Someone suggested a good name a while back and I’ve misplaced it. If you know who you are and could resend, please do.
Microsoft Issue Revealed an Issue with Our Mail Servers
Microsoft’s outlook mail service, servers, starting with *.protection.outlook.com apparently added some servers without adding an SPF record for them.
This revealed a problem with our spf client. It is supposed to not reject but only flag the incoming mail and then it should go to your spam box. Unfortunately even though it was configured not to it is rejecting mail.
I have temporarily disabled SPF checking to allow outlook mail used by many large corporations to get through.
I have contacted Microsoft’s tech contacts to make them aware of their missing SPF record for some of their new servers.
And investigating why our SPF client is rejecting mail when it is not configured to do so, it appears the format of the configuration files has changed substantially since we installed it and it did not update the configuration file when updates updated the software.
The new software has added some additional capabilities not present in the old hence the need for new configuration.
I am uninstalling all the spf related software and re-installing to correct these issues. In the meantime we will still advertise SPF records for outgoing mail but incoming mail is not being checked so be very careful if you receive any mails asking you for authentication as they are more likely than not forged. Do not give banking info or login credentials to ANY e-mail asking, do not follow web links in e-mail.
I will send additional notice when new software is operational.