Network Trouble

     We are seeing significant packet loss from various locations to our servers in the Bellevue CoLo facility.  I have determined the issue to be between Isomedia’s Seattle ring router and Bellevue core router and generated a trouble ticket.  We are not seeing heavy traffic to our facility but it may be another customer in the facility is under DoS attack or there maybe hardware or routing issues.  At any rate a ticket has been generated and we are waiting on a response.

Centos-Stream Broke

     Centos-Stream is broken at present.  I tried to install the current openssl on it and unfortunately sshd was built against 1.1.1b which has some different symbols than 1.1.1k so does not work.  Pulling it out also didn’t fix for reasons I do not understand so doing a fresh install.

 

Kernel, Openssl Upgrade / Reboots Tonight

     I do not normally do kernel updates mid-week, I prefer to wait until Friday on the off chance something goes horribly wrong, to provide the most time to recover before the business week.

     However, a serious vulnerability has been discovered in openssl and I’m going to have to reboot all the machines just to get any old copies of openssl out of memory so might as well do a kernel upgrade at the same time.

     Most machines will remain on openssl 1.1.1f but it will be a patched version that fixes the exploit.  The webserver with any luck will be on openssl 1.1.1k, this is just because it’s already on a self-compiled version of openssl to get the most current encryptions.

     Normally I would start this at 11pm but because of the seriousness of this exploit, I am going to proceed as soon as I have the current software in place on all the machines but some time after 5PM.  The downtime for the entire system should be less than 1/2 hour and any given machine not more than about ten minutes.

Maintenance This Weekend

     I may or may not do kernel upgrades Friday starting at 11PM, there are two new releases since the last but I’m waiting to see if this is stable or not.  If not will do otherwise I’m going to hold off on that until the following weekend.

     The other thing I will be doing over the weekend is upgrading openssl on various servers.  There have been security exploits found in the existing version and so far the vendors haven’t fix it but there will be a new release with it fixed and I already compile it myself on a number of servers because the vendors use old versions that don’t have all the encryption suites I want.

Mail / Upgrades

     After servers were rebooted, postfix did not start ont he mail server and I did not notice before I went to bed.  This affected the ability to send outgoing e-mail.