I will be taking the main server holding home directories down for up to two hours Sunday to replace a failed drive in a RAID array.  No data is lost as the result of this drive being ill since all data is duplicated in the RAID array.  Things will be slower for the twelve hours or so after replacement as the RAID resync’s the new drive.  Because virtually everything depends upon home directories pretty much everything except private virtual servers will be out of service during this interval.

     I’ve repaired Mint, at the heart of the issue was the glibc package name was somehow missing a ‘1’ at the end and so package version mis-matches resulted.  I had to force a manual install and then re-apply several hundred updates.  Because chasing this down was not easy I am making a backup now that it is fixed so I don’t have to do this again.  Mint should be available by approximately 1900 Pacific Standard Time March 1st.

     A routine upgrade exploded today on Mint leaving packages in an inconsistent state of dependency loop hell that I can’t easily fix so I’m going to restore from a backup and then re-apply missing updates.  This will take an hour or more to complete.  It will be unavailable during this time.

     I apologize for the brief interruptions of service around 4pm and 4:20pm today.  Our router was being attacked by a botnet attempting to brute force guess passwords on it.

     To combat this I was able to employ fail2ban, the same software we use on our hosts, as the router is based upon Debian stretch.  But because the router uses an overlay file system I had to change the default logging location to a place where it would save across boots and needed to test this requiring a couple of reboots.

     The attacks from Microsofts servers have not have not completely stopped but they’ve slowed considerably.

     Looking specifically at what is in the logs tripping fail2ban, I find that they are trying to send mail from non-existent domains.  This is something our servers will not accept anyway, so for now I’ve whitelisted the IP space involved which should let legitimate mail through.

     E-mail from outlook.com is having issues right now because at least 750 of their servers are infected with or affected by some kind of botnet malware.

     Our servers are equipped with fail2ban which watches logs for abusive behavior and locks those IP addresses out for a period of time.  The attacks are still ongoing so releasing these blocks do no good and potentially harmful if malware were to spread to our servers or our customers machines.

     I have notified Microsoft of this via their tech and abuse e-mail contacts and I have forwarded a list of the 750 servers thus far affected.  This can affect other domains as they contract with Microsoft for mail delivery.

     Reboots are completed, everything at eskimo.com back operational.

   I plan on rebooting all servers around 11PM tonight.  This is necessary to make various updates, some of which are security related, operational.

     Downtime for any one machine should not exceed 10 minutes.  I expect it to go smooth as other than routine updates supplied by OS vendors, I haven’t made any significant changes.

     Apparently someone is calling my customers with scam calls representing us.  I’m not calling people at random.  If you happen to get one of these calls, please check your caller ID and collect the number and any other identifying information you can get.

     Thank you.

     Graphical Desktop sessions are now working on Centos-Stream.Eskimo.Com via X2go, RDP, VNC, and via the web (host or terminal) Guacamole services.