A routine upgrade exploded today on Mint leaving packages in an inconsistent state of dependency loop hell that I can’t easily fix so I’m going to restore from a backup and then re-apply missing updates.  This will take an hour or more to complete.  It will be unavailable during this time.

     I apologize for the brief interruptions of service around 4pm and 4:20pm today.  Our router was being attacked by a botnet attempting to brute force guess passwords on it.

     To combat this I was able to employ fail2ban, the same software we use on our hosts, as the router is based upon Debian stretch.  But because the router uses an overlay file system I had to change the default logging location to a place where it would save across boots and needed to test this requiring a couple of reboots.

     The attacks from Microsofts servers have not have not completely stopped but they’ve slowed considerably.

     Looking specifically at what is in the logs tripping fail2ban, I find that they are trying to send mail from non-existent domains.  This is something our servers will not accept anyway, so for now I’ve whitelisted the IP space involved which should let legitimate mail through.

     E-mail from outlook.com is having issues right now because at least 750 of their servers are infected with or affected by some kind of botnet malware.

     Our servers are equipped with fail2ban which watches logs for abusive behavior and locks those IP addresses out for a period of time.  The attacks are still ongoing so releasing these blocks do no good and potentially harmful if malware were to spread to our servers or our customers machines.

     I have notified Microsoft of this via their tech and abuse e-mail contacts and I have forwarded a list of the 750 servers thus far affected.  This can affect other domains as they contract with Microsoft for mail delivery.

     Reboots are completed, everything at eskimo.com back operational.

   I plan on rebooting all servers around 11PM tonight.  This is necessary to make various updates, some of which are security related, operational.

     Downtime for any one machine should not exceed 10 minutes.  I expect it to go smooth as other than routine updates supplied by OS vendors, I haven’t made any significant changes.

     Apparently someone is calling my customers with scam calls representing us.  I’m not calling people at random.  If you happen to get one of these calls, please check your caller ID and collect the number and any other identifying information you can get.

     Thank you.

     Graphical Desktop sessions are now working on Centos-Stream.Eskimo.Com via X2go, RDP, VNC, and via the web (host or terminal) Guacamole services.

     Redhat has decided to retire Centos8 early.  Most of what was on Centos8 has been ported to Centos-Stream which is also available here.  So if you were using the hostname Centos8.eskimo.com please change to Centos-Stream.eskimo.com.

     If there is anything that you need that was on Centos8 but it is not on Centos-Stream, please create a ticket at https://www.eskimo.com/support/osTicket/ and I will install it for you.

     The upgrade of Mint completed uneventfully.  You’ll probably not notice much difference, it only affected 160 packages.  This is the smallest distribution upgrade I’ve ever seen for ANY operating system and makes me wonder if Mint is dead.