Received a report that people were unable to login to Debian this afternoon.  I found that ypbind had died but there was no indication in the logs as to why.  I restarted it and logins are functioning normally again.

     Slow service early this morning and the temporary unavailability of mail.eskimo.com was the result of a denial of service attack where upon our name servers were used as amplifiers in a denial of service attack aimed at us.  I had to lower the external view rate limit because of this, hopefully it is still adequate to service legitimate requests.

     There are aspects of this attack that I do not understand.  They forged an address of 204.122.16.248 from outside (udp packets so no three-way connect) and directed requests at 204.122.16.8, so our name servers would attempt to reply to 204.122.16.248 but there was no host on that IP address and the result was that our router didn’t know what to do with it and it overloaded it logging what it considered “Martian” packets.

     The puzzling aspect of this is I have a firewall rule that SHOULD block all traffic from an external interface which has an internal address.  I was able to mitigate the attack by blackholeing 204.122.16.248 at the name servers and rate limiting responses.

     Sorry I neglected to post the completion.  Reboots went smoothly, only Debian failed to properly remount file systems.

     I will be starting reboots now.

     I will be doing another round of reboots sometime after midnight Friday night / Saturday morning and then some machines will be out of service 20-30 minutes for imaging.

     We had a strange data issue this afternoon.

     Around 16:00-16:15 I saw data flow drop to about half of what it had been averaging, then between 16:15-16.20 it dropped to zero.

     By around 17:00 everything was mostly back to normal.  Our bandwidth graph still is not logging but we can get out to the rest of the world.

     Found out there was a problem with power to an entire breaker panel at the Isofusion Bellevue co-location equipment facility that took out their core router.

     Rwhod was broken so mint appeared down in ruptime even though it was up.  This has been repaired.  Most of the software previously installed has been re-installed.  Java is version 10, and is the Oracle release as the included jdk-10 was missing libraries.

     Mint is essentially repaired.  It works with the following desktop environments: KDE. Gnome, LXDE, LXQT, XFCE, and Mate.

     Gnome is a little bastardized.  The default configuration includes no logout button, obviously a problem.

     Adding one is non-standard.  Instead of the normal ALT+Right Click, Mint’s bastardized Gnome requires Windows+Alt+Right Click to add to a panel.  You can use that to add a Logout button to your panel then you will be good.

     All shells should be operational and most of the mailers on the other .deb derivatives are present on Mint.

     As is oh so typical of Mint, the last in place upgrade to Tara left the graphical environment non-functional, so I will be taking Mint down for a while to perform a fresh re-install from the Tara ISO.

     NextCloud is up and running again, https://www.eskimo.com Web Apps -> Nextcloud

     – or – https://www.eskimo.com/nextcloud/

     I was up all night until after 8am getting this to work, and in the process learned something important about innoDB, you can’t just copy the files like you can MyISAM which means I’m going to have to come up with a different method of backing up the database.  Fortunately, I learned it on our data not yours.