We have been occasionally running into the max_connections limit on mysqld.  I’ve bumped it up a few notches so that issue should be resolved.

     Mint is back up and running however not all of the applications are installed and configured.  In particular postfix isn’t setup yet so don’t try to send mail, it will just get stuck in queue or rejected outright.  And on the subject of mail, most mailers are not yet installed.

     Unfortunately, there was no clean way to upgrade from Mint 14 to 17.2 cleanly except to do a clean install.

     Our shell server mint.eskimo.com is presently down for an upgrade to Mint 17.2.  If you need a debian based server in the meantime please use the newly upgraded ubuntu.eskimo.com or debian.eskimo.com.

     We were hit with two denial of service attacks today.  The first was close to 9AM and only lasted a few seconds.  The second was between 3:00pm and 3:45pm lasting 45 minutes.

     After analysis of the traffic, I determined that it was not possible to block this type of attack without disrupting legitimate traffic.

     The real problem is that our router is not able to handle the volume of packets that can be generated easily these days.  Simply put, it doesn’t have enough CPU to examine 100,000 packets per second.

     I ordered a replacement router today, the existing unit is one that we have used since 1995.  The replacement should be here in 7-10 days and should handle more than 10x the traffic in terms of packets per second than our current router is completely swamped by.  In addition, it will change our edge interface from 100mb/s half duplex to 1G/s full duplex.

     It is possible for a denial of service attack to generate as much as 46G/s and that would overwhelm even our new router but the majority of attacks we’ve seen lately have not even been 100mb/s and instead of exhausting bandwidth are exhausting router CPU.

     Our shell server, ubuntu.eskimo.com, has been upgraded to version 15.04 Vivid Vervet.

     Unlike the first attempt when 15.04 first came out, this upgrade went almost flawlessly.  The only boot problem being some features missing in the kernel built that this version requires and the non-existence of postgres user and group (it is in NIS but the start-up sequence with systemd doesn’t have NIS enabled at the time it tries to create the necessary log files).

     These two things have been corrected and now the boot-up is 100% clean and mighty fast (takes about two seconds from launch to login).

     The denial of service attack lasted between about 4:00PM and 4:15PM Pacific Time.  It ended just as I was successful at getting an Isomedia tech on the phone so we weren’t able to identify the source.

     It took me 3 hours to make a trip 22 miles in each direction.  Mariners game screwing up traffic on the way there, concert at the Key Arena screwing up traffic on the way back, plus two accidents.

     Someone launched a denial of service attack against our router at the co-location facility.  Even though it is presently up and routing traffic, it has stopped allowing administrative connections so I can neither monitor traffic or adjust firewall rules.

     For this reason I am making a trip to the co-location facility to reboot the router, this is the only way to restore administrative functionality, and will be out of the office for a couple of hours.

     Cell phone numbers become available and legal for telemarketers to call this month. (Good to know congress is looking out for your interests).  When they call, you will get charged your normal rates for minutes.

     To avoid them waisting your time and money, add your cell phone number to the National Do Not Call Registry.  To do this call 1-888-382-1222 from the number you wish to add.  The whole process takes about 20 seconds.

     Then, 31 days after you have registered, if you receive any telemarketer calls at that number you can file a complaint with the FTC and the offending caller can receive a $500 fine, per call.

     I’ve added Mosh to all of the Linux servers at a customers request.  Mosh is a specialized shell for mobile users that works even as a call user switches cells and IP addresses.  It is designed to work with high latency and intermittent connections.

     Maintenance on all the machines is now complete.  They’re all current with patches and updates and the latest kernels and all imaged and up and available.