Incoming Mail Servers

     The incoming mail servers mx1 and mx2 have been replaced with Ubuntu based servers.  Still using postfix as the MTA but it’s a newer version as is procmail and smartlist.  DKIM is now supported as well so that’s one more tool to stop bogus e-mail.

     Please let me know if you see any more e-mail not scored by spamassassin.

Apache and Apr and Apr-Util Upgraded

     Apache httpd has been upgraded to version 2.4.27, apr has been upgraded to 6.0.2, and apr-util to 6.0.0.  Should not be much change in functionality, there is some improvement in memory footprint.  The less memory code uses, the more is available for cache.

Web Server Stopped Talking to Network

     Our web server stopped talking to the network today.  I have not yet had a chance to do any forensics but I was able to connect to the machine via the Virtual Machine Manager but not ssh or www.  And from what I’ve been able to determine so far even internal connections between applications such as the web server to the mysql database failed.

     I have rebooted it, which restored network connectivity, and will investigate further.

 

Isomedia Co-Location Facility Network Maintenance August 31st

ISOMEDIA will performing scheduled network maintenance on 08/31/2017 at 12:00AM PDT to 04:00AM PDT. During this window, there will multiple periods of increased latency and packet loss, as network protocols re-converge.  These periods may last between 5 and 15 minutes.

All times are estimates based on expected outcomes of the work being performed and previous experience performing the same or similar work. There is always the possibility of some unforeseen bug, or problem, that could extend the maintenance time or cause a disruption in connectivity. Administrators will make every effort to correct the problem, or implement the back out plan quickly, if something does occur.

Spam Scoring

     Spammers have figured out how to bypass spamassassin spam scoring rendering the bulk of our spam filtering capabilities non-functional.  I have not been able to determine how they are doing this yet.  It is happening with both incoming servers so it is not a per server problem.  I’ve also found other people are experiencing this as well but have not found any solutions elsewhere either.

Denial of Service Attack Self Inflicted – Operator Malfunction

     Upon closer investigation, what appeared to be a denial of service attack triggering rate limiting on our name servers and crashes was in fact self inflicted.

     At some point I accidentally copied over the virtual domain configuration file for the slave name servers onto the master name server so it was effectively no longer a master.  It is a hidden server (so the master can not be attacked) but since data from the master server was no longer available, everything went fine until the zone’s on the slaves expired.

     At that point all the slaves contacted the master trying to refresh for each domain and since the master had no data to serve at that point, they could not, so they kept trying until they triggered rate limiting on the master.  Then the slaves did not know how to handle that and just died.

     Once I discovered this I was able to restore the virtual domain configuration file from backups and then the slaves updated their zone files successfully and all was good once again.

 

Name Server Denial of Service Attack

     Someone launched a denial of service attacks which repeatedly caused three of our public name servers to crash earlier but the downtime was less than a minute on each as I have scripts in place that check for the proper operation of our name servers once each minute and relaunch them if they are inoperative, and at no time were all name servers simultaneously out of service.

Maintenance Complete

      Maintenance is completed.  I need to check all the shell servers to make sure they have properly remounted NFS file systems but I will do that when I get home.

      Sorry this took so long, a combination of bugs in Ubuntu’s start-up script and operator error, at one point I powered down the wrong server.