I logged in today at about 2PM to find my INBOX full of spam in spite of just having replaced the mail servers with the newest version of everything to address this.

     Examining the spam I found it was all coming through mx2, and upon further examination, I found the reason for this was that I had forgot to allow smtp and smtps through the firewall on mx1.  That is fixed so both servers are functioning now.

     I examined the spam and found that it was all being properly scored by spam assassin but they have found ways to craft their message to not look like spam and be scored low.

     And it’s not that the new filters are not working at all, there were about 20 spams in my INBOX, but 180 had been properly sent to spam.  They just are sending such a huge volume that even 1-out-of-9 getting through is too much.

     Frustrating this is, but being on the newest operating system with all the newest software provided some more options.  I have now implemented gray listing.  What this does is when a message for a person arrives from an unknown location, it sends a temporary failure result back to the sending site and refuses delivery.

     RFC compliant mail systems will wait a brief period and then retry, but much spam software is not RFC compliant and just blasts it out once and moves on.

     I started with a short interval because I do not want to delay legitimate e-mail unnecessarily but a longer one than initially set may be necessary.

     I have installed the Flowblade video editor on Debian, Mint, and Ubuntu.  I find this editor closer to Premier and more intuitive than most other Linux based video editors I have tried.  The video will be somewhat jerky via x2go when you play it, but the finished file once downloaded or uploaded to youtube or whatever, should be fine.

     I replaced the incoming mail servers yesterday and this resolved the issue of spam not being scored.  For some individuals, even though spam is being scored correctly, it is not being placed in the spam folder.  What those have in common so far is that they all use tcsh for their login shell.

     I found that I had not installed tcsh on these servers so did that around noon today.

     I found there were some other shells missing, basically everything except sh, bash, and dash, and installed those as well. If you get spam in your INBOX and it has a header that shows “X-Spam: Yes”, please do not delete it and contact support.  Thank you.

     The incoming mail servers mx1 and mx2 have been replaced with Ubuntu based servers.  Still using postfix as the MTA but it’s a newer version as is procmail and smartlist.  DKIM is now supported as well so that’s one more tool to stop bogus e-mail.

     Please let me know if you see any more e-mail not scored by spamassassin.

     Apache httpd has been upgraded to version 2.4.27, apr has been upgraded to 6.0.2, and apr-util to 6.0.0.  Should not be much change in functionality, there is some improvement in memory footprint.  The less memory code uses, the more is available for cache.

     Our web server stopped talking to the network today.  I have not yet had a chance to do any forensics but I was able to connect to the machine via the Virtual Machine Manager but not ssh or www.  And from what I’ve been able to determine so far even internal connections between applications such as the web server to the mysql database failed.

     I have rebooted it, which restored network connectivity, and will investigate further.

ISOMEDIA will performing scheduled network maintenance on 08/31/2017 at 12:00AM PDT to 04:00AM PDT. During this window, there will multiple periods of increased latency and packet loss, as network protocols re-converge.  These periods may last between 5 and 15 minutes.

All times are estimates based on expected outcomes of the work being performed and previous experience performing the same or similar work. There is always the possibility of some unforeseen bug, or problem, that could extend the maintenance time or cause a disruption in connectivity. Administrators will make every effort to correct the problem, or implement the back out plan quickly, if something does occur.

     Spammers have figured out how to bypass spamassassin spam scoring rendering the bulk of our spam filtering capabilities non-functional.  I have not been able to determine how they are doing this yet.  It is happening with both incoming servers so it is not a per server problem.  I’ve also found other people are experiencing this as well but have not found any solutions elsewhere either.

     We are experiencing intermittent and at times extreme  packet loss where our equipment is co-located.  We have reported the issue to our data provider there.

     Upon closer investigation, what appeared to be a denial of service attack triggering rate limiting on our name servers and crashes was in fact self inflicted.

     At some point I accidentally copied over the virtual domain configuration file for the slave name servers onto the master name server so it was effectively no longer a master.  It is a hidden server (so the master can not be attacked) but since data from the master server was no longer available, everything went fine until the zone’s on the slaves expired.

     At that point all the slaves contacted the master trying to refresh for each domain and since the master had no data to serve at that point, they could not, so they kept trying until they triggered rate limiting on the master.  Then the slaves did not know how to handle that and just died.

     Once I discovered this I was able to restore the virtual domain configuration file from backups and then the slaves updated their zone files successfully and all was good once again.