On mx1, spamassassin not only is not running but trashed itself to the point where it won’t start.  I’ve taken mx1 out of service and am restoring it from a backup image.  I did store the postfix spool first so no mail will be lost.  Incoming mail can still be processed by mx2 while this is being restored so there is no loss in functionality and traffic right now is low enough that capacity shouldn’t be an issue.

     SpamAssasin’s daemon, spamd, died at some point on mx2 allowing a lot more spam than would normally make it through.  It has been restarted.

     I am in the process of replacing the existing mail servers with new servers with newer anti-spam measures.  The existing servers are three years old now and spammers have adapted to get past the old measures.  The new servers will rely more heavily on Bayesian filtering (which is adaptive) with a much improved back end capable of holding a lot more sample data thus enabling more accurate filtering.

     I have removed outages-list and eskimo-announce.  This news section replaces both of them.  Neither have been used for years legitimately but spammers have been forging source addresses against them in order to cause the bounce to deliver to the forged address.  Since this could potentially get our services black listed for back scatter, I’ve removed them entirely.

     If you wish to have this information delivered to you via e-mail you can use the subscribe function on this page.  Since submissions are by web only using authentication, back scatter can not be generated the way they could with the previous mail lists.

     While I am still working on new mail servers, I’ve updated postfix on the current incoming servers, mx1 and mx2, and client server mail, to the latest version which is postfix 3.0.1.

     The primary reason for doing this is that it added settings for how long mail could be kept in queue before being rejected.  This was necessary because of people sending mail to domains which had MX records but non-functional or non-existent servers at the IP addresses those records resolved to.  It was also necessary because of sites like Yahoo and Google which sometimes won’t accept e-mail as fast as our servers are trying to send it and, if this goes on too long, causes a backlog in our mail queue.

      Still working on new mail servers for better spam control and over-all functionality.  Working with Ubuntu 14.04 LTS rather than Fedora and so far it’s going much more smoothly.   Ubuntu builds low latency kernels as part of the distribution so don’t have to do that by hand, synaptic is far better than any of the package installers provided with Fedora.

     I’ve got most of the necessary pieces installed on one new server, now just have to configure everything.  Once I get one setup, it should be easy to clone the new configuration.

    Our web server got stuck in some sort of loop where it was consuming a lot of CPU but not serving pages effectively at around 1:38 PM.  It slowed the machine down to the point where it was unresponsive to commands.

     At 2:46 PM I gave up trying to troubleshoot and rebooted the machine after which it returned to normal operations.  I was unable to find anything out of the ordinary in the logs that indicated the reason for it’s behaviour.

     After spending a day working on getting a new Fedora 21 server up, I think I’m going to go with Ubuntu instead.  The install tools don’t work properly in Fedora 21, and they’re going to replace it with something new and probably even more broken in Fedora 22.  Too much revolution instead of evolution in this distribution at this time.

     Spam has increased and new techniques spammers are using are getting past the old version of spamassassin we have in place.  The version of spamassassin in place is old because it’s the version provided by CentOS 6 which those servers are based upon.

     I plan on replacing the servers with new Fedora based installations because Fedora is always current and thus it is easy to compile and install current applications like the most current version of SpamAssassin which is 3.4.0.

     The new version of SpamAssassin has a new Bayes engine that has a database back end instead of text files allowing it to store much more information and still process mail in a timely manner and at the same time be more effective.

     It has a number of other improvements as well.

     I will be making the changes first to the incoming mail servers since most spam originates out there on botnets comprised largely of compromised Windows machines and more recently personal routers that are not properly secured.  Once those are up and running, I will put together a replacement client machine.

     The Julia programming language has been added to all of our shell servers except debian.  I can not find a complete stable release for that platform.