Around May 22nd, I became aware that our server was sending out spams via Comcast automatic notification and MX-Toolbox notifications but was unable to determine the sending account owing to the fact that I had not received a single complaint from an end user receiving said spam nor hard either of these automated tools given me enough information to determine it. Comcast in their extremely finite wisdom expunges this information from the headers to protect the privacy of their customers. How the hell they expect us to identify the sender without this information is beyond me.
At any rate on the 28th, I finally saw our outgoing mail server had been clogged by one customer, and further examining some of the queued mail found that it was a phishing scam. This particular customer is a long time customer that I know would not do something like this so it was evident his account had been compromised. I changed the password, contacted the customer with the location of the phishing scam on his website, and applied for delisting of the three RBLs we were on a result.
Two of those three removed us from the RBL but SORBS still has not. I’m still going back and forth with them trying to get this taken care of. I’ve purged all the spam queued on our mail servers and all the bounces attempting to return from spammed addresses.
The mail server is now caught up. The only location I am aware of that is still blocking us is Starbucks. Earlier Yahoo and others were throttling incoming mail from our server resulting in legitimate mail being bounced. But mailq is now clear save for starbucks.
Upgrading to Ubuntu 18.04 LTS seems to have broken mail delivery from the web server, ftp.eskimo.com. This does not affect webmail programs because they are all configured to talk directly to mail.eskimo.com rather than relaying, but it does affect scripts that you may have on your website to send mail to you.
Postfix broke in a way that some mail just disappeared into a black hole. Until I can figure out precisely how it broke (not easy as it is not logging any errors or bouncing the messages) I’ve installed null mailer, a very simple relay only mail server. However, even it is introducing random delays in delivery.
Just wanted you all to be aware of this and I am working towards a solution.
I have tested all machines and all are operational.
The reboots have been completed. I will be checking all servers to make sure NFS properly mounted and NIS properly bound, this will take several hours, but things should be operational now.
To load new kernels that correct some issues with the previous kernel, we will be rebooting all our host machines and most of the guest machines starting at 3AM approximately.
The expected downtime is about five minutes per machine, most things should be complete in 15 minutes, however, occasional NFS problems arise after a reboot and it will take several hours to check all machines for these but most services should be back up by 3:15AM.
Just found out today that one of our customers, Wayne Moddison, passed away in November of last year.
I did not talk to him frequently but he was one of the most pleasant people you could ever want to know and I’m very saddened to hear of his passing.
Taking centos6.eskimo.com down for about 45 minutes to image the virtual machine.
CentOS6 is up and operational on i7-6700k hardware. It is no longer vulnerable to spectre or meltdown exploits and it was the last machine that was vulnerable so all of our systems are now protected from these exploits.
I will be taking CentOS6 down for about 45 minutes tonight, around 1:30AM, to image the machine on the new hardware.
The move of Centos6 is going to take longer than I originally anticipated due both to the size of the machine and the fact that I did not have enough space on the host I had originally intended to move it to so needed to start over moving it to a different machine.
Based upon the rate the copy is going and the amount of data remaining to me moved I estimate another 25 minutes or so. Centos6 should be available again by 16:30 Pacific time.
All machines here have spectre and meltdown exploits mitigated except for CentOS6. Redhat has kicked out a new kernel that addresses the exploit on i7-6700k hardware but does not address it on i7-6850k hardware which is officially not supported by Redhat 6.
Consequently I am in the process of moving this host to an i7-6700k based server. This will take about 1/2 hour during which CentOS6 will be inaccessible. Please use a different shell server during this time. All x2go supported windowing environments are supported on MxLinux and Ubuntu.