MailSpike has dropped us from their blacklist, we’re still on UCEProtect1 but that’s more of an extortion list than a blacklist ($119 to be delisted, I wonder how many takers they get).
Yahoo is still blocking us. They tell us it will come off in time but won’t say when.
At this point we are removed from the SpamCop black list, but still on the mailspike and uceprotect1 lists.
Two more accounts were compromised, these I found no obvious signs of brute force password guessing. One was used to send a large amount of spam last night, again getting us added to SpamCop and MailSpike blacklists.
I went through the logs, found 183 addresses belonging to this spammer botnet, and added them to our firewall.
This morning another account was used, this time only six addresses were involved, I added those to the firewall rule.
I’ve only been able to contact the second customer, but in his case he used the same password at many places. If you’ve been following the news, accounts at Twitter, Facebook, Yahoo, and Google have all been compromised in the last three months, so you can see the importance of using different passwords.
I am working on eliminating any opportunities for these folks to bash password guesses such as ssh, ftp, telnet, rlogin, etc. It will take some time to do this.
After I finish this I intend to run Crack on our password file to find any easily guessable passwords and notify customers of the need to change those.
All of this will not do any good however if the same good passwords are used at other sites which are not secured.
I broke the web server for a short period this evening attempting to install a security suite that would stop password guessing at WordPress login prompts. That software turned out to be badly broke in spite of a five-star rating.
I found different software that functioned without killing the server.
Webmail now has a plugin called “Lockout” installed.
It is configured so that five failed login attempts on the same username will ban that username for fifteen minutes and five failed login attempts on different usernames but the same IP address will ban that IP address for fifteen minutes.
With fail2ban in place, the people attacking the mail system moved to webmail and used it to get the web server banned so that webmail was intermittently broken.
I’ve exempted the local servers so this will no longer happen and I’m working on installing fail2ban on the web, ftp, and shell servers, as well as a necessary plugin that will cause Squirrel Mail to log the address so we can ban attackers of webmail.
I kind of expected an onslaught of bad activity after Microsoft discontinued support for Windows-XP. They’ve basically just added a large amount of vulnerable CPU capacity to the web for these people to form new botnets and launch attacks from.
I have worked through all the Blacklists that were logged, requesting de-listing for our mail server. There are a couple where no mechanism exists to request a de-list, it’s automatic and time based.
There are also some mail sitting in queue because the destination server thinks we are still blacklisted because of the way real time black hole lists work. These lists use DNS to distribute status information for each IP. DNS is cached, and so even when an RBL de-lists our server, a site that our server tried to deliver to while we were listed still sees us as listed until the time to live (TTL) expires for the DNS zone in question.
At this point, our server is fixed and better secured and I’ve contacted every RBL that has a manual list method.
I have added fail2ban which is a program that watches logs for authentication failures and bans the associated IP address after five unsuccessful attempts for ten minutes.
I have added some rate limiting to postfix which should be not affect legitimate mail but will limit the damage in the event accounts are compromised. I’ve also made it a little less forgiving of bad behavior common in some of these spam botnets.
I am watching the logs for rejections and contacting sites and blacklists and requesting our server to be removed as I become aware of them.
Some of the blacklists do not provide a manual removal mechanism and require a fixed interval of time to pass without receiving spam before they will remove our server.
If you receive a bounce message, please read the message, it usually contains a URL where you can submit a request for removal. I am working off the logs as fast as I can.
Our server is on a number of real time black hole lists now because of the spammer that compromised two accounts here and badly abused our server.
Those RBLs which have a mechanism which allows delisting to be requested, I’ve already done so, but some like sorbs, spamcop, and a handful of others only delist an IP if no further spam is received after a set period of time.
There is NOTHING I can do about these. Please complain to the sites that use them.
I am working on fixing the weaknesses that allowed the accounts to be compromised and the server to be abused. This is non-trivial.
A second account was compromised via brute-force password guessing. I’m working on getting fail to ban configured on the mail server (and later other servers) to stop this.