Webmail now has a plugin called “Lockout” installed.
It is configured so that five failed login attempts on the same username will ban that username for fifteen minutes and five failed login attempts on different usernames but the same IP address will ban that IP address for fifteen minutes.
With fail2ban in place, the people attacking the mail system moved to webmail and used it to get the web server banned so that webmail was intermittently broken.
I’ve exempted the local servers so this will no longer happen and I’m working on installing fail2ban on the web, ftp, and shell servers, as well as a necessary plugin that will cause Squirrel Mail to log the address so we can ban attackers of webmail.
I kind of expected an onslaught of bad activity after Microsoft discontinued support for Windows-XP. They’ve basically just added a large amount of vulnerable CPU capacity to the web for these people to form new botnets and launch attacks from.
I have worked through all the Blacklists that were logged, requesting de-listing for our mail server. There are a couple where no mechanism exists to request a de-list, it’s automatic and time based.
There are also some mail sitting in queue because the destination server thinks we are still blacklisted because of the way real time black hole lists work. These lists use DNS to distribute status information for each IP. DNS is cached, and so even when an RBL de-lists our server, a site that our server tried to deliver to while we were listed still sees us as listed until the time to live (TTL) expires for the DNS zone in question.
At this point, our server is fixed and better secured and I’ve contacted every RBL that has a manual list method.
I have added fail2ban which is a program that watches logs for authentication failures and bans the associated IP address after five unsuccessful attempts for ten minutes.
I have added some rate limiting to postfix which should be not affect legitimate mail but will limit the damage in the event accounts are compromised. I’ve also made it a little less forgiving of bad behavior common in some of these spam botnets.
I am watching the logs for rejections and contacting sites and blacklists and requesting our server to be removed as I become aware of them.
Some of the blacklists do not provide a manual removal mechanism and require a fixed interval of time to pass without receiving spam before they will remove our server.
If you receive a bounce message, please read the message, it usually contains a URL where you can submit a request for removal. I am working off the logs as fast as I can.
Our server is on a number of real time black hole lists now because of the spammer that compromised two accounts here and badly abused our server.
Those RBLs which have a mechanism which allows delisting to be requested, I’ve already done so, but some like sorbs, spamcop, and a handful of others only delist an IP if no further spam is received after a set period of time.
There is NOTHING I can do about these. Please complain to the sites that use them.
I am working on fixing the weaknesses that allowed the accounts to be compromised and the server to be abused. This is non-trivial.
A second account was compromised via brute-force password guessing. I’m working on getting fail to ban configured on the mail server (and later other servers) to stop this.
The client mail server is still being beaten by bounces from misconfigured sites that, instead of rejecting mail they will not deliver up front, queue it and bounce it later.
This ran the server out of spool space sometime early this morning and as a result, some mail that couldn’t be immediately delivered, may have been lost.
Sites where immediate delivery may not have been possible tended to be large sites like Yahoo or Google which throttle the rate at which they will accept incoming mail.
A hacker used a brute force attack (kept connecting and using auth to try passwords) and successfully guessed the password of one of our customers.
This was followed by a bunch of computers bashing our mail server to send a few million spams using that customers login credentials. A botnet was apparently involved as the spam originated from many IP addresses.
I have disabled the account in question until I can contact the customer and arrange for a new, stronger, password.
I have deleted all the spam that was still in queue.
I’ve checked blacklists and, where we were listed, requested removal. One automatically removes only after seven days and won’t manually remove without a $119 extortion fee. Another has had us listed since 2007 apparently because one of our customers angered someone in IRC. That’s hard to imagine.
At any rate I’ve got things as cleaned up as they can be for now.
The following servers are restored to service:
scientific.eskimo.com
debian.eskimo.com
mx69.eskimo.com
The following servers did not come back up after the power outage and I will need to make a trip to the co-location facility to fix:
scientific.eskimo.com – shell server
debian.eskimo.com – shell server
mx69.eskimo.com – used to provide mail service from misconfigured servers
ipspace.eskimo.com – advertises availability of 207.54.0.0/19