Two more accounts were compromised, these I found no obvious signs of brute force password guessing. One was used to send a large amount of spam last night, again getting us added to SpamCop and MailSpike blacklists.
I went through the logs, found 183 addresses belonging to this spammer botnet, and added them to our firewall.
This morning another account was used, this time only six addresses were involved, I added those to the firewall rule.
I’ve only been able to contact the second customer, but in his case he used the same password at many places. If you’ve been following the news, accounts at Twitter, Facebook, Yahoo, and Google have all been compromised in the last three months, so you can see the importance of using different passwords.
I am working on eliminating any opportunities for these folks to bash password guesses such as ssh, ftp, telnet, rlogin, etc. It will take some time to do this.
After I finish this I intend to run Crack on our password file to find any easily guessable passwords and notify customers of the need to change those.
All of this will not do any good however if the same good passwords are used at other sites which are not secured.
I broke the web server for a short period this evening attempting to install a security suite that would stop password guessing at WordPress login prompts. That software turned out to be badly broke in spite of a five-star rating.
I found different software that functioned without killing the server.
Webmail now has a plugin called “Lockout” installed.
It is configured so that five failed login attempts on the same username will ban that username for fifteen minutes and five failed login attempts on different usernames but the same IP address will ban that IP address for fifteen minutes.
With fail2ban in place, the people attacking the mail system moved to webmail and used it to get the web server banned so that webmail was intermittently broken.
I’ve exempted the local servers so this will no longer happen and I’m working on installing fail2ban on the web, ftp, and shell servers, as well as a necessary plugin that will cause Squirrel Mail to log the address so we can ban attackers of webmail.
I kind of expected an onslaught of bad activity after Microsoft discontinued support for Windows-XP. They’ve basically just added a large amount of vulnerable CPU capacity to the web for these people to form new botnets and launch attacks from.
I have worked through all the Blacklists that were logged, requesting de-listing for our mail server. There are a couple where no mechanism exists to request a de-list, it’s automatic and time based.
There are also some mail sitting in queue because the destination server thinks we are still blacklisted because of the way real time black hole lists work. These lists use DNS to distribute status information for each IP. DNS is cached, and so even when an RBL de-lists our server, a site that our server tried to deliver to while we were listed still sees us as listed until the time to live (TTL) expires for the DNS zone in question.
At this point, our server is fixed and better secured and I’ve contacted every RBL that has a manual list method.
I have added fail2ban which is a program that watches logs for authentication failures and bans the associated IP address after five unsuccessful attempts for ten minutes.
I have added some rate limiting to postfix which should be not affect legitimate mail but will limit the damage in the event accounts are compromised. I’ve also made it a little less forgiving of bad behavior common in some of these spam botnets.
I am watching the logs for rejections and contacting sites and blacklists and requesting our server to be removed as I become aware of them.
Some of the blacklists do not provide a manual removal mechanism and require a fixed interval of time to pass without receiving spam before they will remove our server.
If you receive a bounce message, please read the message, it usually contains a URL where you can submit a request for removal. I am working off the logs as fast as I can.
Our server is on a number of real time black hole lists now because of the spammer that compromised two accounts here and badly abused our server.
Those RBLs which have a mechanism which allows delisting to be requested, I’ve already done so, but some like sorbs, spamcop, and a handful of others only delist an IP if no further spam is received after a set period of time.
There is NOTHING I can do about these. Please complain to the sites that use them.
I am working on fixing the weaknesses that allowed the accounts to be compromised and the server to be abused. This is non-trivial.
A second account was compromised via brute-force password guessing. I’m working on getting fail to ban configured on the mail server (and later other servers) to stop this.
The client mail server is still being beaten by bounces from misconfigured sites that, instead of rejecting mail they will not deliver up front, queue it and bounce it later.
This ran the server out of spool space sometime early this morning and as a result, some mail that couldn’t be immediately delivered, may have been lost.
Sites where immediate delivery may not have been possible tended to be large sites like Yahoo or Google which throttle the rate at which they will accept incoming mail.
A hacker used a brute force attack (kept connecting and using auth to try passwords) and successfully guessed the password of one of our customers.
This was followed by a bunch of computers bashing our mail server to send a few million spams using that customers login credentials. A botnet was apparently involved as the spam originated from many IP addresses.
I have disabled the account in question until I can contact the customer and arrange for a new, stronger, password.
I have deleted all the spam that was still in queue.
I’ve checked blacklists and, where we were listed, requested removal. One automatically removes only after seven days and won’t manually remove without a $119 extortion fee. Another has had us listed since 2007 apparently because one of our customers angered someone in IRC. That’s hard to imagine.
At any rate I’ve got things as cleaned up as they can be for now.