I will be out of the office today as I have a doctors appointment during this time frame.

     The solution I’ve got in place now is modreqtimeout with header and body timeouts in place so people can’t just create a connection and stall it indefinitely.  This should mitigate the slow httpd style attacks we received yesterday.  This module is included in the stock Apache distribution so my home is that it is more stable and better tested.

     I’ve got a partial fix in for the type of attacks yesterday that does not require mod_qos but while it will keep someone from eating all the available slots, it won’t stop them from placing a heavy load on the server as mod_qos does.

     I am still looking for a more complete but stable fix.  It’s too bad mod_qos wasn’t stable as it did mitigate the attacks very effectively.

     The web server stopped serving all traffic this morning even though it was still running.  I’m assuming this is a bug in mod_qos which I installed yesterday to mitigate a type of denial of service attack as I’ve never seen this behavior from apache before.

     Consequently I’ve removed that module but still have some timeouts in place that should at least reduce the severity of this type of attack and will research the problem with mod_qos to see if either there is a fix or an alternate solution to this problem.

     Had some problems with MySQL but it is repaired.

Current Time: Sunday, 06-Dec-2015 13:50:16 PST

Restart Time: Sunday, 06-Dec-2015 13:38:00 PST

Parent Server Config. Generation: 1

Parent Server MPM Generation: 0

Server uptime: 12 minutes 15 seconds

Server load: 15.81 19.11 21.48

Total accesses: 16901 – Total Traffic: 103.4 MB

CPU Usage: u28.99 s33.72 cu2241.71 cs305.58 – 355% CPU load

23 requests/sec – 144.1 kB/second – 6.3 kB/request

24 requests currently being processed, 476 idle workers

     They are hammering the server now.  Normal traffic is between 4-7 requests/sec save for the occasional special event but it’s hanging in there just fine.

     The 355% CPU load is deceptive as the web server doesn’t know there are multiple cores and take that into account in it’s calculation.

     It is slowing things slightly but still very usable where as prior to installing mod_qos, it was unusuable.

     Obviously having to wait for an attack to be successful, then responding by blocking the source IP is not a good long term strategy so I’ve made changes to Apache to fix this.

     I added a QoS module to Apache and configured it to mitigate this type of attack in the future, actually this type or another where the full request isn’t sent tying up processes.

     Either way there is now a limit on connections per IP that will prevent one attacker from tying up the entire server, and timeouts on connections that are stalled or sending below a threshold (presently sent to 500 characters / second).

     This should mitigate this type of slow http attack in the future.

     We had a denial of service attack against our web server this morning which consisted of someone initiating more than 400 requests from a very slow connection.  This had the effect of launching 400 workers that just backed up a large amount of data in queue because the connection was too slow to receive the data.

     These were all coming from one IP address rather than a BotNet (which would have been ineffective for this type of attack as it would have had the bandwidth to absorb the traffic). The offending IP address has been blocked.

There will be a meeting of and for Eskimo North users on December 19th, 2015, at 2PM at Spiro’s on Aurora.  Hope to see you there!

     Today I noticed we were getting a bunch of requests like this:

     What these are is a hack that attempts to bash usernames and passwords against xmlrpc.php in WordPress to try to hack WordPress accounts.

     To counter this, and generally afford a number of other worthwhile security improvements in WordPress, I recommend installing the “All In One WP Security” security plug-in on your WordPress site and in the WP Security settings under firewall, enable block access to xmlrpc.php unless you have a plugin that requires this.  There are many other useful options in this plugin you may wish to enable as well.