Web Server Slowness

     Today I received a complaint about slow response from our web server, not something we frequently have an issue with.  It took me a while to find what was going on because the hardware was not that busy.

     First, we had been hit with a couple of SYN floods today but they were brief and we have SYN cookies enabled so that should not have caused a problem.

     I could not find any backlog in the network, ping times from our web server to Googles name server was under 2ms, so not a problem with network.

     I looked at the server statistics and it was doing around 20 hits/second an the worker threads were maxed out.  I increased the worker threads in Apache from 1000 to 5000, and the traffic jumped to 46 hits/second with no more lag.

     So the issue turned out to be a software bottleneck in Apache’s configuration.  We haven’t had that much traffic since Milla Jovavich’s website was here many years ago so it had not been tested at that heavy of a load.

     Still there were plenty of hardware resources.  With that traffic and those settings it used about 10GB on a 128GB RAM machine and did not even saturate one core of a six core processor.  Assuming traffic handling scales reasonably linearly with worker threads, we should be good for about 100 hits/second now but there is still plenty of hardware overhead to increase further if need be.