Today I received a complaint about slow response from our web server, not something we frequently have an issue with. It took me a while to find what was going on because the hardware was not that busy.
First, we had been hit with a couple of SYN floods today but they were brief and we have SYN cookies enabled so that should not have caused a problem.
I could not find any backlog in the network, ping times from our web server to Googles name server was under 2ms, so not a problem with network.
I looked at the server statistics and it was doing around 20 hits/second an the worker threads were maxed out. I increased the worker threads in Apache from 1000 to 5000, and the traffic jumped to 46 hits/second with no more lag.
So the issue turned out to be a software bottleneck in Apache’s configuration. We haven’t had that much traffic since Milla Jovavich’s website was here many years ago so it had not been tested at that heavy of a load.
Still there were plenty of hardware resources. With that traffic and those settings it used about 10GB on a 128GB RAM machine and did not even saturate one core of a six core processor. Assuming traffic handling scales reasonably linearly with worker threads, we should be good for about 100 hits/second now but there is still plenty of hardware overhead to increase further if need be.