UPDATE: This article is now slightly dated. In the comments below, Null4Ever provides more contemporary kernel parameter settings.
Siege is a great tool to measure web site performance and establish benchmark metrics. What do you do if siege shows your performance stinks? This document illustrates how to tune your server and validate those tunings with siege.
For the purpose of this exercise we will tune a very old Linux server to effectively serve static content at a rate of 200 requests per second. We’ll start with an out-of-box apache configuration and build from there. By the way, when I say “old” I mean old:
Ben: $ uname -a Linux ben 2.4.10-4GB #1 Fri Sep 28 17:20:21 GMT 2001 i586 unknown
Ben: $ bin/httpd -V Server version: Apache/2.0.47 Server built: Jul 18 2003 18:16:04
Damn, that’s old! Let’s see how this Model-T performs. The first thing we need to do is establish a benchmark:
Bully $ siege -d1 -c200 -t1m http://ben.home.joedog.org/apache_pb.gif Lifting the server siege... done. Transactions: 10230 hits Availability: 100.00 % Elapsed time: 59.90 secs Data transferred: 22.69 MB Response time: 0.15 secs Transaction rate: 170.78 trans/sec Throughput: 0.38 MB/sec Concurrency: 26.18 Successful transactions: 10230 Failed transactions: 0 Longest transaction: 2.34 Shortest transaction: 0.00
There’s no two ways about it: that’s crappy. What steps can we take to improve that? An obvious place to look is the apache configuration. Is the server configured to handle 200 simulataneous connections? If apache is configured with less than 200 workers, then requests are queued until workers are available to handle them. Let’s check our config:
<IfModule prefork.c> StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 150 MaxRequestsPerChild 0 </IfModule>
Sure enough, we have fewer workers than requests. Let’s bump those numbers in order to meet our requirement. While we’re at it, we’ll increase the initial pool so we pay less of a penalty to fork new workers:
<IfModule prefork.c> StartServers 50 MinSpareServers 15 MaxSpareServers 25 MaxClients 225 MaxRequestsPerChild 0 </IfModule>
After apache restart we find thirty-three httpd processes in memory. With a bigger pool in memory, it’s easier to accomodate a large pool of incoming requests. With the expansion of our workers pool, we have the capacity to meet our load requirement. Let’s see how these changes affected our performance:
Lifting the server siege... done. Transactions: 10290 hits Availability: 99.44 % Elapsed time: 59.07 secs Data transferred: 22.83 MB Response time: 0.05 secs Transaction rate: 174.20 trans/sec Throughput: 0.39 MB/sec Concurrency: 9.55 Successful transactions: 10290 Failed transactions: 58 Longest transaction: 3.07 Shortest transaction: 0.00
Our transaction rate improved by nearly 3.5 per second. This is because requests spent less time waiting for a worker to handle them. If you watch the web server’s socket table during a siege, one thing stands out. The number of sockets in TIME_WAIT. This what I saw during the last run:
Ben: $ netstat -a | grep TIME_WAIT | wc -l 8081
If we could recycle those connections, we could improve our performance. The kernel parameter that controls TIME_WAIT reuse is ‘net.ipv4.tcp_tw_recycle’. The sysctl command will enable you to view and set kernel parameters. Let’s check the value on our web server:
Ben: $ sysctl net.ipv4.tcp_tw_recycle net.ipv4.tcp_tw_recycle = 0
We’re not reusing sockets in TIME_WAIT. Let’s change it and view our results:
Ben: $ sysctl -w net.ipv4.tcp_tw_recycle=1 net.ipv4.tcp_tw_recycle = 1
For readers with newer systems, we recommend this additional setting:
sysctl -w net.ipv4.tcp_tw_reuse=1
I set those parameters on both the computer running siege and the server running apache. To make the changes permanent, add them to /etc/sysctl.conf Now let’s run another test:
Lifting the server siege... done. Transactions: 10231 hits Availability: 100.00 % Elapsed time: 31.25 secs Data transferred: 22.69 MB Response time: 0.05 secs Transaction rate: 327.39 trans/sec Throughput: 0.73 MB/sec Concurrency: 16.68 Successful transactions: 10231 Failed transactions: 0 Longest transaction: 3.12 Shortest transaction: 0.00
Our transactions per second sky-rocketed to 327.39! That’s a significant improvement but we still have a problem. Notice the elapsed time. I stopped the siege short of a minute because it was hung. In run after run, siege was hung at about 10,200 transactions. In fact, every time I run the siege it hangs at that number. We’re exhausting something, but what? Let’s check our kernel parameters and see what we find:
Ben: $ /sbin/sysctl -a | grep 102 net.ipv6.neigh.default.gc_thresh3 = 1024 net.ipv6.route.gc_thresh = 1024 net.ipv4.ip_conntrack_max = 10232 net.ipv4.neigh.default.gc_thresh3 = 1024 net.ipv4.route.gc_thresh = 1024 net.ipv4.tcp_max_syn_backlog = 1024 net.core.optmem_max = 10240 kernel.sem = 250 32000 32 1024 kernel.rtsig-max = 1024
It looks like we hit the ip_conntrack_max limit. If kernel dropped packets it will let us know. Let’s check the system logs:
Apr 2 15:00:58 ben kernel: ip_conntrack: table full, dropping packet.
Sure enough. Let’s double it.
Ben: $ sysctl -w net.ipv4.ip_conntrack_max=`echo 10232*2 | bc` net.ipv4.ip_conntrack_max = 20464
One more run, let’s see how we do:
Bully # siege -d1 -c200 -t1m http://ben.home.joedog.org/apache_pb.gif
Lifting the server siege... done. Transactions: 20462 hits Availability: 100.00 % Elapsed time: 59.65 secs Data transferred: 45.39 MB Response time: 0.04 secs Transaction rate: 343.03 trans/sec Throughput: 0.76 MB/sec Concurrency: 15.31 Successful transactions: 20462 Failed transactions: 0 Longest transaction: 1.26 Shortest transaction: 0.00
Much better. With a little bit of tuning, we were able to double our server’s transaction rate.