Low throughput with www/apache22 on FreeBSD 8.1

This was originally posted on blogger here.

Few days ago I installed FreeBSD 8.1 and compiled Apache 2.2.15 from port www/apache22. I tried to transfer some files over the internet with rather good connection (100Mbps in the weakest point, but several hops resulting in RTT ~27ms). But I got only 10 – 15Mbps throughput from FreeBSD on hi-end HW. I also tried to install FTP server (ftp/proftpd) but I got the same result. It was not a network issue or a client problem, because I had full speed transfers from another machine in the same network. What the ****?

Step one… Sysctl parameters. The most ****ing evil parameter is net.inet.tcp.inflight.enable. Disable it! Actually I tuned more parameters relevant to TCP/IP stack and kernel internals:

/etc/sysctl.conf

kern.ipc.maxsockbuf=16777216
kern.ipc.nmbclusters=32768
kern.ipc.somaxconn=32768
kern.maxfiles=65536
kern.maxfilesperproc=32768
kern.maxvnodes=800000
net.inet.tcp.inflight.enable=0
net.inet.udp.maxdgram=57344
net.inet.udp.recvspace=65536
net.local.stream.recvspace=65536
net.inet.tcp.sendbuf_max=16777216

OK, transfers over FTP were then running at 100Mbps. Fine. But HTTP file transfers were still ****ing slow. Locally it was two times slower than FTP and much more slower over the internet (HTTP: 4-10 Mbps vs. FTP: 100Mbps).

Step two... Turning off sendfile(2) in Apache HTTPd.

This is also incredibly ****ing option and there is no documentation pointing at it... Even Google does not know what to do when the Apache ****s up on FreeBSD 8.1. Well, I added one single line to /usr/local/etc/apache22/httpd.conf:

EnableSendfile off

And that's it. Both FTP and HTTP are running at full speed.