Avoid Tracking Memcached Using Ip_conntrack in CentOS Kernel

Published December 10th, 2008

Problem Statement

We recently enabled Ip_conntrack for supporting FTP service using vsftpd daemon on a CentOS server that also runs memcached. A bit later we noticed that /proc/net/ip_conntrack shows a lot of entries for memcached. Since kernel level tracking of connections require memory and CPU resources, we wanted to not track memcached connections on a busy server. Here you will find out what we had to do to overcome this problem.

First Choice: stop using ip_conntrack kernel module on memcached server

This is the best option as tracking connections in kernel is really a memory/CPU resource concern.
Because, conntrac mechanism of the Linux kernal eats up memory and cpu time and may be disabling it can improve overall performance of memcached.

Second Choice: increase the max connection tracking size for the kernel

If you must use ip_conntrack, consider setting the maximum entry size (CONNTRACK_MAX) of the connection tracking table in the kernel to a large number.

Take a look at /proc/sys/net/ipv4/netfilter/ip_conntrack_max
and if it set to a low number. For systems with more than 1GB of RAM, default value is limited to 65536. To calculate what size you can set this to, use the following information:

On i386 architecture:

CONNTRACK_MAX = RAMSIZE (in bytes) / 16384 = RAMSIZE (in MegaBytes) * 64.

So for example, a 32 bits PC with 512MB of RAM can handle 512*1024^2/16384 = 512*64 = 32768 simultaneous connections by default.

But the real formula is:

CONNTRACK_MAX = RAMSIZE (in bytes) / 16384 / (x / 32)

where x is the number of bits in a pointer (for example, 32 or 64 bits). To set a new value run:

echo "new value" > /proc/sys/net/ipv4/netfilter/ip_conntrack_max

For example:

echo 1234567  > /proc/sys/net/ipv4/netfilter/ip_conntrack_max

To make this setting permanent, add this line in your /etc/rc.local file so that when you reboot your CentOS server, it also sets the connection tracking size to the desired value.

Third Choice: use iptables to stop tracking memcached connections

To minimize the resource use (memory/CPU) for connection tracking, you can consider disabling connection tracking of your memcached daemon. For example, if you run memcached on the default 11211 port, you can run the following iptables rules on the memcached CentOS server:

iptables -t raw -I PREROUTING -p tcp --dport 11211 -j NOTRACK
iptables -t raw -I OUTPUT -p tcp --dport 11211 -j NOTRACK

The first iptables rule tells CentOS kernel to not to track incoming traffic from your memcached clients. The second one tells CentOS kernel to not to track outgoing traffic from your memcached daemon.

Category: Performance | Tags: