pitter patter on the keyboard

I noticed one of my new Xen dom0s was coughing up our friend, the ip_conntrack: table full, dropping packet message today. If you like to get your money’s worth out of your dedis the RAM available to dom0 is probably limited – meaning a correspondingly low default ip_conntrack_max. I’m sure you can see how this might be a problem, even more so if it is lower than the ip_conntrack_max of your virtual machines.

None of my previous CentOS dedis had NAT/conntrack modules loaded by default and this dom0 had no need for NAT – being of a fully bridged configuration and routing only public IPs. My first guess was that this dedi’s redhatty initrd loaded the modules through the typical mash-everything-against-the-kernel-and-see-what-sticks approach so I tried removing the NAT and connection tracking related modules:

# rmmod iptable_nat
ERROR: Module iptable_nat is in use

OK, let’s take a look at the tables:

[root@cl-t067-252cl ~]# iptables-save
# Generated by iptables-save v1.3.5 on Sat Jul 21 21:27:40 2012
*nat
:PREROUTING ACCEPT [931:50495]
:POSTROUTING ACCEPT [446:25128]
:OUTPUT ACCEPT [7:502]
-A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p tcp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p udp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE 
COMMIT

It seems I have a subnet I was not aware of…

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Who put that there? libvirt, apparently. According to that article not only is our problem ip_conntrack_max, but:

However, NAT slows down things and only recommended for desktop installations.

Seems highly logical to me. Their solution didn’t look very permanent so I first deleted the symlink in the autostart directory for “default”:

# cd /etc/libvirt/qemu/networks/autostart/
# ls -lsah
total 16K
8.0K drwx------ 2 root root 4.0K Jul 21 21:17 .
8.0K drwx------ 3 root root 4.0K May 14 09:18 ..
   0 lrwxrwxrwx 1 root root   14 Jul 21 21:17 default.xml -> ../default.xml
# mv default.xml
# cd ..
# cp default.xml ~/
# /etc/init.d/libvirtd restart

That didn’t do anything at all. Still had virbr0, still had the iptables rules and still had the kernel modules.

Reboot.

Apparently that was the wrong thing to do. All of my interfaces, bridges, etc seemed to come back up (except virbr0) and the NAT/conntrack modules were missing but not a single VM was routing.

On to their method:

# virsh net-destroy default
# virsh net-undefine default
# service libvirtd restart

Everything looks great. You still have the NAT/conntrack modules loaded but we should be able to take those out one by one.

# lsmod | grep nat
iptable_nat            40517  0 
ip_nat                 52973  2 ipt_MASQUERADE,iptable_nat
ip_conntrack           91749  4 ipt_MASQUERADE,iptable_nat,ip_nat,xt_state
nfnetlink              40457  2 ip_nat,ip_conntrack
ip_tables              55329  2 iptable_nat,iptable_filter
x_tables               50377  7 xt_physdev,ipt_MASQUERADE,iptable_nat,xt_state,ipt_REJECT,xt_tcpudp,ip_tables

Reboot.

Boned again.`Now default.xml is missing (I’m assuming that’s what net-destroy does) – good thing we made a backup first!

# cd /etc/libvirt/qemu/networks/
# cp ~/default.xml ./
# ln -s default.xml autostart/
# reboot

OK. Screw it. We’ll do it the hard way.

#!/bin/bash
ifconfig virbr0 down
iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p tcp -j MASQUERADE --to-ports 1024-65535
iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p udp -j MASQUERADE --to-ports 1024-65535
iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE
iptables -D INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
iptables -D INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
iptables -D INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
iptables -D INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 
iptables -D FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 
iptables -D FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT 
iptables -D FORWARD -i virbr0 -o virbr0 -j ACCEPT 
iptables -D FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 
iptables -D FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
rmmod iptable_nat
rmmod ipt_MASQUERADE
rmmod ip_nat
rmmod xt_state
rmmod ip_conntrack

HOW DO YOU LIKE ME NOW?!

I recently started getting complaints from a client running VoIP services about lag and jitter. I wasn’t able to correlate the issue with any regular indicators and I have a number of people using VoIP services as a client on the same network who haven’t reported any issues. Looking closer at the router I noticed certain daemons lagging despite a low load average and I could see dmesg was flooded with this message:

ip_ct_ras: decoding error: out of bound

There isn’t exactly a heap of documentation on this situation online but I was able to infer it had something to do with H.323. Doing an lsmod I found:

ip_nat_h323
ip_conntrack_h323

I know some VoIP setups have to do a little magic to get around conventional NAT but since this network is completely public I assumed it was safe to rmmod these two modules and things seemed to clear up instantly. I’m still not sure how H.323 connection tracking got involved with good-ol-fashioned DMZ public IP routing and would appreciate any insights you VoIP geniuses could provide – especially since the client who complained is using SIP and not H.323.

Connections in to and out of your network are working sporadically. Your router’s dmesg is flooded with “ip_conntrack: table full, dropping packet.” What do you do?

This condition occurs when the connection tracking table has reached its limit. Connection tracking is a function of Netfilter that stores information like the source and destination IP addresses, port numbers, protocol type, state and timeout of a two-way connection. This facility lets us create sophisticated and informed Netfilter rules in a way that is not possible to accurately derive on a packet header-by-header basis.

The conntrack table takes the form of a memory structure; if there were no constraints on the size of the table it could conceivably start knocking off userspace processes if it became too large (i.e. under DoS conditions). Entries in the conntrack table expire either when their timeout has been reached or the connection has been properly closed. In cases where connections are not being closed according to protocol (poor network connectivity, DoS, spoof attack, etc.) the table can fill rapidly causing an intermittent denial of service condition on your network.

The most prominent symptom of a full connection tracking table is that your old, running connections (secure shell sessions) will continue to function while it becomes impossible to establish new ones. Worse, as the entries continue to time out and the table keeps filling up you may “get lucky” and establish a new connection here and there, making the situation much more confusing.

Depending on the situation you may have one or two options. If you have gobs and gobs of RAM available or the (for example) attack is low-volume you can adjust the entry limit of the table. First, check what the current limit is:

# cat /proc/sys/net/ipv4/ip_conntrack_max
65536

You can see how full the table currently is by running:

# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
62168

ip_conntrack_max is determined as a multiple of how much RAM the system boots up with but generally stops at 65536 regardless. You may find that this isn’t even enough for a high volume network under normal conditions. We can adjust the limit temporarily thus:

# echo 131072 > /proc/sys/net/ipv4/ip_conntrack_max

If this turns out to be your magic bullet and you’re sure no other actions need to be taken to mitigate your particular situation add the following line to /etc/sysctl.conf:

net.ipv4.netfilter.ip_conntrack_max = 131072

To load the value from sysctl.conf run:

# sysctl -p

If you don’t have the option of throwing more RAM at the problem you may be forced to make an executive decision in the interest of preserving network services for legitimate clients. You can decrease the load on the conntrack table by removing rules that use stateful logic (i.e. containing “-t nat” or “-m state”). The brute force option is to rmmod the ip_conntrack module:

# rmmod ip_conntrack

However this may not be possible in all environments. The other option is to flush your rules and set the default policy to allow:

# iptables -P
# iptables -F

This is also typically the effect of

# /etc/init.d/iptables stop
or
# /etc/init.d/firewall stop