pitter patter on the keyboard

I noticed one of my new Xen dom0s was coughing up our friend, the ip_conntrack: table full, dropping packet message today. If you like to get your money’s worth out of your dedis the RAM available to dom0 is probably limited – meaning a correspondingly low default ip_conntrack_max. I’m sure you can see how this might be a problem, even more so if it is lower than the ip_conntrack_max of your virtual machines.

None of my previous CentOS dedis had NAT/conntrack modules loaded by default and this dom0 had no need for NAT – being of a fully bridged configuration and routing only public IPs. My first guess was that this dedi’s redhatty initrd loaded the modules through the typical mash-everything-against-the-kernel-and-see-what-sticks approach so I tried removing the NAT and connection tracking related modules:

# rmmod iptable_nat
ERROR: Module iptable_nat is in use

OK, let’s take a look at the tables:

[root@cl-t067-252cl ~]# iptables-save
# Generated by iptables-save v1.3.5 on Sat Jul 21 21:27:40 2012
*nat
:PREROUTING ACCEPT [931:50495]
:POSTROUTING ACCEPT [446:25128]
:OUTPUT ACCEPT [7:502]
-A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p tcp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p udp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE 
COMMIT

It seems I have a subnet I was not aware of…

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Who put that there? libvirt, apparently. According to that article not only is our problem ip_conntrack_max, but:

However, NAT slows down things and only recommended for desktop installations.

Seems highly logical to me. Their solution didn’t look very permanent so I first deleted the symlink in the autostart directory for “default”:

# cd /etc/libvirt/qemu/networks/autostart/
# ls -lsah
total 16K
8.0K drwx------ 2 root root 4.0K Jul 21 21:17 .
8.0K drwx------ 3 root root 4.0K May 14 09:18 ..
   0 lrwxrwxrwx 1 root root   14 Jul 21 21:17 default.xml -> ../default.xml
# mv default.xml
# cd ..
# cp default.xml ~/
# /etc/init.d/libvirtd restart

That didn’t do anything at all. Still had virbr0, still had the iptables rules and still had the kernel modules.

Reboot.

Apparently that was the wrong thing to do. All of my interfaces, bridges, etc seemed to come back up (except virbr0) and the NAT/conntrack modules were missing but not a single VM was routing.

On to their method:

# virsh net-destroy default
# virsh net-undefine default
# service libvirtd restart

Everything looks great. You still have the NAT/conntrack modules loaded but we should be able to take those out one by one.

# lsmod | grep nat
iptable_nat            40517  0 
ip_nat                 52973  2 ipt_MASQUERADE,iptable_nat
ip_conntrack           91749  4 ipt_MASQUERADE,iptable_nat,ip_nat,xt_state
nfnetlink              40457  2 ip_nat,ip_conntrack
ip_tables              55329  2 iptable_nat,iptable_filter
x_tables               50377  7 xt_physdev,ipt_MASQUERADE,iptable_nat,xt_state,ipt_REJECT,xt_tcpudp,ip_tables

Reboot.

Boned again.`Now default.xml is missing (I’m assuming that’s what net-destroy does) – good thing we made a backup first!

# cd /etc/libvirt/qemu/networks/
# cp ~/default.xml ./
# ln -s default.xml autostart/
# reboot

OK. Screw it. We’ll do it the hard way.

#!/bin/bash
ifconfig virbr0 down
iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p tcp -j MASQUERADE --to-ports 1024-65535
iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p udp -j MASQUERADE --to-ports 1024-65535
iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE
iptables -D INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
iptables -D INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
iptables -D INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
iptables -D INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 
iptables -D FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 
iptables -D FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT 
iptables -D FORWARD -i virbr0 -o virbr0 -j ACCEPT 
iptables -D FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 
iptables -D FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
rmmod iptable_nat
rmmod ipt_MASQUERADE
rmmod ip_nat
rmmod xt_state
rmmod ip_conntrack

HOW DO YOU LIKE ME NOW?!

You may find yourself in a position where it is necessary to remove all the entries in Netfilter’s connection tracking table (ip_conntrack) for a particular criterion, like the source or destination IP.

For example, I recently detected a user on one of my networks engaged in what was likely a TCP denial of service attack against root name servers (despite the odd fact that the destination port was 80). Being a NATted user, all of their connections were being tracked. The default time-out for tracking an established connection being 5 days, simply disconnecting the user at the second layer would not relieve the congestion on my routers within an acceptable time frame.

#  cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
98636

#  cat /proc/net/ip_conntrack | grep "xxx.xxx.xxx.xxx"
tcp      6 416408 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.5.147 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.5.147 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416406 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.9.239 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.9.239 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416400 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.11.231 sport=58968 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.11.231 dst=yyy.yyy.yyy.yyy sport=80 dport=58968 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416387 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.14.37 sport=58968 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.14.37 dst=yyy.yyy.yyy.yyy sport=80 dport=58968 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416381 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.11.103 sport=58968 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.11.103 dst=yyy.yyy.yyy.yyy sport=80 dport=58968 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416275 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.9.57 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.9.57 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 415776 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.1.52 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.1.52 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 417319 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.43.60 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.43.60 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 417319 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.43.13 sport=58968 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.43.13 dst=yyy.yyy.yyy.yyy sport=80 dport=58968 packets=0 bytes=0 mark=0 secmark=0 use=1

...

It is possible to clear tracking entries en masse by removing then reloading the iptables rule that requires them to be tracked in the first place, but on a production gateway this is even less acceptable than waiting for them to expire. Fortunately, we can interact with the ip_conntrack table via conntrack-tools.

Against common sense, conntrack-tools is not available in the repositories for (at least version 5.2) of ClearOS, my favourite router distro. I grabbed a couple recent versions in RPM form but they didn’t feel like playing ball so I ended up with version 0.9.5 from ftp://ftp.pbone.net/mirror/archive.fedoraproject.org/fedora/linux/updates/8/i386.newkey/conntrack-tools-0.9.5-3.fc8.i386.rpm

Apparently newer versions allow one to use -D intuitively, i.e.:

# conntrack -D -s xxx.xxx.xxx.xxx

But this is not the case for at least versions including and prior to 0.97 – these require the d, dport, s and sport flags.

This wonderful person provides a way to pipe the output of conntrack -L (which lists entries the way I’d like to delete them, i.e. -s only) into sed which then breaks the output lines up and awk runs them with conntrack -D appropriately. I had to do some cleanup to get it to work due to the way their blog software mangles punctuation (a lot of my first posts here are mangled in the same way – pobody’s nerfect!):

 conntrack -L -s xxx.xxx.xxx.xxx | sed 's/=/ /g'| awk '{system("conntrack -D -s "$6" -d "$8" -p "$1" --sport="$10" --dport="$12)}'

It should be pretty clear how conntrack -L -s can be modified to work with the destination address or more complicated pattern matching.

Now we can see the ip_conntrack table is at a more reasonable level:

[root@router ~]#  cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
46676