pitter patter on the keyboard

I recently started getting complaints from a client running VoIP services about lag and jitter. I wasn’t able to correlate the issue with any regular indicators and I have a number of people using VoIP services as a client on the same network who haven’t reported any issues. Looking closer at the router I noticed certain daemons lagging despite a low load average and I could see dmesg was flooded with this message:

ip_ct_ras: decoding error: out of bound

There isn’t exactly a heap of documentation on this situation online but I was able to infer it had something to do with H.323. Doing an lsmod I found:

ip_nat_h323
ip_conntrack_h323

I know some VoIP setups have to do a little magic to get around conventional NAT but since this network is completely public I assumed it was safe to rmmod these two modules and things seemed to clear up instantly. I’m still not sure how H.323 connection tracking got involved with good-ol-fashioned DMZ public IP routing and would appreciate any insights you VoIP geniuses could provide – especially since the client who complained is using SIP and not H.323.

You may find yourself in a position where it is necessary to remove all the entries in Netfilter’s connection tracking table (ip_conntrack) for a particular criterion, like the source or destination IP.

For example, I recently detected a user on one of my networks engaged in what was likely a TCP denial of service attack against root name servers (despite the odd fact that the destination port was 80). Being a NATted user, all of their connections were being tracked. The default time-out for tracking an established connection being 5 days, simply disconnecting the user at the second layer would not relieve the congestion on my routers within an acceptable time frame.

#  cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
98636

#  cat /proc/net/ip_conntrack | grep "xxx.xxx.xxx.xxx"
tcp      6 416408 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.5.147 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.5.147 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416406 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.9.239 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.9.239 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416400 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.11.231 sport=58968 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.11.231 dst=yyy.yyy.yyy.yyy sport=80 dport=58968 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416387 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.14.37 sport=58968 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.14.37 dst=yyy.yyy.yyy.yyy sport=80 dport=58968 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416381 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.11.103 sport=58968 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.11.103 dst=yyy.yyy.yyy.yyy sport=80 dport=58968 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 416275 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.9.57 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.9.57 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 415776 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.1.52 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.1.52 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 417319 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.43.60 sport=58967 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.43.60 dst=yyy.yyy.yyy.yyy sport=80 dport=58967 packets=0 bytes=0 mark=0 secmark=0 use=1
tcp      6 417319 ESTABLISHED src=xxx.xxx.xxx.xxx dst=192.168.43.13 sport=58968 dport=80 packets=1 bytes=40 [UNREPLIED] src=192.168.43.13 dst=yyy.yyy.yyy.yyy sport=80 dport=58968 packets=0 bytes=0 mark=0 secmark=0 use=1

...

It is possible to clear tracking entries en masse by removing then reloading the iptables rule that requires them to be tracked in the first place, but on a production gateway this is even less acceptable than waiting for them to expire. Fortunately, we can interact with the ip_conntrack table via conntrack-tools.

Against common sense, conntrack-tools is not available in the repositories for (at least version 5.2) of ClearOS, my favourite router distro. I grabbed a couple recent versions in RPM form but they didn’t feel like playing ball so I ended up with version 0.9.5 from ftp://ftp.pbone.net/mirror/archive.fedoraproject.org/fedora/linux/updates/8/i386.newkey/conntrack-tools-0.9.5-3.fc8.i386.rpm

Apparently newer versions allow one to use -D intuitively, i.e.:

# conntrack -D -s xxx.xxx.xxx.xxx

But this is not the case for at least versions including and prior to 0.97 – these require the d, dport, s and sport flags.

This wonderful person provides a way to pipe the output of conntrack -L (which lists entries the way I’d like to delete them, i.e. -s only) into sed which then breaks the output lines up and awk runs them with conntrack -D appropriately. I had to do some cleanup to get it to work due to the way their blog software mangles punctuation (a lot of my first posts here are mangled in the same way – pobody’s nerfect!):

 conntrack -L -s xxx.xxx.xxx.xxx | sed 's/=/ /g'| awk '{system("conntrack -D -s "$6" -d "$8" -p "$1" --sport="$10" --dport="$12)}'

It should be pretty clear how conntrack -L -s can be modified to work with the destination address or more complicated pattern matching.

Now we can see the ip_conntrack table is at a more reasonable level:

[root@router ~]#  cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
46676

You may find yourself unable to ping the local and remote IPs of the ClearOS router from a host configured with a public IP address participating on a DMZ subnet. I haven’t had time to find out why but it is possible to address this by explicitly accepting pings on the DMZ interface; modify /etc/rc.d/rc.firewall.custom:

iptables -I INPUT -i eth1 -p icmp -m icmp --icmp-type 8 -j ACCEPT # Accept pings on DMZ