pitter patter on the keyboard

I noticed one of my new Xen dom0s was coughing up our friend, the ip_conntrack: table full, dropping packet message today. If you like to get your money’s worth out of your dedis the RAM available to dom0 is probably limited – meaning a correspondingly low default ip_conntrack_max. I’m sure you can see how this might be a problem, even more so if it is lower than the ip_conntrack_max of your virtual machines.

None of my previous CentOS dedis had NAT/conntrack modules loaded by default and this dom0 had no need for NAT – being of a fully bridged configuration and routing only public IPs. My first guess was that this dedi’s redhatty initrd loaded the modules through the typical mash-everything-against-the-kernel-and-see-what-sticks approach so I tried removing the NAT and connection tracking related modules:

# rmmod iptable_nat
ERROR: Module iptable_nat is in use

OK, let’s take a look at the tables:

[root@cl-t067-252cl ~]# iptables-save
# Generated by iptables-save v1.3.5 on Sat Jul 21 21:27:40 2012
*nat
:PREROUTING ACCEPT [931:50495]
:POSTROUTING ACCEPT [446:25128]
:OUTPUT ACCEPT [7:502]
-A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p tcp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p udp -j MASQUERADE --to-ports 1024-65535 
-A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE 
COMMIT

It seems I have a subnet I was not aware of…

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Who put that there? libvirt, apparently. According to that article not only is our problem ip_conntrack_max, but:

However, NAT slows down things and only recommended for desktop installations.

Seems highly logical to me. Their solution didn’t look very permanent so I first deleted the symlink in the autostart directory for “default”:

# cd /etc/libvirt/qemu/networks/autostart/
# ls -lsah
total 16K
8.0K drwx------ 2 root root 4.0K Jul 21 21:17 .
8.0K drwx------ 3 root root 4.0K May 14 09:18 ..
   0 lrwxrwxrwx 1 root root   14 Jul 21 21:17 default.xml -> ../default.xml
# mv default.xml
# cd ..
# cp default.xml ~/
# /etc/init.d/libvirtd restart

That didn’t do anything at all. Still had virbr0, still had the iptables rules and still had the kernel modules.

Reboot.

Apparently that was the wrong thing to do. All of my interfaces, bridges, etc seemed to come back up (except virbr0) and the NAT/conntrack modules were missing but not a single VM was routing.

On to their method:

# virsh net-destroy default
# virsh net-undefine default
# service libvirtd restart

Everything looks great. You still have the NAT/conntrack modules loaded but we should be able to take those out one by one.

# lsmod | grep nat
iptable_nat            40517  0 
ip_nat                 52973  2 ipt_MASQUERADE,iptable_nat
ip_conntrack           91749  4 ipt_MASQUERADE,iptable_nat,ip_nat,xt_state
nfnetlink              40457  2 ip_nat,ip_conntrack
ip_tables              55329  2 iptable_nat,iptable_filter
x_tables               50377  7 xt_physdev,ipt_MASQUERADE,iptable_nat,xt_state,ipt_REJECT,xt_tcpudp,ip_tables

Reboot.

Boned again.`Now default.xml is missing (I’m assuming that’s what net-destroy does) – good thing we made a backup first!

# cd /etc/libvirt/qemu/networks/
# cp ~/default.xml ./
# ln -s default.xml autostart/
# reboot

OK. Screw it. We’ll do it the hard way.

#!/bin/bash
ifconfig virbr0 down
iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p tcp -j MASQUERADE --to-ports 1024-65535
iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p udp -j MASQUERADE --to-ports 1024-65535
iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE
iptables -D INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 
iptables -D INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 
iptables -D INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 
iptables -D INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 
iptables -D FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 
iptables -D FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT 
iptables -D FORWARD -i virbr0 -o virbr0 -j ACCEPT 
iptables -D FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 
iptables -D FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
rmmod iptable_nat
rmmod ipt_MASQUERADE
rmmod ip_nat
rmmod xt_state
rmmod ip_conntrack

HOW DO YOU LIKE ME NOW?!

I recently started getting complaints from a client running VoIP services about lag and jitter. I wasn’t able to correlate the issue with any regular indicators and I have a number of people using VoIP services as a client on the same network who haven’t reported any issues. Looking closer at the router I noticed certain daemons lagging despite a low load average and I could see dmesg was flooded with this message:

ip_ct_ras: decoding error: out of bound

There isn’t exactly a heap of documentation on this situation online but I was able to infer it had something to do with H.323. Doing an lsmod I found:

ip_nat_h323
ip_conntrack_h323

I know some VoIP setups have to do a little magic to get around conventional NAT but since this network is completely public I assumed it was safe to rmmod these two modules and things seemed to clear up instantly. I’m still not sure how H.323 connection tracking got involved with good-ol-fashioned DMZ public IP routing and would appreciate any insights you VoIP geniuses could provide – especially since the client who complained is using SIP and not H.323.

UPDATE CoreNetworks.net has responded and reacted to my criticisms below. Please read the comments on this article to see our exchange.

It has been 30 days since I first told you about CoreNetworks’ unfortunate bandwidth test file placement, which they have still not bothered to correct. Against my better judgement I decided to give them a second chance this morning when, upon configuring my order for 100mbit/s port (+$10) and 3300GB (+$55) add-on I was surprised with this error:

NOTICE: You cannot select 3300 GB transfer and 100Mbps transfer. Your selection has been updated.

Well, why not? That doesn’t seem to make much sense. Since they called the 100 meg port “transfer” and the amount of bandwidth “transfer” as well I assumed some dolt that didn’t know what he was doing was behind this glitch in the order process. So I called.

“3300GB is by definition an unlimited 10mbps port”

In the same breath their rep told me:

• I couldn’t get more than the 2000GB package plus per-GB overages on 100mbit/s despite the fact that they could “technically” give 3300GB to 10mbps port users
  • I guess we are to assume their network is so poorly constructed the extra TB at 100mbit/s would crush them
• If you went with the 10mbps port and the $55 1.3T add-on (3300GB package) the only way to get your money’s worth would be to saturate the connection 100%, 24/7, all month long.
• If 3300GB is synonymous with an unlimited 10mbps port we’re working in terms of egress traffic only

Let’s verify the math to make sure I’m not jumping to any conclusions:

Seconds in a month: 2629743.83
Bits in 3300GB: 28346784153600
Bits per second: 10779294.9
Mbits per second: 10.2799367 (binary) 10.7792949 (decimal)

The fact is no matter how you slice your megabits they are selling you more transit than you can possibly use on a 10mbit/s port. Even if we start with a decimal gigabyte you would have to transmit at 39Kbit/s faster than your port speed all month long just to reach your quota.

It’s not just false advertising. It’s a scam.

Made worse by the fact that their FAQ misconstrues the truth:

What type of connection will I have?
This depends on the type of connection you choose. All servers are connected to 10Mbps switch ports (100Mbps ports are available for certain plans). You are able to select flat-rate capped bandwidth on a per-Mbps basis, or you could opt for burstable metered bandwidth. We make both options available to help you find exactly what you’re looking for. You can upgrade from a 10Mbps port to a 100Mbps port for an added $10/month. This does not change your allowed data transfer amount, and care should be taken to avoid unexpected overages if you decide on this option.

It’s also interesting to note that “flat-rate capped bandwidth on a per-Mbps basis” no longer seems to be available (it was when I was with them aeons ago).

Despite all the nice things I’ve said about their support, their level of sloppiness and shadiness makes CoreNetworks.net inadequate for serious hosting.