Removing virbr0 or Why The Fsck is My Dom0 NATting?
- July 21st, 2012
- By كارما
- Write comment
I noticed one of my new Xen dom0s was coughing up our friend, the ip_conntrack: table full, dropping packet message today. If you like to get your money’s worth out of your dedis the RAM available to dom0 is probably limited – meaning a correspondingly low default ip_conntrack_max. I’m sure you can see how this might be a problem, even more so if it is lower than the ip_conntrack_max of your virtual machines.
None of my previous CentOS dedis had NAT/conntrack modules loaded by default and this dom0 had no need for NAT – being of a fully bridged configuration and routing only public IPs. My first guess was that this dedi’s redhatty initrd loaded the modules through the typical mash-everything-against-the-kernel-and-see-what-sticks approach so I tried removing the NAT and connection tracking related modules:
# rmmod iptable_nat ERROR: Module iptable_nat is in use
OK, let’s take a look at the tables:
[root@cl-t067-252cl ~]# iptables-save # Generated by iptables-save v1.3.5 on Sat Jul 21 21:27:40 2012 *nat :PREROUTING ACCEPT [931:50495] :POSTROUTING ACCEPT [446:25128] :OUTPUT ACCEPT [7:502] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p tcp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p udp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT
It seems I have a subnet I was not aware of…
virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Who put that there? libvirt, apparently. According to that article not only is our problem ip_conntrack_max, but:
However, NAT slows down things and only recommended for desktop installations.
Seems highly logical to me. Their solution didn’t look very permanent so I first deleted the symlink in the autostart directory for “default”:
# cd /etc/libvirt/qemu/networks/autostart/ # ls -lsah total 16K 8.0K drwx------ 2 root root 4.0K Jul 21 21:17 . 8.0K drwx------ 3 root root 4.0K May 14 09:18 .. 0 lrwxrwxrwx 1 root root 14 Jul 21 21:17 default.xml -> ../default.xml # mv default.xml # cd .. # cp default.xml ~/ # /etc/init.d/libvirtd restart
That didn’t do anything at all. Still had virbr0, still had the iptables rules and still had the kernel modules.
Reboot.
Apparently that was the wrong thing to do. All of my interfaces, bridges, etc seemed to come back up (except virbr0) and the NAT/conntrack modules were missing but not a single VM was routing.
On to their method:
# virsh net-destroy default # virsh net-undefine default # service libvirtd restart
Everything looks great. You still have the NAT/conntrack modules loaded but we should be able to take those out one by one.
# lsmod | grep nat iptable_nat 40517 0 ip_nat 52973 2 ipt_MASQUERADE,iptable_nat ip_conntrack 91749 4 ipt_MASQUERADE,iptable_nat,ip_nat,xt_state nfnetlink 40457 2 ip_nat,ip_conntrack ip_tables 55329 2 iptable_nat,iptable_filter x_tables 50377 7 xt_physdev,ipt_MASQUERADE,iptable_nat,xt_state,ipt_REJECT,xt_tcpudp,ip_tables
Reboot.
Boned again.`Now default.xml is missing (I’m assuming that’s what net-destroy does) – good thing we made a backup first!
# cd /etc/libvirt/qemu/networks/ # cp ~/default.xml ./ # ln -s default.xml autostart/ # reboot
OK. Screw it. We’ll do it the hard way.
#!/bin/bash ifconfig virbr0 down iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p tcp -j MASQUERADE --to-ports 1024-65535 iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -p udp -j MASQUERADE --to-ports 1024-65535 iptables -t nat -D POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE iptables -D INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT iptables -D INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT iptables -D INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT iptables -D INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT iptables -D FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -D FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT iptables -D FORWARD -i virbr0 -o virbr0 -j ACCEPT iptables -D FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable iptables -D FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable rmmod iptable_nat rmmod ipt_MASQUERADE rmmod ip_nat rmmod xt_state rmmod ip_conntrack
HOW DO YOU LIKE ME NOW?!
