pitter patter on the keyboard

I like to start my Xen networking by creating two (or more) bridges, one connected to the physical interface on the dom0 and another which is only connected to by virtual machines. This lets me set up an internal private network for cross-VM interaction and administration which is insulated from the DC’s network. If you put a router VM on both bridges which occupies your public IP addresses you can use 1-1 NAT to stack services from different VMs behind one IP and firewall the virtual machines. This is particularly handy where your address space is small but your VMs are many.

Unfortunately, the Debian interfaces configuration scheme doesn’t like setting up bridges with no bridge_ports directive so we have to give it a little boost. You will end up with a configuration file that looks something like this:

auto lo
iface lo inet loopback

iface eth0 inet manual

auto extbr0
iface extbr0 inet static
        bridge_ports eth0
        address x.x.x.x
        netmask 255.255.255.0
        network x.x.x.0
        broadcast x.x.x.255
        gateway x.x.x.1

auto intbr0
iface intbr0 inet manual
        pre-up    brctl addbr $IFACE
        post-down brctl delbr $IFACE
        up        ifconfig    $IFACE up y.y.y.y
        down      ifconfig    $IFACE down

Where y.y.y.y is an internal IP you can use to access the dom0. For example you might prefer not to expose SSH on your dom0 to the wild, you would make sshd listen on this IP and shell in over a VPN.

GRUB2 is highly customizable and feature-rich, however it is a bit of a nightmare if you configure bootloaders regularly. Unfortunately GRUB 0.97 doesn’t support ext4 and – if you were an idiot like me – you might have set up your new dedicated with an ext4 /boot. Now, you have neither the time nor patience to re-install and need to make Xen go.

First, copy your Dom0 .config and kernel to /boot/. /etc/grub.d/20_linux_xen picks up on the CONFIG_XEN_PRIVILEGED_GUEST=y variable.

# cp /usr/src/linux/arch/x86_64/boot/bzImage /boot/kernel-dom0
# cp /usr/src/linux/.config /boot/config-dom0

Next, remove the executable flag from /etc/grub.d/10_linux or you will end up with three or four entries that boot straight to the Dom0 kernel without the hypervisor.

# chmod -x /etc/grub.d/10_linux

To add kernel boot parameters to your grub.cfg (as generated with grub2-mkconfig and 20_linux_xen) add these lines to your /etc/default/grub file:

GRUB_CMDLINE_LINUX_XEN_REPLACE="max_loop=128"
GRUB_CMDLINE_XEN="dom0_mem=512"

The former will place your boot options on the Dom0 kernel’s module line. The latter will place boot options on the xen.gz hypervisor’s kernel line.

Now, generate your new configuration:

# grub2-mkconfig -o /boot/grub2/grub.cfg

Once the config file has been written, determine the position of the Xen hypervisor entry and update GRUB_DEFAULT in /etc/default/grub.

Install GRUB2 to your MBR(s) as necessary:

# grub2-install /dev/sda 

I just encountered this while making a tar backup on one of my virtual filesystems:

lost page write due to I/O error on xvdb1
end_request: I/O error, dev xvdb1, sector 33707688
Buffer I/O error on device xvdb1, logical block 4213461

Unfortunately, my fs suffered corruption.

Some folks, particularly Ubuntu users with kernel 3.2.x seem to say the answer is adding

barrier=0

to your ext3/ext4 mount options.

In my case I’m pretty sure an OOM error killed something critical to the backend on dom0.

Be sure to check your dom0′s dmesg.

UPDATE Uhh yeah, something actually managed to overflow the dom0′s fs. >.> This is what I was getting in the dom0 dmesg:

loop: Write error at byte offset 17820364800, length 4096.

for which there is very little discussion, but a quick df told me what was going on.