pitter patter on the keyboard

I like to start my Xen networking by creating two (or more) bridges, one connected to the physical interface on the dom0 and another which is only connected to by virtual machines. This lets me set up an internal private network for cross-VM interaction and administration which is insulated from the DC’s network. If you put a router VM on both bridges which occupies your public IP addresses you can use 1-1 NAT to stack services from different VMs behind one IP and firewall the virtual machines. This is particularly handy where your address space is small but your VMs are many.

Unfortunately, the Debian interfaces configuration scheme doesn’t like setting up bridges with no bridge_ports directive so we have to give it a little boost. You will end up with a configuration file that looks something like this:

auto lo
iface lo inet loopback

iface eth0 inet manual

auto extbr0
iface extbr0 inet static
        bridge_ports eth0
        address x.x.x.x
        netmask 255.255.255.0
        network x.x.x.0
        broadcast x.x.x.255
        gateway x.x.x.1

auto intbr0
iface intbr0 inet manual
        pre-up    brctl addbr $IFACE
        post-down brctl delbr $IFACE
        up        ifconfig    $IFACE up y.y.y.y
        down      ifconfig    $IFACE down

Where y.y.y.y is an internal IP you can use to access the dom0. For example you might prefer not to expose SSH on your dom0 to the wild, you would make sshd listen on this IP and shell in over a VPN.

I have the good fortune of being forced to migrate over 1,100 e-mail accounts from a 15 year old qmail server with Courier-IMAP.

A script is provided at http://wiki.zimbra.com/wiki/Courier-IMAP_Maildir_to_zmmailbox. It should be easy to run as root and import over NFS. Unfortunately, I ran into trouble with the –noVerify flag, possibly due to the older version of Zimbra I’m importing to:

  * Running import process...           0 ...some messages did not import correctly: check /tmp//import-example.com-user--6108

The output of the log looks like:

addMessage --flags "u"  --noValidation "/Inbox" "/mnt/example.com/user/Maildir//cur/1362778133.14175.smtp.example.com:2,"

To test it you’re going to have to look at the prefix in the script:

/opt/zimbra/bin/zmmailbox -m user@example.com -z addMessage --flags "u"  --noValidation "/Inbox" "/mnt/example.com/user/Maildir//cur/1362778133.14175.smtp.example.com:2,"

Which gave me the following error:

ERROR: zclient.CLIENT_ERROR (unknown folder: --noValidation)

I tried putting the –noValidation flag in every position with no luck, and it is listed in the Zimbra wiki article for zmmailbox at http://wiki.zimbra.com/wiki/Zmmailbox so I am left to assume my version simply does not support it. It doesn’t seem to be consequential anyway so I simply removed it from the script.

#!/bin/bash                                                                                                                                                                                                                                                                    
#
# courier/vpopmail Maildir to Zimbra Import
#
# This script can be stored anywhere, but you should run it while in the root
# of the domain's users.  It looks for the file vpasswd which contains a
# line-separated list of users and uses that to import.  You can also run the
# script with a user name to process a single user.  Additionally, you can
# specify a folder name (courier format) to process a single folder for that
# user.

# We assume the folder structure is like this:
# Inbox: <working directory>/<user>/Maildir/<cur|new>
# Subfolder: <working directory>/<user>/Maildir/.Subfolder/<cur|new>
# If this is not what your structure looks like, you need to change the
# "folderpath" variable construction down further in this script.

# This is the command to run to run mailbox commands.
ZMCMD='/opt/zimbra/bin/zmmailbox -z'

# This will be used for temporary/log files during the import process
TEMP='/tmp/'

# We assume the working directory's name is the domain.
# Otherwise, override this with your actual domain name.
domain=`basename ${PWD}`

echo Process ID: $$

if [[ $1 != "" ]] ; then
  USERS=$1;
else
  USERS=`cat vpasswd | cut -f1 -d:`
fi

for user in ${USERS}; do
  echo "Beginning User: $user..."

  if [[ $2 != "" ]] ; then
    FOLDERS="$user/Maildir/$2/cur";
  else
    FOLDERS=`find $user -type d -name cur | sort`
  fi

  echo "$FOLDERS" | while read line; do
    folderdir=`echo ${line} | cut -f3 -d"/"`
    if [[ ${folderdir} == "cur" ]] ; then
      folderdir="";
    fi 
    folder=`echo ${folderdir} | sed 's/^\.//; s%\.%/%g; s%\&-%\&%g'`
    folderpath=${PWD}/${user}/Maildir/${folderdir}/
    
    # If the folder name is blank, this is the top level folder,
    # Zimbra calls it "Inbox" (so do most clients/servers).
    if [[ $folder == "" ]] ; then
      folder="Inbox";
    fi
    # In Courier IMAP, all folders must be children of the root
    # folder, which means Trash, Junk, Sent, Drafts are typically
    # under Inbox. This is not the case with Zimbra, so we will
    # slide these mailboxes to the top level so they behave properly,
    # For all "non-special" mailboxes, we will keep them as children
    # so they remain where the user had them before.
    if [[ $folder != "Trash" && $folder != "Junk" && $folder != "Sent"
       && $folder != "Drafts" && $folder != "Inbox" ]] ; then
      folder="Inbox/${folder}";
    fi
    echo "* Working on Folder $folder..."

    # Courier allows heirarchy where non-folders (literally nothing) are
    # able to have children.  Zimbra does not.  It's also possible that
    # we will process the folders out of heirarchical order for some reason
    # Here we separate the path and make sure all the parent folders exist
    # before trying to create the folder we're working on.
    parts=(`echo $folder | sed 's% %\x1a%g; s%/% %g'`);
    hier="";
    for i in "${parts[@]}"; do
      hier=`echo ${hier}/$i | sed 's%^/%%; s%\x1a% %g'`;
      ${ZMCMD} -m ${user}@${domain} getFolder "/${hier}" >/dev/null 2>&1 ||
      ( echo -n "  + Creating folder $hier... " &&
      ${ZMCMD} -m ${user}@${domain} createFolder "/${hier}" )
    done

    # Figure out how many messages we have
    count=`find "${folderpath}new/" "${folderpath}cur/" -type f | wc -l`;
    imported=0;
    echo "  * $count messages to process..."

    # Define the temporary file names we will need
    importfn="${TEMP}/import-$domain-$user-$folderdir-$$"
    implogfn="${TEMP}/import-$domain-$user-$folderdir-$$-log"
    impflogfn="${TEMP}/import-$domain-$user-$folderdir-$$-flaglog"
    impflagfn="${TEMP}/import-$domain-$user-$folderdir-$$-flags"
    touch "$importfn"

    # Determine the courier extended flag identifiers ("keywords")
    flagid=0
    if [[ -f "${folderpath}courierimapkeywords/:list" ]] ; then
      extflags="YES"
      cat "${folderpath}courierimapkeywords/:list" 2>/dev/null | while read line; do
        # A blank line indicates the end of the definitions.
        if [[ "${line}" == "" ]]; then break; fi

        # To avoid escape character madness, I'm swapping $ with % here.
        flag=`echo ${line} | sed 's/\\\$/%/'`
        echo courierflag[${flagid}]="'$flag'";
        flagid=$(( flagid + 1 ));

        # Create the tag if it doesn't start with '%'
        if [[ `echo ${flag} | grep '%'` == "" ]] ; then
          echo -n "  + Attemping to create tag ${flag}... " >&2
          ${ZMCMD} -m ${user}@${domain} createTag "${flag}" >&2
        fi

      done > "$impflagfn"
      source "$impflagfn"
    fi

    echo -n "  * Queuing messages for import...        " 

    # Find all "cur" or "new" messages in this folder and import them.
    find "${folderpath}new/" "${folderpath}cur/" -type f | while read msg; do
      flags="";
      tags="";
      msgid=`echo $msg | cut -d: -f1 | sed s%.*/%%`

      # Determine the old maildir style flags
      oldflags=`echo $msg | cut -d: -f2`
      # Replied
      if [[ `echo ${oldflags} | grep 'R'` != "" ]] ; then flags="${flags}r"; fi
      # Seen
      if [[ `echo ${oldflags} | grep 'S'` == "" ]] ; then flags="${flags}u"; fi
      # Trashed
      if [[ `echo ${oldflags} | grep 'T'` != "" ]] ; then flags="${flags}x"; fi
      # Draft
      if [[ `echo ${oldflags} | grep 'D'` != "" ]] ; then flags="${flags}d"; fi
      # Flagged
      if [[ `echo ${oldflags} | grep 'F'` != "" ]] ; then flags="${flags}f"; fi

      # Determine the courier-imap extended flags for this message
      if [[ ${extflags} == "YES" ]] ; then
        oldflags2=`grep $msgid "${folderpath}courierimapkeywords/:list" 2>/dev/null | cut -d: -f2`
        for flag in ${oldflags2}; do
          # Forwarded
          if [[ ${courierflag[$flag]} == '%Forwarded' ]] ; then flags="${flags}w"; fi
          # Sent by me
          if [[ ${courierflag[$flag]} == '%MDNSent' ]] ;   then flags="${flags}s"; fi
          # Convert non-system flags to Zimbra tags
          if [[ `echo ${courierflag[$flag]} | grep '%'` == "" ]] ; then
            tags="${tags},${courierflag[$flag]}"
          fi
        done
        # Clean up the tag list for the command line
        if [[ ${tags} != "" ]]; then
          tags=`echo ${tags} | sed "s/^,\?/--tags \'/; s/\$/\'/"`;
        fi
      fi

      # Log the result of flag processing for debugging
      if [[ $flags != "" || $tags != "" ]] ; then
        echo `date +%c` "$msg had flags $oldflags and $oldflags2, now $flags and $tags in folder $folder" >> "$impflogfn"
      fi

      # Add the command to the queue file to import this message
      echo "addMessage --flags \"${flags}\" ${tags} \"/$folder\" \"${msg}\"" >> "$importfn"

      imported=$(( $imported + 1 ));
      printf "\b\b\b\b\b\b\b\b%7d " $imported;
    done

    echo "...done";

    # Since we redirect the queue file to the mailbox tool, we end with "quit"
    echo "quit" >> "$importfn"

    # We're counting "prompts" from the zmmailbox utility here.  The first
    # one comes up before a message is imported, so we start at -1 to offset
    # its existence.
    imported=-1;

    # We do this redirect because running the command for each message is very
    # slow.  We can't just pass the directory to the command, despite Zimbra's
    # support because we can't tag or flag the messages that way.
    echo -n "  * Running import process...             "
    ${ZMCMD} -m $user@$domain < "${importfn}" 2> "${implogfn}" | while read; do
      imported=$(( $imported + 1 ));
      printf "\b\b\b\b\b\b\b\b%7d " $imported;
    done

    if [[ -s "${implogfn}" ]]; then 
      echo "...some messages did not import correctly: check $importfn";
    else
      echo "...done";
    fi
  done
done
echo "Import Process Complete!"

Now my output for a single account is:

Process ID: 11495
Beginning User: user...
* Working on Folder Inbox/Archive...
  + Creating folder Inbox/Archive... 285
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Drafts...
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Junk...
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Sent...
  * 3 messages to process...
  * Queuing messages for import...      3 ...done
  * Running import process...           3 ...done
* Working on Folder Trash...
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Inbox/new folder...
  + Creating folder Inbox/new folder... 289
  * 5 messages to process...
  * Queuing messages for import...      5 ...done
  * Running import process...           5 ...done
* Working on Folder Inbox...
  * 1 messages to process...
  * Queuing messages for import...      1 ...done
  * Running import process...           1 ...done
Import Process Complete!

It is important to note that running the import script multiple times will result in the e-mails being imported multiple times and it takes a fair amount of time to perform this procedure on one account nevermind one thousand so a strategy should be formulated for dealing with runoff mail before the MX/target is switched.

This raw full disk image is a straight-up install of Ubuntu 12.04.1 Server LTS “Precise Pangolin” on EXT3. No packages were selected for installation other than OpenSSH Server. It will boot under full virtualization platforms (QEMU, VirtualBox, VMWare) without modification (except maybe disk image conversion) but the steps below are required to make it play nicely as a Xen Paravirtualized Guest.

db3fb8b154cb05ab704733bba1a6d70e ubuntu-12.04.1-x86_64.raw.hdd.lzma

The user account is “user”
The user password is “user”

The default network settings are: eth0 DHCP

Fortunately, the generic kernel has Xen PV guest support so the required buggery is mostly limited to making the console work. Unpack and mount the image:

# unlzma ubuntu-12.04.1-x86_64.raw.hdd.lzma
# mkdir /mnt/rawroot
# lomount -diskimage ubuntu-12.04.1-x86_64.raw.hdd -partition 1 /mnt/rawroot

The distributed /boot/grub/grub.cfg script will make pygrub throw up. We’ll drop this (/mnt/rawroot)/boot/grub/grub.conf file in to override it:

default 0
timeout 5
fallback 1

title Ubuntu, with Linux 3.2.0-29-generic
        root=(hd0,0)
        kernel   /boot/vmlinuz-3.2.0-29-generic root=UUID=a52e9498-44e8-4c0d-807e-903d4e19e204 ro console=hvc0
        initrd  /boot/initrd.img-3.2.0-29-generic

title Ubuntu, with Linux 3.2.0-29-generic (recovery mode)
        root=(hd0,0)
        kernel   /boot/vmlinuz-3.2.0-29-generic root=UUID=a52e9498-44e8-4c0d-807e-903d4e19e204 ro recovery nomodeset
        initrd  /boot/initrd.img-3.2.0-29-generic

Now we need to make a getty load on hvc0, create (/mnt/rawroot)/etc/init/hvc0.conf

# hvc0 - getty
#
# This service maintains a getty on hvc0 from the point the system is
# started until it is shut down again.

start on stopped rc or RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -L 115200 hvc0 vt102

We probably want to give our VM a static IP. Edit (/mnt/rawroot)/etc/network/interfaces to reflect:

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.100
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 8.8.8.8

Now we are ready to boot up. Unmount /mnt/rawroot and create a configuration file that looks similar to:

name = "ubuntu"

vcpus = 1
memory = 256
vif = [ 'mac=00:16:3e:ff:00:01,bridge=extbr0' ]
disk = ['file:/xen/ubuntu/ubuntu.hdd,sda,w']

bootloader = "/usr/bin/pygrub"
extra = "xencons=hvc0 console=hvc0"

Ensure the MAC address does not conflict with any existing virtual machines. Assuming the configuration file is named ubuntu.conf run

# xm create ubuntu.conf -c

If everything went well you should be looking at a login prompt shortly after.

Note that if you use cp -ax to transfer the filesystem’s contents to a larger image in the future, references using UUIDs in fstab and grub.conf will have to be changed to either reflect the new UUID or the corresponding /dev node (probably xvda1) and GRUB must be re-installed to the MBR.

If you would like to move the contents to a filesystem-only image (no partition table, MBR, etc.) which is much easier to grow than a full disk image you can drop pygrub and externalize the kernel and initrd images then reference them in the configuration file.