Import Courier-IMAP Maildir E-Mail and IMAP Folders to Zimbra
- March 8th, 2013
- By كارما
- Write comment
I have the good fortune of being forced to migrate over 1,100 e-mail accounts from a 15 year old qmail server with Courier-IMAP.
A script is provided at http://wiki.zimbra.com/wiki/Courier-IMAP_Maildir_to_zmmailbox. It should be easy to run as root and import over NFS. Unfortunately, I ran into trouble with the –noVerify flag, possibly due to the older version of Zimbra I’m importing to:
* Running import process... 0 ...some messages did not import correctly: check /tmp//import-example.com-user--6108
The output of the log looks like:
addMessage --flags "u" --noValidation "/Inbox" "/mnt/example.com/user/Maildir//cur/1362778133.14175.smtp.example.com:2,"
To test it you’re going to have to look at the prefix in the script:
/opt/zimbra/bin/zmmailbox -m user@example.com -z addMessage --flags "u" --noValidation "/Inbox" "/mnt/example.com/user/Maildir//cur/1362778133.14175.smtp.example.com:2,"
Which gave me the following error:
ERROR: zclient.CLIENT_ERROR (unknown folder: --noValidation)
I tried putting the –noValidation flag in every position with no luck, and it is listed in the Zimbra wiki article for zmmailbox at http://wiki.zimbra.com/wiki/Zmmailbox so I am left to assume my version simply does not support it. It doesn’t seem to be consequential anyway so I simply removed it from the script.
#!/bin/bash
#
# courier/vpopmail Maildir to Zimbra Import
#
# This script can be stored anywhere, but you should run it while in the root
# of the domain's users. It looks for the file vpasswd which contains a
# line-separated list of users and uses that to import. You can also run the
# script with a user name to process a single user. Additionally, you can
# specify a folder name (courier format) to process a single folder for that
# user.
# We assume the folder structure is like this:
# Inbox: <working directory>/<user>/Maildir/<cur|new>
# Subfolder: <working directory>/<user>/Maildir/.Subfolder/<cur|new>
# If this is not what your structure looks like, you need to change the
# "folderpath" variable construction down further in this script.
# This is the command to run to run mailbox commands.
ZMCMD='/opt/zimbra/bin/zmmailbox -z'
# This will be used for temporary/log files during the import process
TEMP='/tmp/'
# We assume the working directory's name is the domain.
# Otherwise, override this with your actual domain name.
domain=`basename ${PWD}`
echo Process ID: $$
if [[ $1 != "" ]] ; then
USERS=$1;
else
USERS=`cat vpasswd | cut -f1 -d:`
fi
for user in ${USERS}; do
echo "Beginning User: $user..."
if [[ $2 != "" ]] ; then
FOLDERS="$user/Maildir/$2/cur";
else
FOLDERS=`find $user -type d -name cur | sort`
fi
echo "$FOLDERS" | while read line; do
folderdir=`echo ${line} | cut -f3 -d"/"`
if [[ ${folderdir} == "cur" ]] ; then
folderdir="";
fi
folder=`echo ${folderdir} | sed 's/^\.//; s%\.%/%g; s%\&-%\&%g'`
folderpath=${PWD}/${user}/Maildir/${folderdir}/
# If the folder name is blank, this is the top level folder,
# Zimbra calls it "Inbox" (so do most clients/servers).
if [[ $folder == "" ]] ; then
folder="Inbox";
fi
# In Courier IMAP, all folders must be children of the root
# folder, which means Trash, Junk, Sent, Drafts are typically
# under Inbox. This is not the case with Zimbra, so we will
# slide these mailboxes to the top level so they behave properly,
# For all "non-special" mailboxes, we will keep them as children
# so they remain where the user had them before.
if [[ $folder != "Trash" && $folder != "Junk" && $folder != "Sent"
&& $folder != "Drafts" && $folder != "Inbox" ]] ; then
folder="Inbox/${folder}";
fi
echo "* Working on Folder $folder..."
# Courier allows heirarchy where non-folders (literally nothing) are
# able to have children. Zimbra does not. It's also possible that
# we will process the folders out of heirarchical order for some reason
# Here we separate the path and make sure all the parent folders exist
# before trying to create the folder we're working on.
parts=(`echo $folder | sed 's% %\x1a%g; s%/% %g'`);
hier="";
for i in "${parts[@]}"; do
hier=`echo ${hier}/$i | sed 's%^/%%; s%\x1a% %g'`;
${ZMCMD} -m ${user}@${domain} getFolder "/${hier}" >/dev/null 2>&1 ||
( echo -n " + Creating folder $hier... " &&
${ZMCMD} -m ${user}@${domain} createFolder "/${hier}" )
done
# Figure out how many messages we have
count=`find "${folderpath}new/" "${folderpath}cur/" -type f | wc -l`;
imported=0;
echo " * $count messages to process..."
# Define the temporary file names we will need
importfn="${TEMP}/import-$domain-$user-$folderdir-$$"
implogfn="${TEMP}/import-$domain-$user-$folderdir-$$-log"
impflogfn="${TEMP}/import-$domain-$user-$folderdir-$$-flaglog"
impflagfn="${TEMP}/import-$domain-$user-$folderdir-$$-flags"
touch "$importfn"
# Determine the courier extended flag identifiers ("keywords")
flagid=0
if [[ -f "${folderpath}courierimapkeywords/:list" ]] ; then
extflags="YES"
cat "${folderpath}courierimapkeywords/:list" 2>/dev/null | while read line; do
# A blank line indicates the end of the definitions.
if [[ "${line}" == "" ]]; then break; fi
# To avoid escape character madness, I'm swapping $ with % here.
flag=`echo ${line} | sed 's/\\\$/%/'`
echo courierflag[${flagid}]="'$flag'";
flagid=$(( flagid + 1 ));
# Create the tag if it doesn't start with '%'
if [[ `echo ${flag} | grep '%'` == "" ]] ; then
echo -n " + Attemping to create tag ${flag}... " >&2
${ZMCMD} -m ${user}@${domain} createTag "${flag}" >&2
fi
done > "$impflagfn"
source "$impflagfn"
fi
echo -n " * Queuing messages for import... "
# Find all "cur" or "new" messages in this folder and import them.
find "${folderpath}new/" "${folderpath}cur/" -type f | while read msg; do
flags="";
tags="";
msgid=`echo $msg | cut -d: -f1 | sed s%.*/%%`
# Determine the old maildir style flags
oldflags=`echo $msg | cut -d: -f2`
# Replied
if [[ `echo ${oldflags} | grep 'R'` != "" ]] ; then flags="${flags}r"; fi
# Seen
if [[ `echo ${oldflags} | grep 'S'` == "" ]] ; then flags="${flags}u"; fi
# Trashed
if [[ `echo ${oldflags} | grep 'T'` != "" ]] ; then flags="${flags}x"; fi
# Draft
if [[ `echo ${oldflags} | grep 'D'` != "" ]] ; then flags="${flags}d"; fi
# Flagged
if [[ `echo ${oldflags} | grep 'F'` != "" ]] ; then flags="${flags}f"; fi
# Determine the courier-imap extended flags for this message
if [[ ${extflags} == "YES" ]] ; then
oldflags2=`grep $msgid "${folderpath}courierimapkeywords/:list" 2>/dev/null | cut -d: -f2`
for flag in ${oldflags2}; do
# Forwarded
if [[ ${courierflag[$flag]} == '%Forwarded' ]] ; then flags="${flags}w"; fi
# Sent by me
if [[ ${courierflag[$flag]} == '%MDNSent' ]] ; then flags="${flags}s"; fi
# Convert non-system flags to Zimbra tags
if [[ `echo ${courierflag[$flag]} | grep '%'` == "" ]] ; then
tags="${tags},${courierflag[$flag]}"
fi
done
# Clean up the tag list for the command line
if [[ ${tags} != "" ]]; then
tags=`echo ${tags} | sed "s/^,\?/--tags \'/; s/\$/\'/"`;
fi
fi
# Log the result of flag processing for debugging
if [[ $flags != "" || $tags != "" ]] ; then
echo `date +%c` "$msg had flags $oldflags and $oldflags2, now $flags and $tags in folder $folder" >> "$impflogfn"
fi
# Add the command to the queue file to import this message
echo "addMessage --flags \"${flags}\" ${tags} \"/$folder\" \"${msg}\"" >> "$importfn"
imported=$(( $imported + 1 ));
printf "\b\b\b\b\b\b\b\b%7d " $imported;
done
echo "...done";
# Since we redirect the queue file to the mailbox tool, we end with "quit"
echo "quit" >> "$importfn"
# We're counting "prompts" from the zmmailbox utility here. The first
# one comes up before a message is imported, so we start at -1 to offset
# its existence.
imported=-1;
# We do this redirect because running the command for each message is very
# slow. We can't just pass the directory to the command, despite Zimbra's
# support because we can't tag or flag the messages that way.
echo -n " * Running import process... "
${ZMCMD} -m $user@$domain < "${importfn}" 2> "${implogfn}" | while read; do
imported=$(( $imported + 1 ));
printf "\b\b\b\b\b\b\b\b%7d " $imported;
done
if [[ -s "${implogfn}" ]]; then
echo "...some messages did not import correctly: check $importfn";
else
echo "...done";
fi
done
done
echo "Import Process Complete!"
Now my output for a single account is:
Process ID: 11495 Beginning User: user... * Working on Folder Inbox/Archive... + Creating folder Inbox/Archive... 285 * 0 messages to process... * Queuing messages for import... ...done * Running import process... 0 ...done * Working on Folder Drafts... * 0 messages to process... * Queuing messages for import... ...done * Running import process... 0 ...done * Working on Folder Junk... * 0 messages to process... * Queuing messages for import... ...done * Running import process... 0 ...done * Working on Folder Sent... * 3 messages to process... * Queuing messages for import... 3 ...done * Running import process... 3 ...done * Working on Folder Trash... * 0 messages to process... * Queuing messages for import... ...done * Running import process... 0 ...done * Working on Folder Inbox/new folder... + Creating folder Inbox/new folder... 289 * 5 messages to process... * Queuing messages for import... 5 ...done * Running import process... 5 ...done * Working on Folder Inbox... * 1 messages to process... * Queuing messages for import... 1 ...done * Running import process... 1 ...done Import Process Complete!
It is important to note that running the import script multiple times will result in the e-mails being imported multiple times and it takes a fair amount of time to perform this procedure on one account nevermind one thousand so a strategy should be formulated for dealing with runoff mail before the MX/target is switched.
