=^.^=

Import Courier-IMAP Maildir E-Mail and IMAP Folders to Zimbra

karma

I have the good fortune of being forced to migrate over 1,100 e-mail accounts from a 15 year old qmail server with Courier-IMAP.

A script is provided at http://wiki.zimbra.com/wiki/Courier-IMAP_Maildir_to_zmmailbox. It should be easy to run as root and import over NFS. Unfortunately, I ran into trouble with the --noVerify flag, possibly due to the older version of Zimbra I'm importing to:

  * Running import process...           0 ...some messages did not import correctly: check /tmp//import-example.com-user--6108

The output of the log looks like:

addMessage --flags "u"  --noValidation "/Inbox" "/mnt/example.com/user/Maildir//cur/1362778133.14175.smtp.example.com:2,"

To test it you're going to have to look at the prefix in the script:

/opt/zimbra/bin/zmmailbox -m [email protected] -z addMessage --flags "u"  --noValidation "/Inbox" "/mnt/example.com/user/Maildir//cur/1362778133.14175.smtp.example.com:2,"

Which gave me the following error:

ERROR: zclient.CLIENT_ERROR (unknown folder: --noValidation)

I tried putting the --noValidation flag in every position with no luck, and it is listed in the Zimbra wiki article for zmmailbox at http://wiki.zimbra.com/wiki/Zmmailbox so I am left to assume my version simply does not support it. It doesn't seem to be consequential anyway so I simply removed it from the script.

#!/bin/bash                                                                                                                                                                                                                                                                    
#
# courier/vpopmail Maildir to Zimbra Import
#
# This script can be stored anywhere, but you should run it while in the root
# of the domain's users.  It looks for the file vpasswd which contains a
# line-separated list of users and uses that to import.  You can also run the
# script with a user name to process a single user.  Additionally, you can
# specify a folder name (courier format) to process a single folder for that
# user.

# We assume the folder structure is like this:
# Inbox: <working directory>/<user>/Maildir/<cur|new>
# Subfolder: <working directory>/<user>/Maildir/.Subfolder/<cur|new>
# If this is not what your structure looks like, you need to change the
# "folderpath" variable construction down further in this script.

# This is the command to run to run mailbox commands.
ZMCMD='/opt/zimbra/bin/zmmailbox -z'

# This will be used for temporary/log files during the import process
TEMP='/tmp/'

# We assume the working directory's name is the domain.
# Otherwise, override this with your actual domain name.
domain=`basename ${PWD}`

echo Process ID: $$

if  $1 != ""  ; then
  USERS=$1;
else
  USERS=`cat vpasswd | cut -f1 -d:`
fi

for user in ${USERS}; do
  echo "Beginning User: $user..."

  if  $2 != ""  ; then
    FOLDERS="$user/Maildir/$2/cur";
  else
    FOLDERS=`find $user -type d -name cur | sort`
  fi

  echo "$FOLDERS" | while read line; do
    folderdir=`echo ${line} | cut -f3 -d"/"`
    if  ${folderdir} == "cur"  ; then
      folderdir="";
    fi 
    folder=`echo ${folderdir} | sed 's/^\.//; s%\.%/%g; s%\&-%\&%g'`
    folderpath=${PWD}/${user}/Maildir/${folderdir}/
    
    # If the folder name is blank, this is the top level folder,
    # Zimbra calls it "Inbox" (so do most clients/servers).
    if  $folder == ""  ; then
      folder="Inbox";
    fi
    # In Courier IMAP, all folders must be children of the root
    # folder, which means Trash, Junk, Sent, Drafts are typically
    # under Inbox. This is not the case with Zimbra, so we will
    # slide these mailboxes to the top level so they behave properly,
    # For all "non-special" mailboxes, we will keep them as children
    # so they remain where the user had them before.
    if [[ $folder != "Trash" && $folder != "Junk" && $folder != "Sent"
       && $folder != "Drafts" && $folder != "Inbox" ]] ; then
      folder="Inbox/${folder}";
    fi
    echo "* Working on Folder $folder..."

    # Courier allows heirarchy where non-folders (literally nothing) are
    # able to have children.  Zimbra does not.  It's also possible that
    # we will process the folders out of heirarchical order for some reason
    # Here we separate the path and make sure all the parent folders exist
    # before trying to create the folder we're working on.
    parts=(`echo $folder | sed 's% %\x1a%g; s%/% %g'`);
    hier="";
    for i in "${parts[@]}"; do
      hier=`echo ${hier}/$i | sed 's%^/%%; s%\x1a% %g'`;
      ${ZMCMD} -m ${user}@${domain} getFolder "/${hier}" >/dev/null 2>&1 ||
      ( echo -n "  + Creating folder $hier... " &&
      ${ZMCMD} -m ${user}@${domain} createFolder "/${hier}" )
    done

    # Figure out how many messages we have
    count=`find "${folderpath}new/" "${folderpath}cur/" -type f | wc -l`;
    imported=0;
    echo "  * $count messages to process..."

    # Define the temporary file names we will need
    importfn="${TEMP}/import-$domain-$user-$folderdir-$$"
    implogfn="${TEMP}/import-$domain-$user-$folderdir-$$-log"
    impflogfn="${TEMP}/import-$domain-$user-$folderdir-$$-flaglog"
    impflagfn="${TEMP}/import-$domain-$user-$folderdir-$$-flags"
    touch "$importfn"

    # Determine the courier extended flag identifiers ("keywords")
    flagid=0
    if  -f "${folderpath}courierimapkeywords/:list"  ; then
      extflags="YES"
      cat "${folderpath}courierimapkeywords/:list" 2>/dev/null | while read line; do
        # A blank line indicates the end of the definitions.
        if  "${line}" == "" ; then break; fi

        # To avoid escape character madness, I'm swapping $ with % here.
        flag=`echo ${line} | sed 's/\\\$/%/'`
        echo courierflag[${flagid}]="'$flag'";
        flagid=$(( flagid + 1 ));

        # Create the tag if it doesn't start with '%'
        if  `echo ${flag} | grep '%'` == ""  ; then
          echo -n "  + Attemping to create tag ${flag}... " >&2
          ${ZMCMD} -m ${user}@${domain} createTag "${flag}" >&2
        fi

      done > "$impflagfn"
      source "$impflagfn"
    fi

    echo -n "  * Queuing messages for import...        " 

    # Find all "cur" or "new" messages in this folder and import them.
    find "${folderpath}new/" "${folderpath}cur/" -type f | while read msg; do
      flags="";
      tags="";
      msgid=`echo $msg | cut -d: -f1 | sed s%.*/%%`

      # Determine the old maildir style flags
      oldflags=`echo $msg | cut -d: -f2`
      # Replied
      if  `echo ${oldflags} | grep 'R'` != ""  ; then flags="${flags}r"; fi
      # Seen
      if  `echo ${oldflags} | grep 'S'` == ""  ; then flags="${flags}u"; fi
      # Trashed
      if  `echo ${oldflags} | grep 'T'` != ""  ; then flags="${flags}x"; fi
      # Draft
      if  `echo ${oldflags} | grep 'D'` != ""  ; then flags="${flags}d"; fi
      # Flagged
      if  `echo ${oldflags} | grep 'F'` != ""  ; then flags="${flags}f"; fi

      # Determine the courier-imap extended flags for this message
      if  ${extflags} == "YES"  ; then
        oldflags2=`grep $msgid "${folderpath}courierimapkeywords/:list" 2>/dev/null | cut -d: -f2`
        for flag in ${oldflags2}; do
          # Forwarded
          if  ${courierflag[$flag]} == '%Forwarded'  ; then flags="${flags}w"; fi
          # Sent by me
          if  ${courierflag[$flag]} == '%MDNSent'  ;   then flags="${flags}s"; fi
          # Convert non-system flags to Zimbra tags
          if  `echo ${courierflag[$flag]} | grep '%'` == ""  ; then
            tags="${tags},${courierflag[$flag]}"
          fi
        done
        # Clean up the tag list for the command line
        if  ${tags} != "" ; then
          tags=`echo ${tags} | sed "s/^,\?/--tags \'/; s/\$/\'/"`;
        fi
      fi

      # Log the result of flag processing for debugging
      if  $flags != "" || $tags != ""  ; then
        echo `date +%c` "$msg had flags $oldflags and $oldflags2, now $flags and $tags in folder $folder" >> "$impflogfn"
      fi

      # Add the command to the queue file to import this message
      echo "addMessage --flags \"${flags}\" ${tags} \"/$folder\" \"${msg}\"" >> "$importfn"

      imported=$(( $imported + 1 ));
      printf "\b\b\b\b\b\b\b\b%7d " $imported;
    done

    echo "...done";

    # Since we redirect the queue file to the mailbox tool, we end with "quit"
    echo "quit" >> "$importfn"

    # We're counting "prompts" from the zmmailbox utility here.  The first
    # one comes up before a message is imported, so we start at -1 to offset
    # its existence.
    imported=-1;

    # We do this redirect because running the command for each message is very
    # slow.  We can't just pass the directory to the command, despite Zimbra's
    # support because we can't tag or flag the messages that way.
    echo -n "  * Running import process...             "
    ${ZMCMD} -m $user@$domain < "${importfn}" 2> "${implogfn}" | while read; do
      imported=$(( $imported + 1 ));
      printf "\b\b\b\b\b\b\b\b%7d " $imported;
    done

    if  -s "${implogfn}" ; then 
      echo "...some messages did not import correctly: check $importfn";
    else
      echo "...done";
    fi
  done
done
echo "Import Process Complete!"

Now my output for a single account is:

Process ID: 11495
Beginning User: user...
* Working on Folder Inbox/Archive...
  + Creating folder Inbox/Archive... 285
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Drafts...
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Junk...
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Sent...
  * 3 messages to process...
  * Queuing messages for import...      3 ...done
  * Running import process...           3 ...done
* Working on Folder Trash...
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Inbox/new folder...
  + Creating folder Inbox/new folder... 289
  * 5 messages to process...
  * Queuing messages for import...      5 ...done
  * Running import process...           5 ...done
* Working on Folder Inbox...
  * 1 messages to process...
  * Queuing messages for import...      1 ...done
  * Running import process...           1 ...done
Import Process Complete!

It is important to note that running the import script multiple times will result in the e-mails being imported multiple times and it takes a fair amount of time to perform this procedure on one account nevermind one thousand so a strategy should be formulated for dealing with runoff mail before the MX/target is switched.

Comments

There are no comments for this item.