pitter patter on the keyboard

Round-Robin DNS gets trash-talked a lot because although it is a cheap and easy way to distribute loads it is counter-redundant: the more A records (servers) there are behind a domain the more points of failure there are and the lower your mean time to failure is going to be. The good news is that if one in five web servers/reverse proxies are down then only about one fifth of your audience is unable to connect at any given time.

The answer to this problem is host monitoring. If we can update our DNS records to remove the IPs of downed servers then add them back when the hosts recover no direct intervention on our part is required. Unfortunately, DNS is a heavily cached system so we will have to work with reasonably short timeouts. DNS Made Easy recommends a TTL of no less than 180 seconds as some ISPs are configured to ignore the TTLs of records which they deem are too short and default to a much higher value. The drawback to short TTLs is that you will end up receiving more DNS queries, which is a problem if you use a commercial billed-by-million-queries DNS provider like Amazon’s Route 53 or EasyDNS’s enterprise service.

If your objective is to have web server failover that happens instantly this is simply not the solution for you – you need a load balancer and/or anycast address space. Amazon’s Route53 and DNS Made Easy can be configured to check as often as every minute and it doesn’t make a lot of sense to run a ping/tcp test more often than that. At worst this means that the failover system doesn’t even know there is a problem for up to 60 seconds. Once the failover system updates the records there may be a short delay while the slave name servers synchronize. Then we have to wait for the record to expire at any-given-user’s ISP’s recursive name servers, which could take up to the TTL of your record or longer if their ISP is manipulative. Then you may have to wait for the record to expire in the caching DNS daemon on their home or office router. Then you may have to wait for the record to expire in their OS or browser’s DNS cache. This could take up to 15 minutes even if you use a very low TTL like 180.

So the question is: you already have DNS infrastructure. Why pay these large DNS outfits for host monitoring and DNS failover when it’s not really that great anyway and you can do it just as well as they can?

Just because BIND doesn’t have built-in support? Pshaw!

You could just as easily do the host monitoring with nagios/icinga or use the mysql-bind backend or even some other database-backed name daemon but in this article I’ll show you how to drop in a simple shell script that will work with your existing BIND installation because it demonstrates how mind-numbingly simple this is and why it shouldn’t be charged for as a premium service.

Observe a typical zone file with round-robin:

$TTL 6400       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                201305140       ; Serial
                                28800           ; Refresh
                                7200            ; Retry
                                60480           ; Expire
                                600 )           ; TTL Minimum
@               IN      A       10.0.0.10
@               IN      A       10.0.0.11
@               IN      A       10.0.0.12
@               IN      A       10.0.0.13
@               IN      A       10.0.0.14
*               IN      A       10.0.0.10
*               IN      A       10.0.0.11
*               IN      A       10.0.0.12
*               IN      A       10.0.0.13
*               IN      A       10.0.0.14
ns1             IN      A       10.0.1.10
ns2             IN      A       10.0.1.11
@               IN      NS      ns1.somedomain.com.
@               IN      NS      ns2.somedomain.com.
www             IN      CNAME   somedomain.com.

Our SOA contains the serial which will have to be updated by the script if our changes are to propagate properly. In the zone file on the master server(s) replace the SOA and block of round-robin A records with $INCLUDE statements like this:

$INCLUDE "/var/bind/soa.include"
$INCLUDE "/var/bind/ips.include"
ns1             IN      A       10.0.1.10
ns2             IN      A       10.0.1.11
@               IN      NS      ns1.somedomain.com.
@               IN      NS      ns2.somedomain.com.
www             IN      CNAME   somedomain.com.

Do this for every zone file which is to use this pool of A records. Now we have a centralized place to put the IPs and serial number that come from the shell script.

Create the script on the master name server and chmod +x it, don’t forget to update the paths to reflect your DNS situation. Also note that I’m adding a wildcard subdomain to the pool:

#!/bin/bash
HOSTS="10.0.0.10 10.0.0.11 10.0.0.12 10.0.0.13 10.0.0.14"
COUNT=4
echo "; Generated by monitor.sh $(date)" > /chroot/dns/var/bind/ips.include
for myHost in $HOSTS
do
  count=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')
  if [ $count -eq 0 ]; then
    # 100% failed 
    echo "$(date) $myHost is down" >> /var/log/monitor.log
  else
    echo "@               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include
    echo "*               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include
  fi

done

echo "; Generated by monitor.sh $(date)
\$TTL 300       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                $(date +%s)      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum" > /chroot/dns/var/bind/soa.include

rndc reload

This script will ping each host in the HOSTS array four times. If at least one ping is received the host is written to a new version of ips.include (note the single angle bracket when inserting the date). If all four pings fail a message will be recorded in /var/log/monitor.log. You may want to adjust the number of pings and failure tolerance, or replace the logging line with an e-mail notification instead. Once the ping tests are done a new soa.include is written with an epoch serial number and the zones are reloaded.

At the end of execution you should see something like this in ips.include:

; Generated by monitor.sh Tue May 14 16:15:26 EDT 2013
@               IN      A       10.0.0.10
*               IN      A       10.0.0.10
@               IN      A       10.0.0.11
*               IN      A       10.0.0.11
@               IN      A       10.0.0.12
*               IN      A       10.0.0.12
@               IN      A       10.0.0.13
*               IN      A       10.0.0.13
@               IN      A       10.0.0.14
*               IN      A       10.0.0.14

And in soa.include:

; Generated by monitor.sh Tue May 14 16:15:26 EDT 2013
$TTL 300       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                1368562526      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum

Note that you may need to chown named: the .include files after they are created the first time, depending on your environment.

I switched from using the widely popular YYYYMMDDID format to epoch since the 5 minute interval requires hours, minutes and seconds to be effective and YYYMMDDHHMMSS is too large a value for BIND. This resulted in a lower serial value – you may have to go around to your slaves and manually delete then reload their zone files.

This approach ends up generating a lot of NOTIFY traffic since every 5 minutes (or whatever interval you cron the shell script at) a new serial is loaded and all of your slaves have to be contacted. A more graceful improvement would be to save the state that each host is in inside of a temporary file and only update the serial when there has actually been a change in the status of your pool.

Another neat thing I thought of trying was using something like heartbeat for real-time monitoring and dnsupdate to dynamically update the zone files. This should narrow the propagation latency on your side of the equation down to the barest minimum possible.

The firewall on a RHEL system is configured by default with system-config-firewall, which on the console is an annoying ncurses menu which doesn’t permit adding custom ports/protocols.

The ports you probably want open for Zimbra are:

25
    smtp [mta] - incoming mail to postfix 
80
    http [mailbox] - web mail client 
110
    pop3 [mailbox] 
143
    imap [mailbox] 
443
    https [mailbox] - web mail client over ssl 
465
    smtps [mta] - incoming mail to postfix over ssl (Outlook only) 
587
    smtp [mta] - Mail submission over tls 
993
    imaps [mailbox] - imap over ssl 
995
    pops [mailbox] - pop over ssl 
7071
    https [mailbox] - admin console

The raw iptables configuration is stored in /etc/sysconfig/iptables:

# cat /etc/sysconfig/iptables
# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 25 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 110 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 143 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 456 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 587 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 993 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 995 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 7071 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Ensure the iptables init script is part of your default runlevel.

# chkconfig --level 345 iptables on

Restart it to apply the changes.

/etc/init.d/iptables restart

I have the good fortune of being forced to migrate over 1,100 e-mail accounts from a 15 year old qmail server with Courier-IMAP.

A script is provided at http://wiki.zimbra.com/wiki/Courier-IMAP_Maildir_to_zmmailbox. It should be easy to run as root and import over NFS. Unfortunately, I ran into trouble with the –noVerify flag, possibly due to the older version of Zimbra I’m importing to:

  * Running import process...           0 ...some messages did not import correctly: check /tmp//import-example.com-user--6108

The output of the log looks like:

addMessage --flags "u"  --noValidation "/Inbox" "/mnt/example.com/user/Maildir//cur/1362778133.14175.smtp.example.com:2,"

To test it you’re going to have to look at the prefix in the script:

/opt/zimbra/bin/zmmailbox -m user@example.com -z addMessage --flags "u"  --noValidation "/Inbox" "/mnt/example.com/user/Maildir//cur/1362778133.14175.smtp.example.com:2,"

Which gave me the following error:

ERROR: zclient.CLIENT_ERROR (unknown folder: --noValidation)

I tried putting the –noValidation flag in every position with no luck, and it is listed in the Zimbra wiki article for zmmailbox at http://wiki.zimbra.com/wiki/Zmmailbox so I am left to assume my version simply does not support it. It doesn’t seem to be consequential anyway so I simply removed it from the script.

#!/bin/bash                                                                                                                                                                                                                                                                    
#
# courier/vpopmail Maildir to Zimbra Import
#
# This script can be stored anywhere, but you should run it while in the root
# of the domain's users.  It looks for the file vpasswd which contains a
# line-separated list of users and uses that to import.  You can also run the
# script with a user name to process a single user.  Additionally, you can
# specify a folder name (courier format) to process a single folder for that
# user.

# We assume the folder structure is like this:
# Inbox: <working directory>/<user>/Maildir/<cur|new>
# Subfolder: <working directory>/<user>/Maildir/.Subfolder/<cur|new>
# If this is not what your structure looks like, you need to change the
# "folderpath" variable construction down further in this script.

# This is the command to run to run mailbox commands.
ZMCMD='/opt/zimbra/bin/zmmailbox -z'

# This will be used for temporary/log files during the import process
TEMP='/tmp/'

# We assume the working directory's name is the domain.
# Otherwise, override this with your actual domain name.
domain=`basename ${PWD}`

echo Process ID: $$

if [[ $1 != "" ]] ; then
  USERS=$1;
else
  USERS=`cat vpasswd | cut -f1 -d:`
fi

for user in ${USERS}; do
  echo "Beginning User: $user..."

  if [[ $2 != "" ]] ; then
    FOLDERS="$user/Maildir/$2/cur";
  else
    FOLDERS=`find $user -type d -name cur | sort`
  fi

  echo "$FOLDERS" | while read line; do
    folderdir=`echo ${line} | cut -f3 -d"/"`
    if [[ ${folderdir} == "cur" ]] ; then
      folderdir="";
    fi 
    folder=`echo ${folderdir} | sed 's/^\.//; s%\.%/%g; s%\&-%\&%g'`
    folderpath=${PWD}/${user}/Maildir/${folderdir}/
    
    # If the folder name is blank, this is the top level folder,
    # Zimbra calls it "Inbox" (so do most clients/servers).
    if [[ $folder == "" ]] ; then
      folder="Inbox";
    fi
    # In Courier IMAP, all folders must be children of the root
    # folder, which means Trash, Junk, Sent, Drafts are typically
    # under Inbox. This is not the case with Zimbra, so we will
    # slide these mailboxes to the top level so they behave properly,
    # For all "non-special" mailboxes, we will keep them as children
    # so they remain where the user had them before.
    if [[ $folder != "Trash" && $folder != "Junk" && $folder != "Sent"
       && $folder != "Drafts" && $folder != "Inbox" ]] ; then
      folder="Inbox/${folder}";
    fi
    echo "* Working on Folder $folder..."

    # Courier allows heirarchy where non-folders (literally nothing) are
    # able to have children.  Zimbra does not.  It's also possible that
    # we will process the folders out of heirarchical order for some reason
    # Here we separate the path and make sure all the parent folders exist
    # before trying to create the folder we're working on.
    parts=(`echo $folder | sed 's% %\x1a%g; s%/% %g'`);
    hier="";
    for i in "${parts[@]}"; do
      hier=`echo ${hier}/$i | sed 's%^/%%; s%\x1a% %g'`;
      ${ZMCMD} -m ${user}@${domain} getFolder "/${hier}" >/dev/null 2>&1 ||
      ( echo -n "  + Creating folder $hier... " &&
      ${ZMCMD} -m ${user}@${domain} createFolder "/${hier}" )
    done

    # Figure out how many messages we have
    count=`find "${folderpath}new/" "${folderpath}cur/" -type f | wc -l`;
    imported=0;
    echo "  * $count messages to process..."

    # Define the temporary file names we will need
    importfn="${TEMP}/import-$domain-$user-$folderdir-$$"
    implogfn="${TEMP}/import-$domain-$user-$folderdir-$$-log"
    impflogfn="${TEMP}/import-$domain-$user-$folderdir-$$-flaglog"
    impflagfn="${TEMP}/import-$domain-$user-$folderdir-$$-flags"
    touch "$importfn"

    # Determine the courier extended flag identifiers ("keywords")
    flagid=0
    if [[ -f "${folderpath}courierimapkeywords/:list" ]] ; then
      extflags="YES"
      cat "${folderpath}courierimapkeywords/:list" 2>/dev/null | while read line; do
        # A blank line indicates the end of the definitions.
        if [[ "${line}" == "" ]]; then break; fi

        # To avoid escape character madness, I'm swapping $ with % here.
        flag=`echo ${line} | sed 's/\\\$/%/'`
        echo courierflag[${flagid}]="'$flag'";
        flagid=$(( flagid + 1 ));

        # Create the tag if it doesn't start with '%'
        if [[ `echo ${flag} | grep '%'` == "" ]] ; then
          echo -n "  + Attemping to create tag ${flag}... " >&2
          ${ZMCMD} -m ${user}@${domain} createTag "${flag}" >&2
        fi

      done > "$impflagfn"
      source "$impflagfn"
    fi

    echo -n "  * Queuing messages for import...        " 

    # Find all "cur" or "new" messages in this folder and import them.
    find "${folderpath}new/" "${folderpath}cur/" -type f | while read msg; do
      flags="";
      tags="";
      msgid=`echo $msg | cut -d: -f1 | sed s%.*/%%`

      # Determine the old maildir style flags
      oldflags=`echo $msg | cut -d: -f2`
      # Replied
      if [[ `echo ${oldflags} | grep 'R'` != "" ]] ; then flags="${flags}r"; fi
      # Seen
      if [[ `echo ${oldflags} | grep 'S'` == "" ]] ; then flags="${flags}u"; fi
      # Trashed
      if [[ `echo ${oldflags} | grep 'T'` != "" ]] ; then flags="${flags}x"; fi
      # Draft
      if [[ `echo ${oldflags} | grep 'D'` != "" ]] ; then flags="${flags}d"; fi
      # Flagged
      if [[ `echo ${oldflags} | grep 'F'` != "" ]] ; then flags="${flags}f"; fi

      # Determine the courier-imap extended flags for this message
      if [[ ${extflags} == "YES" ]] ; then
        oldflags2=`grep $msgid "${folderpath}courierimapkeywords/:list" 2>/dev/null | cut -d: -f2`
        for flag in ${oldflags2}; do
          # Forwarded
          if [[ ${courierflag[$flag]} == '%Forwarded' ]] ; then flags="${flags}w"; fi
          # Sent by me
          if [[ ${courierflag[$flag]} == '%MDNSent' ]] ;   then flags="${flags}s"; fi
          # Convert non-system flags to Zimbra tags
          if [[ `echo ${courierflag[$flag]} | grep '%'` == "" ]] ; then
            tags="${tags},${courierflag[$flag]}"
          fi
        done
        # Clean up the tag list for the command line
        if [[ ${tags} != "" ]]; then
          tags=`echo ${tags} | sed "s/^,\?/--tags \'/; s/\$/\'/"`;
        fi
      fi

      # Log the result of flag processing for debugging
      if [[ $flags != "" || $tags != "" ]] ; then
        echo `date +%c` "$msg had flags $oldflags and $oldflags2, now $flags and $tags in folder $folder" >> "$impflogfn"
      fi

      # Add the command to the queue file to import this message
      echo "addMessage --flags \"${flags}\" ${tags} \"/$folder\" \"${msg}\"" >> "$importfn"

      imported=$(( $imported + 1 ));
      printf "\b\b\b\b\b\b\b\b%7d " $imported;
    done

    echo "...done";

    # Since we redirect the queue file to the mailbox tool, we end with "quit"
    echo "quit" >> "$importfn"

    # We're counting "prompts" from the zmmailbox utility here.  The first
    # one comes up before a message is imported, so we start at -1 to offset
    # its existence.
    imported=-1;

    # We do this redirect because running the command for each message is very
    # slow.  We can't just pass the directory to the command, despite Zimbra's
    # support because we can't tag or flag the messages that way.
    echo -n "  * Running import process...             "
    ${ZMCMD} -m $user@$domain < "${importfn}" 2> "${implogfn}" | while read; do
      imported=$(( $imported + 1 ));
      printf "\b\b\b\b\b\b\b\b%7d " $imported;
    done

    if [[ -s "${implogfn}" ]]; then 
      echo "...some messages did not import correctly: check $importfn";
    else
      echo "...done";
    fi
  done
done
echo "Import Process Complete!"

Now my output for a single account is:

Process ID: 11495
Beginning User: user...
* Working on Folder Inbox/Archive...
  + Creating folder Inbox/Archive... 285
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Drafts...
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Junk...
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Sent...
  * 3 messages to process...
  * Queuing messages for import...      3 ...done
  * Running import process...           3 ...done
* Working on Folder Trash...
  * 0 messages to process...
  * Queuing messages for import...        ...done
  * Running import process...           0 ...done
* Working on Folder Inbox/new folder...
  + Creating folder Inbox/new folder... 289
  * 5 messages to process...
  * Queuing messages for import...      5 ...done
  * Running import process...           5 ...done
* Working on Folder Inbox...
  * 1 messages to process...
  * Queuing messages for import...      1 ...done
  * Running import process...           1 ...done
Import Process Complete!

It is important to note that running the import script multiple times will result in the e-mails being imported multiple times and it takes a fair amount of time to perform this procedure on one account nevermind one thousand so a strategy should be formulated for dealing with runoff mail before the MX/target is switched.