pitter patter on the keyboard

Round-Robin DNS gets trash-talked a lot because although it is a cheap and easy way to distribute loads it is counter-redundant: the more A records (servers) there are behind a domain the more points of failure there are and the lower your mean time to failure is going to be. The good news is that if one in five web servers/reverse proxies are down then only about one fifth of your audience is unable to connect at any given time.

The answer to this problem is host monitoring. If we can update our DNS records to remove the IPs of downed servers then add them back when the hosts recover no direct intervention on our part is required. Unfortunately, DNS is a heavily cached system so we will have to work with reasonably short timeouts. DNS Made Easy recommends a TTL of no less than 180 seconds as some ISPs are configured to ignore the TTLs of records which they deem are too short and default to a much higher value. The drawback to short TTLs is that you will end up receiving more DNS queries, which is a problem if you use a commercial billed-by-million-queries DNS provider like Amazon’s Route 53 or EasyDNS’s enterprise service.

If your objective is to have web server failover that happens instantly this is simply not the solution for you – you need a load balancer and/or anycast address space. Amazon’s Route53 and DNS Made Easy can be configured to check as often as every minute and it doesn’t make a lot of sense to run a ping/tcp test more often than that. At worst this means that the failover system doesn’t even know there is a problem for up to 60 seconds. Once the failover system updates the records there may be a short delay while the slave name servers synchronize. Then we have to wait for the record to expire at any-given-user’s ISP’s recursive name servers, which could take up to the TTL of your record or longer if their ISP is manipulative. Then you may have to wait for the record to expire in the caching DNS daemon on their home or office router. Then you may have to wait for the record to expire in their OS or browser’s DNS cache. This could take up to 15 minutes even if you use a very low TTL like 180.

So the question is: you already have DNS infrastructure. Why pay these large DNS outfits for host monitoring and DNS failover when it’s not really that great anyway and you can do it just as well as they can?

Just because BIND doesn’t have built-in support? Pshaw!

You could just as easily do the host monitoring with nagios/icinga or use the mysql-bind backend or even some other database-backed name daemon but in this article I’ll show you how to drop in a simple shell script that will work with your existing BIND installation because it demonstrates how mind-numbingly simple this is and why it shouldn’t be charged for as a premium service.

Observe a typical zone file with round-robin:

$TTL 6400       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                201305140       ; Serial
                                28800           ; Refresh
                                7200            ; Retry
                                60480           ; Expire
                                600 )           ; TTL Minimum
@               IN      A       10.0.0.10
@               IN      A       10.0.0.11
@               IN      A       10.0.0.12
@               IN      A       10.0.0.13
@               IN      A       10.0.0.14
*               IN      A       10.0.0.10
*               IN      A       10.0.0.11
*               IN      A       10.0.0.12
*               IN      A       10.0.0.13
*               IN      A       10.0.0.14
ns1             IN      A       10.0.1.10
ns2             IN      A       10.0.1.11
@               IN      NS      ns1.somedomain.com.
@               IN      NS      ns2.somedomain.com.
www             IN      CNAME   somedomain.com.

Our SOA contains the serial which will have to be updated by the script if our changes are to propagate properly. In the zone file on the master server(s) replace the SOA and block of round-robin A records with $INCLUDE statements like this:

$INCLUDE "/var/bind/soa.include"
$INCLUDE "/var/bind/ips.include"
ns1             IN      A       10.0.1.10
ns2             IN      A       10.0.1.11
@               IN      NS      ns1.somedomain.com.
@               IN      NS      ns2.somedomain.com.
www             IN      CNAME   somedomain.com.

Do this for every zone file which is to use this pool of A records. Now we have a centralized place to put the IPs and serial number that come from the shell script.

Create the script on the master name server and chmod +x it, don’t forget to update the paths to reflect your DNS situation. Also note that I’m adding a wildcard subdomain to the pool:

#!/bin/bash
HOSTS="10.0.0.10 10.0.0.11 10.0.0.12 10.0.0.13 10.0.0.14"
COUNT=4
echo "; Generated by monitor.sh $(date)" > /chroot/dns/var/bind/ips.include
for myHost in $HOSTS
do
  count=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')
  if [ $count -eq 0 ]; then
    # 100% failed 
    echo "$(date) $myHost is down" >> /var/log/monitor.log
  else
    echo "@               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include
    echo "*               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include
  fi

done

echo "; Generated by monitor.sh $(date)
\$TTL 300       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                $(date +%s)      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum" > /chroot/dns/var/bind/soa.include

rndc reload

This script will ping each host in the HOSTS array four times. If at least one ping is received the host is written to a new version of ips.include (note the single angle bracket when inserting the date). If all four pings fail a message will be recorded in /var/log/monitor.log. You may want to adjust the number of pings and failure tolerance, or replace the logging line with an e-mail notification instead. Once the ping tests are done a new soa.include is written with an epoch serial number and the zones are reloaded.

At the end of execution you should see something like this in ips.include:

; Generated by monitor.sh Tue May 14 16:15:26 EDT 2013
@               IN      A       10.0.0.10
*               IN      A       10.0.0.10
@               IN      A       10.0.0.11
*               IN      A       10.0.0.11
@               IN      A       10.0.0.12
*               IN      A       10.0.0.12
@               IN      A       10.0.0.13
*               IN      A       10.0.0.13
@               IN      A       10.0.0.14
*               IN      A       10.0.0.14

And in soa.include:

; Generated by monitor.sh Tue May 14 16:15:26 EDT 2013
$TTL 300       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                1368562526      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum

Note that you may need to chown named: the .include files after they are created the first time, depending on your environment.

I switched from using the widely popular YYYYMMDDID format to epoch since the 5 minute interval requires hours, minutes and seconds to be effective and YYYMMDDHHMMSS is too large a value for BIND. This resulted in a lower serial value – you may have to go around to your slaves and manually delete then reload their zone files.

This approach ends up generating a lot of NOTIFY traffic since every 5 minutes (or whatever interval you cron the shell script at) a new serial is loaded and all of your slaves have to be contacted. A more graceful improvement would be to save the state that each host is in inside of a temporary file and only update the serial when there has actually been a change in the status of your pool.

Another neat thing I thought of trying was using something like heartbeat for real-time monitoring and dnsupdate to dynamically update the zone files. This should narrow the propagation latency on your side of the equation down to the barest minimum possible.

The PS561 Voice Dialer Security Console integrates up to 16 security sensors with your X10 home automation system. When the alarm is tripped the console will dial up to four preprogrammed numbers and allow the party on the other end to listen in. This suffers, of course, from the fact that the mic is in the same package as two peizo sirens.

This model is the predecessor to the SC9000 which is much prettier and includes a touch tone security and X10 module control dial-in menu. Modules advertised for either system are compatible with any X10 console, even the new versions of the door/window sensors:

As you can see, the DS12A is much smaller (and therefore sexier) however the integrated magnetic sensor makes placement less versatile. The new sensors take an additional magnetic sensor through the terminals at the bottom so up to two doors or windows can be monitored with one device. This does not, however, provide exactly the same functionality as the DS10A’s external sensor as the internal sensor must be bypassed if only the external one is to be used.

You may find yourself forced to install these switches upside down on left-to-right opening vectors such as patio doors. This would be fine if the X10 logo wasn’t printed on the front.

The glass break sensors (GB10A) use an adhesive backing to stick right on your window panes. Though they can apparently detect a window breaking within 20 feet to reduce the potential for false alarms I have installed them on every pane of glass (two per window) and put them on their lowest sensitivity.
All of these modules are installed by sliding the console’s switch from the RUN1 or RUN2 positions to the INSTALL position then pressing the TEST button and, when finished installing all devices, returning the console’s switch to RUN1 or RUN2.

When the alarm is tripped the console sends alternating ON/OFF commands to the house and device code you have configured. The original purpose of this was to flash the outside lights to make your house easier for law enforcement/security to identify but some clever duck realized this signal could also be used to set off remote alarms. The PowerHorn (SH10A) is a module that screws into an outlet anywhere in your home and blasts its four peizo sirens much louder than the security console on its own. If you have a large dwelling space multiple units liberally to ensure a traumatic experience for intruders. The only drawback to these sirens is they are prone to false positives; if you remotely turn on and off the lights associated with their house and device code four times quickly they will go off momentarily which can be quite undesirable at some hours.

Last but not least, of course, the KR10A security keyfob. The lights on/off buttons control the lights on the address the console has been configured for so you can, for example, turn on the outside lights when you exit your vehicle. I haven’t had any false positives with the panic feature yet but it should be noted that the cover over the buttons dimples and wears out quickly.

One of the bathroom fans in my new pad was installed by the same Joe Handyman that brought you electrical tape on a wall switch and the decorative phone jack.

Not only is the fan wired in series with the light switch, the switch is screwed into the drywall and the neutral is running up the outside of the wall!

Through a giant, ugly, silicone-plugged hole and into the bathroom:

Connected to the fan with a thin twist of electrical tape!

I cut a hole in the drywall above the switches to help me run the cable around the stud to the right of the door:

When cutting a hole you intend to patch over later make an angled incision in the same fashion one would the top of a jack-o-lantern, such that the wedge can be re-seated with spackling paste without falling through.

North American electrical codes call for at least 14-gague non-metallic sheathed wire for 15A circuits, commonly referred to by the trademark Romex.

The sides of most single-gang electrical boxes can be unscrewed and removed so that two or more boxes can be joined together.

Make sure all of your grounds are securely connected to the electrical box and the neutrals have been connected and capped. I’ve connected a WS12A X10 remote control dimmer and a regular decorator-style toggle switch for the fan.

If you are installing remodelling or “old-work” boxes such as these (you can tell the difference by the tabs on the top and bottom for mounting on the surface of drywall) cut the drywall neatly around it so that it fits snugly in the wall. You can see from the first image that this hole was already made too high to accommodate the original old-work box which was recessed (without removing the tabs) and screwed into the door frame stud like a new-work box; while undesirable this shouldn’t be too much of a problem since the faceplate will cover the gap and the electrical box can be sufficiently secured with the bottom tabs alone.

I don’t recommend mounting new-work boxes until you have a faceplate handy since drywall will not tolerate minor adjustments in screw positions.

Now spackle and re-seat any plugs you’ve removed from the wall. I used Poly Filla Big, a 2 inch putty knife and a damp J-cloth to patch this hole. Simply sand, level and paint once the spackling has set to make it disappear. The hard part will be hiding all that silicone numbnuts left me…