Posts Tagged ‘shell’

Quick and Dirty (and Free!) Host Monitoring for DNS Failover and Round-Robin

Round-Robin DNS gets trash-talked a lot because although it is a cheap and easy way to distribute loads it is counter-redundant: the more A records (servers) there are behind a domain the more points of failure there are and the lower your mean time to failure is going to be. The good news is that if one in five web servers/reverse proxies are down then only about one fifth of your audience is unable to connect at any given time.

The answer to this problem is host monitoring. If we can update our DNS records to remove the IPs of downed servers then add them back when the hosts recover no direct intervention on our part is required. Unfortunately, DNS is a heavily cached system so we will have to work with reasonably short timeouts. DNS Made Easy recommends a TTL of no less than 180 seconds as some ISPs are configured to ignore the TTLs of records which they deem are too short and default to a much higher value. The drawback to short TTLs is that you will end up receiving more DNS queries, which is a problem if you use a commercial billed-by-million-queries DNS provider like Amazon’s Route 53 or EasyDNS’s enterprise service.

If your objective is to have web server failover that happens instantly this is simply not the solution for you – you need a load balancer and/or anycast address space. Amazon’s Route53 and DNS Made Easy can be configured to check as often as every minute and it doesn’t make a lot of sense to run a ping/tcp test more often than that. At worst this means that the failover system doesn’t even know there is a problem for up to 60 seconds. Once the failover system updates the records there may be a short delay while the slave name servers synchronize. Then we have to wait for the record to expire at any-given-user’s ISP’s recursive name servers, which could take up to the TTL of your record or longer if their ISP is manipulative. Then you may have to wait for the record to expire in the caching DNS daemon on their home or office router. Then you may have to wait for the record to expire in their OS or browser’s DNS cache. This could take up to 15 minutes even if you use a very low TTL like 180.

So the question is: you already have DNS infrastructure. Why pay these large DNS outfits for host monitoring and DNS failover when it’s not really that great anyway and you can do it just as well as they can?

Just because BIND doesn’t have built-in support? Pshaw!

You could just as easily do the host monitoring with nagios/icinga or use the mysql-bind backend or even some other database-backed name daemon but in this article I’ll show you how to drop in a simple shell script that will work with your existing BIND installation because it demonstrates how mind-numbingly simple this is and why it shouldn’t be charged for as a premium service.

Observe a typical zone file with round-robin:

$TTL 6400       ; max TTL
@       IN      SOA (
                                201305140       ; Serial
                                28800           ; Refresh
                                7200            ; Retry
                                60480           ; Expire
                                600 )           ; TTL Minimum
@               IN      A
@               IN      A
@               IN      A
@               IN      A
@               IN      A
*               IN      A
*               IN      A
*               IN      A
*               IN      A
*               IN      A
ns1             IN      A
ns2             IN      A
@               IN      NS
@               IN      NS
www             IN      CNAME

Our SOA contains the serial which will have to be updated by the script if our changes are to propagate properly. In the zone file on the master server(s) replace the SOA and block of round-robin A records with $INCLUDE statements like this:

$INCLUDE "/var/bind/soa.include"
$INCLUDE "/var/bind/ips.include"
ns1             IN      A
ns2             IN      A
@               IN      NS
@               IN      NS
www             IN      CNAME

Do this for every zone file which is to use this pool of A records. Now we have a centralized place to put the IPs and serial number that come from the shell script.

Create the script on the master name server and chmod +x it, don’t forget to update the paths to reflect your DNS situation. Also note that I’m adding a wildcard subdomain to the pool:

echo "; Generated by $(date)" > /chroot/dns/var/bind/ips.include
for myHost in $HOSTS
  count=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')
  if [ $count -eq 0 ]; then
    # 100% failed 
    echo "$(date) $myHost is down" >> /var/log/monitor.log
    echo "@               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include
    echo "*               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include


echo "; Generated by $(date)
\$TTL 300       ; max TTL
@       IN      SOA (
                                $(date +%s)      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum" > /chroot/dns/var/bind/soa.include

rndc reload

This script will ping each host in the HOSTS array four times. If at least one ping is received the host is written to a new version of ips.include (note the single angle bracket when inserting the date). If all four pings fail a message will be recorded in /var/log/monitor.log. You may want to adjust the number of pings and failure tolerance, or replace the logging line with an e-mail notification instead. Once the ping tests are done a new soa.include is written with an epoch serial number and the zones are reloaded.

At the end of execution you should see something like this in ips.include:

; Generated by Tue May 14 16:15:26 EDT 2013
@               IN      A
*               IN      A
@               IN      A
*               IN      A
@               IN      A
*               IN      A
@               IN      A
*               IN      A
@               IN      A
*               IN      A

And in soa.include:

; Generated by Tue May 14 16:15:26 EDT 2013
$TTL 300       ; max TTL
@       IN      SOA (
                                1368562526      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum

Note that you may need to chown named: the .include files after they are created the first time, depending on your environment.

I switched from using the widely popular YYYYMMDDID format to epoch since the 5 minute interval requires hours, minutes and seconds to be effective and YYYMMDDHHMMSS is too large a value for BIND. This resulted in a lower serial value – you may have to go around to your slaves and manually delete then reload their zone files.

This approach ends up generating a lot of NOTIFY traffic since every 5 minutes (or whatever interval you cron the shell script at) a new serial is loaded and all of your slaves have to be contacted. A more graceful improvement would be to save the state that each host is in inside of a temporary file and only update the serial when there has actually been a change in the status of your pool.

Another neat thing I thought of trying was using something like heartbeat for real-time monitoring and dnsupdate to dynamically update the zone files. This should narrow the propagation latency on your side of the equation down to the barest minimum possible.

Managing Raw Disk/File System Image Files

Xen users frequently deal with raw file system image files. While this isn’t the ideal method for managing virtual machine storage it is the format of choice for redistribution. This article is a simple cheat sheet that will help you deal with sparse and regular image files.

To create a blank image file run:

dd if=/dev/zero bs=1M count=X > image.img

Where X is the size of the file in MB.

To enlarge an image file run:

dd if=/dev/zero bs=1M count=X >> image.img

Where X is the amount of space you want to add in MB.

To create a sparse file run:

dd if=/dev/zero of=image.img seek=X bs=1M count=0

Where X is the size of the image in MB

To enlarge a sparse file run:

dd if=/dev/zero of=image.img seek=X bs=1M count=0

Where seek=8 should be the current size of the sparse file plus the amount of space you wish to grow the image by in MB.

To enlarge an ext2 or ext3 file system to fill an expanded disk image run:

e2fsck -f image.img
e2resizefs image.img
e2fsck -f image.img

To create an ext2 file system on an image file run:

mke2fs image.img

To create an ext3 file system on an image file run:

mke2fs -j image.img

I often find that mke2fs’ defaults for file systems between 2 and 6 gigs are not efficient for full linux installs, make sure you have enough inodes and to really save on space reduce the block size to 1KB. Optionally you can force the defaults for a “small” file system by using the -T flag:

mke2fs -j -T small image.img

This helps clip off a lot of the wasted space created by the (Gentoo in particular but any given flavour’s) large number of small files.

To make a reiserfs file system run:

mkreiserfs image.img

Note that reiser’s ability to incorporate files smaller than the block size into its B-* tree probably makes assigning a smaller (than 4096B) block size more costly than it’s worth.

To expand a reiserfs file system to fill the available space in an image file run

resize_reiserfs image.img

To make an XFS file system run:

mkfs.xfs image.img

XFS caches heavily making it on one hand a decent file-based image performer at the cost of being slightly more fragile than other solutions. Another bonus is it is designed to scale up easily, however I tend to only use XFS as the “mother” FS on battery-backed hardware RAID setups and use ext3 for VM images due to its comparative resilience and the higher level of crash vulnerability virtual machines have over their host counterparts.

To expand an XFS file system to use all available space on a partition image run:

mount -o loop image.img /mnt/image
xfs_growfs /mnt/image

To mount an image file run

mount -o loop image.img /mnt/image

Remember not to perform any operations (like DD) on an image file while it is mounted.

To chroot into an image file with a working linux installation after mounting run:

mount -o bind /dev /mnt/image/dev
mount -t proc none /mnt/image/proc
chroot /mnt/image /bin/bash

Include /bin/bash where the default shell might be less permissive, i.e. standard sh or leave it off if the type of shell is unimportant.

To copy a file system’s contents verbatim to another file system mount both then run:

cp -ax /mnt/image1/* /mnt/image2/

You should do this to a freshly created sparse or regular file to increase the effectiveness of compression when redistributing an image. In a regular file, the empty space that would otherwise be completely zeroed out instead contains remnants of deleted files.

Mass Virtual Hosting Part Six: (Remote) Apache Vhost Configuration and Privileged Command Execution with Database-Backed Upkeep System

What a mouthful.

The more I worked with mod_vdbh the more I realized it’s not for me. Neither is its more advanced fork, mod_vhs which adds some useful extensions. While database-backed configuration is an elegant solution to mass virtual hosting it falls short in that it lacks these key (for my purposes) abilities or requires hackery that violates the Keep It Simple, Stupid rule

  • Apache configuration can not be customized on a per-vhost basis, requiring the use of generated .htaccess files for tricks that might be implemented in a user-friendly way via a front-end, such as custom error pages.
  • Additional Directory directives (i.e. those that don’t match the wildcard directives for whatever reason) need to be put into flat files anyway.
  • Logs for individual sites must be split from a common log by a resource-consuming external solution
  • ScriptAlias directive (mod_vhs) only works for one global directory for all sites or not at all (mod_vdbh)
  • These modules are unmaintained, if something better comes out there is the whole hassle of migration to contend with.
  • A new version of apache may break old modules, but flat files will always be supported.

This is a serious problem if, like me, you are used to setting up vhost accounts in such a fashion:


How can using flat files for mass virtual hosting be as easy to manage as databases one might ask? The answer is simple: generate the flat files from data stored in a database.

My configuration front-end sits on a web server that is independent from the one that serves the ‘public’ virtual hosts. This necessitated the ability to execute commands remotely as root, such as creating the directories for a new host, while at the same time taking into consideration the security implications of average users being able to pass data to the server as root.

My solution came in the form of a two-part system; a shared database that is used to pass sanity-checked data from the configuration interface and an administrative upkeep script run as root by cron every 5 minutes on the virtual hosting server. The script executes appropriate commands with the data set provided then flags its tasks as completed in the database. By storing arrays of raw data and indicating the job type one can avoid altogether the inherent problem of sending straight-up command line commands to the remote server. Careful variable checking in both the configuration interface where the data is added, then in the upkeep script where the tasks are then run can result in rock solid security, despite the fact that we are talking about translating a user’s actions on a web page to root-privileged commands.

In my frustration with the database modules for Apache I realized that the same system could be adapted to write, delete and overwrite individual apache configuration files based on back-end information. It helps to think of the situation like when one uses memcache or APC variable caching with an SQL database; the actual driving force behind the application is the database information but as it is pulled into memory the cache sits between them. By simply dropping the files into a directory and using a wildcard with an Include directive all it takes is reloading apache once updates have been performed for changes to take effect. Through maintaining a master database from which the files are generated one could easily delete the entire directory, run a regeneration script and they would all reappear.

It’s at this point one may find one’s head shaking: one of the prime benefits of database-backed configuration is that the apache server does not have to be restarted when a new vhost is added. Indeed, it is the very reason most people seem to switch to the solution. However I didn’t say restart, I said reload - it seems a lot of people have overlooked apache’s ability to reload configuration files gracefully – that is without being restarted all at once and without dropping open connections. Fortunately, I have not.

One of the major benefits of this approach weighed against database-only configuration is once apache has been reloaded all of the configuration is loaded into RAM; there is no need to worry about thread safety or hundreds of redundant connections and queries to your database server – a problem that becomes worse as your platform scales up without the aide of some sort of abstraction layer like pooled connections (mod_dbm support currently still in development (or not) for mod_vhs) or caching as could be implemented with much hackery and mysql-proxy.

This article will show you – in much simpler and specific detail – how I have implemented what I call an upkeep system that can manage virtual hosts and run privileged commands passed to it (from Joe User in his web browser) safely on a local or remote server with nothing more than PHP and MySQL. It is not at all hard to imagine this system being adapted to manage multiple servers, the workload distributed or logically divided among them given some automated mechanism or instruction from the configuration interface.

Bear in mind that it’s not the norm to use PHP for server-side-only scripts and you may wish to implement the idea in PERL or Python or something more traditional – but the application is sound, I’m good at PHP and I am secure in my masculinity :) PHP should be available on the web servers we intend to manage anyway, but may be more trouble than it’s worth to install it on, say, an NFS-only server if you want to split up file-related commands from configuration tasks.

The other drawback to this approach is the 5-minute delay between runs of the upkeep script. To address this I simply add a notification event to the database which the configuration interface searches for and reports to the user if they have any tasks pending completion.

First, create the shared table on a mutually-accessible SQL server. Remember to use one account for administration and one for the upkeep script, applying permissions frugally.

 `id` bigint(20) NOT NULL auto_increment,
 `uid` int(11) NOT NULL default '0',
 `date` int(11) NOT NULL default '0',
 `completed` int(11) NOT NULL default '0',
 `status` varchar(30) NOT NULL,
 `type` varchar(30) NOT NULL,
 `data` longtext NOT NULL,
 PRIMARY KEY  (`id`)

And we need one for the virtual hosts:

CREATE TABLE IF NOT EXISTS `virtual_hosts` (
 `id` int(11) NOT NULL auto_increment,
 `date` int(11) NOT NULL,
 `user` varchar(255) NOT NULL,
 `yuid` int(11) NOT NULL,
 `appid` int(11) NOT NULL,
 `server` char(255) NOT NULL,
 `environment_variable` char(255) default NULL,
 `subof` int(11) NOT NULL,
 `firewall` longtext NOT NULL,
 `errorpages` longtext NOT NULL,
 PRIMARY KEY  (`id`),
 KEY `yuid` (`yuid`),
 KEY `server` (`server`)

I’m using the uid field in the first table and the yuid field in the second to store the administrative interface’s account ID number or the person who uses the functions. Next we’re going to need some functions for the administration front-end to interface with:

function hostedUPKEEP($type, $data, $uid=0)
 $data = serialize($data);

 $data = mysql_real_escape_string($data);

 mysql_query("insert into `upkeep` (`uid`, `date`, `status`, `type`, `data`) values ('$uid', '".time()."', 'pending', '$type', '$data')");

function hostedUPKEEPMkdir($user, $group, $path)
 $data['user'] = $user;
 $data['group'] = $group;
 $data['path'] = $path;

 hostedUPKEEP('mkdir', $data);

function hostedVHOSTAddHost($user, $yuid, $appid, $server)
 $firewall = $errorpages = serialize(array());
 $date = time();

 mysql_query("insert into `virtual_hosts` (`user`, `date`, `yuid`, `appid`, `server`, `firewall`, `errorpages`) values ('$user', '$date', '$yuid', '$appid', '$server', '$firewall', '$errorpages')");
 $idgrabr = mysql_query("select * from ``virtual_hosts` where `user` = '$user' and `server` = '$server'");
 $idgrabo = mysql_fetch_object($idgrabr);

 hostedUPKEEPMkdir('root', 'root', "/home/$user");
 hostedUPKEEPMkdir($user, 'hosted', "/home/$user/$server");
 hostedUPKEEPMkdir($user, 'hosted', "/home/$user/$server/htdocs");
 hostedUPKEEPMkdir($user, 'hosted', "/home/$user/$server/log");
 hostedUPKEEPMkdir($user, 'hosted', "/home/$user/$server/cgi-bin");
 hostedUPKEEP('vhost', $idgrabo->id);
 hostedUPKEEP('notification', 'jobs pending notification', $yuid);

Since staff may edit user settings I pass the affected user’s front-end UID to a notification event, the user’s front-end will look for events marked pending with their UID and report that they must wait a little while for changes to take effect. Before passing any data to these functions it is important that you make sure it has been as carefully sanitized as possible. The following is a simple upkeep script that you can drop into /sbin/, chown root: and chmod 700 then add to cron at your preferred interval:


$sql_host = '';
$sql_user = '';
$sql_pass = '';
$sql_base = '';

$sql_h = mysql_pconnect($sql_host, $sql_user, $sql_pass);
$sql_d = mysql_select_db($sql_base, $sql_h);

$forbidden_users = array('www',

function checkname($string)
 global $forbidden_users;
 foreach($forbidden_users as $fuse)
 if($fuse == strtolower($string))
 die("Forbidden User");

$result = mysql_query("select * from `upkeep` where `status` = 'pending' order by `date` asc");
while($object = mysql_fetch_object($result))
 if($object->type == 'mkdir')
 $data = unserialize($object->data);
 exec("mkdir -p ".escapeshellcmd($data['path']));
 exec("chown ".escapeshellcmd(ereg_replace("[^A-Za-z0-9]", '', $data['user'])).":".escapeshellcmd(ereg_replace("[^A-Za-z0-9]", '', $data['group']))." ".escapeshellcmd($data['path']));

 if($object->type == 'vhost')
 $id = $object->data;
 $vhost_result = mysql_query("select * from `virtual_hosts` where `id` = '$id'");
 $vhost_object = mysql_fetch_object($vhost_result);

 $domain = $vhost_object->server;
 $user = ereg_replace("[^A-Za-z0-9]", '', $vhost_object->user);


 $errarray = unserialize($vhost_object->errarray);
 $errorpages = '';
 foreach($errarray as $code => $loc)
 $errorpages .= "\n\tErrorDocument $code $loc";
 $errorpages = '';

 $fwarray = unserialize($vhost_object->fwrarray);
 $firewall = '';
 foreach($fwarray as $address)
 $firewall .= "\n\tDeny from $address";
 $firewall = '';

 $domainesc = str_replace('.', '\.', $domain);

 $file = "<VirtualHost *:80>
\tServerName $domain
\tServerAlias www.$domain
\tDocumentRoot /home/$user/$domain/htdocs
\tScriptAlias /cgi-bin/ /home/$user/$domain/cgi-bin/
\tErrorLog /home/$user/$domain/log/error_log
\tTransferLog /home/$user/$domain/log/access_log{$errorpages}
\t<IfModule mod_rewrite.c>
\t\tRewriteEngine on
\t\tRewriteCond %{REQUEST_METHOD} !^(GET|POST|HEAD)$
\t\tRewriteRule .* - [F]
\t\tRewriteCond %{HTTP_HOST} ^www\.$domainesc$ [NC]
\t\tRewriteRule ^(.*)\$ http://$domainesc/\$1 [R=307,L]
\t<IfModule mod_access.c>
\t\tOrder Allow,Deny
\t\tAllow from all{$firewall}

 $fh = fopen("/etc/apache2/hosted.d/{$user}_{$domain}.conf", 'w');
 fwrite($fh, $file);

 exec("/etc/init.d/apache2 reload");        // Change the path to your apache2ctl if the init script does not support reload.

 mysql_query("update `upkeep` set `status` = 'completed', `completed` = '".time()."' where `id` = '{$object->id}'");


Now create the directory /etc/apache2/hosted.d (or whatever you prefer) and add this directive to the end of your httpd.conf:

Include /etc/apache2/hosted.d/*.conf
Return top
Online Marketing Toplist
Technology Blogs - Blog Rankings

Internet Blogs - BlogCatalog Blog Directory

Bad Karma Networks

Please Donate!

Made in Canada  •  There's a fox in the Gibson!  •  2010-12