I’ve made a minor modification to include support for HTTP Basic Authentication.

class HTTPRequest
{
    var $_fp;          // HTTP socket
    var $_url;         // full URL
    var $_host;        // HTTP host
    var $_protocol;    // protocol (HTTP/HTTPS)
    var $_uri;         // request URI
    var $_port;        // port
    var $_user;        // HTTP Basic Auth User
    var $_pass;        // HTTP Basic Auth Password
   
    // scan url
    function _scan_url()
    {
        $req = $this->_url;
       
        $pos = strpos($req, '://');
        $this->_protocol = strtolower(substr($req, 0, $pos));
       
        $req = substr($req, $pos+3);
        $pos = strpos($req, '/');
        if($pos === false)
            $pos = strlen($req);
        $host = substr($req, 0, $pos);
       
        if(strpos($host, ':') !== false)
        {
            list($this->_host, $this->_port) = explode(':', $host);
        }
        else
        {
            $this->_host = $host;
            $this->_port = ($this->_protocol == 'https') ? 443 : 80;
        }
       
        $this->_uri = substr($req, $pos);
        if($this->_uri == '')
            $this->_uri = '/';
    }
   
    // constructor
    function HTTPRequest($url, $user='', $pass='')
    {
        $this->_url = $url;
        $this->_scan_url();
        $this->_user = $user;
        $this->_pass = $pass;
    }
   
    // download URL to string
    function DownloadToString()
    {
        $crlf = "\r\n";
       
        // generate request
        $req = 'GET ' . $this->_uri . ' HTTP/1.0' . $crlf
            .    'Host: ' . $this->_host . $crlf;
            if(!empty($this->_user))
                 $req .= "Authorization: Basic " . base64_encode($this->_user . ':' . $this->_pass) . $crlf;
       $req .= $crlf;
       
        // fetch
        $this->_fp = fsockopen(($this->_protocol == 'https' ? 'ssl://' : '') . $this->_host, $this->_port);
        fwrite($this->_fp, $req);
        while(is_resource($this->_fp) && $this->_fp && !feof($this->_fp))
            $response .= fread($this->_fp, 1024);
        fclose($this->_fp);
       
        // split header and body
        $pos = strpos($response, $crlf . $crlf);
        if($pos === false)
            return($response);
        $header = substr($response, 0, $pos);
        $body = substr($response, $pos + 2 * strlen($crlf));
       
        // parse headers
        $headers = array();
        $lines = explode($crlf, $header);
        foreach($lines as $line)
            if(($pos = strpos($line, ':')) !== false)
                $headers[strtolower(trim(substr($line, 0, $pos)))] = trim(substr($line, $pos+1));
       
        // redirection?
        if(isset($headers['location']))
        {
            $http = new HTTPRequest($headers['location']);
            return($http->DownloadToString($http));
        }
        else
        {
            return($body);
        }
    }
}

Usage:

$r = new HTTPRequest($url, [username], [password]);
$response = $r->DownloadToString();

The username and password variables are optional.

The answer to this problem is host monitoring. If we can update our DNS records to remove the IPs of downed servers then add them back when the hosts recover no direct intervention on our part is required. Unfortunately, DNS is a heavily cached system so we will have to work with reasonably short timeouts. DNS Made Easy recommends a TTL of no less than 180 seconds as some ISPs are configured to ignore the TTLs of records which they deem are too short and default to a much higher value. The drawback to short TTLs is that you will end up receiving more DNS queries, which is a problem if you use a commercial billed-by-million-queries DNS provider like Amazon’s Route 53 or EasyDNS’s enterprise service.

If your objective is to have web server failover that happens instantly this is simply not the solution for you – you need a load balancer and/or anycast address space. Amazon’s Route53 and DNS Made Easy can be configured to check as often as every minute and it doesn’t make a lot of sense to run a ping/tcp test more often than that. At worst this means that the failover system doesn’t even know there is a problem for up to 60 seconds. Once the failover system updates the records there may be a short delay while the slave name servers synchronize. Then we have to wait for the record to expire at any-given-user’s ISP’s recursive name servers, which could take up to the TTL of your record or longer if their ISP is manipulative. Then you may have to wait for the record to expire in the caching DNS daemon on their home or office router. Then you may have to wait for the record to expire in their OS or browser’s DNS cache. This could take up to 15 minutes even if you use a very low TTL like 180.

So the question is: you already have DNS infrastructure. Why pay these large DNS outfits for host monitoring and DNS failover when it’s not really that great anyway and you can do it just as well as they can?

Just because BIND doesn’t have built-in support? Pshaw!

You could just as easily do the host monitoring with nagios/icinga or use the mysql-bind backend or even some other database-backed name daemon but in this article I’ll show you how to drop in a simple shell script that will work with your existing BIND installation because it demonstrates how mind-numbingly simple this is and why it shouldn’t be charged for as a premium service.

Observe a typical zone file with round-robin:

$TTL 6400       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                201305140       ; Serial
                                28800           ; Refresh
                                7200            ; Retry
                                60480           ; Expire
                                600 )           ; TTL Minimum
@               IN      A       10.0.0.10
@               IN      A       10.0.0.11
@               IN      A       10.0.0.12
@               IN      A       10.0.0.13
@               IN      A       10.0.0.14
*               IN      A       10.0.0.10
*               IN      A       10.0.0.11
*               IN      A       10.0.0.12
*               IN      A       10.0.0.13
*               IN      A       10.0.0.14
ns1             IN      A       10.0.1.10
ns2             IN      A       10.0.1.11
@               IN      NS      ns1.somedomain.com.
@               IN      NS      ns2.somedomain.com.
www             IN      CNAME   somedomain.com.

Our SOA contains the serial which will have to be updated by the script if our changes are to propagate properly. In the zone file on the master server(s) replace the SOA and block of round-robin A records with $INCLUDE statements like this:

$INCLUDE "/var/bind/soa.include"
$INCLUDE "/var/bind/ips.include"
ns1             IN      A       10.0.1.10
ns2             IN      A       10.0.1.11
@               IN      NS      ns1.somedomain.com.
@               IN      NS      ns2.somedomain.com.
www             IN      CNAME   somedomain.com.

Do this for every zone file which is to use this pool of A records. Now we have a centralized place to put the IPs and serial number that come from the shell script.

Create the script on the master name server and chmod +x it, don’t forget to update the paths to reflect your DNS situation. Also note that I’m adding a wildcard subdomain to the pool:

#!/bin/bash
HOSTS="10.0.0.10 10.0.0.11 10.0.0.12 10.0.0.13 10.0.0.14"
COUNT=4
echo "; Generated by monitor.sh $(date)" > /chroot/dns/var/bind/ips.include
for myHost in $HOSTS
do
  count=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')
  if [ $count -eq 0 ]; then
    # 100% failed 
    echo "$(date) $myHost is down" >> /var/log/monitor.log
  else
    echo "@               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include
    echo "*               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include
  fi

done

echo "; Generated by monitor.sh $(date)
\$TTL 300       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                $(date +%s)      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum" > /chroot/dns/var/bind/soa.include

rndc reload

This script will ping each host in the HOSTS array four times. If at least one ping is received the host is written to a new version of ips.include (note the single angle bracket when inserting the date). If all four pings fail a message will be recorded in /var/log/monitor.log. You may want to adjust the number of pings and failure tolerance, or replace the logging line with an e-mail notification instead. Once the ping tests are done a new soa.include is written with an epoch serial number and the zones are reloaded.

At the end of execution you should see something like this in ips.include:

; Generated by monitor.sh Tue May 14 16:15:26 EDT 2013
@               IN      A       10.0.0.10
*               IN      A       10.0.0.10
@               IN      A       10.0.0.11
*               IN      A       10.0.0.11
@               IN      A       10.0.0.12
*               IN      A       10.0.0.12
@               IN      A       10.0.0.13
*               IN      A       10.0.0.13
@               IN      A       10.0.0.14
*               IN      A       10.0.0.14

And in soa.include:

; Generated by monitor.sh Tue May 14 16:15:26 EDT 2013
$TTL 300       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                1368562526      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum

Note that you may need to chown named: the .include files after they are created the first time, depending on your environment.

I switched from using the widely popular YYYYMMDDID format to epoch since the 5 minute interval requires hours, minutes and seconds to be effective and YYYMMDDHHMMSS is too large a value for BIND. This resulted in a lower serial value – you may have to go around to your slaves and manually delete then reload their zone files.

This approach ends up generating a lot of NOTIFY traffic since every 5 minutes (or whatever interval you cron the shell script at) a new serial is loaded and all of your slaves have to be contacted. A more graceful improvement would be to save the state that each host is in inside of a temporary file and only update the serial when there has actually been a change in the status of your pool.

Another neat thing I thought of trying was using something like heartbeat for real-time monitoring and dnsupdate to dynamically update the zone files. This should narrow the propagation latency on your side of the equation down to the barest minimum possible.

First, copy your Dom0 .config and kernel to /boot/. /etc/grub.d/20_linux_xen picks up on the CONFIG_XEN_PRIVILEGED_GUEST=y variable.

# cp /usr/src/linux/arch/x86_64/boot/bzImage /boot/kernel-dom0
# cp /usr/src/linux/.config /boot/config-dom0

Next, remove the executable flag from /etc/grub.d/10_linux or you will end up with three or four entries that boot straight to the Dom0 kernel without the hypervisor.

# chmod -x /etc/grub.d/10_linux

To add kernel boot parameters to your grub.cfg (as generated with grub2-mkconfig and 20_linux_xen) add these lines to your /etc/default/grub file:

GRUB_CMDLINE_LINUX_XEN_REPLACE="max_loop=128"
GRUB_CMDLINE_XEN="dom0_mem=512"

The former will place your boot options on the Dom0 kernel’s module line. The latter will place boot options on the xen.gz hypervisor’s kernel line.

Now, generate your new configuration:

# grub2-mkconfig -o /boot/grub2/grub.cfg

Once the config file has been written, determine the position of the Xen hypervisor entry and update GRUB_DEFAULT in /etc/default/grub.

Install GRUB2 to your MBR(s) as necessary:

# grub2-install /dev/sda 

Open serviceengine.xml and replace all four instances of “8080″ with the non-secure port you specified in ofbiz-containers.xml.

After taking steps to shut down the site you should probably use the find command to check to see if there are any unusual files which have been uploaded recently. If you scramble to close the hole and do updates before this step you will likely end up drowning any suspects in the results.

If you don’t remove, for example, a phishing page before plugging the hole you:

• May never find out it’s there
• Are contributing to phishing
• One day your ISP will probably forward you a sternly worded letter from the victimized institution and threaten to drop your service if the page is not removed in 24 hours

Use your imagination if it’s something worse, like a rootkit or webshell.

It is necessary to determine the earliest possible time the attack could have taken place. It won’t kill you to add a day or two for safety.

# find /var/www/localhost/htdocs/ -type f -ctime -X

Where X is the number of days to look back.

I might just be an idiot, and things may have changed by now but I wasted a solid half hour of my life not finding anywhere to RMA the drive on their Canadian consumer products site.

For your benefit, so you do not end up on hold with consumer support for 20 minutes as I did, the location of the RMA portal:

https://www.lgrepairportal.com

This strategy has numerous benefits, not the least of which is the ability to upgrade libraries and binaries in-place. The software which relies on these libraries continues to run using the version it was started up with, preventing crashes due to version mismatching and giving you time to update the binaries themselves before restarting for minimal downtime.

Unfortunately, this makes things slightly confusing. You may have deleted a 400MB log file expecting to have immediately freed 400MB but df is still reporting that your file system is full. If you know what process “owns” that file it’s usually a simple matter of restarting the corresponding software. You won’t always know this, however, and that’s where some clever use of the lsof command comes in handy.

lsof spits out the size and owner of all open files. If you already know the file you’re looking for it’s as simple as grepping the output:

# lsof | grep "/var/log/zimbra.log"
COMMAND     PID    USER   FD      TYPE             DEVICE    SIZE/OFF       NODE NAME
rsyslogd   1285    root    6w      REG             202,17    65125680    2031675 /var/log/zimbra.log

If we pipe lsof‘s output through awk and sort we can get some useful, human-readable information. This command will give us the 10 largest currently open files, the size of the files in megabytes and the name of the process(es) using them:

# lsof / | awk '{if($7 > 1048576) print $7/1048576 "MB" " " $9 " " $1}' | sort -n -u | tail

For example:

498.804MB /var/log/zimbra.log zimbra

To view more or less than 10 results add -n X where X is the number of lines you would like to see to the tail command.

You can force a manual update by running:

# sudo -u zimbra /opt/zimbra/libexec/zmstatuslog

To re-load the cron settings perform the following:

# cd /opt/zimbra/zimbramon/crontabs
# cat * > crontab.zimbra
# sudo -u zimbra crontab crontab.zimbra