pitter patter on the keyboard

Round-Robin DNS gets trash-talked a lot because although it is a cheap and easy way to distribute loads it is counter-redundant: the more A records (servers) there are behind a domain the more points of failure there are and the lower your mean time to failure is going to be. The good news is that if one in five web servers/reverse proxies are down then only about one fifth of your audience is unable to connect at any given time.

The answer to this problem is host monitoring. If we can update our DNS records to remove the IPs of downed servers then add them back when the hosts recover no direct intervention on our part is required. Unfortunately, DNS is a heavily cached system so we will have to work with reasonably short timeouts. DNS Made Easy recommends a TTL of no less than 180 seconds as some ISPs are configured to ignore the TTLs of records which they deem are too short and default to a much higher value. The drawback to short TTLs is that you will end up receiving more DNS queries, which is a problem if you use a commercial billed-by-million-queries DNS provider like Amazon’s Route 53 or EasyDNS’s enterprise service.

If your objective is to have web server failover that happens instantly this is simply not the solution for you – you need a load balancer and/or anycast address space. Amazon’s Route53 and DNS Made Easy can be configured to check as often as every minute and it doesn’t make a lot of sense to run a ping/tcp test more often than that. At worst this means that the failover system doesn’t even know there is a problem for up to 60 seconds. Once the failover system updates the records there may be a short delay while the slave name servers synchronize. Then we have to wait for the record to expire at any-given-user’s ISP’s recursive name servers, which could take up to the TTL of your record or longer if their ISP is manipulative. Then you may have to wait for the record to expire in the caching DNS daemon on their home or office router. Then you may have to wait for the record to expire in their OS or browser’s DNS cache. This could take up to 15 minutes even if you use a very low TTL like 180.

So the question is: you already have DNS infrastructure. Why pay these large DNS outfits for host monitoring and DNS failover when it’s not really that great anyway and you can do it just as well as they can?

Just because BIND doesn’t have built-in support? Pshaw!

You could just as easily do the host monitoring with nagios/icinga or use the mysql-bind backend or even some other database-backed name daemon but in this article I’ll show you how to drop in a simple shell script that will work with your existing BIND installation because it demonstrates how mind-numbingly simple this is and why it shouldn’t be charged for as a premium service.

Observe a typical zone file with round-robin:

$TTL 6400       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                201305140       ; Serial
                                28800           ; Refresh
                                7200            ; Retry
                                60480           ; Expire
                                600 )           ; TTL Minimum
@               IN      A       10.0.0.10
@               IN      A       10.0.0.11
@               IN      A       10.0.0.12
@               IN      A       10.0.0.13
@               IN      A       10.0.0.14
*               IN      A       10.0.0.10
*               IN      A       10.0.0.11
*               IN      A       10.0.0.12
*               IN      A       10.0.0.13
*               IN      A       10.0.0.14
ns1             IN      A       10.0.1.10
ns2             IN      A       10.0.1.11
@               IN      NS      ns1.somedomain.com.
@               IN      NS      ns2.somedomain.com.
www             IN      CNAME   somedomain.com.

Our SOA contains the serial which will have to be updated by the script if our changes are to propagate properly. In the zone file on the master server(s) replace the SOA and block of round-robin A records with $INCLUDE statements like this:

$INCLUDE "/var/bind/soa.include"
$INCLUDE "/var/bind/ips.include"
ns1             IN      A       10.0.1.10
ns2             IN      A       10.0.1.11
@               IN      NS      ns1.somedomain.com.
@               IN      NS      ns2.somedomain.com.
www             IN      CNAME   somedomain.com.

Do this for every zone file which is to use this pool of A records. Now we have a centralized place to put the IPs and serial number that come from the shell script.

Create the script on the master name server and chmod +x it, don’t forget to update the paths to reflect your DNS situation. Also note that I’m adding a wildcard subdomain to the pool:

#!/bin/bash
HOSTS="10.0.0.10 10.0.0.11 10.0.0.12 10.0.0.13 10.0.0.14"
COUNT=4
echo "; Generated by monitor.sh $(date)" > /chroot/dns/var/bind/ips.include
for myHost in $HOSTS
do
  count=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')
  if [ $count -eq 0 ]; then
    # 100% failed 
    echo "$(date) $myHost is down" >> /var/log/monitor.log
  else
    echo "@               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include
    echo "*               IN      A       $myHost" >> /chroot/dns/var/bind/ips.include
  fi

done

echo "; Generated by monitor.sh $(date)
\$TTL 300       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                $(date +%s)      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum" > /chroot/dns/var/bind/soa.include

rndc reload

This script will ping each host in the HOSTS array four times. If at least one ping is received the host is written to a new version of ips.include (note the single angle bracket when inserting the date). If all four pings fail a message will be recorded in /var/log/monitor.log. You may want to adjust the number of pings and failure tolerance, or replace the logging line with an e-mail notification instead. Once the ping tests are done a new soa.include is written with an epoch serial number and the zones are reloaded.

At the end of execution you should see something like this in ips.include:

; Generated by monitor.sh Tue May 14 16:15:26 EDT 2013
@               IN      A       10.0.0.10
*               IN      A       10.0.0.10
@               IN      A       10.0.0.11
*               IN      A       10.0.0.11
@               IN      A       10.0.0.12
*               IN      A       10.0.0.12
@               IN      A       10.0.0.13
*               IN      A       10.0.0.13
@               IN      A       10.0.0.14
*               IN      A       10.0.0.14

And in soa.include:

; Generated by monitor.sh Tue May 14 16:15:26 EDT 2013
$TTL 300       ; max TTL
@       IN      SOA     ns1.somedomain.com. admin.somedomain.com. (
                                1368562526      ; Serial
                                300             ; Refresh
                                60              ; Retry
                                86400           ; Expire
                                300 )           ; TTL Minimum

Note that you may need to chown named: the .include files after they are created the first time, depending on your environment.

I switched from using the widely popular YYYYMMDDID format to epoch since the 5 minute interval requires hours, minutes and seconds to be effective and YYYMMDDHHMMSS is too large a value for BIND. This resulted in a lower serial value – you may have to go around to your slaves and manually delete then reload their zone files.

This approach ends up generating a lot of NOTIFY traffic since every 5 minutes (or whatever interval you cron the shell script at) a new serial is loaded and all of your slaves have to be contacted. A more graceful improvement would be to save the state that each host is in inside of a temporary file and only update the serial when there has actually been a change in the status of your pool.

Another neat thing I thought of trying was using something like heartbeat for real-time monitoring and dnsupdate to dynamically update the zone files. This should narrow the propagation latency on your side of the equation down to the barest minimum possible.

There appears to be a benign segfault from opendkim when Zimbra is shut down:

opendkim[4057]: segfault at 1f0 ip 0000003044209220 sp 00007fffa235bd48 error 4 in libpthread-2.12.so[3044200000+17000]
opendkim[10772]: segfault at 1f0 ip 0000003044209220 sp 00007fffbd4e47a8 error 4 in libpthread-2.12.so[3044200000+17000]

The only solid reference to this I could find was at http://www.zimbra.com/forums/administrators/58839-mta-logger-logswatch-wont-start-also-opendkim-issue.html. These users appear to be using CentOS 6.3 as well which may indicate the problem is with libpthread and not in fact opendkim.

Apparently some among legion want to DRDoS the root name servers with the same sort of UDP-spoofing DNS amplification attack I have had personal experience defending against.

I’m not going to get into how horribly misguided this is, how negatively this is going to affect the cause’s image or some long-winded speech on ethics. I only plead with interested parties not to get involved.

Attacking “the Internet” itself for any reason makes you an enemy of freedom, an enemy of progress and above all else a disgrace to hacking.

The hottest places in hell are reserved for traitors and heretics.

“The greatest enemy of freedom is a happy slave.”

To protest SOPA, Wallstreet, our irresponsible leaders and the beloved
bankers who are starving the world for their own selfish needs out of
sheer sadistic fun, On March 31, anonymous will shut the Internet down.

———————————————————————–

In order to shut the Internet down, one thing is to be done. Down the
13 root DNS servers of the Internet. Those servers are as follow:

A 198.41.0.4
B 192.228.79.201
C 192.33.4.12
D 128.8.10.90
E 192.203.230.10
F 192.5.5.241
G 192.112.36.4
H 128.63.2.53
I 192.36.148.17
J 192.58.128.30
K 193.0.14.129
L 199.7.83.42
M 202.12.27.33

By cutting these off the Internet, nobody will be able to perform a
domain name look-up, thus, disabling the HTTP Internet, which is,
after all, the most widely used function of the Web. Anybody entering
“http://www.google.com” or ANY other url, will get an error page,
thus, they will think the Internet is down, which is, close enough.
Remember, this is a protest, we are not trying to ‘kill’ the Internet,
we are only temporarily shutting it down where it hurts the most.

While some ISPs uses DNS caching, most are configured to use a low
expire time for the cache, thus not being a valid failover solution
in the case the root servers are down. It is mostly used for speed,
not redundancy.

We have compiled a Reflective DNS Amplification DDoS tool to be used for
this attack. It is based on AntiSec’s DHN, contains a few bugfix, a
different dns list/target support and is a bit stripped down for speed.

The principle is simple; a flaw that uses forged UDP packets is to be
used to trigger a rush of DNS queries all redirected and reflected to
those 13 IPs. The flaw is as follow; since the UDP protocol allows it,
we can change the source IP of the sender to our target, thus spoofing
the source of the DNS query.

The DNS server will then respond to that query by sending the answer to
the spoofed IP. Since the answer is always bigger than the query, the
DNS answers will then flood the target ip. It is called an amplified
because we can use small packets to generate large traffic. It is called
reflective because we will not send the queries to the root name servers,
instead, we will use a list of known vulnerable DNS servers which will
attack the root servers for us.

DDoS request —> [Vulnerable DNS Server ] Normal client requests
\
| ( Spoofed UDP requests
| will redirect the answers
| to the root name server )
|
[ 13 root servers ] * BAM

Since the attack will be using static IP addresses, it will not rely
on name server resolution, thus enabling us to keep the attack up even
while the Internet is down. The very fact that nobody will be able to
make new requests to use the Internet will slow down those who will try
to stop the attack. It may only lasts one hour, maybe more, maybe even
a few days. No matter what, it will be global. It will be known.

———————————————————————–

download link in #opGlobalBlackout
The tool is named “ramp” and stands for Reflective Amplification. It is
located in the \ramp\ folder.

———-> Windows users

In order to run “ramp”, you will need to download and install these two
applications;

WINPCAP DRIVER – www.winpcap.org/install/default.htm
TOR – www.torproject.org/dist/vidalia-bundles/

The Winpcap driver is a standard library and the TOR client is used as
a proxy client for using the TOR network.

It is also recommended to use a VPN, feel free to choose your own flavor of this.

To launch the tool, just execute “\ramp\launch.bat” and wait. The attack
will start by itself.

———-> Linux users

The “ramp” linux client is located under the \ramp\linux\ folder and
needs a working installation of python and scapy.

Read more: www.disclose.tv/forum/on-march-31-anonymous-will-shut-the-internet-down-t67878.html#ixzz1modrC1Jn

“He who sacrifices freedom for security deserves neither.”
Benjamin Franklin

We know you won’t listen. We know you won’t change. We know it’s because
you don’t want to. We know it’s because you like it how it is. You bullied
us into your delusion. We have seen you brutalize harmless old womans who were
protesting for peace. We do not forget because we know you will only use that
to start again. We know your true face. We know you will never stop. Neither
are we. We know.

We are Anonymous.
We are Legion.
We do not Forgive.
We do not Forget.
You know who you are, Expect us.

UPDATE And then there are those who do God’s work: Anonymous hacks Chinese websites