pitter patter on the keyboard

In the last article I told you any username-and-password authentication system that is exposed to the Internet is inherently vulnerable to dictionary and brute force attack. If you must use such an authentication scheme you can defend it by implementing rate control. If you block an attacker from trying to log in for one hour after three failed attempts it would take them a year to try just under 3,000 combinations. In cryptanalytic terms that is abysmal and the odds are on your side that the attacker will have moved on by then.

While porting your ban system to fail2ban might be a great idea it’s probably overkill for situations where you have hundreds of legitimate users who might often forget their credentials; IP-bans are not generally considered good customer service. Many sites, including Google, will present the user with a CAPTCHA after three failed attempts and that’s great but those are getting easier to crack every day.

For the sake of the pseudocode in this article we’re going to assume you want to block the  potential attacker and politely tell them they have either a) failed to log in too many times, please come back in an hour or b) posted too recently, please try again. Since we want to be able to rate control two (and perhaps more in the future) different things and we don’t want to make a mess of our database let’s make one table called ‘greylist’ and use the type column to differentiate:

CREATE TABLE `demo_cat`.`greylist` (
`type` VARCHAR( 30 ) NOT NULL ,
`date` INT NOT NULL ,
`ip` VARCHAR( 15 ) NOT NULL ,
PRIMARY KEY ( `ip` ) ,
INDEX ( `date` )
);

Now in your login script for argument’s sake we’ll say $outcome is a boolean representation of if the authentication was successful or not and $delay is the period of time we want to measure for in seconds. We’ll start off by clearing everything that’s out of date, a relatively inexpensive query to run every time there’s a failure. After the table has been updated we’ll add an entry for the current failure and take a tally of all the entries for the user’s IP. If the tally exceeds the retry $threshold we’ll tell them to buzz off for an hour, change their password, show a captcha or whatever suits your site best.

<?php

if(!$outcome)
{
   $ip = mysql_real_escape_string($_SERVER['REMOTE_ADDR']);
   mysql_query("delete from `greylist` where `type` = 'login' and `date` < '".time()-$offset."'");
   mysql_query("insert into `greylist` (`type`, `date`, `ip`) values ('login', '".time()."', '$ip'')");
   $result = mysql_query("select `ip` from `greylist` where `type` = 'login' and `ip` = '$ip'");
   if(mysql_num_rows($result > $threshold))
   {
      // Too many tries, what now?
   }
   else
   {
      // Please try again
   }
}

?>

It is as simple as that. Now let’s use this to flood-protect our comments box:

<?php

if($_POST)
{
   $ip = mysql_real_escape_string($_SERVER['REMOTE_ADDR']);
   mysql_query("delete from `greylist` where `type` = 'comment' and `date` < '".time()-$offset."'");
   mysql_query("insert into `greylist` (`type`, `date`, `ip`) values ('comment', '".time()."', '$ip'')");
   $result = mysql_query("select `ip` from `greylist` where `type` = 'comment' and `ip` = '$ip'");
   if(mysql_num_rows($result > $threshold))
   {
      // You posted too recently, please wait x seconds before trying again.
   }
   else
   {
      // Continue...
   }
}

?>

A more sophisticated implementation of this concept is in use at Ychan, where users’ posting patterns are analyzed to determine if they are computers, legitimate humans or computers trying to look like humans.

A SYN flood is a type of resource-starvation denial of service (DoS) attack in which the attacker creates enough “half open” connections to render a server inaccessible to the legitimate public. Because the attack takes advantage of weaknesses in the default configuration of most TCP implementations rather than raw strength, one attacker with a relatively low bandwidth connection can quickly take down a much better equipped server. The attacker only needs to send one SYN packet to establish a half-open connection on the defending server, which will in turn attempt to reset the connection a set number of times. Since the handshake has been initialized and the connection is being logged the deed is done; the attacker doesn’t need to respond to the RST packet so the source address can be spoofed, making the task of tracing the attacker virtually impossible and the attack itself very difficult to block.

When you first come under attack it may not seem obvious  what is happening. The targeted host(s) will stop or sporadically respond to your users and you may not even be able to shell into the machine. If you can gain access to the machine the telltale signs are:

• Services are running but using no CPU or I/O
• Traffic graphs flatline but the host(s) remain pingable
• Services appear to be listening on the right ports, the firewall is clear, but you can’t connect to them even locally
• Multiple TCP-based services are affected
• The output of netstat -n indicates an unusually high number of connections in the SYN_RECV state

All or most TCP services will seem to be affected because they all share the same connection queue. Unless your server is very overloaded, even on high traffic sites you should never see more than about 5 or 6 connections in the SYN_RECV state sustained over any period of time – particularly if you reduce the number of retries your kernel attempts as outlined below.

Fortunately there are two ways to address this problem: stack tweaking and syncookies (for BSD/linux, other implementations exist). Since the SYN flood relies on a lengthy timeout and limited number of available connections the obvious first step is to increase these limits. Having a lot of extra RAM comes in handy here since it takes RAM to track the connections. In fact, in preventing most resource starvation tactics throwing more RAM (if available) at the problem is always a good blind first step – though never the solution. We can manipulate these values through the /proc interface:

# echo 3096 > /proc/sys/net/ipv4/tcp_max_syn_backlog

tcp_max_syn_backlog limits the number of half-open connections the kernel will track. This is the limit that gets exhausted when regular users are no longer able to connect.

# echo 2 > /proc/sys/net/ipv4/tcp_syn_retries
# echo 1 > /proc/sys/net/ipv4/tcp_synack_retries

tcp_syn_retries is the number of times the kernel will wait appx. 40 seconds and send out another SYN packet when trying to establish an outbound connection. This won’t do you any good for SYN flood protection but it can mitigate the effects of some amplification/redirection techniques that use your hosts as soldiers. tcp_synack_retries limits the number of times the kernel will retry responding to a half-opened connection. The default is 5 and that means an attacker’s connection could last in the queue for up to 180 seconds. If the attacker can open an easy 300 new half-open connections in that period it becomes clear how quickly your connection queue can be overrun. Setting this value too low can cause problems for people on weak links like dialup.

Obviously this isn’t going to be enough; finite resources will always be finite resources. Syncookies are a genious little invention that in a nutshell validate that traffic coming to the host is sent from a real computer rather than a packet generator by sending a simple type of cryptographic challenge in the headers of outgoing packets that is “responded” to in the headers of incoming packets by the mechanics of tcp itself. Because spoofed traffic doesn’t have a legitimate sending host behind it to  “hear” the challenge it (probably) does not contain  a valid response and the connection is swiftly discarded.

Syncookies are not enabled by default and enabling them will override the value in tcp_max_syn_backlog, but it won’t hurt you to increase it anyway:

# echo 1 > /proc/sys/net/ipv4/tcp_syncookies

Most distributions include a “local” script that runs at the end of init, yours may have one specifically for the firewall. On Gentoo I put these rules in /etc/conf.d/local.start and on ClearOS /etc/rc.d/rc.firewall.local. Note that since NAT doesn’t handle the connections themselves and only passes them through, simply turning on syncookies in your firewall will not protect everything behind it.

If you want to centralize or introduce a degree of separation between your SYN flood protection and regular servers you can use proxies, Squid and Apache both work in reverse and SOCKS proxies may work as well (don’t quote me on that).

I was caught with my pants down once; I hadn’t enabled syncookies on just one VM and it got SYN flooded (murphy’s law of course) and that’s a mistake you only make once. It underscored for me the importance of following some sort of thorough lockdown procedure before you deploy a new machine. That will be the subject of an upcoming article where I will attempt to compile a definitive checklist.

If you are running a virtualized environment or have the space for enough servers the easiest way to mitigate the harm a resource starvation attack can do to the continuity of your operations is to compartmentalize and space services out as much as possible. If you have a web server and a dns server 1-1 NATted to a public address and an attacker hits you on port 80 only the web server is going to lock up, your DNS and therefore mail and so on should continue to operate, until of course they figure it out. If you have to run DNS and mail and web and radius try to run them on different servers rather than one despite the overhead; when one plans a public-facing network one should think less in terms of bare economics and more in terms of capacity to absorb attack.