=^.^=

Stifling Brute Force Attacks with fail2ban

• karma

fail2ban is a package that monitors your log files for failed login attempts and executes a configured action, usually temporarily blocking the attacking IP with iptables for a set duration. Any exposed service that uses a username/password authentication scheme is vulnerable to dictionary and brute force attack, your first defense if you must expose such a service is to make such attacks as costly as possible and that's where fail2ban comes in. By temporarily blocking an address for even 10 minutes after every 3 failed login attempts you make the process several orders of magnitude slower. Since fail2ban reads plain log files and can be configured for any action one clever deployment could see a log server collecting logs from all the hosts on a network and sharing the relevant logs with the firewall via NFS where fail2ban can quickly cut access to the entire network from the attacker with ease. For the purposes of this article we will only focus on locking down SSH on a local host.

fail2ban is probably available in your distribution's package management system. Gentoo users type:

# emerge fail2ban

If the package is not available for your flavour you can compile it from source, available at http://sourceforge.net/project/showfiles.php?group_id=121032&package_id=132537:

# tar xjf fail2ban-*
# cd fail2ban-*/
# ./setup.py install
# cp /usr/local/src/fail2ban-*/files/{your distro or close match here}-init /etc/init.d/fail2ban

Then add the script to the appropriate runlevels. Gentoo users type:

# rc-update add fail2ban default

Despite the name, fail2ban jails are not like chroot or ssh jails. A 'jail' is the combination of a filter and an action. The filters are regular expressions used to search the log files for interesting lines such as login failures. These filters are located in /etc/fail2ban/filter.d/ and the action scripts are located in /etc/fail2ban/filter.d/. By adding to and tying these filters and actions together in /etc/fail2ban/jail.conf you can re-purpose fail2ban to do just about any log event-triggered action imaginable; once you've given it a good mucking about locking down SSH may seem trite.

Open /etc/fail2ban/jail.conf and find [ssh-iptables], change the configuration block to look like this:

[ssh-iptables]
enabled = true
filter = sshd
action = iptables[name=SSH, port=ssh, protocol=tcp]
logpath = /var/log/sshd/current
maxretry = 5
# findtime = 600
# bantime = 600

You may need to edit logpath to reflect your system's settings. Set maxretry to however many failed login attempts you wish to allow over a given amount of time (findtime) until the source address is blocked for a given amount of time (bantime). The default findtime and bantime  is 600 seconds (10 minutes) and only needs to be set if you would like to choose different durations. If you would like to be notified by e-mail when someone has been blocked (probably not a good idea on a busy public server) add this line to the jail:

mail-whois[name=SSH, [email protected]]

Now make sure your SSH daemon is logging in verbose mode, add this line if you must to /etc/ssh/sshd_config:

LogLevel VERBOSE

If your sshd log entries contain the string pam_unix(sshd:auth) (Gentoo users here) you may need to modify the line starting with __daemon_re in /etc/fail2ban/filter.d/common.conf to look like:

__daemon_re = [\[\(]?%(_daemon)s(?:\([^\)]+\))?[\]\)]?:?

and configuration is over. Now start the server:

/etc/init.d/fail2ban start

If you run iptables --list you should see a fail2ban target. Try breaking into SSH from another host, after a few tries you should be blocked from port 22 on the remote host. Running iptables-save will show you a rule under the fail2ban target for the IP that was just blocked. Once the bantime limit has been reached you will regain access.

Defending Against the SYN Flood

• karma

A SYN flood is a type of resource-starvation denial of service (DoS) attack in which the attacker creates enough "half open" connections to render a server inaccessible to the legitimate public. Because the attack takes advantage of weaknesses in the default configuration of most TCP implementations rather than raw strength, one attacker with a relatively low bandwidth connection can quickly take down a much better equipped server. The attacker only needs to send one SYN packet to establish a half-open connection on the defending server, which will in turn attempt to reset the connection a set number of times. Since the handshake has been initialized and the connection is being logged the deed is done; the attacker doesn't need to respond to the RST packet so the source address can be spoofed, making the task of tracing the attacker virtually impossible and the attack itself very difficult to block.

When you first come under attack it may not seem obvious  what is happening. The targeted host(s) will stop or sporadically respond to your users and you may not even be able to shell into the machine. If you can gain access to the machine the telltale signs are:

  • Services are running but using no CPU or I/O
  • Traffic graphs flatline but the host(s) remain pingable
  • Services appear to be listening on the right ports, the firewall is clear, but you can't connect to them even locally
  • Multiple TCP-based services are affected
  • The output of netstat -n indicates an unusually high number of connections in the SYN_RECV state

All or most TCP services will seem to be affected because they all share the same connection queue. Unless your server is very overloaded, even on high traffic sites you should never see more than about 5 or 6 connections in the SYN_RECV state sustained over any period of time - particularly if you reduce the number of retries your kernel attempts as outlined below.

Fortunately there are two ways to address this problem: stack tweaking and syncookies (for BSD/linux, other implementations exist). Since the SYN flood relies on a lengthy timeout and limited number of available connections the obvious first step is to increase these limits. Having a lot of extra RAM comes in handy here since it takes RAM to track the connections. In fact, in preventing most resource starvation tactics throwing more RAM (if available) at the problem is always a good blind first step - though never the solution. We can manipulate these values through the /proc interface:

# echo 3096 > /proc/sys/net/ipv4/tcp_max_syn_backlog

tcp_max_syn_backlog limits the number of half-open connections the kernel will track. This is the limit that gets exhausted when regular users are no longer able to connect.

# echo 2 > /proc/sys/net/ipv4/tcp_syn_retries
# echo 1 > /proc/sys/net/ipv4/tcp_synack_retries

tcp_syn_retries is the number of times the kernel will wait appx. 40 seconds and send out another SYN packet when trying to establish an outbound connection. This won't do you any good for SYN flood protection but it can mitigate the effects of some amplification/redirection techniques that use your hosts as soldiers. tcp_synack_retries limits the number of times the kernel will retry responding to a half-opened connection. The default is 5 and that means an attacker's connection could last in the queue for up to 180 seconds. If the attacker can open an easy 300 new half-open connections in that period it becomes clear how quickly your connection queue can be overrun. Setting this value too low can cause problems for people on weak links like dialup.

Obviously this isn't going to be enough; finite resources will always be finite resources. Syncookies are a genious little invention that in a nutshell validate that traffic coming to the host is sent from a real computer rather than a packet generator by sending a simple type of cryptographic challenge in the headers of outgoing packets that is "responded" to in the headers of incoming packets by the mechanics of tcp itself. Because spoofed traffic doesn't have a legitimate sending host behind it to  "hear" the challenge it (probably) does not contain  a valid response and the connection is swiftly discarded.

Syncookies are not enabled by default and enabling them will override the value in tcp_max_syn_backlog, but it won't hurt you to increase it anyway:

# echo 1 > /proc/sys/net/ipv4/tcp_syncookies

Most distributions include a "local" script that runs at the end of init, yours may have one specifically for the firewall. On Gentoo I put these rules in /etc/conf.d/local.start and on ClearOS /etc/rc.d/rc.firewall.local. Note that since NAT doesn't handle the connections themselves and only passes them through, simply turning on syncookies in your firewall will not protect everything behind it.

If you want to centralize or introduce a degree of separation between your SYN flood protection and regular servers you can use proxies, Squid and Apache both work in reverse and SOCKS proxies may work as well (don't quote me on that).

I was caught with my pants down once; I hadn't enabled syncookies on just one VM and it got SYN flooded (murphy's law of course) and that's a mistake you only make once. It underscored for me the importance of following some sort of thorough lockdown procedure before you deploy a new machine. That will be the subject of an upcoming article where I will attempt to compile a definitive checklist.

If you are running a virtualized environment or have the space for enough servers the easiest way to mitigate the harm a resource starvation attack can do to the continuity of your operations is to compartmentalize and space services out as much as possible. If you have a web server and a dns server 1-1 NATted to a public address and an attacker hits you on port 80 only the web server is going to lock up, your DNS and therefore mail and so on should continue to operate, until of course they figure it out. If you have to run DNS and mail and web and radius try to run them on different servers rather than one despite the overhead; when one plans a public-facing network one should think less in terms of bare economics and more in terms of capacity to absorb attack.

ClearOS PPTP Multi-Subnet Magic

• karma

ClearOS is the latest incarnation of ClarkConnect, a linux-based router/network appliance distribution produced by the similarly re-branded Clear Foundation.

This post will show you how to gently abuse Gateway Mode to set up a single PPTP tunnel for use with multiple private subnets if you need to use a standalone VPN server. This does not apply if your VPN server is the gateway for your private subnets.

If you run a large network chances are you don't want to bugger up your clients' ability to pptp out. That means, for all practical "supported" purposes, you can't use your ClearOS/CC gateway(s) as PPTP servers. The answer of course is to set up a "standalone" VPN server. If you're working with a $0 budget or you pay for your rack space this might not be an attractive solution. You can always turn an existing physical server into a VPN gateway but for the purposes of this guide we're going to assume you have a dedicated server or virtual machine.

In both cases the machine needs to have no less than two ethernet interfaces. If you're using a VM I find that Clear fits nicely into 96 megs of ram with 20 megs room to breathe once you've trimmed off all the crap you don't need (hald, messagebus, iscsid, etc - anything you won't use on a VM unless you're an X nut). Use

Code:
chkconfig --list

then

Code:
chkconfig --level 12345 <service name> off

for every excess service that loads in runlevel3. If you use physical hardware you will need to ensure both NICs are plugged into a switch otherwise they won't come up on boot (or, naturally, you can alter the networking init script).

For this example, you have three private networks at headquarters:

10.0.0.0/8
172.16.0.0/12
192.168.0.0/24

And your client will be on a local NATted subnet of say

192.168.1.0/24

Log into the webconfig of your new VPN server. Make sure Gateway Mode is selected. Set eth0 to external mode and assign a routed public address and gateway to it, this is the IP you're going to come in on. Now for each of the private networks add an IP to a virtual interface ON ETH0. Add an IP somewhere that will never be used to eth1 thus:

Code:
eth0 - 66.66.66.66 netmask 255.255.255.0 gw 66.66.66.1 (this is our public ip) eth0:1 - 10.0.0.66 netmask 255.0.0.0 eth0:2 - 172.16.0.66 netmask 255.240.0.0 eth0:3 - 192.168.0.66 netmask 255.255.255.0 --- eth1 - 192.168.111.66 netmask 255.255.255.0 (put it on a subnet you don't use)

By now you're wondering why we have this dummy eth1 that does nothing. It's very simple - we're tricking CC into routing these subnets over the vpn by using the built in Gateway Mode. I can't stress this enough: what we're doing is not ideologically correct. It is, however, clean and simple.

Enable the PPTP server if you have not already done so and set the IP pools to some unobstructed address space on one of your private subnets. Make sure the ClearOS firewall is happy passing GRE (it will tell you if it's not on the PPTP webconfig page).

Now on the client end you're going to create a PPTP connection just like you always would except for one twist: once the connection is established add the private subnets to the client's routing table on the ppp interface thus:

Code:
# pon clearvpn # route add -net 192.168.0.0/24 dev ppp0 # route add -net 172.16.0.0/12 dev ppp0 # route add -net 10.0.0.0/8 dev ppp0

And presto! You can talk to any host on any of those networks with one IP and one tunnel thanks to a gentle abuse of Gateway Mode. I like to speed this up by adding those lines to a short shell script.

If you're a windows user, leaving Use Remote Gateway checked in the connection preferences will get you the same effect with one small hitch - all of your traffic (web browsing, chat etc) will be routed through your VPN and any time you connect or disconnect whatever open connections you have will be reset.

Without getting too much into details you want to (at a CLI) type:

route print

remember the hex number for your PPP connection. Type:

route add 10.0.0.0 mask 255.0.0.0 192.168.0.66 metric 2 if (hex number)

You probably don't need the metric bit but some of the people connecting to my particular vpn server will sometimes be connecting to their own subnets and that's bad for routing. Substitute 192.168.0.66 with the private IP your VPN server sits on in the same subnet as the PPTP IP pool.

Naturally, replace 10.0.0.0/8 with whatever private subnet you're trying to add (you don't need to do the subnet you're actually put on by the PPTPd, that's automagic). For additional subnets: Lather. Rinse. Repeat.

Everything I've read so far says windows will automagically detect what interface to use based on the gateway you specified but that hasn't worked for me. If it works for you, great! Use the -p flag to save the routes to your registry. You will find however that there is nothing static about windows interface numbers particularly when dealing with on-again-off-again VPN connections.

Gentoo TFTPd

• karma

TFTP is most commonly used these days to remote boot small images, move firmware and configuration details. Despite its name it shares little in common with FTP other than it's primary purpose: moving files. This post will leave you with a configured and running TFTP daemon on Gentoo.

There are a number of TFTP servers in portage, for the sake of expedience we'll only look at atftpd.

# emerge atftp

Now create a root directory to serve files from and change its permissions:

# mkdir /var/tftp
# chown nobody: /var/tftp

Edit atftp's configuration file to look like this:

# nano /etc/conf.d/atftp

# Config file for tftp server

TFTPD_ROOT="/var/tftp"
TFTPD_OPTS="--daemon --user nobody --group nobody"

Now drop the image or whatever you plan on transferring into /var/tftp. Start the server:

# /etc/init.d/atftp start

Run this to make is start on boot:

# rc-update add atftp default

And away you go!

It looks sort of like this...

• karma

When talking about the hosting platform I use to deliver the BKN sites it's hard to get across the makeup of the network to someone who's not familiar with virtual machines and vlans. This diagram illustrates how the network is logically divided and connected:

One managed switch is divided into an external vlan and an internal vlan.  This prevents traffic from the private network from crossing over onto the upstream network which is also patched into the external side of the switch. There are currently four physical servers and each one has a physical (cat6) connection both to the external vlan and the internal vlan ports on the switch. It's safe to think of the virtual machines as tiny servers stuffed in a physical server's package. Inside of the physical server there are virtual network connections that function like real world cables and switches.

Any number of virtual machines may route for the internal vlan and since every physical server is connected to both the external and internal sides of the switch the physical location of the router VM can quickly move to or be replaced at any other server. This opens up not only the option to live-migrate the virtual machine without dropping connections but also run a standby router (or small army) with IP failover that can automatically cut in if the active physical server fails. Not yet implemented yet at this location but I'm working on it.