pitter patter on the keyboard

Plain FTP suffers from a number of problems, foremost is transmission in cleartext. Unless SSL is used (I have come across very few occurrences of its use in the wild) usernames and passwords, as well as the files being transmitted, are sent over the wire unencrypted. This means anyone with a well-placed sniffer or man-in-the-middle setup between the client and server can intercept or mutilate the data without much skill or resource cost. FTP also suffers from NAT issues, in that some users must forward ports or use passive transfer mode – words most of the users who will be using your hosting service have probably never heard, don’t want to hear and have no idea how to implement.

Enter SFTP: file transfer over Secure Shell. Not to be confused with FTPS, or “FTP Secure” (FTP with SSL), SFTP shares nothing in common with FTP other than its purpose: moving files around the network. SFTP uses a single, wholly encrypted tunnel from the client to the server to send and receive files. This avoids the problems associated with NAT and FTP’s dual-connection implementation, provides a relatively secure means of authentication and prevents third-party manipulation of the data in transit. SSH also warns users when a server’s RSA keys change so diligent users can identify and avoid potential man-in-the-middle attacks. SFTP is not, however, a perfect solution. OpenSSH, the implementation which we will be dealing with in this article, can be found on most Linux and BSD servers. Due to its ubiquity it is a prime target for 0-day exploits on one hand, and well hardened against known exploits on the other (you do keep your software patched and up-to-date, right?).

SSH, like any service that uses a username-password authentication scheme – including FTP, is vulnerable to brute-force or dictionary cracking attacks. This problem can be virtually eliminated by disabling user-pass authentication altogether and using on shared keys (please my article Passwordless or Single Password SSH with Key Exchange for instructions on implementing this configuration) however this is probably not an appropriate solution for your public hosting project as the process is difficult for regular users to implement, particularly if they are using a Windows client such as FileZilla. One solution which I highly recommend is fail2ban, it will read your sftpd logs and temporarily block an IP address associated with a specified number of failed attempts over a given period of time. I have provided simple instructions for implementing fail2ban for SSH in my article Stifling Brute Force Attacks with fail2ban. Because we are dealing with a situation where users are prone to forget their password you may wish to use fairly loose criteria in configuring fail2ban, enough failures should be tolerated that a forgetful user won’t be quickly blocked but an automated attack should be picked up. You might also wish to block access to sftpd rather than all ports as affected users might understand sftp disappearing after several failed attempts but could think the server is down if they are unable to access their website (assuming it is hosted from the same server).

The “chroot jail” concept is as old as the hills and has provided a way for us to separate vulnerable services from the filesystem-at-large by faking them into thinking a certain directory is the absolute root (/) of the filesystem. In this article we’re going to apply this concept to regular users using OpenSSH’s built-in functionality available since version 4.9. Before then it was a royal pain to implement whereas most popular FTP daemons had supported the feature out-of-the-box for years, now SFTP can finally be considered a complete and suitable replacement.

One probably does not wish to restrict SSH access for ALL user accounts, since one probably needs remote administrative access to the machine. Therefore one shall create a group to which users who must be chroot jailed will be added. For the purposes of this article we shall call the group hosted but you may call it anything:

# groupadd hosted

Next add users to the group, if you want to make sure these users do not get regular shell access (the ability to log in and run commands etc) be sure to specify a dummy shell, such as /bin/false or /sbin/nologin (Gentoo):

# useradd -G hosted -s /bin/false demo-user
or
# usermod -G hosted -s /bin/false demo-user

Don’t forget to give your test account a password if it is a new account.

# passwd demo-user

Now open your sshd config file (most users: /etc/ssh/sshd_config) and go to the very bottom. If there is a Subsystem sftp line already delete it. Add the following:

Subsystem       sftp    internal-sftp

Match Group hosted
        ChrootDirectory %h
        ForceCommand internal-sftp
	AllowTcpForwarding no

The Match Group directive tells sftpd to use the proceeding settings for any user in the group hosted. Now you must change the owner (and optionally group) of the user’s home directory to root:

# chown root: /home/demo-user

Due to these permissions the user will not be able to create new files at the top level of their directory tree, which to you looks like /home/demo-user and to them /. That’s fine because we’re going to give them a directory for the site they want to host with appropriate ownership, then they can do whatever they want in that. I typically set users up thus:

# mkdir ~demo-user/demo-site.com
# mkdir ~demo-user/demo-site.com/htdocs
# mkdir ~demo-user/demo-site.com/log
# mkdir ~demo-user/demo-site.com/cgi-bin
# chown demo-user: ~demo-user/demo-site.com/ -R

Restart your sshd (/etc/init.d/ssh(d) restart) and try logging in as your jailed user via sftp. If it works, congratulations. If you chose to restrict regular SSH access you may need to include /bin/false (or /sbin/nologin etc) to your /etc/shells valid shells list or the user may not be able to log in at all. Try logging in via SSH to ensure access has been blocked.

In the last article I told you any username-and-password authentication system that is exposed to the Internet is inherently vulnerable to dictionary and brute force attack. If you must use such an authentication scheme you can defend it by implementing rate control. If you block an attacker from trying to log in for one hour after three failed attempts it would take them a year to try just under 3,000 combinations. In cryptanalytic terms that is abysmal and the odds are on your side that the attacker will have moved on by then.

While porting your ban system to fail2ban might be a great idea it’s probably overkill for situations where you have hundreds of legitimate users who might often forget their credentials; IP-bans are not generally considered good customer service. Many sites, including Google, will present the user with a CAPTCHA after three failed attempts and that’s great but those are getting easier to crack every day.

For the sake of the pseudocode in this article we’re going to assume you want to block the  potential attacker and politely tell them they have either a) failed to log in too many times, please come back in an hour or b) posted too recently, please try again. Since we want to be able to rate control two (and perhaps more in the future) different things and we don’t want to make a mess of our database let’s make one table called ‘greylist’ and use the type column to differentiate:

CREATE TABLE `demo_cat`.`greylist` (
`type` VARCHAR( 30 ) NOT NULL ,
`date` INT NOT NULL ,
`ip` VARCHAR( 15 ) NOT NULL ,
PRIMARY KEY ( `ip` ) ,
INDEX ( `date` )
);

Now in your login script for argument’s sake we’ll say $outcome is a boolean representation of if the authentication was successful or not and $delay is the period of time we want to measure for in seconds. We’ll start off by clearing everything that’s out of date, a relatively inexpensive query to run every time there’s a failure. After the table has been updated we’ll add an entry for the current failure and take a tally of all the entries for the user’s IP. If the tally exceeds the retry $threshold we’ll tell them to buzz off for an hour, change their password, show a captcha or whatever suits your site best.

<?php

if(!$outcome)
{
   $ip = mysql_real_escape_string($_SERVER['REMOTE_ADDR']);
   mysql_query("delete from `greylist` where `type` = 'login' and `date` < '".time()-$offset."'");
   mysql_query("insert into `greylist` (`type`, `date`, `ip`) values ('login', '".time()."', '$ip'')");
   $result = mysql_query("select `ip` from `greylist` where `type` = 'login' and `ip` = '$ip'");
   if(mysql_num_rows($result > $threshold))
   {
      // Too many tries, what now?
   }
   else
   {
      // Please try again
   }
}

?>

It is as simple as that. Now let’s use this to flood-protect our comments box:

<?php

if($_POST)
{
   $ip = mysql_real_escape_string($_SERVER['REMOTE_ADDR']);
   mysql_query("delete from `greylist` where `type` = 'comment' and `date` < '".time()-$offset."'");
   mysql_query("insert into `greylist` (`type`, `date`, `ip`) values ('comment', '".time()."', '$ip'')");
   $result = mysql_query("select `ip` from `greylist` where `type` = 'comment' and `ip` = '$ip'");
   if(mysql_num_rows($result > $threshold))
   {
      // You posted too recently, please wait x seconds before trying again.
   }
   else
   {
      // Continue...
   }
}

?>

A more sophisticated implementation of this concept is in use at Ychan, where users’ posting patterns are analyzed to determine if they are computers, legitimate humans or computers trying to look like humans.

fail2ban is a package that monitors your log files for failed login attempts and executes a configured action, usually temporarily blocking the attacking IP with iptables for a set duration. Any exposed service that uses a username/password authentication scheme is vulnerable to dictionary and brute force attack, your first defense if you must expose such a service is to make such attacks as costly as possible and that’s where fail2ban comes in. By temporarily blocking an address for even 10 minutes after every 3 failed login attempts you make the process several orders of magnitude slower. Since fail2ban reads plain log files and can be configured for any action one clever deployment could see a log server collecting logs from all the hosts on a network and sharing the relevant logs with the firewall via NFS where fail2ban can quickly cut access to the entire network from the attacker with ease. For the purposes of this article we will only focus on locking down SSH on a local host.

fail2ban is probably available in your distribution’s package management system. Gentoo users type:

# emerge fail2ban

If the package is not available for your flavour you can compile it from source, available at http://sourceforge.net/project/showfiles.php?group_id=121032&package_id=132537:

# tar xjf fail2ban-*
# cd fail2ban-*/
# ./setup.py install
# cp /usr/local/src/fail2ban-*/files/{your distro or close match here}-init /etc/init.d/fail2ban

Then add the script to the appropriate runlevels. Gentoo users type:

# rc-update add fail2ban default

Despite the name, fail2ban jails are not like chroot or ssh jails. A ‘jail’ is the combination of a filter and an action. The filters are regular expressions used to search the log files for interesting lines such as login failures. These filters are located in /etc/fail2ban/filter.d/ and the action scripts are located in /etc/fail2ban/filter.d/. By adding to and tying these filters and actions together in /etc/fail2ban/jail.conf you can re-purpose fail2ban to do just about any log event-triggered action imaginable; once you’ve given it a good mucking about locking down SSH may seem trite.

Open /etc/fail2ban/jail.conf and find [ssh-iptables], change the configuration block to look like this:

[ssh-iptables]
enabled = true
filter = sshd
action = iptables[name=SSH, port=ssh, protocol=tcp]
logpath = /var/log/sshd/current
maxretry = 5
# findtime = 600
# bantime = 600

You may need to edit logpath to reflect your system’s settings. Set maxretry to however many failed login attempts you wish to allow over a given amount of time (findtime) until the source address is blocked for a given amount of time (bantime). The default findtime and bantime  is 600 seconds (10 minutes) and only needs to be set if you would like to choose different durations. If you would like to be notified by e-mail when someone has been blocked (probably not a good idea on a busy public server) add this line to the jail:

mail-whois[name=SSH, dest=yourmail@mail.com]

Now make sure your SSH daemon is logging in verbose mode, add this line if you must to /etc/ssh/sshd_config:

LogLevel VERBOSE

If your sshd log entries contain the string pam_unix(sshd:auth) (Gentoo users here) you may need to modify the line starting with __daemon_re in /etc/fail2ban/filter.d/common.conf to look like:

__daemon_re = [\[\(]?%(_daemon)s(?:\([^\)]+\))?[\]\)]?:?

and configuration is over. Now start the server:

/etc/init.d/fail2ban start

If you run iptables –list you should see a fail2ban target. Try breaking into SSH from another host, after a few tries you should be blocked from port 22 on the remote host. Running iptables-save will show you a rule under the fail2ban target for the IP that was just blocked. Once the bantime limit has been reached you will regain access.