=^.^=

Mass Virtual Hosting Part Six: (Remote) Apache Vhost Configuration and Privileged Command Execution with Database-Backed Upkeep System

What a mouthful.

The more I worked with mod_vdbh the more I realized it's not for me. Neither is its more advanced fork, mod_vhs which adds some useful extensions. While database-backed configuration is an elegant solution to mass virtual hosting it falls short in that it lacks these key (for my purposes) abilities or requires hackery that violates the Keep It Simple, Stupid rule

  • Apache configuration can not be customized on a per-vhost basis, requiring the use of generated .htaccess files for tricks that might be implemented in a user-friendly way via a front-end, such as custom error pages.
  • Additional Directory directives (i.e. those that don't match the wildcard directives for whatever reason) need to be put into flat files anyway.
  • Logs for individual sites must be split from a common log by a resource-consuming external solution
  • ScriptAlias directive (mod_vhs) only works for one global directory for all sites or not at all (mod_vdbh)
  • These modules are unmaintained, if something better comes out there is the whole hassle of migration to contend with.
  • A new version of apache may break old modules, but flat files will always be supported.

This is a serious problem if, like me, you are used to setting up vhost accounts in such a fashion:

/home/user/website.com/htdocs
/home/user/website.com/log
/home/user/website.com/cgi-bin

How can using flat files for mass virtual hosting be as easy to manage as databases one might ask? The answer is simple: generate the flat files from data stored in a database.

My configuration front-end sits on a web server that is independent from the one that serves the 'public' virtual hosts. This necessitated the ability to execute commands remotely as root, such as creating the directories for a new host, while at the same time taking into consideration the security implications of average users being able to pass data to the server as root.

My solution came in the form of a two-part system; a shared database that is used to pass sanity-checked data from the configuration interface and an administrative upkeep script run as root by cron every 5 minutes on the virtual hosting server. The script executes appropriate commands with the data set provided then flags its tasks as completed in the database. By storing arrays of raw data and indicating the job type one can avoid altogether the inherent problem of sending straight-up command line commands to the remote server. Careful variable checking in both the configuration interface where the data is added, then in the upkeep script where the tasks are then run can result in rock solid security, despite the fact that we are talking about translating a user's actions on a web page to root-privileged commands.

In my frustration with the database modules for Apache I realized that the same system could be adapted to write, delete and overwrite individual apache configuration files based on back-end information. It helps to think of the situation like when one uses memcache or APC variable caching with an SQL database; the actual driving force behind the application is the database information but as it is pulled into memory the cache sits between them. By simply dropping the files into a directory and using a wildcard with an Include directive all it takes is reloading apache once updates have been performed for changes to take effect. Through maintaining a master database from which the files are generated one could easily delete the entire directory, run a regeneration script and they would all reappear.

It's at this point one may find one's head shaking: one of the prime benefits of database-backed configuration is that the apache server does not have to be restarted when a new vhost is added. Indeed, it is the very reason most people seem to switch to the solution. However I didn't say restart, I said reload - it seems a lot of people have overlooked apache's ability to reload configuration files gracefully - that is without being restarted all at once and without dropping open connections. Fortunately, I have not.

One of the major benefits of this approach weighed against database-only configuration is once apache has been reloaded all of the configuration is loaded into RAM; there is no need to worry about thread safety or hundreds of redundant connections and queries to your database server - a problem that becomes worse as your platform scales up without the aide of some sort of abstraction layer like pooled connections (mod_dbm support currently still in development (or not) for mod_vhs) or caching as could be implemented with much hackery and mysql-proxy.

This article will show you - in much simpler and specific detail - how I have implemented what I call an upkeep system that can manage virtual hosts and run privileged commands passed to it (from Joe User in his web browser) safely on a local or remote server with nothing more than PHP and MySQL. It is not at all hard to imagine this system being adapted to manage multiple servers, the workload distributed or logically divided among them given some automated mechanism or instruction from the configuration interface.

Bear in mind that it's not the norm to use PHP for server-side-only scripts and you may wish to implement the idea in PERL or Python or something more traditional - but the application is sound, I'm good at PHP and I am secure in my masculinity :) PHP should be available on the web servers we intend to manage anyway, but may be more trouble than it's worth to install it on, say, an NFS-only server if you want to split up file-related commands from configuration tasks.

The other drawback to this approach is the 5-minute delay between runs of the upkeep script. To address this I simply add a notification event to the database which the configuration interface searches for and reports to the user if they have any tasks pending completion.

First, create the shared table on a mutually-accessible SQL server. Remember to use one account for administration and one for the upkeep script, applying permissions frugally.

CREATE TABLE IF NOT EXISTS `upkeep` (
 `id` bigint(20) NOT NULL auto_increment,
 `uid` int(11) NOT NULL default '0',
 `date` int(11) NOT NULL default '0',
 `completed` int(11) NOT NULL default '0',
 `status` varchar(30) NOT NULL,
 `type` varchar(30) NOT NULL,
 `data` longtext NOT NULL,
 PRIMARY KEY  (`id`)
)

And we need one for the virtual hosts:

CREATE TABLE IF NOT EXISTS `virtual_hosts` (
 `id` int(11) NOT NULL auto_increment,
 `date` int(11) NOT NULL,
 `user` varchar(255) NOT NULL,
 `yuid` int(11) NOT NULL,
 `appid` int(11) NOT NULL,
 `server` char(255) NOT NULL,
 `environment_variable` char(255) default NULL,
 `subof` int(11) NOT NULL,
 `firewall` longtext NOT NULL,
 `errorpages` longtext NOT NULL,
 PRIMARY KEY  (`id`),
 KEY `yuid` (`yuid`),
 KEY `server` (`server`)
);

I'm using the uid field in the first table and the yuid field in the second to store the administrative interface's account ID number or the person who uses the functions. Next we're going to need some functions for the administration front-end to interface with:

function hostedUPKEEP($type, $data, $uid=0)
{
 if(is_array($data))
 $data = serialize($data);

 $data = mysql_real_escape_string($data);

 mysql_query("insert into `upkeep` (`uid`, `date`, `status`, `type`, `data`) values ('$uid', '".time()."', 'pending', '$type', '$data')");
}

function hostedUPKEEPMkdir($user, $group, $path)
{
 $data['user'] = $user;
 $data['group'] = $group;
 $data['path'] = $path;

 hostedUPKEEP('mkdir', $data);
}

function hostedVHOSTAddHost($user, $yuid, $appid, $server)
{
 $firewall = $errorpages = serialize(array());
 $date = time();

 mysql_query("insert into `virtual_hosts` (`user`, `date`, `yuid`, `appid`, `server`, `firewall`, `errorpages`) values ('$user', '$date', '$yuid', '$appid', '$server', '$firewall', '$errorpages')");
 $idgrabr = mysql_query("select * from ``virtual_hosts` where `user` = '$user' and `server` = '$server'");
 $idgrabo = mysql_fetch_object($idgrabr);

 hostedUPKEEPMkdir('root', 'root', "/home/$user");
 hostedUPKEEPMkdir($user, 'hosted', "/home/$user/$server");
 hostedUPKEEPMkdir($user, 'hosted', "/home/$user/$server/htdocs");
 hostedUPKEEPMkdir($user, 'hosted', "/home/$user/$server/log");
 hostedUPKEEPMkdir($user, 'hosted', "/home/$user/$server/cgi-bin");
 hostedUPKEEP('vhost', $idgrabo->id);
 hostedUPKEEP('notification', 'jobs pending notification', $yuid);
}

Since staff may edit user settings I pass the affected user's front-end UID to a notification event, the user's front-end will look for events marked pending with their UID and report that they must wait a little while for changes to take effect. Before passing any data to these functions it is important that you make sure it has been as carefully sanitized as possible. The following is a simple upkeep script that you can drop into /sbin/, chown root: and chmod 700 then add to cron at your preferred interval:

#!/usr/bin/php
<?php

$sql_host = '';
$sql_user = '';
$sql_pass = '';
$sql_base = '';

$sql_h = mysql_pconnect($sql_host, $sql_user, $sql_pass);
$sql_d = mysql_select_db($sql_base, $sql_h);

$forbidden_users = array('www',
'ftp',
'sql',
'mysql',
'database',
'db',
'sftp',
'ftps',
'sync',                                                                                                                          
'shutdown',                                                                                                                
'halt',                                                                                   
'mail',                                                                                                                 
'news',
'uucp',
'operator',
'calendar',
'docs',
'man',
'postmaster',
'cron',
'ftp',
'sshd',
'ssh',
'at',
'squid',
'gdm',
'xfs',
'games',
'named',
'postgres',
'apache',
'admin',
'administrator',
'cyrus',
'vpopmail',
'alias',
'qmaild',
'qmaill',
'qmailp',
'qmailq',
'qmailr',
'qmails',
'postfix',
'smmsp',
'portage',
'guest',
'nobody',
'clamav',
'amavis',
'vmail',
'ntp',
'deleted',
'mrtg',
'sockd',
'lighttpd',
'memcached',
'smokeping',
'rpc',
'anon',
'site',
'sites',
'anonymous',
'pop',
'pop3',
'smtp',
'sendmail',
'information_schema',
'test');

function checkname($string)
{
 global $forbidden_users;
 foreach($forbidden_users as $fuse)
 {
 if($fuse == strtolower($string))
 die("Forbidden User");
 }
}

$result = mysql_query("select * from `upkeep` where `status` = 'pending' order by `date` asc");
while($object = mysql_fetch_object($result))
{
 if($object->type == 'mkdir')
 {
 $data = unserialize($object->data);
 checkname($data['user']);
 checkname($data['group']);
 exec("mkdir -p ".escapeshellcmd($data['path']));
 exec("chown ".escapeshellcmd(ereg_replace("[^A-Za-z0-9]", '', $data['user'])).":".escapeshellcmd(ereg_replace("[^A-Za-z0-9]", '', $data['group']))." ".escapeshellcmd($data['path']));
 }

 if($object->type == 'vhost')
 {
 $id = $object->data;
 $vhost_result = mysql_query("select * from `virtual_hosts` where `id` = '$id'");
 $vhost_object = mysql_fetch_object($vhost_result);

 $domain = $vhost_object->server;
 $user = ereg_replace("[^A-Za-z0-9]", '', $vhost_object->user);

 checkname($user);

 $errarray = unserialize($vhost_object->errarray);
 if(!empty($errarray[0]))
 {
 $errorpages = '';
 foreach($errarray as $code => $loc)
 {
 $errorpages .= "\n\tErrorDocument $code $loc";
 }
 }
 else
 {
 $errorpages = '';
 }

 $fwarray = unserialize($vhost_object->fwrarray);
 if(!empty($fwarray[0]))
 {
 $firewall = '';
 foreach($fwarray as $address)
 {
 $firewall .= "\n\tDeny from $address";
 }
 }
 else
 {
 $firewall = '';
 }

 $domainesc = str_replace('.', '\.', $domain);

 $file = "<VirtualHost *:80>
\tServerName $domain
\tServerAlias www.$domain
\tDocumentRoot /home/$user/$domain/htdocs
\tScriptAlias /cgi-bin/ /home/$user/$domain/cgi-bin/
\tErrorLog /home/$user/$domain/log/error_log
\tTransferLog /home/$user/$domain/log/access_log{$errorpages}
\t<IfModule mod_rewrite.c>
\t\tRewriteEngine on
\t\tRewriteCond %{REQUEST_METHOD} !^(GET|POST|HEAD)$
\t\tRewriteRule .* - [F]
\t\tRewriteCond %{HTTP_HOST} ^www\.$domainesc$ [NC]
\t\tRewriteRule ^(.*)\$ http://$domainesc/\$1 [R=307,L]
\t</IfModule>
\t<IfModule mod_access.c>
\t\tOrder Allow,Deny
\t\tAllow from all{$firewall}
\t</IfModule>
</VirtualHost>";

 $fh = fopen("/etc/apache2/hosted.d/{$user}_{$domain}.conf", 'w');
 fwrite($fh, $file);
 fclose($fh);

 exec("/etc/init.d/apache2 reload");        // Change the path to your apache2ctl if the init script does not support reload.
 }

 mysql_query("update `upkeep` set `status` = 'completed', `completed` = '".time()."' where `id` = '{$object->id}'");
}

?>

Now create the directory /etc/apache2/hosted.d (or whatever you prefer) and add this directive to the end of your httpd.conf:

Include /etc/apache2/hosted.d/*.conf

Mass Virtual Hosting Part Five: Dynamic MySQL Based Apache vhost Configuration with mod_vdbh

Please skip to Part 6 for advanced vhost configuration with database-backed flat files.

mod_vdbh is a relatively obscure gem of an Apache module. It doesn't look like it has been maintained in years and its website is gone so Google (at present) won't give you much on it except the usual package list results and odd blog post like this. Despite living in a day of quadruple-digit version numbers, some software reaches a point where it's "just done." (if you don't believe me look at qmail). I'm hoping that's the case here, because if there's any 0day or problems with future versions of apache we're SOL. FreeBSD and Gentoo keep it in their package managers and that's more or less good enough for me.

The README is hard to find so I'm posting it below for your viewing pleasure:

Configuring mod_vdbh in Apache Configure Files

In order to use mod_vdbh with Apache Web Server server configuration blocks will need to be configured with mod_vdbh configuration directives described in the table below. mod_vdbh configuration directives must be located in a server configuration block (ie <VirtualHost></VirtualHost>).

vdbh    This switch makes mod_vdbh active for the specified server.
vdbh_CLIENT_COMPRESS    Enables the CLIENT_COMPRESS option with a MySQL server allowing the connection data to be compressed. Using this option will likely require more cpu time and less network bandwidth.
vdbh_CLIENT_SSL Enables the CLIENT_SSL option when communicating with a MySQL server.
vdbh_MySQL_Database     Sets the database name to use when running a query for file name translations.
vdbh_MySQL_Table        Sets the table name to use when running a query for file name translations.
vdbh_MySQL_Host_Field   Sets the name of the host field in the table specified by vdbh_MySQL_Table.
vdbh_MySQL_Path_Field   Sets the name of the path field in the table specified by vdbh_MySQL_Table.
vdbh_MySQL_Environment_Field    Sets the name of the environment field in the table specified by vdbh_MySQL_Table. This optional field contains data that will be set to the VDBH_ENVIRONMENT variable.
vdbh_MySQL_Host Sets the internet hostname where the MySQL server is located at. This option is not required and defaults to localhost.
vdbh_MySQL_Port Sets the port number to connect to when making a connection to a MySQL server. This option is not required and defaults to 0 for using a UNIX domain socket.
vdbh_MySQL_Username     Sets the username required to gain access to the MySQL server. This option is not required.
vdbh_MySQL_Password     Sets the password required to gain access to the MySQL server. This option is not required.
vdbh_Path_Prefix        Sets an optional location to prefix translations by. This option is not required.
vdbh_Default_Host       Sets the default host to use if a non-HTTP/1.1 request was received. This option is not required and usually won't do anything because the Apache Web Server by default catches these errors.
vdbh_Declines   Sets a list of glob patterns to match URIs against. If any match occurs then the URI is declined to the next translate phase.

The vdbh_MySQL_Host_Field and vdbh_MySQL_Path_Field along with vdbh_MySQL_Environment_Field are available as environment variables and can be included in logs if a LogFormat is defined for them. The environment variables are labled VDBH_HOST, VDBH_PATH, and VDBH_ENVIRONMENT. Information on how to use LogFormat is available at http://httpd.apache.org/docs/mod/mod_log_config.html. An example configuration may look something like this.

NameVirtualHost 206.9.161.29

<VirtualHost 206.9.161.29>
vdbh On
vdbh_CLIENT_COMPRESS On
vdbh_MySQL_Database virtual_hosts
vdbh_MySQL_Table virtual_hosts
vdbh_MySQL_Host_Field server
vdbh_MySQL_Path_Field path
vdbh_MySQL_Environment_Field environment_variable
vdbh_Default_Host julia.fractal.net
vdbh_Declines .htpasswd *.txt
</VirtualHost>

The corresponding database schema would look like this.

CREATE TABLE virtual_hosts (
server char(255) NOT NULL,
path char(255),
environment_variable char(255),
PRIMARY KEY (server)
);

INSERT INTO virtual_hosts VALUES ('julia.fractal.net','/export/home/mlink/public_html','julia.fractal.net');
INSERT INTO virtual_hosts VALUES ('visualphixation.com','/export/home/carlosp','visualphixation.com');
INSERT INTO virtual_hosts VALUES ('www.visualphixation.com','/export/home/carlosp','www.visualphixation.com');
INSERT INTO virtual_hosts VALUES ('www.fractal.net','/export/web/www.fractal.net','www.fractal.net');
INSERT INTO virtual_hosts VALUES ('fractal.net','/export/web/www.fractal.net','fractal.net');

Other handlers should still work accordingly.  mod_vdbh declares its translate_name phase as AP_HOOK_FIRST so it can run before other translations.  An example configuration allowing mod_tcl in specific directories follows.

<VirtualHost 206.9.161.29>
vdbh On
vdbh_CLIENT_COMPRESS On
vdbh_MySQL_Database virtual_hosts
vdbh_MySQL_Table virtual_hosts
vdbh_MySQL_Host_Field server
vdbh_MySQL_Path_Field path
vdbh_MySQL_Environment_Field environment_variable
vdbh_Default_Host julia.fractal.net
vdbh_Declines .htpasswd *.txt

<Directory /export/web/www.fractal.net>
AddHandler tcl-handler tm

Tcl_ContentHandler content_handler
</Directory>

<Directory /export/web/www.fractal.net/images>
SetHandler default-handler

Options Indexes FollowSymLinks

AllowOverride None

Order allow,deny
Allow from all
</Directory>

<Directory /export/web/www.fractal.net/files>
SetHandler default-handler

Options Indexes FollowSymLinks

AllowOverride None

Order allow,deny
Allow from all
</Directory>
</VirtualHost>

Additional Information

mod_vdbh assumes that its connection to the MySQL server is persistent. If there are excessive disconnections try setting the wait_timeout variable for MySQL to a larger value. Apache Web Server 2.0 is required, and at least MySQL 3.23 is required.

References

mod_vdbh is an Apache 2.0 module using MySQL libraries, more about Apache Web Server can be found at http://www.apache.org/. Documentation regarding MySQL can be found at http://www.mysql.com/

That's right. That's all there is to it. If you've been following the other parts in this series on Mass Virtual Hosting you should have a keen eye for the ways MySQL-backed services can be used (sexually?) to integrate into your custom web hosting front-end - or anything that interfaces with MySQL/ODBC!

Remote Ethernet Packet Capture with Wireshark and tshark over SSH

Wireshark is a powerful and popular packet capture and analysis suite that runs on Windows and most flavours of UNIX. Often one finds one's self in need of its GUI's abilities on remote, headless servers without X windows (and who wants to install X on a server if they don't have to?). One has three options: use a text/ncurses based packet capture system like ettercap to analyze the traffic on the server itself, save packet capture files and move them to your Wireshark host or pipe the output from tshark - Wireshark's text interface - to your client in real-time. The last option suits me best; I don't want to have to learn two packet capture suites if I can only use one and it is often useful to see the packets fly by as they come.

To compile Wireshark without the GUI, and therefore all of its X windows dependencies, on Gentoo:

# USE="-gtk" emerge wireshark

Or disable the GTK use flag in your /etc/make.conf.

The next step is to establish passwordless root ssh access to the target machine. This should only be temporary as it is best practice to disallow any form of remote login for the root user. Please read my previous article, Passwordless or Single Password SSH with Key Exchange but be sure to use a blank passphrase for your key and disregard the part about restricting root access. Once this has been completed and you are able to log in to the target server by simply typing ssh hostname you are ready to begin your packet capture.

On the client which runs the GUI version of Wireshark, open up a shell as root and run the following:

wireshark -k -i < ( ssh -l root xxx.xxx.xxx.xxx /usr/bin/tshark -i eth0 -w - )

Be sure to change the path to tshark if this does not reflect your installation. Adjust the interface (-i flag) to match your target.

Mass Virtual Hosting Part Three: Disk Quotas (including NFS)

Disk quotas allow one to limit the amount of space a user or group may use on a particular filesystem. The traditional linux quota implementation allows two sorts of limit: soft limits, which the user/group may exceed for a given grace period and hard limits which may not be exceeded at all. Soft limits are great in situations where users may need significant amounts of storage only temporarily, as in the case with burning ISOs or intensive rendering software and other temporary-file generating activity. In a webhosting environment one typically offers different plans with a set limit of storage, so soft limits are probably redundant for our purposes. Since quotas can be applied to any user or group they can also be used to ensure particular daemons do not run roughshod over the filesystem, like apache access logs might during an HTTP GET denial of service attack.

If you will be using NFS to remotely mount filesystems you intend to implement quotas on you must implement them on the NFS server and use some sort of UID/GID consistency like NIS or libnss-mysql. Your kernel must be compiled with quota support or have it available as a module to enforce the limits. It is possible to set up quotas on a system running a quota-incapable kernel then activate them later by incorporating quota support. In menuconfig, set this option to module or compiled-in, if you choose to compile it as a module ensure that it is automatically loaded:

  • File systems
    • Quota support

You must also install the userspace tools, emerge quota on Gentoo. Next add usrquota and/or grpquota depending on your needs to the options field of the target filesystem(s), i.e:

/dev/sdX1               /mnt/storage     ext3            defaults,nosuid,noexec,nodev,noatime,usrquota,grpquota   0 0

In this example we are also disabling binary execution and some potentially dangerous filesystem options such as SUID and device files for security purposes. You may remount the filesystem to apply the changes:

# mount -o remount /mnt/storage

Now in the root of every filesystem to have quotas create and secure the quota files:

# touch /mnt/storage/aquota.user
# touch /mnt/storage/aquota.group
# chmod 600 /mnt/storage/aquota.user
# chmod 600 /mnt/storage/aquota.group
# /usr/sbin/quotacheck -avug

Unless you will be exclusively using XFS you must add the quota init script to your runlevel. On Gentoo:

# rc-update add quota boot

Next you must decide how often the quotas are checked, in other words how often the total recorded space users and groups are consuming  is updated. You must weigh the importance of accurate reporting against the potential resource load scanning the filesystem(s) may incur. Drop this scriptlet into a /etc/cron.* directory and chmod +x it:

#!/bin/bash
/usr/sbin/quotacheck -avug

Quotas can be edited by the program edquota, with the -u flag and a user's name or a -g flag and a group's name respectively. Use the -f flag and the target filesystem's mountpoint to restrict operations to one particular filesystem, otherwise edquota will default to all filesystems with quotas enabled. Edquota uses your EDITOR environment variable to load a temporary file containing a tabular representation of a user's soft and hard quotas as well as currently used space. Simply change the soft and hard quota limits and save the file, the new values will be applied immediately. This is how user test's quota looks like when edited with nano:

# edquota -f /mnt/storage -u test

Disk quotas for user test (uid 5000):
  Filesystem                   blocks       soft       hard     inodes     soft     hard
  /dev/sdX1                         0          0          0          0        0        0

It should be noted here that quotas can also be set on the number of inodes a user or group may use, effectively limiting the total number of files they can create. This is probably not practical for our needs, where space is the concern and we will mostly be hosting websites composed of relatively many, relatively small files. The blocks and inodes fields tell us how much the user was using the last time quotacheck was run while the soft and hard fields are their limits respectively. Once you have finished configuring the user or group's quotas simply save and exit the editor and the changes will be saved.

We can use repquota to see a filesystem's overall use on a per-user basis, simply specify the mountpoint of the filesystem in question:

# repquota /mnt/storage/
*** Report for user quotas on device /dev/sdX1
Block grace time: 7days; Inode grace time: 7days
                        Block limits                File limits
User            used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      --  180240       0       0              7     0     0
test      +- 1738760     500     500  3days       5     0     0

The output is similar to edquota. Note the -- and +- column: the first character indicates whether the user's hard quota is over limit or under limit, denoted by the + and - symbols respectively, and the second character represents the soft quota. In this example we can see user test is very much over their 500 block hard limit. They will not be able to create any more files until they have cleared out enough space to put them back under the limit.

Quotas are fully enforced on an NFS server, but to share information about the quotas to NFS clients you must ensure rpc.rquotad is running. On gentoo alter /etc/conf.d/nfs to reflect:

# Optional services to include in default `/etc/init.d/nfs start`
# For NFSv4 users, you'll want to add "rpc.idmapd" here.
NFS_NEEDED_SERVICES="rpc.idmapd rpc.rquotad"

Then restart your nfs init script:

# /etc/init.d/nfs restart

On the NFS client change the share's fstab column so that the options field contains quota, for example:

nfs-server:/mnt/storage        /home   nfs             rsize=32768,wsize=8192,soft,timeo=10,rw,intr,nosuid,noexec,nodev,quota          0 0

Remount the filesystem and you should be able to interface with the quotas on the remote NFS server. Be sure to use the -r flag when modifying quotas from the client(s).

Mass Virtual Hosting Part Two: Easy SFTP Chroot Jail

Plain FTP suffers from a number of problems, foremost is transmission in cleartext. Unless SSL is used (I have come across very few occurrences of its use in the wild) usernames and passwords, as well as the files being transmitted, are sent over the wire unencrypted. This means anyone with a well-placed sniffer or man-in-the-middle setup between the client and server can intercept or mutilate the data without much skill or resource cost. FTP also suffers from NAT issues, in that some users must forward ports or use passive transfer mode - words most of the users who will be using your hosting service have probably never heard, don't want to hear and have no idea how to implement.

Enter SFTP: file transfer over Secure Shell. Not to be confused with FTPS, or "FTP Secure" (FTP with SSL), SFTP shares nothing in common with FTP other than its purpose: moving files around the network. SFTP uses a single, wholly encrypted tunnel from the client to the server to send and receive files. This avoids the problems associated with NAT and FTP's dual-connection implementation, provides a relatively secure means of authentication and prevents third-party manipulation of the data in transit. SSH also warns users when a server's RSA keys change so diligent users can identify and avoid potential man-in-the-middle attacks. SFTP is not, however, a perfect solution. OpenSSH, the implementation which we will be dealing with in this article, can be found on most Linux and BSD servers. Due to its ubiquity it is a prime target for 0-day exploits on one hand, and well hardened against known exploits on the other (you do keep your software patched and up-to-date, right?).

SSH, like any service that uses a username-password authentication scheme - including FTP, is vulnerable to brute-force or dictionary cracking attacks. This problem can be virtually eliminated by disabling user-pass authentication altogether and using on shared keys (please my article Passwordless or Single Password SSH with Key Exchange for instructions on implementing this configuration) however this is probably not an appropriate solution for your public hosting project as the process is difficult for regular users to implement, particularly if they are using a Windows client such as FileZilla. One solution which I highly recommend is fail2ban, it will read your sftpd logs and temporarily block an IP address associated with a specified number of failed attempts over a given period of time. I have provided simple instructions for implementing fail2ban for SSH in my article Stifling Brute Force Attacks with fail2ban. Because we are dealing with a situation where users are prone to forget their password you may wish to use fairly loose criteria in configuring fail2ban, enough failures should be tolerated that a forgetful user won't be quickly blocked but an automated attack should be picked up. You might also wish to block access to sftpd rather than all ports as affected users might understand sftp disappearing after several failed attempts but could think the server is down if they are unable to access their website (assuming it is hosted from the same server).

The "chroot jail" concept is as old as the hills and has provided a way for us to separate vulnerable services from the filesystem-at-large by faking them into thinking a certain directory is the absolute root (/) of the filesystem. In this article we're going to apply this concept to regular users using OpenSSH's built-in functionality available since version 4.9. Before then it was a royal pain to implement whereas most popular FTP daemons had supported the feature out-of-the-box for years, now SFTP can finally be considered a complete and suitable replacement.

One probably does not wish to restrict SSH access for ALL user accounts, since one probably needs remote administrative access to the machine. Therefore one shall create a group to which users who must be chroot jailed will be added. For the purposes of this article we shall call the group hosted but you may call it anything:

# groupadd hosted

Next add users to the group, if you want to make sure these users do not get regular shell access (the ability to log in and run commands etc) be sure to specify a dummy shell, such as /bin/false or /sbin/nologin (Gentoo):

# useradd -G hosted -s /bin/false demo-user
or
# usermod -G hosted -s /bin/false demo-user

Don't forget to give your test account a password if it is a new account.

# passwd demo-user

Now open your sshd config file (most users: /etc/ssh/sshd_config) and go to the very bottom. If there is a Subsystem sftp line already delete it. Add the following:

Subsystem       sftp    internal-sftp
Match Group hosted
        ChrootDirectory %h
        ForceCommand internal-sftp
	AllowTcpForwarding no

The Match Group directive tells sftpd to use the proceeding settings for any user in the group hosted. Now you must change the owner (and optionally group) of the user's home directory to root:

# chown root: /home/demo-user

Due to these permissions the user will not be able to create new files at the top level of their directory tree, which to you looks like /home/demo-user and to them /. That's fine because we're going to give them a directory for the site they want to host with appropriate ownership, then they can do whatever they want in that. I typically set users up thus:

# mkdir ~demo-user/demo-site.com
# mkdir ~demo-user/demo-site.com/htdocs
# mkdir ~demo-user/demo-site.com/log
# mkdir ~demo-user/demo-site.com/cgi-bin
# chown demo-user: ~demo-user/demo-site.com/ -R

Restart your sshd (/etc/init.d/ssh(d) restart) and try logging in as your jailed user via sftp. If it works, congratulations. If you chose to restrict regular SSH access you may need to include /bin/false (or /sbin/nologin etc) to your /etc/shells valid shells list or the user may not be able to log in at all. Try logging in via SSH to ensure access has been blocked.