pitter patter on the keyboard

You have two options for monitoring things like load average and logged in users on a remote host with Nagios or Icinga: SNMP (which you are probably already using for Cacti or a similar graphing/monitoring solution) or the Nagios Remote Plugin Executor (NRPE). NRPE gives one greater flexibility in the kind of data collected and actions executed. Unfortunately, NRPE is not included in Portage so we must compile and configure it on our own.

First, download the NRPE source tarball to your core monitoring server and install check_nrpe:

# mkdir /usr/src/nrpe
# cd /usr/src/nrpe
# wget [tarball]
# tar xf [tarball]
# cd nrpe-[version]
# ./configure
# make all
# cp src/check_nrpe /usr/[lib|lib64]/nagios/plugins/

Now add the command to your Icinga or Nagios config:

define command{
        command_name check_nrpe
        command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

Compile and install the nrpe server on the target host:

# mkdir /usr/src/nrpe
# cd /usr/src/nrpe
# wget [tarball]
# tar xf [tarball]
# cd nrpe-[version]
# ./configure
# make all
# mkdir /etc/nrpe
# cp src/nrpe /usr/bin/
# cp sample-config/nrpe.cfg /etc/nrpe/

Install the nagios-plugins package from portage:

# emerge nagios-plugins

Edit /etc/nrpe/nrpe.cfg with a mind to security (particularly the allowed_hosts directive). The nagios-plugins package has already created the nagios user and group so there is no need to change the defaults. Update the command paths at the end of the file to reflect the location nagios-plugins installed to:

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_root]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/root
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

Start the server as root with the daemoniz flag:

# nrpe -c /etc/nrpe/nrpe.cfg -d

We can see it has dropped down to the nagios user:

# ps aux | grep nrpe
nagios     570  0.0  0.0   4208   948 ?        Ss   16:55   0:00 nrpe -c /etc/nrpe/nrpe.cfg -d

Now we can test the configuration on the Nagios/Icinga core monitoring server:

# /usr/lib64/nagios/plugins/check_nrpe -H [ADDRESS]
NRPE v2.13

If the remote NRPE server’s configuration is working it should respond with NRPE [version]. Now we can set up some services on the monitoring server:

define service{
        host_name               myhost
        service_description     Users
        check_command           check_nrpe!check_users
        max_check_attempts      5
        check_interval          5
        retry_interval          1
        check_period            24x7
        notification_interval   30
        notification_period     24x7
        notification_options    w,c,r
        contact_groups          admins
        }

Note that the argument for check_command is one of the hard-coded commands in the nrpe server’s configuration file. Restart Icinga/Nagios to load the changes and begin monitoring:

# /etc/init.d/icinga restart

An init script will be required to make the NRPE daemon start on boot, create /etc/init.d/nrpe:

#!/sbin/runscript
# Copyright (c) 2012 http://foxpa.ws
# All rights released

description="Runs Nagios Remote Plugin Executor on Gentoo"

depend()
{
        need net
}

start()
{
        ebegin "Starting NRPE"
        start-stop-daemon --start --quiet --user=root --background --exec "/usr/bin/nrpe" -- -c /etc/nrpe/nrpe.cfg -d
        eend ${?}
}

stop()
{
        ebegin "Stopping NRPE"
        start-stop-daemon --stop --quiet --pidfile "/var/run/nrpe.pid"
        eend ${?}
}

Now make it executable and add it to the default runlevel:

# chmod +x /etc/init.d/nrpe
# rc-update add nrpe default

Alternatively, NRPE can be run by xinetd. From the README:

Running Under INETD or XINETD
-----------------------------

If you plan on running nrpe under inetd or xinetd and making use
of TCP wrappers, you need to do the following things:



1) Add a line to your /etc/services file as follows (modify the port
   number as you see fit)

        nrpe            5666/tcp        # NRPE



2) Add entries for the NRPE daemon to either your inetd or xinetd
   configuration files.  Which one your use will depend on which
   superserver is installed on your system.  Both methods are described
   below.  NOTE: If you run nrpe under inetd or xinetd, the server_port
   and allowed_hosts variables in the nrpe configuration file are
   ignored.


   ***** INETD *****
   If your system uses the inetd superserver WITH tcpwrappers, add an
   entry to /etc/inetd.conf as follows:

        nrpe    stream  tcp     nowait  <user> /usr/sbin/tcpd <nrpebin> -c <nrpecfg> --inetd

   If your system uses the inetd superserver WITHOUT tcpwrappers, add an
   entry to /etc/inetd.conf as follows:

        nrpe    stream  tcp     nowait  <user> <nrpebin> -c <nrpecfg> --inetd


   - Replace <user> with the name of the user that the nrpe server should run as.
        Example: nagios
   - Replace <nrpebin> with the path to the nrpe binary on your system.
        Example: /usr/local/nagios/nrpe
   - Replace <nrpecfg> with the path to the nrpe config file on your system.
        Example: /usr/local/nagios/nrpe.cfg


   ***** XINETD *****
   If your system uses xinetd instead of inetd, you'll probably
   want to create a file called 'nrpe' in your /etc/xinetd.d
   directory that contains the following entries:


        # default: on
        # description: NRPE
        service nrpe
        {
                flags           = REUSE
                socket_type     = stream        
                wait            = no
                user            = <user>
                server          = <nrpebin>
                server_args     = -c <nrpecfg> --inetd
                log_on_failure  += USERID
                disable         = no
                only_from       = <ipaddress1> <ipaddress2> ...
        }


   - Replace <user> with the name of the user that the nrpe server should run as.
   - Replace <nrpebin> with the path to the nrpe binary on your system.
   - Replace <nrpecfg> with the path to the nrpe config file on your system.
   - Replace the <ipaddress> fields with the IP addresses of hosts which
     are allowed to connect to the NRPE daemon.  This only works if xinetd was
     compiled with support for tcpwrappers.



3) Restart inetd or xinetd will the following command (pick the
   on that is appropriate for your system:

        /etc/rc.d/init.d/inet restart

        /etc/rc.d/init.d/xinetd restart

   OpenBSD users can use the following command to restart inetd:

        kill -HUP `cat /var/run/inet.pid`



4) Add entries to your /etc/hosts.allow and /etc/hosts.deny
   file to enable TCP wrapper protection for the nrpe service.
   This is optional, although highly recommended.

NOTE Please see Rotate Red5 Logs Without logrotate for a better way to configure logging.

This script assumes you have installed Red5 to /opt/red5, added a non-privileged user named Red5 and applied the appropriate ownership/permissions to the red5/log, red5/webapps and red5/work directories. It is compatible with OpenRC.

#!/sbin/runscript
# Copyright (c) 2011 http://foxpa.ws
# All rights released

description="Runs Red5 streaming media server on Gentoo"

depend()
{
        need net
}

start()
{
        ebegin "Starting Red5"
        start-stop-daemon --start --quiet --user=red5 --background --chdir=/opt/red5/ --stdout /var/log/red5/red5.log --stderr /var/log/red5/red5_errors.log --exec "/opt/red5/red5.sh"
        eend ${?}
}

stop()
{
        ebegin "Stopping Red5"
        cd /opt/red5/
        start-stop-daemon --stop --quiet --exec "/opt/red5/red5.sh"
        ./red5-shutdown.sh &> /dev/null
        eend ${?}
}

Plain FTP suffers from a number of problems, foremost is transmission in cleartext. Unless SSL is used (I have come across very few occurrences of its use in the wild) usernames and passwords, as well as the files being transmitted, are sent over the wire unencrypted. This means anyone with a well-placed sniffer or man-in-the-middle setup between the client and server can intercept or mutilate the data without much skill or resource cost. FTP also suffers from NAT issues, in that some users must forward ports or use passive transfer mode – words most of the users who will be using your hosting service have probably never heard, don’t want to hear and have no idea how to implement.

Enter SFTP: file transfer over Secure Shell. Not to be confused with FTPS, or “FTP Secure” (FTP with SSL), SFTP shares nothing in common with FTP other than its purpose: moving files around the network. SFTP uses a single, wholly encrypted tunnel from the client to the server to send and receive files. This avoids the problems associated with NAT and FTP’s dual-connection implementation, provides a relatively secure means of authentication and prevents third-party manipulation of the data in transit. SSH also warns users when a server’s RSA keys change so diligent users can identify and avoid potential man-in-the-middle attacks. SFTP is not, however, a perfect solution. OpenSSH, the implementation which we will be dealing with in this article, can be found on most Linux and BSD servers. Due to its ubiquity it is a prime target for 0-day exploits on one hand, and well hardened against known exploits on the other (you do keep your software patched and up-to-date, right?).

SSH, like any service that uses a username-password authentication scheme – including FTP, is vulnerable to brute-force or dictionary cracking attacks. This problem can be virtually eliminated by disabling user-pass authentication altogether and using on shared keys (please my article Passwordless or Single Password SSH with Key Exchange for instructions on implementing this configuration) however this is probably not an appropriate solution for your public hosting project as the process is difficult for regular users to implement, particularly if they are using a Windows client such as FileZilla. One solution which I highly recommend is fail2ban, it will read your sftpd logs and temporarily block an IP address associated with a specified number of failed attempts over a given period of time. I have provided simple instructions for implementing fail2ban for SSH in my article Stifling Brute Force Attacks with fail2ban. Because we are dealing with a situation where users are prone to forget their password you may wish to use fairly loose criteria in configuring fail2ban, enough failures should be tolerated that a forgetful user won’t be quickly blocked but an automated attack should be picked up. You might also wish to block access to sftpd rather than all ports as affected users might understand sftp disappearing after several failed attempts but could think the server is down if they are unable to access their website (assuming it is hosted from the same server).

The “chroot jail” concept is as old as the hills and has provided a way for us to separate vulnerable services from the filesystem-at-large by faking them into thinking a certain directory is the absolute root (/) of the filesystem. In this article we’re going to apply this concept to regular users using OpenSSH’s built-in functionality available since version 4.9. Before then it was a royal pain to implement whereas most popular FTP daemons had supported the feature out-of-the-box for years, now SFTP can finally be considered a complete and suitable replacement.

One probably does not wish to restrict SSH access for ALL user accounts, since one probably needs remote administrative access to the machine. Therefore one shall create a group to which users who must be chroot jailed will be added. For the purposes of this article we shall call the group hosted but you may call it anything:

# groupadd hosted

Next add users to the group, if you want to make sure these users do not get regular shell access (the ability to log in and run commands etc) be sure to specify a dummy shell, such as /bin/false or /sbin/nologin (Gentoo):

# useradd -G hosted -s /bin/false demo-user
or
# usermod -G hosted -s /bin/false demo-user

Don’t forget to give your test account a password if it is a new account.

# passwd demo-user

Now open your sshd config file (most users: /etc/ssh/sshd_config) and go to the very bottom. If there is a Subsystem sftp line already delete it. Add the following:

Subsystem       sftp    internal-sftp

Match Group hosted
        ChrootDirectory %h
        ForceCommand internal-sftp
	AllowTcpForwarding no

The Match Group directive tells sftpd to use the proceeding settings for any user in the group hosted. Now you must change the owner (and optionally group) of the user’s home directory to root:

# chown root: /home/demo-user

Due to these permissions the user will not be able to create new files at the top level of their directory tree, which to you looks like /home/demo-user and to them /. That’s fine because we’re going to give them a directory for the site they want to host with appropriate ownership, then they can do whatever they want in that. I typically set users up thus:

# mkdir ~demo-user/demo-site.com
# mkdir ~demo-user/demo-site.com/htdocs
# mkdir ~demo-user/demo-site.com/log
# mkdir ~demo-user/demo-site.com/cgi-bin
# chown demo-user: ~demo-user/demo-site.com/ -R

Restart your sshd (/etc/init.d/ssh(d) restart) and try logging in as your jailed user via sftp. If it works, congratulations. If you chose to restrict regular SSH access you may need to include /bin/false (or /sbin/nologin etc) to your /etc/shells valid shells list or the user may not be able to log in at all. Try logging in via SSH to ensure access has been blocked.