=^.^=

Documentary for Dinner: Vice Fringe: Brainwashed by the Westboro Baptist Church (2012)

karma

Vice Fringe's Brainwashed by the Westboro Baptist Church interviews ex-members and the documentarian-turned-westboroer Stephen Drain.

Apache Reload: Fix seg fault or similar nasty error detected in the parent process

karma

I noticed this nasty wee bugger last week:

[notice] seg fault or similar nasty error detected in the parent process

The first time it happened the drop in traffic did not correlate with the daily init scripts but when the second occurance this morning did I finally had somewhere to start. The default logrotate script for apache on gentoo (and many other flavours) does a graceful reload once the log files have been rotated to release their inodes. Now I could reproduce the issue:

# /etc/init.d/apache2 reload
 * Gracefully restarting apache2 ...
# /etc/init.d/apache2 restart
 * Stopping apache2 ...
httpd (pid 2505?) not running                                                [ ok ]
 * Starting apache2 ...
 * start-stop-daemon: /usr/sbin/apache2 is already running
# /etc/init.d/apache2 stop   
 * Stopping apache2 ...
httpd (pid 2505?) not running
# /etc/init.d/apache2 start
 * Starting apache2 ...
 * start-stop-daemon: /usr/sbin/apache2 is already running

Poking around this seems to usually be caused by loading bad/wrong/nonexistent libraries in the php.ini - and I am guilty of carrying over old php.inis on upgrade. Updating and switching to the distributed php.ini for my installed version of php seems to have solved this issue.

Sometimes it really is my fault :p

Installing NRPE for Nagios/Icinga on Gentoo

karma

You have two options for monitoring things like load average and logged in users on a remote host with Nagios or Icinga: SNMP (which you are probably already using for Cacti or a similar graphing/monitoring solution) or the Nagios Remote Plugin Executor (NRPE). NRPE gives one greater flexibility in the kind of data collected and actions executed. Unfortunately, NRPE is not included in Portage so we must compile and configure it on our own.

UPDATE It is now possible to emerge NRPE via net-analyzer/nrpe

First, download the NRPE source tarball to your core monitoring server and install check_nrpe:

# mkdir /usr/src/nrpe
# cd /usr/src/nrpe
# wget [tarball]
# tar xf [tarball]
# cd nrpe-[version]
# ./configure
# make all
# cp src/check_nrpe /usr/[lib|lib64]/nagios/plugins/

Now add the command to your Icinga or Nagios config:

define command{
        command_name check_nrpe
        command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

Compile and install the nrpe server on the target host:

# mkdir /usr/src/nrpe
# cd /usr/src/nrpe
# wget [tarball]
# tar xf [tarball]
# cd nrpe-[version]
# ./configure
# make all
# mkdir /etc/nrpe
# cp src/nrpe /usr/bin/
# cp sample-config/nrpe.cfg /etc/nrpe/

Install the nagios-plugins package from portage:

# emerge nagios-plugins

Edit /etc/nrpe/nrpe.cfg with a mind to security (particularly the allowed_hosts directive). The nagios-plugins package has already created the nagios user and group so there is no need to change the defaults. Update the command paths at the end of the file to reflect the location nagios-plugins installed to:

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_root]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/root
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

Start the server as root with the daemoniz flag:

# nrpe -c /etc/nrpe/nrpe.cfg -d

We can see it has dropped down to the nagios user:

# ps aux | grep nrpe
nagios     570  0.0  0.0   4208   948 ?        Ss   16:55   0:00 nrpe -c /etc/nrpe/nrpe.cfg -d

Now we can test the configuration on the Nagios/Icinga core monitoring server:

# /usr/lib64/nagios/plugins/check_nrpe -H [ADDRESS]
NRPE v2.13

If the remote NRPE server's configuration is working it should respond with NRPE [version]. Now we can set up some services on the monitoring server:

define service{
        host_name               myhost
        service_description     Users
        check_command           check_nrpe!check_users
        max_check_attempts      5
        check_interval          5
        retry_interval          1
        check_period            24x7
        notification_interval   30
        notification_period     24x7
        notification_options    w,c,r
        contact_groups          admins
        }

Note that the argument for check_command is one of the hard-coded commands in the nrpe server's configuration file. Restart Icinga/Nagios to load the changes and begin monitoring:

# /etc/init.d/icinga restart

An init script will be required to make the NRPE daemon start on boot, create /etc/init.d/nrpe:

#!/sbin/runscript
# Copyright (c) 2012 http://foxpa.ws
# All rights released

description="Runs Nagios Remote Plugin Executor on Gentoo"

depend()
{
        need net
}

start()
{
        ebegin "Starting NRPE"
        start-stop-daemon --start --quiet --user=root --background --exec "/usr/bin/nrpe" -- -c /etc/nrpe/nrpe.cfg -d
        eend ${?}
}

stop()
{
        ebegin "Stopping NRPE"
        start-stop-daemon --stop --quiet --pidfile "/var/run/nrpe.pid"
        eend ${?}
}

Now make it executable and add it to the default runlevel:

# chmod +x /etc/init.d/nrpe
# rc-update add nrpe default

Alternatively, NRPE can be run by xinetd. From the README:

Running Under INETD or XINETD
-----------------------------

If you plan on running nrpe under inetd or xinetd and making use
of TCP wrappers, you need to do the following things:



1) Add a line to your /etc/services file as follows (modify the port
   number as you see fit)

        nrpe            5666/tcp        # NRPE



2) Add entries for the NRPE daemon to either your inetd or xinetd
   configuration files.  Which one your use will depend on which
   superserver is installed on your system.  Both methods are described
   below.  NOTE: If you run nrpe under inetd or xinetd, the server_port
   and allowed_hosts variables in the nrpe configuration file are
   ignored.


   ***** INETD *****
   If your system uses the inetd superserver WITH tcpwrappers, add an
   entry to /etc/inetd.conf as follows:

        nrpe    stream  tcp     nowait  <user> /usr/sbin/tcpd <nrpebin> -c <nrpecfg> --inetd

   If your system uses the inetd superserver WITHOUT tcpwrappers, add an
   entry to /etc/inetd.conf as follows:

        nrpe    stream  tcp     nowait  <user> <nrpebin> -c <nrpecfg> --inetd


   - Replace <user> with the name of the user that the nrpe server should run as.
        Example: nagios
   - Replace <nrpebin> with the path to the nrpe binary on your system.
        Example: /usr/local/nagios/nrpe
   - Replace <nrpecfg> with the path to the nrpe config file on your system.
        Example: /usr/local/nagios/nrpe.cfg


   ***** XINETD *****
   If your system uses xinetd instead of inetd, you'll probably
   want to create a file called 'nrpe' in your /etc/xinetd.d
   directory that contains the following entries:


        # default: on
        # description: NRPE
        service nrpe
        {
                flags           = REUSE
                socket_type     = stream        
                wait            = no
                user            = <user>
                server          = <nrpebin>
                server_args     = -c <nrpecfg> --inetd
                log_on_failure  += USERID
                disable         = no
                only_from       = <ipaddress1> <ipaddress2> ...
        }


   - Replace <user> with the name of the user that the nrpe server should run as.
   - Replace <nrpebin> with the path to the nrpe binary on your system.
   - Replace <nrpecfg> with the path to the nrpe config file on your system.
   - Replace the <ipaddress> fields with the IP addresses of hosts which
     are allowed to connect to the NRPE daemon.  This only works if xinetd was
     compiled with support for tcpwrappers.



3) Restart inetd or xinetd will the following command (pick the
   on that is appropriate for your system:

        /etc/rc.d/init.d/inet restart

        /etc/rc.d/init.d/xinetd restart

   OpenBSD users can use the following command to restart inetd:

        kill -HUP `cat /var/run/inet.pid`



4) Add entries to your /etc/hosts.allow and /etc/hosts.deny
   file to enable TCP wrapper protection for the nrpe service.
   This is optional, although highly recommended.

conftest segfault Emerging PHP 5.3.15

karma

Just had this happen on a 64 bit box after seeing it on a 32 bit virtual machine:

conftest[6750]: segfault at 1 ip 00007f1978ec2739 sp 00007fff4de65190 error 4 in libc-client.so.1.0.0[7f1978e84000+fb000]
conftest[18209]: segfault at 1 ip 00007f034a1de739 sp 00007fff522332a0 error 4 in libc-client.so.1.0.0[7f034a1a0000+fb000]
conftest[30093]: segfault at 1 ip 00007f6f163ea739 sp 00007fff6cd3a0c0 error 4 in libc-client.so.1.0.0[7f6f163ac000+fb000]
conftest[9687]: segfault at 1 ip 00007f9392238739 sp 00007fff4cc9a600 error 4 in libc-client.so.1.0.0[7f93921fa000+fb000]

Seems to occur when emerging PHP 5.3.15 at ./configure though I have seen no problems with the php binary itself.

Documentary for Dinner: Alternative Views - Guest John Stockwell (1988)

karma