Documentary for Dinner: Vice Fringe: Brainwashed by the Westboro Baptist Church (2012)
Vice Fringe's Brainwashed by the Westboro Baptist Church interviews ex-members and the documentarian-turned-westboroer Stephen Drain.
Vice Fringe's Brainwashed by the Westboro Baptist Church interviews ex-members and the documentarian-turned-westboroer Stephen Drain.
I noticed this nasty wee bugger last week:
[notice] seg fault or similar nasty error detected in the parent process
The first time it happened the drop in traffic did not correlate with the daily init scripts but when the second occurance this morning did I finally had somewhere to start. The default logrotate script for apache on gentoo (and many other flavours) does a graceful reload once the log files have been rotated to release their inodes. Now I could reproduce the issue:
# /etc/init.d/apache2 reload * Gracefully restarting apache2 ... # /etc/init.d/apache2 restart * Stopping apache2 ... httpd (pid 2505?) not running [ ok ] * Starting apache2 ... * start-stop-daemon: /usr/sbin/apache2 is already running # /etc/init.d/apache2 stop * Stopping apache2 ... httpd (pid 2505?) not running # /etc/init.d/apache2 start * Starting apache2 ... * start-stop-daemon: /usr/sbin/apache2 is already running
Poking around this seems to usually be caused by loading bad/wrong/nonexistent libraries in the php.ini - and I am guilty of carrying over old php.inis on upgrade. Updating and switching to the distributed php.ini for my installed version of php seems to have solved this issue.
Sometimes it really is my fault :p
You have two options for monitoring things like load average and logged in users on a remote host with Nagios or Icinga: SNMP (which you are probably already using for Cacti or a similar graphing/monitoring solution) or the Nagios Remote Plugin Executor (NRPE). NRPE gives one greater flexibility in the kind of data collected and actions executed. Unfortunately, NRPE is not included in Portage so we must compile and configure it on our own.
UPDATE It is now possible to emerge NRPE via net-analyzer/nrpe
First, download the NRPE source tarball to your core monitoring server and install check_nrpe:
# mkdir /usr/src/nrpe # cd /usr/src/nrpe # wget [tarball] # tar xf [tarball] # cd nrpe-[version] # ./configure # make all # cp src/check_nrpe /usr/[lib|lib64]/nagios/plugins/
Now add the command to your Icinga or Nagios config:
define command{ command_name check_nrpe command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ }
Compile and install the nrpe server on the target host:
# mkdir /usr/src/nrpe # cd /usr/src/nrpe # wget [tarball] # tar xf [tarball] # cd nrpe-[version] # ./configure # make all # mkdir /etc/nrpe # cp src/nrpe /usr/bin/ # cp sample-config/nrpe.cfg /etc/nrpe/
Install the nagios-plugins package from portage:
# emerge nagios-plugins
Edit /etc/nrpe/nrpe.cfg with a mind to security (particularly the allowed_hosts directive). The nagios-plugins package has already created the nagios user and group so there is no need to change the defaults. Update the command paths at the end of the file to reflect the location nagios-plugins installed to:
command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10 command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20 command[check_root]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/root command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200
Start the server as root with the daemoniz flag:
# nrpe -c /etc/nrpe/nrpe.cfg -d
We can see it has dropped down to the nagios user:
# ps aux | grep nrpe nagios 570 0.0 0.0 4208 948 ? Ss 16:55 0:00 nrpe -c /etc/nrpe/nrpe.cfg -d
Now we can test the configuration on the Nagios/Icinga core monitoring server:
# /usr/lib64/nagios/plugins/check_nrpe -H [ADDRESS] NRPE v2.13
If the remote NRPE server's configuration is working it should respond with NRPE [version]. Now we can set up some services on the monitoring server:
define service{ host_name myhost service_description Users check_command check_nrpe!check_users max_check_attempts 5 check_interval 5 retry_interval 1 check_period 24x7 notification_interval 30 notification_period 24x7 notification_options w,c,r contact_groups admins }
Note that the argument for check_command is one of the hard-coded commands in the nrpe server's configuration file. Restart Icinga/Nagios to load the changes and begin monitoring:
# /etc/init.d/icinga restart
An init script will be required to make the NRPE daemon start on boot, create /etc/init.d/nrpe:
#!/sbin/runscript # Copyright (c) 2012 http://foxpa.ws # All rights released description="Runs Nagios Remote Plugin Executor on Gentoo" depend() { need net } start() { ebegin "Starting NRPE" start-stop-daemon --start --quiet --user=root --background --exec "/usr/bin/nrpe" -- -c /etc/nrpe/nrpe.cfg -d eend ${?} } stop() { ebegin "Stopping NRPE" start-stop-daemon --stop --quiet --pidfile "/var/run/nrpe.pid" eend ${?} }
Now make it executable and add it to the default runlevel:
# chmod +x /etc/init.d/nrpe # rc-update add nrpe default
Alternatively, NRPE can be run by xinetd. From the README:
Running Under INETD or XINETD ----------------------------- If you plan on running nrpe under inetd or xinetd and making use of TCP wrappers, you need to do the following things: 1) Add a line to your /etc/services file as follows (modify the port number as you see fit) nrpe 5666/tcp # NRPE 2) Add entries for the NRPE daemon to either your inetd or xinetd configuration files. Which one your use will depend on which superserver is installed on your system. Both methods are described below. NOTE: If you run nrpe under inetd or xinetd, the server_port and allowed_hosts variables in the nrpe configuration file are ignored. ***** INETD ***** If your system uses the inetd superserver WITH tcpwrappers, add an entry to /etc/inetd.conf as follows: nrpe stream tcp nowait <user> /usr/sbin/tcpd <nrpebin> -c <nrpecfg> --inetd If your system uses the inetd superserver WITHOUT tcpwrappers, add an entry to /etc/inetd.conf as follows: nrpe stream tcp nowait <user> <nrpebin> -c <nrpecfg> --inetd - Replace <user> with the name of the user that the nrpe server should run as. Example: nagios - Replace <nrpebin> with the path to the nrpe binary on your system. Example: /usr/local/nagios/nrpe - Replace <nrpecfg> with the path to the nrpe config file on your system. Example: /usr/local/nagios/nrpe.cfg ***** XINETD ***** If your system uses xinetd instead of inetd, you'll probably want to create a file called 'nrpe' in your /etc/xinetd.d directory that contains the following entries: # default: on # description: NRPE service nrpe { flags = REUSE socket_type = stream wait = no user = <user> server = <nrpebin> server_args = -c <nrpecfg> --inetd log_on_failure += USERID disable = no only_from = <ipaddress1> <ipaddress2> ... } - Replace <user> with the name of the user that the nrpe server should run as. - Replace <nrpebin> with the path to the nrpe binary on your system. - Replace <nrpecfg> with the path to the nrpe config file on your system. - Replace the <ipaddress> fields with the IP addresses of hosts which are allowed to connect to the NRPE daemon. This only works if xinetd was compiled with support for tcpwrappers. 3) Restart inetd or xinetd will the following command (pick the on that is appropriate for your system: /etc/rc.d/init.d/inet restart /etc/rc.d/init.d/xinetd restart OpenBSD users can use the following command to restart inetd: kill -HUP `cat /var/run/inet.pid` 4) Add entries to your /etc/hosts.allow and /etc/hosts.deny file to enable TCP wrapper protection for the nrpe service. This is optional, although highly recommended.
Just had this happen on a 64 bit box after seeing it on a 32 bit virtual machine:
conftest[6750]: segfault at 1 ip 00007f1978ec2739 sp 00007fff4de65190 error 4 in libc-client.so.1.0.0[7f1978e84000+fb000] conftest[18209]: segfault at 1 ip 00007f034a1de739 sp 00007fff522332a0 error 4 in libc-client.so.1.0.0[7f034a1a0000+fb000] conftest[30093]: segfault at 1 ip 00007f6f163ea739 sp 00007fff6cd3a0c0 error 4 in libc-client.so.1.0.0[7f6f163ac000+fb000] conftest[9687]: segfault at 1 ip 00007f9392238739 sp 00007fff4cc9a600 error 4 in libc-client.so.1.0.0[7f93921fa000+fb000]
Seems to occur when emerging PHP 5.3.15 at ./configure though I have seen no problems with the php binary itself.