The Wandering Geek

Updated: July 2009

Some time ago I found a useful tool for monitoring and restarting processes on Linux or Unix servers should they unexpectedly die. Psmon is a system monitoring script written in Perl and licensed under the Apache license that is quite useful if you run servers with critical processes on them. You can download the latest version from the psmon homepage (Version 1.39 as of this writing). Read on for tips on installing and configuring it.

Quick Start for the Impatient

Uncompress the tarball and run support/install.sh, then edit /etc/psmon.conf (it is well-commented) and skip below to the section on running psmon.

Detailed Installation

There are some Perl module requirements - you can use this Perl one-liner to check definitively that all the requirements are in place:

perl -e 'foreach ((Config::General,Proc::ProcessTable,Net::SMTP,Unix::Syslog,Getopt::Long)) { eval("use $_;"); print "Module $_ not present\n" if $@;}'

Net::SMTP and Getopt::Long are core Perl modules now, so they will already be present in any modern Linux or BSD. If you don’t have all the required modules, you can install them several ways. You can run an install script supplied with psmon - the installer is a shell script install.sh, located in the support directory of the tarball. When you run it, you’ll need root privileges as it will try to install any missing modules from CPAN. If the install script doesn’t work, or fails to pull in the required modules, you can install the modules manually like this (again, you’ll need root privileges):

for m in Config::General Proc::ProcessTable Net::SMTP Unix::Syslog Getopt::Long; do perl -MCPAN -e "install $m"; done

Remember that you probably won’t need to install Net::SMTP or Getopt::Long. On Debian or Ubuntu, you can just install these packages:

apt-get install libconfig-general-perl libproc-processtable-perl libunix-syslog-perl

On recent Red Hat/Fedora and clones, there may not be a CPAN module, so you will need to install it first, or grab the RPMs of the various missing modules:

yum install perl-Config-General perl-Proc-ProcessTable perl-Unix-Syslog

If you haven’t run the install.sh script, you can then install psmon as a standard Perl module, like so:

tar xvzf psmon-1.39.tar.gz cd psmon-1.39 perl Makefile.PL make sudo make install

Given that it’s pure Perl, it should run on any Unix, although I’ve only run it on Debian, Ubuntu and Red Hat Linux servers.

Configuration

The psmon binary is installed in /usr/local/bin (standard Perl install) or /usr/bin (from using install.sh). Regardless of where it is installed, psmon is configured through the file /etc/psmon.conf. You’ll have to change both lines in the config file that read Disabled True to Disabled False before psmon will function. Once that is done, configuration is pretty simple, especially if you want psmon to just monitor processes and restart them if they die. There are a lot of other things you can do, like monitor CPU usage of a given process and restart it if it is above a certain percentage. See the /etc/psmon.conf that gets installed by default for lots of examples, or the online docs. Here is a simple config file snippet that tells psmon to monitor sshd:

<Process sshd> LogLevel LOG_CRITICAL SpawnCmd /etc/init.d/ssh restart PidFile /var/run/sshd.pid </Process>

The paths in this example are specific to Debian or Ubuntu, here is an example crafted for Red Hat’s Fedora:

<Process sshd> LogLevel LOG_CRITICAL SpawnCmd /etc/init.d/sshd restart PidFile /var/run/sshd.pid </Process>

Modify the paths as appropriate for your distribution.

Running psmon and Getting Alerts

Once configured, you can start psmon:

root@kaylee:~# /usr/local/bin/psmon --daemon Use of uninitialized value in sprintf at /usr/local/bin/psmon line 1447. psmon[29858]: Forking background daemon, process 29859. root@kaylee:~# psmon[29859]: Forking second background daemon, process 29860. root@kaylee:~# ps -fp $(pgrep -d, psmon) UID PID PPID C STIME TTY TIME CMD root 29860 1 0 22:15 pts/5 00:00:00 /usr/bin/perl -w /usr/local/bin/psmon --daemon

You can then run psmon periodically via a cron entry:

*/5 * * * * /usr/local/bin/psmon --daemon --cron

The above will run psmon every five minutes. It will actually spawn a daemon the first time it runs, thereafter it will do nothing if it detects that it is already running, or respawn itself if needed. The ‘–cron’ switch disables ‘already running’ warnings. You can configure psmon to send email every time it has to restart a process with the AdminEmail directive:

AdminEmail doug@example.com

This will use Net::SMTP to send mail via localhost by default - use the ‘SMTPHost’ directive to configure a mail relay if needed:

SMTPHost mail.yourisp.net

Now with psmon running, we can run a simple test. Stop ssh (/etc/init.d/ssh stop or service sshd stop). After 60 seconds (the default check interval), SSH should be restarted and you should get an email that looks like this:

From: "root@kaylee" To: “doug@example.com” Subject: [psmon/kaylee] Spawned ’sshd’ with ‘/etc/init.d/ssh restart’ Command executed: /etc/init.d/ssh restart Exit value: 0 Signal number: 0 Dumped core?: 0 * Restarting OpenBSD Secure Shell server sshd …done.

There will also be a message in your syslog:

Jul 15 22:20:10 kaylee psmon[29860]: Spawned 'sshd' with '/etc/init.d/ssh restart

That’s really all there is to using psmon for basic system monitoring. See the config file and docs that come with the tarball for details on other features.