The Wandering Geek

Killing Processes by Name on Linux or Unix Systems

2009-10-02T04:04:00Z

Finding and killing a process on a Unix or Linux system is typically done by sending it a signal using the kill command, specifying the process ID (PID), which we can grab using ps and grep (use ps -ef on Solaris):


dmaxwell@kaylee:~$ ps ax | grep gedit
11604 ?        Sl     0:01 gedit
11609 pts/5    S+     0:00 grep gedit
dmaxwell@kaylee:~$ kill 11604

But it’s sometimes convenient to want to kill a running process by name, or kill a group of running processes with the same name. The most portable way to do this is with the pkill command - this is present on most Linux, Solaris and BSD systems. The simplest way to use pkill is just to specify the process name:


pkill gedit

This sends a TERM signal to any process whose name matches ‘gedit’, terminating it. If you have a long-running command and can only remember part of the command string, no problem - use -f with pkill:


dmaxwell@kaylee:~$ ps ax | grep name
11902 pts/5    S      5:23 find . -name foo*
11906 pts/5    S+     0:00 grep name
dmaxwell@kaylee:~$ pkill -f name

This would kill the find process (and the grep if it were still running), since part of its full command string contained the substring ‘name’. Using pkill in this way will by default gracefully terminate processes, but for stubborn processes that refuse to die, you can specify a different signal. Here we specify a KILL signal, which immediately ends a process.


pkill -KILL name

You can use numeric signals in place of the signal name, for example ‘-9′ is the KILL signal in the last example.

Killing a group of processes is just as easy. Sometimes this is necessary when system shutdown scripts fail, perhaps due to a missing lockfile. Here we kill all the Apache processes running on our server after the shutdown command fails:


root@kaylee:~# /etc/init.d/apache2 stop
 * Stopping web server apache2                                                                                                                                                                          [ OK ] 
root@kaylee:~# ps ax | grep apache
13124 ?        Ss     0:00 /usr/sbin/apache2 -k start
13129 ?        S      0:00 /usr/sbin/apache2 -k start
13130 ?        S      0:00 /usr/sbin/apache2 -k start
13131 ?        S      0:00 /usr/sbin/apache2 -k start
13132 ?        S      0:00 /usr/sbin/apache2 -k start
13133 ?        S      0:00 /usr/sbin/apache2 -k start
13162 pts/8    S+     0:00 grep apache
root@kaylee:~# pkill apache
root@kaylee:~# ps ax | grep apache
13165 pts/8    S+     0:00 grep apache
root@kaylee:~#

Using pkill -f start would also work here, since each of the Apache command lines contains the substring ’start’. The pkill command has many more options, but one other that might be useful is -u, which will allow you to specify a username or ID. In this example we send a TERM signal to all the processes owned by the user ‘nobody’:


root@kaylee:~# pkill -u nobody

There is a sister command to pkill, pgrep, that takes most of the same options but rather than sending a signal to one or a group of processes, it just displays the process IDs. This can be fed as standard input into other commands. Here is an example: dmaxwell@kaylee:~$ pgrep -d, apache2 14507,14512,14513,14514,14515,14516 dmaxwell@kaylee:~$ ps fvp $(pgrep -d, apache2) PID TTY STAT TIME .. RSS %MEM COMMAND 14507 ? Ss 0:00 .. 11076 0.3 /usr/sbin/apache2 -k start 14512 ? S 0:00 .. 6032 0.2 \_ /usr/sbin/apache2 -k start 14513 ? S 0:00 .. 6028 0.2 \_ /usr/sbin/apache2 -k start 14514 ? S 0:00 .. 6028 0.2 \_ /usr/sbin/apache2 -k start 14515 ? S 0:00 .. 6028 0.2 \_ /usr/sbin/apache2 -k start 14516 ? S 0:00 .. 6028 0.2 \_ /usr/sbin/apache2 -k start

This is particularly useful, since it preserves the header line output by ps, as opposed to something like ps avx | grep apache, which displays the data, but not the column headers. Both pkill and pgrep are documented in the same manual page, so search for either in the FreeBSD or Debian man pages for more info.

Quick Log File Processing with Perl

2009-09-15T01:20:00Z

A common thing to want to do as a sysadmin is match and print text from a file in a particular output format. There are lots of ways to do this using shell tools - grep, sed and awk are used frequently - but I’d like to show you a common Perl idiom for doing this type of task.

Perl was originally designed to be a replacement for the various shell tools, and while it has grown into much more over the years, it is still a great tool to have in your command line toolbox. Here’s an example. Let’s say you want to print the date, time, IP address and URL each time your website is crawled by a Googlebot. The Apache access log will look something like this:


...
10.249.66.234 - - [12/Sep/2009:19:22:51 -0400] "GET /robots.txt HTTP/1.1" 404 424 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
10.249.66.234 - - [12/Sep/2009:19:22:51 -0400] "GET / HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
...

A quick solution is this, all in one line:


serenity:~# perl -wnle 'print "Googlebot accessed \"$4\" from $1 on $2 at $3" if (/^ (\d+\.\d+\.\d+\.\d+) .+? \[ (.+?) : (.+?) \s .+? GET\s+(.+?)\s+HTTP .+ Googlebot/x)' /var/log/apache2/access.log
Googlebot accessed "/robots.txt" from 10.249.66.234 on 12/Sep/2009 at 19:22:51
Googlebot accessed "/" from 10.249.66.234 on 12/Sep/2009 at 19:22:51
serenity:~#

There are four command line options used here:

w: Turn on warnings
n: Loop through the supplied file one line at a time
l: Print a newline after each line of output
e: Execute the Perl code that follows

See the perlrun manpage for details, there is much more to Perl’s command line processing.

I build the regular expression by picking a target line and going through it from left to right, adding expressions as I go. I make use of the /x modifier so that it is easier to read - this makes Perl ignore whitespace in the regexp. I also use Perl’s non-greedy quantifier quite a bit, this is the question mark in expressions like .+? \[. This little snippet matches one or more of any character, followed by a left-bracket. The question mark ensures that the first such left-bracket is matched. Normally Perl’s regexp engine would happily chomp away at characters and match the last left bracket it found in the line. Using the greedy form .+ \[ would work for us, since there is only one such left bracket in each line, but it turns out to be a performance improvement if we are parsing large text files (For more info, I encourage you to read Mastering Regular Expressions by Jeffrey Friedl, or start with the Regular Expression Tutorial).

This method has a few advantages. For one, it relies on just one tool, not a few disparate ones. Perl is portable to many operating systems, so you could use this to parse text files on Windows, for example. You also have the ability to load modules on the command line with the ‘-M’ switch. This gives you access to all of CPAN, potentially a huge time-saver.

Troubleshooting SSH Connections

2009-09-03T02:17:00Z

I’ve helped a few people recently who have had trouble getting OpenSSH working properly; I’ve also had my share of issues over the years. Generally problems with SSH connections fall into two groups - network related and server related. Most of these problems can be fixed fairly quickly if you know what to look for.

Network Related

These will typically be caused by improper routing or firewall configurations. Here are some things to check.

1. If your SSH server sits behind a firewall or router, make sure the default route of your internal SSH server points back to that firewall or router. Seems obvious, but it’s common to forget about the return trip packets need to make. This will display your default gateway:


netstat -rn | grep '^0'

Sometimes the default gateway is just one of your server interfaces, this is OK as long as that interface is directly connected to something that knows how to get back to your client.

2. While you’re at it, make sure the incoming SSH packets are actually getting to your SSH server. Tcpdump works very nicely for this, you’ll need to be root to run it on the server:


tcpdump -n -i eth0 tcp port 22 and host [IP address of client]

Just replace eth0 by your client-facing interface name. If you don’t see incoming SSH packets during connection attempts, it’s probably due to a firewall or router access list.

SSH Server Problems

All of these issues revolve around SSH server configuration settings - not misconfigurations necessarily, just settings you may not be aware of.

1. Permissions can be a problem - in its default configuration, OpenSSH sets StrictModes to yes and won’t allow any connections if the account you’re trying to SSH into has group- or world-writable permissions on its home directory, ~/.ssh directory, or ~/.ssh/authorized_keys file. I typically just make the two directories mode 700 and the authorized_keys file mode 600. The sshd man page suggests this one-liner:


chmod go-w ~/ ~/.ssh ~/.ssh/authorized_keys

2. On Debian or Ubuntu systems, it is possible the keys you are using to connect are blacklisted. This is only an issue on Debian or Debian-based clients, and stems from this now-famous vulnerability in May of 2008. To detect any such blacklisted keys, run ssh-vulnkey on the client, while logged into the account you are connecting from. Debian and Ubuntu SSH servers will reject any such keys unless the PermitBlacklistedKeys directive in the /etc/ssh/sshd_config file is set to no. I don’t recommend you actually leave this security check disabled, but it can be useful to temporarily disable it during testing.

3. Finally, if all else fails, you can see exactly what the SSH server is doing by running it in debug mode on a non-standard port:


/usr/sbin/sshd -d -p 2222

Then, on the client, connect and watch the server output:


ssh -vv -p 2222 [Server IP]

Note the -vv option to provide verbose client output. This alone can sometimes help debug connection issues.

Move Over, Grep. Hello, Ack

2009-08-23T23:20:00Z

As someone who has been using grep and its variants like egrep for years, I admit they have been insanely useful. But every once in a while something comes along that improves an idea so much, you can’t ignore it. Such a thing is Ack, the grep replacement.

I do a lot of software development in large codebases, and the ability to find snippets of text is paramount. Tags can be used and integrated with Emacs (or Vim, we’re not all perfect), which is great for function names, but not useful for general text searches. Using grep in a code repository is a pain, and usually means some sort of hack to ignore VC directories like .svn and RCS. Enter ack - similar to grep but with some more thought behind it. It ignores VC meta-data directories by default and is written in pure Perl - so it’s portable and supports the full Perl regexp syntax. Having a pure-Perl version available with no dependencies also means its easy to install in shared hosting environments, where you don’t have root access.

Install ack by just downloading the standalone version and put it in your command path, use CPAN (cpan App::Ack), or install a pre-packaged binary (On Debian/Ubuntu systems, the package name is ack-grep). Ack output is very readable, with highlighted matches by default as well as line numbers and file names. Here is an example:


dmaxwell@kaylee:~/tmp$ ack-grep -ai 'limit_as.+?\&rlimit' emacs-22.3
emacs-22.3/src/vm-limit.c
76:  getrlimit (RLIMIT_AS, &rlimit);

Here is a screenshot so you can see the highlighting and colorization:

The -ai means ’search all, case insensitively’, and tells Ack to search all filetypes (but still not including common VCS directories or files), while ignoring case. Ack searches are recursive by default, so there is no need for a -r switch. You can see we used Perl’s non-greedy match quantifier in the search regexp, something egrep can’t do. This speeds the search up considerably.

There is much more to ack, read the docs and give it a try. I hope you’ll find it as useful as I have.

Five Minutes to an Even More Secure SSH

2009-08-16T12:42:00Z

One of the most popular articles on this blog was Five-Minutes to a More Secure SSH. My impetus for writing it was seeing too many client’s servers left in a default state where they are vulnerable to brute-force attacks. In it, I basically advocate three things:

Disabling password authentication
Disabling root login
Enabling key-based authentication

Three years later, those recommendations still hold true and I would encourage you to follow them. However, OpenSSH has many features and there is more you can do to secure your SSH servers, without resorting to external software.

Important Notes: The main OpenSSH server configuration file is called sshd_config and will typically be in the /etc/ssh or /etc/sshd directories. Like all of the configuration files used by OpenSSH, it is in plain text and so can be edited with any text editor. After editing your sshd_config file, you will need to reload your SSH server’s configuration - restarting the SSH daemon is not necessary. The command typically looks like this (this is on Debian or Ubuntu): /etc/init.d/ssh reload or (on Red Hat/Fedora): service sshd reload

Also be careful not to lock yourself out of your SSH server when experimenting with these access controls. It’s a good idea to always have two SSH sessions into the server, and to always make backup of the relevant configuration files. If you log out of one session and get denied access, you still have one active session to fix things.

Restricting Users and Hosts

OpenSSH allows you to restrict users and groups by host or IP address. There are four different directives you can use in your sshd_config file (they are evaluated in this order):


DenyUsers
AllowUsers
DenyGroups
AllowGroups

The format for all of them will be the same - a space-separated list of users or group names, with optional host names. Here is an example:


AllowUsers vader@10.0.0.1 maul@sproing.evillittleman.net sidious tyranus@*.evillitleman.net
AllowGroups wheel staff

This tells sshd to only allow connections from the user vader and only from the IP address 10.0.0.1. The user maul is also allowed, but only from the host sproing.evillittleman.net. User sidious is allowed from anywhere, and the user tyranus is also allowed, from any host in the evillittleman.net domain (the asterisk matches zero or more characters).

The AllowGroups line allows login only from users whose primary group name or supplementary group list match one of ‘wheel’ or ’staff’.

Keep in mind that using AllowUsers or AllowGroups means that anyone not matching one of the supplied patterns will be denied access by default. Also, in order for sshd to allow access based on full or partial hostnames, it needs to do a DNS lookup on the incoming IP address. That means the connecting IP address must have a PTR (reverse) entry that maps back to a real hostname. These aren’t hard to get if you have a static IP address, usually your ISP or server hosting provider can do this for you on request. If your server is internal, you probably have your own DNS server and can add appropriate PTR entries yourself.

In addition to the asterisk in hostname or group patterns, you can use a question-mark to mean exactly one character, and an exclamation point to negate the sense of a match:

* - Matches zero or more characters
? - Matches exactly one character
! - Negates the host pattern match

Note: In my tests, using ! to negate the sense of the hostname match did not work with the AllowUsers directive. It only seems to work when used with authorized_keys file restrictions (see below).

Restricting Access and Commands

SSH has the concept of authorized keys. If you are using key-based auth, like I suggested in my first article, the user accounts on the SSH server will have an authorized_keys file (which is by default in the ~/.ssh directory of whatever user account you are logging into). This file lists the public keys, one per line, that are authorized for access to that account. Apart from just specifying which public keys are allowed access, there are a some more options that you can use to further restrict SSH sessions. Here are the most useful ones:

from=”hostname1,hostname2,…” - Restricts access from the specified IP or hostname patterns
command=”command” - Runs the specified command after authentication
no-pty - Does not allocate a pty (does not allow interactive login)
no-port-forwarding - Does not allow port forwarding

Here is an example showing part of an authorized_keys file:


from="deathstar.example.com,!jedi.example.com,10.0.0.?" ssh-rsa AAAAB5...2BQ== vader@evillittleman.net
from="pitofdespair.example.com",command="ls",no-pty,no-port-forwarding ssh-dss AAAAZ7...22Q== droidQBX12@evillittleman.net

The first line allows login with the specified RSA key from deathstar.example.com, from any host with IP address in 10.0.0.[0-9], but not from the host jedi.example.com. The second line merely runs the ‘ls’ command whenever the specified DSA key is used - it does not allow any other commands to be run, does not allow interactive login, and does not allow port-forwarding. It also restricts the source of that key to the host pitofdespair.example.com.

Running sshd on a Non-Standard Port

Admittedly this is an attempt at ’security through obscurity’, but that doesn’t mean it’s not useful when combined with other security measures. You may not be able to restrict access by hostname or IP, for example - you may always be sourcing your connections from a dynamic IP address, or you may not be able to get a proper PTR record created. It’s also very easy to do. In your sshd_config file, just change Port=22 to Port=nnnnn (where nnnnn is some high port), then reload the sshd configuration. How do we pick a port number? Some are better than others. First, assume that most port scans are being done with Nmap, and take a look at the nmap-services file. This is a list of ports that Nmap will use by default if you don’t specify a port range on the command line. It’s probably a fair bet that most script-kiddies are using nmap is this manner. Just pick a high port not on this list, most nmap scans won’t notice it. You can also use multiple Port= directives, meaning you can have sshd listen on multiple ports. Connecting to an alternate port is also very easy, use the following options depending on the command used:


ssh -p 65502 vader@deathstar.example.com
sftp -oPort=65502 vader@deathstar.example.com
scp -P 65502 deathstar_plans.doc vader@deathstar.example.com:

You can also edit your ~/.ssh/config file, and add the Port= directive to one of your host blocks:


...
Host    evil
        Hostname deathstar.example.com
        User vader
        Port 65502
...

Then just connecting with the command ssh evil will connect with the specified user and port.

Hashing Known Hosts Files

When you connect to an SSH server, the ssh client stores the server’s hostname, IP address and host key in a file named known_hosts. It will by default be in your ~/.ssh directory. Having the IP addresses of the servers you connect to regularly in plaintext can be a security risk if you are on a shared host, or your client gets compromised (stolen laptop, for example). An easy way to avoid this problem is to obscure the information in the known_hosts file by hashing it. Hashing your known_hosts file is easy, you just use the ssh-keygen command, giving it the file path.


ssh-keygen -H -f ~/.ssh/known_hosts

While this hashes all existing host keys, any host keys that get added to your known_hosts file after you hash it do not get hashed by default. To make it the default, add the directive HashKnownHosts to your ~/.ssh/config file. Here is an example of hashing a known_hosts file. First, here is what the file looks like beforehand:


dmaxwell@kaylee:~/.ssh$ head known_hosts
10.100.6.151 ssh-rsa AAAAB4NzaC1yc2EAAAABIwAAAIEAuVgRdptT3xsQoGkiNnJb4Zb02p07MaZX02MFs5JhoqmvV5X5Z/LEQH0S7ngSn3b8kQUnocGulJgLchwfThrd/1OkdyOKdpgXxH/rmDXfwh/MZBNBxnMWBa1HpXSc1gxyDfSSxo+VPa1NCP+ob0dWx4sI+JFJ5cVzbQng4rKp3x8=
10.100.6.162 ssh-rsa AAAAB4NzaC1yc2EAAAABIwAAAIEAxpQuMJR4Dq/MmrpUryYlNbP+BIWgJlr0LAfaHTIU64Ho6F58Bb1QzlUeeHQSI9f6qFW9aPsBC3Gd5wgQBUj3byinXXHC/10c3vmb2aEujmyL6en2Pef4AN8bKgaRtJq2G/H4MkPWBzxqZPb/k9c3a26P/DjG4y01TMw9vCld+As=
...

Here we run the ssh-keygen command:


dmaxwell@kaylee:~/.ssh$ ssh-keygen -H -f ~/.ssh/known_hosts
/home/dmaxwell/.ssh/known_hosts updated.
Original contents retained as /home/dmaxwell/.ssh/known_hosts.old
WARNING: /home/dmaxwell/.ssh/known_hosts.old contains unhashed entries
Delete this file to ensure privacy of hostnames

And here is what the file looks like afterward (Note that we deleted the backup file when we were done):


dmaxwell@kaylee:~/.ssh$ head known_hosts
|1|PdThGCuhg23t9bcURxyitJTmfKk=|/z+Xvh4xPuDni8PTB5iK7KKnGdA= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAuVgRdptT3xsQoGkiNnJb4Zb02p07MaZX02MFs5JhoqmvV5X5Z/LEQH0S7ngSn3b8kQUnocGulJgLchwfThrd/1OkdyOKdpgXxH/rmDXfwh/MZBNBxnMWBa1HpXSc1gxyDfSSxo+VPa1NCP+ob0dWx4sI+JFJ5cVzbQng4rKp3x8=
|1|vkLZ22nl30gyJ3gIX74FUF7b3eg=|uy5oSZ8avgZQZE+dwMd/mXGoA38= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAxpQuMJR4Dq/MmrpUryYlNbP+BIWgJlr0LAfaHTIU64Ho6F58Bb1QzlUeeHQSI9f6qFW9aPsBC3Gd5wgQBUj3byinXXHC/10c3vmb2aEujmyL6en2Pef4AN8bKgaRtJq2G/H4MkPWBzxqZPb/k9c3a26P/DjG4y01TMw9vCld+As=
...
dmaxwell@kaylee:~/.ssh$ rm known_hosts.old

Donate!

OpenSSH is an amazing tool, one most system and network admins couldn’t live without. I encourage you to donate to the OpenSSH project.

More Information

The Forgotten Power of Unix Text Utilities

2009-08-10T19:33:00Z

I’m the first to extol the virtues of scripting languages like Python and Perl in particular. But they aren’t always the best tool for the job. It’s often forgotten how powerful the original Unix (and now GNU) text processing utilities are. Recently on linuxquestions.org, someone was asking how to combine specific columns from multiple CSV files into a new CSV file. They had the start of a Perl solution that was not working correctly, and wanted advice on it. My advice was to go with a one-line shell solution which is simply this:


paste -d, <(cut -d, -f3 file1.csv) <(cut -d, -f3 file2.csv) > output.csv

This will combine the third column from each specified file into a new file. It relies on a feature of the more modern Bourne shells, process substitution - the two parts that look like <(…). Here it is in action:


dmaxwell@kaylee:~$ cat foo1.txt
a1,a2,a3
b1,b2,b3
dmaxwell@kaylee:~$ cat foo2.txt
A1,A2,A3
B1,B2,B3
dmaxwell@kaylee:~$ paste -d, <(cut -d, -f3 foo1.txt) <(cut -d, -f3 foo2.txt) 
a3,A3
b3,B3

You can paste columns from as many files as you need here. One catch, of course, is that this only works with simple CSV data - meaning there are no embedded commas in the data fields themselves. But this is much more understandable than any lengthy scripting language solution.

One other tip, if you had to get rid of the first row, which might contain column header data, just pipe the output through tail:


paste -d, <(cut -d, -f8 file1.csv) <(cut -d, -f8 file2.csv) | tail -n +2 > output.csv

Colophon

2009-08-04T15:39:00Z

A note on the new blog format. I've opted for simplicity and changed from Wordpress to PyBlosxom. Wordpress is great, but I found my normal workflow (pretty much all shell and Emacs) conflicted with the web interface. This took a bit more effort to setup, but the ease of use is worth it. You might be interested in the plugins I'm using:

tags
pyarchives
pycalendar
conditionalhttp
readmore

I also wrote a couple of shell scripts, one to publish the posts from a staging directory, and another to save or restore the post mtimes, so I can edit old entries and still preserve the original blog order. I converted old site from its dynamic form to a static one, using wget and Perl's wonderful regular expression engine. That served two purposes - it meant that the pages are now served much more quickly while the original URLs are preserved for the search engines.

Happy Sysadmin Day!

2009-07-31T16:34:00Z

Yup, it’s that time of year again, when we celebrate the back-room geeks that keep the world’s computers and networks afloat. Please remember your sysadmin today. Here are the classic web-guy-vs-sales-dude videos for your enjoyment, parts one and two:

Introduction to the Command Line

2009-07-30T22:22:00Z

There is a manual available from the FSF for those wishing to learn how to use the command line and associated tools. It’s quite good. You can get it online at Flossmanuals, or support the FSF and buy a printed copy. At about 165 pages, it covers all the Bash shell basics, and has sections on the various text utilities, scripting, SSH, text editors and the indispensable GNU screen. It also has a nice command reference as an appendix. Here is an outline of the book content.

Comments on “10 Things for Linux Desktop Evangelists to Ponder”

2009-07-19T11:01:00Z

Technewsworld has an opinion piece listing 10 things needed to bring desktop Linux closer to reality. Here is a snippet:

8. Convince the killer-apps owners to create real and usable ports of their products.
…
7. Find a sponsor willing to step up to real publicity for Linux.
…
5. Pay for Linux!
…
1. Lose the attitude! Lose the edge! Stop whining already!

I’ve said it before, and I’ll say it again (and again), it’s all about the OEMs. None of this stuff matters to anyone but us geeks. People use Windows on the desktop because of the lock Microsoft has on the OEM market. It’s not about the apps, or the OS, or the Free Software. Generally people will use whatever comes with whatever they buy. That’s why Google’s announcement of a new OS was so important - they were very public about the OEM agreements they have in place to put their OS on hardware that consumers will buy. They’re giving plenty of warning to the app developers to get ready, in this case to web-enable their apps.

Monitoring and Alerting on Linux Logfiles

2009-07-17T06:17:00Z

As a sysadmin, I’ve found it’s always useful to monitor system logs on your Linux or Unix servers for specific patterns of activity, things that can indicate security or system issues. Even nicer to get alerts when activity occurs. Some time ago I wanted a simple solution that would allow me to continuously monitor the ClamAV updater (freshclam) logfiles and send email alerts - the result was this script. Recently I wanted something a bit more general, so I wrote this Perl script that monitors any logfile for a specific pattern and generates email or syslog alerts.

Installing Logmon

It needs a few non-core Perl modules to run, namely Mail::Mailer, Proc::Daemon, Unix::Syslog and File::Tail, but these can be installed pretty easily as packaged modules or via CPAN. On Debian/Ubuntu systems, all the needed modules are pre-packaged for you:


apt-get install libmailtools-perl libunix-syslog-perl libfile-tail-perl libproc-daemon-perl

On red Hat/Fedora servers, you can use yum:


yum install perl-MailTools perl-Unix-Syslog perl-File-Tail perl-Proc-Daemon

To pull in all the modules from CPAN, you can use this one-liner:


for m in Mail::Mailer Proc::Daemon Unix::Syslog File::Tail; do perl -MCPAN -e "install $m"; done

Once the modules are installed, download the script and double check that it will run without error, then copy it to your path and make it executable. You should see no errors about missing modules when you run the script under ‘perl -cwT’.


perl -cwT ./logmon.pl
install -m 755 logmon.pl /usr/local/bin/

Running Logmon

It’s meant to be both simple and secure, here is the usage summary:


logmon.pl synopsis: Daemon that periodically checks logfile for a pattern and send alerts
Pattern is always required. If no other options are given, defaults to syslog alerts and monitors /var/log/messages for given pattern.
Usage: logmon.pl  -p pattern [-m alerts@example.com] [-f logfile] [-u run as user] [-g run as group] [-i max interval] [-v] [-d] [-h]
-m: Email destination for alerts
-f: logfile to monitor
-p: Pattern to match against lines in logfile (Perl regexp, match is case-insensitive)
-u: Run with permissions of user
-g: Run with permissions of group
-i: Max time to sleep between checks
-d: Debug output to STDOUT, do not daemonize
-v: Verbose logging (use with caution or you may have endless alerts)
-h: This help text

Running the script is pretty straightforward, you specify a pattern to match against with -p (this is the only required parameter), and optionally an email recipient (-m) and logfile to watch (-f). Here is an example. Let’s say you want to get alerts whenever MySQL detects a crashed table. The syslog event for this looks like this on my Ubuntu box:


Jul 17 08:02:49 kaylee mysqld[1532]: 090717  8:02:49 [ERROR] /usr/sbin/mysqld: Table './mysql/user' is marked as crashed and should be repaired

And here is the command line usage:


logmon.pl -p 'mysqld.+?table.+?crashed' -m you@example.com -u nobody -g adm -f /var/log/syslog -i 30

Note that the pattern match is case-insensitive. When run this way, Logmon will detach itself from your terminal and run as a daemon, checking /var/log/syslog every 30 seconds for the supplied pattern. I recommend you use the -u and -g options to force Logmon to drop it’s privileges, just make sure you specify a user or group that have read-permissions on the specified logfile. On Debian and Ubuntu servers, all the system logs are readable by the group ‘adm’.

Other Options and Tips for Testing Logmon

Logmon will dump alerts to your default system log if you leave off the -m option. If you also use the -v option for verbose logging, these syslog entries and alerts will have pattern and match data. If you end up monitoring the same file you are dumping alerts into you’ll get an endless series of alerts continuously being added to the system log. For this reason when alerts are sent to syslog, by default they are very generic (email alerts are always verbose).

To test Logmon, use the -v and -d options together. Run this way, Logmon will not daemonize itself, and will just print alerts and activity to your console (STDERR).

Errors

Any errors that cause the script to die while it’s running as a daemon can be found in your system log.

Startup and Shutdown

I haven’t yet written any startup scripts for Logmon, although I plan to. For now, just start it from one of your system’s startup scripts, and if you have to stop it you can just use pkill logmon. Please send any bugs or suggestions to the email address in the script header, or leave a comment here.

Process Monitoring on Linux Servers

2009-07-17T03:10:00Z

I’m updating some of the older articles on this blog, making sure the links work, updating the referenced software with newer versions and generally re-testing everything to make sure it still works on the latest crop of distros. Since I’m on the topic of processes, I updated Monitoring Unix System Processes with Psmon. Psmon is a very useful tool for monitoring running processes - every sysadmin should have it in their toolbox. I encourage you to take a look.