Geek Pit

Mar 12th, 2006 by Doug

This article details using Dsh (the distributed/dancer’s shell) and SSH to remotely administer groups of machines. Overview Dsh is a tool that will allow you to use SSH to run commands or spawn login shells on multiple machines at once. When properly used with ssh-agent, dsh can run unattended commands on many different hosts at once. I assume you have a working su and sudo setup. Just in case, here is a tutorial on configuring and using sudo. Installing Dsh On a Debian GNU/Linux-based system, you can install dsh quite simply with apt-get install dsh. If you aren’t using a Debian system, or are on another Unix-based system, you can always install from the dsh sources using the instructions at the dsh home page. Configuring Dsh The main dsh configuration file is /etc/dsh/dsh.conf (or $HOME/.dsh/dsh.conf). Here is a working sample (note that this file is installed for you when you install the Debian dsh package):

#default configuration file for dsh.
#suppled as part of dancer's shell
verbose = 0
remoteshell = ssh
showmachinenames = 0
waitshell=0
#remoteshellopt=...
# default config file end.

Note: In the default config file supplied with Debian Sarge or Ubuntu Hoary, you will have to change the remoteshell option from rsh to ssh, and the waitshell option from 1 to 0. You can also just cut-n-paste the version listed above. Other Dsh Configuration Files There are several other files located in /etc/dsh. Here are their descriptions:

/etc/dsh/machines.list: A list of all hostnames or IP addresses, one per line. These are the remote servers you wish to run commands on.
/etc/dsh/group/all: This is really a symlink to ../machines.list. You can create this with the command cd /etc/dsh/group && ln -s ../machines.list all
/etc/dsh/group/servers: A chosen subset of all the workstation IP’s or hostnames in machines.list, one per line. We will actually use the word servers as part of our dsh commands. You can actually choose any name for this file, and can have multiple group files.

Dsh Command Line Switches The following are the most useful options to dsh:

Switch	Description
-a	All machines
-g servers	Use the group servers
-c	Use concurrent connections
-w	Wait for one machine to finish before moving onto next
-v	Verbose output
-M	Show machine name, useful with -c

Using Dsh - An Example Setting Up SSH and the SSH Authentication Agent In this example, we will use dsh to update and install any packages available from an apt-get update and apt-get dist-upgrade. Since we need root access on each remote host, we will have to create a public/private key pair on the machine we are launching dsh from, under the root account (using ssh-keygen -t dsa). Next, append the public key you just created (this will be in /root/.ssh/id_dsa.pub by default) to each remote host’s /root/.ssh/authorized_keys files. Make sure the remote authorized_keys files all have a permission mode of 0600. Once this is done, you should be able to run the remote commands on all the configured remote machines without being prompted for a passphrase. Here are the steps, in detail:

Login to your own workstation, use su to become root
If you don’t already have a public/private SSH keypair for the root user, run ssh-keygen -t dsa as root. Use the default locations/filenames, and be sure to use a secure passphrase
Append /root/.ssh/id_dsa.pub to the /root/.ssh/authorized_keys file of the other servers you wish to run commands on
Make sure the remote authorized_keys files have permissions of 0600 by running chmod 0600 /root/.ssh/authorized_keys on each remote box
Exit the root shell you entered with su, above. If you are running in console mode (versus X, which always runs under an ssh-agent in Debian and most other GNU/Linux distributions), type ssh-agent bash (or whatever your preferred shell is) at the shell prompt. This will start another shell under its own ssh-agent process that you can access from the newly spawned shell
Type sudo ssh-add, this will add your root private key to the local authentication agent
Verify that the identity is added with ssh-add -l

Running Dsh Now we are ready to actually run dsh. Test dsh access to the workstations you specified in one of your dsh config file groups sudo dsh -M -w -g servers — ‘w’. You may be prompted to accept the remote host key if this is the first time you are logging in to the remote host(s). We have used the -w switch to ensure we process one host at a time, so that we can answer y to the host key question at each machine in turn. You will not be prompted for a passphrase, since you have cached it in the your local SSH authentication agent. Assuming the access works (and you should have the output of the w command on all three hosts sent to your console or xterm), let’s try running apt-get update and apt-get dist-upgrade on each host. This time, you will not be prompted to accept each remote host key: sudo dsh -M -g servers -c — ‘apt-get update’ sudo dsh -M -g servers -c — ‘apt-get -y dist-upgrade’ You should see the output of the commands mixed together, as they are all running concurrently. The -M switch helps in this regard, since it prepends the machine name to each output line. When you are done, all machines in the host group serversshould be updated. Because the dist-upgrade may occasionally prompt for your input (despite the -y), or even abort with an error, you may wish to run it first with -u and -s, coupled with the dsh -w switch, to show which packages will be upgraded, while simulating what will happen if the apt-get command is actually run (sudo dsh -M -g servers -w — ‘apt-get -u -s dist-upgrade’. If you want to only download the packages to be updated, but not actually install them, use the -d switch to apt-get. Sample Dsh Session In this example, the /etc/dsh/group/servers file looks like this:

10.1.1.101
10.1.1.102
10.1.1.103

dmaxwell@lab0:~$ sudo ssh-add
Could not open a connection to your authentication agent.
dmaxwell@lab0:~$ ssh-agent bash
dmaxwell@lab0:~$ sudo ssh-add
Enter passphrase for /root/.ssh/id_dsa:
Identity added: /root/.ssh/id_dsa (/root/.ssh/id_dsa)
dmaxwell@lab0:~$ ssh-add -l
1024 dd:a9:8f:bd:e8:9e:7e:5d:cc:10:9f:e4:a4:c2:52:22 /root/.ssh/id_dsa (DSA)
dmaxwell@lab0:~$ sudo dsh -M -w -g lab — ‘w’
10.1.1.101:  12:58:51 up 239 days,  3:28,  1 user,  load average: 0.00, 0.00, 0.00
10.1.1.101: USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
10.1.1.101: dmaxwell  pts/0    lab0.turinglabs 03Mar05 16:11m  0.00s  0.00s  -bash
10.1.1.102:  12:59:09 up 324 days, 18:07,  1 user,  load average: 0.00, 0.00, 0.00
10.1.1.102: USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
10.1.1.102: dmaxwell  pts/0    lab0.turinglabs 15Mar05 16:12m  0.06s  0.06s  -bash
10.1.1.103:  12:57:06 up 234 days,  3:10,  1 user,  load average: 0.00, 0.00, 0.00
10.1.1.103: USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
10.1.1.103: dmaxwell  pts/1    lab0.turinglabs 27Apr05 15:47m  0.01s  0.01s  -bash
dmaxwell@lab0:~$ exit

Technorati Tags: SSH, Sysadmin, Linux, Unix, Howto

Posted in Uncategorized

3 Responses to “Remotely Administering Groups of Servers With Dsh and SSH”

on 01 Oct 2006 at 12:03 am1Sebas

Really nice and useful, I was using things like:

for f in 1 2 3 4 5 6 7 ; do
ssh root@192.168.204.5$f apt-get install libqt-perl
done

But this is much better.
on 05 Mar 2007 at 5:07 pm2Giantlaser

Consider Cluster SSH. Convenient, if you don’t mind using xterms. Konsole can also copy input in one tab to all other tabs in the same instance, but Cluster SSH is more flexible for this.
on 27 Apr 2007 at 9:40 am3Steven

Thanks for the walkthrough (setting up keys, machine.list, groups, etc.), it was really easy to set up! Dsh is going to be super-useful for me… I am trying to build and administer a simple computing cluster

Trackback URI | Comments RSS

Geek tech-tips, news and commentary

Remotely Administering Groups of Servers With Dsh and SSH

3 Responses to “Remotely Administering Groups of Servers With Dsh and SSH”

Leave a Reply

Posts

Recent Comments

Presentations

Howtos

Blogroll

Links

Meta