Quickly jump to any server with UNIX/Linux SSH jumpbox scripts

- select the contributor at the end of the page -
If you are a UNIX sysadmin for any number of servers, you need to build yourself a Linux secure shell (SSH) jumpbox. Do it now! Having a centralized location that you can use to quickly “jump” to any box saves a whole bunch of time. Not only that, it opens opportunities for speeding up repetitive chores, and even automating tasks.

Here I'll describe some of the things that I have built into my SSH Jumpbox, including some scripts you can put to work for yourself right away.

SSH Keys

It all starts with SSH Keys.

SSH can use a set of private/public keys to authenticate your session. Your private key will reside on your Jumpbox, and your public key will be copied out to each of the target servers you are going to manage.

Instead of using your UNIX password on those servers (which may not all be the same), you will use a single passphrase that goes with your private key to do the authentication. You can even leverage the SSH Authentication Agent so that you only have to type that passphrase a couple of times through the course of your day.

To generate the key, you simply need to use the “ssh-keygen” command.

[me@jumpbox ~]$ ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/home/me/.ssh/id_rsa): {ACCEPT DEFAULT}

Created directory '/home/me/.ssh'.

Enter passphrase (empty for no passphrase): {YOUR PASSPHRASE}

Enter same passphrase again: {YOUR PASSPHRASE}

Your identification has been saved in /home/me/.ssh/id_rsa.

Your public key has been saved in /home/me/.ssh/id_rsa.pub.

The key fingerprint is:

3d:84:37:43:59:05:ca:e6:2d:59:de:90:36:06:17:24 me@jumpbox

Let's have a closer look at what this created:

[me@jumpbox ~]$ ls -l ${HOME}/.ssh

total 8

-rw-------. 1 me me 1743 Jun 26 14:14 id_rsa

-rw-r--r--. 1 me me 390 Jun 26 14:14 id_rsa.pub

In your home directory, you will now have a “.ssh” directory. In that directory you will have two files. “id_rsa” is the private key that will stay on the jumpbox. “id_rsa.pub” is the public key you will be copying to your target servers.

It is now time to upload the private key to your target server. The easiest way to do that is using “sftp”.

[me@jumpbox ~]$ sftp me@target

Connecting to target...

The authenticity of host target (x.x.x.x)' can't be established.

RSA key fingerprint is 40:4b:f7:f3:a1:da:dc:59:9c:19:d5:e3:21:f8:a9:ea.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added target,x.x.x.x (RSA) to the list of known hosts.

me@target's password: {YOUR PASSWORD}

sftp> mkdir .ssh

sftp> chmod 700 .ssh

Changing mode on /home/me/.ssh

sftp> put .ssh/id_rsa.pub .ssh/authorized_keys

Uploading .ssh/id_rsa.pub to /home/me/.ssh/authorized_keys

sftp> chmod 600 .ssh/authorized_keys

Changing mode on /home/me/.ssh/authorized_keys

sftp> quit

So, stepping through the commands above:

If this is the first time you are connecting to this host, SFTP will ask you if you trust this host.

Next, you will be required to provide your UNIX password to get authenticated. Hopefully, this is the last time you will need to type that password.

You will now create the .ssh directory (mkdir .ssh) and lock down the permissions (chmod 700 .ssh) on that directory so that only this user can see files in that directory.

You will then upload the public key discussed before, and store it as an authorized_keys file (put .ssh/id_rsa.pub .ssh/authorized_keys). You will lock it down so that only this user can see and alter this file (chmod 600 .ssh/authorized_keys).

WARNING:

If you already have an authorized_keys file, then append your new key to the existing authorized_keys file rather than overwriting it. SFTP the file up to your server with a different name. Then log in to a shell to add the contents of your new file to the original file.

That should be all you need to do. Now, test your setup.

[me@jumpbox ~]$ ssh me@target

Enter passphrase for key '/home/me/.ssh/id_rsa': {YOUR PASSPHRASE}

[me@target ~]$ exit

[me@jumpbox ~]$

If it didn't work, the troubleshooting section at the end of this article might be helpful to you.

SSH Configuration Options

There are a few common SSH client options that you may want to consider.

EscapeChar @

SSH provides an “escape” mechanism to allow you to interact with the SSH client from an active session. The most common use of this would be to kill the current session if for some reason your shell is hung.

The default escape character is a tilde (~). One problem: that's the default escape character for “tip” on Solaris, and for some Lights Out Management Tools. If you're running multiple tools, you won't know which tool is going to catch the escape character first.

The solution is to change your SSH escape character to something unique. I like to use “@”. No other tool that I am aware of uses that. Now, if your session is hung, simply press “@.” and, you will drop your connection. You can also press “@?” to see what other options are available to you.

ForwardX11 yes

You will need to forward X11 traffic to be able to open graphic windows from the target server. There will be more on this topic later.

StrictHostKeyChecking no

The SSH client maintains a cache of host public keys. When you connect to new hosts, you will be regularly prompted to add entries to the known_hosts file. Disable this option and SSH will automatically cache these entries without asking you first.

All these options are available from the command line using the -o option.

Because you will be using them over and over again, store them in the SSH client config file.

You will want to store them in ${HOME}/.ssh/config.

You will also need to change the permission on the file so only you can access it:chmod 600 ${HOME}/.ssh/config.

SSH Authentication Agent (ssh-agent)

The next useful feature is the SSH Authentication Agent. The idea here is to load up your SSH Key (i.e. enter your passphrase just once), and then use that key to establish new sessions.

Here is a sample session:

[me@jumpbox ~]$ ssh-agent

SSH_AUTH_SOCK=/tmp/ssh-jVTDW14623/agent.14623; export SSH_AUTH_SOCK;

SSH_AGENT_PID=14624; export SSH_AGENT_PID;

echo Agent pid 14624;

[me@jumpbox ~] SSH_AUTH_SOCK=/tmp/ssh-jVTDW14623/agent.14623

[me@jumpbox ~] export SSH_AUTH_SOCK

[me@jumpbox ~] SSH_AGENT_PID=14624

[me@jumpbox ~] export SSH_AGENT_PID

[me@jumpbox ~]$ ssh-add –t 14400

Enter passphrase for /home/me/.ssh/id_rsa: {YOUR PASSPHRASE}

Identity added: /home/me/.ssh/id_rsa (/home/me/.ssh/id_rsa)

Lifetime set to 14400 seconds

[me@jumpbox ~]$ ssh me@target

Last login: Thu Jun 27 12:58:16 2013 from jumpbox

[me@target ~]$ exit

[me@jumpbox ~]

Let's step through the transaction.

The “ssh-agent” command starts up the agent. It reports to the user where it's running by giving us some environment variables to load into the shell. Set those variables as detailed by the agent output.

Next, load up our current private key and its passphrase. It will be stored in the agent. The -t option sets the “time-to-live” for the key. My preference is a four-hour time-to-live (4 hours * 60 Mins/Hour * 60 Secs/Min = 14,440 Secs). This means you are prompted for a key when you start your day, and again when you come back from lunch.

Start up a session that uses that key. Notice it did not prompt for a passphrase. It retrieved that information from the agent.

The ssh2host Script

Here is a script that ties all of this together. On the first run, it starts up the SSH Authentication Agent, loads your private key into memory, and then starts an SSH session to the client. On subsequent runs, it connects to the existing agent and the keys it has loaded, so you won't be prompted for your password again.

But here's the really neat thing. It creates hard links to this script with names that match your target hosts (and makes sure those files are in your system path). You can now SSH to these systems by simply typing their name.

#!/bin/bash ############################################## # This script should be hard-linked with the name # of the target server. It will try to ssh to # a client with that name. (ln ssh2host target) ############################################## # Where to store and read information about current agent. SSH_AGENT_INFO=${HOME}/.ssh_agent_info [ -f ${SSH_AGENT_INFO} ] && . ${SSH_AGENT_INFO} > /dev/null # The SSH_AGENT_PID should have the PID for the current agent. # And there should be only one ssh-agent running with that PID. ALL_GOOD=YES # Assume all is good to start. if [ ! -n "${SSH_AGENT_PID}" ] # The variable is not set. then ALL_GOOD=NO else # The ssh-agent isn't running. if [ `ps -ef | grep "ssh-agent" | grep "${SSH_AGENT_PID}" |\ grep -v grep | wc -l` -ne 1 ] then ALL_GOOD=NO fi fi # If we had problems - start the ssh-agent up. [ "${ALL_GOOD}" = "NO" ] && /usr/bin/ssh-agent > ${SSH_AGENT_INFO} && \ . ${SSH_AGENT_INFO} > /dev/null # Is the key loaded? If not, load it now. /usr/bin/ssh-add -l > /dev/null [ $? -ne 0 ] && /usr/bin/ssh-add -t 14400 # Time to start the SSH session to the host.. /usr/bin/ssh -X -e @ -o StrictHostKeyChecking=no `basename $0`

This is the script in action. My target server is called “target”.

[me@jumpbox ~]$ target

Enter passphrase for /home/me/.ssh/id_rsa: {ENTER PASSPHRASE}

Identity added: /home/me/.ssh/id_rsa (/home/me/.ssh/id_rsa)

Lifetime set to 14400 seconds

Last login: Thu Jun 27 16:36:13 2013 from target

[me@target ~]$ exit

logout

Connection to target closed.

[me@jumpbox ~]$ target

Last login: Thu Jun 27 16:36:44 2013 from target

[me@target ~]$ exit

logout

Connection to target closed.

Troubleshooting:

Stale Host Keys

Take a look at the files in your .ssh directory. You'll notice that you now have a known_hosts file. This is where the target host keys are stored.

[me@jumpbox ~]$ ls -l ${HOME}/.ssh

total 12

-rw-------. 1 me me 1743 Jun 26 15:17 id_rsa

-rw-r--r--. 1 me me 390 Jun 26 15:17 id_rsa.pub

-rw-r--r--. 1 me me 790 Jun 26 15:27 known_hosts

If you change the IP on a server, or if you re-install the OS on a server, you will very likely end up with discrepancies in your known_hosts file, and the following threating error message will be presented to you during login. If you're sure there is nothing fishy going on, simply edit the known_hosts file and remove the offending line as identified in the error message.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that the RSA host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

40:4b:f7:f3:a1:da:dc:59:9c:19:d5:e3:21:f8:a9:ea.

Please contact your system administrator.

Add correct host key in /home/me/.ssh/known_hosts to get rid of this message.

Offending key in /home/me/.ssh/known_hosts:2

Password authentication is disabled to avoid man-in-the-middle attacks.

Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.

X11 forwarding is disabled to avoid man-in-the-middle attacks.

DNS Issues

If after executing the SSH command, the connection is established, but it takes a minute or so to actually get anywhere, look for DNS issues.

Ensure both your Jumpbox and your target servers are properly set up (both forward and reverse) on your DNS server. If this is not possible, then fake out DNS by putting IP's into /etc/hosts on both the Jumpbox and the target server.

You can also look at disabling DNS on the SSH Daemon on the target server. (Look for the “UseDNS” entry in /etc/ssh/sshd_config file on your target server).

Host Side Firewalls

IP Tables are enabled on most Linux systems. You'll need to open up the SSH port on the target server to ensure that SSH traffic can get into your target server.

/sbin/iptables -A INPUT -p tcp --dport 22 -j ACCEPT

Another common method to restrict access to your target servers is TCP Wrappers. If that is enabled on your target host, you should add an entry to your hosts.allow file for the SSH protocol.

echo “sshd: ALL” >> /etc/hosts.allow

File Permissions

SSH likes its settings files locked down. File and directory modes should all be set so that no “group” or “other” permissions are granted.

chmod –R go= ${HOME}/.ssh

You may also want to look at the SSH Daemon on your target server and disable the “StrictModes” directive in /etc/ssh/sshd_config file.

Root Logins

Direct root logins are discouraged as a general practice for security reasons. However, that's one practice that I usually ignore so that I can do other interesting things.

If you're looking to log in as root directly, look for the “PermitRoot” directive in the /etc/ssh/sshd_config file on your target server. I recommend the “without-password” setting. This will allow you to log in using a private/public key pair, and still prevent direct root access with username/password.

Ready to test your skills in Unix? See how they stack up with this assessment from Smarterer. Start this Unix test now

Get our content first. In your inbox.

Contributor

Nuno Paixao

is a Systems and Storage Analyst from Saint John, NB, Canada. He has a Bachelor of Science in Computing Science from Dalhousie University. He has worked in the IT industry for the last 20 years and his areas of expertise include Unix (Linux and Solaris), storage (EMC and IBM), backups (Veritas Netbackup), clustering (Veritas Storage Foundation/VCS), and virtualization (VMware). For more info, visit his website and follow him on Twitter @nunopaixao.