SSH - using the shell remotely

If you came to Ubuntu from a Windows background, or are otherwise familiar with Windows (let's face it, not many of us are in a position where they never use Windows at all!), you'll probably be familiar with Windows Remote Desktop, which is often used as a troubleshooting tool. You can do the same kind of thing in Ubuntu, but because the Linux command line is so much more powerful and flexible than its Windows counterpart, it's often much more practical to use a command-line only tool for the same purpose. This is often a better solution because sending a constantly updated image of your desktop to somewhere else is inevitably going to use a lot more bandwidth than a few shell commands, and it's very likely you can resolve the issue from the shell. It can also be useful for connecting to servers, shell accounts, web hosts etc.

Telnet was the original method of connecting to a remote computer, however it has declined in popularity due to security issues. SSH has largely replaced it because it's more secure, and that's what I'm going to tell you about today.

SSH requires two components to work - a client and a server. Think of it like surfing the web - when you visit a web page, you use a client (the web browser) to connect to a server (the web server). Similarly, with SSH you use an SSH client to connect to an SSH server on the machine you want to connect to. The most common SSH client and server on Linux are maintained by the OpenSSH project. Ubuntu ships with the OpenSSH client by default, and if you have a machine you'd like to be able to connect to remotely via SSH, just install the openssh-server package in the usual way.

How do you use it? It's simple. Open a terminal and enter something using the following format:

ssh user@server

Let's go into this in a little more detail. First of all, user is just the user name you want to log in as on the remote server. For instance, if you've installed openssh-server on your desktop and your user account is called eric on that machine, then you should put eric as the username. If you want to connect to a shell account in the name of ebarnes, then it should be ebarnes.

The server can be either a domain name or an IP address. So if you have a shell account on a machine which has the domain name myfreeshellaccount.com and your user name is ebarnes, you'd enter ssh ebarnes@myfreeshellaccount.com. If you want to connect to a machine that doesn't have a domain name associated with it, then you'd need to enter the IP address to connect to. There's nothing stopping you connecting to another machine on your home network via this method, so if you have an old machine running Ubuntu Server, for instance, on your home network, and it's been assigned the IP address 192.168.1.4 by your router, you can just connect like this:

ssh ebarnes@192.168.1.4

If you'd like to be able to use SSH to connect to your home machine from outside your home network, that's a bit more demanding. You'd need to either obtain a static IP address from your ISP, which can be expensive, or use a service like DynDNS, and would also need to set up port forwarding on your router. However, there's all sorts of possibilites that offers, such as managing BitTorrent downloads remotely by using a command-line client like rTorrent, or administering your machine remotely over SSH.

Once you're connected, SSH works just like any other shell session - you can do exactly the same things with SSH as you would in the terminal. When you're finished, just type in exit or hit Ctrl-D as you would normally to close the shell and that will finish your session.

Note that you aren't limited to connecting to an SSH server from Linux or Unix hosts. Although most Linux or Unix-based operating systems ship with an SSH client, you can use PUTTY to connect from a Windows box (so, for instance, if you have a desktop running Ubuntu with an OpenSSH server installed, and you want to remote into it from your laptop running Windows 7, you can do that). There's even SSH clients for a number of smartphones, including the iPhone and Android, so you can remote into your machine from anywhere.

If you want to try it out, and you have an old desktop you don't use, it's quite interesting to install Ubuntu Server on it, connect it to your router and disconnect the monitor, managing it entirely via SSH. It can be a useful resource to have around, and offers a lot of possibilities, such as:

• File server
• uPnP server (allows you to stream content to many electronic devices, including the Sony PlayStation 3)
• BitTorrent box

By managing it entirely via SSH, you don't need the additional overhead of running a desktop, and can devote more system resources to whatever job you want it to do. You can use SSH to install whatever software you want via apt-get to customise it for the job you want it to do.

Alternatively, if you don't have an old machine spare or don't wish to try this, why not try it in a virtual machine? Virtualbox is good for this - just install Ubuntu Server in a VM, and under the network settings, attach it to a bridged adapter. This will mean the virtual machine gets allocated its own IP address by your router, and you can connect to it via SSH.

If you have a web hosting account, you may be able to connect to this via SSH - the host will provide the details if this is an option. Finally, another option is to get a shell account - essentially it's a user account on a remote server. I highly recommend devio.us, who are a free shell account provider running on OpenBSD.

Have fun with SSH! Next time I'll show you a few tricks to make your experience with it better!

Going further with environment variables

First of all, I apologise for how long it's taken to get round to this (merely days away from being a full year) but I've been busy! I was frantically busy at work for much of the last year, and have my own web development studies to deal with too, so I've only just gotten around to doing this post. Hopefully I should be able to maintain a better frequency of posts in future!

Last time I explained a bit about how environment variables work. Now we'll get to use them a little. For instance, say you want to set up a shortcut to a specific folder called mystuff to make it easy to get to. Well, you can easily set up an environment variable to hold the path to that folder. Just enter something like this:

export mystuff=/home/user/my/stuff

This will set mystuff up as a local variable that you can call at any time in the same way you'd refer to it normally, like this:

cd $mystuff

Note the dollar sign is only used when calling it, not when setting it.

If you want to set variables permanently, just add them to the .profile file in your home directory. Then, when you next start up a shell, it will automatically load these settings.

There's a number of different settings you can add to customise your bash shell. For instance, the other day I added this to my .profile so that Vim was set as my text editor when using Subversion:

export SVN_EDITOR=/usr/bin/vim

I also added the following to my shell account at Devio.us (who, incidentally are hands down the best shell account providers I have ever found, and I recommend) so I could use Vim in colour, even over SSH:

export TERM=xterm-256color

There's a huge range of settings you can change to customise your bash shell, so I won't attempt to cover them all. Instead, I'll refer you to the guide in the Ubuntu documentation. A few I will mention, however, are PAGER (typically /usr/bin/less), EDITOR (the default text editor, you can set it to /usr/bin/vim, /usr/bin/emacs, or /usr/bin/nano, or any other CLI editor you want), VISUAL (the visual editor, normally /usr/bin/gedit, although may be different on Kubuntu or Xubuntu), or BROWSER (pretty much what you'd expect).

Now, a related subject to environment variables is aliases. These are handy shortcuts you can write for commonly used commands, and can be a great timesaver. For instance, if you often connect to a specific host via ssh, you can write an alias for it. To set up an alias, just enter something like the following:

alias example="ssh user@server"

Now, just enter example, on its own, into the shell, and it will run the command. But, of course, this is only temporary. To make it permanent, set up a file in your home directory called .bash_aliases and add the alias you want to set up to it. Once you've done that, every time you start a new shell, all the aliases you've defined in the file will be available to you!

I hope this has given you some idea of how to customise your bash shell to your own liking and improve your workflow in the process!

Introducing Environment Variables

What are environment variables? If you're even vaguely familiar with any programming language you'll have heard the term variable, which is a method of storing some information, such as a letter, number, or string of text, in a convenient manner that makes it easy to manipulate. Just as an example, here's a line of Python code (you don't have to know Python, I've just used it because it's the programming language I know best):

name = raw_input("Please enter your name: ")

This should be fairly clear. The variable here is name, because it holds the response to the question.

Now, the shell is in fact a programming language of sorts, so you can do the exact same thing in the shell. However, the syntax (grammar) is different, as it differs between programming languages. Here's an example which sets up the variable example to represent a string of text:

matthew@trinity:~$ example="This is an example"

This should be easy to follow. Please note, however, that the bash shell, at least, is less forgiving than many programming languages - it won't tolerate spaces between the elements of the command. Also, while you can set up a variable using a number or a single word and not use quotes, if you want to put in a string of more than one items, you MUST use either single or double quotes.
Now you've set this, it's easy to view it using the echo command, which for those of you with programming experience is like print in Python or Basic. Note when printing the variable you precede it with a dollar sign:

matthew@trinity:~$ echo $example
This is an example

Environment variables are divided into two types - global and local. Local variables are limited to the current shell session - in other words they're lost once you exit, and if you have two shell sessions and define a local variable in one of them, it's not available in the other. Global variables are present throughout the system, and are often defined for you by the system. By convention, they are usually expressed in uppercase.
You can view the global environment variables with the printenv command:

matthew@trinity:~$ printenv
KDE_MULTIHEAD=false
SSH_AGENT_PID=5704
DM_CONTROL=/var/run/xdmctl
TERM=xterm
SHELL=/bin/bash
XDG_SESSION_COOKIE=efec6dfd9a5f464a8c8a6760490c6b47-1243367179.617199-1707425974
XDM_MANAGED=/var/run/xdmctl/xdmctl-:0,maysd,mayfn,sched,rsvd,method=classic
GTK2_RC_FILES=/home/matthew/.gtkrc-2.0-kde:/home/matthew/.kde/share/config/gtkrc-2.0
GTK_RC_FILES=/etc/gtk/gtkrc:/home/matthew/.gtkrc:/home/matthew/.kde/share/config/gtkrc
GS_LIB=/home/matthew/.fonts
WINDOWID=31468213
KDE_FULL_SESSION=true
USER=matthew
LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.svgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:
SSH_AUTH_SOCK=/tmp/ssh-YTSIDl5600/agent.5600
SESSION_MANAGER=local/trinity:/tmp/.ICE-unix/5751
DCOP_YAKUAKE_SESSION=4
KONSOLE_DCOP=DCOPRef(yakuake,konsole)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
DESKTOP_SESSION=default
KONSOLE_DCOP_SESSION=DCOPRef(yakuake,session-1)
PWD=/home/matthew
KDE_SESSION_UID=
LANG=en_GB.UTF-8
HISTCONTROL=ignoreboth
HOME=/home/matthew
SHLVL=1
XCURSOR_THEME=DMZ-White
LOGNAME=matthew
LESSOPEN=| /usr/bin/lesspipe %s
DISPLAY=:0
LESSCLOSE=/usr/bin/lesspipe %s %s
COLORTERM=
DCOP_YAKUAKE_TERMINAL=4
_=/usr/bin/printenv

These variables can be viewed just like any other. I'll point out three for you in particular. SHELL represents the default shell you have set:

matthew@trinity:~$ echo $SHELL
/bin/bash

HOME represents your home directory:

matthew@trinity:~$ echo $HOME
/home/matthew

And finally, PATH represents where your system looks by default for executable files. The different folders are show, separated by commas:

matthew@trinity:~$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

This last one is worth mentioning because it can cause problems. Your PATH variable essentially refers to all the places your system will look for an executable. For instance, Firefox on my system is in /usr/bin/firefox. If I type firefox into the terminal and press Enter, the system will search through all the folders in PATH until it finds a file called firefox in one of them, whereupon it executes it.

Note that the current directory (which would be represented by a period/full stop)does not appear. That's a security feature common to most Unix-like operating systems - by not looking in the current directory by default, it makes it harder to run a recently downloaded executable file, and that along with the permissions system is one of the things that makes Linux more generally secure than Windows. This is why if you want to run an executable file in the current directory, you need to precede its name with ./ to show that it's in the current directory, such as ./configure for example, or provide the full path to it, such as /home/matthew/configure.

Local environment variables don't have a specific command to print them. However, you can use the set command to display all environment variables, both global and local. I haven't repeated this here because there was a LOT of it, but if you want to try it just enter the following:

matthew@trinity:~$ set

Next time, we'll take a further look at environment variables, and go into more details about how you can use them.

INX

If you've been following along for a while, you should have picked up a fair bit of knowledge about how to use the command line. You should be comfortable using a non-graphical text editor such as Vim or nano, installing software using apt-get or dpkg, and viewing text files using cat or less. But I strongly suspect you may not be ready to use the command line exclusively!

However, you may be surprised by just how much can be done from the command line. Did you know you can listen to music from the command line? Or watch videos? Read email? Chat via IRC or IM? Even surf the Web (and not just using a text-only browser such as Lynx, I'm talking about with graphics, although you shouldn't expect the same experience as you get in Firefox).

Furthermore, there's a lot you can get from an interactive teaching method, rather than just following along with my blog posts. For that reason, I highly recommend that if you can spare the bandwidth, you download a copy of INX. It's a live CD, based on Ubuntu, that doesn't include either a desktop or an X window server. In other words, it's command line only!

Now, I'm not dreaming of suggesting you install this in place of your regular Ubuntu or Ubuntu-derived distro install, but it's a great teaching tool. It includes the excellent Ceni network manager, which I've encountered before in Sidux, another distro which like Ubuntu is based on Debian. Ceni is surprisingly intuitive to use and isn't appreciably harder to use than Network Manager in Ubuntu.

INX is also a great showcase for GNU Screen, which is included along with a full tutorial on how to use it. There's also a great introduction which shows just how much the command line can do. On top of that, there's a very easy tutorial that teaches you some of the basics of the bash shell, in an interactive and fun way. All the way through you're given menus to navigate, and you'll never be left at a blank prompt with nothing obvious to do (unless you ask to!).

INX is a great way to learn more about the Linux command line, and I strongly recommend everyone who reads this downloads a copy if they can, and either runs it in Virtualbox or burns it to a CD and boots from it. It's based on Ubuntu Hardy so it should work fine on most hardware. Try it out, and even if you always swore you'd never touch the command line you'll soon lose your fear and will begin to appreciate just why so many Linux and Unix users are ardent fans of the command line. It's an interface that in many ways is more powerful and quicker to use than a point-and-click graphical desktop, and INX will help you get the best out of it. And as it's a live CD you don't have to worry about messing anything up!

Redirection

If you've understood pipes reasonably well, redirection should be easy to understand as well. It's a similar concept in that it involves doing something with the output of a command.

Redirection allows you to send the output of a command into a text file, or send the contents of a file as input to a command. It's used in a similar fashion to pipes, and is represented by the > symbol.

Here's an example:

ls -A > list.txt

Try running this and it will generate a list of all the files and folders in your home directory and save it as list.txt. If you then open the created text file using less or cat, you can see the effect for yourself.

Please note, however, that if you use this with an existing file it will be overwritten. However, by using the double redirection sign(>>) you can append the results to the file, so they are added to whatever is there. Try running a variant of the same command:

ls -A >> list.txt

If you've done it right, there will be two entries for each item in list.txt.

You can also use redirection in the opposite direction to send the contents of a file to a command in order to be written to standard output, as in this example:

matthew@trinity:~/Development/Python/PPAB2ED$ grep game < list.txt
game_over_2.py
game_over2.py
game_over.py

In this case I'd already used ls -A > list.txt to provide a list of the files in a directory I'd used for Python programming. This command allows grep to sort through the file and pick out those lines that have the word "game" in them. OK, there are other ways to do this using pipes, such as cat list.txt | grep game, but this demonstrates how redirection works

You can also combine redirection each way within one command. In this case, I use grep to sort through list.txt for lines featuring the word "game", but I then send the output of that to a new file, game_list.txt:

matthew@trinity:~$ grep game < list.txt > game_list.txt
matthew@trinity:~$ cat game_list.txt
game_over_2.py
game_over2.py
game_over.py

I've had to shorten the path to fit this in, but it's still in the same directory. This demonstrates how you can combine redirection in both ways.

It's quite common to see redirection used in shell scripts (programs written in bash) for various purposes, such as capturing errors, editing text files (I've seen it used to edit your /etc/apt/sources.list in order to add new repositories) and other purposes. It's not of use to the average user as often as pipes are, but it's still worth knowing about.

Pipes

One of the original philosophies behind Unix is that each program should do one thing only, and do it well, and by and large most command-line applications in Ubuntu continue that tradition. By sticking to just that one task, they remain efficient and powerful.

But sometimes you may need more complex functionality, and this is where piping comes into its own. Piping allows you to combine commands in a simple but effective way that, if well used, makes them more powerful than the sum of their parts.

The concept is simple. Say it's a hot day and you want to water the plants in your garden. You have a tap on the outside wall of your house, but that's nowhere near the plants. You need to take the output of the tap (water) and use it as input to the plants.

How do you do it? You use a hosepipe, and that's exactly what a pipe is in Linux and Unix. It takes the output of one command and uses it as input for another.

The pipe character looks like this:

|

Where you find it differs on different keyboards, and some really do their best to hide it away! It's often shown as broken in the middle. On my Kubuntu machine it's on the same key as the backslash (\) character. Make a mental note of where it is, as you'll find it very handy!

Here's a simple example. On my Kubuntu laptop, I want to run ls -A, which shows all the files and folders in a directory, including hidden ones, within my /home directory. However, there's quite a lot here so if I display it all, it'll just run off the screen. But if I pipe it into the less command, I can easily scroll up and down through them. Here's how you do it:

ls -A | less

Try it out! This is a common way of making directories like this more readable. Note that less didn't need a file name specified, instead it accepted the output of ls -A as input. Most commands will accept this fine.

Remember I said that grep was more effective when combined with pipes? Well, here's an example where I'm again running ls -A in my /home directory. However, here I want to find files or folders with bash in their names, so I use grep to filter the output for responses that include bash:

matthew@trinity:~$ ls -A | grep bash
.bash_history
.bash_logout
.bashrc

You can use more than one pipe if you wish. To the best of my knowledge, there's no upper limit! Here's an example I used the other day when trying out OpenBSD, another open-source Unix-based operating system similar to Linux. I wasn't sure what device the CD drive was recorded as, so I ran the following:

ls /dev | grep -i cd | less

The first part listed all the files or folders in the /dev directory (in other words, devices on the computer). I then piped the output into grep, which picked out the files or folders which had cd in their names (note the use of -i to make grep case-insensitive). Finally, I piped the results of that into less so I could easily move up and down through the responses. Although this wasn't in Linux, it would work exactly the same in Ubuntu as in OpenBSD.

Pipes are extremely powerful. They let you "glue" commands together to make them more useful. You should already be able to see just how useful pipes can be if used well. Next time we'll deal with a related concept, redirection.

Using grep to find text

The grep command is one of the most useful in Linux. It may not seem that exciting at first, but once you learn about pipes (next time!) you'll begin to see why it's so indispensable.

Put simply, grep searches through one or more text files for a specific word or phrase. For any lines it finds which include the specified word or phrase, it will display them on the screen. It will also work with standard output (such as the output of the cat command, for example), when combined with pipes.

Let's take a simple example: Your name is Eric and you know that somewhere in your /home directory is a report you've written for work. Let's say you know that it has the word competition in it. So you can instruct grep to search through all the files in your home directory like this:

grep -r competition /home/eric/*

Note the use of the * wild card character, meaning it will search through every file in /home/eric. Also, I've used the -r option here, short for recursive (similar to with rm), so it will also search every folder within /home/eric.

However, grep is case sensitive, so if you wanted to make it include results in both upper and lower case, you would use the -i option, like this:

grep -ir competition /home/eric/*

If you're entering a phrase with more than one word, you'll need to put it in quotes, like this:

grep "Hello there" hello.txt

grep is most commonly used just with one file, as in this example, where I'm using to search for the word universe in /etc/apt/sources.list:

matthew@trinity:~$ grep universe /etc/apt/sources.list
## universe WILL NOT receive any review or updates from the Ubuntu security
deb http://gb.archive.ubuntu.com/ubuntu/ hardy universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy universe
deb http://gb.archive.ubuntu.com/ubuntu/ hardy-updates universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-updates universe
# deb http://gb.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
# deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu hardy-security universe
deb-src http://security.ubuntu.com/ubuntu hardy-security universe
matthew@trinity:~$

While grep is a simple utility, and is therefore fairly easy to explain, it's incredibly useful once you get the hang of it. If you want to know more about the various options available, you may want to examine the man page for grep:

man grep

As stated above, its real power comes when you combine it with pipes, which I'll be demonstrating next time.