Managing your email with fetchmail, procmail and mutt

Let's face it, email can be a pain. Sure it's useful, but between the social network stuff, the marketing emails and other stuff, there's a lot to wade through, and that's without even counting the spam. Graphical email clients may be full of features and relatively easy to use, but after a point it becomes worthwhile considering a more powerful solution.

My favourite email client these days is mutt. I have a small mail server (actually a Pogoplug running Debian, since it's got better ARM support than Ubuntu) that I use to back up all my email from my Gmail account, as well as several others, and it's an extremely efficient way to deal with your email. If necessary, I can view it in a graphical client like Mozilla Thunderbird, but for most of my needs mutt is more than sufficient. Traditionally, the Unix way of doing things is to have many smaller applications that will do one job and do it well, and once you get a setup like this working, it's incredibly powerful and flexible as a result.

So today we're going to cover creating a similar setup on whatever computer you want. If you have a machine that is always on, you could set up an SMTP and IMAP or POP server (I use Postfix and Dovecot), obtain a fully-qualified domain name and run your own mail server, which is a powerful and flexible option. Or you could just get and sort your email automatically, or just back it up offline.

We're going to create a simple setup in this tutorial that will use fetchmail to grab emails from one or more email accounts, then use procmail to filter them into the folders you designate, before using mutt to read them. From here it's fairly trivial to expand upon this - you could set up SpamAssassin to filter out spam emails, for example.

Get the required packages in the usual way:

sudo apt-get install procmail fetchmail mutt

Once these are installed, you can start working on the configuration files. Don't panic! I've found that fetchmail, procmail and mutt are all well-documented, there's plenty of great reference materials online, and once you're done you can easily move your configuration files to a new computer if necessary.

So first of all, create the .forward file in your home directory with the following command:

echo "|/usr/bin/procmail" > .forward

Of course, you could do this with your favourite text editor, but why bother waiting for it to open? This just displays |/usr/bin/procmail to standard output and then redirects it into a new file, thus saving you keystrokes!

Next, you need to create the configuration for fetchmail. Create a new file in your home directory called .fetchmailrc and enter something like this:

set daemon 3600
set logfile /home/user/.fetchmail.log
set no bouncemail
poll pop.gmail.com proto POP3 auth password no dns user "user@gmail.com" pass "password" is user keep ssl

mda "/usr/bin/procmail -d %T"

This example is for a Gmail account and uses the POP3 protocol to download email (I find that for fetchmail, POP is probably a better choice than IMAP in most cases since with IMAP, if you've already read something it won't get downloaded via IMAP). Note the first line - this specifies an interval in seconds between fetchmail checking for new mail. Here I've set it to download once per hour since I use this for backing up my email, but set it to whatever you want. If you want you can then add more email accounts after the end of the settings for that one, but before the line that starts with mda, which tells fetchmail to send all the fetched emails to procmail for delivery, so for instance you could download your work and home email into the same mailbox.

Don't forget to change user to the name of your user account on your machine, and the email address and password for those for the email account(s) you want to use fetchmail with

You also need to set the permissions correctly for your .fetchmailrc. Enter the following command to correct these:

chmod 700 .fetchmailrc

So with fetchmail configured, we'll move on to procmail. Now, procmail is an insanely powerful tool and I can only hope to give you a very brief overview of what it can do, including:

• Automatically sort emails from or to specific email addresses into specified folders


• Break up mailing list digests into individual messages (this is very useful!)


• Automatically respond to emails


• Work as a spam filter (although something like SpamAssassin is considerably more effective

So create a new text file in your home directory called .procmailrc and enter the following:

SHELL=/bin/bash
PATH=/usr/sbin:/usr/bin
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$HOME/.procmail.log
LOG=""
VERBOSE=yes

Once you have the filter working how you want it, you may wish to consider changing VERBOSE to no so the log is not so large.

Now you need to think about what kind of format and folders you want to use. The two mail formats for mail files are mbox and Maildir. While mbox uses a single text file to represent a mailbox, Maildir uses a folder that contains three further folders called cur, new and tmp. Maildir is generally considered superior so we will go with that, however if you want to use mbox instead, all you need to do is drop the trailing / from the mailbox name - the setup above uses a Maildir called Maildir, so if you wanted to use a single mbox file called Mail, you would change that line to MAILDIR=$HOME/Mail for example.

Assuming you want to use Maildir, create a folder in your home directory called Maildir, then go into it and create three directories called cur, new and tmp. Then create three more called .Drafts, .Sent , and .Spam and create the cur, new and tmp directories inside each of them. If you'd like to create more folders for specific purposes, create them in Maildir with the name preceded by a . and create the same cur, new and tmp directories inside each one. For instance, if you want to have a mailbox for emails relating to Linux, you might create it as .Linux inside the Maildir folder.

Now return to your .procmailrc. When configuring procmail, you're creating a series of rules, and if an email matches one of those rules, the action specified in the rule will be taken. As a simple example, here's a rule from my .procmailrc:

# Drop all emails from National Express East Anglia, as there is no good reason to retain these
:0:
*^From:.*(noreply\@nexusalpha\.com)
/dev/null

In this case, these emails are essentially worthless after I have read them, as they're merely to advise if my train to or from work is on time, so I see no reason to keep them around. Rules for procmail are referred to as recipes, and generally consist of three parts.

The first part doesn't mean much - it simply designates the start of the recipe. However, the trailing colon ensures that only one message gets written at the same time - this isn't really necessary for Maildir, but it's a good idea to use it if you're using mbox.

The second part describes a pattern that an email should match. Here, we're looking for emails from the email address noreply@nexusalpha.com. Note that as procmail uses regular expressions, some characters will need to be escaped by preceding them with a \ character, including @ and . characters. Note that you can also specify multiple addresses by separating them with a pipe character, as in this example:

*^From:.*(bob\@company\.com|dave\@company\.com)

This will match any of the email address quoted in the rule. Also note that you can include more than one place to find the string you're searching for at once by placing it in brackets and separating them with a pipe, as in this example:

*^(To|Cc|Bcc):.*(bob\@company\.com|dave\@company\.com)

You can search for a string in From, To, Cc, Bcc, Subject, or practically anywhere else in an email if you want. You can also require that multiple conditions be met, as in this example:

*^From:.*(bob\@company\.com)
*^Subject:.*(joke)

This example would match anything from bob@company.com with joke in the subject.

Finally, the third part shows what should be done with the email once it's been matched against the pattern. In the example above, we sent it to /dev/null, effectively deleting it, which is useful as long as you know you don't want to keep email that matches that pattern. More normally, you're going to want to put it in a specific folder. Here's how you might do that:

:0:
*^Subject:.*(Linux)
$MAILDIR/.Linux/

This would send anything with Linux in the subject line to the folder .Linux. By writing some rules carefully, you can generally rely on procmail delivering them to the right location. When you receive emails, procmail will work down the configuration file until it finds a pattern that matches. If it doesn't the email will go in the default folder specified above.

Npw, a couple of very handy tricks I've learned with procmail. First, a lot of spam emails tend not to be specifically addressed to your email address, so a lot of them can be caught by sending anything not specifically sent to your email address to the spam folder, as in this example:

:0:
* !(To|Cc).*(bob\@company\.com)
$MAILDIR/.Spam/

Here the ! negates the condition so that anything not for bob@company.com gets sent to the Spam folder. Note that this will also catch many mailing list emails, however, since they are also often not sent directly to your email address.

Finally, another favourite of mine is this one, which will break up a mailing list digest into individual messages using the formail tool:

:0:
*^From:.*(mailinglist)
| formail +1 -ds procmail

Note that if you then want this to go into a specific folder, you'll need to set up another rule to catch the individual messages. Fortunately, these are normally sent to the mailing list address, so you should be able to use something like this after the recipe above:

:0:
*^(To|Cc):(mailinglist)
$MAILDIR/mailinglist/

If you want to know more, refer to the man pages for procmail, procmailex, procmailrc and procmailsc. In particular, procmailex contains lots of great examples, and there are loads of procmail recipes around on the web that you may want to borrow snippets from.

Finally, time to configure mutt. Here's a sample mutt configuration for you - save this as .muttrc:

set mbox_type=Maildir

set spoolfile="~/Maildir/"
set folder="~/Maildir/"
set mask="!^\\.[^.]"
set record="+.Sent"
set postponed="+.Drafts"

mailboxes ! + `\
for file in ~/Maildir/.*; do \
box=$(basename "$file"); \
if [ ! "$box" = '.' -a ! "$box" = '..' -a ! "$box" = '.customflags' \
-a ! "$box" = '.subscriptions' ]; then \
echo -n "\"+$box\" "; \
fi; \
done`

macro index c "?" "open a different folder"
macro pager c "?" "open a different folder"

# Move spam to the Spam folder more easily
macro index X "s=.Spam\n" "file as Spam"
macro pager X "s=.Spam\n" "file as Spam"

set sort = 'threads'
set sort_aux = 'reverse-last-date-received'
set auto_tag = yes
ignore "Authentication-Results:"
ignore "DomainKey-Signature:"
ignore "DKIM-Signature:"
hdr_order Date From To Cc
alternative_order text/plain text/html *
auto_view text/html

# Editor
set editor = "vim"

# Colours
color index red black ~N
color index red black ~O

This is a fairly simple muttrc that I use, but you may want to take note of a few things. This shows any unread emails in red, making it easy to see at a glance. By hitting c, you can switch to a different mailbox, and by hitting Shift-X, you can send an email to the Spam folder.

With this in place, you should be ready to go. Enter the following command:

fetchmail -vk

This will launch fetchmail. Note that if you have a lot of email to download, it can take a while. Also note that your .procmailrc may not be perfect and you may find things winding up in the wrong folders, so you may wish to make a backup - there is a good example of how to do this in procmailex.

Have fun getting fetchmail, procmail and mutt working, and feel free to share some of the procmail recipes you find or come up with here if you think they're something others may be interested in! I've found the sheer power of procmail addictive - to quote Douglas Adams, "I am rarely happier than when spending an entire day programming my computer to perform automatically a task that it would otherwise take me a good ten seconds to do by hand.", and after you've gotten stuck into configuring procmail, you may well agree!

Using Screen

The bash shell is great, but it can be awkward to use because a task will continue to run until it's completed. It's possible to push a task into the background, but this isn't the best way to do so. What you really need is a way to have more than one shell operating at once.

Now, on a graphical desktop this is no problem, as you can just open a new terminal, or a new tab in the same terminal. However, this is less practical on a command-line only system, or one you've remoted into via SSH. But there is a solution, and it's called Screen.

Screen is a terminal multiplexer - it allows you to access multiple terminal sessions inside a single terminal window or remote terminal session (including SSH). Think of it as being like having a tabbed terminal, but in the command line. You can leave Screen running and detach from a session, then reconnect to it later, so for instance if you had an Ubuntu Server install you could connect to it via SSH, issue a command to update the system, then detach and attach again later once it had finished.

Screen is almost certainly preinstalled on your system, but if not you can install it via apt-get in the usual way (sudo apt-get install screen). Once you have it installed, you can start it by issuing the following command:

screen

You're now running screen! You should be confronted by a standard copyright notice (you can remove this by editing /etc/screenrc and removing the hash at the start of the line that reads #startup_message off).

All of screen's internal commands require you hit Ctrl-a, followed by the appropriate key for what you want to do. Try hitting Ctrl-a, then c, and you'll create a new virtual console. Then hit Ctrl-a, then n to cycle to the next virtual console, or Ctrl-a then p to go to the previous one. Ctrl-a, then " will get you a list of all currently running virtual consoles that you can select from using the cursor keys, while Ctrl-a then d will detach you from your current screen session (it will continue to run in the background, so you could potentially log out and leave the processes to run while you do something else). When you want to reattach, just start screen with screen -r and it will reattach you to the running processes. To stop screen, just exit each of the running consoles with Ctrl-d or the exit command in the usual way.

Screen is of limited use on a graphical desktop, but you can still use it there to get used to the idea. It's most useful when dealing with command-line systems. For more details, check out the man page for Screen, or try searching on Google. It's a great tool that makes a lot of tasks easier, and is indispensable if you have to log into a server via SSH a lot. It's also quite handy if you use a lot of command-line tools because it means you can have everything in one terminal session - for instance, you could have mutt in one screen, irssi in another, lynx in a third, Vim in a fourth, and be compiling code using GCC in a fifth, all at the same time, and in a way that makes it easy to switch between them.

It's also worth checking out byobu, which is an Ubuntu-specific tool that makes Screen a lot easier to use if you're new to it. Byobu makes it easy to set Screen up according to your preferences, without having to edit the configuration files by hand.

SSH - using the shell remotely

If you came to Ubuntu from a Windows background, or are otherwise familiar with Windows (let's face it, not many of us are in a position where they never use Windows at all!), you'll probably be familiar with Windows Remote Desktop, which is often used as a troubleshooting tool. You can do the same kind of thing in Ubuntu, but because the Linux command line is so much more powerful and flexible than its Windows counterpart, it's often much more practical to use a command-line only tool for the same purpose. This is often a better solution because sending a constantly updated image of your desktop to somewhere else is inevitably going to use a lot more bandwidth than a few shell commands, and it's very likely you can resolve the issue from the shell. It can also be useful for connecting to servers, shell accounts, web hosts etc.

Telnet was the original method of connecting to a remote computer, however it has declined in popularity due to security issues. SSH has largely replaced it because it's more secure, and that's what I'm going to tell you about today.

SSH requires two components to work - a client and a server. Think of it like surfing the web - when you visit a web page, you use a client (the web browser) to connect to a server (the web server). Similarly, with SSH you use an SSH client to connect to an SSH server on the machine you want to connect to. The most common SSH client and server on Linux are maintained by the OpenSSH project. Ubuntu ships with the OpenSSH client by default, and if you have a machine you'd like to be able to connect to remotely via SSH, just install the openssh-server package in the usual way.

How do you use it? It's simple. Open a terminal and enter something using the following format:

ssh user@server

Let's go into this in a little more detail. First of all, user is just the user name you want to log in as on the remote server. For instance, if you've installed openssh-server on your desktop and your user account is called eric on that machine, then you should put eric as the username. If you want to connect to a shell account in the name of ebarnes, then it should be ebarnes.

The server can be either a domain name or an IP address. So if you have a shell account on a machine which has the domain name myfreeshellaccount.com and your user name is ebarnes, you'd enter ssh ebarnes@myfreeshellaccount.com. If you want to connect to a machine that doesn't have a domain name associated with it, then you'd need to enter the IP address to connect to. There's nothing stopping you connecting to another machine on your home network via this method, so if you have an old machine running Ubuntu Server, for instance, on your home network, and it's been assigned the IP address 192.168.1.4 by your router, you can just connect like this:

ssh ebarnes@192.168.1.4

If you'd like to be able to use SSH to connect to your home machine from outside your home network, that's a bit more demanding. You'd need to either obtain a static IP address from your ISP, which can be expensive, or use a service like DynDNS, and would also need to set up port forwarding on your router. However, there's all sorts of possibilites that offers, such as managing BitTorrent downloads remotely by using a command-line client like rTorrent, or administering your machine remotely over SSH.

Once you're connected, SSH works just like any other shell session - you can do exactly the same things with SSH as you would in the terminal. When you're finished, just type in exit or hit Ctrl-D as you would normally to close the shell and that will finish your session.

Note that you aren't limited to connecting to an SSH server from Linux or Unix hosts. Although most Linux or Unix-based operating systems ship with an SSH client, you can use PUTTY to connect from a Windows box (so, for instance, if you have a desktop running Ubuntu with an OpenSSH server installed, and you want to remote into it from your laptop running Windows 7, you can do that). There's even SSH clients for a number of smartphones, including the iPhone and Android, so you can remote into your machine from anywhere.

If you want to try it out, and you have an old desktop you don't use, it's quite interesting to install Ubuntu Server on it, connect it to your router and disconnect the monitor, managing it entirely via SSH. It can be a useful resource to have around, and offers a lot of possibilities, such as:

• File server
• uPnP server (allows you to stream content to many electronic devices, including the Sony PlayStation 3)
• BitTorrent box

By managing it entirely via SSH, you don't need the additional overhead of running a desktop, and can devote more system resources to whatever job you want it to do. You can use SSH to install whatever software you want via apt-get to customise it for the job you want it to do.

Alternatively, if you don't have an old machine spare or don't wish to try this, why not try it in a virtual machine? Virtualbox is good for this - just install Ubuntu Server in a VM, and under the network settings, attach it to a bridged adapter. This will mean the virtual machine gets allocated its own IP address by your router, and you can connect to it via SSH.

If you have a web hosting account, you may be able to connect to this via SSH - the host will provide the details if this is an option. Finally, another option is to get a shell account - essentially it's a user account on a remote server. I highly recommend devio.us, who are a free shell account provider running on OpenBSD.

Have fun with SSH! Next time I'll show you a few tricks to make your experience with it better!

Going further with environment variables

First of all, I apologise for how long it's taken to get round to this (merely days away from being a full year) but I've been busy! I was frantically busy at work for much of the last year, and have my own web development studies to deal with too, so I've only just gotten around to doing this post. Hopefully I should be able to maintain a better frequency of posts in future!

Last time I explained a bit about how environment variables work. Now we'll get to use them a little. For instance, say you want to set up a shortcut to a specific folder called mystuff to make it easy to get to. Well, you can easily set up an environment variable to hold the path to that folder. Just enter something like this:

export mystuff=/home/user/my/stuff

This will set mystuff up as a local variable that you can call at any time in the same way you'd refer to it normally, like this:

cd $mystuff

Note the dollar sign is only used when calling it, not when setting it.

If you want to set variables permanently, just add them to the .profile file in your home directory. Then, when you next start up a shell, it will automatically load these settings.

There's a number of different settings you can add to customise your bash shell. For instance, the other day I added this to my .profile so that Vim was set as my text editor when using Subversion:

export SVN_EDITOR=/usr/bin/vim

I also added the following to my shell account at Devio.us (who, incidentally are hands down the best shell account providers I have ever found, and I recommend) so I could use Vim in colour, even over SSH:

export TERM=xterm-256color

There's a huge range of settings you can change to customise your bash shell, so I won't attempt to cover them all. Instead, I'll refer you to the guide in the Ubuntu documentation. A few I will mention, however, are PAGER (typically /usr/bin/less), EDITOR (the default text editor, you can set it to /usr/bin/vim, /usr/bin/emacs, or /usr/bin/nano, or any other CLI editor you want), VISUAL (the visual editor, normally /usr/bin/gedit, although may be different on Kubuntu or Xubuntu), or BROWSER (pretty much what you'd expect).

Now, a related subject to environment variables is aliases. These are handy shortcuts you can write for commonly used commands, and can be a great timesaver. For instance, if you often connect to a specific host via ssh, you can write an alias for it. To set up an alias, just enter something like the following:

alias example="ssh user@server"

Now, just enter example, on its own, into the shell, and it will run the command. But, of course, this is only temporary. To make it permanent, set up a file in your home directory called .bash_aliases and add the alias you want to set up to it. Once you've done that, every time you start a new shell, all the aliases you've defined in the file will be available to you!

I hope this has given you some idea of how to customise your bash shell to your own liking and improve your workflow in the process!

Introducing Environment Variables

What are environment variables? If you're even vaguely familiar with any programming language you'll have heard the term variable, which is a method of storing some information, such as a letter, number, or string of text, in a convenient manner that makes it easy to manipulate. Just as an example, here's a line of Python code (you don't have to know Python, I've just used it because it's the programming language I know best):

name = raw_input("Please enter your name: ")

This should be fairly clear. The variable here is name, because it holds the response to the question.

Now, the shell is in fact a programming language of sorts, so you can do the exact same thing in the shell. However, the syntax (grammar) is different, as it differs between programming languages. Here's an example which sets up the variable example to represent a string of text:

matthew@trinity:~$ example="This is an example"

This should be easy to follow. Please note, however, that the bash shell, at least, is less forgiving than many programming languages - it won't tolerate spaces between the elements of the command. Also, while you can set up a variable using a number or a single word and not use quotes, if you want to put in a string of more than one items, you MUST use either single or double quotes.
Now you've set this, it's easy to view it using the echo command, which for those of you with programming experience is like print in Python or Basic. Note when printing the variable you precede it with a dollar sign:

matthew@trinity:~$ echo $example
This is an example

Environment variables are divided into two types - global and local. Local variables are limited to the current shell session - in other words they're lost once you exit, and if you have two shell sessions and define a local variable in one of them, it's not available in the other. Global variables are present throughout the system, and are often defined for you by the system. By convention, they are usually expressed in uppercase.
You can view the global environment variables with the printenv command:

matthew@trinity:~$ printenv
KDE_MULTIHEAD=false
SSH_AGENT_PID=5704
DM_CONTROL=/var/run/xdmctl
TERM=xterm
SHELL=/bin/bash
XDG_SESSION_COOKIE=efec6dfd9a5f464a8c8a6760490c6b47-1243367179.617199-1707425974
XDM_MANAGED=/var/run/xdmctl/xdmctl-:0,maysd,mayfn,sched,rsvd,method=classic
GTK2_RC_FILES=/home/matthew/.gtkrc-2.0-kde:/home/matthew/.kde/share/config/gtkrc-2.0
GTK_RC_FILES=/etc/gtk/gtkrc:/home/matthew/.gtkrc:/home/matthew/.kde/share/config/gtkrc
GS_LIB=/home/matthew/.fonts
WINDOWID=31468213
KDE_FULL_SESSION=true
USER=matthew
LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.svgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:
SSH_AUTH_SOCK=/tmp/ssh-YTSIDl5600/agent.5600
SESSION_MANAGER=local/trinity:/tmp/.ICE-unix/5751
DCOP_YAKUAKE_SESSION=4
KONSOLE_DCOP=DCOPRef(yakuake,konsole)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
DESKTOP_SESSION=default
KONSOLE_DCOP_SESSION=DCOPRef(yakuake,session-1)
PWD=/home/matthew
KDE_SESSION_UID=
LANG=en_GB.UTF-8
HISTCONTROL=ignoreboth
HOME=/home/matthew
SHLVL=1
XCURSOR_THEME=DMZ-White
LOGNAME=matthew
LESSOPEN=| /usr/bin/lesspipe %s
DISPLAY=:0
LESSCLOSE=/usr/bin/lesspipe %s %s
COLORTERM=
DCOP_YAKUAKE_TERMINAL=4
_=/usr/bin/printenv

These variables can be viewed just like any other. I'll point out three for you in particular. SHELL represents the default shell you have set:

matthew@trinity:~$ echo $SHELL
/bin/bash

HOME represents your home directory:

matthew@trinity:~$ echo $HOME
/home/matthew

And finally, PATH represents where your system looks by default for executable files. The different folders are show, separated by commas:

matthew@trinity:~$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

This last one is worth mentioning because it can cause problems. Your PATH variable essentially refers to all the places your system will look for an executable. For instance, Firefox on my system is in /usr/bin/firefox. If I type firefox into the terminal and press Enter, the system will search through all the folders in PATH until it finds a file called firefox in one of them, whereupon it executes it.

Note that the current directory (which would be represented by a period/full stop)does not appear. That's a security feature common to most Unix-like operating systems - by not looking in the current directory by default, it makes it harder to run a recently downloaded executable file, and that along with the permissions system is one of the things that makes Linux more generally secure than Windows. This is why if you want to run an executable file in the current directory, you need to precede its name with ./ to show that it's in the current directory, such as ./configure for example, or provide the full path to it, such as /home/matthew/configure.

Local environment variables don't have a specific command to print them. However, you can use the set command to display all environment variables, both global and local. I haven't repeated this here because there was a LOT of it, but if you want to try it just enter the following:

matthew@trinity:~$ set

Next time, we'll take a further look at environment variables, and go into more details about how you can use them.

INX

If you've been following along for a while, you should have picked up a fair bit of knowledge about how to use the command line. You should be comfortable using a non-graphical text editor such as Vim or nano, installing software using apt-get or dpkg, and viewing text files using cat or less. But I strongly suspect you may not be ready to use the command line exclusively!

However, you may be surprised by just how much can be done from the command line. Did you know you can listen to music from the command line? Or watch videos? Read email? Chat via IRC or IM? Even surf the Web (and not just using a text-only browser such as Lynx, I'm talking about with graphics, although you shouldn't expect the same experience as you get in Firefox).

Furthermore, there's a lot you can get from an interactive teaching method, rather than just following along with my blog posts. For that reason, I highly recommend that if you can spare the bandwidth, you download a copy of INX. It's a live CD, based on Ubuntu, that doesn't include either a desktop or an X window server. In other words, it's command line only!

Now, I'm not dreaming of suggesting you install this in place of your regular Ubuntu or Ubuntu-derived distro install, but it's a great teaching tool. It includes the excellent Ceni network manager, which I've encountered before in Sidux, another distro which like Ubuntu is based on Debian. Ceni is surprisingly intuitive to use and isn't appreciably harder to use than Network Manager in Ubuntu.

INX is also a great showcase for GNU Screen, which is included along with a full tutorial on how to use it. There's also a great introduction which shows just how much the command line can do. On top of that, there's a very easy tutorial that teaches you some of the basics of the bash shell, in an interactive and fun way. All the way through you're given menus to navigate, and you'll never be left at a blank prompt with nothing obvious to do (unless you ask to!).

INX is a great way to learn more about the Linux command line, and I strongly recommend everyone who reads this downloads a copy if they can, and either runs it in Virtualbox or burns it to a CD and boots from it. It's based on Ubuntu Hardy so it should work fine on most hardware. Try it out, and even if you always swore you'd never touch the command line you'll soon lose your fear and will begin to appreciate just why so many Linux and Unix users are ardent fans of the command line. It's an interface that in many ways is more powerful and quicker to use than a point-and-click graphical desktop, and INX will help you get the best out of it. And as it's a live CD you don't have to worry about messing anything up!

Redirection

If you've understood pipes reasonably well, redirection should be easy to understand as well. It's a similar concept in that it involves doing something with the output of a command.

Redirection allows you to send the output of a command into a text file, or send the contents of a file as input to a command. It's used in a similar fashion to pipes, and is represented by the > symbol.

Here's an example:

ls -A > list.txt

Try running this and it will generate a list of all the files and folders in your home directory and save it as list.txt. If you then open the created text file using less or cat, you can see the effect for yourself.

Please note, however, that if you use this with an existing file it will be overwritten. However, by using the double redirection sign(>>) you can append the results to the file, so they are added to whatever is there. Try running a variant of the same command:

ls -A >> list.txt

If you've done it right, there will be two entries for each item in list.txt.

You can also use redirection in the opposite direction to send the contents of a file to a command in order to be written to standard output, as in this example:

matthew@trinity:~/Development/Python/PPAB2ED$ grep game < list.txt
game_over_2.py
game_over2.py
game_over.py

In this case I'd already used ls -A > list.txt to provide a list of the files in a directory I'd used for Python programming. This command allows grep to sort through the file and pick out those lines that have the word "game" in them. OK, there are other ways to do this using pipes, such as cat list.txt | grep game, but this demonstrates how redirection works

You can also combine redirection each way within one command. In this case, I use grep to sort through list.txt for lines featuring the word "game", but I then send the output of that to a new file, game_list.txt:

matthew@trinity:~$ grep game < list.txt > game_list.txt
matthew@trinity:~$ cat game_list.txt
game_over_2.py
game_over2.py
game_over.py

I've had to shorten the path to fit this in, but it's still in the same directory. This demonstrates how you can combine redirection in both ways.

It's quite common to see redirection used in shell scripts (programs written in bash) for various purposes, such as capturing errors, editing text files (I've seen it used to edit your /etc/apt/sources.list in order to add new repositories) and other purposes. It's not of use to the average user as often as pipes are, but it's still worth knowing about.

Pipes

One of the original philosophies behind Unix is that each program should do one thing only, and do it well, and by and large most command-line applications in Ubuntu continue that tradition. By sticking to just that one task, they remain efficient and powerful.

But sometimes you may need more complex functionality, and this is where piping comes into its own. Piping allows you to combine commands in a simple but effective way that, if well used, makes them more powerful than the sum of their parts.

The concept is simple. Say it's a hot day and you want to water the plants in your garden. You have a tap on the outside wall of your house, but that's nowhere near the plants. You need to take the output of the tap (water) and use it as input to the plants.

How do you do it? You use a hosepipe, and that's exactly what a pipe is in Linux and Unix. It takes the output of one command and uses it as input for another.

The pipe character looks like this:

|

Where you find it differs on different keyboards, and some really do their best to hide it away! It's often shown as broken in the middle. On my Kubuntu machine it's on the same key as the backslash (\) character. Make a mental note of where it is, as you'll find it very handy!

Here's a simple example. On my Kubuntu laptop, I want to run ls -A, which shows all the files and folders in a directory, including hidden ones, within my /home directory. However, there's quite a lot here so if I display it all, it'll just run off the screen. But if I pipe it into the less command, I can easily scroll up and down through them. Here's how you do it:

ls -A | less

Try it out! This is a common way of making directories like this more readable. Note that less didn't need a file name specified, instead it accepted the output of ls -A as input. Most commands will accept this fine.

Remember I said that grep was more effective when combined with pipes? Well, here's an example where I'm again running ls -A in my /home directory. However, here I want to find files or folders with bash in their names, so I use grep to filter the output for responses that include bash:

matthew@trinity:~$ ls -A | grep bash
.bash_history
.bash_logout
.bashrc

You can use more than one pipe if you wish. To the best of my knowledge, there's no upper limit! Here's an example I used the other day when trying out OpenBSD, another open-source Unix-based operating system similar to Linux. I wasn't sure what device the CD drive was recorded as, so I ran the following:

ls /dev | grep -i cd | less

The first part listed all the files or folders in the /dev directory (in other words, devices on the computer). I then piped the output into grep, which picked out the files or folders which had cd in their names (note the use of -i to make grep case-insensitive). Finally, I piped the results of that into less so I could easily move up and down through the responses. Although this wasn't in Linux, it would work exactly the same in Ubuntu as in OpenBSD.

Pipes are extremely powerful. They let you "glue" commands together to make them more useful. You should already be able to see just how useful pipes can be if used well. Next time we'll deal with a related concept, redirection.

Using grep to find text

The grep command is one of the most useful in Linux. It may not seem that exciting at first, but once you learn about pipes (next time!) you'll begin to see why it's so indispensable.

Put simply, grep searches through one or more text files for a specific word or phrase. For any lines it finds which include the specified word or phrase, it will display them on the screen. It will also work with standard output (such as the output of the cat command, for example), when combined with pipes.

Let's take a simple example: Your name is Eric and you know that somewhere in your /home directory is a report you've written for work. Let's say you know that it has the word competition in it. So you can instruct grep to search through all the files in your home directory like this:

grep -r competition /home/eric/*

Note the use of the * wild card character, meaning it will search through every file in /home/eric. Also, I've used the -r option here, short for recursive (similar to with rm), so it will also search every folder within /home/eric.

However, grep is case sensitive, so if you wanted to make it include results in both upper and lower case, you would use the -i option, like this:

grep -ir competition /home/eric/*

If you're entering a phrase with more than one word, you'll need to put it in quotes, like this:

grep "Hello there" hello.txt

grep is most commonly used just with one file, as in this example, where I'm using to search for the word universe in /etc/apt/sources.list:

matthew@trinity:~$ grep universe /etc/apt/sources.list
## universe WILL NOT receive any review or updates from the Ubuntu security
deb http://gb.archive.ubuntu.com/ubuntu/ hardy universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy universe
deb http://gb.archive.ubuntu.com/ubuntu/ hardy-updates universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-updates universe
# deb http://gb.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
# deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu hardy-security universe
deb-src http://security.ubuntu.com/ubuntu hardy-security universe
matthew@trinity:~$

While grep is a simple utility, and is therefore fairly easy to explain, it's incredibly useful once you get the hang of it. If you want to know more about the various options available, you may want to examine the man page for grep:

man grep

As stated above, its real power comes when you combine it with pipes, which I'll be demonstrating next time.

Viewing text files

One of the central philosophies behind Unix, which Linux has inherited from it, is that configuration files should be plain text. This has the advantage that you can easily edit these files, even if for some reason you can't start the X window server. So an unbootable system can be repaired (hopefully!) by editing a few text files. It also means you can make changes to your system remotely with ease, by logging on via telnet or ssh (don't worry if you don't know what these are!)
For this reason, Linux has a lot of incredibly powerful tools to enable you to manipulate text files. I'm going to go through several that you can use to view text files from the command line.
The most basic of these is cat. This just prints the file as standard output - in other words, it just dumps it into your terminal, as in this example:

matthew@morpheus:~$ cat /etc/apt/sources.list
#deb cdrom:[Ubuntu 8.04 _Hardy Heron_ - Release i386 (20080423)]/ hardy main restricted
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.

deb http://gb.archive.ubuntu.com/ubuntu/ hardy main restricted
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://gb.archive.ubuntu.com/ubuntu/ hardy-updates main restricted
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## universe WILL NOT receive any review or updates from the Ubuntu security
## team.
deb http://gb.archive.ubuntu.com/ubuntu/ hardy universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy universe
deb http://gb.archive.ubuntu.com/ubuntu/ hardy-updates universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://gb.archive.ubuntu.com/ubuntu/ hardy multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy multiverse
deb http://gb.archive.ubuntu.com/ubuntu/ hardy-updates multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-updates multiverse

## Uncomment the following two lines to add software from the 'backports'
## repository.
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
# deb http://gb.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
# deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository. This software is not part of Ubuntu, but is
## offered by Canonical and the respective vendors as a service to Ubuntu
## users.
# deb http://archive.canonical.com/ubuntu hardy partner
# deb-src http://archive.canonical.com/ubuntu hardy partner

deb http://security.ubuntu.com/ubuntu hardy-security main restricted
deb-src http://security.ubuntu.com/ubuntu hardy-security main restricted
deb http://security.ubuntu.com/ubuntu hardy-security universe
deb-src http://security.ubuntu.com/ubuntu hardy-security universe
deb http://security.ubuntu.com/ubuntu hardy-security multiverse
deb-src http://security.ubuntu.com/ubuntu hardy-security multiverse

cat is fine for small text files, but for long ones like this example, the text just scrolls off the screen. So for longer files, it's better to use an alternative.
To give you more control when viewing text files, someone came up with the more command, which allows you to scroll through pages of text (for this reason it's known as a pager). The more command has now been largely replaced by the less command, which is more advanced, but does essentially the same thing. However, both are available in Ubuntu.
Try using less to get a feel for it:

less /etc/apt/sources.list

You can scroll up and down using the cursor keys. Alternatively, if you're used to the Vim text editor, you can use j and k to scroll up and down, same as you would in Vim. To exit, press Q.
Finally, two commands that you're unlikely to need unless you're running Ubuntu as a server are head and tail. As the name suggests, these allow you to view the beginning (head) or the end (tail) of a text file, as in this example for head:

matthew@morpheus:~$ head /etc/apt/sources.list
#deb cdrom:[Ubuntu 8.04 _Hardy Heron_ - Release i386 (20080423)]/ hardy main restricted
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.

deb http://gb.archive.ubuntu.com/ubuntu/ hardy main restricted
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://gb.archive.ubuntu.com/ubuntu/ hardy-updates main restricted

And this example for tail:

matthew@morpheus:~$ tail /etc/apt/sources.list
## users.
# deb http://archive.canonical.com/ubuntu hardy partner
# deb-src http://archive.canonical.com/ubuntu hardy partner

deb http://security.ubuntu.com/ubuntu hardy-security main restricted
deb-src http://security.ubuntu.com/ubuntu hardy-security main restricted
deb http://security.ubuntu.com/ubuntu hardy-security universe
deb-src http://security.ubuntu.com/ubuntu hardy-security universe
deb http://security.ubuntu.com/ubuntu hardy-security multiverse
deb-src http://security.ubuntu.com/ubuntu hardy-security multiverse

Head and tail are primarily used for log files, which can contain hundreds of lines of text. For example, someone administering a web server would no doubt find tail invaluable to examine the server logs.

By default head and tail display 10 lines of text, but you can change this with the -n modifier, as in this example:

matthew@morpheus:~$ tail -n5 /etc/apt/sources.list
deb-src http://security.ubuntu.com/ubuntu hardy-security main restricted
deb http://security.ubuntu.com/ubuntu hardy-security universe
deb-src http://security.ubuntu.com/ubuntu hardy-security universe
deb http://security.ubuntu.com/ubuntu hardy-security multiverse
deb-src http://security.ubuntu.com/ubuntu hardy-security multiverse

This applies to head as well.
Over the next few sessions, we'll go into some of the other useful tools that are available for manipulating text files in Ubuntu.

Handy shortcuts in the bash shell

You may have noticed that I've pushed the use of the command line (or, to give it the proper name, the shell), over graphical utilities much of the time. This is deliberate - the shell is a fast, flexible and powerful interface. Once you've got a bit of experience, most things can be done a lot faster from the shell. Now that I'm used to apt-get and apt-cache, I almost never use Synaptic to install anything in Ubuntu as it's so much quicker and more efficient to use the shell.

There are several different shells available for Unix-like operating systems, such as the original Bourne shell, the C shell, the Korn shell (used in Sun's Solaris operating system, among others), the Z shell (which was used in early versions of Mac OS X), and others. Linux uses the bash shell (in common with more recent versions of Mac OS X and OpenSolaris), which stands for Bourne Again SHell. This is a free software rewrite of the Bourne shell, but with many of the more advanced features of other shells incorporated into it.

Bash includes a number of very handy shortcuts that really make a difference. Don't you hate having to type out filenames in full, such as package-0.8.1.tar.gz? Me too, but bash has an answer - autocompletion.

To use autocompletion, start typing a command, then press tab. If bash has enough to confirm what you're typing, it will automatically fill in the rest. For example:

sudo apt-get upd

Press tab here, and there's enough for bash to autocomplete to this:

sudo apt-get update

But what if you enter this:

sudo apt-get up

Here, there's more than one option. The final word could be update or upgrade, so bash doesn't know how to complete it. Instead, when you press tab, you'll hear a bell to show that bash can complete it. But press it again and you'll be shown a list of possible options, as in this example:

matthew@morpheus:~$ sudo apt-get up
update upgrade
matthew@morpheus:~$ sudo apt-get up

Note that although the line you were on ends, what you'd entered is copied to the new line so you don't have to retype it.

This works for filenames and commands, so you could use autocompletion in this example with the file package-0.8.1.tar.gz:

tar -xzf pack

Assuming there wasn't another file in the current directory whose first four letters were pack, this would autocomplete fine. You can also use autocompletion to fill in the names of folders. If you had a path to a file like this:

/usr/share/doc

You could type one or two letters for each folder within the path, and press tab to fill in the rest of the folder name.

Autocompletion is a very handy feature in bash that can save you a lot of time, and makes using the shell a lot quicker and easier.

There are some other handy keyboard shortcuts you might want to be aware of. I've mentioned in a previous post that if you've started a process and would like to cancel it, you can press Ctrl-C and the process will stop dead.

Another handy shortcut is Ctrl-D, which acts the same as the exit command ie. it closes down the shell you've been using. However, it's possible to have one shell running inside another so this doesn't necessarily mean it closes the terminal emulator. For instance, if you open the terminal, you have one shell running. If you then enter bash and press enter, this opens a new shell and places you in it (this would be a subshell). By pressing Ctrl-D here, you'd be returned to your original shell. Pressing it again would close the terminal.

Another is Ctrl-L, which clears the terminal. A similar effect can be had by entering the clear command.

If you want to repeat previous commands, you can use the up and down cursor keys to scroll through previously used commands in the current session. For example, if you entered ls, then less text.txt, you could press up twice to get ls again.

Finally, two things you may want to know are how to copy and paste. Copying is Ctrl-Shift-C, while pasting is Ctrl-Shift-V or Shift-Insert.

If you're interested in learning a few more shortcuts, check out this link for more details. I'd recommend you get familiar with at least the commands mentioned above as these are extremely labour-saving. Also, over the next few posts I plan to delve into the shell a little more and you'll probably find these shortcuts invaluable.

I'd also recommend that you find a good command-line editor as these are invaluable for editing configuration files. I highly recommend you take the time to learn Vim using the tutorial mode built into it (just open the shell and enter vimtutor), as although Vim may seem tough at first, it's one of those things like touch-typing - at first it slows you down, but soon enough you'll be faster than you ever were. It's also a great text editor for if you want to program - I'm currently learning Python and I find it incredibly useful. But if you try it and it really doesn't work for you, try nano instead. Even though it's a command-line editor, it's simple to use.

Finally, if you're really finding bash an uphill struggle and would prefer something that really goes out of its way to help beginners, why not try fish? Short for Friendly Interactive SHell, this is a good choice for newbies, and a great learning tool. But don't give up on bash! Use fish to learn the ropes if you like it, but bash is more powerful and flexible in the long term. You can get fish from the repositories:

sudo apt-get install fish

Then, to run it, just enter the following:

fish

This will set fish running as a subshell within bash.

Getting started with Wine

Ubuntu offers a huge range of software from the repositories that cover all kinds of needs. For any one piece of software you can name that you use in Windows, there's a pretty good chance that something else exists in Linux that can do the same job, often for free. OK, some of them may not be as well known as their Windows counterparts, and they may require a little adjustment, but this is fairly trivial. For most people, Ubuntu can do everything they use Windows for.

But what if you absolutely need to use a specific application? Or you have other software that you bought for Windows and would like to be able to run? Or even those games you may still have?

Well, the answer is Wine. It stands for Wine Is Not an Emulator. It's a compatibility layer for running Windows programs. Essentially, it allows you to run many Windows programs on Linux. Wine recently released a stable 1.0 version after 15 years in development, and although it's by no means perfect, you may be surprised at just how good it is. I have successfully run two of my favourite games on it (Homeworld and Homeworld 2, the second of which doesn't seem to work in Vista!), and many other games are well supported and will work straight away. For some others, you can get them running with a little work - the website features a search engine so you can check the compatibility of something before you try it. If you have some Windows games you haven't been able to run, why not dig them out and give them a try?

Wine is in the Ubuntu repository, but unless you're already running the next version, Intrepid Ibex, this version may not be very up to date. It's probably better to go with the official version.

To install the current version of Wine, you need to add another repository. Open a terminal and enter the following:

wget -q http://wine.budgetdedicated.com/apt/387EE263.gpg -O- | sudo apt-key add -

This will download the APT key for the Wine repository. Now, to add the repository to you /etc/apt/sources.list:

sudo wget http://wine.budgetdedicated.com/apt/sources.list.d/hardy.list -O /etc/apt/sources.list.d/winehq.list

This is the version for Hardy Heron. For Intrepid Ibex, you need to enter this instead:

sudo wget http://wine.budgetdedicated.com/apt/sources.list.d/intrepid.list -O /etc/apt/sources.list.d/winehq.list

Now to update it:

sudo apt-get update

And to install Wine:

sudo apt-get install wine

Now, Wine creates its own entry in your applications menu, as shown here on my Kubuntu Hardy system:


This is how it looks in Kubuntu with Tasty Menu, but any Ubuntu variant should show the same four items for a newly installed copy of Wine: Notepad, Uninstall, Browse C Drive and Configure Wine.

Wine works by creating a folder in your home directory called .wine. This is then set up to trick the applications into thinking they are running on Windows. Choosing the Browse C Drive option lets you open a folder inside .wine called drive_c, which emulates the C drive (the main hard drive) on a Windows install. This is where anything you install using Wine will reside.

Notepad is a clone of the Windows version, and Uninstall is essentially a copy of the Windows Add/Remove software dialogue. That leaves Configure Wine, which comes up with this:


This dialogue lets you configure Wine to help make things work. For instance, you can specify for it to emulate different versions of Windows - even to the point of being able to run one as XP, another as Vista, and a third as Windows 98, for example! This gives you huge scope to tinker and get software working.

For the most part, getting something working is easy. For a Windows EXE file, I find that whether you're using Gnome or KDE, you can just click on it as usual and it will run. You can also run it from the command line with something like this:

wine setup.exe

Other types of files are generally just as easy to run. If you have an application on a CD or DVD that you wish to install, it won't usually autorun the way it does in Windows, but instead you have to browse through the disc yourself and locate the appropriate file (which is usually setup.exe, autorun.exe, or something similar). Follow the installer's instructions as usual and it should install fine.

For more information about using Wine, I'd suggest going to their website and having a good look around. Nearly every piece of software you can think of has been tried with Wine, and they give ratings for how well it works in Wine, often including instructions.

Now, I won't claim that Wine is perfect. Not everything will work on it, and not everything that does work on it works well. But you may be surprised at how well it does do, and it's improving all the time. If there's one or two applications you use in Windows that are keeping you from making the switch to Ubuntu, why not try Wine to see if they will run?

Tar files

It's quite likely that if you've been using Linux for any great length of time, you'll have discovered tar files. If you go to a website and download something which is offered as the Linux version of an application, the chances are it will be a tar archive (although some offer separate RPM and deb packages for different distros). Tar files are widely used in Linux, as well as in other Unix-like operating systems.
So what are they? Well, if you've come from a Windows background, you should be familiar with .zip files. These don't usually contain an application, but instead contain a collection of files. The .zip format is simply a convenient way to store them as it rolls them into one file and compresses it, making it ideal for distributing these files across the Internet or storing them.
Well, tar files are Linux and Unix's equivalent. Tar stands for Tape Archive, and the term comes from backing up to a magnetic tape. Like .zip files, tar files are usually used as a convenient and easy way to store or distribute a collection of files. However, it's far more common for tar files to be used to distribute applications than it is for .zip files. If you download a non distro-specific Linux binary, that will usually be packaged as a tar file. Also, if you download source code for an application, you can generally expect it to be packaged as a tar file.
Although it's best to use a deb package in Ubuntu where possible, there are times when you can only get something as a tar package. So being able to deal with tar packages is a necessary Linux skill. In addition, they're ideal for backing up files in case something goes wrong.
Tar files aren't compressed by default. There are two utilities available to compress them: bzip2and gzip. Normally you'll be able to tell which one has been used from the file extensions used. This example is compressed using bzip2:

package.tar.bz2

While this is compressed using gzip

package.tar.gz

That's pretty simple to follow, but remember that unlike Windows, Linux doesn't rely on file extensions to ascertain what a file is in quite the same way, and someone can easily give a package a completely different extension. So you may see variations such as .tgz.

There are plenty of graphical applications available to deal with these packages, but as usual we're going to go the command-line route! For this example I'm going to use the Wordpress software, as this is available as a gzipped tar package, and is a mere 1MB in size.

Click on this link to download the current version of Wordpress. If the package winds up on your desktop, move it to your /home directory to make it more convenient to work with.

Now, to extract the contents of an uncompressed tar archive, you would enter the following:

tar -xf packagename.tar

The x tells tar that you want to extract the files, while the f indicates that the filename for the package follows.

But this won't deal with the compression. To make it work in this case, you need to tell tar to uncompress the archive as well. In order to do so, tar needs to know whether the package uses bzip2 or gzip compression.


In our example, the package uses gzip. To handle this, add a z to the options. So, to uncompress and extract Wordpress, we need to enter the following:

tar -xzf wordpress-2.6.2.tar.gz

This will create a new folder called wordpress, which contains all the files you need to get Wordpress working on your system. But we're not going to do that (at least, not until at least after the end of the lesson!). Instead, we're going to become familiar with the tar command by using it to both create and extract archives.

If you run ls, you'll notice the original archive is still there. Move it to another directory so it's out of the way, but still there in case you need it. Now, we'll use the wordpress folder you just extracted to create a bzipped tar file.

Now, because you're creating an archive rather than extracting one, you don't use the x option. Instead you use the c option. This is easy to remember as it's c to create, x to extract. You need to include the f option as again you need to specify the filename. The difference is, you also need to specify the path to the folder you want to create an archive from. This makes the default something like this:

tar -cf packagename.tar /folder

But as with extracting the archive, you need to add options to compress it. Fortunately, you can used the same option for each type of compression whether you're creating or extracting, so with gzip you would always put a z. For bzip2, you use j. So, to create an archive of the wordpress folder and compress it with bzip2, you'd enter the following:

tar -cjf wordpress.tar.bz2 wordpress

Notice that again, the original folder is still there. Remove it with the following command to get it out of the way:

rm -rf wordpress

Now, let's just extract this again, then create another archive using and you'll be familiar with extracting and creating both gzipped and bzipped tar files. Enter the following to extract your bzipped tar file:

tar -xjf wordpress.tar.bz2

That should extract the file once more as wordpress. Finally, lets turn it into a gzipped tar file:

tar -czf wordpress.tar.gz wordpress

That's it! Now you should be sufficiently savvy with tar files to be able to extract and create them as you wish!
There are many more options available for tar, such as v, which verbosely lists extracted files (so as it extracts files, it lists them), among others. If you want to know more, I suggest you study the man page for tar, using the following command:

man tar

One final note. Ubuntu does include the unzip utility, which enables you to extract Windows .zip files using the following command:

unzip file.zip

So if you need to transfer something across from Windows to Linux, it's fine to zip it up in Windows and extract it in Linux.
Dealing with tar files is necessary for installing any software that isn't specifically packaged for your distro. If you download a tar file for an application, to install it you need to extract the archive and look in the resulting folder for a text file called something like INSTALL or README. You will usually find something that will give instructions on how to install it. For source code, there will usually be instructions on how to compile the application from source.

Permissions

Something we haven't touched on yet is permissions in Ubuntu. The concept of permissions is highly important to how Linux works.
As you may know, Linux is a clone of the Unix operating system. Unix was originally designed to be used on huge systems which might have hundreds or even thousands of users, such as in universities, and it was therefore important to establish who was entitled to do what. The concept of groups therefore came about. By creating groups, and making individual users members of these groups, system administrators could set what users could do quickly and easily.
For example, refer back to the VirtualBox install article a while back and you'll notice that in order to use VirtualBox, a user needs to be a member of a group called "vboxusers". By adding a user name "fred"to this group, this shows that the administrator has given permission for the user fred to use VirtualBox. Now, to a certain extent this is a little redundant on a desktop system that will have only a handful of users, but it illustrates how Unix-like operating systems such as Linux handle files. In Ubuntu, a normal user will belong to a group based on their username, but other Linux distributions may instead have a group called "users".
A user account has limits. Each user has their own directory in the /home folder (so in this example, fred has /home/fred), in which they can save and edit files, but normally they can't save elsewhere. Users can move around, but some files are out of bounds. Some files (normally system files, such as your /etc/apt/sources.list), can be read, but not written to, so you could open it in a text editor, edit it and save a copy in your own home folder, for example, but you couldn't edit and save the original. This is enforced by the use of file permissions.
Each file or directory is owned by a user. For instance, if the user fred rips a CD using Sound Juicer and save it to a directory called Music in his /home directory, he will be the owner of the audio files created. He's therefore free to do as he wishes with these files. If he then logs out and another user, jane, logs in, she is not the owner and what she can do with these files will be subject to the permissions fred has set for these files. He can if he wishes make these files available for other users to edit or delete if they wish, or he can make them read-only. If he prefers not to share these, he can deny anyone else access to the files.
Another thing to note is the Unix philosophy that "everything is a file". This includes folders, so fred can also make entire folders read only, or can deny others access to them completely. So he could make his entire /home directory available for others to access or edit, or keep people from accessing it entirely.
There are three sets of permissions for each file or folder. There is one for the user, one for the specified group, and one for everyone else. To demonstrate this, open your file manager (Nautilus in Ubuntu, Konqueror or Dolphin in Kubuntu and Thunar in Xubuntu). Right-click on a file or folder and select Properties. Select the Permissions tab and you'll see something like this:


You may want to open this image in a separate tab so you can see it better. As you can see, each set of users has a dropdown box for you to set the permissions, and Group has an additional dropdown so you can set the permissions for each group. So far, this may have seemed quite technical, but hopefully this makes it a bit easier to understand!

There's three types of access you can set for a file:

• You can set out who is allowed to read a file;
• You can set out who is allowed to write to a file;
• And you can set who can execute (run) a file

Normally, a file in your /home directory which you own will allow you to read or write to it, but not execute it (this is just another one of the reasons why Linux is regarded as being more secure than Windows, as by default a freshly downloaded file can't be executed straight away). So, if you wanted to run it, you'd have to change the permissions on it. For instance, if you write a shell script to automate a task for you, you'll need to give yourself permission to run it.

One other thing to remember: Opening a folder is executing it, not reading it. So if you want to prevent other people looking in a folder, it should not be executable for others.

To illustrate the permissions on the files in a directory, run the following command:

ls -lh

You should see something like this:

matthew@matthew-laptop2:~/Demonstration$ ls -lh
total 8.0K
drwxr-xr-x 2 matthew matthew 4.0K 2008-09-18 20:31 Example
-rw-r--r-- 1 matthew matthew 13 2008-09-18 20:30 example.txt
matthew@matthew-laptop2:~/Demonstration$

Now, pay attention to the string of characters at the start of each sentence. The d at the start of the Example line indicates this is a directory, while this is a dash for example.txt, showing that this is not a directory.

The next three characters give the permissions for the owner. The r shows this can be read, the w shows this can be written, and the x shows it's executable. As example.txt shows a dash instead of an x, it shows it's not executable, but the r and w show it can still be read or written to.

The next three characters show the permissions for the group specified (more on this a bit later). The format is the same. Then, the final three characters are the permissions for everyone else.

You'll notice there are two names afterwards. The first is the name of the owner, while the second is the owner's group. As stated previously, in Ubuntu each user has their own group. The rest is information about the file.

OK, so we've established what permissions are. Now, how can we change them? You can easily use a file manager to change these, as demonstrated in the screenshot above, but as usual I'm going to point you towards the terminal as the best way to do it.

First, to change permissions, you use the chmod command. In this example, to make the example.txt document available so that everyone can read and write to it, you enter this:

chmod a+rw example.txt

This is quite easy to follow - all users (a) add (+) read (r) and write (w) permissions. Another example:

chmod a-w example.txt

This means all users (a) take away (-) write (w) permission. You can edit permissions for just the user as follows:

chmod u+rw example.txt

Similarly, you can edit it for the group like this:

chmod g+rw example.txt

Or you can edit it for others:

chmod o+rw example.txt

Please note that you can leave out the first character, which means it will default to a (all), as in this example:

chmod +x example.txt

Now, say you've written a shell script called housekeeping to automate a common task for you. To enable this to run, you need to make it executable, like this:

chmod +x housekeeping

OK, so now you should understand how to change permissions for a file or folder. But what about changing the user who owns it? There's an easy way to do this, using the chown (CHange OWNer) command. Please note, this must be run using sudo for security reasons. So if fred wants jane to own his example.mp3 file, he should run the following:

sudo chown jane example.mp3

He can also change the group at the same time:

sudo chown jane.examplegroup example.mp3

Or, he can change the group alone with chgrp (CHange GRouP):

sudo chgrp examplegroup example.mp3

This is a fairly complex concept, so you may want to spend some time getting familiar with the idea of permissions.

Manipulating files and folders

Apologies for the delay since my last post - as I don't have access to the Internet during the week at present and my parent's Internet connection has been playing up, it's been hard to get any blogging done! Hopefully that will change soon!
Anyway, manipulating files and folders is something that you need to know about, seeing that Linux applies the Unix philosophy of "everything is a file". So I'll walk you through some basic commands.
First up is mkdir. This command makes a new folder. So, if you're in your /home directory and you enter the following:

mkdir Work

You'll create a new directory called Work. Never forget that Linux, unlike Windows, is case sensitive. Also, if you want to create a new directory anywhere other than in your own /home directory, you'll need to use sudo.
To remove an empty directory, you can use rmdir, as in this example:

rmdir Work

Now, to remove a file, you use the rm command, as in this example where we're deleting the document Example.html:

rm Example.html

Like most commands in Linux, rm has a number of options you can specify. A word of warning: watch out for the command rm -rf. The r option means remove things recursively (so you can remove a directory AND everything inside it), while the f option forces it, even if it gives a warning. You can specify the folder you want to delete with its path, so NEVER run the following command (unless you actually want to destroy your system!):

sudo rm -rf /

This means "delete the whole filesystem recursively, ignoring all warnings, with root access", essentially meaning delete everything!. I've tried this on VirtualBox installs of Ubuntu and it will destroy everything on your system! That said, the command rm -rf can be quite handy for deleting a folder and it's contents, but be VERY careful with it!
To copy a file, use the cp command, with the source followed by the destination, as in this example:

cp Example.html /media/USB

You can change the name of the destination by specifying the file name as well, like this:

cp Example.html /media/USB/Example2.html

Moving a file is similar to copying it, with similar syntax:

mv Example.html /media/USB

And again you can change the name, in the same way as with cp. Or you can use it to rename a file without moving it, like this:

mv Example.html Example2.html

Now, moving files and folders is all very well, but how can you show them? Simple, use the ls command:

ls

To show hidden files and folders as well, use this:

ls -A

As usual, if you want to know more about these commands, refer to the man pages for more specific guidelines. Hopefully this has given you enough to get started. I recommend spending a little time becoming comfortable with these commands, they'll give you a sound grasp of the fundamentals of using the command line.

Navigating folders from the command line

If you've followed the posts on this blog for a while, you should be reasonably comfortable using the command line to install software, edit configuration files and so on. You should also be aware of the layout of the Linux filesystem. However, you may not be aware of how to navigate it without using a graphical file manager such as Nautilus or Konqueror. So in this post I'm going to go through basic commands for navigating from the command line.

First of all is the cd command. This is short for "change directory". The default location for you to start is in your home directory, and entering the following:

cd

will take you to your home directory. You can also use the ~ character (known as a tilde) as shorthand for your home directory, which is useful if referring to your home directory from elsewhere. So, entering the following:

cd ~

will give the same results as the previous command.

You can also navigate up one directory as follows:

cd ..

And can go back to the previous directory using this command:

cd -

These commands can be used as part of longer paths. For instance, say you are currently in /var/www but want to go to a folder called Music in your home folder. You can go there with this command:

cd /home/username/Music

Or you can use this:

cd ~/Music

Now, say you want to go from there to another folder in your home directory called Pictures. You can enter this:

cd ../Pictures

Then, to head back to the Music folder, you can enter the following:

cd -

And to head to the root of the filesystem, just enter the following:

cd /

Now, how can you see where you are? Easy, just enter the pwd command (short for Print Working Directory). Here I'm in the Desktop directory:

/home/matthew/Desktop

Got that? I recommend you spend a little time getting comfortable with these commands as they are the first building blocks in becoming proficient at the command line. Next time I'll go into creating and deleting files and folders.

Recreating my Ubuntu desktop

I've been asked how to recreate the desktop I showed in the previous post in Ubuntu. For those who haven't seen it, here it is:


OK, first of all there's a number of applications you need (I'm assuming you're working from a fresh install). So run the following command:

sudo apt-get install avant-window-navigator awn-manager compizconfig-settings-manager emerald gnome-do gnome-do-plugins

Strictly speaking, Gnome Do isn't required as it doesn't contribute to the appearance, but it's a very handy application launcher.

Once these are installed, you'll need to remove the bottom panel. Move any applets you want to keep up to the top panel, then right-click on the bottom panel and select Remove This Panel. To set Avant Window Navigator to run when you start up, go to System>Preferences>Sessions and select Add. Then enter Avant Window Navigator as the name, and /usr/bin/avant-window-navigator as the command. You can put anything you like for the description (I put "Mac OS X-style dock bar").

Now to grab the packages you need:

1. Genesis Emerald Theme
2. Blue-Joy GTK Theme
3. Oxygen Refit 2 Icon Theme
4. Peace Wallpaper
   

Now open System>Preferences>Appearance. From the Theme tab, click on Install. You'll be able to browse your filesystem for the theme packages you just downloaded. Use this method to install the Blue-Joy GTK theme and the Oxygen Refit 2 icon theme. For the background, go to the Background tab, click on Add, and select it using the file browser. You might want to copy it to your/usr/share/backgrounds so all users can have it - this can be done with the following command:

sudo cp (insert name of image here) /usr/share/backgrounds

Finally, to install the Emerald theme. Go to System>Preferences>Emerald Theme Manager. Click on Import, select the theme package, and click on Open. You should then be able to select the theme package from the list of those installed.

However, it won't have any immediate effect as Emerald isn't running. To start it up, enter the following command at the terminal:

emerald --replace

To make Emerald always be the default window decorator, open CompizConfig Settings Manager and look for Window Decorator under Effects. Check the box, then click on the icon to adjust the settings. Replace whatever is in the Command box with emerald --replace.

If you now log out, restart your X window server using the Options menu from the login page, then log back in, Emerald and AWN should be running straight away. You can then add whatever launchers you want, but by default AWN only has the launcher applet. If you want to add Ubuntu Tweak so you can add more applets to AWN, please refer to my previous post.

If you're interested in recreating the dock bar I've got, here it is from left to right:

1. File Browser applet
2. Firefox launcher
3. Exaile music player launcher
4. VLC media player launcher
5. GIMP launcher
6. Evolution launcher
7. Pidgin launcher
8. Jokosher audio editor launcher
9. Kino launcher
10. F-Spot launcher
11. Skype launcher
12. Last.fm launcher (not the Last.fm applet, I couldn't get that to work)
13. XChat IRC client launcher
14. Time Vault launcher
15. Ubuntu Tweak launcher
16. Gnome Terminal launcher
17. Showdesktop applet
18. Stacks Trasher applet

I personally really like this desktop. I'm normally more of a KDE guy (I have a separate Kubuntu install on another computer, while this computer is dual-booting Vista and Ubuntu), but I really enjoy working in this kind of environment. I've added two more virtual desktops so as to get the cube effect, and the whole thing looks amazing.

One problem with using this theme is that a lot of things are black that aren't normally (such as the paper in OpenOffice!) and I haven't yet found a way round this, so this might be a problem. I'm hoping the theme's author will be able to resolve this.