Wednesday 14 January 2009

Pipes

One of the original philosophies behind Unix is that each program should do one thing only, and do it well, and by and large most command-line applications in Ubuntu continue that tradition. By sticking to just that one task, they remain efficient and powerful.

But sometimes you may need more complex functionality, and this is where piping comes into its own. Piping allows you to combine commands in a simple but effective way that, if well used, makes them more powerful than the sum of their parts.

The concept is simple. Say it's a hot day and you want to water the plants in your garden. You have a tap on the outside wall of your house, but that's nowhere near the plants. You need to take the output of the tap (water) and use it as input to the plants.

How do you do it? You use a hosepipe, and that's exactly what a pipe is in Linux and Unix. It takes the output of one command and uses it as input for another.

The pipe character looks like this:
|

Where you find it differs on different keyboards, and some really do their best to hide it away! It's often shown as broken in the middle. On my Kubuntu machine it's on the same key as the backslash (\) character. Make a mental note of where it is, as you'll find it very handy!

Here's a simple example. On my Kubuntu laptop, I want to run ls -A, which shows all the files and folders in a directory, including hidden ones, within my /home directory. However, there's quite a lot here so if I display it all, it'll just run off the screen. But if I pipe it into the less command, I can easily scroll up and down through them. Here's how you do it:
ls -A | less

Try it out! This is a common way of making directories like this more readable. Note that less didn't need a file name specified, instead it accepted the output of ls -A as input. Most commands will accept this fine.

Remember I said that grep was more effective when combined with pipes? Well, here's an example where I'm again running ls -A in my /home directory. However, here I want to find files or folders with bash in their names, so I use grep to filter the output for responses that include bash:
matthew@trinity:~$ ls -A | grep bash
.bash_history
.bash_logout
.bashrc

You can use more than one pipe if you wish. To the best of my knowledge, there's no upper limit! Here's an example I used the other day when trying out OpenBSD, another open-source Unix-based operating system similar to Linux. I wasn't sure what device the CD drive was recorded as, so I ran the following:
ls /dev | grep -i cd | less

The first part listed all the files or folders in the /dev directory (in other words, devices on the computer). I then piped the output into grep, which picked out the files or folders which had cd in their names (note the use of -i to make grep case-insensitive). Finally, I piped the results of that into less so I could easily move up and down through the responses. Although this wasn't in Linux, it would work exactly the same in Ubuntu as in OpenBSD.

Pipes are extremely powerful. They let you "glue" commands together to make them more useful. You should already be able to see just how useful pipes can be if used well. Next time we'll deal with a related concept, redirection.

Tuesday 13 January 2009

Using grep to find text

The grep command is one of the most useful in Linux. It may not seem that exciting at first, but once you learn about pipes (next time!) you'll begin to see why it's so indispensable.

Put simply, grep searches through one or more text files for a specific word or phrase. For any lines it finds which include the specified word or phrase, it will display them on the screen. It will also work with standard output (such as the output of the cat command, for example), when combined with pipes.

Let's take a simple example: Your name is Eric and you know that somewhere in your /home directory is a report you've written for work. Let's say you know that it has the word competition in it. So you can instruct grep to search through all the files in your home directory like this:
grep -r competition /home/eric/*

Note the use of the * wild card character, meaning it will search through every file in /home/eric. Also, I've used the -r option here, short for recursive (similar to with rm), so it will also search every folder within /home/eric.

However, grep is case sensitive, so if you wanted to make it include results in both upper and lower case, you would use the -i option, like this:
grep -ir competition /home/eric/*

If you're entering a phrase with more than one word, you'll need to put it in quotes, like this:
grep "Hello there" hello.txt

grep is most commonly used just with one file, as in this example, where I'm using to search for the word universe in /etc/apt/sources.list:
matthew@trinity:~$ grep universe /etc/apt/sources.list
## universe WILL NOT receive any review or updates from the Ubuntu security
deb http://gb.archive.ubuntu.com/ubuntu/ hardy universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy universe
deb http://gb.archive.ubuntu.com/ubuntu/ hardy-updates universe
deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-updates universe
# deb http://gb.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
# deb-src http://gb.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu hardy-security universe
deb-src http://security.ubuntu.com/ubuntu hardy-security universe
matthew@trinity:~$

While grep is a simple utility, and is therefore fairly easy to explain, it's incredibly useful once you get the hang of it. If you want to know more about the various options available, you may want to examine the man page for grep:
man grep
As stated above, its real power comes when you combine it with pipes, which I'll be demonstrating next time.