Managing your email with fetchmail, procmail and mutt

Let's face it, email can be a pain. Sure it's useful, but between the social network stuff, the marketing emails and other stuff, there's a lot to wade through, and that's without even counting the spam. Graphical email clients may be full of features and relatively easy to use, but after a point it becomes worthwhile considering a more powerful solution.

My favourite email client these days is mutt. I have a small mail server (actually a Pogoplug running Debian, since it's got better ARM support than Ubuntu) that I use to back up all my email from my Gmail account, as well as several others, and it's an extremely efficient way to deal with your email. If necessary, I can view it in a graphical client like Mozilla Thunderbird, but for most of my needs mutt is more than sufficient. Traditionally, the Unix way of doing things is to have many smaller applications that will do one job and do it well, and once you get a setup like this working, it's incredibly powerful and flexible as a result.

So today we're going to cover creating a similar setup on whatever computer you want. If you have a machine that is always on, you could set up an SMTP and IMAP or POP server (I use Postfix and Dovecot), obtain a fully-qualified domain name and run your own mail server, which is a powerful and flexible option. Or you could just get and sort your email automatically, or just back it up offline.

We're going to create a simple setup in this tutorial that will use fetchmail to grab emails from one or more email accounts, then use procmail to filter them into the folders you designate, before using mutt to read them. From here it's fairly trivial to expand upon this - you could set up SpamAssassin to filter out spam emails, for example.

Get the required packages in the usual way:

sudo apt-get install procmail fetchmail mutt

Once these are installed, you can start working on the configuration files. Don't panic! I've found that fetchmail, procmail and mutt are all well-documented, there's plenty of great reference materials online, and once you're done you can easily move your configuration files to a new computer if necessary.

So first of all, create the .forward file in your home directory with the following command:

echo "|/usr/bin/procmail" > .forward

Of course, you could do this with your favourite text editor, but why bother waiting for it to open? This just displays |/usr/bin/procmail to standard output and then redirects it into a new file, thus saving you keystrokes!

Next, you need to create the configuration for fetchmail. Create a new file in your home directory called .fetchmailrc and enter something like this:

set daemon 3600
set logfile /home/user/.fetchmail.log
set no bouncemail
poll pop.gmail.com proto POP3 auth password no dns user "user@gmail.com" pass "password" is user keep ssl

mda "/usr/bin/procmail -d %T"

This example is for a Gmail account and uses the POP3 protocol to download email (I find that for fetchmail, POP is probably a better choice than IMAP in most cases since with IMAP, if you've already read something it won't get downloaded via IMAP). Note the first line - this specifies an interval in seconds between fetchmail checking for new mail. Here I've set it to download once per hour since I use this for backing up my email, but set it to whatever you want. If you want you can then add more email accounts after the end of the settings for that one, but before the line that starts with mda, which tells fetchmail to send all the fetched emails to procmail for delivery, so for instance you could download your work and home email into the same mailbox.

Don't forget to change user to the name of your user account on your machine, and the email address and password for those for the email account(s) you want to use fetchmail with

You also need to set the permissions correctly for your .fetchmailrc. Enter the following command to correct these:

chmod 700 .fetchmailrc

So with fetchmail configured, we'll move on to procmail. Now, procmail is an insanely powerful tool and I can only hope to give you a very brief overview of what it can do, including:

• Automatically sort emails from or to specific email addresses into specified folders


• Break up mailing list digests into individual messages (this is very useful!)


• Automatically respond to emails


• Work as a spam filter (although something like SpamAssassin is considerably more effective

So create a new text file in your home directory called .procmailrc and enter the following:

SHELL=/bin/bash
PATH=/usr/sbin:/usr/bin
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$HOME/.procmail.log
LOG=""
VERBOSE=yes

Once you have the filter working how you want it, you may wish to consider changing VERBOSE to no so the log is not so large.

Now you need to think about what kind of format and folders you want to use. The two mail formats for mail files are mbox and Maildir. While mbox uses a single text file to represent a mailbox, Maildir uses a folder that contains three further folders called cur, new and tmp. Maildir is generally considered superior so we will go with that, however if you want to use mbox instead, all you need to do is drop the trailing / from the mailbox name - the setup above uses a Maildir called Maildir, so if you wanted to use a single mbox file called Mail, you would change that line to MAILDIR=$HOME/Mail for example.

Assuming you want to use Maildir, create a folder in your home directory called Maildir, then go into it and create three directories called cur, new and tmp. Then create three more called .Drafts, .Sent , and .Spam and create the cur, new and tmp directories inside each of them. If you'd like to create more folders for specific purposes, create them in Maildir with the name preceded by a . and create the same cur, new and tmp directories inside each one. For instance, if you want to have a mailbox for emails relating to Linux, you might create it as .Linux inside the Maildir folder.

Now return to your .procmailrc. When configuring procmail, you're creating a series of rules, and if an email matches one of those rules, the action specified in the rule will be taken. As a simple example, here's a rule from my .procmailrc:

# Drop all emails from National Express East Anglia, as there is no good reason to retain these
:0:
*^From:.*(noreply\@nexusalpha\.com)
/dev/null

In this case, these emails are essentially worthless after I have read them, as they're merely to advise if my train to or from work is on time, so I see no reason to keep them around. Rules for procmail are referred to as recipes, and generally consist of three parts.

The first part doesn't mean much - it simply designates the start of the recipe. However, the trailing colon ensures that only one message gets written at the same time - this isn't really necessary for Maildir, but it's a good idea to use it if you're using mbox.

The second part describes a pattern that an email should match. Here, we're looking for emails from the email address noreply@nexusalpha.com. Note that as procmail uses regular expressions, some characters will need to be escaped by preceding them with a \ character, including @ and . characters. Note that you can also specify multiple addresses by separating them with a pipe character, as in this example:

*^From:.*(bob\@company\.com|dave\@company\.com)

This will match any of the email address quoted in the rule. Also note that you can include more than one place to find the string you're searching for at once by placing it in brackets and separating them with a pipe, as in this example:

*^(To|Cc|Bcc):.*(bob\@company\.com|dave\@company\.com)

You can search for a string in From, To, Cc, Bcc, Subject, or practically anywhere else in an email if you want. You can also require that multiple conditions be met, as in this example:

*^From:.*(bob\@company\.com)
*^Subject:.*(joke)

This example would match anything from bob@company.com with joke in the subject.

Finally, the third part shows what should be done with the email once it's been matched against the pattern. In the example above, we sent it to /dev/null, effectively deleting it, which is useful as long as you know you don't want to keep email that matches that pattern. More normally, you're going to want to put it in a specific folder. Here's how you might do that:

:0:
*^Subject:.*(Linux)
$MAILDIR/.Linux/

This would send anything with Linux in the subject line to the folder .Linux. By writing some rules carefully, you can generally rely on procmail delivering them to the right location. When you receive emails, procmail will work down the configuration file until it finds a pattern that matches. If it doesn't the email will go in the default folder specified above.

Npw, a couple of very handy tricks I've learned with procmail. First, a lot of spam emails tend not to be specifically addressed to your email address, so a lot of them can be caught by sending anything not specifically sent to your email address to the spam folder, as in this example:

:0:
* !(To|Cc).*(bob\@company\.com)
$MAILDIR/.Spam/

Here the ! negates the condition so that anything not for bob@company.com gets sent to the Spam folder. Note that this will also catch many mailing list emails, however, since they are also often not sent directly to your email address.

Finally, another favourite of mine is this one, which will break up a mailing list digest into individual messages using the formail tool:

:0:
*^From:.*(mailinglist)
| formail +1 -ds procmail

Note that if you then want this to go into a specific folder, you'll need to set up another rule to catch the individual messages. Fortunately, these are normally sent to the mailing list address, so you should be able to use something like this after the recipe above:

:0:
*^(To|Cc):(mailinglist)
$MAILDIR/mailinglist/

If you want to know more, refer to the man pages for procmail, procmailex, procmailrc and procmailsc. In particular, procmailex contains lots of great examples, and there are loads of procmail recipes around on the web that you may want to borrow snippets from.

Finally, time to configure mutt. Here's a sample mutt configuration for you - save this as .muttrc:

set mbox_type=Maildir

set spoolfile="~/Maildir/"
set folder="~/Maildir/"
set mask="!^\\.[^.]"
set record="+.Sent"
set postponed="+.Drafts"

mailboxes ! + `\
for file in ~/Maildir/.*; do \
box=$(basename "$file"); \
if [ ! "$box" = '.' -a ! "$box" = '..' -a ! "$box" = '.customflags' \
-a ! "$box" = '.subscriptions' ]; then \
echo -n "\"+$box\" "; \
fi; \
done`

macro index c "?" "open a different folder"
macro pager c "?" "open a different folder"

# Move spam to the Spam folder more easily
macro index X "s=.Spam\n" "file as Spam"
macro pager X "s=.Spam\n" "file as Spam"

set sort = 'threads'
set sort_aux = 'reverse-last-date-received'
set auto_tag = yes
ignore "Authentication-Results:"
ignore "DomainKey-Signature:"
ignore "DKIM-Signature:"
hdr_order Date From To Cc
alternative_order text/plain text/html *
auto_view text/html

# Editor
set editor = "vim"

# Colours
color index red black ~N
color index red black ~O

This is a fairly simple muttrc that I use, but you may want to take note of a few things. This shows any unread emails in red, making it easy to see at a glance. By hitting c, you can switch to a different mailbox, and by hitting Shift-X, you can send an email to the Spam folder.

With this in place, you should be ready to go. Enter the following command:

fetchmail -vk

This will launch fetchmail. Note that if you have a lot of email to download, it can take a while. Also note that your .procmailrc may not be perfect and you may find things winding up in the wrong folders, so you may wish to make a backup - there is a good example of how to do this in procmailex.

Have fun getting fetchmail, procmail and mutt working, and feel free to share some of the procmail recipes you find or come up with here if you think they're something others may be interested in! I've found the sheer power of procmail addictive - to quote Douglas Adams, "I am rarely happier than when spending an entire day programming my computer to perform automatically a task that it would otherwise take me a good ten seconds to do by hand.", and after you've gotten stuck into configuring procmail, you may well agree!