Multiple Gmail accounts & labels with Emacs

Multiple Gmail accounts & labels with Emacs

Intro

For quite some time, e-mail seemed like an anomaly in my workflow. I am a long time Gmail user, and my decade-old account has a somewhat formidable quantity of labels and filters. My messages are often assigned multiple labels, and I also like to keep only a bunch of messages in the inbox.

Although, in my opinion, Gmail web UI was and still is leagues ahead of many of its competitors and even allows keyboard-centric workflow, it’s awkward to use with a keyboard-driven browser, and for no money on Earth I would enable browser notifications.

Any classical IMAP/SMTP client is hard to use in my case, because a message with multiple labels is copied to IMAP folders for each of the label plus the inbox folder, and the copies look like different messages from the client-side. For example, a message can be read in one label and unread in another.

For a few years, my solution was Mailspring, which provides first-class support for labels. However, it has a feature to deploy spy pixels on emails (and offers no protection from them, obviously), the client is Electron-based with a mouse-driven interface, and the sync engine was closed-source at the time.

So, I found an alternative in Emacs+notmuch+lieer and ditched one more proprietary app (the last big one I can’t let go of is DataGrip).

Notmuch’s tags are just as advanced as Gmail’s labels, so I have basically the same mail structure accessible from Emacs, Gmail Android client and even the web UI when I don’t have access to the first two.

Also, I think the setup I describe here is pretty straightforward and less complex than many I encountered, but my impression is not the most reliable source of such knowledge.

In any case, what follows is a description of my current workflow with instructions of varying levels of precision of how to get there.

Setting up

Gmail

Before we start, some setup is required for the Gmail account.

First, as there is no way to enable SMTP without IMAP on Gmail, you have to set “Enable IMAP” in the “Forwarding and POP/IMAP” tab in the settings. If you use two-factor auth, generate an app password.

Also, make sure your labels do not contain whitespaces because if they do, you will have to type them in quotes all the time.

lieer

lieer (formerly gmailieer) is a program that uses Gmail API to download email and synchronize Gmail labels with notmuch tags. Because of its usage of Gmail API instead of IMAP, there are no problems with duplicating emails in different labels, etc.

As I need to use multiple versions of Python & Node.js for other reasons, I manage my installations of them with Anaconda (Miniconda, to be precise). You may instead use venv or even the system-wide installation of Python and omit the conda clauses, but in my experience Anaconda makes life easier in that regard.

# Create an environment with the name "mail"
conda create --name mail
# Activate the environment
conda activate mail
# Install Python
conda install python
# Download and install lieer
git clone https://github.com/gauteh/lieer.git
cd lieer
pip install .

After which we may check if the gmi executable is available:

which gmi

/home/pavel/Programs/miniconda3/envs/mail/bin/gmi

Notmuch

Notmuch is present in most of the package repositories, so you can install it with your package manager, which is pacman in my case.

sudo pacman -S notmuch

After the installation, run notmuch setup. That will inquire the parameters and create the .notmuch-config file with the answers.

Your full name [Pavel]: Pavel Korytov
Your primary email address [pavel@pdsk.(none)]: thexcloud@gmail.com
Additional email address [Press 'Enter' if none]:
Top-level directory of your email archive [/home/pavel/mail]: /home/pavel/Mail
Tags to apply to all new messages (separated by spaces) [unread inbox]: new
Tags to exclude when searching messages (separated by spaces) [deleted spam]:

It is important to set the new tag for the new messages instead of the default unread and inbox.

Next, add the rule to ignore JSON files to the [new] section of the .notmuch-config file, so it would look like this:

[new]
tags=new
ignore=/.*[.](json|lock|bak)$/

That is needed to ignore the lieer config files. Although, as I have noticed, notmuch is generally pretty good at detecting wrong files in its directories, an explicit ignore rule won’t hurt.

Now, create the mail directory and run the notmuch new command. As notmuch has probably already noticed you, it uses the maildir format, which basically means that one message is stored in one file.

# The same directory mentioned in the 4th question
mkdir ~/Mail
# Initialize notmuch
notmuch new

Add an account

After that, we can create a directory for a mail account and initialize lieer.

cd ~/Mail
# Use whatever name you want
mkdir thexcloud
cd thexcloud
# Intialize lieer
gmi init thexcloud@gmail.com

Running gmi init will run an OAuth authentication to your Gmail account. The credentials will be stored in .credentials.gmailieer.json file, so make sure not to expose it somewhere.

We also can add a few settings for lieer, which will make life easier. First, dots seem to be less awkward to type than slashes for the nested tags:

gmi set --replace-slash-with-dot

Then, we don’t want the new tag to be pushed back to Gmail

gmi set --ignore-tags-local new

Now we can finally download the mail directory. To initiate the download, run

gmi sync

The first download can easily take several hours, depending on the size of your email and the speed of your internet connection, but subsequent runs will be much faster.

The last thing to do here is to add the gmi sync command to notmuch’s pre-new hook, so that the email will be synchronized on the notmuch new command.

# Create the hooks folder
mkdir -p ~/Mail/.notmuch/hooks
# Create the file
cd ~/Mail/.notmuch/hooks
cat > pre-new <<EOF
#!/bin/bash
eval "$(conda shell.bash hook)"
conda activate mail
(cd /home/pavel/Mail/thexcloud/ && gmi sync)
EOF
chmod +x pre-new

Side note: as a hook for conda tends to be rather slow, I run the gmi command with system-wide Python as follows:

#!/bin/bash
GMI="/home/pavel/Programs/miniconda3/envs/mail/bin/gmi"
(cd /home/pavel/Mail/thexcloud/ && $GMI sync)

Which doesn’t seem to cause any particular trouble in that case.

Emacs

There are plenty of different frontends for notmuch (even GUI apps), but the one I’m sticking with the Emacs.

Configuration for Emacs is pretty straightforward, but you probably want to use the notmuch package which came with the system package, because otherwise, you may end up with different versions of frontend and backend.

That’s how it can be done with use-package:

(use-package notmuch
  :ensure nil
  :commands (notmuch)
  :config
  (add-hook 'notmuch-hello-mode-hook
            (lambda () (display-line-numbers-mode 0))))

The only notable observation here is that display-line-numbers-mode seems to break formatting of the notmuch-hello page.

If you use evil-mode, you also should enable the evil-collection mode for notmuch.

Now run M-x notmuch and the notmuch-hello page should appear. Running notmuch-poll-and-refresh-this-buffer (gR with evil bindings) will run the notmuch new command and refresh the buffer. All the syncronized messages should be present.

I should note that notmuch frontend for Emacs is the most user-friendly Emacs app I have seen so far. UI, commands and keybindings are self-descriptive, all the options can be configured with the build-in customize interface. It may be useful to look through emacs tips at the official site and notmuch man pages, in particular syntax for notmuch queries.

Reading mail

notmuch-search-show-thread (RET) opens the thread under the cursor.

notmuch-show-view-part (. v with evil) opens an attachment with associations defined in .mailcap file. Mine looks like this:

audio/*; mpc add %s

image/*; feh %s

application/msword; /usr/bin/xdg-open %s
application/pdf; zathura %s
application/postscript ; zathura %s

text/html; /usr/bin/xdg-open %s

Here watch out for the last line, default version of which may be set as follows:

text/html; /usr/bin/xdg-open %s ; copiousoutput

Which causes a temporary file to be deleted before it could be opened because recent versions of xdg-open do not block the input.

As expected, Emacs mail reader does not trigger any spy pixels or other tracking contents of email (not any I know of, at least). However, opening an HTML email in a browser will even run embedded JavaScript. Therefore, in no case open emails you do not trust with xdg-open. Even if you use NoScript, the browser will still load all the CSS, videos and even iframes, which can be used to track you.

Even Gmail web UI is preferable to view the message in a browser, because the former blocks most of the malicious stuff and does not seem to leak your IP to the sender, for what it’s worth.

Sending mail

To start composing a message, run notmuch-mua-new-mail (C with evil bindings).

After doing so, C-c C-c will run notmuch-mua-send-and-exit, which will invoke the function stated in the message-send-mail-function variable. The default value of the variable is sendmail-query-once, which will inquire the parameters and save them as custom variables.

If SMTP is used, send-mail-function will be set to the one from the built-it smtpmail package. SMTP parameters for Gmail are listed here.

Authorization parameters will be saved to your authinfo file. If you didn’t have one, the plaintext .authinfo will be created, so it’s reasonable to encrypt it:

cd ~
gpg -o .authinfo.gpg -c --cipher-algo AES256 .authinfo

However, if you plan to use multiple accounts with different SMTP servers, it makes more sense to use something like MSMTP to manage multiple accounts. Here are a couple of examples (1, 2) how to do that.

Another alternative for Gmail is to use lieer as sendmail program. That may make sense if you don’t want to enable IMAP and SMTP on your account.

There are also a bunch of ways to set up address completion if the built-in completion based on notmuch database does not suffice.

I also use LanguageTool for Emacs to do a spell checking of important emails (integrations like that really make Emacs shine). For some reason, developers don’t give a link to download the server on the frontpage, so here it is. And here is the relevant part of my Emacs config:

(use-package langtool
  :straight t
  :commands (langtool-check)
  :config
  (setq langtool-language-tool-server-jar "/home/pavel/Programs/LanguageTool-5.1/languagetool-server.jar")
  (setq langtool-mother-tongue "ru"))

As a last note here, to set up a signature create the .signature file in the $HOME directory. If you need more complex logic here, for instance, different signatures for different accounts, you can put an arbitrary expression to the mail-signature variable or apply this gnus-alias tip.

Another account

Adding an account

Now we can send and receive mail from one account. Adding another account is also pretty easy.

If another account is Gmail, the process starts the same as before:

# Create a directory
mkdir -p ~/Mail/progin6304
cd ~/Mail/progin6304
# OAuth
gmi init progin6304@gmail.com
# Settings
gmi set --replace-slash-with-dot

However, before running gmi sync for the second account, we want to make sure that we can distinguish the message from different accounts. To do that, I add the main for the main account and progin for the second account. We also don’t want these labels to be pushed:

cd ~/Mail/thexcloud
gmi set --ignore-tags-local new,mail,progin
cd ~/Mail/progin6304
gmi set --ignore-tags-local new,mail,progin

Now we can use notmuch’s post-new hook to tag the messages based on their folder as follows:

cd ~/Mail/.notmuch/hooks
cat > post-new <<EOF
#!/bin/bash
notmuch tag +main "path:thexcloud/** AND tag:new"
notmuch tag +progin "path:progin6304/** AND tag:new"
notmuch tag -new "tag:new"
EOF
chmod +x post-new

Now it finally makes sense why we wanted to use the new tag in the first place. In principle, any kind of tagging logic can be applied here, but for the reasons I stated earlier, I prefer to set up filters in the Gmail web interface.

The last thing to do is to modify the pre-new hook:

#!/bin/bash
GMI="/home/pavel/Programs/miniconda3/envs/mail/bin/gmi"
(cd /home/pavel/Mail/thexcloud/ && $GMI sync)
(cd /home/pavel/Mail/progin6304/ && $GMI sync)

After which we can finally tag the existing messages and download ones from the new account

notmuch tag +main "path:thexcloud/**"
notmuch new

The obvious problem, however, is that the messages are fetched sequentially, which is rather slow. A solution is to use something like GNU Parallel:

#!/bin/bash
GMI="/home/pavel/Programs/miniconda3/envs/mail/bin/gmi"
parallel -j0 "(cd /home/pavel/Mail/{}/ && $GMI sync)" ::: thexcloud progin6304

I haven’t encountered any trouble with that solution so far (and I don’t see anything thread-unsafe in the lieer code), but I’ll keep an eye on that.

In principle, it shouldn’t be too hard to add a normal IMAP account as well with mbsync, but I expect it would require something like iterating through the directory structure and assigning notmuch labels based on that. I’ll probably try that some time in the future.

Emacs

With that done, I also want separate entries on the start page for each of the accounts. Doing that is easy enough, just modify the notmuch-saved-searches variable with customize-group or like this:

(setq notmuch-saved-searches
   '((:name "inbox (main)" :query "tag:inbox AND tag:main")
     (:name "unread (main)" :query "tag:unread AND tag:main")
     (:name "sent (main)" :query "tag:sent AND tag:main")
     (:name "all mail (main)" :query "tag:main")
     (:name "inbox (progin)" :query "tag:inbox AND tag:progin")
     (:name "unread (progin)" :query "tag:unread AND tag:progin")
     (:name "sent (progin)" :query "tag:sent AND tag:progin")
     (:name "all main (progin)" :query "tag:progin")
     (:name "drafts" :query "tag:draft")))

Notification for new messages

Now, we can send and receive mail, but we also probably want notifications for new emails. To do that, I wrote a simple script:

#!/bin/bash
# To run notify-send from cron
export DISPLAY=:0
# A file with last time of sync
CHECK_FILE="/home/pavel/Mail/.last_check"
QUERY="tag:unread"
ALL_QUERY="tag:unread"
# If the file exists, check also the new messages from the last sync
if [ -f "$CHECK_FILE" ]; then
    DATE=$(cat "$CHECK_FILE")
    QUERY="$QUERY and date:@$DATE.."
fi

notmuch new
NEW_UNREAD=$(notmuch count "$QUERY")
ALL_UNREAD=$(notmuch count "$ALL_QUERY")

# I don't really care if there are unread messages for which I've already seen a notification
if [ $NEW_UNREAD -gt 0 ]; then
    MAIN_UNREAD=$(notmuch count "tag:unread AND tag:main")
    PROGIN_UNREAD=$(notmuch count "tag:unread AND tag:progin")
    read -r -d '' NOTIFICATION <<EOM
$NEW_UNREAD new messages
$MAIN_UNREAD thexcloud@gmail.com
$PROGIN_UNREAD progin6304@gmail.com
$ALL_UNREAD total
EOM
    notify-send "New Mail" "$NOTIFICATION"
fi

# Save sync timestamp
echo "$(date +%s)" > $CHECK_FILE

The script is launched with cron every 5 minutes:

*/5 * * * * bash /home/pavel/bin/scripts/check-email

Here’s how the notification looks like:

Caveats

  • lieer has an extensive list of caveats concerning Gmail API
  • Make sure that you understand the implications of lieer’s --ignore-tags-locally and --ignore-tags-remote
  • If two of your accounts receive the same email, it will be stored as one email in notmuch, so tags from these accounts will be merged and pushed back on the next sync. To solve that, you can set tags from one account to be ignored on the rest of the accounts
  • A sent email is being downloaded again on the next sync. Not a great deal, but it is somewhat annoying to download recently sent attachments.
Table of Contents