March 2011 – Project Reclaim

Making the Thunderbird interface more Gmail-y

As part of Project Reclaim, I’m gearing up to move my email off of Gmail and onto my own server. Email is, and long has been, central to my life online – it’s my main point of contact for so many personal and professional connections, and my email archives are the closest thing I’ve got to a record of my online activity. So I’m keen to make the move as smooth as I can.

For that reason, I’m handling the transition in stages. The first stage involves transitioning email client software away from the Gmail website.

Choosing a client

I’ve chosen Thunderbird as my alternative, for a couple reasons.

It’s open-source.
It’s highly extensible and customizable.
It works across platforms. That’s important, because I’ll be migrating away from OSX.

Choosing Thunderbird is not without its sacrifices. For one thing, moving to an OS-native application, rather than an app that runs in a browser window like Gmail does, means that I’ll no longer be able to count on having a consistent UI and feature set between different devices. In some cases, this is not a huge loss. The UI for Gmail on my Android phone is really quite different from the normal web version, and it’s never bothered me very much. My biggest worry, though, is that I’ll have multiple workstations – a primary work machine and a netbook, for example – with different email setups. I’m hoping to mitigate the problem by coming up with some idiot-proof backup and syncing methods for the fairly small number of files that comprise Thunderbird’s configuration. (This’ll be necessary for other software transitions as well, like my gradual move to Vim. It’d be quite easy with Dropbox and some strategically-placed symbolic links, but I’m trying to break the Dropbox habit too :) )

It’s a considerable comfort that, now that I have a smart phone (and am thus no longer reliant on public or borrowed computers for email access), the vast, vast majority of my email use is centered on a handful of devices, all of which I own. The last time I checked my email on a device other than my own was probably three years ago. Weird, now that I think of it.

The other sacrifice is related to UX. I happen to like Gmail’s interface. In particular, I’ve grown quite used to Gmail’s thoughtful keyboard shortcuts, which make it possible to do nearly navigation without touching the mouse. Coming up with a reasonable facsimile of these shortcuts in Thunderbird would be the biggest part of my configuration process.

Keyboard Shortcuts

Thunderbird has pretty good keyboard shortcuts out of the box. I didn’t feel like learning a whole new system, though, so I wanted a way to map Gmail-style shortcuts onto Thunderbird. There used to be a Thunderbird extension to do just that, called GmailUI. But the GmailUI website suggests that the extension is only compatible with Thunderbird versions 0.8-2.0 (Thunderbird’s currently in 3.1), which would explain why the extension doesn’t show up in a search on tho Mozilla repo (the “Expression Search” plugin does come up, which is a fork of a part of GmailUI that I’ll talk about in a minute – but it doesn’t do keyboard shortcuts).

So I looked for a more general method for customizing Thunderbird’s keyboard shortcuts, and found it with Keyconfig. It’s pretty straightforward to remap keystrokes using Keyconfig (Tools > Keyconfig), though it can be a bit of a pain because many of the standard Gmail shortcuts (like j and k for up/down navigation) are already in use by Thunderbird, so that changing one shortcut often means making two. You might find it helpful to borrow my configuration, which I’ve pasted below. Add these lines to your user.js config file (create if it doesn’t exist):

	user_pref("keyconfig.main.key_killThread", "][I][");
	user_pref("keyconfig.main.key_markJunk", "meta shift][J][");
	user_pref("keyconfig.main.key_markReadByDate", "meta shift][D][");
	user_pref("keyconfig.main.key_markThreadAsRead", "meta shift][R][");
	user_pref("keyconfig.main.key_newMessage", "meta shift][M][");
	user_pref("keyconfig.main.key_newMessage2", "meta shift][N][");
	user_pref("keyconfig.main.key_nextMsg", "][J][");
	user_pref("keyconfig.main.key_previousMsg", "][K][");
	user_pref("keyconfig.main.key_reply", "][R][");
	user_pref("keyconfig.main.key_replyall", "shift][R][");
	user_pref("keyconfig.main.key_toggleMessagePane", "][V][");
	user_pref("keyconfig.main.xxx_key74_SwitchPaneFocus(event);", "][D][");

Briefly, this does the following. First, it maps some familiar keystroke combos from Gmail to Thunderbird: j and k to up and down, and r and R to Reply and Reply To All. Second, because two-stroke codes from Gmail (like gi for Go To > Inbox) don’t seem to be supported natively by Thunderbird, I’ve mapped D to SwitchPaneFocus, which lets me get back and forth between the folders pane, the message list pane, and the single message pane, for easier navigation. As I get more comfortable with it, I might write my own extension that ports over some of the other most convenient Gmail shortcuts, but for now this covers a good 80% of what I might regularly use.

The other big shortcut missing from Thunderbird is y, which is the Archive command in Gmail. For that purpose, I installed Nostalgy, which allows you to move messages with keyboard shortcuts. I don’t think I’ve set this up in 100% the right way, but here’s how I’ve approximated Gmail’s y using Nostalgy. First, at Tools > Nostalgy > Keys, I’ve set ‘Save message’ to ‘shift Y’ and ‘Save as suggested’ to ‘Y’. ‘Save as suggested’ seems to work on a session basis; it suggests the folders that you’ve used since the last time you started Thunderbird. Thus, every time I start Thunderbird, the first time I want to archive a message, I use the more verbose shift-Y. A dialog at the bottom of the window suggests places where I might put the message; I select Gmail’s All Mail folder. Then, the next time I want to archive a message, I can use y by itself to go to the suggested (i.e. the last-used) location. Since I just throw all of my email into All Mail – no complex tagging or organization – this is all I need.

Mimicking Search

One of the best things about Gmail its easy searchability. Gmail searches are generally fast, accurate, facet-able, and easy to navigate. Mimicking this in Thunderbird is, at one level, impossible; you’ll never have access to Google’s search algorithms. (Though probably this doesn’t matter much, since searching on a finite corpus like an email archive doesn’t require fancy logic like what’s in PageRank.) But I’ve got a few tips for making Thunderbird search a bit more like Gmail.

Searching vs filtering – The main search box, accessible by Cmd-K/Ctrl-K, is a true search. The nice thing about it is that it searches through all available metadata fields, and in all folders. The bad thing about it is that the search results interface, which comes up in a new tab, is totally different from the standard message viewing interface, and has (so far as I can tell) absolutely no keyboard shortcuts. As an alternative, you can use the Filter mode, which is available via the Quick Filter Bar (View > Toolbars > Quick Filter Bar). That allows you to filter the messages in your current view according to various search terms, within the default UI. If you’re like me, and you have an empty inbox and an All Mail file with all the things you want to search, the Filter method will mean switching to All Mail before you start searching.
Sort search results by date – A small annoyance: when you do a Cmd-K search, results are, by default, in order by Relevance. Usually this is not what I want (I’m looking for that person who emailed me about BuddyPress yesterday) – I’d be much better served by Date sorting. Search Results Sort By Date Not Relevance is an aptly-named extension to change this behavior.
Gmail-type expression searches – Gmail search has a handful of handy expressions for narrowing your search: from:x will search for ‘x’ in the From field only; attachment:yes will limit results to those that have attachments, etc. Enabling this sort of filtering in Thunderbird was part of the aforementioned GmailUI. A kind-hearted dev has ported it over to Expression Search, which adds the same syntax to a special Expression box on the Quick Filter bar. (This last point is important – I spent some time messing around, thinking that the extension was broken, because I was expecting it to interact with the main search bar. It works only on the Filter bar.)

As a side note related to search, I fear that one of the things I will lose irrevocably when I move away from Gmail as a provider is mobile search. Gmail searches take place on Gmail’s servers, so they work equally well on a mobile device. Thunderbird searches depend on local copies of the mail archive, which is impractical on a mobile phone. Suggestions for getting around this restriction are quite welcome.

Bonus! Offline Access and New Email Throttling

Moving to a non-web-based email client is not all bad. For one thing, moving to a local application means that my email – archives and all – are available offline. I know that Google Gears and some of the new HTML5 goodies mitigate the issue somewhat with respect to Gmail, but offline access for a webapp is always going to be something of a hack.

On a related note, one thing that I have always hated about using Gmail is how it weaves together the process of checking for new email and accessing old email. I like to check for new mail at specified intervals only (once every few hours); anything more than that is extremely distracting. In Gmail, this meant closing the browser tab. Yet, fairly frequently, I find that I need to access an older email from my archive in order to do a specific task. In Gmail, this meant checking my new email as well. Now that I’ve moved to Thunderbird, I can access my archive without checking for new mail – the way it ought to be!

Setting it up this way was not straightforward. There are a few things you’ve got to do:

In Tools > Account Settings > Server Settings, configure the ‘Check for new messages at startup’ and ‘Check for new messages every x minutes’ however you’d like. I have them both disabled, so that email can only be checked manually.
Here’s the tricky part: Gmail, in its futuristic wisdom, uses a special protocol called IDLE to push new email to remote clients – bypassing the settings from step (1). (This took me half a day to figure out.) Disable this feature at Tools > Account Settings > Server Settings > Advanced > ‘Use IDLE command if the server supports it’.

I’m planning to spend a few weeks improving and getting used to this setup before starting the migration to self-hosted email.

Project Reclaim

Update: I have begun aggregating these posts at projectreclaim.net.

Lately I have been feeling increasingly uneasy about the state of my digital affairs. I am a leader on a number of open source software projects that pride themselves on, among other things, their ability to enable users to “own their own data”. Moreover, I am trained as a philosopher, and have spent a pretty fair amount of time reading and thinking carefully about the nature of data and our relationships with it. If anyone is in a position to develop and advocate for good models of digital independence, I am.

Yet, when I look around my digital world, I see instance after instance where I am, to a greater or lesser extent, completely reliant on the good will of commercial entities and their propietary systems. To wit:

My Twitter account is a big part of my online identity
The last five years of my private correspondence, personal and professional, is in Gmail
I use Dropbox for syncing documents between devices (like my blog_sandbox.txt file, where I’m writing this post!)
I use Picasa Web Albums to back up and share photos
I have a Mozy account to back up the rest of my important files
Until recently, I had an iPhone. I still use a Mac
I use Remember The Milk for task management
I store source code for all my projects in Github

Some of these are products; some are services. Some are free; some of them I pay for. And – for sure – some of the companies behind the products and services listed above are more evil than others. So I don’t want to pretend that my reliance on each of them is equally bad. But each item on this list plays a crucial role in my digital life, and each one of them operates in a way that is beyond my control, both literally (I can’t modify the source code) and more figuratively (questions about ownership, exportability, transportability are icky).

I’m planning to extricate myself.

Project Reclaim

In order to make it sound a bit fancier, I’m giving my project a name: Project Reclaim. ‘Reclaim’ because it’s a manifestation of my desire to fight the inertia that leads us to give up control over our computing experiences, my desire to reclaim control and ownership. ‘Project’ because this will be hard, and ongoing. And why give it a name at all? I’m hoping that, by being public about it – putting my experiences in a series of blog posts and tweets under a common tag – that I’ll be able to hold myself accountable, and hopefully guide others who are hoping to reclaim their lives a bit as well.

In short, Project Reclaim is the process of weaning oneself off of digital platforms that are closed source and/or under the control of others.

Methodology

How will Project Reclaim actually work?

Assess the situation I’ll first need a way of figuring out which systems and platforms are worth moving away from, what their replacements should be, and in what order I should effect the transition. I’ve got a few rules of thumb.
- Open source is better than closed source. I write open-source software for a living. I believe that, on balance, it makes better software. And I believe that using software where one has access to the source code is a necessary component of maximizing one’s digital autonomy. Thus: if the third-party system I’m currently using is also a benefactor of open-source communities (like, say, wordpress.com), it makes it less urgent to move away. And, when selecting replacements, select open source if at all possible.
- Paying is better than getting something for free This might seem like a contrast to the previous rule, but I don’t think it is. When you use a free service, somebody’s paying the bills. Usually that means targeted advertising – think Facebook and Gmail. Paying service fees, on the other hand, and agreeing to the contract that comes with it, generally has the effect of making the relationship more transparent. Of course, this is far from absolute, but it seems reasonable in a broad sense. Plus, I like to support developers and services that are truly valuable.
- Go for the low-hanging fruit In cases like email, there are well-established, straightforward (though not necessarily easy…ugh) ways of fending for yourself. No need to invent the wheel. On the other hand, some of the areas where alternatives are less obvious – social networking-type data springs to mind – also happen to be areas where I have some expertise and leverage. So, in those cases, it might be worth innovating.
- Get the important stuff first My email history is more important to me than my Twitter history; the convenience of Github is more valuable to me than the convenience of Dropbox. Plan the Reclaim accordingly.
- Get the vulnerable stuff first Recent statements by Twitter have made me think that the way I interact with the services is more subject to change in the upcoming months than, say, the way I interact with Gmail. That’s frightening. The more profit-hungry the company is – and, thus, the more disinclined to have the customer’s freedoms in mind – the more urgent it is to pull yourself out.
Clearly, some of these considerations are at odds with each other. But they give a rough framework for deciding whether, when, and how to carry out the mission of Project Reclaim.
Make the switch Here’s where the action happens: I do what I need to do to move myself to the replacement.
Write about it This weekend I spent an afternoon on the problem of Twitter, and I ran into a ton of technical problems that remain unresolved. I imagine that there will be similar hurdles for each part of the project. I’m hoping that, by writing about the problems (and, where they exist, the solutions) I can help other people to take some of the same steps themselves, or even to spur someone really smart to come up with better solutions than the ones that currently exist.

What I expect from myself

My goal, ultimately, is to move away from third-party, closed-source services and platforms altogether. It might take some time. So I’ll make some interim goals: by the end of 2011, I’m hoping to have my email moved, my microblogging federated, my own backup system on my own server space, and my computer running an open-source OS.

Even if I manage to meet this goal, there’s a very real sense in which Project Reclaim will necessarily be an exercise in futility. I’ll always have to buy server space, and who’s to say that Amazon or Slicehost won’t go berserk tomorrow? I’ll always have to connect to the internet, which leaves me perpetually at the mercy of the ISPs, who are IMO more evil than all of the other service providers put together. It’s a depressing state of affairs: the kind of autonomy I want might be impossible given the way that the economy works. I take some solace in the fact that philosophers have spilled much ink over the problem of free will without coming up with a clear formulation of exactly what kind of autonomy would be worth arguing for. At least I’m not alone in my delusion.

That said, it’s a fight that I feel I have a responsibility to fight. If I’m going to continue to argue for the use and development of open source software, I have to start putting my money where my mouth is. And so, to me, Project Reclaim is less about my being a paragon of virtue, and more about my wanting to sleep a bit better at night.

Do I think that everyone should do this? People should prefer open solutions to closed ones, all things being equal. But generally, all things are not equal. Most people don’t have the time to write their own software, to run their own servers. For those people, decisions about their digital life are (rightly, I think) made more on the basis of aesthetics and convenience than lofty concepts like Autonomy and Ownership. But there are a few considerations that are perhaps relevant for the kinds of people who read my blog:

Open source developers who tout the importance of data ownership and other such freedoms have a special responsibility to model best behavior in these areas.
Academics, more than anyone, should be sensitive to the dangers of leaving the crucial pieces of one’s online self in the hands of corporate entities. That’s true for personal artifacts like email, but perhaps doubly so for scholarly work that ought to be part of a public trust.
Educators, like open source developers, should model best practices, encouraging students to take control over their digital identities.

So, while I wouldn’t belabor the point for the average Joe, I do think that people who consider themselves members of one of these groups – as most people reading my blog probably do – that they should think carefully about their relationship with the tools and services that enable their digital existence.

To freedom!