Project Reclaim and the email dilemma

One of the main 2011 goals for Project Reclaim is to get my email out of Gmail. Heavy reliance on Gmail raises a number of red flags. For one thing, email is central to my business and personal life online, and provides the best archive of my online past (get the important stuff first). For another, Gmail is ad-supported, in a way that has rankled since Gmail went public: it “reads” your email and serves ads based on what it finds. No one really talks about it anymore, but it still kind of bugs me – so I want to move to a non-free system (paying is better than getting something for free).

It’s taken me a while to make the move, though, for two main reasons.

  1. Email is tricky. Good, free mail server software is easy to find. But it’s not necessarily easy to set up and maintain. If the outgoing server isn’t configured correctly, your messages will get marked as spam. If you haven’t got constantly monitored spam filters on your incoming mail, you’ll be inundated with garbage. And the issues of backups and reliability, while certainly important in the case of (say) self-hosted websites, are many times more important with email: if the server goes down, emails may get altogether lost in the ether.

    I’ve set up and configured email servers before, and it hasn’t been very fun. When deciding how to solve the Gmail conundrum, I needed to take this fact into consideration. I started to do a bit of research on paid email hosting, and found good reviews of Rackspace’s hosted email service. The service is pretty affordable, and I knew from years of Slicehost use (now owned by Rackspace) that customer service and support would be good.

  2. I needed a good address. I own a lot of domain names, but most of them are lame, and none lent themselves very neatly to an email address. For instance, when your domain name is boonebgorges.com, what’s the email account name? ‘boone’? The cool factor there is pretty low. And I am a cool guy, so that’s important.

    Some of the obvious domains are taken. boone.com is wasted on dry-erase boards. gorges.com could never be wrested from the clutches of “one of the oldest family owned Volvo franchises in the United States”. But there was hope – or should I say habĂ­a esperanza – that I might get the fairly unused gorg.es. In fact, my brother and I had been working on that project for a couple of years, but it was only a few months ago that the owner finally relented, and the domain name was transferred to the Gorges boys.

So, about two months ago, I made the switch. For now, I just set it up as another account in Thunderbird (more on my Thunderbird setup). I created a generic “Archive” directory on my gorg.es account (to mimic Gmail’s All Mail) and pointed my ‘Y’ shortcut to that directory. I’m using K-9 Mail on my Android phone, which I set up to save the entire Archive directory, so I’d have good local email search on my phone. Little by little, I’m moving over my email correspondence to the new, awesome address. Bye bye, Gmail!

Building a baby photo site with WordPress

My wife and I just had our first baby, which is the occassion for much nachas and, by extension, picture sharing. Facebook is, for better or for worse (emphasis on the latter), the de facto place for such sharing to happen. For a number of reasons – a desire to be somewhat selective about who sees my family pictures, my Project Reclaim sensibilities, the fact that I don’t have a Facebook account and generally think that Facebook is an evil company – I don’t want to use FB for this purpose. As in the case of my Twitpic-like photoblog, I figured I could use WordPress to set something up that was nearly as seamless as Facebook, or Google+, or Flickr, or whatever.

The criteria

There were a few things I wanted out of the baby site.

  1. Easy (or zero) login for users
  2. Control over who has access
  3. Optional email notification for new content
  4. Easy, javascripty gallery browsing

When I started, I was pretty sure that I’d be able to get all of these things pretty easily, using existing WordPress plugins. I was both right and wrong about this: plugins exist for all of these purposes, but none of them were very easy to implement. As a result, I ended up building several pieces from scratch. I’ll go through each of the criteria, talk a bit more about what I was looking for, and then say something about how it was achieved. By doing this, and sharing the code (spoiler alert: https://github.com/boonebgorges/Hard-G/tree/master/wp-content), I’m hoping that I can help others with similar sensibilities to get started on their own sites.

Non-sucky registration and login

I love WordPress, and I understand the important reasoning behind the decisions that led to the design, but WP’s user registration system sucks. I didn’t want just anyone to be able to create and activate an account. I didn’t want users to have to click an activation link. I didn’t want users to have randomly generated passwords that would need changing later on. And I wanted users to have the option of logging in a non-WP way.

Several of these problems could be solved by using Facebook logins. I’m not willing to give my photos over to the horrific FB leviathan, but I’m happy to piggyback on their login APIs if it will save my family and friends a few headaches. I wanted my users to have the option of clicking a single button that would give my site the ability to provision them based on their persistent Facebook login.

I started by looking at some popular Facebook Connect plugins from the wordpress.org plugin repository. I didn’t really like them. Most were linked to the Comments section of blog posts, while I wanted to use the logins for overall site access. Most were dependent on Javascript for logins, while I wanted to handle logins on the server side. Most used an outdated version of FB’s API (or at least of the PHP API classes that FB offers). And, to be blunt, most were too much of a mess, having been retrofitted many times over, and as a result next to impossible to extend. I tried modifying one or two of the more popular FB-WP plugins to do what I wanted, but I ended up writing so much garbage spaghetti code that I decided to cut my losses and start from scratch.

So I boned up a bit on the FB API, and wrote a small plugin that I call Wally Login. Together with the registration page template from my custom theme (a child of TwentyEleven), it does a couple of key things.

Your choice

Your choice

  • Rudimentary access control · If a non-logged-in user tries to visit any page on the site (other than the registration page), he is redirected to the Register page.
  • FB login integration · If a user clicks the “Log me in using Facebook” link, they’re directed to the FB authorization page for my website (which is registered as a Facebook app). There, they’re asked to approve the app – a one-time process – and are then returned to my site. Based on the display name, email address, etc that I get from FB, I create a WP user corresponding to the FB account. On future visits, approved users who are logged into Facebook will be automatically logged into my WP site whenever they visit it (an important point, because FB cookies are persistent over browser sessions, while WP logins, by default, are not). As a result, in the best-case scenario, a user will authorize their FB account with my site one time, and will never again have to think about authorization on Wally’s page.
  • A customized WP registration process · If users opt not to go the FB route, they can create a WP account directly on the site. I wanted to avoid sending users to an unthemed wp-login.php or wp-signup.php page, so I cribbed a few lines of code from BuddyPress and made my own registration and login dialogs. Wally’s site is part of a larger WP network, but I wanted to bypass WPMS’s built-in registration stuff (which requires users to activate their accounts, and is thus generally too hard for newbies to grok). My custom registration therefore creates the user directly (with wp_insert_user()), using a password that he provides, and skips the activation email. (By bypassing account activation, I’m removing an important spam prevention tool. More on that in the next section.)
  • Customized email notifications · Because I’m not using the built-in registration process, I needed to write my own email notifications for account applications and approvals.

If you decide to use my code, keep in mind that it’s not particulary beautiful. I wrote it for my own use, which means that it will take a bit of elbow grease to get it to work on your own site. In particular, if I were writing something for more general distribution, I would not be so reliant on a theme template as I am here. But if you’re looking to create a site like mine, this code is a great place to start – especially the FB integration stuff, which has made the registration and login process about as smooth as it can get.

Access control

The final important thing that the Wally Login plugin does is to provide me (the site admin) with control over who has access to the site. There are a couple ways I could have approached this issue. One is to whitelist users ahead of time. The problem with this is that I’m bound to forget some names, get email addresses wrong, and run into other problems that stem from my unfortunate lack of omniscience. Another strategy is the invitation code. When unique to the individual, this method suffers from the same drawbacks of the whitelist; when non-unique (ie when everyone uses the same invitation code) it takes away much of the security, as the code can be passed around quite easily; either way, invitation codes are clunky, easily misplaced, and all too often mistyped.

Thanks for applying

Thanks for applying

As a result, I ended up going with a third option: an application and approval process. Here’s the idea, conceptually. Anyone who wants can create an account on my site (either through Facebook or natively; see the previous section). However, the account does not actually allow access to the site unless the account is also approved by the administrators. Thus, after the initial application, two emails are sent: one to the applicant saying “thanks for applying, please be patient”, and one to me saying “there’s a new applicant, please approve them”. Then I go to my approval interface and click the Approve button (if I want), which marks the user as approved in my database and sends them an email saying “You’re in!”

Here’s a brief description of how it works technically. All applicants have a WP account created for them. Every new account is marked, at the time of creation, with user_status = ‘2’, and I make sure that no page other than Register can be viewed by an account with user_status = ‘2’. In this way, I am turning the idea of activation around a bit – natively, WP makes the user do the activation, but in my case I do it. The admin tool I use to activate users is my Unconfirmed plugin, designed for a slightly different purpose but quite at home here. (For technical reasons, Unconfirmed needs users to have activation keys; thus Wally Login also generates some dummy keys during the user creation process so that Unconfirmed will work right.) Unconfirmed, in turn, does the work of flipping user_status to 0 upon approval.

Taken together, Wally Login and Unconfirmed (with custom WP registration, FB integration, user approval by admin, etc) has given me a comfortable level of access control, without making the process unduly difficult for my users.

Email notification for new content

One of the biggest drawbacks of creating a standalone picture site instead of using an all-purpose social network (in practice, this means Facebook) is that the standalone site is likely to be forgotten. FB collects all of your network’s activity into a single stream; it’s highly unlikely, on the other hand, that Wally’s site will become part of anyone’s daily routine, so that they stop by to check for new content. For that reason, good email notification of new content is essential to making the site work.

Dead simple email subscription

Dead simple email subscription

I first tried using the popular Subscribe2 to handle these notifications. But I ran into a bunch of problems. For one thing, I didn’t like that S2’s subscription management happened in the Dashboard – I want to keep my users on the front end. S2’s category-based subscription is too complicated for my site, where people are either going to want to subscribe to all posts or to none at all. And the widget that comes with S2, for display on the front-end of the site, is pretty much atrocious. (Sorry. The rest of the plugin is nice. But that widget sucks.) At first I tried solving these problems just by building my own widget for S2, one that would tell the user whether he was currently subscribed, and show an Unsubscribe/Subscribe button, as appropriate. But, given the structure of S2’s data (which is somewhat arcane, and in any case far too complicated for my purposes), it ended up being a lot harder than it should have been.

So – wait for it – I wrote something from scratch. It is dead simple. Two parts: (1) a widget, which does exactly what I describe in the foregoing paragraph; and (2) hooks into publish_post to send an email to all subscribed users (along with some gentle checks to make sure dupes are not sent). This plugin has no admin UI and no options, because I don’t need any of those things.

Pretty galleries

Since the main point of the site would be to look at lots of pictures, it was quite important to have an easy, pretty way to do so. By “easy” I mean, primarily, navigable by keyboard; by “pretty”, I mean, primarily, bigger than the content area of a typical blog post. Less important, but still desirable, was the admin interface: I wanted it to be easy to upload lots of pictures at once, to add captions and other metadata if necessary, and to turn it all into a gallery that would look good on the front end.

Pretty, easy

Pretty, easy

You know the drill: I tried a couple of the more popular free plugins, but all of them were annoying in one way or another, and each one was way overengineered for my meager needs. I was especially disappointed by the back-end admin for the popular gallery plugins, which I found lugubrious, unintuitive, and impossible to extend. After some consideration, I decided that I actually preferred WP’s native Add Media interface for uploading photos and adding metadata, and that I was perfectly happy with the way that WP’s gallery shortcode displayed content on the front end, at least when viewing thumbnails.

So the only thing I really needed was to implement the javascript that would allow for keyboard navigation and lightboxing of gallery photos. Thanks in part to his extremely uncreative and literal plugin naming schema, I found Viper007Bond’s jQuery Lightbox For Native Galleries plugin. It does almost exactly what I want, right out of the box.

I did make a few minor mods, though. First, the plugin is a bit greedy in the way that it filters the output of get_attachment_link(), which was either breaking things (as in the case of comment_post_redirect on attachment posts) or making it hard to display links to the attachment page instead of the raw attachment file. The former problem I solved with a filter; for the latter problem, I was a bit lazy, so I modded the plugin itself in addition to adding an explicit ‘lightbox’ class to attachment links. This combination of hacks makes it work perfectly for my purposes.

Odds and ends

A little bonus

A little bonus

With my absolute requirements met, I was able to add a few other goodies to the site. My theme is a child of Twenty Eleven, which I’m pretty much using as-is. But I’ve added a few fun bits. First, on each attachment page, I added Download links, so that users could download images of various resolutions for printing or editing. I messed with the WP Admin Bar so that users coming from Facebook wouldn’t see Log Out and some other inappropriate links. And under each thumbnail in Gallery view, I’ve added Download/Comments links, so that users could bypass the jQuery lightbox and go straight to the attachment permalink if they wanted.

It took some work, but I think I’ve ended up with a site that is nice to use and easy to maintain, without resorting to the extreme discomfort associated with Facebook. Hooray!

Making the Thunderbird interface more Gmail-y

As part of Project Reclaim, I’m gearing up to move my email off of Gmail and onto my own server. Email is, and long has been, central to my life online – it’s my main point of contact for so many personal and professional connections, and my email archives are the closest thing I’ve got to a record of my online activity. So I’m keen to make the move as smooth as I can.

For that reason, I’m handling the transition in stages. The first stage involves transitioning email client software away from the Gmail website.

Choosing a client

I’ve chosen Thunderbird as my alternative, for a couple reasons.

  1. It’s open-source.
  2. It’s highly extensible and customizable.
  3. It works across platforms. That’s important, because I’ll be migrating away from OSX.

Choosing Thunderbird is not without its sacrifices. For one thing, moving to an OS-native application, rather than an app that runs in a browser window like Gmail does, means that I’ll no longer be able to count on having a consistent UI and feature set between different devices. In some cases, this is not a huge loss. The UI for Gmail on my Android phone is really quite different from the normal web version, and it’s never bothered me very much. My biggest worry, though, is that I’ll have multiple workstations – a primary work machine and a netbook, for example – with different email setups. I’m hoping to mitigate the problem by coming up with some idiot-proof backup and syncing methods for the fairly small number of files that comprise Thunderbird’s configuration. (This’ll be necessary for other software transitions as well, like my gradual move to Vim. It’d be quite easy with Dropbox and some strategically-placed symbolic links, but I’m trying to break the Dropbox habit too :) )

It’s a considerable comfort that, now that I have a smart phone (and am thus no longer reliant on public or borrowed computers for email access), the vast, vast majority of my email use is centered on a handful of devices, all of which I own. The last time I checked my email on a device other than my own was probably three years ago. Weird, now that I think of it.

The other sacrifice is related to UX. I happen to like Gmail’s interface. In particular, I’ve grown quite used to Gmail’s thoughtful keyboard shortcuts, which make it possible to do nearly navigation without touching the mouse. Coming up with a reasonable facsimile of these shortcuts in Thunderbird would be the biggest part of my configuration process.

Keyboard Shortcuts

Thunderbird has pretty good keyboard shortcuts out of the box. I didn’t feel like learning a whole new system, though, so I wanted a way to map Gmail-style shortcuts onto Thunderbird. There used to be a Thunderbird extension to do just that, called GmailUI. But the GmailUI website suggests that the extension is only compatible with Thunderbird versions 0.8-2.0 (Thunderbird’s currently in 3.1), which would explain why the extension doesn’t show up in a search on tho Mozilla repo (the “Expression Search” plugin does come up, which is a fork of a part of GmailUI that I’ll talk about in a minute – but it doesn’t do keyboard shortcuts).

So I looked for a more general method for customizing Thunderbird’s keyboard shortcuts, and found it with Keyconfig. It’s pretty straightforward to remap keystrokes using Keyconfig (Tools > Keyconfig), though it can be a bit of a pain because many of the standard Gmail shortcuts (like j and k for up/down navigation) are already in use by Thunderbird, so that changing one shortcut often means making two. You might find it helpful to borrow my configuration, which I’ve pasted below. Add these lines to your user.js config file (create if it doesn’t exist):

	user_pref("keyconfig.main.key_killThread", "][I][");
	user_pref("keyconfig.main.key_markJunk", "meta shift][J][");
	user_pref("keyconfig.main.key_markReadByDate", "meta shift][D][");
	user_pref("keyconfig.main.key_markThreadAsRead", "meta shift][R][");
	user_pref("keyconfig.main.key_newMessage", "meta shift][M][");
	user_pref("keyconfig.main.key_newMessage2", "meta shift][N][");
	user_pref("keyconfig.main.key_nextMsg", "][J][");
	user_pref("keyconfig.main.key_previousMsg", "][K][");
	user_pref("keyconfig.main.key_reply", "][R][");
	user_pref("keyconfig.main.key_replyall", "shift][R][");
	user_pref("keyconfig.main.key_toggleMessagePane", "][V][");
	user_pref("keyconfig.main.xxx_key74_SwitchPaneFocus(event);", "][D][");
	

Briefly, this does the following. First, it maps some familiar keystroke combos from Gmail to Thunderbird: j and k to up and down, and r and R to Reply and Reply To All. Second, because two-stroke codes from Gmail (like gi for Go To > Inbox) don’t seem to be supported natively by Thunderbird, I’ve mapped D to SwitchPaneFocus, which lets me get back and forth between the folders pane, the message list pane, and the single message pane, for easier navigation. As I get more comfortable with it, I might write my own extension that ports over some of the other most convenient Gmail shortcuts, but for now this covers a good 80% of what I might regularly use.

The other big shortcut missing from Thunderbird is y, which is the Archive command in Gmail. For that purpose, I installed Nostalgy, which allows you to move messages with keyboard shortcuts. I don’t think I’ve set this up in 100% the right way, but here’s how I’ve approximated Gmail’s y using Nostalgy. First, at Tools > Nostalgy > Keys, I’ve set ‘Save message’ to ‘shift Y’ and ‘Save as suggested’ to ‘Y’. ‘Save as suggested’ seems to work on a session basis; it suggests the folders that you’ve used since the last time you started Thunderbird. Thus, every time I start Thunderbird, the first time I want to archive a message, I use the more verbose shift-Y. A dialog at the bottom of the window suggests places where I might put the message; I select Gmail’s All Mail folder. Then, the next time I want to archive a message, I can use y by itself to go to the suggested (i.e. the last-used) location. Since I just throw all of my email into All Mail – no complex tagging or organization – this is all I need.

Bonus! Offline Access and New Email Throttling

Moving to a non-web-based email client is not all bad. For one thing, moving to a local application means that my email – archives and all – are available offline. I know that Google Gears and some of the new HTML5 goodies mitigate the issue somewhat with respect to Gmail, but offline access for a webapp is always going to be something of a hack.

On a related note, one thing that I have always hated about using Gmail is how it weaves together the process of checking for new email and accessing old email. I like to check for new mail at specified intervals only (once every few hours); anything more than that is extremely distracting. In Gmail, this meant closing the browser tab. Yet, fairly frequently, I find that I need to access an older email from my archive in order to do a specific task. In Gmail, this meant checking my new email as well. Now that I’ve moved to Thunderbird, I can access my archive without checking for new mail – the way it ought to be!

Setting it up this way was not straightforward. There are a few things you’ve got to do:

  1. In Tools > Account Settings > Server Settings, configure the ‘Check for new messages at startup’ and ‘Check for new messages every x minutes’ however you’d like. I have them both disabled, so that email can only be checked manually.
  2. Here’s the tricky part: Gmail, in its futuristic wisdom, uses a special protocol called IDLE to push new email to remote clients – bypassing the settings from step (1). (This took me half a day to figure out.) Disable this feature at Tools > Account Settings > Server Settings > Advanced > ‘Use IDLE command if the server supports it’.

I’m planning to spend a few weeks improving and getting used to this setup before starting the migration to self-hosted email.

Related posts:

  1. Project Reclaim and the email dilemma
  2. Automated and redundant Wordpress backup via email
  3. Making Userthemes work on WordPress 3.0