In-house email subscriptions

In a move that’s typical of Google, Feedburner received the “update” treatment two years ago: its support for RSS-to-email notifications was turned down. I had been relying on this service to offer email subscriptions for this blog for many years, but as the service vanished, I had to find a replacement.

As I did not have a lot of time nor desire to assess alternatives, I took the bait from a salesperson from follow.it and I moved my subscribers to their free service. And let me tell you that… I have not been happy with it. If you have received any of their emails, you well know that they look like spam: my content appears as just one tiny text line among various sketchy links and large unrelated images. In fact, I have been ashamed of these emails.

Was there an easy solution to this problem? Sure. I could have upgraded to the “Super-cool” follow.it plan in order to customize the look-and-feel of the emails… but paying a subscription just for that one feature seemed out of proportion: there are other features I want to offer in this blog that a static site cannot provide, and if I had to pay for one, I’d rather get others too.

Over this same time period, you might recall that I have been working on a little service to support the dynamic aspects of this blog. In particular, this service has been taking care of privacy-respecting analytics, post voting, and comments. Its name: EndTRACKER—an awful name in retrospect because invasively tracking readers is the last thing I want to do, so the name has to change.

Given that I already had a custom service supporting this blog, I thought the obvious: why not make it handle email subscriptions too? That would be the last feature I needed to have a dynamic-feeling blog without abandoning the static page builder I use. And thus I got my hands dirty to make this work.

The first thing I did was to add the subscription and unsubscription flows. This was ready pretty quickly because I could reuse code from the EndBASIC Service. Thus, at the beginning of May, I dropped follow.it altogether. As a little bonus of this migration, I was able to display the total number of subscribers next to the subscription box, and also add a subscription box to the EndBASIC web site “for free”.

But then what? After I made that (premature) move to a custom subscription process… the next problem arose: how could I actually send notifications to you all every time there is a new post?

There are many trivial solutions to this problem, like manually copying the (small) list of subscribers into Outlook and composing an email, or creating a Google Groups read-only mailing list, or giving up altogether on self-hosting and moving to Substack. But… none of these felt right. I wanted automation, so automation I built.

Since then, I’ve been busy at work in my very-limited free time coming up with the necessary automation to scrape the RSS feed of an arbitrary site and to reliably send email notifications for any new entries to that site’s subscribers.

Does that sound like a trivial (and fun!) feature to build? Yes, yes it does. But making it work reliably and safely has been incredibly time-consuming. To give you a glimpse of what was involved:

• I started by creating a persistent task processing queue (because why not): a service that monitors a stored description of tasks and executes them in the context of a serverless runtime engine, with automatic retries if things go wrong.
• Then I bundled this new service into EndTRACKER to run periodic tasks and defined 1. a task to scrape RSS feeds, storing their processed contents into a database, and 2. another task that fans out and schedules finer-grained tasks for individual email notifications.
• Then I had to add safety measures to abort early if the notification logic decides to do the wrong thing (just in case).
• Then I had to add throttles to not exceed daily email submission limits.
• Then I had to write a small HTML processor to sanitize the content of the feed items before injecting them into emails.
• While, all along, writing a lot of test cases: 40% of the code supporting this feature are tests because I’m really concerned about the automation going rogue and spamming you all. And I still feel that test coverage is insufficient.

All of this so that you can be notified of this and future posts in the form of concise, spam-free emails. Yay?

In the next post, I’ll take a deep dive into the teeny-tiny persistent task queue service I wrote, which I’ve open-sourced as part of III-IV. This service is written in Rust, runs in the context of the Azure Functions serverless runtime, and is backed by PostgreSQL.

Oh, and by the way: I’m planning on opening up EndTRACKER to fellow static site creators: if you have a static blog and have wanted to add select dynamic features to it, this may be the right choice for you! EndTRACKER is intended to be a collection of REST services you can pick and choose from, allowing you to integrate them into your site with whatever look-and-feel you like, all while providing a hosted admin interface. Take a look at the new WIP website, which currently (only) includes a demo of the statistics features. Subscribe to updates to receive an early-adopter invite when it’s ready to launch!