In the past I've raved about self-hosting your own content, but what about consuming others' content? The default way to go about this is to install a plethora of apps onto your phone and maybe have a few bookmarks in your browser, then proceed to tirelessly scroll through each feed searching for gems. Having lived through this experience for far too long, I can say not only that it doesn't work well, but it also leaves me exhausted.
Luckily an alternative to the madness has been around for quite a while now: Really Simple Syndication (RSS). Generally when people talk about using RSS they think of setting up an account on something like Feedly and adding a bunch of blogs and newspaper publications. This is great, however I wanted to take my RSS setup a step further by:
- Self-hosting all pieces of the stack, from aggregation to presentation.
- Treat RSS as my single 'feed' of content- essentially replacing all other scrolling on the web. This means replacing Twitter, Instagram, and even e-mail newsletters.
Why go through all of this effort? Mostly for fun, but also because I'm frustrated with the idea that companies are watching everything I do. I am keenly aware of much can be revealed about me through observing the content I consume, and self-hosting my means of discovering and reading content is another layer of protection against prying eyes. Remember kids, if the product you're using is free you're the product.
If you're busy and don't need the details, the tl;dr of my setup is that I hobbled together the following components to form a pretty robust way of aggregation most all of the content I consume into a nice RSS interface.
- FreshRSS as the RSS aggregator
- RSS Bridge to convert various social media feeds into RSS feeds (eg Twitter, Instagram, etc.)
- Kill The Newsletter to convert email newsletters into RSS feeds
- Reeder as the frontend client on my Mac and iPhone
Without further ado, let's get into the grimy details.
Whenever self-hosting, the first task to consider is where and how you are going to run everything. Generally for a project like this, a cheap VPS like Scaleway's Stardust instances would be the way to go, however I happen to have a dedicated IP address on my residential connection, a spare computer, and a strong desire control the stack all the way down to the bare metal (a potent combination).
The server itself runs Debian and Docker, which I found to be a fairly low-maintenance setup. To manage the complexity of running multiple services, I use docker-compose and keep everything within a single git repository which looks something like this:
docker-compose.yaml config/ caddy/ Caddyfile rss-bridge/ whitelist.txt volumes/ freshrss/ caddy/ scripts/ backup
Everything is checked into git with the exception of the
volumes directory, which is gitignored as it contains data that changes frequently/is generally too large for git to handle. Even so, I use
borgbackup to keep it backed up in case things go seriously wrong (which is handled by
scripts/backup, a simple bash script). I've published a sample repository here that you can check out to see what this looks like when it all comes together.
FreshRSS and RSS Bridge have HTTP frontends that need to be exposed to the internet at large, so we'll need to use a reverse proxy in order to properly expose them on different domains and to set up SSL. Normally I default to nginx for these sorts of things, but this time around I wanted to try something new and went with Caddy. I have to say, after a few months of running of this setup I've been very pleasantly surprised with it. The configuration file is stupid easy to deal with and I never once have needed to think about creating or renewing SSL certs (it handles it automatically with Let's Encrypt). Honestly, this section would be double the length if I were still with nginx, as setting up
The heart and soul of any RSS setup is the aggregator. This is the piece of software that periodically fetches each of your subscribed RSS feeds, stores the content, and keeps track of read status. I ended up defaulting to FreshRSS, as my RSS client of choice is Reeder and it comes with an option to use it as a backend out-of-the-box. That being said, the web UI that comes along with FreshRSS is pretty mediocre, so your mileage may vary if you don't have a separate client. Thankfully all of our common interactions like reading/managing subscriptions can be done through Reeder itself.
Setting up FreshRSS is pretty straightforward, just a single entry in our
freshrss: image: freshrss/freshrss volumes: - './volumes/freshrss/data:/var/www/FreshRSS/data' - './volumes/freshrss/ext:/var/www/FreshRSS/extensions' environment: - 'CRON_MIN=4,34' - 'TZ=America/New_York' expose: - 8080
Once it's running and accessible via Caddy you'll need to complete the setup process via the web interface (as described in the README) and then link it to Reeder (or whatever RSS frontend you end up choosing).
As mentioned, almost all interactions with FreshRSS can be handled via your RSS frontend, however there is one feature in particular that's only accessible in the web UI that I've found useful: keyword filtering. There are a few different blogs that I subscribe to which have topics I'm not too interested in. Things like announcing new podcast episodes or basically anything about cars. In the subscription editor of FreshRSS you can automatically mark articles with certain keywords as read, which is helpful for keeping the signal to noise ratio high on particularly active feeds:
Converting Social Networks to RSS
If you're just looking to aggregate RSS feeds you can probably stop there, as FreshRSS will do everything you need. Unfortunately I've found that many interesting people that I'd like to follow don't always publish on a platform that's RSS friendly. Mostly these are the major social networks, who are particularly protective about content because they want to keep you on their apps to serve ads.
I wasn't able to find a reliable hosted social-media-to-RSS converter readily available on the internet, however in my search I stumbled upon RSS-Bridge. This project appeared to fill in the gap nicely, as it's self-hosted, has a wide variety of plugins for converting various sources into RSS feeds, and appears to be actively maintained by the community. Set up is also pretty straightforward, just another line in our
rss-bridge: image: rssbridge/rss-bridge:latest container_name: rss_bridge volumes: - './config/rssbridge/whitelist.txt:/app/whitelist.txt' expose: - 80
Note that the whitelist is optional, I just use it to enable only the plugins I actually need (Instagram, Facebook, and Twitter) rather than having the giant list of things enabled.
Another important note is that Twitter and Instagram like to periodically change their layouts in order to break scrapers like RSS Bridge. Thankfully the RSS-Bridge community appears to be pretty on-top of these changes and updates things fairly quickly as they break. The only downside for me is that this also means I need to periodically update to the latest version, something that can be solved pretty easily with a quick bash script:
#!/usr/bin/env bash docker-compose kill rss-bridge docker-compose rm rss-bridge docker rmi rssbridge/rss-bridge docker-compose up -d rss-bridge
Email newsletters are generally the bane of my existence, however they've managed to become a common way of keeping up with various activist groups, artsy collectives, local businesses, etc. Rather than having them clog up my email inbox (and cause me to never read them), I found Kill The Newsletter to be a great service for converting them into RSS feeds I can subscribe to.
All you have to do is give it a feed name (which will show up in your RSS reader) and it'll output an email address to use with the newsletter and an RSS feed that you can subscribe to with FreshRSS. Easy peasy.
If you're looking to be an overachiever, Kill The Newsletter is actually an open source project, however I've yet to be bold enough to set up my own email infrastructure. For now, the hosted solution works great (thanks Leandro!).
Not too bad- right? If any of this is intimidating or ends up being too much effort, remember you can always bail and use something like Feedly to get yourself started. Transferring subscriptions between RSS aggregators is straightforward thanks to the OPML standard which seems to be universally supported among aggregators.
Of course, if you have any additional questions about my setup, please don't hesitate to drop me a line. Better yet, if you decide to start using RSS for yourself be sure to subscribe to my blog's feed.
Bonus: Feeds I like
In no particular order, here are some interesting things I follow:
- Hacker News RSS - more specifically the front page feed, with a filter of all posts > 250 points. This helps keep the signal to noise ratio high.
- Kenji López-Alt - Great cooking videos from the author of my favorite cookbook, The Food Lab. YouTube actually supports RSS out of the box, so you should just be able to paste in a channel URL into FreshRSS and have it subscribe like any other blog.
- Craig Mod - A guy who blogs about walking around Japan. His email newsletters have been a great way to travel vicariously during the pandemic.
- Mi Niu York - Not sure how I found this one, but it's an interesting weekly newsletter from Irene Pedruelo, a fellow New Yorker that seems to have great taste in articles, art, poetry, etc.