Maintaining a zero-maintenance website
I just shut down www.bookfetch.co.nz after the maintenance burden became too big.
Bookfetch was always supposed to run with minimal intervention, so reducing maintenance cost was a big guiding principle in the development I did. But despite my best intentions, so many things still needed fixing and updating along the way.
Here’s the maintenance log for my zero-maintenance webapp.
November 2012 - Bookfetch is born
It’s a price comparison website: you tell it a book, and it tells you how much that book costs at different retailers. The site get book details and prices from a combination of scraping and official APIs.
The site is hosted on Google App Engine, a hosting service that I don’t have to administer. The hope is that the managed App Engine platform will mean I don’t have to do any server maintenance.
November 2012 - Google deprecates datastore
Mere days after I launch Google deprecates the datastore library I’m using. I dutifully rewrite Bookfetch’s models to use the new ndb library. A sign of things to come.
March 2013 - Barnes & Noble update their website…
…breaking my scraper. A few months later I give up on trying to keep scrapers up to date, instead just removing the retailer if its scraper fails. There are enough retailers with stable APIs to keep the site useful.
April 2014 - Book Depository merges their .co.uk and .com sites
This affects the book details API, price API, and affiliate link format.
June 2014 - HTTPS becomes a de-facto requirement
Over a couple of years, the types of websites that need encryption has gone from “just banks” to “all of them”.
In 2014 there are are two options for free SSL certificates: CloudFlare and StartSSL. I randomly wisely choose CloudFlare, which is just as well because in 2016 all major browsers stop trusting StartSSL certificates after they are found guilty of shenanigans.
June 2014 - I corrupt both my Dropbox backup and git repo
After treading unknowingly on thin ice for two years, I learn why you aren’t supposed to keep a git repository in Dropbox. I can’t remember exactly what happened or how I fixed it. I do know there was a lot of swearing in my git commit log around that time.
April 2017 - Google deprecates App Engine SDK
The App Engine SDK – used for uploading code and running a development server – is retired in favour of the Google Cloud SDK. The new SDK has a command for uploading code but not for running a development server.
The documentation for running a dev sever appears a few week after the old SDK download link is removed.
May 2017 - fishpond.co.nz starts returning $0 as the price for some books…
…causing rampant numerical issues.
Nov 2017 - Yahoo terminates currency conversion API
I don’t notice for months due to strong caching and weak logging. Move to new provider.
Jan 2020 - Amazon restricts API usage
Amazon limits usage of its Product API based on the number of sales driven by your account. I don’t refer enough sales so I get errors on the API, which is used for the Bookfetch book search.
With the Amazon book details API failing, I only now realise that the Google book details API changed years ago to requiring authentication, and the Book Depository API had been offline since the UK merger in 2014. I didn’t notice at the time because I’d designed book details fetching to be Zero-Maintenance: if one API failed I just tried the next one. Well now all three were broken.
Jan 2020 - Bookfetch is retired
Book search no longer works.
Of the 30 retailers I set up in the beginning, only 5 are left. All but 2 pulled their affiliate program over the last 8 years.
It was sad to finally pull the plug. Bookfetch taught me webdev. It landed me my first internship and my first job. It had long ceased to be profitable on an hour-spent basis, but the Amazon API change was the last straw.
I’ll miss you Bookfetch.
Takeaways
- A managed service (like Google App Engine) is a maintenance tradeoff. You don’t need to think about updates, scaling, or hardware. Instead you’re at the mercy of product managers and API deprecators. Despite the turmoil running Bookfetch I’d host it again on App Engine in a heartbeat, I got off lightly compared to 8 years of devops.
- Scrapers need ongoing maintenance. I knew this going in, but now I truly know it.
- Stacking unreliable scrapers or 3rd party APIs improves reliability. As does caching. But you still need to log and alert failures, otherwise you’ll be left surprised when the last scraper in the stack dies or the cache expires.
- Given enough time, 3rd parties will get bored of providing you with a service. This counts double if you’re not paying for that service. When redesigning this blog in January 2020 I removed or self-hosted all critical 3rd party components: maps, fonts, JS libraries.