Making Persistent Build Folders in Netlify

Static Site Generators are all-or-nothing. Each time they build a new version of the site, they throw away everything that was created before and start from scratch.

That’s usually what you want to ensure everything is up-to-date. But there are special cases when keeping parts of the previous build around makes sense. For example, If you fetch lots of data from an external source during your build, it might make sense to cache that data and re-use it again in the future.

I recently found such a case when working on the Eleventy webmentions feature. For each build, a script queries the webmention.io API and fetches all the webmentions for the site. That can be a lot of data - and most of it stays the same, so fetching everything new again each time is sort of wasteful.

# A Cache Folder

A better solution is to store the fetched webmentions locally in a separate _cache folder as JSON and give them a lastFetched timestamp. On the next go, we can load the old data straight from there and only query the API for webmentions newer than that timestamp.

My webmentions code does exactly that - but it had a big problem: that only worked locally. Since Netlify (where my site is hosted) throws everything out the window each time, I couldn’t use the cache there.

# Plugins to the Rescue

To edit anything related to the Netlify build process itself, you need a build plugin. There is a directory of plugins available for you to choose from, but you can also define your own plugins and deploy them alongside the rest of your code.

To define a custom plugin, make a new directory called plugins and within that, a new directory for your code:

_cache
_site
src
plugins
└── webmention-cache
    ├── index.js
    └── manifest.yml
package.json
netlify.toml

Your plugin should contain at least two files: a manifest with some metadata, and the actual plugin code.
For the manifest file, let’s just set a name:

# manifest.yml
name: webmention-cache

The meat of the plugin is in the index.js file. There are lots of things you could do here- but for this usecase, it’s enough to define an object with two functions. These are hooks that will be called on specific parts of the build process that Netlify runs.

Both functions will be given some arguments, and among them is the utils object we can use to access the internal build cache:

// index.js
module.exports = {
    // Before the build runs,
    // restore a directory we cached in a previous build.
    // Does not do anything if:
    //  - the directory already exists locally
    //  - the directory has never been cached
    async onPreBuild({ utils }) {
        await utils.cache.restore('./_cache')
    },
    // After the build is done,
    // cache directory for future builds.
    // Does not do anything if:
    //  - the directory does not exist
    async onPostBuild({ utils }) {
        await utils.cache.save('./_cache')
    }
}

1. The onPreBuild hook looks for a previously cached _cache folder and restores it within the build.
2. The onPostBuild hook takes the final build output, looks for changes in the _cache folder and saves it for later.

Because these hooks only look at changes that happen between the start and end of your build, your code needs to create the cache directory itself and write files to it as it runs. You can do that by using node’s filesystem functions, similiar to what I’ve done here.

# Register the Plugin

The last thing to do is to let the Netlify build script know you intend to use your plugin. You can register it with a line in your netlify.toml configuration file:

# netlify.toml
[[plugins]]
    package = "./plugins/webmention-cache"

Now, when you run a new build, you should see something like these lines in your deploy log:

2:02:08 PM: ❯ Loading plugins
2:02:08 PM:    - ./plugins/webmention-cache from netlify.toml
2:02:08 PM: ​
2:02:08 PM: ────────────────────────────────────────────────────────────────
2:02:08 PM:   1. onPreBuild command from ./plugins/webmention-cache         
2:02:08 PM: ────────────────────────────────────────────────────────────────
2:02:09 PM: ​
2:02:09 PM: (./plugins/webmention-cache onPreBuild completed in 302ms)
2:02:09 PM: ​
2:02:09 PM: ────────────────────────────────────────────────────────────────
2:02:09 PM:   2. build.command from netlify.toml                            
2:02:09 PM: ────────────────────────────────────────────────────────────────
2:02:09 PM: ​
2:02:09 PM: $ npm run build

...[lines omitted]...

2:02:12 PM: >>> 4240 webmentions loaded from cache
2:02:12 PM: >>> 6 new webmentions fetched from https://webmention.io/api
2:02:12 PM: >>> webmentions saved to _cache/webmentions.json

...[lines omitted]...

2:02:29 PM: ────────────────────────────────────────────────────────────────
2:02:29 PM:   3. onPostBuild command from ./plugins/webmention-cache        
2:02:29 PM: ────────────────────────────────────────────────────────────────
2:02:29 PM: ​
2:02:29 PM: (./plugins/webmention-cache onPostBuild completed in 53ms)

And that’s about it!