It is no secret that, in software development, the edit+build+test cycle must be as short as possible. The delay between saving a file and seeing the results has to be minimal and in the order of a few seconds, or else developers lose focus and productivity suffers.

It’s equally important to ensure that the code is held to certain quality standards. Compiler warnings, for example, are part of any compilation and catch a set of common problems. But there are a lot more health checks that can be performed, such as ensuring that the code matches predefined coding guidelines, running a more aggressive linter to catch bugs that compiler warnings don’t notice, or even using ASAN or TSAN to validate the code’s memory and thread safety.

The problem is: while compiler warnings come for free as part of the edit+build+test cycle, all of these other checks are expensive—and sometimes very expensive. To make matters worse, running these extra checks might require installing heavyweight tools that are not desirable on every developer’s machine.

So when do we run these extra checks? Ideally we would run them as part of every build because, if we get to the test stage with code that is not up to our standards, we may be wasting time validating “unclean” code. Unfortunately, if we make these checks part of the build itself, just like compiler warnings are, we lengthen the build times and slow the development cycle. Bad idea.

We can loosen our strictness and say that the code must only pass health checks at commit time, not at build time. Which is actually a good idea: even though health checks that catch runtime bugs may be very useful at every build, others, such as those that validate the coding style, are a distraction when trying to quickly iterate on a change.

Postponing the health checks from build time to commit time is thus a big improvement. And if we are talking about a centralized version control system, we’d be done because these checks would run on the server before accepting a commit. But that’s not how the world operates today: distributed version control systems like Git are the norm, and in these, commits happen locally. In fact, these systems encourage frequent, small commits. Sometimes they even replay commits en masse in an automated manner (git rebase). Thus if we add our health checks to a commit-time hook (pre-commit in Git parlance), we are almost back to square one: some routine operations become slow and we still need costly tools on every developer’s machine.

If we are left without the options of running these health checks at build time and at commit time, what then? We can offload these checks to a different machine and we can require these to run only at branch merge time. In popular terms: run the health checks when a Pull Request (PR) is created on GitHub and require that they pass before the PR can be merged. The unfortunate tradeoff here is that it might take a long while for a developer to go from local branch to PR creation, and thus it might take a long while for them to discover lingering problems in their code. This is an issue that can be solved with training though: get those developers to push their changes to a remote branch at regular intervals so that these checks run periodically. Or, really: make them work on smaller PRs.

In this post, I will outline how to leverage the “new” cool hotness, GitHub Actions, to run code health checks whenever someone creates a PR. To make the case study specific, I’ll talk about running Rust sanity checks via cargo clippy and cargo publish --dry-run, both of which are very expensive operations, and I will refer to EndBASIC’s configuration for real-world examples.

In a sense, this is the second edition of my “Offloading maintenance tasks to Travis CI” post from 2015 updated to the modern times. And I’m posting this now in preparation for another article that will build on this one—so make sure to come back in a few days 😉

Goal

Our goal is to:

1. Create a GitHub Actions workflow that runs costly code validation operations.
2. Require this workflow to pass for all incoming changes to the main branch of our Git repository.
3. Prevent the main branch from being tampered with via commits that didn’t go through these checks.

Let’s look at the steps involved in reaching these goals.

First: create health validation scripts

The first step in creating the GitHub Actions workflow is to write one or more scripts to execute the checks we want to run. For our example, we will create two separate scripts: one to validate the code style, and another to validate the code release readiness. We will use two separate scripts because these two steps require disjoint builds (debug vs. release), which means that we can and should run them in parallel via different remote jobs.

Let’s create the script to validate the code style and save it as .github/workflows/lint.sh:

#! /bin/sh
set -eux

rustup component add clippy
rustup component add rustfmt

# Run all of our project through the Clippy linter.
cargo clippy --all-features --all-targets -- -D

# Ensure all of our project's formatting maches our rustfmt configuration.
cargo fmt -- --check

And then let’s create the script to validate the code release readiness and save it as .github/workflows/package.sh:

#! /bin/sh
set -eux

# Validate that our code is always in a publishable state.
cargo publish --dry-run

Don’t forget to make these two scripts executable!

At this point, you should be able to manually run these scripts on your machine from the top-level directory of your project and get them to succeed. This won’t prove that they’ll work fine on the environments used by GitHub Actions, but it’ll get you closer to that goal.

Second: Create the GitHub Actions workflow

With our health validation scripts in place, we can proceed to define the GitHub Actions workflow. This workflow will contain two separate jobs, one for each of the scripts we defined, so that they can run in parallel.

Our configuration, which you can store as .github/workflows/health.yml (and which I took straight from EndBASIC), might look like this:

name: Health checks

on: [push, pull_request]

jobs:
    lint:
        runs-on: ubuntu-latest
        steps:
            - uses: actions-rs/toolchain@v1
              with:
                  profile: minimal
                  toolchain: stable
                  components: clippy, rustfmt
            - uses: actions/checkout@v2
            - run: ./.github/workflows/lint.sh
    package:
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v2
            - run: ./.github/workflows/package.sh

There are two key details to note from this code snippet:

• The first is that, while your project might support more than just Linux, you most likely don’t need to run the health checks on every platform you support. The outcome of validating the coding style, for example, is going to be the same no matter whether you run the test on Linux or on Windows. Therefore, pick the platform with the fastest runtime and restrict your health check jobs to run on it.

• The second is that you want to run the Rust code validation checks using the latest available build. You can see that I overrode the preinstalled Rust toolchain in the GitHub Actions executors with the latest stable build by using the actions-rs/toolchain@v1 action. This is because Clippy in particular keeps gaining new checks, and if you don’t keep up to date with them, your interactive builds might fail later on while the CI ones will not. (Whether keeping up with Clippy is a good idea or not is up for debate, but I’m doing that in my project so this is a good compromise.)

Third: Test and submit your changes

Getting scripts to successfully run on remote machines, whose environments don’t necessarily match your local machine, can be exasperating. It’s common to push a commit with your shiny new configuration and then see the workflow immediately fail remotely. And thus it’s common to end up with a chain of commits with messages like “One more try” and “Oh FFS, one more!” to see if the workflow will run.

This is normal. But you don’t want these intermediate commits to end up in the project’s history.

To prevent that from happening, and to avoid embarrassment, do all of this preparatory work in a separate branch, say health, and push that branch to the project’s repository in whichever nasty state you need. Do as many intermediate commits as necessary to make things work. When the GitHub Actions workflow finally passes, run a git rebase -i main to squash all commits into just one and fix the commit message. Only at that point merge the health branch into main.

Fourth: Keep the main branch clean

At this point, our GitHub Actions code health workflow is up and running. But if we aren’t careful, it is still easy to push changes to the main branch that haven’t gone through validation. If we do that, either intentionally or by mistake, the next person that submits a PR will encounter failures that are not due to their changes and be very frustrated. Which is a very bad situation to be in: you truly need HEAD to be green at all times.

Thus the last step in our ordeal is to protect the main branch (and any other authoritative branches that we maintain) so that they cannot be tampered with. And the reason we have to do this last is because we need the workflow to be in place.

For this step:

1. Go to the repository’s settings on GitHub.
2. Click on Branches.
3. Click on Add rule under Branch protection rules.
4. Enter the name of your main branch.
5. Select the Require status checks to pass before merging and then tick the workflow jobs we created.
6. (Optional but recommended:) Tick Include administrators so that nobody can make mistakes and tick Require branches to be up to date before merging to ensure the tests are fresh.

With that, you should be up and running.

You can see a sample failure to get a sense of the kinds of things this new workflow would catch and how things would look like.

And that’s all folks. In the next post, I’ll cover a teeny tiny trick that the Google stack supports to prevent premature PR merges and how we can replicate it under a scenario like this. See you soon. And happy 2021!