How we worked around some quirks and limitations of GitHub Actions to skip CI jobs that aren't necessary in certain scenarios, without breaking branch protection rules that normally require those CI jobs to succeed.
The Pants documentation site is generated from markdown sources in a top-level docs/ folder in the repo. So editing the docs requires a pull request, a review, a passing CI, followed by a publishing step.
In order to encourage documentation updates, we want to make them less burdensome, and one way to do that is by not running a full CI workflow. Why bother running tests and lint checks when we know that no code has changed?
This seems like it should be straightforward, but it's actually surprisingly finicky. Read on to find out how we did it!
First pass: path filtering
One seemingly simple way to achieve this is to use GitHub Actions' path filtering with `paths-ignore`. In fact, their usage example is exactly our docs/ use case! On every CI workflow, set
paths-ignore:
- "docs/**"
And that workflow will only run if at least one changed file is outside the docs/ directory. Problem solved, right?
Branch protection racket
Not so fast… There's a catch:
We have a branch protection rule on our main branch that prevents a merge unless various CI jobs, such as testing and linting, pass. If the jobs don't run due to path filtering, GitHub actions considers them pending, not passing, so we won't be able to merge.
The workaround is, as is so often the case, to add a level of indirection:
Instead of having a branch protection rule that directly requires those test and lint jobs to pass, have a rule that requires some single job named "Merge OK" to pass. Since branch protection rules only look at the name of the job, we can have two "Merge OK" jobs - one that runs for regular changes that affect code, and depends on all the test and lint jobs passing before it can run, and another, for "docs only" changes, that runs immediately. Now, problem solved, right?
The trouble with path filtering
Not so fast… There is another catch:
Path filtering can only handle "run a workflow if any path is under docs/" or "skip a workflow if all paths are under docs/". It does not provide "run a workflow if all paths are under docs/". So there is no way to use path filtering to run that other "Merge OK" job in the "docs only" case.
To solve this, we don't use GitHub Actions' built-in path filtering. Instead, we use the changed-files action, and check the files it reports in the "if" condition of the job. Surely, now, problem solved, right?
Skipping successfully
Not so fast… Try not to fall over when I tell you that there is yet another catch:
GitHub Actions treats jobs skipped due to an "if" condition as successful. So we can't have our branch protection rule check for a "Merge OK" job directly: The "docs only" variant of that job will always be successful, either because this is a "docs only" change, so it actually ran, or because this is not a "docs only" change, so the job was skipped due to its "if" condition.
The workaround here is to add, you guessed it, another level of indirection:
Instead of two "Merge OK" jobs, we have two "Set Merge OK" jobs, and a single "Merge OK" job that depends on both of them. The "Set Merge OK" jobs set an output, and the "Merge OK" job checks for the presence of that output. A job needs to actually run to set an output, so if "Merge OK" does see that output we know that one of the "Set Merge OK" jobs ran, which means one of two things:
- This was a "docs only" change
- This was not a "docs only" change, and all the regular CI jobs passed
And now, finally, problem solved!
You can see how all this fits together here, and that YAML is generated by this script.
And since if you're reading this you probably care about CI performance, you may want to learn more about Pants, and how it can help speed up your builds. If you're curious, come and say hi on our Slack!