See the example-python repository for an [example GitHub Actions worfklow](🔗).
## Directories to cache
init-pants` GitHub Action
If you're using GitHub Actions to run your CI workflows, then you can use our [standard action](🔗) to set up and cache the Pants bootstrap state. Otherwise, read on to learn how to configure this manually.
In your CI's config file, we recommend caching these directories:
$HOME/.cache/pants/setup`<br> This is the Pants bootstrap directory. Cache this against the version, as specified in `
pants.toml`. See the [pantsbuild/example-python](🔗) repo for an example of how to generate an effective cache key for this directory in GitHub Actions.
$HOME/.cache/pants/named_caches`<br> Caches used by some underlying tools. Cache this against the inputs to those tools. For the `
pants.backend.python` backend, named caches are used by PEX, and therefore its inputs are your lockfiles. Again, see [pantsbuild/example-python](🔗) for an example.
If you're not using a fine-grained [remote caching](🔗) service, then you may also want to preserve the local Pants cache at `
$HOME/.cache/pants/lmdb_store`. This has to be invalidated on any file that can affect any process, e.g., `
hashFiles('**/*')` on GitHub Actions.
Computing such a coarse hash, and saving and restoring large directories, can be unwieldy. So this may be impractical and slow on medium and large repos.
A [remote cache service](🔗) integrates with Pants's fine-grained invalidation and avoids these problems, and is recommended for the best CI performance.
See [Troubleshooting](🔗) for how to change these cache locations.
Nuking the cache when too big
In CI, the cache must be uploaded and downloaded every run. This takes time, so there is a tradeoff where too large a cache will slow down your CI.
You can use this script to nuke the cache when it gets too big:
Tip: check cache performance with `
Set the option `
[stats].log = true` in `
pants.ci.toml` for Pants to print metrics of your cache's performance at the end of the run, including the number of cache hits and the total time saved thanks to caching, e.g.:
You can also add `
plugins = ["hdrhistogram"]` to the `
[GLOBAL]` section of `
pants.ci.toml` for Pants to print histograms of cache performance, e.g. the size of blobs cached.
Rather than storing your cache with your CI provider, remote caching stores the cache in the cloud, using gRPC and the open-source Remote Execution API for low-latency and fine-grained caching.
This brings several benefits over local caching:
All machines and CI jobs share the same cache.
Remote caching downloads precisely what is needed by your run—when it's needed—rather than pessimistically downloading the entire cache at the start of the run.
No download and upload stage for your cache.
No need to "nuke" your cache when it gets too big.
See [Remote Caching and Execution](🔗) for more information.
## Recommended commands
With both approaches, you may want to shard the input targets into multiple CI jobs, for increased parallelism. See [Advanced Target Selection](🔗). (This is typically less necessary when using [remote caching](🔗).)
### Approach #1: only run over changed files
Because Pants understands the dependencies of your code, you can use Pants to speed up your CI by only running tests and linters over files that actually made changes.
We recommend running these commands in CI:
Because most linters do not care about a target's dependencies, we lint all changed files and targets, but not any dependees of those changes.
Meanwhile, tests should be rerun when any changes are made to the tests _or_ to dependencies of those tests, so we use the option `
check` should also run on any transitive changes.
See [Advanced target selection](🔗) for more information on `
--changed-since` and alternative techniques to select targets to run in CI.
This will not handle all cases, like hooking up a new linter
For example, if you add a new plugin to Flake8, Pants will still only run over changed files, meaning you may miss some new lint issues.
For absolute correctness, you may want to use Approach #2. Alternatively, add conditional logic to your CI, e.g. that any changes to `
pants.toml` trigger using Approach #2.
GitHub Actions: use `
To use `
--changed-since`, you may want to use the [Checkout action](🔗).
By default, Checkout will only fetch the latest commit; you likely want to set `
fetch-depth` to fetch prior commits.
GitLab CI: disable shallow clones or fetch main branch
GitLab's merge pipelines make a shallow clone by default, which only contains recent commits for the feature branch being merged. That severely limits `
--changed-since`. There are two possible workarounds:
Clone the entire repository by going to "CI / CD" settings and erase the number from the "Git shallow clone" field of the "General pipelines" section. Don't forget to "Save changes". This has the advantage of cloning everything, which also is the biggest long-term disadvantage.
A more targeted and hence light-weight intervention leaves the shallow clone setting at its default value and instead fetches the `
main` branch as well:
git branch` commands are only included to print out all available branches before and after fetching `
### Approach #2: run over everything
Alternatively, you can simply run over all your code. Pants's caching means that you will not need to rerun on changed files.
However, when the cache gets too big, it should be nuked (see "Directories to cache"), so your CI may end up doing more work than Approach #1.
This approach works particularly well if you are using [remote caching](🔗).
## Configuring Pants for CI: `
Sometimes, you may want config specific to your CI, such as turning on test coverage reports. If you want CI-specific config, create a dedicated `
pants.ci.toml` [config file](🔗). For example:
Then, in your CI script or config, set the environment variable `
PANTS_CONFIG_FILES=pants.ci.toml` to use this new config file, in addition to `
### Tuning resource consumption (advanced)
Pants allows you to control its resource consumption. These options all have sensible defaults. In most cases, there is no need to change them. However, you may benefit from tuning these options.
process_execution_local_parallelism`](🔗): number of concurrent processes that may be executed locally.
rule_threads_core`](🔗): number of threads to keep active to execute `
rule_threads_max`](🔗): maximum number of threads to use to execute `
Memory usage options:
pantsd`](🔗): enable or disable the Pants daemon, which uses an in-memory cache to speed up subsequent runs after the first run in CI.
pantsd_max_memory_usage`](🔗): reduce or increase the size of Pantsd's in-memory cache.
The default test runners for these CI providers have the following resources. If you are using a custom runner, e.g. enterprise, check with your CI provider.
|GitHub Actions, Linux||2||7 GB||[link](🔗)|
|Travis, Linux||2||7.5 GB||[link](🔗)|
|Circle CI, Linux, free plan||2||4 GB||[link](🔗)|
|GitLab, Linux shared runners||1||3.75 GB||[link](🔗)|
## Tip: store Pants logs as artifacts
We recommend that you configure your CI system to store the pants log (`
.pantd.d/pants.log`) as a build artifact, so that it is available in case you need to troubleshoot CI issues.
Different CI providers and systems have different ways to configure build artifacts:
Circle CI - [Storing artifacts](🔗)
Github Actions - [Storing Artifacts](🔗) - [example in the pants repo](🔗)
Bitbucket pipelines - [Using artifacts](🔗)
Jenkins - [Recording artifacts](🔗)
It's particularly useful to configure your CI to always upload the log, even if prior steps in your pipeline failed.