Using Pants in CI

📘

Example .travis.yml

See https://github.com/pantsbuild/example-python/blob/master/.travis.yml for an example of how to configure Pants with Travis CI.

pants.ci.toml

We recommend creating a pants.ci.toml config file with options specific to your CI setup.

[GLOBAL]
dynamic_ui = false
# Ensure colors are used (if your CI provider supports it). 
colors = true

# Limit the maximum number of concurrent processes. Change this
# to a number that makes sense for your CI setup, based on 
# the number of cores/threads.
process_execution_local_parallelism = 4  

[python-setup]
# Limit the maximum number of concurrent jobs used to resolve third
# party dependencies. The total level of parallelism will be
# `process_execution_local_parallelism x resolver_jobs`.
resolver_jobs = 1

Then, in your CI script or config, set the environment variable PANTS_CONFIG_FILES to use this new config file, in addition to the default pants.toml. For example:

export PANTS_CONFIG_FILES=pants.ci.toml

# Then, your normal CI setup
./pants test ::
...

Directories to cache

In your CI's config file, we recommend caching these directories:

• $HOME/.cache/pants/setup: the initial bootstrapping of Pants.
• $HOME/.cache/pants/named_caches: caches of tools like pip and PEX.
• $HOME/.cache/pants/lmdb_store: cached content for prior Pants runs.

📘

Nuking the cache when too big

In CI, the cache must be uploaded and downloaded every run. This takes time, so there is a tradeoff where too large of a cache will slow down your CI.

You can use this script to nuke the cache when it gets too big:

Bash

function nuke_if_too_big() {
  path=$1
  limit_mb=$2
  size_mb=$(du -m -d0 ${path} | cut -f 1)
  if (( ${size_mb} > ${limit_mb} )); then
    echo "${path} is too large (${size_mb}mb), nuking it."
    rm -rf ${path}
  fi
}

nuke_if_too_big ~/.cache/pants/lmdb_store 2048
nuke_if_too_big ~/.cache/pants/setup 768
nuke_if_too_big ~/.cache/pants/named_caches 1024

Recommended commands

Because Pants understands the dependencies of your code, you can use Pants to speed up your CI by only running tests and linters over files that actually made changes.

We recommend running these commands in CI:

$ <<pantscmd>> --version  # This will bootstrap Pants
$ <<pantscmd>> --changed-since=origin/master lint
$ <<pantscmd>> \
  --changed-since=origin/master \
  --changed-include-dependees=transitive \
  test

Because most linters do not care about a target's dependencies, we lint all changed targets, but not any dependees of those changed targets.

Meanwhile, tests should be rerun when any changes are made to the tests or to dependencies of those tests, so we use the option --changed-include-dependees=transitive.

See Advanced target selection for more information on --changed-since and alternative techniques to select targets to run in CI.