Skip to main content
Version: 2.14 (deprecated)

test

Run tests with Pytest.

Pants uses the Pytest test runner to run Python tests. You may write your tests in Pytest-style, unittest-style, or mix and match both.

Benefit of Pants: runs each file in parallel

Each file gets run as a separate process, which gives you fine-grained caching and better parallelism. Given enough cores, Pants will be able to run all your tests at the same time.

This also gives you fine-grained invalidation. If you run ./pants test ::, and then you only change one file, then only tests that depended on that changed file will need to rerun.

Examples​

 # Run all tests in the repository.
❯ ./pants test ::

# Run all the tests in this directory.
❯ ./pants test helloworld/util:

# Run just the tests in this file.
❯ ./pants test helloworld/util/lang_test.py

 # Run just one test.
❯ ./pants test helloworld/util/lang_test.py -- -k test_language_translator

Pytest version and plugins​

To change the Pytest version, set the version option in the [pytest] scope.

To install any plugins, add the pip requirement string to extra_requirements in the [pytest] scope, like this:

pants.toml
[pytest]
version = "pytest>=5.4"
extra_requirements.add = [
  "pytest-django>=3.9.0,<4",
  "pytest-rerunfailures==9.0",
]

If you change either version or extra_requirements, Pants's default lockfile for Pytest will not work. Either set the lockfile option to a custom path or "<none>" to opt out. See Third-party dependencies.

Alternatively, if you only want to install the plugin for certain tests, you can add the plugin to the dependencies field of your python_test / python_tests target. See Third-party dependencies for how to install Python dependencies. For example:

pytest-django==3.10.0
Tip: Using the pytest-xdist plugin

Pants' default concurrency will run each test target in parallel. If you want to additionally parallelize within each of your targets, you can do so by enabling built-in support for the pytest-xdist plugin:

pants.toml
[pytest]
xdist_enabled = True

This will cause Pants to pass -n <concurrency> when running pytest.

By default, Pants will automatically compute the value of <concurrency> for each target based on the number of tests defined in the file and the number of available worker threads. You can instead set a hard-coded upper limit on the concurrency per target:

BUILD
python_test(name="tests", source="tests.py", xdist_concurrency=4)

To explicitly disable the use of pytest-xdist for a target, set xdist_concurrency=0. This can be necessary for tests that are not safe to run in parallel.

When pytest-xdist is in use, the PYTEST_XDIST_WORKER and PYTEST_XDIST_WORKER_COUNT environment variables will be automatically set. You can use those values to avoid collisions between parallel tests (i.e. by using PYTEST_XDIST_WORKER as a suffix for generated database names / file paths).

Note that use of pytest-xdist may cause high-level pytest fixtures to execute more often than expected. See the pytest-xdist docs here for more details, and tips on how to mitigate this.

Controlling output​

By default, Pants only shows output for failed tests. You can change this by setting --test-output to one of all, failed, or never, e.g. ./pants test --output=all ::.

You can permanently set the output format in your pants.toml like this:

pants.toml
[test]
output = "all"
Tip: Use Pytest options to make output more or less verbose

See "Passing arguments to Pytest".

For example:

❯ ./pants test project/app_test.py -- -q

You may want to permanently set the Pytest option --no-header to avoid printing the Pytest version for each test run:

[pytest]
args = ["--no-header"]

Passing arguments to Pytest​

To pass arguments to Pytest, put them at the end after --, like this:

❯ ./pants test project/app_test.py -- -k test_function1 -vv -s

You can also use the args option in the [pytest] scope, like this:

pants.toml
[pytest]
args = ["-vv"]
Tip: some useful Pytest arguments

See https://docs.pytest.org/en/latest/usage.html for more information.

  • -k expression: only run tests matching the expression.
  • -v: verbose mode.
  • -s: always print the stdout and stderr of your code, even if a test passes.
How to use Pytest's --pdb option

You must run ./pants test --debug for this to work properly. See the section "Debugging Tests" for more information.

Config files​

Pants will automatically include any relevant config files in the process's sandbox: pytest.ini, pyproject.toml, tox.ini, and setup.cfg.

conftest.py​

Pytest uses conftest.py files to share fixtures and config across multiple distinct test files.

The default sources value for the python_test_utils target includes conftest.py. You can run ./pants tailor :: to automatically add this target:

./pants tailor ::
Created project/BUILD:
  - Add python_sources target project
  - Add python_tests target tests
  - Add python_test_utils target test_utils

Pants will also infer dependencies on any confest.py files in the current directory and any ancestor directories, which mirrors how Pytest behaves. This requires that each conftest.py has a target referring to it. You can verify this is working correctly by running ./pants dependencies path/to/my_test.py and confirming that each conftest.py file shows up. (You can turn off this feature by setting conftests = false in the [python-infer] scope.)

Setting environment variables​

Test runs are hermetic, meaning that they are stripped of the parent ./pants process's environment variables. This is important for reproducibility, and it also increases cache hits.

To add any arbitrary environment variable back to the process, you can either add the environment variable to the specific tests with the extra_env_vars field on python_test / python_tests targets or to all your tests with the [test].extra_env_vars option. Generally, prefer the field extra_env_vars field so that more of your tests are hermetic.

With both [test].extra_env_vars and the extra_env_vars field, you can either hardcode a value or leave off a value to "allowlist" it and read from the parent ./pants process's environment.

[test]
extra_env_vars = ["VAR1", "VAR2=hardcoded_value"]

Force reruns with --force​

To force your tests to run again, rather than reading from the cache, run ./pants test --force path/to/test.py.

Debugging Tests​

Because Pants runs multiple test targets in parallel, you will not see your test results appear on the screen until the test has completely finished. This means that you cannot use debuggers normally; the breakpoint will never show up on your screen and the test will hang indefinitely (or timeout, if timeouts are enabled).

Instead, if you want to run a test interactively—such as to use a debugger like pdb—run your tests with ./pants test --debug. For example:

def test_debug():
    import pdb; pdb.set_trace()
    assert 1 + 1 == 2

If you use multiple files with test --debug, they will run sequentially rather than in parallel.

Tip: using ipdb in tests

ipdb integrates IPython with the normal pdb debugger for enhanced features like autocomplete and improved syntax highlighting. ipdb is very helpful when debugging tests.

To be able to access ipdb when running tests, add this to your pants.toml:

[pytest]
extra_requirements.add = ["ipdb"]

Then, you can use import ipdb; ipdb.set_trace() in your tests.

To run the tests you will need to add -- -s to the test call since ipdb will need stdin and pytest will capture it.

❯ ./pants test --debug  <target>   -- -s
Tip: using the VS Code (or any DAP-compliant editor) remote debugger in tests
  1. In your editor, set your breakpoints and any other debug settings (like break-on-exception).
  2. Run your test with ./pants test --debug-adapter.
  3. Connect your editor to the server. The server host and port are logged by Pants when executing test --debug-adapter. (They can also be configured using the [debug-adapter] subsystem).

Run your test with ./pants test --debug as usual.

Tip: using the IntelliJ/PyCharm remote debugger in tests

First, add this to your pants.toml:

[pytest]
extra_requirements.add = ["pydevd-pycharm==203.5419.8"]  # Or whatever version you choose.

Now, use the remote debugger as usual:

  1. Start a Python remote debugging session in PyCharm, say on port 5000.
  2. Add the following code at the point where you want execution to pause and connect to the debugger:
import pydevd_pycharm
pydevd_pycharm.settrace('localhost', port=5000, stdoutToServer=True, stderrToServer=True)

Run your test with ./pants test --debug as usual.

Timeouts​

Pants can cancel tests which take too long. This is useful to prevent tests from hanging indefinitely.

To add a timeout, set the timeout field to an integer value of seconds, like this:

BUILD
python_test(name="tests", source="tests.py", timeout=120)

When you set timeout on the python_tests target generator, the same timeout will apply to every generated python_test target.

BUILD
python_tests(
    name="tests",
    overrides={
        "test_f1.py": {"timeout": 20},
        ("test_f2.py", "test_f3.py"): {"timeout": 35},
    },
)

You can also set a default value and a maximum value in pants.toml:

pants.toml
[test]
timeout_default = 60
timeout_maximum = 600

If a target sets its timeout higher than [test].timeout_maximum, Pants will use the value in [test].timeout_maximum.

Tip: temporarily ignoring timeouts

When debugging locally, such as with pdb, you might want to temporarily disable timeouts. To do this, set --no-test-timeouts:

$ ./pants test project/app_test.py --no-test-timeouts

Test utilities and resources​

Test utilities​

Use the target type python_source for test utilities, rather than python_test.

To reduce boilerplate, you can use either the python_sources or python_test_utils targets to generate python_source targets. These behave the same, except that python_test_utils has a different default sources to include conftest.py and type stubs for tests (like test_foo.pyi). Use ./pants tailor :: to generate both these targets automatically.

For example:

# The default `sources` includes all files other than
# `!*_test.py`, `!test_*.py`, and `tests.py`, and `conftest.py`.
python_sources(name="lib")

# We leave off the `dependencies` field because Pants will infer
# it based on import statements.
python_tests(name="tests")

Assets​

Refer to Assets for how to include asset files in your tests by adding to the dependencies field.

It's often most convenient to use file / files and relocated_files targets in your test code, although you can also use resource / resources targets.

Testing your packaging pipeline​

You can include the result of ./pants package in your test through the runtime_package_dependencies field. Pants will run the equivalent of ./pants package beforehand and copy the built artifact into the test's chroot, allowing you to test things like that the artifact has the correct files present and that it's executable.

This allows you to test your packaging pipeline by simply running ./pants test ::, without needing custom integration test scripts.

To depend on a built package, use the runtime_package_dependencies field on the python_test / python_tests target, which is a list of addresses to targets that can be built with ./pants package, such as pex_binary, python_awslambda, and archive targets. Pants will build the package before running your test, and insert the file into the test's chroot. It will use the same name it would normally use with ./pants package, except without the dist/ prefix (set by the output_path field).

For example:

# This target teaches Pants about our non-test Python files.
python_sources(name="lib")

pex_binary(
    name="bin",
    entry_point="say_hello.py",
)

python_tests(
    name="tests",
    runtime_package_dependencies=[":bin"],
)

Coverage​

To report coverage using Coverage.py, set the option --test-use-coverage:

❯ ./pants test --use-coverage helloworld/util/lang_test.py

Or to permanently use coverage, set in your config file:

pants.ci.toml
[test]
use_coverage = true
Failure to parse files?

Coverage defaults to running with Python 3.6+ when generating a report, which means it may fail to parse Python 2 syntax and Python 3.8+ syntax. You can fix this by changing the interpreter constraints for running Coverage:

# pants.toml
[coverage-py]
interpreter_constraints = [">=3.8"]

However, if your repository has some Python 2-only code and some Python 3-only code, you will not be able to choose an interpreter that works with both versions. So, you will need to set up a .coveragerc config file and set ignore_errors = True under [report], like this:

# .coveragerc
[report]
ignore_errors = True

ignore_errors = True means that those files will simply be left off of the final coverage report.

(Pants should autodiscover the config file .coveragerc. See coverage-py.)

There's a proposal for Pants to fix this by generating multiple reports when necessary: https://github.com/pantsbuild/pants/issues/11137. We'd appreciate your feedback.

Coverage will report data on any files encountered during the tests. You can filter down the results by using the option --coverage-py-filter and passing the name(s) of modules you want coverage data for. Each module name is recursive, meaning submodules will be included. For example:

❯ ./pants test --use-coverage helloworld/util/lang_test.py --coverage-py-filter=helloworld.util
❯ ./pants test --use-coverage helloworld/util/lang_test.py --coverage-py-filter='["helloworld.util.lang", "helloworld.util.lang_test"]'
Set global_report to include un-encountered files

By default, coverage.py will only report on files encountered during the tests' run. This means that your coverage score may be misleading; even with a score of 100%, you may have files without any tests.

Instead, you can set global_report = true:

pants.toml
[coverage-py]
global_report = true

Coverage.py will report on all files it considers importable, i.e. files at the root of the tree, or in directories with a __init__.py file. It may still omit files in implicit namespace packages that lack __init__.py files. This is a shortcoming of Coverage.py itself.

Pants will default to writing the results to the console, but you can also output in HTML, XML, JSON, or the raw SQLite file:

pants.toml
[coverage-py]
report = ["raw", "xml", "html", "json", "console"]

You can change the output dir with the output_dir option in the [coverage-py] scope.

You may want to set [coverage-py].fail_under to cause Pants to gracefully fail if coverage is too low, e.g. fail_under = 70.

You may use a Coverage config file, e.g. .coveragerc or pyproject.toml. Pants will autodiscover the config file for you, and you can also set [coverage-py].config in your pants.toml to point to a non-standard location. You must include relative_files = True in the [run] section for Pants to work.

.coveragerc
[run]
relative_files = True
branch = True

When generating HTML, XML, and JSON reports, you can automatically open the reports through the option --test-open-coverage.

JUnit XML results​

Pytest can generate JUnit XML result files. This allows you to hook up your results, for example, to dashboards.

To save JUnit XML result files, set the option [test].report, like this:

pants.toml
[test]
report = true

This will default to writing test reports to dist/test/reports. You may also want to set the option [pytest].junit_family to change the format. Run ./pants help-advanced pytest for more information.