test

Pants uses the popular Pytest test runner to run Python tests. You may write your tests in Pytest-style, unittest-style, or mix and match both.

👍
Benefit of Pants: runs each test target in parallel
Each test target gets run as a separate process, which gives you fine-grained caching and better parallelism. Given enough cores, Pants will be able to run all your tests at the same time.
This also gives you fine-grained caching. If you run ./pants test ::, and then you only change one target, then only that target will need to rerun and the others will use the cache.

Pytest version and plugins

The Pytest community has developed hundreds of plugins to add additional functionality.

To install any plugins, add the Pip requirement string to pytest_plugins in the scope [pytest], like this:

[pytest]
version = "pytest>=5.4"
pytest_plugins.add = [
  "pytest-django>=3.9.0,<4",
  "pytest-rerunfailures==9.0",
]

🚧
Testing Python 2 code? Use Pytest 4.x
By default, Pants uses Pytest 5.x, which only supports Python 3 code. If you need to run Python 2 tests, set the option version in the scope pytest to pytest>=4.6,<5.

🚧
Avoid the pytest-xdist plugin
We do not recommend using this plugin because its concurrency conflicts with Pants' own parallelism. Using Pants will bring you similar benefits to pytest-xdist already: Pants will run each test target in parallel.

📘
Tip: plugins for better Pytest output
Add pytest-icdiff and pygments to the option pytest_plugins for much better error messages and diffs from Pytest.

Passing arguments to Pytest

To pass arguments to Pytest, put them at the end after --, like this:

$ <<pantscmd>> test project/app_test.py -- -k test_function1 -vv -s

You can also use the args option in the [pytest] scope, like this:

[pytest]
args = ["-vv"]

📘
Tip: some useful Pytest arguments
See https://docs.pytest.org/en/latest/usage.html for more information.

-k expression: only run tests matching the expression.

-v: verbose mode.

-s: always print the stdout and stderr of your code, even if a test passes.

🚧
How to use Pytest's --pdb option
You must run <<pantscmd>> test --debug for this to work properly. See the next section "Running tests interactively" for more information.

Running tests interactively

Because Pants runs multiple test targets in parallel, you will not see your test results appear on the screen until the test has completely finished. This means that you cannot use debuggers normally; the breakpoint will never show up on your screen and the test will hang indefinitely (or timeout, if timeouts are enabled).

Instead, if you want to run a test interactively—such as to use a debugger like pdb—run your tests with <<pantscmd>> test --debug. For example:

def test_debug():
    import pdb; pdb.set_trace()
    assert 1 + 1 == 2

$ <<pantscmd>> test --debug test_debug_example.py

===================================================== test session starts =====================================================
platform darwin -- Python 3.6.10, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /private/var/folders/sx/pdpbqz4x5cscn9hhfpbsbqvm0000gn/T/.tmpn2li0z
plugins: cov-2.8.1, timeout-1.3.4
collected 6 items

test_debug_example.py
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /private/var/folders/sx/pdpbqz4x5cscn9hhfpbsbqvm0000gn/T/.tmpn2li0z/test_debug_example.py(11)test_debug()
-> assert 1 + 1 == 2
(Pdb) 1 + 1
2

You may only run one test target at a time when using --debug.

📘
Use --debug to force re-running a test
Normally, Pants will cache test results. If you run <<pantscmd>> test app_test.py, then run the same command, Pants will nearly instantly give you the result from its cache.
However, <<pantscmd>> test --debug will not use the cache. This can be helpful, for example, to check if a test is flaky.

📘
Tip: using ipdb in tests
ipdb integrates IPython with the normal pdb debugger for enhanced features like autocomplete and improved syntax highlighting. ipdb is very helpful when debugging tests.
To be able to access ipdb when running tests, add this to your pants.toml:
[pytest]
pytest_plugins.add = ["ipdb"]
Then, you can use import ipdb; ipdb.set_trace() in your tests.

Using timeouts

Pants can cancel tests which take too long. This is useful to prevent tests from hanging indefinitely.

To add a timeout for a particular python_tests target, set the timeout field to an integer value of seconds, like this:

python_tests(
    name="tests",
    dependencies=[
        ":lib",
    ],
    timeout=120, # seconds.
)

You can also set a default value and a maximum value in pants.toml:

[pytest]
timeout_default = 60
timeout_maximum = 600

If a target sets its timeout higher than --pytest-timeout-maximum, Pants will use the value in --pytest-timeout-maximum.

📘
Tip: temporarily ignoring timeouts
When debugging locally, such as with pdb, you might want to temporarily disable timeouts. To do this, set --no-pytest-timeouts:
$ ./pants test project/app_test.py --no-pytest-timeouts

Saving JUnit XML results

Pytest can generate JUnit XML result files. This allows you to hook up your results, for example, to dashboards.

To save JUnit XML result files, set the option junit_xml_dir in the [pytest] scope, like this:

[pytest]
junit_xml_dir = "dist/pytest_results"

You may also want to set the option junit_family in the [pytest] scope to change the format. Run <<pantscmd>> help-advanced pytest for more information.

Examples

 # Run all tests in the repository.
<<pantscmd>> test ::

# Run all the tests in this target.
<<pantscmd>> test helloworld/util:test  

# Run just the tests in this file.
<<pantscmd>> test helloworld/util/lang_test.py  

 # Run just one test.
<<pantscmd>> test helloworld/util/lang_test.py --pytest-args='-k test_language_translator'