Third-party dependencies

Basic setup

To use third-party dependencies in your project, create a regular Pip requirements.txt file.

Then, in the same directory, create a BUILD file that invokes the python_requirements() macro:

flask>=1.1.2,<1.3
requests==2.23.0

python_requirements()

Pants will use the python_requirements() macro to create a distinct python_requirement_library target for each specified requirement. Now, you can specify that your Python code uses a third-party dependency by adding it to the dependencies field for the target.

For example:

python_library(
    dependencies=[
        # The `//` signals that the target is at the root of your project.
        "//:flask",
        "//:requests",
    ],
)

import flask
import requests

...

📘

Installing from version control or local files

Sometimes, you may want to install from version control, like Git.

You might be used to using pip's proprietary VCS-style requirements for this, like git+https://github.com/django/django.git#egg=django. However, this proprietary format does not work with Pants.

Instead of pip VCS-style requirements:

git+https://github.com/django/django.git#egg=Django
git+https://github.com/django/django.git@stable/2.1.x#egg=Django
git+https://github.com/django/django.git@fd209f62f1d83233cc634443cfac5ee4328d98b8#egg=Django

Use direct references from PEP 440:

Django@ git+https://github.com/django/django.git
Django@ git+https://github.com/django/django.git@stable/2.1.x
Django@ git+https://github.com/django/django.git@fd209f62f1d83233cc634443cfac5ee4328d98b8

You can also install from local files using PEP 440 direct references. You must use an absolute path to the file, and you should ensure that the file exists on your machine.

Django @ file:///Users/pantsbuild/prebuilt_wheels/django-3.1.1-py3-none-any.whl

Pip still works with these PEP 440-compliant formats, so you won't be losing any functionality by switching to using them.

Using a lockfile (recommended)

Lockfiles are important for reproducible builds. They ensure that the selected versions of all transitive dependencies are consistent, even when third-party libraries release new versions.

Pants uses Pip's constraints file feature to support lockfiles. Constraints files tell Pip to use the specified version of a library when encountered, even if it overrides what was specified in your requirements.txt.

🚧

Constraints files are not strongly validated

It is possible to put conflicting versions in a constraints file, and to leave off constraints for some of your dependencies; Pip doesn't check for this. Unfortunately, the onus is on you to make sure that your constraints file is sound.

To use this feature, create a constraints.txt file that pins versions of all your transitive third-party dependencies.

Then tell Pants about the file with the requirement_constraints option in the [python-setup] scope, like this:

certifi==2019.6.16
chardet==3.0.2
idna==2.7
requests==2.23.0
urllib3==1.25.6

[python-setup]
requirement_constraints = "constraints.txt"

Pants will then pass this constraints file to Pip whenever resolving third party dependencies.

📘

Why not just put everything in requirements.txt?

This is possible but is not very ergonomic or maintainable. See https://kenreitz.org/essays/a-better-pip-workflow.

With Pants, if you put everything in requirements.txt, this means that you will end up with a target for every dependency—including dependencies of your dependencies—even though you likely only want targets for the dependencies you directly use.

📘

How do I generate a lockfile?

The simplest approach is to create a virtual environment for your project (see below). Then, with the virtual environment activated, run pip install -r requirements.txt and pip freeze --all > constraints.txt.

You may alternatively use a tool like Poetry or Pipenv, then use their export commands to generate a constraints.txt file.

🚧

Need support for multiple lockfiles?

For now, Pants only supports one single global lockfile. But, we are considering adding support for multiple lockfiles. Please message us on Slack if you would like this feature - we would appreciate feedback on your use case.

Defining inline requirements

Sometimes you may want to use a certain dependency without adding it to your global requirements.txt. For example, you may want to use a different version than what is in requirements.txt. Or, you may want to only use the requirement in one location and don't want it to discoverable by all targets.

Pants allows you to define inline requirements with the target type python_requirement_library, like this:

python_requirement_library(
    name="old_requests",
    requirements=[
        python_requirement("requests==2.8.0"),
    ],
)

python_binary(
    name="app",
    sources=["app.py"],
    dependencies=[
        ":old_requests",
    ],
)

Using custom repositories

If you host your own wheels at a custom index (aka "cheese shop"), you can instruct Pants to use it with the option indexes in the [python-repos] scope.

[python-repos]
indexes.add = ["https://custom-cheeseshop.net/simple"]

To exclusively use your custom index—i.e. to not use PyPI—use indexes = [..] instead of indexes.add = [..].

Tip: Set up a virtual environment (optional)

It's often useful to build a virtual environment (virtualenv) containing all your third-party dependencies. For example, many IDEs use such a virtualenv to understand imports.

Pants itself doesn't need this virtualenv. But it may be useful for working with your tooling outside Pants.

You can create a virtualenv using standard Python tooling—such as Python's builtin venv module—along with running Pants to query for all of your Python requirements.

For example:

#!/usr/bin/env bash

set -euo pipefail

PYTHON_BIN=python3
VIRTUALENV=build-support/.venv
PIP="${VIRTUALENV}/bin/pip"
CONSTRAINTS_FILE=constraints.txt

"${PYTHON_BIN}" -m venv "${VIRTUALENV}"
"${PIP}" install pip --upgrade
"${PIP}" install -r <(./pants dependencies --type=3rdparty ::)
echo "# Generated by build-support/generate_constraints.sh on $(date)" > "${CONSTRAINTS_FILE}"
"${PIP}" freeze --all >> "${CONSTRAINTS_FILE}"