HomeDocs
DocsCommunityTestimonialsUsersGitHubTwitterBlogJobsTermsPrivacyCookies
TermsPrivacyCookies
Hey! These docs are for version 2.10, which is no longer officially supported. Click here for the latest version, 2.18!


When your Python code imports Protobuf generated files, Pants will detect the imports and run the Protoc compiler to generate those files.

Example repository

See [the Python example repository](🔗) for an example of using Protobuf to generate Python.

Benefit of Pants: generated files are always up-to-date

With Pants, there's no need to manually regenerate your code or check it into version control. Pants will ensure you are always using up-to-date files in your builds.

Thanks to fine-grained caching, Pants will regenerate the minimum amount of code required when you do make changes.

## Step 1: Activate the Protobuf Python backend

Add this to your `pants.toml`:



This adds the new [`protobuf_source`](🔗) target, which you can confirm by running `./pants help protobuf_source`.

To reduce boilerplate, you can also use the [`protobuf_sources`](🔗) target, which generates one `protobuf_source` target per file in the `sources` field.



Enable the MyPy Protobuf plugin

The [MyPy Protobuf plugin](🔗) generates [`.pyi` type stubs](🔗). If you use MyPy through Pants's [check goal](🔗), this will ensure MyPy understands your generated code.

To activate, set `mypy_plugin = true` in the `[python-protobuf]` scope:



MyPy will use the generated `.pyi` type stub file, rather than looking at the `.py` implementation file.

## Step 2: Set up the `protobuf` and `grpcio` runtime libraries

Generated Python files require the [`protobuf` dependency](🔗) for their imports to work properly. If you're using gRPC, you also need the [`grpcio` dependency](🔗).

Add `protobuf`—and `grpcio`, if relevant— to your project, e.g. your `requirements.txt` (see [Third-party dependencies](🔗)).



Pants will then automatically add these dependencies to your `protobuf_source` targets created in the next step.

## Step 3: Generate `protobuf_sources` target

Run [`./pants tailor`](🔗) for Pants to create a `protobuf_sources` target wherever you have `.proto` files:



Pants will use [dependency inference](🔗) for any `import` statements in your `.proto` files, which you can confirm by running `./pants dependencies path/to/file.proto`. You should also see the `python_requirement` target for the `protobuf` library from the previous step.

If you want gRPC code generated for all files in the folder, set `grpc=True`.



If you only want gRPC generated for some files in the folder, you can use the `overrides` field:



## Step 4: Confirm Python imports are working

Now, you can import the generated Python module in your Python code. For example, to import `project/example/f.proto`, add `import project.example.f_pb2` to your code.

If you have [source roots](🔗) other than the repository root, remove the source root from the import. For example, `src/protos/example/f.proto` gets stripped to `import example.f_pb2`. See the below section on source roots for more info.

Pants's dependency inference will detect Python imports of Protobuf modules, which you can confirm by running `./pants dependencies path/to/file.py`.

If gRPC is activated, you can also import the module with `_pb2_grpc` at the end, e.g. `project.example.f_pb2_grpc`.



Run `./pants export-codegen ::` to inspect the files

`./pants export-codegen ::` will run all relevant code generators and write the files to `dist/codegen` using the same paths used normally by Pants.

You do not need to run this goal for codegen to work when using Pants; `export-codegen` is only for external consumption outside of Pants.

You likely need to add empty `**init**.py` files

By default, Pants will generate the Python files in the same directory as the `.proto` file. To get Python imports working properly, you will likely need to add an empty `**init**.py` in the same location, and possibly in ancestor directories.

See the below section "Protobuf and source roots" for how to generate into a different directory. If you use this option, you will still likely need an empty `**init**.py` file in the destination directory.

## Protobuf and source roots

By default, generated code goes into the same [source root](🔗) as the `.proto` file from which it was generated. For example, a file `src/proto/example/f.proto` will generate `src/proto/example/f_pb2.py`.

However, this may not always be what you want. In particular, you may not want to have to add `**init**py` files under `src/proto` just so you can import Python code generated to that source root.

You can configure a different source root for generated code by setting the `python_source_root` field:



Now `src/proto/example/f.proto` will generate `src/python/example/f_pb2.py`, i.e., the generated files will share a source root with your other Python code.

Set the `.proto` file's `package` relative to the source root

Remember that the `package` directive in your `.proto` file should be relative to the source root.

For example, if you have a file at `src/proto/example/subdir/f.proto`, you'd set its `package` to `example.subdir`; and in your Python code, `from example.subdir import f_pb2`.

## Multiple resolves

If you're using [multiple resolves](🔗) (i.e. multiple lockfiles), then you may need to set the `python_resolve` field. `protobuf_source` targets only work with a single resolve, meaning, for example, that a `python_source` target that uses the resolve 'a' can only depend on Protobuf targets that also uses this same resolve.

By default, `protobuf_source` / `protobuf_sources` targets use the resolve set by the option `[python].default_resolve`. To use a different resolve, set the field `python_resolve: str` to one of the values from the option `[python].resolves`.

You must also make sure that any resolves that use codegen include `python_requirement` targets for the `protobuf` and `grpcio` runtime libraries from Step 2. Pants will eagerly validate this for you.

For example:



Pants 2.11 will be adding support for using the same `protobuf_source` target with multiple resolves through a new `parametrize()` feature.