When your Python code imports Protobuf generated files, Pants will detect the imports and run the Protoc compiler to generate those files.
See [the Python example repository](🔗) for an example of using Protobuf to generate Python.
Benefit of Pants: generated files are always up-to-date
With Pants, there's no need to manually regenerate your code. Pants will ensure you are always using up-to-date files in your builds.
Thanks to fine-grained caching, Pants will regenerate the minimum amount of code required when you do make changes.
## Step 1: Activate the Protobuf Python backend
Add this to your `
This adds the new [`
protobuf_source`](🔗) target, which you can confirm by running `
./pants help protobuf_source`.
To reduce boilerplate, you can also use the [`
protobuf_sources`](🔗) target, which generates one `
protobuf_source` target per file in the `
Enable the MyPy Protobuf plugin
The [MyPy Protobuf plugin](🔗) generates [`
.pyi` type stubs](🔗). If you use MyPy through Pants's [typecheck goal](🔗), this will ensure MyPy understands your generated code.
To activate, set `
mypy_plugin = true` in the `
MyPy will use the generated `
.pyi` type stub file, rather than looking at the `
.py` implementation file.
Want to use other protocols, like Thrift?
Please message us on [Slack](🔗) if you would like support for more protocols. We would be happy to either add support to the core Pants distribution or to help you to write a plugin.
## Step 2: Set up the `
protobuf` and `
grpcio` runtime libraries
Generated Python files require the [`
protobuf` dependency](🔗) for their imports to work properly. If you're using gRPC, you also need the [`
First, add `
grpcio`, if relevant— to your project, e.g. your `
requirements.txt` (see [Third-party dependencies](🔗)).
Then, add the targets' addresses to the option `
runtime_dependencies` in the `
[python-protobuf]` scope. Pants will use this to automatically add the target(s) to the `
dependencies` field for every `
protobuf_source` target you write.
## Step 3: Generate `
./pants tailor`](🔗) for Pants to create a `
protobuf_sources` target wherever you have `
Pants will use [dependency inference](🔗) for any `
import` statements in your `
.proto` files, which you can confirm by running `
./pants dependencies path/to/file.proto`. You can also manually add to the `
If you want gRPC code generated for all files in the folder, set `
If you only want gRPC generated for some files in the folder, you can use the `
## Step 4: Confirm Python imports are working
Now, you can import the generated Python module in your Python code. For example, to import `
project/example/f.proto`, add `
import project.example.f_pb2` to your code.
If you have [source roots](🔗) other than the repository root, remove the source root from the import. For example, `
src/protos/example/f.proto` gets stripped to `
import example.f_pb2`. See the below section on source roots for more info.
Pants's dependency inference will detect Python imports of Protobuf modules, which you can confirm by running `
./pants dependencies path/to/file.py`.
If gRPC is activated, you can also import the module with `
_pb2_grpc` at the end, e.g. `
./pants export-codegen ::` to inspect the files
./pants export-codegen ::` will run all relevant code generators and write the files to `
dist/codegen` using the same paths used normally by Pants.
You do not need to run this goal for codegen to work when using Pants; `
export-codegen` is only for external consumption outside of Pants.
You likely need to add empty `
By default, Pants will generate the Python files in the same directory as the `
.proto` file. To get Python imports working properly, you will likely need to add an empty `
__init__.py` in the same location, and possibly in ancestor directories.
See the below section "Protobuf and source roots" for how to generate into a different directory. If you use this option, you will still likely need an empty `
__init__.py` file in the destination directory.
## Protobuf and source roots
By default, generated code goes into the same [source root](🔗) as the `
.proto` file from which it was generated. For example, a file `
src/proto/example/f.proto` will generate `
However, this may not always be what you want. In particular, you may not want to have to add `
__init__py` files under `
src/proto` just so you can import Python code generated to that source root.
You can configure a different source root for generated code by setting the `
src/proto/example/f.proto` will generate `
src/python/example/f_pb2.py`, i.e., the generated files will share a source root with your other Python code.
Set the `
.proto` file's `
package` relative to the source root
Remember that the `
package` directive in your `
.proto` file should be relative to the source root.
For example, if you have a file at `
src/proto/example/subdir/f.proto`, you'd set its `
package` to `
example.subdir`; and in your Python code, `
from example.subdir import f_pb2`.