When your Python code imports Protobuf generated files, Pants will detect the imports and run the Protoc compiler to generate those files.
Example repository
See [the Python example repository](🔗) for an example of using Protobuf to generate Python.
Benefit of Pants: generated files are always up-to-date
With Pants, there's no need to manually regenerate your code. Pants will ensure you are always using up-to-date files in your builds.
Thanks to fine-grained caching, Pants will regenerate the minimum amount of code required when you do make changes.
## Step 1: Activate the Protobuf Python backend
Add this to your `pants.toml
`:
This adds the new [`protobuf_source
`](🔗) target, which you can confirm by running `./pants help protobuf_source
`.
To reduce boilerplate, you can also use the [`protobuf_sources
`](🔗) target, which generates one `protobuf_source
` target per file in the `sources
` field.
Enable the MyPy Protobuf plugin
The [MyPy Protobuf plugin](🔗) generates [`
.pyi
` type stubs](🔗). If you use MyPy through Pants's [typecheck goal](🔗), this will ensure MyPy understands your generated code.To activate, set `
mypy_plugin = true
` in the `[python-protobuf]
` scope:MyPy will use the generated `
.pyi
` type stub file, rather than looking at the `.py
` implementation file.
Want to use other protocols, like Thrift?
Please message us on [Slack](🔗) if you would like support for more protocols. We would be happy to either add support to the core Pants distribution or to help you to write a plugin.
## Step 2: Set up the `protobuf
` and `grpcio
` runtime libraries
Generated Python files require the [`protobuf
` dependency](🔗) for their imports to work properly. If you're using gRPC, you also need the [`grpcio
` dependency](🔗).
First, add `protobuf
`—and `grpcio
`, if relevant— to your project, e.g. your `requirements.txt
` (see [Third-party dependencies](🔗)).
Then, add the targets' addresses to the option `runtime_dependencies
` in the `[python-protobuf]
` scope. Pants will use this to automatically add the target(s) to the `dependencies
` field for every `protobuf_source
` target you write.
## Step 3: Generate `protobuf_sources
` target
Run [`./pants tailor
`](🔗) for Pants to create a `protobuf_sources
` target wherever you have `.proto
` files:
Pants will use [dependency inference](🔗) for any `import
` statements in your `.proto
` files, which you can confirm by running `./pants dependencies path/to/file.proto
`. You can also manually add to the `dependencies
` field.
If you want gRPC code generated for all files in the folder, set `grpc=True
`.
If you only want gRPC generated for some files in the folder, you can use the `overrides
` field:
## Step 4: Confirm Python imports are working
Now, you can import the generated Python module in your Python code. For example, to import `project/example/f.proto
`, add `import project.example.f_pb2
` to your code.
If you have [source roots](🔗) other than the repository root, remove the source root from the import. For example, `src/protos/example/f.proto
` gets stripped to `import example.f_pb2
`. See the below section on source roots for more info.
Pants's dependency inference will detect Python imports of Protobuf modules, which you can confirm by running `./pants dependencies path/to/file.py
`.
If gRPC is activated, you can also import the module with `_pb2_grpc
` at the end, e.g. `project.example.f_pb2_grpc
`.
Run `
./pants export-codegen ::
` to inspect the files`
./pants export-codegen ::
` will run all relevant code generators and write the files to `dist/codegen
` using the same paths used normally by Pants.You do not need to run this goal for codegen to work when using Pants; `
export-codegen
` is only for external consumption outside of Pants.
You likely need to add empty `
__init__.py
` filesBy default, Pants will generate the Python files in the same directory as the `
.proto
` file. To get Python imports working properly, you will likely need to add an empty `__init__.py
` in the same location, and possibly in ancestor directories.See the below section "Protobuf and source roots" for how to generate into a different directory. If you use this option, you will still likely need an empty `
__init__.py
` file in the destination directory.
## Protobuf and source roots
By default, generated code goes into the same [source root](🔗) as the `.proto
` file from which it was generated. For example, a file `src/proto/example/f.proto
` will generate `src/proto/example/f_pb2.py
`.
However, this may not always be what you want. In particular, you may not want to have to add `__init__py
` files under `src/proto
` just so you can import Python code generated to that source root.
You can configure a different source root for generated code by setting the `python_source_root
` field:
Now `src/proto/example/f.proto
` will generate `src/python/example/f_pb2.py
`, i.e., the generated files will share a source root with your other Python code.
Set the `
.proto
` file's `package
` relative to the source rootRemember that the `
package
` directive in your `.proto
` file should be relative to the source root.For example, if you have a file at `
src/proto/example/subdir/f.proto
`, you'd set its `package
` to `example.subdir
`; and in your Python code, `from example.subdir import f_pb2
`.