When you depend on a protobuf_library
in a Python target (like a python_library
), Pants will run the Protoc compiler to generate Python code that you can import and use like normal Python code.
Example repository
See the Python example repository for an example of using Protobuf to generate Python.
Benefit of Pants: generated files are always up-to-date
With Pants, there's no need to manually regenerate your code. Pants will ensure you are always using up-to-date files in your builds.
Thanks to fine-grained caching, Pants will regenerate the minimum amount of code required when you do make changes.
Step 1: Activate the Protobuf Python backend
Add this to your pants.toml
:
[GLOBAL]
backend_packages.add = [
"pants.backend.codegen.protobuf.python",
"pants.backend.python",
]
This adds the new protobuf_library
target, which you can confirm by running ./pants help protobuf_library
.
Enable the MyPy Protobuf plugin
The MyPy Protobuf plugin generates
.pyi
type stubs. If you use MyPy through Pants's typecheck goal, this will ensure MyPy understands your generated code.To activate, set
mypy_plugin = true
in the[python-protobuf]
scope:[python-protobuf] mypy_plugin = true
MyPy will use the generated
.pyi
type stub file, rather than looking at the.py
implementation file.
Want to use other protocols, like Thrift?
Please message us on Slack if you would like support for more protocols. We would be happy to either add support to the core Pants distribution or to help you to write a plugin.
Step 2: Set up the protobuf
and/or grpcio
runtime libraries
protobuf
and/or grpcio
runtime librariesGenerated Python files require the protobuf
library for their imports to work properly. If you're using gRPC, you also need the grpcio
library.
First, add protobuf
βand grpcio
, if relevantβ to your requirements.txt
(see Third-party dependencies).
grpcio==1.32.0
protobuf>=3.12.1
Then, add the targets' addresses to the option runtime_dependencies
in the [python-protobuf]
scope. Pants will use this to automatically add the target(s) to the dependencies
field for every protobuf_library()
target you write.
[python-protobuf]
# Use the path to your 3rd-party requirements,
# e.g. `3rdparty/python:protobuf`.
runtime_dependencies = ["//:grpcio", "//:protobuf"]
Step 3: Define a protobuf_library
target
protobuf_library
targetWherever you create your .proto
files, add a protobuf_library
.
# `sources` defaults to `['*.proto']`.
protobuf_library()
syntax = "proto3";
package project.example;
message Example {
...
}
Pants will use dependency inference for any import
statements in your .proto
files, which you can confirm by running ./pants dependencies path/to/file.proto
. You can also manually add to the dependencies
field.
If you want gRPC code generated, set grpc=True
.
protobuf_library(grpc=True)
Step 4: Use the protobuf_library
in your Python code
protobuf_library
in your Python codeNow, you can import the generated Python module in your Python code. For example, to import project/example/f.proto
, add import project.example.f_pb2
to your code.
Pants's dependency inference will detect Python imports of Protobuf modules, which you can confirm by running ./pants dependencies path/to/file.py
.
If gRPC is activated, you can also import the module with _pb2_grpc
at the end, e.g. project.example.f_pb2_grpc
.
from project.example.f_pb2 import HelloReply
from project.example.f_pb2_grcp import GreeterServicer
Run
./pants export-codegen ::
to inspect the files
./pants export-codegen ::
will run all relevant code generators and write the files todist/codegen
using the same paths used normally by Pants.You do not need to run this goal for codegen to work when using Pants;
export-codegen
is only for external consumption outside of Pants.
You likely need to add empty
__init__.py
filesBy default, Pants will generate the Python files in the same directory as the
.proto
file. To get Python imports working properly, you will likely need to add an empty__init__.py
in the same location, and possibly in ancestor directories. You do not need to add apython_library()
target; Pants will automatically include the file.See the below section "Protobuf and source roots" for how to generate into a different directory. If you use this option, you will still likely need an empty
__init__.py
file in the destination directory.
Protobuf and source roots
By default, generated code goes into the same source root as the .proto
file from which it was generated. For example, a file src/proto/example/f.proto
will generate src/proto/example/f_pb2.py
.
However this may not always be what you want. In particular, you may not want to have to add __init__py
files under src/proto
just so you can import Python code generated to that source root.
You can configure a different source root for generated code by setting the python_source_root
field:
protobuf_library(
python_source_root='src/python'
)
Now src/proto/example/f.proto
will generate src/python/example/f_pb2.py
, i.e., the generated files will share a source root with your other Python code.
Set the
.proto
file'spackage
relative to the source rootRemember that the
package
directive in your.proto
file should be relative to the source root.For example, if you have a file at
src/proto/example/subdir/f.proto
, you'd set itspackage
toexample.subdir
; and in your Python code,from example.subdir import f_pb2
.
Updated 9 days ago