Protobuf

Pants can generate Python code from the popular Protobuf (Protocol Buffer) format.

When you depend on a protobuf_library in a Python target (like a python_library), Pants will run the Protoc compiler to generate Python code that you can import and use like normal Python code.

📘

Example repository

See the Python example repository for an example of using Protobuf to generate Python.

Step 1: Activate the Protobuf Python backend

Add this to your pants.toml:

[GLOBAL]
backend_packages2.add = [
  "pants.backend.codegen.protobuf.python",
  "pants.backend.python",
]

This adds the new protobuf_library target, which you can confirm by running <<pantscmd>> target-types --details=protobuf_library.

🚧

Want to use other protocols, like Thrift?

Please message us on Slack if you would like support for more protocols. We would be happy to either add support to the core Pants distribution or to help you to write a plugin.

Step 2: Set up the Protobuf Python runtime library

Generated Python files require the protobuf library for their imports to work properly.

First, add protobuf to your requirements.txt (see Third-party dependencies).

protobuf>=3.12.1

Then, add the address to the target to the option runtime_targets in the [protoc] scope. Pants will use this to automatically add the target to the dependencies field for every protobuf_library() target you write.

[protoc]
runtime_targets = ["//:protobuf"]

Step 3: Define a protobuf_library target

Wherever you create your .proto files, add a protobuf_library.

# `sources` defaults to `['*.proto']`.
protobuf_library()

syntax = "proto3";

package protos;

message Example {
  ...
}

Your protobuf_library can optionally depend on other protobuf_library targets through the dependencies field.

protobuf_library(
    dependencies=[
        "protos",
    ],
)

syntax = "proto3";

package protos.subdir;

import "example.proto";

message SubdirExample {
  ...
}

Step 4: Use the protobuf_library in your Python code

Now, you can add the protobuf_library to the dependencies field of your Python targets. Pants will generate the Python code automatically for you.

In your Python file, import the module with the name _pb2 at the end, e.g. protos/example.proto becomes proto.example_pb2.

python_library(
    dependencies=[
        "protos",
        "protos/subdir",
    ],
)

from protos.example_pb2 import Example
from protos.subdir.example_pb2 import SubdirExample

# See https://developers.google.com/protocol-buffers/docs/pythontutorial
# for how to use the generated code in your project.

📘

Protobuf and source roots

Pants will strip the source root for you.

For example, if you have a file at src/protos/example/f.proto, Pants will treat this as example/f.proto. So—in this example—in the .proto file, set the package as example; and in your Python code, import example.f_pb2.

🚧

Want to open up the generated files?

Please message us on Slack if you would like to be able to save the generated files to your repository. We are considering adding this feature and would appreciate your feedback.

For now, you can add the protobuf_library() as a dependency of a python_binary(), then run <<pantscmd>> binary on the binary target, and finally inspect the built PEX by using unzip.