When your Python code imports Protobuf generated files, Pants will detect the imports and run the Protoc compiler to generate those files.

📘

Example repository

See [the Python example repository](🔗) for an example of using Protobuf to generate Python.

👍

Benefit of Pants: generated files are always up-to-date

With Pants, there's no need to manually regenerate your code. Pants will ensure you are always using up-to-date files in your builds.

Thanks to fine-grained caching, Pants will regenerate the minimum amount of code required when you do make changes.

## Step 1: Activate the Protobuf Python backend

Add this to your `pants.toml`:

This adds the new [`protobuf_source`](🔗) target, which you can confirm by running `./pants help protobuf_source`.

To reduce boilerplate, you can also use the [`protobuf_sources`](🔗) target, which generates one `protobuf_source` target per file in the `sources` field.

📘

Enable the MyPy Protobuf plugin

The [MyPy Protobuf plugin](🔗) generates [`.pyi` type stubs](🔗). If you use MyPy through Pants's [typecheck goal](🔗), this will ensure MyPy understands your generated code.

To activate, set `mypy_plugin = true` in the `[python-protobuf]` scope:

Text



MyPy will use the generated `.pyi` type stub file, rather than looking at the `.py` implementation file.

🚧

Want to use other protocols, like Thrift?

Please message us on [Slack](🔗) if you would like support for more protocols. We would be happy to either add support to the core Pants distribution or to help you to write a plugin.

## Step 2: Set up the `protobuf` and `grpcio` runtime libraries

Generated Python files require the [`protobuf` dependency](🔗) for their imports to work properly. If you're using gRPC, you also need the [`grpcio` dependency](🔗).

First, add `protobuf`—and `grpcio`, if relevant— to your project, e.g. your `requirements.txt` (see [Third-party dependencies](🔗)).

Then, add the targets' addresses to the option `runtime_dependencies` in the `[python-protobuf]` scope. Pants will use this to automatically add the target(s) to the `dependencies` field for every `protobuf_source` target you write.

## Step 3: Generate `protobuf_sources` target

Run [`./pants tailor`](🔗) for Pants to create a `protobuf_sources` target wherever you have `.proto` files:

Pants will use [dependency inference](🔗) for any `import` statements in your `.proto` files, which you can confirm by running `./pants dependencies path/to/file.proto`. You can also manually add to the `dependencies` field.

If you want gRPC code generated for all files in the folder, set `grpc=True`.

If you only want gRPC generated for some files in the folder, you can use the `overrides` field:

## Step 4: Confirm Python imports are working

Now, you can import the generated Python module in your Python code. For example, to import `project/example/f.proto`, add `import project.example.f_pb2` to your code.

If you have [source roots](🔗) other than the repository root, remove the source root from the import. For example, `src/protos/example/f.proto` gets stripped to `import example.f_pb2`. See the below section on source roots for more info.

Pants's dependency inference will detect Python imports of Protobuf modules, which you can confirm by running `./pants dependencies path/to/file.py`.

If gRPC is activated, you can also import the module with `_pb2_grpc` at the end, e.g. `project.example.f_pb2_grpc`.

📘

Run `./pants export-codegen ::` to inspect the files

`./pants export-codegen ::` will run all relevant code generators and write the files to `dist/codegen` using the same paths used normally by Pants.

You do not need to run this goal for codegen to work when using Pants; `export-codegen` is only for external consumption outside of Pants.

🚧

You likely need to add empty `__init__.py` files

By default, Pants will generate the Python files in the same directory as the `.proto` file. To get Python imports working properly, you will likely need to add an empty `__init__.py` in the same location, and possibly in ancestor directories.

See the below section "Protobuf and source roots" for how to generate into a different directory. If you use this option, you will still likely need an empty `__init__.py` file in the destination directory.

## Protobuf and source roots

By default, generated code goes into the same [source root](🔗) as the `.proto` file from which it was generated. For example, a file `src/proto/example/f.proto` will generate `src/proto/example/f_pb2.py`.

However, this may not always be what you want. In particular, you may not want to have to add `__init__py` files under `src/proto` just so you can import Python code generated to that source root.

You can configure a different source root for generated code by setting the `python_source_root` field:

Now `src/proto/example/f.proto` will generate `src/python/example/f_pb2.py`, i.e., the generated files will share a source root with your other Python code.

📘

Set the `.proto` file's `package` relative to the source root

Remember that the `package` directive in your `.proto` file should be relative to the source root.

For example, if you have a file at `src/proto/example/subdir/f.proto`, you'd set its `package` to `example.subdir`; and in your Python code, `from example.subdir import f_pb2`.