Integrating new tools without plugins
adhoc_tool target allows you to execute "runnable" targets inside the Pants sandbox. Runnable targets include first-party sources that can be run with
pants run, 3rd-party dependencies like
jvm_artifact, or even executables that exist on your system and managed externally to Pants.
adhoc_tool provides you with the building blocks needed to put together a custom build process without needing to develop and maintain a plugin. The level of initial effort involved in using
adhoc_tool is significantly lower than that of writing a plugin, so it's well-suited to consuming one-off scripts, or for rapidly prototyping a process before actually writing a plugin. The tradeoff is that there is more manual work involved in defining build processes that reflect your codebase's structure, and that the targets that define the tools you consume are less easy to reuse.
antlr demo in the
example-adhoc respository shows an example of running a JVM-based tool to transparently generate Python code that can be used in another language:
args=["Expr.g4", "-Dlanguage=Python3", "-o", "expr_parser", "-package", "expr_parser",],
# These are consumed by `antlr`, but are not relevant to this target's dependents.
# These are needed by the code that is output by this target
"Runnable" targets are targets that Pants knows how to execute within its sandbox. Generally, these correspond to targets that can be executed with the
pants run goal, and include first-party source files, as well as third-party dependencies.
The tool will be run with values from
args specified as arguments. By default, the process' working directory will be the directory where the
BUILD file is defined. This can be adjusted using the
runnable targets must be pure functions
When run by
adhoc_tool, Pants assumes that the inputs provided to the process -- that is, the values of the
adhoc_tool's fields, and the contents of the runnable and execution dependencies -- fully describe the output. Output values will be cached by Pants, and future invocations with identical inputs will be retrieved from the cache instead of being re-executed. If your process has behavior that is not fully defined by its inputs, Pants' behavior may be unexpected or inconsistent.
runnable targets must be idempotent
adhoc_tool processes may be cancelled or retried any number of times, so it is important that any side effects are idempotent. That is, it should not matter if it is run several times, or only partially.
adhoc_tool has more complexity surrounding dependencies compared with Pants' first-class targets. This is because you need to do manual work to set up the execution environment, which is usually taken care of by plugin code.
adhoc_tool has three dependencies fields:
output_dependencies, which defines dependencies that are required to effectively consume the output of the tool, e.g. runtime libraries for generated code bindings. Any targets that (transitively) depend on the
adhoc_tooltarget will also transitively depend on these dependencies.
execution_dependencies, which define data dependencies required for the tool to produce its output. These are not considered when resolving transitive dependencies that include this
runnable_dependencies, which define runnables that the
adhoc_toolneeds on its
PATHto execute as a subprocess. These are also not considered when resolving transitive dependencies. The discussion of
system_binarylater in this page shows one key use of
output_dependencies is used because the tool produces Python-based bindings that depend on a runtime library.
execution_dependencies specifies the sources that are consumed by the tool, but do not need to be consumed by subsequent build steps.
adhoc_tool targets are run to produce outputs that can be supplied to other targets. These can be in the form of files or directories that are output directly by the tools: use the
output_files field to capture individual files, or
output_directories to capture entire directories as output.
Files are captured relative to the build root by default: this is useful when passing results to further
adhoc_tool targets defined in the same
BUILD file. If this behavior is not right for you, for example, if you are producing an artifact for packaging, you can change the root of the outputs using the
Finally, if you want to capture
stderr from your tool, you can use the
stderr fields. These specify filenames where those streams will be dumped once the process completes. Note that these files are specified in addition to those from the
output_files field, and an error will occur if the filename occurs in the outputs arising from
output_directories and the contents of that file are different.
Chaining processes together
adhoc_tool targets that's used to produce a resource file.
To get the best cache efficiency, it can make sense to break your
adhoc_tool into smaller incremental steps. For example, if your process needs to fetch dependencies and then build a library based on those dependencies and some first-party source files, having one
adhoc_tool for each of those steps means that the dependency-fetching stage will only be re-run when your requirements change, and not when the first-party source files change.
Generally, if you are chaining
adhoc_tool targets together , it will be easier to use the default
root_output_directory fields for each step that will be consumed by an
adhoc_tool in the same
BUILD file. Change the
root_output_directory only for targets that are intended to be used in other places or ways.
Wrapping generated sources for use by other targets
Our Antlr demo demonstrates wrapping the outputs of
adhoc_tool targets for use as Python sources.
file sources by default. This can be acceptable if generating assets that do not need to be consumed as source files for another Pants backend. Other Pants backends need generated sources to be marked as actual source files.
There are several targets included in Pants with the prefix
experimental_wrap_as_. These act as a source target that can be used as a dependency in a given language backend, with the caveat that dependency inference is not available.
Using externally-managed tools
Some build processes need to make use of tools that can't be modeled within a Pants codebase. The
system_binary target lets you make use of a binary that is installed on the system.
system_binary targets may be specified as
runnable_dependency values for
system_binary will search for a binary in pre-defined or user-supplied search paths with a given
binary_name. To improve reproducibility, it's possible to test matching binaries with sample arguments, to see if its output matches a given regular expression. This can be used to match against version strings. If a such a binary does not exist on the system where Pants is being run, any build involving this
system_binary target will fail.
When specified as a
runnable_dependency, the binary will be available on the
PATH with the target name of the dependency. This can be important if the
runnable field invokes a subprocess (for example,
yarn tries to invoke a binary called
node as its interpreter).
Running shell scripts
shell_source targets are not runnable. In the meantime, it is possible to run a shell script as an
adhoc_tool through the following approach:
- Declare a
system_binarytarget referring to your preferred shell (e.g.
- Declare an
adhoc_tool, with the
runnablefield pointing at your
system_binarytarget, add your
execution_dependency, and provide your script's path relative to the buildroot as the first value in
This is clearly not as convenient as directly running a
shell_source target, and we anticipate adding support in a future version of Pants.