Streaming Workunit Handlers

Overview

Pants uses an internal tracing format called "workunits" for tracing execution timing. Workunits are mostly equivalent to tracing spans in other systems. The engine supports "streaming workunit handlers" to process and send workunits to monitoring systems.

Handler Setup

To setup a streaming workunit handler, plugins should do the following:

1. First, implement the WorkunitsCallbackFactoryRequest union with a rule which returns a WorkunitsCallbackFactory so the engine can gather workunit handler callbacks.

2. The WorkunitsCallbackFactory.callback_factory when invoked should either return a WorkunitsCallback, or else None if no handler should be activated.

3. The WorkunitsCallback can be any type which is callable and accepts the signature declared in the Pants source, and also implements a can_finish_async property which informs Pants whether Pants should wait or not for the callback to complete at the end of a run. The parameters passed to the callable are:

• started_workunits: Workunits which were started.
• completed_workunits: Workunits which completed.
• finished: When True, the end of the session has been reached and the callback will not be invoked any more.
• context: Useful context including access to the engine, and session-specific metrics like counters and observation histograms.

Workunit Format

Workunits are untyped Python dictionaries and have the following structure:

Field	Type	Sample	Description
name	string	pants.backend.project_info.list_targets.list_targets	Name of the workunit.
span_id	string	"56a74a32fdade07b"	Unique identifider for this workunit.
parent_ids	list[string]	['56a74a32fdade07b']	List of the span IDs which are the parent(s) of this workunit in the tracing graph.
level	string (enum)	DEBUG	The logging level for this workunuit. One of ERROR, WARN, INFO, DEBUG, or TRACE.
description	string	"list goal'"	Human-readable description of the task represented by the workunit.
start_secs	integer	1738032069	Numer of seconds since the unix epoch UTC for the start of the workunit's execution.
start_nanos	integer	440807186	Sub-second nanoseconds comoponent of the workunit's start time.
duration_secs	integer	0	Seconds component of the duration of time during which this workunit executed. to compute the end time, add this value (along with duration_nanos) to the time represented by start_secs and start_nanos.
duration_nanos	integer	18695352	Sub-second nanoseconds component of the time during which this workunit executed.
metadata	object		Workunit-specific mapping of metadata values. Metadata may be added by Pants rules to annonate a workunit's execution. (See the EngineAwareParameter.metadata and EngineAwareReturnType.metadata methods for more details.)
artifacts	object		Workunit-specific mapping of artifacts added by Pants rules to annotate workunit execution. (See the EngineAwareReturnType.artifacts method for more details.) The mapping will be filled with the name of the artifact as the key and a value represening either a "file digest" or "digest snapshot" represented by the following instance classes: FileDigest with fingerprint and serialized_bytes_length properties Snapshot with digest, dirs, and files properties. digest has fingerprint and serialized_bytes_length properties.
counters	object		DEPRECATED. This used to store the counters which were incremented during the execution of this partcular workunit. For various reasons, Pants does not track counter increments to workunuits any more here. Use the global counters for a session instead which can be obtaind from the StreamingWorkunitContext.get_metrics method.

See the workunit_to_py_value function for details of how a Rust workunit value is exposed to Python rule code.

Known Metadata

There is no specification currently for the metadata generated by Pants (besides the source code which is of course not an actual specification). This section has been derived empiracally by observing actual traces. Pants makes no promises not to change these schemas.

Process Metadata

This metadata appears on process spans:

Key	Type	Sample	Description
definition	object	see below	Process definiiton which was executed by Pants. See sample JSON below this table.
environment_type	string (enum)	local	The kind of environment in which the process executed (originally executed if cached). Valid values are docker, local, remote, and workspace corresponding to the docker_environment, local_environment, remote_environment, and experimental_workspace_environment target types, respectively.
exit_code	int	0	Exit code of the process execution
saved_by_cache_ms	int	62	Time saved by using a cache over re-executing this process (if this process was cached) in milliseconds.
source	string (enum)	HitLocally	Represents whether process was actually run or was cached. Valid values: Ran: The process was actually executed. HitLocally: The process output was cached locally. HitRemotely: The process output was cached remotely.
total_elapsed_ms	int	100	Total time to execute the process originally in milliseconds.

Sample JSON for the process definition (as of Pants 2.24.x):

{
    "argv": [
        "__coursier/coursier_post_process_stderr.py",
        "/bin/bash",
        "__coursier/coursier_fetch_wrapper_script.sh",
        "__coursier/./cs-x86_64-apple-darwin",
        "coursier_report.json",
        "--intransitive",
        "com.lihaoyi:sourcecode_2.13:0.3.0"
    ],
    "env": {
        "COURSIER_ARCHIVE_CACHE": ".cache/arc",
        "COURSIER_CACHE": ".cache/82ce2b8f4bc8f448df4f6ed5355885af1faeecda677ab193fad51e95d47d1677/jdk",
        "COURSIER_JVM_CACHE": ".cache/v1"
    },
    "working_directory": null,
    "input_digests": {
        "complete": {
            "digest": {
                "fingerprint": "2c6a51e70e246e42a6242160a954e8c54d9c35b9ec6dc6baa9fa7748ae8e95ed",
                "size_bytes": 85
            }
        },
        "nailgun": {
            "digest": {
                "fingerprint": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
                "size_bytes": 0
            }
        },
        "inputs": {
            "digest": {
                "fingerprint": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
                "size_bytes": 0
            }
        },
        "immutable_inputs": {
            "__coursier": {
                "digest": {
                    "fingerprint": "8c70f32ac92ed8ac5c7471940f415b6f9194a8a1bf5ea6a781d1a37236f115f2",
                    "size_bytes": 429
                }
            }
        },
        "use_nailgun": []
    },
    "output_files": [
        "coursier_report.json"
    ],
    "output_directories": [
        "classpath"
    ],
    "timeout": null,
    "execution_slot_variable": null,
    "concurrency_available": 0,
    "description": "Fetching with coursier: com.lihaoyi:sourcecode_2.13:0.3.0",
    "level": "DEBUG",
    "append_only_caches": {
        "coursier": ".cache",
        "python_build_standalone": ".python-build-standalone"
    },
    "jdk_home": null,
    "cache_scope": "Successful",
    "execution_environment": {
        "name": null,
        "platform": "Macos_x86_64",
        "strategy": "Local"
    },
    "remote_cache_speculation_delay": {
        "secs": 0,
        "nanos": 0
    },
    "attempt": 0
}