Writing Python lockfile metadata to separate files

Image generated by ChatGPT

The Pants Python backend uses lockfiles extensively to support repeatable, secure builds.

These lockfiles are in a proprietary JSON format defined by the underlying Pex tool. However, Pants associates some extra metadata with each such Pex-created lockfile.

Until recently, this metadata was tacked on to the lockfile as front matter. Pants would remove this metadata before passing the lockfile on to Pex. This was convenient in some ways, but had the very unfortunate effect of making those lockfiles invalid JSON, and unreadable by Pex when used outside of Pants. This was a regrettable design choice which we are now remedying.

Starting in the just-released Pants 2.30.0, this lockfile metadata can be written to a separate file. Specifically, if enabled, the metadata for path/to/lockfile.json will be stored in a sibling file path/to/lockfile.json.metadata.

What does this mean for me?

We recommend enabling this feature after you upgrade to Pants 2.30.0 or higher. To do so set the following option in your pants.toml:

[python]
separate_lockfile_metadata_file = true

This will become the default in Pants 2.31.0, and eventually the prepended header will be phased out entirely.

Note that even with this option enabled, Pants will read the prepended header if it finds one. This option only controls where Pants places this metadata when you generate a new lockfile.

Nonetheless, you may want to get ready for the future (and enjoy the benefits of unbroken JSON) by converting your existing lockfiles now. You can do so by regenerating them, but of course that might pick up dependency changes you don't want to deal with at the moment.

Your other option is to run the following script on your existing lockfiles. Simply copy this into a file, say convert_lockfiles.py:

import sys

for path in sys.argv[1:]:
    with open(path, "r") as fp:
        lines = fp.read().splitlines(keepends=False)
        metadata_lines = []
        in_metadata_block = False
        delim = lines[0].split()[0] + " "
        lines_iter = iter(lines)
        while True:
            line = next(lines_iter)
            if line == f"{delim}This lockfile was autogenerated by Pants. To regenerate, run:":
                next(lines_iter)
                cmd = next(lines_iter)[len(delim):].strip()
            if line == f"{delim}--- END PANTS LOCKFILE METADATA ---":
                remaining_lines = list(lines_iter)
                break
            elif in_metadata_block:
                metadata_lines.append(line[len(delim):])
            elif line == f"{delim}--- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---":
                in_metadata_block = True
        metadata_lines.insert(1, f'"description": "This lockfile was generated by Pants. To regenerate, run: {cmd}",')
        metadata = "\n".join(metadata_lines) + "\n"
        with open(f"{path}.metadata", "w") as fp:
            fp.write(metadata)
        with open(path, "w") as fp:
+            fp.write("\n".join(remaining_lines).strip() + "\n")

then run python convert_lockfiles.py path/to/lockfile1 path/to/lockfile2 ...

and voila, your lockfiles will be nicely split up without incurring any other perturbations!