Updating Rust crates used by Chromium

This document describes how Chromium updates crates.io Rust crates that Chromium depends on.

Staffing

We have a weekly rotation of Google engineers responsible for creating and landing CLs that update Rust crates.

Google engineers can join the rotation by emailing [email protected].

Initial setup

The “Rust: periodic update of 3rd-party crates” rotation requires access to an up-to-date Chromium repo. One way to start a shift is to run git fetch, git checkout origin/main, and gclient sync (but other workflows should also work - e.g. ones based on git-new-workdir).

Avoiding conflicts

Before creating a CL stack, check for open CLs with the cratesio-autoupdate tag. Such CLs tend to conflict, so coordinate with owners of any open CLs.

Automated step: `create_update_cl.py`

The first actual step of the rotation is running create_update_cl.py. You must invoke it from within the src/ directory of a Chromium repository checkout, and it depends on depot_tools and git being present in the PATH.

$ cd ~/chromium/src  # or wherever you have your checkout
$ tools/crates/create_update_cl.py auto

In auto mode, it runs gnrt update to discover crate updates and then for each update creates a new local git branch (and a Gerrit CL unless invoked with --no-upload). Each branch contains an update created by gnrt update <old crate id>, gnrt vendor, and gnrt gen. Depending on how many crates are updated, the script may need 10-15 minutes to run.

The script should Just Work in most cases, but sometimes it may fail when dealing with a specific crate update. See Recovering from script failures below for what to do when that happens.

Before the auto-generated CLs can be landed, you will need to get an LGTM from //third_party/rust/OWNERS. See //third_party/rust/OWNERS-security-review.md for a review checklist.

New transitive dependencies

Notes from //third_party/rust/OWNERS-security-review.md apply:

The dependency will need to go through security review.
An FYI email should be sent to [email protected] in order to record the addition.

Optional: Adding the transitive dependency in its own CL

If the new crate is non-trivial, it's possible to split the additional crate into its own CL, however then it will default to global visibility and allowing non-test use.

gnrt add and gnrt vendor can add the dependency to a fresh checkout.
Mark the crate as being for third-party code only by setting allow_first_party_usage to false for the crate in third_party/rust/chromium_crates_io/gnrt_config.toml.
If the crates making use of the transitive dependency are only allowed in tests, then set group = 'test' for the crate in third_party/rust/chromium_crates_io/gnrt_config.toml. This reduces the level of security review required for the library.
gnrt gen will then generate the GN rules.
Rebase the roll CL on top of the changes to make sure the choices made above are correct. gn gen will fail in CQ if the crate was placed in the 'test' group but needs to be visible outside of tests.

Potential additional steps

The create_update_cl.py script may stop early if it detects that gnrt vendor or gnrt gen have reported any warnings or errors (e.g. a “License file not found for crate foo” warning). In this case, manual intervention is needed to finish the update CL. It's probably best to finish and land the CLs created so far before trying to restart the script in order to create the remaining CLs.

Landing the CL

Other than the above, the CL can go through the normal, plain-vanilla, manual review and landing process.

git cl upload
Get a review from one of //third_party/rust/OWNERS
Land the CL using CQ+2

Checking for new major versions

Note that create_update_cl.py auto will by default only handle minor version updates (e.g. 123.1 => 123.2, or 0.123.1 => 0.123.2). Major version changes (e.g. 1.0 => 2.0, which may include breaking API changes and other breaking changes) need to be handled separately - this section describes what to do.

Detecting available major version updates

As part of the rotation, one should attempt to check for new major versions of direct Chromium dependencies (i.e. dependencies directly listed in third_party/rust/chromium_crates_io/Cargo.toml). To discover direct and transitive dependencies with a new major version, you can use the command below (running it in the final update CL branch - after all the minor version updates):

$ tools/crates/run_cargo.py -Zunstable-options -C third_party/rust/chromium_crates_io -Zbindeps update --dry-run --verbose
...
   Unchanged serde_json_lenient v0.1.8 (latest: v0.2.0)
   Unchanged syn v1.0.109 (latest: v2.0.53)
...

Major version update: Workflow A: Single update CL

If updating to a new major version doesn‘t require lots of Chromium changes, then it may be possible to land the update in a single CL. This is typically possible when the APIs affected by the major version’s breaking change either weren't used by Chromium, or were used only in a handful of places.

Warning: Sometimes a new major version may be API compatible, but may introduce breaking changes in the behavior of the existing APIs.

To update:

tools/crates/create_update_cl.py auto -- some_crate_name --breaking
Follow the manual steps from the minor version update rotation for review, landing, etc.

Major version update: Workflow B: Incremental transition

When lots of first-party code depends on the old major version, then the transition to the new major version may need to be done incrementally. In this case the transition can be split into the following steps:

Open a new bug to track the transition
- TODO: Figure out how to tag/format the bug to make it easy to discover in future rotations
Land the new major version, so that the old and the new versions coexist. To do this follow the process for importing a new crate as described in docs/rust.md (i.e. edit Cargo.toml to add the new version, run gnrt vendor, and so forth).
Incrementally transition first-party code to the new major version
Remove the old major version. To do this follow a similar process as above (i.e. edit Cargo.toml to remove the old version, run gnrt vendor, and so forth). Any leftover files in //third_party/rust/<crate>/<old epoch> should also be removed.

Note that the following Cargo.toml syntax allows two versions of a crate to coexist:

[dependencies.serde_json_lenient_old_epoch]
package = "serde_json_lenient"
version = "0.1"

[dependencies.serde_json_lenient]
version = "0.2"

Other ways to use `create_update_cl.py`

`auto` mode

Extra arguments passed to create_update_cl.py auto end up being passed to cargo update. For a complete list of available options, see Cargo documentation here), but the most common scenarios are covered in the sections below.

Updating all crates during the weekly rotation

tools/crates/create_update_cl.py auto with no extra arguments will attempt to discover minor version updates for all crates that Chromium depends on and for their transitive dependencies.

Updating the minor version of a single crate

tools/crates/create_update_cl.py auto -- some_crate_name can be used to trigger a minor version update of a single crate.

Updating the major version of a single crate

tools/crates/create_update_cl.py auto -- some_crate_name --breaking can be used to trigger a major version update of a single crate

`manual` mode

For maximal control, the script can be used in manual mode:

Prepare Cargo.toml change:
1. git checkout origin/main
2. git checkout -b manual-update-of-foo
3. Edit third_party/rust/chromium_crates_io/Cargo.toml to change the crate version of the crate (or crates) you want to update. Important: Do not edit Cargo.lock (e.g. don't run gnrt vendor etc.).
4. git add third_party/rust/chromium_crates_io/Cargo.toml
5. git commit -m "Manual edit of Cargo.toml"
6. git cl upload -m "Manual edit of Cargo.toml" --bypass-hooks --skip-title --force
Run the helper script as follows: tools/crates/create_update_cl.py manual --title "Roll foo crate to new version X"
- This will run gnrt vendor to discover and execute updates that were requested by the manual edits of Cargo.toml in the previous steps.
- This will automatically add more details to the CL description
- To make the review easier, one of the patchsets covers just the path changes. For example - see the delta here.

Recovering from script failures

Sometimes the create_update_cl.py script will fail when dealing with a specific crate update. The general workflow in this case is to

fix the issue in a separate CL, and 2) restart the tool from the middle by using --upstream-branch that points to the last successful update branch (or to the fix CL) rather than defaulting to origin/main.

Examples of a few specific situations that may lead to script failure:

An update brought in a new crate, but gnrt didn‘t recognize new crate’s license kind or license file. In that case a prerequisite CL needs to be landed first, teaching gnrt about the new license kinds/files (in readme.rs). You can see an example CL with such a fix.
Patches from //third_party/rust/chromium_crates_io/patches/ no longer apply cleanly to the new version of a crate. In that case the crate update CL needs to 1) first update the patches, and then 2) update the crate as usual. This is not very well supported by the script... But something like this should work:
- Checkout a new branch:
```
$ git checkout rust-crates-update--last-successful-update
$ git checkout -b fix-patches-for-foo
$ git branch --set-upstream-to=rust-crates-update--last-successful-update
```
- Fix the patches and upload as a temporary / throw-away CL (this CL can't be landed on its own - it needs to be combined with the actual update CL):
```
$ # Fix the patches
$ git commit -a -m ...
$ git cl upload
```
- Restart the script (the CL created by the script can't be landed as-is / on its own - it needs to be combined with the fixed patches in the step below) with --upstream-branch parameter:
```
$ tools/crates/create_update_cl.py auto \
    --upstream-branch=fix-patches-for-foo \
    -- name-of-failed-crate
```
- Combine the branches:
```
$ git map-branches -v # to orient yourself
$ git checkout rust-crates-update--new-successful-update
$ git branch --set-upstream-to=rust-crates-update--last-successful-update
$ git cl upload -m Rebasing... # --bypass-hooks as needed
```
//third_party/rust/chromium_crates_io/gnrt_config.toml needs to be updated to work with a new crate version. The same workflow should work as for fixing //third_party/rust/chromium_crates_io/patches/ (see the item above).