Ability to query the set of activated features

[thesayyn]

Problem

We are people interested in making rust support in Bazel better, and one of the requirements is we determine the dependency graph for a given workspace using the information provided by Cargo.

rules_rust implements a whole set of tools to accommodate for these shortcomings https://github.com/bazelbuild/rules_rust/tree/main/crate_universe

Proposed Solution

I am proposing to change cargo to record features in use for all the crates recorded in the lockfile.

[[package]]
name = "clap"
version = "4.5.37"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eccb054f56cbd38340b380d4a8e69ef1f02f1af43db2f0cc817a4774d80ae071"
dependencies = [
 "clap_builder",
 "clap_derive",
]
features = ["color", "default", "derive", "error-context", "help", "std", "suggestions", "usage", "wrap_help"]

Currently cargo.lock does not record the unified list of features (in use) for all the crates listed in the lockfile.

Listing clap in Cargo.toml

clap = { version = "4.5.37", features = ["derive", "env"] }

Gets recorded in the Cargo.lock as

[[package]]
name = "clap"
version = "4.5.37"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eccb054f56cbd38340b380d4a8e69ef1f02f1af43db2f0cc817a4774d80ae071"
dependencies = [
 "clap_builder",
 "clap_derive",
]

Currently, the lockfile entry does not record a crucial information about what features are in use by the crate making it not really useful for tools other than cargo itself.

Only place where the information about the features are recorded is target/<cfg>/.fingerprint/clap-<hash>/lib-clap.json, as a workaround we could read this file, due to nature of target folder being VCS ignored forces users to run cargo first whenever they invoke bazel.

Ideally Cargo.lock should be the only source of truth to go from set of dependencies into a full dependency graph.

Notes

No response

[epage]

As an FYI I made a crude re-ordering of content to better fit the template. Wanting more information in a lockfile is a solution to a problem, not the problem itself. We need to make sure we focus on the root problem to keep ourselves open to ways to solve this.

In general, we try to keep information to a minimum in lockfiles to avoid churn and conflicts. For instance, #5979 is requesting us to include less information. The lockfile as of today also doesn't even include the minimal set of dependencies which I assume you would want (#10801). Another consideration for lockfiles is stability: only explicit user actions should change the file.

The question of features-in-use is also dependent on

• The crate in question. With resolver = "2", the features activated are different depending on whether a dependency is being built for a build script or not. For resolver = "1" behavior, you can use cargo metadata (see also cargo-metadata always resolves features at the workspace level #7754).
• The command-line flags

We have a GSoC project / Project Goal for creating more general cargo plumbing commands. The intent would be that you could resolve dependencies and features like cargo would.

[weihanglo]

Zulip thread https://rust-lang.zulipchat.com/#narrow/channel/334885-t-cargo.2Fbuild-integration/topic/Better.20Bazel.20integration.20and.20missing.20features.20in.20Cargo/with/537920481

though the above comment from epage is pretty comprehensive already

[thesayyn]

In general, we try to keep information to a minimum in lockfiles to avoid churn and conflicts. For instance, #5979 is requesting us to include less information.

To me a person who works with lockfiles all the time, this sounds like the opposite of what you'd want to do and i don't really see the point of trying to keep a lockfile stable if something in the dependency graph has changed. From reproducible builds point of view, a lock file ought to contain all the necessary information to go from a text file to set of rustc commands to invoke.

I understand that improvement above is dependent on what features you want to compile with, but that i don't see how that is preventing the lockfile from recording the default features (no additional --features flag present on the command line) at the end of the day the lockfile is representation of cargo.toml and we already list packages and their features there.

FWIW, This is a similar problem with requirements_lock.txt not providing enough information, if you are familiar with the python ecosystem.

[thesayyn]

i'll try stay curious a little more, do you have any suggestions for a solution that does not require making api calls to crates.io or invoking a command perform a re-resolution?

[bjorn3]

Even without --features the set of features that will be used depends on which package in a workspace you are building. Or the exact target you are building for (given that target specific dependencies can enable features, which with the v2 resolver only have effect when compiling for said target). Also a single package can get compiles twice with different features enabled in a single compilation if it is used as both build and runtime dependency.

i'll try stay curious a little more, do you have any suggestions for a solution that does not require making api calls to crates.io or invoking a command perform a re-resolution?

If you have vendored all dependencies, cargo metadata will not contact crates.io, but use the Cargo.toml files in the vendored packages. If you haven't vendored dependencies and cargo hasn't cached the regsistry entries for the dependencies, then there is no way to determine the enabled features without fetching those registry entries from crates.io.

To me a person who works with lockfiles all the time, this sounds like the opposite of what you'd want to do and i don't really see the point of trying to keep a lockfile stable if something in the dependency graph has changed.

If you enable/disable a feature you still have the same dependencies. They just might not get compiled or get compiled with different flags.

Edit: Cargo changes the lockfile only if Cargo.toml changes since the lockfile was written or you ask it to update a dependency. It will never change it as a result of a change in the build commando. That would void the reason cargo has lockfiles.

[epage]

a lock file ought to contain all the necessary information to go from a text file to set of rustc commands to invoke.

That is a lot larger of a scope than requested here and is sounding more similar to build-plan or unit-graph as you are looking to have

• all build units
• all of the information derived from Cargo.toml that would lead to compilation flags (e.g. [lints], [profiles], etc)
• Build scripts are an inherent limit in this

If we included effectively the unit graph in the file, that would lead to the problems noted

• upgrading cargo would cause the file to change
• the file would be large containing all of that
• the file would change regularly as you edit things locally
• the file would be ripe for merge conflicts when dealing with dependency upgrades

From reproducible builds point of view, a lock file ought to contain all the necessary information to go from a text file to set of rustc commands to invoke.

For reproducible builds, more likely what would be wanted is for cargo to create a dump of rustc invocations, making the assumption that build scripts will not need to be re-run.

I understand that improvement above is dependent on what features you want to compile with, but that i don't see how that is preventing the lockfile from recording the default features (no additional --features flag present on the command line) at the end of the day the lockfile is representation of cargo.toml and we already list packages and their features there.

That then becomes limited in value if whats recorded is only one instance of the features but there can be many more combinations a query may need to do. Also, this runs counter to lockfiles intending to be environment or run independent.

i'll try stay curious a little more, do you have any suggestions for a solution that does not require making api calls to crates.io or invoking a command perform a re-resolution?

Currently, our official answer plans to require running a command.

As a stop gap, hakari is a community effort for doing feature resolution. I do not know of its requirements or limitations.

[illicitonion]

I think the biggest complexity here is the interaction between features and platforms. Because different platforms may enable both different dependencies and different features by default, across the entire transitive dep graph, this is hard information to encode in a single format - you effectively end up with something conditional similar to parsing through the Cargo.toml files yourself.

What we do in Bazel currently is perform cargo tree look-ups for the cartesian product of supported host and target platforms (not cargo metadata look-ups, for the already mentioned lack of support for resolver-v2), which are indeed annoyingly costly. I haven't looked at it in depth, but the fact that cargo resolve-features accepts a Vec<String> for targets looks promising as an API for being able to reduce this down to one query rather than n (or at least, a step there - I don't see a way of specifying the host platform ?yet)?

It does feel like this should be something that could be much more efficiently implemented in a single pass through the resolver (keeping track of what was conditional one what), rather than having to run lots of queries and then post-process them to look for what was common. Hopefully this could form a nice plumbing command, if it doesn't fit nicely into one of the exist ones.

[epage]

I don't see a way of specifying the host platform ?yet)?

Feel free to open an issue on the repo!

rather than having to run lots of queries and then post-process them to look for what was common. Hopefully this could form a nice plumbing command, if it doesn't fit nicely into one of the exist ones.

It sounds like you are asking for cargo plumbing resolve-features but then acknowledge something else might be needed. Is there something I'm missing?

[epage]

Forgot to mention as part of all of this that if you are ok with doing a build, rust-lang/rfcs#3553 would be a way to then capture what happened.