Closed Bug 2065363 Opened 2 months ago Closed 1 month ago

Add `./mach vendor node` to vendor `node_modules` needed during the build

Categories

(Firefox Build System :: General, task)

Product:
Firefox Build System
Component:
General
Type:
task

Tracking

(firefox157+ fixed)

Status:
RESOLVED FIXED
Milestone:
157 Branch
Tracking Flags:
Tracking Status
firefox157 + fixed

People

(Reporter: ahochheiden, Assigned: ahochheiden)

References

(Blocks 3 open bugs)

Details

Attachments

(10 files)

Bug 2065363 - Add a `pnpm` fetch task r?#build
48 bytes, text/x-phabricator-request
Details | Review
Bug 2065363 - Add `./mach vendor node` for node packages needed at build time r?#build,nalexander,mconley
48 bytes, text/x-phabricator-request
Details | Review
Bug 2065363 - Vendor the node packages needed to build newtab r?#build
48 bytes, text/x-phabricator-request
Details | Review
Bug 2065363 - Build the newtab bundles from the vendored node packages r?#build,mconley
48 bytes, text/x-phabricator-request
Details | Review
Bug 2065363 - Check the licenses of the vendored node packages r?#build,#linter-reviewers
48 bytes, text/x-phabricator-request
Details | Review
Bug 2065363 - Update newtab's webpack and Babel past their published advisories r?mconley
48 bytes, text/x-phabricator-request
Details | Review
Bug 2065363 - Integrate `pnpm audit` into `mozlint` r?#build,#linter-reviewers
48 bytes, text/x-phabricator-request
Details | Review
Bug 2065363 - List the vendored node packages in `about:license` r?#build,sylvestre
48 bytes, text/x-phabricator-request
Details | Review
Bug 2065363 - Prune the vendored node tree to what the build reads r?#build
48 bytes, text/x-phabricator-request
Details | Review
Bug 2065363 - Refuse to vendor node packages when the manifests disagree r?#build
48 bytes, text/x-phabricator-request
Details | Review

Bug 2025793 made newtab bundling depend on node_modules during the build. In our CI, the node_modules were supplied via toolchain and npm ci was added as a local fallback. The problem is that downstream builders are not covered. They have no network access, and don't consume our toolchains. The simplest fix is to just vendor the necessary node_modules in the tree, which will work for all paths.

Blocks: 2063266

I used ./mach vendor python as a template for this. We strip everything we
don't need, including anything that varies by host, so vendoring on any
platform yields identical results.

third_party/node/node_modules is generated by ./mach vendor node.

The toolkit/content/license.html change was done manually.

Nothing is installed during the build now, and it can run without network access.
Webpack's inputs are recorded and hashed rather than returned as dependencies,
since the FasterMake backend never includes the dependency files it writes.

The node-licenses lint normally reinstalls node_modules with npm. For
pnpm-managed vendored trees, it checks the existing tree instead and reports
an error when it is missing.

Duplicate of this bug: 2065426
Blocks: 2065426
Attachment #9629202 - Attachment description: Bug 2065363 - Add a `pnpm` fetch task in CI r?#build → Bug 2065363 - Add a `pnpm` fetch task r?#build

webpack 5.109 no longer provides terser-webpack-plugin, so the minifier is
declared and imported under its own name. The emitted bundles change.

Duplicate of this bug: 2066663
Depends on: 2068750
Blocks: 2068750
No longer depends on: 2068750

Documentation, tests, type declarations, source maps, bin shims and the
bookkeeping pnpm keeps for itself are all pruned. License text stays, and a
README stays only when a package has no license text of its own.

Packages restricted by os, cpu or libc are pruned too, because whether
pnpm installs them depends on the vendoring host. The executable bit is
cleared because a Windows checkout cannot carry it. ./mach vendor node should
yield identical output on every host platform.

A pnpm other than the one package.json names can resolve the same manifest
differently, so its version is checked before anything runs. A consumer
manifest that pins another version of a vendored package would build against
something other than it declares, so after any requested add or remove the
versions have to agree before the lock file is resolved and the tree installed.

Consumers are listed in CONSUMER_MANIFESTS, newtab alone for now. Two
consumers that want different versions of one package fail this check rather
than getting two copies. Once consumers keep their build dependencies apart
from their test dependencies, third_party/node can become a pnpm workspace
with the consumers as its packages and the check can go.

Pushed by ahochheiden@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/33b62710206c https://hg.mozilla.org/integration/autoland/rev/f2e7b69bf9ed Add a `pnpm` fetch task r=firefox-build-system-reviewers,sylvestre https://github.com/mozilla-firefox/firefox/commit/75867e16f887 https://hg.mozilla.org/integration/autoland/rev/91fec5ba39ad Update newtab's webpack and Babel past their published advisories r=mconley https://github.com/mozilla-firefox/firefox/commit/fd238dcdaa38 https://hg.mozilla.org/integration/autoland/rev/e3ac66486a44 Add `./mach vendor node` for node packages needed at build time r=mconley,firefox-build-system-reviewers,glandium https://github.com/mozilla-firefox/firefox/commit/d203e9e21b3b https://hg.mozilla.org/integration/autoland/rev/0afb735f0c5c Prune the vendored node tree to what the build reads r=firefox-build-system-reviewers,glandium https://github.com/mozilla-firefox/firefox/commit/1f1567bf6454 https://hg.mozilla.org/integration/autoland/rev/ea8218b9e9e9 Refuse to vendor node packages when the manifests disagree r=firefox-build-system-reviewers,glandium https://github.com/mozilla-firefox/firefox/commit/d0470670a103 https://hg.mozilla.org/integration/autoland/rev/64e16150b730 Vendor the node packages needed to build newtab r=firefox-build-system-reviewers,sylvestre https://github.com/mozilla-firefox/firefox/commit/f174ceb3cf4e https://hg.mozilla.org/integration/autoland/rev/668e494beeef List the vendored node packages in `about:license` r=sylvestre,firefox-build-system-reviewers,sergesanspaille https://github.com/mozilla-firefox/firefox/commit/95aa7c1db362 https://hg.mozilla.org/integration/autoland/rev/5fd880e0eec1 Build the newtab bundles from the vendored node packages r=mconley,ai4dev-reviewers,padenot https://github.com/mozilla-firefox/firefox/commit/bbd47a1061fe https://hg.mozilla.org/integration/autoland/rev/5713dbdae4ab Check the licenses of the vendored node packages r=firefox-build-system-reviewers,linter-reviewers,Standard8,sergesanspaille https://github.com/mozilla-firefox/firefox/commit/97313bb6ae66 https://hg.mozilla.org/integration/autoland/rev/d85da6e65e08 Integrate `pnpm audit` into `mozlint` r=linter-reviewers,ahal
Regressions: 2070982
Regressions: 2071041
No longer duplicate of this bug: 2066663
Blocks: 2065418
No longer blocks: 2065426
No longer duplicate of this bug: 2065426
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: