fix(#1621,#1592): cross-instance coherence for ContentCache + navigator cache

[jung-thomas]

What & why

Fixes #1621 (and the render-cache facet of #1592): the per-process ContentCache in srv/lib/content-store.js — both bare-<slug> tutorial HTML and render:<slug> catalog (group/mission SSR) entries — and the navigator response cache in navigator-catalog.js are invalidated only on the srv instance that handled the publish/admin write. With 2+ CF instances the others keep serving stale HTML until LRU eviction or cf restart:

• Content ContentCache invalidation is instance-local — stale HTML served fleet-wide after publish #1621: a publish (hana-cloud-mission-trial-3) served two different documents round-robined across instances, both X-Content-Source: cache, while /content/hashes returned a single current hash — divergence was purely in-memory.
• Tutorials in Wrong have wrong ordering #1592: an Admin-UI group reorder didn't propagate; one instance served the corrected order, the other the stale one.

Approach

There is no pub/sub bus available (CAP messaging unconfigured; @cap-js-community/websocket has no Redis adapter, so broadcasts don't cross instances). Rather than move the hot HTML cache onto cds-caching (no L1 — every op is a HANA round-trip, deleteByTag is an O(n) full-store scan), this keeps the fast in-memory caches and adds a shared generation token:

• Writers (publish/commit/rollback + the admin.after nav-invalidating hook) call bumpCacheGeneration() — sets a new token in the already-wired caching service. In hybrid/prod that service is store: "cds" → a shared HANA table, so the bump is visible to all instances. Writers still do their immediate local invalidation too, so the writing instance is correct with zero delay.
• Readers call refreshCacheGeneration() before trusting a local cache entry (both the render: catalog path and the bare-slug tutorial path, plus the navigator handler). It's TTL-gated (default 5s, CONTENT_CACHE_CHECK_TTL_MS) and coalesced, so it's at most one lightweight caching.get per 5s per instance. When the token changed, all registered local invalidators fire (content-store drops the whole ContentCache; navigator clears its response).

Fail-open everywhere: a caching outage or throw → falls back to today's behaviour (local cache only), never a 500. In the default/test profile the caching store is memory (per-process), so it's a no-op across processes — correct, since unit tests are single-process.

Staleness bound: cross-instance convergence within CONTENT_CACHE_CHECK_TTL_MS (5s) of a write — well under the <=60s admin-to-visitor delay already documented for Alerts.

Tradeoff (called out for review): one generation token covers both content and catalog caches, and a generation change triggers a full cache.invalidate(). So a catalog admin write also drops peers' tutorial-content cache entries (mild over-invalidation; they repopulate from the DB, and content pages are Akamai-fronted). This keeps the mechanism simple and matches the publish-path's existing full-wipe semantics. Splitting into two tokens (content vs catalog) is a possible follow-up if the extra origin reads ever matter.

Files

• srv/lib/content-cache-coherence.js (new) — the shared-generation primitive: bumpCacheGeneration, refreshCacheGeneration, onCacheGenerationChange; TTL-gated + coalesced reads; fail-open; test seams.
• srv/lib/content-store.js — register cache.invalidate() on generation change; refreshCacheGeneration() before the render: and bare-slug reads; bumpCacheGeneration() after each cache.invalidate().
• srv/lib/navigator-catalog.js — register local clear + refresh before serving cache.
• srv/server.js — bumpCacheGeneration() in the admin.after nav-invalidating hook.
• test/unit/content-cache-coherence.test.js (new) — peer-bump detection, TTL gating, self-bump no-op, post-bump peer change, fail-open (read + write), null-token no-op.

Testing

• New unit suite: 7/7 pass (deterministic via fake timers; simulates a peer instance by writing the shared token directly).
• Regressions green: test/render-cache-invalidation.test.js, test/unit/content-cache-metrics.test.js, test/lib/content-store.test.js (47).

On the #1621 acceptance "hybrid test that simulates two cache holders": a true 2-instance CF test isn't feasible in CI, but the coherence primitive that guarantees the fan-out is unit-tested directly (peer-bump -> local invalidators fire). Verified in PROD out-of-band that a cf restart converges instances (the manual workaround this replaces).

Related

• Migration root-cause + one-shot PROD data repair for Tutorials in Wrong have wrong ordering #1592 (the wrong group ordering itself) is on a separate branch (worktree-fix-1592-group-ordering): the IMS->CAP migration ordered group members by CHILD_TASK_ID instead of TASK_ORDER. This PR is the cache-coherence half only.