Skip to content

perf(concepts): parallelize /build/concepts link-table reads - #2302

Merged
jung-thomas merged 1 commit into
DEVfrom
perf/build-concepts-parallel
Sep 14, 2026
Merged

jung-thomas merged 1 commit into
DEVfrom
perf/build-concepts-parallel

Conversation

@jung-thomas

Copy link
Copy Markdown
Contributor

Problem

/build/concepts takes ~50–60s on DEV — past the 30s smoke-gate timeout. Surfaced as 4 smoke failures (all pure 30s timeouts, not 4xx/5xx) in the DEV 1.25.0 deploy: concepts-page, concepts-route, asset-retention, explore-route. The endpoint returns correct 200s, just slowly.

Not a 1.25.0 regression — published-concepts-query.js last changed 2026-08-09; the slowness is structural.

Root cause

buildConceptsPayload issued its ~11 independent link-table reads (teaches, edges, journeys, blogs, missions, videos, apiDocs, samples, helpDocs, events) strictly sequentially — each awaited before the next. Each is an unbounded full-table scan filtered in Node (deliberate, to dodge the HANA IN (ids) packet-size bug, #1103). On HANA that's ~11 serial round-trips over multi-thousand-row tables per cold call.

Fix

The reads are all keyed off idsSet (computed once in step 1) with no inter-query dependencies → wrap them in one Promise.all wave. Steps 3+4 stay grouped (they jointly build requires/relatedTo/requiredBy) and run their two edge reads in parallel too. Stitch (step 6) and groupBy unchanged; identical columns, filters, sorts, per-concept caps.

Impact / risk

  • No user impact was ever in play — /concepts/* pages are served from HANA BLOBs (fast, verified 0.8s). This endpoint feeds only the Hugo concept-render build pipeline and the smoke gate.
  • Behavior identical: same reads, same output shape. All 744 concepts-related unit tests pass.
  • Concurrency note: ~12 in-flight db.run may briefly exceed the HANA pool (default max 10); excess queries queue rather than fail — still overlaps, no correctness risk.

Verification

  • npx vitest run --project unit over all concepts suites → 744 passed, 1 skipped.
  • Recommend confirming wall-clock on DEV post-merge (expect well under the 30s smoke bound); the 4 smoke timeouts should clear.

buildConceptsPayload issued its ~11 independent link-table reads (teaches,
edges, journeys, blogs, missions, videos, apiDocs, samples, helpDocs, events)
strictly sequentially. On HANA that is ~11 serial round-trips over multi-
thousand-row tables per cold call, pushing /build/concepts to ~50-60s and past
the 30s smoke-gate timeout (4 smoke tests failed as pure timeouts on DEV
1.25.0; endpoint returned correct 200s, just slowly).

The reads are all keyed off idsSet (computed once in step 1) with no inter-
query dependencies, so wrap them in a single Promise.all wave. Steps 3+4 stay
grouped (they jointly build requires/relatedTo/requiredBy) and now run their
two edge reads in parallel too. Stitch (step 6) and groupBy unchanged; same
columns, filters, sorts, and per-concept caps.

No user impact was in play - /concepts/* pages are served from HANA BLOBs
(fast); this endpoint feeds only the Hugo concept-render build pipeline and
the smoke gate. Not a 1.25.0 regression (query helper last changed 2026-08-09);
the slowness is structural and pre-existing.

All 744 concepts-related unit tests pass; payload shape unchanged.
@jung-thomas
jung-thomas merged commit c929478 into DEV Sep 14, 2026
5 checks passed
@jung-thomas
jung-thomas deleted the perf/build-concepts-parallel branch September 14, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant