Repository navigation
perf(concepts): parallelize /build/concepts link-table reads - #2302
Merged
Merged
Conversation
buildConceptsPayload issued its ~11 independent link-table reads (teaches, edges, journeys, blogs, missions, videos, apiDocs, samples, helpDocs, events) strictly sequentially. On HANA that is ~11 serial round-trips over multi- thousand-row tables per cold call, pushing /build/concepts to ~50-60s and past the 30s smoke-gate timeout (4 smoke tests failed as pure timeouts on DEV 1.25.0; endpoint returned correct 200s, just slowly). The reads are all keyed off idsSet (computed once in step 1) with no inter- query dependencies, so wrap them in a single Promise.all wave. Steps 3+4 stay grouped (they jointly build requires/relatedTo/requiredBy) and now run their two edge reads in parallel too. Stitch (step 6) and groupBy unchanged; same columns, filters, sorts, and per-concept caps. No user impact was in play - /concepts/* pages are served from HANA BLOBs (fast); this endpoint feeds only the Hugo concept-render build pipeline and the smoke gate. Not a 1.25.0 regression (query helper last changed 2026-08-09); the slowness is structural and pre-existing. All 744 concepts-related unit tests pass; payload shape unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
/build/conceptstakes ~50–60s on DEV — past the 30s smoke-gate timeout. Surfaced as 4 smoke failures (all pure 30s timeouts, not 4xx/5xx) in the DEV 1.25.0 deploy:concepts-page,concepts-route,asset-retention,explore-route. The endpoint returns correct 200s, just slowly.Not a 1.25.0 regression —
published-concepts-query.jslast changed 2026-08-09; the slowness is structural.Root cause
buildConceptsPayloadissued its ~11 independent link-table reads (teaches, edges, journeys, blogs, missions, videos, apiDocs, samples, helpDocs, events) strictly sequentially — eachawaited before the next. Each is an unbounded full-table scan filtered in Node (deliberate, to dodge the HANAIN (ids)packet-size bug, #1103). On HANA that's ~11 serial round-trips over multi-thousand-row tables per cold call.Fix
The reads are all keyed off
idsSet(computed once in step 1) with no inter-query dependencies → wrap them in onePromise.allwave. Steps 3+4 stay grouped (they jointly build requires/relatedTo/requiredBy) and run their two edge reads in parallel too. Stitch (step 6) andgroupByunchanged; identical columns, filters, sorts, per-concept caps.Impact / risk
/concepts/*pages are served from HANA BLOBs (fast, verified 0.8s). This endpoint feeds only the Hugo concept-render build pipeline and the smoke gate.db.runmay briefly exceed the HANA pool (default max 10); excess queries queue rather than fail — still overlaps, no correctness risk.Verification
npx vitest run --project unitover all concepts suites → 744 passed, 1 skipped.