Repository navigation
fix(kg): dedupe tag/product vertex arms to stop KG Louvain job crash - #2496
Merged
Merged
Conversation
KG_PG_VERTICES_V's tag and product arms used SELECT DISTINCT, but both arms also select LABEL. Two Tags rows sharing the same NAME (or product suffix) but differing in LABEL therefore both survive DISTINCT and emit a duplicate VERTEX_KEY. On PROD, 'sustainability for human capital management' had two rows whose LABEL differed only by a double space, producing a duplicate 'tag:<name>' key. HANA's GraphEngine rejects the whole KG_PG_WORKSPACE on a duplicate vertex key ([4907] range 1), crashing all three nightly KG jobs (kg-pagerank, kg-communities/Louvain, kg-wcc). Alert: ScheduledJobFailed kg-communities, 2026-09-24. Replace DISTINCT with the same ROW_NUMBER() OVER (PARTITION BY ...) derived-table dedupe already used by the concept and teched arms, so each key is emitted exactly once regardless of label drift. Tags has no MODIFIEDAT column, so order by LEGACYID DESC, ID for determinism. The taggedWith/aboutProduct edge arms key their tag/product endpoint on NAME/suffix alone, so any surviving row keeps edge targets resolvable. Validated against live PROD data: rewritten tag arm yields 265 rows / 265 distinct keys (was 266/265) and the known dup collapses to one vertex; product arm stays consistent. Extends the hybrid dedupe test with the exact PROD tag shape (same NAME, differing LABEL).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
kg-communities(Louvain) nightly job crashed on PROD (ScheduledJobFailed, 2026-09-24 03:57 UTC):HANA's GraphEngine rejects the whole
KG_PG_WORKSPACEwhen anyVERTEX_KEYrepeats, so this crashes all three nightly KG jobs (kg-pagerank,kg-communities/Louvain,kg-wcc).Root cause
KG_PG_VERTICES_V's tag and product arms usedSELECT DISTINCT, but both arms also selectLABEL. TwoTagsrows sharing the sameNAME(orsoftware-product>suffix) but differing inLABELboth surviveDISTINCTand emit the sametag:<name>/product:<suffix>key twice — the exact trap the concept arm's header already warns about.On PROD,
sustainability for human capital managementhad twoTagsrows whoseLABELdiffered only by a double space (Sustainability for...vsSustainability for...), producing the duplicate key. (12 other tag NAMEs have multiple rows today with identical labels — masked byDISTINCT, but one label edit away from the same crash.)Fix
Replace
DISTINCTwith theROW_NUMBER() OVER (PARTITION BY ... ORDER BY LEGACYID DESC, ID)derived-table dedupe already used by the concept and teched arms — exactly one vertex per key regardless of label drift.Tagshas noMODIFIEDAT, so ordering is byLEGACYID DESC, ID. ThetaggedWith/aboutProductedge arms key their endpoint onNAME/suffix alone, so any surviving row keeps edge targets resolvable.Validation (live PROD data, read-only)
Ran the rewritten arm SQL against the real prod HDI container:
tag:sustainability for human capital managementrowsAlso extends
test/hybrid/kg-vertex-dedupe.test.jswith a tag-arm case seeding two rows (same NAME, differing LABEL) — the exact PROD shape.Deploy note
This is an HDI view change — needs a DB deploy to take effect (the currently-deployed prod view still shows the 2× dup). Once deployed, the next KG job run recovers on its own (jobs are fail-open; no data backfill needed).