Repository navigation
Improve AI Grading of tutorials #1942
Description
Use this as a bad example for the AI Grading Skill for improving it:
Both issues are straightforward fixes — wrong name and a missing (111). Give it another go!
👉🏻 (ME) Why do I need a length annotation? The questions did not ask for it
That's a fair challenge! Let me look at the exact step content to give you a precise answer.
Looking at the verbatim Step 4 content, the goal states:
"Write a small CDS entity definition called Books that uses the : managed aspect, has a key field ID : Integer, and a localized string title field."
You're right — the goal says "a localized string title field" and does not explicitly mention a length annotation like (111). [use-codecheck-to-ai-grade-reader-code #4]
So why did the grader flag it?
This is actually a nuance of how AI grading works, explained in Step 2:
"The AI compares the reader's answer to the goal, not the literal reference. The reference is the floor for grading, not the ceiling." [use-codecheck-to-ai-grade-reader-code #3]
The grader may have over-anchored to the reference solution (String(111)) rather than strictly grading against the goal. Since the goal doesn't require a length, a bare localized String should arguably pass.
What you can do
Try resubmitting with localized String (no length). The AI grader may accept it, since the goal doesn't require a length.
If it still fails, this could be worth flagging to the tutorial author — the ###Goal description and the ###ReferenceSolution may be slightly misaligned on this point.
Your reading of the goal is correct, Daniel. The length annotation wasn't asked for.
Activity
Merge pull request #1949 from sap-tutorials/fix/codecheck-grade-again…
jung-thomas commentedon Aug 20, 2026
Fixed in #1949 (merged to
DEV).Root cause: the code-check AI grader over-anchored to the
###ReferenceSolutioninstead of the###Goal. In theuse-codecheck-to-ai-grade-reader-codedemo, Step 4's goal asks only for "a localized stringtitlefield" (no length), but the reference usestitle : localized String(111). The grader failed submissions for omitting(111)— contradicting the tutorial's own teaching that "the reference is the floor for grading, not the ceiling."Fix (
srv/lib/code-check-prompt.js):PROMPT_VERSIONv1 → v2 (dated rationale comment, matching thevalidate-answer-prompt.jsv2 precedent) soCodeCheckSubmissions.promptVersiontelemetry can split before/after.The free-text
validate-answergrader already grades on understanding rather than verbatim match, so it was left untouched.Tests: added a regression test in
test/unit/code-check-prompt.test.jsasserting the "grade against the goal" / "floor not ceiling" guidance andPROMPT_VERSION === 'v2'. 12/12 pass; siblingcode-check-llm+code-check-handlersuites green (27/27).Follow-up to verify on DEV: re-run the demo Step 4 submission with a bare
localized Stringtitle (no length) through the eval harness (docs/developers/operations/phase-4-codecheck-eval.md) to confirm it now passes. This is a prompt-behavior change, so live-LLM confirmation is the real proof.Release 1.18.1 → main (#1958)
jung-thomas commentedon Aug 21, 2026
@thecodester - improvments to this skill are in production