fix(quiz): fix grading bugs causing correct answers to be marked wrong#1064
fix(quiz): fix grading bugs causing correct answers to be marked wrong#1064karan033-cybertech wants to merge 3 commits into
Conversation
Three separate bugs were causing correct quiz answers to be evaluated incorrectly: 1. SOLQuestionRenderer: parameterized questions lost their shuffle when processing text tags — shuffledLotItems was overwritten with lotItems (unshuffled), causing option index mismatch between what student saw and what grader compared against. Fixed by mapping over shuffledLotItems instead of lotItems. 2. DESQuestionGrader: placeholder grading awarded full question points (Math.round(this.question.points)) to every descriptive answer including blank submissions. Fixed to award 0 score until real LLM grading is integrated. 3. quiz.tsx SELECT_MANY_IN_LOT: when a lotItem _id failed to convert from buffer to hex, the fallback returned index.toString() which was then silently filtered out by .filter(id => !id.match(/digit/)). This caused valid selected answers to be dropped before submission, resulting in incorrect grading. Fixed to detect conversion failures early and skip the whole submission rather than sending a corrupted partial answer set. Fixes vicharanashala#714
When a lotItem _id failed to convert from buffer to hex, the fallback returned index.toString() which was silently filtered out by .filter(id => !id.match(/digit/)). This caused valid selected answers to be dropped before submission, resulting in incorrect grading. Fixed to detect conversion failures early and skip the whole submission rather than sending a corrupted partial answer set. Fixes vicharanashala#714
aloktripathi1
left a comment
aloktripathi1
left a comment
There was a problem hiding this comment.
None of the three fixes have tests. Bug 1 and Bug 2 are deterministic and seem straightforward to unit test. Worth adding before merging?
| // TODO: Integrate with LLM-based evaluation engine later | ||
| // This is a dummy placeholder response | ||
| return { | ||
| status: 'PARTIAL', |
There was a problem hiding this comment.
Returning status: 'PARTIAL' with score: 0 seems contradictory. PARTIAL
usually implies some credit was earned. Would it make more sense to use
'INCORRECT' here or introduce a 'PENDING' status to signal the answer
is waiting for LLM grading?
| if (resolvedIds.every(id => id !== null)) { | ||
| saveAnswer.lotItemIds = resolvedIds as string[]; | ||
| } else { | ||
| console.error(`[quiz] One or more selected answers could not be resolved for question ${question.id}. Skipping submission for this question.`); |
There was a problem hiding this comment.
If any ID conversion fails, the entire question submission is skipped.
If a student selects 5 answers and 1 fails, they lose all 5.
Would it be better to submit only the successfully resolved IDs instead
of skipping the whole question? The console.error is not visible to the
student either, so they would have no idea their answer was not saved.
- DESQuestionGrader: change status from PARTIAL to INCORRECT since score is 0 and no credit is awarded - quiz.tsx: submit partially resolved IDs instead of skipping entire question when some _id conversions fail
Thanks for the review @aloktripathi1! Addressed all three points in the latest commit. |
2 participants
Problem — Closes #714
Three bugs were causing correct quiz answers to be marked wrong.
Bug 1 — SOLQuestionRenderer: shuffle lost on parameterized questions
When
parameterMapwas present, shuffled options were overwritten withunshuffled
lotItems, causing index mismatch between what student sawand what grader checked.
Fix: Map over
shuffledLotItemsinstead oflotItems.Bug 2 — DESQuestionGrader: blank answers got full marks
Placeholder grading awarded
Math.round(this.question.points)to everydescriptive answer including blank submissions.
Fix: Award
score: 0until real LLM grading is integrated.Bug 3 — quiz.tsx: selected answers silently dropped
For SELECT_MANY_IN_LOT, if
_idconversion failed, fallbackindex.toString()was immediately filtered out — silently droppingthe student's selection before submission.
Fix: Detect failures early, log warning, skip submission instead
of sending corrupted partial answer set.
Files changed
backend/src/modules/quizzes/question-processing/renderers/SOLQuestionRenderer.tsbackend/src/modules/quizzes/question-processing/graders/DESQuestionGrader.tsfrontend/src/components/quiz.tsxS