- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates rubric drift in LLM-based grading, where changes in answer order, length, formatting, or the inclusion of irrelevant information can influence grading outcomes despite unchanged satisfaction of scoring criteria. To systematically evaluate this issue, the authors introduce EduR2Bench, a benchmark spanning four STEM subjects, and assess grading consistency using Criterion Flip Rate and related metrics. They further propose RITE, a training framework that combines criterion-level correctness supervision with score-distribution alignment across equivalent answers. Experimental results show that standard correctness training can mitigate rubric drift, while RITE achieves further improvements in grading consistency.
Critical strengths
- A dataset with diverse answer variants and extensive experimental evaluation. EduR2Bench covers four STEM subjects and includes changes in answer order, length, format, and irrelevant information. It provides a useful resource for systematically evaluating grading behavior under different answer presentations. The paper also includes comparisons across multiple models and external methods, ablations under different supervision settings, tests on unseen transformation families, and validation using human-written rewrites and distractors. The overall scope of the work is substantial.
- A clear evaluation of rubric drift and an effective approach to mitigating it. The paper examines whether grading is affected by factors unrelated to the scoring criteria when criterion satisfaction remains unchanged. This perspective complements average grading-quality metrics and helps identify sensitivity to answer presentation. RITE further reduces drift on the evaluated short-answer tasks, with meaningful improvements in some subjects, supporting the value of the proposed method.
Critical weaknesses
- The validity of the scoring criteria needs stronger evidence. Data construction, equivalence judgments, label inheritance, and final evaluation depend on the predefined criteria, so their completeness matters. The paper reports that Fable 5 drafted the criteria, that the authors reviewed them, and that all variants were manually checked. It also reports 96.2% agreement (Fleiss' κ=0.91) on a stratified sample of criterion decisions and 95.0% agreement with teaching-assistant annotations on RiceChem. These are meaningful validation steps. Still, agreement on applying criteria does not by itself establish that the criteria are complete or accommodate alternative correct solutions. A small set of representative criteria, human revisions, and adjudicated disagreements would help readers independently assess this remaining issue. The evaluation is limited to short-answer tasks; the paper acknowledges broader task types as future work. This narrows external validity rather than directly undermining the results for the stated short-answer setting.
- The paper provides a direct comparison under the intended original-only supervision setting, so the incremental benefit of RITE is more clearly isolated than this concern initially suggests. On Computer Science, Table 4 reports that adding RITE's equivalence loss improves QWK from 0.737 to 0.753 and reduces CFR from 11.7% to 5.2%; Appendix A.5 also reports seed-paired changes. Table 3 shows gains over PBCE-Orig across all four subjects. These results support an incremental benefit beyond ordinary correctness training. The remaining uncertainty is how robust and practically meaningful the effect is with three seeds, especially across subjects. The full-label setting behaves differently: RITE-All is worse than PBCE-All on the reported CS metrics, consistent with the authors' explanation that equivalence regularization can over-regularize when every variant is labeled. The paper should make this setting dependence prominent when describing RITE's advantage. The paper's own experiments show that ordinary correctness supervision can substantially reduce drift, while Table 4 indicates that explicit equivalence supervision adds value in the original-only setting. A clearer discussion of the effect size and its uncertainty across the four subjects would help establish when this additional constraint is most useful.
Questions for authors
- Can you provide stronger evidence for criterion construction and data that reviewers can inspect? Please explain how reference answers were decomposed into scoring criteria, the subject expertise of the criterion reviewers, and how the review process addressed omissions, redundancy, ambiguity, and alternative correct solutions. Could you provide data for inspection, or samples stratified by subject and difficulty, containing the original questions, reference answers, student answers, generated criteria, and final labels, together with representative human revisions and disagreements? This would help reviewers directly assess the appropriateness of the criteria. Please also distinguish validation of the criteria themselves from agreement in judging whether answers satisfy given criteria.
- The experiments focus on short-answer grading, which is also the scope stated in the paper’s limitations. Could you make the intended scope explicit and avoid implying generalization to longer-form or other task types? If broader claims are intended, please indicate what additional evidence would support them.
- The paper already reports the key original-only comparison (Table 4) and seed-paired changes (Appendix A.5). Could you clarify how to interpret the effect size given that the comparison uses three seeds, and provide uncertainty estimates for the cross-subject original-only results? Please also explain the setting dependence: RITE improves over PBCE-Orig when variants are unlabeled, while RITE-All underperforms PBCE-All in the reported CS comparison. This distinction would help readers understand when the equivalence loss is useful and avoid treating its benefit as uniform across supervision regimes.
- Are the variant families sufficiently distinct in the factors they change? The paper gives operational definitions, but some surface properties may still overlap. For example, both Length and Distractor increase text length, while Position and Style alter how content is organized. Could you clarify with paired before-and-after examples which properties each operator changes and preserves? A length-matched Length-versus-Distractor analysis could further separate content effects from length effects, although the current family-level results remain useful for measuring the overall drift associated with each defined transformation.
3 comments
Thanks to the authors for their responses. I believe my original score remains appropriate, and I will therefore maintain it.
Thanks for your prompt response. We appreciate your positive feedback w. high confidence.
Response to Reviewer EzJT
Thank you for your valuable feedback. We hope the clarifications and results below directly address your concerns.
(1) Decomposition. Fable 5 decomposed each reference answer into 1–4 independently verifiable binary criteria, one per scoring point. Each rubric was reviewed against the question and reference answer and frozen before variant generation and labeling.
(2) Reviewer expertise. CS criteria (114) were reviewed by a CS Ph.D. candidate with an M.S. in CS; Physics, Chemistry, and Earth Science criteria (173) by a physics Ph.D. candidate with a B.S. in physics. For each of these three subjects, an independent reviewer with a master's degree in the subject who teaches or works professionally in it also checked the final criteria under the four checks in (3); none proposed a change.
(3) Review checks. All 176 rubrics were reviewed under four checks; 31 were revised (36 edits): omissions (every scoring point covered; 8 criteria added), redundancy (one criterion per scoring point; 5 overlapping pairs merged, 6 bundled criteria split), ambiguity (hit evidence statable before reading any answer; 17 criteria rewritten), and alternative correct solutions (criteria are content propositions, not keywords, phrased at the level of the claim the question tests; any wording conveying the proposition is a hit).
(4) Data for inspection. Tab. 8 gives one complete CS group (question, criteria, answer, all four variants, gold labels); the revision adds one group each for Physics, Chemistry, and Earth Science, with source reference answers and full-, partial-, and no-credit cases (new Tab. 9).
(5) Validity vs. agreement. The 96.2% agreement (Fleiss' κ = 0.91) and 95.0% agreement with teaching-assistant labels (Tab. 23) measure consistency in applying given criteria; all 19 adjudicated cases concern this application, not the criteria. Criterion validity rests on (1)–(4), now reported in App. A.4 (L1106–1125).
Our claims concern STEM short-answer grading; we do not claim that the findings directly generalize to essays, long-form responses, or other grading formats, and now state this scope explicitly (L97–99).
The effect is large in every subject: even after small-sample correction, Hedges' g ≥ 1.9, above the 0.8 threshold for a large effect, and every paired 95% CI excludes zero. We extended the seed-paired analysis of App. A.5 from CS to all four subjects (Tab. 5, right): under original-only correctness supervision, RITE reduces mean CFR by 38%–61% relative to PBCE-Orig, with reductions in all 12 seed pairs (Qwen3-8B; CFR %, mean±SD, 3 seeds):
The equivalence loss is useful when correctness supervision does not cover the variants, consistent with the complementary roles in Thm. 2. When every variant receives inherited labels, these encode the equivalence and the constraint does not help (CS: 5.8% → 7.7%; Tab. 4). The Introduction now makes this dependence prominent (L95–97).
Yes, each operator changes a different property, and Distractor drifts more than Length within the same added-length ranges. Position changes order but not format; Style changes format but not order; Length elaborates existing content; Distractor adds new content irrelevant to every criterion. Tab. 8 shows all four on one answer; the App. A.4 checklist specifies what each preserves.
We added a length-stratified analysis (CS test, Qwen3-8B; family-specific CFR, %; new Tab. 10):
Even the longest Length additions drift less than the shortest Distractor additions. In both bins where the two families overlap, Distractor has higher CFR than Length, with gaps of 7.2–10.2 points, while CFR within each family changes by at most 2.1 points across bins. Thus, the larger Distractor effect is not explained by longer answers: adding task-irrelevant content is harder than elaborating existing evidence.