acceptodds
Under review as a conference paper at ICLR 2027

RITE: Investigating and Mitigating Rubric Drift in LLM-Based Grading

Abstract

Large language models are increasingly used to grade open-ended student responses, yet construct-irrelevant variation can change their judgments. We call this failure rubric drift: grading changes despite fixed questions, rubrics, and rubric-relevant evidence. We introduce EduR²Bench, with 1,060 verified rubric-equivalence groups and 3,297 variants across four STEM subjects, and Criterion Flip Rate (CFR), a label-free drift diagnostic. Rubric drift is widespread across open-weight, frontier, and specialized graders and is distinct from grading accuracy. Our central finding is supervision-induced invariance (SII): correctness supervision on original responses reduces CFR on unsupervised equivalent variants from 26.3% to 11.7%, accompanied by contraction of upper-middle-layer representation gaps. However, SII is limited by supervision coverage, label reliability, and transformation-family transfer. These boundaries motivate Rubric-Invariant Training via Equivalence (RITE), which combines correctness grounding with equivalence supervision. Using labels only on original responses, RITE reduces CFR to 5.2% while preserving grading quality and retains substantial benefit on a held-out transformation family and unseen human-written rewrites. Together, our results characterize when rubric invariance emerges from supervision and how to strengthen it explicitly.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI for education, large language models, automated grading, rubric drift, rubric invariance, robustness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EzJTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates rubric drift in LLM-based grading, where changes in answer order, length, formatting, or the inclusion of irrelevant information can influence grading outcomes despite unchanged satisfaction of scoring criteria. To systematically evaluate this issue, the authors introduce EduR2Bench, a benchmark spanning four STEM subjects, and assess grading consistency using Criterion Flip Rate and related metrics. They further propose RITE, a training framework that combines criterion-level correctness supervision with score-distribution alignment across equivalent answers. Experimental results show that standard correctness training can mitigate rubric drift, while RITE achieves further improvements in grading consistency.

Critical strengths
  1. A dataset with diverse answer variants and extensive experimental evaluation. EduR2Bench covers four STEM subjects and includes changes in answer order, length, format, and irrelevant information. It provides a useful resource for systematically evaluating grading behavior under different answer presentations. The paper also includes comparisons across multiple models and external methods, ablations under different supervision settings, tests on unseen transformation families, and validation using human-written rewrites and distractors. The overall scope of the work is substantial.
  2. A clear evaluation of rubric drift and an effective approach to mitigating it. The paper examines whether grading is affected by factors unrelated to the scoring criteria when criterion satisfaction remains unchanged. This perspective complements average grading-quality metrics and helps identify sensitivity to answer presentation. RITE further reduces drift on the evaluated short-answer tasks, with meaningful improvements in some subjects, supporting the value of the proposed method.
Critical weaknesses
  1. The validity of the scoring criteria needs stronger evidence. Data construction, equivalence judgments, label inheritance, and final evaluation depend on the predefined criteria, so their completeness matters. The paper reports that Fable 5 drafted the criteria, that the authors reviewed them, and that all variants were manually checked. It also reports 96.2% agreement (Fleiss' κ=0.91) on a stratified sample of criterion decisions and 95.0% agreement with teaching-assistant annotations on RiceChem. These are meaningful validation steps. Still, agreement on applying criteria does not by itself establish that the criteria are complete or accommodate alternative correct solutions. A small set of representative criteria, human revisions, and adjudicated disagreements would help readers independently assess this remaining issue. The evaluation is limited to short-answer tasks; the paper acknowledges broader task types as future work. This narrows external validity rather than directly undermining the results for the stated short-answer setting.
  2. The paper provides a direct comparison under the intended original-only supervision setting, so the incremental benefit of RITE is more clearly isolated than this concern initially suggests. On Computer Science, Table 4 reports that adding RITE's equivalence loss improves QWK from 0.737 to 0.753 and reduces CFR from 11.7% to 5.2%; Appendix A.5 also reports seed-paired changes. Table 3 shows gains over PBCE-Orig across all four subjects. These results support an incremental benefit beyond ordinary correctness training. The remaining uncertainty is how robust and practically meaningful the effect is with three seeds, especially across subjects. The full-label setting behaves differently: RITE-All is worse than PBCE-All on the reported CS metrics, consistent with the authors' explanation that equivalence regularization can over-regularize when every variant is labeled. The paper should make this setting dependence prominent when describing RITE's advantage. The paper's own experiments show that ordinary correctness supervision can substantially reduce drift, while Table 4 indicates that explicit equivalence supervision adds value in the original-only setting. A clearer discussion of the effect size and its uncertainty across the four subjects would help establish when this additional constraint is most useful.
Questions for authors
  1. Can you provide stronger evidence for criterion construction and data that reviewers can inspect? Please explain how reference answers were decomposed into scoring criteria, the subject expertise of the criterion reviewers, and how the review process addressed omissions, redundancy, ambiguity, and alternative correct solutions. Could you provide data for inspection, or samples stratified by subject and difficulty, containing the original questions, reference answers, student answers, generated criteria, and final labels, together with representative human revisions and disagreements? This would help reviewers directly assess the appropriateness of the criteria. Please also distinguish validation of the criteria themselves from agreement in judging whether answers satisfy given criteria.
  2. The experiments focus on short-answer grading, which is also the scope stated in the paper’s limitations. Could you make the intended scope explicit and avoid implying generalization to longer-form or other task types? If broader claims are intended, please indicate what additional evidence would support them.
  3. The paper already reports the key original-only comparison (Table 4) and seed-paired changes (Appendix A.5). Could you clarify how to interpret the effect size given that the comparison uses three seeds, and provide uncertainty estimates for the cross-subject original-only results? Please also explain the setting dependence: RITE improves over PBCE-Orig when variants are unlabeled, while RITE-All underperforms PBCE-All in the reported CS comparison. This distinction would help readers understand when the equivalence loss is useful and avoid treating its benefit as uniform across supervision regimes.
  4. Are the variant families sufficiently distinct in the factors they change? The paper gives operational definitions, but some surface properties may still overlap. For example, both Length and Distractor increase text length, while Position and Style alter how content is organized. Could you clarify with paired before-and-after examples which properties each operator changes and preserves? A length-matched Length-versus-Distractor analysis could further separate content effects from length effects, although the current family-level results remain useful for measuring the overall drift associated with each defined transformation.
3 comments
AuthorsOpenReview ↗

Response to Reviewer EzJT

Thank you for your valuable feedback. We hope the clarifications and results below directly address your concerns.

W1/Q1: Validity of the scoring criteria.

(1) Decomposition. Fable 5 decomposed each reference answer into 1–4 independently verifiable binary criteria, one per scoring point. Each rubric was reviewed against the question and reference answer and frozen before variant generation and labeling.

(2) Reviewer expertise. CS criteria (114) were reviewed by a CS Ph.D. candidate with an M.S. in CS; Physics, Chemistry, and Earth Science criteria (173) by a physics Ph.D. candidate with a B.S. in physics. For each of these three subjects, an independent reviewer with a master's degree in the subject who teaches or works professionally in it also checked the final criteria under the four checks in (3); none proposed a change.

(3) Review checks. All 176 rubrics were reviewed under four checks; 31 were revised (36 edits): omissions (every scoring point covered; 8 criteria added), redundancy (one criterion per scoring point; 5 overlapping pairs merged, 6 bundled criteria split), ambiguity (hit evidence statable before reading any answer; 17 criteria rewritten), and alternative correct solutions (criteria are content propositions, not keywords, phrased at the level of the claim the question tests; any wording conveying the proposition is a hit).

(4) Data for inspection. Tab. 8 gives one complete CS group (question, criteria, answer, all four variants, gold labels); the revision adds one group each for Physics, Chemistry, and Earth Science, with source reference answers and full-, partial-, and no-credit cases (new Tab. 9).

  • Human revision (CS): "a constructor has no return type" was extended with "whereas a function does", the contrast the reference answer scores.
  • Adjudicated disagreement (Chemistry): for "because both solutions are saturated (at the same temperature), they must have the same concentration", two annotators marked hit and one miss on "Yes because if they are saturated … Take that stuff out then they both have the same concentration", adjudicated as hit because it links saturation to equal concentration.

(5) Validity vs. agreement. The 96.2% agreement (Fleiss' κ = 0.91) and 95.0% agreement with teaching-assistant labels (Tab. 23) measure consistency in applying given criteria; all 19 adjudicated cases concern this application, not the criteria. Criterion validity rests on (1)–(4), now reported in App. A.4 (L1106–1125).

Q2: Intended scope.

Our claims concern STEM short-answer grading; we do not claim that the findings directly generalize to essays, long-form responses, or other grading formats, and now state this scope explicitly (L97–99).

W2/Q3: Effect size, uncertainty, and supervision regime.

The effect is large in every subject: even after small-sample correction, Hedges' g ≥ 1.9, above the 0.8 threshold for a large effect, and every paired 95% CI excludes zero. We extended the seed-paired analysis of App. A.5 from CS to all four subjects (Tab. 5, right): under original-only correctness supervision, RITE reduces mean CFR by 38%–61% relative to PBCE-Orig, with reductions in all 12 seed pairs (Qwen3-8B; CFR %, mean±SD, 3 seeds):

SubjectPBCE-OrigRITE95% CI for ΔCFRHedges' g
CS11.7±1.85.2±0.7[−10.4, −2.6]3.8
Physics19.1±0.97.6±0.8[−13.2, −9.8]11.1
Chemistry16.1±2.56.2±1.1[−13.3, −6.5]4.1
Earth Sci.21.5±4.513.4±1.5[−15.6, −0.6]1.9

The equivalence loss is useful when correctness supervision does not cover the variants, consistent with the complementary roles in Thm. 2. When every variant receives inherited labels, these encode the equivalence and the constraint does not help (CS: 5.8% → 7.7%; Tab. 4). The Introduction now makes this dependence prominent (L95–97).

Q4: Are the transformation families distinct?

Yes, each operator changes a different property, and Distractor drifts more than Length within the same added-length ranges. Position changes order but not format; Style changes format but not order; Length elaborates existing content; Distractor adds new content irrelevant to every criterion. Tab. 8 shows all four on one answer; the App. A.4 checklist specifies what each preserves.

We added a length-stratified analysis (CS test, Qwen3-8B; family-specific CFR, %; new Tab. 10):

Added wordsn (L/D)Base LBase DPBCE-Orig LPBCE-Orig D
≤1522/1099.119.33.010.7
16–2340/5110.019.63.310.5
>23 (Length only)98/–11.2–3.1–

Even the longest Length additions drift less than the shortest Distractor additions. In both bins where the two families overlap, Distractor has higher CFR than Length, with gaps of 7.2–10.2 points, while CFR within each family changes by at most 2.1 points across bins. Thus, the larger Distractor effect is not explained by longer answers: adding task-irrelevant content is harder than elaborating existing evidence.

Thanks to the authors for their responses. I believe my original score remains appropriate, and I will therefore maintain it.

Thanks for your prompt response. We appreciate your positive feedback w. high confidence.

Reviewer_9cYXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies rubric drift: changes in an LLM grader’s criterion-level decisions when an answer’s presentation changes without changing its grading-relevant evidence. It introduces EduR²Bench, comprising verified equivalent-answer groups across four STEM subjects, and evaluates consistency alongside grading correctness. RITE combines criterion supervision on original answers with consistency regularization of the induced total-score distributions across equivalent variants. Experiments across two backbone models, multiple baselines, supervision and noise settings, and human-written variants support improvements in criterion-level stability while generally maintaining or improving grading quality.

Critical strengths
  1. The evaluation separates invariance from correctness and measures drift at the criterion level, where unchanged total scores can obscure inconsistent feedback. This is a useful and consequential improvement over assessing average grading accuracy alone.
  2. The experimental evidence is relatively comprehensive: multiple subjects and backbones, human-written variants, supervision and noise analyses, and negative results help establish both the benefits and boundaries of the approach. The appendix also explicitly examines criterion changes versus score changes rather than leaving this potential failure mode unaddressed.
Critical weaknesses

Total-score consistency does not guarantee criterion-level invariance, as the paper itself acknowledges; the empirical improvements should therefore carry the contribution rather than a stronger theoretical interpretation.

Also, original-answer-only supervision should not automatically be interpreted as an annotation-cost reduction: once equivalence has been verified, criterion labels can in principle be inherited by variants. These limitations affect the magnitude and framing of the contribution, but I do not consider them sufficient to outweigh the benchmark and empirical evidence.

Questions for authors

Does the original-answer-only supervision claim imply a reduction in human annotation cost, or only a different use of the available supervision? Please clarify the cost accounting, including equivalence verification and the possibility of inheriting labels for PBCE-All. This would help assess the contribution’s data-efficiency implications.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9cYX

We sincerely thank you for the insightful comment and positive feedback.

W1: Theoretical interpretation vs. empirical contribution.

We agree that total-score consistency is a necessary, rather than sufficient, condition for criterion-level invariance, as different criterion-level errors could theoretically cancel out to yield the same total score. This scope is already stated in the paper (Sec. 2; Prop. 2).

As you kindly noted, our empirical findings demonstrate that RITE's improvements extend beyond total-score consistency to substantial, consistent reductions in criterion flip rate (CFR) across subjects and backbones. In particular, our appendix analysis of criterion versus total-score changes shows that, for every trained grader, no group changes a criterion judgment while keeping its total score unchanged, whereas for the untrained grader 3.1–5.6% of groups do so (Tab. 19). Relative to original-only correctness training, CFR falls from 11.7 / 19.1 / 16.1 / 21.5% to 5.2 / 7.6 / 6.2 / 13.4% on the four subjects (Tab. 3), with the same pattern on Llama-3.1-8B (Tab. 24).

In the revision, we carefully calibrate our framing in the Introduction, clarifying any wording that might imply a strict theoretical guarantee of criterion-level invariance from total-score regularization: the theoretical guarantees concern total-score distributions and explain the complementary roles of correctness and equivalence supervision, while the criterion-level gains are established empirically across subjects and backbones (L90–91).

W2/Q1: Does original-only supervision reduce annotation cost?

No. We did not claim lower human annotation cost. "Original-only" describes where direct correctness supervision is applied in the training objective. Once rubric equivalence has been verified, the variant criterion labels can be inherited from the original. PBCE-All does exactly this. RITE and PBCE-All therefore require the same human inputs: (1) construction/review of the original criterion labels, and (2) generation and verification that each variant preserves rubric-relevant evidence and the criterion-level gold state (all 3,297 variants were manually verified; App. A.4).

The distinction is how the same verified equivalence structure is used during training. PBCE-All treats inherited variant labels as direct correctness targets, whereas RITE applies correctness loss only to the original and uses the variants through a relational equivalence constraint. With the same human inputs, RITE achieves lower CFR than PBCE-All across all four subjects and both backbones (Tab. 3; Tab. 24). These results indicate a more effective use of the same verified data to improve criterion-level stability.

In the revision, we replace potentially ambiguous phrases such as "using labels only on original responses" and "without a single variant label" with wording that explicitly refers to the scope of the correctness loss (Abstract, Introduction, Sec. 2, Tab. 3), and state in Sec. 3 that RITE and PBCE-All have the same annotation cost (L426–428).

Thank you for the response, which addressed most of my concerns. I believe this is a good paper, so I will maintain my positive assessment.

AuthorsOpenReview ↗

Response to Reviewer 9cYX

Thank you very much for your recognition! We appreciate your constructive suggestions, which helped us improve the paper.

Reviewer_U4FyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies changes in LLM grading judgments under answer transformations that preserve rubric-relevant evidence. It introduces EduR^2Bench and a criterion-level flip metric, finds that supervision on original responses also improves consistency on equivalent variants, and proposes RITE to strengthen this effect through score-distribution alignment. Experiments across four subjects and two backbones report improved consistency and grading quality, with additional tests on held-out transformation families and human-written rewrites.

Critical strengths
  1. The benchmark provides manually verified equivalence groups, question-level splits, and separate measures of correctness and consistency, making the evaluation reasonably well controlled.
  2. The experimental validation is substantial, including supervised augmentation and consistency baselines, loss-design ablations, a second backbone, and unseen human-written rewrites. The paper explicitly acknowledges that consistency alone can produce degenerate graders and that score-level alignment does not guarantee criterion-level agreement.
Critical weaknesses
  1. The distinction from prior work is not sufficiently established. Inconsistency under meaning-preserving variations and consistency-based training have already been studied in “Measuring and Improving Consistency in Pretrained Language Models” (TACL 2021) and “Unsupervised Data Augmentation for Consistency Training” (NeurIPS 2020). Verified rubric equivalence and criterion-level measurement provide a useful educational setting, but the paper does not clearly establish what new understanding this setting reveals. In particular, SII accompanies substantial accuracy improvements and is not separated from ordinary supervised generalization.
  2. RITE’s methodological contribution remains limited. It combines correctness supervision with consistency regularization, with total-score distribution alignment as its main distinctive choice. Relevant baselines and ablations are included, but the direct evidence favoring this choice is concentrated on the CS validation set. Moreover, verified equivalence allows variant labels to be inherited from the original, so the advantage over PBCE-All concerns the training objective rather than necessarily reducing supervision. The results support a useful recipe, but a broader methodological insight is not yet clear.
Questions for authors
  1. Beyond the educational setting and criterion-level granularity, what does rubric drift reveal that previous work on semantic consistency does not?
  2. Does SII remain evident when comparing models with similar criterion-level accuracy, or can the reduced disagreement largely be explained by fewer grading errors?
  3. When should total-score alignment outperform criterion-level consistency or direct label inheritance? Does this advantage hold across held-out subjects and both backbones?
5 comments
AuthorsOpenReview ↗

Response to Reviewer U4Fy

We appreciate your valuable feedback. We add clarifications and new results to distinguish our contribution from prior work.

W1/Q1: What does rubric drift reveal beyond semantic consistency?

The key distinction is that rubric equivalence is task-conditioned, not semantic. Elazar et al. (TACL'21) study consistency across meaning-preserving query paraphrases, while UDA (NeurIPS'20) enforces consistency under label-preserving augmentations. Here the question and rubric are fixed, while the judged answer changes. Two answers are equivalent only in rubric-relevant evidence; they need not be semantically equivalent. E.g., Distractor variants add new semantic content that should not affect grading.

This exposes an invariance not captured by paraphrase consistency: stability to alternative expressions does not imply the ability to ignore newly added but task-irrelevant content. Holding out Distractor in training raises its CFR from 1.3±0.6% to 6.5±1.0% (~5×), whereas holding out Position, Length, or Style causes no comparable degradation (Tab. 2).

Our main finding is supervision-induced invariance (SII). Consistency training explicitly teaches that x and T(x) should agree; SII asks whether ordinary correctness supervision, with no equivalence objective and no supervision on transformed inputs, can itself induce knowledge of which input differences should not matter. It can, but only partially and within the support of reliable supervision. This yields our grounding–invariance distinction: correctness supervision specifies what prediction is correct; equivalence supervision specifies which input differences should not change it. The former can induce the latter through shared-parameter generalization, but does not guarantee it (coverage, label-noise, and transfer; Sec. 3).

W1/Q2: Is SII merely fewer grading errors?

No. We added an analysis on the CS groups whose original answer all four graders judge correctly on every criterion in every seed (59 group–criterion pairs; Tab. 5):

ModelBasePBCE-OrigPBCE-AllRITE
CFR, all26.311.75.85.2
CFR, common-correct20.39.04.53.4

Even with no anchor errors to remove, original-only supervision more than halves CFR (20.3% → 9.0%). Thus, SII persists when all graders correctly judge the originals.

W2/Q3: Why total-score alignment, and when does it help?

Each loss has a distinct role, and both are needed. L_PBCE tells the grader what grade is correct by grounding each criterion of the original to its gold label; L_RI tells it which answers must receive the same grade by aligning the total-score distributions of verified equivalent answers. Without L_PBCE, the grader stays inaccurate (QWK 0.316) even as CFR falls to 11.9%; without L_RI, drift persists (CFR 11.7%) even as QWK reaches 0.737; together they more than halve CFR and raise QWK (5.2%, 0.753; CS, original-only supervision, Tab. 4).

Total-score alignment preserves this division of labor: it complements criterion-level correctness supervision by constraining consistency at the score-distribution level. Criterion-level alignment ties each variant's criterion predictions to the anchor's, which can propagate criterion-specific anchor errors; total-score alignment constrains the task-level outcome without requiring criterion-by-criterion agreement. On the test sets of all four subjects and both backbones (CFR %, mean±SD, 3 seeds; Tab. 17):

BackboneSubjectLabel inherit. (PBCE-All)Criterion align.Total-score align. (RITE)
Qwen3-8BCS5.8±1.66.7±1.05.2±0.7
Physics12.4±1.28.5±0.87.6±0.8
Chemistry12.8±0.96.9±1.06.2±1.1
Earth Sci.16.2±2.815.0±1.613.4±1.5
Llama-3.1-8BCS9.2±1.37.1±1.05.8±0.7
Physics11.4±1.68.3±0.87.4±1.0
Chemistry12.1±1.59.6±1.28.3±1.1
Earth Sci.13.4±1.711.7±1.310.2±1.3

RITE has the lowest CFR in all 8 settings; all 24 paired seeds favor total-score over criterion-level alignment, with QWK/C-F1 also higher. Its margin over criterion-level alignment is largest where QWK is lowest (Earth Science), and its margin over label inheritance extends to variations unseen in training (held-out Distractor: 3.1–3.8% vs. 6.5%, Tab. 2; human rewrites, Tab. 22).

The broader pattern is that equivalence regularization helps when direct correctness supervision leaves equivalent inputs underconstrained; once all variants carry inherited labels, it mildly over-regularizes (CFR 5.8% → 7.7%; Tab. 4). More or finer consistency constraints are not inherently better; their value depends on what correctness supervision already identifies.

Finally, we did not intend to claim reduced human annotation cost: verified variants can inherit labels, as in PBCE-All. PBCE-All and RITE differ in how verified equivalence is used, as inherited correctness targets or as a relational invariance constraint, and at the same cost RITE has lower CFR in all 8 settings.

Thanks for the extra experiments and clarifications. I understand the distinction from semantic paraphrasing: the answer can change in meaning while still satisfying the same rubric. But to me, this still fits within label-preserving robustness, where the task determines which changes should leave the label unchanged.

The new experiments make a stronger case that the method works, and I appreciate the work on the dataset. What I’m still missing is a clear takeaway beyond that existing framework. SII also still seems compatible with ordinary supervised generalization. So while the response helps with the empirical concerns, my concern about the added insight remains. I’m keeping my score of 2.

AuthorsOpenReview ↗

Response to Reviewer U4Fy

Thank you for the continued discussion. We're happy to provide new takeaways as follows for your consideration.

(1) Accuracy does not determine invariance. For a single pair (x,T(x))(x, T(x)), risk-based generalization controls error rates on originals and variants, eoe_o and e1e_1, which only bound CFR within [∣eo−e1∣,eo+e1][|e_o−e_1|, e_o+e_1]. Whether errors on xx and T(x)T(x) coincide is a separate property. We now report error consistency [1] between each grader's predictions on xx and T(x)T(x), which conditions on accuracy.

Gradereoe_o / e1e_1 (%)Error consistency
PBCE-Orig12.9 / 14.10.73
PBCE-All12.6 / 13.20.89
RITE9.9 / 10.50.86

PBCE-Orig and PBCE-All have nearly identical error rates, yet their error consistency is 0.73 vs. 0.89, with no seed overlap. RITE reaches a comparable level (0.86) with the correctness loss applied only to originals.

(2) How verified equivalence is encoded matters for transfer, at equal information. PBCE-All and RITE-All use identical variants and identical inherited labels. Encoding equivalence as per-point labels fits seen families better (5.8% vs 7.7% CFR), while encoding it as a relation transfers better to an unseen family (held-out Distractor 6.5% vs 3.1%, 3/3 seeds). The same holds on Llama-3.1-8B (9.0% vs 5.2%, 3/3 seeds). Label-preserving augmentation treats variants as extra (labeled or unlabeled) data and is agnostic to this choice.

We have added this discussion to App. A.7.

Reference:

[1] Geirhos, R., et al. "Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency", NeurIPS 2020.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response Summary for AC

Hello AC,

Thank you very much for your time and for overseeing our submission. We are very grateful for your help in coordinating the review process.

1. Summary

We would like to express our sincere gratitude to all reviewers for their insightful comments and constructive suggestions.

Contribution. This paper studies rubric drift, where LLM graders change their judgments on answers that carry the same rubric-relevant evidence. We contribute EduR²Bench (1,060 verified equivalence groups across four STEM subjects) with the label-free Criterion Flip Rate (CFR), show that correctness supervision alone removes only part of the drift (CFR 26.3% → 11.7%), and propose RITE, which lowers CFR to 5.2% while preserving grading quality and gives the lowest CFR across four subjects and two backbones.

  • Before the discussion{\color{orange}\textbf{Before the discussion}}, 9cYX and EzJT (confidence 4) rated the paper 3. 9cYX noted that the remaining limitations are not "sufficient to outweigh the benchmark and empirical evidence".
  • During the discussion{\color{orange}\textbf{During the discussion}}, U4Fy raised the rating from 2 to 3: "After reconsidering the results showing improvements in both stability and grading quality, along with the value of the dataset, I'm now leaning toward acceptance." 9cYX stated that our response "addressed most of my concerns" and that "I believe this is a good paper, so I will maintain my positive assessment." EzJT confirmed the score: "I believe my original score remains appropriate, and I will therefore maintain it."

All revisions are marked in blue in the updated PDF.

2. Key Strengths

Reviewers highlighted strengths across five dimensions:

  • Problem and Evaluation: Evaluates invariance separately from correctness, at the criterion level (9cYX, U4Fy, EzJT)
  • Benchmark: Manually verified and well-controlled, covering four STEM subjects (U4Fy, EzJT)
  • Effectiveness: An effective approach that improves both stability and grading quality (U4Fy, 9cYX, EzJT)
  • Empirical Rigor: Substantial validation with strong baselines, ablations, and tests on unseen variations (U4Fy, 9cYX, EzJT)
  • Soundness and Clarity: No substantive concerns about soundness (9cYX); clear and easy to follow (U4Fy, 9cYX)

3. Key Concerns and Our Responses

Key ConcernReviewersOur Response & Revision
What rubric drift reveals beyond semantic consistency and accuracyU4FyShowed that rubric equivalence is task-conditioned, not semantic (Tab. 2), and that accuracy does not determine invariance. Added in App. A.7, Tab. 25.
Evidence across subjects and backbones; effect sizeU4Fy, EzJTDemonstrated that RITE has the lowest CFR in all 8 settings, with large and consistent effects. Added in Tab. 5, 17.
Framing: theory, annotation cost, scope9cYX, U4Fy, EzJTClarified that guarantees concern total-score distributions, that RITE and PBCE-All have the same annotation cost, and that the scope is STEM short-answer grading. Revised in Introduction, Sec. 3.
Validity of scoring criteriaEzJTDocumented the criterion review procedure and added complete examples for three more subjects. Added in App. A.4, Tab. 9.
Distinctness of transformation familiesEzJTShowed that Distractor drifts more than Length at the same added length. Added in Tab. 10.

We thank the reviewers and the AC again for their time and feedback.

Bests, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.