acceptodds
Under review as a conference paper at ICLR 2027

THE HITCHHIKERS GUIDE TO RUBRIC QUALITY UNDERSTANDING AND ENRICHMENT

Abstract

Rubrics distill notions of expert quality and measure agent performance. However, the quality of rubrics themselves have not been systematically measured and are often left to downstream performance.We import apparatuses from measurement theory built for exactly this: quantitative signals based on the rubric's content, and introduce the RubrIc-Failure Taxonomy (RIFT), of nine possible ways a rubric fails, organized under reliability and content validity. Every mode leaves a distinct signature. To show the signals track failure causally, we seed 720 corruptions, injecting each RIFT mode into clean rubrics at known severity levels. A linear probe over the signals identifies which mode was injected at accuracy, beating for a frontier model asked to name the failure directly. Surprisingly across GDPval and Terminal-Bench, 10 of 48 expert-authored rubrics weight their criteria backwards, putting more of the score on requirements an expert panel judged less essential. This means a response can fail what matters most and still be graded well. This paper serves as a comprehensive guide on how to understand failure modes in rubrics and create better versions using quality signals, causal experiments, and provides a taxonomy with its rules and examples.

Primary area:
datasets and benchmarks
Keywords:
Rubrics, Quality Signals, Measurement Theory, Miscalibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MVtQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a framework for evaluating the quality of rubrics themselves and diagnosing their failure modes. Drawing on measurement theory, it introduces four families of quality signals: Content Validity Ratio, Domain Mapping, Cross-lens Expert Judging, and Weight Calibration. It also proposes RIFT, a taxonomy comprising nine failure modes. The authors apply controlled corruptions to 60 human-authored rubrics from GDPval and Terminal-Bench to analyze how different failure modes and corruption severity affect these signals. The experiments show distinct signal patterns across failure modes and suggest that some relevance metrics may not adequately capture structural problems in rubrics. The authors also compare a feature-based classifier with a direct LLM judge and examine the alignment between criterion weights and the essentiality judgments of a simulated expert panel.

Critical strengths
  • Treating a rubric as a measurement instrument and analyzing its quality from the perspectives of reliability and content validity is an interesting approach.

  • Comparing each rubric before and after corruption, while varying corruption severity, provides a useful way to analyze the relationship between failure modes and quality signals.

  • The finding that non-atomic corruption, which combines multiple criteria, can increase weighted relevance is useful. It shows that averaging criterion-level relevance can overlook structural problems in rubrics.

Critical weaknesses
  • The need for the full set of quality signals and the incremental diagnostic value of its components are not sufficiently justified. The Introduction and Sec3 explain the questions addressed by the four signal families. However, it remains unclear why so many signals need to be considered together and what additional information each provides over simpler combinations. The finding that different signals respond to different failure modes is useful, but it does not, by itself, establish the necessity of the full signal set. Moreover, the mediation analysis in Sec 5.3 examines only relationships among a subset of signals under the Hackable failure mode.

  • Sec3 presents numerous concepts and formulas from measurement theory in a limited space. Although some definitions and interpretations are provided, the explanation is insufficient for readers unfamiliar with measurement theory to follow how expert judgments are converted into the final signals. For ML/AI audience, a more detailed explanation should be provided, at least in the appendix.

  • Also, the rules underlying Items Good, Items Review, and Items Bad, as well as the definition of Candidate Item Mean in Fig3 and Table 4, are not clearly connected to the method description. A table summarizing each signal’s definition, aggregation rule, range, thresholds where applicable, and interpretation would be helpful. The appendix should include a worked example tracing the process from expert judgments to signal computation and failure interpretation for a single task and rubric. The interpretation caveats discussed elsewhere should also be presented alongside the signal definitions, particularly for signals whose increase or decrease does not consistently indicate an improvement or deterioration in rubric quality.

  • The experimental setup is described across multiple sections, and some evaluation procedures are not sufficiently specified. Corpus and injection paragraph in Sec5 describes the dataset size, corruption levels, and primary models. However, the settings for feature scoring, the direct classification baseline, and cross-judge validation are scattered across different parts of the paper, making the overall experimental design difficult to follow. Although Sec 5 introduces GPT-5.5 as the feature scorer and Sonnet as the corruption model, the direct classification baseline using Opus is first introduced in the results paragraph of Sec 5.1.

  • The scope of cross-model validation is limited. The additional judge models in Appendix D are a positive inclusion, but this analysis only reproduces the decrease in mean CVR for three failure modes: Hackable, Rigid, and Misaligned. It does not establish whether the full signal vector and failure-mode classification results remain consistent across judge models. This makes it difficult to assess how sensitive the broader conclusions are to the choice of feature scorer. Although Sec 5 points to Appendix D, I had to consult the appendix to understand which additional judges were used and the scope of this validation.

  • Research on LLM-as-a-Judge and rubric evaluation is cited, but its implications for the proposed evaluation pipeline require further discussion. The quality signals depend on LLM-generated personas and objectives, as well as LLM-based essentiality and relevance judgments. Therefore, more discussion is needed of evaluator reliability, persona sensitivity, and alignment with human judgments. The current explanation does not clearly establish the extent to which the signals reflect rubric defects and the extent to which they may reflect the evaluator’s judgment patterns.

  • The comparison between the feature-based classifier and the direct LLM judge does not clearly use matched evaluation conditions. For the reported 75.0% accuracy result in Sec 5.1, the classifier uses the 17-signal delta vector between a corrupted rubric and its clean parent to distinguish eight failure modes. In contrast, the direct LLM judge receives the task instruction and rubric, predicts one of the eight failure modes or “none,” and is evaluated on 540 cases, including 60 clean rubrics. As described, the two evaluations appear to differ in their evaluation sets, label spaces, and access to a clean reference. Without matched conditions, it is difficult to attribute the reported accuracy gap to the additional diagnostic value of the quality signals. A comparison using the same evaluation set, label space, and reference information is needed.

Questions for authors

I think the paper’s topic interesting, and I think the attempt to examine a wide range of aspects of rubric quality is meaningful. However, the motivation for examining so many quality signals together, and the specific problem this combination addresses, were not communicated clearly enough to me. Explaining this motivation more clearly in the Introduction would make the need for the work and its contribution more persuasive.

In particular, the limited explanation for ML/AI readers had a substantial impact on my score. Given that ICLR is an ML/AI venue, readers should not be assumed to have a background in measurement theory. Readers unfamiliar with the field should still be able to understand the intuition, purpose, and application of the concepts presented. I believe this accessibility is necessary to build a meaningful interdisciplinary bridge between measurement theory and ML/AI. The current manuscript provides insufficient background explanation, making it difficult to follow how the theoretical concepts and formulas connect to the problem statement.

The experimental setup was also difficult to understand. Some methodological details were missing or scattered across multiple sections, and the roles and purposes of some models were not sufficiently explained before their results were presented. I also had to consult the appendix to understand several details needed to interpret the experiments. A dedicated Experimental Setup subsection would help explain the purpose of each experiment, the role of each model, the evaluation procedures, and the interpretation of the metrics.

Space constraints may explain some of these presentation choices. However, the main text should provide sufficient information to understand the study’s motivation and experimental design. These concerns about motivation, accessibility, and the experimental setup are the main reasons for my recommendation to reject the paper.

6 comments

Thank you for acknowledging our interesting methods to rubric understanding, improvement, and our controlled interventions and findings of structural problems with rubrics. We also thank you for your careful reading and comments on our paper.

You mentioned that the most substantive effect on your score and comment is a lack of explanation and theoretical grounding, and improving accessibility for ML readers, along with definitions, examples, and changes added to the paper. These now have a concrete fix in the paper: a motivating sentence, the theory appendix with the signal table and worked example, and a dedicated setup paragraph. The new experiments also sharpen the contribution. The signals measure rubric validity in a way that catches what a frontier LLM judge misses, raising its accuracy from 89.2%89.2\% to 96.5%96.5\% with statistical significance. This is a concrete, ML-facing use of a measurement-theory bridge that you described as "meaningful".

A. Expanding upon the quality signals

What we added

  1. Background for ML readers (new Appendix). The appendix has two short subsections. In a camera ready version we will add and make these richer.

    • Measurement theory explains what makes a score meaningful (validity vs. reliability, construct-irrelevant variance, underrepresentation) and maps each idea to the signal that operationalizes it.
    • Education theory shows that RIFT's failure modes were documented in classroom rubrics decades ago. Popham's overly general criteria became our subjective-language mode. Sadler's missing and double-counted criteria became our missing-criteria and redundant modes. Brookhart's "counting" criteria became our hackable mode.
  2. Signal table (new Table). For each of the 17 signals, the table gives its definition and aggregation rule, its range, its thresholds where applicable, its healthy direction, and the injected modes that move it. It also states the caveats you asked for. Weighted relevance rises under non-atomic and redundant corruption, so higher is not uniformly better. The adjacent and background shares have no healthy direction.

  3. Worked example (new Appendix). We trace one task rubric from end to end. Four of its eight criteria are rewritten into hackable form, e.g. "explains that readouts disagree across cohorts" becomes "includes the words 'pathway activation' and 'sequencing batches'". The example then follows the pipeline:

    • all 5 personas now rate the rewritten criteria non-essential (CVR 1→−11\to-1);
    • the rewrites move from the core domain ring to the general ring;
    • their relevance drops from 3.03.0 to 1.01.0–2.02.0;
    • the 17-signal vector changes, while the signals that stay flat (item count, redundancy) rule out the structural modes;
    • the classifier and the judge both output hackable.
  4. Why 17 signals (ablation, new Appendix). Each lens carries information the others lack:

FeaturesAccuracy (8 labels, chance 12.5)
All 17 signals75.075.0
Without CVR65.065.0
Without cross-lens34.634.6
Without item counts53.553.5
Without domain mapping75.075.0

Domain mapping adds little accuracy. We keep it because it tells an author where an item drifted, not just that it did, and the paper now says so. The signals also add +7.3+7.3 pts to a frontier judge.

Changes to the paper

  • Introduction: one sentence motivating the signal vector (each signal answers a different validity question)
  • §3: each lens points to Table 4. Good/review/bad items and candidate item mean are now defined.
  • §5: We state each model's role: GPT-5.5 personas score the signals, Sonnet 4.6 injects corruptions, Opus 4.8 is the direct judge, and Opus 4.8, Haiku 4.5 and Gemini 3.1 Pro form the cross-judge panel. It also gives the evaluation protocol (leave-one-task-out CV, task-cluster bootstrap, McNemar). Line 290
  • §5, injection framing: injection is a controlled experiment where the defect is known by construction. Claims are scoped to recognizing injected single-mode defects.
  • §5 / Appendix D: the scope of the cross-judge check (mean CVR, three modes) is now stated in the main text in line 295.

B. Matched classifier–judge comparison

You are right that the original comparison was unmatched. We now give both methods the same examples, label set and access to the clean rubric (judge: Opus 4.8).

With clean rubric (8 labels, n=480n=480)Without (9 labels, n=540n=540)
Signals (LR)75.075.0 [71.5, 78.3]39.339.3 [35.6, 43.3]
LLM judge89.289.2 [87.7, 90.8]56.956.9 [52.0, 61.3]
Judge + signals96.596.5 [95.0, 97.7]59.659.6 [55.6, 63.9]

Under matched conditions the judge beats the signals alone. The signals are a complement: adding them fixes 38 judge errors and introduces 3 (p=1.1×10−8p=1.1\times10^{-8}). The gain comes from ungrounded verification, where the judge's recall is 0.200.20 and the combination's is 0.830.83. In practice, rubric defects and labels often overlap such as in Wang et al 2025, Sharma et al. 2026. This has been added to the conclusion in line 527+

Changes: the abstract and §5.1 now clarify the injection detection, updated numbers and the complementarity claim. Appendices give per-mode recall, absolute-signal probes, ablations and robustness checks.

Evaluator reliability

The signals are validated against defects whose ground truth is known. They shift in the predicted direction for every mode, scale with corruption dose, and replicate under an independent Claude/Gemini panel (hackable d=−2.83d=-2.83 under GPT-5.5 vs. −2.51-2.51 under the panel). We state the remaining gap, item-level agreement with human experts, as a limitation and as future work.

Dear Reviewer,

We have invested considerable effort in preparing the rebuttals are we are excited by the improvements. Your suggestions have made our claims considerably stronger. We would love for you to consider increasing your score for our work.

If you have further questions or concerns, please raise them promptly. We would like to make the most of the discussion period through active and constructive communication.

Even if you have no additional comments, a brief acknowledgment would be greatly appreciated.

Thank you so much for your time!

Reviewer_XcihOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates rubric quality assessment and failure diagnosis in LLM evaluation. Drawing on measurement theory, it introduces four families of quality signals and RIFT, a taxonomy covering nine rubric failure modes. The experiments construct 720 corrupted variants from 60 expert-authored rubrics in GDPval and Terminal-Bench. A classifier using 17 quality signals achieves 75.0% accuracy in identifying injected failure modes, compared with 56.7% for direct LLM classification. The paper also reports that 10 of 48 analyzable expert-authored rubrics exhibit negative correlations between criterion weights and panel-assessed essentiality. These results provide useful evidence about rubric quality, although their implications for practical diagnosis remain uncertain.

Critical strengths

1.Relevant research problem. The paper examines rubric quality directly rather than relying solely on downstream evaluation performance, addressing an important issue in the reliability of LLM evaluation.

  1. Meaningful methodological integration. The combination of measurement-theoretic quality signals and the RIFT taxonomy provides a systematic approach to identifying and interpreting rubric deficiencies.

  2. Systematic controlled evaluation. The experiments examine different failure modes and corruption levels through paired comparisons, with multiple-testing correction and additional analyses across datasets and evaluation models.

Critical weaknesses
  1. The core classification results depend on access to reference rubrics.

Section 5.1 uses differences in 17 quality signals between original rubrics and corrupted variants to achieve 75.0% classification accuracy. This setup is appropriate for studying controlled interventions, but practical diagnosis typically involves rubrics without known clean references. Although the paper mentions population-level reference distributions as an alternative, it does not evaluate classification performance under this condition.

The comparison with direct LLM classification also involves unequal information. The quality-signal classifier uses differences from the original rubric, while the LLM baseline receives only the task description and current rubric. Consequently, the 75.0% versus 56.7% accuracy gap does not fully establish an advantage under equivalent diagnostic conditions.

  1. Identifying injected defects does not necessarily establish the ability to diagnose genuine rubric deficiencies.

The experiments construct failure modes by deleting, duplicating, merging, or rewriting criteria. Some operations introduce structural changes that are directly reflected in the quality signals. For instance, Missing Criteria, Redundant Criteria, and Non-Atomicity can alter criterion counts or organization. Classification performance may therefore partly reflect recognition of these editing operations rather than semantic deficiencies.

Similar structural changes do not always indicate the same problem in practice, and real rubrics may contain multiple overlapping failures. The current experiments primarily consider individual injected defects, leaving diagnostic performance on naturally occurring and coexisting deficiencies unclear.

  1. The validity of simulated expert judgments remains insufficiently established.

Several quality signals rely on five task-specific expert personas generated using GPT-5.5. While this enables scalable evaluation, agreement among simulated personas does not necessarily correspond to independent human expert judgments. Appendix D provides some cross-model evidence, but does not establish this correspondence.

This concern also affects the weight-calibration analysis in Section 5.4. The reported negative correlations in 10 of 48 rubrics indicate disagreement between criterion weights and panel-assessed essentiality. However, essentiality and appropriate scoring weight are not equivalent, since weights may also reflect error severity, task priorities, and scoring rules. These correlations alone do not establish genuine weight-allocation errors.

Questions for authors

How does the proposed classifier identify failure modes when no clean reference rubric is available? Is the reported 75.0% accuracy representative of such settings?

To what extent does the 18.3-percentage-point advantage over direct LLM classification result from differences in the information available to the two methods?

Can the quality signals distinguish naturally occurring or overlapping failure modes, rather than primarily identifying structural changes introduced by predefined corruption operations?

How closely do the simulated expert panel's essentiality judgments reflect actual scoring priorities? To what extent do the negative correlations in Section 5.4 demonstrate genuine weight-allocation problems?

2 comments

Thank you for acknowledging our relevant research problem, meaningful methods and systematic evaluations. We thank you for your careful review and the four precise questions. Your questions led to the paper's strongest new result. Under matched information the signals do not replace an LLM judge, but they fix its main blind spot and raise its accuracy from 89.2% to 96.5%. That gives the measurement-theory approach you called a "meaningful methodological integration" a concrete role in LLM evaluation. All claims are now scoped to what the experiments test.

1. New results

A. Reference access and the matched comparison.

Q1: performance without a clean reference. We now evaluate the population-reference alternative you mention, z-scoring each signal against clean rubrics from the same benchmark family, along with raw absolute signals. The 75.0% is not representative of the reference-free setting:

Probe without a reference8 labels (chance 12.5)9 labels, +clean (chance 11.1)
Absolute signals45.4 [40.8, 50.2]39.3 [35.6, 43.3]
Population-normalized45.8 [40.8, 51.3]40.4 [36.3, 44.8]

Q2: how much of the 18.3-point gap is information? All of it. With identical examples, labels and access to the clean rubric, the judge (Claude Opus 4.8) beats the signals alone:

With clean rubric (8 labels)Without (9 labels)
Signals (LR)75.0 [71.5, 78.3]39.3 [35.6, 43.3]
LLM judge89.2 [87.7, 90.8]56.9 [52.0, 61.3]
Judge + signals96.5 [95.0, 97.7]59.6 [55.6, 63.9]

The finding that holds is that the signals complement the judge. With the clean rubric, adding them fixes 38 judge errors.

B. Injected vs. genuine defects (Q3). You are right that three modes change the item count by construction. Two results separate structural edits from semantic ones:

  • Removing all four item-count features leaves 53.5% [49.8, 57.3] accuracy (4.3×4.3\times chance).
  • The five rewrite modes edit the wording of criteria and leave the item count unchanged. The signals still recover them: hackable 0.870.87, ungrounded verification 0.780.78, rigid 0.680.68, subjective language 0.530.53, misaligned 0.420.42. Their mean recall is 0.660.66, against a chance level of 0.1250.125.

We do not have naturally occurring or overlapping defects labeled by independent human experts. We scope all claims to recognizing injected single-mode defects and state this as a limitation.

C. Simulated experts and weight calibration (Q4). We agree that persona panels are not a substitute for human experts, and we do not claim they are. Our claim is narrower: the signals computed from the panels are valid measures, and we test that against defects whose ground truth is known:

  • Known-groups validity: every injected mode produces a significant signature in the predicted direction (46/136 mode × signal cells, FDR q<.05q<.05).
  • Specificity: the redundancy signal responds to the redundant mode and to no other.
  • Dose-response: damage scales with corruption rate (hackable CVR slope −1.55-1.55 per unit ρ\rho, p=1.6×10−12p=1.6\times10^{-12}).
  • Not model-specific: an independent panel (Claude Opus 4.8, Haiku 4.5, Gemini 3.1 Pro) shares no model with GPT-5.5 and reproduces the CVR collapse:
ModeGPT-5.5 ddIndependent panel ddpp (panel)
Hackable−2.83-2.83−2.51-2.512.6×10−272.6\times10^{-27}
Rigid−2.04-2.04−1.89-1.892.9×10−212.9\times10^{-21}
Misaligned−1.85-1.85−2.24-2.248.5×10−258.5\times10^{-25}

On §5.4 we agree. Essentiality and appropriate weight are different constructs: weights also encode error severity and scoring rules. The 10/48 negative correlations show disagreement between author weights and panel essentiality. They are a flag for review, not established weight-allocation errors.

2. Changes to the paper

  • Abstract and §5.1: the matched comparison and complementarity claim replace 75.0 vs. 56.7.
  • Appendix I (new, Extended Classification Results): reference-free and population-normalized probes, item-count ablations, per-mode recall, and robustness.
  • §5.4: reframed as disagreement between weights and panel essentiality, with the severity and scoring-rule caveat.
  • §5 / Appendix D: the scope of the cross-judge check is stated in the main text 295.
  • Limitations: natural and overlapping defects; item-level calibration of persona judgments against human experts is left for future work.

Dear Reviewer,

We have invested considerable effort in preparing the rebuttals are we are excited by the improvements. Your suggestions have made our claims considerably stronger. We would love for you to consider increasing your score for our work.

If you have further questions or concerns, please raise them promptly. We would like to make the most of the discussion period through active and constructive communication.

Even if you have no additional comments, a brief acknowledgment would be greatly appreciated.

Thank you so much for your time!

Reviewer_hGE2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RIFT, a nine-mode taxonomy of rubric failures, alongside content-validity, domain-alignment, and weighting signals. Controlled corruptions of 60 rubrics test whether these signals identify particular defects, with a separate study of weight miscalibration. The goal is useful: give rubric designers feedback before relying on downstream model performance.

Critical strengths

I like the combination of an actionable taxonomy and controlled interventions, particularly the finding that merging criteria can improve relevance scores while making a rubric harder to grade. Holding task-level instruments fixed and leaving entire tasks out of classifier training are sensible design choices. The additional judges provide useful corroboration for the three large content-validity effects, and the taxonomy's examples make the RIFT framework approachable.

Critical weaknesses

I have one major (minor/major, can be fixed by wording changes but also changes the impact of the claims in the abstract) qualm about the experimental reporting. The headline 75.0% versus 56.7% comparison is not matched: the classifier receives clean-to-corrupted feature differences and predicts eight injected modes, while the direct judge sees individual rubrics and predicts nine labels, including clean examples (Section 5.1). This does not establish the claimed 18.3-point advantage under comparable conditions. More broadly, the necessity ratings come from simulated LLM personas, so the 10/48 negative correlations establish disagreement with that instrument, not independently verified mistakes by human rubric authors. The claim that signals are independent also does not follow from nonsignificant mediation estimates (Section 5.3); synthetic corruption recognition and real-world diagnostic validity remain distinct.

Questions for authors
  1. Can you compare both methods on identical examples, label sets, and access to clean rubrics, and report classifier performance using only absolute features?
  2. How much accuracy remains after removing item-count features, and on naturally occurring or mixed defects labeled by independent human experts?

Q1 is the big question there and aligns with my major qualm above, and I think Q2 falls out of a satisfactory answer to Q1.

2 comments

Thank you for acknowledging our actionable taxonomy, controlled interventions and insightful findings. Also thanks for the precise questions and suggested fixes. Your suggestion to reframe some of our wording and comparison experiments made our paper and claims about quality signals stronger. You described the main issue as fixable by wording that also changes the impact of the abstract's claims. The matched experiments replace a confounded comparison with a cleaner and more useful finding. Measurement-theory signals catch what a frontier judge misses, and together they reach 96.5% where the judge alone reaches 89.2%. We have rewritten some of our claims so they match the experiments.

1. New results

A. Matched comparison (main concern, Q1). You are right that the original comparison was unmatched. The abstract says the probe identifies which mode was injected, which assumes a clean parent, but the judge never saw one. We now give both methods identical examples, label sets and access to the clean rubric (judge: Claude Opus 4.8, same prompt as §5.1).

With clean rubric (8 labels, n=480n=480)Without (9 labels, n=540n=540)
Signals (LR)75.0 [71.5, 78.3]39.3 [35.6, 43.3]
LLM judge89.2 [87.7, 90.8]56.9 [52.0, 61.3]
Judge + signals96.5 [95.0, 97.7]59.6 [55.6, 63.9]

Under matched conditions the judge beats the signals alone, so we reframe the abstract to show the new results. The result that holds is that the signals complement the judge. With the clean rubric, adding the signals fixes 38 judge errors.

Absolute features and item counts (Q1, Q2).

FeaturesWith clean parent (8)Absolute (8)Absolute (9, +clean)
All 17 signals75.045.4 [40.8, 50.2]39.3 [35.6, 43.3]
Without item counts (13)53.5 [49.8, 57.3]33.8 [30.2, 37.5]29.6 [26.3, 33.0]
Item counts only (4)49.0 [46.0, 51.9]25.622.4

Chance is 12.5% with 8 labels and 11.1% with 9. Absolute signals identify the mode at more than 3×3\times chance; z-scoring within benchmark gives 45.8%.

B. Scope of the validity claims.

  • Personas. Agreed. The 10/48 negative correlations show disagreement with the with the persona instrument and we have reframed this in the paper. The personas are generated from the task instruction alone, never from the rubric they judge. The CVR collapse replicates under an independent Claude/Gemini panel (hackable d=−2.83d=-2.83 under GPT-5.5 vs. −2.51-2.51 under the panel).
  • Mediation. Agreed: a non-significant indirect effect is not evidence of independence. The ablation shows instead that the lenses carry non-redundant information. Without cross-lens features accuracy is 34.634.6, without CVR 65.065.0, and without domain mapping 75.075.0 (unchanged).
  • Synthetic vs. natural (Q2). We do not have naturally occurring or mixed defects labeled by independent human experts.Due to resource constraints, we unfortunately cannot build that set within the rebuttal window. We scope every claim to recognizing injected single-mode defects.

2. Changes to the paper

  • Abstract: "Given a rubric and its clean parent, a linear probe over the signals identifies the injected mode at 75.0% accuracy; a frontier judge given the same pair reaches 89.2%, and adding the signals to the judge raises accuracy to 96.5%."
  • §5.1: a matched-comparison paragraph and table.
  • Appendix I (new, Extended Classification Results): CIs, per-mode recall, absolute-signal probes, ablations, and robustness to classifier family and dose.
  • §5.3: "no evidence of mediation"; the case for the full vector now rests on the ablation.
  • §5.4: negative correlations are framed as disagreement with the panel instrument.
  • Limitations: natural and mixed defects, and item-level agreement with human experts.

Dear Reviewer,

We have invested considerable effort in preparing the rebuttals are we are excited by the improvements. Your suggestions have made our claims considerably stronger. We would love for you to consider increasing your score for our work.

If you have further questions or concerns, please raise them promptly. We would like to make the most of the discussion period through active and constructive communication.

Even if you have no additional comments, a brief acknowledgment would be greatly appreciated.

Thank you so much for your time!

Comments on the paper

1 comment

Thank you for all of your careful comments, feedback, and suggestions.

We thank the reviewers for the acknowledgement that our paper is timely, provides a new actionable taxonomy, grounded quality signals, a systematic controlled intervention, and shows downstream effects of failure modes on overall rubric instruments.

We also them for their suggested changes and thorough and deep reviews. These changes have helped us improve the paper overall and establish stronger claims. With our new experiments we find that our signals complement the judges while being deeply grounded in theoretical knowledge. Here’s a summary of our claims, changes, and takeaways.

  1. The main claim of the paper is the importance of grounding modern evaluation systems in decades of established theory. To support the readers with a deeper background and relevant literature, in the new draft we add a new appendix A - Quality signals. This section talks about both measurement theory and education theory and how literature from both these domains motivate building our quality signals. In this section we show how established theoretical practices translate directly to our quality signals, for example, how double-counted criterion in education theory becomes “redundant” or how “overly general criterion” by Popham becomes “subjective language”. This new section provides readers transitional context from theory to practice, while grounding our research.

  2. Exciting New Results! Aligning with the reviewers’ suggestions on comparing the judge and logistic model with a clean reference we find exciting new results from our quality signals. We ran new matched experiments in which a frontier judge (Claude Opus 4.8) and a logistic regression (LR) over the 17 signals see the same rubrics, with or without the clean parent, and a second LR combines the judge’s label with the signals (chance: 12.5% with 8 labels, 11.1% with 9). with clean and corrupted rubrics. Chance is 12.5% with 8 labels and 11.1% with 9. Absolute signals identify the mode at more than 3x chance. A frontier model accuracy with the signals also jumps up to 96.5% - further proving that our quality signals are consistent and complement frontier models and help statistical models alike.

  3. Added paper depth. Taking into account your comments, we add Table 4 to the appendix showing every quality signal in detail. We also add Appendix H to show a full worked example as suggested. We have added more theory, explanations, grounding, and experiments. As the paper moves to a camera ready state we can commit to further enriching the background section added during this rebuttal period.

  4. Clarifying the injection mode detection and signal. We reframe the abstract to make the injection drawing sentence much clearer. We clearly state that “Given the clean parent, a linear probe over the signals identifies the injected mode at 75.0%75.0\% accuracy and a frontier judge at 89.2%89.2\%; combining the judge's prediction with the signals reaches 96.5%96.5\%. Without a reference, both the probe and the judge remain statistically significantly above chance (39.3% and 56.9% vs. 11.1%”. We also use the added space to improve the conclusion of this paper. We draw the fact that our signals complement the judges while being deeply grounded in theoretical knowledge. In here we clearly mention in lines 532 onwards the limitations related to naturally occurring defects, expert panel simulations and more.

  5. We explain the presence of a reference - Our reference-based setting matches how expert rubrics are built in practice. Benchmark rubrics are drafted, reviewed and revised over several rounds: Such as in ProfBench (Wang et al., 2025), PaperBench rubrics which undergo “several rounds of internal review” (Starace et al., 2025); ResearchRubrics (Sharma et. al. 2026) also states that rubrics are “iteratively refined”; the assessment literature prescribes the same piloting and revision (Moskal & Leydens, 2000). Therefore when such an edit history exists (as it often does), the signals apply exactly as in our experiments; when it does not, as for rubrics an LLM generates in one pass, the reference-free results above still hold at more than 3× chance.

We thank you once again for your time, comments, feedback and detailed observations which helped us make this paper and its claims stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.