- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a framework for evaluating the quality of rubrics themselves and diagnosing their failure modes. Drawing on measurement theory, it introduces four families of quality signals: Content Validity Ratio, Domain Mapping, Cross-lens Expert Judging, and Weight Calibration. It also proposes RIFT, a taxonomy comprising nine failure modes. The authors apply controlled corruptions to 60 human-authored rubrics from GDPval and Terminal-Bench to analyze how different failure modes and corruption severity affect these signals. The experiments show distinct signal patterns across failure modes and suggest that some relevance metrics may not adequately capture structural problems in rubrics. The authors also compare a feature-based classifier with a direct LLM judge and examine the alignment between criterion weights and the essentiality judgments of a simulated expert panel.
Critical strengths
-
Treating a rubric as a measurement instrument and analyzing its quality from the perspectives of reliability and content validity is an interesting approach.
-
Comparing each rubric before and after corruption, while varying corruption severity, provides a useful way to analyze the relationship between failure modes and quality signals.
-
The finding that non-atomic corruption, which combines multiple criteria, can increase weighted relevance is useful. It shows that averaging criterion-level relevance can overlook structural problems in rubrics.
Critical weaknesses
-
The need for the full set of quality signals and the incremental diagnostic value of its components are not sufficiently justified. The Introduction and Sec3 explain the questions addressed by the four signal families. However, it remains unclear why so many signals need to be considered together and what additional information each provides over simpler combinations. The finding that different signals respond to different failure modes is useful, but it does not, by itself, establish the necessity of the full signal set. Moreover, the mediation analysis in Sec 5.3 examines only relationships among a subset of signals under the Hackable failure mode.
-
Sec3 presents numerous concepts and formulas from measurement theory in a limited space. Although some definitions and interpretations are provided, the explanation is insufficient for readers unfamiliar with measurement theory to follow how expert judgments are converted into the final signals. For ML/AI audience, a more detailed explanation should be provided, at least in the appendix.
-
Also, the rules underlying Items Good, Items Review, and Items Bad, as well as the definition of Candidate Item Mean in Fig3 and Table 4, are not clearly connected to the method description. A table summarizing each signal’s definition, aggregation rule, range, thresholds where applicable, and interpretation would be helpful. The appendix should include a worked example tracing the process from expert judgments to signal computation and failure interpretation for a single task and rubric. The interpretation caveats discussed elsewhere should also be presented alongside the signal definitions, particularly for signals whose increase or decrease does not consistently indicate an improvement or deterioration in rubric quality.
-
The experimental setup is described across multiple sections, and some evaluation procedures are not sufficiently specified. Corpus and injection paragraph in Sec5 describes the dataset size, corruption levels, and primary models. However, the settings for feature scoring, the direct classification baseline, and cross-judge validation are scattered across different parts of the paper, making the overall experimental design difficult to follow. Although Sec 5 introduces GPT-5.5 as the feature scorer and Sonnet as the corruption model, the direct classification baseline using Opus is first introduced in the results paragraph of Sec 5.1.
-
The scope of cross-model validation is limited. The additional judge models in Appendix D are a positive inclusion, but this analysis only reproduces the decrease in mean CVR for three failure modes: Hackable, Rigid, and Misaligned. It does not establish whether the full signal vector and failure-mode classification results remain consistent across judge models. This makes it difficult to assess how sensitive the broader conclusions are to the choice of feature scorer. Although Sec 5 points to Appendix D, I had to consult the appendix to understand which additional judges were used and the scope of this validation.
-
Research on LLM-as-a-Judge and rubric evaluation is cited, but its implications for the proposed evaluation pipeline require further discussion. The quality signals depend on LLM-generated personas and objectives, as well as LLM-based essentiality and relevance judgments. Therefore, more discussion is needed of evaluator reliability, persona sensitivity, and alignment with human judgments. The current explanation does not clearly establish the extent to which the signals reflect rubric defects and the extent to which they may reflect the evaluator’s judgment patterns.
-
The comparison between the feature-based classifier and the direct LLM judge does not clearly use matched evaluation conditions. For the reported 75.0% accuracy result in Sec 5.1, the classifier uses the 17-signal delta vector between a corrupted rubric and its clean parent to distinguish eight failure modes. In contrast, the direct LLM judge receives the task instruction and rubric, predicts one of the eight failure modes or “none,” and is evaluated on 540 cases, including 60 clean rubrics. As described, the two evaluations appear to differ in their evaluation sets, label spaces, and access to a clean reference. Without matched conditions, it is difficult to attribute the reported accuracy gap to the additional diagnostic value of the quality signals. A comparison using the same evaluation set, label space, and reference information is needed.
Questions for authors
I think the paper’s topic interesting, and I think the attempt to examine a wide range of aspects of rubric quality is meaningful. However, the motivation for examining so many quality signals together, and the specific problem this combination addresses, were not communicated clearly enough to me. Explaining this motivation more clearly in the Introduction would make the need for the work and its contribution more persuasive.
In particular, the limited explanation for ML/AI readers had a substantial impact on my score. Given that ICLR is an ML/AI venue, readers should not be assumed to have a background in measurement theory. Readers unfamiliar with the field should still be able to understand the intuition, purpose, and application of the concepts presented. I believe this accessibility is necessary to build a meaningful interdisciplinary bridge between measurement theory and ML/AI. The current manuscript provides insufficient background explanation, making it difficult to follow how the theoretical concepts and formulas connect to the problem statement.
The experimental setup was also difficult to understand. Some methodological details were missing or scattered across multiple sections, and the roles and purposes of some models were not sufficiently explained before their results were presented. I also had to consult the appendix to understand several details needed to interpret the experiments. A dedicated Experimental Setup subsection would help explain the purpose of each experiment, the role of each model, the evaluation procedures, and the interpretation of the metrics.
Space constraints may explain some of these presentation choices. However, the main text should provide sufficient information to understand the study’s motivation and experimental design. These concerns about motivation, accessibility, and the experimental setup are the main reasons for my recommendation to reject the paper.
6 comments
Domain mapping adds little accuracy. We keep it because it tells an author where an item drifted, not just that it did, and the paper now says so. The signals also add pts to a frontier judge.
Changes to the paper
- Introduction: one sentence motivating the signal vector (each signal answers a different validity question)
- §3: each lens points to Table 4. Good/review/bad items and candidate item mean are now defined.
- §5: We state each model's role: GPT-5.5 personas score the signals, Sonnet 4.6 injects corruptions, Opus 4.8 is the direct judge, and Opus 4.8, Haiku 4.5 and Gemini 3.1 Pro form the cross-judge panel. It also gives the evaluation protocol (leave-one-task-out CV, task-cluster bootstrap, McNemar). Line 290
- §5, injection framing: injection is a controlled experiment where the defect is known by construction. Claims are scoped to recognizing injected single-mode defects.
- §5 / Appendix D: the scope of the cross-judge check (mean CVR, three modes) is now stated in the main text in line 295.
B. Matched classifier–judge comparison
You are right that the original comparison was unmatched. We now give both methods the same examples, label set and access to the clean rubric (judge: Opus 4.8).
| With clean rubric (8 labels, ) | Without (9 labels, ) | |
|---|---|---|
| Signals (LR) | [71.5, 78.3] | [35.6, 43.3] |
| LLM judge | [87.7, 90.8] | [52.0, 61.3] |
| Judge + signals | [95.0, 97.7] | [55.6, 63.9] |
Under matched conditions the judge beats the signals alone. The signals are a complement: adding them fixes 38 judge errors and introduces 3 (). The gain comes from ungrounded verification, where the judge's recall is and the combination's is . In practice, rubric defects and labels often overlap such as in Wang et al 2025, Sharma et al. 2026. This has been added to the conclusion in line 527+
Changes: the abstract and §5.1 now clarify the injection detection, updated numbers and the complementarity claim. Appendices give per-mode recall, absolute-signal probes, ablations and robustness checks.
Evaluator reliability
The signals are validated against defects whose ground truth is known. They shift in the predicted direction for every mode, scale with corruption dose, and replicate under an independent Claude/Gemini panel (hackable under GPT-5.5 vs. under the panel). We state the remaining gap, item-level agreement with human experts, as a limitation and as future work.
Dear Reviewer,
We have invested considerable effort in preparing the rebuttals are we are excited by the improvements. Your suggestions have made our claims considerably stronger. We would love for you to consider increasing your score for our work.
If you have further questions or concerns, please raise them promptly. We would like to make the most of the discussion period through active and constructive communication.
Even if you have no additional comments, a brief acknowledgment would be greatly appreciated.
Thank you so much for your time!
Thank you for acknowledging our interesting methods to rubric understanding, improvement, and our controlled interventions and findings of structural problems with rubrics. We also thank you for your careful reading and comments on our paper.
You mentioned that the most substantive effect on your score and comment is a lack of explanation and theoretical grounding, and improving accessibility for ML readers, along with definitions, examples, and changes added to the paper. These now have a concrete fix in the paper: a motivating sentence, the theory appendix with the signal table and worked example, and a dedicated setup paragraph. The new experiments also sharpen the contribution. The signals measure rubric validity in a way that catches what a frontier LLM judge misses, raising its accuracy from 89.2% to 96.5% with statistical significance. This is a concrete, ML-facing use of a measurement-theory bridge that you described as "meaningful".
A. Expanding upon the quality signals
What we added
Background for ML readers (new Appendix). The appendix has two short subsections. In a camera ready version we will add and make these richer.
Signal table (new Table). For each of the 17 signals, the table gives its definition and aggregation rule, its range, its thresholds where applicable, its healthy direction, and the injected modes that move it. It also states the caveats you asked for. Weighted relevance rises under non-atomic and redundant corruption, so higher is not uniformly better. The adjacent and background shares have no healthy direction.
Worked example (new Appendix). We trace one task rubric from end to end. Four of its eight criteria are rewritten into hackable form, e.g. "explains that readouts disagree across cohorts" becomes "includes the words 'pathway activation' and 'sequencing batches'". The example then follows the pipeline:
Why 17 signals (ablation, new Appendix). Each lens carries information the others lack: