- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes Interventional Transfer (IT) to evaluate automatically generated rubrics against expert rubrics. It edits a response to satisfy or violate one rubric and measures changes under the other. Experiments on 105 HealthBench examples find that degradation transfers more strongly than improvement. The authors use this asymmetry to distinguish the usefulness of generated rubrics for guiding optimization and detecting degradation.
Critical strengths
None
Critical weaknesses
-
I find the methodological novelty limited. Prior work already uses generated checklists to revise responses, and controlled response perturbations to evaluate rubric sensitivity. The main addition here is to organize these operations into four directional experiments with normalized score changes. This is a reasonable evaluation protocol, but the conceptual and technical advance appears incremental. The paper also does not show that the proposed metrics predict downstream optimization or monitoring performance better than existing metrics, which would provide stronger evidence for their added value.
-
The main experimental finding is difficult to separate from the intervention procedure. The framework assumes that edits make the target criteria pass or fail, but I could not find reported attainment rates. Low transfer may therefore reflect unsuccessful editing as well as differences between rubrics. In addition, Table A.9 shows that degradation interventions make much larger changes than improvement interventions: whole-rubric degradation shortens responses to 42–62% of their original length. Removing substantial content could make both rubrics lose passing criteria regardless of their detailed alignment. The single-criterion experiments help, but negative edits are still larger. Without verifying intervention success and controlling the amount of editing, the interpretation of the pass/fail asymmetry remains uncertain.
Questions for authors
None
3 comments
Thank you for the clarifications and additional analyses, which have substantially alleviated my concerns. My only remaining concern is about novelty, primarily the extent of the methodological advance, which goes beyond the question of whether an identical experimental setup has appeared in prior work. Taking the additional experiments and the other reviewers’ comments into account, I am raising my score to Weak Reject.
Thank you for raising your score.
We would characterize the novelty of our work differently from it being not identical to existing experimental setups. We are solving a different problem from prior work: using rubrics to evaluate rubrics.
The single-criterion interventions are a methodological novelty with direct applications to improving rubric generation. In Figure 6 of the paper, we show that expert rubrics are more redundant than generated rubrics, with implications for creating less redundant rubrics. In the discussion on page 8 around table A.6, we argue that the asymmetry uncovered by our method is not attainable with existing methods. In our response to nQj2 we show that criteria level interventions have implications for optimization unlikely to be uncovered using existing work. We fail to see how one could use, for example, the `generic’ perturbations of (Chen et al. 2026) to show that initially-failing generated criteria are harmful at the same rate as being helpful, but have positive marginal utility nevertheless.
Our method is useful in ways other works are not. We view this as novelty. It is difficult to dispute questions of novelty without references / citations to the papers that you feel make our contribution not novel. If you have such references in mind, we would appreciate you sharing them, so we can incorporate those into the paper and better position against existing work.
Thank you for the feedback.
Weakness 1: Limited novelty: other work uses rubrics to revise responses
We disagree with the claim that there is limited novelty here. We are aware of (and cite) prior work that incorporates some of the same ingredients that we use, but our goals, methods, and conclusions have novelty. Much prior work uses generated rubrics to improve responses in various ways; some works improve and evaluate rubric generation using methods we cover in Section 4.3 and Appendix B, but we propose a different method for evaluating rubrics. Our IT method, and its extension into single-criterion interventions, leads us to new findings: the pass/fail asymmetry, the relative redundancy of expert rubrics, and task-specific generator strengths.
To illustrate how our work differs from prior work that edits responses using checklists, we consider STICK (Cook et al., 2024). In their paper, an LLM responds and generates a checklist, answers each checklist question for its own response, and then rewrites the whole response to change the NO answers to YES. In contrast to our optimization task, they edit with respect to a checklist generated by the response model because their goal is to get a better response. Our edits are with respect to the models under evaluation because our goal is to evaluate rubric generators. Our papers are complementary, but different.
To illustrate how our work differs from prior work that evaluates rubrics using perturbations, we consider "Retrieval-Augmented Agentic Rubric Generation" (Chen et al., 2026). They evaluate rubrics by whether they prefer a response to one that was intentionally degraded with an error from a set of generic clinical errors (e.g., omitting a red-flag warning), not with an instance-specific criterion. The strength of our method for degrading responses is that we change the response in ways that a human cares about only for that response. The two methods are different.
To our knowledge, no prior work considers single-criterion interventions, transfer of interventions on the expert rubric to the generated rubric, or the pass/fail asymmetry. We would welcome any references the reviewer has in mind.
Weakness 2: Table A.9: edit to fail is larger, so effects might be due to edit size
In Table A.9, for single-criterion edits, fail edits aren't uniformly larger than pass edits. Fail edits reduce response length by 8%, and pass edits increase response length by 6-15%.
We conducted a further edit-size-controlled analysis to investigate the concern you raised, and saw that the asymmetry persists even under edit-size-controlled evaluation. In the table below, we measure edit size as the word-level Levenshtein distance d between the original and the edited responses (substitutions + insertions + deletions). The table shows the fraction of single-criterion edits that move at least one criterion of the other rubric (relevance), in each bin of edit size d:
Interventions on cg (relevance: whether a c∗ moved):
Interventions on c∗ (relevance: whether a cg moved):
Fail edits transfer more than pass edits when controlling for edit size, and larger edits mostly transfer more for both edit types. When we remove the largest fail edits (8-11%) until the mean edit sizes are equal, the gap between fail and pass edits remains: the relevance gap between fail and pass edits becomes 0.29 [0.21, 0.35] for interventions on cg and 0.36 [0.31, 0.41] for interventions on c∗. With all edits, it is 0.31 and 0.38. Note that whole-rubric fail edits are larger (mean d 224-289 words, against 69-96), partly because they target more criteria (6.4-9.0, against 3.0-4.0).
Edit size does not explain the asymmetry.
Other Weaknesses
On downstream performance (Weakness 1 last sentence), please see our response to reviewer nQj2, "Questions 1 and 2": criterion-level IT led to a change in rubric generation that improved revision-based optimization on HealthBench and PRBench.
On attainment rates (Weakness 2), please see our response to reviewer o1SW, "Weakness 1: Unreported flip success rate of the intervention (henceforth 'fliprates'). Question 1: What are the fliprates?" Whole-rubric interventions move 98% of targeted criteria to pass and 93% to fail. Interventions to fail succeed less often but transfer more. So failed edits do not underlie the asymmetry.