acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Rubric Generation with Interventional Transfer

Abstract

Instance-specific rubrics are common in AI benchmarks where reliable evaluation requires specific expert knowledge. This approach is difficult to scale, prompting research into the generation of rubrics with large language models (LLMs). However, even when expert rubrics are available as references, it is unclear how to productively evaluate the quality of generated rubrics at scale. In this paper, we introduce a method for the evaluation of rubric generation, which we call Interventional Transfer (IT), based on the idea that two rubrics are similar if they move together when a response is perturbed to pass/fail one of them. In contrast to existing approaches for evaluation of rubric generation, we argue that different forms of interventional transfer can be used to evaluate the utility of generated rubrics for different tasks. For instance, we apply this approach in a case study on HealthBench, where we demonstrate an asymmetry in rubrics generated by Qwen3.8-27B, Deepseek-V4-Flash, and Opus-5, used to evaluate responses from GPT-5.6-Terra. Perturbations that degrade responses according to the generated/expert rubric transfer into lower scores on the corresponding expert/generated rubric, but perturbations that improve on one rubric do not reliably transfer into higher scores on the other. We argue that this finding has implications for the usage of LLM-generated rubrics for performance monitoring and hill-climbing. We contrast our approach with existing approaches for rubric evaluation, which do not surface the same asymmetry that we observe.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM evaluation, rubric generation, LLM-as-a-judge, rubric evaluation, instance-specific rubrics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YLAsOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Interventional Transfer (IT) to evaluate automatically generated rubrics against expert rubrics. It edits a response to satisfy or violate one rubric and measures changes under the other. Experiments on 105 HealthBench examples find that degradation transfers more strongly than improvement. The authors use this asymmetry to distinguish the usefulness of generated rubrics for guiding optimization and detecting degradation.

Critical strengths

None

Critical weaknesses
  1. I find the methodological novelty limited. Prior work already uses generated checklists to revise responses, and controlled response perturbations to evaluate rubric sensitivity. The main addition here is to organize these operations into four directional experiments with normalized score changes. This is a reasonable evaluation protocol, but the conceptual and technical advance appears incremental. The paper also does not show that the proposed metrics predict downstream optimization or monitoring performance better than existing metrics, which would provide stronger evidence for their added value.

  2. The main experimental finding is difficult to separate from the intervention procedure. The framework assumes that edits make the target criteria pass or fail, but I could not find reported attainment rates. Low transfer may therefore reflect unsuccessful editing as well as differences between rubrics. In addition, Table A.9 shows that degradation interventions make much larger changes than improvement interventions: whole-rubric degradation shortens responses to 42–62% of their original length. Removing substantial content could make both rubrics lose passing criteria regardless of their detailed alignment. The single-criterion experiments help, but negative edits are still larger. Without verifying intervention success and controlling the amount of editing, the interpretation of the pass/fail asymmetry remains uncertain.

Questions for authors

None

3 comments

Thank you for the feedback.

Weakness 1: Limited novelty: other work uses rubrics to revise responses

We disagree with the claim that there is limited novelty here. We are aware of (and cite) prior work that incorporates some of the same ingredients that we use, but our goals, methods, and conclusions have novelty. Much prior work uses generated rubrics to improve responses in various ways; some works improve and evaluate rubric generation using methods we cover in Section 4.3 and Appendix B, but we propose a different method for evaluating rubrics. Our IT method, and its extension into single-criterion interventions, leads us to new findings: the pass/fail asymmetry, the relative redundancy of expert rubrics, and task-specific generator strengths.

To illustrate how our work differs from prior work that edits responses using checklists, we consider STICK (Cook et al., 2024). In their paper, an LLM responds and generates a checklist, answers each checklist question for its own response, and then rewrites the whole response to change the NO answers to YES. In contrast to our optimization task, they edit with respect to a checklist generated by the response model because their goal is to get a better response. Our edits are with respect to the models under evaluation because our goal is to evaluate rubric generators. Our papers are complementary, but different.

To illustrate how our work differs from prior work that evaluates rubrics using perturbations, we consider "Retrieval-Augmented Agentic Rubric Generation" (Chen et al., 2026). They evaluate rubrics by whether they prefer a response to one that was intentionally degraded with an error from a set of generic clinical errors (e.g., omitting a red-flag warning), not with an instance-specific criterion. The strength of our method for degrading responses is that we change the response in ways that a human cares about only for that response. The two methods are different.

To our knowledge, no prior work considers single-criterion interventions, transfer of interventions on the expert rubric to the generated rubric, or the pass/fail asymmetry. We would welcome any references the reviewer has in mind.

Weakness 2: Table A.9: edit to fail is larger, so effects might be due to edit size

In Table A.9, for single-criterion edits, fail edits aren't uniformly larger than pass edits. Fail edits reduce response length by 8%, and pass edits increase response length by 6-15%.

We conducted a further edit-size-controlled analysis to investigate the concern you raised, and saw that the asymmetry persists even under edit-size-controlled evaluation. In the table below, we measure edit size as the word-level Levenshtein distance d between the original and the edited responses (substitutions + insertions + deletions). The table shows the fraction of single-criterion edits that move at least one criterion of the other rubric (relevance), in each bin of edit size d:

Interventions on cgc^g (relevance: whether a c∗c^{\ast} moved):

dd[0,8][8,14][14,21][21,33][33,63]63+
pass0.120.150.180.190.210.05
fail0.340.380.470.450.570.66

Interventions on c∗c^{\ast} (relevance: whether a cgc^g moved):

dd[0,9][9,14][14,20][20,30][30,56]56+
pass0.100.160.180.180.200.23
fail0.330.420.540.550.630.76

Fail edits transfer more than pass edits when controlling for edit size, and larger edits mostly transfer more for both edit types. When we remove the largest fail edits (8-11%) until the mean edit sizes are equal, the gap between fail and pass edits remains: the relevance gap between fail and pass edits becomes 0.29 [0.21, 0.35] for interventions on cgc^g and 0.36 [0.31, 0.41] for interventions on c∗c^{\ast}. With all edits, it is 0.31 and 0.38. Note that whole-rubric fail edits are larger (mean d 224-289 words, against 69-96), partly because they target more criteria (6.4-9.0, against 3.0-4.0).

Edit size does not explain the asymmetry.

Other Weaknesses

On downstream performance (Weakness 1 last sentence), please see our response to reviewer nQj2, "Questions 1 and 2": criterion-level IT led to a change in rubric generation that improved revision-based optimization on HealthBench and PRBench.

On attainment rates (Weakness 2), please see our response to reviewer o1SW, "Weakness 1: Unreported flip success rate of the intervention (henceforth 'fliprates'). Question 1: What are the fliprates?" Whole-rubric interventions move 98% of targeted criteria to pass and 93% to fail. Interventions to fail succeed less often but transfer more. So failed edits do not underlie the asymmetry.

Thank you for the clarifications and additional analyses, which have substantially alleviated my concerns. My only remaining concern is about novelty, primarily the extent of the methodological advance, which goes beyond the question of whether an identical experimental setup has appeared in prior work. Taking the additional experiments and the other reviewers’ comments into account, I am raising my score to Weak Reject.

Thank you for raising your score.

We would characterize the novelty of our work differently from it being not identical to existing experimental setups. We are solving a different problem from prior work: using rubrics to evaluate rubrics.

The single-criterion interventions are a methodological novelty with direct applications to improving rubric generation. In Figure 6 of the paper, we show that expert rubrics are more redundant than generated rubrics, with implications for creating less redundant rubrics. In the discussion on page 8 around table A.6, we argue that the asymmetry uncovered by our method is not attainable with existing methods. In our response to nQj2 we show that criteria level interventions have implications for optimization unlikely to be uncovered using existing work. We fail to see how one could use, for example, the `generic’ perturbations of (Chen et al. 2026) to show that initially-failing generated criteria are harmful at the same rate as being helpful, but have positive marginal utility nevertheless.

Our method is useful in ways other works are not. We view this as novelty. It is difficult to dispute questions of novelty without references / citations to the papers that you feel make our contribution not novel.  If you have such references in mind, we would appreciate you sharing them, so we can incorporate those into the paper and better position against existing work.

Reviewer_nQj2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose Interventional Transfer (IT), a novel metric for evaluating the quality of rubrics by measuring how scores change when a response is perturbed. They apply this approach to a case study on HealthBench and show that degradations can transfer to lower scores on pairs of rubrics but improvements do not reliably transfer into higher scores on pairs of rubrics.

Critical strengths
  • Clear presentation.
  • A novel metric for evaluating rubric quality given expert crafted that have been established as trusted.
  • Interesting findings. Specifically,
    1. asymmetry in the sensitivity of changes to scores from rubrics.
    2. absolute quality is higher for all rubrics on tasks that intervene to fail than on tasks that intervene to pass. This is connected to similarites between existing passing criteria.
  • The CRITERIA-LEVEL INTERVENTIONS AND MEASURES findings based on categorizing criteria into categories is a nice strategy to get actionable insights to feed back into improving the rubric generator.
Critical weaknesses
  • The proposed metric is dependent on having expert rubrics that can be trusted as high quality. Human-crafted rubrics may themselves have issues causing poor downstream outcomes.
  • The proposed metric is dependent on the model used for rubric-based evaluation.
  • The connection to using this metric to improve rubric generation is missing or not clear.
Questions for authors
  • How do the authors suggest using the proposed metric? what are the actionable insights you can use to improve rubric generation?
Reviewer_o1SWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Interventional Transfer (IT), a framework for evaluating the quality of automatically generated rubrics by editing responses to satisfy or violate one rubric and observing the resulting changes in scores under another rubric. The authors combine intervention directions and target rubrics into four settings, corresponding to applications such as optimization targets, improvement detection, and degradation monitoring. In a case study on HealthBench, generated and expert rubrics exhibit stronger transfer under degradation interventions than under improvement interventions.

Critical strengths
  1. The research question is clearly defined and practically relevant.
  2. The four intervention settings are interpretable and have clear connections to downstream applications.
Critical weaknesses
  1. The effects of intervention effectiveness and rubric quality are not sufficiently disentangled. IT scores depend on both rubric content and the editing model’s capabilities. Although the paper provides editing prompts and analyzes the extent of modifications, it does not systematically report the success rates of flipping targeted criteria. Consequently, the sources of low transfer scores remain unclear.
  2. The generalizability of the findings is limited. The main analysis uses 105 questions from HealthBench and excludes the Hard subset. Although the appendix extends the analysis to additional response models, the applicability of the findings to other domains and more challenging questions remains to be established.
Questions for authors
  1. What are the target criterion flip success rates for each of the four intervention settings? Does the main asymmetry persist after accounting for intervention success rates?
  2. Are the rubric generator rankings and main conclusions stable when different editing and scoring models are used?

Comments on the paper

3 comments

The rebuttal substantially strengthens the paper and mitigates several of my concerns. However, the robustness of the conclusions to the choice of intervention editor remains untested, so I would retain my Weak Reject rating.

Thank you for the follow-up. We ran the Figure 5 pipeline with GLM-5.3-Flash as the editor in place of GPT-5.6-terra, but still on GPT-5.6-terra responses. The asymmetry also holds under both editors: interventions to fail transfer more than interventions to pass. The other Figure 5 results are robust to GLM doing the edits:

  1. Training target.
    • Values: The IT scores are about the same. Each generator's score changes by at most 0.03, and Opus-5 is the top generator with both editors (0.11).
    • Significance: Opus-5 is still significantly above Qwen3.8-27B (0.11 with Terra, 0.09 with GLM). With GLM, Opus-5 is now also significantly above DeepSeek-V4-Flash. This is not a new conclusion: the point estimate is positive with both editors (0.06 with Terra, 0.08 with GLM), and the lower end of the GLM interval is 0.00.
  2. Detect good responses.
    • Values: The IT scores are about the same. All three are low with both editors (0.06 to 0.15 with Terra, 0.05 to 0.09 with GLM).
    • Significance: No difference between generators is significant with either editor. All three paired differences have the same sign with both editors.
  3. Detect bad responses.
    • Values: All three generators score 0.05 to 0.07 lower with GLM. DeepSeek-V4-Flash is the top generator with both editors.
    • Significance: No change. Both significant differences stay, with almost the same values (DeepSeek − Qwen: 0.10 vs 0.12; Opus − DeepSeek: −0.10 vs −0.10).
  4. Degradation flag.
    • Values: The IT scores are about the same. Each generator's score changes by at most 0.04, and Opus-5 is the top generator with both editors.
    • Significance: With Terra, Opus-5 is significantly above both other generators. With GLM, these two differences are no longer significant. The point estimates change by only 0.01 (Opus − Qwen: 0.03 vs 0.02; Opus − DeepSeek: 0.06 vs 0.05) and keep their sign. The loss of significance comes from wider GLM intervals, not from smaller differences.

IT score Q, by edit model

TaskGeneratorGPT-5.6-terra editsGLM-5.3-Flash edits
Training targetQwen3.8-27B-0.00 [-0.06, 0.04]0.03 [-0.04, 0.08]
DeepSeek-V4-Flash0.05 [-0.01, 0.11]0.03 [-0.04, 0.10]
Opus-50.11 [0.02, 0.17]0.11 [0.02, 0.20]
Detect good responsesQwen3.8-27B0.15 [0.04, 0.25]0.09 [-0.08, 0.22]
DeepSeek-V4-Flash0.11 [-0.05, 0.23]0.05 [-0.14, 0.19]
Opus-50.06 [-0.01, 0.13]0.05 [-0.05, 0.11]
Detect bad responsesQwen3.8-27B0.46 [0.41, 0.52]0.39 [0.34, 0.44]
DeepSeek-V4-Flash0.56 [0.50, 0.62]0.51 [0.45, 0.56]
Opus-50.46 [0.41, 0.51]0.41 [0.36, 0.46]
Degradation flagQwen3.8-27B0.61 [0.56, 0.66]0.59 [0.54, 0.64]
DeepSeek-V4-Flash0.58 [0.54, 0.62]0.56 [0.52, 0.61]
Opus-50.65 [0.61, 0.69]0.61 [0.56, 0.66]

Paired differences between generators (* = interval excludes 0, i.e. a bracket in Figure 5)

TaskDifferenceGPT-5.6-terra editsGLM-5.3-Flash edits
Training targetDeepSeek − Qwen0.05 [-0.02, 0.13]0.01 [-0.06, 0.07]
Opus − Qwen0.11 [0.04, 0.17] *0.09 [0.02, 0.16] *
Opus − DeepSeek0.06 [-0.02, 0.13]0.08 [0.00, 0.16] *
Detect good responsesDeepSeek − Qwen-0.03 [-0.18, 0.09]-0.04 [-0.21, 0.10]
Opus − Qwen-0.09 [-0.22, 0.04]-0.04 [-0.16, 0.09]
Opus − DeepSeek-0.05 [-0.18, 0.10]-0.01 [-0.13, 0.15]
Detect bad responsesDeepSeek − Qwen0.10 [0.05, 0.14] *0.12 [0.07, 0.16] *
Opus − Qwen-0.00 [-0.04, 0.04]0.02 [-0.02, 0.06]
Opus − DeepSeek-0.10 [-0.14, -0.06] *-0.10 [-0.13, -0.06] *
Degradation flagDeepSeek − Qwen-0.03 [-0.07, 0.01]-0.03 [-0.08, 0.03]
Opus − Qwen0.03 [0.00, 0.07] *0.02 [-0.03, 0.08]
Opus − DeepSeek0.06 [0.03, 0.10] *0.05 [0.00, 0.10]

Thank you for the additional experiments, which substantially address my remaining concern about editor dependence. While some ranking differences are no longer statistically significant, the main asymmetry remains consistent. I am therefore updating my recommendation to Weak Accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.