acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation

Abstract

LLM judges increasingly evaluate responses against fine-grained rubric checklists. Evaluating all rubrics in a single pass is efficient, but we find it introduces *rubric interference*: the verdict on one rubric shifts depending on which other rubrics are co-evaluated. In a preliminary study, only one-third of samples receive fully consistent verdicts across rubric sets of varying composition. We formalize this problem and develop a measurement framework that probes interference through three controlled operations: rubric set expansion, reordering, and noise injection. To mitigate interference without external supervision, we propose **Self-Anchored Rubric Alignment (SARA)**. SARA uses a model's own single-rubric judgments as stable anchors and aligns multi-rubric outputs to these anchors through on-policy self-distillation. We validate SARA on three datasets (HealthBench, FLASK, ResearchQA) across Qwen3 and Llama-3.1 model families. SARA improves rubric-level consistency by up to 16% and triples sample-level exact match, while preserving agreement with both the base model and GPT-4.1. We further show that judge consistency directly affects downstream reinforcement learning: a SARA-aligned reward model reduces inter-run reward variance by 5x under rubric-order perturbation, confirming that consistent rubric evaluation is a practical requirement for stable policy optimization. Our code is available at https://anonymous.4open.science/r/SARA-22E2/

Primary area:
foundation or frontier models, including LLMs
Keywords:
evaluation methodologies, metrics, distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ExLCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies rubric interference in LLM judges: a judgment on a fixed response and rubric can change when other rubrics are added, reordered, or replaced. It introduces a measurement framework covering rubric-set expansion, reordering, and irrelevant-rubric injection. Its proposed method, Self-Anchored Rubric Alignment (SARA), uses on-policy self-distillation to align joint-evaluation analysis tokens with an EMA teacher’s distributions under single-rubric contexts, alongside a preservation loss for verdict and formatting tokens. Experiments on HealthBench, ResearchQA, and FLASK across four models report generally improved consistency. Additional experiments examine cross-dataset transfer and reduced sensitivity of downstream RL training to rubric order.

Critical strengths

A meaningful and clearly testable reliability problem. Changes in unrelated evaluation context should not arbitrarily alter a judgment on a fixed criterion. The paper makes this concern concrete through controlled expansion, reordering, and noise-injection tests. This measurement framework is useful for assessing whether joint rubric evaluation can reliably replace separate evaluations.

Evidence of consistency improvements across several settings. The experiments cover three datasets, multiple model sizes, and two model families, with additional cross-dataset transfer results. The downstream RL experiment also connects judge consistency to sensitivity under rubric-order perturbations. Together, these results provide substantive support for the method’s ability to improve behavioral stability.

Critical weaknesses

Preservation of judgment quality is insufficiently established. The main metrics measure consistency between evaluation modes rather than correctness against human judgments. Agreement with the original model measures behavioral preservation, while agreement with GPT-4.1 provides only an automated proxy for quality. Moreover, Table 4 reports a decline in Qwen3-8B’s joint-mode agreement with GPT-4.1 from 0.773 to 0.731 on HealthBench. Since the supervision comes from the model’s own single-rubric behavior, the method may preserve or reinforce existing errors. The paper does not establish that this occurs, but its current evaluation does not adequately exclude this possibility, limiting support for the claim that consistency improves without quality degradation.

The incremental benefit over simple self-training is not sufficiently demonstrated. In Table 3, SFTiso slightly exceeds SARA in rubric-level agreement and sample-level exact match, while SARA performs better in κ and agreement with the original model. Greater preservation of the original model’s behavior does not establish better judgment quality. This central comparison is also limited to one model–dataset setting, and the description does not clearly establish whether SFTiso receives equivalent rubric-order and noise augmentation. Consequently, the evidence does not sufficiently isolate the value of SARA’s additional mechanisms from simpler training alternatives.

Questions for authors

Could you compare the base model, SFTiso, and SARA against human judgments, particularly on cases where training changes the verdict? A breakdown of corrected judgments versus newly introduced errors would help assess possible error reinforcement. How should the Qwen3-8B decline in Table 4 be interpreted?

Does SARA retain an advantage over SFTiso under matched data augmentation and comparable training budgets? Could this comparison be extended beyond Qwen3-14B on ResearchQA, with both judgment-quality and computational-cost measurements? Which ablations isolate the effects of the distillation mechanism from rubric shuffling and noise injection?

Could you more explicitly distinguish the contribution from existing on-policy self-distillation and prior observations of multi-criteria instruction interference, including Self-Distilled Reasoner and When Gradients Collide? In particular, which findings establish the necessity of the proposed segment-wise alignment and preservation design?

1 comment
AuthorsOpenReview ↗

A kind reminder

Dear Reviewer,

As the rebuttal deadline approaches, we would appreciate any follow-up feedback on whether our responses— particularly the boundary condition discussion for Qwen3-8B and the novelty distinction from prior work— address your concerns. We would be happy to provide further clarification if needed.

Thank you for your time and consideration.

Reviewer_xvRoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The authors identify rubric interference, where a judge’s output on the same response and rubric changes depending on which other rubrics are evaluated together or their order
  • They introduce SARA, an OPSD method that trains a rubric judge using an EMA teacher evaluating each rubric in isolation
  • Experiments on HealthBench, FLASK, and ResearchQA show improved isolation–joint consistency in most settings and reduced sensitivity to rubric reordering and irrelevant rubrics
  • Authors also ran downstream GRPO experiments which shows approximately 5× lower reward standard deviation across rubric-order variants, while the trained policies achieve similar average evaluation scores.
Critical strengths
  • The authors identify a practically relevant reliability problem in rubric-based judging: evaluating criteria together can change individual verdicts. Their controlled experiments characterize this sensitivity to rubric composition, ordering, and irrelevant criteria pretty well.

  • The SARA method is intuitive and practical: align joint-rubric evaluation with the model’s own isolated-rubric reasoning through on-policy self-distillation. It requires neither external labels nor a stronger teacher, making it potentially applicable to training more capable judges.

Critical weaknesses
  • The results do not sufficiently establish SARA’s advantage over simple SFT: the SFT baseline achieves similar improvements in judge consistency. Any comparison of judgment quality and training cost would improvements would help clarify why SARA is preferable.

  • The downstream RL results show reduced reward variability, but no improvement in the trained policy’s average quality. Evidence of better policy quality or faster convergence, especially against the SFT-trained judge as baseline, would strengthen the practical motivation

Questions for authors

As commented in the weakness section - I'm wondering if there's any additional evidence suggesting the SARA is superior than then SFT baseline, in terms of both judge behavior and downstream RL training?

2 comments
AuthorsOpenReview ↗

A kind reminder

Dear Reviewer,

As the rebuttal deadline approaches, we would appreciate any follow-up feedback on whether the RL experiment sufficiently demonstrates SARA's advantage over SFT_iso. We would be happy to provide further clarification if needed.

Thank you for your time and consideration.

Thanks author for the clarification and I agree that the evidence in the responses stregnthened the results. However, I'd like to maintain my original score based on the overall contribution of this paper.

Reviewer_ek8pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies rubric interference in LLM judges: a model’s judgment on a fixed response and criterion can change depending on which other criteria are evaluated alongside it. It proposes SARA, an on-policy self-distillation method that uses isolation-mode evaluations to guide joint evaluation. Experiments across three datasets and two model families show improved isolation–joint consistency in most settings. A downstream RL experiment also finds reduced sensitivity to rubric ordering when SARA is used as the reward model.

Critical strengths

The paper identifies a concrete reliability issue in rubric-based evaluation and provides a clear framework for testing it through rubric expansion, reordering, and noise injection. This framework is useful independently of the proposed method. SARA improves isolation–joint consistency across most tested settings, covering multiple model sizes, two model families, and three datasets without external labels. The cross-dataset transfer results provide further evidence that the gains are not entirely dataset-specific.

Critical weaknesses
  1. Improved consistency does not necessarily imply better judgment. Both isolation and joint predictions can change during SARA training, so higher agreement does not tell us whether joint evaluation becomes more reliable or whether both modes converge toward the same mistakes. Section 4.3 offers some reassurance, but agreement with the base model is not a measure of correctness, and agreement with GPT-4.1 is mixex, for example, it drops from .773 to .731 for Qwen3-8B in joint mode. The claim of preserving judgment quality therefore needs stronger evidence. Besides, the paper treats isolation-mode judgments as the preferred reference, but does not establish when this preference is justified. Co-present rubrics may provide useful context or clarify otherwise underspecified criteria, so a change in judgment need not indicate harmful interference. Aligning joint judgments with isolation judgments could therefore remove useful contextual information as well as unwanted bias. The current consistency metrics do not distinguish these possibilities.
  2. The benefit over simpler training remains unclear. In Table 3, SFT_iso slightly outperforms SARA on agreement and exact match, although SARA better preserves the base model’s predictions. It is unclear how valuable this preservation is without an independent quality measure. The experiments also do not separate the effect of the proposed distillation from rubric shuffling and noise injection, and the RL comparison omits SFT_iso. These gaps make it difficult to assess whether SARA’s additional complexity is justified.
Questions for authors
  1. Using the frozen base model’s isolation predictions as a reference, how much of the consistency gain comes from changes in joint predictions versus changes in isolation predictions?
  2. On a small, independently adjudicated sample of changed verdicts, does SARA correct more errors than it introduces? This would help resolve the concern about judgment quality.
  3. Does SFT_iso receive the same shuffling and noise augmentation as SARA? If given the same augmentations, does SARA retain a meaningful advantage?
  4. How does an SFT_iso judge perform in the RL stability experiment? Comparable gains would weaken the case for SARA, while a clear advantage would strengthen it.
  5. Among cases where isolation and joint judgments disagree, can the authors independently assess which judgment is better supported? In particular, does the answer differ between self-contained rubrics and rubrics that benefit from shared context? This would help establish when alignment toward isolation is warranted.
1 comment
AuthorsOpenReview ↗

A kind reminder

Dear Reviewer,

As the rebuttal deadline approaches, we would appreciate any follow-up feedback on whether our responses (particularly the verdict-change decomposition and the three-way RL comparison) address your concerns. We would be happy to provide further clarification if needed.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.