- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies rubric interference in LLM judges: a judgment on a fixed response and rubric can change when other rubrics are added, reordered, or replaced. It introduces a measurement framework covering rubric-set expansion, reordering, and irrelevant-rubric injection. Its proposed method, Self-Anchored Rubric Alignment (SARA), uses on-policy self-distillation to align joint-evaluation analysis tokens with an EMA teacher’s distributions under single-rubric contexts, alongside a preservation loss for verdict and formatting tokens. Experiments on HealthBench, ResearchQA, and FLASK across four models report generally improved consistency. Additional experiments examine cross-dataset transfer and reduced sensitivity of downstream RL training to rubric order.
Critical strengths
A meaningful and clearly testable reliability problem. Changes in unrelated evaluation context should not arbitrarily alter a judgment on a fixed criterion. The paper makes this concern concrete through controlled expansion, reordering, and noise-injection tests. This measurement framework is useful for assessing whether joint rubric evaluation can reliably replace separate evaluations.
Evidence of consistency improvements across several settings. The experiments cover three datasets, multiple model sizes, and two model families, with additional cross-dataset transfer results. The downstream RL experiment also connects judge consistency to sensitivity under rubric-order perturbations. Together, these results provide substantive support for the method’s ability to improve behavioral stability.
Critical weaknesses
Preservation of judgment quality is insufficiently established. The main metrics measure consistency between evaluation modes rather than correctness against human judgments. Agreement with the original model measures behavioral preservation, while agreement with GPT-4.1 provides only an automated proxy for quality. Moreover, Table 4 reports a decline in Qwen3-8B’s joint-mode agreement with GPT-4.1 from 0.773 to 0.731 on HealthBench. Since the supervision comes from the model’s own single-rubric behavior, the method may preserve or reinforce existing errors. The paper does not establish that this occurs, but its current evaluation does not adequately exclude this possibility, limiting support for the claim that consistency improves without quality degradation.
The incremental benefit over simple self-training is not sufficiently demonstrated. In Table 3, SFTiso slightly exceeds SARA in rubric-level agreement and sample-level exact match, while SARA performs better in κ and agreement with the original model. Greater preservation of the original model’s behavior does not establish better judgment quality. This central comparison is also limited to one model–dataset setting, and the description does not clearly establish whether SFTiso receives equivalent rubric-order and noise augmentation. Consequently, the evidence does not sufficiently isolate the value of SARA’s additional mechanisms from simpler training alternatives.
Questions for authors
Could you compare the base model, SFTiso, and SARA against human judgments, particularly on cases where training changes the verdict? A breakdown of corrected judgments versus newly introduced errors would help assess possible error reinforcement. How should the Qwen3-8B decline in Table 4 be interpreted?
Does SARA retain an advantage over SFTiso under matched data augmentation and comparable training budgets? Could this comparison be extended beyond Qwen3-14B on ResearchQA, with both judgment-quality and computational-cost measurements? Which ablations isolate the effects of the distillation mechanism from rubric shuffling and noise injection?
Could you more explicitly distinguish the contribution from existing on-policy self-distillation and prior observations of multi-criteria instruction interference, including Self-Distilled Reasoner and When Gradients Collide? In particular, which findings establish the necessity of the proposed segment-wise alignment and preservation design?
A kind reminder
Dear Reviewer,
As the rebuttal deadline approaches, we would appreciate any follow-up feedback on whether our responses— particularly the boundary condition discussion for Qwen3-8B and the novelty distinction from prior work— address your concerns. We would be happy to provide further clarification if needed.
Thank you for your time and consideration.