acceptodds
Under review as a conference paper at ICLR 2027

Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence

Abstract

Rubric evolution offers a promising approach to improving the quality of rubrics generated by large language models (LLMs). Central to this process is rubric comparison, which identifies the better of two rubrics and guides the direction of evolution. However, accurate rubric comparison is difficult, which presents two challenges. It should reflect downstream task performance, which is essential for assessing rubric utility but often prohibitively expensive to evaluate. It should discourage unnecessary criteria, which increase verification costs and may dilute the influence of essential criteria. To address these challenges, we introduce Rubrics on Trial, a multi-agent framework that evolves rubrics by comparing synthetic response pairs. To address the first challenge, the framework compares synthetic responses that satisfy the respective rubrics, providing a proxy for downstream performance without training a separate policy for each rubric. To address the second challenge, it assesses the necessity of a candidate criterion by independently generating high-quality alternative responses that violate it and comparing them with edited versions that satisfy it. A rubric is favored when it improves response quality in both comparisons, and the resulting comparison signal is further incorporated for rubric evolution. Extensive experiments demonstrate that Rubrics on Trial improves the quality of generated rubrics and leads to better downstream task performance. Code is available at https://anonymous.4open.science/r/Trial-9464.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Model, LLM based rubric generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QHwZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a multi-agent framework called “Rubrics on Trial,” which evolves scoring criteria for specific queries by comparing pairs of synthetic responses—including alternative responses designed to identify unnecessary evaluation criteria. Experimental results show that, compared to several existing scoring criteria generation methods, this approach achieves improvements in both preference evaluation based on scoring criteria and downstream RLHF performance.

Critical strengths
  1. The proposed synthetic response comparison provides an interesting approach to evaluating rubric quality without repeatedly training separate policies.

  2. The experiments cover multiple preference benchmarks and downstream RLHF tasks. The ablation studies demonstrate the contributions of iterative evolution and both synthetic response pairs, with consistent improvements over several baselines.

Critical weaknesses
  1. Downstream RLHF performance remains relatively weak in several settings (e.g., 20.56% on AlpacaEval 2.0 LC for Qwen2.5-3B). The paper also lacks comparisons with alternative preference optimization approaches such as DPO and SimPO, as well as recent self-improvement methods, making it difficult to assess the broader effectiveness of the proposed framework.

  2. The method largely extends existing iterative rubric evolution approaches (e.g., RRD and ChaseTail) by incorporating synthetic response comparisons and additional criterion filtering. While these modifications improve performance, the underlying framework remains similar to prior work, and the paper does not sufficiently demonstrate a substantial methodological advance beyond existing rubric refinement techniques. For me, it's a lack of novelty.

Questions for authors

Why does the paper only compare downstream RLHF performance against rubric-generation methods? Could the authors include comparisons with DPO, SimPO, or recent self-improvement approaches under comparable data and computational budgets to better establish the practical effectiveness of the proposed framework?

Reviewer_whEgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the rubric comparison problem: identifying the better of two rubrics and guides the direction of evolution.

It identifies two challenges: 1) it should reflect downstream task performance, 2) it should discourage unnecessary criteria.

To address it, it introduces Rubrics on Trial, a multi-agent framework that compares synthetic responses that satisfy the respective rubrics and assesses the necessity of a candidate criterion by independently generating high-quality alternative responses that violate it and comparing them with edited versions that satisfy it. A rubric is favored when it improves response quality in both comparisons, and the resulting comparison signal is further incorporated for rubric evolution.

Critical strengths
  • The experiments are comprehensive, covering rubric comparison accuracy, preference accuracy on reward-model benchmarks, and downstream RLHF performance.
  • The ablation studies are thorough and isolate the contribution of each key component.
Critical weaknesses
  • The two challenges identified in the paper, reflecting downstream performance and discouraging unnecessary criteria, are already well recognized in the rubric-based evaluation literature.
  • The technical novelty is limited. The framework design, which compares rubrics via responses that satisfy or violate a candidate criterion, is fairly natural, and the findings are not particularly surprising.
  • The paper lacks a thorough cost analysis, in particular a cost comparison with baseline methods.
Questions for authors
  • Do the baseline methods also have access to the reference answers? If the baselines generate rubrics from the query alone, the comparison may be unfair.
  • How does the method compare with directly using the same LLM as a comparative judge, without rubrics?
Reviewer_KwnsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method to iteratively edit the rubric’s text; it uses existing LLMs to generate and compare evidence for each edit. The main idea is to test each proposed criterion using two response pairs. The evaluation on broad domains and benchmarks demonstrates the superior performance of the method.

Critical strengths
  1. Broad empirical coverage. The paper evaluates rubric-based response ranking across knowledge, reasoning, mathematics, coding, chat, and safety. It also evaluates downstream policy improvement on general instruction-following benchmarks and transfer to healthcare.

  2. Evidence of generalization and practical usefulness. The gains persist across different generation models and final verifiers, and the resulting rubrics improve several smaller downstream policies. The paper also reports how accuracy, token consumption, and active request time change with the evolution budget, making the computational trade-off visible.

  3. Overall, the method is interesting and natural to follow.

Critical weaknesses

The method largely relies on a powerful LLM, deepseek-v4-flash here, to propose and iteratively improve the rubrics.

This should limit the capability to scale the proposer itself, for example, to scale deepseek-flash to get even better performance.

Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.