acceptodds
Under review as a conference paper at ICLR 2027

RubricLearner: Learning Evaluation Rubrics through Diagnosis-Guided Refinement

Abstract

As language models take on open-ended work such as deep research and office assistance, response quality hinges on many purpose-dependent requirements rather than a single correct answer, so evaluation depends on rubrics that state what a response must satisfy and how shortcomings affect its assessment. A rubric is nevertheless an empirical artifact: drafted criteria can miss the distinctions a task actually requires, and piecemeal score-driven edits risk discarding still-valid guidance while overfitting to the errors they fix. We introduce RubricLearner, a framework that improves evaluation rubrics through a diagnosis-guided refinement loop. Each iteration contrasts erroneous scoring justifications with reference judgments and distills conditional rules specifying when a lesson applies, what to attend to, and which mistake to avoid. Accepted rules accumulate, and the complete rubric is regenerated from them; a regenerated rubric is retained only when it improves validation performance. Because each rule states when it applies and accepted rules are retained rather than overwritten, the accumulated guidance stays specific to its context and grows across iterations. With Qwen3.7-Max as the judge, RubricLearner outperforms a state-of-the-art baseline across eight public benchmarks, an average relative gain of 3.51%. Beyond these scores, RubricLearner produces higher-quality rubrics with fewer defective criteria, and the improvement extends to unseen examples.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Rubrics, Evaluation, Diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8trVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RubricLearner, a method for learning evaluation rubrics through diagnosis-guided refinement. This method keeps the judge fixed, transforms scoring errors into conditional rules containing applicability conditions, positive principles, and negative lessons, retains accepted rules in an append-only manner, and regenerates complete rubrics based on these rules. Candidate rubrics are only accepted if they achieve better results on independent validation sets. The authors evaluated the rubrics on eight benchmarks, four judgment formats, and three judges, and reported supplementary analyses including criterion defect, held-out performance, and diagnosis budget. The main results show that, at Qwen3.7-Max, RubricLearner achieves an average task-wise relative improvement of 3.51% compared to Learnable Skills, with a lower average defect rate.

Critical strengths

1.The method is clearly designed and auditable. RubricLearner represents error diagnosis as rules with applicable conditions and separates the accepted rule history from the final rubric. This helps limit the error generalization of rules while preserving the basis for modifications and version history, which is valuable for practical evaluation workflows.

2.The experiments are comprehensive. The paper covers eight benchmarks, four judgment formats, and multiple judges, and analyzes the method from the perspectives of accuracy, criterion defect, held-out behavior, and diagnosis budget, rather than just reporting a single aggregate score.

Critical weaknesses

1.The main performance comparisons lack a unified matched evaluation protocol. Appendix B.1 explicitly states that each method in Table 1 uses its own learning and inference protocols, and some methods also use different data sizes, temperatures, and voting settings; furthermore, the main table evaluates the full dataset, not a unified held-out set. Therefore, the 3.51% relative improvement emphasized in the paper cannot be considered a direct comparison under strictly controlled conditions, which weakens the main empirical claim.

2.Evidence for the core mechanisms and generalization conclusions is insufficient. The paper does not separately ablate the conditional rule structure, append-only retention, complete-rubric regeneration, and validation gate, thus making it impossible to determine which design contributes to the performance improvement. On the common HelpSteer3 held-out set, RubricLearner and Eval-Skill correctly identified 316 and 315 samples, respectively (p=1.0); their development score definitions and inference protocols also differ. Therefore, a smaller development-to-held-out gap does not sufficiently demonstrate stronger generalization ability.

Questions for authors
  1. Could the authors re-evaluate the main method under the same held-out split, judge, decoding configuration, learning budget, and inference protocol, and evaluate the stability of the results using multiple random seeds, confidence intervals, or significance tests?

  2. Could the authors perform component ablation on conditional structure, append-only rules, complete-rubric regeneration, and validation-based selection respectively? Furthermore, could you clarify whether Algorithm 1, after resetting RbestR_{\mathrm{best}} and mbestm_{\mathrm{best}}, might accept candidates weaker than those incumbent in the previous epoch, and whether the final output is the globally best rubric across all epochs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8trV

Thank you for your careful review, your recognition of the method's inspectable design, and your constructive suggestions for the empirical analysis.

For Weakness 1 / Question 1 and Weakness 2: Matched settings, unseen performance, and uncertainty

Thank you for raising this concern. We would like to clarify that Table 1 is a fair comparison: all learning-based methods use the same learning examples, and all methods are evaluated on the same full datasets with the same judge, temperature, and voting configuration within each judge block. The blocks use Qwen3-8B, Qwen3.7-Max, and DeepSeek-V4-Flash. RubricLearner uses 100 diagnosis examples and a separate 100-example validation set, without updating judge parameters. The inclusion of learning examples in the full-dataset evaluation applies equally to the learning-based methods; RubricLearner receives no additional learning-data access. Method-specific protocols refer to internal learning procedures, not different learning data or judge-side evaluation settings. We will clarify this in our future version to avoid misunderstanding.

The 3.51% gain concerns the shared full-dataset evaluation. To complement these full-dataset results, and following your suggestion, we add a supplementary unseen-only evaluation on PPE-IFEval and RewardBench2 under Qwen3.7-Max. We reuse the final selected rubrics without further learning or selection, evaluating all methods on the same subsets that exclude every example exposed during learning, generation context, or validation across all epochs.

MethodPPE-IFEval full (Acc.)PPE-IFEval unseen (Acc.)RewardBench2 full (Acc.)RewardBench2 unseen (Acc.)
Learnable Assessment Skills0.8860.8810.8590.854
Eval-Skill0.8850.8790.8800.875
RubricLearner0.9030.8970.8930.887

Regarding the HelpSteer3 analysis, we would like to clarify that cumulative evolution describes Eval-Skill's learning procedure, whereas both methods' reported scores evaluate their final selected rubrics. Development accuracy measures performance on each method's selection data; test accuracy evaluates the same rubric on the 384 shared unseen examples under matched test-time settings.

Our analysis focuses on performance retention when moving to unseen examples. RubricLearner moves from 80.00% to 82.29%, while Eval-Skill drops from 87.83% to 82.03%. The reported 8.09-percentage-point contrast quantifies the difference in these development-to-test changes; McNemar p=1.0 concerns a different quantity—the final test-accuracy difference. RubricLearner maintains its development-level performance on the shared test set, whereas Eval-Skill's higher development accuracy does not carry over.

For Weakness2: Component effects

We appreciate your suggestion to examine the components separately. As a supplementary component analysis, we compare the complete method with four single-component controls, keeping the judge, data splits, initialization, and candidate budget matched. Removing conditions preserves principles and negative lessons; removing append-only retention permits rule revision and deletion. Local editing uses the same accumulated guidance. The no-gate control accepts all candidates, follows its own updated history, and returns the last version. In this table, removing conditions, retention, or validation yields larger score reductions than replacing regeneration with local editing. The ablation distinguishes these component-specific effects rather than assuming equal importance for every component.

ConfigurationHelpSteer3 Acc.FollowBench Acc.BiGGen-Bench Pearson
RubricLearner0.8230.8060.607
w/o applicability conditions0.8100.7800.580
w/o append-only retention0.8120.7860.572
Local editing instead of regeneration0.8190.8000.599
w/o validation gate0.8100.7870.582

For Question2: Epoch reset

Regarding your question about epoch resets, we would like to clarify that selecting the highest recorded score across epochs would not establish global optimality, particularly because validation subsets change between epochs. Our method carries accepted rules forward while resetting the selection baseline for each new validation subset. It returns the best accepted rubric within the final epoch, incorporating the accumulated learning experience. Selection is therefore based on within-epoch validation comparisons, rather than a ranking of scores obtained on different subsets.

Thank you for the additional experiments. The unseen-only results address much of my concern. RubricLearner retains nearly all of its full-dataset lead on both benchmarks, with gains of 1.6 percentage points on PPE-IFEval and 3.3 on RewardBench2 over the first baseline.

Could you confirm whether “Learnable Assessment Skills” in the new table is the “Learnable Skills” baseline in Table 1, and whether the HelpSteer3 held-out scores were rerun with matched decoding and voting settings?

Reviewer_hqiKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RubricLearner, a framework that improves evaluation rubrics while keeping the judge fixed. It diagnoses disagreements with reference labels, converts them into conditional rules, and accumulates accepted rules to regenerate rubrics. Updates are selected using validation performance. Experiments examine evaluation accuracy, rubric defects, held-out performance, and diagnosis-data requirements across several benchmarks.

Critical strengths

Separating accumulated diagnostic rules from the generated rubric provides a clear and inspectable representation of evaluation guidance. Conditioning lessons on their applicability is a sensible way to avoid turning local corrections into overly broad evaluation preferences. The paper examines both scoring performance and rubric quality, and the appendix provides useful disclosures about evaluation protocols and statistical comparisons.

Critical weaknesses
  1. The evaluation does not adequately support the headline performance and generalization claims. Section 4.1 evaluates on the full dataset, including development examples, while Appendix B.1 states that methods do not share a matched held-out protocol. The reported 8.09% generalization-gap difference also compares incompatible development statistics: validation accuracy versus cumulative evolution-trial accuracy. On the shared HelpSteer3 test set, the methods differ by only one correct prediction, with McNemar p = 1.0. These results do not establish the claimed advantage under a controlled generalization comparison.
  2. The contribution of the proposed rule structure and retention strategy is not isolated. The paper acknowledges that error diagnosis, rubric regeneration, and validation-based selection already exist in prior work. Its distinctive claims concern conditional rules and append-only retention, but the scaffold ablation removes the entire learning loop rather than testing these choices separately. Moreover, Appendix B.1 identifies Learnable Skills as a historical implementation rather than the cited Learnable Assessment Skills method, making the headline comparator’s identity unclear. The current evidence does not establish which proposed design accounts for the gains.
Questions for authors
  1. What are the results when all methods use the same development/test split, judge configuration, and comparable learning and inference budgets, with evaluation restricted to unseen examples?
  2. Given the different definitions of the development scores, what evidence supports improved generalization beyond comparable accuracy on the shared HelpSteer3 test set?
  3. Under a matched pipeline and search budget, what happens when conditional rules become unconditional, or accepted rules can be revised and deleted rather than only appended?
  4. What exactly is the Learnable Skills baseline, how does it relate to the cited published method, and which comparison supports calling the reported improvement state of the art?
2 comments
AuthorsOpenReview ↗

Response to Reviewer hqiK

Thank you for your careful reading and detailed, constructive feedback. We appreciate the opportunity to clarify our experimental setup and findings, as well as your suggestions for strengthening the empirical analysis. We respond to your concerns below.

For Weakness 1 / Question 1: Fair comparison and unseen-only evaluation

Thank you for raising this concern. We would like to clarify that Table 1 is a fair comparison: all learning-based methods use the same learning examples, and all methods are evaluated on the same full datasets with the same judge, temperature, and voting configuration within each judge block. RubricLearner uses 100 diagnosis examples and a separate 100-example validation set, and the inclusion of learning examples in the full-dataset evaluation applies equally to the learning-based methods. RubricLearner receives no additional learning-data access.

To complement these full-dataset results, and following your suggestion, we add a supplementary unseen-only evaluation on PPE-IFEval and RewardBench2 under Qwen3.7-Max. We reuse the final selected rubrics without further learning or selection, evaluating methods on the same subsets that exclude every example exposed during learning, generation context, or validation across all epochs. These supplementary results would extend the main-table findings to unseen examples, with accuracy gains over Eval-Skill of 1.8 percentage points on PPE-IFEval and 1.2 on RewardBench2.

MethodPPE-IFEval full (Acc.)PPE-IFEval unseen (Acc.)RewardBench2 full (Acc.)RewardBench2 unseen (Acc.)
Learnable Assessment Skills0.8860.8810.8590.854
Eval-Skill0.8850.8790.8800.875
RubricLearner0.9030.8970.8930.887

For Weakness 1 / Question 2: What the generalization result measures

We apologize for not clearly distinguishing Eval-Skill's learning procedure from the definition of the reported scores. Cumulative evolution concerns how the rubric is learned; for both methods, the development and test scores correspond to the final selected rubric, rather than an aggregation across intermediate candidates.

Our analysis focuses on performance retention when moving to unseen examples. The reported 8.09-percentage-point contrast is the difference between the two methods' development-to-test changes, not an inference from the one-example test difference. RubricLearner maintains its development-level performance on the shared test set, whereas Eval-Skill's higher development accuracy does not carry over.

For Weakness 2 / Question 3: Isolating conditions and retention

Thank you for suggesting component-level controls. To complement the existing scaffold comparison, we include two single-component ablations: remove explicit applicability conditions while retaining principles and negative lessons, or permit revision and deletion of accepted rules. All three configurations retain diagnosis, complete regeneration, and validation, with the same judge, splits, initialization, and candidate budget. The pattern separates the two effects: removing conditions particularly affects FollowBench, whereas permitting edits to accepted rules gives the larger reduction on BiGGen-Bench.

ConfigurationHelpSteer3 Acc.FollowBench Acc.BiGGen-Bench Pearson
RubricLearner0.8230.8060.607
w/o applicability conditions0.8100.7800.580
w/o append-only retention0.8120.7860.572

For Weakness 2 / Question 4: Comparator identity and the SOTA result

Thank you for asking us to clarify the baseline. We would like to clarify that Learnable Skills refers to our reimplementation of the cited Learnable Assessment Skills method. As no publicly released official implementation was available to us at the time of our experiments, we implemented the method based on its published description. The name refers to the cited method, not a separate baseline; we will clarify this relationship in the appendix.

We compare against leading iterative rubric-learning and skill-learning methods, including Learnable Assessment Skills and Eval-Skill, alongside Auto-Rubric for preference-based rubric induction. These baselines cover recent advances in learning evaluation guidance from examples and feedback. Against these methods, RubricLearner achieves the highest pairwise and scale averages under all three judge models, supporting its state-of-the-art performance in the evaluated setting.

Finally, thank you again for your time and thoughtful suggestions. We hope this response clarifies our evaluation protocol and methodological contribution, and we welcome any further questions.

Thank you for the additional experiments. I am still quite confused by three contradictions between the response and the submitted appendix:

  1. Evaluation protocols. The response says evaluation datasets, temperature, and voting settings are the same within each judge block. B.1 says the main table uses “each method’s own learning and evaluation protocol” (p. 12, lines 629–631). Eval-Skill’s RewardBench2 result uses 1,263 test items and average@3, “not accuracy on our 1,756-item evaluation” (p. 13, lines 648–652). Were the baselines rerun, or does Table 1 use the results described in B.1?
  2. Development scores. The response says both development scores evaluate the final selected rubrics. B.3 defines Eval-Skill’s score as “an evolution-stage cumulative trial accuracy, 404/460,” with different samples and statistical units (p. 14, lines 702–705). Which definition is correct, and what does this imply for the 8.09-point gap comparison?
  3. Baseline identity. The response calls Learnable Skills a reimplementation of Learnable Assessment Skills, while B.1 says it is “not Learnable Assessment Skills or Eval-Skill” (p. 12, lines 633–634). Which method was evaluated in Table 1?

Please clarify which account is correct in each case and what needs correcting in the paper or results.

Reviewer_G76oOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RubricLearner, an iterative method to improve rubrics qualities from end judge's preference. In each improvement step, the judge justification is contrasted with reference-side explanation, producing three improvements including applicability condition, positive principle, negative lessons, used to propose regenerated rubrics that are gated by validation improvement. On eight benchmarks, RubricLearner outperforms state-of-the-art baselines with Qwen-3.7 max as judge and extends to unseen examples.

Critical strengths
  1. The problem tackled is a timely one as rubrics are very commonly used in less verifiable domains, while rubrics quality itself suffers from various error modes. The paper would make meaningful community contributions to automating rubrics generation and improvements.
  2. The paper is empirically extensive in benchmark and baselinechoices, held-out evaluations, and contains insightful analysis.
  3. Although on a limited experiment scale, the method seems to have better generalization with smallers dev-minus-test gaps. This might have indicated a future direction for the community.
Critical weaknesses
  1. Table 1 reports the evaluation performance on the full set of each dataset, which contains the development set. Could the author clarify how much etimated contamination would be causing as other baselines are compared with the same standard? Could the author also report if other baselines approaches the problems by taking a dev sample set and report on the full set performance? Without such reports it will be unsound to make the main claim supported by table 1.
  2. I don't usually pick on novelty but the paper is more of an incremental work. The iterative refinement loop idea and the idea of learning by error analysis seems overlapping with other literatures. The paper does introduces design choices like condition applicability and retention gating by validation.
Questions for authors
  1. Could the author estimate the potential contamination introduced in table 1? Could the author also clearly report the dev-test composition of other baselines, if any?
  2. Could the author explain the design choices and clarify novelty more specifically?
1 comment
AuthorsOpenReview ↗

Response to Reviewer G76o

Thank you for your thoughtful review and for recognizing the importance of the problem and the breadth of our evaluation.

For Weakness1 / Question1: Shared learning data and evaluation overlap

Thank you for raising this concern. We would like to clarify that Table 1 is a fair comparison: all learning-based methods use the same learning examples, and all methods are evaluated on the same full datasets with the same judge, temperature, and voting configuration within each judge block. RubricLearner uses 100 diagnosis examples and a separate 100-example validation set, and the inclusion of learning examples in the full-dataset evaluation applies equally to the learning-based methods. RubricLearner receives no additional learning-data access.

To complement these full-dataset results, and following your suggestion, we add a supplementary unseen-only evaluation on PPE-IFEval and RewardBench2 under Qwen3.7-Max. We reuse the final selected rubrics without further learning or selection, evaluating methods on the same subsets that exclude every example exposed during learning, generation context, or validation across all epochs. These supplementary results would extend the main-table findings to unseen examples, with accuracy gains over Eval-Skill of 1.8 percentage points on PPE-IFEval and 1.2 on RewardBench2.

MethodPPE-IFEval full (Acc.)PPE-IFEval unseen (Acc.)RewardBench2 full (Acc.)RewardBench2 unseen (Acc.)
Learnable Assessment Skills0.8860.8810.8590.854
Eval-Skill0.8850.8790.8800.875
RubricLearner0.9030.8970.8930.887

For Weakness2 / Question2: What the design contributes

We appreciate your suggestion to make the design motivations more explicit. The contribution is learning reusable evaluation guidance from diagnosed errors, with explicit control over its scope and preservation. Diagnosis contrasts a reference-side explanation with the judge's erroneous justification to identify the mistaken evaluation principle. Each lesson records when it applies, what to reward, and what reasoning error to avoid. For example, discouraging added commentary in grammar correction should not penalize elaboration in creative writing.

Accepted lessons persist while the complete rubric can be reorganized, with validation determining whether additions are retained. To test whether scoping and preservation matter beyond a generic refinement loop, we compare the full method with two controls: remove applicability conditions, or allow accepted rules to be revised and deleted. The table shows lower performance when applicability conditions or append-only retention are removed. The proposed contribution is the organization of diagnosis-derived guidance, not iteration itself.

ConfigurationFollowBench Acc.BiGGen-Bench Pearson
RubricLearner0.8060.607
w/o applicability conditions0.7800.580
w/o append-only retention0.7860.572

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.