- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RubricLearner, a method for learning evaluation rubrics through diagnosis-guided refinement. This method keeps the judge fixed, transforms scoring errors into conditional rules containing applicability conditions, positive principles, and negative lessons, retains accepted rules in an append-only manner, and regenerates complete rubrics based on these rules. Candidate rubrics are only accepted if they achieve better results on independent validation sets. The authors evaluated the rubrics on eight benchmarks, four judgment formats, and three judges, and reported supplementary analyses including criterion defect, held-out performance, and diagnosis budget. The main results show that, at Qwen3.7-Max, RubricLearner achieves an average task-wise relative improvement of 3.51% compared to Learnable Skills, with a lower average defect rate.
Critical strengths
1.The method is clearly designed and auditable. RubricLearner represents error diagnosis as rules with applicable conditions and separates the accepted rule history from the final rubric. This helps limit the error generalization of rules while preserving the basis for modifications and version history, which is valuable for practical evaluation workflows.
2.The experiments are comprehensive. The paper covers eight benchmarks, four judgment formats, and multiple judges, and analyzes the method from the perspectives of accuracy, criterion defect, held-out behavior, and diagnosis budget, rather than just reporting a single aggregate score.
Critical weaknesses
1.The main performance comparisons lack a unified matched evaluation protocol. Appendix B.1 explicitly states that each method in Table 1 uses its own learning and inference protocols, and some methods also use different data sizes, temperatures, and voting settings; furthermore, the main table evaluates the full dataset, not a unified held-out set. Therefore, the 3.51% relative improvement emphasized in the paper cannot be considered a direct comparison under strictly controlled conditions, which weakens the main empirical claim.
2.Evidence for the core mechanisms and generalization conclusions is insufficient. The paper does not separately ablate the conditional rule structure, append-only retention, complete-rubric regeneration, and validation gate, thus making it impossible to determine which design contributes to the performance improvement. On the common HelpSteer3 held-out set, RubricLearner and Eval-Skill correctly identified 316 and 315 samples, respectively (p=1.0); their development score definitions and inference protocols also differ. Therefore, a smaller development-to-held-out gap does not sufficiently demonstrate stronger generalization ability.
Questions for authors
-
Could the authors re-evaluate the main method under the same held-out split, judge, decoding configuration, learning budget, and inference protocol, and evaluate the stability of the results using multiple random seeds, confidence intervals, or significance tests?
-
Could the authors perform component ablation on conditional structure, append-only rules, complete-rubric regeneration, and validation-based selection respectively? Furthermore, could you clarify whether Algorithm 1, after resetting and , might accept candidates weaker than those incumbent in the previous epoch, and whether the final output is the globally best rubric across all epochs?
2 comments
Thank you for the additional experiments. The unseen-only results address much of my concern. RubricLearner retains nearly all of its full-dataset lead on both benchmarks, with gains of 1.6 percentage points on PPE-IFEval and 3.3 on RewardBench2 over the first baseline.
Could you confirm whether “Learnable Assessment Skills” in the new table is the “Learnable Skills” baseline in Table 1, and whether the HelpSteer3 held-out scores were rerun with matched decoding and voting settings?
Response to Reviewer 8trV
Thank you for your careful review, your recognition of the method's inspectable design, and your constructive suggestions for the empirical analysis.
For Weakness 1 / Question 1 and Weakness 2: Matched settings, unseen performance, and uncertainty
Thank you for raising this concern. We would like to clarify that Table 1 is a fair comparison: all learning-based methods use the same learning examples, and all methods are evaluated on the same full datasets with the same judge, temperature, and voting configuration within each judge block. The blocks use Qwen3-8B, Qwen3.7-Max, and DeepSeek-V4-Flash. RubricLearner uses 100 diagnosis examples and a separate 100-example validation set, without updating judge parameters. The inclusion of learning examples in the full-dataset evaluation applies equally to the learning-based methods; RubricLearner receives no additional learning-data access. Method-specific protocols refer to internal learning procedures, not different learning data or judge-side evaluation settings. We will clarify this in our future version to avoid misunderstanding.
The 3.51% gain concerns the shared full-dataset evaluation. To complement these full-dataset results, and following your suggestion, we add a supplementary unseen-only evaluation on PPE-IFEval and RewardBench2 under Qwen3.7-Max. We reuse the final selected rubrics without further learning or selection, evaluating all methods on the same subsets that exclude every example exposed during learning, generation context, or validation across all epochs.
Regarding the HelpSteer3 analysis, we would like to clarify that cumulative evolution describes Eval-Skill's learning procedure, whereas both methods' reported scores evaluate their final selected rubrics. Development accuracy measures performance on each method's selection data; test accuracy evaluates the same rubric on the 384 shared unseen examples under matched test-time settings.
Our analysis focuses on performance retention when moving to unseen examples. RubricLearner moves from 80.00% to 82.29%, while Eval-Skill drops from 87.83% to 82.03%. The reported 8.09-percentage-point contrast quantifies the difference in these development-to-test changes; McNemar p=1.0 concerns a different quantity—the final test-accuracy difference. RubricLearner maintains its development-level performance on the shared test set, whereas Eval-Skill's higher development accuracy does not carry over.
For Weakness2: Component effects
We appreciate your suggestion to examine the components separately. As a supplementary component analysis, we compare the complete method with four single-component controls, keeping the judge, data splits, initialization, and candidate budget matched. Removing conditions preserves principles and negative lessons; removing append-only retention permits rule revision and deletion. Local editing uses the same accumulated guidance. The no-gate control accepts all candidates, follows its own updated history, and returns the last version. In this table, removing conditions, retention, or validation yields larger score reductions than replacing regeneration with local editing. The ablation distinguishes these component-specific effects rather than assuming equal importance for every component.
For Question2: Epoch reset
Regarding your question about epoch resets, we would like to clarify that selecting the highest recorded score across epochs would not establish global optimality, particularly because validation subsets change between epochs. Our method carries accepted rules forward while resetting the selection baseline for each new validation subset. It returns the best accepted rubric within the final epoch, incorporating the accumulated learning experience. Selection is therefore based on within-epoch validation comparisons, rather than a ranking of scores obtained on different subsets.