acceptodds
Under review as a conference paper at ICLR 2027

RubricRL: Rubric-Aware Reinforcement Learning for LLM Evaluation

Abstract

Reliable LLM judging requires not only learning , but also determining for each instance. Existing judge-training methods mainly optimize the former while leaving instance-specific evaluation criteria implicit, whereas rubric-guided approaches make these criteria explicit but typically optimize rubric generation separately from downstream judging. We introduce **RubricRL**, a reinforcement learning framework that couples instance-specific rubric learning with rubric-conditioned judging. RubricRL uses downstream judging performance to optimize instance-specific evaluation rubrics and, in turn, leverages the learned rubrics to guide score prediction within a shared policy model. We further introduce a distributional second-moment reward for absolute scoring quality and a distributional pairwise score-gap reward for relative scoring quality. Extensive experiments across diverse backbones and benchmarks show that RubricRL consistently improves LLM judging, achieves competitive performance with strong closed-source LLMs, and generates rubrics that provide judging performance comparable to human-validated criteria.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, Instance-Specific Rubrics, Reinforcement Learning, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uwN3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RubricRL turns task-specific evaluation criteria into something a model can learn. A single model switches between two objectives. It learns to generate rubrics from downstream judging feedback. It also learns to score answers with those rubrics. Its distributional rewards target both absolute score accuracy and score gaps between answers.

Critical strengths
  1. The method learns evaluation criteria without annotated rubrics. This addresses a practical bottleneck, since score labels can simultaneously supervise rubric generation and scoring.
  2. The fixed-judge experiment isolates the value of learned criteria. Rubrics from the base model can hurt performance, whereas learned rubrics improve it.
Critical weaknesses
  1. RubricRL reduces the need for human annotation, but reliable supervision is still required. Both rubric generation and judging learn from UltraFeedback's GPT-4 annotations, so they may inherit the annotator's errors. Human-rated tests show transfer, but they do not directly test whether the method corrects those errors. When GPT-4 and human scores disagree, does RubricRL agree with humans more often than the baselines?
Questions for authors

See Critical Weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer uwN3

W1: Performance comparison on the FLASK-Eval where GPT-4 and human annotations disagree

We thank the reviewer for raising this important concern. To directly address this question, we conduct additional experiments on FLASK-Eval, which provides both GPT-4-generated and human annotations for the same responses. Specifically, we construct a disagreement subset consisting of instances where GPT-4 and human scores differ, and evaluate our method and competing baselines against human annotations on this subset.

As shown in the table, RubricRL achieves the best performance across all four metrics. Notably, RubricRL also substantially outperforms GPT-4 itself. These results demonstrate that, despite being trained on GPT-4-annotated UltraFeedback data, RubricRL achieves better alignment with human judgments than both GPT-4 and existing judge-training baselines, even on instances where GPT-4 and human evaluations disagree. Furthermore, as reported in Tables 1 and 3 of the paper, RubricRL achieves better performance on the human-annotated FLASK-Eval and HelpSteer3 benchmarks. On BiGGen-Bench, Figure 3 shows that RubricRL-generated rubrics achieve performance comparable to human-validated rubrics. Together, these results provide further evidence of RubricRL's effectiveness on human-annotated evaluation benchmarks.

We attribute this improvement to three complementary factors. (1) Rather than directly predicting a score, RubricRL first identifies task-specific evaluation criteria through instance-specific rubric generation (what to evaluate) and then assesses the response conditioned on these criteria (how to evaluate). This rubric-first, judgment-second process enables RubricRL to learn human-relevant evaluation preferences rather than merely imitating GPT-4's individual judgments, allowing it to make more human-aligned judgments even when GPT-4 annotations are inaccurate and making it less sensitive to instance-level annotation noise. (2) RubricRL also benefits from human-grounded evaluation criteria during SFT initialization, as Feedback-Collection includes 50 human-authored seed rubrics, exposing the model to human-designed criteria about what aspects of responses should be evaluated. (3) GPT-4 annotations provide a useful supervisory signal that broadly reflects human preferences, despite inevitable annotation noise. The original UltraFeedback study [1] reports that GPT-4 achieves 59.7% agreement with individual human annotators and 68.6% agreement with the majority votes of three annotators on 400 comparison pairs. This suggests that GPT-4 annotations capture substantial aspects of collective human preferences, providing a meaningful foundation for learning human-aligned evaluation behaviors.

MethodMAE ↓RMSE ↓Pearson ↑Spearman ↑
Base0.98281.27040.57670.5281
TRACT0.77841.03590.63100.5799
J11.13321.40460.52400.5280
REAL0.77461.00610.65520.5570
GPT-40.93041.12560.60020.5118
RubricRL0.62060.81430.68480.6015

[1] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. ULTRAFEEDBACK: boosting language models with scaled AI feedback. In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 384, 9722–9744.

Thanks to the authors for the additional experiment. It helps address most of my concerns.

Thank you for your positive feedback. We appreciate your time and consideration.

Reviewer_PepqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the pointwise scoring task of Large Language Models as Judges (LLM-as-a-Judge), proposing that reliable evaluation requires not only learning "how to evaluate" but also determining "what to evaluate" for each specific sample. Existing judge training methods mostly implicitly encapsulate the evaluation criteria for specific samples, while existing rubric-guided methods typically optimize rubric generation independently from the downstream judging process. To address this, the authors propose RubricRL, a reinforcement learning framework that couples "sample-specific rubric generation (IRG)" and "conditional judge scoring (RCJ)" within a shared policy model:

  • Alternating dual-role policy optimization: The model alternately performs MM steps of judge policy optimization and NN steps of rubric policy optimization under the same base parameters. In the rule learning phase, the model samples multiple specific rule samples based solely on task instructions and general dimensionality criteria (without touching candidate answers), and utilizes the rewards from downstream judges' scores as sequence-level credit allocation signals. In the judge learning phase, the generated rule samples are fixed to guide evaluation inference and score prediction.

  • Distributional Rewards and Generalized Policy Gradient (GPG): To simultaneously optimize absolute scoring accuracy and relative ranking quality, the authors propose a Distributional Second-Moment Reward (which can be decomposed into expected score error and prediction variance) and a Distributional Pairwise Score-Gap Reward, both acting on the predicted score distribution. Since these rewards explicitly depend on the score distribution of the policy parameter θ\theta, the authors employ the Generalized Policy Gradient (GPG) to combine the Trajectory-Level Policy Gradient (GRPO) with a directly differentiable score distribution loss.

  • Experimental Validation: The authors conducted extensive experiments on 5 open-source model bases (Qwen3.5-4B/9B, Gemma-4-E4B, Phi-4-mini, Mistral-3-3B) and 5 evaluation benchmarks (UltraFeedback 1K, Feedback Bench, FLASK-Eval, HelpSteer3, BiGGen-Bench), demonstrating that RubricRL outperforms existing referee training baselines in terms of overall error (MAE/RMSE) and correlation (Pearson/Spearman), and that the generated sample-specific rules achieve evaluation results comparable to manually validated rules under a fixed referee.

Critical strengths
  • The paper demonstrates a precise problem identification and a reasonable two-stage decoupling design (Motivation & Formulation):

    • The paper astutely points out the core contradiction in single-point LLM judging: the general scoring criteria are too abstract, while manually writing sample-level rules is costly, and independently generated automatic rules lack downstream alignment feedback.

    • When generating sample-specific rules (IRGs), the model strictly filters candidate responses aa based solely on the task instruction xx and the general rule rgenr^{\mathrm{gen}}. This design is crucial, effectively preventing the model from "tailor-making" bias rules based on specific responses, ensuring the objectivity and consistency of evaluation scales for different responses to the same question.

  • The empirical evaluation is comprehensive, and the independent verification of the rule quality is convincing:

    • The experiments cover five different families of state-of-the-art open-source models from 3B to 9B, as well as five highly regarded evaluation benchmarks, fully demonstrating the method's generalization ability across model architectures.

    • The controlled variable experiments on BiGGen-Bench in Section 4.3 are well-designed: by freezing the same Qwen3.5-9B referee and replacing only the instance rules generated by different methods, it directly demonstrates that the rules learned by RubricRL are of high quality (superior to OpenRubrics, MAPLE, and zero-shot generated rules, and approaching human-verified BiGGen rules), eliminating concerns that "performance improvements come solely from referee-side training."

Critical weaknesses

Potential "Reward Hacking" Risks and Insufficient Cross-Model Generalization Analysis in Rule Generation (IRG):

  • During the rule policy optimization phase, IRG is only supervised by the downstream judge's reward RJ,kR_{\mathrm{J},k} and an extremely simple heuristic reward format (whether it contains 1–5 segment markers, whether it is empty, whether its length exceeds the limit). Since IRG and RCJ share parameters and co-evolve, theoretically, IRG may generate "shortcut" or templated text that is difficult for humans to understand but triggers specific scoring biases in RCJ, rather than truly faithful evaluation rules for the task.

  • Although Section 4.3 validates the effectiveness of freezing the referee in the same family Qwen3.5-9B, there is a lack of cross-family rule transfer experiments (e.g., using RubricRL trained with Qwen3.5 to generate rules and feeding them to frozen Llama/Gemma or closed-source models as referees), and there is also a lack of direct quantitative evaluation of the generated rules themselves in terms of atomicity, coverage, illusion rate, etc. (e.g., directly evaluating rule quality on RubricBench).

Questions for authors
  • Regarding a fair comparison of test-time computation cost with RAIL:

    • If the baseline methods (especially TRACT and REAL) are also treated with +RAIL (calculating the expected score from the score distribution) and multi-track sampling averaging (e.g., averaging G=4G=4 inference tracks to match the 4x inference cost of RubricRL + K4G1), how would the baseline methods perform? Would RubricRL still maintain a significant advantage?

    • Please provide a quantitative comparison table of training phase (time/memory usage per step) and inference phase (average number of tokens generated and actual wall-clock latency) with TRACT, J1, and REAL.

  • Regarding the cross-model Rubric Transfer capability of the rules:

    • In the BiGGen-Bench experiments in Section 4.3, the test judges used the Qwen3.5-9B, which is from the same source as the training base. If the instance rules generated by RubricRL trained by Qwen3.5-4B/9B are provided to frozen judges from completely different families (such as Gemma-4-E4B or closed-source models), can they still achieve results that are better than OpenRubrics/MAPLE and close to those of human-made rules?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Pepq (1/3)

W1: Cross-Model Generalization Analysis of IRG

To investigate whether RubricRL-generated rubrics are specific to the co-trained Qwen judge, we conduct cross-family transfer experiments on BiGGen-Bench. Specifically, we use the IRG trained with Qwen3.5-9B to generate instance-specific rubrics and feed them to two frozen judges from different model families: Gemma-4-E4B and DeepSeek-V4-Flash. We compare six settings: (1) No Rubric, (2) rubrics generated by the base Qwen3.5-9B, (3) OpenRubrics, (4) MAPLE, (5) RubricRL-Qwen3.5-9B, and (6) human-validated BiGGen rubrics.

As shown in the table, RubricRL consistently outperforms all automatic rubric generation baselines across both external judges and all four evaluation metrics. Specifically, it achieves MAEs of 0.816 and 0.777 on Gemma and DeepSeek, respectively, substantially outperforming MAPLE. Moreover, its performance is comparable to that of human-validated rubrics across both judges. These results demonstrate that RubricRL-generated rubrics capture transferable, task-relevant evaluation criteria rather than merely exploiting scoring patterns specific to the co-trained Qwen judge. Their consistent effectiveness across different judge families provides further evidence against the concern that RubricRL relies solely on judge-specific shortcuts.

Gemma-4-E4B as JudgeMAE ↓RMSE ↓Pearson ↑Spearman ↑
No Rubric0.9391.35456.453.0
Base Qwen3.5-9B1.0831.53153.351.9
OpenRubrics0.8621.31257.154.3
MAPLE0.8531.13458.254.6
RubricRL-Qwen3.5-9B0.8161.09164.861.8
Human Rubrics0.8211.06564.461.1
DeepSeekV4Flash as JudgeMAE ↓RMSE ↓Pearson ↑Spearman ↑
No Rubric0.8651.27758.856.1
Base Qwen3.5-9B0.9871.43155.554.8
OpenRubrics0.8571.28858.256.3
MAPLE0.8341.13460.057.2
RubricRL-Qwen3.5-9B0.7770.97366.264.4
Human Rubrics0.7810.97866.863.9
AuthorsOpenReview ↗

Response to Reviewer Pepq (2/3)

W2: Quantitative Evaluation of Instance-Specific Rubric Quality

To directly evaluate the quality of generated rubrics, we conduct an additional experiment on RubricBench [1], adopting its rubric alignment metrics, including Rubric Recall, Hallucination Rate, and Structural F1. Since RubricRL generates five-level score-band rubrics, we use GPT-6.1-Sol to decompose them into atomic evaluation criteria before applying the alignment metrics. We compare RubricRL with five existing rubric generation methods: Auto-Rubric, RocketEval, CheckEval, TICK, and OpenRubric. As shown in the table, RubricRL achieves a Structural F1 of 33.0, outperforming Auto-Rubric, TICK, and OpenRubric. These results demonstrate that RubricRL achieves reasonable alignment with human-annotated evaluation criteria, although it does not outperform all existing methods.

We attribute the performance gap partly to two factors. First, despite format conversion, RubricRL's score-band rubrics and RubricBench's atomic checklist-style criteria differ in their representation and granularity, and the conversion may inevitably lose some scoring granularity and the hierarchical relationships among the five score levels. Second, RubricRL is trained on UltraFeedback, whereas RubricBench covers a different distribution of tasks and evaluation scenarios, introducing a cross-dataset generalization gap.

GeneratorRubric Recall ↑Hallucination Rate ↓Structural F1 ↑
Auto-Rubric [2]40.4%76.2%28.3
RocketEval [3]35.4%54.1%38.3
CheckEval [4]53.8%68.7%38.2
TICK [5]26.3%74.1%24.8
OpenRubric [6]47.5%72.6%31.5
RubricRL37.5%70.5%33.0

[1] Zhang Q, Zhou J, Wang Y, et al. RubricBench: Aligning model-generated rubrics with human standards[J]. arXiv preprint arXiv:2603.01562, 2026.

[2] Lipeng Xie, Sen Huang, Zhuo Zhang, Anni Zou, Yunpeng Zhai, Dingchao Ren, Kezun Zhang, Haoyuan Hu, Boyin Liu, Haoran Chen, Zhaoyang Liu, and Bolin Ding. 2025. Auto-rubric: Learning to extract generalizable criteria for reward modeling. Preprint, arXiv:2510.17314.

[3] Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. 2025. Rocketeval: Efficient automated LLM evaluation via grading checklist. In The Thirteenth International Conference on Learning Representations.

[4] Yukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim. 2025. CheckEval: A reliable LLM-as-a-judge framework for evaluating text generation using checklists. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15782–15809, Suzhou, China. Association for Computational Linguistics.

[5] Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. 2024. Ticking all the boxes: Generated checklists improve llm evaluation and generation. Preprint, arXiv:2410.03608.

[6] Liu T, Xu R, Yu T, et al. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 17417-17437.

W3: Potential Reward Hacking Risks in IRG

We thank the reviewer for raising this important concern. To address it, we conducted two additional experiments on the above responses (W1&W2). These results suggest that the learned rubrics capture transferable, task-relevant evaluation, high-quality criteria rather than merely exploiting scoring patterns specific to the co-trained judge.

Furthermore, our method incorporates three design choices that help mitigate potential reward hacking. (1) Response-independent generation. IRG only conditions on the instruction and generic rubric, without access to candidate responses, preventing it from tailoring evaluation criteria to specific responses. (2) Explicit generation constraints. Our IRG prompt requires concrete, instruction-specific criteria and meaningful distinctions between adjacent score levels, discouraging vague or templated rubrics. (3) Alternating optimization with stop-gradient. During Rubric Policy Optimization, gradients are stopped through the inactive Judge role, reducing direct optimization interference between the two objectives.

AuthorsOpenReview ↗

Response to Reviewer Pepq (3/3)

W4: Fairness comparison of test-time scaling

We thank the reviewer for this valuable suggestion. To ensure a fairer comparison, we additionally evaluate TRACT, J1, and REAL with four independent judge inference tracks, aggregating their predictions through voting, as a counterpart to RubricRL's K4G1 setting. We also apply RAIL to all three baselines to examine the effect of expected-score decoding.

As shown in the table, RubricRL + K4G1+RAIL achieves the best performance across all four metrics compared with the four-inference baselines. We attribute these gains to two complementary factors. (1) RubricRL explicitly optimizes the predictive score distribution through the distributional second-moment and pairwise score-gap rewards, making it better suited to leverage RAIL's expected-score decoding at inference time. (2) K4G1 generates multiple instance-specific rubrics, providing diverse task-specific evaluation criteria rather than merely repeating judge inference, which helps improve the robustness of the final judgment.

MethodMAE ↓RMSE ↓Pearson ↑Spearman ↑
TRACT0.6751.04166.761.9
+ 4 inferences0.6810.98767.262.5
+ RAIL0.6900.88569.264.7
J11.0301.35757.959.0
+ 4 inferences0.9581.32157.459.3
+ RAIL1.0171.37156.458.1
REAL0.6410.98566.658.3
+ 4 inferences0.6390.97467.158.6
+ RAIL0.7060.97363.260.2
RubricRL0.6660.89666.162.3
+ K4G10.6290.84269.364.2
+ RAIL0.6120.81770.565.6

W5: Computational Cost Analysis

We thank the reviewer for requesting a quantitative comparison of computational costs. We profile the training and inference efficiency of RubricRL, TRACT, J1, and REAL under the same hardware and evaluation setup. The results are reported in the tables. During training, RubricRL requires 147.68 seconds per step and 61.51 GiB of peak GPU memory. Its training cost is higher than TRACT and J1, but remains substantially lower than REAL, which requires 299.46 seconds per step and 74.09 GiB of GPU memory. During inference, RubricRL generates an average of 173.88 tokens per response, with an average latency of 2.829 seconds. While its inference cost is higher than TRACT and J1 due to the additional rubric generation process, it is lower than REAL in both generated tokens and latency.

Overall, RubricRL introduces additional computational overhead compared with simpler judge-training methods, reflecting the cost of explicitly generating instance-specific rubrics. However, its computational requirements remain lower than REAL in both training time and inference cost, while providing the performance improvements reported in our experiments.

MethodTraining Time (s/step)Peak GPU Memory (GiB)
RubricRL147.6861.51
TRACT25.3641.79
J196.7243.89
REAL299.4674.09
MethodAvg. generated tokens / responseAverage time / response (s)
RubricRL173.882.829
TRACT43.160.509
J186.021.102
REAL238.333.729
Reviewer_PSeEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies pointwise LLM judging by combining instance-specific rubric generation (IRG) with rubric-conditioned judging (RCJ), alternately optimizing both roles within a shared policy. The rubric generator receives only the task and generic rubric, obtaining sequence-level credit from downstream judging performance; the judge learns to produce an evaluation rationale and numerical score given the rubric. The paper introduces a distributional second-moment reward and a distributional pairwise score-gap reward for responses to the same question, combining trajectory-level policy optimization with direct gradients through the score distribution. Experiments compare five backbones on four scoring benchmarks and evaluate rubric sources with a fixed judge on BiGGen-Bench. The results support overall effectiveness, with advantages varying across models, metrics, and inference settings.

Critical strengths
  1. The cumulative ablation provides convincing evidence for the method. Table 2 successively adds RL, GPG, the second-moment reward, the score-gap reward, and RPO, reducing MAE from 1.0000 to 0.7187 and increasing Pearson from 48.73 to 67.19, with both metrics improving at each step. Within the preceding component combination, RPO further reduces MAE from 0.7875 to 0.7187. RMSE does not improve at every step, so these results establish incremental evidence conditional on the preceding components, rather than monotonic gains on every metric or fully independent contributions from each component.

  2. The distributional reward design has substance. The second-moment reward operates directly on the full predictive score distribution, with Proposition 1 decomposing it into expected-score error and predictive variance; the score-gap reward further constrains the ordering and magnitude of quality differences between responses to the same question. These designs are reusable for numerical judge training. Relative to Rubric-ARM, the specific novelty lies in combining pointwise numerical scoring, a shared policy, and distributional rewards; the broader idea of alternating rubric generation and judge learning already has a precedent.

  3. The experimental coverage is broad. Tables 1/3 cover five backbones from four model families, four scoring benchmarks, and four metrics, complemented by score-band analysis in Figure 5, K4G1/RAIL inference settings, and hyperparameter sensitivity analysis. The results support the method's overall utility, although advantages vary by backbone, benchmark, metric, and inference setting rather than holding in every column.

  4. The rubric-source comparison includes a useful negative control. Section 4.3 compares rubric-free judging, base-model-generated rubrics, and trained RubricRL rubrics. Base-model rubrics do not outperform rubric-free judging, showing that adding rubric text does not automatically improve scoring. RubricRL rubrics perform well with the tested fixed judge, supporting the practical value of the trained rubrics; this control alone does not establish a general advantage across judges.

  5. The method is clearly structured, and the comparison discussion reports important limitations. The roles of IRG, RCJ, RPO, and JPO are easy to follow. Section 4.4 explicitly acknowledges that several closed-source models achieve higher correlations, locating the main current advantage in absolute numerical scoring. Related Work also directly discusses the closest alternating-learning work and the distinctions of this paper. These statements help readers assess the scope of the contribution.

Critical weaknesses
  • W1: The provenance of scoring supervision needs to be distinguished more clearly from the interpretation of human alignment and the closed-source comparison. As described in the original dataset papers (Cui et al., 2024; Kim et al., 2024a), UltraFeedback scores come from GPT-4, while Feedback-Collection and Feedback Bench are predominantly constructed with GPT-4. Section 4.1 and Appendices 8.2–8.3 do not state these sources directly, yet Figures 1/2 use “Human Score.” Model-derived reference scores should be distinguished from direct human ratings. Feedback-Collection includes human-authored seed rubrics, so the concern is the absence of direct human ratings as supervision in the described training, rather than an absence of all human input. For each metric, macro-averaging the unenhanced results in Tables 1/3 over the ten backbone–benchmark combinations formed by the five backbones and FLASK-Eval/HelpSteer3 shows overall improvements for RubricRL over TRACT (reviewer recomputation), although individual columns remain mixed. The mixed averages in Tables 1/3 do not separately characterize improvements in agreement with human evaluations. Figure 4 provides valid in-distribution evidence on held-out UltraFeedback, but the roles of adaptation to the training-label source and more general judging ability remain unresolved. The sources should be explicit, results should be reported separately by provenance, and the Figure 4 configuration should be clarified; broader interpretations of superiority to closed-source models need human-annotated or out-of-distribution evidence, or correspondingly narrower claims. These concerns do not directly invalidate comparisons among the trained judging methods on the tested tasks.

  • W2: Cross-judge generalization of rubric quality and reporting of key-result stability remain limited. Fixing Qwen3.5-9B and varying rubric sources in Section 4.3 is a valid conditional comparison. However, the specific frozen-judge checkpoint is unclear, and one judge does not establish a general advantage for the rubrics with other model families. Adaptation between generated rubrics and the training judge is possible, but the current experiments neither establish nor rule out this effect; weaker base-model rubrics cannot establish it on their own. Clarifying the checkpoint and testing a fixed judge from another family, or restricting the conclusion to the current judge, would address this generalization issue. A further fixable reporting gap is that the number of independent training runs and variability across runs for key configurations are unspecified. Figure 5 does provide 95% CIs, but their statistical unit should be made explicit. The paper provides configurations, derivations, and prompt templates; no code link appears in the manuscript, and the availability of anonymous implementation materials and any subsequent release plan remain unclear.

Questions for authors
  1. Could you state the score and generation provenance of UltraFeedback, Feedback-Collection, and Feedback Bench in Section 4.1 and the appendix, revise “Human Score” in the figures, and report the results in Tables 1/3 separately by annotation source? This would distinguish agreement with model-derived reference scores from agreement with direct human evaluations.
  2. What backbones/checkpoints or API versions, prompting and demonstration setups, score-extraction methods, and sampling budgets are used in Figure 4? Does RubricRL use K4G1 and/or RAIL, and how do these results correspond to Tables 1/3? Given that this comparison uses UltraFeedback, the same data source used for training, could you provide a corresponding comparison on one human-annotated benchmark or explicitly restrict the claimed advantage to numerical scoring under the current in-distribution setting?
  3. Which checkpoint is used as the frozen Qwen3.5-9B judge in Section 4.3? Fixing the judge and varying rubric sources is a useful conditional comparison; does the relative advantage of RubricRL rubrics transfer to one frozen judge from another model family? If this has not been tested, could the rubric-quality conclusion be explicitly restricted to downstream scoring with the current judge?
  4. How many independent training runs were actually performed for the key configurations? Could you explain the statistical unit and construction of the confidence intervals in Figure 5 and provide enough uncertainty information to assess the stability of the main baseline differences, the RPO increment, and the rubric-source comparison? A full rerun of every hyperparameter setting is not required.
  5. Related Work explains the distinction from Rubric-ARM, and Tables 10/11 provide useful comparisons of alternating versus joint optimization and shared versus separate parameters. Acknowledging the difference between its pairwise preference supervision and this paper's pointwise numerical scoring, would it be possible to compare transferable alternating rubric-learning designs on a common subset or under an explicitly controlled protocol? If a direct comparison is not applicable, could you explain which differences prevent a fair comparison and which relative design advantages the existing ablations support? This question does not request direct comparison of scores from unmatched cross-paper settings.
  6. The paper already provides configurations, rubric examples, and IRG/RCJ prompt templates. Could you clarify whether the implementation, training/evaluation scripts, and generated rubrics are included in anonymous supplementary materials and, if not, whether there is a subsequent release plan? This question concerns the availability of reproducibility materials and does not request identity disclosure or a public repository during review.
7 comments
AuthorsOpenReview ↗

Response to Reviewer PSeE (1/4)

W1: Clarification of Annotation Provenance and Result Reporting

We thank the reviewer for pointing this out. We have clarified the annotation provenance of UltraFeedback, Feedback-Collection, Feedback Bench, FLASK-Eval, HelpSteer3 and BigGen-Bench in Appendices 8.2–8.3. We have also revised the terminology in Figures 1–2, replacing "Human Score" with "Annotation Score". We have added annotation-source markers to the benchmark names in Tables 1 and 3 to distinguish model-generated from human annotations. We have also added a table reporting the macro-averaged results separately for model-annotated and human-annotated benchmarks, and our method continues to outperform competing baselines in most evaluation settings. All corresponding revisions are highlighted in blue in the revised manuscript.

MethodModel Avg. MAE ↓Model Avg. RMSE ↓Model Avg. Pearson ↑Model Avg. Spearman ↑Human Avg. MAE ↓Human Avg. RMSE ↓Human Avg. Pearson ↑Human Avg. Spearman ↑
Generative Reward Models
Prometheus 20.8991.27258.456.71.1091.53729.524.3
Selene1 Mini0.7811.11363.664.01.0391.38041.439.2
DeepSeek-GRM0.8331.21164.163.40.9281.31141.135.4
Qwen3.5-4B
CoT0.9501.30263.563.60.9681.38447.343.7
TRACT0.7441.02674.174.40.8211.19153.248.2
J10.9351.27264.666.11.0261.38648.247.6
REAL0.7691.11668.167.60.7941.13752.445.4
RubricRL0.7081.00975.275.90.8051.07753.048.3
+ K4G10.7150.97277.178.10.7801.03655.049.3
+ RAIL0.7270.96577.578.60.7651.01156.151.2
Gemma-4-E4B
CoT0.8421.25466.466.51.1131.52845.343.5
TRACT0.8031.16663.563.90.9861.30945.040.9
J10.8711.23967.667.50.9871.35848.945.4
REAL0.7601.10968.768.90.9381.25550.646.6
RubricRL0.6880.98174.174.70.8301.09951.045.3
+ K4G10.6910.93975.976.80.8111.06552.246.4
+ RAIL0.6990.92576.677.60.7991.03953.347.8
Qwen3.5-9B
CoT0.8881.30366.166.51.1041.52250.248.8
TRACT0.6890.96374.174.70.8161.18055.751.7
J10.8961.28364.966.21.1061.46048.848.4
REAL0.7391.11770.170.00.8491.23453.449.5
RubricRL0.7001.01375.676.60.8091.07256.651.5
+ K4G10.6970.98077.378.70.7901.04458.053.1
+ RAIL0.6720.96778.079.50.7721.01659.054.4
Phi-4-mini
CoT1.1521.59248.948.21.0541.49320.819.2
TRACT0.9241.22560.361.90.9141.20635.532.4
J10.8871.23159.056.50.9921.36527.922.5
REAL1.1001.53755.955.01.0391.40427.926.3
RubricRL0.6740.95271.772.40.9351.20438.035.3
+ K4G10.7060.92973.073.60.9771.24941.738.3
+ RAIL0.7070.90974.374.90.9651.23242.839.4
Mistral-3-3B
CoT1.0041.41553.053.61.2501.54035.435.1
TRACT0.8701.13264.664.80.8621.15143.638.7
J10.8541.15964.864.80.9671.29339.837.0
REAL0.8411.25162.961.81.0051.38241.339.2
RubricRL0.7081.01672.573.40.9281.17442.138.0
+ K4G10.7170.98074.175.10.9011.13044.939.8
+ RAIL0.7140.96574.876.00.8971.13044.940.1
AuthorsOpenReview ↗

Response to Reviewer PSeE (2/4)

W2: Experimental Configuration of Figure 4

We thank the reviewer for requesting these clarifications. RubricRL, TRACT, J1, and REAL all use Qwen3.5-9B [1] as the backbone and are trained on UltraFeedback under the same data and backbone settings. The closed-source API baselines include GPT-5.6-Terra [2], Claude Sonnet 5 [3], Gemini 3.6 Flash [4], and DeepSeek-V4-Flash [5]. The prompts used by RubricRL for instance-specific rubric generation (IRG) and rubric-conditioned judging (RCJ) are provided in Appendix Figures 8 and 9, respectively. We use the [RESULT] marker to identify and extract the final numerical score. The model is instructed to output the score in the format [RESULT] <score>, where the score is an integer between 1 and 5. We use the default sampling configurations for all models, including Qwen3.5-9B and the closed-source API models. As specified in Appendices 8.1.2 and 8.1.3, the maximum generation lengths are 256 tokens for Judging and 220 tokens for Rubric.

[1] https://huggingface.co/Qwen/Qwen3.5-9B

[2] https://developers.openai.com/api/docs/models/gpt-5.6-terra

[3] https://platform.claude.com/docs/en/models/sonnet-5/overview

[4] https://aistudio.google.com/docs/models/gemini-3.6-flash

[5] https://api-docs.deepseek.com/

W3: Does RubricRL use K4G1 and/or RAIL in the Figure 4 experiment, and how do these results correspond to Tables 1/3?

Yes, we use both K4G1 and RAIL for RubricRL in the Figure 4 experiment.

Tables 1 and 3 report three configurations: RubricRL (without K4G1 or RAIL), RubricRL + K4G1, and RubricRL + K4G1 + RAIL. The results show that the standard RubricRL already achieves strong performance in most evaluation settings without additional inference-time enhancements. Furthermore, K4G1 and RAIL can be incorporated to further improve its performance. For fair comparison, we additionally evaluate TRACT, J1, and REAL with four independent judge inference tracks, aggregating their predictions through voting, as a counterpart to RubricRL's K4G1 setting. We also apply RAIL to all three baselines to examine the effect of expected-score decoding.

As shown in the table, RubricRL + K4G1+RAIL achieves the best performance across all four metrics compared with the four-inference baselines. We attribute these gains to two complementary factors. (1) RubricRL explicitly optimizes the predictive score distribution through the distributional second-moment and pairwise score-gap rewards, making it better suited to leverage RAIL's expected-score decoding at inference time. (2) K4G1 generates multiple instance-specific rubrics, providing diverse task-specific evaluation criteria rather than merely repeating judge inference, which helps improve the robustness of the final judgment.

MethodMAE ↓RMSE ↓Pearson ↑Spearman ↑
TRACT0.6751.04166.761.9
+ 4 inferences0.6810.98767.262.5
+ RAIL0.6900.88569.264.7
J11.0301.35757.959.0
+ 4 inferences0.9581.32157.459.3
+ RAIL1.0171.37156.458.1
REAL0.6410.98566.658.3
+ 4 inferences0.6390.97467.158.6
+ RAIL0.7060.97363.260.2
RubricRL0.6660.89666.162.3
+ K4G10.6290.84269.364.2
+ RAIL0.6120.81770.565.6

W4: Evaluation on Human-Annotated Benchmarks in Figure 4 experiment

We thank the reviewer for this suggestion. To further examine whether RubricRL's advantage generalizes beyond GPT-4-annotated data, we conduct an additional comparison on FLASK-Eval, which provides direct human annotations. We evaluate RubricRL and its enhanced variant against four closed-source LLMs under the same evaluation protocol.

As shown in the table, standard RubricRL achieves the lowest MAE and RMSE, as well as the highest Pearson correlation. Claude Sonnet 5 achieves a slightly higher Spearman correlation. Furthermore, RubricRL with K4G1 and RAIL achieves the best performance across all four metrics.

These results provide further evidence that RubricRL's numerical scoring advantage is not limited to the GPT-4 annotation distribution but also generalizes to direct human evaluations, demonstrating competitive performance against strong closed-source LLMs.

MethodMAE ↓RMSE ↓Pearson ↑Spearman ↑
GPT-5.6-Terra0.9811.35264.261.5
Claude-Sonnet-50.9641.30169.165.9
Gemini-3.6-Flash0.9831.37865.463.4
DeepSeek-V4-Flash0.8821.26766.462.6
J11.1541.49356.557.5
REAL0.7481.11866.262.6
RubricRL0.6790.88869.965.5
RubricRL + K4G1 + RAIL0.6390.83871.868.0
AuthorsOpenReview ↗

Response to Reviewer PSeE (3/4)

W5: Frozen Judge Checkpoint in Section 4.3

We use the base Qwen3.5-9B checkpoint [1] as the frozen Judge in Section 4.3, rather than the RubricRL-trained model. This ensures a fair comparison by keeping the Judge fixed, and it allows us to fairly compare the effectiveness of different rubric sources while keeping the Judge model unchanged.

[1] https://huggingface.co/Qwen/Qwen3.5-9B

W6: Cross-Family Rubric Transfer

To investigate whether the advantage of RubricRL-generated rubrics generalizes across judge architectures, we conduct cross-family transfer experiments on BiGGen-Bench. Specifically, we use the IRG trained with Qwen3.5-9B to generate instance-specific rubrics and provide them to two frozen judges from different model families: Gemma-4-E4B and DeepSeek-V4-Flash. We compare six settings: (1) No Rubric, (2) rubrics generated by the base Qwen3.5-9B, (3) OpenRubrics, (4) MAPLE, (5) RubricRL-Qwen3.5-9B, and (6) human-validated BiGGen rubrics.

As shown in the table, RubricRL consistently outperforms all automatic rubric generation baselines across both external judges and all four evaluation metrics. Moreover, its performance is comparable to that of human-validated rubrics across both judges. These results demonstrate that RubricRL-generated rubrics capture transferable, task-relevant evaluation criteria, and their effectiveness is not limited to the co-trained Qwen judge but extends to frozen judges from different model families.

Gemma-4-E4B as JudgeMAE ↓RMSE ↓Pearson ↑Spearman ↑
No Rubric0.9391.35456.453.0
Base Qwen3.5-9B1.0831.53153.351.9
OpenRubrics0.8621.31257.154.3
MAPLE0.8531.13458.254.6
RubricRL-Qwen3.5-9B0.8161.09164.861.8
Human Rubrics0.8211.06564.461.1
DeepSeekV4Flash as JudgeMAE ↓RMSE ↓Pearson ↑Spearman ↑
No Rubric0.8651.27758.856.1
Base Qwen3.5-9B0.9871.43155.554.8
OpenRubrics0.8571.28858.256.3
MAPLE0.8341.13460.057.2
RubricRL-Qwen3.5-9B0.7770.97366.264.4
Human Rubrics0.7810.97866.863.9

W7: Training Stability Across Random Seeds.

We thank the reviewer for raising this important question. Our original experiments were conducted with random seed 13. To further assess training stability, we independently retrained RubricRL with two additional seeds (42 and 123) using Qwen3.5-4B, and evaluated all three runs on four benchmarks. As shown in the table, we report the mean and standard deviation across three independent training runs. The consistently small standard deviations demonstrate that RubricRL achieves stable performance across different random seeds while maintaining its overall advantage over competing baselines.

MethodMAE ↓RMSE ↓Pearson ↑Spearman ↑
UltraFeedback 1K0.715 ± 0.0020.995 ± 0.00267.2 ± 0.167.9 ± 0.2
Feedback Bench0.694 ± 0.0030.964 ± 0.01083.9 ± 0.184.6 ± 0.2
FLASK-Eval0.665 ± 0.0020.898 ± 0.00266.5 ± 0.362.8 ± 0.1
HelpSteer30.945 ± 0.0041.259 ± 0.01039.8 ± 0.134.0 ± 0.1

W8: Confidence Interval Construction.

We thank the reviewer for the clarification request. We compute 95% confidence intervals using 10,000 bootstrap resamples at the instruction–criterion level, keeping responses within each cluster together. For each resample, we recompute the RMSE or standard deviation and obtain the confidence intervals from the 2.5th and 97.5th percentiles.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their constructive feedback. We would like to briefly highlight the key strengths recognized by the reviewers and the additional evidence provided during the rebuttal.

1. Key strengths recognized by the reviewers

Clear motivation and rubric-learning framework. Reviewer Pepq recognized our well-motivated two-stage design for learning what to evaluate and how to evaluate, while Reviewer uwN3 highlighted the ability to learn evaluation criteria without requiring annotated rubrics during RL training. Reviewer PSeE noted that the method is clearly structured.

Principled reward design. Reviewer PSeE highlighted the technical contributions of our distributional rewards and the convincing evidence from cumulative ablations.

Comprehensive experimental validation. Reviewers Pepq and PSeE highlighted our extensive evaluations across model families and benchmarks, while Pepq and uwN3 particularly recognized the fixed-judge experiments as evidence of the effectiveness of learned rubrics.

2. Additional evidence from the rebuttal

We conducted substantial new experiments, including (i) human-annotated evaluations and GPT-4–human disagreement analysis, demonstrating stronger alignment with human judgments; (ii) cross-family rubric transfer to frozen Gemma and DeepSeek judges, showing effectiveness beyond the co-trained judge; and (iii) direct rubric-quality evaluation, multi-seed stability analysis, and additional test-time computation comparisons. We also clarified annotation provenance and experimental details in the revised manuscript.

3. Positive reviewer feedback

Reviewer PSeE explicitly confirmed that our clarifications “have resolved all my concerns.” Reviewer uwN3 stated that our additional experiment “helps address most of my concerns,” while Reviewer Pepq acknowledged the additional experiments and clarifications.

We believe these results further strengthen the paper's contributions and address the major concerns raised during review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.