- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RubricRL turns task-specific evaluation criteria into something a model can learn. A single model switches between two objectives. It learns to generate rubrics from downstream judging feedback. It also learns to score answers with those rubrics. Its distributional rewards target both absolute score accuracy and score gaps between answers.
Critical strengths
- The method learns evaluation criteria without annotated rubrics. This addresses a practical bottleneck, since score labels can simultaneously supervise rubric generation and scoring.
- The fixed-judge experiment isolates the value of learned criteria. Rubrics from the base model can hurt performance, whereas learned rubrics improve it.
Critical weaknesses
- RubricRL reduces the need for human annotation, but reliable supervision is still required. Both rubric generation and judging learn from UltraFeedback's GPT-4 annotations, so they may inherit the annotator's errors. Human-rated tests show transfer, but they do not directly test whether the method corrects those errors. When GPT-4 and human scores disagree, does RubricRL agree with humans more often than the baselines?
Questions for authors
See Critical Weaknesses above.
3 comments
Thanks to the authors for the additional experiment. It helps address most of my concerns.
Thank you for your positive feedback. We appreciate your time and consideration.
Response to Reviewer uwN3
We thank the reviewer for raising this important concern. To directly address this question, we conduct additional experiments on FLASK-Eval, which provides both GPT-4-generated and human annotations for the same responses. Specifically, we construct a disagreement subset consisting of instances where GPT-4 and human scores differ, and evaluate our method and competing baselines against human annotations on this subset.
As shown in the table, RubricRL achieves the best performance across all four metrics. Notably, RubricRL also substantially outperforms GPT-4 itself. These results demonstrate that, despite being trained on GPT-4-annotated UltraFeedback data, RubricRL achieves better alignment with human judgments than both GPT-4 and existing judge-training baselines, even on instances where GPT-4 and human evaluations disagree. Furthermore, as reported in Tables 1 and 3 of the paper, RubricRL achieves better performance on the human-annotated FLASK-Eval and HelpSteer3 benchmarks. On BiGGen-Bench, Figure 3 shows that RubricRL-generated rubrics achieve performance comparable to human-validated rubrics. Together, these results provide further evidence of RubricRL's effectiveness on human-annotated evaluation benchmarks.
We attribute this improvement to three complementary factors. (1) Rather than directly predicting a score, RubricRL first identifies task-specific evaluation criteria through instance-specific rubric generation (what to evaluate) and then assesses the response conditioned on these criteria (how to evaluate). This rubric-first, judgment-second process enables RubricRL to learn human-relevant evaluation preferences rather than merely imitating GPT-4's individual judgments, allowing it to make more human-aligned judgments even when GPT-4 annotations are inaccurate and making it less sensitive to instance-level annotation noise. (2) RubricRL also benefits from human-grounded evaluation criteria during SFT initialization, as Feedback-Collection includes 50 human-authored seed rubrics, exposing the model to human-designed criteria about what aspects of responses should be evaluated. (3) GPT-4 annotations provide a useful supervisory signal that broadly reflects human preferences, despite inevitable annotation noise. The original UltraFeedback study [1] reports that GPT-4 achieves 59.7% agreement with individual human annotators and 68.6% agreement with the majority votes of three annotators on 400 comparison pairs. This suggests that GPT-4 annotations capture substantial aspects of collective human preferences, providing a meaningful foundation for learning human-aligned evaluation behaviors.
[1] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. ULTRAFEEDBACK: boosting language models with scaled AI feedback. In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 384, 9722–9744.