- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates factors contributing to reward over-optimization in rubric-based reinforcement learning (RL) for large language models (LLMs) by examining four hypotheses: (1) suboptimal reference guidance; (2) stronger judges produce better rubrics; (3) the more rubrics there are, the noisier the evaluation becomes; and (4) the reward function is not the right choice to utilize rubrics. The authors conduct a series of controlled experiments across medical and scientific tasks. Their results suggest that stronger reference guidance and rubric generators do not alleviate reward over-optimization, whereas reducing the number of rubric criteria and replacing weighted-sum rubric rewards with rubric-guided preference rewards can improve training outcomes.
Critical strengths
-
Rather than proposing another rubric-based RL algorithm, the paper tries to investigate factors in rubric construction and reward design that may contribute to reward over-optimization. This is an important and well-motivated problem in rubric-based RL.
-
The empirical observations are interesting. In particular, the results suggest that higher-quality reference answers, stronger rubric generators, and a larger number of rubric criteria do not necessarily lead to better optimization outcomes.
-
The experimental evaluation is extensive, covering both medical and scientific domains. The authors also employ both length-controlled rubric-based evaluations and rubric-free preference evaluations, providing a broader perspective on the observed performance degradation.
Critical weaknesses
-
The experimental evidence does not clearly establish that the observed performance degradation is caused by reward over-optimization.
-
The paper aims to investigate reward over-optimization in rubric-based RL, but there remains a gap between the empirical observations and the claimed phenomenon. The main evidence appears to be that performance under both the gold verifier and the preference evaluator declines as RL training progresses. However, reward over-optimization is typically characterized by an increase in the training reward accompanied by a deterioration in performance under an independent evaluator. The paper does not report the corresponding weak-verifier reward trajectories alongside the gold-verifier evaluation curves. Without demonstrating that the policy increasingly optimizes the training reward while its externally evaluated performance deteriorates, it is difficult to distinguish reward over-optimization from ineffective or unstable RL training.
-
Equation (3) does not, by itself, establish reward over-optimization. The inequality only indicates a discrepancy between the weak and gold verifiers and does not necessarily imply that the weak verifier systematically overestimates response quality. Such discrepancies could also arise from under-crediting. More direct evidence is needed to establish how these disagreements contribute to reward over-optimization during training.
-
-
The experimental results are somewhat surprising, given that rubric-based rewards have been successfully used in LLM training. In several experiments, performance under external evaluators appears to decline throughout training, rather than initially improving before deteriorating. This raises a fundamental question: does it indicate that the training reward is misaligned with the evaluation objective from the outset? In a commonly studied reward over-optimization setting, the proxy reward continues to increase, while the gold reward initially improves and subsequently declines (Gao et al., ICML 2023). Although this pattern is not a necessary condition for reward over-optimization, the consistently declining evaluation performance observed in this paper warrants further investigation into the quality of the training reward and the stability of the RL optimization procedure.
-
The comparison in Sec. 5.4 does not isolate the effect of reward aggregation. The authors aim to investigate the effect of the reward aggregation function while keeping the underlying rubric-level judgments unchanged. However, the comparison between weighted-sum rubric rewards and rubric-guided preference rewards changes more than the aggregation function. Specifically, it changes the judgment procedure, the reward formulation, and the use of pairwise comparisons. In particular, the rubric-guided preference method does not first generate the same binary judgment vector $ \mathbf{g} $ used by the weighted-sum approach. Therefore, the reported improvement demonstrates the effectiveness of rubric-guided preference rewards in this experimental setting, but does not establish that changing the aggregation function alone is responsible for the improvement. Additional controlled comparisons are needed to disentangle these factors.
Questions for authors
-
Could the authors report the weak-verifier reward trajectories alongside the gold-verifier evaluation curves? Does the weak-verifier reward consistently increase while the gold-verifier reward decreases on both the training and evaluation sets? Have the authors considered using a stronger verifier for training as a positive control?
-
Could the authors conduct an experiment in which the model is trained using the gold verifier? If performance still declines under this setting, it would be important to further investigate the RL optimization procedure, training stability, rubric specification, and evaluation protocol.
-
Could the authors provide direct criterion-level evidence supporting the mechanism proposed in Eq. (3), such as an analysis of how disagreements between the weak and gold verifiers evolve during training?
-
Could the authors provide additional ablation studies for Sec. 5.4 to better isolate the effect of reward aggregation? For example, it would be helpful to compare different aggregation functions applied to the same rubric-level judgment vector $ \mathbf{g} $, examine preference-based formulations constructed from the same judgments, and compare rubric-free and rubric-guided preference judgments, i.e., $ P(y \succ y' \mid x) $ and $ P(y \succ y' \mid x, C) $, respectively.
4 comments
Dear Reviewer,
Thank you for your thoughtful feedback and for reading our rebuttal. Since the discussion deadline is approaching, we wanted to check whether our rebuttal has addressed your concerns. If there are any remaining issues, please let us know—we would be happy to provide further clarification or additional experimental results if helpful.
Thank you again for your valuable time!
I thank the authors for their response and the additional experiments. I appreciate their efforts, and I believe the new reward trajectories have partly addressed my concern in W1 regarding the evidence for reward over-optimization. However, several concerns raised in my other weaknesses and questions remain insufficiently addressed. For example, it remains unclear why the gold-verifier performance consistently declines from the very beginning of RL training, rather than initially improving before deteriorating. Furthermore, the authors' response labeled Q2 appears to address my W3 rather than my original Q2 in the Questions for Authors section. I wonder whether some of my questions may have been overlooked. While I appreciate the additional evidence and clarifications, I do not currently see sufficient reasons to increase my score.
Thank you for your response. We would like to clarify that, in Fig. 11, we show that rubric-based RL suffers from verbosity bias, where the model progressively increases response length during training without corresponding improvements in response quality. This explains why the length-controlled gold-verifier performance declines from the beginning of RL training, rather than initially improving before deteriorating. Unlike previous works that directly use the raw gold-verifier score to evaluate performance. Furthermore, prior work [1] also reports reward hacking without observing a deterioration in the raw reward score, suggesting that reward over-optimization does not necessarily manifest as an initial improvement followed by a decline in evaluation performance.
Regarding Q2, we additionally examine the training-verifier reward throughout the RL process and observe that it consistently increases, even as gold-verifier performance declines. This suggests that the observed degradation is not primarily due to optimization instability or failure to optimize the specified reward, but rather to a misalignment between the training reward and the gold-verifier evaluation. These findings provide further evidence that rubric-based RL can over-optimize the training reward without improving actual response quality.
[1] Reward Hacking in Rubric-Based Reinforcement Learning. ICML 2026 AIWILD
Thank you for dedicating your time and expertise to review our submission. Please find our responses below.
Q1. The experimental evidence does not clearly establish that the observed performance degradation is caused by reward over-optimization, and the training stability of RL Fine-tuning. Answer: Thank you for this suggestion. We have updated Figure 2 to include both training and gold verifier reward curves throughout training (Fig ). The results clearly demonstrate that training reward consistently increases while gold verifier performance decreases. Interestingly, rubric-based rewards improve performance under the weak verifier, whereas preference-based rewards lead to a decline under the same verifier. Nevertheless, a performance gap between the weak and strong verifiers persists even with preference-based rewards.
Furthermore, we also provide training reward dynamics in RL Fine-tuning, we observe that training rewards consistently increase across both domains without noticeable signs of instability. These results provide evidence that the observed performance degradation stems from reward over-optimization rather than instability in the RL training process.
Q2. The comparison in Sec. 5.4 does not isolate the effect of reward aggregation. The authors aim to investigate the effect of the reward aggregation function while keeping the underlying rubric-level judgments unchanged. However, the comparison between weighted-sum rubric rewards and rubric-guided preference rewards changes more than the aggregation function. The reported improvement demonstrates the effectiveness of rubric-guided preference rewards in this experimental setting, but does not establish that changing the aggregation function alone is responsible for the improvement. Additional controlled comparisons are needed to disentangle these factors. Answer: Thank you for raising this important concern. We acknowledge that our comparison changes both the reward formulation and the judgment procedure, rather than isolating the effect of reward aggregation alone. However, our primary intention in Section 5.4 is to demonstrate that alternative ways of utilizing rubric information can mitigate reward over-optimization, rather than to attribute the improvement exclusively to reward aggregation.
Our experiments provide empirical evidence for the existence of such alternatives, showing that rubric-guided preference rewards can mitigate reward over-optimization compared to explicit weighted-sum rubric rewards. We believe this finding is meaningful in its own right and motivates further investigation into how different mechanisms for incorporating rubric information affect reward over-optimization.
We have revised Section 5.4 to clarify this objective, avoid attributing the improvements solely to reward aggregation, and acknowledge the need for additional controlled studies to disentangle the underlying mechanisms.