acceptodds
Under review as a conference paper at ICLR 2027

Understanding Reward Over-optimization in Rubric-based Reinforcement Learning

Abstract

Reinforcement Learning (RL) with rubric-based rewards has become a canonical framework for post-training large language models (LLMs) in open-ended domains. However, recent studies have shown that rubric-based RL remains susceptible to reward over-optimization, where the RL-trained policy exploits spurious patterns that satisfy the rubric without necessarily improving the underlying quality of its responses. We first demonstrate the prevalence of this phenomenon in medical and scientific tasks, observing substantial reward over-optimization under both rubric-based and rubric-free evaluation. Motivated by this observation, we systematically investigate the underlying causes through a series of controlled experiments. Our results rule out several potential explanations, showing that neither suboptimal reference guidance or limited coverage of diverse reference responses, nor the use of stronger judges for rubric construction, can fully account for the observed over-optimization. In contrast, we find that factors such as the number of rubrics associated with each prompt, or using preference-based reward with rubric guidance reward, can substantially mitigate reward over-optimization. Taken together, our findings highlight the importance of rubric quantity and reward design in understanding and mitigating reward over-optimization in rubric-based RL

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Alignment, Rubric evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ajSzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates factors contributing to reward over-optimization in rubric-based reinforcement learning (RL) for large language models (LLMs) by examining four hypotheses: (1) suboptimal reference guidance; (2) stronger judges produce better rubrics; (3) the more rubrics there are, the noisier the evaluation becomes; and (4) the reward function is not the right choice to utilize rubrics. The authors conduct a series of controlled experiments across medical and scientific tasks. Their results suggest that stronger reference guidance and rubric generators do not alleviate reward over-optimization, whereas reducing the number of rubric criteria and replacing weighted-sum rubric rewards with rubric-guided preference rewards can improve training outcomes.

Critical strengths
  1. Rather than proposing another rubric-based RL algorithm, the paper tries to investigate factors in rubric construction and reward design that may contribute to reward over-optimization. This is an important and well-motivated problem in rubric-based RL.

  2. The empirical observations are interesting. In particular, the results suggest that higher-quality reference answers, stronger rubric generators, and a larger number of rubric criteria do not necessarily lead to better optimization outcomes.

  3. The experimental evaluation is extensive, covering both medical and scientific domains. The authors also employ both length-controlled rubric-based evaluations and rubric-free preference evaluations, providing a broader perspective on the observed performance degradation.

Critical weaknesses
  1. The experimental evidence does not clearly establish that the observed performance degradation is caused by reward over-optimization.

    • The paper aims to investigate reward over-optimization in rubric-based RL, but there remains a gap between the empirical observations and the claimed phenomenon. The main evidence appears to be that performance under both the gold verifier and the preference evaluator declines as RL training progresses. However, reward over-optimization is typically characterized by an increase in the training reward accompanied by a deterioration in performance under an independent evaluator. The paper does not report the corresponding weak-verifier reward trajectories alongside the gold-verifier evaluation curves. Without demonstrating that the policy increasingly optimizes the training reward while its externally evaluated performance deteriorates, it is difficult to distinguish reward over-optimization from ineffective or unstable RL training.

    • Equation (3) does not, by itself, establish reward over-optimization. The inequality only indicates a discrepancy between the weak and gold verifiers and does not necessarily imply that the weak verifier systematically overestimates response quality. Such discrepancies could also arise from under-crediting. More direct evidence is needed to establish how these disagreements contribute to reward over-optimization during training.

  2. The experimental results are somewhat surprising, given that rubric-based rewards have been successfully used in LLM training. In several experiments, performance under external evaluators appears to decline throughout training, rather than initially improving before deteriorating. This raises a fundamental question: does it indicate that the training reward is misaligned with the evaluation objective from the outset? In a commonly studied reward over-optimization setting, the proxy reward continues to increase, while the gold reward initially improves and subsequently declines (Gao et al., ICML 2023). Although this pattern is not a necessary condition for reward over-optimization, the consistently declining evaluation performance observed in this paper warrants further investigation into the quality of the training reward and the stability of the RL optimization procedure.

  3. The comparison in Sec. 5.4 does not isolate the effect of reward aggregation. The authors aim to investigate the effect of the reward aggregation function while keeping the underlying rubric-level judgments unchanged. However, the comparison between weighted-sum rubric rewards and rubric-guided preference rewards changes more than the aggregation function. Specifically, it changes the judgment procedure, the reward formulation, and the use of pairwise comparisons. In particular, the rubric-guided preference method does not first generate the same binary judgment vector $ \mathbf{g} $ used by the weighted-sum approach. Therefore, the reported improvement demonstrates the effectiveness of rubric-guided preference rewards in this experimental setting, but does not establish that changing the aggregation function alone is responsible for the improvement. Additional controlled comparisons are needed to disentangle these factors.

Questions for authors
  1. Could the authors report the weak-verifier reward trajectories alongside the gold-verifier evaluation curves? Does the weak-verifier reward consistently increase while the gold-verifier reward decreases on both the training and evaluation sets? Have the authors considered using a stronger verifier for training as a positive control?

  2. Could the authors conduct an experiment in which the model is trained using the gold verifier? If performance still declines under this setting, it would be important to further investigate the RL optimization procedure, training stability, rubric specification, and evaluation protocol.

  3. Could the authors provide direct criterion-level evidence supporting the mechanism proposed in Eq. (3), such as an analysis of how disagreements between the weak and gold verifiers evolve during training?

  4. Could the authors provide additional ablation studies for Sec. 5.4 to better isolate the effect of reward aggregation? For example, it would be helpful to compare different aggregation functions applied to the same rubric-level judgment vector $ \mathbf{g} $, examine preference-based formulations constructed from the same judgments, and compare rubric-free and rubric-guided preference judgments, i.e., $ P(y \succ y' \mid x) $ and $ P(y \succ y' \mid x, C) $, respectively.

4 comments

Thank you for dedicating your time and expertise to review our submission. Please find our responses below.

Q1. The experimental evidence does not clearly establish that the observed performance degradation is caused by reward over-optimization, and the training stability of RL Fine-tuning. Answer: Thank you for this suggestion. We have updated Figure 2 to include both training and gold verifier reward curves throughout training (Fig ). The results clearly demonstrate that training reward consistently increases while gold verifier performance decreases. Interestingly, rubric-based rewards improve performance under the weak verifier, whereas preference-based rewards lead to a decline under the same verifier. Nevertheless, a performance gap between the weak and strong verifiers persists even with preference-based rewards.

Furthermore, we also provide training reward dynamics in RL Fine-tuning, we observe that training rewards consistently increase across both domains without noticeable signs of instability. These results provide evidence that the observed performance degradation stems from reward over-optimization rather than instability in the RL training process.

Q2. The comparison in Sec. 5.4 does not isolate the effect of reward aggregation. The authors aim to investigate the effect of the reward aggregation function while keeping the underlying rubric-level judgments unchanged. However, the comparison between weighted-sum rubric rewards and rubric-guided preference rewards changes more than the aggregation function. The reported improvement demonstrates the effectiveness of rubric-guided preference rewards in this experimental setting, but does not establish that changing the aggregation function alone is responsible for the improvement. Additional controlled comparisons are needed to disentangle these factors. Answer: Thank you for raising this important concern. We acknowledge that our comparison changes both the reward formulation and the judgment procedure, rather than isolating the effect of reward aggregation alone. However, our primary intention in Section 5.4 is to demonstrate that alternative ways of utilizing rubric information can mitigate reward over-optimization, rather than to attribute the improvement exclusively to reward aggregation.

Our experiments provide empirical evidence for the existence of such alternatives, showing that rubric-guided preference rewards can mitigate reward over-optimization compared to explicit weighted-sum rubric rewards. We believe this finding is meaningful in its own right and motivates further investigation into how different mechanisms for incorporating rubric information affect reward over-optimization.

We have revised Section 5.4 to clarify this objective, avoid attributing the improvements solely to reward aggregation, and acknowledge the need for additional controlled studies to disentangle the underlying mechanisms.

Dear Reviewer,

Thank you for your thoughtful feedback and for reading our rebuttal. Since the discussion deadline is approaching, we wanted to check whether our rebuttal has addressed your concerns. If there are any remaining issues, please let us know—we would be happy to provide further clarification or additional experimental results if helpful.

Thank you again for your valuable time!

I thank the authors for their response and the additional experiments. I appreciate their efforts, and I believe the new reward trajectories have partly addressed my concern in W1 regarding the evidence for reward over-optimization. However, several concerns raised in my other weaknesses and questions remain insufficiently addressed. For example, it remains unclear why the gold-verifier performance consistently declines from the very beginning of RL training, rather than initially improving before deteriorating. Furthermore, the authors' response labeled Q2 appears to address my W3 rather than my original Q2 in the Questions for Authors section. I wonder whether some of my questions may have been overlooked. While I appreciate the additional evidence and clarifications, I do not currently see sufficient reasons to increase my score.

Thank you for your response. We would like to clarify that, in Fig. 11, we show that rubric-based RL suffers from verbosity bias, where the model progressively increases response length during training without corresponding improvements in response quality. This explains why the length-controlled gold-verifier performance declines from the beginning of RL training, rather than initially improving before deteriorating. Unlike previous works that directly use the raw gold-verifier score to evaluate performance. Furthermore, prior work [1] also reports reward hacking without observing a deterioration in the raw reward score, suggesting that reward over-optimization does not necessarily manifest as an initial improvement followed by a decline in evaluation performance.

Regarding Q2, we additionally examine the training-verifier reward throughout the RL process and observe that it consistently increases, even as gold-verifier performance declines. This suggests that the observed degradation is not primarily due to optimization instability or failure to optimize the specified reward, but rather to a misalignment between the training reward and the gold-verifier evaluation. These findings provide further evidence that rubric-based RL can over-optimize the training reward without improving actual response quality.


[1] Reward Hacking in Rubric-Based Reinforcement Learning. ICML 2026 AIWILD
Reviewer_CKLWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Building on Mahmoud et al. (2026), the paper asks why rubric-based RL over-optimizes. A Qwen2.5-3B-Instruct policy is trained with GRPO against a Qwen2.5-14B-Instruct rubric judge, and GPT-OSS-120B acts as gold verifier on HealthBench-Hard and ResearchQA, reporting a length-controlled rubric score and a win rate over the base model. The authors test four suspected causes: poor reference answers, a weak rubric generator, too many criteria per prompt, and weighted-sum aggregation, which they replace with a rubric-guided pairwise preference reward. They conclude that better references and generators do not resolve over-optimization, whereas fewer criteria and the preference reward substantially mitigate it.

Critical strengths
  1. The decomposition is useful, and its negative results are informative. Isolating four design choices under a fixed policy and verifier makes the comparisons interpretable, and the finding that stronger references and rubric generators do not prevent degradation is directly relevant to anyone building rubric rewards.

  2. Changing the aggregation has a large effect. The preference reward ends more than ten points above the weighted sum on ResearchQA's rubric score and HealthBench's win rate (Fig. 10), far beyond the noise visible at step 0. If it holds up, this is an actionable recipe.

Critical weaknesses
  1. Several figures do not support their conclusions, and the training comparisons appear to be single runs. µ_original in Fig. 4, the evidence for H1, coincides point for point with D1-5 in Fig. 8, the evidence for H3, while µ_original elsewhere is the default run labelled D7-20. Whichever label is wrong, a conclusion falls: either coverage no longer consistently worsens ResearchQA's rubric score, or the main H3 evidence disappears. The H2 text says the strong generator performs best on HealthBench-Hard where Fig. 6 shows the weak one ending higher, and Fig. 9 reports as improvement win rates that never reach parity with the base model. Training reports no seeds or error bars, yet the same base model's step-0 score differs by nearly three points across figures, so several smaller differences, including parts of the H3 and H4 evidence, cannot be told apart from chance. For an empirical paper, it is then unclear which conclusions hold.

  2. The H3 and H4 interventions are confounded, and close prior work is missing. H3 blames the number of criteria, so D1-5 and D7-20 must differ only in count, yet their construction is never described; nor does Eq. 3 imply the effect, since under the normalized sum a fixed per-criterion disagreement rate gives the same expected weak-gold gap at any count. For H4, Eq. 4 leaves the reference policy and judge unspecified, and the win-rate evaluation the Abstract calls rubric-free shares its rubric-guided prompt with the training reward (App. F), favoring H4 on those panels. Related Work calls rubric generation's role largely unexplored, but the uncited Zhang et al. ("Chasing the Tail", ICLR 2026) report that rubrics elicited from great and diverse off-policy responses substantially mitigate over-optimization, in tension with H1. The evidence supports that these recipes degrade less in one run, not that quantity matters more than quality.

Questions for authors
  1. Which label is correct for the curve shared by Fig. 4 and Fig. 8, and are there additional training seeds behind the main comparisons? If not, multi-seed reruns would materially strengthen the final version.
  2. Were D1-5 and D7-20 built by dropping criteria from the same rubrics, by reselecting examples, or by regenerating rubrics? A random criterion-dropping control on the same rubrics would isolate the count.
  3. What are the reference policy and judge in Eq. 4, and does the H4 gain in win rate survive an evaluation prompt independent of the training reward? A comparison with the implicit aggregation of Rubrics as Rewards (Gunjal et al., ICLR 2026), the natural intermediate baseline, would also help.
  4. Which checkpoints produce Fig. 9, and what are the fitted parameters, given that the fitted form assumes zero gain at the start while win rate starts near parity?
3 comments

Thank you for dedicating your time and expertise to review our submission. Please find our responses below.

Q1. μoriginal\mu_{\text{original}} in Fig. 4, the evidence for H1, coincides point for point with D1-5 in Fig. 8, the evidence for The H2 text says the strong generator performs best on HealthBench-Hard where Fig. 6 shows the weak one ending higher, and Fig. 9 reports as improvement win rates that never reach parity with the base model. Training reports no seeds or error bars, yet the same base model's step-0 score differs by nearly three points across figures, so several smaller differences, including parts of the H3 and H4 evidence, cannot be told apart from chance. Answer: Thank you for raising these concerns. We apologize for the lack of clarity in our figures and have made the following corrections and clarifications.

  • Figure 4 (H1) and Figure 8 (H3): In Figure 4, both μoriginal\mu_{\text{original}} and μcoverage\mu_{\text{coverage}} use 1–5 rubric criteria, rather than the default 7–20 criteria used for μoriginal\mu_{\text{original}} in Figure 3. This design allows us to isolate the effect of reference coverage while controlling for the number of criteria, as increasing reference coverage could potentially lead to more generated criteria. Therefore, the overlapping curves in Figures 4 and 8 are expected and do not indicate a labeling error. We have clarified this experimental setup in the revised manuscript to avoid misunderstanding.
  • Figure 6 (H2): We apologize for the incorrect description in the text. The relative performance of the strong and weak rubric generators on HealthBench-Hard was mistakenly reversed. We have corrected the statement to align with Figure 6.
  • Figure 9 (H3): We agree that the base model's win rate against itself should be 50%, and the curves remaining below 50% indicate degradation rather than improvement. Our intended conclusion is that using fewer rubric criteria mitigates the rate of performance degradation compared to using more criteria, rather than necessarily improving performance over the base model. We have updated Figure 9 to include the initial base-model win rate and clarified this distinction in the text.
  • To further strengthen our results, we have added standard deviations across three independent training runs with different random seeds in our revised paper. The results remain consistent with our original findings, further supporting the robustness of our conclusions regarding the proposed hypotheses.

Q2. Eq. 3 does not imply the effect, since under the normalized sum a fixed per-criterion disagreement rate gives the same expected weak-gold gap at any count. For H4, Eq. 4 leaves the reference policy and judge unspecified, and the win-rate evaluation the Abstract calls rubric-free shares its rubric-guided prompt with the training reward (App. F), favoring H4 on those panels. The evidence supports that these recipes degrade less in one run, not that quantity matters more than quality. Answer: We agree that if the expected signed discrepancy is the same across criteria, increasing the number of criteria leaves the expected normalized weak–gold reward gap unchanged. Eq. 3 quantifies the probability of at least one disagreement.

The same mean gap can still produce different GRPO updates. For example, consider two rollouts, (A) and (B), that the gold verifier scores zero on every criterion. With two equally weighted criteria, suppose the weak verifier incorrectly credits both rollouts on one criterion, giving normalized rewards of ((0.50,0.50)). Now add two criteria on which the weak verifier incorrectly credits only (B). The normalized weak rewards become ((0.25,0.75)), with the same mean weak–gold gap of (0.50). In the first case, both rollouts receive zero reward advantage. In the second, GRPO favors (B). This illustrates the result highlighted by [1], that criteria affect group-relative advantages when their scores vary across rollouts.

Q3. Related Work calls rubric generation's role largely unexplored, but the uncited Zhang et al. ("Chasing the Tail", ICLR 2026) report that rubrics elicited from great and diverse off-policy responses substantially mitigate over-optimization, in tension with H1. Answer: Thank you for pointing out this omission. We have revised the Related Work section to cite Zhang et al. (ICLR 2026), qualify our claim that rubric generation is largely unexplored, and discuss how their findings relate to H1. Zhang et al. mitigate over-optimization by iteratively refining rubrics to distinguish among strong, diverse off-policy responses.

Q4. Were D1-5 and D7-20 built by dropping criteria from the same rubrics, by reselecting examples, or by regenerating rubrics? Answer: Thank you for raising this question. We clarify that D1−5\mathcal{D}_{1-5} and D7−20\mathcal{D}_{7-20} are constructed by regenerating rubrics with different numbers of criteria, rather than dropping criteria from existing rubrics or reselecting examples. The corresponding generation prompts are provided in Appendix E. Notably, our experiments vary only the number of rubric criteria while maintaining the samerubric-generation protocol. This controlled setup minimizes potential confounding factors.

Q5. What are the reference policy and judge in Eq. 4, and does the H4 gain in win rate survive an evaluation prompt independent of the training reward? A comparison with the implicit aggregation of Rubrics as Rewards (Gunjal et al., ICLR 2026) would also help. Answer: Thank you for raising this question. In Eq. 4, the reference policy is original response used in RaR-Science/Medicine, and the judge remains Qwen2.5-14B-Instruct. Regarding the suggested baseline, our rubric-based preference comparison is conceptually similar to the implicit aggregation approach proposed in [2], as both replace explicit aggregation of criterion-level scores with preference-based judgments. Thus, our H4 experiments already investigate a closely related reward formulation.


[1] Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR. Arxiv 2605.20164.
[2] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. ICLR 2026.

Dear Reviewer,

Thank you for your thoughtful feedback and for reading our rebuttal. Since the discussion deadline is approaching, we wanted to check whether our rebuttal has addressed your concerns. If there are any remaining issues, please let us know—we would be happy to provide further clarification or additional experimental results if helpful.

Thank you again for your valuable time!

Reviewer_GyW2OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies reward over-optimization in rubric-based RL. The experimental setup uses Qwen2.5-3B-Instruct as the policy, Qwen2.5-14B-Instruct as the training verifier, and GPT-OSS-120B as the stronger evaluation verifier. The authors train on RaR-Medicine and RaR-Science and evaluate on HealthBench-Hard and ResearchQA using length-controlled rubric scores and preference-based evaluation. The authors investigate four hypotheses concerning reference guidance, rubric-generator strength, the number of rubric criteria, and reward formulation. The results suggest that stronger reference responses, broader reference coverage, and a stronger rubric generator do not consistently mitigate over-optimization. In contrast, using fewer rubric criteria and replacing weighted criterion-satisfaction rewards with rubric-guided pairwise preference rewards reduce over-optimization in the studied settings.

Critical strengths
  1. The paper tackles the important problem of reward over-optimization in rubric-based RL.

  2. The paper is well presented and easy to follow.

Critical weaknesses
  1. Limited analysis of judge-specific preferences. Each combination of judge and prompt template may have its own preferences. For example, some judges may prioritize completeness, while others place greater importance on conciseness. The experiments primarily use Qwen2.5-14B-Instruct as the training verifier and GPT-OSS-120B as the evaluation verifier, so the findings may depend on the preferences of this particular pair. Ablations with additional judges, an ensemble of judges, or an analysis of the current judges’ preferences would help clarify this concern.

  2. Narrow domain coverage. The experiments focus only on medicine and science, which provides limited support for broader claims about reward over-optimization in rubric-based RL.

  3. Limited statistical support for hypothesis testing. Although the paper reports repeated evaluation with different random seeds, it does not clearly provide formal statistical tests or uncertainty estimates across independent training runs. These would help assess how reliably the results support or contradict the proposed hypotheses.

  4. Limited comparisons of rubric-generation methods. The paper studies rubric generation conditioned on prompts and reference responses, including multiple-reference guidance. However, it does not compare with alternative rubric-generation and refinement methods such as [1, 2]. These comparisons would help assess whether the findings extend beyond the rubric-generation approach used in the paper.

References

[1] Shen et al. Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks.

[2] Liu et al. OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment.

Questions for authors
  1. Figures 8–9 show that fewer rubric criteria mitigate over-optimization in the studied setting. Could the authors provide evidence that this effect is driven by fewer exploitable disagreements between verifiers, as proposed in 5.3, rather than changes in criterion content or difficulty?

  2. The introduction describes changing the aggregation scheme while keeping criterion-level judgments identical, whereas Appendix F describes a holistic pairwise judgment. Could the authors clarify whether the comparison changes only reward aggregation or also the judging procedure?

  3. GPT-OSS-120B is itself a proxy evaluator rather than ground truth. Could the authors justify treating it as a gold verifier?

  4. If rubric criteria can explicitly encourage conciseness and penalize unnecessary verbosity, why is additional length control needed during evaluation?

  5. What is the performance gap between Qwen2.5-14B-Instruct and GPT-OSS-120B specifically as verifiers on these tasks?

  6. Could the benefit of using fewer rubric criteria depend on the limited capacity of Qwen2.5-3B-Instruct rather than the number of criteria itself?

  7. Could the observed results primarily reflect the use of Qwen2.5-14B-Instruct as a relatively weak training verifier? Would the same conclusions hold with a stronger training verifier?

  8. Could the authors add the training verifier’s reward curve alongside the gold verifier’s curves in Figure 2? This would help establish whether training reward increases while evaluation performance decreases.

  9. Could the authors clarify the choice of Qwen2.5-3B-Instruct rather than a more recent Qwen3 model? Experiments across additional model families and scales would help assess the generality of the findings.

3 comments

Thank you for dedicating your time and expertise to review our submission. Please find our responses below.

Q1. Limited analysis of judge-specific preferences. Answer: Thank you for raising this concern. Our study focuses on rubric-based rewards, which were proposed to reduce the implicit preference biases of LLM judges [1]. Under this protocol, the verifier only assesses whether a response satisfies each predefined, concrete, and human-interpretable criterion, while the rubric weights determine the relative importance of each criterion. The verifier does not provide preference judgments or overall response scores. Furthermore, we include an independent preference-based evaluation to assess whether improvements in rubric rewards reflect genuine improvements in response quality, rather than merely exploiting the rubric-based evaluation procedure at the expense of performance under a difference evaluation procedure, providing a broader perspective on the observed performance degradation as acknowledged by Reviewer ajSz.

Q2. Narrow domain coverage. Answer: Thank you for raising this concern. We would like to clarify that our experimental evaluation follows established protocols from an extensive body of prior work on rubric-based rewards, which primarily focuses on medical and scientific domains [1,2,3,4]. Furthermore, as reviewer ajSz also acknowledges, our evaluation is extensive, covering multiple tasks across both medical and scientific domains. We therefore believe that our experimental coverage is sufficiently broad to support our findings within these established settings.

Q3. Limited statistical support for hypothesis testing. Answer: Thank you for your suggestion. We have added standard deviations across three independent training runs with different random seeds in our revised paper. The results remain consistent with our original findings, further supporting the robustness of our conclusions regarding the proposed hypotheses.

Q4. Limited comparisons of rubric-generation methods. Answer: Thank you for your suggestion. We would like to clarify that our study follows the established protocol widely adopted in prior rubric-based RL studies, where rubrics are generated conditioned on prompts and reference responses. Our primary objective is not to compare rubric-generation methods, but rather to systematically investigate how these commonly adopted rubric-generation practices contribute to reward over-optimization during RL training.

Furthermore, the multi-reference-guided rubric generation proposed in [1] is closely related to the approach we already investigate in Section 5.1, where we examine the effects of conditioning rubric generation on multiple reference responses. Therefore, our experiments already cover an important aspect of the alternative methods suggested.

Q5. Evidence that fewer rubric criteria can mitigate reward over-optimization rather than changes in criterion content or difficulty. Answer: Thank you for your question. As discussed in Section 5.3, we provide a theoretical motivation for why increasing the number of rubric criteria introduces more opportunities for exploitable disagreements between the weak and gold verifiers. Our empirical results in Figures 8–9 consistently show that reducing the number of rubric criteria mitigates reward over-optimization, supporting the mechanism proposed in Section 5.3.

We acknowledge that criterion content and difficulty may also influence the extent of verifier disagreement. However, our experiments vary ONLY the number of rubric criteria while maintaining the same domains, underlying task distribution (Appendix Section D), and rubric-generation protocol, thereby minimizing potential confounding factors associated with differences in criterion content and difficulty.

Q6. Clarify whether the comparison changes only reward aggregation or also the judging procedure? Answer: Thank you for raising this concern. We clarify that the comparison involves a change in the judging procedure, rather than merely the reward aggregation scheme. Specifically, rubric-based rewards aggregate criterion-level satisfaction judgments, whereas preference-based rewards use a holistic pairwise comparison to assess response quality. Therefore, the two approaches differ in both how responses are evaluated and how rewards are assigned.

Q7. GPT-OSS-120B is itself a proxy evaluator rather than ground truth. Answer: Thank you for raising this concern. Following prior work, we use GPT-OSS-120B as a stronger independent evaluator to assess underlying improvements, rather than treating it as a replacement for human ground truth. Prior studies have also demonstrated its strong performance as an LLM judge, supporting its use as a reference evaluator.

Importantly, using a relatively weaker training verifier is intentional, as our study aims to investigate reward over-optimization arising from exploitable verifier limitations. This setting allows us to systematically analyze how the policy exploits imperfections in the reward signal and evaluate potential mitigation strategies. While stronger training verifiers may reduce such vulnerabilities, whether they eliminate over-optimization remains an open question beyond the scope of our study.

Q8. If rubric criteria can explicitly encourage conciseness and penalize unnecessary verbosity, why is additional length control needed during evaluation Answer: Prior work on criterion-fulfillment benchmarks and reward over-optimization, has shown that more verbose responses tend to receive higher scores because they provide more opportunities to satisfy rubric criteria [1,2,3]. Even when rubrics explicitly encourage conciseness, this inherent bias may persist, motivating the need for length-controlled evaluation to disentangle genuine improvements in response quality from gains driven by verbosity. Additionally, length-controlled evaluation has also been adopted in prior rubric-based benchmarks [4,5], further validating our evaluation protocol.

Q9. Could the authors clarify the choice of Qwen2.5-3B-Instruct rather than a more recent Qwen3 model? Answer: Thank you for raising this concern. The Qwen2.5 model family has been extensively used in prior rubric-based RL research [1,2,7], motivating our choice to follow established experimental protocols. While additional model families and scales could strengthen generality, attributing our findings to Qwen2.5-3B-Instruct's limited capacity remains speculative, without clear theoretical motivation or empirical evidence. To the best of our knowledge, prior work has not identified such capacity limitations in Qwen2.5 that would explain our observations.

Q11. Could the authors add the training verifier’s reward curve alongside the gold verifier’s curves in Figure 2? Answer: Thank you for your suggestion. We have updated Figure 2 to include both training and gold verifier reward curves throughout the training process. The results clearly demonstrate that the training reward consistently increases while gold verifier performance declines under rubric-based evaluation, providing direct evidence of reward over-optimization. Interestingly, rubric-based rewards improve performance under the weak verifier, whereas preference-based rewards lead to a decline under the same verifier. Nevertheless, a performance gap between the weak and strong verifiers persists even with preference-based rewards.


[1] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. ICLR 2026.
[2] Reward Hacking in Rubric-Based Reinforcement Learning. Arxiv 2605.12474.
[3] SERPO: Self-Evolving Rubric Policy Optimization. Arxiv 2607.26873.
[4] Enhancing Rubric-based RL via Self-Distillation. Arxiv 2607.18082.
[5] MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcome. ICLR 2026.
[6] HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats. OpenAI report.
[7] Checklists Are Better Than Reward Models For Aligning Language Models. NeurIPS 2025 Spotlight.

Dear Reviewer,

Thank you for your thoughtful feedback and for reading our rebuttal. Since the discussion deadline is approaching, we wanted to check whether our rebuttal has addressed your concerns. If there are any remaining issues, please let us know—we would be happy to provide further clarification or additional experimental results if helpful.

Thank you again for your valuable time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.