- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses a key limitation of traditional rubric-based rewards: they often fail to effectively distinguish rollouts within the same group, resulting in insufficient training signals for GRPO. To address this issue, the authors propose DivR, which uses an LLM judge to identify meaningful behavioral differences among current rollouts and construct rewards based on these differences.
Critical strengths
1.The paper provides extensive experiments involving multiple models.
2.The paper is very well written and easy to follow, particularly the experimental section, which clearly addresses each research question (RQ).
Critical weaknesses
-
Existing studies, such as OnlineRubrics, OpenRS, and BRPO, have already explored constructing rewards from rollout differences or using relative comparisons. The main novelty of DivR lies in combining Divergence Extraction with Shared-baseline Relative Scoring. The authors should further clarify the technical novelty of this combination compared with existing approaches.
-
The scale of the policy models evaluated is limited. The paper only uses Qwen2.5-3B for RL training, without investigating whether the proposed method provides similar benefits for larger models, such as 7B, 14B, or beyond. This limits the generalizability of the findings.
Questions for authors
see the weaknesses
4 comments
A Gentle Follow-up on Our Responses
Dear reviewer JZiz:
Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.
We sincerely look forward to your feedback.
Thanks for your careful rebuttal. I will make my decision soon based on other reviewers' replies.
Thank you again for your thoughtful feedback and for taking the time to review our rebuttal.
In response to your concerns, we have provided further clarifications on DivR's technical novelty and its distinctions from existing relative reward methods, including OnlineRubrics, OpenRS, and BRPO. We have also added a new experiment using Qwen2.5-7B to investigate whether the benefits of DivR extend to larger policy models.
In response to the concerns raised by other reviewers, we have also made several further clarifications and improvements:
- Explained the relationship between absolute scoring and reward collapse;
- Clarified the role of the shared observed baseline;
- Added fairer controlled comparison;
- Discussed reward hacking and groups with limited meaningful behavioral variation;
- Strengthened the ablation studies;
- Refined the description of the causal intervention experiment and clarified the use of equations.
We are also encouraged that another reviewer (927m) has found our responses helpful and acknowledged that several of their concerns have been addressed.
As the rebuttal period is approaching its end, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your concerns. If you find the additional clarifications and experimental results convincing, we would be grateful if you could consider raising your evaluation score accordingly.
Thank you once again for your time and consideration. We sincerely look forward to your feedback.
W1: The Novelty Boundary Relative to Existing Methods
Thank you for raising this concern. DivR's core contribution is a training-oriented guideline: the rubric should match the capability of the model, since only then does it provide a better optimization signal. OnlineRubrics and OpenRS also evolve rubrics, but neither states a criterion for how the rubric should evolve.
Concretely:
OnlineRubrics extracts criteria from current–control response pairs and appends them to a fixed absolute rubric. It does improve rubric coverage, but it does not ensure that the resulting criteria distinguish the current rollout group: criteria elicited from a single local pair need not lie on a dimension that varies across the whole group, and once appended to an absolute rubric they are applied to every response in the same way. Two observations support this. First, the elicited-only reward is negatively correlated with the independent HealthBench ordering, that is, the direction these criteria induce does not track quality; we suspect this is because pairwise elicitation tends to phrase criteria as behaviors to avoid (59% of the elicited criteria), and such criteria are already satisfied by almost every response in a group.
Second, OnlineRubrics answers "where do responses diverge" and "which response is better" in a single step, so a difference in style alone is still assigned a quality direction, in 31.8% of comparisons against 17.3% for DivR.
DivR separates these two steps: divergence is extracted at the group level with no direction attached, and a direction is assigned only where the divergence corresponds to a quality difference, measured against the observed high-frequency baseline.
BRPO compares each response with a sampled anchor, but does not explicitly identify the behavioral dimensions underlying the comparison.
OpenRS is based on BRPO. It generates criteria and weights separately for each candidate–anchor pair, so a shared anchor fixes the comparison target but not the comparison dimensions: with the same anchor, the same dimension can carry different weights, or be dropped, across candidates.
The choice of anchor also injects substantial uncertainty: switching to another anchor can change the ordering considerably.
Its rank consistency after resampling the anchor is also much lower than DivR's:
DivR instead extracts one set of dimensions, weights, and observed baselines from the complete rollout group and holds them fixed while scoring every response. It also separates divergence discovery from quality judgment, allowing behaviorally different responses to receive equal scores when the difference is not quality-relevant.
Therefore, DivR is not simply a combination of existing components; it constructs a capability-matched, group-shared reward coordinate specifically for group-relative credit assignment. We have clarified this distinction and included the detailed comparisons in Appendix B.3.
W2: Limited Experimental Scope
Thank you for raising this concern. DivR is a general method, so in principle it applies to policies of any scale. We agree, however, that training larger models would make the results more convincing. We have therefore added an experiment with a Qwen2.5-7B policy:
On groups that contain little meaningful variation, our response to the previous question already adds the explanation and an example. We hope this addresses the concern.