acceptodds
Under review as a conference paper at ICLR 2027

DivR: Grounding LLM Reinforcement Learning in Rollout Behavioral Divergence

Abstract

LLM-as-a-judge has become a practical approach for providing rewards in reinforcement learning on open-ended tasks, where correctness is difficult to verify automatically. A common approach is to let a judge LLM observe current rollouts and generate quality rubrics for scoring them. We argue that rubrics designed for evaluation should not be directly reused for training: evaluation rubrics only need to judge the absolute quality of an individual response, whereas training rewards must also distinguish the relative quality of rollouts within the same group. We propose Divergence-Point Reward (DivR), which identifies quality-relevant behavioral divergences among current rollouts and scores responses relative to their peers. Across HealthBench and WritingBench, DivR outperforms both judge-generated and dataset-specific rubric rewards, improving over GenR by 11.6–21.8% under independent HealthBench judges and achieving the strongest ranking across judges on WritingBench. Mechanistic analyses show that divergence-grounded reward dimensions are more reliable and preserve informative group-relative signals as the policy improves. Our code is available at https://anonymous.4open.science/r/divr_public-FA48.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, large language models, rubric-based rewards, adaptive reward design, LLM-as-a-judge, GRPO, open-ended generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JZizOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses a key limitation of traditional rubric-based rewards: they often fail to effectively distinguish rollouts within the same group, resulting in insufficient training signals for GRPO. To address this issue, the authors propose DivR, which uses an LLM judge to identify meaningful behavioral differences among current rollouts and construct rewards based on these differences.

Critical strengths

1.The paper provides extensive experiments involving multiple models.

2.The paper is very well written and easy to follow, particularly the experimental section, which clearly addresses each research question (RQ).

Critical weaknesses
  1. Existing studies, such as OnlineRubrics, OpenRS, and BRPO, have already explored constructing rewards from rollout differences or using relative comparisons. The main novelty of DivR lies in combining Divergence Extraction with Shared-baseline Relative Scoring. The authors should further clarify the technical novelty of this combination compared with existing approaches.

  2. The scale of the policy models evaluated is limited. The paper only uses Qwen2.5-3B for RL training, without investigating whether the proposed method provides similar benefits for larger models, such as 7B, 14B, or beyond. This limits the generalizability of the findings.

Questions for authors

see the weaknesses

4 comments

W1: The Novelty Boundary Relative to Existing Methods

Thank you for raising this concern. DivR's core contribution is a training-oriented guideline: the rubric should match the capability of the model, since only then does it provide a better optimization signal. OnlineRubrics and OpenRS also evolve rubrics, but neither states a criterion for how the rubric should evolve.

Concretely:

  1. OnlineRubrics extracts criteria from current–control response pairs and appends them to a fixed absolute rubric. It does improve rubric coverage, but it does not ensure that the resulting criteria distinguish the current rollout group: criteria elicited from a single local pair need not lie on a dimension that varies across the whole group, and once appended to an absolute rubric they are applied to every response in the same way. Two observations support this. First, the elicited-only reward is negatively correlated with the independent HealthBench ordering, that is, the direction these criteria induce does not track quality; we suspect this is because pairwise elicitation tends to phrase criteria as behaviors to avoid (59% of the elicited criteria), and such criteria are already satisfied by almost every response in a group.

    OnlineRubrics rewardSpearman r with independent HealthBench ordering
    static+0.073
    static + elicited−0.006
    elicited only−0.287

    Second, OnlineRubrics answers "where do responses diverge" and "which response is better" in a single step, so a difference in style alone is still assigned a quality direction, in 31.8% of comparisons against 17.3% for DivR.

    MethodDivROnlineRubrics
    Misattribution rate on style-only pairs0.1730.318

    DivR separates these two steps: divergence is extracted at the group level with no direction attached, and a direction is assigned only where the divergence corresponds to a quality difference, measured against the observed high-frequency baseline.

  2. BRPO compares each response with a sampled anchor, but does not explicitly identify the behavioral dimensions underlying the comparison.

  3. OpenRS is based on BRPO. It generates criteria and weights separately for each candidate–anchor pair, so a shared anchor fixes the comparison target but not the comparison dimensions: with the same anchor, the same dimension can carry different weights, or be dropped, across candidates.

    DimensionSettings across candidates
    EmpathyR0/R1/R5: core (w=5); R3/R4/R6: important (w=2); R7: not evaluated
    Language qualityR3: core (w=5); R4/R5/R6: veto; others: not evaluated
    # CriteriaR0: 4; others: 5

    The choice of anchor also injects substantial uncertainty: switching to another anchor can change the ordering considerably.

    AnchorR0R1R2R3R4R5R6R7
    R2rank5rank2rank8rank6rank1rank2rank4rank7
    R7rank1rank8rank7rank2rank3rank4rank5rank6

    Its rank consistency after resampling the anchor is also much lower than DivR's:

    MethodRank consistency
    OpenRS0.663
    DivR0.784

DivR instead extracts one set of dimensions, weights, and observed baselines from the complete rollout group and holds them fixed while scoring every response. It also separates divergence discovery from quality judgment, allowing behaviorally different responses to receive equal scores when the difference is not quality-relevant.

Therefore, DivR is not simply a combination of existing components; it constructs a capability-matched, group-shared reward coordinate specifically for group-relative credit assignment. We have clarified this distinction and included the detailed comparisons in Appendix B.3.

W2: Limited Experimental Scope

Thank you for raising this concern. DivR is a general method, so in principle it applies to policies of any scale. We agree, however, that training larger models would make the results more convincing. We have therefore added an experiment with a Qwen2.5-7B policy:

Methodqwen3.7-plusdeepseek
GenR0.46100.3830
DivR0.47680.4015

On groups that contain little meaningful variation, our response to the previous question already adds the explanation and an example. We hope this addresses the concern.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Responses

Dear reviewer JZiz:

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Thanks for your careful rebuttal. I will make my decision soon based on other reviewers' replies.

Thank you again for your thoughtful feedback and for taking the time to review our rebuttal.

In response to your concerns, we have provided further clarifications on DivR's technical novelty and its distinctions from existing relative reward methods, including OnlineRubrics, OpenRS, and BRPO. We have also added a new experiment using Qwen2.5-7B to investigate whether the benefits of DivR extend to larger policy models.

In response to the concerns raised by other reviewers, we have also made several further clarifications and improvements:

  • Explained the relationship between absolute scoring and reward collapse;
  • Clarified the role of the shared observed baseline;
  • Added fairer controlled comparison;
  • Discussed reward hacking and groups with limited meaningful behavioral variation;
  • Strengthened the ablation studies;
  • Refined the description of the causal intervention experiment and clarified the use of equations.

We are also encouraged that another reviewer (927m) has found our responses helpful and acknowledged that several of their concerns have been addressed.

As the rebuttal period is approaching its end, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your concerns. If you find the additional clarifications and experimental results convincing, we would be grateful if you could consider raising your evaluation score accordingly.

Thank you once again for your time and consideration. We sincerely look forward to your feedback.

Reviewer_927mOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors introduce DivR, a reward construction method for reinforcement learning on open-ended language generation tasks. The authors argue that evaluation-oriented rubrics, which primarily assess the absolute quality of individual responses, may fail to provide sufficiently discriminative reward signals for group-relative policy optimization. To address this issue, DivR introduces a two-stage reward construction procedure. First, an LLM judge identifies quality-relevant behavioral divergences among responses sampled from the current policy and constructs a set of divergence dimensions with importance weights and shared baselines defined by the most frequently observed behaviors. Second, each response is evaluated relative to these baselines, and the resulting dimension-wise scores are aggregated into a scalar reward for GRPO training.

Critical strengths
  1. The paper is well-written and easy to follow;

  2. The experiments demonstrate promising improvements over conventional rubric-based rewards.

Critical weaknesses
  1. The core methodological ideas overlap substantially with existing approaches, OpenRS [1] and OnlineRubrics [2]. OpenRS already employs explicit semantic difference extraction, adaptive weighted evaluation criteria, and criterion-wise relative scoring with a five-level scale. EvoRubric has also investigated dynamically updating reward criteria to maintain discriminative supervision during policy optimization.

  2. In Table 3(a), removing the shared baseline slightly improves performance under the Qwen3-max evaluator (0.326 versus 0.318), while DivR achieves relatively modest improvements under DeepSeek and Gemini. Without repeated runs or uncertainty estimates, these differences are insufficient to establish that the shared-baseline mechanism is consistently beneficial.

  3. The targeted intervention experiment provides positive evidence that modifying DivR-identified divergence locations can affect independently evaluated response quality. However, the interventions differ from the controls in both the edited locations and the nature of the edits: the former involve changes toward high-frequency behaviors, whereas the latter primarily involve stylistic rewrites. Consequently, the experiment does not fully isolate the causal contribution of the identified divergence locations.

  4. The formulation in Equations (2)&(3) introduces quantities such as Kπ(c)K_\pi(c), Δπ(c)\Delta_\pi(c), and Kr(c)K_r(c), but their mathematical definitions and estimators are not specified precisely.

[1] Jia, R., Yang, Y., Wang, W., Wu, Y., Gai, Y., Tao, S., ... & Jiang, G. (2026). Open rubric system: Scaling reinforcement learning with pairwise adaptive rubric. arXiv preprint arXiv:2602.14069.

[2] Rezaei, M., Vacareanu, R., Wang, Z., Wang, C., Liu, B., He, Y., & Akyürek, A. F. (2025). Online rubrics elicitation from pairwise comparisons. arXiv preprint arXiv:2510.07284.

Questions for authors

Please see the Weaknesses

5 comments

W1: Differences from OpenRS, OnlineRubrics, and Other Relative Reward Methods

Thank you for pointing us to the closest related work; we agree these methods deserve a fuller discussion. DivR's core contribution is a training-oriented guideline: the rubric should match the capability of the model, since only then does it provide a better optimization signal. OnlineRubrics and OpenRS also evolve rubrics, but neither states a criterion for how the rubric should evolve.

Methodologically, DivR extracts one shared set of dimensions, weights, and baseline from the group, so that every rollout is scored against the same reference matched to the policy's capability. Both baselines evolve rubrics via pairwise comparison, but their shortcomings prevent rubrics from matching model capability.

(a) OnlineRubrics compares current-policy and control-policy responses and appends differences not yet covered by the existing rubric as new criteria to a fixed absolute rubric. Because these criteria come from a local pair, they need not be discriminative across the whole rollout group. OnlineRubrics is already a training baseline in Table 2, and it trails DivR:

Methoddeepseekgemini
OnlineRubrics0.1540.302
DivR0.3180.433

A further analysis finds that 59% of its elicited criteria are phrased as behaviors to avoid, which most responses already satisfy. As shown in Appendix B.3, adding elicited criteria lowers rather than raises the agreement between the reward and the independent HealthBench ordering:

OnlineRubrics rewardSpearman r with independent HealthBench ordering
staticr=+0.073
static + elicitedr=-0.006
elicited onlyr=-0.287

OnlineRubrics settles both at once: a criterion written from a single current–control comparison already carries a quality direction. On responses that differ only in style, OnlineRubrics assigns a quality direction in 31.8% of the comparisons, against 17.3% for DivR. Separating the discovery of an in-group divergence from the judgment of its direction therefore avoids imposing a reward direction on meaningless differences.

MethodDivROnlineRubrics
Misattribution rate on style-only pairs0.1730.318

(b) OpenRS samples one response as the anchor, forms pairs with all other responses, and generates a rubric, weights, and scores separately for every pair. A shared anchor fixes the object of comparison but neither the dimensions nor the scale. With the anchor fixed at R2, the dimension "empathy" carries weight 5 for some candidates and 2 for others; "language quality" varies likewise. Scalar rewards within one group can therefore summarize different semantic dimensions and scoring rules, while GRPO still centers them when computing advantages.

DimensionSettings across candidates
EmpathyR0/R1/R5: core (w=5); R3/R4/R6: important (w=2); R7: not evaluated
Language qualityR3: core (w=5); R4/R5/R6: veto; others: not evaluated
# CriteriaR0: 4; others: 5

The anchor choice also injects substantial uncertainty: switching to another anchor can change the ordering considerably. For one group, switching the anchor from R2 to R7 reorders most responses, whereas DivR avoids this by fixing the high-frequency behavior as the baseline.

MethodRank consistency
OpenRS0.663
DivR0.784
AnchorR0R1R2R3R4R5R6R7
anchor: R2rank5rank2rank8rank6rank1rank2rank4rank7
anchor: R7rank1rank8rank7rank2rank3rank4rank5rank6

By contrast, DivR applies the same criteria, weights, and comparison anchor to every response in the group, so that scores reflect the same set of quality differences and can be used directly for group-relative credit assignment.

DivR is therefore not just another rubric-evolution mechanism: it guides reward construction with the quality-relevant behavioral differences learnable under the current policy.

In addition, pairwise construction incurs more API calls and longer training time.

We have expanded the related-work section accordingly and added a detailed comparison in Appendix B.3.

W2: Instability of the Shared-Baseline Ablation Result

Thank you. Full DivR is better under both cross-family evaluators: 0.227 vs. 0.214 under DeepSeek and 0.349 vs. 0.338 under Gemini. The w/o baseline variant is slightly higher under Qwen3-max (0.326 vs. 0.318), yet averaged over the three evaluators full DivR is still slightly higher (0.298 vs. 0.293).

We understand that this may look like a marginal effect. We therefore reran the no-baseline ablation with a different seed:

Methodqwen3.7-plusdeepseek-v4-flash
no_baseline0.19610.1846
DivR0.23070.2273

With this run the advantage of DivR over the ablation is clearer. We hope this resolves the concern.

W3: Issues with the Experimental Setup of the Causal Intervention Experiment

Thank you for identifying this problem. Our control does change two variables at once, which is not a rigorous design. In practice, however, the intervention arm alone already establishes our claim, namely that the dimensions identified by DivR are causally related to response quality.

DivR does not search for the text locations with the largest global effect on quality; it searches for behavioral dimensions that satisfy two conditions at once:

  1. the dimension actually varies within the current rollout group;
  2. intervening along it affects independently evaluated quality.

A dimension can be very important for quality, but if all rollouts behave identically on it, it cannot produce a reward with a distribution, nor provide credit assignment for the current GRPO group.

Group-level divergence extraction guarantees the first condition; the intervention experiment tests the second. In the intervention arm, a divergence point is rewritten toward the high-frequency behavior and then scored. The mean score difference from the original response is 0.055 with p<0.001, which is significant. The original experiment therefore suffices to support our claim that the identified dimensions are causally related to response quality.

The additional analysis is consistent with this reading: the intervention effect increases monotonically with DivR's importance weight.

Weight# divergence pointsIntervention − control
537+0.069
446+0.062
345+0.045
222+0.033

We have revised section 5.4 to reduce this possible misunderstanding.

W4: Unclear Definitions in Equations (2) and (3)

Thank you for the reminder. The expositions need a clearer definition. We define $K_\pi(c)$ as the policy's expected latent capability on dimension $c$, $\Delta_\pi(c)$ as its typical rollout-level sampling variation, and $K_r(c)$ as the capability threshold required to satisfy rubric $r$. We also clarify that these quantities are conceptual rather than explicitly estimated by DivR.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Responses

Dear reviewer 927m:

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_EnkBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DivR constructs rewards around the behavioral dimensions that distinguish the current rollouts. A judge identifies differences within the group and scores each response relative to the most common observed behavior. The motivation is that rubrics useful for evaluation may provide little training signal when responses receive similar absolute scores. Criteria that initially distinguish responses can also stop providing signal as training progresses.

The experiments use Qwen2.5-3B as the policy, with Qwen3-8B and Qwen3-max as training judges, across medical QA and writing tasks, with Gemini and DeepSeek as additional evaluators.

Critical strengths
  • Concrete and relevant failure diagnosis. The saturation examples show how coarse rubric scores can discard meaningful distinctions between responses.
  • The medical example is particularly useful: the absolute rubric assigns identical scores, while DivR captures differences within the same group.
  • The equal-score preference experiment supports the motivation. A preference counts as stable only when the same response wins all four comparisons, including reversed presentation orders.
  • Substantially fewer saturated rewards in the reported signal analysis: 2.4% of DivR rollouts reach the ceiling, compared with 41.8% for GenR. Groups with zero reward-derived training signal also drop from 10.8% to 2.5%.
Critical weaknesses
  • The distinction between training and evaluation rubrics feels overstated. Absolute scoring does not inherently cause reward collapse. A sufficiently discriminative absolute score can still provide useful GRPO advantages.
  • The role of the observed baseline needs a clearer explanation. Subtracting a common baseline alone cannot improve GRPO, since group centering already removes that offset. Any benefit must come from how the reference changes the judge's judgments.
  • The main comparison changes more than the source of the criteria. GenR uses binary item grades, while DivR uses five levels and additional instructions for invalid or avoidant responses. This makes the performance and criterion-liveness differences harder to attribute specifically to divergence grounding.
  • Potential reward-hacking concern. Keeping rewards discriminative could encourage optimization of minor or judge-specific preferences once meaningful quality differences become small. More reward variation alone does not establish a better learning signal.
  • Limited experimental scope: one 3B policy, one epoch, and two task domains. Multiple judges strengthen the evaluation, but leave open whether the gains persist with larger policies, longer training, or groups containing little meaningful variation.
Questions for authors
  1. How much of the improvement remains when GenR uses five-level grading and the same instructions for invalid or avoidant responses?
  2. What happens when a rollout group contains few or no meaningful behavioral differences? Can DivR avoid inventing distinctions simply to maintain reward variation?
  3. As training progresses, do the remaining reward differences continue to track independently measured quality, rather than increasingly reflecting judge preferences?
  4. Do the gains persist with larger policies and longer training?
4 comments

W1: The Relationship Between Reward Collapse and Absolute Scoring

Thank you for raising this concern. Absolute scoring does make reward collapse more likely: as shown in Fig. 1(b), 16.7% and 10.8% of rollout groups under the two absolute-scoring methods receive an identical reward and therefore no gradient, and this is largely because more than 40% of responses hit the score ceiling under both methods. DivR reaches 2.5% and 2.4% on these two measures, which substantially alleviates the problem.

MethodGroups with zero gradient (%)Rollouts at ceiling (%)
ConR16.742.0
GenR10.841.8
DivR2.52.4

Here's one example:

MethodQuantityR0R1R2R3R4R5R6R7Std
ConRReward1.0001.0001.0001.0001.0001.0001.0001.0000.0000
Advantage0.000.000.000.000.000.000.000.00—
GenRReward1.0001.0000.9001.0000.9251.0001.0001.0000.0384
Advantage0.570.57−2.030.57−1.380.570.570.57—
DivRReward1.2430.3780.1080.7570.4320.4590.4861.5410.4512
Advantage1.26−0.66−1.260.18−0.54−0.48−0.421.92—

Appendix C.3 gives a second example. This happens because absolute scoring discards many differences that are in fact separable. As shown in Fig. 1(c), responses that absolute-scoring methods treat as equal in quality are often separable in practice. Addressing this requires two things: finding the locations where quality actually differs, and constructing a comparable ruler that makes the difference visible. This is precisely the idea behind DivR, and the reason DivR reduces the risk of reward collapse at its source.

W2: The Role of the Shared Observed Baseline

Thank you for this comment. We use the high-frequency behavior as the observed baseline, which acts as a ruler that keeps all judgments on the same scale. Because high-frequency behavior lies in the dense region of the current behavior distribution, it also mitigates the failure in which an over-high or over-low reference compresses responses of different quality into the same score band, so that the reward remains discriminative. The observed baseline is therefore an essential component of DivR. In the example below, the run with the observed baseline separates the responses and assigns different scores, whereas removing it yields one identical score.

HealthBench#2616 Provider involvement (w=4)

MethodR0R1R2R3R4R5R6R7Std
With observed baseline002001110.70
Without observed baseline222222220.00

You are certainly right that GRPO itself subtracts rmeanr_{mean}. But that subtraction happens after scoring, when the advantage is computed, whereas DivR's observed baseline exists so that scoring itself returns a comparable and discriminative reward. If the reward is not discriminative, or is of poor quality, nothing at the advantage step can recover it. The benefit of the observed baseline therefore does come from how the reference changes the judge's judgments, not from GRPO.

We have explained the role of the observed baseline more clearly in the revision in section 4.3 and Appendix B.1.

W3: Matched GenR Experiment

This is a fair point. To address your concern, we have added an experiment that uses GenR-generated criteria together with DivR's five-level scale and the same rules for invalid or avoidant responses. The results are:

Methodqwen3.7-plusdeepseek-v4-flash
GenR-50.21540.1908
DivR0.23070.2273

The results show that the gain of DivR persists once the scoring granularity and the handling of abnormal responses are held fixed.

W4: Concerns About Reward Hacking, and What Happens When a Group Contains No Meaningful Divergence

Reward variation is a basic signal: optimization is only possible when reward variance exists, and this is the premise of our method as well. For example, 16.7% of rollout groups under ConR have zero reward variance.

Variance measures the quality differences among the model's responses that the judge model is able to detect. We therefore do not claim that a larger reward variance always means a better learning signal; but a variance of zero, or a very small one, is harmful to training, because it means that under the given rubric the judge sees essentially no quality difference among the policy's responses. Figure 6(b) shows that as response quality improves, the discriminability of ConR and GenR drops markedly, while DivR retains within-group variation close to its global level. DivR thus still provides a stable signal late in training, when the policy has improved and the other methods can no longer separate responses by quality.

MethodConRGenRDivR
Within-group variance in the high-quality zone / overall variance0.5620.7460.962

When there is no meaningful difference, DivR does not force an optimization signal. Even if a behavioral difference is observed during divergence extraction, if it does not correspond to a quality difference, relative scoring assigns 0 to every response on that dimension. Fig. 6(a) shows that even at the end of training, 20.5% of DivR's rubric dimensions give 0 to all responses to the same query. The number of non-discriminative rubric dimensions is of course far higher for GenR and ConR.

MethodConRGenRDivR
Non-discriminative rubric dimensions at the end of training61.3%61.0%20.5%

One example is HealthBench prompt 4272. This query contains a divergence point about the timing and duration of naps, with the observed baseline "Advise short naps earlier in the day and discourage late or long naps." Although the responses differ slightly in the stated duration (20–30 min), all of them recommend a reasonable short nap, so this dimension receives 0 for every response. At the same time, other dimensions in the same group (specific interventions such as melatonin or sleep supplements and CBT-I, and circadian strategies such as light exposure and gradual adjustment) still separate the responses.

This case shows that DivR can acknowledge behavioral difference with equivalent quality on a single dimension while retaining training signal on other quality-relevant dimensions of the same group, instead of inflating every behavioral difference into a quality preference merely to preserve variance. We have added a discussion of this case to Section 5.5.

W5: Limited Experimental Scope

Thank you for raising this concern. DivR is a general method, so in principle it applies to policies of any scale. We agree, however, that training larger models would make the results more convincing. We have therefore added an experiment with a Qwen2.5-7B policy:

Methodqwen3.7-plusdeepseek-v4-flash
GenR0.46100.3830
DivR0.47680.4015

On groups that contain little meaningful variation, our response to the previous question already adds the explanation and an example. We hope this addresses the concern.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Responses

Dear reviewer EnkB:

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

I thank the authors for the insightful answers. Having read the other reviewers' concerns as well as the authors' responses, I maintain my weak acceptance rating.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful and constructive feedback, which has helped us improve the clarity, analysis, and experimental validation of our work. In response, we have made several clarifications and additions:

  • Relationship between absolute scoring and reward collapse: Explained how score saturation in absolute-scoring methods leads to zero-gradient groups, supported by additional quantitative evidence and examples.
  • Shared observed baseline: Clarified how the baseline serves as a common reference that shapes the judge's scoring decisions, thereby making reward judgments more discriminative.
  • Fairer comparisons: Added a more controlled comparison with existing methods, showing that DivR maintains its advantage when scoring granularity and instructions for handling invalid or avoidant responses are kept consistent.
  • Reward hacking under meaningless divergence: Clarified how DivR avoids rewarding behavioral differences that do not correspond to quality differences, supported by additional quantitative analysis and examples.
  • Ablation studies: Added a repeated ablation experiment with a different random seed, providing stronger evidence for the contribution of the shared baseline.
  • Causal intervention: Refined the description of the intervention experiment and added analysis showing how intervention effects vary with the importance weights of divergence dimensions.
  • Formulation: Clarified the definitions and conceptual roles of the quantities introduced in Equations (2) and (3).
  • Larger policy model: Added experiments with Qwen2.5-7B, showing that DivR's advantages extend beyond the original 3B policy.

Regarding the novelty of DivR, our contribution is to address the mismatch between evaluation-oriented rubrics and the evolving capabilities of policies during RL training. We propose aligning reward criteria with quality-relevant behavioral differences that are both observable and learnable under the current policy, and realize this principle through group-level divergence extraction and shared-baseline relative scoring. This provides a consistent reward framework for group-relative credit assignment, distinguishing DivR from existing pairwise or locally constructed approaches.

We are grateful for the reviewers' careful assessment and constructive suggestions. We hope our clarifications, additional experiments, and revisions have adequately addressed the concerns raised and made our contributions clearer.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.