acceptodds
Under review as a conference paper at ICLR 2027

ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning

Abstract

Reinforcement learning has emerged as an effective paradigm for enhancing the mathematical reasoning capabilities of large language models. Among existing policy optimization methods, Proximal Policy Optimization (PPO) remains particularly appealing because its learned critic can, in principle, provide token-level credit assignment. However, in mathematical reasoning tasks characterized by long reasoning horizons and sparse outcome rewards, reliable token-level credit assignment remains challenging. The standard critic often fails to accurately evaluate intermediate reasoning states, resulting in noisy advantage estimates and suboptimal policy updates. In this paper, we propose ReDiPPO, a Reference-guided and Discrepancy-aware PPO framework for mathematical reasoning. ReDiPPO introduces a reference-guided critic that uses reference answers as training-time privileged signals to provide more accurate value estimation. Meanwhile, it retains a standard critic and quantifies the token-level reference-standard discrepancy between the standard value estimate and the reference-guided value estimate. This discrepancy serves as an indicator of difficult reasoning states and is used to reweight the corresponding token-level advantages during PPO optimization. Extensive experiments on diverse mathematical reasoning benchmarks demonstrate that ReDiPPO improves value-estimation accuracy and consistently outperforms strong policy optimization baselines, including PPO, DAPO, and GSPO, in final reasoning performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Mathematical Reasoning, Proximal Policy Optimization, Credit Assignment, Value Estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xZvmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work addresses the issue that PPO’s critic has difficulty accurately evaluating intermediate reasoning states, thereby affecting advantage estimation. To address this problem, it allows one critic to access reference answers during training, while retaining a standard critic without access to these answers. The prediction discrepancy between the two is used to adjust token-level update weights, while the actor always receives only the original prompt. Experiments are conducted on six mathematical reasoning benchmarks: AIME24, AIME25, AIME26, HMMT25, Minerva Math, and OlympiadBench.

Critical strengths

One advantage of this method is that it not only improves the critic’s value estimation using reference answers, but also reweights token-level advantages through the discrepancy between the reference-guided critic and the standard critic. The component ablations examine the roles of these two components and show incremental performance improvements across three different backbones.

Critical weaknesses
  1. The discrepancy alone cannot determine which critic is more reliable. ∣Vref−Vstd∣|V^{\mathrm{ref}}-V^{\mathrm{std}}| is a symmetric discrepancy measure that does not necessarily equal the standard critic’s true error or establish that high-discrepancy positions should receive larger weights for their reference-guided advantages. Positive weights and clipping only constrain the reweighting magnitude, without establishing the reliability of this reweighting with respect to the original expected-return objective.
  2. The metrics used are insufficient to fully support the value-calibration interpretation. According to the formulas in the paper, EV is insensitive to a constant shift in value predictions, while PSA only evaluates response ranking. Improvements in both therefore cannot rule out systematic value bias or independently establish that local advantages are more accurate.
  3. Experimentally, there is a lack of independent value validation using multiple continuations from the prompt-only actor under the same prefix. Existing segment-level EV/PSA analyses and reference-answer/reference-solution ablations have not fully examined whether the reference-guided critic overestimates the actor’s actual success probability or undervalues different yet correct trajectories.
  4. There is a lack of controls that isolate the contribution of reference–standard discrepancy. Comparisons with the prediction discrepancy between two standard critics, other uncertainty measures, or difficulty-based weighting methods, as well as comparisons between response-level and token-level reweighting, have not been fully explored.
  5. The main table does not report results and uncertainty across multiple independent training seeds. The authors report that, in the tested settings, this method increases training time by approximately 16.0%–23.9% compared with PPO, but do not provide a PPO comparison under the same computational budget. Therefore, whether the performance gains justify the additional computational cost still requires further verification.
Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Author Response to Official Review by Reviewer xZvm

We thank the reviewer for recognizing the combined contribution of reference-guided value estimation and discrepancy-aware token weighting. Your comments motivated additional prefix-continuation evaluations and controlled policy-training ablations, which we report below.

1. W1: Critic Reliability and the Justification for Reweighting

The discrepancy alone cannot determine which critic is more reliable.

Positive weights and clipping only constrain the reweighting magnitude, without establishing the reliability of this reweighting with respect to the original expected-return objective.

Thank you for raising this concern. The purpose of discrepancy is to identify tokens that merit greater emphasis during policy optimization.

We additionally assess the relative reliability of the two critics through independent Monte Carlo (MC) continuations from the prompt-only actor in our combined response to W2 and W3 below. Here, we first use controlled experiments to demonstrate the effectiveness of discrepancy-based token selection.

We hypothesize that large disagreements identify partial reasoning states whose policy updates can benefit from greater emphasis, and test this hypothesis through controlled token-selection experiments as below.

High-discrepancy token selection. To test the effectiveness of selecting tokens by discrepancy, we conduct experiments on Qwen3-4B-Instruct-2507 in which only 20% of the valid tokens in each response contribute to the actor update. All variants use reference-guided advantages. Within each response, we select tokens with the highest or lowest raw absolute critic discrepancy, or sample randomly. All three variants use the same selection fraction and unit weights. A binary actor-loss mask is applied after full-response advantage whitening.

High-20% exceeds Random-20% on all six benchmarks, with average performance ordered High > Random > Low. These results support discrepancy as an empirical criterion for identifying useful update positions. The all-token +Ref baseline reported in our response to W4 exceeds all three selection variants, motivating ReDiPPO's use of discrepancy-guided emphasis with supervision at every valid response token.

Token selectionAIME24AIME25AIME26HMMT25MinervaOlympiadAvg.Δ Avg. vs. Random (pp)
High-discrepancy 20%56.4644.5853.3328.1362.2274.3953.19+1.27
Random 20%55.9443.8551.9827.5058.3673.8751.920.00
Low-discrepancy 20%55.9442.6052.0825.8345.3173.1849.16-2.76

We further clarify the scope of unbiased estimation in our training procedure and its relationship to the original expected-return objective.

Critic learning uses terminal MC targets independently of discrepancy weighting. Both critics are supervised by the same terminal returns, which avoid bootstrapping bias. For continuations sampled from a fixed rollout policy without outcome-based filtering, these returns are unbiased samples of its value. Discrepancy weights enter only the actor objective.

The actor uses centered and normalized advantages. Whitening sets the mean of the weighted advantages to zero over valid batch tokens. Finite-batch centering and scaling can introduce gradient bias, while GRPO [1] provides an established example of effective LLM training with normalized advantages. Its group-mean centering alone introduces a constant rescaling relative to the leave-one-out estimator.

ReDiPPO applies discrepancy weighting as a bounded modification of the PPO surrogate. Token-dependent weights further change the relative contributions to the actor update. Their effectiveness is supported by the fixed-budget token-selection experiments above and the response-level, shuffled-weight, and entropy-weighting controls presented in our response to W4.

We will clarify this motivation and include the token-selection analysis in Sections 4.3 and 4.4.

[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

AuthorsOpenReview ↗

Author Response to Official Review by Reviewer xZvm

2. W2 & W3: Independent Monte Carlo Validation of Critic Quality

EV is insensitive to a constant shift in value predictions, while PSA only evaluates response ranking. Improvements in both therefore cannot rule out systematic value bias or independently establish that local advantages are more accurate.

There is a lack of independent value validation using multiple continuations from the prompt-only actor under the same prefix.

Existing segment-level EV/PSA analyses and reference-answer/reference-solution ablations have not fully examined whether the reference-guided critic overestimates the actor's actual success probability or undervalues different yet correct trajectories.

We thank the reviewer for carefully examining our critic-quality analysis, highlighting the limitations of EV and PSA, and suggesting independent prefix-continuation validation.

During RL training, our primary goal in monitoring the critics is to compare their relative predictive performance. We therefore use explained variance (EV), a long-established statistical measure widely adopted for evaluating value predictions in PPO-based LLM RL. [1, 2, 3]

We agree that absolute prediction error and signed bias are needed to assess calibration alongside EV and PSA. We have therefore added an independent Monte Carlo evaluation of value predictions.

(1) Independent MC validation. We evaluate both critics at 6,144 prefixes from 256 problems in the training set. For value estimation, we use 32 independent continuations per prefix. For each fixed prefix, the value target is

V^MC(st)=1K∑k=1KRt,k,K=32.\widehat V_{\mathrm{MC}}(s_t)=\frac{1}{K}\sum_{k=1}^{K}R_{t,k},\qquad K=32.

(2) Value accuracy, calibration, and bias. Mean squared error (MSE) measures value-prediction error against independent MC targets. Expected calibration error (ECE) measures how well predicted success probabilities match observed success rates. Mean signed error captures average overestimation or underestimation.

Reference conditioning improves value accuracy and calibration. Predictions better match the actor's continuation success probabilities. The reference-guided critic underestimates success on average, with a smaller absolute bias than the standard critic's overestimation.

CriticValue MSE ↓ECE ↓Mean signed error
Standard critic0.105900.13712+0.10837
Reference-guided critic0.080250.07834−0.04013

Signed error is the predicted value minus the Monte Carlo success estimate. Mean signed error closer to zero indicates less average bias.

(3) Alternative correct paths. Our core method supplies only the ground-truth final answer as critic-side reference information. In Appendix C.1, we compare reference answers with reference solutions and find that solution conditioning yields slightly lower downstream policy performance. We hypothesize that a specific reference solution may bias the critic toward that derivation, potentially leading it to undervalue alternative yet correct reasoning paths. This motivates our use of concise reference answers, which leave the intermediate solution path unspecified.

We are currently working on step-level scoring guided by high-quality reference solutions, with the aim of improving fine-grained credit assignment.

We will add a new subsection, C.3, to Appendix C to present these results and analyses, together with details of the continuation protocol and ECE binning.

[1] Reporting Explained Variance, 1981

[2] VinePPO: Refining Credit Assignment in RL Training of LLMs, 2024

[3] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks, 2025

AuthorsOpenReview ↗

Author Response to Official Review by Reviewer xZvm

3. W4: Alternative Discrepancy, Uncertainty, and Difficulty Controls

Comparisons with the prediction discrepancy between two standard critics, other uncertainty measures, or difficulty-based weighting methods, as well as comparisons between response-level and token-level reweighting, have not been fully explored.

Thank you for highlighting this gap and suggesting these controls. We acknowledge that our original analysis of discrepancy-based weighting was insufficient. Building on the token-selection experiments presented in our response to W1, we have added weighting controls on Qwen3-4B-Instruct-2507 to examine the weighting signal, response-level aggregation, and token alignment.

All five variants use reference-guided advantages and every valid response token. (a) Reference-guided PPO (+Ref) uses unit weights. (b) Sequence-level discrepancy weighting broadcasts one weight derived from ∣V‾iref−V‾istd∣|\overline V_i^{\mathrm{ref}}-\overline V_i^{\mathrm{std}}| to all tokens within each response. (c) Within-response shuffle permutes clipped discrepancy weights within each response before advantage whitening. (d) Token-entropy weighting substitutes full-vocabulary policy entropy for critic discrepancy under the same token-weight transformation. This provides a simpler heuristic that assigns larger update weights to tokens with higher policy uncertainty. (e) Full ReDiPPO applies normalized, clipped discrepancy weights to their corresponding token positions.

Weighting variantAIME24AIME25AIME26HMMT25MinervaOlympiadAvg.
Reference-guided PPO (+Ref)57.7148.9655.5228.9662.3275.1754.77
Sequence-level discrepancy59.7948.7555.0029.7962.4174.7355.08
Within-response shuffle55.3145.7354.4828.2361.7672.8753.06
Token-entropy weighting58.4448.4353.7528.5461.0774.2254.08
Full ReDiPPO60.6849.7956.7730.6362.8475.4556.03

Shuffling reduces average performance below +Ref despite preserving each response's weight distribution, supporting the importance of token alignment. Entropy weighting also falls below +Ref, and sequence-level weighting gives a smaller average gain. Full ReDiPPO achieves the highest performance on all six benchmarks among these variants.

The sequence control uses the absolute difference of response-mean values, normalized across responses and clipped before broadcast. This aggregation permits cancellation of signed differences and changes the weighting statistic. The shuffle control directly tests position alignment while preserving each response's clipped-weight distribution.

We will include these controls and their analysis in Section 4.3 (Ablation Studies) and Section 4.4 (Analysis of Reference-Standard Discrepancy).


4. W5: Independent Training Seeds and Computational Cost

The main table does not report results and uncertainty across multiple independent training seeds.

The authors report that, in the tested settings, this method increases training time by approximately 16.0%–23.9% compared with PPO, but do not provide a PPO comparison under the same computational budget.

Thank you for raising these questions. Due to computational constraints, each configuration in Table 1, Figure 3, and the additional training experiments uses one independent RL training run. Although we have not conducted multi-seed runs, ReDiPPO consistently outperforms PPO across different model backbones and evaluation datasets, providing empirical support for its effectiveness.

To assess the benefit of additional training, we extended PPO training on Qwen3-4B-Instruct-2507 to give it a larger H200 GPU-hour budget than ReDiPPO. Although ReDiPPO takes longer per training step, it reaches higher final performance. Extending PPO training yields modest improvements, but its performance remains below ReDiPPO even with a larger compute budget.

Qwen3-4B-Instruct-2507GPU-hours (H200)Gradient stepsSix-benchmark avg. (%)
ReDiPPO309.635256.03
PPO248.335253.66
PPO (extended training)361.248054.68

We will incorporate these results and timing details into Appendix C.2.

Reviewer_UvGnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies credit assignment for llm reasoning. They propose ReDiPPO, which gives the reference answer only to the critic as training-time privileged information, to let the critic model provide better value estimation. It also reweights each token’s advantage by how far apart this answer-aware critic and a standard critic are. Across six math benchmarks and three backbones, ReDiPPO beats other baselines with no extra inference cost.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The empirical results show a consistent improvement over PPO/DAPO/GSPO baselines.
  3. The method is simple and intuitively reasonable, and it incurs no additional cost.
Critical weaknesses
  1. The paper only do experiments on math. There are no other tasks like coding and agentic tasks.
  2. Letting the critic see training-time information that the policy cannot is the established asymmetric actor-critic idea [1]. Recent work [2,3] arrived at the same answer-conditioned critic independently (where the paper does not cite). The contribution reads more as an incremental application than a new idea.
  3. The discrepancy weight uses the signed gap vref−vstdv^{ref} − v^{std}. In this regard, tokens where the answer-aware critic is more pessimistic (e.g., a confidently wrong path) get down-weighted, which contradicts the paper’s own motivation. There is also no theoretical justification and no ablation against simpler weightings.

[1] Asymmetrc Actor Critic for Image-Based Robot Learning

[2] Best Practice Critic Optimization

[3] le Critique: Privileged Value Functions for LLM Reinforcement Learning

Questions for authors
  1. See weakness.
  2. [2] finds that giving the reference answer to the critic speeds up training but causes earlier overfitting on a small dataset, while it helps on a larger one. Have you observed similar overfitting with ReDiPPO under longer training or smaller datasets, and do you think the critic pretraining or the discrepancy reweighting helps mitigate it?
8 comments
AuthorsOpenReview ↗

Author Response to Official Review by Reviewer UvGn

We thank reviewer UvGn for recognizing the clear presentation, consistent empirical improvements, and simple design of ReDiPPO. We also appreciate your observation that the method adds no critic-side inference computation.

1. W1: Evaluation Beyond Mathematical Reasoning

The paper only do experiments on math. There are no other tasks like coding and agentic tasks.

We thank the reviewer for raising this important concern. We agree that evaluating ReDiPPO on diverse tasks is important for understanding its generalizability beyond mathematical reasoning.

To further investigate its applicability, we have conducted additional experiments on a Logic Reasoning benchmark Knights & Knaves [1], which provides an evaluation domain beyond mathematical problem solving. ReDiPPO achieves the highest accuracy in every reported role group. These results extend the empirical evidence to logical reasoning. The additional results are summarized below:

Qwen3-4B-Instruct-25074–8 Roles9 Roles10 Roles11 RolesAvg.
Vanilla72.9967.9454.4437.8158.30
DAPO73.7168.8157.0040.0059.88
PPO74.3365.5957.6939.1459.19
PPO+Ref73.9569.1956.1339.2559.63
ReDiPPO76.1669.3158.5641.3861.35

We acknowledge that coding and agentic tasks involve additional challenges, such as program execution, tool interaction, and multi-step environment feedback, which are not fully covered by the current experiments. Therefore, we will clarify the current scope of our evaluation and avoid overgeneralizing the results to these domains. We leave further validation on coding and agentic tasks to future work.

We will add a new subsection to Appendix C to present the Knights & Knaves results and training details.

AuthorsOpenReview ↗

Author Response to Official Review by Reviewer UvGn

2. W2: Relationship to Asymmetric Actor-Critic, BPCO, and Le Critique

Letting the critic see training-time information that the policy cannot is the established asymmetric actor-critic idea. Recent work arrived at the same answer-conditioned critic independently. The contribution reads more as an incremental application than a new idea.

We thank the reviewer for pointing out these related works. We will expand the related-work discussion and refine our contribution statements to acknowledge prior research and clarify the following distinctions.

(1) Asymmetric Actor-Critic [1]. We acknowledge asymmetric actor-critic as an established RL paradigm and will add the corresponding citation. The original work studies image-based robotic control. ReDiPPO targets mathematical reasoning in LLMs, where long reasoning horizons and sparse terminal rewards make intermediate-state value estimation challenging. In addition to using reference answers as privileged critic information, ReDiPPO retains a standard critic and uses the absolute differences between the two critics' predictions to construct token-level policy update weights. The component ablation in Figure 3 (Section 4.3) shows that this weighting mechanism yields further gains over reference-guided PPO, supporting its complementary contribution to answer-conditioned value estimation.

(2) BPCO [2] and Le Critique [3]. BPCO, Le Critique, and ReDiPPO all incorporate reference information into critic learning for LLM RL, providing complementary evidence for the promise and value of this approach. Building on this shared foundation, each work pursues a distinct direction for further improvement. BPCO develops a recipe for stable critic-based RL through DPPO, bounded value prediction, Monte Carlo targets, unnormalized advantages, and length-adaptive GAE, with optional privileged information. Le Critique studies privileged value functions and adaptive group–value baseline interpolation through TETHER to reduce policy-gradient variance. ReDiPPO constructs normalized, clipped token weights from the absolute value differences between reference-guided and standard critics, then applies these weights to reference-guided advantages. Its dual-critic design thus uses both privileged information and critic disagreement in policy optimization. We will incorporate both related works into the revision.

(3) Independent and concurrent development. Le Critique and BPCO were publicly released on August 17 and August 24, 2026, respectively. The work presented in our submission had already been completed before these releases. We therefore regard them as closely related, independently developed concurrent works and will acknowledge this relationship in the revision.

(4) Empirical evidence for discrepancy-aware reweighting. In our response to W3 below, we present additional ablations that provide further evidence for the effectiveness of discrepancy-aware token weighting.

[1] Asymmetrc Actor Critic for Image-Based Robot Learning

[2] Best Practice Critic Optimization

[3] le Critique: Privileged Value Functions for LLM Reinforcement Learning

AuthorsOpenReview ↗

Author Response to Official Review by Reviewer UvGn

3. W3: Absolute Discrepancy, Weight Interpretation, and Simpler Controls

The discrepancy weight uses the signed gap.

There is also no theoretical justification and no ablation against simpler weightings.

Thank you for raising these concerns and suggesting comparisons with simpler weightings. We acknowledge that our original analysis of discrepancy-based weighting was insufficient. We have added two groups of controlled ablations to evaluate discrepancy-based token selection and alternative weighting strategies. Regarding the sign of the gap, ReDiPPO uses the absolute critic discrepancy to construct token weights, as defined below.

(1) Absolute discrepancy determines the weight regardless of which critic predicts the larger value. Our method defines

\qquad w_{i,t}=\operatorname{clip}\left(1+\frac{e_{i,t}-\mu_e}{\sigma_e+\epsilon_e},0.5,2.0\right).$$ For example, $V^{\mathrm{ref}}=0.2$ and $V^{\mathrm{std}}=0.8$ produce the same discrepancy as the reversed pair. Within the same rollout batch, both pairs therefore receive the same weight. A more pessimistic reference-guided prediction is weighted according to the magnitude of its disagreement with the standard critic. These positive weights multiply the reference-guided advantages before masked whitening. We use discrepancy as **an empirical proxy** for the sensitivity of value predictions to reference information. We hypothesize that large disagreements identify partial reasoning states that are challenging for reference-blind value estimation and may benefit from greater emphasis during policy optimization. The resulting weights define a bounded, empirically motivated modification of the PPO surrogate, which we evaluate through the following controls. **(2) High-discrepancy tokens yield more effective updates.** To test the effectiveness of high-discrepancy token selection, we conduct controlled experiments in which **only 20%** of the valid tokens in each response contribute to the actor update. We use Qwen3-4B-Instruct-2507 with reference-guided advantages throughout. Within each response, we select the 20% of valid tokens with the **highest** or **lowest** raw absolute critic discrepancy, or sample **randomly**. All three variants use the same selection fraction and unit weights. A binary actor-loss mask is applied after full-response advantage whitening. | Token selection | AIME24 | AIME25 | AIME26 | HMMT25 | Minerva | Olympiad | Avg. | Δ Avg. vs. Random (pp) | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | **High-discrepancy 20%** | **56.46** | **44.58** | **53.33** | **28.13** | **62.22** | **74.39** | **53.19** | **+1.27** | | Random 20% | 55.94 | 43.85 | 51.98 | 27.50 | 58.36 | 73.87 | 51.92 | 0.00 | | Low-discrepancy 20% | 55.94 | 42.60 | 52.08 | 25.83 | 45.31 | 73.18 | 49.16 | -2.76 | High-20% exceeds Random-20% on all six benchmarks, with average performance ordered High > Random > Low. This supports discrepancy as a criterion for identifying useful update positions. See the remaining rebuttal for W3 in the next response. ↓
Reviewer_1QzFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReDiPPO to improve value estimation and token-level credit assignment in mathematical reasoning with sparse outcome rewards. The method introduces a reference-guided critic that receives the reference answer as training-time privileged information and supplies the baseline for actor advantages. A standard critic is retained, and the absolute discrepancy between the two critics is normalized and clipped to construct token-level advantage weights. The policy itself does not receive the reference answer. Experiments cover three model backbones and six mathematical reasoning benchmarks, supplemented by component ablations, critic-quality analyses, and training-time measurements. ReDiPPO achieves the highest average performance on all three backbones, improving over PPO by 1.19–2.37 percentage points.

Critical strengths
  1. The problem is well motivated, and the proposed design directly addresses it. Reliable value estimation under long reasoning horizons and sparse terminal rewards is an important challenge for critic-based RL. Providing existing reference answers to the critic is a reasonable way to simplify value prediction without requiring additional process annotations.

  2. The empirical improvements are consistent across backbones, with reasonably controlled comparisons. ReDiPPO achieves the highest average score on all three backbones. The component ablations also show improvements from reference conditioning and further gains from discrepancy reweighting. The PPO baseline shares the critic-pretraining protocol, dynamic sampling, and Clip-Higher settings, making it a useful controlled comparison. Together, these results provide substantive evidence that the proposed modifications can be beneficial.

Critical weaknesses
  1. The specific contribution of token-level discrepancy weighting has not been sufficiently isolated. Figure 3 supports the usefulness of adding discrepancy weighting, but Figure 5 primarily establishes a response-level association between average discrepancy and difficulty, while Figure 6 shows improved value prediction toward the end of responses. These observations do not directly establish that critic discrepancy identifies the particular token positions whose policy updates should be emphasized.

  2. Consequently, it remains unclear whether the gains depend on token-specific localization or arise from a more general reweighting of difficult responses. This distinction matters because token-level discrepancy weighting is a central methodological contribution.

  3. Some related works are not evaluated or discussed, such AAPO, Dr.GRPO, SAO, etc. (SAO is not published, but can be discussed. Experiment evaluation is not necessary)

[1] AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin ACL 2026

[2] Understanding R1-Zero-Like Training: A Critical Perspective, COLM 2025

[3] Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning, Arixv 2026

Questions for authors
  1. Clarification of experimental repetitions and checkpoint selection. Appendix A specifies two epochs of critic pretraining. Separately, how many independent RL training runs were conducted for each configuration in Table 1 and Figure 3?

  2. Validation of the central reweighting mechanism. Could the authors provide one targeted control, such as assigning a uniform weight to each response based on its average discrepancy, or shuffling token weights within each response? This would help determine whether the gains depend on the correspondence between weights and specific token positions.

  3. Clarification of the performance–cost trade-off. Does the timing analysis in Figure 8 include critic pretraining? Could existing training logs support a comparison at matched wall-clock time for one representative backbone? This would help assess the practical benefit of maintaining the additional critic.

4 comments
AuthorsOpenReview ↗

Author Response to Official Review by Reviewer 1QzF

Thank you very much to reviewer 1QzF for your attention to and recognition of our work. Your feedback is greatly encouraging to us. We appreciate the clarity and constructiveness of your comments, which help us further improve the paper.

1. W1, W2 & Q2: Isolating the Contribution of Token-Level Discrepancy Weighting

The specific contribution of token-level discrepancy weighting has not been sufficiently isolated.

It remains unclear whether the gains depend on token-specific localization or arise from a more general reweighting of difficult responses.

Could the authors provide one targeted control, such as assigning a uniform weight to each response based on its average discrepancy, or shuffling token weights within each response?

Thank you for highlighting this gap and suggesting these controls. We acknowledge that our original analysis of token-level discrepancy weighting was insufficient. To better isolate its contribution, we have added two groups of controlled ablations on Qwen3-4B-Instruct-2507, examining token selection and alternative weighting strategies.

To test the effectiveness of high-discrepancy token selection, we conduct controlled experiments in which only 20% of the valid tokens in each response contribute to the actor update.

We use Qwen3-4B-Instruct-2507 with reference-guided advantages throughout. Within each response, we select the 20% of valid tokens with the highest or lowest raw absolute critic discrepancy, or sample randomly. All three variants use the same selection fraction and unit weights. A binary actor-loss mask is applied after full-response advantage whitening, so the comparison tests which positions contribute to policy optimization.

High-discrepancy tokens yield more effective updates. High-20% exceeds Random-20% on all six benchmarks, with average performance ordered High > Random > Low. This supports discrepancy as a criterion for identifying useful update positions.

Token selectionAIME24AIME25AIME26HMMT25MinervaOlympiadAvg.Δ Avg. vs. Random (pp)
High-discrepancy 20%56.4644.5853.3328.1362.2274.3953.19+1.27
Random 20%55.9443.8551.9827.5058.3673.8751.920.00
Low-discrepancy 20%55.9442.6052.0825.8345.3173.1849.16-2.76

To further evaluate the effectiveness of discrepancy-based token weighting, we design an ablation study with five variants, all using every valid response token. (a) Reference-guided PPO (+Ref) uses unit weights. (b) Sequence-level discrepancy weighting broadcasts one weight derived from ∣V‾iref−V‾istd∣|\overline V_i^{\mathrm{ref}}-\overline V_i^{\mathrm{std}}| to all tokens within each response to test response-level weighting. (c) Within-response shuffle permutes clipped discrepancy weights within each response to test position alignment. (d) Token-entropy weighting substitutes full-vocabulary policy entropy for critic discrepancy under the same token-weight transformation to test an alternative uncertainty signal. (e) Full ReDiPPO applies normalized, clipped discrepancy weights to their corresponding token positions.

Weighting variantAIME24AIME25AIME26HMMT25MinervaOlympiadAvg.
Reference-guided PPO (+Ref)57.7148.9655.5228.9662.3275.1754.77
Sequence-level discrepancy59.7948.7555.0029.7962.4174.7355.08
Within-response shuffle55.3145.7354.4828.2361.7672.8753.06
Token-entropy weighting58.4448.4353.7528.5461.0774.2254.08
Full ReDiPPO60.6849.7956.7730.6362.8475.4556.03

Shuffling reduces average performance below +Ref despite preserving each response's weight distribution, supporting the importance of token alignment. Entropy weighting also falls below +Ref, and sequence-level weighting gives a smaller average gain. The all-token +Ref baseline exceeds High-20%, favoring broad supervision with discrepancy-guided emphasis.

We will include these controls and their analysis in 4.3 Ablation Studies and 4.4 Analysis of Reference-Standard Discrepancy.

AuthorsOpenReview ↗

Author Response to Official Review by Reviewer 1QzF

2. Q1: Independent Training Runs and Checkpoint Selection

Appendix A specifies two epochs of critic pretraining. Separately, how many independent RL training runs were conducted for each configuration in Table 1 and Figure 3?

Thank you for requesting these details. Due to computational constraints, each configuration in Table 1, Figure 3, and the additional training experiments uses one independent RL training run. The two epochs in Appendix A specify critic-pretraining duration, while avg@32 and avg@8 specify response sampling during evaluation. For Table 1 and Figure 3, every configuration is evaluated at the predetermined checkpoint saved after 352 gradient steps.

Although we have not conducted repeated training runs with different random seeds, ReDiPPO consistently outperforms PPO across different model backbones and evaluation datasets, providing empirical support for its effectiveness.


3. Q3: Training-Time Accounting and Performance–Cost Comparison

Does the timing analysis in Figure 8 include critic pretraining? Could existing training logs support a comparison at matched wall-clock time for one representative backbone?

Thank you for raising the performance–cost trade-off. Figure 8 reports only the time consumed by a single RL training step. Critic pretraining is performed offline before RL to provide a warm start for value learning. We will clarify this measurement scope in the figure caption.

To assess the benefit of additional training, we extended PPO training on Qwen3-4B-Instruct-2507 to give it a larger H200 GPU-hour budget than ReDiPPO.

Although ReDiPPO takes longer per training step, it reaches higher final performance. Extending PPO training yields modest improvements, but its performance remains below ReDiPPO even with a larger compute budget.

Qwen3-4B-Instruct-2507GPU-hours (H200)Gradient stepsSix-benchmark avg. (%)
ReDiPPO309.635256.03
PPO248.335253.66
PPO (extended training)361.248054.68

We will incorporate these results and timing details into Appendix C.2 and add a figure comparing PPO and ReDiPPO, with cumulative training time on the x-axis and evaluation performance on the y-axis.


4. W3: Discussion of AAPO, Dr.GRPO, and SAO

Some related works are not evaluated or discussed, such AAPO, Dr.GRPO, SAO, etc. (SAO is not published, but can be discussed. Experiment evaluation is not necessary)

Thank you for these related-work suggestions. They motivate a broader baseline comparison and a discussion of critic-training practice.

AAPO [1] and Dr.GRPO [2] provide valuable advances in GRPO-style optimization. AAPO introduces margin-augmented advantages to strengthen learning when group-relative advantages approach zero, and we will add it to Related Work. Dr.GRPO [2] is already cited, and we have added an experiment on Qwen3-4B-Instruct. Dr.GRPO achieves a six-benchmark average of 52.98%, compared with 53.84% for DAPO, as summarized below.

Qwen3-4B-InstructAIME24AIME25AIME26HMMT25MinervaOlympiadAverage
DAPO55.4246.2554.9529.6461.7075.0753.84
Dr.GRPO57.9248.0251.1528.8557.7274.2052.98

SAO [3] offers valuable PPO-style actor–critic training practices for agentic tasks. Its single-rollout asynchronous framework emphasizes stable value learning, including value pretraining to address critic cold-start problems. This observation aligns with the importance of critic pretraining in our training protocol.

We will add discussion of AAPO and SAO in Section 2 Related Work.

[1] AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin

[2] Understanding R1-Zero-Like Training: A Critical Perspective

[3] Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

AuthorsOpenReview ↗

Looking Forward to Your Response

Dear reviewer 1QzF,

Thank you again for the time you have devoted to reviewing our submission and our rebuttal. As the discussion period is drawing to a close, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns, or if any further clarification would be helpful on our side.

Best regards,

Authors of submission 24619

Thanks a lot for your response. Most of concerns have been addressed. I understand that the multiple seeds experiment is hard to be done. The additional experimental evaluation on added paper is not necessary. I would like to maintain my positive score as I think it is proper to assess the paper. Thanks.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Paper Revision

In response to the reviewers’ comments, we have revised the manuscript, with the additions highlighted in blue. The major changes are as follows:

Main Paper:

  • Section 2 (lines 96–101): Added discussions of AAPO and SAO. (Reviewer 1QzF W3)
  • Section 4.3, Table 2 (lines 430–452): Added ablations comparing discrepancy-aware token reweighting with response-level, shuffled, and entropy-based weighting. (Reviewer 1QzF W1, W2 & Q2; Reviewer UvGn W3; Reviewer xZvm W4)
  • Section 4.4, Table 3 (lines 473–485, 505–509): Added fixed-20% token-selection experiments to evaluate the effectiveness of discrepancy-based token selection. (Reviewer 1QzF W1, W2 & Q2; Reviewer UvGn W3; Reviewer xZvm W1 & W4)

Appendix:

  • Appendix B.4 (lines 793–800): Clarified the number of independent training runs and checkpoint-selection protocol. (Reviewer 1QzF Q1; Reviewer xZvm W5)
  • Appendix C.2 (lines 888–890): Clarified that Figure 8 measures per-step RL training time and excludes critic pretraining. (Reviewer 1QzF Q3)
  • Appendix C.2.1, Table 6 and Figure 9 (lines 914–917, 944–961): Added extended PPO training results and performance-versus-training-time curves to assess the performance–cost trade-off. (Reviewer 1QzF Q3; Reviewer xZvm W5)
  • Appendix C.3, Table 7 (lines 965–1001): Added independent Monte Carlo evaluation of critic accuracy, calibration, and bias. (Reviewer xZvm W1, W2 & W3)
  • Appendix C.4, Table 8 (lines 1003–1025): Added Knights & Knaves results and analysis of discrepancy weighting in a small-data setting. (Reviewer UvGn W1, Q1 & Q2)
  • Appendix D (lines 1026–1057): Added comparisons with asymmetric actor-critic, BPCO, and Le Critique, clarifying methodological differences and the development timeline. (Reviewer UvGn W2 & Q1)
AuthorsOpenReview ↗

Summary for the Area Chair

Dear Area Chair,

We sincerely thank the reviewers for their feedback and continued discussion. Reviewer 1QzF recognized the importance of value estimation under sparse rewards, the controlled PPO comparison, and the consistent improvements across backbones. Reviewer UvGn highlighted the clear presentation, the simple and intuitive design, and the empirical gains over the evaluated baselines. Reviewer xZvm acknowledged that the component ablations show incremental benefits from both reference conditioning and discrepancy-based token reweighting.

During the rebuttal, we provided point-by-point clarifications and added extensive new experiments in response to all reviewer concerns. Among the three reviewers, reviewer 1QzF stated that "Most of concerns have been addressed" and maintained their positive score.

Reviewer UvGn raised a follow-up question about the different weighting directions in GEAR and ReDiPPO. We clarified their distinct signals and the experimental basis for our weighting; the reviewer has not yet confirmed whether this response addresses the concern.

Reviewer xZvm acknowledged that the added calibration analyses and weighting controls addressed the corresponding concerns and raised their score from 1 to 2, while retaining concerns about reliability and experimental design. We responded with discrepancy-stratified critic evaluation and clarified the training-seed limitations, control design, and timing definitions.

Below is a concise summary of how the rebuttal resolved the issues and questions.

  1. Clarifying the contribution and relationship to prior work. (Reviewer 1QzF W3; Reviewer UvGn W2 and follow-up)

    • We clarify that ReDiPPO combines reference-guided value estimation with token reweighting derived from reference–standard critic disagreement. We discuss asymmetric actor-critic, BPCO, and Le Critique, and expand the related-work discussion to include AAPO and SAO.
    • In response to the GEAR comparison, we distinguish its self-distillation likelihood signal from ReDiPPO's discrepancy between estimated success probabilities.
  2. Isolating the contribution of token-level discrepancy weighting. (Reviewer 1QzF W1, W2 & Q2; Reviewer UvGn W3; Reviewer xZvm W1 & W4)

    • We use discrepancy to construct token-update masks that retain only 20% of tokens per response, demonstrating that high-discrepancy selection yields more effective updates than random or low-discrepancy selection.
    • We add ablations on weighting, showing that aligned discrepancy weighting outperforms response-level, shuffled, and entropy-based alternatives.
  3. Validating critic accuracy and reliability at high discrepancy. (Reviewer xZvm W1–W3 and follow-up)

    • Independent MC evaluation assesses value accuracy, calibration, and bias; discrepancy-stratified analysis shows that the reference-guided critic is more accurate on average across all buckets, with larger absolute MSE reductions at higher discrepancy.
  4. Extending evaluation beyond mathematical reasoning. (Reviewer UvGn W1 & Q2; Reviewer xZvm W5)

    • We add Knights & Knaves logical reasoning experiments. ReDiPPO improves upon reference-guided PPO across all four evaluated role-count groups.
  5. Clarifying reproducibility and computational cost. (Reviewer 1QzF Q1 & Q3; Reviewer xZvm W4, W5 and follow-up)

    • We disclose one independent RL training run per configuration and the predetermined checkpoint-selection protocol.
    • We add extended PPO training results and performance-versus-training-time curves.

Overall, the rebuttal provides additional evidence, expanded analysis, and clearer technical explanations aimed at resolving all reviewer concerns. We hope this summary helps the AC assess the revised submission efficiently.

Best regards,

Authors of submission 24619

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.