- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work addresses the issue that PPO’s critic has difficulty accurately evaluating intermediate reasoning states, thereby affecting advantage estimation. To address this problem, it allows one critic to access reference answers during training, while retaining a standard critic without access to these answers. The prediction discrepancy between the two is used to adjust token-level update weights, while the actor always receives only the original prompt. Experiments are conducted on six mathematical reasoning benchmarks: AIME24, AIME25, AIME26, HMMT25, Minerva Math, and OlympiadBench.
Critical strengths
One advantage of this method is that it not only improves the critic’s value estimation using reference answers, but also reweights token-level advantages through the discrepancy between the reference-guided critic and the standard critic. The component ablations examine the roles of these two components and show incremental performance improvements across three different backbones.
Critical weaknesses
- The discrepancy alone cannot determine which critic is more reliable. is a symmetric discrepancy measure that does not necessarily equal the standard critic’s true error or establish that high-discrepancy positions should receive larger weights for their reference-guided advantages. Positive weights and clipping only constrain the reweighting magnitude, without establishing the reliability of this reweighting with respect to the original expected-return objective.
- The metrics used are insufficient to fully support the value-calibration interpretation. According to the formulas in the paper, EV is insensitive to a constant shift in value predictions, while PSA only evaluates response ranking. Improvements in both therefore cannot rule out systematic value bias or independently establish that local advantages are more accurate.
- Experimentally, there is a lack of independent value validation using multiple continuations from the prompt-only actor under the same prefix. Existing segment-level EV/PSA analyses and reference-answer/reference-solution ablations have not fully examined whether the reference-guided critic overestimates the actor’s actual success probability or undervalues different yet correct trajectories.
- There is a lack of controls that isolate the contribution of reference–standard discrepancy. Comparisons with the prediction discrepancy between two standard critics, other uncertainty measures, or difficulty-based weighting methods, as well as comparisons between response-level and token-level reweighting, have not been fully explored.
- The main table does not report results and uncertainty across multiple independent training seeds. The authors report that, in the tested settings, this method increases training time by approximately 16.0%–23.9% compared with PPO, but do not provide a PPO comparison under the same computational budget. Therefore, whether the performance gains justify the additional computational cost still requires further verification.
Questions for authors
See weaknesses.
5 comments
Author Response to Official Review by Reviewer xZvm
2. W2 & W3: Independent Monte Carlo Validation of Critic Quality
EV is insensitive to a constant shift in value predictions, while PSA only evaluates response ranking. Improvements in both therefore cannot rule out systematic value bias or independently establish that local advantages are more accurate.
There is a lack of independent value validation using multiple continuations from the prompt-only actor under the same prefix.
Existing segment-level EV/PSA analyses and reference-answer/reference-solution ablations have not fully examined whether the reference-guided critic overestimates the actor's actual success probability or undervalues different yet correct trajectories.
We thank the reviewer for carefully examining our critic-quality analysis, highlighting the limitations of EV and PSA, and suggesting independent prefix-continuation validation.
During RL training, our primary goal in monitoring the critics is to compare their relative predictive performance. We therefore use explained variance (EV), a long-established statistical measure widely adopted for evaluating value predictions in PPO-based LLM RL. [1, 2, 3]
We agree that absolute prediction error and signed bias are needed to assess calibration alongside EV and PSA. We have therefore added an independent Monte Carlo evaluation of value predictions.
(1) Independent MC validation. We evaluate both critics at 6,144 prefixes from 256 problems in the training set. For value estimation, we use 32 independent continuations per prefix. For each fixed prefix, the value target is
(2) Value accuracy, calibration, and bias. Mean squared error (MSE) measures value-prediction error against independent MC targets. Expected calibration error (ECE) measures how well predicted success probabilities match observed success rates. Mean signed error captures average overestimation or underestimation.
Reference conditioning improves value accuracy and calibration. Predictions better match the actor's continuation success probabilities. The reference-guided critic underestimates success on average, with a smaller absolute bias than the standard critic's overestimation.
| Critic | Value MSE ↓ | ECE ↓ | Mean signed error |
|---|---|---|---|
| Standard critic | 0.10590 | 0.13712 | +0.10837 |
| Reference-guided critic | 0.08025 | 0.07834 | −0.04013 |
Signed error is the predicted value minus the Monte Carlo success estimate. Mean signed error closer to zero indicates less average bias.
(3) Alternative correct paths. Our core method supplies only the ground-truth final answer as critic-side reference information. In Appendix C.1, we compare reference answers with reference solutions and find that solution conditioning yields slightly lower downstream policy performance. We hypothesize that a specific reference solution may bias the critic toward that derivation, potentially leading it to undervalue alternative yet correct reasoning paths. This motivates our use of concise reference answers, which leave the intermediate solution path unspecified.
We are currently working on step-level scoring guided by high-quality reference solutions, with the aim of improving fine-grained credit assignment.
We will add a new subsection, C.3, to Appendix C to present these results and analyses, together with details of the continuation protocol and ECE binning.
[1] Reporting Explained Variance, 1981
[2] VinePPO: Refining Credit Assignment in RL Training of LLMs, 2024
[3] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks, 2025
Author Response to Official Review by Reviewer xZvm
3. W4: Alternative Discrepancy, Uncertainty, and Difficulty Controls
Comparisons with the prediction discrepancy between two standard critics, other uncertainty measures, or difficulty-based weighting methods, as well as comparisons between response-level and token-level reweighting, have not been fully explored.
Thank you for highlighting this gap and suggesting these controls. We acknowledge that our original analysis of discrepancy-based weighting was insufficient. Building on the token-selection experiments presented in our response to W1, we have added weighting controls on Qwen3-4B-Instruct-2507 to examine the weighting signal, response-level aggregation, and token alignment.
All five variants use reference-guided advantages and every valid response token. (a) Reference-guided PPO (+Ref) uses unit weights. (b) Sequence-level discrepancy weighting broadcasts one weight derived from to all tokens within each response. (c) Within-response shuffle permutes clipped discrepancy weights within each response before advantage whitening. (d) Token-entropy weighting substitutes full-vocabulary policy entropy for critic discrepancy under the same token-weight transformation. This provides a simpler heuristic that assigns larger update weights to tokens with higher policy uncertainty. (e) Full ReDiPPO applies normalized, clipped discrepancy weights to their corresponding token positions.
| Weighting variant | AIME24 | AIME25 | AIME26 | HMMT25 | Minerva | Olympiad | Avg. |
|---|---|---|---|---|---|---|---|
| Reference-guided PPO (+Ref) | 57.71 | 48.96 | 55.52 | 28.96 | 62.32 | 75.17 | 54.77 |
| Sequence-level discrepancy | 59.79 | 48.75 | 55.00 | 29.79 | 62.41 | 74.73 | 55.08 |
| Within-response shuffle | 55.31 | 45.73 | 54.48 | 28.23 | 61.76 | 72.87 | 53.06 |
| Token-entropy weighting | 58.44 | 48.43 | 53.75 | 28.54 | 61.07 | 74.22 | 54.08 |
| Full ReDiPPO | 60.68 | 49.79 | 56.77 | 30.63 | 62.84 | 75.45 | 56.03 |
Shuffling reduces average performance below +Ref despite preserving each response's weight distribution, supporting the importance of token alignment. Entropy weighting also falls below +Ref, and sequence-level weighting gives a smaller average gain. Full ReDiPPO achieves the highest performance on all six benchmarks among these variants.
The sequence control uses the absolute difference of response-mean values, normalized across responses and clipped before broadcast. This aggregation permits cancellation of signed differences and changes the weighting statistic. The shuffle control directly tests position alignment while preserving each response's clipped-weight distribution.
We will include these controls and their analysis in Section 4.3 (Ablation Studies) and Section 4.4 (Analysis of Reference-Standard Discrepancy).
4. W5: Independent Training Seeds and Computational Cost
The main table does not report results and uncertainty across multiple independent training seeds.
The authors report that, in the tested settings, this method increases training time by approximately 16.0%–23.9% compared with PPO, but do not provide a PPO comparison under the same computational budget.
Thank you for raising these questions. Due to computational constraints, each configuration in Table 1, Figure 3, and the additional training experiments uses one independent RL training run. Although we have not conducted multi-seed runs, ReDiPPO consistently outperforms PPO across different model backbones and evaluation datasets, providing empirical support for its effectiveness.
To assess the benefit of additional training, we extended PPO training on Qwen3-4B-Instruct-2507 to give it a larger H200 GPU-hour budget than ReDiPPO. Although ReDiPPO takes longer per training step, it reaches higher final performance. Extending PPO training yields modest improvements, but its performance remains below ReDiPPO even with a larger compute budget.
| Qwen3-4B-Instruct-2507 | GPU-hours (H200) | Gradient steps | Six-benchmark avg. (%) |
|---|---|---|---|
| ReDiPPO | 309.6 | 352 | 56.03 |
| PPO | 248.3 | 352 | 53.66 |
| PPO (extended training) | 361.2 | 480 | 54.68 |
We will incorporate these results and timing details into Appendix C.2.
Author Response to Official Review by Reviewer xZvm
We thank the reviewer for recognizing the combined contribution of reference-guided value estimation and discrepancy-aware token weighting. Your comments motivated additional prefix-continuation evaluations and controlled policy-training ablations, which we report below.
1. W1: Critic Reliability and the Justification for Reweighting
Thank you for raising this concern. The purpose of discrepancy is to identify tokens that merit greater emphasis during policy optimization.
We additionally assess the relative reliability of the two critics through independent Monte Carlo (MC) continuations from the prompt-only actor in our combined response to W2 and W3 below. Here, we first use controlled experiments to demonstrate the effectiveness of discrepancy-based token selection.
We hypothesize that large disagreements identify partial reasoning states whose policy updates can benefit from greater emphasis, and test this hypothesis through controlled token-selection experiments as below.
High-discrepancy token selection. To test the effectiveness of selecting tokens by discrepancy, we conduct experiments on Qwen3-4B-Instruct-2507 in which only 20% of the valid tokens in each response contribute to the actor update. All variants use reference-guided advantages. Within each response, we select tokens with the highest or lowest raw absolute critic discrepancy, or sample randomly. All three variants use the same selection fraction and unit weights. A binary actor-loss mask is applied after full-response advantage whitening.
High-20% exceeds Random-20% on all six benchmarks, with average performance ordered High > Random > Low. These results support discrepancy as an empirical criterion for identifying useful update positions. The all-token +Ref baseline reported in our response to W4 exceeds all three selection variants, motivating ReDiPPO's use of discrepancy-guided emphasis with supervision at every valid response token.
We further clarify the scope of unbiased estimation in our training procedure and its relationship to the original expected-return objective.
Critic learning uses terminal MC targets independently of discrepancy weighting. Both critics are supervised by the same terminal returns, which avoid bootstrapping bias. For continuations sampled from a fixed rollout policy without outcome-based filtering, these returns are unbiased samples of its value. Discrepancy weights enter only the actor objective.
The actor uses centered and normalized advantages. Whitening sets the mean of the weighted advantages to zero over valid batch tokens. Finite-batch centering and scaling can introduce gradient bias, while GRPO [1] provides an established example of effective LLM training with normalized advantages. Its group-mean centering alone introduces a constant rescaling relative to the leave-one-out estimator.
ReDiPPO applies discrepancy weighting as a bounded modification of the PPO surrogate. Token-dependent weights further change the relative contributions to the actor update. Their effectiveness is supported by the fixed-budget token-selection experiments above and the response-level, shuffled-weight, and entropy-weighting controls presented in our response to W4.
We will clarify this motivation and include the token-selection analysis in Sections 4.3 and 4.4.
[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models