- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes πPPO, which improves value estimation in RLVR by conditioning the critic on correctness-labeled reference solutions sampled for the same prompt, excluding the target trajectory itself. The actor’s inputs and the original reward objective remain unchanged.
On Qwen3-4B and Qwen3-8B, πPPO achieves average accuracies of 50.3% and 51.6% across five mathematical reasoning benchmarks, outperforming the strongest baseline, DAPO, by 4.1 and 2.3 percentage points. The paper also includes context ablations, value-estimation diagnostics, and experiments with smaller critics.
Critical strengths
-
The method makes effective use of information already available during training. Multiple rollouts naturally provide successful and unsuccessful attempts at the same problem. Using these attempts to assist the critic is a simple and practical design that preserves the actor’s inference interface and the original task reward.
-
The empirical improvements are supported by several complementary experiments. The gains are consistent across both actor sizes. Removing correctness labels, using references with the same outcome, or providing only the ground-truth answer substantially reduces performance, supporting the importance of the context construction. The smaller-critic experiments further suggest that the approach can compensate for some limitations in critic capacity.
Critical weaknesses
-
The computational cost is unclear. πPPO adds up to 8,192 tokens of critic context, so matching training steps does not ensure comparable compute. GPU configuration and training time are not reported, and it is unclear whether token counts include reference context and critic warm-up. This leaves both the gains under equal compute budgets and the savings from smaller critics uncertain.
-
The theoretical interpretation needs clarification. Proposition 1 establishes non-increasing optimal prediction risk under function-class inclusion, but does not demonstrate reduced irreducible error. With a fixed policy and independent rollouts, reference trajectories should be conditionally independent of the target’s future return given the full prompt and prefix. The gains may instead reflect easier learning for a finite-capacity critic; the explanation should distinguish these possibilities.
As a secondary limitation, training experiments focus on mathematics. The additional transfer evaluations do not establish effectiveness in code generation or agent training.
Questions for authors
-
Compute accounting: Does Table 1 include the critic’s reference tokens and warm-up costs? Please report the GPU configuration and total GPU-hours for the main comparisons. Do the gains remain under the same overall compute budget?
-
Context length and truncation: What is the maximum total critic input length, including the prompt, references, and target response? Please report the actual length distribution and explain what is truncated when this limit is reached. How sensitive is performance to reference length, particularly under a fixed total context budget?
-
Validation beyond mathematics: Could you provide results from training πPPO on a non-mathematical RLVR dataset, such as code generation with unit-test rewards or agent tasks with verifiable success? The current GPQA-Diamond and MMLU-Pro transfer evaluations do not directly establish effectiveness in these training settings.
-
Theoretical interpretation: With a fixed policy and independently sampled rollouts, what additional assumptions allow the references to reduce irreducible uncertainty given the full target state? If the benefit instead comes from easier learning for a finite-capacity critic, could the theoretical discussion distinguish these explanations?
4 comments
Response to Reviewer afVk [2/2]
Q2: Context length and truncation: What is the maximum total critic input length, including the prompt, references, and target response? Please report the actual length distribution and explain what is truncated when this limit is reached. How sensitive is performance to reference length, particularly under a fixed total context budget?
RE Q2:
Thank you for your question. We have added detailed length hyperparameters in Appendix C.1 (Table 3). Specifically, the 8,192-token limit (pi_contrast_max_tokens) applies to each reference individually and matches the maximum rollout length (we have made this clearer in the revision). Thus, no additional reference truncation is triggered on the critic side in our main experiments.
Under this setting, the actual critic input length distribution over the 250 training steps is:
| Backbone | Prompt | Single Rollout | Batch-Max Priv. Prompt | Effective Total Critic Input (Mean ± SD / Median / P10–P90) | Clamped |
|---|---|---|---|---|---|
| Qwen3-4B | 105 ± 4 | 4,515 ± 827 | 16,693 ± 416 (max 17,633) | 13,721 ± 2,480 / 14,494 / 9,425–15,801 | 0.0% |
| Qwen3-8B | 105 ± 4 | 4,619 ± 1,027 | 16,609 ± 661 (max 17,852) | 14,031 ± 3,080 / 15,325 / 8,178–16,506 | 0.0% |
Effective critic input combines the prompt, two sibling references plus template overhead, and the target rollout, averaging ~2.97× the state-only critic input.
When a reference exceeds its limit, our implementation truncates it from the left, preserving the final answer that echoes the provided correctness label, keeping the appended reference block coherent.
Regarding the context budget, we have updated the compute comparison in our response to W1 & Q1. Memory bottlenecks from long contexts can also be mitigated using systems techniques (e.g., context parallelism in Megatron). We currently retain complete references within the context budget used in our experiments.
Q3: Validation beyond mathematics: Could you provide results from training PPO on a non-mathematical RLVR dataset, such as code generation with unit-test rewards or agent tasks with verifiable success? The current GPQA-Diamond and MMLU-Pro transfer evaluations do not directly establish effectiveness in these training settings.
RE Q3:
Thank you for your question. Following your suggestion, we additionally validated our method on code generation, training Qwen3-4B on the coding subset of Eurus-2-RL-Data [1] for 250 steps under the same RLVR setup as our main experiments. We evaluated our method on HumanEval+, MBPP+, and LiveCodeBench (v6). The results are summarized below and have also been added to Appendix E.2 (Table 6).
| Method | HumanEval+ | MBPP+ | LiveCodeBench | Overall |
|---|---|---|---|---|
| GRPO | 85.3 | 69.0 | 29.4 | 61.2 |
| DAPO | 83.5 | 68.7 | 30.2 | 60.8 |
| PPO | 84.7 | 70.3 | 27.1 | 60.7 |
| VAPO | 79.9 | 71.6 | 28.4 | 60.0 |
| PPO | 83.1 | 74.3 | 29.4 | 62.3 |
| PPO (0.6B small critic) | 81.6 | 74.2 | 30.0 | 61.9 |
Avg@4 accuracy (%); Overall is the mean across the three benchmarks.
We mainly compare the baselines with the asymmetric version of PPO (0.6B small critic), where the overall compute is aligned with vanilla PPO and DAPO. Under this setting, asymmetric PPO achieves 61.9% Overall accuracy, outperforming both actor–critic baselines (60.7% for PPO and 60.0% for VAPO) and critic-free baselines (61.2% for GRPO and 60.8% for DAPO), while symmetric PPO further reaches 62.3%. These results confirm the effectiveness of PPO in non-mathematical RLVR training settings.
[1] Cui G, Yuan L, Wang Z, et al. Process reinforcement through implicit rewards[J]. Transactions on Machine Learning Research (TMLR), 2025.
Thank you again for your thoughtful review. We hope the clarifications and new experiments above help address your remaining concerns.
Thank you for your response, which has addressed my concerns. I will maintain my current score.
Thank you for the follow-up. We are glad that our clarifications and additional experiments have addressed your concerns. We sincerely appreciate your careful review and constructive feedback, which have helped us improve the paper.
Response to Reviewer afVk [1/2]
We sincerely thank you for your thoughtful comments and constructive questions. We are encouraged that you find our method a simple and practical use of the verified rollouts already available during training, and consider its gains well supported by complementary ablations and smaller-critic experiments. To address your remaining concerns, we make the following detailed response.
RE W1 & Q1:
Table 1 reports only actor-side token usage during policy training (“Gen.” and “Train.”), including critic warm-up but excluding additional critic-side reference processing.
Appendix F (Table 9) now reports end-to-end GPU-hours on 16 NVIDIA H20 GPUs, including rollout generation, actor and critic computation, reference processing, and full value pretraining. Based on this metric, we evaluate two compute-aligned comparisons:
Asymmetric πPPO matched to baseline compute (250 steps). Pairing πPPO with compact critics (0.6B for Qwen3-4B; 1.7B for Qwen3-8B) matches the total compute of 250-step PPO and DAPO (4B: 1,263 vs. 1,215 and 1,226 GPU-h; 8B: 1,548 vs. 1,565 and 1,534 GPU-h). Under this aligned budget, asymmetric πPPO reaches 47.8% / 49.9% Overall accuracy on 4B / 8B, outperforming DAPO (46.2% / 49.3%) and PPO (43.5% / 47.9%).
Symmetric πPPO vs. compute-aligned baselines (Qwen3-4B). Conversely, to evaluate symmetric πPPO under an equal budget, we also align compute by extending the baselines beyond 250 updates (specifically, PPO for 356 steps, DAPO for 303 steps, GRPO for 425 steps, and VAPO for 403 steps). This additional training lifts PPO (43.5% → 46.4%) and DAPO (46.2% → 47.4%), while GRPO and VAPO reach 45.4% and 44.3% Overall; yet 250-step symmetric πPPO (50.3%) still outperforms all baselines.
Together, these two comparisons consolidate our evaluation and confirm πPPO’s gains under equal compute, both at the baseline budget that asymmetric πPPO naturally matches and at symmetric πPPO’s budget with extended baselines.
RE W2 & Q4:
Thank you for your insightful comments. Proposition 1 decomposes the optimal prediction-risk gap into Bayes-risk reduction and the difference in approximation errors between the two critic classes.
In the original Eqs. (3)–(4), we write Ciσ as trajectory sets {y(j)}, leaving the verifier labels implicit in the σ∈{+,−} grouping (and fallback [y⋆]), whereas I in fact carries those outcomes (As the reviewer notes, if I contains only independently sampled rollouts from the same fixed policy, it does not reduce Bayes risk given the full state s.).
We now write them explicitly as (y(j),r(x,y(j))) in the revised Eqs. (3)–(4). In this way, for the irreducible Bayes risk, strict reduction Var(R)(Δπ,+−Δπ)>0 holds iff R⊥⊥I∣s, which holds in our setting whenever the external verification rule r(x,⋅) involves task criteria beyond x. We have updated the manuscript and added an explicit discussion of this condition in Appendix B.3.
For finite-capacity approximation, reducing the critic from 8B to 1.7B decreases EV by only 0.027 for πPPO, compared with 0.082 for vanilla PPO (Fig. 3). This gap provides empirical support and should correspond solely to the approximation-error term in Proposition 1.