acceptodds
Under review as a conference paper at ICLR 2027

Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR

Abstract

Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-critic methods such as PPO address this by learning value functions to construct token-level advantages. Their effectiveness, however, hinges on reliable value estimation, a difficult task requiring the critic to both assess progress toward a correct solution and anticipate an evolving policy's future behavior; errors in either can compromise credit assignment and destabilize online training. In this paper, we revisit the standard state-only formulation of value estimation and propose **PPO**, a self-privileged actor-critic framework. By reusing verified same-prompt rollouts as contrastive evidence, PPO helps the critic assess intermediate reasoning against successful and failed attempts, while preserving standard policy optimization and the deployment interface. Experiments show that PPO consistently improves value-estimation quality by a substantial margin and outperforms representative actor-critic and critic-free RLVR baselines on challenging mathematical reasoning benchmarks, while remaining effective even when paired with substantially smaller asymmetric critics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Reasoning, Post-Training, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_afVkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes πPPO, which improves value estimation in RLVR by conditioning the critic on correctness-labeled reference solutions sampled for the same prompt, excluding the target trajectory itself. The actor’s inputs and the original reward objective remain unchanged.

On Qwen3-4B and Qwen3-8B, πPPO achieves average accuracies of 50.3% and 51.6% across five mathematical reasoning benchmarks, outperforming the strongest baseline, DAPO, by 4.1 and 2.3 percentage points. The paper also includes context ablations, value-estimation diagnostics, and experiments with smaller critics.

Critical strengths
  1. The method makes effective use of information already available during training. Multiple rollouts naturally provide successful and unsuccessful attempts at the same problem. Using these attempts to assist the critic is a simple and practical design that preserves the actor’s inference interface and the original task reward.

  2. The empirical improvements are supported by several complementary experiments. The gains are consistent across both actor sizes. Removing correctness labels, using references with the same outcome, or providing only the ground-truth answer substantially reduces performance, supporting the importance of the context construction. The smaller-critic experiments further suggest that the approach can compensate for some limitations in critic capacity.

Critical weaknesses
  1. The computational cost is unclear. πPPO adds up to 8,192 tokens of critic context, so matching training steps does not ensure comparable compute. GPU configuration and training time are not reported, and it is unclear whether token counts include reference context and critic warm-up. This leaves both the gains under equal compute budgets and the savings from smaller critics uncertain.

  2. The theoretical interpretation needs clarification. Proposition 1 establishes non-increasing optimal prediction risk under function-class inclusion, but does not demonstrate reduced irreducible error. With a fixed policy and independent rollouts, reference trajectories should be conditionally independent of the target’s future return given the full prompt and prefix. The gains may instead reflect easier learning for a finite-capacity critic; the explanation should distinguish these possibilities.

As a secondary limitation, training experiments focus on mathematics. The additional transfer evaluations do not establish effectiveness in code generation or agent training.

Questions for authors
  1. Compute accounting: Does Table 1 include the critic’s reference tokens and warm-up costs? Please report the GPU configuration and total GPU-hours for the main comparisons. Do the gains remain under the same overall compute budget?

  2. Context length and truncation: What is the maximum total critic input length, including the prompt, references, and target response? Please report the actual length distribution and explain what is truncated when this limit is reached. How sensitive is performance to reference length, particularly under a fixed total context budget?

  3. Validation beyond mathematics: Could you provide results from training πPPO on a non-mathematical RLVR dataset, such as code generation with unit-test rewards or agent tasks with verifiable success? The current GPQA-Diamond and MMLU-Pro transfer evaluations do not directly establish effectiveness in these training settings.

  4. Theoretical interpretation: With a fixed policy and independently sampled rollouts, what additional assumptions allow the references to reduce irreducible uncertainty given the full target state? If the benefit instead comes from easier learning for a finite-capacity critic, could the theoretical discussion distinguish these explanations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer afVk [1/2]

We sincerely thank you for your thoughtful comments and constructive questions. We are encouraged that you find our method a simple and practical use of the verified rollouts already available during training, and consider its gains well supported by complementary ablations and smaller-critic experiments. To address your remaining concerns, we make the following detailed response.


W1: The computational cost is unclear. π\piPPO adds up to 8,192 tokens of critic context, so matching training steps does not ensure comparable compute. GPU configuration and training time are not reported, and it is unclear whether token counts include reference context and critic warm-up. This leaves both the gains under equal compute budgets and the savings from smaller critics uncertain.

Q1: Compute accounting: Does Table 1 include the critic’s reference tokens and warm-up costs? Please report the GPU configuration and total GPU-hours for the main comparisons. Do the gains remain under the same overall compute budget?

RE W1 & Q1:

Table 1 reports only actor-side token usage during policy training (“Gen.” and “Train.”), including critic warm-up but excluding additional critic-side reference processing.

Appendix F (Table 9) now reports end-to-end GPU-hours on 16 NVIDIA H20 GPUs, including rollout generation, actor and critic computation, reference processing, and full value pretraining. Based on this metric, we evaluate two compute-aligned comparisons:

  1. Asymmetric π\piPPO matched to baseline compute (250 steps). Pairing π\piPPO with compact critics (0.6B for Qwen3-4B; 1.7B for Qwen3-8B) matches the total compute of 250-step PPO and DAPO (4B: 1,263 vs. 1,215 and 1,226 GPU-h; 8B: 1,548 vs. 1,565 and 1,534 GPU-h). Under this aligned budget, asymmetric π\piPPO reaches 47.8% / 49.9% Overall accuracy on 4B / 8B, outperforming DAPO (46.2% / 49.3%) and PPO (43.5% / 47.9%).

  2. Symmetric π\piPPO vs. compute-aligned baselines (Qwen3-4B). Conversely, to evaluate symmetric π\piPPO under an equal budget, we also align compute by extending the baselines beyond 250 updates (specifically, PPO for 356 steps, DAPO for 303 steps, GRPO for 425 steps, and VAPO for 403 steps). This additional training lifts PPO (43.5% → 46.4%) and DAPO (46.2% → 47.4%), while GRPO and VAPO reach 45.4% and 44.3% Overall; yet 250-step symmetric π\piPPO (50.3%) still outperforms all baselines.

Together, these two comparisons consolidate our evaluation and confirm π\piPPO’s gains under equal compute, both at the baseline budget that asymmetric π\piPPO naturally matches and at symmetric π\piPPO’s budget with extended baselines.

W2: The theoretical interpretation needs clarification. Proposition 1 establishes non-increasing optimal prediction risk under function-class inclusion, but does not demonstrate reduced irreducible error. With a fixed policy and independent rollouts, reference trajectories should be conditionally independent of the target’s future return given the full prompt and prefix. The gains may instead reflect easier learning for a finite-capacity critic; the explanation should distinguish these possibilities.

Q4: Theoretical interpretation: With a fixed policy and independently sampled rollouts, what additional assumptions allow the references to reduce irreducible uncertainty given the full target state? If the benefit instead comes from easier learning for a finite-capacity critic, could the theoretical discussion distinguish these explanations?

RE W2 & Q4:

Thank you for your insightful comments. Proposition 1 decomposes the optimal prediction-risk gap into Bayes-risk reduction and the difference in approximation errors between the two critic classes.

In the original Eqs. (3)–(4), we write Ciσ\mathcal{C}_i^\sigma as trajectory sets {y(j)}\{\boldsymbol{y}^{(j)}\}, leaving the verifier labels implicit in the σ∈{+,−}\sigma \in \{+, -\} grouping (and fallback [y⋆][\boldsymbol{y}^\star]), whereas I\mathcal{I} in fact carries those outcomes (As the reviewer notes, if I\mathcal{I} contains only independently sampled rollouts from the same fixed policy, it does not reduce Bayes risk given the full state ss.).

We now write them explicitly as (y(j),r(x,y(j)))(\boldsymbol{y}^{(j)}, r(\boldsymbol{x}, \boldsymbol{y}^{(j)})) in the revised Eqs. (3)–(4). In this way, for the irreducible Bayes risk, strict reduction Var(R)(Δπ,+−Δπ)>0\mathrm{Var}(R)(\Delta^{\pi,+} - \Delta^\pi) > 0 holds iff R̸⊥⊥I∣sR\not\mathrel{\perp\mkern-10mu\perp}\mathcal{I}\mid s, which holds in our setting whenever the external verification rule r(x,⋅)r(\boldsymbol{x}, \cdot) involves task criteria beyond x\boldsymbol{x}. We have updated the manuscript and added an explicit discussion of this condition in Appendix B.3.

For finite-capacity approximation, reducing the critic from 8B to 1.7B decreases EV by only 0.027 for π\piPPO, compared with 0.082 for vanilla PPO (Fig. 3). This gap provides empirical support and should correspond solely to the approximation-error term in Proposition 1.

AuthorsOpenReview ↗

Response to Reviewer afVk [2/2]

Q2: Context length and truncation: What is the maximum total critic input length, including the prompt, references, and target response? Please report the actual length distribution and explain what is truncated when this limit is reached. How sensitive is performance to reference length, particularly under a fixed total context budget?

RE Q2:

Thank you for your question. We have added detailed length hyperparameters in Appendix C.1 (Table 3). Specifically, the 8,192-token limit (pi_contrast_max_tokens) applies to each reference individually and matches the maximum rollout length (we have made this clearer in the revision). Thus, no additional reference truncation is triggered on the critic side in our main experiments.

Under this setting, the actual critic input length distribution over the 250 training steps is:

BackbonePromptSingle RolloutBatch-Max Priv. PromptEffective Total Critic Input (Mean ± SD / Median / P10–P90)Clamped
Qwen3-4B105 ± 44,515 ± 82716,693 ± 416 (max 17,633)13,721 ± 2,480 / 14,494 / 9,425–15,8010.0%
Qwen3-8B105 ± 44,619 ± 1,02716,609 ± 661 (max 17,852)14,031 ± 3,080 / 15,325 / 8,178–16,5060.0%

Effective critic input combines the prompt, two sibling references plus template overhead, and the target rollout, averaging ~2.97× the state-only critic input.

When a reference exceeds its limit, our implementation truncates it from the left, preserving the final answer that echoes the provided correctness label, keeping the appended reference block coherent.

Regarding the context budget, we have updated the compute comparison in our response to W1 & Q1. Memory bottlenecks from long contexts can also be mitigated using systems techniques (e.g., context parallelism in Megatron). We currently retain complete references within the context budget used in our experiments.

Q3: Validation beyond mathematics: Could you provide results from training π\piPPO on a non-mathematical RLVR dataset, such as code generation with unit-test rewards or agent tasks with verifiable success? The current GPQA-Diamond and MMLU-Pro transfer evaluations do not directly establish effectiveness in these training settings.

RE Q3:

Thank you for your question. Following your suggestion, we additionally validated our method on code generation, training Qwen3-4B on the coding subset of Eurus-2-RL-Data [1] for 250 steps under the same RLVR setup as our main experiments. We evaluated our method on HumanEval+, MBPP+, and LiveCodeBench (v6). The results are summarized below and have also been added to Appendix E.2 (Table 6).

MethodHumanEval+MBPP+LiveCodeBenchOverall
GRPO85.369.029.461.2
DAPO83.568.730.260.8
PPO84.770.327.160.7
VAPO79.971.628.460.0
π\piPPO83.174.329.462.3
π\piPPO (0.6B small critic)81.674.230.061.9

Avg@4 accuracy (%); Overall is the mean across the three benchmarks.

We mainly compare the baselines with the asymmetric version of π\piPPO (0.6B small critic), where the overall compute is aligned with vanilla PPO and DAPO. Under this setting, asymmetric π\piPPO achieves 61.9% Overall accuracy, outperforming both actor–critic baselines (60.7% for PPO and 60.0% for VAPO) and critic-free baselines (61.2% for GRPO and 60.8% for DAPO), while symmetric π\piPPO further reaches 62.3%. These results confirm the effectiveness of π\piPPO in non-mathematical RLVR training settings.

[1] Cui G, Yuan L, Wang Z, et al. Process reinforcement through implicit rewards[J]. Transactions on Machine Learning Research (TMLR), 2025.


Thank you again for your thoughtful review. We hope the clarifications and new experiments above help address your remaining concerns.

Thank you for your response, which has addressed my concerns. I will maintain my current score.

Thank you for the follow-up. We are glad that our clarifications and additional experiments have addressed your concerns. We sincerely appreciate your careful review and constructive feedback, which have helped us improve the paper.

Reviewer_znLiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

To improve the accuracy of critic value estimates, the paper proposes πPPO, which augments the value model’s input with other solutions to the same problem as reference context. The experimental results reported by the authors indicate that this approach is effective in the tested settings.

Critical strengths
  1. The method makes effective use of rollout data, reusing existing samples to provide richer information for value estimation and policy optimization.
  2. The paper presents a comprehensive experimental evaluation.
  3. The paper is clearly written, well organized, and easy to follow.
Critical weaknesses
  1. The paper’s central contribution is to augment the critic’s input with additional privileged information. However, this general idea has already been explored in several related works [1–4]. The remaining distinction appears to lie primarily in how the privileged context is constructed. In its current form, this variation does not convincingly establish a substantive methodological advance over existing approaches.
  2. The paper does not report the main benchmark results across multiple independent RL training seeds or provide corresponding uncertainty estimates. Averaging multiple generations per test problem reduces evaluation sampling noise, but does not capture variability in the RL training process.

[1] Pinto L, Andrychowicz M, Welinder P, et al. Asymmetric actor critic for image-based robot learning[J]. arXiv preprint arXiv:1710.06542, 2017.

[2] Yu C, Velu A, Vinitsky E, et al. The surprising effectiveness of ppo in cooperative multi-agent games[J]. Advances in neural information processing systems, 2022, 35: 24611-24624.

[3] Zhou Y, Jiang S, Tian Y, et al. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks[J]. arXiv preprint arXiv:2503.15478, 2025.

[4] Venkatraman S, Dinot M, Aitchison L. Le Critique: Privileged Value Functions for LLM Reinforcement Learning[J]. arXiv preprint arXiv:2608.16739, 2026.

Questions for authors
  1. I am not convinced by the reported results in their current form. The paper reports consistent improvements over all compared RL baselines across the mathematical benchmarks, despite the primary modification being additional context for the critic. The magnitude and consistency of these gains require stronger verification. Could you provide results across multiple independent training seeds and additional controlled experiments to establish their reliability and attribution to the proposed modification?

  2. Privileged information may substantially simplify the critic’s prediction task, but more accurate value estimates do not automatically imply better policy optimization. An action-independent baseline leaves the expected gradient unchanged in the ideal, unclipped policy-gradient setting. Could you therefore provide direct evidence showing how the proposed method improves finite-sample gradient estimation, affects gradient variance, or interacts with PPO clipping, beyond improvements in critic prediction accuracy?

2 comments
AuthorsOpenReview ↗

Response to Reviewer znLi [1/2]

Thank you for your time and effort invested in reviewing our work. We are encouraged that you find our method effective in reusing existing rollouts for value estimation, and consider our experimental evaluation comprehensive and our paper clearly written. For your comments and questions, we would like to respond as follows:


W1: The paper’s central contribution is to augment the critic’s input with additional privileged information. However, this general idea has already been explored in several related works [1–4]. The remaining distinction appears to lie primarily in how the privileged context is constructed. In its current form, this variation does not convincingly establish a substantive methodological advance over existing approaches.

RE W1:

We thank the reviewer for the thoughtful comments on the conceptual connections between our work and the literature.

Early asymmetric actor–critic methods [1,2] stemmed from partially observable settings with unequal access to state information during training, such as camera observations for the actor and full simulator states for the critic [1]. Le Critique [4] explores privileged value functions for LLMs concurrently with our work.

Our insight is that the same rollouts used for policy optimization can serve a second role in bootstrapping a better training procedure: their success–failure contrast can further be exploited to provide the critic with evidence of the task’s value structure in RLVR with binary rewards. (SWEET-RL [3] learns a turn-level advantage model through Bradley–Terry preference learning and uses it to construct preference pairs for DPO. This focus on preference-based policy training is less directly related to the LUPI (learning using privileged information) formulation studied here.)

We have accordingly added a new paragraph to Section 6 (Related Work), providing a broader and more detailed discussion of these works and their relationship to our approach, building on the reviewer's helpful comments.

W2: The paper does not report the main benchmark results across multiple independent RL training seeds or provide corresponding uncertainty estimates. Averaging multiple generations per test problem reduces evaluation sampling noise, but does not capture variability in the RL training process.

Q1: I am not convinced by the reported results in their current form. The paper reports consistent improvements over all compared RL baselines across the mathematical benchmarks, despite the primary modification being additional context for the critic. The magnitude and consistency of these gains require stronger verification. Could you provide results across multiple independent training seeds and additional controlled experiments to establish their reliability and attribution to the proposed modification?

RE W2 & Q1:

Our main experiments were conducted with the default data seed 0 in VeRL. To complement the experiments with more statistical evidence, we reran the experiments with seeds 42 and 1234 for PPO, DAPO, π\piPPO and its asymmetric critic version on both Qwen3-4B and Qwen3-8B. The summarized results are as follows (with detailed per-seed and per-benchmark results updated in Appendix E.3, Table 7).

MethodQwen3-4B (Seed 0 / 42 / 1234)Qwen3-4B (Mean ± SD)Qwen3-8B (Seed 0 / 42 / 1234)Qwen3-8B (Mean ± SD)
DAPO46.2 / 44.2 / 44.144.8 ± 1.249.3 / 48.4 / 49.849.2 ± 0.7
PPO43.5 / 45.1 / 42.743.8 ± 1.247.9 / 49.5 / 49.949.1 ± 1.1
π\piPPO50.3 / 48.9 / 46.948.7 ± 1.751.6 / 50.4 / 51.651.2 ± 0.7
π\piPPO (small critic)47.8 / 46.1 / 44.346.1 ± 1.849.9 / 49.4 / 50.149.8 ± 0.4

Overall accuracy (%) across the five math benchmarks (mean ± sample SD over seeds 0, 42, and 1234; small critic denotes 0.6B for Qwen3-4B and 1.7B for Qwen3-8B).

Across three training seeds, both π\piPPO variants retain higher mean Overall accuracy than PPO and DAPO on both backbones, including under matched compute budgets with the small critic. These results strengthen the empirical evidence for π\piPPO’s effectiveness and the robustness of its Overall gains to training variability.

Regarding attribution, the compared policy-gradient methods target the same reward objective through different gradient estimators. We attribute π\piPPO’s gains primarily to improved baseline quality (e.g., better distinguishing promising reasoning continuations and tracking local changes in expected return), supported by our value-estimation analyses (see also the gradient-variance evidence for Q2 in the next comment).

AuthorsOpenReview ↗

Response to Reviewer znLi [2/2]

Q2: Privileged information may substantially simplify the critic’s prediction task, but more accurate value estimates do not automatically imply better policy optimization. An action-independent baseline leaves the expected gradient unchanged in the ideal, unclipped policy-gradient setting. Could you therefore provide direct evidence showing how the proposed method improves finite-sample gradient estimation, affects gradient variance, or interacts with PPO clipping, beyond improvements in critic prediction accuracy?

RE Q2:

Thank you for this insightful question.

Our EV and value-loss results (Fig. 2), together with the MC-based analyses (Figs. 4–6), support improved baseline quality. In our experiments, we use each rollout batch for a single actor update, following DeepSeekMath [5] and Open-Reasoner-Zero [6], so clipping is inactive in the exact on-policy case (with occasional activations due to training–inference mismatch); we have made this, together with all detailed training hyperparameters, explicit in the revised Appendix C.1.

In this way, as the reviewer notes, for the ideal, unclipped estimator, a valid baseline preserves the expected gradient while potentially reducing its finite-sample variance [7], and this variance can be reflected by EV: since the baseline-dependent part of token-gradient variance is E[(R−bt)2∥∇θlog⁡πθ(at∣st)∥2]\mathbb{E}\big[(R - b_t)^2 \Vert\nabla_\theta \log \pi_\theta(a_t \mid s_t)\Vert^2\big] and Var(R−bt)=(1−EV) Var(R)\mathrm{Var}(R - b_t) = (1 - \mathrm{EV})\,\mathrm{Var}(R), the baseline-induced gradient variance scales directly with 1−EV1 - \mathrm{EV}.

We also examine this empirically. Since gradient variance also depends on the actor's score function, we fix the final-step Qwen3-4B π\piPPO actor and its rollouts (16 batches of 32 training prompts, 16 rollouts each) and vary only the baseline across EV levels: b=Rˉb=\bar{R}, the same critic with the privileged block removed (Vϕ+(s,∅)V_\phi^+(s,\varnothing)), and the full critic (Vϕ+(s,I)V_\phi^+(s,\mathcal{I})):

BaselineEVVar(AA)E∥g^∥2\mathbb{E}\Vert\hat{g}\Vert^21−EV1-\mathrm{EV}
b=Rˉb=\bar{R}0.0000.2291 (1.00)3.02e-3 (1.00)1.00
Vϕ+(s,∅)V_\phi^+(s,\varnothing)0.2780.1668 (0.73)2.32e-3 (0.77)0.72
Vϕ+(s,I)V_\phi^+(s,\mathcal{I})0.6280.0862 (0.38)7.67e-4 (0.25)0.37

Parentheses give ratios to b=Rˉb=\bar{R}.

The results are consistent with our analysis: both Var\mathrm{Var} (0.73,0.380.73, 0.38) and measured gradient variance E∥g^∥2\mathbb{E}\Vert\hat{g}\Vert^2 (0.77,0.250.77, 0.25) scale closely with 1−EV1-\mathrm{EV} (0.72,0.370.72, 0.37) as expected from our derivation.

Thus, improved baseline quality, reflected in higher EV, can translate into lower variance in policy-gradient estimates, providing a more reliable gradient signal for policy optimization.

[5] Shao Z, Wang P, Zhu Q, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models[J]. arXiv preprint arXiv:2402.03300, 2024.

[6] Hu J, Zhang Y, Han Q, et al. Open-Reasoner-Zero: An open source approach to scaling up reinforcement learning on the base model[J]. arXiv preprint arXiv:2503.24290, 2025.

[7] Greensmith E, Bartlett P L, Baxter J. Variance reduction techniques for gradient estimates in reinforcement learning[J]. Journal of Machine Learning Research, 2004, 5: 1471–1530.


Thank you again for your time and effort evaluating our paper. We hope our responses and these additional results could help resolve your concerns.

Reviewer_P87MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes πPPO, an actor–critic framework that improves value estimation in RLVR by incorporating privileged information from verified same-prompt rollouts. Specifically, it samples a group of rollouts for each prompt and uses successful and unsuccessful sibling trajectories as privileged context for the critic, while retaining the standard PPO policy optimization objective. Experiments on mathematical reasoning benchmarks show improved value estimation and downstream performance over existing RLVR baselines, including with substantially smaller critics.

Critical strengths
  1. The proposed approach is simple and intuitive, leveraging verified sibling rollouts to improve critic value estimation without modifying the PPO policy optimization objective.

  2. The method consistently outperforms several actor–critic and critic-free RLVR baselines on mathematical reasoning benchmarks, with improvements in both value estimation and downstream accuracy.

  3. The method maintains competitive performance with substantially smaller critics, suggesting its potential for more parameter-efficient actor–critic training.

Critical weaknesses
  1. The proposed method introduces additional computational overhead compared to standard PPO, as it requires sampling a group of GG rollouts per prompt and processing additional reference trajectories in the critic. However, the paper does not provide wall-clock time or FLOPs comparisons to quantify this overhead.

  2. The theoretical results do not establish the superiority of the proposed method and are insufficiently connected to the empirical findings. In particular, Proposition 1 only establishes a general property of conditional variance, while Proposition 2 provides a necessary condition relating value prediction error to success–failure contrast. Neither result explains why the proposed contrastive privileged context should lead to better value estimation or improved policy optimization, leaving a gap between the theoretical analysis, the proposed method, and the experimental results.

  3. The experimental evaluation is relatively limited, as the main experiments focus exclusively on mathematical reasoning tasks using two models from the same family. Although the paper includes additional evaluations on GPQA-Diamond and MMLU-Pro, the improvements on these benchmarks are relatively marginal. More comprehensive experiments across diverse tasks and model families are needed to demonstrate the generalizability of the proposed method.

  4. The experiments lack an ablation on the rollout group size GG, which is a key component of the proposed method. Since the availability and diversity of successful and unsuccessful reference trajectories depend on the group size, it is important to investigate how GG affects value estimation, downstream performance, and computational cost.

  5. The experimental results lack sufficient evidence of robustness. In particular, the paper does not report results across multiple random seeds or provide variance estimates for the main benchmark comparisons, making it difficult to assess whether the reported improvements are statistically reliable.

Questions for authors
  1. How does the computational cost of πPPO compare with standard PPO and other RLVR baselines in terms of training FLOPs, wall-clock time, and memory consumption, considering both grouped rollout generation and the additional reference trajectories processed by the critic?

  2. Could the authors further clarify how Propositions 1 and 2 theoretically justify the proposed contrastive privileged context? In particular, under what conditions can independently sampled sibling rollouts improve value estimation beyond the information already contained in the target state?

  3. Can the authors provide additional experiments on more diverse tasks and model families to demonstrate the generalizability of πPPO beyond mathematical reasoning?

  4. How sensitive is πPPO to the rollout group size GG? Could the authors investigate different group sizes and analyze their effects on reference diversity, value estimation quality, downstream performance, and computational efficiency?

  5. Could the authors report results across multiple random seeds, including standard deviations or confidence intervals, to establish the statistical reliability of the reported improvements?

6 comments
AuthorsOpenReview ↗

Response to Reviewer P87M [1/4]

Thank you for your careful review and constructive feedback, which have been crucial in improving our paper. We are glad that you regard our approach as simple and intuitive and see its competitive performance with smaller critics as promising for parameter-efficient actor–critic training. We would like to address your concerns as follows:


W1: The proposed method introduces additional computational overhead compared to standard PPO, as it requires sampling a group of (G) rollouts per prompt and processing additional reference trajectories in the critic. However, the paper does not provide wall-clock time or FLOPs comparisons to quantify this overhead.

Q1: How does the computational cost of π\piPPO compare with standard PPO and other RLVR baselines in terms of training FLOPs, wall-clock time, and memory consumption, considering both grouped rollout generation and the additional reference trajectories processed by the critic?

RE W1 & Q1:

Thank you for your question. We would like to kindly explain that grouped rollouts per prompt do not themselves introduce additional computation in our setting, since the number of trajectories used per optimization step is fixed and equal across all methods, as reflected by the comparable “Train.” tokens in Table 1.

We have added a computation overhead section in Appendix F (Table 9) to quantify how the critic’s additional reference context affects training cost, measured in end-to-end GPU-hours on 16 NVIDIA H20 GPUs (including rollout generation, actor and critic computation, and full value pretraining). The main results are:

  1. Computational overhead. Over 250 steps on Qwen3-4B / 8B, PPO and DAPO take 1,215 / 1,565 and 1,226 / 1,534 total GPU-h; asymmetric π\piPPO (0.6B / 1.7B critic) matches this budget at 1,263 / 1,548 GPU-h, while symmetric π\piPPO takes 1,681 / 2,333 GPU-h. Meanwhile, Table 9 also reports step-level MFU: 47.7% / 55.2% for symmetric π\piPPO, compared with 34.1% / 43.6% for PPO and 26.9% / 36.1% for DAPO. Peak GPU memory is comparable across methods in our runs, as it is dominated by the rollout KV-cache allocation under the same decoding configuration.

  2. Performance under matched compute. At the baseline-level budget (250 steps), asymmetric π\piPPO reaches 47.8% / 49.9% Overall accuracy on 4B / 8B, outperforming DAPO (46.2% / 49.3%) and PPO (43.5% / 47.9%). When extending 4B baselines to match symmetric π\piPPO’s training GPU-hours (356 steps for PPO, 303 steps for DAPO, 425 steps for GRPO, and 403 steps for VAPO), 250-step symmetric π\piPPO (50.3%) still outperforms all baselines. We have updated these results in Appendix F (Table 10).

AuthorsOpenReview ↗

Response to Reviewer P87M [2/4]

W2: The theoretical results do not establish the superiority of the proposed method and are insufficiently connected to the empirical findings. In particular, Proposition 1 only establishes a general property of conditional variance, while Proposition 2 provides a necessary condition relating value prediction error to success–failure contrast. Neither result explains why the proposed contrastive privileged context should lead to better value estimation or improved policy optimization, leaving a gap between the theoretical analysis, the proposed method, and the experimental results.

Q2: Could the authors further clarify how Propositions 1 and 2 theoretically justify the proposed contrastive privileged context? In particular, under what conditions can independently sampled sibling rollouts improve value estimation beyond the information already contained in the target state?

RE W2 & Q2:

Thank you for your insightful comments. In our setting (γ=λ=1\gamma = \lambda = 1 at critic side, single on-policy update per batch), the critic directly serves as the token-level reward baseline. We clarify both links below and have updated Eqs. (3)–(4) and Appendix B.3 accordingly:

  1. How Propositions 1–2 justify contrastive context. Proposition 1 decomposes the optimal MSE gap into Bayes-risk reduction and the critic approximation-error gap, where the Bayes-risk reduction equals the gain in Bayes-optimal success–failure contrast in Proposition 2, Var(R)(Δπ,+−Δπ)≥0\mathrm{Var}(R)(\Delta^{\pi,+} - \Delta^\pi) \ge 0 (Appendix B.3). Unlabeled independent siblings satisfy R⊥⊥I∣sR\mathrel{\perp\mkern-10mu\perp}\mathcal{I}\mid s and do not reduce Bayes risk; however, I\mathcal{I} pairs siblings with verifier outcomes (y(j),r(x,y(j)))(\boldsymbol{y}^{(j)}, r(\boldsymbol{x}, \boldsymbol{y}^{(j)})) (made explicit in the revised Eqs. 3–4), yielding strict reduction (R̸⊥⊥I∣sR\not\mathrel{\perp\mkern-10mu\perp}\mathcal{I}\mid s) whenever r(x,⋅)r(\boldsymbol{x}, \cdot) involves task criteria beyond x\boldsymbol{x}. Beyond the Bayes-risk term, opposite-outcome pairs also reduce the approximation-error term by providing same-prompt anchors for a finite-capacity critic, consistent with Table 2 and the smaller EV drop when shrinking the critic from 8B to 1.7B (0.0270.027 for π\piPPO vs. 0.0820.082 for PPO, Fig. 3). Together, these additions help bridge the gap between Propositions 1–2 and our method by clarifying the conditions under which sibling rollouts effectively improve value estimation beyond ss.

  2. Why a more accurate baseline improves optimization. An action-independent baseline leaves the expected policy gradient unchanged, while its accuracy governs the finite-sample gradient variance [1]: for token baseline btb_t, the baseline-dependent variance term E[(R−bt)2∥∇θlog⁡πθ(at∣st)∥2]\mathbb{E}\big[(R - b_t)^2 \Vert\nabla_\theta \log \pi_\theta(a_t \mid s_t)\Vert^2\big] scales with Var(R−bt)=(1−EV) Var(R)\mathrm{Var}(R - b_t) = (1 - \mathrm{EV})\,\mathrm{Var}(R). Hence, the higher EV of π\piPPO's critic (Fig. 2) directly translates into lower-variance gradient estimates, curbing noisy updates and stabilizing policy optimization [2, 3].

[1] Greensmith E, Bartlett P L, Baxter J. Variance reduction techniques for gradient estimates in reinforcement learning[J]. Journal of Machine Learning Research, 2004, 5: 1471–1530.

[2] Chung W, Thomas V, Machado M C, et al. Beyond variance reduction: Understanding the true impact of baselines on policy optimization[C]//International Conference on Machine Learning. PMLR, 2021: 1999–2009.

[3] Mei J, Chung W, Thomas V, et al. The role of baselines in policy gradient optimization[J]. Advances in Neural Information Processing Systems, 2022, 35: 17818–17830.

AuthorsOpenReview ↗

Response to Reviewer P87M [3/4]

W3: The experimental evaluation is relatively limited, as the main experiments focus exclusively on mathematical reasoning tasks using two models from the same family. Although the paper includes additional evaluations on GPQA-Diamond and MMLU-Pro, the improvements on these benchmarks are relatively marginal. More comprehensive experiments across diverse tasks and model families are needed to demonstrate the generalizability of the proposed method.

Q3: Can the authors provide additional experiments on more diverse tasks and model families to demonstrate the generalizability of π\piPPO beyond mathematical reasoning?

RE W3 & Q3:

Thank you for the suggestion. To evaluate the generalizability of π\piPPO beyond mathematical reasoning, we conducted additional RLVR training experiments on code generation. Specifically, we trained Qwen3-4B for 250 steps on the coding split of Eurus-2-RL-Data [4] following the same RLVR protocol as our main experiments, and evaluated performance across HumanEval+, MBPP+, and LiveCodeBench (v6) (now included in Appendix E.2, Table 6):

MethodHumanEval+MBPP+LiveCodeBenchOverall
GRPO85.369.029.461.2
DAPO83.568.730.260.8
PPO84.770.327.160.7
VAPO79.971.628.460.0
π\piPPO83.174.329.462.3
π\piPPO (0.6B small critic)81.674.230.061.9

Avg@4 accuracy (%); Overall is the mean across the three benchmarks.

We mainly compare the baselines with asymmetric π\piPPO using a 0.6B small critic, whose overall compute is aligned with vanilla PPO and DAPO. In this setting, asymmetric π\piPPO achieves 61.9% Overall accuracy, outperforming the actor–critic baselines PPO (60.7%) and VAPO (60.0%), as well as the critic-free baselines GRPO (61.2%) and DAPO (60.8%). Symmetric π\piPPO further improves the Overall accuracy to 62.3%. These results provide additional evidence for the effectiveness of π\piPPO in non-mathematical RLVR training settings.

Given the rebuttal budget, we prioritized code-generation comparisons against all baselines and multi-seed validation (W5) on the widely used Qwen3 backbones.

[4] Cui G, Yuan L, Wang Z, et al. Process reinforcement through implicit rewards[J]. Transactions on Machine Learning Research (TMLR), 2025.

W4: The experiments lack an ablation on the rollout group size (G), which is a key component of the proposed method. Since the availability and diversity of successful and unsuccessful reference trajectories depend on the group size, it is important to investigate how (G) affects value estimation, downstream performance, and computational cost.

Q4: How sensitive is π\piPPO to the rollout group size (G)? Could the authors investigate different group sizes and analyze their effects on reference diversity, value estimation quality, downstream performance, and computational efficiency?

RE W4 & Q4:

We thank the reviewer for this suggestion. We compare G=4G=4 with our default G=8G=8 using a 4B actor, a 0.6B critic, and two references per target. We keep 2,048 responses per step (256 prompts × 8 versus 512 prompts × 4), both settings are trained for 100 steps under otherwise identical configurations.

Reference availability and value estimation (averaged over steps 1–100):

Asymmetric π\piPPOG=8G=8G=4G=4
Targets with a correct/incorrect pair (%)63.251.7
EV0.4510.459
Var(A)/Var(R)\mathrm{Var}(A)/\mathrm{Var}(R)0.5500.541

The sibling candidate pool shrinks from seven to three. Aggregate EV and advantage variance (≈1−EV\approx 1-\mathrm{EV}; see our response to W2 & Q2) remain comparable, supporting retained value-estimation quality for the complete method.

Downstream performance (G=4G=4, step 100):

MethodAIME24AIME25BeyondAIMEHMMT FebHMMT NovOverall
GRPO58.152.531.531.738.342.4
DAPO56.350.829.429.636.940.6
π\piPPO (Asym)58.353.832.531.339.243.0

π\piPPO matches or exceeds both other group-based baselines (+2.4 over DAPO). Moving from G=8G=8 to G=4G=4, DAPO's dynamic sampling finds fewer mixed-outcome groups (39.5% → 28.9%), under the same 2×2\times oversampled rejection pool as the main text, so ~42% of its batch becomes zero-advantage filler, whereas π\piPPO's learned baseline does not rely on within-group outcome variance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

We sincerely thank the reviewers for their constructive feedback and thoughtful suggestions. We summarize below the additional experiments, analyses, and manuscript revisions addressing the main concerns raised.

  1. The performance advantage persists at comparable total training budgets. Pairing π\piPPO with a compact critic matches the GPU-hours of PPO and DAPO while reaching 47.8% and 49.9% Overall accuracy on Qwen3-4B and Qwen3-8B, against 46.2% and 49.3% for DAPO. Conversely, extending the 4B baselines by 1.2–1.7$\times$ more updates to match symmetric π\piPPO's budget of 1,681 GPU-hours lifts the strongest of them to 47.4%, still below π\piPPO's 50.3% (Appendix F, Tables 9–10).

  2. The improvement extends beyond mathematical reasoning. We trained Qwen3-4B on the coding subset of Eurus-2-RL-Data for 250 steps and evaluated it on HumanEval+, MBPP+, and LiveCodeBench v6. Symmetric and asymmetric πPPO achieve Overall accuracies of 62.3% and 61.9%, respectively, exceeding the strongest baseline at 61.2%. These results support the applicability of privileged value estimation to code generation under verifiable rewards (Appendix E.2; Table 6).

  3. The gains are consistent across training seeds. We added runs with seeds 42 and 1234 alongside the original seed 0 for PPO, DAPO, πPPO, and asymmetric πPPO on both Qwen3-4B and Qwen3-8B. πPPO achieves the highest Overall accuracy in all six combinations of seed and backbone. The asymmetric variant also achieves higher mean Overall accuracy than PPO and DAPO on both backbones. Appendix E.3 and Table 7 report individual runs and means ± sample standard deviations.

  4. Theoretical justification for privileged value estimation. Proposition 1 separates Bayes risk from critic approximation error. In our Bayesian predictive setting, verified sibling outcomes provide evidence about shared, unknown task criteria; Appendix B.3 characterizes when this reduces Bayes risk. For the approximation-error component, the smaller EV drop when reducing the critic from 8B to 1.7B (0.027 for πPPO versus 0.082 for PPO) is consistent with contrastive references easing value approximation under limited critic capacity (Section 3.1; Figure 3).

We expanded the implementation details and training configuration (Appendix C.1; Table 3) and added discussion of asymmetric actor–critic methods and concurrent work on privileged value estimation (Section 6). Revisions are highlighted in blue.

We hope these additional results and clarifications help address the reviewers' concerns and further support the contributions of πPPO. We would be glad to provide any further details during the discussion period, and we would sincerely appreciate it if these clarifications could be taken into account in your final evaluation. Thank you again for your time and thoughtful feedback.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.