acceptodds
Desk rejected at ICLR 2027

PPO+: Effective and Efficient Policy Optimization with Self-Valuing Transformers

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning is central to improving language model reasoning, yet conventional PPO often trails GRPO-style methods and incurs extra cost from a separate critic. At the same time, PPO provides explicit token-level values that can guide learning from sparse rewards, leaving its potential underexplored. We introduce **PPO+**, which improves how value signals guide policy learning, integrates value estimation into the policy itself, and reuses these signals for prompt selection. PPO+ uses both token-level value estimates and within-prompt response comparisons to construct advantages for a single PPO objective. Motivated by our finding that policy representations already contain useful value information, we introduce a *Self-Valuing Transformer (SVT)* that reuses the policy backbone with a small detached value branch. We further introduce *Value-Guided Selection (VGS)*, which combines historical outcomes with early value estimates to prioritize prompts likely to yield mixed groups. Across five mathematical reasoning benchmarks, PPO+ improves over DAPO by 8.09 points on Qwen3-4B while using slightly less overall training time than the GRPO-style baseline. These results suggest that value estimation can become an internal policy capability, enabling stronger and more efficient reinforcement learning.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5fD2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PPO+, which improves policy optimization and rollout efficiency by integrating group-relative token advantages, a Self-Valuing Transformer, and Value-Guided Selection. Experiments cover five mathematical reasoning benchmarks, and the method demonstrates superior average performance compared to DAPO across three model settings.

Critical strengths
  1. Group-relative advantages improve PPO across three model settings, supporting the combination of token-level values and within-prompt comparisons.
  2. SVT reuses the policy backbone via a detached value branch, maintaining performance while avoiding the need for a full critic.
  3. VGS employs six sets of paired trials across two frozen checkpoints, providing evidence of reduced sampling time.
  4. Checkpoint selection utilizes an independent held-out set, and primary results report both avg@k and pass@k.
Critical weaknesses
  1. The overall efficiency comparison excludes the value warmup phase, which is insufficient to substantiate a lower overall training cost compared to DAPO.
  2. The SVT ablation simultaneously alters the critic architecture, trainable parameters, and learning rate; thus, it is impossible to isolate whether the performance gains stem from representation sharing.
  3. The experiments lack relevant value-based baselines such as VAPO and SAO, leaving the advantages of PPO+ over existing PPO improvements unclear.
  4. The history-only selection analysis provides only partial diagnostic results and lacks a comprehensive training comparison, making it impossible to verify the ultimate benefit of using prefix values versus a simpler selector.
Questions for authors

NaN

5 comments

Thanks for recognizing our contributions to group-relative token-level credit assignment, efficient value reuse through SVT, and sampling-time reduction through VGS.

We apologize for the delayed response, as the additional RL experiments required substantial computation time. Detailed responses to the concerns are provided below.

W1: The overall efficiency comparison excludes the value warmup phase, which is insufficient to substantiate a lower overall training cost compared to DAPO.

Thanks for raising this point. We now report end-to-end training costs for both non-thinking settings, including shared trajectory generation, value warm-up, and online training. The 32,768 trajectories used for warm-up are pre-generated for prompt selection and shared across methods. All compute costs are reported in H20 GPU-hours, with peak memory in GB per GPU.

MethodShared Gen.Warm-upOnlineTotalPeak Memory
4B Non-Thinking
DAPO73.5—4064.34137.884.5
PPO73.5184.76368.56626.788.4
PPO+73.534.73904.64012.889.2
30B-A3B Non-Thinking
DAPO137.1—9408.59545.689.9
PPO137.1633.111168.411938.691.2
PPO+137.1247.28736.7912192.3

Warm-up accounts for only 0.9% and 2.7% of PPO+'s total GPU-hours, respectively. After accounting for all three stages, PPO+ reduces total GPU-hours by 3.0% and 4.4% relative to DAPO, respectively, while also substantially reducing compute compared to PPO.

Please refer to the revised Analysis section (Lines 501–513) for details.

W2: The SVT ablation simultaneously alters the critic architecture, trainable parameters, and learning rate; thus, it is impossible to isolate whether the performance gains stem from representation sharing.

We conduct an additional controlled ablation on both Qwen3-4B and Qwen3-30B-A3B non-thinking settings. Starting from the full SVT configuration, we remove policy-representation sharing while keeping the remaining architecture and training settings unchanged. The results are reported below (Avg.@k / Pass@k):

SettingMethodAIME 2026HMMT Nov 2025HMMT Feb 2026AIME 2025BeyondAIMEAvg.
4BSVT (w/o Rep. Sharing)61.3 / 83.346.3 / 76.736.4 / 48.559.2 / 83.341.0 / 57.048.8 / 69.8
4BSVT (Ours)64.2 / 80.046.3 / 70.039.0 / 48.565.4 / 86.736.0 / 56.050.2 / 68.2
30B-A3BSVT (w/o Rep. Sharing)66.7 / 83.350.8 / 73.339.0 / 57.665.8 / 83.342.3 / 55.052.9 / 70.5
30B-A3BSVT (Ours)67.5 / 86.752.9 / 76.739.8 / 60.669.2 / 90.044.0 / 57.054.7 / 74.2

SVT with representation sharing consistently outperforms its non-sharing counterpart across both model settings, demonstrating the benefit of reusing policy representations for value estimation.

W3: The experiments lack relevant value-based baselines such as VAPO and SAO, leaving the advantages of PPO+ over existing PPO improvements unclear.

We add VAPO and SAO under the same value warm-up protocol as PPO+. The results (five-benchmark Avg.@k) are summarized below:

Method4B Non-Thinking4B Thinking30B-A3B Non-Thinking
PPO35.64841
VAPO44.2 (+8.6)54.1 (+6.1)47.1 (+6.1)
SAO46.6 (+11.0)55.2 (+7.2)50.9 (+9.9)
PPO+ (Ours)50.1 (+14.5)56.9 (+8.9)54.5 (+13.5)

PPO+ consistently outperforms both VAPO and SAO across all three settings.

The details are provided in the revised PDF (Lines 270–309).

W4: The history-only selection analysis provides only partial diagnostic results and lacks a comprehensive training comparison, making it impossible to verify the ultimate benefit of using prefix values versus a simpler selector.

We conduct additional full-training comparisons between History-only selection and the full History + Prefix (VGS) selector on Qwen3-4B and Qwen3-30B-A3B in non-thinking mode, keeping other training settings unchanged. We report reasoning performance, average prompt consumption per round, and total training GPU-hours, including shared trajectory generation, value warm-up, and online training:

SettingSelectorAvg.@k / Pass@k (%)Prompts/roundTotal GPU-hours
4B-NTHistory only49.6 / 69.1306.45093.2
4B-NTHistory + Prefix (VGS)50.1 / 67.9274.34012.8
30B-A3B-NTHistory only53.4 / 74.0303.211093.5
30B-A3B-NTHistory + Prefix (VGS)54.5 / 74.1276.89121

Prefix values consistently improve Avg.@k and efficiency across both settings.

AuthorsOpenReview ↗

Follow-Up on Review and Feedback

Dear Reviewer 5fD2,

We hope this message finds you well.

We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.

As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.

Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.

Kind regards,

Paper 44224 Authors

Reviewer_ztfPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PPO+, a value-based reinforcement learning framework for mathematical reasoning in large language models that aims to close both the performance gap and the systems overhead of conventional PPO relative to critic-free group methods such as DAPO. Motivated by a layerwise representation probe (Section 2.1, Figure 2) and a rollout cost breakdown (Section 2.2, Figure 3), PPO+ integrates three components (Section 3, Figure 1):

  1. Group-Relative Advantage Estimation (Section 3.1, Eqs. 1–2): Computes token-level GAE Ag,i,t=∑l=0Lg,i−t(γλi)lδg,i,t+lA _ {g,i,t} = \sum _ {l=0}^{L _ {g,i}-t}(\gamma\lambda _ i)^l \delta _ {g,i,t+l} along each trajectory and normalizes these token advantages across the K=4K=4 responses of prompt xgx _ g with equal response weighting (A^g,i,t=(Ag,i,t−μg)/max⁡(σg2,ϵ)\widehat{A} _ {g,i,t} = (A _ {g,i,t} - \mu _ g)/\sqrt{\max(\sigma _ g^2, \epsilon)}).
  2. Self-Valuing Transformer (SVT, Section 3.2, Eq. 3): Replaces a separate full-backbone critic with a lightweight two-block Transformer value branch attached to stop-gradient policy hidden states Vϕ(s)=vϕ(sg[hθ(T)(s)])V _ \phi(s) = v _ \phi(\mathrm{sg}[h _ \theta^{(T)}(s)]) at layer T=36T=36 (Qwen3-4B) or T=48T=48 (Qwen3-30B-A3B).
  3. Value-Guided Selection (VGS, Section 3.3, Eq. 4 & Appendix B.4 Eq. 8): Uses a 9-feature logistic model combining historical pass rate, prompt value v0v _ 0, and four 20-token prefix values v1,…,v4v _ 1, \dots, v _ 4 to prioritize candidate prompts likely to yield informative mixed-outcome groups (0<∑i=1KRi<K0 < \sum _ {i=1}^K R _ i < K).

Evaluated on Qwen3-4B (non-thinking and thinking modes) and Qwen3-30B-A3B (non-thinking) trained on DAPO-Math-17k across five competition-math benchmarks (AIME 2026 @8, HMMT Nov 2025 @8, HMMT Feb 2026 @8, AIME 2025 @8, and BeyondAIME @4, Table 1), PPO+ improves the five-benchmark avg@k average over DAPO from 42.0% to 50.1% (4B non-thinking), 54.1% to 56.9% (4B thinking), and 49.6% to 54.5% (30B-A3B). While the group-normalized token-level GAE result is compelling and the paired VGS sampling benchmark (Section 5.3, Figure 5b) is carefully designed, I lean toward weak rejection because the main experiments rely on single training runs without statistical uncertainty, omit closely related value-based reasoning RL baselines (VAPO, SAO), exclude a substantial 32,768-trajectory / 1,024-update value warmup phase from the headline training-time comparison, and rely on calibrated estimates for 9 of the 12 points in the SVT optimization plot (Figure 5a).

Critical strengths

1. Group-relative token-level GAE (Eqs. 1–2) delivers large, consistent gains over conventional PPO across model scales and thinking modes.
Normalizing token-level GAE advantages across the K=4K=4 responses of each prompt with equal response weight (Eq. 2) lifts five-benchmark avg@k from 35.6% to 49.1% (+13.5 pp) on Qwen3-4B non-thinking, from 48.0% to 55.2% (+7.2 pp) on Qwen3-4B thinking, and from 41.0% to 48.7% (+7.7 pp) on Qwen3-30B-A3B (Table 1). This cleanly demonstrates that conventional PPO's poor performance on reasoning benchmarks stems largely from lacking prompt-relative baseline normalization rather than an inherent flaw in token-level value estimation.

2. Cohesive value-reuse architecture (SVT + VGS) supported by a controlled, order-balanced sampling benchmark.
Attaching a two-block value branch to stop-gradient policy representations (SVT) avoids maintaining a second full LM backbone and further improves 30B-A3B accuracy from 48.7% to 54.7% (+6.0 pp in Table 1). Reusing short-prefix value predictions (VGS, 4 prefixes ×\times 20 tokens, costing only 1.5% of sampling time) cuts completed prompt groups per round by 10.4%–24.8% in Table 1 and reduces paired sampling wall-clock time by 3.3% (u320: 30.98 →\to 29.97 min) and 6.3% (u480: 34.17 →\to 32.01 min) under a rigorous 6-permutation crossover protocol on 32 H200 GPUs (Section 5.3, Figure 5b, Appendix C.3).

Critical weaknesses

1. Incomplete efficiency accounting that excludes the 1,024-update value warmup phase and relies on heterogeneous historical runs.
The abstract and Section 4 highlight that PPO+ reduces "overall training time" relative to DAPO (e.g., 12.2 vs. 12.7 min/round on 4B non-thinking, 17.2 vs. 19.1 min/round on 4B thinking, and 27.3 vs. 29.4 min/round on 30B-A3B in Table 1). However, Time (min) in Table 1 measures only the mean logged online round time and explicitly excludes the initial value-model warmup stage—which requires generating 32,768 full trajectories (8,192 prompts ×4\times 4) from the starting policy and running 1,024 value-branch updates at batch size 64 before online RL even begins (Section 4.1 p. 6, Appendix B.3 p. 15). Because critic-free DAPO requires zero value warmup (whereas PPO+ only saves 0.5 to 2.1 minutes per online round over 100–125 rounds, i.e., roughly 60–250 minutes total), including the 32,768-trajectory generation and 1,024-step warmup cost could easily reverse the end-to-end wall-clock comparison against DAPO. Furthermore, on Qwen3-4B thinking, adding SVT actually increases online round time relative to + Group-Rel. Adv. (19.7 vs. 18.8 min), and Appendix B.2 (p. 14) acknowledges that serving precision, parallelism, and clipping settings varied across some historical runs.

2. Single-seed training without statistical uncertainty, missing value-based reasoning RL baselines (VAPO, SAO), and unclarified Routing Replay on 30B.
All training trajectories in Table 1, Figure 4, and Figure 5(a) come from single training runs (Appendix C.1, p. 17), with no training-seed standard deviations or benchmark bootstrap confidence intervals (and even in Table 2b, the paired stratified bootstrap CIs mentioned in Appendix C.4 are omitted from the table). While the +7.2 to +13.5 pp jump from Group-Rel. Adv. is large, the incremental accuracy changes from + SVT on 4B (+1.1 and +1.0 pp) and + VGS (-0.1, +0.7, and -0.2 pp) are small enough to fall within single-run RL variance on 30-problem AIME/HMMT sets. Moreover, although Section 6 discusses directly competing value-based reasoning RL methods such as VAPO and SAO, the experiments compare only against conventional PPO and DAPO; and in the Qwen3-30B-A3B block of Table 1, only the DAPO + RR row is labeled with Routing Replay (+ RR), leaving unclear whether PPO and PPO+ also used Routing Replay or differed in MoE routing stabilization.

Questions for authors
  1. Assessment-critical (End-to-end training time and GPU-hours including value warmup):
    How many wall-clock hours and GPU-hours are spent generating the 32,768 warmup trajectories (8,192 prompts ×4\times 4) and running the 1,024 value-warmup updates for conventional PPO and SVT/PPO+ across the three setups in Table 1? When this warmup cost is added to the 100–125 online training rounds under identical serving precision and parallelism, what is the true end-to-end training time and total GPU-hour cost of PPO+ vs. DAPO?

  2. Assessment-critical (30B Routing Replay parity, and VAPO/SAO baselines):

    • Can you provide prompt-level paired bootstrap confidence intervals for the Qwen3-4B non-thinking progression in Table 1 (as well as the CIs mentioned in Appendix C.4 for Table 2b), and compare against VAPO or SAO?
    • In the Qwen3-30B-A3B block of Table 1, was MoE Routing Replay (RR) enabled only for DAPO + RR, or also for PPO, + Group-Rel. Adv., + SVT, and + VGS (PPO+)?
  3. Clarification (Omitted formulas, Figure 5a direct evaluations, and Figure 4 round count):

    • Please provide the exact mathematical formulas for (a) the length-adaptive GAE λi\lambda _ i (with scale α=1.5\alpha = 1.5 in Table 3), (b) the within-group / between-group weight = 1 / 0 setting, and (c) the online Gaussian posterior update for VGS (Appendix B.4).
    • Can you report the direct five-benchmark evaluation scores for the 9 calibrated points in Figure 5(a), and clarify whether the five evaluation benchmarks were used to select the 2L / 8V configuration? Also, why does Figure 4 plot up to 150 training rounds when Section 4.1 specifies 125 rounds for non-thinking models?
6 comments

Thanks for recognizing the consistent performance gains of our approach, the effectiveness of SVT in improving reasoning performance, and the role of VGS in reducing sampling costs.

We apologize for the delayed response, as the additional RL experiments required substantial computation time. Detailed responses to the concerns are provided below.

Q1: Assessment-critical (End-to-end training time and GPU-hours including value warmup)

Thanks for raising this point. We now report end-to-end training costs for both non-thinking settings, including shared trajectory generation, value warm-up, and online training. The 32,768 trajectories used for warm-up are pre-generated for prompt selection and shared across methods. All compute costs are reported in H20 GPU-hours, with peak memory in GB per GPU.

MethodShared Gen.Warm-upOnlineTotalPeak Memory
4B Non-Thinking
DAPO73.5—4064.34137.884.5
PPO73.5184.76368.56626.788.4
PPO+73.534.73904.64012.889.2
30B-A3B Non-Thinking
DAPO137.1—9408.59545.689.9
PPO137.1633.111168.411938.691.2
PPO+137.1247.28736.7912192.3

Warm-up accounts for only 0.9% and 2.7% of PPO+'s total GPU-hours, respectively. After accounting for all three stages, PPO+ reduces total GPU-hours by 3.0% and 4.4% relative to DAPO, respectively, while also substantially reducing compute compared to PPO.

Please refer to the revised Analysis section (Lines 501–513) for further details.

Q2: Single-seed training without statistical uncertainty, missing value-based reasoning RL baselines (VAPO, SAO), and unclarified Routing Replay on 30B.

Thanks for raising these concerns. We address them in three ways.

  1. Seed robustness. We repeat Qwen3-4B non-thinking PPO+ with three random seeds (20261001–20261003). The Avg.@k results show little variation across seeds (50.1, 49.9, 50.1), indicating that our main findings are not sensitive to the random seed.

  2. Additional value-based baselines. We add VAPO and SAO under the same value warm-up protocol as PPO+. The results (five-benchmark Avg.@k) are summarized below:

    Method4B Non-Thinking4B Thinking30B-A3B Non-Thinking
    PPO35.64841
    VAPO44.2 (+8.6)54.1 (+6.1)47.1 (+6.1)
    SAO46.6 (+11.0)55.2 (+7.2)50.9 (+9.9)
    PPO+ (Ours)50.1 (+14.5)56.9 (+8.9)54.5 (+13.5)

    PPO+ consistently outperforms both VAPO and SAO across all three settings.

  3. Routing Replay for MoE. All methods evaluated on Qwen3-30B-A3B use the same Routing Replay (RR) configuration, ensuring a consistent comparison across methods.

We have added the VAPO and SAO comparisons (Lines 270–309) and the Routing Replay clarification (Line 307) to the revised manuscript.

Q3: Clarification (Omitted formulas, Figure 5a direct evaluations, and Figure 4 round count):

Thanks for raising these clarification questions. We address the three points below.

  1. Missing mathematical formulas. We have added all three requested mathematical definitions, including length-adaptive GAE and within-/between-group weighting in Section 3.1 (Lines 168–195), and the online Gaussian posterior update for VGS in Appendix B.4 (Lines 885–903). Please refer to the revised PDF for details.

  2. Direct evaluations and configuration selection. We have directly evaluated all nine previously calibrated checkpoints on the five benchmarks and replaced the estimates in Figure 5(a) with measured results. The five-benchmark averages for all twelve checkpoints are reported below.

    Configuration360 Updates480 Updates600 Updates
    1L / 8V44.2344.9345.64
    1L / 16V43.4944.4144.98
    2L / 8V45.2347.9850.26
    2L / 16V44.3347.8348.17

    The 2L/8V configuration was selected based on Avg.@4 on a 128-question held-out subset drawn from the five evaluation benchmarks, with matched actor-update budgets. We have explicitly documented this selection procedure in the revised manuscript (Lines 989–1003).

  3. Figure 4 training rounds. The 125-round statement in Section 4.1 was a typo. All non-thinking runs used 150 training rounds, consistent with Figure 4. We have corrected the text accordingly (Line 262).

Reviewer_kwFUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GRPO is more popular than PPO in post-training today since PPO needs an extra critic model and is harder to train and use. The authors think that value estimation in PPO is very useful and should still be considered. So they make three improvements. First, they add GRPO's group comparison to token-level advantages, so that different rollouts for the same question can be compared. Second, the authors find that the policy model already contains value information in its hidden states. So they add a small value branch to the policy model backbone instead of using an independent critic model. Third, they use value predictions from short prefixes and historical outcomes to select questions that are more likely to produce both correct and incorrect rollouts, reducing unnecessary generation. Experiments show that PPO+ outperforms DAPO on several math reasoning benchmarks and also slightly reducing training time.

Critical strengths
  1. The motivation is clear. The authors use the experiments in Section 2 to show that an extra critic model is unnecessary, so it is possible to reduce the computation cost of PPO by reusing the policy model as the critic model.

  2. Most post-training work today focuses on GRPO, but this paper instead studies how to improve PPO by combining token-level advantages with group-level comparisons. The ablation study shows that group-relative advantages improve performance, while SVT and VGS reduce the training cost.

Critical weaknesses
  1. The authors' writing is ambiguous and makes people think these modules are new. However, some already exist. For example, the authors propose SVT, which can reuse the policy backbone and avoid using an independent critic model. [1] is a classical actor-critic work that already uses a shared network for policy and value prediction. The Hugging Face TRL library also provides AutoModelForCausalLMWithValueHead directly on an LM. These indicate that shared networks are common in both research and engineering. SVT might be a bit different from the traditional shared backbone, but the authors should discuss this and add a baseline for comparison in the results section. Otherwise, I cannot determine whether the improvement comes from the shared backbone or SVT.

  2. Section 3.1 uses GAE to compute token-level advantages, then performs normalization across multiple rollouts from the same prompt. This advantage normalization is also commonly used. [2] also studies how to improve value-based PPO. The authors need to clearly state the novelty instead of simply saying they combine PPO and GRPO, and add more baselines.

  3. In Section 2.1, the authors want to show that policy hidden states contain value information. However, in Figure 2, the best recovery layer is Actor Layer 20. This observation is different from the design of SVT, which uses the policy's last layer.

  4. Based on Section 4.1, the critic and SVT need 1,024 value updates using 32,768 trajectories for warm-up. However, the costs reported in Table 1 do not include this warm-up cost. VGS also needs extra prefix generation and value prediction, which introduce additional costs. In Section 5.3, VGS reduces wall-clock sampling time by only 4.9%. As a paper focusing on efficiency, the authors need to report comprehensive end-to-end training costs, including value warm-up, online training, GPU hours, and peak GPU memory.

References

[1] Asynchronous Methods for Deep Reinforcement Learning (Mnih et al., ICML 2016)

[2] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks

Questions for authors

see weakness.

11 comments

Thanks for recognizing our motivation for reusing policy representations to avoid a separate critic, as well as the contributions of SVT and VGS to training efficiency.

We apologize for the delayed response, as the additional RL experiments required substantial computation time. Detailed responses to the concerns are provided below.

W1: Shared networks are common in both research and engineering. SVT might be a bit different from the traditional shared backbone, but the authors should discuss this and add a baseline for comparison in the results section.

Thanks for highlighting prior shared actor-critic designs, including A3C and Hugging Face TRL. While backbone sharing itself is not new, a gap remains between this established concept and its implementation in stable, high-performing LLM RL algorithms. We address this gap through two SVT designs:

  1. Gradient isolation: Detached policy representations enable independent value-branch optimization while retaining reusable actor features.
  2. Value-specific transformation: A lightweight causal Transformer branch further processes these representations for token-level value prediction, beyond a direct linear readout.

To isolate these two factors, we conduct controlled comparisons (five-benchmark Avg.@k):

ArchitectureGradientValue Branch4B Non-Thinking4B Thinking30B-A3B Non-Thinking
Shared Linear (TRL)SharedLinear41.945.145
Shared SVTShared2-block Transformer44.2 (+2.3)47.1 (+2.0)46.9 (+1.9)
Detached LinearDetachedLinear47.1 (+5.2)52.9 (+7.8)48.8 (+3.8)
Detached SVT (Ours)Detached2-block Transformer50.2 (+8.3)56.2 (+11.1)54.7 (+9.7)

Detached SVT consistently outperforms conventional Linear and shared Critic across all settings, confirming that SVT's improvements extend beyond backbone sharing alone.

Please refer to the revised PDF for all the details. We clarify our contributions in the Introduction (Lines 87–90), elaborate on SVT's design in the Method section (Lines 197–210), add a TRL-style Shared Value Head baseline to the main results (Lines 270–369, 364–370), and provide controlled comparisons in the analysis (Lines 416–424).

W2: Advantage normalization is also commonly used. [2] also studies how to improve value-based PPO. The authors need to clearly state the novelty instead of simply saying they combine PPO and GRPO, and add more baselines.

Thanks for raising this concern. We do not claim advantage normalization itself as novel.

  1. PPO+ adopts a distinct and more effective normalization strategy than existing baselines. PPO+ computes token-level GAE first, then applies response-equal, prompt-relative normalization, followed by prompt-balanced PPO optimization. This differs from conventional PPO, VAPO, and SAO. Results show that PPO+ consistently outperforms both VAPO and SAO across all three settings. The methods and results are summarized below (five-benchmark Avg.@k):

    MethodAdvantage NormalizationSampling4B Non-Thinking4B Thinking30B-A3B Non-Thinking
    PPOBatch-wideSingle35.64841
    VAPOBatch-wideGroup44.2 (+8.6)54.1 (+6.1)47.1 (+6.1)
    SAOBatch-wideSingle46.6 (+11.0)55.2 (+7.2)50.9 (+9.9)
    PPO + Group-Rel. Adv.Prompt-wiseGroup49.1 (+13.5)55.2 (+7.2)48.7 (+7.7)
    PPO+ (Ours)Prompt-wiseGroup50.1 (+14.5)56.9 (+8.9)54.5 (+13.5)
  2. The individual contribution of each design is validated through controlled ablations. The ablations show consistent improvements across all three settings. The results are summarized below:

    MethodNormalization ScopeResponse WeightingPPO Loss Aggregation4B Non-Thinking4B Thinking30B-A3B Non-Thinking
    PPO BaselineBatch-wideToken-weightedOriginal PPO35.64841
    Prompt-relativePrompt-wiseToken-weightedOriginal PPO44.2 (+8.6)51.7 (+3.7)46.5 (+5.5)
    Response-equalPrompt-wiseResponse-equalOriginal PPO47.3 (+11.7)53.4 (+5.4)47.2 (+6.2)
    PPO + Group-Rel. Adv.Prompt-wiseResponse-equalPrompt-balanced49.1 (+13.5)55.2 (+7.2)48.7 (+7.7)

We also want to emphasize that advantage normalization is only one component of PPO+, while SVT and VGS constitute its architectural and algorithmic contributions.

Please refer to the revised PDF for all the details. We clarify our contributions in the Introduction (Lines 87–90), elaborate on the advantage estimation design in the Method section (Lines 170–195), add VAPO and SAO to the main results (Lines 270–309), and provide controlled ablations in the analysis (Lines 407–415).

W3: In Section 2.1, the authors want to show that policy hidden states contain value information. However, in Figure 2, the best recovery layer is Actor Layer 20. This observation is different from the design of SVT, which uses the policy's last layer.

Thanks for raising this concern. We choose the final policy layer for SVT for two reasons:

  1. Our layer ablation in Section 5.2 validates this choice. We favor Layer 36 for its stronger within-prompt ranking, which better serves SVT's credit-assignment role. Related details are already included in the original version. We extract them below for your convenience:

    LayerMSE ↓AUC ↑Pair Acc. ↑
    160.2440.60257.6
    200.2450.59057.6
    240.2470.57453.2
    320.2510.56454.4
    360.2420.57767.1
  2. Final-layer attachment provides a simple and transferable rule across backbones. To examine whether an earlier attachment at Layer 20 offers any advantage in training performance or efficiency, we conduct a direct comparison on Qwen3-4B non-thinking. Layer 36 achieves better reasoning performance with virtually identical training cost. Results are shown below:

    Attachment LayerAIME 2026HMMT 2025HMMT 2026AIME 2025BeyondAIMEAvg.@k / Pass@kTraining Time (min)
    Layer 2062.5 / 83.346.3 / 63.333.7 / 39.462.9 / 80.041.8 / 55.049.4 / 64.212.2
    Layer 36 (Ours)60.8 / 86.748.3 / 63.339.8 / 57.662.1 / 76.739.3 / 55.050.1 / 67.912.2

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions. We appreciate their recognition of PPO+'s value-based token-level credit assignment (kwFU-S2, ztfP-S1, 5fD2-S1), the effectiveness of SVT in reusing policy representations (kwFU-S1, ztfP-S2, 5fD2-S2), and the benefits of VGS for improving sampling efficiency (ztfP-S2, 5fD2-S3). In response, we have conducted additional experiments and revised the manuscript to strengthen its methodological clarity and empirical evidence.

Key improvements based on reviewer feedback:

  1. Methodological Clarification and Baselines: We clarify the novelty of PPO+'s advantage estimation and SVT design, provide the missing mathematical definitions, and add controlled advantage-estimation ablations. We also introduce a TRL-style shared value head baseline and comparisons with VAPO and SAO. (kwFU-W1/W2, ztfP-Q2/Q3, 5fD2-W3)

  2. SVT Architecture and Representation Sharing: We conduct additional controlled ablations to isolate the contribution of policy-representation sharing across Qwen3-4B and Qwen3-30B-A3B, and compare different value-branch attachment layers. (kwFU-W1/W3, 5fD2-W2)

  3. Comprehensive Efficiency and VGS Evaluation: We report end-to-end training costs covering shared trajectory generation, value warm-up, and online training, including GPU-hours and peak memory. We also add full-training comparisons between history-only and history-plus-prefix selection across two model scales, demonstrating the benefits of prefix values. (kwFU-W4, ztfP-Q1, 5fD2-W1/W4)

  4. Experimental Robustness and Reproducibility: We add multi-seed experiments, clarify Routing Replay across all 30B-A3B methods, replace calibrated estimates in Figure 5(a) with direct five-benchmark evaluations, document the 2L/8V selection procedure, and correct the training-round inconsistency. (ztfP-Q2/Q3)

Overall, these revisions reinforce PPO+'s central idea: leveraging value signals more effectively for policy learning, more efficiently through representation reuse, and more broadly for rollout selection. They further substantiate PPO+'s three contributions: prompt-relative token-level advantage estimation, efficient value estimation through the Self-Valuing Transformer (SVT), and value-guided prompt selection through VGS.

We sincerely thank all reviewers again for their constructive feedback, which has helped us substantially improve the clarity, rigor, and completeness of our work.

Overall Significance and Novelty.

Beyond the individual improvements, the central novelty of PPO+ lies in making value estimation an integral capability of policy optimization, rather than merely an auxiliary mechanism for advantage computation. By leveraging value signals to guide both policy learning and rollout selection, PPO+ provides a unified approach to improving credit assignment and training efficiency. This design is supported by consistent performance gains across three model settings, improvements over the TRL-style shared value head, and comprehensive end-to-end efficiency evaluations. In particular, PPO+ reduces total GPU-hours by 39.4% and 23.6% compared with conventional PPO, while VGS further reduces training costs by 21.2% and 17.8% compared with history-only selection. We believe these results demonstrate that PPO+ offers not merely an incremental combination of existing techniques, but a meaningful methodological contribution to LLM reinforcement learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.