- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PPO+, which improves policy optimization and rollout efficiency by integrating group-relative token advantages, a Self-Valuing Transformer, and Value-Guided Selection. Experiments cover five mathematical reasoning benchmarks, and the method demonstrates superior average performance compared to DAPO across three model settings.
Critical strengths
- Group-relative advantages improve PPO across three model settings, supporting the combination of token-level values and within-prompt comparisons.
- SVT reuses the policy backbone via a detached value branch, maintaining performance while avoiding the need for a full critic.
- VGS employs six sets of paired trials across two frozen checkpoints, providing evidence of reduced sampling time.
- Checkpoint selection utilizes an independent held-out set, and primary results report both avg@k and pass@k.
Critical weaknesses
- The overall efficiency comparison excludes the value warmup phase, which is insufficient to substantiate a lower overall training cost compared to DAPO.
- The SVT ablation simultaneously alters the critic architecture, trainable parameters, and learning rate; thus, it is impossible to isolate whether the performance gains stem from representation sharing.
- The experiments lack relevant value-based baselines such as VAPO and SAO, leaving the advantages of PPO+ over existing PPO improvements unclear.
- The history-only selection analysis provides only partial diagnostic results and lacks a comprehensive training comparison, making it impossible to verify the ultimate benefit of using prefix values versus a simpler selector.
Questions for authors
NaN
5 comments
W3: The experiments lack relevant value-based baselines such as VAPO and SAO, leaving the advantages of PPO+ over existing PPO improvements unclear.
We add VAPO and SAO under the same value warm-up protocol as PPO+. The results (five-benchmark Avg.@k) are summarized below:
| Method | 4B Non-Thinking | 4B Thinking | 30B-A3B Non-Thinking |
|---|---|---|---|
| PPO | 35.6 | 48 | 41 |
| VAPO | 44.2 (+8.6) | 54.1 (+6.1) | 47.1 (+6.1) |
| SAO | 46.6 (+11.0) | 55.2 (+7.2) | 50.9 (+9.9) |
| PPO+ (Ours) | 50.1 (+14.5) | 56.9 (+8.9) | 54.5 (+13.5) |
PPO+ consistently outperforms both VAPO and SAO across all three settings.
The details are provided in the revised PDF (Lines 270–309).
W4: The history-only selection analysis provides only partial diagnostic results and lacks a comprehensive training comparison, making it impossible to verify the ultimate benefit of using prefix values versus a simpler selector.
We conduct additional full-training comparisons between History-only selection and the full History + Prefix (VGS) selector on Qwen3-4B and Qwen3-30B-A3B in non-thinking mode, keeping other training settings unchanged. We report reasoning performance, average prompt consumption per round, and total training GPU-hours, including shared trajectory generation, value warm-up, and online training:
| Setting | Selector | Avg.@k / Pass@k (%) | Prompts/round | Total GPU-hours |
|---|---|---|---|---|
| 4B-NT | History only | 49.6 / 69.1 | 306.4 | 5093.2 |
| 4B-NT | History + Prefix (VGS) | 50.1 / 67.9 | 274.3 | 4012.8 |
| 30B-A3B-NT | History only | 53.4 / 74.0 | 303.2 | 11093.5 |
| 30B-A3B-NT | History + Prefix (VGS) | 54.5 / 74.1 | 276.8 | 9121 |
Prefix values consistently improve Avg.@k and efficiency across both settings.
Follow-Up on Review and Feedback
Dear Reviewer 5fD2,
We hope this message finds you well.
We have carefully addressed all your questions and concerns, including conducting additional experiments as requested, and have provided detailed responses in the rebuttal.
As the rebuttal deadline is approaching, we would deeply appreciate it if you could share your updated thoughts based on the rebuttal and paper revision, or do not hesitate to let us know if you have additional questions, and we will respond promptly.
Thank you again for your thoughtful review and your invaluable contributions to the quality of this paper.
Kind regards,
Paper 44224 Authors
Thanks for recognizing our contributions to group-relative token-level credit assignment, efficient value reuse through SVT, and sampling-time reduction through VGS.
We apologize for the delayed response, as the additional RL experiments required substantial computation time. Detailed responses to the concerns are provided below.
Thanks for raising this point. We now report end-to-end training costs for both non-thinking settings, including shared trajectory generation, value warm-up, and online training. The 32,768 trajectories used for warm-up are pre-generated for prompt selection and shared across methods. All compute costs are reported in H20 GPU-hours, with peak memory in GB per GPU.
Warm-up accounts for only 0.9% and 2.7% of PPO+'s total GPU-hours, respectively. After accounting for all three stages, PPO+ reduces total GPU-hours by 3.0% and 4.4% relative to DAPO, respectively, while also substantially reducing compute compared to PPO.
Please refer to the revised Analysis section (Lines 501–513) for details.
We conduct an additional controlled ablation on both Qwen3-4B and Qwen3-30B-A3B non-thinking settings. Starting from the full SVT configuration, we remove policy-representation sharing while keeping the remaining architecture and training settings unchanged. The results are reported below (Avg.@k / Pass@k):
SVT with representation sharing consistently outperforms its non-sharing counterpart across both model settings, demonstrating the benefit of reusing policy representations for value estimation.