- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SAPO, a method that adds special tokens to vocabulary to let LLM directly predict the state value and Q value. At the next token of input context, SAPO use the difference of logits of two special tokens to predict V. At the next token of LLM reponse, SAPO use the difference to predict Q. Then, SAPO use V to calculate the advantage for policy optimization jointly with the regression optimization of V and Q prediction. Experiments show significant improvement compared with GRPO and PPO.
Critical strengths
- SAPO achieves more efficient training and comparable performance at the same time compare with PPO.
Critical weaknesses
- Underlying Mechanism is not Clear: According to previous experience, optimize multiple different targets in one LLM backbone might cause "alignment tax", because these targets can have totally different nature and contradict with each other. There is no theoritical or emperical explanation on why SAPO adding and training the special tokens won't affect the original capability of LLM.
Questions for authors
- Many previous works find PPO's critic cannot provide a good state value estimation without proper value pretraining, but experiments in this paper show PPO significantly outperform GRPO, which is different from experiment result in "Group-in-Group Policy Optimization for LLM Agent Training". What’s the training recipe of PPO? Is there critic warmup or value pretrain? And what is the training recipes of SAPO? According to Figure 3, it seems that there is no warmup or value pretrain, then why the initial inaccurate value estimation does not degrade the policy performance?
- The experiments show SAPO's performance is comparable with PPO. However, PPO’s critic has far more parameter, which should lead to better performance. Is there any more theoretical or emperical analysis (like the ablation study) on why two special tokens can achieve the performance of full critic network?
4 comments
> Re W1: Does joint policy/value learning cause optimization conflict or damage the original LLM capability?
We first emphasize that SAPO uses separate logits and gradients for next-token prediction objective and value learning objective, and no logit is optimized by both losses. We also provide a more detailed and carefully gradient conflict analysis among these objectives below.
We separately backpropagated the policy, V, and Q objectives on the same diagnostic micro-batch, aggregated FSDP-sharded gradient statistics over the full model, and report mean ± standard deviation. In Table 1, the policy–V and policy–Q cosines are near zero, indicating near-orthogonality rather than conflict, while V-Q cosines show mild positive alignment. Moreover, the joint gradient remained a policy-descent direction at all measured checkpoints. We therefore find no evidence of harmful parameter competition in the measured regime.
Table 1. Statistics on gradient metrics among Policy, V, and Q Objectives in SAPO during training time using three consecutive checkpoints.
| Metric | Definition | Result |
|---|---|---|
| Policy–V gradient cosine | ||
| Policy–Q gradient cosine | ||
| V–Q gradient cosine | ||
| Total policy alignment |
Background: Assuming non-zero gradients and positive loss coefficients, a positive cosine indicates first-order alignment, where the auxiliary gradient reinforces the policy direction; a indicates some gradient conflicts; and a indicates orthogonality, where it neither counteracts nor reinforces the policy gradient, with the objectives optimizing along different directions in the shared parameter space.
Thank you for the clarification. The additional ablation study and the statistics on gradient well address my concerns. I will update my score to weak accept.
Response to Reviewer Xbi7
We thank you for your constructive feedback and positive assessment.
Dear Reviewer Xbi7:
Thank you for your review comments. We fully understand your concerns, and however, some of the core issues raised appear to stem from a misunderstanding of our paper, likely due to a lack of clear clarity in our original presentation. We hope that our detailed responses and experiments below will fully address your concerns and encourage you to reconsider your evaluation of our work. Below, we provide a point-by-point response.
> Re Q1: Why is PPO much stronger than the PPO reported in GiGPO? Are value pretraining or warm-up used?The PPO results are not directly comparable because the two papers use different advantage-estimation settings. GiGPO reports PPO with token-level GAE, whereas our PPO baseline uses turn-level GAE, matching SAPO for a fair comparison. This distinction is stated in Page 7, Sec. 5.1, i.e., “Notably, PPO employs turn-level GAE consistented with SAPO, rather than the token-level GAE reported in GiGPO Feng et al. (2026)”.
Training and evaluation curves for SAPO, PPO, and GRPO using Qwen2.5-1.5B/7B on ALFWorld and WebShop can be viewed via the
Anonymous Link. And, we will fully open-source all of our experimental code, just as we would be happy to submit the complete code here.All methods with value-learning in our paper use neither value pretraining nor warm-up. Initial value estimates need not be accurate enough to select actions directly: they serve as baselines and bootstrap terms for GAE. Also, Figure 3 confirms that V and Q progressively become correlated with Monte Carlo returns. To our knowledge, PPO results in GiGPO paper on verl-agent do not enable a warm-up during the actual training process. Our own reproduction of these PPO experiments also showed no significant impact from this settings.
> Re Q2: Full ablation analysis for why SAPO with logit-based value achieves high performance.We first clarify that the dual-logit readout is only one contributor to SAPO’s performance, while constituting the architectural basis for its unified policy–value formulation. Below, we provide more fine-grained ablation comparisons of the logit-based value design.
Table 1 shows that (a) replacing the logit-based readout with a conventional value-head network (w/ Value Head) consistently degrades performance across experimental settings, supporting the effectiveness of the joint PPO+SARSA optimization. Disabling value scaling also substantially reduces SAPO’s performance across all environments and model sizes, while using a single-logit readout leads to a moderate degradation. These results indicate that (b) constraining the value-target scale and using the dual-logit mechanism improve the effectiveness and stability of value learning. Moreover, the ablation studies in the paper demonstrate that (c) auxiliary Q-value learning also contributes positively to SAPO’s performance.
Table 1. Additional ablation studies.