acceptodds
Under review as a conference paper at ICLR 2027

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning (RL) has emerged as an important post-training approach for enhancing the capabilities of Large Language Models (LLMs). However, existing methods face a trade-off between policy performance and resource efficiency. Conventional Proximal Policy Optimization (PPO) implementations incur substantial memory overhead from a separate critic, whereas critic-free group-relative methods require multiple rollouts and face potential learning bottlenecks on long-horizon tasks. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), an efficient PPO-style framework that unifies policy optimization and value learning within a single causal language model. SAPO exploits the autoregressive structure of LLMs to sequentially generate action and value estimation at distinct causal boundaries with shared parameters, and then jointly optimizes the PPO objectives and an auxiliary on-policy SARSA objective with turn-level generalized advantage estimation, where the latter is designed to facilitate value learning. Extensive experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B and Qwen3-14B demonstrate that SAPO reduces peak GPU memory usage by 23.1% and per-iteration runtime by 24.8% over strong PPO baseline, while matching or slightly improving task success rate. Our experiments also show that SAPO outperforms Group Relative Policy Optimization (GRPO) and recent cutting-edge variants in both task success and training stability.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Agentic RL, LLMs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xbi7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAPO, a method that adds special tokens to vocabulary to let LLM directly predict the state value and Q value. At the next token of input context, SAPO use the difference of logits of two special tokens to predict V. At the next token of LLM reponse, SAPO use the difference to predict Q. Then, SAPO use V to calculate the advantage for policy optimization jointly with the regression optimization of V and Q prediction. Experiments show significant improvement compared with GRPO and PPO.

Critical strengths
  1. SAPO achieves more efficient training and comparable performance at the same time compare with PPO.
Critical weaknesses
  1. Underlying Mechanism is not Clear: According to previous experience, optimize multiple different targets in one LLM backbone might cause "alignment tax", because these targets can have totally different nature and contradict with each other. There is no theoritical or emperical explanation on why SAPO adding and training the special tokens won't affect the original capability of LLM.
Questions for authors
  1. Many previous works find PPO's critic cannot provide a good state value estimation without proper value pretraining, but experiments in this paper show PPO significantly outperform GRPO, which is different from experiment result in "Group-in-Group Policy Optimization for LLM Agent Training". What’s the training recipe of PPO? Is there critic warmup or value pretrain? And what is the training recipes of SAPO? According to Figure 3, it seems that there is no warmup or value pretrain, then why the initial inaccurate value estimation does not degrade the policy performance?
  2. The experiments show SAPO's performance is comparable with PPO. However, PPO’s critic has far more parameter, which should lead to better performance. Is there any more theoretical or emperical analysis (like the ablation study) on why two special tokens can achieve the performance of full critic network?
4 comments

Dear Reviewer Xbi7:

Thank you for your review comments. We fully understand your concerns, and however, some of the core issues raised appear to stem from a misunderstanding of our paper, likely due to a lack of clear clarity in our original presentation. We hope that our detailed responses and experiments below will fully address your concerns and encourage you to reconsider your evaluation of our work. Below, we provide a point-by-point response.

> Re Q1: Why is PPO much stronger than the PPO reported in GiGPO? Are value pretraining or warm-up used?

The PPO results are not directly comparable because the two papers use different advantage-estimation settings. GiGPO reports PPO with token-level GAE, whereas our PPO baseline uses turn-level GAE, matching SAPO for a fair comparison. This distinction is stated in Page 7, Sec. 5.1, i.e., “Notably, PPO employs turn-level GAE consistented with SAPO, rather than the token-level GAE reported in GiGPO Feng et al. (2026)”.

Training and evaluation curves for SAPO, PPO, and GRPO using Qwen2.5-1.5B/7B on ALFWorld and WebShop can be viewed via the Anonymous Link. And, we will fully open-source all of our experimental code, just as we would be happy to submit the complete code here.

All methods with value-learning in our paper use neither value pretraining nor warm-up. Initial value estimates need not be accurate enough to select actions directly: they serve as baselines and bootstrap terms for GAE. Also, Figure 3 confirms that VV and QQ progressively become correlated with Monte Carlo returns. To our knowledge, PPO results in GiGPO paper on verl-agent do not enable a warm-up during the actual training process. Our own reproduction of these PPO experiments also showed no significant impact from this settings.

> Re Q2: Full ablation analysis for why SAPO with logit-based value achieves high performance.

We first clarify that the dual-logit readout is only one contributor to SAPO’s performance, while constituting the architectural basis for its unified policy–value formulation. Below, we provide more fine-grained ablation comparisons of the logit-based value design.

Table 1 shows that (a) replacing the logit-based readout with a conventional value-head network (w/w/ Value Head) consistently degrades performance across experimental settings, supporting the effectiveness of the joint PPO+SARSA optimization. Disabling value scaling also substantially reduces SAPO’s performance across all environments and model sizes, while using a single-logit readout leads to a moderate degradation. These results indicate that (b) constraining the value-target scale and using the dual-logit mechanism improve the effectiveness and stability of value learning. Moreover, the ablation studies in the paper demonstrate that (c) auxiliary QQ-value learning also contributes positively to SAPO’s performance.

Table 1. Additional ablation studies.

MethodPickCleanCoolLookHeatPick2All
Qwen2.5-1.5B-Instruct
SAPO96.4 ± 1.481.2 ± 5.1100.0 ± 0.0100.0 ± 0.097.2 ± 3.287.7 ± 5.194.6 ± 1.4
−-Value Scaling95.3 ± 3.270.6 ± 25.588.3 ± 9.588.3 ± 11.582.9 ± 11.745.2 ± 6.880.5 ± 2.9
w/w/ Single Logit99.0 ± 1.983.4 ± 17.395.5 ± 6.4100.0 ± 0.085.7 ± 6.173.9 ± 10.089.9 ± 3.3
w/w/ Value Head93.4 ± 4.290.2 ± 7.491.5 ± 6.292.9 ± 6.085.3 ± 5.58.8 ± 6.377.3 ± 4.3
Qwen2.5-7B-Instruct
SAPO98.8 ± 1.685.5 ± 11.299.1 ± 2.0100.0 ± 0.095.6 ± 4.792.3 ± 8.296.1 ± 1.1
−-Value Scaling89.9 ± 4.171.7 ± 7.188.4 ± 8.491.3 ± 6.690.4 ± 7.772.6 ± 15.785.7 ± 2.1
w/w/ Single Logit100.0 ± 0.086.9 ± 9.4100.0 ± 0.093.1 ± 7.083.5 ± 7.383.9 ± 5.092.2 ± 1.7
w/w/ Value Head98.1 ± 1.981.2 ± 6.793.0 ± 4.988.9 ± 4.483.4 ± 10.592.6 ± 3.990.8 ± 2.3

> Re W1: Does joint policy/value learning cause optimization conflict or damage the original LLM capability?

We first emphasize that SAPO uses separate logits and gradients for next-token prediction objective and value learning objective, and no logit is optimized by both losses. We also provide a more detailed and carefully gradient conflict analysis among these objectives below.

We separately backpropagated the policy, V, and Q objectives on the same diagnostic micro-batch, aggregated FSDP-sharded gradient statistics over the full model, and report mean ± standard deviation. In Table 1, the policy–V and policy–Q cosines are near zero, indicating near-orthogonality rather than conflict, while V-Q cosines show mild positive alignment. Moreover, the joint gradient remained a policy-descent direction at all measured checkpoints. We therefore find no evidence of harmful parameter competition in the measured regime.

Table 1. Statistics on gradient metrics among Policy, V, and Q Objectives in SAPO during training time using three consecutive checkpoints.

MetricDefinitionResult
Policy–V gradient cosinecos⁡(gπ,gV)\cos(g_\pi,g_V)0.0023±0.00960.0023\pm0.0096
Policy–Q gradient cosinecos⁡(gπ,gQ)\cos(g_\pi,g_Q)0.001±0.00350.001\pm0.0035
V–Q gradient cosinecos⁡(gV,gQ)\cos(g_V,g_Q)0.112±0.2790.112\pm0.279
Total policy alignment⟨gπ,gjoint⟩/∥gπ∥2\langle g_\pi,g_{\mathrm{joint}}\rangle/\lVert g_\pi\rVert^21.0003±0.00311.0003\pm0.0031

Background: Assuming non-zero gradients and positive loss coefficients, a positive cosine cos⁡(⋅)>0\cos(\cdot)>0 indicates first-order alignment, where the auxiliary gradient reinforces the policy direction; a cos⁡(⋅)<0\cos(\cdot)<0 indicates some gradient conflicts; and a cos⁡(⋅)=0\cos(\cdot)=0 indicates orthogonality, where it neither counteracts nor reinforces the policy gradient, with the objectives optimizing along different directions in the shared parameter space.

Thank you for the clarification. The additional ablation study and the statistics on gradient well address my concerns. I will update my score to weak accept.

AuthorsOpenReview ↗

Response to Reviewer Xbi7

We thank you for your constructive feedback and positive assessment.

Reviewer_5rSYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces SAPO (Single-Rollout Autoregressive Policy Optimization), an RL algorithm for LLM agents. SAPO targets two common limitations of existing LLM agent RL: vanilla PPO requires a separate critic network, incurring high memory overhead, while GRPO and related group-relative methods need multiple rollouts per prompt and suffer from poor temporal credit assignment. The core design reuses the autoregressive LLM backbone for both policy and value prediction. It leverages two reserved special tokens (w+,w−w^+,w^-) at causal boundaries to read state-value V and action-value Q without interfering with action generation. The joint loss combines clipped PPO objective, GAE advantage estimation, on-policy SARSA Q regression, KL penalty and entropy regularization. Unlike GRPO, SAPO only samples one trajectory per task prompt, eliminating per-prompt group sampling. Experiments on ALFWorld and WebShop show SAPO outperforms vanilla PPO and GRPO, reducing iteration wall-clock time by ~33% compared to separate-critic PPO while maintaining competitive success rates.

Critical strengths
  1. The problem is timely and practically relevant for LLM agent training. Reducing memory and sampling cost is a key bottleneck in agentic RL.
  2. The algorithmic design is conceptually clean.
Critical weaknesses
  1. The evaluation benchmark suite is narrow. Results are limited to ALFWorld and WebShop, both relatively short-horizon agent environments. There is no evidence on long-horizon tasks, tool-use or OOD transfer, making it hard to confirm whether SAPO’s credit assignment and shared-value design generalize beyond these two environments.
  2. Missing controlled comparisons with closely related single-rollout baselines (SAO, SPO). These methods also train agents with one rollout per prompt, yet the paper does not compare under identical model sizes and compute budgets. Without this comparison, the unique advantage of SAPO is insufficiently justified.
  3. The special token mechanism requires stronger ablation. Injecting reserved tokens to read V/Q modifies the native autoregressive pipeline, raising portability concerns. A critical ablation baseline (predicting values at trajectory end without w+/w−w^+/w^-) is absent, so we cannot isolate performance gains from the token design versus the joint PPO+SARSA objective. There is also limited analysis on whether joint policy/V/Q optimization causes parameter competition or representation drift.

Minor Concerns: Several ablation studies are incomplete, including sensitivity to loss weights and GAE λ\lambda. Reproducibility is a concern given the nonstandard special-token value readout; releasing full training code would significantly strengthen the submission. The paper also lacks analysis of failure modes under sparse or noisy reward settings.

Questions for authors
  1. Empirically, SAPO demonstrates clear computational efficiency gains. Its single-rollout formulation removes the requirement to collect multiple trajectories for each prompt, which is attractive for interactive agent environments where grouped sampling is difficult.
  2. Risk of slicing with PlanPO (2608.17289), a concurrent paper. The authors must clearly demarcate the two contributions and ideally provide a small fusion experiment to demonstrate orthogonality.
  3. Single-rollout training naturally has higher gradient variance than GRPO’s group sampling. The paper relies empirically on GAE and batch normalization for stabilization, but lacks quantitative gradient variance measurements or discussion about convergence and parameter conflicts in multi-task supervision on a single backbone.
4 comments

Dear Reviewer 5rSY:

We thank the reviewer for the valuable feedback. To address the reviewer’s concerns, we add a new benchmark, the SPO baseline, comprehensive ablations, and gradient analysis, further demonstrating SAPO’s computational efficiency and performance. Our point-by-point responses follow.

> Re Q1: SAPO demonstrates clear computational efficiency gains.

We thank the reviewer for the positive assessment. SAPO aims to share PPO’s actor and critic in an elegant and scalable manner without extensive manual engineering, achieving low compute and memory overhead while retaining PPO-level learning capability and performance. We also note that the “33%” figure in the Official Review’s Paper Summary is a typo, the reported per-iteration runtime is 23.1%.

> Re Q2: Risk of slicing with PlanPO.

We apologize for this oversight during code packaging. “PlanPO” is a file or field entirely unrelated to this work, and the corresponding code is never invoked during execution. It can therefore be safely ignored.

> Re Q3/W3: Quantitative gradient conflict analysis and discussion.

We first emphasize that SAPO uses separate logits and gradients for next-token prediction and value learning, and no logit is optimized by both losses. We also provide a more detailed and carefully gradient conflict analysis below.

With three consecutive checkpoints (150-152 steps), we separately backpropagated the policy, V, and Q objectives on the same diagnostic micro-batch, aggregated FSDP-sharded gradient statistics over the full model, and report mean ± standard deviation.

In Table 1, the policy–V, policy–Q, and V–Q gradient cosines were 0.0023, 0.001, and 0.112, respectively, indicating near-orthogonality between the policy and auxiliary objectives and mild positive alignment between V and Q, with no evidence of substantial pairwise gradient conflict. Total policy alignment remained 1.0003±0.00311.0003\pm0.0031, and the joint gradient was a policy-descent direction at all three checkpoints. Thus, we find no evidence that joint policy/V/Q supervision settings in SAPO causes harmful parameter competition on the shared backbone.

Table 1. Gradient Metric among Policy, V, and Q Objectives in SAPO.

MetricDefinitionResult
Policy–V gradient cosinecos⁡(gπ,gV)=⟨gπ,gV⟩∥gπ∥2∥gV∥2\cos(g_\pi,g_V)=\frac{\langle g_\pi,g_V\rangle}{\lVert g_\pi\rVert_2\lVert g_V\rVert_2}0.0023±0.00960.0023\pm0.0096
Policy–Q gradient cosinecos⁡(gπ,gQ)=⟨gπ,gQ⟩∥gπ∥2∥gQ∥2\cos(g_\pi,g_Q)=\frac{\langle g_\pi,g_Q\rangle}{\lVert g_\pi\rVert_2\lVert g_Q\rVert_2}0.001±0.00350.001\pm0.0035
V–Q gradient cosinecos⁡(gV,gQ)=⟨gV,gQ⟩∥gV∥2∥gQ∥2\cos(g_V,g_Q)=\frac{\langle g_V,g_Q\rangle}{\lVert g_V\rVert_2\lVert g_Q\rVert_2}0.112±0.2790.112\pm0.279

Background: Assuming non-zero gradients and positive loss coefficients, a positive cosine cos⁡(⋅)>0\cos(\cdot)>0 indicates first-order alignment, where the auxiliary gradient reinforces the policy direction; a cos⁡(⋅)<0\cos(\cdot)<0 indicates gradient conflict, where it partially opposes the policy direction; and a cos⁡(⋅)=0\cos(\cdot)=0 indicates orthogonality, where it neither counteracts nor reinforces the policy gradient, with the objectives optimizing along different directions in the shared parameter space.

> Re W1: The evaluation benchmark suite is narrow.

In Table 2, we add experiments on the new SciWorld benchmark to support the scalability of SAPO. The results further demonstrates that SAPO achieves performance close to PPO while significantly reducing runtime, and substantially outperforms GRPO.

Table 2. Score comparison using the Qwen2.5-7B-Instruct model. Each result reports the mean and standard deviation over three seeds. We also report the runtime required with 50 steps under the same devices: 2 NVIDIA H200 GPUs and 16 CPUs on an Ubuntu 2020 system.

MethodSeed 1 ↑Seed 2 ↑Seed 3 ↑Mean ± Std ↑50-Step Runtime (hour) ↓
GRPO36.375036.039133.289135.23 ± 1.389.4 h
PPO51.085952.062548.859450.67 ± 1.3412.1 h
SAPO51.593849.679750.234450.50 ± 0.808.7 h

> Re W2: Missing controlled comparisons with closely related single-rollout baselines (SAO, SPO).

To address the reviewer’s concern, we add the recent single-rollout baseline SPO and SharedPPO w/ value head (a PPO baseline that shares the actor–critic backbone through a value head). We cannot currently compare with SAO because it requires asynchronous infrastructure and specialized long-horizon environments.

As shown in Table 3, SAPO outperforms both baselines on ALFWorld and WebShop with Qwen2.5-1.5B and Qwen2.5-7B. SPO’s value tracker is task-specific: it maintains historical samples for each task and is sensitive to initialization, making performance vulnerable to configuration choices. And, value head introduces greater system complexity through an additional value head, including its optimizer, FSDP wrapping, network design, and initialization. Overall, SAPO provides a more scalable and efficient approach to single-rollout policy optimization.

Table 3. Score comparison on ALFWorld and Webshop benchmarks. Each result reports the mean and standard deviation over three seeds.

ALFWorld:

MethodPickLookCleanHeatCoolPick2All ↑
Qwen2.5-1.5B-Instruct
SharedPPO w/ value head93.4 ± 4.290.2 ± 7.491.5 ± 6.292.9 ± 6.085.3 ± 5.58.8 ± 6.377.3 ± 4.3
SPO86.8 ± 6.474.2 ± 20.178.5 ± 11.389.8 ± 7.854.5 ± 5.446.9 ± 10.471.7 ± 4.0
SAPO96.4 ± 1.481.2 ± 5.1100.0 ± 0.0100.0 ± 0.097.2 ± 3.287.7 ± 5.194.6 ± 1.4
Qwen2.5-7B-Instruct
SharedPPO w/ value head98.1 ± 1.981.2 ± 6.793.0 ± 4.988.9 ± 4.483.4 ± 10.592.6 ± 3.990.8 ± 2.3
SPO85.7 ± 4.258.5 ± 8.494.4 ± 5.689.4 ± 10.673.6 ± 9.481.1 ± 4.182.0 ± 3.6
SAPO98.8 ± 1.685.5 ± 11.299.1 ± 2.0100.0 ± 0.095.6 ± 4.792.3 ± 8.296.1 ± 1.1

Webshop:

MethodScore ↑Succ. ↑
Qwen2.5-1.5B-Instruct
SharedPPO w/ value head56.9 ± 3.852.3 ± 3.8
SPO75.2 ± 1.863.8 ± 2.4
SAPO80.1 ± 2.166.0 ± 1.5
Qwen2.5-7B-Instruct
SharedPPO w/ value head72.7 ± 0.858.6 ± 3.4
SPO73.3 ± 1.760.9 ± 2.2
SAPO88.4 ± 1.783.6 ± 1.9

> Re W3: The special token mechanism requires stronger ablation.

In addition to the component ablations reported in the paper, we provide more fine-grained comparisons of the logit-based value design. Table 4 shows that replacing the logit-based readout with a conventional value-head network (w/w/ Value Head) consistently degrades performance across experimental settings, supporting the effectiveness of the joint PPO+SARSA optimization. Disabling value scaling also substantially reduces SAPO’s performance across all environments and model sizes, while using a single-logit readout leads to a moderate degradation. These results indicate that constraining the value-target scale and using the dual-logit mechanism improve the effectiveness and stability of value learning.

Table 4. Additional ablation studies.

MethodPickCleanCoolLookHeatPick2All
Qwen2.5-1.5B-Instruct
SAPO96.4 ± 1.481.2 ± 5.1100.0 ± 0.0100.0 ± 0.097.2 ± 3.287.7 ± 5.194.6 ± 1.4
−-Value Scaling95.3 ± 3.270.6 ± 25.588.3 ± 9.588.3 ± 11.582.9 ± 11.745.2 ± 6.880.5 ± 2.9
w/w/ Single Logit99.0 ± 1.983.4 ± 17.395.5 ± 6.4100.0 ± 0.085.7 ± 6.173.9 ± 10.089.9 ± 3.3
w/w/ Value Head93.4 ± 4.290.2 ± 7.491.5 ± 6.292.9 ± 6.085.3 ± 5.58.8 ± 6.377.3 ± 4.3
Qwen2.5-7B-Instruct
SAPO98.8 ± 1.685.5 ± 11.299.1 ± 2.0100.0 ± 0.095.6 ± 4.792.3 ± 8.296.1 ± 1.1
−-Value Scaling89.9 ± 4.171.7 ± 7.188.4 ± 8.491.3 ± 6.690.4 ± 7.772.6 ± 15.785.7 ± 2.1
w/w/ Single Logit100.0 ± 0.086.9 ± 9.4100.0 ± 0.093.1 ± 7.083.5 ± 7.383.9 ± 5.092.2 ± 1.7
w/w/ Value Head98.1 ± 1.981.2 ± 6.793.0 ± 4.988.9 ± 4.483.4 ± 10.592.6 ± 3.990.8 ± 2.3

Dear Reviewer 5rSY,

We would greatly appreciate it if you pay attention to our response and the newly added supporting evidence. We have highlighted the key findings to make the update concise and easy to assess, and we hope you find the results relevant and engaging.

By the way, for a more intuitive comparison, you can see the [Anonymous Link], which provides the training/evaluation curves of SAPO, PPO and GRPO using Qwen-1.5B/7B/14B models.

Best regards,

The Authors

Dear Reviewer 5rSY,

As the author–reviewer discussion deadline is approaching, we sincerely remind the reviewer to pay attention to our rebuttal. Your any feedback is very important to this work. Thank you very much.

Best regards,

The Authors

Reviewer_jtpPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on agentic RL training, in which they propose SAPO (Single-rollout Autoregressive Policy Optimization) to unify policy optimization and value learning within a single causal language model. It aims to achieve better resource efficiency than conventional PPO implementations and better policy performance than critic-free group-relative methods like GRPO. Value estimation is generated from token logits rather than a separate critic model, which is more computational efficient. Experiments on ALFWorld and WebShop show the effectiveness and efficiency of the proposed method.

Critical strengths
  • The motivation of this paper is sensible. To improve the training efficiency and performance of agentic RL training is an important topic as LLMs are applied to more diverse and complex domains.
  • Experiments show that efficiency improvement of SAPO over PPO, and the improvement on task performance.
Critical weaknesses
  • Some related baselines may be missed to validate the design choice of SAPO. For example, what are the specific advantages of using token logits to derive the value, compared to using a value head to compute it from the embeddings?
  • More complex environments and other model families can be used to demonstrate the broad effectiveness of the proposed method.
Questions for authors
  • Could the authors show the difference between using token logits to derive values and using a value head to get them?
  • Do the training objectives interfere each other if the embedding at a position needs to predict the next token and derive values at the same time?
  • How is the total training cost of SAPO compared to other baselines?
  • Do the authors have experiments on other model families and benchmarks?
4 comments

Dear Reviewer jtpP:

Thank you for your careful review and critical comments. To address your concerns, we conducted key auxiliary experiments within the limited time available to highlight the scalable performance of SAPO over existing related methods. We respond to the questions and weaknesses point by point below.

> Re Q1/W1: The difference between using token logits to derive values and using a value head and the specific advantages with token logits

The differences between using token logits to derive values, as in SAPO, and using a value head, temporarily referred to as SharedPPO w/ value head, mainly lie in:

a). Implementation and learning property differences: The value-head method introduces an additional deployment path and a non-native network module, resulting in higher system complexity. For example, this method requires modifications to head-network registration, the optimizer, FSDP wrapping, and so on. It also involves the network size and initialization tricks, and requires numerical matching with the original embedding states. In contrast, token logits are fully integrated with LLMs and therefore involve almost none of these issues.

b). Scalability differences: The token-logits-based SAPO method can be quickly applied to new training frameworks and LLM families, such as Llama and Deepseek models. In contrast, applying the value-head method across frameworks and LLM families incurs higher costs, as it requires consideration of systematic issues such as vLLM/FSDP2 and hyperparameters. All of the above require precise manual tuning, and improper settings are more likely to result in poor performance.

Benefiting from the advantages of using token logits to derive values, we can efficiently implement the SAPO insight on top of them. The experiments provided below also empirically demonstrate that the value-head approach suffers from scalability instability.

> Re Q2: Do the training objectives interfere with each other?

Next-token prediction learning and value learning do not interfere with each other when the embedding at a position needs to predict the next token and derive values. The gradients at the logit positions used to define values participate only in value prediction and learning and are excluded from inference and training related to next-token prediction, as detailed in Equation (3).

> Re Q3: How does the total training cost of SAPO compare with other baselines?

In addition to the per-iteration runtime comparison already reported in the paper, we provide a comprehensive comparison of the total training costs below.

As shown in Table 1, the total training time of SAPO on the ALFWorld, WebShop, and SciWorld environments using Qwen2.5-1.5B/7B, and Llama3.2-3B models is significantly lower than that of PPO, with an average reduction of 26.1%, and is comparable to that of GRPO.

Table 1. Complete runtime comparison within 150-step traning in ALFWorld and WebShop and 50-step traning in SciWorld.

MethodALFWorld-1.5BALFWorld-7BALFWorld-3BWebShop-1.5BWebShop-7BSciWorld-7B
H200 GPU122122
GRPO17.9h26.4h25.5h7.2h13.5h9.4h
PPO25.5h33.6h35.2h8.7h17.9h12.1h
SAPO17.6h25.1h24.4h7.5h13.0h8.7h
SAPO vs. PPO↓ 31.0%↓ 25.3%↓ 30.7%↓13.8%↓ 27.4%↓ 28.1%

> Re Q4/W2: Do the authors have experiments on other model families and benchmarks?

We add experiments on Llama-3.2-3B-Instruct and the SciWorld benchmark to support the stronger performance and scalability of SAPO.

First, as shown in Table 2, the results on the new Llama3.2-3B model demonstrate that SAPO significantly outperforms PPO and GRPO on both ALFWorld and WebShop. On this model, GRPO fails to obtain a score on WebShop and also performs poorly on ALFWorld. Overall, the results show that SAPO can still be stably extended to a new model family and achieve performance no worse than the standard PPO baseline at a low resource cost.

Incidentally, for the Llama model, we select the <|reserved_special_token_0|> and <|reserved_special_token_1|> tokens from the logits to define values. Previously, for the Qwen model, we used the <|vision_start|> and <|vision_end|> tokens. These tokens were selected simply according to their semantics.

Table 2. Comparison of Success Rate on ALFWorld and WebShop using Llama3.2-3B. Each result reports the mean and standard deviation over three seeds.

ALFWorld:

MethodPickLookCleanHeatCoolPick2All
PPO91.5 ± 1.074.3 ± 13.990.9 ± 4.567.6 ± 16.780.5 ± 7.679.8 ± 4.283.6 ± 1.1
GRPO64.3 ± 7.150.7 ± 20.846.0 ± 8.618.5 ± 7.829.0 ± 4.141.2 ± 5.144.3 ± 2.6
SAPO94.6 ± 1.971.6 ± 10.892.6 ± 2.897.6 ± 3.494.6 ± 4.586.3 ± 2.491.1 ± 2.6

WebShop:

MethodScore ↑Succ. ↑
PPO85.3 ± 1.972.4 ± 0.7
GRPO0.0 ± 0.00.0 ± 0.0
SAPO88.8 ± 0.581.8 ± 2.1

Second, as shown in Table 3 and Table 4, the comparison on the new SciWorld benchmark further demonstrates that SAPO achieves performance close to PPO while significantly reducing runtime, and substantially outperforms GRPO. PPO requires 39.1% more runtime than SAPO. In addition, the previously mentioned SharedPPO w/ value head method does not perform well on SciWorld.

Table 3. Score comparison of the methods using the Qwen2.5-7B-Instruct model with 50 training steps. We also report the full runtime under the same device, e.g., 2 NVIDIA H200 GPUs and 16 CPUs on an Ubuntu 2020 system.

MethodSeed 1Seed 2Seed 3Mean Score ± Std ↑50-Step Runtime (hour) ↓
GRPO36.375036.039133.289135.23 ± 1.389.4 h
SharedPPO w/ value head16.640617.226615.289116.39 ± 0.8110.7 h
PPO51.085952.062548.859450.67 ± 1.3412.1 h
SAPO51.593849.679750.234450.50 ± 0.808.7 h

Table 4. Per-stage runtimes in the first step, measured in seconds.

MethodGen ↓Old Log Prob ↓Reward ↓Ref ↓Advantage ↓Update Actor ↓Values ↓Update Critic ↓Step ↓
SharedPPO w/ value head309.0256.710.8470.740.10213.6800654.35
GRPO274.6756.750.8073.940.28215.2600624.47
PPO300.1553.670.8169.560.09242.1548.74150.91868.80
SAPO277.5357.830.8268.690.11213.3300621.49

> Re W1: SAPO vs. "SharedPPO w/ value head" baseline comparison.

In Figure 5, the results demonstrate that SAPO outperforms the "SharedPPO w/ value head" approach across both the AlfWorld and WebShop environments, as well as in both Qwen-1.5B/7B/14B models. Furthermore, we observed that the "SharedPPO w/ value head" method exhibits significantly higher variance than SAPO in all experiments. These findings empirically suggest that the SAPO settings offers superior overall learning performance. An other comparison results see Re Q4/W2.

Table 5. Success rate comparison. Each result reports the mean and standard deviation over three seeds.

ALFWorld:

MethodPick ↑Look ↑Clean ↑Heat ↑Cool ↑Pick2 ↑All ↑
Qwen2.5-1.5B-Instruct
SharedPPO w/ value head93.4 ± 4.290.2 ± 7.491.5 ± 6.292.9 ± 6.085.3 ± 5.58.8 ± 6.377.3 ± 4.3
SAPO96.4 ± 1.481.2 ± 5.1100.0 ± 0.0100.0 ± 0.097.2 ± 3.287.7 ± 5.194.6 ± 1.4
Qwen2.5-7B-Instruct
SharedPPO w/ value head98.1 ± 1.981.2 ± 6.793.0 ± 4.988.9 ± 4.483.4 ± 10.592.6 ± 3.990.8 ± 2.3
SAPO98.8 ± 1.685.5 ± 11.299.1 ± 2.0100.0 ± 0.095.6 ± 4.792.3 ± 8.296.1 ± 1.1
Qwen3-14B-Instruct
SharedPPO w/ value head100.0 ± 0.0100.0 ± 0.0100.0 ± 0.078.1 ± 8.893.6 ± 2.091.4 ± 7.694.0 ± 1.6
SAPO99.0 ± 1.895.8 ± 7.2100.0 ± 0.097.6 ± 4.198.7 ± 2.2100.0 ± 0.098.7 ± 0.5

Webshop:

MethodScore ↑Succ. (%)↑
Qwen2.5-1.5B-Instruct
SharedPPO w/ value head56.9 ± 3.852.3 ± 3.8
SAPO80.1 ± 2.166.0 ± 1.5
Qwen2.5-7B-Instruct
SharedPPO w/ value head72.7 ± 0.858.6 ± 3.4
SAPO88.4 ± 1.783.6 ± 1.9
Qwen3-14B-Instruct
SharedPPO w/ value head77.1 ± 0.168.0 ± 1.4
SAPO90.0 ± 1.384.4 ± 0.8

Finally, together with the ablation experiments (some in Reviewer 5rSY), we believe that the superiority and stability of SAPO stems from components such as scaled value learning and auxiliary Q-learning, which are built upon the natural coupling between LLMs and logit-based values while incorporating the autoregressive property of LLMs. Notably, the “SharedPPO w/ value head” adopts the conventional value-head setting, in which a fully connected layer is added after the embedding layer to predict scalar values.

AuthorsOpenReview ↗

Update Qwen3-14B-Instruct Comparison in [Re W1].

Dear Reviewer jtpP:

We have just completed training and updated the comparative data for the Qwen3-14B model, evaluating SAPO vs. "SharedPPO w/ value head" baseline. This baseline serves as a critical point of comparison, please refer to the detailed training curves at [Anonymous Link].

Beyond SAPO’s higher success rate, we also observed encouraging results regarding runtime efficiency. When run on the same hardware configuration (pods with 32 H20 GPUs), the "SharedPPO w/ value head" approach required 17.1h (vs.14.0h for SAPO) on Webshop, and 34.8h (vs. 22.1h for SAPO) on AlfWorld to complete 150 training steps. This indicates that, despite the "value head" settings nearly entirely sharing the LLM backbone, it incurs an additional training time overhead of 22.1% and 57.5%, respectively, compared to SAPO. We attribute this to the factors mentioned in [Re Q1/W1], i.e. the value head introduces an additional deployment path and a non-native network module, which increases system complexity (for instance, potentially impacting FSDP and the optimizer, even though we have registered the head network with the LLM backbone). Overall, SAPO’s objective is exactly this: to integrate policy and value learning elegantly and efficiently within a full single LLM.

We sincerely encourage the reviewers to revisit our work and consider the new supporting evidence. We believe you will find it engaging, and it will not require a significant time commitment, as we have highlighted the key points.

Best regards,

Authors

Dear Reviewer jtpP,

As the author–reviewer discussion deadline is approaching, we sincerely remind the reviewer to pay attention to our rebuttal. Your feedback is very important to this work. Thank you very much.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.