acceptodds
Under review as a conference paper at ICLR 2027

SWPO: Score-Weighted Critic Fitting for Multi-Turn Policy Optimization

Abstract

As language agents generate longer reasoning and tool-use trajectories, learning from sparse outcome rewards becomes increasingly difficult. Critic-free group methods can receive weak or noisy learning signals in this setting, motivating PPO's learned state-dependent baseline. Turn-level PPO aligns optimization with complete reasoning-and-action decisions and can produce substantially shorter responses, but aggregating token scores and likelihood ratios can also increase update variability. Optimal-baseline theory motivates score-energy-weighted critic fitting to address this variability, yet existing OPT logit proxies require vocabulary-wide computation. We introduce SWPO, which constructs a cheap scalar proxy from stored sampled-token probabilities and sums it within each turn to weight critic fitting. The method retains the turn-level PPO actor and admits an exact factor-of-two logit-space bound, with a conditional interpretation of population baseline error. Experiments on search and embodied tasks support summed scalar weighting among the tested formulations, preserve shorter responses in the search-training comparison, and achieve competitive performance with the summed OPT proxy. SWPO reduces measured actor-update time by about 40% relative to summed OPT, with little additional time over the PPO baselines.

Primary area:
reinforcement learning
Keywords:
multi turn, PPO, GRPO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8fCgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SWPO, a method that weights turn-level critic fitting using the sum of (1−p)2(1-p)^2 over sampled tokens while retaining the turn-level PPO actor. This inexpensive proxy approximates gradient-energy weighting without vocabulary-wide computation. The paper provides a token-level logit-gradient bound and a conditional analysis of baseline approximation error. Experiments with Qwen3-4B/8B on search and embodied tasks compare PPO and group-based baselines, alongside controlled critic-weighting ablations.

Critical strengths

The controlled ablations keep the actor and critic architecture fixed, providing evidence that the proposed weighting contributes beyond the turn-level PPO backbone. SWPO also achieves competitive performance with summed OPT at substantially lower actor-update cost, supporting its practical value as an inexpensive alternative to vocabulary-wide proxy computation.

Critical weaknesses
  1. SWPO uses token probabilities to estimate which turns have larger policy gradients. However, the theoretical bound concerns gradients with respect to logits, not the full model parameters. Please compare the proposed weights with actual turn-level parameter-gradient norms to verify whether higher weights correspond to larger gradients.

  2. In the critic-weighting ablations, the variants differ in both relative sample weights and overall loss scale. Please clarify whether the performance differences arise from which turns receive more weight or are also influenced by changes in critic optimization due to the loss scale.

  3. Each turn’s critic weight is computed from its token probabilities at generation time and remains fixed during subsequent updates. After policy updates, PPO clipping may suppress that turn’s contribution to the actor gradient, even when its critic weight remains high. Please examine how often this occurs and whether the fixed weights still emphasize turns that influence the actor update.

Questions for authors

Please see weaknesses.

2 comments

Thank you for these questions. We provide full parameter-gradient measurements, a unit-mean weighting control, and actual-update clipping measurements.

1. Full parameter gradients. Let nn index turns, SnS_n be the starting history, AnA_n the generated turn, LnL_n its token count, an,ta_{n,t} token tt, and Zn,tZ_{n,t} its preceding context. Let θ\theta denote actor parameters, θ0\theta_0 the frozen rollout parameters, and πθ\pi_\theta the token policy. Define pn,t=πθ0(an,t∣Zn,t)p_{n,t}=\pi_{\theta_0}(a_{n,t}\mid Z_{n,t}), parameter-score energy ρn\rho_n and proxy WnW_n:

\qquad W_n=\sum_{t=1}^{L_n}(1-p_{n,t})^2.$$ We use one backward pass per turn through all trained actor parameters, including the cross-token terms in the squared norm. Across 12 frozen checkpoints from seed 42, we sample $J=16$ held-out states and $M=32$ alternatives per state: **6,144 measured turns**. Let $F$ denote the sum of exact token logit-score energies, computed across the vocabulary. At state $s_j$, let $m$ index alternatives and $X_{jm}$ be a candidate weight $X\in\{1,W,F,L,W/L\}$. Let $\hat\mu_{X,j}$ and $\hat\mu_{\rho,j}$ be the respective sample means. Define normalized discrepancy $\hat\eta_X(s_j)$ and its state average $\bar\eta_X$: $$\hat\eta_X(s_j)=\frac1M\sum_{m=1}^{M} \left|\frac{\rho_{jm}}{\hat\mu_{\rho,j}}-\frac{X_{jm}}{\hat\mu_{X,j}}\right|, \qquad \bar\eta_X=\frac1J\sum_{j=1}^{J}\hat\eta_X(s_j).$$ Smaller discrepancy means closer agreement with relative full parameter-gradient energy. Final-checkpoint results are: | Environment | Uniform | $W$ | $F$ | $L$ | $W/L$ | Partial rank correlation of $W,\rho$ | | --- | --- | --- | --- | --- | --- | --- | | Search | 0.717 | **0.185** | 0.186 | 0.386 | 0.397 | 0.732 | | WebShop | 0.781 | **0.174** | 0.171 | 0.436 | 0.417 | 0.752 | | ALFWorld | 0.699 | **0.190** | 0.187 | 0.415 | 0.394 | 0.763 | The correlations control turn length and state. At all 12 checkpoints, $W$ has lower discrepancy than uniform, length-only and mean-token weights; its discrepancy differs from $F$ by at most 0.0025. This directly validates the proxy against parameter-space measurements rather than relying only on the logit-space bound. The connection to the baseline is explicit. For completed return $G\in[0,1]$, let $\mathbb E_s$ denote expectation conditional on turn-start state $s$. For $X\in\{\rho,W\}$ with positive mean, define the population weighted baseline $b_X^*(s)=\mathbb E_s[XG]/\mathbb E_s[X]$. Let $\eta_W(s)$ be the population counterpart of the discrepancy above. Our analysis gives $$\left|b_W^*(s)-b_\rho^*(s)\right|\le\frac12\eta_W(s).$$ Thus the measured normalized-energy agreement probes the quantity governing the proxy-baseline approximation. **2. Relative weighting at controlled mean scale.** We added normalized SWPO while retaining original raw SWPO and uniform Turn PPO. Let $\mathcal B$ be all valid turns in the full rollout batch across data-parallel workers, $\overline W_{\mathcal B}$ their mean raw weight and $\widetilde W_n$ the detached applied weight: $$\overline W_{\mathcal B}=\frac1{|\mathcal B|}\sum_{n\in\mathcal B}W_n, \qquad \widetilde W_n=\frac{W_n}{\overline W_{\mathcal B}}, \qquad \frac1{|\mathcal B|}\sum_{n\in\mathcal B}\widetilde W_n=1.$$ The scalar is fixed during that batch's critic updates; other settings match Turn PPO. Three independent Qwen3-8B training seeds give the matched-compute results below. Search is equal-weight macro Pass@1 over HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle, NQ, TriviaQA and PopQA. Other tasks report single-attempt success; differences are percentage points (pp), with paired 95% Student CIs across seeds. | Environment | GPU-hours | Normalized SWPO - Turn PPO, pp [95% CI] | | --- | --- | --- | | Search | 256 | **2.14 [0.46, 3.82]** | | WebShop | 120 | **2.07 [1.03, 3.10]** | | ALFWorld | 40 | **2.60 [1.11, 4.09]** | Selective weighting therefore retains gains when its mean is fixed to the uniform baseline's unit scale. **3. Fixed weights and clipping.** Each fresh rollout batch receives one full-batch actor optimizer step and one PPO epoch, with no intervening sequential actor updates. Let $r_n(\theta)$ be the complete-turn importance ratio and $\Pi_\theta$ the turn probability. At actual gradient evaluation, $$\theta=\theta_0 \quad\Longrightarrow\quad r_n(\theta)=\frac{\Pi_\theta(A_n\mid S_n)}{\Pi_{\theta_0}(A_n\mid S_n)}=1.$$ This is strictly inside the PPO clipping interval [0.8, 1.2]. Across 48 protocol checks, $\max_n|r_n-1|=8.94\times10^{-7}$; both the suppressed-turn fraction and suppressed critic-weight mass are **zero**. The batch is then discarded, and fresh rollouts receive recomputed weights. Consequently, the suggested fixed-weight/repeated-update clipping mismatch does not occur in our protocol.

Thank you for the additional experiments and clarifications, which address my concerns in W1 and W3. The unit-mean weighting control also helps address W2, although some uncertainty remains about whether loss-scale changes contribute to the gains of raw SWPO. Could you also report a direct comparison between raw and normalized SWPO under the same compute budget and paired seeds? This would help clarify the remaining concern.

Reviewer_S2PDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWPO, a method for weighting critic fitting in turn-level PPO. It uses sampled-token probabilities to construct a cheap approximation of policy-score energy and sums these weights within each turn. The method keeps the turn-level PPO actor unchanged and aims to improve the baseline used for policy optimization.

Critical strengths
  1. The method is simple and inexpensive to implement, since it only requires stored sampled-token probabilities.
  2. The paper provides a clear theoretical motivation from optimal-baseline theory and an exact bound relating the scalar proxy to the logit-score norm.
  3. The ablations compare several weighting choices, including uniform, length-only, mean, and summed weights. The results show competitive performance with the fuller OPT proxy at a lower actor-update cost.
Critical weaknesses
  1. The search evaluation could be more complete. The paper evaluates on 2WikiMultiHopQA, MuSiQue, HotpotQA, and Bamboogle, but does not include NQ, TriviaQA, or PopQA from the Search-R1 evaluation suite. Adding these datasets would help assess whether the gains extend beyond multi-hop QA.
  2. The baseline comparison is missing GSPO. Although GSPO uses a different baseline construction and optimization granularity, it is a relevant comparison given this paper’s use of aggregated importance ratios and clipping. Including GSPO under comparable training and evaluation settings would help clarify the practical advantage of SWPO.
Questions for authors

Why is the proposed critic weighting specifically used with turn-level PPO? Could the same scalar-proxy idea be adapted to token-level critic fitting and used with token-level PPO? A comparison would help clarify whether the benefit depends on turn-level optimization or applies more generally.

Why do the authors use the product of token-level importance ratios rather than their geometric mean? The product may become sensitive to turn length and trigger clipping even when individual token probabilities change only slightly. Could the authors discuss this choice and compare the two formulations?

1 comment

Thank you for recognizing SWPO's simplicity, theoretical motivation and efficiency. We added the requested QA evaluations, faithful GSPO baseline, token-level extension and ratio comparison. These experiments use Qwen3-8B and three independent training seeds (42, 123, 2025). Search denotes equal-weight macro Pass@1 over HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle, NQ, TriviaQA and PopQA; WebShop/ALFWorld report single-attempt success. Differences are percentage points (pp), with paired 95% Student CIs across training seeds.

1. Additional QA coverage. NQ, TriviaQA and PopQA are evaluation only; they enter neither training nor checkpoint selection. Mean Pass@1 (%), using 500 fixed examples per dataset with shared decoding and scoring, is:

MethodNQTriviaQAPopQA
Turn PPO43.3365.9346.67
GSPO43.0066.1346.47
SWPO47.3369.4049.87

The added-three macro improvement over Turn PPO is 3.56 pp [1.46, 5.65], demonstrating gains beyond the original multi-hop evaluation.

2. GSPO and token-level comparisons. GSPO uses its sequence-normalized importance ratio and group-relative advantages with eight rollouts per prompt. We match the backbone, data, generated-trajectory budget, evaluation and scheduled endpoints. This faithful GSPO baseline is distinct from geometric-ratio SWPO, which retains a learned critic.

Scheduled comparison, pp [95% CI]SearchWebShopALFWorld
SWPO - GSPO4.11 [2.51, 5.72]4.47 [2.46, 6.47]6.47 [5.43, 7.50]
Token SWPO - Token PPO2.09 [0.74, 3.43]2.47 [0.02, 4.92]1.53 [0.50, 2.57]

The scalar construction applies directly at token granularity. Let tt index generated tokens, ata_t be the sampled token, ZtZ_t its preceding prefix, πold\pi_{\mathrm{old}} the rollout policy, and pt=πold(at∣Zt)p_t=\pi_{\mathrm{old}}(a_t\mid Z_t). Let GtG_t be the same token-level return target as Token PPO, Vϕ(Zt)V_\phi(Z_t) the prefix critic with parameters ϕ\phi, and Et\mathbb E_t the empirical average over valid tokens. With detached weight qt=(1−pt)2q_t=(1-p_t)^2, the critic loss LVtoken\mathcal L_V^{\mathrm{token}} is

=\mathbb E_t\left[q_t\bigl(G_t-V_\phi(Z_t)\bigr)^2\right].$$ Keeping the Token PPO actor, this extension improves over Token PPO in all three environments. Thus the proxy is useful beyond turn-level optimization. Turn-level fitting additionally aligns the critic state and actor action with a complete reasoning-and-action decision. **3. Product versus geometric ratios.** Let $n$ index turns, $L_n$ be the generated turn length, $a_{n,t}$ its token at position $t$, $Z_{n,t}$ its prefix, and $\pi_\theta$ the current policy with parameters $\theta$. Define token ratio $u_{n,t}$, product ratio $r_n^{\mathrm p}$ and geometric ratio $r_n^{\mathrm g}$: $$u_{n,t}=\frac{\pi_\theta(a_{n,t}\mid Z_{n,t})}{\pi_{\mathrm{old}}(a_{n,t}\mid Z_{n,t})}, \qquad r_n^{\mathrm p}=\prod_{t=1}^{L_n}u_{n,t}, \qquad r_n^{\mathrm g}=(r_n^{\mathrm p})^{1/L_n}.$$ The product is the exact likelihood ratio of the complete turn, matching the turn-score derivation. The geometric variant length-normalizes it. With the same raw SWPO critic weighting, their performance is: | Product - geometric SWPO, pp [95% CI] | Search | WebShop | ALFWorld | | --- | --- | --- | --- | | Difference | 0.53 [-1.15, 2.22] | -0.33 [-2.88, 2.22] | 0.07 [-1.94, 2.07] | Aggregate means differ by at most **0.54 pp**, supporting comparable observed performance with both ratios. This reinforces the practical applicability of the critic-weighting proxy across these actor formulations. Regarding length-sensitive clipping, our actor takes one full-batch optimizer step per fresh rollout batch, with one PPO epoch. Ratios are evaluated before that step, when actor and rollout parameters coincide. Across the protocol checks, the maximum absolute ratio departure from one is $8.94\times10^{-7}$ and the suppressed-turn fraction is zero. Thus the proposed repeated-update clipping mechanism does not arise in this training protocol.
Reviewer_xVgVOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SWPO, a turn-level PPO variant for multi-turn LLM agents trained from sparse outcome rewards. The actor multiplies token likelihood ratios within a turn and applies PPO clipping once to the whole-turn ratio, a state-only critic is queried at turn boundaries.

Critical strengths

1.The proposed modification is simple, inexpensive, and precisely specified.

Critical weaknesses
  1. The advance over existing score-weighted baseline is limited. SWPO retains the established optimal-baseline objective and the same turn-level PPO actor. The results offer limited additional insight into the underlying learning problem. Consequently, the contribution depends heavily on demonstrating a consequential practical benefit from this simplification.

2.The experiments do not establish a compelling performance–compute advantage over the simpler unweighted alternative. The strongest efficiency result compares SWPO against summed OPT, but unweighted Turn PPO is itself cheaper: SWPO takes 7.6% more actor-update time and produces slightly longer search responses. Its performance improvements are also mixed. SWPO exceeds the strongest main baseline on the primary Avg@4 endpoint in only six of twelve task–model pairs, and is below unweighted Turn PPO on both Pass@1 and Avg@4 at both Bamboogle model scales. Competitive performance with a more expensive proxy supports the proposed approximation, but it does not establish that score-weighted critic fitting is a better use of training resources than continuing the simpler Turn PPO baseline. It could help to have a comparison at matched total computation, or computation required to reach a common reward level

Questions for authors
  1. How many independent training runs were used for each configuration? If repeated runs are available, could you summarize whether the main comparisons in Tables 1–2 are consistent across runs? Results from a small number of representative settings would help clarify the strength of the empirical conclusions.

  2. Could you provide a small diagnostic comparing the proposed weight WnW_n with the parameter-space turn-score energy ρn\rho_n, where computing these gradients is feasible? A limited measurement at a fixed checkpoint would help illustrate how well the inexpensive proxy captures the quantity motivating the method.

  3. What accounts for the additional actor-update cost of summed OPT in Table 4? Does the overhead mainly come from the vocabulary-wide reduction, materializing probabilities, additional forward computation, or communication across devices? A brief breakdown would help explain the timing and utilization differences, and clarify whether the relative saving is expected to persist with further implementation optimization.

  4. It would be informative to examine how weighted fitting changes critic prediction errors across turns. For example, comparing held-out critic MSE on turns with larger and smaller WnW_n could illustrate whether the method allocates fitting effort as intended.

1 comment

Thank you for identifying the central practical question: whether SWPO is a better use of computation than uniform Turn PPO. Our new repeated experiments directly answer this question.

1. Contribution and matched-compute benefit. SWPO combines a sampled-token-only proxy, an exact logit-space bound and conditional baseline-error analysis, and learned turn-level critic fitting. Retaining the actor objective isolates the practical contribution of critic weighting.

We added three independent Qwen3-8B training runs per configuration (seeds 42, 123, 2025), sharing initialization, data, rewards and evaluation settings. Search is equal-weight macro Pass@1 over HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle, NQ, TriviaQA and PopQA; other environments report single-attempt success. For seed-paired percentage-point differences, let dˉ\bar d and sds_d be their mean and sample SD. Intervals below are paired 95% Student CIs, dˉ±4.30265sd/3\bar d\pm4.30265s_d/\sqrt3.

Both methods receive the same allocated GPU-hours on eight H200 GPUs, including rollouts, tool waits, actor/critic updates, proxy/reference computation, communication and checkpoint I/O; evaluation is accounted for separately.

EnvironmentGPU-hoursTurn PPO (%)SWPO (%)Gain, pp [95% CI]
Search25637.5940.763.17 [1.59, 4.76]
WebShop12045.6048.202.60 [1.11, 4.09]
ALFWorld4079.6082.873.27 [2.23, 4.30]

SWPO improves in every seed pair in all three environments. Giving Turn PPO additional updates within the same compute budget does not recover SWPO's performance. At scheduled endpoints the gains are also 3.34/3.07/3.60 pp. On Bamboogle, the new three-seed Pass@1 is 33.87% versus 31.47%, a 2.40 pp gain [0.41, 4.39].

2. Full parameter-gradient diagnostic. Let nn index turns, SnS_n be the starting history, AnA_n the generated turn, LnL_n its token count, pn,tp_{n,t} the sampled-token probability at position tt, and Πθ(An∣Sn)\Pi_\theta(A_n\mid S_n) its probability under actor parameters θ\theta. At frozen sampling parameters θ0\theta_0, define full parameter-score energy ρn\rho_n and proxy WnW_n:

\qquad W_n=\sum_{t=1}^{L_n}(1-p_{n,t})^2.$$ Each measurement differentiates the summed turn log-probability through all trained actor parameters. We measured 6,144 turns from seed 42: 16 held-out states per checkpoint, 32 alternatives per state, across 12 checkpoints. Let $F$ be the summed OPT logit-score energy, satisfying $W\le F\le2W$. For candidate $X\in\{1,W,F,L,W/L\}$ at state $s$, let $m$ index its $M=32$ alternatives, $\hat\mu_X(s)$ and $\hat\mu_\rho(s)$ be the sample means of $X$ and $\rho$, and $\hat\eta_X(s)$ the relative-energy discrepancy: $$\hat\eta_X(s)=\frac1M\sum_{m=1}^{M} \left|\frac{\rho_m}{\hat\mu_\rho(s)}-\frac{X_m}{\hat\mu_X(s)}\right|.$$ Final-checkpoint state averages and partial rank correlations are: | Environment | Discrepancy: $W$ | $F$ | Length | Correlation of $W,\rho$ controlling length/state | | --- | --- | --- | --- | --- | | Search | 0.185 | 0.186 | 0.386 | 0.732 | | WebShop | 0.174 | 0.171 | 0.436 | 0.752 | | ALFWorld | 0.190 | 0.187 | 0.415 | 0.763 | Across all checkpoints, $W$ is closer to parameter energy than uniform, length-only and mean-token weights; its discrepancy differs from OPT by at most 0.0025. Thus the evidence directly concerns full parameter gradients, beyond the logit bound. **3. OPT overhead.** Twenty synchronized same-batch profiler repeats give Search actor-update times of 282.21 s (SWPO) and 471.18 s (summed OPT), a 40.11% reduction. Of the 188.97 s gap, 74.59 s is vocabulary reduction, 45.58 s communication, 67.17 s proxy handling, and 1.62 s base computation. No full probability tensor is materialized in this replay. SWPO avoids vocabulary-wide proxy reduction by using stored sampled-token entries. Three-seed end-to-end savings are **21.19%/17.03%/19.16%** for Search/WebShop/ALFWorld, distinct from actor-only savings. **4. Held-out critic errors.** We evaluated both frozen online critics on common completed branches. Define high-weight turns by $W>20$ and low-weight turns by $W\le20$. Relative MSE changes for SWPO versus uniform fitting are: | Environment | High-weight MSE | Low-weight MSE | | --- | --- | --- | | Search | **-8.00%** | +32.61% | | WebShop | **-7.03%** | +27.36% | | ALFWorld | **-13.06%** | +20.07% | This is the requested allocation pattern: lower error on high-weight turns. The measured parameter-score-weighted squared error also falls by 11.10%/7.77%/10.79%. Together with the matched-compute gains, these diagnostics substantiate SWPO's practical contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.