- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SWPO, a method that weights turn-level critic fitting using the sum of over sampled tokens while retaining the turn-level PPO actor. This inexpensive proxy approximates gradient-energy weighting without vocabulary-wide computation. The paper provides a token-level logit-gradient bound and a conditional analysis of baseline approximation error. Experiments with Qwen3-4B/8B on search and embodied tasks compare PPO and group-based baselines, alongside controlled critic-weighting ablations.
Critical strengths
The controlled ablations keep the actor and critic architecture fixed, providing evidence that the proposed weighting contributes beyond the turn-level PPO backbone. SWPO also achieves competitive performance with summed OPT at substantially lower actor-update cost, supporting its practical value as an inexpensive alternative to vocabulary-wide proxy computation.
Critical weaknesses
-
SWPO uses token probabilities to estimate which turns have larger policy gradients. However, the theoretical bound concerns gradients with respect to logits, not the full model parameters. Please compare the proposed weights with actual turn-level parameter-gradient norms to verify whether higher weights correspond to larger gradients.
-
In the critic-weighting ablations, the variants differ in both relative sample weights and overall loss scale. Please clarify whether the performance differences arise from which turns receive more weight or are also influenced by changes in critic optimization due to the loss scale.
-
Each turn’s critic weight is computed from its token probabilities at generation time and remains fixed during subsequent updates. After policy updates, PPO clipping may suppress that turn’s contribution to the actor gradient, even when its critic weight remains high. Please examine how often this occurs and whether the fixed weights still emphasize turns that influence the actor update.
Questions for authors
Please see weaknesses.
2 comments
Thank you for the additional experiments and clarifications, which address my concerns in W1 and W3. The unit-mean weighting control also helps address W2, although some uncertainty remains about whether loss-scale changes contribute to the gains of raw SWPO. Could you also report a direct comparison between raw and normalized SWPO under the same compute budget and paired seeds? This would help clarify the remaining concern.
Thank you for these questions. We provide full parameter-gradient measurements, a unit-mean weighting control, and actual-update clipping measurements.
1. Full parameter gradients. Let n index turns, Sn be the starting history, An the generated turn, Ln its token count, an,t token t, and Zn,t its preceding context. Let θ denote actor parameters, θ0 the frozen rollout parameters, and πθ the token policy. Define pn,t=πθ0(an,t∣Zn,t), parameter-score energy ρn and proxy Wn:
\qquad W_n=\sum_{t=1}^{L_n}(1-p_{n,t})^2.$$ We use one backward pass per turn through all trained actor parameters, including the cross-token terms in the squared norm. Across 12 frozen checkpoints from seed 42, we sample $J=16$ held-out states and $M=32$ alternatives per state: **6,144 measured turns**. Let $F$ denote the sum of exact token logit-score energies, computed across the vocabulary. At state $s_j$, let $m$ index alternatives and $X_{jm}$ be a candidate weight $X\in\{1,W,F,L,W/L\}$. Let $\hat\mu_{X,j}$ and $\hat\mu_{\rho,j}$ be the respective sample means. Define normalized discrepancy $\hat\eta_X(s_j)$ and its state average $\bar\eta_X$: $$\hat\eta_X(s_j)=\frac1M\sum_{m=1}^{M} \left|\frac{\rho_{jm}}{\hat\mu_{\rho,j}}-\frac{X_{jm}}{\hat\mu_{X,j}}\right|, \qquad \bar\eta_X=\frac1J\sum_{j=1}^{J}\hat\eta_X(s_j).$$ Smaller discrepancy means closer agreement with relative full parameter-gradient energy. Final-checkpoint results are: | Environment | Uniform | $W$ | $F$ | $L$ | $W/L$ | Partial rank correlation of $W,\rho$ | | --- | --- | --- | --- | --- | --- | --- | | Search | 0.717 | **0.185** | 0.186 | 0.386 | 0.397 | 0.732 | | WebShop | 0.781 | **0.174** | 0.171 | 0.436 | 0.417 | 0.752 | | ALFWorld | 0.699 | **0.190** | 0.187 | 0.415 | 0.394 | 0.763 | The correlations control turn length and state. At all 12 checkpoints, $W$ has lower discrepancy than uniform, length-only and mean-token weights; its discrepancy differs from $F$ by at most 0.0025. This directly validates the proxy against parameter-space measurements rather than relying only on the logit-space bound. The connection to the baseline is explicit. For completed return $G\in[0,1]$, let $\mathbb E_s$ denote expectation conditional on turn-start state $s$. For $X\in\{\rho,W\}$ with positive mean, define the population weighted baseline $b_X^*(s)=\mathbb E_s[XG]/\mathbb E_s[X]$. Let $\eta_W(s)$ be the population counterpart of the discrepancy above. Our analysis gives $$\left|b_W^*(s)-b_\rho^*(s)\right|\le\frac12\eta_W(s).$$ Thus the measured normalized-energy agreement probes the quantity governing the proxy-baseline approximation. **2. Relative weighting at controlled mean scale.** We added normalized SWPO while retaining original raw SWPO and uniform Turn PPO. Let $\mathcal B$ be all valid turns in the full rollout batch across data-parallel workers, $\overline W_{\mathcal B}$ their mean raw weight and $\widetilde W_n$ the detached applied weight: $$\overline W_{\mathcal B}=\frac1{|\mathcal B|}\sum_{n\in\mathcal B}W_n, \qquad \widetilde W_n=\frac{W_n}{\overline W_{\mathcal B}}, \qquad \frac1{|\mathcal B|}\sum_{n\in\mathcal B}\widetilde W_n=1.$$ The scalar is fixed during that batch's critic updates; other settings match Turn PPO. Three independent Qwen3-8B training seeds give the matched-compute results below. Search is equal-weight macro Pass@1 over HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle, NQ, TriviaQA and PopQA. Other tasks report single-attempt success; differences are percentage points (pp), with paired 95% Student CIs across seeds. | Environment | GPU-hours | Normalized SWPO - Turn PPO, pp [95% CI] | | --- | --- | --- | | Search | 256 | **2.14 [0.46, 3.82]** | | WebShop | 120 | **2.07 [1.03, 3.10]** | | ALFWorld | 40 | **2.60 [1.11, 4.09]** | Selective weighting therefore retains gains when its mean is fixed to the uniform baseline's unit scale. **3. Fixed weights and clipping.** Each fresh rollout batch receives one full-batch actor optimizer step and one PPO epoch, with no intervening sequential actor updates. Let $r_n(\theta)$ be the complete-turn importance ratio and $\Pi_\theta$ the turn probability. At actual gradient evaluation, $$\theta=\theta_0 \quad\Longrightarrow\quad r_n(\theta)=\frac{\Pi_\theta(A_n\mid S_n)}{\Pi_{\theta_0}(A_n\mid S_n)}=1.$$ This is strictly inside the PPO clipping interval [0.8, 1.2]. Across 48 protocol checks, $\max_n|r_n-1|=8.94\times10^{-7}$; both the suppressed-turn fraction and suppressed critic-weight mass are **zero**. The batch is then discarded, and fresh rollouts receive recomputed weights. Consequently, the suggested fixed-weight/repeated-update clipping mismatch does not occur in our protocol.