- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReSPO (Reshaped Sequence Policy Optimization), a sequence-level policy optimization method designed to address gradient starvation in off-policy reinforcement learning for LLM reasoning. The authors identify two limitations of clipping-based methods such as GRPO and GSPO: (1) positive-advantage trajectories with low importance ratios receive vanishing gradient weights, preventing the model from effectively learning from under-generated successful responses; and (2) negative-advantage trajectories with high importance ratios can dominate policy-gradient updates, potentially destabilizing training.
To address these issues, ReSPO introduces a sign-dependent importance-weight reshaping kernel derived from a weighted α-Bregman divergence minimization followed by a KL projection with moment constraints.
Critical strengths
-
The paper highlights an important limitation of conventional importance-weight clipping in off-policy RLVR. The asymmetric treatment of positive and negative advantages provides an intuitive explanation for why standard clipping may allocate gradient contributions inefficiently, particularly when rollouts are reused across multiple optimization steps.
-
The authors derive the reshaping kernel through a weighted Bregman divergence objective and a subsequent KL projection. The derivation offers a principled interpretation of the proposed kernel as a trade-off between proximity to the behavior and current policies, together with moment control.
-
ReSPO achieves the highest late-stage training score in five of six model–reuse settings and consistently outperforms clipping-based baselines in the reported late-stage training comparisons. The gains are particularly noticeable at a rollout-reuse ratio of (N=32), suggesting that the method can better tolerate substantial off-policy drift.
Critical weaknesses
Major Question
-
The theoretical derivation establishes optimality with respect to a weighted Bregman divergence objective, followed by a KL projection under moment constraints. However, it remains unclear how minimizing these divergences translates into improved expected reward.
-
The exponential tilting parameters are selected through tail requirements and derivative-matching conditions rather than by solving for a prescribed variance budget. This leaves a gap between the theoretical moment-control formulation and the practical algorithm.
-
The main design requirements are characterized by a small set of tail conditions, which could also be satisfied by substantially simpler heuristic weighting functions, such as piecewise clipping strategies or smooth bounded kernels. For instance, and satisfy the desired tail conditions without requiring the proposed variational construction. While I acknowledge that the Bregman divergence formulation provides a principled optimization-based interpretation, the specific hyperparameters are still selected based on manually imposed design choices, such as tail behavior and derivative matching, rather than being derived from an optimality criterion directly related to policy improvement. Therefore, it remains unclear whether the proposed variational formulation offers meaningful advantages over simpler heuristic alternatives.
-
Insufficient analysis of the negative branch. The paper motivates the negative branch by arguing that high-importance-ratio negative trajectories can dominate gradient updates and destabilize optimization. However, the empirical analysis focuses primarily on preserving learning signals for low-ratio positive trajectories, while providing limited evidence for the proposed negative-gradient dominance mechanism. I feel the author can discuss how the method help negative branch in mitigating collapse. For example, [1,2] discussed how the large negative gradient can lead to training collapse. Some discussion or even analysis is helpful.
Minor Question
- Limited consideration of absolute trajectory likelihood. ReSPO determines its gradient scaling coefficient solely from the importance ratio between the current and behavior policies. However, absolute value matters, for example, positive trajectories with large importance ratios but with both rollout likelihood being low could still require a large ratio without decaying. It would be interesting to investigate whether incorporating trajectory rarity or length-normalized likelihood provides complementary information.
[1] On group relative policy optimization collapse in agent search: The lazy likelihood-displacement
[2] On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
Questions for authors
see weakness
4 comments
Minor Q1: Low likelihood and large importance ratios.
We examine positive-advantage responses whose current-policy mean token log likelihood is in the bottom 10% within their source-policy, response-length, and training-stage group, with .
This defines rarity using current-policy likelihood, complementing Appendix E.3's ratio analysis. Since , the behavior-policy probability is even lower for each selected response. For ReSPO's positive branch:
The coefficient is therefore below 27.1% of its value at .
Across 112,640 saved response observations from four 1.7B, diagnostic trajectories over rollouts 2–23, 16,560 have positive advantage. Counts retain repeated observations at different reuse positions. Of the positives, 16,362 are in groups with at least 20 positives and have valid comparison thresholds:
| Source policy | Eligible positive observations | Current likelihood in bottom 10% and | Share of eligible positives |
|---|---|---|---|
| ReSPO | 4,582 | 47 | 1.03% |
| VESPO | 4,293 | 53 | 1.23% |
| GRPO | 3,701 | 29 | 0.78% |
| GSPO | 3,786 | 45 | 1.19% |
This joint case accounts for about 1% of eligible positives. For ReSPO, the 47 observations are 10.2% of its 463 bottom-decile positives. Using full sequence log likelihood with the same grouping gives 50/4,582 (1.09%).
Large-ratio suppression also downweights this small subset of low-likelihood positives in the diagnostic sample. Although the low-likelihood positives with large are very rare, we agree that integrating absolute current-policy likelihood alongside into the weighting rule represents a promising direction for future research.
References
[1] Metelli et al. (2018). Policy Optimization via Importance Sampling. NeurIPS.
[2] Agapiou et al. (2017). Importance Sampling: Intrinsic Dimension and Computational Cost. Statistical Science, 32(3), 405–431.
[3] Deng et al. (2025). On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization. NeurIPS.
[4] Deng et al. (2026). On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacement. ICML.
Thank you for your response, most of my concern are addressed, I maintain my positive score
Thank you for your constructive feedback. We are glad that the revisions have addressed your concerns.
Minimizing Fβ selects the base measure and kernel through a reward-gradient MSE upper bound. Under Appendix A.7's assumptions, reducing this bound raises a lower bound on expected one-step reward improvement. This gives a reward-based rationale for choosing the base measure τ0∗ and hence ϕ0=τ0∗/μ. Common response C2 gives the equations, the positivity condition, and the role of KL projection.
Controlling the second moment of change-of-measure weights is standard in importance sampling and off-policy optimization [1, 2]. We require finite projection budgets C1,C2 and obtain their values after choosing the kernel shape. So, we first derive the kernel shape based on the finite constraints on moments and mass, then determine the kernel based on analytical heuristics (Sec. 3.3), and then we can calculate the variance upper bound under this choice. We give details below.
The KL projection gives the exponential-tilt family. For a base kernel ϕ0, rescaling to ϕ(1)=1 gives: ϕλ(W)=ϕ0(W)exp!(λ(1−ϕ0(W))). Tail behavior and local branch matching determine the specific shape. Tail requirements select the branch families; we use symmetric interpolation β(±)=1/2, set λ(+)=2 for a flat positive-branch slope at W=0, and choose λ(−)=2 to match both branch slopes at W=1 (Sec. 3.3; Appendix A.3).
Once the shape is fixed, we recover finite budgets from the kernel before rescaling. For either branch, take ϕ∗(W)=ϕ0(W)e−2ϕ0(W). Under the behavior policy μ, set:
\qquad C_2=\mathbb E_\mu[\phi^*]\leq\frac{1}{2e}.$$ Here the budgets equal the attained moments and depend on the policy pair and branch; $e^{-2}$ and $1/(2e)$ are uniform upper bounds, following from $\phi_0\ge0$. With these moment values as budgets, $\phi^*$ satisfies the KL-projection conditions with product-moment multiplier $\lambda=2$ and mass multiplier $\gamma=0$. **We rescale by $c=e^2$ to enforce $\phi(1)=1$.** The implemented kernel is $\phi=e^2\phi^*$, so its product moment and mass are: $$\mathbb E_\mu[\phi\phi_0]=e^2C_1\le1, \qquad \mathbb E_\mu[\phi]=e^2C_2\le\frac e2.$$ The second moment scales by $e^4$: $\mathbb E_\mu[\phi^2]=e^4\mathbb E_\mu[(\phi^*)^2]$. Since $0\le\phi\le e/2$, $\mathbb E_\mu[\phi^2]\le e^2/4$. Thus **$1$, $e/2$, and $e^2/4$ are uniform upper bounds on the rescaled proxy moment, mass, and second moment**, respectively. Appendix A.7 uses these rescaled moments and KL divergence in the final gradient-error bound (Common response **C2**). > ### Major Q3: Rational alternatives. We implement your proposed kernel. On Qwen3-1.7B at $N=32$, ReSPO exceeds the proposed rational pair in early peak (**15.7% vs. 12.3%**), late mean (**27.6% vs. 22.7%**), and final training accuracy (**26.7% vs. 22.8%**). After 1,024 updates, its six-benchmark pass@1 is **34.5% vs. 32.3%**, with higher scores on five of six benchmarks. **This comparison supports the importance of kernel shape beyond its qualitative tail constraints.** Common response **C2** gives the base-kernel gradient-error criterion; **JsYL W2** reports the exact kernels and full results. > ### Major Q4: Negative-branch analysis. We directly test ReSPO's negative tail by removing its tilt, giving $\phi^{(-)}(W)=\sqrt W$ with the positive branch fixed (**Common response C4; Appendix E.6, Figure 6**). Across 40 sampled mini-batches in the first eight rollouts of Qwen3-1.7B, $N=32$, **9/2,822 negative responses (0.32%)** have $\log W\gt 5$ but contribute **21.7%** of negative loss-derivative mass. At rollout 2, reuse 16, just **3/57** negatives contribute **76.2%**. Here mass is the sum of absolute loss derivatives with respect to token log probabilities. Applying the original tilt to the same policy and responses reduces this share below $10^{-7}\\%$. The actual training curves reach **13.6% with tilt versus 11.6% without** at rollout 8, with similar mean lengths (**1,129 versus 1,136 tokens**). **The negative tilt suppresses the concentration of negative loss derivatives on rare high-ratio responses.** This supports our choice $\phi^{(-)}(\infty)=0$, so that extremely high-ratio negative responses receive vanishing weight. This sequence-level tail test complements the token-level analyses in NTHR [3] and LLDS [4]; Common response **C4** gives the complete definition and details.