acceptodds
Under review as a conference paper at ICLR 2027

ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning

Abstract

Reinforcement learning from verifiable rewards (RLVR) frequently reuses rollouts across multiple policy updates, increasing the mismatch between the current policy and the data-generating policy. We identify a sign-dependent gradient starvation problem in clipped policy optimization: clipping suppresses under-generated positive responses at the low-importance-weight tail while permitting severely over-generated negative responses to dominate the high-weight tail. To address this, we propose **ReSPO** (Reshaped Sequence Policy Optimization), which replaces clipping with a smooth, two-branch sequence-level kernel derived from an -divergence variational objective and an exponential variance-control tilt. The positive branch preserves a nonzero gradient weight for under-generated positive responses, while the negative branch suppresses heavily over-generated negative responses. We demonstrate that ReSPO effectively learns from long positive reasoning trajectories during early training, even when accumulated policy drift relegates them to the low-importance-weight tail. On dense and MoE Qwen3 models, ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under a rollout reuse, validating our approach on importance-weight tail control in off-policy learning.

Primary area:
reinforcement learning
Keywords:
off-policy learning, MoE, weight reshaping

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Pi56OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReSPO (Reshaped Sequence Policy Optimization), a sequence-level policy optimization method designed to address gradient starvation in off-policy reinforcement learning for LLM reasoning. The authors identify two limitations of clipping-based methods such as GRPO and GSPO: (1) positive-advantage trajectories with low importance ratios receive vanishing gradient weights, preventing the model from effectively learning from under-generated successful responses; and (2) negative-advantage trajectories with high importance ratios can dominate policy-gradient updates, potentially destabilizing training.

To address these issues, ReSPO introduces a sign-dependent importance-weight reshaping kernel derived from a weighted α-Bregman divergence minimization followed by a KL projection with moment constraints.

Critical strengths
  1. The paper highlights an important limitation of conventional importance-weight clipping in off-policy RLVR. The asymmetric treatment of positive and negative advantages provides an intuitive explanation for why standard clipping may allocate gradient contributions inefficiently, particularly when rollouts are reused across multiple optimization steps.

  2. The authors derive the reshaping kernel through a weighted Bregman divergence objective and a subsequent KL projection. The derivation offers a principled interpretation of the proposed kernel as a trade-off between proximity to the behavior and current policies, together with moment control.

  3. ReSPO achieves the highest late-stage training score in five of six model–reuse settings and consistently outperforms clipping-based baselines in the reported late-stage training comparisons. The gains are particularly noticeable at a rollout-reuse ratio of (N=32), suggesting that the method can better tolerate substantial off-policy drift.

Critical weaknesses

Major Question

  1. The theoretical derivation establishes optimality with respect to a weighted Bregman divergence objective, followed by a KL projection under moment constraints. However, it remains unclear how minimizing these divergences translates into improved expected reward.

  2. The exponential tilting parameters are selected through tail requirements and derivative-matching conditions rather than by solving for a prescribed variance budget. This leaves a gap between the theoretical moment-control formulation and the practical algorithm.

  3. The main design requirements are characterized by a small set of tail conditions, which could also be satisfied by substantially simpler heuristic weighting functions, such as piecewise clipping strategies or smooth bounded kernels. For instance, ϕ+(W)=2/(1+W)\phi^+(W)=2/(1+W) and ϕ−(W)=4W/(1+W)2\phi^-(W)=4W/(1+W)^2 satisfy the desired tail conditions without requiring the proposed variational construction. While I acknowledge that the Bregman divergence formulation provides a principled optimization-based interpretation, the specific hyperparameters are still selected based on manually imposed design choices, such as tail behavior and derivative matching, rather than being derived from an optimality criterion directly related to policy improvement. Therefore, it remains unclear whether the proposed variational formulation offers meaningful advantages over simpler heuristic alternatives.

  4. Insufficient analysis of the negative branch. The paper motivates the negative branch by arguing that high-importance-ratio negative trajectories can dominate gradient updates and destabilize optimization. However, the empirical analysis focuses primarily on preserving learning signals for low-ratio positive trajectories, while providing limited evidence for the proposed negative-gradient dominance mechanism. I feel the author can discuss how the method help negative branch in mitigating collapse. For example, [1,2] discussed how the large negative gradient can lead to training collapse. Some discussion or even analysis is helpful.

Minor Question

  1. Limited consideration of absolute trajectory likelihood. ReSPO determines its gradient scaling coefficient solely from the importance ratio between the current and behavior policies. However, absolute value matters, for example, positive trajectories with large importance ratios but with both rollout likelihood being low could still require a large ratio without decaying. It would be interesting to investigate whether incorporating trajectory rarity or length-normalized likelihood provides complementary information.

[1] On group relative policy optimization collapse in agent search: The lazy likelihood-displacement

[2] On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization

Questions for authors

see weakness

4 comments

Major Q1: Divergence minimization and reward.

Minimizing Fβ\mathcal F_\beta selects the base measure and kernel through a reward-gradient MSE upper bound. Under Appendix A.7's assumptions, reducing this bound raises a lower bound on expected one-step reward improvement. This gives a reward-based rationale for choosing the base measure τ0∗\tau_0^* and hence ϕ0=τ0∗/μ\phi_0=\tau_0^*/\mu. Common response C2 gives the equations, the positivity condition, and the role of KL projection.

Major Q2: Prescribed variance budgets.

Controlling the second moment of change-of-measure weights is standard in importance sampling and off-policy optimization [1, 2]. We require finite projection budgets C1,C2C_1,C_2 and obtain their values after choosing the kernel shape. So, we first derive the kernel shape based on the finite constraints on moments and mass, then determine the kernel based on analytical heuristics (Sec. 3.3), and then we can calculate the variance upper bound under this choice. We give details below.

The KL projection gives the exponential-tilt family. For a base kernel ϕ0\phi_0, rescaling to ϕ(1)=1\phi(1)=1 gives: ϕλ(W)=ϕ0(W)exp⁡!(λ(1−ϕ0(W))).\phi_\lambda(W)=\phi_0(W)\exp\\!\bigl(\lambda(1-\phi_0(W))\bigr). Tail behavior and local branch matching determine the specific shape. Tail requirements select the branch families; we use symmetric interpolation β(±)=1/2\beta^{(\pm)}=1/2, set λ(+)=2\lambda^{(+)}=2 for a flat positive-branch slope at W=0W=0, and choose λ(−)=2\lambda^{(-)}=2 to match both branch slopes at W=1W=1 (Sec. 3.3; Appendix A.3).

Once the shape is fixed, we recover finite budgets from the kernel before rescaling. For either branch, take ϕ∗(W)=ϕ0(W)e−2ϕ0(W)\phi^*(W)=\phi_0(W)e^{-2\phi_0(W)}. Under the behavior policy μ\mu, set:

\qquad C_2=\mathbb E_\mu[\phi^*]\leq\frac{1}{2e}.$$ Here the budgets equal the attained moments and depend on the policy pair and branch; $e^{-2}$ and $1/(2e)$ are uniform upper bounds, following from $\phi_0\ge0$. With these moment values as budgets, $\phi^*$ satisfies the KL-projection conditions with product-moment multiplier $\lambda=2$ and mass multiplier $\gamma=0$. **We rescale by $c=e^2$ to enforce $\phi(1)=1$.** The implemented kernel is $\phi=e^2\phi^*$, so its product moment and mass are: $$\mathbb E_\mu[\phi\phi_0]=e^2C_1\le1, \qquad \mathbb E_\mu[\phi]=e^2C_2\le\frac e2.$$ The second moment scales by $e^4$: $\mathbb E_\mu[\phi^2]=e^4\mathbb E_\mu[(\phi^*)^2]$. Since $0\le\phi\le e/2$, $\mathbb E_\mu[\phi^2]\le e^2/4$. Thus **$1$, $e/2$, and $e^2/4$ are uniform upper bounds on the rescaled proxy moment, mass, and second moment**, respectively. Appendix A.7 uses these rescaled moments and KL divergence in the final gradient-error bound (Common response **C2**). > ### Major Q3: Rational alternatives. We implement your proposed kernel. On Qwen3-1.7B at $N=32$, ReSPO exceeds the proposed rational pair in early peak (**15.7% vs. 12.3%**), late mean (**27.6% vs. 22.7%**), and final training accuracy (**26.7% vs. 22.8%**). After 1,024 updates, its six-benchmark pass@1 is **34.5% vs. 32.3%**, with higher scores on five of six benchmarks. **This comparison supports the importance of kernel shape beyond its qualitative tail constraints.** Common response **C2** gives the base-kernel gradient-error criterion; **JsYL W2** reports the exact kernels and full results. > ### Major Q4: Negative-branch analysis. We directly test ReSPO's negative tail by removing its tilt, giving $\phi^{(-)}(W)=\sqrt W$ with the positive branch fixed (**Common response C4; Appendix E.6, Figure 6**). Across 40 sampled mini-batches in the first eight rollouts of Qwen3-1.7B, $N=32$, **9/2,822 negative responses (0.32%)** have $\log W\gt 5$ but contribute **21.7%** of negative loss-derivative mass. At rollout 2, reuse 16, just **3/57** negatives contribute **76.2%**. Here mass is the sum of absolute loss derivatives with respect to token log probabilities. Applying the original tilt to the same policy and responses reduces this share below $10^{-7}\\%$. The actual training curves reach **13.6% with tilt versus 11.6% without** at rollout 8, with similar mean lengths (**1,129 versus 1,136 tokens**). **The negative tilt suppresses the concentration of negative loss derivatives on rare high-ratio responses.** This supports our choice $\phi^{(-)}(\infty)=0$, so that extremely high-ratio negative responses receive vanishing weight. This sequence-level tail test complements the token-level analyses in NTHR [3] and LLDS [4]; Common response **C4** gives the complete definition and details.

Minor Q1: Low likelihood and large importance ratios.

We examine positive-advantage responses whose current-policy mean token log likelihood is in the bottom 10% within their source-policy, response-length, and training-stage group, with W=πθ(o)/μ(o)>3W=\pi_\theta(o)/\mu(o)\gt 3.

This defines rarity using current-policy likelihood, complementing Appendix E.3's ratio analysis. Since W>3W\gt 3, the behavior-policy probability is even lower for each selected response. For ReSPO's positive branch:

W>3⟹ϕ(+)(W)=1+W2e1−W<2e−2≈0.271W\gt 3\quad\Longrightarrow\quad \phi^{(+)}(W)=\frac{1+W}{2}e^{1-W}\lt 2e^{-2}\approx0.271

The coefficient is therefore below 27.1% of its value at W=1W=1.

Across 112,640 saved response observations from four 1.7B, N=32N=32 diagnostic trajectories over rollouts 2–23, 16,560 have positive advantage. Counts retain repeated observations at different reuse positions. Of the positives, 16,362 are in groups with at least 20 positives and have valid comparison thresholds:

Source policyEligible positive observationsCurrent likelihood in bottom 10% and W>3W\gt 3Share of eligible positives
ReSPO4,582471.03%
VESPO4,293531.23%
GRPO3,701290.78%
GSPO3,786451.19%

This joint case accounts for about 1% of eligible positives. For ReSPO, the 47 observations are 10.2% of its 463 bottom-decile positives. Using full sequence log likelihood with the same grouping gives 50/4,582 (1.09%).

Large-ratio suppression also downweights this small subset of low-likelihood positives in the diagnostic sample. Although the low-likelihood positives with large WW are very rare, we agree that integrating absolute current-policy likelihood πθ(o)\pi_\theta(o) alongside WW into the weighting rule represents a promising direction for future research.


References

[1] Metelli et al. (2018). Policy Optimization via Importance Sampling. NeurIPS.
[2] Agapiou et al. (2017). Importance Sampling: Intrinsic Dimension and Computational Cost. Statistical Science, 32(3), 405–431.
[3] Deng et al. (2025). On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization. NeurIPS.
[4] Deng et al. (2026). On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacement. ICML.

Thank you for your response, most of my concern are addressed, I maintain my positive score

Thank you for your constructive feedback. We are glad that the revisions have addressed your concerns.

Reviewer_JsYLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how importance weighting affects RLVR training when rollout data become stale. The main concern is that positive responses with small importance ratios receive little learning signal, while negative responses with large ratios can dominate the update. ReSPO replaces clipping with separate smooth weighting functions for positive and negative advantages, derived using an α-divergence objective and exponential tilting. Experiments with dense and MoE Qwen3 models show improvements at higher staleness, alongside analyses of training dynamics, response length, and weight allocation.

Critical strengths
  1. The method is clearly explained. The paper connects the desired tail behavior to the two branches and gives explicit reasons for the parameter choices. The gradient formulation and log-space implementation also make the method relatively easy to follow and implement (pp. 5–6 and Appendix B).

  2. The experiments cover dense and MoE models, three staleness settings, and relevant baselines including GRPO, GSPO, and VESPO. ReSPO obtains the highest observed final-evaluation macro-average in all four model–staleness combinations at N = 16 and 32 (Table 1). These results provide useful evidence that the method helps in the higher-staleness settings studied here.

  3. The paper includes useful analyses beyond the final benchmark scores. The training curves, comparisons within shared response-length ranges, and measurements of weight allocation help readers understand how training changes under ReSPO. The appendix also documents the algorithm and training configurations in detail.

Critical weaknesses
  1. My main concern is that the experiments do not separate the effects of the positive and negative branches. Both are changed relative to VESPO, but Section 4.3 largely explains the gains through retaining low-weight positive responses and learning earlier from long positive trajectories. Tables 5–7 provide useful evidence, although the accompanying changes in weights, lengths, and scores do not tell us how much the negative branch contributes or whether the two branches need to work together. The negative-tail treatment also receives less empirical analysis. In one representative setting, replacing each branch separately while keeping the other from VESPO would help test the proposed explanation.

  2. The paper could be clearer about what its specific kernel design adds. Variational sequence reweighting is closely related to VESPO, and preserving learning signals from low-ratio positive responses has also been studied in TOPR (Le Roux et al., NeurIPS 2025). The derivation explains how ReSPO satisfies the chosen tail and local-shape requirements. What remains unclear empirically is whether the gains depend on these particular curves or could also be obtained with a simpler weighting rule that has similar tail behavior. Such a comparison would help establish the value of the new construction. The discussion of prior work should also make clear which ideas are shared and which parts are new.

  3. As a smaller point, the explanation for GRPO/GSPO needs more direct evidence. GRPO uses token-level ratios, and GSPO uses a length-normalized sequence ratio, whereas Appendix E.3 focuses on the raw sequence ratio W. A small W can still correspond to a length-normalized ratio close to 1, so it is not clear that the low-W responses identified in the appendix also receive very small weights in GSPO. Examining the effective weights in these baselines would help establish how serious the proposed imbalance is in practice. If the current diagnostics mainly explain the difference from VESPO, I suggest limiting this part of the claim to that comparison.

  4. The lines in Figure 2 are too thick, which makes the curves hard to distinguish and the details difficult to read. Thinner lines (or distinct markers/line styles) would improve readability.

Questions for authors

The authors are encouraged to address the weakness above.

4 comments

W1: Positive and negative contributions.

Let R/VR/V denote ReSPO/VESPO [1] and +/−+/- their advantage branches. On 30B at N=32N=32, using R+R^+ with V−V^- raises observed six-benchmark pass@1 from 49.3% to 50.9%. Using R−R^- with V+V^+ gives an unstable run and 37.6% pass@1. With R+R^+ fixed, using R−R^- instead of V−V^- raises late penalized training accuracy by 6.3 pp and held-out mean by 0.3 pp (Tables 9–10; Appendix E.5). The benefit of R−R^- depends on its positive partner and is clearest in late training reward (Common response C3).

Common response C4 and Appendix E.6 isolate the negative tilt. With R+R^+ fixed and ϕ(−)(W)=W\phi^{(-)}(W)=\sqrt W (no tilt), only 0.32% of sampled negative responses have log⁡W>5\log W\gt 5, but they carry 21.7% of negative loss-derivative mass over the first eight rollouts. Restoring the tilt on the same batches suppresses this concentration, supporting our choice ϕ(−)(∞)=0\phi^{(-)}(\infty)=0 to downweight extremely high-ratio negative responses.

W2: Novelty and simpler curves.

TOPR [2] and REAL [3] also preserve ϕ(+)(0)>0\phi^{(+)}(0)\gt 0, but retain ϕ(−)(∞)>0\phi^{(-)}(\infty)\gt 0; ReSPO suppresses both negative tails (Table 11; Common response C1). Common response C2 explains how minimizing the base kernel's gradient-error upper bound raises the corresponding expected-reward improvement lower bound under Appendix A.7's assumptions.

Comparison with a simpler kernel. We tested Pi56's rational pair, which satisfies ReSPO's four qualitative tail constraints and equals one at W=1W=1:

h+(W)=21+W,h−(W)=4W(1+W)2h_+(W)=\frac{2}{1+W},\qquad h_-(W)=\frac{4W}{(1+W)^2}

Using the Qwen3-1.7B N=32N=32 configuration from Sec. 4.1 with group-standardized advantages, detached coefficients, and token-mean aggregation, we evaluated this pair after 1,024 policy updates:

BenchmarkRational pairReSPO
AIME256.5%8.8%
AIME248.3%11.7%
AMC2344.2%48.8%
OlympiadBench33.5%36.7%
MinervaMath30.3%28.0%
MATH50070.7%72.9%
Macro average32.3%34.5%

Both use the Table 1 evaluation protocol, and ReSPO's entries match Table 1. ReSPO has a 2.2 pp higher macro pass@1 and higher scores on five of six benchmarks. Training results are:

Training accuracyRational pairReSPO
Early peak (through 256 updates)12.3%15.7%
Late mean (896–1,024 updates)22.7±0.422.7_{\pm0.4}\\%27.6±0.4\mathbf{27.6}_{\pm0.4}\\%
Final (1,024 updates)22.8%26.7%

The late mean is weighted by policy steps; subscripts are temporal standard deviations. Kernel shape matters beyond matching the tails: ReSPO has a higher early peak, late mean, final training accuracy, and held-out mean in this comparison. Its base kernel also has the gradient-error rationale in C2, complementing the tail comparison in C1.

W3: Low-ratio positive responses and the sequence-level objective.

Our low-WW gradient-starvation analysis primarily explains the difference between ReSPO and VESPO [1]: ReSPO preserves low-WW positive signals, while VESPO reduces them. A small raw WW alone does not imply small GRPO/GSPO weights. We admit it and limit the current diagnostics in the difference from VESPO.

GRPO [4] and GSPO [5] also aim to maximize sequence-level reward, and they introduce additional bias. Their surrogate objectives use token ratios and the length-normalized ratio W1/TW^{1/T}, respectively, where TT is response length. Common response C2 explains these importance corrections, and Appendix A.6 derives GSPO's length-dependent bias.

W4: Figure 2 readability.

We update Figure 2 to use visible dashes; method colors and line styles help distinguish overlapping curves.


References

[1] Shen et al. (2026). VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training. arXiv:2602.10693.
[2] Le Roux et al. (2025). Tapered Off-Policy REINFORCE: Stable and Efficient Reinforcement Learning for Large Language Models. NeurIPS.
[3] Zhai et al. (2026). Rewards as Labels: Revisiting RLVR from a Classification Perspective. arXiv:2602.05630.
[4] Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
[5] Zheng et al. (2025). Group Sequence Policy Optimization. arXiv:2507.18071.

Dear JsYL,

Thank you again for your time and constructive feedback. We have carefully addressed your concerns point by point in the rebuttal and revised manuscript. As the discussion period comes to a close, we would appreciate your thoughts on these responses and welcome any remaining questions or points that need further clarification.

Thanks for your response. I have raised my score.

Thank you for your constructive feedback. We appreciate your positive response to our rebuttal.

Reviewer_igAVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReSPO to address two issues in off-policy RLVR: weak learning signals for positive-advantage samples with low importance ratios, and dominant gradients from negative-advantage samples with high ratios. Using a variational formulation and exponential tilting, it constructs separate sequence-level weighting kernels for positive and negative advantages. Experiments on Qwen3-1.7B and Qwen3-30B-A3B cover different levels of policy staleness and six math benchmarks, reporting training results and some generalization gains.

Critical strengths

Clear motivation and technical details. The positive branch retains a nonzero weight as (W→0)(W \to 0), while the negative branch downweights extreme ratios at both ends. The paper clearly explains its differences from VESPO and provides tail analysis and a stable implementation, making the technical contribution well specified.

Critical weaknesses
  1. The discussion of related work is not thorough enough. Asymmetric updates for positive and negative samples are already an established line of work. REAL identifies two similar types of gradient mismatch, SAPO uses asymmetric smooth gating, and BAPO introduces adaptive clipping to address imbalanced positive and negative contributions in off-policy learning. Although ReSPO builds directly on VESPO, comparisons with these methods in both performance and mechanism remain important for establishing its effectiveness.

  2. The core design choices lack ablation support. Compared with VESPO, ReSPO changes both branches and the exponential tilting form, but does not include ablations that mix and match the original and modified branches. This makes it hard to tell whether both branch changes actually help.

Questions for authors

Why does ReSPO underperform VESPO at (N=8)? Could you provide further analysis to explain this gap?

5 comments

W1: REAL, SAPO, and BAPO comparisons.

On Qwen3-30B-A3B at N=32N=32, ReSPO has the highest late penalized training accuracy among REAL [1], SAPO [2], BAPO [3], and TOPR [4] (58.8% vs. 49.6% for BAPO, the closest baseline), and the highest observed six-benchmark pass@1 (51.2% vs. 49.4% for REAL; Tables 12–13). ReSPO shares TOPR/REAL's nonzero low-ratio positive limit, but unlike TOPR/REAL, it suppresses the high-ratio negative tail, and has fewer response-cap hits (Sec. 4.3; Appendix H). Common response C1 gives the full comparison.

Training dynamics. REAL's pre-clip gradient norm rises from 0.75 to 16.54 between updates 512 and 1,024 as mean response length grows from 2,108 to 3,460 tokens. SAPO's norm reaches 3.20 at update 800, followed by 7.48% penalized training accuracy and 259-token responses at update 832. Training reward decreases accompany over-lengthening in REAL and shortening in SAPO (Figure 10; Appendix H.2). These results are consistent with ReSPO's moderate response-length growth supporting long reasoning while limiting length penalties.

W2: Mixing branches and changing the tilt.

Let R/VR/V denote ReSPO/VESPO [5] and +/−+/- their advantage branches. R+R−R^+R^- leads other three. At fixed R−R^-, using R+R^+ instead of V+V^+ raises observed six-benchmark pass@1 by 13.6 pp. At fixed R+R^+, using R−R^- instead of V−V^- raises late penalized training accuracy by 6.3 pp and held-out mean by 0.3 pp (Tables 9–10; Appendix E.5). The positive-branch swap shows the larger held-out performance difference. Our negative-branch design follows tail requirements and scale matching with the positive branch. Common response C3 reports all four pairs and their scale matching.

In Common response C4, we remove the exponential tilt on the negative branch. Only 9/2,822 negative-response observations (0.32%) have log⁡W>5\log W>5, yet they carry 21.7% of the pooled negative loss-derivative mass. The negative tilt therefore suppresses derivative concentration on rare high-ratio responses. At rollout 8, training accuracy is 13.6% with tilt versus 11.6% without.

Q1: Performance at N=8N=8.

Compared with VESPO, ReSPO has higher early peaks and late training means on both models; the held-out differences are small. Figure 2 and Table 1 report the following (ReSPO / VESPO):

ModelEarly training peakLast-128 training meanFinal held-out pass@1
Qwen3-1.7B19.2% / 16.1%28.6% / 27.5%34.8% ± 1.6 pp / 35.3% ± 1.5 pp
Qwen3-30B-A3B46.8% / 41.4%59.5% / 53.8%49.5% ± 1.9 pp / 49.2% ± 1.8 pp

The 30B training score includes the length penalty; held-out error bars are bootstrap standard errors. ReSPO is 0.5 pp lower on 1.7B and 0.3 pp higher on 30B, both small relative to the evaluation uncertainty.

Logged 1.7B histories show a smaller average drift proxy −log⁡W/T-\log W/T at N=8N=8 over updates 256–1,024: 0.89×10−40.89\times10^{-4} versus 4.44×10−44.44\times10^{-4} at N=32N=32 for ReSPO, and 0.66×10−40.66\times10^{-4} versus 3.29×10−43.29\times10^{-4} for VESPO. Here W=πθ(o)/μ(o)W=\pi_\theta(o)/\mu(o) and TT is response length. Smaller drift is consistent with less benefit from low-ratio correction at N=8N=8. The larger drifts and held-out gains happen at N=16,32N=16,32, especially on the studied MoE model.


References

[1] Zhai et al. (2026). Rewards as Labels: Revisiting RLVR from a Classification Perspective. arXiv:2602.05630.
[2] Gao et al. (2025). Soft Adaptive Policy Optimization. arXiv:2511.20347.
[3] Xi et al. (2025). BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping. arXiv:2510.18927.
[4] Le Roux et al. (2025). Tapered Off-Policy REINFORCE: Stable and Efficient Reinforcement Learning for Large Language Models. NeurIPS.
[5] Shen et al. (2026). VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training. arXiv:2602.10693.

Dear Reviewer igAV

Thank you again for your time and constructive feedback. We have carefully addressed your concerns point by point in the rebuttal and revised manuscript. As the discussion period comes to a close, we would appreciate your thoughts on these responses and welcome any remaining questions or points that need further clarification.

AuthorsOpenReview ↗

Brief Summary of Rebuttal and Additional Experiments

Dear Reviewer igAV,

Thank you again for your constructive feedback and time in reviewing our work. Following up on our earlier detailed response, we would like to briefly summarize the key additions we made to address your points. All tables and figures below are in the updated paper. C1, C3, C4 are the common responses.

We have shown ReSPO’s distinct tail behaviors among the compared baselines: ϕ(+)(0)>0\phi^{(+)}(0)>0, while the other three tails vanish (Table 11, Figure 9, and C1). We have completed comparisons with all requested baselines on Qwen3-30B-A3B at N=32N=32, where ReSPO achieves the highest training score (Table 12 and Figure 10) and six-benchmark mean pass@1 (Table 13 and C1). Response-length analyses show moderate length growth and fewer response-cap hits than REAL/TOPR, showing that ReSPO can learn longer correct reasoning trajectories while avoiding generating too long responses (Sec. 4.3 and C1).

The branch-swapping ablations support the contribution of ReSPO’s positive branch, with the full ReSPO pair attaining the highest training score and held-out mean (C3). Removing the negative tilt, giving ϕ(−)(W)=W\phi^{(-)}(W)=\sqrt W, produces concentrated loss derivatives on rare high-ratio negative responses and lower training accuracy (C4). Overall, these results support ReSPO’s kernel design and empirical effectiveness. Finally, ReSPO's observed held-out gains are generally larger at N=16,32N=16,32 than at N=8N=8, consistent with the fact that the policy drift will be larger at larger rollout reuse NN.

Thank you for suggesting these ablations, which further highlight the advantages of our method. As the discussion period is drawing to a close, we would greatly appreciate your thoughts on these responses. If your concerns have been addressed, we would be grateful if you could raise your score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common response

We thank all reviewers for their thoughtful feedback and recognition of the clear motivation and two-branch formulation (igAV, JsYL), the Bregman/KL interpretation (Pi56), and the empirical results and diagnostics on dense and MoE models (JsYL, Pi56).

We have added baseline and branch-ablation results, method comparisons, and a theoretical link between gradient-error bounds and expected reward improvement. Below, we summarize these updates and address the shared concerns and individual comments.


Core Contributions of Our Work

  1. Sign-dependent sequence reweighting: ReSPO preserves low-ratio positive-advantage signals and suppresses both negative-advantage tails to control gradient allocation under rollout reuse.
  2. Objective-based kernel construction: Starting from sequence-level expected reward, we select a base measure through Bregman interpolation and reshape it through KL projection. Appendix A.7 links base selection to a smaller gradient-error upper bound and a larger expected-reward improvement lower bound under its stated assumptions.
  3. Gains at larger rollout reuse: ReSPO has larger observed gains at N=16,32N=16,32 than at N=8N=8, especially on the studied MoE model. This is consistent with greater benefits under larger policy drift.

Updates of Experimental Results during Rebuttal

  • Sec. 4.3 and Appendix H.2: Added TOPR, REAL, SAPO, and BAPO comparisons on Qwen3-30B-A3B at N=32N=32, including training rewards, six-benchmark evaluations, response lengths, and gradient norms (igAV, JsYL).
  • Appendix E.5: Added training and held-out comparisons of all four ReSPO/VESPO branch combinations to examine each branch and their interaction (igAV, JsYL, Pi56).
  • Appendix E.6: Added a negative-tilt ablation with high-ratio negative-response diagnostics and training curves over the first eight rollouts (JsYL W1; Pi56 Major Q4).
  • Individual responses: Added rational-kernel comparisons and low-likelihood response counts (JsYL W2; Pi56 Major Q3 and Minor Q1).

Updates of In-depth Discussions during Rebuttal

  • Appendix A.7 and C2: Connected base-kernel selection, gradient-error bounds, and expected reward improvement under stated assumptions (JsYL, Pi56).
  • Sec. 4.3 and Appendices A.6/H.1: Clarified importance ratios, branch coefficients, tail behavior, and response-length growth in relation to learning and length penalties (igAV, JsYL, Pi56).
  • igAV Q1, referring to Figure 2 and Table 1: Clarified the N=8N=8 training and held-out comparisons considering reported evaluation uncertainty (igAV).

Below, we address the shared concerns in detail.

C1. Contribution and related methods

Shared concern: igAV W1; JsYL W2; Pi56 Major Q3. How does ReSPO differ from existing asymmetric methods and simpler kernels?

ReSPO combines a nonzero low-WW positive coefficient with vanishing other tails. TOPR and REAL also preserve low-ratio positive signals, but retain a nonzero high-ratio negative coefficient. VESPO suppresses both low- and high-WW tails on both branches.

Write wt=πθ(yt∣ht)/μ(yt∣ht)w_t=\pi_\theta(y_t\mid h_t)/\mu(y_t\mid h_t), W=∏twtW=\prod_t w_t, and s=W1/Ts=W^{1/T}, where μ\mu is the behavior policy, hth_t is the prompt and preceding tokens, and TT is response length. The table compares positive/negative token score-gradient coefficient magnitudes ϕ(+),ϕ(−)\phi^{(+)},\phi^{(-)}, factoring out reward/advantage magnitudes and ratio-independent scales. Each row uses its own input xx; tail pairs are ordered (ϕ(+),ϕ(−))(\phi^{(+)},\phi^{(-)}). Appendix H.1 gives the derivations and reductions.

MethodInput xxPositive ϕ(+)(x)\phi^{(+)}(x)Negative ϕ(−)(x)\phi^{(-)}(x)x→0x\to0x→∞x\to\infty
ReSPOWW1+x2e1−x\frac{1+x}{2}e^{1-x}x,e2(1−x)\sqrt{x}\\,e^{2(1-\sqrt{x})}(e/2,0)(e/2,0)(0,0)(0,0)
VESPO [1]WWx2e3(1−x)x^2e^{3(1-x)}x3e2(1−x)x^3e^{2(1-x)}(0,0)(0,0)(0,0)(0,0)
TOPR [2]WW11min⁡(x,1)\min(x,1)(1,0)(1,0)(1,1)(1,1)
GRPO [3] (dual clip)wtw_tx1x<ux\mathbf1_{x\lt u}x1ℓ<x<dx\mathbf1_{\ell\lt x\lt d}(0,0)(0,0)(0,0)(0,0)
GSPO [4]ssx1x<ux\mathbf1_{x\lt u}x1x>ℓx\mathbf1_{x\gt \ell}(0,0)(0,0)(0,∞)(0,\infty)
SAPO [5]wtw_t4xp+(x)(1−p+(x))4xp_+(x)(1-p_+(x))4xp−(x)(1−p−(x))4xp_-(x)(1-p_-(x))(0,0)(0,0)(0,0)(0,0)
BAPO [6] (dual clip)wtw_tx1x<ux\mathbf1_{x\lt u}x1ℓ<x<dx\mathbf1_{\ell\lt x\lt d}(0,0)(0,0)(0,0)(0,0)
REAL [7]ssui+/D+u_i^+/D_+ui−/D−u_i^-/D_-(1,0)(1,0)(0,1)(0,1)

Here ℓ,u\ell,u are the lower/upper ratio bounds: fixed for GRPO/GSPO and batch-adaptive for BAPO. The tested GRPO/BAPO variants use negative dual-clip cap d=3d=3; without this cap, their negative high-ratio limit is ∞\infty. For SAPO, p±(x)=σ(κ±(x−1))p_\pm(x)=\sigma(\kappa_\pm(x-1)), with σ\sigma the logistic sigmoid and κ±>0\kappa_\pm\gt 0. For REAL, ui±=si∓1/κu_i^\pm=s_i^{\mp1/\kappa} and D±=1+∑j∈O±uj±D_\pm=1+\sum_{j\in\mathcal O_\pm}u_j^\pm, where κ>0\kappa\gt 0 and O±\mathcal O_\pm are the positive/negative label groups; its row holds same-label peers fixed in a retained mixed-label group and omits 1/(κTi)1/(\kappa T_i).

Training results (Table 12, Appendix H.2). Length-penalized training accuracy for Qwen3-30B-A3B at N=32N=32. The early peak covers the first 256 policy updates, the late mean covers updates 896–1,024, and the final score is taken at update 1,024. Subscripts indicate temporal standard deviations.

StatisticTOPRREALSAPOBAPOReSPO
Early peak0.3790.4490.4270.3970.418
Late mean0.418±0.0120.418_{\pm 0.012}0.420±0.0080.420_{\pm 0.008}0.379±0.1010.379_{\pm 0.101}0.496±0.0370.496_{\pm 0.037}0.588±0.007\mathbf{0.588}_{\pm 0.007}
Final0.4360.4180.5030.5220.566

Held-out evaluation (Table 13, Appendix H.2). Final-checkpoint pass@1 for Qwen3-30B-A3B at N=32N=32.

MethodAIME25AIME24AMC23OlympiadBenchMinervaMathMATH500Avg.
TOPR0.1690.2650.7200.5180.4260.8520.492±0.0220.492_{\pm 0.022}
REAL0.1980.2710.7000.5040.4460.8460.494±0.0230.494_{\pm 0.023}
SAPO0.1270.2560.6610.4450.4110.8120.452±0.0220.452_{\pm 0.022}
BAPO0.1630.2330.6910.5130.4470.8470.482±0.0220.482_{\pm 0.022}
ReSPO0.2020.2690.7950.5100.4510.8440.512±0.018\mathbf{0.512}_{\pm 0.018}

Additional baseline comparison (Sec. 4.3). On 30B at N=32N=32, REAL and TOPR have higher observed six-benchmark pass@1 than SAPO/BAPO, and ReSPO has the highest mean. Over the first 512 updates, ReSPO, TOPR, and REAL (ϕ(+)(0)>0\phi^{(+)}(0)\gt 0) show net response-length growth, while the ϕ(+)(0)=0\phi^{(+)}(0)=0 methods shorten overall (Figures 8 and 10). ReSPO's late mean length is 1,403 tokens, above the latter methods and below TOPR/REAL (2,744/3,419). TOPR/REAL's more frequent response-cap hits contribute to lower length-penalized training accuracy (Table 12).

In summary, these results suggest that ReSPO supports response lengths suitable for learning longer correct reasoning trajectories. Its late training responses are longer than those of methods with ϕ(+)(0)=0\phi^{(+)}(0)=0, while avoiding the excessive growth and frequent cap hits seen with TOPR and REAL. This balance limits length penalties and achieves higher training rewards and held-out accuracy.

C2. Sequence-reward objective and expected reward improvement

Shared concern: JsYL W2/W3; Pi56 Major Q1/Q3. Why use the full sequence ratio, and how does kernel selection relate to expected reward improvement?

For the sequence-level objective J=Eo∼πθR(o)J=\mathbb E_{o\sim\pi_\theta}R(o), with parameter-independent reward and common policy support (Appendix A.7), the exact off-policy reward gradient is

\qquad W=\pi_\theta/\mu.$$ **GRPO [3] and GSPO [4] also aim to improve sequence-level reward; they differ in their surrogates and importance corrections.** GRPO uses token ratios rather than the full sequence correction $W$. Such token-wise weighting can introduce noisy updates and training instability, especially for long responses [4]. GSPO uses $W^{1/T}$ to reduce variance, introducing length-dependent bias [1]: at fixed $T$, $\mu W^{1/T}=\mu^{1-1/T}\pi_\theta^{1/T}$ shifts the effective measure toward the behavior policy as length grows (Appendix A.6). **The gradient-error bound provides a reward-based selection criterion.** Let $\widehat g_\phi$ be Appendix A.7's kernel-weighted estimator from $G$ independent samples, and write $\mathcal G(\phi)$ for its mean-squared error (denoted $\mathcal E_G(\phi)$ in the appendix). Under its bounded reward-score gradient assumption $\\|R\nabla\log\pi_\theta\\|\le H$ and generator curvature assumption $f''\ge m\gt 0$ on the common ratio/kernel range, $$\begin{aligned} \mathcal G(\phi) &:=\mathbb E\\|\widehat g_\phi-\nabla J(\theta)\\|^2 \le U_\beta(\phi) =\frac{2H^2}{G}+C_\beta\mathcal F_\beta(\phi),\\\\ C_\beta &=\max\left\\{\frac{2H^2}{m\beta},\frac{4H^2}{mG(1-\beta)}\right\\}. \end{aligned}$$ Here $\beta\in(0,1)$ is the interpolation parameter, and the scalar Bregman divergence $d_f$ generated by $f$ defines: $$\mathcal F_\beta(\phi) =(1-\beta)\mathbb E_\mu[d_f(\phi(W),1)] +\beta\mathbb E_\mu[d_f(\phi(W),W)].$$ With $H,m,G,\beta$ and the generator fixed, **minimizing $\mathcal F_\beta$ minimizes $U_\beta$**. **$\mathcal F_\beta$ is the ratio-space counterpart of Eq. (6):** under $\tau=\mu\phi$, the two objectives use positive sequence weights and have the same unconstrained pointwise minimizer (Appendix A.7). Thus Eq. (6) selects the same base measure $\tau_0^*=\mu\phi_0$ and kernel $\phi_0$. For an estimator $\widehat g$ with $\mathbb E\\|\widehat g-\nabla J(\theta)\\|^2\le U$, an $L$-smooth objective $J$ and ascent step $\theta^+=\theta+\eta_{\mathrm{step}}\widehat g$, with $0\lt \eta_{\mathrm{step}}\le1/L$, give $$\mathbb E[J(\theta^+)]-J(\theta) \ge\frac{\eta_{\mathrm{step}}}{2} \left(\\|\nabla J(\theta)\\|^2-U\right).$$ At fixed policy and step size, reducing $U$ raises this lower bound; $U\lt \\|\nabla J(\theta)\\|^2$ guarantees positive expected reward improvement. For the base kernel, minimizing $\mathcal F_\beta$ minimizes $U_\beta$ and maximizes the corresponding lower bound. The final two-branch kernel uses a separate MSE bound combining displacement and moment terms after KL projection and rescaling (Appendix A.7). ---

C3. Branch contributions on Qwen3-30B-A3B

Shared concern: igAV W2; JsYL W1; Pi56 Major Q4. How do the two branches contribute?

Let R/VR/V denote ReSPO/VESPO and +/−+/- their advantage branches. Appendix E.5 compares all four pairs on Qwen3-30B-A3B at N=32N=32 after 1,024 policy updates, using group-standardized advantages, detached coefficients, and token-mean aggregation.

Branch pairLast-128 penalized training meanFinal penalized training accuracySix-benchmark pass@1
ReSPO (R+R−R^+R^-)58.8±0.7\mathbf{58.8}_{\pm0.7}56.651.2±1.8\mathbf{51.2}_{\pm1.8}
VESPO (V+V−V^+V^-)52.3±0.852.3_{\pm0.8}50.349.3±1.849.3_{\pm1.8}
R+V−R^+V^-52.5±1.352.5_{\pm1.3}54.850.9±2.350.9_{\pm2.3}
V+R−V^+R^-40.5±2.240.5_{\pm2.2}35.937.6±2.037.6_{\pm2.0}

All entries are percentages, matching Tables 9–10. Training subscripts are temporal standard deviations; held-out subscripts are bootstrap standard errors.

The two pairs retaining R+R^+ have the highest observed held-out means. At fixed R−R^-, replacing R+R^+ with V+V^+ lowers the mean by 13.6 pp. At fixed R+R^+, replacing R−R^- with V−V^- lowers it by 0.3 pp and lowers late penalized training accuracy by 6.3 pp. The larger held-out difference under the positive-branch swap supports the contribution of R+R^+.

The negative-branch design follows tail requirements and scale matching with the positive branch. The selected kernels have equal peaks and matching values and slopes at W=1W=1 (Sec. 3.3; Appendix A.3):

\max_{W\ge0}\phi^{(+)}(W) &=\max_{W\ge0}\phi^{(-)}(W)=\frac e2,\\\\ \phi^{(+)}(1)&=\phi^{(-)}(1)=1,\\\\ \phi^{(+)}{}'(1)&=\phi^{(-)}{}'(1)=-\frac12. \end{aligned}$$ These conditions guide the selected kernel's tail behavior and local shape. --- > ### C4. Negative tilt and high-ratio responses **Shared concern: JsYL W1; Pi56 Major Q4.** What does the negative tilt contribute when the positive branch is fixed? **Appendix E.6, Figure 6** removes the negative exponential tilt, replacing $\phi^{(-)}(W)=\sqrt W\\,e^{2(1-\sqrt W)}$ with $\sqrt W$, while retaining the positive branch and other settings. We examine Qwen3-1.7B at $N=32$ over the **first eight rollouts (256 policy updates)**, with a 15,360-token response limit. For response $i$ of length $T_i$, its loss-derivative mass is $c_i=\sum_{t=1}^{T_i}|\partial\mathcal L_{\mathrm{policy}}/\partial\ell_{i,t}|$, where $\ell_{i,t}$ is its token log probability. This measures allocation before multiplication by parameter score gradients. We record reuse positions 1, 8, 16, 24, and 32 in each rollout. Across these 40 mini-batches, only **9/2,822 negative-response observations (0.32%)** have $\log W\gt 5$, yet they carry **21.7%** of the pooled negative loss-derivative mass without tilt. At reuse 16 of rollouts 2, 3, 5, and 8, respectively, **3/57, 1/58, 4/74, and 1/74** negative responses carry **76.2%, 45.3%, 40.5%, and 12.1%** of their batch's negative derivative mass. Applying the original tilt to the same responses, with policy, advantages, and reduction fixed, reduces each share below $10^{-7}\\%$. The actual training curves compare this ablation with the existing ReSPO reference. At rollout 8, training accuracy is **13.6% with tilt versus 11.6% without**, while mean response lengths are similar (**1,129 versus 1,136 tokens**). **The negative tilt suppresses derivative concentration on rare high-ratio responses; this difference appears early, while aggregate response lengths remain close.** This supports our choice $\phi^{(-)}(\infty)=0$: vanishing tail weights limit the influence of extremely high-ratio negative responses on the loss derivatives. **References (Common response)** [1] Shen et al. (2026). VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training. arXiv:2602.10693. [2] Le Roux et al. (2025). Tapered Off-Policy REINFORCE: Stable and Efficient Reinforcement Learning for Large Language Models. NeurIPS. [3] Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. [4] Zheng et al. (2025). Group Sequence Policy Optimization. arXiv:2507.18071. [5] Gao et al. (2025). Soft Adaptive Policy Optimization. arXiv:2511.20347. [6] Xi et al. (2025). BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping. arXiv:2510.18927. [7] Zhai et al. (2026). Rewards as Labels: Revisiting RLVR from a Classification Perspective. arXiv:2602.05630.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.