acceptodds
Under review as a conference paper at ICLR 2027

When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL

Abstract

Reinforcement learning (RL) has shown great promise in large language models (LLMs) post-training, which typically rely on token-level clipping to maintain stability during optimization. Despite the empirical success of GRPO-style methods, we identify a fundamental and previously overlooked challenge in this popular Outcome-Supervised RL (OSRL) paradigm. We reveal that in OSRL, where advantages are shared across tokens within a response, importance sampling (IS) ratios also act as token-level weights beyond their traditional role of distribution correction that allocate the shared advantage signal across tokens. We show that this hidden role shift induces a critical mismatch for positive-advantage tokens, leading to unbalanced token weighting between positive and negative tokens. Specifically, it suppresses the update of underrepresented tokens that are lagging behind, while over-amplifying already high-probability tokens. This mismatch results in rich-get-richer dynamics that over-reinforce confident tokens, weaken catch-up learning that drive entropy collapse, excessive repetition, and premature convergence. To address this, we propose Asymmetric Importance Sampling Policy Optimization (ASPO), a simple yet effective strategy that reverses the ratio-induced weighting of positive-advantage tokens, while stabilizing extreme updates and maintaining gradient flow for unmasked tokens. This mismatch correction aligns their weighting trend with the learning dynamics of negative ones. Comprehensive experiments across math reasoning and coding benchmarks demonstrate that ASPO significantly mitigates entropy collapse, improves training stability, and enhances performance over strong GRPO-based baselines. Our analysis provides new insights into the role of token-level weighting in OSRL and highlights the critical importance of correcting ratio-induced weighting in LLM RL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_whtBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how token-level importance ratios affect learning when a response-level advantage is shared across tokens. It proposes Asymmetric Importance Sampling Policy Optimization (ASPO), which reverses the weighting of positive-advantage tokens while retaining directional masking and adding gradient-preserving dual clipping. Experiments on mathematical reasoning and coding report improved accuracy and more stable training than GRPO across several model sizes. The results suggest practical value, although the proposed explanation of credit misallocation and some essential implementation details require further clarification.

Critical strengths

(1) ASPO improves average performance over GRPO across all tested model sizes in mathematics and both tested sizes in coding. For example, the 4B math average increases from 61.6 to 65.0, while the 1.5B coding average increases from 26.8 to 31.0. These consistent gains support the practical usefulness of the method.

(2)The paper combines gradient analysis with useful diagnostic experiments. Removing importance weights or averaging them within positive responses improves performance and training stability. The additional dual-clipping ablation shows gradient spikes without the stabilizer, providing evidence beyond final benchmark scores.

Critical weaknesses

Major concerns

  1. The gradient-equivalence argument does not establish the central explanation of credit misallocation. The same identity also applies to token-specific advantages, and a larger ratio weight does not necessarily produce a larger parameter update because the score gradient varies as well. The analysis therefore does not adequately establish that the proposed weighting mismatch causes entropy collapse, rather than merely correlating with it.

  2. The dual-clipping operation is not specified precisely enough to reproduce. The paper does not provide the complete clipped objective or pseudocode, nor does it explicitly state the dual-clip threshold used in the experiments. The description of clipping values while retaining gradients leaves important implementation choices unresolved, particularly because the ablation identifies this operation as necessary for controlling gradient spikes.

  3. The experiments do not sufficiently isolate the benefit of reciprocal weighting from other stabilization choices. The existing ablation removes dual clipping from ASPO but does not compare weighting rules under the same stabilizer. Positive-response averaging already achieves 64.0 on the 4B math evaluation, compared with 65.0 for ASPO. A matched comparison with this simpler alternative and a relevant clipping method such as CISPO would clarify the contribution.

Minor concerns

  1. The paper does not report training-seed counts, uncertainty intervals, or checkpoint-selection rules. Sampling multiple completions per question does not measure training-run variability, making the relatively small gain over positive-response averaging difficult to assess.

  2. The OOD evaluation lacks benchmark-specific prompting, answer-scoring, and sampling details. These are particularly relevant to interpreting the large improvement in the 8B average, from 46.1 to 67.4, and should be specified for reproducibility.

Questions for authors
  1. What assumptions or direct measurements connect the importance-ratio weights to actual token-update magnitudes and reward-relevant credit, after accounting for the score gradients?

  2. Could the authors provide the complete ASPO update, including the dual-clip threshold, stop-gradient placement, masking order, and number of optimization passes per rollout batch?

  3. Under identical stabilization and training budgets, does reciprocal weighting consistently outperform positive-response averaging and a relevant clipping baseline across independent training seeds, using a stated checkpoint-selection rule?

3 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/3)

We thank the reviewer for the constructive comments. Our responses to all weaknesses and questions are as follows.


Weaknesses

Major concerns

W1: The gradient-equivalence argument does not establish the central explanation of credit misallocation. The same identity also applies to token-specific advantages, and a larger ratio weight does not necessarily produce a larger parameter update because the score gradient varies as well. The analysis therefore does not adequately establish that the proposed weighting mismatch causes entropy collapse, rather than merely correlating with it.

A: We agree that the weighted log-likelihood identity holds generally and that parameter updates depend on both the scalar weight and the score gradient ∇θlog⁡πθ\nabla_\theta \log \pi_\theta. We clarify our evidence across three points:

  1. Role of IS ratios under shared advantages: Unlike token-level PPO where rtir_t^i multiplies a token-specific advantage AtiA_t^i, OSRL broadcasts a single response-level advantage (Ati=A^iA_t^i = \hat{A}^i) to all tokens. Thus, rtir_t^i becomes the only token-varying scalar modulating A^i\hat{A}^i within a response.

  2. Direct measurements of parameter updates: On Qwen3-4B, we measured the relative L1 update norm and the effective rank of updates (RankMe) in three matrices in the transformer block (mlp.down_proj, self_attn.o_proj, and self_attn.k_proj).

MethodMLP down-proj. norm (×10−5\times 10^{-5})Attn out-proj. norm (×10−5\times 10^{-5})Attn key-proj. norm (×10−5\times 10^{-5})Avg. norm (×10−5\times 10^{-5})Avg. Effective Rank (RankMe)Avg. Score
GRPO4.564.454.824.611225.5761.6
GRPO w/o IS4.544.444.874.621232.5863.5
GRPO w/ Pos. Resp. IS Mean4.574.464.774.601230.5764.0
ASPO4.584.474.764.601236.3065.0

All four methods had similar mean relative update norms in these matrices, from 4.60×10−54.60\times10^{-5} to 4.62×10−54.62\times10^{-5}. ASPO had the highest mean RankMe (1236.30). These measurements describe parameter updates after the token gradients are combined.

  1. Controlled interventions: Sections 3.2 and 4.3 vary only the positive-token weights under identical data, clipping (ϵ=0.2\epsilon=0.2), and KL regularization (β=0.001\beta=0.001). Progressively removing (rti→1.0r_t^i \to 1.0), averaging (rti→rˉir_t^i \to \bar{r}^i), and inverting (rti→1/rtir_t^i \to 1/r_t^i) positive ratios directly slows entropy collapse, reduces repetition, and lowers positive clip ratios (Figure 2(b)–(d)).

We have added these measurements and clarifications to the revised manuscript.


W2: The dual-clipping operation is not specified precisely enough to reproduce. The paper does not provide the complete clipped objective or pseudocode, nor does it explicitly state the dual-clip threshold used in the experiments. The description of clipping values while retaining gradients leaves important implementation choices unresolved, particularly because the ablation identifies this operation as necessary for controlling gradient spikes.

A: We use a dual-clipping threshold of c=3.0c = 3.0, following the standard implementation in verl [1]. In the revised manuscript, we have added the complete mathematical objective and exact masking/clipping steps to the main text (Section 5), and placed the full pseudocode in the Appendix due to length constraints (see also Q2 below). We will release our full codebase upon acceptance.

AuthorsOpenReview ↗

Rebuttal (Part 2/3)

W3: The experiments do not sufficiently isolate the benefit of reciprocal weighting from other stabilization choices. The existing ablation removes dual clipping from ASPO but does not compare weighting rules under the same stabilizer. Positive-response averaging already achieves 64.0 on the 4B math evaluation, compared with 65.0 for ASPO. A matched comparison with this simpler alternative and a relevant clipping method such as CISPO would clarify the contribution.

A: Below we compare the weighting rules and clipping methods on Qwen3-4B under the exact same training setup:

MethodPositive Weighting RuleStabilization MechanismAvg. Score
GRPOStandard ratio (rtir_t^i)Hard clip (ϵ=0.2\epsilon = 0.2)61.6
GRPO w/ Pos. Resp. MeanResponse mean ratio (rˉi\bar{r}^i)Hard clip (ϵ=0.2\epsilon = 0.2)64.0
ASPO w/o Soft Dual ClipReciprocal ratio (1/rti1/r_t^i)Hard clip (ϵ=0.2\epsilon = 0.2)64.7
CISPOStandard ratio (rtir_t^i)Soft clip64.1
Full ASPOReciprocal ratio (1/rti1/r_t^i)Hard mask (ϵ=0.2\epsilon = 0.2) + soft dual clip (c=3.0c = 3.0)65.0
  1. Under the identical hard-clipping stabilizer (ϵ=0.2\epsilon = 0.2): Reciprocal weighting without dual clipping (64.7) still outperforms both positive-response averaging (64.0) and standard GRPO (61.6). Note also that because response-averaged ratios rˉi\bar{r}^i remain tightly concentrated around 1.01.0 (Figure 3), adding a soft dual-clip cap at c=3.0c = 3.0 to positive-response averaging never triggers and leaves its behavior unchanged.
  2. Role of soft dual clipping (c=3.0c = 3.0): Adding soft dual clipping yields a smaller secondary gain (64.7 →\to 65.0) by preventing gradient spikes when 1/rti1/r_t^i becomes large (Figure 8).
  3. Comparison with CISPO: CISPO (64.1) applies soft clipping without inverting positive weights, trailing both reciprocal weighting with hard clipping (64.7) and Full ASPO (65.0). Thus, the main gain comes from inverting the positive weighting direction rather than from soft clipping alone.

Minor concerns

W4: The paper does not report training-seed counts, uncertainty intervals, or checkpoint-selection rules. Sampling multiple completions per question does not measure training-run variability, making the relatively small gain over positive-response averaging difficult to assess.

A: For the Qwen3-4B math comparisons, we evaluated every method at step 400. We used this fixed checkpoint for both the single-run and three-seed results. Each rollout iteration includes four optimizer steps.

Across 3 independent training seeds on Qwen3-4B, Full ASPO achieves 65.0±0.20565.0 \pm 0.205, outperforming GRPO (61.7±0.37461.7 \pm 0.374), positive-response averaging (63.8±0.17063.8 \pm 0.170), and CISPO (63.9±0.26263.9 \pm 0.262) (see full table in Q3). We have added these multi-seed results and checkpoint rules to the revised manuscript.


W5: The OOD evaluation lacks benchmark-specific prompting, answer-scoring, and sampling details. These are particularly relevant to interpreting the large improvement in the 8B average, from 46.1 to 67.4, and should be specified for reproducibility.

A: We have added the complete OOD evaluation protocol (Table 3) to Appendix C:

  • Prompting & Scoring: We evaluate Qwen3-4B and Qwen3-8B in non-thinking mode. Each zero-shot prompt gives the question and answer options. We ask the model to reason step by step and place its final answer in \boxed{}. A rule-based verifier compares the answer with the reference option.
  • Sampling: We use vLLM with temperature 0.60.6 and a maximum output length of 32,76832{,}768 tokens. We report avg@1 on ARC-Challenge and MMLU-Pro and avg@4 on GPQA-Diamond.
  • Qwen3-8B results: In non-thinking mode, base Qwen3-8B scores 8.6 on GPQA-Diamond and 29.6 on MMLU-Pro. The average OOD score rises from 46.1 with GRPO to 67.4 with ASPO. Figure 6(b) also shows that GRPO loses entropy faster during training.
AuthorsOpenReview ↗

Rebuttal (Part 3/3)

Questions

Q1: What assumptions or direct measurements connect the importance-ratio weights to actual token-update magnitudes and reward-relevant credit, after accounting for the score gradients?

A:

  1. Token-update magnitudes after accounting for score gradients: Treating wtiw_t^i in wtiA^i∇θlog⁡πθ(oti∣sti)w_t^i \hat{A}^i \nabla_\theta \log \pi_\theta(o_t^i \mid s_t^i) as a relative token weight assumes that score-gradient norms ∥∇θlog⁡πθ∥\|\nabla_\theta \log \pi_\theta\| within a response do not systematically cancel wtiw_t^i. Because actual weight updates ΔW\Delta W combine both wtiw_t^i and the backpropagated score gradients, we directly measured ∥ΔW∥\|\Delta W\| and effective rank (RankMe) across three modules on Qwen3-4B (see table in W1). Total update norms remain matched across all methods (≈4.60×10−5\approx 4.60 \times 10^{-5}), while ASPO achieves the highest effective rank (1236.30 vs. 1225.57 for GRPO), confirming more balanced updates across directions.

  2. Scope regarding reward-relevant credit: Because OSRL only observes a response-level reward, no outcome-based method can identify true causal token-level credit. In our paper, "credit allocation" refers strictly to how the shared response advantage A^i\hat{A}^i is distributed across tokens during gradient updates.


Q2: Could the authors provide the complete ASPO update, including the dual-clip threshold, stop-gradient placement, masking order, and number of optimization passes per rollout batch?

A: Here is the exact specification of the ASPO update (added to Section 5 of the main text, with full pseudocode in the Appendix due to length):

  • Optimization passes per rollout batch: Each rollout batch of 64 prompts (16 responses per prompt) is split into 4 mini-batches of 16 prompts, performing 1 optimization epoch (4 gradient steps) per rollout batch.
  • Hyperparameters: Clipping range ϵlow=ϵhigh=0.2\epsilon_{\text{low}} = \epsilon_{\text{high}} = 0.2, dual-clip threshold c=3.0c = 3.0, and KL coefficient β=0.001\beta = 0.001.
  • Step-by-step update and stop-gradient placement: Let rti(θ)=πθ(oti∣q,o<ti)πθold(oti∣q,o<ti)r_t^i(\theta) = \frac{\pi_\theta(o_t^i \mid q, o_{<t}^i)}{\pi_{\theta_{\text{old}}}(o_t^i \mid q, o_{<t}^i)} and let sg⁡(⋅)\operatorname{sg}(\cdot) denote stop-gradient:
    1. Step 1 (Hard Token Masking, computed before flipping): mti={I ⁣(rti(θ)≤1+ϵhigh),if A^i≥0,I ⁣(rti(θ)≥1−ϵlow),if A^i<0.m_t^i = \begin{cases} \mathbb{I}\!\left(r_t^i(\theta) \le 1 + \epsilon_{\text{high}}\right), & \text{if } \hat{A}^i \ge 0, \\ \mathbb{I}\!\left(r_t^i(\theta) \ge 1 - \epsilon_{\text{low}}\right), & \text{if } \hat{A}^i < 0. \end{cases}
    2. Step 2 (Asymmetric Ratio Flipping): r~ti(θ)={πθold(oti∣q,o<ti) πθ(oti∣q,o<ti)sg⁡ ⁣(πθ2(oti∣q,o<ti)),if A^i≥0,rti(θ),if A^i<0.\tilde{r}_t^i(\theta) = \begin{cases} \dfrac{\pi_{\theta_{\text{old}}}(o_t^i \mid q, o_{<t}^i)\,\pi_\theta(o_t^i \mid q, o_{<t}^i)}{\operatorname{sg}\!\left(\pi_\theta^2(o_t^i \mid q, o_{<t}^i)\right)}, & \text{if } \hat{A}^i \ge 0, \\ r_t^i(\theta), & \text{if } \hat{A}^i < 0. \end{cases}
    3. Step 3 (Soft Dual Clipping at c=3.0c = 3.0, capping values while keeping gradients): r^ti(θ)={r~ti(θ),if r~ti(θ)≤c,c⋅πθ(oti∣q,o<ti)sg⁡ ⁣(πθ(oti∣q,o<ti)),if r~ti(θ)>c.\hat{r}_t^i(\theta) = \begin{cases} \tilde{r}_t^i(\theta), & \text{if } \tilde{r}_t^i(\theta) \le c, \\ c \cdot \dfrac{\pi_\theta(o_t^i \mid q, o_{<t}^i)}{\operatorname{sg}\!\left(\pi_\theta(o_t^i \mid q, o_{<t}^i)\right)}, & \text{if } \tilde{r}_t^i(\theta) > c. \end{cases}
    4. Final Objective: Let V\mathcal V be the set of valid response tokens in the mini-batch. We maximize JASPO(θ)=1∣V∣∑(i,t)∈V[mti r^ti(θ) A^i−β D^KL,ti]\mathcal J_{\mathrm{ASPO}}(\theta)=\frac{1}{|\mathcal V|}\sum_{(i,t)\in\mathcal V}\left[m_t^i\,\hat r_t^i(\theta)\,\hat A^i-\beta\,\widehat D_{\mathrm{KL},t}^i\right].

Q3: Under identical stabilization and training budgets, does reciprocal weighting consistently outperform positive-response averaging and a relevant clipping baseline across independent training seeds, using a stated checkpoint-selection rule?

A: For all Qwen3-4B methods, each rollout iteration uses 64 prompts and four optimizer steps with 16 prompts per mini-batch. We evaluated each method at step 400. The table reports single-run scores and three-seed results (mean ±\pm standard deviation) across the six math benchmarks.

MethodSingle-Run Avg.3-Seed Avg. ±\pm Std
GRPO61.661.7±0.37461.7 \pm 0.374
Positive-response averaging (rˉi\bar{r}^i + hard clip)64.063.8±0.17063.8 \pm 0.170
CISPO64.163.9±0.26263.9 \pm 0.262
Full ASPO65.065.0±0.20565.0 \pm 0.205

At step 400, Full ASPO had the highest mean score across three independent training seeds (65.0±0.20565.0 \pm 0.205), followed by CISPO (63.9±0.26263.9 \pm 0.262), positive-response averaging (63.8±0.17063.8 \pm 0.170), and GRPO (61.7±0.37461.7 \pm 0.374).


Thanks again for your valuable feedback. We would be happy to address any further questions.

References

[1] HybridFlow: A Flexible and Efficient RLHF Framework. EuroSys 2025.

Reviewer_FjBzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the role of importance sampling (IS) ratios in token-level gradient weighting for Outcome-Supervised Reinforcement Learning (OSRL). The authors observe that, in GRPO-style methods with shared response-level advantages, IS ratios not only serve their traditional distribution-correction role but also influence the gradient contributions of individual tokens. For positive-advantage responses, standard GRPO tends to assign larger update weights to tokens whose probabilities have already increased, while assigning smaller weights to tokens whose probabilities have decreased relative to the old policy. The authors argue that this weighting pattern may lead to rich-get-richer dynamics and entropy collapse.

To address this issue, the paper proposes Asymmetric Importance Sampling Policy Optimization (ASPO), which reverses the ratio-induced weighting for positive-advantage tokens and incorporates a dual-clipping mechanism to improve training stability. Experiments across multiple model scales on mathematical reasoning and coding benchmarks demonstrate relatively consistent performance improvements over standard GRPO.

Critical strengths

S1. The paper provides an insightful perspective on the role of IS ratios in OSRL by analyzing their effects on token-level gradient weighting.

S2. ASPO is simple and easy to implement, without introducing significant additional computational overhead.

S3. The experiments cover multiple model scales, ranging from 1.5B to 30B, and demonstrate relatively consistent improvements on mathematical reasoning and coding tasks.

S4. The paper provides useful analyses of training dynamics, together with supporting ablation studies that help explain the behavior of the proposed method.

Critical weaknesses

W1. The original IS ratio has a principled distribution-correction interpretation and is closely connected to the policy optimization objective. ASPO modifies this weighting through reciprocal IS weights for positive-advantage tokens, but it remains unclear why the resulting optimization objective should be more effective in improving the expected RL return. Moreover, positive-advantage responses may contain redundant or incorrect intermediate tokens, so decreasing their probabilities does not necessarily indicate insufficient learning. Further clarification of the motivation and applicability of the proposed catch-up learning principle would strengthen the paper.

W2. The main controlled experiments primarily compare ASPO with standard GRPO and simplified IS-weighting variants, with limited comparisons to other relevant methods. For example, DAPO, CISPO, GSPO, SAPO, and GMPO also investigate related issues involving IS ratios, clipping, or gradient weighting. Comparisons with a few representative methods under comparable training settings would help better establish the distinctive contributions of ASPO.

W3. The ablation results in the appendix show that removing IS weights or replacing positive-token ratios with response-level mean ratios already leads to noticeable improvements over standard GRPO. This raises the question of whether ASPO's gains arise specifically from the proposed catch-up learning mechanism or from more general changes in gradient weighting and effective optimization strength. Additional comparisons with alternative weighting functions would help clarify the source of these improvements and the necessity of reciprocal weighting.

Questions for authors

Q1. Why should tokens whose probabilities have decreased relative to the old policy within positive-advantage responses receive stronger reinforcement? Would it be useful to distinguish critical tokens from redundant, unnecessary, or even incorrect intermediate tokens? Further clarification of the motivation and applicability of this assumption would be helpful.

Q2. How does ASPO compare with existing GRPO variants that address related issues? Could the authors provide controlled comparisons with a few representative baselines to further demonstrate its distinctive advantages?

Q3. What motivates the specific choice of reciprocal IS weights? Have the authors explored alternative weighting functions that could achieve similar catch-up learning effects?

2 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for the constructive comments. Our responses to all weaknesses and questions are as follows.


W1 & Q1

A: Thanks for raising this question. ASPO is motivated by how importance-ratio weighting shapes token updates under shared outcome supervision. Standard RL post-training algorithm's (e.g., GRPO) ratio weighting produces severe amplification on positive-advantage responses. ASPO replaces it with diminishing reinforcement as the token moves in the advantage-implied direction, while preserving the original surrogate gradient at the rollout policy. This is the basis of our catch-up principle.

  1. Catch-up learning makes the weighting consistent with directional progress: Let d(A,r)=sign⁡(A)log⁡rd(A,r) = \operatorname{sign}(A)\log r measure progress in the advantage-implied direction. In GRPO, the scalar coefficient magnitude is ∣A∣e−d\vert{}A\vert{}e^{-d} for A<0A<0 (penalizing less as progress increases) but ∣A∣ed\vert{}A\vert{}e^{d} for A>0A>0 (amplifying updates as progress increases). Using 1/rt1/r_t for A>0A>0 yields ∣A∣e−d\vert{}A\vert{}e^{-d} for both signs, preventing already-reinforced tokens from dominating the shared positive signal while preserving the reinforcement sign. Here, "catch-up" refers to progress relative to the rollout policy. ASPO assigns diminishing update coefficients as tokens move in the encouraged direction, rather than further amplifying positive tokens that have already moved in that direction.

  2. Relative ratio (rt=πθ/πoldr_t = \pi_\theta / \pi_{\text{old}}) vs. absolute probability (πθ\pi_\theta): ASPO weights active positive tokens by the reciprocal of the policy ratio 1/rt1/r_t, not by absolute token probability πθ\pi_\theta. A token only receives a larger weight (1/rt>11/r_t > 1) when its probability has decreased relative to the rollout policy (rt<1r_t < 1). Conversely, tokens whose probabilities have already increased (rt>1r_t > 1) are down-weighted (1/rt<11/r_t < 1).

  3. Redundant or incorrect intermediate tokens: We agree that outcome-level supervision cannot identify causal token-level correctness, which is a limitation shared by GRPO and all OSRL methods. However, redundant or filler tokens in positive responses often carry high probability and become further reinforced during updates (rt>1r_t > 1), in which case ASPO reduces their update weight (1/rt<11/r_t < 1) compared to GRPO. We will clarify this scope and limitation in the revised version.


W2 & Q2

A: We evaluated ASPO against DAPO, CISPO, and GSPO on Qwen3-4B under the exact same training data, hyperparameters, and evaluation setup across six math benchmarks:

MethodAverage score
GRPO61.6
DAPO63.4
CISPO64.1
GSPO63.8
ASPO65.0

While DAPO relaxes upper clipping bounds, CISPO preserves gradients of clipped tokens via soft clipping, and GSPO computes ratios at the sequence level, none of them corrects the intra-response ratio weighting direction for active positive tokens. By inverting positive-token weights, ASPO achieves the highest average score (65.0) among all variants. We have included this controlled comparison in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

W3 & Q3

A:

  1. Theoretical uniqueness of α=−1\alpha = -1 (1/rt1/r_t): Under directional progress d(A,r)=sign⁡(A)log⁡rd(A,r) = \operatorname{sign}(A)\log r, suppose we require a single progress-response function ϕ(d)\phi(d) such that c(A,r)=∣A∣ϕ(d(A,r))c(A,r) = \vert{}A\vert{}\phi(d(A,r)) for both positive and negative advantages, while keeping GRPO's negative branch (c(A,r)=∣A∣r=∣A∣e−dc(A,r) = \vert{}A\vert{}r = \vert{}A\vert{}e^{-d} for A<0A<0). This uniquely determines ϕ(d)=e−d\phi(d) = e^{-d}, which yields c(A,r)=∣A∣/rc(A,r) = \vert{}A\vert{}/r (α=−1\alpha = -1) for A>0A>0. Any other exponent α≠−1\alpha \neq -1 breaks this symmetry between positive and negative progress updates.

  2. Empirical comparison across the rtαr_t^\alpha family: We evaluated positive-token weighting functions rtαr_t^\alpha for α∈{−1.0,−0.5,0.0,0.5,1.0}\alpha \in \{-1.0, -0.5, 0.0, 0.5, 1.0\} on Qwen3-4B under identical training settings:

Variant (α\alpha)Average score
1.01.0 (GRPO)61.6
0.50.562.3
0.00.062.0
−0.5-0.563.1
−1.0-1.0 (ASPO)65.0

Negative exponents (α<0\alpha < 0) consistently outperform non-negative ones, and α=−1.0\alpha = -1.0 achieves the strongest performance (65.0), supporting our theoretical derivation.


Thanks again for your valuable feedback. We would be happy to address any further questions.

Reviewer_p597OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how importance ratios redistribute token-level updates when all tokens in a response share an outcome-based advantage. ASPO changes the effective ratio weighting for positive advantages while retaining clipping and adding soft dual clipping. Experiments on mathematical reasoning and code generation report improvements over GRPO across several model sizes.

Critical strengths
  1. The method is simple and addresses a concrete optimization issue. The comparisons with removing importance ratios and using response-level mean ratios help diagnose the role of token weighting.
  2. The empirical coverage is substantial. For example, Table 1 reports average math improvements from 61.6 to 65.0 for Qwen3-4B and from 63.3 to 65.7 for Qwen3-8B. These results support the practical value of ASPO.
Critical weaknesses
  1. The causal interpretation is stronger than the evidence. Proposition 3.1 gives a weighted log-likelihood view of the gradient, but does not establish that importance weighting is inherently inappropriate under outcome supervision. The diagnostics also change update magnitude, so improved credit allocation is not fully separated from a change in effective optimization strength.
  2. ASPO combines asymmetric weighting with a stabilization mechanism. The dual-clipping ablation is useful, but a matched-update comparison would make the source of the gains more convincing.
Questions for authors
  1. After matching update norms or effective step sizes, does asymmetric weighting still outperform the no-ratio and response-mean controls?
  2. Can the authors clarify which conclusions concern practical token weighting and which, if any, establish a problem with importance sampling as distribution correction?
2 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

We thank the reviewer for the constructive comments. Our responses to all weaknesses and questions are as follows.


Weaknesses

W1: The causal interpretation is stronger than the evidence. Proposition 3.1 gives a weighted log-likelihood view of the gradient, but does not establish that importance weighting is inherently inappropriate under outcome supervision. The diagnostics also change update magnitude, so improved credit allocation is not fully separated from a change in effective optimization strength.

A: Proposition 3.1 identifies a concrete effect of how a shared response-level outcome signal is distributed across token score gradients, while the empirical question is whether changing this weighting improves learning beyond what can be achieved by changing overall update strength.

For an active token in GRPO with response advantage A^i\hat{A}^i, the gradient contribution is

gi,t=ri,tA^i∇θlog⁡πθ(oi,t∣si,t).g_{i,t} = r_{i,t}\hat{A}^i\nabla_\theta\log\pi_\theta(o_{i,t}\mid s_{i,t}).

For two active tokens t,ut, u in the same response (A^i≠0\hat{A}^i \neq 0), the ratio of their scalar coefficients is ri,t/ri,ur_{i,t}/r_{i,u}. This characterization exposes an optimization effect of the ratio within the shared-advantage surrogate. Here, "credit allocation" refers strictly to how this shared response-level training signal is distributed across tokens, rather than causal credit toward the reward.

Defining directional progress as d(A,r)=sign⁡(A)log⁡rd(A, r) = \operatorname{sign}(A)\log r, standard GRPO's coefficient decreases with progress for A<0A<0 (∣A∣e−d\vert{}A\vert{}e^{-d}) but increases with progress for A>0A>0 (∣A∣ed\vert{}A\vert{}e^{d}), creating a self-reinforcing loop on already confident positive tokens. ASPO replaces the positive coefficient with 1/ri,t1/r_{i,t} to enforce negative feedback for both signs. Combined with our controlled ablations (Section 3.2, Section 4.3, and Table 4), this directly motivates the asymmetric rule.


W2: ASPO combines asymmetric weighting with a stabilization mechanism. The dual-clipping ablation is useful, but a matched-update comparison would make the source of the gains more convincing.

A: The component ablation below on Qwen3-4B isolates the effect of ratio flipping from the masking and clipping stabilizers under the exact same training setup:

VariantAverage score
GRPO61.6
GRPO + flipped positive ratio64.4
GRPO + flipped positive ratio + hard clipping64.7
GRPO + flipped positive ratio + soft dual clipping64.5
Full ASPO65.0

Flipping the positive ratio accounts for most of the gain (61.6 →\to 64.4), while adding hard masking (64.7) and soft dual clipping (65.0) stabilizes extreme updates and brings further improvement. Specifically, reciprocal weighting reallocates the positive learning signal, hard masking filters out already over-reinforced tokens (rti>1+ϵr_t^i > 1+\epsilon), and soft dual clipping caps extreme reciprocal weights (1/rti>c1/r_t^i > c).


Questions

Q1: "After matching update norms or effective step sizes, does asymmetric weighting still outperform the no-ratio and response-mean controls?"

A: We used the same learning rate for all four methods and measured the resulting parameter updates. On Qwen3-4B, we tracked three matrices in the final transformer block (mlp.down_proj, self_attn.o_proj, and self_attn.k_proj). For each matrix, we measured the relative L1 update norm ∥ΔW∥1/∥Wprev∥1\|\Delta W\|_1/\|W_{\mathrm{prev}}\|_1 and the effective rank of ΔW\Delta W (RankMe).

MethodMLP down-proj. norm (×10−5\times 10^{-5})Attn out-proj. norm (×10−5\times 10^{-5})Attn key-proj. norm (×10−5\times 10^{-5})Avg. norm (×10−5\times 10^{-5})Avg. Effective Rank (RankMe)Avg. Score
GRPO4.564.454.824.611225.5761.6
GRPO w/o IS4.544.444.874.621232.5863.5
GRPO w/ Pos. Resp. IS Mean4.574.464.774.601230.5764.0
ASPO4.584.474.764.601236.3065.0

The four methods had similar mean relative update norms in these three matrices, ranging from 4.60×10−54.60\times10^{-5} to 4.62×10−54.62\times10^{-5}. ASPO had the highest mean RankMe (1236.30). It also had the highest average score (65.0), compared with 63.5 for GRPO w/o IS and 64.0 for positive-response averaging. We have added these results to the revised manuscript.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

Q2: Can the authors clarify which conclusions concern practical token weighting and which, if any, establish a problem with importance sampling as distribution correction?

A: Our conclusions concern the optimization role of token weighting in GRPO-style outcome-supervised RL. Importance sampling has its standard likelihood-ratio interpretation, and the current-to-old ratio retains its role in the PPO surrogate. When a single response-level advantage is broadcast to all tokens, that same ratio also determines the relative scalar coefficients applied to the shared training signal.

ASPO acts on this latter role:

  • Characterization: For active tokens, the proposition expresses the surrogate gradient as weighted token score gradients. With a shared response-level advantage, the IS ratios determine the relative scalar weights of these token contributions. This establishes an update-weighting structure.
  • Optimization design (directional progress): The directional-progress analysis shows that GRPO's raw coefficient magnitude decreases with progress for negative advantages but increases for positive advantages. ASPO preserves the uncapped negative-advantage coefficient and obtains reciprocal weighting for positive advantages, with hard masking and coefficient capping in the implemented objective. This provides a rationale for changing the update feedback.
  • Empirical effectiveness: Across four model scales (1.5B, 4B, 8B, and 30B-A3B), ASPO improves average benchmark scores over GRPO. The Qwen3-4B and Qwen3-8B training curves also show slower entropy loss with ASPO. On Qwen3-4B, the four methods in Q1 had similar early update sizes in the three matrices we measured.

Thanks again for your valuable feedback. We would be happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.