- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how token-level importance ratios affect learning when a response-level advantage is shared across tokens. It proposes Asymmetric Importance Sampling Policy Optimization (ASPO), which reverses the weighting of positive-advantage tokens while retaining directional masking and adding gradient-preserving dual clipping. Experiments on mathematical reasoning and coding report improved accuracy and more stable training than GRPO across several model sizes. The results suggest practical value, although the proposed explanation of credit misallocation and some essential implementation details require further clarification.
Critical strengths
(1) ASPO improves average performance over GRPO across all tested model sizes in mathematics and both tested sizes in coding. For example, the 4B math average increases from 61.6 to 65.0, while the 1.5B coding average increases from 26.8 to 31.0. These consistent gains support the practical usefulness of the method.
(2)The paper combines gradient analysis with useful diagnostic experiments. Removing importance weights or averaging them within positive responses improves performance and training stability. The additional dual-clipping ablation shows gradient spikes without the stabilizer, providing evidence beyond final benchmark scores.
Critical weaknesses
Major concerns
-
The gradient-equivalence argument does not establish the central explanation of credit misallocation. The same identity also applies to token-specific advantages, and a larger ratio weight does not necessarily produce a larger parameter update because the score gradient varies as well. The analysis therefore does not adequately establish that the proposed weighting mismatch causes entropy collapse, rather than merely correlating with it.
-
The dual-clipping operation is not specified precisely enough to reproduce. The paper does not provide the complete clipped objective or pseudocode, nor does it explicitly state the dual-clip threshold used in the experiments. The description of clipping values while retaining gradients leaves important implementation choices unresolved, particularly because the ablation identifies this operation as necessary for controlling gradient spikes.
-
The experiments do not sufficiently isolate the benefit of reciprocal weighting from other stabilization choices. The existing ablation removes dual clipping from ASPO but does not compare weighting rules under the same stabilizer. Positive-response averaging already achieves 64.0 on the 4B math evaluation, compared with 65.0 for ASPO. A matched comparison with this simpler alternative and a relevant clipping method such as CISPO would clarify the contribution.
Minor concerns
-
The paper does not report training-seed counts, uncertainty intervals, or checkpoint-selection rules. Sampling multiple completions per question does not measure training-run variability, making the relatively small gain over positive-response averaging difficult to assess.
-
The OOD evaluation lacks benchmark-specific prompting, answer-scoring, and sampling details. These are particularly relevant to interpreting the large improvement in the 8B average, from 46.1 to 67.4, and should be specified for reproducibility.
Questions for authors
-
What assumptions or direct measurements connect the importance-ratio weights to actual token-update magnitudes and reward-relevant credit, after accounting for the score gradients?
-
Could the authors provide the complete ASPO update, including the dual-clip threshold, stop-gradient placement, masking order, and number of optimization passes per rollout batch?
-
Under identical stabilization and training budgets, does reciprocal weighting consistently outperform positive-response averaging and a relevant clipping baseline across independent training seeds, using a stated checkpoint-selection rule?
3 comments
Rebuttal (Part 2/3)
W3: The experiments do not sufficiently isolate the benefit of reciprocal weighting from other stabilization choices. The existing ablation removes dual clipping from ASPO but does not compare weighting rules under the same stabilizer. Positive-response averaging already achieves 64.0 on the 4B math evaluation, compared with 65.0 for ASPO. A matched comparison with this simpler alternative and a relevant clipping method such as CISPO would clarify the contribution.
A: Below we compare the weighting rules and clipping methods on Qwen3-4B under the exact same training setup:
| Method | Positive Weighting Rule | Stabilization Mechanism | Avg. Score |
|---|---|---|---|
| GRPO | Standard ratio () | Hard clip () | 61.6 |
| GRPO w/ Pos. Resp. Mean | Response mean ratio () | Hard clip () | 64.0 |
| ASPO w/o Soft Dual Clip | Reciprocal ratio () | Hard clip () | 64.7 |
| CISPO | Standard ratio () | Soft clip | 64.1 |
| Full ASPO | Reciprocal ratio () | Hard mask () + soft dual clip () | 65.0 |
- Under the identical hard-clipping stabilizer (): Reciprocal weighting without dual clipping (
64.7) still outperforms both positive-response averaging (64.0) and standard GRPO (61.6). Note also that because response-averaged ratios remain tightly concentrated around (Figure 3), adding a soft dual-clip cap at to positive-response averaging never triggers and leaves its behavior unchanged. - Role of soft dual clipping (): Adding soft dual clipping yields a smaller secondary gain (
64.765.0) by preventing gradient spikes when becomes large (Figure 8). - Comparison with CISPO: CISPO (
64.1) applies soft clipping without inverting positive weights, trailing both reciprocal weighting with hard clipping (64.7) and Full ASPO (65.0). Thus, the main gain comes from inverting the positive weighting direction rather than from soft clipping alone.
Minor concerns
W4: The paper does not report training-seed counts, uncertainty intervals, or checkpoint-selection rules. Sampling multiple completions per question does not measure training-run variability, making the relatively small gain over positive-response averaging difficult to assess.
A: For the Qwen3-4B math comparisons, we evaluated every method at step 400. We used this fixed checkpoint for both the single-run and three-seed results. Each rollout iteration includes four optimizer steps.
Across 3 independent training seeds on Qwen3-4B, Full ASPO achieves , outperforming GRPO (), positive-response averaging (), and CISPO () (see full table in Q3). We have added these multi-seed results and checkpoint rules to the revised manuscript.
W5: The OOD evaluation lacks benchmark-specific prompting, answer-scoring, and sampling details. These are particularly relevant to interpreting the large improvement in the 8B average, from 46.1 to 67.4, and should be specified for reproducibility.
A: We have added the complete OOD evaluation protocol (Table 3) to Appendix C:
- Prompting & Scoring: We evaluate Qwen3-4B and Qwen3-8B in non-thinking mode. Each zero-shot prompt gives the question and answer options. We ask the model to reason step by step and place its final answer in
\boxed{}. A rule-based verifier compares the answer with the reference option. - Sampling: We use vLLM with temperature and a maximum output length of tokens. We report
avg@1on ARC-Challenge and MMLU-Pro andavg@4on GPQA-Diamond. - Qwen3-8B results: In non-thinking mode, base Qwen3-8B scores
8.6on GPQA-Diamond and29.6on MMLU-Pro. The average OOD score rises from46.1with GRPO to67.4with ASPO. Figure 6(b) also shows that GRPO loses entropy faster during training.
Rebuttal (Part 3/3)
Questions
Q1: What assumptions or direct measurements connect the importance-ratio weights to actual token-update magnitudes and reward-relevant credit, after accounting for the score gradients?
A:
-
Token-update magnitudes after accounting for score gradients: Treating in as a relative token weight assumes that score-gradient norms within a response do not systematically cancel . Because actual weight updates combine both and the backpropagated score gradients, we directly measured and effective rank (
RankMe) across three modules on Qwen3-4B (see table in W1). Total update norms remain matched across all methods (), while ASPO achieves the highest effective rank (1236.30vs.1225.57for GRPO), confirming more balanced updates across directions. -
Scope regarding reward-relevant credit: Because OSRL only observes a response-level reward, no outcome-based method can identify true causal token-level credit. In our paper, "credit allocation" refers strictly to how the shared response advantage is distributed across tokens during gradient updates.
Q2: Could the authors provide the complete ASPO update, including the dual-clip threshold, stop-gradient placement, masking order, and number of optimization passes per rollout batch?
A: Here is the exact specification of the ASPO update (added to Section 5 of the main text, with full pseudocode in the Appendix due to length):
- Optimization passes per rollout batch: Each rollout batch of 64 prompts (16 responses per prompt) is split into 4 mini-batches of 16 prompts, performing 1 optimization epoch (4 gradient steps) per rollout batch.
- Hyperparameters: Clipping range , dual-clip threshold , and KL coefficient .
- Step-by-step update and stop-gradient placement: Let and let denote stop-gradient:
- Step 1 (Hard Token Masking, computed before flipping):
- Step 2 (Asymmetric Ratio Flipping):
- Step 3 (Soft Dual Clipping at , capping values while keeping gradients):
- Final Objective: Let be the set of valid response tokens in the mini-batch. We maximize .
Q3: Under identical stabilization and training budgets, does reciprocal weighting consistently outperform positive-response averaging and a relevant clipping baseline across independent training seeds, using a stated checkpoint-selection rule?
A: For all Qwen3-4B methods, each rollout iteration uses 64 prompts and four optimizer steps with 16 prompts per mini-batch. We evaluated each method at step 400. The table reports single-run scores and three-seed results (mean standard deviation) across the six math benchmarks.
| Method | Single-Run Avg. | 3-Seed Avg. Std |
|---|---|---|
| GRPO | 61.6 | |
| Positive-response averaging ( + hard clip) | 64.0 | |
| CISPO | 64.1 | |
| Full ASPO | 65.0 |
At step 400, Full ASPO had the highest mean score across three independent training seeds (), followed by CISPO (), positive-response averaging (), and GRPO ().
Thanks again for your valuable feedback. We would be happy to address any further questions.
References
[1] HybridFlow: A Flexible and Efficient RLHF Framework. EuroSys 2025.
Rebuttal (Part 1/3)
We thank the reviewer for the constructive comments. Our responses to all weaknesses and questions are as follows.
Weaknesses
Major concerns
W1: The gradient-equivalence argument does not establish the central explanation of credit misallocation. The same identity also applies to token-specific advantages, and a larger ratio weight does not necessarily produce a larger parameter update because the score gradient varies as well. The analysis therefore does not adequately establish that the proposed weighting mismatch causes entropy collapse, rather than merely correlating with it.
A: We agree that the weighted log-likelihood identity holds generally and that parameter updates depend on both the scalar weight and the score gradient ∇θlogπθ. We clarify our evidence across three points:
Role of IS ratios under shared advantages: Unlike token-level PPO where rti multiplies a token-specific advantage Ati, OSRL broadcasts a single response-level advantage (Ati=A^i) to all tokens. Thus, rti becomes the only token-varying scalar modulating A^i within a response.
Direct measurements of parameter updates: On Qwen3-4B, we measured the relative L1 update norm and the effective rank of updates (
RankMe) in three matrices in the transformer block (mlp.down_proj,self_attn.o_proj, andself_attn.k_proj).All four methods had similar mean relative update norms in these matrices, from 4.60×10−5 to 4.62×10−5. ASPO had the highest mean RankMe (
1236.30). These measurements describe parameter updates after the token gradients are combined.We have added these measurements and clarifications to the revised manuscript.
W2: The dual-clipping operation is not specified precisely enough to reproduce. The paper does not provide the complete clipped objective or pseudocode, nor does it explicitly state the dual-clip threshold used in the experiments. The description of clipping values while retaining gradients leaves important implementation choices unresolved, particularly because the ablation identifies this operation as necessary for controlling gradient spikes.
A: We use a dual-clipping threshold of c=3.0, following the standard implementation in
verl[1]. In the revised manuscript, we have added the complete mathematical objective and exact masking/clipping steps to the main text (Section 5), and placed the full pseudocode in the Appendix due to length constraints (see also Q2 below). We will release our full codebase upon acceptance.