- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets off-policy mismatch in RL post-training of LLMs, caused by rollout reuse, asynchronous rollouts and train–inference mismatch. It proposes Selective Importance Sampling (SIS). At each token, the behavior policy is treated as a proposal for the current policy, and an acceptance test is drawn with probability w_t/M_t, where w_t is the token importance ratio and M_t is the largest ratio over the vocabulary. Accepted tokens get weight one ("on-policy"); rejected tokens keep their ratio, which the base algorithm then processes. SIS is a plug-in for GRPO, DAPO and GSPO with small overhead. Experiments on dense and MoE Qwen3 models (and a Llama model) for math and search-agent tasks report average gains over each base algorithm.
Critical strengths
- Simple and cheap.
SIS only changes the per-token ratio, needs no extra forward pass, and adds about one percent wall-clock time. It is easy to adopt in existing GRPO-style pipelines.
- Useful lens and diagnostics.
Viewing off-policy correction as token-level rejection sampling with a computable top-K envelope is an interesting angle. Table 2's unified view of stabilization tricks as reweightings g(w), and the accept-rate and token-level analyses, make the method's behavior easy to inspect.
Critical weaknesses
Gains not separated from noise or truncation. There are no seeds or error bars, and several gains are about one point. K differs per algorithm without a validation protocol; at other K, GSPO+SIS only matches or trails GSPO (Table 3). Gains do not grow with staleness (Table 8). Please add seeds on one setting, the deterministic weight above, and a truncation baseline.
Questions for authors
If accepted tokens use π_θ/sg[π_θ], they bypass the clip. What fraction of accepted tokens have ratios outside the clip range, and how much of the gain does this un-clipping explain?
Clarification: SIS is not a clipping-reduction method
After presenting the requested experiment, we would like to clarify the positioning of SIS. The goal of SIS is not to reduce clipping, but to eliminate the off-policy mismatch that makes correction necessary in the first place.
Clipping and SIS act on different objects. Clipping bounds the importance ratio after the mismatch has occurred, and is active only for tokens outside the clipping range. SIS acts on the mismatch itself: a token-level acceptance–rejection test certifies whether a token can be treated as a sample from the current policy (Proposition 1), and an accepted token needs no correction at all (independently of whether its ratio would ever have been clipped. )
Following the same setup as Table 1, we analyze the first 24 gradient steps of GRPO+SIS on Qwen3-8B-Base for mathematical reasoning. We use a clipping range of [0.8, 1.2] for the importance ratio. We find that 1.55% of tokens have ratios outside this range, and SIS accepts 57.47% of these out-of-range tokens, allowing them to bypass clipping.
To assess their contribution, we remove these bypassed tokens from training and report the results as follow:
So, interpreting SIS primarily as a clipping-reduction method misunderstand our central contribution. As shown in this table, removing the bypassed tokens just leads to a modest performance drop. Thus, reducing clipping contributes only a small part of SIS’s gains.
Our core contribution is a rejection-sampling mechanism that converts off-policy tokens into on-policy tokens, removing their need for importance correction—even when their original IS ratios lie within the clipping range. Section 3.3 explains why this matters: these conversions reduce cumulative log-importance deviation and yield a strictly sharper upper bound on off-policy approximation error(Theorem 1, Proposition 3), which Figure 2(C) confirms empirically.