acceptodds
Under review as a conference paper at ICLR 2027

Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment

Abstract

Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded. A natural idea is "transferring" these off-policy token into on-policy token, so that the importance scores for correction are unnecessary. Following this idea, we propose Selective Importance Sampling (SIS), which is inspired by rejection sampling. Concretely, SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test: accepted tokens are viewed as on-policy, so that receive unit importance score, while rejected tokens retain the standard IS correction. Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators. The SIS acts as a plug-in that only modifies the importance ratio in the policy loss, adding negligible wall-clock overhead, and can be combine with a vast vary of RL post-training algorithms. Experiments on dense and MoE LLMs across math and agent benchmarks show that SIS consistently improves all objectives, while providing substantially stronger robustness under off-policy data.

Primary area:
reinforcement learning
Keywords:
RL Post-training, Rejection Sampling, Off-Policy Gradient

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_t24SOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets off-policy mismatch in RL post-training of LLMs, caused by rollout reuse, asynchronous rollouts and train–inference mismatch. It proposes Selective Importance Sampling (SIS). At each token, the behavior policy is treated as a proposal for the current policy, and an acceptance test is drawn with probability w_t/M_t, where w_t is the token importance ratio and M_t is the largest ratio over the vocabulary. Accepted tokens get weight one ("on-policy"); rejected tokens keep their ratio, which the base algorithm then processes. SIS is a plug-in for GRPO, DAPO and GSPO with small overhead. Experiments on dense and MoE Qwen3 models (and a Llama model) for math and search-agent tasks report average gains over each base algorithm.

Critical strengths
  1. Simple and cheap.

SIS only changes the per-token ratio, needs no extra forward pass, and adds about one percent wall-clock time. It is easy to adopt in existing GRPO-style pipelines.

  1. Useful lens and diagnostics.

Viewing off-policy correction as token-level rejection sampling with a computable top-K envelope is an interesting angle. Table 2's unified view of stabilization tricks as reweightings g(w), and the accept-rate and token-level analyses, make the method's behavior easy to inspect.

Critical weaknesses

Gains not separated from noise or truncation. There are no seeds or error bars, and several gains are about one point. K differs per algorithm without a validation protocol; at other K, GSPO+SIS only matches or trails GSPO (Table 3). Gains do not grow with staleness (Table 8). Please add seeds on one setting, the deterministic weight above, and a truncation baseline.

Questions for authors

If accepted tokens use π_θ/sg[π_θ], they bypass the clip. What fraction of accepted tokens have ratios outside the clip range, and how much of the gain does this un-clipping explain?

Reviewer_2LWEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

In LLM reinforcement learning, actor-policy divergence can lead to training instability and even collapse. To address this issue, the paper proposes Selective Importance Sampling (SIS), a technique that combines rejection sampling with importance sampling to mitigate the distribution mismatch between the rollout and training policies. To make the approach computationally efficient, the authors introduce a top-K approximation to the token distribution. The paper demonstrates that SIS can serve as a lightweight plug-in for popular RL objectives, including GRPO, DAPO, and GSPO, improving training performance and stability across various off-policy training scenarios.

Critical strengths
  1. The paper is clearly written and easily understandable.

  2. The paper provides comprehensive empirical evaluations across multiple model architectures, RL objectives, and benchmarks. Especially the model sizes, the paper train Qwen3-30BA3B and Qwen3-14B model on both math and searchr1-like settings.

Critical weaknesses
  1. Important seminar baseline not thoroughly discussed. Truncated importance sampling (TIS) is a very important baseline that is also lightweight, widely used, and effective. The paper does cited and refer to another closely related work CISPO, which is very similar to TIS but with a stop-gradient. Although the paper evaluates several off-policy correction methods, the improvements over strong existing alternatives are less convincing. In Table 2, CISPO achieves an average accuracy of 54.44, outperforming SIS combined with GRPO (52.59). The best-performing configuration, SIS combined with DAPO (55.14), only marginally exceeds CISPO, while also using a different optimization objective. This makes it difficult to attribute the improvement specifically to SIS. I encourage the authors to provide more controlled comparisons with strong existing correction methods, such as evaluating SIS on top of CISPO, to better establish the incremental benefit of the proposed approach.

  2. The paper doesn't cite or discuss the following paper, which is published before and also discuss the combination of rejection sampling and importance sampling. Although the math formulation in the paper is different, it is important for the authors to discuss the additional advantages of the SIS technique over Jackpot. Chen, Z., Liu, H., Zhou, Y., Zheng, H., & Chen, B. (2026). Jackpot: Optimal budgeted rejection sampling for extreme actor-policy mismatch reinforcement learning. arXiv preprint arXiv:2602.06107.

  3. The theoretical analysis establishes that SIS achieves a tighter upper bound on the approximation error than standard token-level importance sampling. However, this does not necessarily imply a reduction in the actual gradient estimation bias. In particular, although accepted tokens follow the target-policy distribution conditionally, the proposed estimator retains rejected tokens with their original importance weights. The expected modified importance weight therefore generally differs from the standard importance ratio, potentially introducing additional bias. For example, in the single-token setting, standard token-level importance sampling is already an unbiased estimator, whereas the proposed SIS modification can introduce bias. I encourage the authors to clarify this distinction and provide a more direct analysis or empirical evaluation of the bias–variance tradeoff, ideally comparing gradient estimation errors against an on-policy reference.

I strongly appreciate the authors' presentation of a huge and solid evaluation. However, my biggest issues remain. More controlled CISPO/TIS experiments with even more severe actor-policy mismatch to showcase SIS extra corrective effect and a clear Jackpot comparison could meaningfully change the assessment without requiring the authors to redo their entire evaluation.

Questions for authors
  1. The paper provides a tighter theoretical upper bound on gradient approximation error, but this does not necessarily imply lower actual estimation bias. I suggest including a controlled numerical study comparing the gradient bias, variance, and mean-squared error of SIS against standard importance sampling and other correction methods under varying degrees of policy divergence. A small-scale setting with an accurately computable reference policy gradient would help validate whether the proposed method achieves a better bias–variance tradeoff.
  2. Does SIS show empirical advantage over established techniques that are importance sampling only? TIS or CISPO. One more round of settings to present is CISPO + SIS (since the authors already run CISPO), which would directly test whether SIS provides incremental value over this strong alternative of relying on importance sampling only.
  3. How does SIS differ from previous techniques like Jackpot that also discuss merging rejection sampling to importance sampling? What additional advantage does SIS bring?
Reviewer_jx3oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective Importance Sampling (SIS) for off-policy reinforcement learning of LLMs. A token-level rejection test assigns unit importance weight to accepted tokens while retaining the original correction for rejected tokens. The paper introduces a top-K approximation, provides conditional sampling and gradient-bias results, and integrates SIS into GRPO, DAPO, and GSPO. Experiments cover dense and MoE models on mathematical reasoning and agentic search tasks.

Critical strengths
  1. SIS is simple and practical. It can be integrated into several existing RL objectives without an additional learned model, and the reported overhead is small in the profiled setting.
  2. The experiments cover multiple training objectives, model architectures, and task domains. The consistent average improvements, additional Llama results, and tests under rollout staleness provide useful evidence of practical value.
Critical weaknesses
  1. Missing comparisons with closely related methods. The paper cites SAPO and CFPO, but neither appears in Table 2. Both address training stability without hard clipping, which overlaps with the motivation for SIS. Comparing them under the same training setup would help clarify the practical benefit of SIS.
  2. The results in Table 1 are difficult to reconcile with Figure 4. Table 1 reports 27.40 on AIME25 for Qwen3-30B-A3B-Base with DAPO+SIS, whereas the corresponding curve in Figure 4 peaks at approximately 25.94. The paper does not explain how checkpoints were selected or whether the table and figure use the same runs and evaluation settings.
Questions for authors
  1. How does SIS compare with SAPO and CFPO using the same backbone, training data, and compute budget?
  2. How were checkpoints selected for Table 1? Please explain the difference from Figure 4, including whether the results use different runs, checkpoints, or evaluation protocols.
  3. Section 4.4 states that K=10K=10 is used throughout, but Tables 4–5 list different values. Which settings produced the main results, and how was KK selected?
2 comments
AuthorsOpenReview ↗

Clarification: SIS is not a clipping-reduction method

After presenting the requested experiment, we would like to clarify the positioning of SIS. The goal of SIS is not to reduce clipping, but to eliminate the off-policy mismatch that makes correction necessary in the first place.

Clipping and SIS act on different objects. Clipping bounds the importance ratio after the mismatch has occurred, and is active only for tokens outside the clipping range. SIS acts on the mismatch itself: a token-level acceptance–rejection test certifies whether a token can be treated as a sample from the current policy (Proposition 1), and an accepted token needs no correction at all (independently of whether its ratio would ever have been clipped. )

Following the same setup as Table 1, we analyze the first 24 gradient steps of GRPO+SIS on Qwen3-8B-Base for mathematical reasoning. We use a clipping range of [0.8, 1.2] for the importance ratio. We find that 1.55% of tokens have ratios outside this range, and SIS accepts 57.47% of these out-of-range tokens, allowing them to bypass clipping.

To assess their contribution, we remove these bypassed tokens from training and report the results as follow:

MethodMATH500AMC23AIME24AIME25Avg
GRPO80.474.1423.4418.2349.05
GRPO + SIS w/o bypassed tokens84.279.0124.8718.9051.75
GRPO + SIS85.679.5325.8319.3852.59

So, interpreting SIS primarily as a clipping-reduction method misunderstand our central contribution. As shown in this table, removing the bypassed tokens just leads to a modest performance drop. Thus, reducing clipping contributes only a small part of SIS’s gains.

Our core contribution is a rejection-sampling mechanism that converts off-policy tokens into on-policy tokens, removing their need for importance correction—even when their original IS ratios lie within the clipping range. Section 3.3 explains why this matters: these conversions reduce cumulative log-importance deviation and yield a strictly sharper upper bound on off-policy approximation error(Theorem 1, Proposition 3), which Figure 2(C) confirms empirically.

AuthorsOpenReview ↗

Request for Replying

Dear reviewer:

As the discussion period is drawing to a close, we would greatly appreciate it if you could review our responses above and let us know whether they address your concerns. We would be happy to discuss any remaining questions and provide further clarification.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Call for AC's Discretion (Thanks to all the reviewer for your attention on SIS)

We thank all three reviewers for their careful reading of our paper and the Area Chair for guiding the discussion. We especially appreciate the follow-up comments from Reviewers t24S and 2LWE, both of whom raised their ratings to 3 after considering the additional evidence. All three reviews recognized the practical value of SIS as a simple yet effective approach to off-policy training. To help the Area Chair and reviewers navigate the discussion, we summarize the main clarifications and conclusions from the additional experiments below:

We clarified how SIS differs from Jackpot. Although SIS and Jackpot both use token-level importance ratios for rejection sampling. Beyond this, We further prove that, under this acceptance rule, accepted tokens is on policy and therefore require no further importance correction. Jackpot still recalculates the IS weights of the accepted tokens. Moreover, SIS achieved higher average accuracy than Jackpot in both the standard training setting and the multi-seed comparison under high rollout staleness. (2LWE, W2/Q3 and follow-up, point 1/3)

With the base objective held fixed, SIS improved accuracy on all four benchmarks (2LWE, W1/Q2), and the multi-seed comparison confirmed an average gain at fixed KK (t24S, W1). Most of the gain persisted after removing tokens that bypassed clipping (t24S, Q1). Across three checkpoint pairs, SIS reduced gradient variance and brought the estimated gradient closer to the sequence-level importance-sampling reference (2LWE, W3/Q1 and follow-up, point 2).

The original submission already included CISPO in Table 2 as a soft-clipping baseline. At Reviewer jx3o’s request, we added direct comparisons with two further baselines, SAPO and CFPO, under the same training setup. All tested SIS variants achieved higher average accuracy than both (jx3o, W1/Q1). GRPO+SIS also outperformed GRPO+TIS on all four benchmarks (2LWE, W1/Q2). Finally, we corrected the description of KK, explained how checkpoints were selected, and clarified that the Table 1 result corresponds to the DAPO+SIS curve with R3 in Figure 4 (jx3o, W2/Q2–Q3).

We hope this summary helps the reviewers and the Area Chair assess the contribution of SIS. By identifying which off-policy tokens can be used without importance correction, SIS improves existing RL objectives under both standard and high rollout staleness with negligible training overhead. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.