acceptodds
Under review as a conference paper at ICLR 2027

Bounding Token Divergence to Unlock Stable Reinforcement Learning for Large Language Models

Abstract

Decoupled reinforcement learning from verifiable rewards (RLVR) relies on separate inference and training engines that, despite sharing identical weights, assign different token-level probabilities due to floating-point non-determinism and stochastic expert routing. The resulting importance-sampling corruption destabilizes policy-gradient training, particularly for Mixture-of-Experts (MoE) models. We first introduce IcePop, a token-level ratio mask that eliminates extreme discrepancies between inference and training probabilities and restores training stability. Analysis reveals, however, that any constant ratio threshold over-masks low-probability tokens because the softmax partition function attenuates perturbations for frequent tokens while leaving rare tokens disproportionately vulnerable to them. We then propose KPop, which replaces the fixed ratio window with a symmetric binary KL divergence criterion. The binary KL naturally widens its tolerance at low token probability and tightens it at high probability, matching the heteroscedastic noise structure. The symmetric formulation further eliminates directional leakage inherent in one-sided constraints. Experiments on dense and MoE architectures across mathematical reasoning benchmarks show that KPop retains significantly more rare-token gradient signal than existing methods while improving both training stability and final evaluation accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_B9mhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses token-level probability discrepancies between training and inference engines, which can be further amplified in MoE models by differences in expert routing. Directly using these mismatched probabilities to compute importance-sampling weights can produce anomalous gradients, destabilize training, and even cause training to collapse. To mitigate this problem, the paper proposes KPop, which filters unreliable tokens using a symmetric binary KL-divergence criterion. Experiments evaluate KPop on a 4B dense model and MoE models with up to 100B total parameters.

Critical strengths
  1. Addresses an important practical issue. Even when the inference and training engines use identical model weights, they can assign different probabilities to the same token; discrepancies may be amplified by MoE routing. The paper introduces filtering rules to mitigate this problem.
  2. Provides a clear theoretical rationale. The analysis explains why IcePop disproportionately filters low-probability tokens and derives KPop’s wider tolerance for such tokens.
  3. Incurs little additional computational cost. KPop uses token probabilities already computed by the inference and training engines, requiring neither an additional forward pass nor full-vocabulary logits.
Critical weaknesses
  1. DPPO outperforms KPop on the dense model, achieving 70.2/61.4/38.3 compared with KPop’s 64.0/58.0/27.0 on AIME24/AIME25/HMMT25. Would combining the two methods improve performance?
  2. The optimal δ\delta appears to vary across models. Does this mean that δ\delta needs to be tuned separately for each model?
  3. Could the MoE evaluation include additional baselines, such as DPPO and Seq-TIS?
Questions for authors

Please refer to the weaknesses discussed above.

1 comment
Reviewer_B9mhOpenReview ↗

Official Comment by Reviewer B9mh

Thank you for your response. I still have one remaining concern regarding dense models: why do the two engines agree on almost every token? Moreover, if there is no significant training–inference mismatch in dense models, why does KPOP still outperform the baseline?

Reviewer_dFTHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studied the problem of train-inference mismatch. The authors develop on top of an existing algorithm IcePop's token-level mask part. IcePop's token-level mask defines a fixed region for importance sampling ratio, which will inevitably eliminating more low training probability token, as it is the denominator and provide larger range of the IS ratio, and it can be verified through experiments. The authors believe those low probability tokens have critical value in training and should relax the condition of elimination/masking out. The author proposed KPop that uses minimum of reverse and forward binary KL divergence to measure the rollout probability and training probability, whose local geometry determines more relaxation on the small training probability mask. Experiments show that KPop is more efficient than IcePop and GRPO, and more stable compared to GRPO.

Critical strengths

The TIM problem is an important problem in LLM RL training. Works that can effectively address it is thus important. Figure 3 is very illustrative and intuitively shows the difference between KPop and IcePop. The KPop indeed shows improvement compared to the IcePop baseline and also maintains stable during the training.

Critical weaknesses

I don't see much of the value in Section 4.1's analysis, whose bound is also not taken in the proposed KPop algorithm. If it is to justify that more low probability tokens will be eliminated with fixed clipping ratio, the empirical results already suffice.

There are many ways to relax the clipping ratio, if the only advantage of KL compared to the σ(1−p)\sigma (1-p) derived from the noise scale analysis is it adds more relaxation with order (1−p)/p\sqrt{ (1- p) / p }, simply consider |p - q| < \delta can relax it even more. I don't see the reason why not continue such relaxation.

I hate to say this but there are too many existing works on TIM with update of clipping, thus sufficient ablation is needed to really show the necessity of a new incremental method. The essential trade-off of efficiency and stability is what the community need, e.g., what is the actual desired probability-dependent clipping region.

Questions for authors

If we look at Fig. 3 closely, the IcePop scatters seem to be a bit more concentrated than that of KPop. Should we compare with the baseline IcePop with more relaxed clipping region?

To comprehensively comparing KPop with IcePop, it would be useful to run with a sweep of hyperparameters, which will show actual efficiency v.s. stability trade-off by those two algorithms, that will truly show the advantage of the geometry of KL confined region performing better than linear boundary shape in IcePop.

Reviewer_6NR1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Current RL training usually relies on different inference and training frameworks, for example, SGLang for inference and Megatron for training. But even if they use the same model weights, differences in kernels and MoE routing can make them output different probabilities for the same token. This makes training unstable and sometimes causes collapse. Overall I agree this is an important problem in MoE posttraining.

Existing methods such as IcePop filter out tokens with extreme probability ratios. However, the authors find that IcePop tends to filter out more low-probability tokens, and these tokens might be important for RL exploration. Therefore, KPop proposes a new token filtering method by replacing the fixed ratio threshold with symmetric binary KL. Experiments show that KPop can keep training stable and achieve better accuracy than IcePop.

Critical strengths
  1. The motivation is clear and easy to follow. Section 4 analyzes the unbalanced filtering problem in IcePop. A fixed ratio threshold is more likely to filter low probability tokens. The authors try to explain this phenomenon through the mechanism of softmax.

  2. The authors derive the relationship between token probability and log ratio noise. They also verify this relationship with different models and noise settings. These analyses provide a theoretical motivation for using a probability adaptive threshold, instead of simply trying different masking functions. Besides, the authors analyze the directional leakage problem of one sided binary KL. This further explains why they choose symmetric binary KL.

  3. The proposed method is simple and easy to implement. It replaces the fixed ratio threshold in IcePop with symmetric binary KL. It can adjust the tolerance based on token probability and check the probability differences in both directions. More importantly, it only needs the token probabilities already computed by the inference and training engines. It does not require extra forward passes or the full vocabulary probability distribution. Therefore, it can be easily added to existing RL training frameworks.

Critical weaknesses
  1. In section 4, the authors claims that IcePop filters too many low probability tokens, and KPop can keep more of them. However, the experiments do not rule out a simpler explanation. Maybe the threshold of IcePop is too strict, so it removes some useful training signals. If we adjust the ratio threshold of IcePop to keep the same percentage of tokens as KPop, can it achieve similar results? I think this comparison is worth to be the baseline and important to show the real benefit of KPop.

  2. The authors show that low probability tokens are more sensitive to softmax. Therefore, a fixed ratio threshold is more likely to filter these tokens. However, a larger ratio difference does not mean that the difference is safe. It also does not mean that keeping these tokens will always improve training. The authors argue that low probability tokens are important for RL exploration. But they do not directly analyze whether the tokens filtered by IcePop but kept by KPop really provide more reliable or useful gradients. I think more analysis is needed to support this claim.

  3. All training experiments use only one random seed, so it is hard to know whether the performance improvements are stable. Also, the paper mainly reports the best checkpoint accuracy. Since training stability is an important motivation of this paper, I think the authors should also report the final checkpoint performance, variance across different seeds, and training collapse rate. Besides, Section 5.5 argues that KPop can be combined with DPPO. However, the authors only compare them separately on a dense model. They do not test the combined method, especially on MoE models. Since DPPO performs much better than KPop on the dense model, I think it would be useful to verify whether combining them can bring further improvements.

Questions for authors

see weakness

Comments on the paper

1 comment

Dear Reviewers,

Thank you for the review. The authors have responded to the points you raised, and the response period closes on 18 November (AoE). I would be grateful if you could post a brief reply on whether your concerns are resolved, partly resolved, or still open, so that the authors have a fair chance to address them. If anything still needs clarifying, please ask now while they can still respond.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.