- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses token-level probability discrepancies between training and inference engines, which can be further amplified in MoE models by differences in expert routing. Directly using these mismatched probabilities to compute importance-sampling weights can produce anomalous gradients, destabilize training, and even cause training to collapse. To mitigate this problem, the paper proposes KPop, which filters unreliable tokens using a symmetric binary KL-divergence criterion. Experiments evaluate KPop on a 4B dense model and MoE models with up to 100B total parameters.
Critical strengths
- Addresses an important practical issue. Even when the inference and training engines use identical model weights, they can assign different probabilities to the same token; discrepancies may be amplified by MoE routing. The paper introduces filtering rules to mitigate this problem.
- Provides a clear theoretical rationale. The analysis explains why IcePop disproportionately filters low-probability tokens and derives KPop’s wider tolerance for such tokens.
- Incurs little additional computational cost. KPop uses token probabilities already computed by the inference and training engines, requiring neither an additional forward pass nor full-vocabulary logits.
Critical weaknesses
- DPPO outperforms KPop on the dense model, achieving 70.2/61.4/38.3 compared with KPop’s 64.0/58.0/27.0 on AIME24/AIME25/HMMT25. Would combining the two methods improve performance?
- The optimal appears to vary across models. Does this mean that needs to be tuned separately for each model?
- Could the MoE evaluation include additional baselines, such as DPPO and Seq-TIS?
Questions for authors
Please refer to the weaknesses discussed above.
Official Comment by Reviewer B9mh
Thank you for your response. I still have one remaining concern regarding dense models: why do the two engines agree on almost every token? Moreover, if there is no significant training–inference mismatch in dense models, why does KPOP still outperform the baseline?