- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how numerical discrepancies between LLM inference and training engines can destabilize reinforcement learning, even when identical model weights and BF16 precision are used. The authors focus on rounding-location mismatches introduced by kernel fusion and examine three computation regions: RoPE, SwiGLU, and residual RMSNorm. They propose selectively aligning training-side rounding locations with inference by retaining intermediate computations in FP32 while preserving the original inference engine, major kernels, and learning objective. Experiments across four Qwen-family models show that aligning all three regions prevents training collapse, while RoPE-only alignment is sufficient in the evaluated settings. Additional experiments quantify the effects of individual regions on training–inference discrepancies, analyze BF16 rounding characteristics, and measure the computational overhead of the proposed interventions.
Critical strengths
Interesting observation with practical relevance. The paper shows that aligning only the RoPE rounding locations can prevent RL training collapse without requiring complete numerical consistency between inference and training engines. This finding is practically useful, as it suggests that targeted numerical alignment may be sufficient to improve training stability with relatively low computational overhead.
Critical weaknesses
1. Limited methodological novelty. Prior work, particularly Panda (2026), has already identified and addressed rounding-location mismatches in RoPE, SwiGLU, and normalization operations. This paper largely reuses these established techniques, with its main contribution being an empirical ablation showing that RoPE-only alignment can prevent training collapse. While this observation is useful, the incremental contribution appears insufficiently novel for ICLR.
2. Limited generalizability to real-world LLM RL training. The experiments are restricted to Qwen-family models, mathematical reasoning tasks, and a specific vLLM–Hugging Face training stack. It remains unclear whether RoPE-only alignment is effective across other model architectures, inference engines, and large-scale RL training configurations. Consequently, the results do not provide sufficient evidence that the proposed approach generalizes to practical LLM RL systems.
Questions for authors
1. Clarification of novelty. Prior work, particularly Panda (2026), has already identified and addressed rounding-location mismatches in RoPE, SwiGLU, and normalization operations. Could the authors clarify the main methodological novelty of this paper beyond applying existing alignment techniques and conducting component-level ablations? What fundamentally new insights does this work provide over prior studies?
2. Generalizability of the findings. The experiments focus exclusively on Qwen-family models and mathematical reasoning tasks. What evidence supports the effectiveness of RoPE-only alignment in other model architectures or more realistic large-scale RL training settings?
3. Understanding the stabilization mechanism. The paper shows that RoPE-only alignment prevents training collapse despite relatively modest reductions in overall training–inference KL divergence. Could the authors explain why correcting RoPE-related discrepancies is particularly effective compared with other numerical mismatches? Is there evidence that this observation reflects a general mechanism rather than an implementation-specific effect?
Thank you for your comments on our contribution, the scope of our results, and the stabilization mechanism.
Q1: Contribution. Our contribution is controlled evidence that selective alignment can stabilize BF16 RL while other numerical differences between training and inference remain. Our execution-mode controls, zero-mismatch reference, and targeted interventions support this approach, which preserves the existing engines, major kernels, and learning objective. Full-stack alignment, as implemented by Panda (2026), involves substantial engineering to obtain identical token log probabilities, including writing or modifying kernels for many operations and implementing custom backward passes. Our selective alignment requires no new or rewritten kernels: it adjusts intermediate precision and rounding points using existing operations. We now state the selective-alignment finding explicitly in the Abstract, Introduction, and Discussion.
Q2: Scope. Our evidence covers four Qwen-family models and two mathematical-reasoning datasets. Within these settings, RoPE-path alignment prevents the degradation observed in the BF16 baselines. The Discussion explains this scope. Evaluating other architectures, engines, and larger training configurations remains future work.
Q3: Stabilization mechanism. A small KL reduction can matter if it removes errors that strongly affect model updates. The Discussion presents this as a possible explanation for RoPE's effectiveness in the tested settings. Section 7 and Appendix E now strengthen the empirical comparison: RoPE-path alignment remains stable, while FP32 LM-head computation and token-level truncated importance sampling show training degradation.