acceptodds
Under review as a conference paper at ICLR 2027

Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning

Abstract

Reinforcement learning for large language models often uses separate systems for inference and training. Even with identical weights and BF16 tensors, implementation differences can make an intended on-policy update off-policy and destabilize training. Existing remedies align the entire inference and training stacks, change global precision, or modify the learning algorithm. We ask whether aligning selected parts of the execution stack can stabilize BF16 RL while other implementation differences remain. We align training-side rounding locations with inference in three regions—RoPE, SwiGLU, and residual RMSNorm—by retaining selected intermediates in FP32 until their BF16 outputs. Across four Qwen-family models, aligning all three regions prevents the collapse observed under the default BF16 baseline; RoPE-path alignment alone also suffices in the studied runs, despite remaining numerical discrepancies. The interventions keep the inference engine, matrix-multiplication and attention kernels, and learning objective unchanged.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning, Training–Inference Mismatch

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CWCSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how numerical discrepancies between LLM inference and training engines can destabilize reinforcement learning, even when identical model weights and BF16 precision are used. The authors focus on rounding-location mismatches introduced by kernel fusion and examine three computation regions: RoPE, SwiGLU, and residual RMSNorm. They propose selectively aligning training-side rounding locations with inference by retaining intermediate computations in FP32 while preserving the original inference engine, major kernels, and learning objective. Experiments across four Qwen-family models show that aligning all three regions prevents training collapse, while RoPE-only alignment is sufficient in the evaluated settings. Additional experiments quantify the effects of individual regions on training–inference discrepancies, analyze BF16 rounding characteristics, and measure the computational overhead of the proposed interventions.

Critical strengths

Interesting observation with practical relevance. The paper shows that aligning only the RoPE rounding locations can prevent RL training collapse without requiring complete numerical consistency between inference and training engines. This finding is practically useful, as it suggests that targeted numerical alignment may be sufficient to improve training stability with relatively low computational overhead.

Critical weaknesses

1. Limited methodological novelty. Prior work, particularly Panda (2026), has already identified and addressed rounding-location mismatches in RoPE, SwiGLU, and normalization operations. This paper largely reuses these established techniques, with its main contribution being an empirical ablation showing that RoPE-only alignment can prevent training collapse. While this observation is useful, the incremental contribution appears insufficiently novel for ICLR.

2. Limited generalizability to real-world LLM RL training. The experiments are restricted to Qwen-family models, mathematical reasoning tasks, and a specific vLLM–Hugging Face training stack. It remains unclear whether RoPE-only alignment is effective across other model architectures, inference engines, and large-scale RL training configurations. Consequently, the results do not provide sufficient evidence that the proposed approach generalizes to practical LLM RL systems.

Questions for authors

1. Clarification of novelty. Prior work, particularly Panda (2026), has already identified and addressed rounding-location mismatches in RoPE, SwiGLU, and normalization operations. Could the authors clarify the main methodological novelty of this paper beyond applying existing alignment techniques and conducting component-level ablations? What fundamentally new insights does this work provide over prior studies?

2. Generalizability of the findings. The experiments focus exclusively on Qwen-family models and mathematical reasoning tasks. What evidence supports the effectiveness of RoPE-only alignment in other model architectures or more realistic large-scale RL training settings?

3. Understanding the stabilization mechanism. The paper shows that RoPE-only alignment prevents training collapse despite relatively modest reductions in overall training–inference KL divergence. Could the authors explain why correcting RoPE-related discrepancies is particularly effective compared with other numerical mismatches? Is there evidence that this observation reflects a general mechanism rather than an implementation-specific effect?

1 comment

Thank you for your comments on our contribution, the scope of our results, and the stabilization mechanism.

Q1: Contribution. Our contribution is controlled evidence that selective alignment can stabilize BF16 RL while other numerical differences between training and inference remain. Our execution-mode controls, zero-mismatch reference, and targeted interventions support this approach, which preserves the existing engines, major kernels, and learning objective. Full-stack alignment, as implemented by Panda (2026), involves substantial engineering to obtain identical token log probabilities, including writing or modifying kernels for many operations and implementing custom backward passes. Our selective alignment requires no new or rewritten kernels: it adjusts intermediate precision and rounding points using existing operations. We now state the selective-alignment finding explicitly in the Abstract, Introduction, and Discussion.

Q2: Scope. Our evidence covers four Qwen-family models and two mathematical-reasoning datasets. Within these settings, RoPE-path alignment prevents the degradation observed in the BF16 baselines. The Discussion explains this scope. Evaluating other architectures, engines, and larger training configurations remains future work.

Q3: Stabilization mechanism. A small KL reduction can matter if it removes errors that strongly affect model updates. The Discussion presents this as a possible explanation for RoPE's effectiveness in the tested settings. Section 7 and Appendix E now strengthen the empirical comparison: RoPE-path alignment remains stable, while FP32 LM-head computation and token-level truncated importance sampling show training degradation.

Reviewer_QTPnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies numerical disagreement between inference and training in LLM reinforcement learning. Nominally BF16 implementations can round at different points: fused operations may retain FP32 intermediates where separate operations store BF16 values. The authors align training-side rounding in the RoPE path, SwiGLU, and residual RMSNorm, keeping the inference engine, attention and matrix-multiplication kernels, and RL objective fixed. Across four Qwen-family models, aligning all three regions prevents the baseline's collapse; aligning the RoPE path alone, including Q/K normalization where present, also suffices. Execution-mode controls, static discrepancy measurements, a zero-mismatch backend, and an additional DAPO experiment support the result. A separate analysis characterizes the BF16 cell phases of intermediate values.

Critical strengths
  1. The execution-mode controls give the stability result a solid experimental basis. Both matched modes train stably, while both crossed modes collapse. Holding eager training fixed and changing inference from compiled to eager execution restores stability without increasing training-side precision. The zero-mismatch backend provides a useful reference, and the targeted interventions show that eliminating every numerical discrepancy is unnecessary in these runs. Keeping the objective fixed makes the comparisons easy to interpret.

  2. The intervention offers a practical alternative to aligning the entire execution stack. RoPE-path alignment works for all four tested models, with a further successful run on DAPO using DeepSeek-R1-Distill-Qwen-1.5B. Its reported forward-and-backward overhead is only 0.90–2.04%, with unchanged peak allocated memory; this measurement excludes rollouts and optimizer updates. The fixed-weight analysis is also useful: RoPE alignment reduces estimated KL by only 3.4–22.1%, yet the subsequent RL runs remain stable. The supplement supplies version-pinned dependencies, launch scripts, and the targeted implementation.

Critical weaknesses
  1. The reliability of the stability result across independent runs is unclear. Figures 1, 3, and 6 give trajectories without a reported training-seed count or between-run variation. The repeated AIME generations in Appendix A.2 measure evaluation variability, not variation in whether training collapses. The consistent separation across model sizes and the additional dataset is encouraging. Still, for a paper whose main contribution is preventing collapse, the frequency of that outcome is an important missing result. Coverage is also confined to Qwen-family models and mathematical reasoning, as the authors acknowledge.

  2. The analysis offers limited guidance on which regions need alignment. Shapley attribution measures changes in the static KL estimate, but SwiGLU-only and residual-RMSNorm-only alignment are not tested during RL. It is therefore unclear whether RoPE is a particularly important region or one of several small interventions that would suffice. The Fourier measurements do not resolve this: residual sums have stronger phase concentration than RoPE products, and Appendix A.5 correctly explains why concentration alone does not determine rounding error. The authors present these measurements as descriptive, which is appropriate. My concern is the resulting limit on the paper's explanatory contribution, especially since matching RoPE and SwiGLU rounding is already described in prior work (Panda, 2026). The controlled evidence for selective sufficiency remains valuable.

Questions for authors
  1. How many independent training runs underlie Figures 1, 3, and 6, and how was collapse identified? If repetitions are available, please report collapse frequency and variation in final accuracy.
  2. Have you tested SwiGLU-only or residual-RMSNorm-only alignment in RL? Their stability outcomes would help assess how informative the static attribution is for choosing an intervention.
  3. How uncertain are the paired KL differences in Table 1? Response- or prompt-level confidence intervals would be particularly helpful for the 3.4% RoPE-path reduction on Qwen3-1.7B.
3 comments

Thank you for your questions on experiments, region selection, and KL uncertainty.

Q1: Independent runs. Figures 1, 3, and 6 show one run per condition. Limited computing resources prevented us from repeating these experiments before submission. Appendix E now reports three runs per condition on DeepSeek-R1-Distill-Qwen-1.5B: RoPE-path alignment remains stable in all three, while the baseline, FP32 LM-head computation, and token-level truncated importance sampling (token TIS) each show degradation in all three. RoPE runs average 26.92% AIME 2025 accuracy (sample standard deviation across runs: 1.26 percentage points). Appendix A.2 now defines training degradation as a sustained decline in training accuracy, abnormal growth in training–inference KL, or abnormal outputs such as excessive repetition.

Q2: Region selection. The stability of SwiGLU-only and residual-RMSNorm-only alignment remains to be evaluated. The general starting point for applying our method is to align all regions with mismatched rounding locations. In our models, these are RoPE, SwiGLU, and residual RMSNorm; aligning all three stabilizes training with modest overhead. After verifying stability, smaller subsets can be tested to reduce cost, as illustrated by RoPE-path alignment in our evaluations. We have rewritten the Takeaway around this procedure.

Q3: KL uncertainty. We report five additional measurements here. Each measurement samples new responses to the same prompts. The table shows the mean results for RoPE-path alignment. As in Table 1, we report estimated per-token KL from vLLM to Hugging Face. All KL values are in 10−410^{-4} nats/token.

ModelBaseline KLRoPE-path KLReduction (%)KL reduction [95% confidence interval]
DeepSeek-R1-Distill-Qwen-1.5B6.39695.038421.2%1.3584 [1.0481, 1.6688]
Qwen3-1.7B7.66957.04178.2%0.6277 [0.2329, 1.0225]
Qwen3-8B-Base4.06533.75967.5%0.3057 [0.0064, 0.6049]

KL reduction is baseline minus RoPE-path KL; a positive value means improvement. The percentage is the mean reduction divided by the mean baseline KL. The confidence intervals are calculated from the five baseline-minus-RoPE differences using Student's tt-distribution. All three intervals are above zero. For Qwen3-1.7B, the new mean reduction is 8.2%, rather than the original 3.4% estimate in Table 1.

Thank you for the rebuttal response. However, for such a paper, where numerical accuracy is discussed, I think it's very crucial to provide very comprehensive ablations or experiments, otherwise the conclusion may be false positive. Thanks for the detialed response of the authors for adding additional experiments, e.g., regarding region selection, KL uncertainty, etc. While I choose to maintain my score, since it's fair enough.

Thank you for the follow-up. We agree that controlled ablations are important for assessing the reliability of the result.

Region selection. Section 7 and Appendix E.1, Figure 8, now compare the three single-region interventions on DeepSeek-R1-Distill-Qwen-1.5B. SwiGLU-only and residual-RMSNorm-only interventions both exhibit training degradation, while RoPE-path alignment remains stable. This supports the importance of region selection and agrees with the static attribution in Section 6.

Reviewer_sKHcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training-inference mismatch in BF16 LLM reinforcement learning. The authors identify three regions where fused inference kernels and eager training implementations round intermediate values differently: RoPE, SwiGLU, and residual RMSNorm. They show that aligning these rounding locations on the training side prevents RL collapse across four Qwen models, with RoPE alignment alone being sufficient in their experiments. The paper also includes a compiled/eager comparison and an analysis of how each region contributes to policy mismatch.

Critical strengths
  1. The compiled/eager experiment in Section 4 is interesting. Both matched configurations train stably, while the mismatched configurations collapse. In particular, eager/eager training is stable without increasing precision. This provides good evidence that execution differences matter, rather than BF16 precision alone.

  2. The proposed fix is simple and practical. RoPE-only alignment adds around 1-2% training-computation overhead and stabilizes all four models in the reported experiments. The zero-mismatch baseline is also a useful control, since it shows that complete numerical agreement is not necessary for stability.

Critical weaknesses
  1. The stability claims need stronger experimental evidence. Figure 3 appears to report single runs, with no variance across seeds. This is particularly concerning for Qwen3-1.7B, where RoPE alignment reduces static KL by only 3.4%, and the KL trajectories are very similar until the baseline collapses. Multiple seeds are needed to establish whether this is a reliable improvement.

  2. The mechanism is not sufficiently isolated. The intervention both increases intermediate precision and changes rounding locations. The compiled/eager comparison is suggestive, but compilation can change more than rounding behavior. The paper also does not provide direct numerical comparisons of intermediate outputs between the engines before and after alignment. Interestingly, SwiGLU-only alignment increases KL for two models in Table 1, which is not adequately explained.

The comparison with existing remedies is also limited. In particular, MiniMax-M1 and ScaleRL discuss FP32 computation at the LM head, which seems directly relevant here. It is unclear whether the main experiments already use FP32 logits, and whether that simpler intervention would be sufficient. The paper also does not compare against FP16 or importance-sampling correction under the same setup.

Questions for authors
  1. How many seeds were used for Figure 3? Can the authors provide the distribution of collapse steps for the baseline and RoPE-only alignment?

  2. Can the authors provide direct measurements of intermediate-output agreement with vLLM before and after alignment? Does using FP32-RoPE training with eager vLLM change the stability results?

  3. Why does SwiGLU-only alignment increase KL for two models in Table 1?

  4. What precision is used for LM-head computation and log-softmax in the main RL experiments? Does FP32 LM-head computation alone stabilize training?

  5. How does RoPE alignment compare against FP16 or importance-sampling correction on the same setup?

2 comments

Thank you for your questions about training reliability, the alignment mechanism, and comparisons with other methods.

Q1: Training reliability. Figure 3 reports one run per condition. Limited computing resources prevented us from repeating the training experiments before submission. Appendix E (Figure 7) now adds three runs per condition on DeepSeek-R1-Distill-Qwen-1.5B. All three RoPE-path runs remain stable; the three BF16 baselines show sustained accuracy degradation, detected at steps 585, 614, and 714. Appendix A.2 now defines training degradation as a sustained decline in training accuracy, abnormal growth in training–inference KL, or abnormal outputs such as excessive repetition.

Q2: Mechanism. We use FP32 to make training round intermediate values at the same points as fused inference. The control in Section 4 also shows that stability can improve without increasing training precision: keeping eager training unchanged and switching inference from compiled to eager restores stability. Interestingly, FP32-RoPE training with eager vLLM also remains stable, although the reason is still unclear. The requested before/after comparisons of intermediate values remain future work.

Q3: SwiGLU and KL. One possible explanation is that rounding errors from different operations partly cancel: changing one operation can then increase or decrease the total mismatch. We conducted five additional measurements. Each measurement samples new responses to the same prompts. In these measurements, the effect of SwiGLU-only alignment on Qwen3-1.7B remains uncertain, while KL decreases on Qwen3-8B-Base. As in Table 1, we report estimated per-token KL from vLLM to Hugging Face. All KL values below are in 10−410^{-4} nats/token.

ModelBaseline KLSwiGLU-only KLReduction (%)KL reduction [95% confidence interval]
DeepSeek-R1-Distill-Qwen-1.5B6.39696.29911.5%0.0978 [-0.2288, 0.4244]
Qwen3-1.7B7.66957.6981-0.4%-0.0286 [-0.3031, 0.2459]
Qwen3-8B-Base4.06533.87204.8%0.1933 [0.0647, 0.3218]

KL reduction is baseline minus SwiGLU-only KL; a positive value means improvement. The percentage is this reduction divided by baseline KL. The confidence intervals are calculated from the five baseline-minus-SwiGLU differences using Student's tt-distribution.

Q4–5: Precision and comparisons. We clarified in Appendix A.2 that the baseline uses BF16 LM-head matrix multiplication and FP32 log-softmax. Section 7 and Appendix E now compare FP32 LM-head computation in both training and inference, and token-level truncated importance sampling (token TIS). Both controls show training degradation, while RoPE-path alignment remains stable.

For FP16, Qi et al. (2025) demonstrate stable RL training by switching the global training and inference precision to FP16. Its narrower numerical range can require loss scaling and synchronized overflow checks, adding complexity to distributed training. Our approach retains BF16 as the global format and aligns selected intermediate computations. A comparison with FP16 under our settings remains future work.

We have added Qwen3-1.7B repetitions and numerical comparisons.

Reliability and KL diagnostics. Appendix E.2, Figures 9–10, adds three Qwen3-1.7B runs per condition. All three RoPE-path runs remain stable; two baseline runs are stopped after repetitive outputs, with one also showing accuracy degradation. AIME 2025 accuracy is 32.44 ± 1.88% versus 25.83 ± 3.61%, respectively.

The reported KL averages inference–training discrepancy over all response tokens. The degenerate runs produce long repetitive tails on which the two implementations largely agree. As these low-discrepancy tokens occupy a larger share of the response, the average KL falls even though output quality deteriorates. Response length and output audits therefore help distinguish this degeneration from improved stability.

Compilation. With arithmetic operations and elementary-function approximations held fixed, compilation primarily changes rounding behavior and reduction order. Section 5.1 explains this distinction and directly tests the rounding change in RoPE on identical inputs: compiled outputs differ from BF16 eager outputs but exactly match eager computation with FP32 products and sums before the final BF16 cast.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.