acceptodds
Under review as a conference paper at ICLR 2027

Defeating the Training-Inference Mismatch via FP16

Abstract

Reinforcement learning (RL) fine-tuning of large language models (LLMs) often suffers from instability due to the numerical mismatch between the training and inference policies. While prior work has attempted to mitigate this issue through algorithmic corrections or engineering alignments, we show that floating point precision is a major and controllable contributing factor. The widely adopted BF16, despite its large dynamic range, introduces large rounding errors that break the consistency between training and inference. In this work, we demonstrate that simply reverting to **FP16** substantially reduces this mismatch. The change is simple, fully supported by modern frameworks with only a few lines of code change, and requires no modification to the model architecture or learning algorithm. Our results suggest that using FP16 uniformly yields more stable optimization, faster convergence, and stronger performance across diverse tasks, algorithms and frameworks. We hope these findings motivate a broader reconsideration of precision trade-offs in RL fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Large Language Models, Training-Inference Mismatch, FP16

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_j4wwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript presents an empirical study that establishes an effective yet simple solution to the training-inference mismatch problem of RL fine-tuning of LLMs. The main discovery is that using FP16 instead of BF16 can significantly improve the stability and performance of RL fine-tuning.

Critical strengths

The main strength of the work is the clear message conveyed by the experiments: changing BF16 to FP16 could be an easy but very effective solution to the training-inference mismatch problem of RL. BF16 is currently the dominant precision format used by the community, so this work should be inspiring and potentially have a large impact.

Critical weaknesses

The current conclusion is mainly based on end-point performances within a small number of setups. Though it is clear that FP16 outperforms BF16 consistently in most cases, the current investigation is still a bit superficial. In order to prove that the performance improvement is mainly due to a better matching between training and inference, more in-depth investigations are needed. For example, it is also possible that the performance gain is mainly due to the higher precision of gradient calculation or a more consistent inference results because of the improved precision. How to detach the effect of improved precision from the effect of better training-inference match is critical to further understand the phenomenon.

Questions for authors
  1. Do you think it is feasible to further explore the best tradeoff between exponent bits and mantissa bits. It is possible that there exists a better bit allocation other than both BF16 and FP16 for RL fine-tuning. This could be an interesting future direction.
  2. The improvement of using FP16 is relatively smaller in MOE models. What could be the possible causes?
2 comments

We thank the reviewer for the insightful and valuable feedback. We respond to individual points from your review below.

Weakness: In order to prove that the performance improvement is mainly due to a better matching between training and inference, more in-depth investigations are needed. For example, it is also possible that the performance gain is mainly due to the higher precision of gradient calculation or a more consistent inference results because of the improved precision. How to detach the effect of improved precision from the effect of better training-inference match is critical to further understand the phenomenon.

Section 4.3 was actually intended to conduct this ablation. In Figure 5, FP16 for training/inference only collapse eventually, while FP16 for both training and inference improves stably. Additionally, Figure 3 shows that training-inference mismatch is an important indicator for training collapse. So we put most of our focus on training-inference mismatch.

Q1: Do you think it is feasible to further explore the best tradeoff between exponent bits and mantissa bits. It is possible that there exists a better bit allocation other than both BF16 and FP16 for RL fine-tuning. This could be an interesting future direction.

Thanks for this interesting question. We think it is possible to have a better bit allocation. However, it requires hardware support for such precision formats, which is beyond the scope of this work and could be an interesting future direction.

Q2: The improvement of using FP16 is relatively smaller in MOE models. What could be the possible causes?

For MoE models, the top-k router for expert selection is discontinuous, so even small numerical errors can result in different activated experts, which amplifies the training-inference mismatch. In such scenarios, FP16 is still helpful because improved precision can make the router more consistent.

A common strategy for MoE RL training is rollout routing replay (aka R3), which reuses the same experts activated during inference. With R3, the numerical behavior is identical to a dense model because the discontinuity is eliminated. In our experiments, we did not use R3, which might be one reason why the improvement is relatively smaller (but still noticeable).

Thanks for your response. I will maintain my rating.

Reviewer_g14gOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the training-inference mismatch in LLM reinforcement learning and argues that floating point precision is an important source of this problem. The main finding is that replacing BF16 with FP16 can substantially reduce the mismatch between the training and inference policies, leading to more stable RL training and better performance.

The paper evaluates this across several RL algorithms, model families, frameworks, and tasks, including math, code, science, and RLHF. I am not an expert in numerical precision or large-scale training infrastructure, so my assessment focuses more on the empirical evidence and the scope of the claims than on the low-level systems explanation.

Critical strengths
  1. The paper identifies a simple and practically useful finding: switching from BF16 to FP16 can significantly improve RL training stability.

  2. The experiments are broad, covering multiple algorithms, model families, frameworks, and task domains, which makes the empirical results fairly convincing.

Critical weaknesses
  1. Some claims are stronger than the evidence supports. In particular, describing training-inference mismatch as fundamentally a numerical precision problem seems too strong, since the paper also discusses other possible sources of mismatch.

  2. The paper attributes the improvement mainly to FP16’s higher precision, but FP16 training also uses loss scaling and other mixed-precision safeguards. The role of these factors could be explained more clearly.

Questions for authors
  1. Would the authors consider weakening the claim that training-inference mismatch is fundamentally a numerical precision problem, and instead describe precision as one major source?

  2. How much of the improvement comes specifically from FP16 precision versus dynamic loss scaling and other mixed-precision techniques?

  3. Are there settings where FP16 does not help or is less stable than BF16?

2 comments

We thank the reviewer for the insightful and valuable feedback. We respond to individual points from your review below.

Q1: Would the authors consider weakening the claim that training-inference mismatch is fundamentally a numerical precision problem, and instead describe precision as one major source?

We only make this claim in Conclusion, line 481. We will weaken this claim, aligning with the one in abstract: floating point precision is a major and controllable contributing factor.

Q2: How much of the improvement comes specifically from FP16 precision versus dynamic loss scaling and other mixed-precision techniques?

We intentionally disabled other safeguards like torch.autocast in our experiments, so most of the improvement came from FP16 precision. For dynamic loss scaling, it is a must for FP16 training (unnecessary for BF16), because the gradient scale is typically small.

Q3: Are there settings where FP16 does not help or is less stable than BF16?

we honestly report all of our results without hiding any failed/negative FP16 runs.

Thank you for your responses. Unfortunately, I do not have sufficient expertise in this area to make a reliable judgment on the remaining technical concerns. I wish you the best of luck with your paper.

Reviewer_PRjUOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies why RL fine-tuning of LLMs is often unstable. In common RL frameworks, responses are generated by an inference engine while gradients are computed by a training engine, and the two give slightly different token probabilities even with the same weights. The authors argue that this mismatch mainly comes from the low precision of BF16, and propose using FP16 for both engines, which has three more mantissa bits. They show that FP16 greatly reduces the mismatch, and that RL training becomes more stable and converges faster across algorithms (GRPO, GSPO, and other variants), models (dense models up to 14B, and a 30B MoE model), training frameworks (VeRL and Oat), and a range of tasks. With FP16, even a basic importance-weighted policy gradient works well, and the differences between algorithms mostly disappear.

Critical strengths
  1. The idea is simple and useful. The paper points out that BF16 itself is a main cause of the mismatch between training and inference in RL fine-tuning, which is an interesting finding. The fix is just to switch to FP16: no change to the model or the algorithm, and no extra training cost. It cuts the mismatch a lot (sequence-level KL 7.64 vs. 0.32) and makes training much more stable.
  2. The experiments cover many settings. FP16 helps across different RL algorithms, dense and MoE models up to 30B, full and LoRA fine-tuning, two RL frameworks, and both RLVR and RLHF.
Critical weaknesses

The paper attributes the gains to a smaller training-inference mismatch, but other explanations are not ruled out. Training in different precisions may lead to different solutions, and FP16 training may reach a solution that is more robust to the inference precision. This would explain why, in Figure 5, FP16 training with BF16 inference still works better than BF16 training with FP16 inference, and the authors themselves call high-precision training "the critical factor". Likewise, BF16 training with FP32 inference may work because only a few inference modules are sensitive to precision, not because the overall mismatch is removed.

Questions for authors
  1. Have the authors tried raising precision only for selected operators (e.g., attention, KV cache) while keeping the rest in BF16? This would show where the mismatch mainly comes from, and might keep BF16's wider range while giving similar stability.
  2. Could the authors take BF16- and FP16-trained checkpoints at the same step and measure their mismatch under the same inference precisions? This would show whether FP16 training indeed reaches solutions that are more robust to the inference precision.
  3. Most base models today are pretrained in BF16, and some in FP8 (e.g., DeepSeek-V3). Given FP16's smaller range, what do the authors expect when applying FP16 post-training on top of such models? Similarly, as RL systems move toward FP8 rollouts for speed, do the authors expect FP16 training to remain useful in that setting?
4 comments

We thank the reviewer for the insightful and positive feedback. We respond to individual points from your review below.

Q1: Have the authors tried raising precision only for selected operators (e.g., attention, KV cache) while keeping the rest in BF16? This would show where the mismatch mainly comes from, and might keep BF16's wider range while giving similar stability.

The motivation of this work is to provide a universal, clean and simple solution to training-inference mismatch. However, such ablations are often implementation-dependent, thus the conclusion may change for different frameworks. For example, Megatron and transformers may have different implementations on attention.

Q2: Could the authors take BF16- and FP16-trained checkpoints at the same step and measure their mismatch under the same inference precisions? This would show whether FP16 training indeed reaches solutions that are more robust to the inference precision.

Actually we have similar evaluations in Appendix D. Table 4 and 5 show that, at deployment, FP16 and BF16 are interchangeable, for both BF16- and FP16-trained checkpoints.

Q3: Most base models today are pretrained in BF16, and some in FP8 (e.g., DeepSeek-V3). Given FP16's smaller range, what do the authors expect when applying FP16 post-training on top of such models? Similarly, as RL systems move toward FP8 rollouts for speed, do the authors expect FP16 training to remain useful in that setting?

As discussed in Appendix B.3 and D, inference under FP16 is feasible and comparable to BF16 even they are trained in other precisions, for example, Qwen3-235B-A22B. With dynamic loss scaling, FP16 training is usually also feasible for such models.

Regarding even lower precision for speed, it actually depends. For example, Kimi K2 series use A16W4 (BF16 for activations, INT4 for MoE weights), where FP16 can still help without loss of speed. For A8W8 (e.g. DeepSeek-V3) together with techniques like per-tensor scaling, FP16 is not an economic option any more, but it's a different tradeoff.

Thanks for the response. However, my main concern about whether the gains come from the reduced mismatch or from FP16 training itself is not resolved. Appendix D compares the final accuracy under different inference precisions, while my question was about the mismatch of BF16- and FP16-trained checkpoints, so it does not separate the two effects. Therefore, I lower my rating to 3.

We thank the reviewer for the feedback. We misunderstood the question in our initial rebuttal, and we would like to clarify it again.

Could the authors take BF16- and FP16-trained checkpoints at the same step and measure their mismatch under the same inference precisions? This would show whether FP16 training indeed reaches solutions that are more robust to the inference precision.

We think this comparison is already implied in Figure 3. The training-inference mismatch keeps going larger for BF16 training, while it remains stable for FP16 training. If we take the checkpoints at some step and measure the mismatch under BF16, the mismatch of BF16 checkpoint will be much larger than the initial value. Given FP16 maintains the mismatch under FP16 precision, we suppose the mismatch under BF16 won't change too much as well. And results from Appendix D prove that the solution should be similar under FP16 and BF16.

Additionally, as the training dynamics are quite different for BF16 and FP16 (see Figure 3), it's also hard to reach the desired conclusions by taking the checkpoints at the same step.

The reason why FP16 shows a stable training dynamics is, FP16 has a much smaller training-inference mismatch, thus the ratio is highly concentrated around 1. Conversely, the ratio for BF16 is much easier to explode, resulting in larger optimization bias and variance, thus easier to collapse or get unstable.

whether the gains come from the reduced mismatch or from FP16 training itself

Section 4.3 was actually intended to conduct this ablation. In Figure 5, FP16 for training only collapse eventually, while FP16 for both training and inference improves stably. Additionally, Figure 3 shows that training-inference mismatch is an important indicator for training collapse. So we believe the gains mainly come from the reduced mismatch. However, FP16 training solely also has benefits in some extent, as in Figure 5.

Thanks for the clarification. My concerns have been resolved, and I have raised my rating to 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.