acceptodds
Under review as a conference paper at ICLR 2027

Where the FP4 Pretraining Gap Sits, and Why That Decides What Transfers Across Scale

Abstract

Training a language model in 4-bit floating point (FP4) is faster than in BF16 but ends at a higher loss. Recipes narrow this gap by keeping some computations, or the end of training, in higher precision, and these choices are usually tuned on small models. We ask where the gap comes from in one FP4 recipe at three model sizes, and find that the answer changes with scale. In a small model nearly all of the gap comes from rounding in the forward pass; as the model grows this share falls, and rounding the gradients, alone and together with the forward pass, takes over. At every size a similar share of the gap is the cost of the rounding still in progress, which a switch to BF16 removes at once, while the rest stays in the weights. Since higher precision in one part of training removes at most the share of the gap that arises there, where the gap sits decides what transfers across scale: a higher-precision forward pass removes most of the gap in a small model but less than half in a large one, while a short switch to BF16 at the end removes the same share at every size. A few runs at the target scale measure these shares before any recipe is trained.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
FP4 training, NVFP4, low-precision training, quantization, mixed precision, precision allocation, LLM pretraining, scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CwgtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper provides an empirical study of the error from FP4 training. It breaks down the loss gap from FP4 training into error from the forwards pass, backwards pass, and "excess" error caused by correlations between quantizing both the FW and BW pass. The paper studies RTN W4A4 FW passes and unbiased backwards passes, as has been adopted in the FP4 training literature.

The main findings of the paper are that the relative effect of forwards pass quantization goes down with scale, whereas backwards pass quantization does not. Gradient variance from stochastic rounding adds a constant/ slightly growing loss gap as training progresses.

The paper claims to study FP4 training across different scales, but all models in the paper are tiny and much smaller than models one would reasonably train in FP4 to get real-world speedups. It is not clear to me if the trends in the paper hold at actual LLM-sized models (eg 10 or 100B parameters), or if they reverse at some point since the paper only studies ~100M to 1B parameter models.

This paper does not contain any real theory to explain the effects they observed and is largely just an empirical study on various FP4 training configs. The writing in the paper also seems heavily LLM edited / generated and is hard to follow. There are a lot of unnatural word and phrase choices that native English speakers would not use.

Critical strengths

The findings in this paper are interesting and are "new" in the sense that I am not aware of other papers that have looked into this for FP4 training. However, given the small scale of the models and limited analysis, it is not clear to me how valuable these results are vs the same experiments run on larger models.

Critical weaknesses

See above. The models used are tiny and there is limited theoretical analysis on why these behaviors might be happening. I also found it odd that NVFP4 clipping did not hurt the model, since this makes the gradient biased.

Questions for authors

See above.

Reviewer_oi96OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the gap in the final pre-training loss between NVFP4 and BF16 training for one NVFP4 recipe on Llama-style models at 60M, 130M and 1B scales. The models are trained on FineWeb-Edu with the GPT-2 tokenizer for 1.2B, 2.6B and 20B tokens respectively. For their analysis, they split the gap two ways, by tensor (ie round only weights and activations, only gradients, or all three), and by time (a run is switched from NVFP4 to BF16 at a fraction t of training). The gap that can be closed is the recoverable gap and the gap that remains at the end is the persistent gap. Using this setup, the main claims of the paper are that an allocation of higher precision removed at most the share of the gap it acts on. For instance, running the forward GEMM in MXFP8 instead of NVFP4 removes at most the activations and weights share, and a BF16 tail removes at most the gap recoverable when it starts. They also show that the share of gap from activations and weights falls from 89% at 60M to 42% at 1B. In addition, at the 1B scale, the gradient gap grows through training (this doesn't happen at smaller scales), and at this scale, there is an "excess" gap from quantizing all three together which is about 24% of the full gap. Finally, a BF16 tail removes about 25% of the gap at every scale, and all of it within 1000 steps. Mechanism ablations attribute the forward gap to the mean squared rounding error rather than bias, clipping or outliers. This gap grows as the model converges, and larger models are less sensitive to the same error. The gradient gap is attributed to unbiased noise.

Critical strengths
  1. I like the problem statement and the way this paper approaches the problem.
  2. BF16 tails are paired with their NVFP4 continuations, so both branches see the same batches.
  3. The authors report seed floors at every scale and also report when the differences are below the floor so not significant which is nice to see.
  4. Some of the findings are practically useful, like the finding that a 1000 step BF16 tail is enough and longer training doesn't help much more.
Critical weaknesses
  1. The writing can be improved. It reads like a major chunk of the paper, if not all of it, is written by Claude.
  2. Given that we already see some clear differences as we scale to 1B, it is not clear if the findings will generalize to larger models.
  3. It is also not clear how this will scale with more data, and when the data is not repeated.
  4. Overall, this is a collection of empirical findings but I did not find it particularly insightful in terms of what is the underlying cause for some things like the residual gap, why the 1B behaves differently, etc.
Questions for authors

Can you report held-out validation loss at 1B?

Reviewer_PgyhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Motivated by conflicting recipes for partial higher-precision training across model sizes, the authors study one FP4 recipe at the 60M, 130M, and 1B scale, and find that in small models, the gap in loss (wrt precision) is primarily attributed to rounding from the forward pass, while in larger models, the primary factor contributing to the gap in loss shifts to being rounding the gradient. Across all model sizes, switching to BF16 towards the end of training removes the same share.

Critical strengths

This paper is well motivated; careful analysis of where higher-precision should be allocated and understanding how it changes with scale is very useful to the community.

Critical weaknesses

Main concerns are with regard to the scientific rigour of the paper, and are expressed below:

  1. More evidence is needed for how the results scale: How does the excess gap scale with model size? The excess only shows up at 1B, so it is hard to tell whether it comes from model size or is an artifact specific to the 1B run. The authors could run more model sizes between 130M and 1B to get a smoother curve and more convincing results. Also how does the full gap scale in general? In Figure 4 there is no clean monotonic trend for the full gap, which one might expect: from smallest to largest final gap, the order is 130M, 60M, 1B. It would also be interesting to see how phenomena scales with token budget.
  2. Inconsistent switching between BF16 and MXFP8 in the definitions: The ceiling is measured with the gradients in BF16, while the gap reduction from the MXFP8 forward run is measured with the gradients still in NVFP4. Why are the two defined differently, especially when they are compared directly against each other?
  3. Summing the individual gaps does not show independence: Throughout the paper (especially for the 60M and 130M results), claims rely on the idea that rounding weights, activations, and gradients each affect the gap independently. The authors take the gap from separate runs on each factor and treat it as that factor's share of the total gap. However, it is not clear that these contributions are independent of one another in full training, so this argument does not feel well founded. For example, the paper claims that "the gap from rounding activations and weights is rounding error on the activations entering the forward GEMM," but it is not clear that activations and weights act independently of one another in how they add to the gap. The lack of an excess at the small scales (60M and 130M) is not rigorous evidence of independence, and it also limits how far these claims hold at larger scale.
  4. Lack of tuning: The same hyperparameters are used across one model size. However, it's unclear that the best hyperparameters for different recipes stay consistent (etc, unclear that the hyperparameters when rounding gradients should be the best hyperparameters when rounding activations) — therefore it's hard to discern if results for different factors are just an artifact of not-tuning-enough when rounding that specific factor.
Questions for authors

How does the excess gap scale with model size? How does the full gap scale with model size? Is tuning across the different recipes needed? How does tuning the runs for each factor change the results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.