- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In low-precision training, the forward pass rounds intermediate tensors to lower precision. Later, the backward pass may use these tensors to compute gradients for several parameters. This paper looks at which version each gradient calculation should use: the original value, the rounded value used in the forward pass, or a separately rounded copy. The authors call these choices a “backward-state policy” and argue that they affect the gradients used to update the model.
To work out which choices are appropriate, the authors define a reference gradient based on what actually happened in the forward pass. They use the straight-through convention, meaning gradients pass through rounding unchanged. Under this reference, different gradient calculations may need different versions of the same tensor. For example, suppose a normalization output is rounded before going into a linear layer. The linear layer’s weight gradient needs the rounded input, because that is what the layer actually multiplied. But the normalization layer’s scale gradient needs the normalized input before rounding. So always picking the most numerically accurate version does not necessarily preserve the reference gradients.
The paper also looks at using substitute copies. A substitute is acceptable if it gives the reference gradient on average, with the actual forward pass and the gradient coming from later layers held fixed. It is not enough for the copy itself to be accurate on average: the forward rounding error can be correlated with the incoming gradient and change the result. So the authors work out the requirements for each gradient calculation separately. They also introduce a checker for the stated policies and their random dependencies. The checker reports which conditions hold, which fail, and which it cannot decide.
Critical strengths
-
I like the attention comparison in Figure 1. In the simulated FP8 backward, the precision and rounding rule stay the same, but changing which copy the correction term reads changes whether training succeeds. That makes the effect of this choice pretty clear.
-
The normalization example is useful too. It shows why simply using the most accurate copy everywhere can still change the gradients. Different gradients may need different versions of the same tensor, which makes the main idea more concrete.
-
The tests on PyTorch and Transformer Engine are a nice addition. They connect the discussion to code people actually use and give some support for checking backward operations individually, even when the final loss looks fine.
Critical weaknesses
-
The extra computation is not evaluated. The repair in Section 2.3 adds a BF16 attention recomputation before the native FP8 backward. This adds work, but the paper reports neither time per training step nor throughput. It is unclear how much of FP8’s efficiency remains after this repair.
-
The memory cost is also unclear. Different backward-state policies may retain additional copies or create temporary high-precision tensors. The paper should specify which tensors remain live and report peak memory. Without this, the practical cost of following the proposed requirements is hard to assess.
-
The practical comparison with existing approaches is missing. COAT (Xi et al., 2025) evaluates both memory savings and training speed, while Attn-QAT (Zhang et al., 2026) provides a closely related high-precision-output strategy for attention backward. A matched comparison with the relevant components of these approaches would clarify whether independent rounding offers a useful tradeoff between stability, memory, and runtime.
Questions for authors
-
Comparison with existing fixes. Why is there no matched full-run comparison with an Attn-QAT-style high-precision output [1] or the attention stabilization proposed by Qiu & Yao [2], adapted to the same FP8 setting? Table 9 includes a short BF16-output control, but what evidence establishes the benefit of independent rounding over these alternatives?
-
Memory and computation costs. What are the peak-memory and per-step runtime costs of the extra BF16 attention recomputation in Section 2.3? Please include temporary buffers and retained copies. COAT [3] evaluates both memory and training speed; does your repaired FP8 implementation still offer an advantage over BF16, and how does its cost compare with saving a BF16 output under the same setup?
-
Limited training at the larger scale. The 1.5B experiment uses one seed and 5,000 total steps, corresponding to approximately 5.24B tokens. Using the approximate 20-token-per-parameter Chinchilla convention [4], this is only about 0.17× the reference budget. What evidence supports stability over longer training, beyond the reported short schedule?
-
Interpretation of the native FP8 results. Which controls separate the effect of independent rounding from changes in the output source and rounding rule? Section 2.3 changes both, while Figure 1 isolates the difference using an emulated backward. How strongly do the existing native experiments establish independence itself as the reason for stabilization?
References:
[1] Zhang et al. (2026). Attn-QAT: 4-Bit Attention With Quantization-Aware Training.
[2] Qiu & Yao (2025). Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention.
[3] Xi et al. (2025). COAT: Compressing Optimizer States and Activations for Memory-Efficient FP8 Training.
[4] Hoffmann et al. (2022). Training Compute-Optimal Large Language Models.
3 comments
Thank you for the response. My concerns about the complete end-to-end cost of the high-precision-output strategy and the longer-run validation of the targeted FP8 correction remain only partially resolved.
The revised manuscript also contains extensive changes and additional experimental evidence. Evaluating these changes requires a substantial reassessment of the paper, beyond checking the responses to individual questions. Within the remaining discussion period, I cannot verify all of the revised material with sufficient care to reach a confident updated assessment.
I will therefore maintain my original rating, while lowering my confidence to reflect the extent of the revisions and the material that requires further assessment. I respectfully leave the final decision to the AC, considering the revised manuscript and the discussion as a whole. Thank you to the AC for their time and careful consideration.
Reply to Reviewer zqmr: longer-run validation of the targeted FP8 correction
Thank you for your reply, and for stating which concerns remain open.
On the longer-run validation of the targeted FP8 correction: both runs with the correction train from scratch over the full 20k-step schedule, which exceeds the Chinchilla budget at 390M.
On the two seeds where TE's unchanged attention fails, they end at a validation loss of 2.75, against 2.74 for BF16 training. Section 2.3 reports both runs, and Appendix D.3 gives their schedule.
Reply to Reviewer zqmr: cost, Attn-QAT, COAT and Qiu & Yao, longer 1.5B training, and the controls
Thank you for the review, and for the close reading of Figure 1 and the normalization example.
The §2.3 recomputation was built as a diagnostic, not as an implementation, and it is slower than BF16 training. Saving a BF16 output costs far less: 0.5 GiB of peak memory and no measurable step time.
A step of the §2.3 run takes 1.42 times as long as with TE's FP8 attention, against 1.29 times for BF16 training, at the same peak memory as TE's. That peak memory includes the temporary buffers and retained copies you ask about, because it is the largest memory PyTorch allocated on one GPU over the run. The 0.5 GiB is the cost of keeping a BF16 copy of each attention output until the backward. Computing that output, as Attn-QAT does, takes an extra product in the forward that TE's fused kernel does not let us time. Table 12 gives the step time and peak memory of five attention paths at 390M on eight H20 GPUs.
The revision adds matched full runs for Attn-QAT's choice. Attn-QAT's high-precision output gives the attention backward's correction term D=rowsum(dO∘O) the value our reference prescribes. A correction to TE's FP8 backward that gives D this value makes training normal on 390M seeds 0 and 2, which fail with TE's attention unchanged, ending at a validation loss of 2.75 against 2.74 for BF16.
With the correction replaced by zeros, the same code fails, so the result comes from the correction rather than from the code around it.
The revision adds a second 1.5B seed and continues seed 0 to step 9999, about 10.5B tokens. In both, replacing only the attention backward with a BF16 one trains normally and ends within 0.004 nats of BF16 training.
For training beyond the Chinchilla budget, the evidence is at 390M. Its 20k-step schedule exceeds that budget, and the BF16 backward trains normally over all of it on all three seeds.
The revision adds such a control inside TE's own FP8 attention backward. Figure 1b repeats Figure 1's comparison there with the output source and its rounding distribution fixed, so the two runs of each pair differ only in which rounding D reads. In all three pairs, reusing the forward's rounding fails and a new rounding trains normally.
A second control also keeps TE's own rounding rule. D reads TE's round-to-nearest output as usual, while the next layer consumes a stochastic rounding instead, so the error D reads no longer enters the loss. From both of Figure 1's 390M checkpoints, training is then normal, while unchanged training fails from seed 0 and reaches attention logits of 103 from seed 1. Appendix E.5 also measures the error of D inside TE's kernel, where reading the rounding the network consumed gives a larger mean signed error than an independent rounding in most layers.