acceptodds
Under review as a conference paper at ICLR 2027

Backward-State Policy Is Part of the Learning Algorithm

Abstract

Low-precision training rounds tensors that the backward pass reads again, often for several gradients; each use can read the forward's rounded value, the original, or a new random rounding. This backward-state policy looks like a memory and precision detail, settled by copy accuracy and final loss. We argue that it is part of the learning algorithm, and that neither check shows whether it is right. Copy accuracy does not decide the outcome: in three pairs of 390M runs with an emulated FP8 backward, training fails when attention's backward reuses the forward's rounded output and succeeds with a new rounding from the same distribution. Even the most accurate copy, the original itself, can be wrong by our reference: the gradient of the forward pass as it actually ran, with gradients passed through rounding unchanged. For example, a normalization output stored in low precision feeds two gradients: the gain's gradient needs the original, but the next layer's weight gradient needs the rounded value that layer multiplied. Final loss, the other check, does not rule out the error of reading the original for both: it persists in models trained with such a store, while planned loss comparisons stay within a margin fixed in advance. We therefore derive from this reference which value each use must read, or which substitute gives the same gradient on average with the forward held fixed, and check these per-use requirements on single operators, without training. In three tests using PyTorch and Transformer Engine, the requirements predicted beforehand whether reuse changes what the backward computes on average relative to an independent copy, and every prediction held. Backward-state policy is thus part of the learning algorithm: it should be specified and checked use by use, not settled by copy accuracy and final loss.

Primary area:
optimization
Keywords:
low-precision training, backward-state policy, automatic differentiation, stochastic rounding, gradient correctness, training stability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zqmrOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In low-precision training, the forward pass rounds intermediate tensors to lower precision. Later, the backward pass may use these tensors to compute gradients for several parameters. This paper looks at which version each gradient calculation should use: the original value, the rounded value used in the forward pass, or a separately rounded copy. The authors call these choices a “backward-state policy” and argue that they affect the gradients used to update the model.

To work out which choices are appropriate, the authors define a reference gradient based on what actually happened in the forward pass. They use the straight-through convention, meaning gradients pass through rounding unchanged. Under this reference, different gradient calculations may need different versions of the same tensor. For example, suppose a normalization output is rounded before going into a linear layer. The linear layer’s weight gradient needs the rounded input, because that is what the layer actually multiplied. But the normalization layer’s scale gradient needs the normalized input before rounding. So always picking the most numerically accurate version does not necessarily preserve the reference gradients.

The paper also looks at using substitute copies. A substitute is acceptable if it gives the reference gradient on average, with the actual forward pass and the gradient coming from later layers held fixed. It is not enough for the copy itself to be accurate on average: the forward rounding error can be correlated with the incoming gradient and change the result. So the authors work out the requirements for each gradient calculation separately. They also introduce a checker for the stated policies and their random dependencies. The checker reports which conditions hold, which fail, and which it cannot decide.

Critical strengths
  1. I like the attention comparison in Figure 1. In the simulated FP8 backward, the precision and rounding rule stay the same, but changing which copy the correction term reads changes whether training succeeds. That makes the effect of this choice pretty clear.

  2. The normalization example is useful too. It shows why simply using the most accurate copy everywhere can still change the gradients. Different gradients may need different versions of the same tensor, which makes the main idea more concrete.

  3. The tests on PyTorch and Transformer Engine are a nice addition. They connect the discussion to code people actually use and give some support for checking backward operations individually, even when the final loss looks fine.

Critical weaknesses
  1. The extra computation is not evaluated. The repair in Section 2.3 adds a BF16 attention recomputation before the native FP8 backward. This adds work, but the paper reports neither time per training step nor throughput. It is unclear how much of FP8’s efficiency remains after this repair.

  2. The memory cost is also unclear. Different backward-state policies may retain additional copies or create temporary high-precision tensors. The paper should specify which tensors remain live and report peak memory. Without this, the practical cost of following the proposed requirements is hard to assess.

  3. The practical comparison with existing approaches is missing. COAT (Xi et al., 2025) evaluates both memory savings and training speed, while Attn-QAT (Zhang et al., 2026) provides a closely related high-precision-output strategy for attention backward. A matched comparison with the relevant components of these approaches would clarify whether independent rounding offers a useful tradeoff between stability, memory, and runtime.

Questions for authors
  1. Comparison with existing fixes. Why is there no matched full-run comparison with an Attn-QAT-style high-precision output [1] or the attention stabilization proposed by Qiu & Yao [2], adapted to the same FP8 setting? Table 9 includes a short BF16-output control, but what evidence establishes the benefit of independent rounding over these alternatives?

  2. Memory and computation costs. What are the peak-memory and per-step runtime costs of the extra BF16 attention recomputation in Section 2.3? Please include temporary buffers and retained copies. COAT [3] evaluates both memory and training speed; does your repaired FP8 implementation still offer an advantage over BF16, and how does its cost compare with saving a BF16 output under the same setup?

  3. Limited training at the larger scale. The 1.5B experiment uses one seed and 5,000 total steps, corresponding to approximately 5.24B tokens. Using the approximate 20-token-per-parameter Chinchilla convention [4], this is only about 0.17× the reference budget. What evidence supports stability over longer training, beyond the reported short schedule?

  4. Interpretation of the native FP8 results. Which controls separate the effect of independent rounding from changes in the output source and rounding rule? Section 2.3 changes both, while Figure 1 isolates the difference using an emulated backward. How strongly do the existing native experiments establish independence itself as the reason for stabilization?

References:

[1] Zhang et al. (2026). Attn-QAT: 4-Bit Attention With Quantization-Aware Training.

[2] Qiu & Yao (2025). Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention.

[3] Xi et al. (2025). COAT: Compressing Optimizer States and Activations for Memory-Efficient FP8 Training.

[4] Hoffmann et al. (2022). Training Compute-Optimal Large Language Models.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer zqmr: cost, Attn-QAT, COAT and Qiu & Yao, longer 1.5B training, and the controls

Thank you for the review, and for the close reading of Figure 1 and the normalization example.

W1, W2 / Q2: What are the peak-memory and per-step runtime costs of the extra BF16 attention recomputation in Section 2.3? Please include temporary buffers and retained copies. … does your repaired FP8 implementation still offer an advantage over BF16, and how does its cost compare with saving a BF16 output under the same setup?

The §2.3 recomputation was built as a diagnostic, not as an implementation, and it is slower than BF16 training. Saving a BF16 output costs far less: 0.5 GiB of peak memory and no measurable step time.
A step of the §2.3 run takes 1.42 times as long as with TE's FP8 attention, against 1.29 times for BF16 training, at the same peak memory as TE's. That peak memory includes the temporary buffers and retained copies you ask about, because it is the largest memory PyTorch allocated on one GPU over the run. The 0.5 GiB is the cost of keeping a BF16 copy of each attention output until the backward. Computing that output, as Attn-QAT does, takes an extra product in the forward that TE's fused kernel does not let us time. Table 12 gives the step time and peak memory of five attention paths at 390M on eight H20 GPUs.

 

W3 / Q1: Why is there no matched full-run comparison with an Attn-QAT-style high-precision output [1] or the attention stabilization proposed by Qiu & Yao [2], adapted to the same FP8 setting? … what evidence establishes the benefit of independent rounding over these alternatives?

The revision adds matched full runs for Attn-QAT's choice. Attn-QAT's high-precision output gives the attention backward's correction term D=rowsum(dO∘O)D=\mathrm{rowsum}(dO\circ O) the value our reference prescribes. A correction to TE's FP8 backward that gives DD this value makes training normal on 390M seeds 0 and 2, which fail with TE's attention unchanged, ending at a validation loss of 2.75 against 2.74 for BF16.
With the correction replaced by zeros, the same code fails, so the result comes from the correction rather than from the code around it.

  • Independent rounding. We do not propose it as an alternative to these methods. Figure 1 uses it only as a probe, because it is as accurate as the forward's own rounding but is not the rounding the loss saw.
  • Qiu & Yao. Their stabilization changes the softmax normalization inside the forward kernel, and TE's FP8 forward is a fused cuDNN kernel that we cannot modify. To isolate their failure, however, they compute DD from the softmax probabilities instead of the saved output, and the runs above add the change this makes to the gradients of TE's FP8 backward.
  • COAT. Its relevant component here is attention, which COAT keeps in BF16 together with its saved tensors. In our setting, BF16 attention with FP8 linear layers takes 1.07 times the step time of TE's FP8 attention.

 

Q3: The 1.5B experiment uses one seed and 5,000 total steps, corresponding to approximately 5.24B tokens. … What evidence supports stability over longer training, beyond the reported short schedule?

The revision adds a second 1.5B seed and continues seed 0 to step 9999, about 10.5B tokens. In both, replacing only the attention backward with a BF16 one trains normally and ends within 0.004 nats of BF16 training.
For training beyond the Chinchilla budget, the evidence is at 390M. Its 20k-step schedule exceeds that budget, and the BF16 backward trains normally over all of it on all three seeds.

 

Q4: Which controls separate the effect of independent rounding from changes in the output source and rounding rule? … How strongly do the existing native experiments establish independence itself as the reason for stabilization?

The revision adds such a control inside TE's own FP8 attention backward. Figure 1b repeats Figure 1's comparison there with the output source and its rounding distribution fixed, so the two runs of each pair differ only in which rounding DD reads. In all three pairs, reusing the forward's rounding fails and a new rounding trains normally.
A second control also keeps TE's own rounding rule. DD reads TE's round-to-nearest output as usual, while the next layer consumes a stochastic rounding instead, so the error DD reads no longer enters the loss. From both of Figure 1's 390M checkpoints, training is then normal, while unchanged training fails from seed 0 and reaches attention logits of 10310^3 from seed 1. Appendix E.5 also measures the error of DD inside TE's kernel, where reading the rounding the network consumed gives a larger mean signed error than an independent rounding in most layers.

Thank you for the response. My concerns about the complete end-to-end cost of the high-precision-output strategy and the longer-run validation of the targeted FP8 correction remain only partially resolved.

The revised manuscript also contains extensive changes and additional experimental evidence. Evaluating these changes requires a substantial reassessment of the paper, beyond checking the responses to individual questions. Within the remaining discussion period, I cannot verify all of the revised material with sufficient care to reach a confident updated assessment.

I will therefore maintain my original rating, while lowering my confidence to reflect the extent of the revisions and the material that requires further assessment. I respectfully leave the final decision to the AC, considering the revised manuscript and the discussion as a whole. Thank you to the AC for their time and careful consideration.

AuthorsOpenReview ↗

Reply to Reviewer zqmr: longer-run validation of the targeted FP8 correction

Thank you for your reply, and for stating which concerns remain open.

On the longer-run validation of the targeted FP8 correction: both runs with the correction train from scratch over the full 20k-step schedule, which exceeds the Chinchilla budget at 390M.
On the two seeds where TE's unchanged attention fails, they end at a validation loss of 2.75, against 2.74 for BF16 training. Section 2.3 reports both runs, and Appendix D.3 gives their schedule.

Reviewer_r2SaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a study of low-precision training, specifically the backward pass, where it studies how the rounded values saved from the forward pass in a low-precision setting is used to compute gradients and how it affect the optimization process. The authors propose that the choice of which value the backward reads, the rounded value (R), the unrounded original (U), or a new random rounded value (N), should be evaluated at each use of a given tensor against a reference gradient, rather than by how accurate the saved copy is or by final loss. The experiments show that which of R/U/N is correct differs from what is expected, as since the loss was computed from the rounded value, the gradient flowing back is correlated with that rounding error, so reusing the same rounded copy in the backward introduces a systematic bias, while the error of an independent rounding averages out. The authors also show that a single tensor can need different values at different uses of R/U/N, and that models trained with a backward pass that is different from the reference reach a similar final loss but have different weights.

Critical strengths
  1. The main experiment is a valid test of the paper's central claim, where compare the effect of using rounding values of different distributions in the backward pass compared to the forward pass.

  2. The results provided are non-trivial, and the general idea of the section "one tensor, two uses" can apply beyond just FP8 attention.

Critical weaknesses
  1. The main issue of this paper is its presentation as the abstract relies on terms that are not defined until later, or only implicitly, e.g. "copy accuracy", "the original" (of what?), "wrong by our reference", and the claim that the policy "looks like a memory and precision detail, settled by copy accuracy and final loss" reads as a straw position that the reader cannot place. The U/R/N notation that makes Section 3 readable appears first in the caption of Figure 3, and the sentence "the gain's gradient needs the original, but the next layer's weight gradient needs the rounded value that layer multiplied" is not interpretable without Eq. 2. The base idea of Section 2.1 is that the error in the attention backwards correction term is non-zero on average when the incoming gradient depends on the rounding error of the copy it reads, and zero on average when the copy accuracy error is drawn independently and stating it simply, with a short glossary mapping "use", "store", "reference" and "copy accuracy" to standard concepts, would make the contribution visible on the first page. As written, I needed the appendices to understand the abstract.

  2. Another issues I have is that the paper treats shifts from the reference gradient as an error, but I am not convinced that the reference is sufficiently justified. The authors use the straight-through gradient of the forward pass as it ran, but this is itself an approximation. In Section 3.3, U/U performs comparably to U/R, and R/R even achieves a lower final loss in some settings. The paper shows that these policies produce different weights, but I am not sure it establishes why following the reference is preferable. The toy model in Section 3.2 helps motivate the argument, but I would like to see a clearer explanation of why the reference should be the target in the larger training experiments or some difference in metrics which can evaluate this difference in model weights.

  3. Figure 1 uses stochastic rounding to demonstrate the effect of reusing rounded values, but Transformer Engine uses deterministic round-to-nearest. It is therefore unclear whether the same effect explains the failure observed in the shipped implementation. The experiments with Transformer Engine change multiple factors at once, so they do not isolate the effect of reusing rounded values. I would like to see an experiment that isolates this effect while keeping the rounding method and other factors fixed. Without this, the paper's explanation for the practical failure remains plausible, but not convincingly demonstrated.

Questions for authors
  1. What does the analysis predict for round-to-nearest, which is what TE ships? With deterministic rounding there is no independent copy, so is the shipped failure explained by the same rowsum term acting as a bias, or by something else? Is it possible to run an experiment on TE's own backward that changes only the dependence between the saved output's error and dO, while leaving the rounding mode and the source output unchanged?

  2. [1] already locates the failure specified in D and attribute it to biased rounding of O. What does the reuse-versus-independent framing predict that their paper does not? A concrete case where the two accounts disagree, and the experiment that distinguishes them, would clarify the novelty of Section 2.

  3. What should a practitioner actually change, and at what cost? Reading the original rounded value at the gain use defeats part of the memory saving, and the per-use assignment for attention in Appendix B.4 needs a rounding the forward did not consume. Could you report the memory and compute overhead of the reference-following policies relative to TE's current backward, and say whether the executable checker of Section 4.1 scales beyond the small finite cases in Appendix C?

[1] Qiu H. et. al., Why Low-Precision Transformer Training Fails, ICLR 2026

4 comments
AuthorsOpenReview ↗

Reply to Reviewer r2Sa: presentation and the reference

Thank you for the careful review, and for proposing the round-to-nearest experiment, which the revision runs.

W1: … the abstract relies on terms that are not defined until later, or only implicitly, e.g. "copy accuracy", "the original" (of what?), "wrong by our reference", and the claim that the policy "looks like a memory and precision detail, settled by copy accuracy and final loss" reads as a straw position that the reader cannot place. The U/R/N notation that makes Section 3 readable appears first in the caption of Figure 3, and the sentence "the gain's gradient needs the original, but the next layer's weight gradient needs the rounded value that layer multiplied" is not interpretable without Eq. 2. … with a short glossary mapping "use", "store", "reference" and "copy accuracy" to standard concepts, would make the contribution visible on the first page.

The revision makes each of these changes.

  • The abstract defines its terms before using them and states the reference before judging anything by it. The paper now says "the unrounded value" instead of "the original".
  • The view that the policy is a memory and precision detail is now placed in practice. The abstract says that implementations make this choice to save memory and that two checks seem to settle it, and Section 1 names examples, from FlashAttention's saved output to activation-compressed training.
  • Table 1 on page 2 maps use, store, reference, copy accuracy and the other terms to standard concepts, and defines U, R and N. The reference, for example, is the straight-through estimator of the executed forward. Section 3.1 introduces U, R and N in its text.
  • The gain sentence now states its setting, a normalization output whose rounded value feeds the next layer, and gives each reason. The gain's gradient needs the unrounded value because the gain scaled unrounded values, and the next layer's weight gradient needs the rounded value because that layer multiplied it.
  • The abstract and Section 1 state the idea of Section 2.1 in one sentence. Attention's backward multiplies its copy of the output by a gradient computed from the forward's rounding, so that gradient can depend on that rounding's error but not on a new rounding's.

 

W2: … I am not convinced that the reference is sufficiently justified. The authors use the straight-through gradient of the forward pass as it ran, but this is itself an approximation. … I would like to see a clearer explanation of why the reference should be the target in the larger training experiments or some difference in metrics which can evaluate this difference in model weights.

We take the reference as the target because the forward that actually ran determines it, so every read has one definite gradient to be checked against. We do not claim that following it lowers final loss, and at 6 bits R/R ends lower. Departures from the reference range from that lower loss to the failures of Section 2, so final loss cannot show which gradient training followed. §3.3 now states this.

  • In the larger runs, following the reference at one read, the attention backward's correction term DD, decided the outcome. Reading the forward's rounding there departs from the reference and fails, while in §2.3 a correction that gives DD the reference's value trains normally on 390M seeds 0 and 2, which fail with TE's attention unchanged.
  • Straight-through is an approximation, but in the two-parameter model of Section 3.2, where the expected loss can be computed exactly, following the reference keeps the minimum of the expected loss. The reference's assignment has zero average update at that minimum, while reading the rounded value at both uses has zero average update at another point, where the expected loss is higher.
  • A metric for the weights. The trained gains separate policies that final loss does not. The two policies whose gain use reads the rounded value end with a lower root-mean-square gain than the other five, in every layer and on both seeds, as Figure 5 shows.
AuthorsOpenReview ↗

Reply to Reviewer r2Sa, continued: round-to-nearest, Qiu & Yao, the practitioner's change and the checker

W3 / Q1: What does the analysis predict for round-to-nearest, which is what TE ships? With deterministic rounding there is no independent copy, so is the shipped failure explained by the same rowsum term acting as a bias, or by something else? Is it possible to run an experiment on TE's own backward that changes only the dependence between the saved output's error and dO, while leaving the rounding mode and the source output unchanged?

The analysis predicts the same rowsum term, and the revision tests this on TE's own backward. Once the error that DD reads from TE's round-to-nearest output no longer enters the loss, training from both of Figure 1's 390M checkpoints is normal. Unchanged, it fails from seed 0 and reaches attention logits of 10310^3 from seed 1.
Reading TE's rounded output adds rowsum(dO∘E)\mathrm{rowsum}(dO\circ E) to DD, and dOdO can depend on the error EE because the next layer consumed that rounding. To keep EE out of the loss without changing the rounding mode or the source output, DD keeps reading TE's output, and the next layer consumes a stochastic rounding instead. That change alone does not avoid the failure, since in Figure 1b the next layer also consumes a stochastic rounding and every run whose DD reads that same rounding fails.

 

Q2: [1] already locates the failure specified in D and attribute it to biased rounding of O. What does the reuse-versus-independent framing predict that their paper does not? A concrete case where the two accounts disagree, and the experiment that distinguishes them …

The accounts disagree when the rounding is unbiased. Their account, biased rounding of OO, gives no reason for two copies from one unbiased distribution to differ, while ours predicts that they can differ when only one of them is the rounding the loss saw. Figure 1 tests this case, with an emulated backward and inside TE's own. In all three pairs, reusing the forward's rounding fails and a new rounding trains normally.
We build on their location of the failure in DD, and §2.1 now states where the two accounts diverge.

 

Q3: What should a practitioner actually change, and at what cost? Reading the original rounded value at the gain use defeats part of the memory saving, and the per-use assignment for attention in Appendix B.4 needs a rounding the forward did not consume. Could you report the memory and compute overhead of the reference-following policies relative to TE's current backward, and say whether the executable checker of Section 4.1 scales beyond the small finite cases in Appendix C?

A practitioner should give each backward read the value its forward operation used, and change only the reads that receive something else. Reads whose operation multiplied the rounded value, such as weight gradients, keep reading the saved rounded copy, so its memory saving stays. The cost falls on reads that need the unrounded value, such as the gain's gradient and attention's correction term DD.

  • Where a change matters. Compared with a new rounding, reusing the forward's rounding can change a read's gradient on average only if the read's incoming gradient depends on that rounding, as at DD. Section 5 therefore recommends testing this dependence read by read.
  • The gain use and Appendix B.4. At the gain use, following the reference means keeping or recomputing the unrounded normalization output for that one read, while the weight use keeps reading the few-bit copy. B.4's new rounding needs no saved tensor, since it can be drawn from the unrounded probabilities that FlashAttention-style backwards already recompute.
  • Cost at DD. DD needs rowsum(P∘dP)\mathrm{rowsum}(P\circ dP), where PP holds the softmax probabilities. Keeping a separate BF16 output for DD until the backward, as in Attn-QAT, adds 0.5 GiB of peak memory and no measurable step time to TE's FP8 attention at 390M, though computing that output takes an extra product in the forward that TE's fused kernel does not let us time. Computing DD from PP in the backward instead adds 15% to one call of an FP8 attention backward that we wrote, compared with the same kernel reading the saved output. §2.3 reports both costs.
  • The checker works from the policy as written down, one use at a time: the value each use reads, the random draws on which its gap from the reference and its incoming gradient depend, and the possible outcomes of those draws. The rules behind its verdicts do not depend on size, but as built it compares each use's rule with the reference's over every declared outcome, so it runs on small finite cases like those of Appendix C, not on full-size tensors. For shipped operators, Section 4.3 instead tests the requirements' predictions directly, including TE's fused FP8 attention backward at our 390M attention shape.

The authors have addressed the issues I raised, I shall raise my score to a 3 (weak accept).

Thank you very much for reading our replies and for raising your score.

Reviewer_pnpVOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper is an empirical study on the dynamics of low-precision training. The perspective provided is that the quantization rules are part of the learning algorithm, not simply accounting artifacts. The authors test this by revealing numerous failure modes when quantization is not taken into account between the forward and backward passes. If the forward pass uses rounding, then re-using the same rounding during the backward pass introduces bias. The backward pass should inject fresh, independent noise to remain bias free. The paper also proposes a practical methodology for ensuring low-precision training does not suffer these shortcomings.

Critical strengths

The paper addresses an important question: how to do low-precision training well. It runs a comprehensive set of experiments and makes a crucial observation: that quantization should be part of the training algorithm and not treated as simply numerical artifacts. If we train in low precision by introducing rounding in arbitrary places we run the risk of biasing gradients and introducing error that makes training diverge. This is observed by the paper in numerous places and mitigating this problem is a very interesting and important problem. The paper essentially suggests that practitioners be mindful to apply in the backward pass in a way compatible with the rounding process applied in the forward pass. This needs to be done on a case-by-case basis and properly evaluated before training starts or as training continues. Overall, I think that calling the community's attention to these details is important and has the impact to save much training time and compute.

I also like the hints of theoretical and empirical insights that accompany the paper. The figures and plots are useful for understanding.

Critical weaknesses
  1. The paper is really difficult to read, with a lot of verbosity and dense writing. I found it very difficult to understand the deeper technical details, though the big picture is clear from the start. As a result, I think it is imperative for the writing to be simplified so the penultimate points the paper wants to drive forward be better communicated.
  2. I am not sure whether the insights the paper provides are significant enough for an ICLR publication. The insight that quantization must be part of training is widely accepted fact and the paper is not the first, nor the last, to argue that quantization must be viewed as part of the learning process. In addition, the theory is relatively weak and mainly complimentary. I view the contribution mostly as a diagnostic lens: very useful certainly, but not confident it belongs in ICLR at its current state.
  3. Statistics would need strengthening. Only 3 seeds are used for 390M and only 1 for the 1.5B models. Furthermore, only the TE case is a clear indication of failure. Would it be encountered elsewhere?

Overall, the writing is my biggest concern. Apart from that, however, the contribution seems overstated to an extent. The insights are very useful for practitioners, however, so I believe a careful re-write and reframing can make the paper substantially more impactful.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Reply to Reviewer pnpV: presentation, the question the paper asks, and the evidence

Thank you for the review, and for the view that these details can save much training time and compute.

W1: The paper is really difficult to read, with a lot of verbosity and dense writing. … Overall: … the contribution seems overstated to an extent. … a careful re-write and reframing can make the paper substantially more impactful.

We agree that the submission was hard to read. The revision rewrites the abstract and Section 1 in standard terms around one argument and defines each term in a new Table 1. So that no claim reads as broader than its evidence, each result is now stated with the setting where it was tested.
The argument is that the value a backward read receives can change the gradient that training follows, so the backward-state policy is part of the learning algorithm. Section 2 shows that this choice can decide whether training works, and Section 3 judges which value is right against a reference. Since final loss cannot show which gradient training followed, Section 4 checks each read against that reference at its operator, without training.

 

W2: … The insight that quantization must be part of training is widely accepted fact … In addition, the theory is relatively weak and mainly complimentary. I view the contribution mostly as a diagnostic lens …

The paper asks a narrower question than whether quantization belongs in training. Quantization-aware training makes the forward account for rounding. Here the forward has already rounded, and the question is which value each backward read receives. Shipped kernels make this choice, and it can decide whether training fails.
Inside Transformer Engine's own FP8 attention backward, for example, two equally accurate copies of the attention output led training to opposite outcomes in all three tested pairs, as Figure 1b shows.

  • The theory shows why the value must be chosen read by read. For the probabilities that FP8 attention rounds, we prove, under stated assumptions, that no single substitute is always right on average at both of their reads.
  • A prescriptive rule. From the reference we derive which values each read may receive, as a requirement checked at its operator without training. In three tests on PyTorch and TE operators, every prediction we derived about reuse held.

 

W3: Statistics would need strengthening. Only 3 seeds are used for 390M and only 1 for the 1.5B models. Furthermore, only the TE case is a clear indication of failure. Would it be encountered elsewhere?

The revision adds a second 1.5B seed, a 1.5B run to step 9999, and Figure 1's three pairs repeated inside TE's own backward. Outside TE, Qiu & Yao trace a failure of BF16 FlashAttention to the attention backward's correction term, the term our experiments vary.

  • Statistics. At 390M, the contrast is the same on every seed. TE's FP8 attention fails on all three, and replacing only its backward with a BF16 one trains normally on all three. At 1.5B, the BF16 backward ends within 0.004 nats of BF16 on the second seed and on seed 0 continued to step 9999. On the second seed, TE's FP8 attention does not fail by step 4999 but ends 0.13 nats above BF16 in validation loss.
  • Elsewhere. FlashAttention's backward computes this term from the saved output just as TE's does, and the failure Qiu & Yao report occurs in BF16 training, without TE or FP8.

I thank their reviewers for their clarification, rewrite and additional experiments. I think the improvements made to the manuscript are in the right direction to help clarify and explain the contributions. My concerns on reproducibility are also ameliorated by the revision's additional runs. I maintain reservations regarding the significance of the contribution, but now I acknowledge that the point of the paper is in contrast to the insights of quantization-aware training.

I will raise my score to (2) in light of these explanations. The writing can be improved to simplify things and I still find the theory mainly complimentary, so I'm hesitant to recommend acceptance. That being said, I maintain that the findings are insightful and interesting to the community.

Thank you very much for reading our replies and for raising your score. We are glad that the revision has addressed some of your concerns, and we will further improve the writing.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to the AC and all reviewers: the argument and what the revision adds

We thank the three reviewers for their careful reviews and reply to every point in the per-reviewer threads. The revised PDF sets new and changed text in blue.

The argument
In low-precision training, the backward pass reads tensors that the forward pass rounded. Each read can receive the forward's rounded value, the unrounded value or a new rounding. We argue that this choice, the backward-state policy, is part of the learning algorithm, because the value a read receives can change the gradient that training follows. Section 2 shows that the choice can decide whether training works. A choice that matters this much needs a way to judge which value is right, so Section 3 judges each read against a reference, the straight-through gradient of the forward as it ran. It finds that one tensor can need different values at different reads, and that final loss can look normal while a read receives the wrong one. Section 4 therefore derives from the reference a requirement for each read and checks it at its operator, without training.

Reviewers r2Sa and pnpV found the submission hard to read, and we apologize for it. The revision rewrites the abstract and Section 1 around this argument in standard terms, and a new Table 1 maps each term to a standard concept.

What the revision adds

  • Transformer Engine's own backward. Reviewers zqmr and r2Sa ask whether Figure 1, which uses an emulated backward and stochastic rounding, holds for the attention TE ships. Figure 1b repeats its comparison inside TE's own FP8 attention backward. In all three pairs, training fails when the backward's correction term reads the forward's rounding of the attention output, and trains normally when it reads a new rounding from the same distribution. Since TE rounds to nearest, §2.1 also adds a test under that rounding, in which training is normal once the error the term reads no longer enters the loss.
  • The value the correction term should read. Reviewers zqmr and r2Sa ask how the paper relates to Attn-QAT and Qiu & Yao. Both give this term the value our reference prescribes. Attn-QAT does so with a separate high-precision output, and Qiu & Yao do so to isolate their failure. Figure 1's new rounding serves only as a probe, since it is as accurate as the forward's own rounding but is not the rounding the loss saw. In §2.3, a correction added to the gradients of TE's FP8 backward, which gives the term this value, makes training normal on two 390M seeds where TE's attention fails.
  • Cost. Reviewers zqmr and r2Sa ask what these choices cost. Table 12 gives the step time and peak memory of five attention paths. Keeping a BF16 tensor the size of each attention output until the backward, which is the memory Attn-QAT's output needs, adds 0.5 GiB of peak memory and no measurable step time.
  • Longer training at 1.5B. Reviewers zqmr and pnpV ask for more seeds and longer training at 1.5B. §2.2 adds a second seed and continues seed 0 to step 9999. In both, replacing only the attention backward with a BF16 one trains normally and ends within 0.004 nats of BF16 training.

Appendices A, D and E give the related work, setups and full results behind these additions, and the supplementary material gives their code and result data.

AuthorsOpenReview ↗

Summary of the author–reviewer discussion

As the author–reviewer discussion comes to a close, we thank the three reviewers for their careful reviews and replies, and the AC for handling our submission.

What is new. When the backward pass reads a tensor that the forward rounded, it can receive the forward's rounded value, the unrounded value or a new rounding. The paper argues that this choice is part of the learning algorithm, not a memory detail. Three results support this, and the reviewers noted each:

  1. Two equally accurate copies lead training to opposite outcomes. In an emulated FP8 backward and again inside Transformer Engine's own, reusing the forward's rounding of the attention output made training fail, while a new rounding from the same distribution trained normally. Reviewer r2Sa called the experiment "a valid test of the paper's central claim", and Reviewer zqmr wrote that it "makes the effect of this choice pretty clear".
  2. One tensor can need different values at different reads. Under the paper's reference, the straight-through gradient of the forward as it ran, even the exact unrounded value can be wrong at one of a tensor's reads, and final loss does not reveal it. Reviewer r2Sa found the results "non-trivial" and wrote that the idea "can apply beyond just FP8 attention".
  3. Requirements that can be checked at each operator, without training. Their predictions about reusing the forward's rounding held in all three tests on PyTorch and Transformer Engine operators. Reviewer pnpV wrote after the discussion that "the findings are insightful and interesting to the community".

Main concerns and our responses.

  • Novelty (pnpV, r2Sa). Quantization-aware training makes the forward account for rounding, while the paper asks which value each backward read receives once the forward has rounded. Reviewer pnpV agreed "that the point of the paper is in contrast to the insights of quantization-aware training". We build on Qiu & Yao, who trace a failure of BF16 FlashAttention to the same part of attention's backward, but their explanation, biased rounding, cannot account for two copies from one unbiased distribution behaving differently. Reviewer r2Sa, who asked where the two accounts disagree, replied that "The authors have addressed the issues I raised".
  • Cost (zqmr, r2Sa). Each read should receive the value its forward operation used, so weight gradients keep reading the saved rounded copy and its memory saving stays. The cost falls only on reads that need the unrounded value. For attention, keeping a BF16 copy of the output for the backward adds 0.5 GiB of peak memory in our 390M runs and no measurable step time.
  • Seeds and longer runs (zqmr, pnpV). The revision adds runs with more seeds and longer schedules. On "the longer-run validation of the targeted FP8 correction" that Reviewer zqmr asked about, the corrected runs train normally over the full schedule and end at nearly the same validation loss as BF16. Reviewer pnpV wrote that "My concerns on reproducibility are also ameliorated by the revision's additional runs".
  • Presentation (r2Sa, pnpV). The revision rewrites the abstract and Section 1 in standard terms, and a new Table 1 maps each term to a standard concept. Reviewer pnpV found the changes "in the right direction to help clarify and explain the contributions", and we will keep simplifying the writing.

All new results are in the revised PDF, with their code and result data in the supplementary material, and our general comment below lists what the revision adds. We thank the reviewers and the AC again for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.