acceptodds
Under review as a conference paper at ICLR 2027

Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training

Abstract

BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are widely trusted. When we used FlashAttention-3 to pretrain a 450M-parameter transformer on 50B tokens, however, we ran into a problem: training was healthy for 25B tokens, then the gradient norm grew a thousandfold and the loss ended 0.2 nats above FP32 attention, without a single NaN. Recomputing the attention backward of just two layers in FP32 removes almost all of the excess gradient. Part of the cause is known: a fused multiply-add in the forward softmax, so far treated as an extreme-input NaN case and never fixed in FlashAttention-3. Repairing it stops the blow-up, but the query gradient is still wrong by more than its own size, and training still drives attention logits to thousands of times their size under accurate gradients. The remaining error comes from a broken conservation law. The softmax score gradient sums to zero along every row, which makes the query gradient blind to where the keys sit as a group; rounding it to BF16 leaves a small nonzero sum that leaks the mean key into the gradient, and the leak grows exactly as late training makes keys large and attention sharp. We introduce GProj (gauge projection), which restores the zero sum after the cast with two rank-one corrections. It cuts the remaining median query/key gradient errors from 219%/13% to 0.34%/0.37%, on par with FP32 attention, for 4.7% more time per training step. In matched from-scratch runs it trains to the same loss as FP32 attention, while FlashAttention-3 and key smoothing both destabilize.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
FlashAttention, numerical precision

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_x6MUOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates a late-training instability in BF16 FlashAttention-3 and identifies two sources of gradient error: inaccuracies in the saved forward output and a violation of the softmax score gradient's zero-sum property after BF16 rounding. The latter introduces an error proportional to a common key offset. The proposed GProj correction restores this constraint in the backward contractions. Numerical tests and matched 450M-model training runs show gradient accuracy and final training loss comparable to FP32 attention, with a reported 4.7% increase in training-step time.

Critical strengths

The diagnosis is convincing. Keeping the forward pass and loss bitwise identical while replacing only the attention backward isolates the numerical error from ordinary training instability. The separation of the saved-output error from the remaining BF16 cast error is particularly useful: it explains why repairing the forward pass can stabilize training without making the gradients accurate.

The mathematical analysis leads directly to a practical correction. The paper identifies the erroneous contribution from a nonzero gradient row sum and removes it using the actual mass of the rounded probabilities. Analytic examples, synthetic tests, separate query/key corrections, and full training comparisons support this explanation. Recovering FP32-level query/key gradient accuracy without switching all attention computation to FP32 is a valuable result.

Critical weaknesses

The paper explains the numerical error and how it can amplify during training, but the difference between the Full-AttnRes model and the standard-residual control remains less well understood. Why does the error become severe in the former but remain small in the latter? Appendix K documents a difference in query scale, but does not establish what causes it. Clarifying whether this is related to the residual architecture or other differences in the training setup would help explain when the failure is likely to occur.

Questions for authors

What evidence helps explain why the Full-AttnRes model enters the problematic regime while the standard-residual control does not? How much of this difference can be attributed to the architecture rather than differences in model size and training setup?

2 comments

Thank you for pinpointing the open question. We agree that Appendix K documents the difference in query scale without explaining it. We ran the comparison that answers it.

The failure reproduces in a standard-residual transformer once size and training setup are matched. The control of Appendix K differs from the Full-AttnRes model in two ways: the residual architecture, and the size (it has 102M parameters and 12 layers, against 450M). To separate the two we trained a standard pre-norm transformer with the same width and depth as the 450M Full-AttnRes model, without attention over residual depth, with the same recipe, data order and 50B-token schedule, once with unmodified FA3 and once with the forward repair plus GProj. Checkpoints were measured exactly as in Table 8 (exact FP32 attention, the same two fixed documents; errors are relative to the true gradient).

tokensFA3 run: layer-1 logit scalelayer-1 rms QQ / KKlayer-1 rows with pmax⁡>0.999p_{\max}>0.999layer-1 dQdQ error, FA3 / cuDNNGProj run: largest logit scale in any layer
23.1B1,37734 / 4115%0.36 / 0.3612
33.6B10,89996 / 11447%1.5 / 0.5414
39.8B38,278183 / 20967%2.3 / 0.5215
50.0B89,977282 / 31972%8.1 / 1.0515

Under FA3, layer 1 of the standard transformer reaches logit scale 9.0×10⁴ with 72% of its rows concentrated on one key, and FA3's query-gradient error there is 8.1 times the true gradient. In the GProj run no layer exceeds logit scale 15. The losses coincide for the first 5,000 updates and then separate, ending 0.015 nats apart; the FA3 gradient norm rises from 0.17 to 0.53 while the GProj one falls to 0.05.

This answers the question in three parts.

  • The residual architecture does not decide whether the failure occurs; model size and setup do. With size, recipe and data matched, the standard transformer enters the regime too. The control of Appendix K stayed out of it because, as a smaller model, its query RMS stayed below 45, which is below the scale at which the error appears (in a standard block scaled to a fixed operand RMS, the FA3 error is below 1% up to RMS 64, 10% at RMS 128 and 50% at RMS 212).
  • The architecture changes where the failure appears and how strong it is. In Full-AttnRes the regime appears in layers 5 and 11 and costs 0.2 nats; in the standard transformer it appears in layer 1 and costs 0.015 nats. We do not yet have an explanation for which layer is affected.
  • The gradient error sustains the growth in both architectures. In each, the arm with an accurate attention gradient keeps every layer at logit scale below 30, from the same initialization and data (Table 8 and the table above). The setup is the precondition: Muon without query–key normalization is the setting in which the literature reports large attention logits (Wortsman et al., 2023; the Kimi K2 report).

An independent public comment in this thread reports the same late-checkpoint gradient blow-up, and the same recovery with GProj, in a third model: a 52.5M standard residual Transformer trained with BF16 attention. We have not reproduced it.

Revision. Appendix K gains the matched standard-transformer runs with this table, and Section 6 states when the failure is expected: at large query/key scale, which the standard architecture also reaches at this size and recipe.

Thank you for the additional matched experiments. Reproducing the failure in a standard-residual Transformer under comparable model size and training conditions largely resolves my concern about whether the issue is specific to Full-AttnRes. The results strengthen the paper’s conclusions, and I maintain my clear acceptance recommendation.

Reviewer_mhPQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper diagnoses silent late-training gradient corruption in BF16 FlashAttention-3. It separates an existing forward softmax/saved-output problem from a second error: casting the softmax score gradient to BF16 violates its exact zero-row-sum property, causing a spurious query-gradient component proportional to the mean key. The proposed GProj method projects the cast gradient back onto the zero-sum subspace using low-rank corrections for dQ and dK. Evidence includes an analytic construction, error bounds, fixed-forward diagnostic interventions, synthetic tests, and matched 450M-model pretraining. On eight captured inputs, median dQ/dK relative errors decrease from 219%/13.3% for forward-repaired FA3 to 0.342%/0.371% with GProj; a 50B-token run matches FP32 attention training loss at roughly 4.7% additional step time (Tables 1, 4, 8).

Critical strengths
  1. Mechanistic diagnosis, not just a patch. Equations (3)–(7), the exact four-key counterexample, and the fixed-forward interventions provide a coherent explanation for how a nearly accurate forward can yield seriously wrong gradients. This suggests a useful invariant-based way to test low-precision kernels.
  2. Strong targeted validation at practical cost. Identical-forward ablations isolate the backward fix (Table 1); the synthetic suite and long pretraining comparison support the utility of GProj, whose measured full-step overhead is modest (Table 4).
Critical weaknesses
  1. Limited evidence that the late-training pathology is typical. The principal 50B-token result uses a single seed and a particular 450M Full-AttnRes architecture with QK normalization disabled (Appendices G–H). The plain-transformer controls in Appendix K show relatively small numerical discrepancies and do not reproduce the same mechanism-driven blow-up. This limits how strongly the results establish broad training benefits.
  2. Narrow performance and implementation coverage. GProj is evaluated on Hopper/BF16, head dimension 64, and one main sequence length for the full-step overhead; its additional dK pass can become more expensive when attention dominates compute (Section 6). Results on other head dimensions, sequence lengths, GPU generations, and common attention configurations would materially strengthen the deployment claim.
Questions for authors
  1. Can the authors provide another full-length training seed or a second standard transformer architecture that enters the high-logit regime, to establish the frequency and robustness of the reported failure and GProj's benefit?
  2. How does complete-step overhead change with sequence length, head dimension (e.g., 128), and attention-heavy model configurations? Is the second dK pass still attractive against optimized FP32 fused attention in these regimes?
  3. Could the authors clarify whether the projection alone, without subtract-before-scale forward repair, was tested in training? This would help separate the necessity of the two fixes beyond local gradient ablations.
Reviewer_Dez4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates numerical instability in BF16 FlashAttention-3 during large-scale Transformer pretraining. The authors identify two sources of gradient errors: (1) an FMA-related forward softmax issue that introduces errors in the saved attention output, and (2) a violation of the zero-sum property of the softmax score gradient caused by BF16 rounding.

The main contribution is GProj (Gauge Projection), which restores the zero-sum property after the BF16 cast through two rank-one corrections in the query and key gradient computations. The authors provide a mathematical analysis of the resulting gradient error, kernel-level implementation, synthetic numerical evaluations, and from-scratch training experiments with a 450M-parameter Transformer over 50B tokens.

The results show that GProj substantially reduces query and key gradient errors compared with FA3-SBS, achieving median relative errors of 0.342% and 0.371%, respectively, with approximately 4.7% additional training-step overhead. GProj also achieves a final training loss comparable to FP32 attention in the reported experiments.

Critical strengths
  1. The paper identifies a zero-sum constraint in softmax gradients that ensures query gradient invariance. It demonstrates how BF16 casting breaks this property and proposes a mathematically sound, low-overhead projection to restore it.
  2. The proposed GProj significantly improves gradient accuracy, reducing median query and key errors to near-FP32 levels (~0.34% and 0.37%). It adds only a 4.7% training-step overhead, offering valuable numerical and engineering insights.
Critical weaknesses
  1. Insufficient forward numerical accuracy and incomplete comparison with prior work.
  • FA3-SBS only addresses a specific FMA-related error in the forward softmax computation. It does not fundamentally eliminate numerical errors in attention output accumulation and storage.
  • Qiu and Yao (ICLR 2026) have shown that retaining the attention output in FP32 can address an important source of forward numerical bias. The current paper does not adequately establish why FA3-SBS is a sufficient forward baseline.
  • A controlled comparison combining FP32 attention outputs with native and GProj backward passes is missing. Such an experiment is necessary to isolate the actual contribution of GProj beyond improving the accuracy of the saved attention output.
  1. Unclear statistical and training implications of the proposed gradient leakage.
  • Although the paper shows that BF16 casting breaks the zero-sum property and introduces a key-mean-dependent gradient error, it does not establish whether this error produces persistent biased updates across training steps.
  • An instantaneous gradient error does not necessarily imply systematic bias accumulation; it may instead increase gradient variance.
  • The paper lacks statistical analysis of the signed mean, variance, and temporal correlation of the row-mass leakage. It also does not quantify how much this leakage contributes to accumulated parameter-gradient errors and training instability.
Questions for authors
  1. Can the authors evaluate FP32 attention outputs with both native and GProj backward passes to isolate the additional benefit of GProj?
  2. Does the row-mass leakage exhibit a persistent directional bias across training steps, or does it primarily increase gradient variance? Can the authors provide statistics on its mean, variance, and temporal correlation?

Comments on the paper

1 comment
~jun_xu21OpenReview ↗

Independent validation of SBS and GProj: accurate gradients and fast BF16 training in audio entropy modeling

We would like to share our experience applying the ideas in this paper to Transformer-based entropy modeling of discrete audio codec tokens. Our model is a standard residual Transformer with approximately 52.5M parameters, trained on RTX 4090 GPUs with a head dimension of 128.

We encountered severe gradient distortion at late checkpoints during BF16 attention training. The paper provided a useful explanation and motivated us to independently implement and evaluate its proposed corrections. In a fixed-forward diagnostic, we kept the forward outputs bitwise identical and changed only the attention backward computation, using the same BF16 forward with FP64 attention backward as the reference. For one late checkpoint evaluated on two fixed inputs, the relative L2 gradient error decreased from 4011%–4793% with native BF16 attention to 0.62%–0.71% with GProj. At an early checkpoint, the corresponding native BF16 errors were only 0.24%–0.30%.

We subsequently trained a model from scratch using SBS + GProj for 60 epochs. Under approximately equal training exposure, we compared actual compression performance across six operating points on the same 40 audio recordings. Native BF16 training produced compressed-payload rates 1.99%–36.19% higher than the FP32-trained baseline, indicating substantially poorer compression, particularly at the lower-rate operating points. With SBS + GProj, the relative payload-rate differences narrowed to −2.47% to +0.06%, recovering compression performance comparable to FP32. All models were evaluated using the same tokens, FP32 inference, and arithmetic-coding pipeline.

Retaining BF16’s speed advantage was an important motivation for us, rather than simply switching to FP32 training. In an autoregressive-model training benchmark with cached codec tokens, using the same RTX 4090 and batch size of four, our SBS + GProj implementation reduced training-step time from 0.521 seconds to 0.104 seconds—approximately a 5× speedup over our FP32 implementation, with both running in eager mode. Relative to native BF16, the corrections introduced a moderate overhead: in a matched dual-RTX-4090 training-compute benchmark with a global batch size of 48, step time increased from 0.485 to 0.607 seconds, approximately 25%, while throughput decreased from 99.0 to 79.1 samples per second, retaining approximately 80% of native BF16 throughput. This provided a useful practical trade-off: compression performance comparable to FP32 while preserving most of BF16’s training throughput.

We found this work valuable both for diagnosing a real numerical issue and for obtaining compression performance comparable to FP32 at substantially higher training speed. Our experience provides an additional application example in a standard residual Transformer, an audio-compression task, and a different hardware configuration from the paper's main experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.