- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates a late-training instability in BF16 FlashAttention-3 and identifies two sources of gradient error: inaccuracies in the saved forward output and a violation of the softmax score gradient's zero-sum property after BF16 rounding. The latter introduces an error proportional to a common key offset. The proposed GProj correction restores this constraint in the backward contractions. Numerical tests and matched 450M-model training runs show gradient accuracy and final training loss comparable to FP32 attention, with a reported 4.7% increase in training-step time.
Critical strengths
The diagnosis is convincing. Keeping the forward pass and loss bitwise identical while replacing only the attention backward isolates the numerical error from ordinary training instability. The separation of the saved-output error from the remaining BF16 cast error is particularly useful: it explains why repairing the forward pass can stabilize training without making the gradients accurate.
The mathematical analysis leads directly to a practical correction. The paper identifies the erroneous contribution from a nonzero gradient row sum and removes it using the actual mass of the rounded probabilities. Analytic examples, synthetic tests, separate query/key corrections, and full training comparisons support this explanation. Recovering FP32-level query/key gradient accuracy without switching all attention computation to FP32 is a valuable result.
Critical weaknesses
The paper explains the numerical error and how it can amplify during training, but the difference between the Full-AttnRes model and the standard-residual control remains less well understood. Why does the error become severe in the former but remain small in the latter? Appendix K documents a difference in query scale, but does not establish what causes it. Clarifying whether this is related to the residual architecture or other differences in the training setup would help explain when the failure is likely to occur.
Questions for authors
What evidence helps explain why the Full-AttnRes model enters the problematic regime while the standard-residual control does not? How much of this difference can be attributed to the architecture rather than differences in model size and training setup?
2 comments
Thank you for the additional matched experiments. Reproducing the failure in a standard-residual Transformer under comparable model size and training conditions largely resolves my concern about whether the issue is specific to Full-AttnRes. The results strengthen the paper’s conclusions, and I maintain my clear acceptance recommendation.
Thank you for pinpointing the open question. We agree that Appendix K documents the difference in query scale without explaining it. We ran the comparison that answers it.
The failure reproduces in a standard-residual transformer once size and training setup are matched. The control of Appendix K differs from the Full-AttnRes model in two ways: the residual architecture, and the size (it has 102M parameters and 12 layers, against 450M). To separate the two we trained a standard pre-norm transformer with the same width and depth as the 450M Full-AttnRes model, without attention over residual depth, with the same recipe, data order and 50B-token schedule, once with unmodified FA3 and once with the forward repair plus GProj. Checkpoints were measured exactly as in Table 8 (exact FP32 attention, the same two fixed documents; errors are relative to the true gradient).
Under FA3, layer 1 of the standard transformer reaches logit scale 9.0×10⁴ with 72% of its rows concentrated on one key, and FA3's query-gradient error there is 8.1 times the true gradient. In the GProj run no layer exceeds logit scale 15. The losses coincide for the first 5,000 updates and then separate, ending 0.015 nats apart; the FA3 gradient norm rises from 0.17 to 0.53 while the GProj one falls to 0.05.
This answers the question in three parts.
An independent public comment in this thread reports the same late-checkpoint gradient blow-up, and the same recovery with GProj, in a third model: a 52.5M standard residual Transformer trained with BF16 attention. We have not reproduced it.
Revision. Appendix K gains the matched standard-transformer runs with this table, and Section 6 states when the failure is expected: at large query/key scale, which the standard architecture also reaches at this size and recipe.