acceptodds
Under review as a conference paper at ICLR 2027

One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse

Abstract

A bfloat16 transformer can train normally for many steps, then collapse abruptly. Prior work links the collapse to attention errors whose shared structure lets rounding contributions compound, and shows QK normalization disrupts that compounding. Distinct low-precision errors trigger the same collapse, leaving unclear whether each needs a fix at its source or one shared route can be blocked instead. We isolated the fault behind a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where an fp32 streaming core repairs it, and turned it into an assay for moving a controlled error across sources. Using it, we found that errors placed outside attention still drove the same QK spectral runaway, and that correcting only QK kept training stable while the fault stayed active. This is a *source-channel dissociation*: fault source is not failure channel. It held across tested architectures and scales, and reproduced on a second GPU architecture. As a causal probe, projecting each update off the current QK weights' leading three singular directions held the query projection's largest singular value to 11.1, whereas removing equal energy elsewhere left it at 237: the QK channel causally drives the early runaway. What lets the injected error in is *temporal sign-coherence*, its per-head sign persisting across steps, not aggregate deviation; once inside, the runaway shows as attention-logit saturation. If one channel carries these faults, a QK-local action switched on mid-run should contain each. *QK-Guard*, a dormant controller, tests this by switching on parameter-free QK normalization at the first monitored threshold crossing. On the runs designated for this test, the QK-local action prevented the failure of each matched or same-configuration unguarded run; on plain GPT-2, all 12 final train and validation losses were within 0.03 nat of same-configuration always-on QK-norm, and both methods ran 60k steps without collapse. Intervention at the QK locus therefore suffices in place of a fix at each source.

Primary area:
optimization
Keywords:
low-precision training, training stability, attention collapse, query-key channel, temporal sign-coherence, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2eamOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how numerical errors from different sources can lead to attention instability through a shared query–key pathway. Using a reproduced BF16 streaming-accumulator fault and controlled weight-gradient quantization, it studies QK spectral growth through gradient replacement and direction-specific interventions. It also introduces QKGuard, which monitors attention logits and permanently enables parameter-free per-head QK normalization after a threshold crossing.

Critical strengths
  1. The paper goes beyond correlating spectral growth with training collapse. Selective QK-gradient correction while leaving the fault active, together with direction-specific interventions and matched-energy controls, provides meaningful causal evidence for the QK pathway in the tested settings.

  2. The protection mechanism is evaluated across multiple architectures, fault locations, seeds, and training horizons. Matched guarded and unguarded runs, along with longer-duration experiments, strengthen the evidence that QK normalization can prevent collapse under the studied conditions.

Critical weaknesses
  1. The practical representativeness of the failure settings remains insufficiently established. The initial collapse arises in a particular implementation that stores streaming-attention carries in BF16, whereas standard FP32-accumulating implementations do not collapse within the reported comparison windows. Additional evidence primarily uses controlled post-reduction weight-gradient quantization. These experiments are useful for causal analysis, but do not establish how frequently the identified pathway is consequential in commonly used, correctly implemented low-precision training workflows. This limits the demonstrated practical scope of QKGuard.

  2. The incremental benefit of threshold-triggered protection over established, always-on QK normalization is unclear. QKGuard leaves the numerical error source active and intervenes downstream; preventing collapse therefore does not by itself establish recovery of fault-free training quality. The reported closeness to always-on QK normalization supports the effectiveness of delayed intervention, but does not establish a clear advantage in quality, efficiency, or preservation of the original training behavior. The mechanism analysis is more compelling than the demonstrated added value of the proposed deployment strategy.

Questions for authors
  1. Do the existing experiments include naturally occurring instability in a standard low-precision training pipeline without the BF16-carry implementation or imposed post-reduction gradient quantization? If so, please identify the configuration and evidence connecting that instability to the same QK pathway. Otherwise, please clarify the intended scope of the practical claims.

  2. What measurable benefit does QKGuard provide over always-on QK normalization? Please clarify whether the current results establish an advantage in training quality or total runtime, including monitoring overhead, and how guarded endpoints compare with a fault-free reference rather than only an always-normalized reference.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer 2eam: standard training pipelines and what the switch adds

Thank you for the review, and for crediting the causal design.

W1: … These experiments are useful for causal analysis, but do not establish how frequently the identified pathway is consequential in commonly used, correctly implemented low-precision training workflows. … Q1: Do the existing experiments include naturally occurring instability in a standard low-precision training pipeline without the BF16-carry implementation or imposed post-reduction gradient quantization? If so, please identify the configuration and evidence connecting that instability to the same QK pathway. …

Yes. The new §2.2 shows that PyTorch's fused bf16 attention, with no injected error and no custom kernel, collapses GPT-2 small when the learning rate stays near its peak without weight decay.
All 6 runs at peak learning rates 6e-4 and 8e-4 collapse. The kernel is PyTorch's own scaled-dot-product attention under bf16 autocast, which accumulates in fp32. The instability follows the same QK pathway as the faulty kernel: σ(WQ)\sigma(W^Q) starts to run away before every collapse, from under 10 to peaks over 400, and switching on QK normalization when the monitored attention-logit statistic crosses 30 rescues all 6 runs. With attention computed explicitly in fp32 and everything else unchanged, none of 6 runs collapses. The weight-gradient experiments you mention ran on this same fused path, and as our general comment explains, the revision now reports them as fused-attention runs.

 

W2: … preventing collapse therefore does not by itself establish recovery of fault-free training quality. … The mechanism analysis is more compelling than the demonstrated added value of the proposed deployment strategy. Q2: What measurable benefit does QKGuard provide over always-on QK normalization? Please clarify whether the current results establish an advantage in training quality or total runtime, including monitoring overhead, and how guarded endpoints compare with a fault-free reference rather than only an always-normalized reference.

No. The current results show no advantage over QK normalization from the start in training quality and do not establish one in runtime.
As point 2 of our general comment explains, the revision presents the switch as a test of the analysis, which we agree is the stronger part of the paper. Guarded runs end where runs with QK normalization from the start end, and also close to runs in which the error is removed. Under fused attention, guarded runs reach validation loss 3.20–3.26, and runs of the same recipe with fp32 attention 3.18–3.25. With the faulty kernel at 350M, switching on QK normalization when the statistic crosses 30 ends at 4.04 train loss, and switching the kernel's computation to fp32 at the same moment ends at 4.03. For runtime, Appendix C shows that monitoring adds no measurable time per update, and after the switch the run pays the cost of QK normalization.

Reviewer_kqtWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the instability of some transformer architectures when trained in bf16 without QK norm. It shows that the initial cause of a large loss spike can come from multiple places in the network and can be detected in the QK activations, and that applying QK norm only once the initial signs of the spike are detected can prevent the run from diverging.

Critical strengths

S1. The work demonstrates successfully preventing a run from a loss spike that would otherwise experience one, in a somewhat realistic (if outdated) setting.

Critical weaknesses

W1. The proposed method, QK-Guard, is a way of patching stability for architectures that don't train stably in bf16 without QK norm. However, the work does not clearly characterize (1) if this is an actual concern for modern architectures (my experience is generally no, because of post-norm); (2) the benefit of doing this instead of just applying QK norm from the start of training.

W2. The primary takeaway I got from the paper is "QK norm can be turned on partway through training if stability issues begin to arise," which exists in the literature: https://marin.community/blog/2025/10/28/marin-32b-retro/. This work should cite and discuss the Marin report.

W3. The paper is extremely hard to follow. It uses dense, non-standard jargon and in parts reads like an experiment log rather than a conference publication. I strongly suspect that the text was largely or wholly generated by an AI model. While I generally would not consider presentation issues a critical weakness of a paper, in this case I think the presentation significantly impacts the potential value of the work.

Questions for authors

Q1. When do you think using QK-Guard is preferable to using QK-norm?

Q2. Another point of the work is that the source of the eventual instability is not necessarily in that layer's attention, even if the actual instability starts with QK spectral norms spiking. Can you elaborate on the importance or actionability of this insight?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer kqtW: modern architectures, QK normalization from the start, the Marin report and presentation

Thank you for the review, and for noting that the switch prevents a collapse that would otherwise occur.

W1: … the work does not clearly characterize (1) if this is an actual concern for modern architectures (my experience is generally no, because of post-norm) …

Post-norm leaves the queries and keys unnormalized, so the attention logits can still grow with the query and key weights, and in our runs they do. QK normalization bounds the logits directly and prevented collapse in every setting where we tested it.

  • Under PyTorch's fused attention at learning rate 1e-3, two sandwich-norm runs still show the growth, with σ(WQ)\sigma(W^Q) reaching 354 and 393. Neither collapses within 20,000 steps, while a pre-norm run with the same settings as one of them collapses at 7,700. In that matched pair, post-norm thus prevented the collapse, as your experience suggests, without stopping the growth.
  • With the reproduced bf16 kernel behind the FlashAttention collapse that Qiu & Yao reported, all three post-norm runs collapse, one of them with the OLMo 2 ordering, while both post-norm variants with QK normalization stay healthy.
  • The LLaMA-style model, with RoPE and RMSNorm, collapses in 4 of 4 seeds with that kernel and in none with QK normalization.

 

W1: … (2) the benefit of doing this instead of just applying QK norm from the start of training. Q1: When do you think using QK-Guard is preferable to using QK-norm?

Where QK normalization can be used from the start, we would use it, as many current models do. A mid-run switch such as QK-Guard is the option for a run already under way without it, as in the Marin case.
As point 2 of our general comment explains, the paper uses the switch to test its analysis during training.

 

W2: The primary takeaway I got from the paper is "QK norm can be turned on partway through training if stability issues begin to arise," which exists in the literature … This work should cite and discuss the Marin report.

Thank you for the pointer. §5.4 of the revision now cites and discusses the Marin report. The revised paper's takeaway is the analysis of how low-precision attention errors collapse training, which also explains why such a switch works for them.
The Marin report switches a 32B run to QK normalization to end a loss-instability episode. Our analysis shows that low-precision attention errors compound in the QK weights across training steps, so normalizing queries and keys stops the failure while the error source stays active. The paper tests this with the switch in controlled comparisons: with the faulty kernel, at a fixed moment, QK normalization and a QK clip rescue the run, while the softmax-side fix proposed for the kernel's rounding bias and a tenfold learning-rate cut do not. The analysis also shows that how fast an error compounds depends on whether its sign persists across steps, not only on its size.

 

W3: The paper is extremely hard to follow. It uses dense, non-standard jargon and in parts reads like an experiment log rather than a conference publication. I strongly suspect that the text was largely or wholly generated by an AI model. …

We agree that the submitted text was hard to follow. The revision rewrites the abstract and the relevant sections in plain language, drops the terms we had coined, defines the remaining ones in a glossary in Appendix E and adds Table 1, which maps each experiment to its finding. As our AI use statement says, we used language-model tools in drafting and polishing the text, and we take responsibility for it.

 

Q2: Another point of the work is that the source of the eventual instability is not necessarily in that layer's attention … Can you elaborate on the importance or actionability of this insight?

What stands of this point is that, under both sources, the failure runs through the QK projections, so it can be stopped there without first finding or repairing its source.
The faulty kernel corrupts every gradient, yet restoring only the QK gradient prevents collapse in four architectures, and QK normalization switched on after the runaway has begun rescues every tested run whose unguarded counterpart collapses. The part about sources outside attention did not hold up under an fp32-attention control and is withdrawn, as our general comment explains.

A second finding is also actionable: the size of a kernel's per-step error does not tell a harmless error from one that compounds. Scrambling the sign of each element of the injected error, at unchanged magnitude, collapses none of 4 runs, against 5 of 5 for the unaltered error. A low-precision kernel should therefore be checked for whether its error keeps its sign across steps, not only for its size.

Thanks for the response.

Where QK normalization can be used from the start, we would use it, as many current models do. A mid-run switch such as QK-Guard is the option for a run already under way without it, as in the Marin case.

I think this is, unfortunately, a critical flaw of the paper. Marin has already demonstrated the same method for this edge case, and so it's not clear what contribution this work makes.

The revision rewrites the abstract and the relevant sections in plain language, drops the terms we had coined, defines the remaining ones in a glossary in Appendix E and adds Table 1, which maps each experiment to its finding.

Can you upload the revision?

AuthorsOpenReview ↗

Reply to Reviewer kqtW's follow-up: the revised PDF and what the paper adds to the Marin report

Thank you for the follow-up.

Can you upload the revision?

It is uploaded: the PDF linked at the top of this page is the revised version.
Its abstract and §1 now state the paper's contribution directly, Table 1 maps each experiment to its finding, and §5.4 discusses the Marin report.

 

… Marin has already demonstrated the same method for this edge case, and so it's not clear what contribution this work makes.

We agree that switching on QK normalization mid-run is not new. The paper's contribution is an explanation of the failure that such a switch stops.
The Marin report shows that the switch ended the loss spikes of a 32B run, and it leaves their cause open. For low-precision attention errors, the paper shows two things:

  1. The errors compound in the QK weights across training steps: with the faulty kernel, every gradient carries the error, yet restoring only the QK gradient prevents collapse in four architectures. This is why acting on queries and keys stops the collapse, while generic fixes such as gradient clipping did not stop the failure, in our runs or in the Marin run.
  2. How fast an error compounds depends on whether its sign persists from step to step, not only on its size, so a kernel's per-step error size alone cannot tell a harmless error from one that will compound. The Marin team's debugging notes in issue #1368 conjecture a similar idea for whole gradients, that gradients stay more aligned than usual over a few consecutive steps before a spike, but the notes record no test of it. We test the idea directly on the error, by injecting the faulty kernel's exact output error into an otherwise fp32 kernel: giving the whole error a random sign at each step delays collapse about tenfold, and regrouping the same signs into long runs restores the early collapse.
Reviewer_8ce6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how low-precision numerical errors can cause transformer training to collapse. It first traces a reproduced GPT-2-class failure to the low-precision accumulator in streaming softmax and shows that an FP32 streaming core repairs it. The authors then introduce quantization errors in different parts of the model, including outside attention. Across the tested settings, correcting the query and key (QK) gradient path can keep training stable while the original fault remains active. The authors also show that removing the components of optimizer updates aligned with the leading directions of the QK weights suppresses growth in their largest singular values. They use these findings to motivate QK-Guard, which periodically checks attention logits and permanently enables per-head QK normalization after the monitored statistic crosses a threshold.

Critical strengths

The fault-routing experiments are the strongest contribution. The authors independently vary where the error is introduced and where gradients are corrected. This directly tests whether an error originating outside attention can be contained by intervening only on QK.

The directional intervention has a convincing control. Removing update components along the leading QK weight directions held the largest query-weight singular value between 11.1 and 15.4 across four runs. Removing equal energy elsewhere, with the total update norm matched, let it reach 237 to 491. Two control runs collapsed and two showed spectral growth without collapsing during the observation period. This distinction makes the evidence for the cause of spectral growth stronger than the evidence that spectral growth alone causes collapse.

The guard is evaluated on matched training trajectories. In six additional rope pairs, every unguarded run collapsed after the trigger point and every guarded run reached the planned horizon. In the 60K-step experiment, all three guarded runs and all three always-normalized runs finished with similar validation losses, while all three unprotected runs collapsed.

Critical weaknesses

The sign experiments show that error magnitude alone cannot explain the observed failures. They do not fully isolate temporal sign persistence as the cause. Randomizing each head’s error sign at every step preserves its per-step magnitude, but changes both how long a sign persists and how much signed bias accumulates over time. The paper’s conclusion would be stronger if it separated those effects. The practical benefit of delaying normalization is unclear. QK-Guard and always-on QK normalization reach nearly identical validation losses in the 60K-step comparison, and the guard activates in the reported fault-free run. The rescue results establish that delayed activation can work. They leave open when a practitioner would prefer it to always-on normalization, especially once periodic monitoring is included.

The presentation makes it difficult to track which experiments support each claim. The reproduced kernel fault, injected backward error, and quantized weight gradients are distinct interventions. A compact experiment table in the main text could identify the fault, correction, number of seeds, run horizon, and whether the outcome was spectral growth or loss collapse.

Questions for authors

Can you vary sign persistence while matching each head’s accumulated signed error and error magnitude over a fixed window? For example, could you compare long blocks of consistent signs with an interleaved sequence built from the same signed contributions? This would clarify whether persistence adds explanatory power beyond accumulated bias.

What does QK-Guard offer over always-on QK normalization? Please report the monitoring overhead, the fraction of training steps for which normalization stays off, and how often the guard activates across several fault-free seeds.

What evidence distinguishes a shared failure mechanism from several failure modes that respond to the same QK intervention? In particular, how does the spectral account apply to the RoPE failure that the paper describes as spectrally flat?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer 8ce6: sign persistence, the switch, the experiment table and the shared mechanism

Thank you for the detailed review and the concrete comparison you propose. Of the fault-routing experiments you rate strongest, the gradient corrections stand: with the faulty kernel, restoring only the QK gradient prevents collapse in four architectures, and restoring the MLP gradient does not. The quantized weight gradients no longer count as a separate error source, as our general comment explains. The equal-energy control you note, re-run with four new seeds over 32k steps, now collapses in all four.

Q1: Can you vary sign persistence while matching each head’s accumulated signed error and error magnitude over a fixed window? For example, could you compare long blocks of consistent signs with an interleaved sequence built from the same signed contributions? …

We matched the number of + and − steps in every 2,000-step block and varied only their order. Grouping the signs into long stretches, as in your example, instead of ordering them at random moves collapse from 16,400 and 16,300 steps to 1,300 and 1,100.
Each schedule multiplies the whole injected error by one sign per step, and within a seed all schedules start from the same state and see the same data. With the sign fixed at +1, which leaves the error unaltered, training collapses at 900 and 1,900 steps, and at −1 at 2,200 and 1,600, so reversing the error's direction does not explain the delay. In a follow-up that flips the sign about every 125 steps, all four runs still collapse by step 3,200, far earlier than with random signs. The schedules fix the signs exactly but not each head's accumulated error, because the injected error is recomputed from each run's own state.

 

W2: … the guard activates in the reported fault-free run. … They leave open when a practitioner would prefer it to always-on normalization, especially once periodic monitoring is included. Q2: What does QK-Guard offer over always-on QK normalization? Please report the monitoring overhead, the fraction of training steps for which normalization stays off, and how often the guard activates across several fault-free seeds.

Where QK normalization can be used from the start, a practitioner should use it, and the switch does not improve on it.
As point 2 of our general comment explains, the paper uses the switch to test its analysis, and it serves runs already under way without QK normalization. The quantities you ask for:

  • Monitoring overhead: no measurable time per update, +0.36% with a 95% interval of −3.6% to +4.3%.
  • Steps with normalization off: 28–39% of training in the six guarded 20,000-step runs, and 7–10% in the three guarded 60k-step runs.
  • Activation without an injected error: the statistic crosses 30 in none of 8 runs with fp32 attention and in all 9 with fused bf16 attention at peak learning rates 3e-4 to 8e-4, 7 of which collapse within 20,000 steps. The run you note uses fused attention at 1e-3. Without the guard, its configuration also crosses 30 and σ(WQ)\sigma(W^Q) reaches 380.1, but it does not collapse in 9,900 steps.
  • Cost on a healthy run: switching QK normalization on at step 6,000 of a run with a decayed learning rate and weight decay 0.1 raises the final validation loss by 0.0155 and 0.0170 nats.

 

W3: … A compact experiment table in the main text could identify the fault, correction, number of seeds, run horizon, and whether the outcome was spectral growth or loss collapse.

We added it as Table 1, which gives for each experiment the error source, what is changed, the number of seeds, the outcome measured and the finding it supports. Run horizons vary within an experiment, so the result tables and Appendix A give them.

 

Q3: What evidence distinguishes a shared failure mechanism from several failure modes that respond to the same QK intervention? In particular, how does the spectral account apply to the RoPE failure that the paper describes as spectrally flat?

Shared mechanism. Beyond responding to the same intervention, both sources build the failure up in the QK weights over training steps. Under fused attention, removing from each update its component along the QK weights' three leading singular directions stops the runaway, while removing the same amount in other directions does not. With the faulty kernel, restoring only the QK gradient prevents collapse in every architecture tested, and its injected error collapses training far sooner when its sign persists from step to step, as the schedules above show. Both sources also show the same σ(WQ)\sigma(W^Q) runaway on plain GPT-2.

RoPE. "Spectrally flat" meant that σ(WQ)\sigma(W^Q) does not accelerate before the RoPE collapse, as Appendix A.5 now says. The failure still runs through the QK projections: restoring only the QK gradient prevents collapse in all three RoPE seeds, restoring the MLP gradient rescues none, and the monitored statistic crosses 30 at least 1,500 steps before the unguarded RoPE run collapses.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to the AC and all reviewers: the revised claim, the shared concerns and the revisions

We thank the three reviewers for their careful reviews. We reply to every point in the per-reviewer threads. This comment gives the revised claim, the three concerns the reviews share and the list of revisions.

The revised claim
Low-precision attention errors collapse training by compounding in the QK weights across training steps. The revised paper shows this for two sources of such errors in GPT-2-class models: the reproduced streaming-softmax kernel and PyTorch's fused bf16 attention.

The revision withdraws one claim of the submission, that errors arising outside attention also reach the QK projections. The submitted §3.2 based this claim on runs that rounded weight gradients to an MXFP4-style four-bit format. Those runs used fused bf16 attention at learning rate 1e-3 without weight decay, a setting in which the fused attention alone makes the query projection's largest singular value, σ(WQ)\sigma(W^Q), run away. With attention computed in fp32, the same rounding collapses none of 6 runs, so the collapses in the submitted runs cannot be attributed to the rounding. The other experiments on those runs stand as fused-attention experiments: removing the QK weights' leading directions from each update, the mid-run switch to QK normalization and the 60k-step runs.

1. The failure also arises with PyTorch's standard fused attention
Reviewers 2eam and kqtW ask whether this failure matters in practice. With PyTorch's fused bf16 attention and no injected error, GPT-2 small collapses in all 6 runs when the learning rate stays near its peak without weight decay, while with attention computed in fp32 none of 6 runs collapses. The failure follows the same path as with the faulty kernel: σ(WQ)\sigma(W^Q) runs away before the collapse, and acting on the QK projections stops it.

2. The contribution is the analysis, and QK-Guard is how the paper tests it
All three reviews ask what QK-Guard adds over QK normalization from the start of training. The paper's contribution is the analysis of how low-precision attention errors collapse training. Because the errors compound in the QK weights across training steps, they can be stopped at the QK projections without repairing their source. How fast an error compounds depends on whether its sign persists from step to step, not only on its size.

QK-Guard tests during training whether the failure can still be stopped at the QK projections after it has begun. It switches on QK normalization once a monitored attention-logit statistic crosses 30, when σ(WQ)\sigma(W^Q) has begun to run away. With the error source still active, it rescues every tested run whose unguarded counterpart collapses, under both sources. With the faulty kernel, the softmax-side fix that Qiu & Yao proposed for the kernel's rounding bias and a tenfold learning-rate cut, applied at the same moment, do not rescue the run.

In training quality, QK-Guard gains nothing over QK normalization from the start: over 60k steps, both end at validation loss 3.09–3.13. Where QK normalization can be used from the start, it should be. A mid-run switch serves a run already under way without it, as in the Marin 32B run that Reviewer kqtW cites.

3. Presentation
Reviewers 8ce6 and kqtW found the submitted text hard to follow, and we apologize for it. The revision rewrites the abstract and the relevant sections in plain language, drops the terms we had coined, defines the remaining ones in a glossary and adds a table that maps each experiment to its finding.

Revisions

  • Abstract and §1: revised around the claim above.
  • Table 1: new, the experiments behind each finding.
  • §2: §2.1 now attributes the reproduced collapse to the kernel's bf16 attention computation as a whole, since keeping only its running output and softmax statistics in fp32 still collapses 2 of 2 runs. The new §2.2 adds PyTorch's fused bf16 attention as the second source, with fp32 controls.
  • §3: post-norm runs with the faulty kernel in §3.1. In §3.2, the direction-removal runs under fused attention and a 32k-step re-run of their control, which removes the same amount of each update in other directions. Table 3b gives the observation horizon of the QK-normalized LLaMA-style seeds as 15,800 steps, where the submission printed 19,900. Their outcome, 0 of 4 collapse, is unchanged.
  • §4 and Figure 3: new sign schedules that fix the numbers of + and − steps and vary only their order.
  • §5: the switch on the LLaMA-style model and under fused attention, and the Marin 32B report.
  • §6: a new Related Work section. The conclusion is now §7.
  • Appendices: A gives the protocols of the new runs, with their outcomes in Table 5. C adds what the switch costs. E is a new glossary.
  • Supplementary material: result data and analysis code for the new runs.
AuthorsOpenReview ↗

Summary of the author–reviewer discussion

At the close of the author–reviewer discussion, we thank the three reviewers for their careful reviews and the AC for handling our submission.

What is new. Prior work fixed a bf16 FlashAttention collapse by correcting the kernel's biased rounding, and posited that query–key (QK) normalization keeps such errors from compounding. The paper explains how low-precision attention errors collapse training: they compound in the QK weights across training steps. Three results support it, and the reviewers noted each:

  1. The collapse can be stopped at the QK weights while its source stays active. The faulty kernel corrupts every gradient, yet restoring only the QK gradient prevents collapse in four architectures. Switching on QK normalization mid-run rescues every tested run whose counterpart without the switch collapses, while with the faulty kernel, the fix proposed for its rounding and a tenfold learning-rate cut, applied at the same moment, do not. Reviewer 2eam wrote that the paper "goes beyond correlating spectral growth with training collapse".
  2. The errors compound along the QK weights' leading singular directions. Under fused attention, removing these directions from each update keeps the query weights' largest singular value bounded, while removing as much of each update elsewhere does not. Reviewer 8ce6 wrote that "The directional intervention has a convincing control".
  3. Whether an error compounds depends on whether its sign persists, not only on its size. With the faulty kernel's exact output error injected into an otherwise fp32 kernel, giving the whole error a random sign at each step delays collapse about tenfold, and regrouping the same signs into long runs restores the early collapse. A low-precision kernel should therefore be checked for whether its error keeps its sign across steps. Reviewer 8ce6 wrote that "The sign experiments show that error magnitude alone cannot explain the observed failures".

Main concerns and our responses.

  • What QK-Guard adds over QK normalization from the start (all reviewers). The paper's contribution is the analysis, and the mid-run switch is how it tests the analysis after the failure has begun. Where QK normalization can be used from the start, it should be; the switch ends at the same loss and serves runs already under way. Reviewer 2eam found that "The mechanism analysis is more compelling than the demonstrated added value of the proposed deployment strategy", and the revised abstract and Section 1 lead with that analysis.
  • The Marin 32B report (kqtW). We agree that the mid-run switch is not new: the report used it to end a 32B run's loss spikes and leaves their cause open. For low-precision attention errors, the paper explains the failure that such a switch stops: why acting on queries and keys works where generic fixes do not, and which errors compound. The Marin team's notes conjecture that gradients stay more aligned than usual over a few consecutive steps before a spike, but record no test of it; the third result tests the idea on the error itself.
  • Practical relevance (2eam, kqtW). PyTorch's own fused bf16 attention, with no injected error, collapses GPT-2 small in all 6 runs when the learning rate stays near its peak without weight decay, and in none of 6 with fp32 attention, through the same QK path. Post-norm, which in kqtW's experience protects modern architectures, leaves queries and keys unnormalized; with the faulty kernel, all three post-norm runs collapse, while variants with QK normalization stay healthy.
  • Presentation (8ce6, kqtW). The revision rewrites the abstract and Section 1 in standard terms, drops the terms we had coined, defines the rest in a glossary, and adds Table 1, the experiment table Reviewer 8ce6 suggested.

All new results are in the revised PDF, with their data and code in the supplementary material, and our general comment below lists each revision. We thank the reviewers and the AC again for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.