acceptodds
Under review as a conference paper at ICLR 2027

Repairing Attention Collapse Through the Loss with a Spectrally Gated Penalty

Abstract

Attention collapse can stall transformer training without a numerical failure: attention concentrates on a few tokens and the loss stops improving. We ask whether a term in the loss can bring a run back from collapse. This is not obvious, because a penalty on attention entropy acts through the softmax gradient, and that gradient becomes small as attention saturates. We propose SpecGate, an attention-entropy penalty whose weight is set for each head by a gate that follows the spectral sharpness of the head's attention scores and passes no gradient of its own. In a 124M-parameter language model, SpecGate brings runs from an established collapse to 0.07 nats above the healthy run of the same seed, against 1.05 nats without treatment, and raises the loss of healthy runs by 0.004 nats. The same penalty at constant weight, including a separately tuned one, ends 0.12 to 0.14 nats above SpecGate. Starting SpecGate at the first spectral alarm rather than after collapse has set in lowers the final loss by 0.22 nats, and stopping it 250 updates after the start of repair raises the final loss by 0.10 nats relative to continuous treatment. At equal numbers of updates, a gate on the entropy deficit and a fixed schedule are equivalent to the spectral gates within 0.02 nats; across the tested interventions, when and how strongly the penalty acts moved the final loss more than the signal that sets its weight. After collapse, QK-Clip, which rescales the weights, ends 0.03 nats above SpecGate at its published threshold; with a threshold tuned on this task for the final loss, it ends 0.02 nats below SpecGate and runs faster.

Primary area:
optimization
Keywords:
attention collapse, attention entropy, training instability, transformers, spectral norm, regularization, loss-based repair

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_abZ8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether a loss term alone can bring a transformer back from an established attention collapse after the collapse has happened. It does not change the architecture or the optimizer step. The proposed method, SpecGate (Eq. 2), is an attention-entropy penalty. Each head's weight comes from a detached logistic gate on the spectral sharpness of that head's row-centered score matrix. The authors train a 124M language model on WikiText-103 with a high learning rate so that attention collapses, then continue training from the collapsed checkpoint with different fixes. SpecGate recovers most of the lost performance: the final validation loss is close to that of a run that never collapsed. It also beats the same penalty applied at a fixed weight. However, replacing the spectral gate with simpler rules, such as an entropy-based gate or a fixed schedule, gives nearly the same result. The authors conclude that when and how strongly the penalty is applied matters more than the exact signal that controls it.

Critical strengths
  • Attention collapse is an important problem for training stability.
  • The experiments use an actual language model, not just a toy setting.
  • The experiments are carefully controlled. All methods continue from the same saved checkpoint, so the comparisons are fair. The paper also reports results that do not favor SpecGate, such as a tuned QK-Clip performing better.
Critical weaknesses
  1. The method and the contribution are hard to locate. The gate is defined in Eq. (2), but the full procedure (Algorithm 1) appears only in Appendix D. The main text never clearly separates the novel part from what is borrowed from prior entropy penalties such as EAR and Jha & Reagen (2025). Please present SpecGate as a self-contained method in the main text, with the algorithm, the hyperparameters (λ, s0, w), and an explicit statement of what is new relative to prior gated entropy regularizers.

  2. The results are hard to interpret. The abstract and main text report a long list of loss differences in nats (e.g., "0.07 nats above the healthy run", "0.004 nats", "0.12 to 0.14 nats", "0.22 nats", "0.02 nats below") without saying which are large and which are negligible. Readers cannot tell whether a given improvement is meaningful. Please put the numbers in context, e.g., as perplexity or as the fraction of the collapse gap that is recovered, and summarize the key comparisons in one main-text table with a clear takeaway.

  3. The motivation for a loss-based remedy is unclear. The paper restricts itself to loss-based fixes ("which change neither the architecture nor the optimizer step" in L.54) without explaining why architecture- or optimizer-side fixes are insufficient. The paper's own results suggest they are not: a tuned QK-Clip beats SpecGate and runs faster, QK-norm and plain training with a tuned learning rate both end below SpecGate, and SpecGate adds about 36% compute per update. When should a practitioner choose SpecGate over these alternatives?

  4. The experimental setting is limited. All main results come from one 124M model on WikiText-103, with context 256 and 5,000 updates of 4,096 tokens (about 20M tokens), and collapse is induced by a high learning rate. The generality of the conclusions is unclear.

  5. The spectral gate does not seem necessary. Replacing the spectral gate with simpler rules, either an entropy-based gate or a fixed schedule, gives nearly the same results (Sec. 5.3), and the fixed schedule is even slightly better at equal compute time. This weakens the paper's main novelty claim.

Questions for authors
  • Why should a practitioner prefer a loss-based repair over QK-Clip or QK-norm/QK-Guard, given that a tuned QK-Clip is better and faster in your own comparisons?
  • If the spectral gate can be replaced by an entropy-deficit gate or a fixed schedule with no loss in quality, what remains specifically "spectral" about the contribution?
Reviewer_yYnLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors analyze whether an attention-entropy penalty can repair established attention collapse. Here, SpecGate weights each head's penalty using a detached gate based on spectral sharpness. Paired continuations of collapsed checkpoints indicate a substantial recovery in a 124M language model and improvements over constant-weight penalties. Ablations suggest that penalty strength and its evolution over training are more relevant than the particular gating signal.

Critical strengths
  1. I think the controlled design convincingly establishes repair within the tested setting. SpecGate improves final loss by 0.98 nats over untreated continuation across eight seeds, bringing it within 0.07 nats of healthy training. Also the fresh paired comparisons also indicate an advantage over constant-weight penalties. I consider this might be the paper's most solid result.

  2. The ablations identify a useful distinction between weighting and signal choice with comparisons against a tuned constant coefficient, a mean gate, an entropy-deficit gate, and a replayed schedule showing that the observed repair does not require spectral feedback or precise per-head allocation. From my point of view, this narrows the explanation beyond simply observing that the proposed method works.

Critical weaknesses
  1. One of my largest concerns is that, after collapse, the entropy-deficit gate and replayed schedule finish approximately 0.004-0.005 nats above SpecGate. Moreover, the spectral statistic may be useful for detection, but the paper does not compare outcomes when the same treatment is triggered by spectral versus entropy alarms? (or it was not clear to me). For this reason, I find the methodological contribution beyond appropriately weighted entropy regularization unclear or not well enough explained/established.

  2. Further, I find the scope of the repair finding to remain narrow, with most evidence coming from a 5,000-update, high-learning-rate 124M recipe. These experiments indicate partial transfer, but do not show reliable recovery across substantially different training regimes.

  3. More related to the presentation aspect: The paper is 47 pages long. I strongly suggest moving parts to a repository; presenting so much information can overwhelm readers (me included) and make it difficult to review with this much additional material.

Questions for authors
  • On the existing transfer configurations, does spectral gating improve on entropy-based gating or the replayed schedule at matched compute?
  • Does starting the same treatment from a spectral rather than entropy alarm improve recovery outcomes, beyond the reported difference in detection specificity?
  • What concrete training constraint or failure regime favors SpecGate over tuned QK-Clip?
  • Could you consider reducing the size of the Appendix? It is way too long, in my opinion, and with so much information, it is easy to lose focus.
Reviewer_Fi1AOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aimed at mitigating the attention collapse, where softmax attention converge towards one-hot vector, by modifying the loss function. Authors have introduced SpecGate, which pre-empts the entropy collapse by avoiding the gradient update, which is indicated by the weight value set by a gating mechanism. The effectiveness of SpecGate is shown on 125M GPT-2 models, and compared with other approaches designed to prevent the attention pathologies.

Critical strengths

First, the idea of preventing the entropy collapse through loss function is an interesting approach. Prior work used architectural intervention to resolve this, such as sigma-parameterziation, QK-Norm, weight and spectral normalization etc.

Second, authors have conducted a comprehensive evaluation, in terms of loss reduction, with the existing methods for preventing attention patahogies.

Critical weaknesses

The main shortcoming is the scale of model (125M), there can be numerous approach at this scale which can prevent attention pathologies. The effectiveness must be shown to 24 to 36 layers deeper LLMs, such as smolLM families models.

Also, it is not clear, why anyone use the author's loss-based method over the well-known architectural intervention (QK-Norm, QK-Clip, etc.). Precisely, the motivation is not evident from the paper.

Questions for authors

To claim, attention collapse can be solved through loss function modification, authors should demonstrate this on Softmax-only model used in Jha & Reagan 2025. Their paper showed that, when all nonlinear layers are removed (mainly LayerNorms and GELU in FFN), except Softmax in attention, then attention collapses in deeper layers. Can authors apply their method on GPT-2 125M Softmax-only models, and then observe how entropy collapses recovered in deeper layers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.