acceptodds
Under review as a conference paper at ICLR 2027

Spectral Alignment for Early Detection of Loss Explosion in Language Models

Abstract

Loss explosions can invalidate expensive language-model training runs, yet common scalar monitors such as weight and gradient norms are often ambiguous or late because their scales vary across models and layers. We introduce Spectral Alignment (SA), a data-conditioned and scale-robust monitor of the signed cosine alignment between layer inputs and the principal left singular direction of a weight matrix. Across a mini-batch, mixed-sign SA permits cancellation in the first-order spectral update. We show that a sufficiently one-sided SA distribution, together with compatible downstream gradients, produces positive spectral drift, and derive conditions under which a persistent SA warning precedes a spectral-norm threshold. These results provide a conditional mechanism rather than a universal failure guarantee. In two language-model failure settings, directional alignment collapse emerges substantially before loss explosion and provides a clearer warning than the monitored scalar baselines. SA also selects healthier checkpoints for retraining, leading to lower loss than later pre-failure checkpoints. With configurable monitoring cost through sampled layers, power iteration, and offline computation, SA provides an actionable signal for early intervention and recovery.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GWknOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Spectral Alignment (SA), the signed cosine similarity between layer inputs and a weight matrix’s principal left singular direction, as an early-warning indicator of training instability. The central idea is that amplification depends on both weight geometry and input direction. The theoretical analysis connects sufficiently one-sided alignment to spectral growth under compatible downstream gradients. Experiments in two language-model failure settings show that changes in SA precede loss explosion and can inform checkpoint selection for recovery.

Critical strengths
  1. An interpretable perspective on training instability. SA captures directional relationships between inputs and weights that scalar magnitude-based monitors may overlook. Its invariance to positive scalar rescaling makes it a useful complementary diagnostic.
  2. A connection between theory and empirical observations. The analysis provides a conditional explanation for spectral growth, while the experiments demonstrate an observable pre-failure alignment pattern.
  3. Practical relevance. The monitoring procedure and checkpoint-selection experiments suggest useful applications to early intervention and recovery during language-model training.
Critical weaknesses
  1. The main theoretical analysis considers gradient descent or SGD, whereas the experiments use AdamW. Although the appendix discusses the extension, empirical validation of the relevant conditions would strengthen the connection between the theoretical mechanism and the observed failures. I consider this a minor limitation.
Questions for authors
  1. Does one-sided spectral alignment tend to reinforce itself during subsequent training? Can its persistence be derived from the coupled dynamics of inputs, weights, and downstream gradients, and could this analysis help predict the time to instability?
2 comments

Thank you for the positive review.

W1 — The theory considers GD/SGD while the experiments use AdamW.
We compared the first-order spectral increment predicted from the raw gradient (Eq. 10) with the increment induced by the actual AdamW update, −η u₁ᵀP_t v₁ − ηλσ₁. When the SGD prediction is positive, the AdamW increment is also positive in 80% of steps. This is consistent with the role of the first moment: Adam's exponential average tends to keep the sign of the update and damp occasional opposite-sign steps. We will report this measurement in Appendix B as an empirical check of the sign condition of Proposition 3.2 under AdamW.

Q1 — Does one-sided SA reinforce itself, and can this predict the time to instability?
We have not derived this formally. Doing so would require modelling how the input distribution, shaped by all upstream layers, responds to the growing principal direction, together with the downstream gradients that set the sign of the drift. Empirically, short excursions can occur and recover, whereas in the unstable runs the one-sided state persists until failure. For the FA case, Qiu & Yao (2026) describe a mechanism in which the low-rank structure that makes the rounding error coherent is maintained by the resulting updates, which suggests one route to persistence in that setting.

Thank you for the clarification and the additional empirical analysis.

The reported sign agreement between the SGD prediction and the actual AdamW update helps address my concern about the connection between the theory and the experiments. While this does not fully close the theoretical gap, it provides useful empirical support. Please include the measurement setup in the revision, particularly which runs, layers, and training intervals were used to obtain the reported 80%.

Regarding the persistence of one-sided spectral alignment, I appreciate the distinction between the empirical observations and what has been established theoretically. A formal analysis of the coupled dynamics remains an interesting direction for future work, and I do not view its absence as a major weakness of the current paper.

Overall, the response addresses my main concern sufficiently, and I maintain my weak acceptance rating.

Reviewer_AtStOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Spectral Alignment (SA), a scale-robust metric that measures the signed cosine similarity between layer inputs and the principal left singular vector of a weight matrix, aiming to detect training instability before loss explosion. The authors provide a theoretical analysis connecting persistent one-sided SA distributions to spectral growth under compatible downstream gradients. Experiments on two language-model instability settings demonstrate that SA can provide early warnings before loss explosion and help identify checkpoints for more effective training recovery.

Critical strengths
  1. Practically important problem. The paper addresses early detection of loss explosions in language-model training, a practically important problem given the substantial computational cost of failed training runs. An effective early-warning mechanism could reduce wasted computation and improve training reliability.
  2. An ablation comparing the current heuristic with the theoretically motivated mass-based statistic would clarify the detector design. Promising results across two distinct failure settings. The experiments cover both low-precision Flash Attention instability and high-learning-rate FFN instability. In both cases, the authors observe a persistent one-sided shift in SA before visible loss explosion, suggesting that the signal may capture behavior shared across different instability mechanisms.
Critical weaknesses
  1. Mismatch between the theoretical alignment measure and the practical detector. The theoretical analysis characterizes directional coherence using the signed SA masses M+M_+ and M−M_-, while the practical detector relies on the absolute mean, absolute median, and a ratio of conditional mean magnitudes. These quantities are not equivalent. For example, a batch containing 99 samples with SA =0.5=0.5 and one sample with SA =−0.5=-0.5 has CSA=0.98C_{SA}=0.98, but the conditional-mean ratio is r=1r=1, preventing the proposed detector from firing.
Questions for authors
  1. Unclear prediction horizon and practical actionability. The reported intervals between SA collapse and loss explosion vary substantially across experiments (approximately 4,400 steps in the FA case versus 571 steps in the FFN case). How should practitioners interpret an SA warning in terms of the expected time to failure? Without a reliable estimate of the remaining training steps before loss explosion, it is unclear how to decide whether to stop training immediately, continue monitoring, or roll back to an earlier checkpoint.
  2. Unclear underlying causes of directional alignment collapse. What drives directional alignment collapse during training? While the paper establishes a conditional relationship between SA collapse and spectral growth, the underlying causes of this phenomenon remain unclear. Is it primarily triggered by numerical instability, excessive learning rates, particular data distributions, initialization, or other aspects of the optimization dynamics?
3 comments

Thank you for the review and for the counterexample.

W1 — The practical detector is not equivalent to the signed-mass criterion (99:1 example).
We agree that the two are not equivalent in general, and the reviewer's example shows a case where the conditional-mean ratio would miss a small M_o/M_d. In the revision we replace the conditional-mean ratio with the signed-mass ratio ρ = M_o/M_d from Proposition 3.2, so a checkpoint is abnormal when the absolute mean and absolute median of SA both exceed 0.2 and ρ < 0.1, with the same thresholds as before. This makes the detector test the quantity in the proposition directly. In both failure settings the onsets stay the same (t₀ = 6,621 for FA and 76 for FFN), and neither stable run raises an alarm. In the stable runs ρ also becomes small during transient skew, but the mean and median stay below 0.05, so the magnitude conditions are still needed. The updated values and plots are in Tables 4–5 and Figure 9.

Q1 — Interpreting the prediction horizon.
The lead differs across settings, and our analysis does not give a time-to-failure estimate, so SA is better used as a trigger than as a countdown. For example, one can save a checkpoint when SA first becomes abnormal and intervene once the abnormality persists or a scalar monitor confirms it.

Q2 — Underlying causes of directional alignment collapse.
We do not yet have a general answer. SA couples the input distribution, shaped by all upstream layers, with the weight geometry and the downstream gradients, so its evolution depends jointly on data, numerical precision and optimization, and the factor that initiates the collapse may differ across settings. For the FA setting, Qiu & Yao (ICLR 2026, arXiv:2510.04212) analyse the failure in detail: biased rounding in the BF16 attention backward pass interacts with emergent, similar low-rank representations, so the errors accumulate along a consistent direction in the weight update instead of cancelling. For the high-learning-rate FFN setting, the cause is still open.


Thank you for the response. I am keeping my rating at 3. However, if the AC recommends rejection, I would have no objection.

I find this year’s four-point scale too coarse to convey that judgment, especially hearing that 2/3 of the papers this year are rated 2. I give the paper a 3 because I see practical value for large-scale training infrastructure. However, I think the paper still falls short in two respects.

  1. Checkpointing at the first alarm. Saving a consistent checkpoint is expensive at scale. With synchronous checkpointing, the entire training job has to pause while the state is written. There is also a more fundamental question: by the time SA first raises an alarm, has the weight spectrum already deteriorated to the point where that checkpoint is unsuitable for recovery? The paper neither accounts for the cost of checkpointing at the first alarm nor tests whether resuming from that checkpoint can recover training.

  2. What drives directional alignment collapse. Although the authors cited some references to give a explanation, the need for stochastic rounding in low-precision gradient computations is already widely recognized. Rounding to nearest is much more likely than stochastic rounding to produce errors that accumulate in a consistent direction. I have an additional hypothesis. Token frequencies in natural language follow a Zipf-like distribution, so a small number of tokens account for a large share of occurrences. I would examine the gradient spectra in the very first training steps. You may that frequent tokens such as the, and, and \n contribute disproportionately to the leading singular directions, and that their repeated occurrence keeps those directions aligned across steps. A high learning rate could then amplify the effect of repeated updates along those directions and drive optimization into an ill-conditioned regime.

This is a valuable research problem, but the current paper does not investigate it in enough depth. I encourage the authors to pursue it further.

Again, I rate at 3 mainly for the practical value of the problem the paper focused, and the effort made by the authors. I think the paper could investigate deeper. I'd appreciate it if the AC would consider my opinion carefully.

Thank you for the follow-up and support.

Checkpointing at the first alarm.
At scale, a checkpoint costs little next to training time and the compute a failed run wastes. BLOOM-176B saved 2.3 TB with optimizer states in about 40 s, 0.37% of training time [1]. ByteCheckpoint reports a 0.59 s training stall for a 405B model [2], and Nemotron 3 Ultra cut it from about 60 s to under 1 s with asynchronous saving [3].

Section 4.4 tests whether such checkpoints can recover training. Resuming from the last checkpoint before the abnormal window (FA 6,580, FFN 75) gives lower loss than resuming from a later pre-explosion checkpoint (FA 7,200, FFN 200) (Figure 8): earlier intervention recovers better. We also resumed from a confirmed-alarm checkpoint with a modified attention backward: the final loss is 3.3, versus 7.6 for the original run, which diverges.

Token frequency.
The 178 most frequent tokens (0.35% of the vocabulary) cover ~50% of occurrences. Over the first 10 batches, they supply 99.96% of the FA query-gradient projection onto its leading singular direction and 92.0% onto the full gradient (FFN: 98.0%/64.2%).

SA, however, barely depends on frequency. Early in FA training, when frequent tokens dominate the leading direction, query SA is 0.003. At FA step 10,580, frequent and other tokens have the same SA: 0.018 vs. 0.018 in the stable run and 0.856 vs. 0.855 in the unstable run. Across modules and steps, the median absolute correlation between log frequency and per-token SA is 0.15–0.19.

FA: a possible mechanism.
Setup. We resume the unstable FA run from its step-6,580 checkpoint with two batch orders. Stochastic rounding of the attention output does not delay the query alarm (step 6,620, as with round-to-nearest), and one run still diverges. We therefore examine the backward row statistic of Qiu & Yao and its effect on layer-1 Q/K SA.

Row error. For query ii of a head, let PijP_{ij} be the attention weights, Zi=∑jPijZ_i=\sum_jP_{ij}, VjV_j the values, and gig_i the loss gradient at the BF16 output O~i\tilde O_i. The score gradient is Pij(gi⊤Vj−xi)P_{ij}(g_i^\top V_j-x_i). FlashAttention uses xi=di=gi⊤O~ix_i=d_i=g_i^\top\tilde O_i; the exact value is the row mean μi=∑jPijgi⊤Vj/Zi\mu_i=\sum_jP_{ij}g_i^\top V_j/Z_i. With ei=di−μie_i=d_i-\mu_i, each score gradient carries an extra −Pijei-P_{ij}e_i, so the row sums to −Ziei-Z_ie_i, not 0. Since gig_i depends on the output error, eˉ\bar e can be nonzero even if that error is unbiased.

Effect on SA. In the notation of Section 3 (fi=hiWf_i=h_iW, top singular triplet (σ1,u1,v1)(\sigma_1,u_1,v_1)), let h^i=hi/∥hi∥\hat h_i=h_i/\lVert h_i\rVert, m=1N∑i=1Nh^im=\tfrac1N\sum_{i=1}^N\hat h_i with m u1>0m\,u_1>0, so mean SA is m u1m\,u_1. Let the row vector ϵi\epsilon_i be the change eie_i makes to −∂L/∂fi-\partial L/\partial f_i (Q: ϵi=ei∑jPijkj/d\epsilon_i=e_i\sum_jP_{ij}k_j/\sqrt{d} for keys kjk_j), and bϵ=∑i∥hi∥ϵib_\epsilon=\sum_i\lVert h_i\rVert\epsilon_i.

  1. Weight side. The extra step on WW is E=∑ihi⊤ϵiE=\sum_ih_i^\top\epsilon_i; if σ1≫σ2\sigma_1\gg\sigma_2, u1u_1 turns along (I−u1u1⊤)Ev1(I-u_1u_1^\top)Ev_1. Since Ev1=(bϵv1)m⊤+∑i∥hi∥(ϵiv1)(h^i−m)⊤Ev_1=(b_\epsilon v_1)m^\top+\sum_i\lVert h_i\rVert(\epsilon_iv_1)(\hat h_i-m)^\top, bϵv1>0b_\epsilon v_1>0 turns u1u_1 toward mm unless the rest or AdamW dominate.
  2. Input side. The change to −∂L/∂hi-\partial L/\partial h_i is ϵiW⊤\epsilon_iW^\top, which contains σ1(ϵiv1)u1⊤\sigma_1(\epsilon_iv_1)u_1^\top. Whether mm then turns to u1u_1 depends on upstream layers.

Formation (steps 6,621–6,640). Each step we apply an AdamW update with did_i and one with μi\mu_i from one state; EE is their weight-gradient gap, m^=m/∥m∥\hat m=m/\lVert m\rVert. Median signed cosines (Q; K): Ev1Ev_1 with m^\hat m, .96; .98; mean ϵiW⊤\epsilon_iW^\top with u1u_1, .40–.54; .80–.84. After 160 updates, query SA is 0.90–0.93 with did_i and 0.03 with μi\mu_i; μi\mu_i only in the weight or only in the upstream gradient gives 0.03 or 0.13–0.16.

Accumulation. EE gives only 11–14% of each step's turn of u1u_1, but keeps its direction while the rest cancels: over 20 steps, ∥∑tEt∥/∑t∥Et∥\lVert\sum_tE_t\rVert/\sum_t\lVert E_t\rVert is .94–.99, vs. .12–.35 for the μi\mu_i gradient. For K, ∑tEt\sum_tE_t reaches 96–99% of the summed gradient norm.

Plateau. Write mean SA as RcRc, R=∥m∥R=\lVert m\rVert, c=m^ u1c=\hat m\,u_1. By step 6,801, cc is .988 (Q) and .975 (K), leaving at most 1.2% and 2.5% headroom, and with angle θ\theta, Δc≈sin⁡θ Δθ\Delta c\approx\sin\theta\,\Delta\theta is small. A turn of u1u_1 raises cc only along (I−u1u1⊤)m^(I-u_1u_1^\top)\hat m; the turn due to EE has cosine .37 (Q), .49 (K) with it, vs. .73–.88 in formation. Though ∥E∥\lVert E\rVert nearly equals the gradient norm, one-step query SA moves at most 0.0012; the angle still shrinks slowly (Q 10.8°→8.9°, K 13.6°→12.9°). The row error thus explains how alignment forms and saturates.

FFN.
The cause in FFN remains open, and we will keep studying what drives the SA shift in both settings.

[1] S. Bekman, ML Engineering, "Fault tolerance".
[2] B. Wan et al., ByteCheckpoint, NSDI 2025, Table 8.
[3] NVIDIA, Nemotron 3 Ultra Technical Report, 2026.

Reviewer_ednDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spectral Alignment, which monitors signed input alignment with a weight matrix's leading singular direction to detect training instability. Conditional theoretical analysis and two language-model failure cases suggest that Spectral Alignment can provide early warnings and guide rollback checkpoint selection.

Critical strengths
  • SA provides a scale-invariant, data-dependent view of training instability. The analysis clearly distinguishes alignment collapse from the additional gradient conditions needed for spectral growth.
  • Both failure cases show an alignment shift before loss explosion. The checkpoint recovery experiments suggest that the signal could support practical intervention.
Critical weaknesses
  • Evidence is limited to two failure settings, without reported multi-seed detection results or false-alarm rates. Layer-selection and threshold-calibration procedures are also insufficiently specified for assessing prospective detection.
  • The reported warning onset is difficult to reconcile with the detector rule. In Table 4, several checkpoints after step 76 fail the stated thresholds, while the persistence parameter P is unspecified, leaving the actual online warning time unclear.
Questions for authors
  • What values of P, monitoring interval, and sampled batch size were used? In Table 4, steps 126 and 151 fail the mean-magnitude threshold, and step 176 fails the conditional-mean ratio threshold. How is step 76 identified as the start of a sustained abnormal window?
  • Were monitored layers and thresholds selected without access to subsequent failure trajectories? Can you report detection and false-alarm results across additional seeds with these choices fixed?
  • Can SA and scalar baselines be compared at matched false-alarm rates using calibrated thresholds or change-detection rules? Different absolute scales alone do not establish that scalar monitors provide later warnings.
  • Which FA projection is monitored? Section 4.1 specifies the second layer’s key projection, whereas Table 3 and Figure 9 identify its query projection. Please also clarify whether Figure 8 reports training or validation loss.
2 comments

Thank you for the detailed review.

Q1 — Values of P, monitoring interval and sampled batch size.
The monitoring interval is τ_m = 10 updates for FA and 25 for FFN. SA is computed on 4,000 tokens sampled from the training batch at that step (the training batch has 512 sequences / 512k tokens for FA and 128 sequences / 16k tokens for FFN). We use P = 6 for FA and P = 2 for FFN, so the persistence window spans 50 updates in the FA setting. The onset t₀ in Table 2 is the first abnormal checkpoint, and the online alarm is confirmed at t₀ + (P−1)τ_m, i.e. step 6,671 for FA and step 101 for FFN. We will state these settings explicitly in the revision.

Q1 — Why is step 76 the start of a sustained abnormal window, given that steps 126–176 fail a threshold?
With the signed-mass ratio now used as the alarm statistic (see our reply to Reviewer AtSt), only steps 126 and 151 fail a threshold: the mean drops to about −0.15, and all three conditions hold again from step 176 onward. We interpret this as a transient fluctuation, likely caused by batch-to-batch variation in the input data, rather than a return to stable training. In the stable run, SA stays well within the normal range over the same window. We will distinguish the onset from the confirmed alarm in the appendix.

Q2 — Were monitored layers and thresholds selected without access to the failure trajectory?
Yes. Since SA costs only one power iteration per matrix, we compute it for every module in every layer and raise an alarm when any module satisfies the rule. Baselines are aggregated in the same way.

Q2–Q3 — Multi-seed detection, false-alarm rates, and comparison with scalar baselines.
We ran 5 stable/unstable pairs with different seeds in each setting, using seed 0 for calibration and the remaining 4 pairs for testing. The SA thresholds are those in the paper and are shared by both settings. Each baseline threshold is set to the tightest value that does not fire on the stable run of seed 0. Lead is the minimum number of steps from alarm to explosion.

MonitorFA thresholdFA TPRFA FPRFA leadFFN thresholdFFN TPRFFN FPRFFN lead
SAsame as paper4/40/44100same as paper4/41/4470
Weight spectral norm>304/42/43100>203/41/4390
Gradient norm>1.3 (before clip)4/42/43400>0.92/41/4320
Max activation>1403/41/43000>774/42/4260
Stable rank<4.03/42/43200<4.844/41/4300

SA detects every failure in both settings and gives the longest lead, using one threshold set for both models, whereas the scalar thresholds are recalibrated for each setting. We will add these results to the revision.

Q4 — Which FA projection is monitored?
Thanks for catching this. The query and key projections of layer 1 show the same failure signature, as expected, since they share the same input and their gradients are coupled through the attention logits. In the revision, all FA results use the query projection of layer 1, and the labels are corrected throughout. The key projection is shown in Figure 10 (Appendix A.4).

Q4 — Training or validation loss in Figure 8?
Training loss. We will make the caption, text and axis label consistent.


Dear Reviewer ednD, as the discussion period is ending soon, we would appreciate knowing whether our response has addressed your concerns, and we are happy to clarify anything that remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.