acceptodds
Under review as a conference paper at ICLR 2027

The Mechanism of Training Collapse in Ultra-Deep Transformers

Abstract

Ultra-deep transformers can collapse in training: after a period of successful loss descent a run jumps, often within ten optimizer steps, onto a persistent degenerate plateau, a near-zero predictor for a flow-matching DiT and a near-unigram predictor for a language model. We study this collapse on diffusion transformers of 256–1024 residual sublayers and language models of 256–2048, pushed past their stability boundary by the learning rate, with kernel probes of the residual-stream write matrices, loss profiles along committed AdamW updates, and interventions. Our hypothesis is a condition on one optimizer step, written in output space: the step is harmful when the tangent kernel's gain along the loss gradient, times the learning rate, carries the output past the radius within which the loss still decreases. The tolerable learning rate falls with depth, as a power law over the depths tested; the gain along the gradient rises before any loss symptom and has fallen from its peak on every terminal plateau; and neither the amplified direction nor continued growth of the kernel is required for the instability. GROW, a retraction of each write matrix onto a fixed-Gram set after every AdamW step, is an intervention on this geometry, not a tuned optimizer: keeping the Post-LN forward pass and the zero-initialized write matrices, at four depths from 256 to 2048 residual sublayers it trains through a constant dose that collapses most or all AdamW seeds at the first three, and raising the scale of its fixed factor brings its sample quality to within two FID, inside the run-to-run spread, of a healthy low-rate AdamW reference at every depth.

Primary area:
optimization
Keywords:
training instability, training collapse, ultra-deep transformers, Post-LN, residual stream, write matrices, empirical tangent kernel, learning-rate stability boundary, diffusion transformers, language models, constrained optimization, Stiefel manifold

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fiJ6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates persistent training collapse in ultra-deep transformers using tangent-kernel diagnostics and interventions. It proposes an output-space instability condition and introduces GROW, a fixed-Gram constraint on residual write matrices, evaluated primarily on diffusion transformers.

Critical strengths
  • The distinction between instability onset and the terminal degenerate plateau is useful and supported by detailed measurements.
  • Multiple interventions, numerical checks, and explicit reporting of experimental limitations strengthen the empirical study.
Critical weaknesses
  1. The proposed mechanism is not directly validated under the training dynamics used in the experiments. Equation (2) describes a linearized gradient step restricted to write matrices, whereas the experiments update all parameters with AdamW, including momentum, preconditioning, and clipping. Neither t/rt/r nor FkF_k measures Γ(e)\Gamma(e). Committed-update loss profiles show that harmful steps occur, but do not quantitatively connect them to the proposed kernel condition.

  2. Persistent collapse remains unexplained. A condition for one-step loss increase does not explain why training converges to a zero-output or unigram-like plateau, or why it fails to recover. This distinction matters because the frozen-kernel experiment produces instability without the same plateau. The experiments describe the collapse dynamics in detail, but do not identify what sustains the degeneration.

  3. GROW does not isolate the hypothesized causal factor. Retraction changes weight geometry, filters update directions, and reduces update magnitude at the same time. The fixed 0.68 scaling control neither matches update norms at each step nor is calibrated to the final configuration. Tangent/normal sensitivity measurements and counterfactual tests are explicitly left untested, so the link between the removed update component and instability remains unverified.

  4. Evidence for broader applicability and practical advantage is limited. GROW is not evaluated on language models, and the comparisons allocate 4000 training steps to GROW versus 3500 to AdamW, on top of the additional retraction cost. There is no direct comparison with related manifold optimizers. Moreover, AdamW already survives the tested constant dose at the greatest depth. GROW appears useful in the studied DiT settings, but these results do not establish whether its benefits extend beyond those settings or justify its additional cost.

  5. Poor readability makes the technical argument difficult to assess. The abstract is overloaded, and the main text moves between diagnostics and experimental variants without clearly explaining what each is meant to establish. Frequent qualifications and appendix references make it hard to follow how the evidence supports the proposed mechanism. In particular, the connection between the output-space analysis and GROW’s weight-space constraint needs a more direct explanation.

Questions for authors

Typos

Appendix E refers to lines 4–10 of Algorithm 1 as the retraction procedure; the loop extends through line 14.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fiJ6: the listed soundness concerns are limitations the paper states; scope of the claims

We thank the reviewer for the detailed review. Several of the weaknesses listed quote limitations that the paper itself states; we mark where, and say what the paper does and does not claim.

W1 (mechanism not validated under AdamW; t/r and F_k do not measure Γ). The second sentence is the paper's own: Sec. 2.2 says both quantities describe the residual's coupling to high-gain directions, "not Γ itself", and App. D repeats it. Eq. 2 is presented as a hypothesis in a linearized model; Eq. 1, on which the committed-step profiles rest, is exact for any optimizer. The link between the two is supported by timing at three depths (t/r leaves its band before the validation symptom), the absence of damage after any snapshot with all stored gains shrinking (15 collapses), a 0.9% quantitative closure of the measured gain (App. C), α_safe of 1.0 on quiet vs 0-0.25 on burst steps, and the Gram-freeze and GROW interventions. We agree this is not a per-step measurement of Γ on the AdamW update and do not claim one.

W2 (persistent collapse unexplained). We agree, and Sec. 3.3 states: "the experiments do not establish what is required for the plateau to form". Our claim concerns the onset of instability; Sec. 1 defines "mechanism" as the one-step condition plus the links we could observe and intervene on. The frozen-kernel experiment the reviewer cites is how we establish this separation, which the reviewer lists as a strength. We will make the scope explicit in the abstract.

W3 (GROW does not isolate the causal factor). We agree. Sec. 4 states that GROW "acts in two places" and that the runs "do not separate which of the two carries the stability"; Sec. 5 describes the 0.68 control as "not a per-step match, nor the ratio at α²=16" and lists the counterfactuals that "were not run". What the controls do show is that neither zero writer decay (2/2 collapse) nor a constant step reduction (rescues the quiet seed, not the violent one) reproduces GROW's survival across seeds. GROW is offered as an intervention on write geometry that moves the boundary, not as an identified cause.

W4 (applicability and cost).

  • 4000 vs 3500 steps: for the stability claim this difference works against GROW, not for it. Every AdamW collapse occurs within 250-1675 steps of residence, i.e. by step 2675, well inside its 3500-step window, while GROW is exposed to the same dose for 500 more steps. For sample quality the extra steps do favour GROW, and Sec. 5 says "No comparison below is at equal compute" and "equal-compute equivalence with a tuned AdamW is not established".
  • D=1024: that AdamW survives the dose there is our own reported result (abstract: "the first three"; Table 1); GROW's ramp boundary at that depth is still 1.50x AdamW's.
  • Cost is reported: 25-30% per step at D=256 (App. E).
  • GROW on LMs and a head-to-head with Manifold Muon variants were not run; both are stated (Sec. 5, Sec. 6).

W5 (readability). We accept this. We will shorten the abstract, add a roadmap stating what each diagnostic is meant to establish, and move the output-space to weight-space link (Eq. 3: what one write matrix can add to its sublayer Jacobian is bounded by its Gram) to the opening of Sec. 4.

Typo. Thank you; the retraction loop is lines 4-14 of Algorithm 1 (line 15 is logging). Corrected.

In summary, we do not dispute that the paper leaves the plateau and GROW's causal factor open. We ask the reviewer to consider whether a soundness score of 1 is warranted when the listed concerns are limitations the paper states and does not claim beyond, and no reported result is identified as incorrect.

Reviewer_WbaROpenReview ↗
Rating
3(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

This submission studies collapse during the training of ultra-deep transformers. Here, collapse refers to a phenomenon in which, after a period of successful loss descent, training moves onto a persistent degenerate plateau and the model produces only low-information predictions, e.g., a near-zero predictor for flow-matching Diffusion Transformers (DiTs) and a near-unigram predictor for language models (LMs). The authors analyze collapse through what one optimizer step does to the output, focusing on the write matrices of the residual stream. More specifically, they examine the gain of the empirical tangent kernel for the weights of the attention output projections and MLP down projections with respect to the current loss gradient, and try to explain collapse through this relationship. They further trace the committed AdamW updates to measure when a step becomes harmful, and compare differences across depth and normalization schemes, as well as interventions that remove specific directions and experiments that freeze only the behavior immediately before collapse. In addition, they introduce GROW, which constrains the target weights so harmful writes do not become too large, and evaluate whether it can keep the Post-LN forward pass and the zero-initialized write matrices.

Critical strengths
  1. Although this is not my area of expertise, I understand the focus on the persistent degenerate plateau, the dynamic tracking of the coupling between the gradient and the gain, and the inclusion of task differences and counterfactual interventions as a substantial source of novelty.
  2. Although high gain is also observed during healthy training, the paper shows that, at collapse, the gradient becomes strongly coupled to high-gain directions. The resulting empirical characterization of the shared features and distinct collapse dynamics of DiTs and LMs is valuable to the community.
  3. GROW preserves the Post-LN forward pass and zero initialization, while showing that stability can be improved across multiple depths and runs, so it stands as a meaningful intervention.
  4. The measurements and implementation are carefully validated. For example, for GROW, the paper clearly states which controls were performed and which were not, including factor scale, Gram shape, update length, and weight decay. The paper also makes clear that the strength of the evidence is not uniform across all claims, which strengthens the overall credibility of the results.
Critical weaknesses
  1. I do not see any critical weakness, though I could not thoroughly check all technical details. There are remaining limitations, such as the fact that the contribution of a fixed envelope on the realized write and a direction selection on each candidate update is not disentangled in explaining why GROW stabilizes training, that the experiments do not establish what is required for the plateau to form, and that equal-compute equivalence with a tuned AdamW is not established. However, all of these points are explicitly acknowledged by the authors and do not undermine the reliability of the conclusions or the main claims.
  2. The main GROW evaluation is concentrated on DiT, so the generality of its practical effectiveness still requires further verification. However, this is also not a critical issue that undermines the paper's main claims.
Questions for authors
  1. The Reproducibility Statement says that code, configurations, and run ledgers accompany the submission, but I do not see Supplementary Material on OpenReview. Please confirm.
  2. The analysis throughout the paper focuses on the attention output projections and MLP down projections, and excludes the MLP up-projection from both the analysis and the constraint. Why is it reasonable to exclude it? Since the up-projection helps determine the features that the down-projection writes into the residual stream, it would be helpful to provide a brief rationale or an ablation showing to what extent its contribution to collapse can be ignored.
1 comment
AuthorsOpenReview ↗

Response to Reviewer WbaR: correction on supplementary material, and the up-projection

We thank the reviewer for the careful reading, and in particular for checking which controls were and were not run.

Q1 (supplementary material). Thank you for checking; you are right, and we apologize. No code, configurations or run ledgers are attached to the submission. When the Reproducibility Statement was written we had internal approval to release them; that approval was later withdrawn under our organization's confidentiality requirements, and we failed to update the sentence. It is inaccurate as submitted, and we will correct it in the revision to state plainly that the code is not released.

What the paper itself provides for reproduction is unchanged: Algorithm 1 gives the full GROW step, including the numerics of the retraction for both factors (App. E); P0 is derived from a single integer seed; App. D gives the model and optimizer configurations and the probe definitions; App. F lists every run family with its protocol, dose, seeds and step counts; and App. B/H give per-seed readings. GROW adds no component beyond the AdamW step and the retraction, so it can be reimplemented from Algorithm 1 on any Post-LN trainer. We are happy to answer any implementation question here.

Q2 (why the MLP up-projection is excluded). Three reasons, and one honest caveat. (i) Structural: the attention output projection and the MLP down-projection are the only matrices through which a sublayer writes linearly into the residual stream (u_l = W_l h_l, Sec. 2.2). The up-projection acts on the read side; in our notation it is part of Φ_l, and Eq. 3 bounds what the write matrix can add to the sublayer Jacobian for any Φ_l. (ii) Calibration: on released healthy models the write-matrix kernel carries 0.83 of the residual's full-parameter gain (Table 5, App. D). (iii) Sufficiency: GROW constrains only the write matrices and leaves the up-projection on plain AdamW, yet every GROW arm survives a dose that collapses AdamW at D=128/256/512 (Table 1). Constraining the read side was not needed for stability in these runs. Caveat: in pathological LM states the write-matrix share drops to 0.38-0.43 of the full squared gain (Table 5), so the read side is not negligible there, and Sec. 4 states that the bound "constrains neither the read side Φ_l ... nor the composition across sublayers". We did not run an ablation that also constrains the up-projection, and we will say so explicitly next to the rationale above in Sec. 2.2.

On the two limitations noted (the envelope vs. direction-selection split, and GROW being evaluated on the DiT): we agree, and both will stay stated as limitations in the main text rather than in the appendix.

Reviewer_dXqaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies loss spikes and persistent training collapse in very deep Transformers architectures through the empirical NTK of residual-stream write matrices (e.g., attention output projections and FFN down-projections). The paper combines a local stability argument with measurements during training and interventions on these matrices.

Critical strengths

The main strength is the substantial empirical investigation across depths and two tasks. Focusing on residual-stream write matrices provides a concrete architectural perspective. The reported GROW results also suggest that constraining these matrices can prevent collapse under settings where the AdamW baseline fails.

Critical weaknesses

Soundness of the analysis and novelty:

  1. The linearized update and its stability condition are not novel and already estabilished in the NTK and edge-of-stability literature [1,2]. The potential contribution is hence limited to empirical findings and interventions in sections 3, 4, and 5. This distinction needs to be clearer.
  2. The connection between the local stability argument and observed collapse remains unclear. The authors acknowledge the differences between their kernel probes and full-parameter AdamW training, and measure loss along actual updates. These profiles show that some updates increase loss, but do not directly test the kernel condition in Equation 2. Given the large linearization error reported in Appendix D, stronger evidence is needed to connect the kernel diagnostics to those harmful updates and the subsequent persistent collapse.

Presentation: the paper is hard to follow. It moves between linearized gradient descent, kernel diagnostics, and AdamW updates without a clear connection.

[1] https://arxiv.org/abs/1806.07572 [2] https://arxiv.org/abs/2603.05002

Questions for authors

Could the authors address the following questions, apart from the concerns that I raised above?

  1. How does the directional NTK analysis differ from e.g., [3], which already connects loss spikes to large residual components in the eigenspace of the NTK?
  2. What empricial evidence connects the measured kernel diagnostics to harmful actual AdamW updates, particularly where the linearization error is large?

[3] https://arxiv.org/pdf/2306.04815

1 comment
AuthorsOpenReview ↗

Response to Reviewer dXqa: what is claimed as new, and the evidence linking kernel diagnostics to AdamW updates

We thank the reviewer for the comments and respond to each point.

W1 (the linearized update and stability condition are not novel). We agree, and the paper does not claim them. Sec. 2.2 introduces K as "the empirical tangent kernel (Jacot et al., 2018)" and states that Eq. 2 for an eigenvector "reads ηλ < 2, the stability condition of gradient descent on a quadratic (Cohen et al., 2021)"; Sec. 6 repeats this attribution. What we claim is narrower: (a) the directional quantity Γ(e), which separates the kernel's gain from its coupling to the current loss gradient, so that large gain alone (which we measure in 13 released healthy checkpoints, 25-390x) does not imply a harmful step; and (b) the measurements and interventions of Secs. 3-5. We will reword the first contribution in Sec. 1 so that this division is explicit, as the reviewer asks.

W2 / Q2 (what connects the kernel diagnostics to harmful AdamW updates, given the large linearization error). We state in Sec. 2.2 and App. D that t/r and F_k "are not measurements of Γ in Equation 2", so we do not claim a direct test of Eq. 2 under AdamW. The large linearization error (118% on the burst step) is the reason the analysis rests on Eq. 1, which is an exact identity in the realized output displacement and needs no linearization, and the reason the profiles in Fig. 2b are plotted against realized displacement. The evidence linking the kernel side to the committed updates is:

  • Timing: at D=256/512/1024 the LM's t/r leaves its healthy band before any validation symptom, which follows within 20 steps (Sec. 3.3, App. H).
  • No counterexample in the timing set: over fifteen collapses, no damage followed a snapshot at which every stored direction's gain was shrinking (Sec. 3.3).
  • Quantitative closure: at the amplifier snapshot, overlap x sqrt(λ0) predicts the measured gain to 0.9% (1793 vs 1809, App. C).
  • Committed steps: α_safe is 1.0 on quiet steps and 0-0.25 on burst steps of the same run (App. C).
  • Intervention: holding WW^T fixed prevents the failure for ~20x the failure step (Sec. 4), and under GROW the pre-clip gradient peak at the AdamW boundary step is 0.48 against 22.6 (Sec. 5). This is correlational plus interventional evidence, not a per-step measurement of Γ along the AdamW update, and the paper says so. We will collect these five items in one paragraph at the end of Sec. 3 so the chain is visible in one place.

Q1 (difference from [3]). As we understand [3], it studies loss spikes that recover (catapults) under GD/SGD and relates them to the top NTK eigenspace. We study (i) a non-recovering plateau, (ii) committed AdamW steps with clipping and momentum, (iii) stacks of 256-2048 sublayers, and (iv) the coupling over time: the high-gain directions are present throughout healthy training while the residual stays decoupled from them, and our two counterfactuals show that neither the amplified direction nor continued kernel growth is required for the instability. We cite the catapult analysis of Lewkowycz et al. (2020) and will add [3] and [2] to Sec. 6 with this comparison.

Presentation. We will add a short roadmap at the start of Sec. 2 stating which quantity belongs to which setting: Eq. 1 (exact, any optimizer), Eq. 2 (linearized GD, used as a hypothesis), t/r and F_k (probe-batch coupling), α_safe (the committed AdamW step).

We note the reviewer's soundness score indicates minor or fixable concerns; we hope the clarification that W1 concerns a claim the paper attributes to prior work, and the evidence listed under W2, address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.