- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates persistent training collapse in ultra-deep transformers using tangent-kernel diagnostics and interventions. It proposes an output-space instability condition and introduces GROW, a fixed-Gram constraint on residual write matrices, evaluated primarily on diffusion transformers.
Critical strengths
- The distinction between instability onset and the terminal degenerate plateau is useful and supported by detailed measurements.
- Multiple interventions, numerical checks, and explicit reporting of experimental limitations strengthen the empirical study.
Critical weaknesses
-
The proposed mechanism is not directly validated under the training dynamics used in the experiments. Equation (2) describes a linearized gradient step restricted to write matrices, whereas the experiments update all parameters with AdamW, including momentum, preconditioning, and clipping. Neither nor measures . Committed-update loss profiles show that harmful steps occur, but do not quantitatively connect them to the proposed kernel condition.
-
Persistent collapse remains unexplained. A condition for one-step loss increase does not explain why training converges to a zero-output or unigram-like plateau, or why it fails to recover. This distinction matters because the frozen-kernel experiment produces instability without the same plateau. The experiments describe the collapse dynamics in detail, but do not identify what sustains the degeneration.
-
GROW does not isolate the hypothesized causal factor. Retraction changes weight geometry, filters update directions, and reduces update magnitude at the same time. The fixed 0.68 scaling control neither matches update norms at each step nor is calibrated to the final configuration. Tangent/normal sensitivity measurements and counterfactual tests are explicitly left untested, so the link between the removed update component and instability remains unverified.
-
Evidence for broader applicability and practical advantage is limited. GROW is not evaluated on language models, and the comparisons allocate 4000 training steps to GROW versus 3500 to AdamW, on top of the additional retraction cost. There is no direct comparison with related manifold optimizers. Moreover, AdamW already survives the tested constant dose at the greatest depth. GROW appears useful in the studied DiT settings, but these results do not establish whether its benefits extend beyond those settings or justify its additional cost.
-
Poor readability makes the technical argument difficult to assess. The abstract is overloaded, and the main text moves between diagnostics and experimental variants without clearly explaining what each is meant to establish. Frequent qualifications and appendix references make it hard to follow how the evidence supports the proposed mechanism. In particular, the connection between the output-space analysis and GROW’s weight-space constraint needs a more direct explanation.
Questions for authors
Typos
Appendix E refers to lines 4–10 of Algorithm 1 as the retraction procedure; the loop extends through line 14.
Response to Reviewer fiJ6: the listed soundness concerns are limitations the paper states; scope of the claims
We thank the reviewer for the detailed review. Several of the weaknesses listed quote limitations that the paper itself states; we mark where, and say what the paper does and does not claim.
W1 (mechanism not validated under AdamW; t/r and F_k do not measure Γ). The second sentence is the paper's own: Sec. 2.2 says both quantities describe the residual's coupling to high-gain directions, "not Γ itself", and App. D repeats it. Eq. 2 is presented as a hypothesis in a linearized model; Eq. 1, on which the committed-step profiles rest, is exact for any optimizer. The link between the two is supported by timing at three depths (t/r leaves its band before the validation symptom), the absence of damage after any snapshot with all stored gains shrinking (15 collapses), a 0.9% quantitative closure of the measured gain (App. C), α_safe of 1.0 on quiet vs 0-0.25 on burst steps, and the Gram-freeze and GROW interventions. We agree this is not a per-step measurement of Γ on the AdamW update and do not claim one.
W2 (persistent collapse unexplained). We agree, and Sec. 3.3 states: "the experiments do not establish what is required for the plateau to form". Our claim concerns the onset of instability; Sec. 1 defines "mechanism" as the one-step condition plus the links we could observe and intervene on. The frozen-kernel experiment the reviewer cites is how we establish this separation, which the reviewer lists as a strength. We will make the scope explicit in the abstract.
W3 (GROW does not isolate the causal factor). We agree. Sec. 4 states that GROW "acts in two places" and that the runs "do not separate which of the two carries the stability"; Sec. 5 describes the 0.68 control as "not a per-step match, nor the ratio at α²=16" and lists the counterfactuals that "were not run". What the controls do show is that neither zero writer decay (2/2 collapse) nor a constant step reduction (rescues the quiet seed, not the violent one) reproduces GROW's survival across seeds. GROW is offered as an intervention on write geometry that moves the boundary, not as an identified cause.
W4 (applicability and cost).
W5 (readability). We accept this. We will shorten the abstract, add a roadmap stating what each diagnostic is meant to establish, and move the output-space to weight-space link (Eq. 3: what one write matrix can add to its sublayer Jacobian is bounded by its Gram) to the opening of Sec. 4.
Typo. Thank you; the retraction loop is lines 4-14 of Algorithm 1 (line 15 is logging). Corrected.
In summary, we do not dispute that the paper leaves the plateau and GROW's causal factor open. We ask the reviewer to consider whether a soundness score of 1 is warranted when the listed concerns are limitations the paper states and does not claim beyond, and no reported result is identified as incorrect.