- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces Spectral Alignment (SA), the signed cosine similarity between layer inputs and a weight matrix’s principal left singular direction, as an early-warning indicator of training instability. The central idea is that amplification depends on both weight geometry and input direction. The theoretical analysis connects sufficiently one-sided alignment to spectral growth under compatible downstream gradients. Experiments in two language-model failure settings show that changes in SA precede loss explosion and can inform checkpoint selection for recovery.
Critical strengths
- An interpretable perspective on training instability. SA captures directional relationships between inputs and weights that scalar magnitude-based monitors may overlook. Its invariance to positive scalar rescaling makes it a useful complementary diagnostic.
- A connection between theory and empirical observations. The analysis provides a conditional explanation for spectral growth, while the experiments demonstrate an observable pre-failure alignment pattern.
- Practical relevance. The monitoring procedure and checkpoint-selection experiments suggest useful applications to early intervention and recovery during language-model training.
Critical weaknesses
- The main theoretical analysis considers gradient descent or SGD, whereas the experiments use AdamW. Although the appendix discusses the extension, empirical validation of the relevant conditions would strengthen the connection between the theoretical mechanism and the observed failures. I consider this a minor limitation.
Questions for authors
- Does one-sided spectral alignment tend to reinforce itself during subsequent training? Can its persistence be derived from the coupled dynamics of inputs, weights, and downstream gradients, and could this analysis help predict the time to instability?
2 comments
Thank you for the clarification and the additional empirical analysis.
The reported sign agreement between the SGD prediction and the actual AdamW update helps address my concern about the connection between the theory and the experiments. While this does not fully close the theoretical gap, it provides useful empirical support. Please include the measurement setup in the revision, particularly which runs, layers, and training intervals were used to obtain the reported 80%.
Regarding the persistence of one-sided spectral alignment, I appreciate the distinction between the empirical observations and what has been established theoretically. A formal analysis of the coupled dynamics remains an interesting direction for future work, and I do not view its absence as a major weakness of the current paper.
Overall, the response addresses my main concern sufficiently, and I maintain my weak acceptance rating.
Thank you for the positive review.
W1 — The theory considers GD/SGD while the experiments use AdamW.
We compared the first-order spectral increment predicted from the raw gradient (Eq. 10) with the increment induced by the actual AdamW update, −η u₁ᵀP_t v₁ − ηλσ₁. When the SGD prediction is positive, the AdamW increment is also positive in 80% of steps. This is consistent with the role of the first moment: Adam's exponential average tends to keep the sign of the update and damp occasional opposite-sign steps. We will report this measurement in Appendix B as an empirical check of the sign condition of Proposition 3.2 under AdamW.
Q1 — Does one-sided SA reinforce itself, and can this predict the time to instability?
We have not derived this formally. Doing so would require modelling how the input distribution, shaped by all upstream layers, responds to the growing principal direction, together with the downstream gradients that set the sign of the drift. Empirically, short excursions can occur and recover, whereas in the unstable runs the one-sided state persists until failure. For the FA case, Qiu & Yao (2026) describe a mechanism in which the low-rank structure that makes the rounding error coherent is maintained by the resulting updates, which suggests one route to persistence in that setting.