- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the optimization behavior of Spectral GD in deep residual networks in the near-identity target regime. It argues that the earlier motivation for Spectral GD’s advantage, derived for deep linear networks, does not directly apply to deep residual networks. For an aligned model with isotropic inputs, it derives exact dynamics of both continuous-time GD and Spectral GD. For a conditional block model, it proves a criterion for when Spectral GD decreases the correction risk faster than GD. LLM experiments and synthetic experiments examine the practical side of the benefits of Spectral GD and Muon.
Critical strengths
-
The paper examines whether explanations of spectral optimization in plain deep linear networks extend to near-identity residual networks. It distinguishes full-block conditioning from residual-branch optimization geometry, and separates correction size from the factors that determine learning speed.
-
Theorem 5 gives an explicit condition under which Spectral GD initially reduces unweighted correction error faster than GD at matched speed. This provides a testable prediction, although its connection to practical performance remains to be established.
-
The skip-strength ablation directly measures both residual-branch and full-block Jacobians. It provides controlled evidence that a relatively uniform full-block spectrum can coexist with a concentrated residual-branch spectrum.
Critical weaknesses
-
The comparison in Section 4 does not isolate the effect of residual parameterization. The substitution turns the plain deep linear network into a residual network. With corresponding initializations, the two models follow identical optimization trajectories under the same optimizer. The comparison therefore does not separate an effect of skip connections from differences in initialization. I would recommend narrowing the motivation to the claim that, in the near-identity regime relevant to residual networks, the classical explanation for Spectral GD’s advantage no longer applies directly.
-
The strength and practical relevance of Assumptions 1 and 3 need further discussion. Assumption 1 requires isotropic inputs and a common orthogonal eigenbasis for the network layers and the target. It is clear how this simplifies the analysis. However, its connection to practice and the strength of the assumption need clarification. Appendix A.2 relaxes isotropy only for input covariances that commute with the target. For Assumption 3, the main text would benefit from a short explanation of its effect on Eq. (33). Writing the resulting loss from Appendix A.4 would explain why describes the geometry and why Eq. (38) is called unweighted. This would make the whole argument easier to follow. Diagonalizing and does not imply that shares their left and right bases. Appendix B.1 considers approximate local alignment, but the experiments do not assess whether these perturbations are small. The paper should discuss the practical plausibility of Assumption 3.
-
The experiments do not verify the near-identity Jacobian regime used in the theory. Figure 2 measures the relative forward change. It describes the behavior on activations, but does not bound . Figures 5 and 6 provide information about the Jacobian spectrum, but this still does not bound , as discussed in the theory.
Comment on Figure 5 The network in Appendix C.4 includes LayerNorm, which is invariant to adding a constant to every coordinate. Thus, when the block Jacobian has a zero singular value, which possibly explains the very large condition number in figure 5.
Minor presentation issues
- The loss used to derive Eq. (13) and (14) is undefined in the main text. The definition currently appears in Appendix A.1.
- used in Proposition 2 is undefined in the main text.
- Table 1 labels the second optimizer as SpecGD, while Section 6 attributes the same results to Muon.
The weaknesses above seem fixable. If the authors address them, I am happy to increase my score.
Questions for authors
-
What motivates the use of the unweighted risk in Theorem 5 instead of the block-conditional loss in Eqs. (29) and (33)?
-
Do the synthetic experiments in Appendix C satisfy condition (41), and are the initial Frobenius speeds matched as in Theorem 5? Could you compare cases where the condition holds and fails?
3 comments
Q1:
Our motivation is that recovering the underlying residual correction is a meaningful objective in its own right, beyond minimizing the loss weighted by the current training geometry.
In our aligned conditional model, the training loss and the unweighted correction risk are respectively
=\frac12\sum_i c_i(w_i-d_i)^2, \qquad \mathcal R(W) =\frac12\sum_i(w_i-d_i)^2.$$ The coefficients $c_i$, determined by the input/output geometry, control how strongly different correction errors contribute to the training loss and how quickly GD learns them. The unweighted risk instead directly measures the fidelity of the learned correction to its target, without introducing the same geometry-dependent weighting into its evaluation. It therefore provides a complementary measure of correction recovery. This distinction also has implications for generalization under distribution shifts. Directions that are weakly represented during training, due to small $c_i$, may become more important under a different evaluation distribution. A model that more accurately recovers these corrections may consequently achieve better predictions after such a shift. ### Q2: We have conducted additional synthetic experiments using the exact squared Frobenius correction risk and matched initial Frobenius speeds. For the original 64-mode correction-spectrum family, we use $$d_i=0.1\rho^{i-1}, \qquad c_i=0.9^{i-1}, \qquad i=1,\ldots,64.$$ We evaluate the difference between the two sides of condition (41), $$m=\frac{\sum_i d_i}{\sqrt r}-\frac{\sum_i c_i d_i^2}{\|c\odot d\|_2},$$ with learning rates $\eta_G=1/\|c\odot d\|_2$ and $\eta_S=1/\sqrt r$, so that both methods have unit initial Frobenius velocity. | $\rho$ | Condition margin $m$ | Faster initial risk decrease | |---:|---:|---| | 1.00 | +0.364624 | SpecGD | | 0.99 | +0.208087 | SpecGD | | 0.98 | +0.106071 | SpecGD | | 0.97 | +0.038981 | SpecGD | | 0.96 | −0.005617 | GD | | 0.95 | −0.035614 | GD | The observed initial risk ordering agrees with Theorem 5 in all six settings. We further perform a controlled intervention that changes only the correspondence between correction amplitudes and geometry coefficients. At $\rho=0.95$, replacing $c_i=0.9^{i-1}$ with the reversed pairing $c_i=0.9^{64-i}$ changes the margin from -0.035614 to +0.166325. The initial risk advantage correspondingly reverses from GD to SpecGD, while both spectra, the correction target, and the matched update magnitude remain unchanged. This directly verifies that the relative arrangement of correction amplitudes and optimization geometry determines the risk advantage predicted by Thm 5. The original Appendix C experiments used fixed learning rates (i.e., not matched Frobenius norm) and reported a different mode-error metric in the correction-spectrum sweep. We will clarify those settings and add the new matched-speed experiments, including both positive and negative cases of condition (41), to the revised appendix.Thank you for the detailed response and the additional experiments. The clarifications and new results resolve my main concerns, and I am raising my score to 3.
We thank the reviewer for the constructive and insightful comments. We especially appreciate the observation about Figure 5. The explanation based on LayerNorm invariance is compelling and provides a much clearer interpretation of the reported condition numbers. We agree with the suggested change and will revise both the figure description and the surrounding presentation accordingly.
W1:
We agree with the reviewer's observation that the change of variables Vℓ=I+Wℓ gives equivalent optimization trajectories with suitable initializations. Consequently, the comparison in Sec.4 is best understood as a comparison between optimization regimes, rather than as a difference caused by the parameterization alone. We will follow the reviewer's suggestion and narrow the corresponding statements in the introduction and Sec.4 to the near-identity regime relevant to residual networks.
W2:
We directly examined Assumption 3 in the 48-layer model. Let
H_{\mathrm{in}} = V_{\mathrm{in}} \Lambda V_{\mathrm{in}}^\top,\qquad \Delta^\star = U_\Delta D V_\Delta^\top$$ We then visualize three closely related matrices in Fig.8 of the Appendix C.6 of the updated version of the paper. In the left panel, we show $\left|U_{\mathrm{out}}^\top \Delta^\star V_{\mathrm{in}}\right|$(elementwise magnitudes of the corresponding matrices)which is exactly the matrix $M$ obtained by expressing the learned correction in the output and input geometry eigenbases. If Assumption 3 holds approximately, this matrix should be close to diagonal. In the middle panel, we show $\left|U_\Delta^\top H_{\mathrm{out}} U_\Delta\right|$, which measures whether the left singular vectors of $\Delta^\star$ are close to eigenvectors of $H_{\mathrm{out}}$. In the right panel, we show $\left|V_\Delta^\top H_{\mathrm{in}} V_\Delta\right|$,which measures the analogous property on the input side. All three panels exhibit a clear diagonal pattern. This indicates that the learned correction and the two geometry factors share a substantial amount of common spectral structure. These observations are also consistent with Aubry et al. (*Transformer Block Coupling and its Correlation with Generalization in LLMs*, ICLR 2025), who identify coupling among leading singular vectors of Transformer block Jacobians across layers and tokens. ### W3: We have directly measured $J_{\mathrm{block}}-I$ in trained 48-layer Transformers. For the full block, we estimate the singular-value distribution of $J_{\mathrm{block}}-I$ using stochastic Lanczos quadrature on real text sequences. The estimates are averaged across 48 layers: | Singular values of $J_{\mathrm{block}}-I$ | AdamW | Muon | |---|---:|---:| | At most 0.2 | 73.3% | 63.1% | | At most 0.5 | 95.2% | 92.1% | | At most 1.0 | 99.7% | 99.7% | These measurements directly support a small residual spectral bulk in trained 48-layer Transformers, which support our hypothesis of the near-identity regime.