acceptodds
Under review as a conference paper at ICLR 2027

Spectral Optimization in Residual Networks: Learning Anisotropic Corrections

Abstract

Spectral optimizers such as Muon are often studied through deep linear models, where repeated matrix products create mode-dependent learning rates that become more uneven with depth. Modern pre-norm residual networks operate in a different regime. Their blocks can remain close to the identity map, which makes the classical product-induced factor nearly constant across modes. Meanwhile, the identity path can hide substantial anisotropy in the trainable residual correction. We argue that residual connections therefore shift, rather than remove, the relevant conditioning problem: the identity path stabilizes the full map, while the optimizer must learn a residual correction that has anisotropic structure. Under anisotropic residual-branch geometry in deep linear networks, gradient descent learns correction modes unevenly, while spectral updates balance progress across unlearned directions. Our results suggest that Muon's benefit in deep residual architectures arises because residual parameterization exposes a local spectral correction problem well matched to spectral optimization.

Primary area:
optimization
Keywords:
Deep residual linear networks, Muon, Optimization theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7Fb5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the optimization behavior of Spectral GD in deep residual networks in the near-identity target regime. It argues that the earlier motivation for Spectral GD’s advantage, derived for deep linear networks, does not directly apply to deep residual networks. For an aligned model with isotropic inputs, it derives exact dynamics of both continuous-time GD and Spectral GD. For a conditional block model, it proves a criterion for when Spectral GD decreases the correction risk faster than GD. LLM experiments and synthetic experiments examine the practical side of the benefits of Spectral GD and Muon.

Critical strengths
  1. The paper examines whether explanations of spectral optimization in plain deep linear networks extend to near-identity residual networks. It distinguishes full-block conditioning from residual-branch optimization geometry, and separates correction size from the factors that determine learning speed.

  2. Theorem 5 gives an explicit condition under which Spectral GD initially reduces unweighted correction error faster than GD at matched speed. This provides a testable prediction, although its connection to practical performance remains to be established.

  3. The skip-strength ablation directly measures both residual-branch and full-block Jacobians. It provides controlled evidence that a relatively uniform full-block spectrum can coexist with a concentrated residual-branch spectrum.

Critical weaknesses
  1. The comparison in Section 4 does not isolate the effect of residual parameterization. The substitution Vℓ=I+WℓV_\ell=I+W_\ell turns the plain deep linear network into a residual network. With corresponding initializations, the two models follow identical optimization trajectories under the same optimizer. The comparison therefore does not separate an effect of skip connections from differences in initialization. I would recommend narrowing the motivation to the claim that, in the near-identity regime relevant to residual networks, the classical explanation for Spectral GD’s advantage no longer applies directly.

  2. The strength and practical relevance of Assumptions 1 and 3 need further discussion. Assumption 1 requires isotropic inputs and a common orthogonal eigenbasis for the network layers and the target. It is clear how this simplifies the analysis. However, its connection to practice and the strength of the assumption need clarification. Appendix A.2 relaxes isotropy only for input covariances that commute with the target. For Assumption 3, the main text would benefit from a short explanation of its effect on Eq. (33). Writing the resulting loss from Appendix A.4 would explain why cic_i describes the geometry and why Eq. (38) is called unweighted. This would make the whole argument easier to follow. Diagonalizing HoutH_{out} and HinH_{in} does not imply that Δ⋆\Delta^{\star} shares their left and right bases. Appendix B.1 considers approximate local alignment, but the experiments do not assess whether these perturbations are small. The paper should discuss the practical plausibility of Assumption 3.

  3. The experiments do not verify the near-identity Jacobian regime used in the theory. Figure 2 measures the relative forward change. It describes the behavior on activations, but does not bound ∥Jblock−I∥2\Vert J_{\mathrm{block}}-I\Vert_2. Figures 5 and 6 provide information about the Jacobian spectrum, but this still does not bound ∥Jblock−I∥2\Vert J_{\mathrm{block}}-I\Vert_2, as discussed in the theory.

Comment on Figure 5 The network in Appendix C.4 includes LayerNorm, which is invariant to adding a constant to every coordinate. Thus, when α=0\alpha = 0 the block Jacobian has a zero singular value, which possibly explains the very large condition number in figure 5.

Minor presentation issues

  • The loss used to derive Eq. (13) and (14) is undefined in the main text. The definition currently appears in Appendix A.1.
  • rir_i used in Proposition 2 is undefined in the main text.
  • Table 1 labels the second optimizer as SpecGD, while Section 6 attributes the same results to Muon.

The weaknesses above seem fixable. If the authors address them, I am happy to increase my score.

Questions for authors
  1. What motivates the use of the unweighted risk RR in Theorem 5 instead of the block-conditional loss L\mathcal{L} in Eqs. (29) and (33)?

  2. Do the synthetic experiments in Appendix C satisfy condition (41), and are the initial Frobenius speeds matched as in Theorem 5? Could you compare cases where the condition holds and fails?

3 comments

We thank the reviewer for the constructive and insightful comments. We especially appreciate the observation about Figure 5. The explanation based on LayerNorm invariance is compelling and provides a much clearer interpretation of the reported condition numbers. We agree with the suggested change and will revise both the figure description and the surrounding presentation accordingly.

W1:

We agree with the reviewer's observation that the change of variables Vℓ=I+WℓV_\ell=I+W_\ell gives equivalent optimization trajectories with suitable initializations. Consequently, the comparison in Sec.4 is best understood as a comparison between optimization regimes, rather than as a difference caused by the parameterization alone. We will follow the reviewer's suggestion and narrow the corresponding statements in the introduction and Sec.4 to the near-identity regime relevant to residual networks.

W2:

We directly examined Assumption 3 in the 48-layer model. Let

H_{\mathrm{in}} = V_{\mathrm{in}} \Lambda V_{\mathrm{in}}^\top,\qquad \Delta^\star = U_\Delta D V_\Delta^\top$$ We then visualize three closely related matrices in Fig.8 of the Appendix C.6 of the updated version of the paper. In the left panel, we show $\left|U_{\mathrm{out}}^\top \Delta^\star V_{\mathrm{in}}\right|$(elementwise magnitudes of the corresponding matrices)which is exactly the matrix $M$ obtained by expressing the learned correction in the output and input geometry eigenbases. If Assumption 3 holds approximately, this matrix should be close to diagonal. In the middle panel, we show $\left|U_\Delta^\top H_{\mathrm{out}} U_\Delta\right|$, which measures whether the left singular vectors of $\Delta^\star$ are close to eigenvectors of $H_{\mathrm{out}}$. In the right panel, we show $\left|V_\Delta^\top H_{\mathrm{in}} V_\Delta\right|$,which measures the analogous property on the input side. All three panels exhibit a clear diagonal pattern. This indicates that the learned correction and the two geometry factors share a substantial amount of common spectral structure. These observations are also consistent with Aubry et al. (*Transformer Block Coupling and its Correlation with Generalization in LLMs*, ICLR 2025), who identify coupling among leading singular vectors of Transformer block Jacobians across layers and tokens. ### W3: We have directly measured $J_{\mathrm{block}}-I$ in trained 48-layer Transformers. For the full block, we estimate the singular-value distribution of $J_{\mathrm{block}}-I$ using stochastic Lanczos quadrature on real text sequences. The estimates are averaged across 48 layers: | Singular values of $J_{\mathrm{block}}-I$ | AdamW | Muon | |---|---:|---:| | At most 0.2 | 73.3% | 63.1% | | At most 0.5 | 95.2% | 92.1% | | At most 1.0 | 99.7% | 99.7% | These measurements directly support a small residual spectral bulk in trained 48-layer Transformers, which support our hypothesis of the near-identity regime.

Q1:

Our motivation is that recovering the underlying residual correction is a meaningful objective in its own right, beyond minimizing the loss weighted by the current training geometry.

In our aligned conditional model, the training loss and the unweighted correction risk are respectively

=\frac12\sum_i c_i(w_i-d_i)^2, \qquad \mathcal R(W) =\frac12\sum_i(w_i-d_i)^2.$$ The coefficients $c_i$, determined by the input/output geometry, control how strongly different correction errors contribute to the training loss and how quickly GD learns them. The unweighted risk instead directly measures the fidelity of the learned correction to its target, without introducing the same geometry-dependent weighting into its evaluation. It therefore provides a complementary measure of correction recovery. This distinction also has implications for generalization under distribution shifts. Directions that are weakly represented during training, due to small $c_i$, may become more important under a different evaluation distribution. A model that more accurately recovers these corrections may consequently achieve better predictions after such a shift. ### Q2: We have conducted additional synthetic experiments using the exact squared Frobenius correction risk and matched initial Frobenius speeds. For the original 64-mode correction-spectrum family, we use $$d_i=0.1\rho^{i-1}, \qquad c_i=0.9^{i-1}, \qquad i=1,\ldots,64.$$ We evaluate the difference between the two sides of condition (41), $$m=\frac{\sum_i d_i}{\sqrt r}-\frac{\sum_i c_i d_i^2}{\|c\odot d\|_2},$$ with learning rates $\eta_G=1/\|c\odot d\|_2$ and $\eta_S=1/\sqrt r$, so that both methods have unit initial Frobenius velocity. | $\rho$ | Condition margin $m$ | Faster initial risk decrease | |---:|---:|---| | 1.00 | +0.364624 | SpecGD | | 0.99 | +0.208087 | SpecGD | | 0.98 | +0.106071 | SpecGD | | 0.97 | +0.038981 | SpecGD | | 0.96 | −0.005617 | GD | | 0.95 | −0.035614 | GD | The observed initial risk ordering agrees with Theorem 5 in all six settings. We further perform a controlled intervention that changes only the correspondence between correction amplitudes and geometry coefficients. At $\rho=0.95$, replacing $c_i=0.9^{i-1}$ with the reversed pairing $c_i=0.9^{64-i}$ changes the margin from -0.035614 to +0.166325. The initial risk advantage correspondingly reverses from GD to SpecGD, while both spectra, the correction target, and the matched update magnitude remain unchanged. This directly verifies that the relative arrangement of correction amplitudes and optimization geometry determines the risk advantage predicted by Thm 5. The original Appendix C experiments used fixed learning rates (i.e., not matched Frobenius norm) and reported a different mode-error metric in the correction-spectrum sweep. We will clarify those settings and add the new matched-speed experiments, including both positive and negative cases of condition (41), to the revised appendix.

Thank you for the detailed response and the additional experiments. The clarifications and new results resolve my main concerns, and I am raising my score to 3.

Reviewer_PPfDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyzes why muon optimizer can benefit residual networks. The authors investigate aligned linear models and think the classical depth-based explanation is weakened when the target is nearly identity. The authors distinguish the amount of correct in each spectral direction from how the inputs and outputs in each direction influence the learning speed. Theoretical analysis is provided and synthetic experiments are conducted to validate the claims.

Critical strengths
  1. Sections 4 and 5 show that large differences in residual magnitude do not necessarily imply different learning speed. This helps avoid considering residual anisotropy alone as evidence of optimization difficulty. It distinguishes what the network need to learn from how the inputs and other factors produce weaker gradients in some directions.
Critical weaknesses
  1. The additional insight beyond existing spectral balancing explanation is limited. The direction-wise dynamic in Section 5.2 resembles those in Vasudeva et al. (2026) that ordinary gd depends on directional weights while spectral gd at a constant speed until reach the target. Thus, the proposed mechanism does not require depth-based or residual connections. Moreover, the paper does not sufficiently distinguish the effect of residual connection from initialization and the nearly identity condition.

  2. The experiments cannot effectively validate the proposed mechanism is the root of the advatange of muon. The skip strength experiments measure change in branch and full block jacobians but use adamw instead of how the relative benefit of muon changes with skip strength. Figure 2 and table 1 show optimizer performance and directional imbalance but do not verify that the condition required by theorem 5 holds during training and explains the gains.

Questions for authors
  1. What new insight does Section 5.2 provide beyond Vasudeva et al. (2026)? Which part of the explanation requires depth or residual connections?

  2. Can the authors compare Muon and AdamW under different skip strengths to test whether residual connections explain Muon's relative advantage? Does the condition in Theorem 5 hold during training, and is there evidence that it explains the gains?

1 comment

We thank the reviewer for the thoughtful comments.

W1 & Q1:

We thank the reviewer for helping us clarify the positioning of our contribution relative to Vasudeva et al. (2026). We agree that SpecGD's constant absolute mode-wise velocity is an established property and does not inherently require residual connections. Our contribution is identifying why and under what conditions spectral optimization remains relevant in the near-identity residual regime.

The existence of residual connections makes it easier for the model to enter the near-identity regime under appropriate initialization, while leaving the trainable branches to learn potentially anisotropic corrections. Sec.4 shows that, in this regime, the classical depth-induced mode imbalance is substantially suppressed: even highly anisotropic correction amplitudes need not produce unequal fractional learning rates under GD. Sec.5 then identifies what remains relevant to spectral optimization: the identity path can mask substantial residual-branch anisotropy, while the actual learning dynamics depend on the input/output geometry cic_i, not merely the correction amplitudes did_i. Beyond the established equalization property, Theorem 5 provides an explicit matched-speed criterion characterizing when spectral updates improve the unweighted correction risk.

W2 & Q2:

We conducted additional experiments with a 48-layer Transformer with different MLP skip strength. The final validation cross entropy results are:

Skip α\alphaAdamW CEMuon CEAdamW − Muon
0.805.65183.36682.2850
0.903.96843.23170.7367
0.953.40143.21620.1852
0.983.27093.20990.0609
1.003.25443.20660.0478

Changing skip strength substantially changes the measured input/output optimization geometry and the relative performance of the two optimizers.

To examine the effect of geometry more directly, we conducted a fixed-target intervention. We first independently fitted reference correction targets Δ⋆\Delta^\star using pretrained models with α=1\alpha=1. We constructed 12 settings from two source optimizers (AdamW and Muon), three checkpoints, and two Layer-24 correction types (down projection and square residual probe). For each setting, we kept Δ⋆\Delta^\star fixed while replacing the input/output geometry matrices HinH_{\mathrm{in}} and HoutH_{\mathrm{out}} with those measured from models trained at different skip strengths. This changes the geometry governing optimization without changing the correction to be learned.

We optimized each resulting quadratic problem using AdamW and Muon from the same initialization. We measured the unweighted correction risk. The table reports how many of the same 12 settings achieved lower correction risk with Muon:

Geometry source α\alphaMuon lower correction setting
0.8012/12
0.909/12
0.957/12
0.984/12
1.003/12

The results show a clear reversal in the relative performance of the optimizers. Since the correction target is fixed within each setting, changing the optimization geometry alone can reverse the relative performance of AdamW and Muon.

Reviewer_cGZYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the usual deep linear story for why Muon helps still holds in residual networks, and finds it mostly does not: near identity, depth splits the target correction across layers additively rather than compressing modes, so the classical equalization picture flips. In a single residual block, anisotropic input and output geometry makes GD learn modes at uneven rates while an idealized spectral update (SpecGD) removes that factor, and Theorem 5 gives an exact condition for when SpecGD cuts the correction risk faster at matched initial speed; the experiments are diagnostics only and never test this mechanism in a real network.

Critical strengths
  • The exact mode analysis is clean. Proposition 2 gives exact aligned mode dynamics and shows the depth-based equalization of Vasudeva et al. does not carry over to residual networks, and the Section 5.2 decomposition separates the correction amplitude d_i from the local geometry factor c_i = gamma_i lambda_i, which makes precise what an anisotropic optimizer-facing problem means.
  • Theorem 5 is stated honestly, as an if and only if condition on the initial risk decrease at matched initial Frobenius speed (Eq. 41). Proposition 6 and Theorem 7 also partly answer the alignment objection, with finite-time stability under approximate alignment and dynamics for factorized residual branches.
  • The experiments at least measure the quantities the theory is about, not just final loss. The skip-strength sweep uses exact Jacobians checked against automatic differentiation, and Appendix C.5 measures the effective rank of the local optimization geometry directly.
Critical weaknesses
  • All the theory is in aligned linear models with near-identity targets, and the step to real pre-norm nonlinear networks is asserted, not analyzed. The mechanism also rests on Assumption 3, joint diagonalizability of H_out, the conditional target correction, and H_in, and the paper never measures how close a trained network comes to any of this.
  • No experiment tests the mechanism in a real network. The decoder run compares Muon with AdamW, not the GD versus SpecGD pair the theory analyzes, and the 130M C4 run reports only geometry effective ranks, with no loss comparison and no variation of a quantity the theory says should move the optimizer gap.
  • Theorem 5 compares only the risk derivative at t = 0, under an unweighted Frobenius risk that is not the training loss, so the abstract's suggestion that Muon's benefit arises from this local correction problem is more than an initial-time, local, linearized result can carry. The novelty over Saxe et al., Vasudeva et al., and Sun et al. is also incremental: a useful reframing plus a local result, not an explanation of Muon in Transformers.
Questions for authors
  1. Can you measure alignment in the trained decoder model? For example, the principal angles between the eigenvectors of H_in and H_out and the singular vectors of the conditional correction for one trained block. I want to know how close Assumption 3 comes to holding in practice.
  2. Theorem 5 is about the risk derivative at t = 0. In the conditional model of Theorem 4, does the SpecGD advantage in unweighted correction risk last over the full trajectory, or can GD reach lower risk at finite time in some regimes?
  3. The theory says the spectral advantage is governed by the anisotropy of c_i relative to d_i. Can you vary that quantity in a real network, for example by changing skip strength, and check whether the Muon versus GD gap moves in the predicted direction?
1 comment

We thank the reviewer for the careful assessment and constructive suggestions.

W1 & Q1:

We directly examined Assumption 3 in the 48-layer model. Let

H_{\mathrm{in}} = V_{\mathrm{in}} \Lambda V_{\mathrm{in}}^\top,\qquad \Delta^\star = U_\Delta D V_\Delta^\top$$ We then visualize three closely related matrices in Fig.8 of the Appendix C.6 of the updated version of the paper. In the left panel, we show $\left|U_{\mathrm{out}}^\top \Delta^\star V_{\mathrm{in}}\right|$(elementwise magnitudes of the corresponding matrices)which is exactly the matrix $M$ obtained by expressing the learned correction in the output and input geometry eigenbases. If Assumption 3 holds approximately, this matrix should be close to diagonal. In the middle panel, we show $\left|U_\Delta^\top H_{\mathrm{out}} U_\Delta\right|$, which measures whether the left singular vectors of $\Delta^\star$ are close to eigenvectors of $H_{\mathrm{out}}$. In the right panel, we show $\left|V_\Delta^\top H_{\mathrm{in}} V_\Delta\right|$,which measures the analogous property on the input side. All three panels exhibit a clear diagonal pattern. This indicates that the learned correction and the two geometry matrices share a substantial amount of common spectral structure. These observations are also consistent with Aubry et al. (*Transformer Block Coupling and its Correlation with Generalization in LLMs*, ICLR 2025), who identify coupling among leading singular vectors of Transformer block Jacobians across layers and tokens. ### W2 & Q3: We conducted additional experiments with a 48-layer Transformer with different MLP skip strength. The final validation cross entropy results are: | Skip $\alpha$ | AdamW CE | Muon CE | AdamW − Muon | |---:|---:|---:|---:| | 0.80 | 5.6518 | 3.3668 | 2.2850 | | 0.90 | 3.9684 | 3.2317 | 0.7367 | | 0.95 | 3.4014 | 3.2162 | 0.1852 | | 0.98 | 3.2709 | 3.2099 | 0.0609 | | 1.00 | 3.2544 | 3.2066 | 0.0478 | Changing skip strength substantially changes the measured input/output optimization geometry and the relative performance of the two optimizers. To examine the effect of geometry more directly, we conducted a **fixed-target intervention**. We first independently fitted reference correction targets $\Delta^\star$ using pretrained models with $\alpha=1$. We constructed 12 settings from two source optimizers (AdamW and Muon), three checkpoints, and two Layer-24 correction types (down projection and square residual probe). For each setting, we kept $\Delta^\star$ fixed while replacing the input/output geometry matrices $H_{\mathrm{in}}$ and $H_{\mathrm{out}}$ with those measured from models trained at different skip strengths. This changes the geometry governing optimization without changing the correction to be learned. We optimized each resulting quadratic problem using AdamW and Muon for 1024 steps from the same initialization. We measured the unweighted correction risk. The table reports how many of the same 12 settings achieved lower correction risk with Muon: | Geometry source $\alpha$ | Muon lower correction setting | |---:|---:| | 0.80 | 12/12 | | 0.90 | 9/12 | | 0.95 | 7/12 | | 0.98 | 4/12 | | 1.00 | 3/12 | The results show a clear reversal in the relative performance of the optimizers. Since the correction target is fixed within each setting, changing the optimization geometry alone can reverse the relative performance of AdamW and Muon. ### W3 & Q2: We have extended Thm 5 to provide an explicit finite-time guarantee using the exact trajectories from Thm 4. Under matched initial Frobenius speeds, define the condition margin $$m=\frac{\sum_i d_i}{\sqrt r}-\frac{\sum_i c_i d_i^2}{\|c\odot d\|_2}.$$ Let $D(t)=\mathcal R_{\mathrm{GD}}(t)-\mathcal R_{\mathrm{Spec}}(t)$. Theorem 4 gives the exact risk trajectories $$\mathcal R_{\mathrm{GD}}(t)=\frac12\sum_i d_i^2 e^{-2c_it/\\|c\odot d\\|_2},\qquad \mathcal R_{\mathrm{Spec}}(t)=\frac12\sum_i\left(d_i-\frac{t}{\sqrt r}\right)_+^2.$$ We have $D(0)=0$ and $D'(0)=m$. The GD risk is convex in time, while the second derivative of the SpecGD risk is at most one wherever it exists. Consequently $D(t)\ge mt-\frac{t^2}{2}$.Thus, whenever the condition in Thm 5 holds strictly, SpecGD has lower correction risk throughout the explicit interval $0<t<2m$, and $D(t)\ge mt/2$ for $0<t\le m$. This quantitative finite-time result follows under the same assumptions as Thm 5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Contributions and Rebuttal

Dear Area Chair,

We would like to briefly summarize the main contributions of our work and the additional results provided during the rebuttal. We sincerely thank all reviewers for their constructive feedback and thoughtful assessment.

We appreciate the reviewers' recognition of several central contributions of our paper. Reviewer cGZY highlighted the clean exact mode analysis, the distinction between correction amplitudes and optimization geometry, and the theoretical analysis in Theorem 5. Reviewer PPfD emphasized the insight that anisotropic residual corrections do not necessarily imply uneven learning rates, and the importance of distinguishing what needs to be learned from the geometry governing its optimization. Reviewer 7Fb5 recognized the motivation for revisiting classical spectral optimization explanations in residual networks, the distinction between full block conditioning and residual branch geometry, and the experimental evidence supporting this perspective. Together, these assessments highlight the main contribution of our work: identifying a correction-learning regime in residual networks where the classical depth based explanation becomes insufficient, and characterizing the role of anisotropic optimization geometry.

During the rebuttal, we further strengthened the theoretical results. We extended Theorem 5 to provide an explicit finite-time guarantee under the same assumptions, moving beyond the initial risk derivative comparison. Additional synthetic experiments verify the predicted risk ordering in both positive and negative cases, including a reversal when the correspondence between correction amplitudes and optimization geometry is changed.

We also substantially expanded the empirical evidence. Direct Jacobian measurements support the presence of a near-identity regime. The joint-diagonalization diagnostics reveal structured alignment between the learned correction and the input/output geometry, which supports Assumption 3. Moreover, experiments varying the skip strength demonstrate substantial changes in the relative validation performance of AdamW and Muon, which establish a connection between our theoretical analysis and optimization behaviors in real LLMs.

We are also grateful that Reviewer 7Fb5 explicitly acknowledged that the additional experiments and clarifications resolved their main concerns and increased their rating.

We hope that the clarified theoretical contribution and strengthened empirical evidence will be helpful in the final assessment. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.