acceptodds
Under review as a conference paper at ICLR 2027

Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization

Abstract

The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this transformation in feature learning remain poorly understood. In this work, we investigate this question through a tractable factual-recall model, where a fact maps each subject–relation pair to an answer, and a linear transformer learns the subject- and relation-dependent information required to recover this mapping. The transformer is optimized with gradient flow (GF), spectral GF, or Sign GF, which are continuous-time limits of gradient descent, Muon, and Adam, respectively. Prior studies (Nichani et al., 2025) have shown that when the number of subjects exceeds the number of relations, GF learns relation-dependent information before subject-dependent information, producing a feature-separation phase during training. We characterize this separation with the learning times when the subject- and relation-dependent components of the prediction reach a target accuracy. With subjects and relations, GF has a learning-time ratio of , whereas Spectral GF reduces this ratio to . In addition, for fixed and , the subject- and relation-dependent errors decay as in training time under GF, but as under spectral GF. Finally, we show that GF and spectral GF are equivariant under orthogonal transformations of the token embeddings, whereas Sign GF is not: Different orthonormal embeddings can potentially produce no feature separation, a large feature-separation phase, or even a reversed learning order. These results provide a mechanistic view of how spectral orthogonalization can fundamentally reshape feature-learning dynamics.

Primary area:
learning theory
Keywords:
Muon, factual recall, associative memory, feature learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FxbQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a novel quantity to study the behavior and advantage of Muon: the learning-time ratio. This analysis is novel in the existing literature. In addition, the authors derive the full characterization of the Muon and GD learning dynamics, which is of independent interest.

Critical strengths
  1. The main novelty of this work is the full characterization of the learning dynamics and the newly proposed quantity: the learning-time ratio.

  2. The analysis of Adam under the unitary transformation is also interesting and intuitive.

Critical weaknesses
  1. It is not clear why the data model assumes that the last token in the prompt is EoS, which is not aligned with the common practice of LLMs. It will be better to explain whether the proof still holds if the last token zTz_T is not EoS. I think WKQW_{KQ} will not have the form in (3.2) under this setting.

  2. It is not clear why we need to consider the ratio between TST_S and TRT_R. Since it is a relative quantity, it does not quantify the `` duration of the feature-separation phase’’ in Line 263. Instead, it just measures the relative size of this gap. However, the practical performance of Muon should result from its absolute value.

  3. The paper mainly focuses on the setting S>RS>R. It is better to discuss what happens if R≥SR\geq S.

  4. In addition, the paper should explicitly explain the errors decaying as 1/(T log T) from Theorem 3.4.

Questions for authors
  1. Please explain what results can be extended to the setting where the last token is not EoS.

  2. Please explain why we need to consider the ratio between TST_S and TRT_R.

4 comments

Many thanks for your positive review and thoughtful feedback!

Q1. The data model assumes that the last token in the prompt is EoS, which is not aligned with the common practice of LLMs. Does the proof still hold if the last token zTz_T is not EoS? The reviewer thinks WKQW_{KQ} will not have the form in (3.2) in that setting. Which results extend?

A1. Thank you for the question. Following [1], EOS denotes the token at which the answer is predicted, and it serves as the query that retrieves the subject and the relation from the context. Our results extend whenever the last token does not depend on the fact. For example, in a prompt format with a fixed ending word, such as "is" in "The capital of France is" or "Answer:" in a question–answer format, all results hold verbatim with ϕ(EOS)\boldsymbol\phi(\mathrm{EOS}) in (3.2) replaced by the embedding of that word. More generally, if the last token is drawn from a set Q\mathcal Q of template tokens independently of the fact, WKQ\mathbf W_{KQ} becomes (βS∑sϕ(s)+βR∑rϕ(r))(∑τ∈Qϕ(τ))⊤\big(\beta_S\sum_s\boldsymbol\phi(s)+\beta_R\sum_r\boldsymbol\phi(r)\big)\big(\sum_{\tau\in\mathcal Q}\boldsymbol\phi(\tau)\big)^\top, which is still rank one, and the reduced dynamics coincide with ours after rescaling the initialization and the time (by ∣Q∣\sqrt{|\mathcal Q|} for GF and ∣Q∣1/4|\mathcal Q|^{1/4} for Spectral GF), so all results extend. When the last token depends on the subject or the relation (e.g., relation-specific template words such as "born in"), we agree that WKQ\mathbf W_{KQ} no longer has the form (3.2), since it may need a separate query column for each value of that feature, and the analysis of this general case is substantially more difficult. We leave it for future work.


Q2. Why consider the ratio between TST_S and TRT_R? As a relative quantity it does not quantify the "duration of the feature-separation phase" (L263); it only measures the relative size of the gap, whereas the practical performance of Muon should come from the absolute value.

A2. Thank you for the question.

Absolute learning times are set by the learning rate. With learning rate γ\gamma, each of our dynamics reads Θ˙=−γF(Θ)\dot{\boldsymbol\Theta}=-\gamma F(\boldsymbol\Theta) with F=∇LF=\nabla L, msign(∇L)\mathrm{msign}(\nabla L) or sign(∇L)\mathrm{sign}(\nabla L), and the change of time τ=γt\tau=\gamma t gives TSδ(γ)=TSδ(1)/γT_S^\delta(\gamma)=T_S^\delta(1)/\gamma and TRδ(γ)=TRδ(1)/γT_R^\delta(\gamma)=T_R^\delta(1)/\gamma. Either time can thus be made arbitrarily small or large by the choice of γ\gamma (similarly for any schedule, via τ=∫0tγ(s)ds\tau=\int_0^t\gamma(s)ds; Appendix C), so comparing absolute times across optimizers compares their learning rates rather than their update rules.

The ratio removes the learning rate. The common factor 1/γ1/\gamma cancels in TSδ/TRδT_S^\delta/T_R^\delta, so the ratio depends only on the update rule and the task. Under GF, TSδ/TRδ=Θ~(S/R)T_S^\delta/T_R^\delta=\tilde\Theta(\sqrt{S/R}) for every learning rate: a larger learning rate accelerates subjects and relations by the same factor, so no tuning of GD can shorten the feature-separation phase relative to relation learning. Only changing the update direction, as spectral orthogonalization does, reduces the ratio to Θ~(1)\tilde\Theta(1).

Q3. The paper mainly focuses on the setting S>RS > R. What happens if R≥SR \ge S?

A3. Thank you for the suggestion. Subjects and relations play symmetric roles in our setting: both are sampled uniformly and independently, placed at random positions, and determine the answer through the bijection a∗(s,r)a_*(s,r), so relabeling relations as subjects gives the same task with (S,R)(S,R) swapped. Hence the case R≥SR\ge S follows from our results by exchanging the roles of SS and RR (and of TSδT_S^\delta and TRδT_R^\delta): e.g., GF then learns subjects first with TRδ/TSδ=Θ~(R/S)T_R^\delta/T_S^\delta=\tilde\Theta(\sqrt{R/S}), while Spectral GF still has a ratio of Θ~(1)\tilde\Theta(1).


Q4. The paper should explicitly explain how the errors decaying as 1/(Tlog⁡T)1/(T\log T) follow from Theorem 3.4.

A4. Thank you; we have added Corollary 3.11 (Section 3, in blue; proof in Appendix E.4). Treat SS and RR as constants and write tt for training time. Since the margin qSq_S increases, the subject error at time tt is the δ\delta with TSδ=tT_S^\delta=t. As δ→0\delta\to0, the second term of Theorem 3.4 dominates, so t=TSδ≍δ−1log⁡δ−1⟺δ−1≍tlog⁡t⟺δ≍1tlog⁡t,t=T_S^\delta\asymp\frac{\delta^{-1}}{\log\delta^{-1}}\quad\Longleftrightarrow\quad\delta^{-1}\asymp t\log t\quad\Longleftrightarrow\quad\delta\asymp\frac{1}{t\log t}, where we used log⁡δ−1≍log⁡t\log\delta^{-1}\asymp\log t; the same holds for the relation error. For comparison, Theorem 3.8 gives t≍log⁡δ−1t\asymp\sqrt{\log\delta^{-1}} for Spectral GF, i.e., δ=exp⁡(−Θ(t2))\delta=\exp(-\Theta(t^2)), as also stated in the corollary.


References

[1] Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding factual recall in transformers via associative memories. 2025.

Thanks for the detailed response. My concerns about Q1, Q3, and Q4 are resolved. For Q1, my suggestion is that the author should not frame it as EoS. Instead, it is much better to frame it as the system template of an LLM, which always appears before the response and is independent of the question. This makes this formulation more aligned with a practical LLM.

For Q2, according to my understanding, it is a drawback of the gradient flow formulation adopted in the paper. This formulation makes it hard to describe the following important practical fact: at some given step, Muon has a lower loss than Adam.

Overall, I keep my positive rating of this work and maintain my score.

Thank you for the follow-up and for keeping your positive rating!

Q1. Thank you for the suggestion. We have added this discussion after the data model (Section 2.1, in blue).

Q2. Thank you for clarifying your concern. We agree that the gradient-flow formulation does not capture the upper limit on the learning rate imposed by the stability of discrete-time updates with stochastic gradients, which is what makes the loss at a given step comparable across optimizers in practice. Analyzing this limit requires a discrete-time and stochastic analysis, which we leave for future work.

Nevertheless, the learning-time ratio is informative in this setting. Heuristically, the peak learning rate is set so that the learning of the faster-moving component, the relation-dependent one, remains stable. With tuned learning rates, the relation learning times of different optimizers should therefore be similar, and the ratio TSδ/TRδT_S^\delta/T_R^\delta then determines the subject learning time, and hence the loss at a given step. A smaller ratio, as for Muon, thus means that the subject-dependent component, and with it the whole fact, is learned earlier.

Our experiment in Appendix B.3 supports this picture. With learning rates tuned separately for each optimizer, the relation error becomes small early in training for SGD, Adam, and Muon alike, whereas Muon reduces the subject error several times earlier than SGD and Adam and has the lowest overall error at the end of training.

Reviewer_WSccOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes how Muon’s spectral orthogonalization balances subject and relation learning in a simplified factual-recall model. It derives learning- time bounds, examines embedding dependence, and supports the analysis with simulations and small-scale training experiments.

Critical strengths

• I find the quantitative explanation of balanced feature learning useful, particularly its separation from the slowdown at high confidence.

• The embedding-dependent learning-order examples are informative, and the stochastic training experiments help connect the analysis to practical optimization.

Critical weaknesses

• The target-dependent initialization already makes the correct answer the highest-scoring one. The theoretical “accuracy” therefore measures probability concentration rather than the emergence of correct greedy predictions.

• The practical validation remains limited, with insufficient detail about initialization, learning-rate selection, and the candidate-probability metrics.

Questions for authors
  1. Could you clarify the distinction between correct-answer probability and greedy accuracy, including the metrics in Figure 8?

  2. What initialization and learning-rate selection procedures were used in the practical experiments?

  3. Could you correct the inequality directions in Lemma E.2’s proof and the apparently swapped subject/relation logarithms in Theorem E.3?

3 comments

Many thanks for your positive feedback!

Q1. The target-dependent initialization already makes the correct answer the highest-scoring one, so the theoretical "accuracy" measures probability concentration rather than the emergence of correct greedy predictions. Could you clarify the distinction between correct-answer probability and greedy accuracy, including the (candidate-probability) metrics in Figure 8?

A1. Thank you for the question. We agree that, with our initialization, the correct answer already has the highest score, so greedy (arg-max) accuracy is trivially perfect on the manifold and is not informative. However, a language model generates its answer by sampling from the predicted next-token distribution. Greedy accuracy only describes temperature-zero decoding and ignores how much probability mass is placed on wrong answers. This mass is exactly what characterizes the feature-separation phase: the model already places most of its mass on answers of the queried relation, but spreads it over the answers of wrong subjects, so sampling produces plausible yet factually wrong answers. The correct-answer probability is also the quantity that the cross-entropy objective optimizes.

The metrics in Figure 8 follow the same convention: they are probabilities, not greedy accuracies. For each question about a person and an attribute, we take 64 candidate answers:

  • the correct answer;
  • 31 answers of the same attribute that belong to other people;
  • 32 answers of other attributes.

We normalize the model probabilities of these complete answers over the 64 candidates. Let pfactp_{\rm fact} be the probability of the correct answer and prelp_{\rm rel} the total probability of the 32 answers of the queried attribute. The three curves are then:

  • overall error (solid): 1−pfact1-p_{\rm fact};
  • relation error (dotted): 1−prel1-p_{\rm rel}, the probability of an answer of a wrong attribute;
  • subject error (dashed): 1−pfact/prel1-p_{\rm fact}/p_{\rm rel}, the probability of a wrong person's answer given an answer of the queried attribute.

Thus pfactp_{\rm fact} is the product of the subject and relation components, as in Proposition 3.1. These two errors are the empirical counterparts of 1−(1+e−qS)−11-(1+e^{-q_S})^{-1} and 1−(1+e−qR)−11-(1+e^{-q_R})^{-1} there.

We have stated these definitions in the caption of Figure 8 and in Appendix B.3 (in blue).

Q2. The practical validation remains limited, with insufficient detail about initialization and learning-rate selection. What initialization and learning-rate selection procedures were used in the practical experiments?

A2. Thank you for the question. Appendices B.2 and B.3 now describe both procedures in full (in blue).

Single-layer attention (B.2).

  • Initialization. The model starts on M\mathcal M with α0=β0=0.01\alpha_0=\beta_0=0.01. Every entry ϕ(u)⊤Wϕ(v)\boldsymbol\phi(u)^\top\mathbf W\boldsymbol\phi(v) of WOV\mathbf W_{OV} and WKQ\mathbf W_{KQ} is then perturbed by Gaussian noise with standard deviation 10−410^{-4}, so training leaves M\mathcal M.
  • Learning rates. Each optimizer uses one constant learning rate for all SS and RR: 0.01 for SGD, 0.004 for Muon and 3e-4 for Adam. We do not tune them, because we measure learning times by the cumulative learning rate ηt\eta t. A constant learning rate then only rescales time (Appendix C), and it cancels in the ratio TSδ/TRδT_S^\delta/T_R^\delta, which we compute within each run.
  • Robustness. Across 3 seeds, the ratios of SGD, Muon and Adam with one-hot embeddings differ by at most 2.4% of their mean.

Transformer on synthetic biographies (B.3).

  • Initialization. The Pythia-31M model is trained from scratch, with the Gaussian initialization of standard deviation 0.02 from the Pythia configuration.
  • Learning rates. We swept five learning rates per optimizer, spaced by a factor of about 3, with seed 0. We selected the one with the lowest overall error on held-out phrasings, averaged over steps 40k–50k: 1e-3 for SGD (grid 3e-4 to 3e-2), 1e-3 for Adam (3e-5 to 3e-3) and 1.2e-3 for Muon (1.2e-4 to 1.2e-2). Every selected value lies inside its grid, and Table 3 lists all scores.
  • Seeds. We then trained each optimizer at its selected learning rate from three further initialization seeds. Within each seed, the three optimizers start from the same weights and see the same data.
  • Shared settings. The embeddings, output head, LayerNorms and biases use a shared Adam with learning rate 3e-4 in all runs. All runs use 100 warmup steps followed by cosine decay, with no weight decay and no gradient clipping.

With tuned learning rates, Muon brings the held-out subject error below 0.5 by step 5k in every seed, against 18k–19k steps for SGD and 17k–28k for Adam. Table 4 reports the errors with their spread across seeds.


Q3. Could you correct the inequality directions in Lemma E.2's proof and the apparently swapped subject/relation logarithms in Theorem E.3?

A3. Thank you for the careful proofreading. We have corrected these typos in the revision (in blue): in the proof of Lemma E.2, both inequalities in the bound on dβS/dt\mathrm d\beta_S/\mathrm dt now read ≥\ge (giving the lower bound 1/5S1/\sqrt{5S}), and Theorem E.3 now has (S−1)(S-1) in TSδT_S^\delta and (R−1)(R-1) in TRδT_R^\delta. We also fixed αS(TS0.5)\alpha_S(T_S^{0.5}) to βS(TS0.5)\beta_S(T_S^{0.5}) at the end of the proof of Lemma E.2. None of them affects the conclusions.

Dear Reviewer WScc,

As the discussion period ends tomorrow, we would greatly appreciate it if you could take a look at our rebuttal and let us know whether it addresses your concerns. A brief summary of our response:

  • Correct-answer probability vs. greedy accuracy: we explained why the correct-answer probability is the relevant measure for sampling-based generation and the feature-separation phase, and stated the definitions of the error metrics in Figure 8.
  • Initialization and learning-rate selection: we described both procedures in Appendix B, and reran the transformer experiment with a learning-rate sweep for each optimizer and three seeds, reporting the spread across seeds.
  • Typos: we fixed the inequality directions in the proof of Lemma E.2 and the logarithms in Theorem E.3.

We hope these address your concerns, and we would be happy to answer any further questions before the discussion period ends.

Reviewer_uYKzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks why the Muon optimizer, which replaces each matrix-valued update by its orthogonalized version (keeping the singular vectors of the update and setting all of its singular values to one), might help models learn facts. It studies the factual-recall task of Nichani et al. (2025), in which each fact maps a subject and a relation (for example, a person and one of their attributes) to an answer, learned by a single-layer linear attention model. Gradient descent, Muon and Adam are modeled by idealized continuous-time versions: gradient flow (GF), spectral GF and sign GF. The central quantity is how long the model takes to learn the subject-dependent part of the answer compared with the relation-dependent part. With SS subjects and RR relations, where subjects outnumber relations, the paper proves that, up to logarithmic factors, the ratio of these two learning times is of order S/R\sqrt{S/R} under GF, so GF learns relations first and the gap widens as S/RS/R grows, while under spectral GF the ratio is of order one.

It also shows that spectral GF avoids most of the slowdown that GF suffers as the target accuracy approaches 100%. Finally, it proves that GF and spectral GF are unaffected by rotations of the token embeddings, whereas sign GF is not: depending on the embeddings, an Adam-like method can learn the two parts at similar speeds, at very different speeds, or even in the reverse order. Experiments in the appendix compare SGD, Muon and AdamW on the same task and on a 35M-parameter language model trained on synthetic biographies.

Critical strengths

Some of the key strengths of the paper are as follows:

  • All three idealized optimizers keep the parameters on a small set of matrices described by a few scalars (an invariant manifold), on which the prediction splits exactly into a subject part and a relation part (Props. 3.1, 3.3 and 3.7, Lemma D.1). Both learning-time results come with matching upper and lower bounds (Thms. 3.4 and 3.8), so the gap between GF and spectral GF is a genuine separation within the model rather than an artifact of loose bounds. The factorization also gives a transparent picture of the "hallucination" phase, in which the model already produces an answer of the right type but not yet the one belonging to the queried subject (L256-262). As far as I checked, the proofs in App. E are careful, and the issues I noticed are typos that do not change the conclusions.

  • The sign GF constructions (Table 2, Fig. 1a) show that an Adam-like method can shorten, lengthen, or even reverse the learning order depending only on the basis of the embeddings, which is a concrete illustration of Adam's lack of rotation invariance. In the small experiments (Figs. 5-6), the ratio of learning times for SGD grows with S/RS/R while that for Muon stays close to 1, in line with the theory.

Critical weaknesses

The paper seems to have some major concerns as follows:

  • Orthogonalization is not separated from normalization, although the subtitle asks about the role of orthogonalization. On the invariant manifold, the gradient of the merged key-query matrix WKQW_{KQ} has rank one, so orthogonalizing it amounts to normalizing it (L364-366). The argument that spectral GF escapes the softmax saturation (L387-393, Remark 3.9) only uses the fact that the updates have unit size, and the conclusion itself attributes this to "the spectral normalization of Muon" (L480-481). Consistently, sign GF and AdamW also show only a logarithmic dependence on 1/δ1/\delta, where δ\delta is the target error (L473-474, L854-855), and Table 2 shows that sign GF learns the two parts at comparable speeds (a ratio of order one up to logarithmic factors) for five of the eight embedding choices, including the all-one-hot case. Yet the paper attributes both of its main effects to orthogonalization (L91-92, L403-404), including the reweighting of the gradient of the merged output-value matrix WOVW_{OV} (L417-419), without comparing against gradient descent with normalized step sizes. Closely related work does include such a control: Vasudeva et al. (2026) report that their gap persists against GD with normalized step sizes. A per-matrix normalized GF baseline (normalizing by the spectral or the Frobenius norm) on the reduced equations would be cheap and would make the attribution precise.
  • The headline claims go beyond what is proved and shown. The abstract states that, for fixed SS and RR, the errors decay as 1/(Tlog⁡T)1/(T\log T) in training time TT under GF but as exp⁡(−poly(T))\exp(-\mathrm{poly}(T)) under spectral GF, where poly(T)\mathrm{poly}(T) is a polynomial in TT (L24-25), but these rates are not stated or proved anywhere in the body. Like Table 1 (L115-119), they compare absolute times across optimizers whose time units are not comparable, while App. C (L937-939) itself argues that only the ratio of learning times is intrinsic; in Table 1, sign GF with one-hot embeddings even has learning times that do not grow with SS and RR (up to logarithmic factors), faster than spectral GF. The title claim is not established either. The main text contains no optimizer experiments and never refers to Appendix B, each optimizer uses a single learning rate with no reported tuning (L805-806), B.2 reports averages over 3 seeds without spread (L809), and in the one realistic experiment all three optimizers "reach similar accuracies after step 30000" (L917-918). In Fig. 8, Muon's relation error also appears to end well above those of SGD and AdamW and close to its own subject error, which does not match L913-915. "Faster" or "more balanced" would describe the evidence better than "better".

Additional, less decisive concerns:

  • Novelty over the closest cited work is modest. Wang et al. (2025) already show that Muon balances learning in associative memories while sign-based updates depend on the embedding (acknowledged at L471-472), Vasudeva et al. (2026) show that spectral GD learns all principal components at equal rates, and Li et al. (2026) report an exponential speedup of Muon over GD in a noiseless associative-memory model, which parallels the second main claim. Related work appears only in Appendix A, which the main text never references.
  • The idealized flows change discontinuously with the parameters, and their well-posedness is not discussed: invariance of the manifold gives a solution on it, not uniqueness, and sign GF also needs the unstated convention sign(0)=0\mathrm{sign}(0)=0. Calling them continuous-time limits of Muon and Adam (L16-17) also drops momentum, the Newton-Schulz approximation used by Muon, and Adam's numerical-stability constant ϵ\epsilon.
  • Scope: the manifold is built from the answer map itself (Eq. 3.3), so starting on it with positive coefficients already aligns WOVW_{OV} with the correct answer directions, and the spectral GF analysis also needs the subject and relation coefficients to start at the same value (L376-380). Thm. 3.4 is stated for S,R≥2S,R\geq 2, but its formula is undefined at S=2S=2.
  • Proof typos (conclusions unaffected): L1471 says qS≥1q_S\geq 1, where qSq_S is the subject margin, while L1466 has qS≤0q_S\leq 0; Thm. E.3 (L1632, L1648) has (R−1)(R-1) where the proof (L1638) uses (S−1)(S-1); and since two distinct positions are drawn from [T−1][T-1], where here TT is the context length (L139), L135 should require T≥3T\geq 3.
Questions for authors

I have the following questions regarding validity and clarification; depending on the answers, I am willing to change the scores:

  1. (Additional evidence; could change my score.) What do per-matrix normalized GF controls (normalizing each matrix update by its spectral or Frobenius norm) give on the manifold, both for the ratio of learning times and for the dependence on the target error δ\delta? Which parts of the two main claims remain specific to orthogonalization, and how would L383-393, L417-419 and the title change?

  2. (Clarification; could change my score.) What is new relative to Wang et al. (2025), Vasudeva et al. (2026) and Li et al. (2026), in particular the speedup of Muon over GD reported by Li et al.?

  3. (Clarification; could change my rating.) Were learning rates tuned separately for each optimizer? Please report sweeps and multi-seed results with spread for B.2 and B.3, and clarify how the 0.2 multiplier (L806-807) scales the Muon update relative to AdamW.

  4. (Clarification.) Can the decay rates in the abstract be stated as a corollary with explicit exponents, and under which normalization of time is it meaningful to compare them across optimizers?

  5. (Clarification.) How do the predictions change when training starts from a small random initialization off the manifold? Which embedding dimension dd, context length TT, number of noise tokens ∣N∣|\mathcal N| and initialization does B.2 use?

  6. (Clarification.) Why do the AdamW learning-time ratios for Hadamard embeddings change with SS and RR (Figs. 5e, 6d) when Table 2 predicts no such dependence, and why do some AdamW curves stop early?

10 comments

Q1. Orthogonalization is not separated from normalization, although the subtitle asks about the role of orthogonalization. On the invariant manifold the gradient of WKQW_{KQ} has rank one, so orthogonalizing it amounts to normalizing it (L364–366), and the escape from softmax saturation (L387–393, Remark 3.9) only uses that updates have unit size. Sign GF / AdamW also show only a log⁡(1/δ)\log(1/\delta) dependence (L473–474, L854–855), and Table 2 gives a ratio of order one for five of the eight embeddings. Yet both main effects, including the reweighting of the WOVW_{OV} gradient (L417–419), are attributed to orthogonalization. What does a per-matrix normalized GF baseline (normalizing each update by its spectral or Frobenius norm, as in [1]) give on the manifold, for both the learning-time ratio and the dependence on δ\delta? Which parts of the two main claims remain specific to orthogonalization, and how would L383–393, L417–419 and the title change?

A1. Thank you for the careful reading. There may be a misunderstanding of our terminology: we do not mean to distinguish orthogonalization from normalization. In fact, orthogonalization cannot be separated from normalization: spectral orthogonalization msign(G)=UV⊤\mathrm{msign}(\mathbf G)=\mathbf U\mathbf V^\top sets every singular value of the update to one, i.e., it normalizes the gradient separately in each singular subspace. It therefore has two components, scale control (the update has unit size) and per-subspace normalization (all singular directions move at the same rate), and the two benefits of Muon come from these two components. The weakened softmax saturation (L387–394) comes from scale control. The balanced feature learning (L383–386) comes from per-subspace normalization: on the manifold, the subject and relation components of ∇WOVL\nabla_{\mathbf W_{OV}}L lie in orthogonal singular subspaces, and msign\mathrm{msign} sets both of their coefficients to one. Both benefits are thus intrinsic to spectral orthogonalization, which provides them at once, in contrast to the entry-wise normalization of Sign GF (Adam), whose behavior depends on the embeddings (Section 4). We have added this discussion before Theorem 3.8 (in blue).

Normalization by the gradient norm. The per-matrix normalization G/∥G∥\mathbf G/\|\mathbf G\| suggested by the reviewer only provides scale control. It coincides with msign\mathrm{msign} for the rank-one gradient of WKQ\mathbf W_{KQ}, but it rescales the subject and relation components of ∇WOVL\nabla_{\mathbf W_{OV}}L by the same factor and keeps the imbalance of GF. A full analysis of this dynamics is challenging: both WOV\mathbf W_{OV} and WKQ\mathbf W_{KQ} then follow dynamics like those of WKQ\mathbf W_{KQ} in Spectral GF, with the subject and relation parameters coupled through a normalizing factor. The initial phase with small weights is tractable. There the softmax is nearly uniform, and with α0=β0=:x0\alpha_0=\beta_0=:x_0, normalizing the gradient of each matrix by its Frobenius norm gives αS=βS=:xS\alpha_S=\beta_S=:x_S and αR=βR=:xR\alpha_R=\beta_R=:x_R with x˙S≈xSS (xS2+xR2)1/2,x˙R≈xRR (xS2+xR2)1/2.\dot x_S\approx\frac{x_S}{\sqrt S\,(x_S^2+x_R^2)^{1/2}},\qquad\dot x_R\approx\frac{x_R}{\sqrt R\,(x_S^2+x_R^2)^{1/2}}. Hence (Sx˙S)2+(Rx˙R)2=1(\sqrt S\dot x_S)^2+(\sqrt R\dot x_R)^2=1 as in Lemma E.1, and xS=x0(xR/x0)R/Sx_S=x_0(x_R/x_0)^{\sqrt{R/S}}, the same trajectory as GF. The relation parameters grow linearly (xR≃x0+t/Rx_R\simeq x_0+t/\sqrt R), but the subject parameters grow only as tR/St^{\sqrt{R/S}}, whereas under Spectral GF both grow linearly (Lemmas E.1 and E.2). For example, with S=64S=64, R=16R=16, and x0=0.01x_0=0.01, when the relation component reaches accuracy 0.50.5, xSx_S is only about 6% of the value needed for the subject component to reach 0.50.5. The subject is thus learned mainly after the relation component saturates. Normalization by the gradient norm keeps the feature separation, although its scale control weakens the slowdown from softmax saturation.

We also simulated the reduced dynamics with WOV\mathbf W_{OV} and WKQ\mathbf W_{KQ} normalized separately (α0=β0=0.01\alpha_0=\beta_0=0.01). The learning-time ratio grows with S/RS/R (Table A), whereas that of Spectral GF stays close to 11 (Figure 2), and the learning times grow only logarithmically in δ−1\delta^{-1} (Table B), unlike the Θ~(δ−1)\tilde\Theta(\delta^{-1}) of GF. Normalizing the whole gradient jointly gives the same ratios, with all times larger by a factor of 2\sqrt2.

Table A. Learning-time ratio TSδ/TRδT_S^\delta/T_R^\delta of normalized GF (separate normalization) at δ=0.1\delta=0.1 (rows: SS; columns: RR).

SS \ RR481632641282565121024
410.7180.4100.3050.2470.2040.1690.1410.118
81.39310.6560.4040.3190.2640.2210.1850.156
162.4411.52410.6080.4020.3260.2730.2310.195
323.2742.4731.64510.5740.4010.3310.2800.237
644.0503.1362.4861.74210.5510.4000.3340.284
1284.9063.7913.0652.4931.81610.5340.4000.337
2565.9024.5303.6583.0222.4981.87210.5220.400
5127.0865.3984.3343.5762.9942.5011.91510.513
10248.5066.4335.1324.2133.5202.9722.5031.9501

Table B. Learning times of normalized GF (separate normalization) versus the target error δ\delta, (S,R)=(64,16)(S,R)=(64,16).

δ\deltaTSδT_S^\deltaTRδT_R^\deltaTSδ/TRδT_S^\delta/T_R^\delta
0.554.6018.432.962
0.162.3225.072.486
0.063163.9126.872.378
0.039865.4029.732.200
0.025166.8535.991.858
0.015868.2752.151.309
10−210^{-2}69.6863.121.104
10−310^{-3}76.5674.361.030
10−410^{-4}83.0181.211.022
10−510^{-5}88.9587.291.019
10−610^{-6}94.4692.901.017

Q2. The headline claims go beyond what is proved and shown. The abstract states that, for fixed SS and RR, the errors decay as 1/(Tlog⁡T)1/(T\log T) under GF but as exp⁡(−poly(T))\exp(-\mathrm{poly}(T)) under spectral GF (L24–25), but these rates are not stated or proved anywhere in the body. Can they be stated as a corollary with explicit exponents?

A2. Thank you. We have added Corollary 3.11 (Section 3, in blue; proof in Appendix E.4), which inverts Theorems 3.4 and 3.8 to express the errors ϵS(t),ϵR(t)\epsilon_S(t),\epsilon_R(t) of the subject- and relation-dependent components as functions of the training time tt. Treating SS, RR and the initialization as constants, GF: t≍δ−1log⁡δ−1  ⟺  ϵ(t)=Θ(1tlog⁡t),Spectral GF: t≍log⁡δ−1  ⟺  ϵ(t)=exp⁡(−Θ(t2)),\text{GF: } t\asymp\frac{\delta^{-1}}{\log\delta^{-1}}\iff\epsilon(t)=\Theta\Big(\frac{1}{t\log t}\Big),\qquad\text{Spectral GF: } t\asymp\sqrt{\log\delta^{-1}}\iff\epsilon(t)=\exp\big(-\Theta(t^2)\big), so poly(T)=T2\mathrm{poly}(T)=T^2 in the abstract. The corollary compares the form of the decay (polynomial vs. exponential) rather than absolute times, and this form is unchanged by a constant learning rate, which only rescales tt (Appendix C).


Q3. Absolute learning times are compared across optimizers whose time units are not comparable (abstract; Table 1, L115–119), although App. C (L937–939) argues that only the ratio of learning times is intrinsic. In Table 1, sign GF with one-hot embeddings even has learning times that do not grow with SS and RR (up to log factors), i.e., faster than spectral GF. Under which normalization of time is a cross-optimizer comparison meaningful?

A3. Thank you for raising this point. Since a constant learning rate only rescales time (Appendix C), learning times are defined only up to the choice of learning rate, and our basic rule is that the learning rate may be rescaled by factors depending on SS and RR, but not on the target error δ\delta. This is standard in hyperparameter tuning: the learning rate may be chosen according to the model and the data, but it is fixed before training and cannot depend on the accuracy one eventually aims for.

Under this rule, the dependence of absolute learning times on SS and RR is not comparable across optimizers. Therefore, the short absolute learning times of Sign GF with one-hot embeddings do not imply that Adam is faster than Muon. This is why we study feature separation through the ratio TSδ/TRδT_S^\delta/T_R^\delta: any SS- and RR-dependent rescaling of the learning rate cancels in the ratio, which therefore measures how an optimizer allocates progress between the subject- and relation-dependent components.

In contrast, since the learning rate cannot depend on δ\delta, the dependence of the absolute learning times on δ\delta is intrinsic, and this is the only place where we compare absolute learning times. There, GF requires time Θ~(δ−1)\tilde\Theta(\delta^{-1}) due to softmax saturation, whereas Spectral GF requires time only polylogarithmic in δ−1\delta^{-1} (equivalently, the error decays as Θ(1/(tlog⁡t))\Theta(1/(t\log t)) vs. exp⁡(−Θ(t2))\exp(-\Theta(t^2)); Corollary 3.11). In short, comparisons across optimizers are meaningful for quantities invariant under δ\delta-independent rescaling of time, namely the ratio TSδ/TRδT_S^\delta/T_R^\delta and the growth of the learning times as δ→0\delta\to0.

Comments on the paper

1 comment

We thank all reviewers for their careful reading and constructive comments. We have uploaded a revised manuscript, with all changes marked in blue. The main changes are:

Theory

  • Corollary 3.11 states the error decay rates in training time under GF and Spectral GF, with a proof in Appendix E.4.
  • Before Theorem 3.8, we discuss the roles of scale control and per-subspace normalization in spectral orthogonalization.
  • After Assumption 3.2, we discuss the well-posedness of the reduced dynamics, and Section 2.3 now states the convention sign(0)=0\mathrm{sign}(0)=0.
  • We have fixed the typos pointed out by the reviewers in Theorem 3.4, the data model, Theorem E.3, and the proofs in Appendix E.

Experiments

  • Appendix B.2 now gives the full configuration of the single-layer attention experiments, with updated figures. It also discusses why the learning-time ratios of Adam depend mildly on SS and RR. Since no weight decay is used, we now refer to AdamW as Adam.
  • We have rerun the transformer experiment in Appendix B.3 with a learning-rate sweep for each optimizer and three random seeds. The section now gives the full setup and the definitions of the error metrics, and it reports the errors with their spread across seeds.
  • The conclusion now summarizes the experimental results and refers to Appendix B.

Related work

  • The introduction now points to the related work in Appendix A, and Appendix A has been extended.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.