- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a novel quantity to study the behavior and advantage of Muon: the learning-time ratio. This analysis is novel in the existing literature. In addition, the authors derive the full characterization of the Muon and GD learning dynamics, which is of independent interest.
Critical strengths
-
The main novelty of this work is the full characterization of the learning dynamics and the newly proposed quantity: the learning-time ratio.
-
The analysis of Adam under the unitary transformation is also interesting and intuitive.
Critical weaknesses
-
It is not clear why the data model assumes that the last token in the prompt is EoS, which is not aligned with the common practice of LLMs. It will be better to explain whether the proof still holds if the last token is not EoS. I think will not have the form in (3.2) under this setting.
-
It is not clear why we need to consider the ratio between and . Since it is a relative quantity, it does not quantify the `` duration of the feature-separation phase’’ in Line 263. Instead, it just measures the relative size of this gap. However, the practical performance of Muon should result from its absolute value.
-
The paper mainly focuses on the setting . It is better to discuss what happens if .
-
In addition, the paper should explicitly explain the errors decaying as 1/(T log T) from Theorem 3.4.
Questions for authors
-
Please explain what results can be extended to the setting where the last token is not EoS.
-
Please explain why we need to consider the ratio between and .
4 comments
Q3. The paper mainly focuses on the setting . What happens if ?
A3. Thank you for the suggestion. Subjects and relations play symmetric roles in our setting: both are sampled uniformly and independently, placed at random positions, and determine the answer through the bijection , so relabeling relations as subjects gives the same task with swapped. Hence the case follows from our results by exchanging the roles of and (and of and ): e.g., GF then learns subjects first with , while Spectral GF still has a ratio of .
Q4. The paper should explicitly explain how the errors decaying as follow from Theorem 3.4.
A4. Thank you; we have added Corollary 3.11 (Section 3, in blue; proof in Appendix E.4). Treat and as constants and write for training time. Since the margin increases, the subject error at time is the with . As , the second term of Theorem 3.4 dominates, so where we used ; the same holds for the relation error. For comparison, Theorem 3.8 gives for Spectral GF, i.e., , as also stated in the corollary.
References
[1] Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding factual recall in transformers via associative memories. 2025.
Thanks for the detailed response. My concerns about Q1, Q3, and Q4 are resolved. For Q1, my suggestion is that the author should not frame it as EoS. Instead, it is much better to frame it as the system template of an LLM, which always appears before the response and is independent of the question. This makes this formulation more aligned with a practical LLM.
For Q2, according to my understanding, it is a drawback of the gradient flow formulation adopted in the paper. This formulation makes it hard to describe the following important practical fact: at some given step, Muon has a lower loss than Adam.
Overall, I keep my positive rating of this work and maintain my score.
Thank you for the follow-up and for keeping your positive rating!
Q1. Thank you for the suggestion. We have added this discussion after the data model (Section 2.1, in blue).
Q2. Thank you for clarifying your concern. We agree that the gradient-flow formulation does not capture the upper limit on the learning rate imposed by the stability of discrete-time updates with stochastic gradients, which is what makes the loss at a given step comparable across optimizers in practice. Analyzing this limit requires a discrete-time and stochastic analysis, which we leave for future work.
Nevertheless, the learning-time ratio is informative in this setting. Heuristically, the peak learning rate is set so that the learning of the faster-moving component, the relation-dependent one, remains stable. With tuned learning rates, the relation learning times of different optimizers should therefore be similar, and the ratio then determines the subject learning time, and hence the loss at a given step. A smaller ratio, as for Muon, thus means that the subject-dependent component, and with it the whole fact, is learned earlier.
Our experiment in Appendix B.3 supports this picture. With learning rates tuned separately for each optimizer, the relation error becomes small early in training for SGD, Adam, and Muon alike, whereas Muon reduces the subject error several times earlier than SGD and Adam and has the lowest overall error at the end of training.
Many thanks for your positive review and thoughtful feedback!
Q1. The data model assumes that the last token in the prompt is EoS, which is not aligned with the common practice of LLMs. Does the proof still hold if the last token zT is not EoS? The reviewer thinks WKQ will not have the form in (3.2) in that setting. Which results extend?
A1. Thank you for the question. Following [1], EOS denotes the token at which the answer is predicted, and it serves as the query that retrieves the subject and the relation from the context. Our results extend whenever the last token does not depend on the fact. For example, in a prompt format with a fixed ending word, such as "is" in "The capital of France is" or "Answer:" in a question–answer format, all results hold verbatim with ϕ(EOS) in (3.2) replaced by the embedding of that word. More generally, if the last token is drawn from a set Q of template tokens independently of the fact, WKQ becomes (βS∑sϕ(s)+βR∑rϕ(r))(∑τ∈Qϕ(τ))⊤, which is still rank one, and the reduced dynamics coincide with ours after rescaling the initialization and the time (by ∣Q∣ for GF and ∣Q∣1/4 for Spectral GF), so all results extend. When the last token depends on the subject or the relation (e.g., relation-specific template words such as "born in"), we agree that WKQ no longer has the form (3.2), since it may need a separate query column for each value of that feature, and the analysis of this general case is substantially more difficult. We leave it for future work.
Q2. Why consider the ratio between TS and TR? As a relative quantity it does not quantify the "duration of the feature-separation phase" (L263); it only measures the relative size of the gap, whereas the practical performance of Muon should come from the absolute value.
A2. Thank you for the question.
Absolute learning times are set by the learning rate. With learning rate γ, each of our dynamics reads Θ˙=−γF(Θ) with F=∇L, msign(∇L) or sign(∇L), and the change of time τ=γt gives TSδ(γ)=TSδ(1)/γ and TRδ(γ)=TRδ(1)/γ. Either time can thus be made arbitrarily small or large by the choice of γ (similarly for any schedule, via τ=∫0tγ(s)ds; Appendix C), so comparing absolute times across optimizers compares their learning rates rather than their update rules.
The ratio removes the learning rate. The common factor 1/γ cancels in TSδ/TRδ, so the ratio depends only on the update rule and the task. Under GF, TSδ/TRδ=Θ~(S/R) for every learning rate: a larger learning rate accelerates subjects and relations by the same factor, so no tuning of GD can shorten the feature-separation phase relative to relation learning. Only changing the update direction, as spectral orthogonalization does, reduces the ratio to Θ~(1).