acceptodds
Under review as a conference paper at ICLR 2027

After the Fact: Coupled Updates Shape the Implicit Bias of Muon

Abstract

When Muon updates one matrix layer, its motion changes gradients seen by surrounding parameters, so learning depends on how those signals are transformed downstream. We study this interaction through covariance, starting from Qwen3-0.6B Muon trajectories where similar NS5 update magnitudes can hide very different local responses, including negative elasticities. Persistent motion then turns these responses into curvature gradients. For stable cycles, we characterize when coupled spectral updates admit scalar curvature objectives, identify power compatibility, and show why cube-root updates remain aligned under decay. Under noisy training, Gaussian averaging can restore stability even when instantaneous responses are negative, while finite covariance fluctuations can change whether a scalar objective exists. Together, these results connect spectral response, persistent matrix motion, and optimizer coupling within one theory of curvature dynamics.

Primary area:
optimization
Keywords:
training dynamics, matrix optimizers, Muon, implicit bias, spectral optimization

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_YZRsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how coupled spectral optimizers shape implicit curvature dynamics in a quadratic fast–slow matrix model. For persistent oscillations, it characterizes which fast–slow update pairs induce a scalar curvature objective, highlighting power-update compatibility and the special role of cube-root updates under decay. In a Gaussian small-step regime, it shows that the fast matrix has a positive averaged restoring response despite negative instantaneous spectral responses, and that finite covariance fluctuations can destroy scalar-potential representations that normalizing after averaging would preserve. Measurements from a Qwen3-0.6B trajectory motivate the analysis.

Critical strengths

The paper gives a precise account of when coupled optimizers follow a single implicit objective and when they do not. It classifies which pairs of spectral updates make the slow curvature dynamics equivalent to minimizing a scalar objective. Power updates are always compatible, and the cube-root update is uniquely robust once weight decay is added. Moreover, the paper shows that a stable, well-behaved fast equilibrium can still produce slow dynamics that no scalar objective describes (Theorem K.2). Together, these results set clear limits on the common practice of interpreting an optimizer’s implicit bias through a single objective.

Critical weaknesses
  1. The results are not connected to how curvature actually evolves in a network. The model treats curvature as a matrix that is trained directly. In a real network, curvature is produced by other parameters, and the paper does not derive what its results imply for them. This substantially limits the interpretation of the results as an explanation of coupled Muon training.

  2. The paper does not show that lacking a scalar objective changes training. The negative results establish that the averaged curvature force cannot be expressed as a gradient up to a positive scalar multiplier in the specified geometry, but the paper does not demonstrate how this obstruction affects curvature trajectories or eventual solution selection. The Gaussian effect is proven only for sufficiently wide matrices, and it shrinks in proportion to 1/width with a small constant. The NS5 example is a single hand-picked 3×3 case. The experiments only verify derivative formulas, on Qwen matrices far larger than the analyzed models. A controlled comparison of the two normalization orders, tracking their resulting curvature trajectories, would help establish whether the mathematical distinction produces an appreciable dynamical effect.

Questions for authors

Under what conditions on a parameterization C(y)C(y) and an optimizer acting on yy do the induced curvature dynamics satisfy the model analyzed in the paper? Does scalar compatibility or incompatibility in the independent CC coordinates transfer under those conditions? A concrete derivation for a coupled parameter model would materially strengthen the interpretation.

Reviewer_NaG3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the implicit bias of Muon through the coupled dynamics of different parameter groups. The authors develop a theoretical framework to understand how matrix updates influence the gradients received by other parameters and shape the resulting curvature dynamics.

Overall, I appreciate the theoretical contributions of this paper. However, I have concerns about the gap between the theoretical model and practical neural network training, the restrictive assumptions, and the lack of experimental validation.

Critical strengths
  1. Theoretical value. The paper provides an interesting theoretical perspective on the implicit bias of Muon and the interactions between different parameter groups. I believe this is a meaningful research direction.

  2. Reasonably clear presentation. I was able to follow most of the paper and understand its main ideas, although some technical details and derivations were difficult for me to fully understand.

Critical weaknesses

1. The gap between the theoretical model and practical Transformer training.

The paper considers a simplified quadratic loss:

ℓ(C,E)=12tr⁡(E⊤CE)+⟨Ξ,E⟩F,\ell(C,E)=\frac12\operatorname{tr}(E^\top CE)+\langle\Xi,E\rangle_F,

where CC is a positive definite curvature matrix.

However, the Hessian of a real neural network is generally indefinite, with many near-zero or negative eigenvalues. Moreover, the interactions between different parameter groups in Transformers are much more complicated than those captured by this model.

While I understand the need for simplification in theoretical analysis, I am not fully convinced that this model accurately captures the optimization dynamics of practical LLM training. It remains unclear how well the theoretical conclusions generalize to real Transformers.

2. Restrictive theoretical assumptions.

Several theoretical results rely on relatively strong assumptions, including positive definite and slowly varying curvature, Gaussian and isotropic noise, and additional conditions such as analyticity, the number of independent curvature directions, and restrictions on elasticity.

These assumptions make the analysis tractable but may limit the applicability of the conclusions to practical neural network training. I would like to better understand whether the main insights still hold when these assumptions are relaxed.

3. Lack of experimental validation of practical value.

Although the paper includes measurements from Qwen3-0.6B training trajectories, I do not think these experiments sufficiently demonstrate the practical value of the theoretical findings.

In particular, it remains unclear whether the proposed theoretical insights can help achieve more stable neural network training or lower training loss.

I believe additional experiments connecting the theoretical findings to practical training behavior would significantly strengthen the paper.

I appreciate the theoretical value of this work and find the general research direction interesting. However, the gap between the theoretical setting and realistic Transformer training, the restrictive assumptions, and the limited experimental validation make it difficult for me to assess the practical significance of the results.

Questions for authors

no

Reviewer_vzGtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how a Muon update affects the surrounding parameters, with a particular focus on covariance based measures of the local response. The motivating observation appears to come from Qwen3 0.6B training trajectories, where updates with similar NS5 magnitudes can nevertheless produce very different local responses. The paper then introduces a number of concepts, including persistent motion, negative elasticities, stable cycles, Gaussian averaging, Gaussian equilibrium, and spectral universality, together with several mathematical setups intended to characterize these phenomena.

However, I could not identify the corresponding results in the paper. The paper is extremely hard to read. I'm not sure the authors even know their own results because the motivation is through Qwen but Qwen isn't even in the paper. Instead, the paper largely introduces a sequence of definitions and mathematical constructions that appear to be defined only for synthetic setups. I could not find a clear demonstration that these constructions describe the Qwen3 trajectories that motivate the paper, nor could I determine what concrete conclusions the authors draw about Qwen3 training.

As written, there is a substantial mismatch between the paper described in the abstract and the paper that is actually presented.

Critical strengths

The motivating question is interesting and potentially important. Understanding how a Muon update affects not only the updated parameters themselves but also the local response of the surrounding parameter configuration could provide useful insight into why spectral optimizers behave differently from more conventional optimization methods.

The question is interesting, but the current presentation is horrendous. It isn't clear if the paper is on synthetic setup or if it is being applied after training Qwen with Muon. There are no conclusions from the result. It just introduces a bunch of definitions.

Critical weaknesses

The main weakness is that I **could not identify what the actual results of the paper are.**This is not just a mismatch between the abstract and the body. More generally, the paper does not present clear conclusions anywhere. It reads largely as a sequence of definitions, constructions, and new terminology, without stating what these objects establish or what the reader is supposed to learn from them.

Moreover the abstract strongly motivates the work through Qwen3 0.6B Muon trajectories, but the body appears to move almost entirely to synthetic setups, and the connection back to Qwen3 is never established. Terms such as persistent motion, stable cycles, Gaussian averaging, Gaussian equilibrium, and spectral universality are introduced, but it is often unclear whether they are empirical observations, assumptions, definitions, or consequences of the theory. I could not find a clear statement of what was actually observed on Qwen3, what was proved or derived in the synthetic model, or why the latter explains the former.

The Qwen3 experimental setup itself is also essentially missing. Since the paper relies so heavily on these trajectories as motivation, it should clearly specify how Qwen3 0.6B was trained with Muon, which layers or parameter groups used Muon, what optimizer hyperparameters were chosen, what dataset and training regime were used, and how the trajectories and response statistics were collected. Without this information, the central empirical motivation is not reproducible and is difficult to evaluate.

The presentation makes this much worse. The paper repeatedly introduces additional formalism, such as the matrix layer setup, spectral updates, and the fast update, without clearly explaining why these constructions are needed or what conclusions they lead to. Overall, the paper feels like a collection of definitions rather than a complete scientific argument. There is no clear progression from motivating question, to analysis, to result, to interpretation, and in its current form I do not think the paper communicates a coherent contribution.

Questions for authors
  1. The paper contains too much jargon. What are the actual conclusions drawn from the Qwen trajectories, stated in clear and intuitive terms?
  2. What is the Qwen3 experimental setup that you used? What dataset, what layers were trained with Muon, how were the other layers trained, what hyperparameters were used? Do any of these things change the conclusions (which I don't actually know)?
Reviewer_hrV3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how updates to one parameter group influence another, motivated by Muon. In an idealized model, it analyzes how persistent matrix motion drives changes in curvature and when these changes can be described by a scalar objective. It characterizes how this depends on the coupled update rules, noise, and the order of normalization and averaging.

Critical strengths

Substantial mathematical treatment of coupled optimizer dynamics. Studying implicit bias through the interaction between parameter groups is an interesting perspective.

The NS5 example makes the distinction between similar update magnitudes and very different local responses concrete. The effect of normalization/averaging order on the existence of a scalar objective is interesting.

Critical weaknesses
  • My main concern is readability. The paper uses a lot of non-standard terminology, with little intuition to explain the main ideas. Although the terms are formally defined, it is often difficult to understand what the results mean and why they matter. That is, there is a lack of explanation of the high-level picture, which is almost essential to understand what the manuscript's main picture is. This starts in the abstract: terms such as "scalar curvature objectives" and "negative elasticities" are not standard optimization terminology, making the point of the paper difficult to understand from the get-go. Same applies to e.g. "cube-root updates remain aligned under decay" - it is really unclear what this means. Examples from the intro include "NS5 produces fairly similar update magnitudes even though its local responses vary sharply across singular directions and often become negative." (it is unclear what is meany by local responses here) or "fast matrix" or "fast updates", again, terms that are being properly defined much later. Overall, I think the manuscript requires major edits to make it much more clear to read.

  • The other concern is the connection to Muon. Much of the paper studies general coupled dynamics in an idealized model, with independently trainable curvature and either stable two-cycles or Gaussian equilibrium. It is unclear how these regimes correspond to actual network dynamics. The experiments examine Muon’s response to perturbations, but do not test the predicted interaction between parameter groups during training. Despite the Muon-specific theoretical examples, it remains unclear how much the theory explains about Muon’s implicit bias. I think this connection needs to be established more clearly, or the framing adjusted.

  • Lastly, my concern is significance of the result itself - although otherwise this is a significant achievement. The paper spends considerable effort establishing when the coupled dynamics admit a scalar objective - yet, along the lines of the aforementioned lack of high-level picture, it is not clear from the paper why this property informs the training dynamics.

Questions for authors
  1. Could you explain how the main results fit together and what the overall takeaway is for Muon?
  2. Your analysis derives stable cycles and Gaussian equilibrium in the proposed model. Do you happen to have evidence suggesting that either regime approximates Muon’s dynamics during neural-network training?
  3. Why is the existence of a scalar objective for the effective curvature dynamics important? What concrete consequences follow from its existence or absence for optimization or implicit bias?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.