acceptodds
Under review as a conference paper at ICLR 2027

Muon on the Quotient: Gauge-Equivariant Descent for Homogeneous Networks

Abstract

Positively homogeneous networks, including ReLU networks and the SwiGLU blocks of modern transformers, have a rescaling symmetry. That is, scaling up the incoming weights of a hidden unit and scaling down its outgoing weights leaves the function unchanged. Orthogonalized optimizers such as Muon do not respect this symmetry, thereby equivalent parameterizations of the same function receive different updates, and a function-preserving rescaling can lead Muon to a sharper basin with a higher loss. We introduce Gauge-Muon, a family of optimizers that keeps Muon's orthogonalized update but applies it to gradients rescaled by per-unit factors computed from the weights, and then undoes the rescaling. Every member of the family is exactly equivariant under hidden-unit rescaling at every finite step size, and performs exact steepest descent in the quotient geometry that identifies all equivalent parameterizations. We further show how the family's exponent allocates functional motion across hidden units, and derive an invariant capacity bound on generalization. Empirically, Gauge-Muon is unaffected by rescalings that degrade Muon and improves on Muon in image classification and GPT-2 language modeling, with the most consistent gains in mixture-of-experts models with narrow experts.

Primary area:
optimization
Keywords:
Muon, Gauge-Muon, ReLU, SwiGLU

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CYA3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of neural network optimization when there are many-to-one mappings from the parameter space to the function space. Specifically, it provides a way to mod out the positive rescaling gauge symmetry of shallow ReLU networks, when optimizing with Muon. The way this is achieved is by canonicalizing the gradients of the two weight matrices so that they are invariant to rescalings. The paper names this approach Gauge-Muon. The paper proves the invariance of the updates as well as proving that the update corresponds to steepest descent on the quotient of the parameter space by the positive rescaling symmetry gauge group. Furthermore, the paper provides a simple first-order bound on the change of the functional value computed by hidden neuron ii, when the weights are updated by a single Gauge-Muon step. The bound is dependent on the choice of a hyperparameter α\alpha in the canonicalization, with a phase shift in behaviour around α=1\alpha=1 (at α=1\alpha=1 the bound is uniform over hidden index ii). The paper also proves that the framework allows for modding out positive rescaling symmetries of shallow SwiGLU networks when optimizing with Muon.

The paper validates their findings with experiments, showing accuracy gains over Muon and other optimizers for image classification tasks within the scope of the experiments, as well as showing gains in certain regimes for language tasks within the scope of the experiments.

Critical strengths

The method introduced by the paper deals with a legitimate issue in neural network optimization. Specifically, it deals with the issue of many-to-one mappings from parameters to functions hindering optimization of neural networks. Not only do the authors prove mathematically that their methods work they also demonstrate empirically that the approach can improve training. Compared with other invariant optimizers, they see empirical gains, which could be attributable to the lack of an LMO for those invariant optimizers. Compared to non-invariant optimizers, they see performance gains which could be attributable to the lack of invariance for those optimizers.

Summary: The results deserve to be published since the problem considered is a legitimate problem, both theoretically and in practice, and the solution is quite elegant. I believe that the paper merits consideration to appear at ICLR as long as any critical concerns are dealt with appropriately.

Critical weaknesses

Weakness 1:

The activations ReLU\mathrm{ReLU} and SiLU\mathrm{SiLU} both have the additional affine symmetry \begin{equation} \sigma(x)=\sigma(-x) + x.\qquad\qquad (1) \end{equation} This symmetry can lead to further gauge symmetries depending on the architecture and parameter configuration. For example, in the SwiGLU\mathrm{SwiGLU} case it leads to the following degeneracy (and who knows what else):

Suppose that we have a network f:R→Rf:\mathbb{R}\to \mathbb{R} defined by

f(x)=W(SiLU(Ux)⊙(Vx)),f(x)=W(\mathrm{SiLU}(Ux)\odot (Vx)), where W∈R1×2W\in \mathbb{R}^{1\times 2}, U,V∈R2×1U,V\in \mathbb{R}^{2\times 1}. In the case $W=\begin{pmatrix} w&-w \end{pmatrix}$, $U=\begin{pmatrix} u & -u \end{pmatrix}^T$, $V=\begin{pmatrix} v&v \end{pmatrix}^T$, we have that Equation (1) yields

\begin{align*} f(x)&=\begin{pmatrix} w&-w \end{pmatrix}\Bigg(\begin{pmatrix} \mathrm{SiLU}(ux) \\ \mathrm{SiLU}(-ux) \end{pmatrix}\odot \begin{pmatrix} vx \\ vx \end{pmatrix}\Bigg)\\&= \begin{pmatrix} w&-w \end{pmatrix}\Bigg(\begin{pmatrix} \mathrm{SiLU}(ux)vx \\ \mathrm{SiLU}(-ux)vx \end{pmatrix}\Bigg)\\&=w\mathrm{SiLU}(ux)vx-w\mathrm{SiLU}(-ux)vx\\&=w(\mathrm{SiLU}(ux)-\mathrm{SiLU}(-ux))vx=wuvx^2, \end{align*} where Equation (1) was used in the last step. Clearly, there is a symmetry where we can proportionally rescale UU and VV or UU and WW, not just WW and VV. So, there are further gauge symmetries than the ones mentioned in the paper. Fortunately, this oversight does not affect the ReLU\mathrm{ReLU} case since a careful calculation shows that for both kinds of ReLU\mathrm{ReLU} networks considered in the paper (with and without identity skip connection), the additional symmetry of the activation does not yield further symmetries of the network affecting the optimization.

This does not necessarily require a solution in the sense of finding the correct gauge-invariant update for the optimizer, but I believe that this should, at the very least, be mentioned in the paper as a limitation for the SwiGLU\mathrm{SwiGLU} case. The literature on parameter symmetry and identifiablity has many works which could be cited for the symmetry in Equation (1), for example, compare with equation (2) in Agrawal and Ostrowski 2022.

Weakness 2:

There are many grammatical errors and difficult-to-understand sentences and paragraphs which I believe need to be fixed in order for this paper to be publishable.

I have listed below the grammar errors and typos I found when reading normally, along with a couple of examples of sentences which I believe are hard to understand. I think that it would be a good idea to carefully read through the paper and rewrite for an even clearer exposition.

Line 127 : "opertion" →\to "operation"

Line 137: "Consider" →\to "Let" and "...be diff..." →\to "...be a diff..."

Line 155: "in" →\to "is"

Footnote 1: "Talor" →\to "Taylor"

Line 174: Consider rewriting the sentence starting with "We call..." I believe it is hard to understand if one does not already know what is meant.

Line 175: "incomming" →\to "incoming"

Line 194: Unclear what is meant by "Muon optimizer solves optimization setting involved matrices."

Line 252: "how the optimization perform" is not correct

Line 282: "To this end" →\to "So far"

Line 288: This sentence needs to be rewritten and perhaps expanded upon. It is not grammatically correct and difficult to understand.

Line 371: The setup could be made clearer. The sentence "We use GPT-2... to evaluate our models." does not seem correct. It is different optimizers being compared, right?

Line 416: "...it falls behind we..." →\to "...it falls behind. We..."

Lines 432-434: Figure 6: Something is weird with the numbers in the figure. They have a different font and are hard to read.

Line 1945 "unitand" →\to "unit and"

Summary: Neither weakness is major. However, I consider both to be severe enough that they merit addressing before the paper can be published.

Weakness 1 is a legitimate scope choice, but it should be made clear in the text that there are further gauge symmetries that are not considered. Alternatively, the issue should be resolved, but this could potentially require a lot of work.

Weakness 2 is simply something that deserves to be fixed. I think that any paper published at this conference should be easy to read. The multitude of grammatical errors, typos, and difficult-to-understand sentences should thus be fixed before publication.

My score would change to at least 3 if both concerns are dealt with appropriately.

Reference used:

Agrawal and Ostrowski, 2022, "A classification of GG-invariant shallow neural networks".

Questions for authors

Clarification requests:

Am I correct in understanding that the results are for shallow NNs? If so, it should be clarified early in the paper and mentioned in the conclusion. To be clear, I agree that shallow nets merit study, but it should be mentioned.

Am I correct in understanding that in the language task, the new optimizer did not manage to beat Muon in the two 1B parameter experiments with dense networks? If so, this should also be clarified in the conclusion of the paper.

On Line 227 in Theorem 4.2, what is a "side norm"?

Questions relating to future work:

What do you believe is the reason for the "failure" of Gauge-Muon in the dense 1B regime? How likely is it that the gains from Gauge-Muon will persist at scale?

Reviewer_KYuEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a variant of Muon that is equivariant to the rescaling symmetry of ReLU and SwiGLU networks. They do it by rescaling the gradient by diagonal factors built from each unit’s incoming and outgoing weight norms, applies Muon’s polar map, and maps the result back. They prove equivariance and a quotient steepest-descent characterization, and report gains over Muon on image classification and GPT-2 small language modeling.

Critical strengths

The method is simple and cheap: two diagonal rescalings, one before and one after Muon's polar map. Equivariance holds exactly at every finite step (Thm 4.2), not only to first order, and the proof is short and transparent. The construction extends cleanly to SwiGLU blocks and to hidden-unit permutations (App. E).

Critical weaknesses

W1

The main issue is that there is no convincing argument regarding the need for this equivariance. Among the variants of the method shown, either there is no benefit in the final result or there are other strong mechanisms involved that seem to make the difference. At standard initialization, Gauge-Muon and Muon reach comparable solutions. Differences appear only under synthetic rescalings (q ≥ 2) or off-optimal learning rates, and only on a FashionMNIST MLP. This is while the pure method from Def. 4.1 is 3% worse than standard Muon. In LM, the gains appear only with a geometric-mean normalization. This normalization does not affect equivariance. Yet in the SwiGLU MoE setting it turns a 3.1% loss into a 3.1% gain. That is the relative-update principle of LARS and others (and none of them is cited). It was hard to follow on what works and where. The presentation sometimes doesn't distinct the elements that made a difference. E.g., the normalization that the LM results depend on is only in the appendix.

 

W2

While the authors seem to object to it, this is still really a ``Path-SGD-Muon''. For a two-layer block, Gauge-Muon with α=1\alpha = 1 applies Path-SGD's preconditioner symmetrically around Muon's polar map. With A1=diag(1/∥vi∥)A_1 = \mathrm{diag}(1/\|v_i\|), Path-SGD updates UU by −ηA1(A1GU)-\eta A_1 (A_1 G_U), and Gauge-Muon by −ηA1 polar(A1GU)-\eta A_1\, \mathrm{polar}(A_1 G_U). The VV update is analogous. Replacing the polar map by the identity recovers Path-SGD exactly. Sec. 4 states that the method cannot be viewed as a spectral restatement of Path-SGD because of the coupling through singular vectors. I find this not convincing because that coupling is what any spectral version of Path-SGD would have.

Questions for authors

The questions match the weaknesses:

  1. Can you show a clear advantage for the equivariance?
  2. Can you justify in what way this is not Path-SGD-Muon and clarify what you mean by "cannot be viewed as a spectral restatement of Path-SGD because of the coupling through singular vectors"?
Reviewer_AK7zOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a variant of the Muon optimizer that accounts for the unit rescaling symmetries in MLPs: if all of the weights for a unit are multiplied by some a > 0, and the incoming weights applied to that unit in the following layer are multiplied by 1/a, then the final network remains functionally equivalent. The proposed optimizer is equivalent to Muon with the addition that it respects this invariance in weight space unlike vanilla Muon which can change update direction based on differences in scaling.

They argue that this is materially different from prior work, and that experimentally it is superior to vanilla Muon.

Critical strengths

The core research question is sound, the approach appears technically correct, and the development of symmetry-aware optimizers is an important research direction IMHO.

The development of a rescaling-invariant Muon is appealing, and a potentially interesting development.

The presented performance wins relative to vanilla Muon appear modest but appear consistent which is a satisfying result if it holds up. A principled improvement to Muon with real, robust gains could be very impactful, even if those gains are modest.

I found the relationship between Gauge-Muon and MoE models to be esp interesting. I found this to be unexpected and curious. I would be keen to see that covered more thoroughly.

Critical weaknesses

The primary weakness of this paper is its experimental methodology. I have 3 core concerns.

  1. The paper uses different models on different datasets, and different training paradigms in different datasets: full training from random init in some, frozen-backbone head training in others, and finetuning in yet others. The lack of consistency smells strongly of cherry picking. By the authors own count they have 7 distinct experimental setups (line 353).
  2. There are elementary baselines that are not considered: IMHO an essential baseline here is projected Muon (a la projected gradient descent) in which the rows of U are constrained to have unit l2 norm (or alternatively constrain the columns of V).
  3. Finally (and less important than 1 and 2), the language model experiments are quite small, only 124M parameter model. I would be wary of drawing any conclusions without seeing experiments at >1B parameter scale.
  4. AFAICT there is no systematic study varying \alpha.

I cannot in good faith recommend this paper for acceptance without a clean, coherent experimental methodology. FWIW my bar for performance is low: a modest win is a win, but this win must be robust and the evidence must be convincing beyond a reasonable doubt.

Unfortunately these shortcomings reduce my score below the acceptance threshold. I am not opposed to the research agenda here, but the paper in its current form fails to do this research question justice, esp wrt the experiments.

Other concerns, roughly in descending order of importance:

  • The writing is terse, jargon-heavy, and extensively invokes terminology without defining it. For example, terms like gauge invariance, positive homogeneity, orbit, polar mapping, path mass are thrown around without any definition or introduction. I am left with the impression that much of the paper was written by AI, or was written with the intent to show off the authors' command of fancy words and terminology. In any case communication does not appear to have been an objective. Once we make it into section 2, the situation improves somewhat.
  • It's unclear to me what the takeaway is in Fig 1. All of the plots look approximately the same. What is the purpose of this figure?
  • theorem 4.2 leaves the defn of U^+ and V^+ implicit
  • Figure 5 is confusing: what is the relationship of these two plots to each other? what does "150M" vs "1B" mean? From the body text I am guessing this is the number of training tokens but it is unclear in the plots as currently presented.
  • labels like "(a) 1B tokens" inside the plots in fig 6 are confusing and unnecessary. if they are needed, they should below plots.
  • line 31: "important alternative" what does "important" mean exactly? why are they important and what makes them important?
  • "path mass" is used in some cases and "path-mass" is used in others.
  • line 176: typo: "incomming"
  • line 282: "To this end," does not make sense in this context
Questions for authors

We introduce Gauge-Muon, a family of optimizers

it's really just one optimizer, right?

How does Gauge-Muon affect other optimization outcomes? Eg I would be curious if it has an impact on the sharpness of the final loss basin.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.