- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tackles the problem of neural network optimization when there are many-to-one mappings from the parameter space to the function space. Specifically, it provides a way to mod out the positive rescaling gauge symmetry of shallow ReLU networks, when optimizing with Muon. The way this is achieved is by canonicalizing the gradients of the two weight matrices so that they are invariant to rescalings. The paper names this approach Gauge-Muon. The paper proves the invariance of the updates as well as proving that the update corresponds to steepest descent on the quotient of the parameter space by the positive rescaling symmetry gauge group. Furthermore, the paper provides a simple first-order bound on the change of the functional value computed by hidden neuron , when the weights are updated by a single Gauge-Muon step. The bound is dependent on the choice of a hyperparameter in the canonicalization, with a phase shift in behaviour around (at the bound is uniform over hidden index ). The paper also proves that the framework allows for modding out positive rescaling symmetries of shallow SwiGLU networks when optimizing with Muon.
The paper validates their findings with experiments, showing accuracy gains over Muon and other optimizers for image classification tasks within the scope of the experiments, as well as showing gains in certain regimes for language tasks within the scope of the experiments.
Critical strengths
The method introduced by the paper deals with a legitimate issue in neural network optimization. Specifically, it deals with the issue of many-to-one mappings from parameters to functions hindering optimization of neural networks. Not only do the authors prove mathematically that their methods work they also demonstrate empirically that the approach can improve training. Compared with other invariant optimizers, they see empirical gains, which could be attributable to the lack of an LMO for those invariant optimizers. Compared to non-invariant optimizers, they see performance gains which could be attributable to the lack of invariance for those optimizers.
Summary: The results deserve to be published since the problem considered is a legitimate problem, both theoretically and in practice, and the solution is quite elegant. I believe that the paper merits consideration to appear at ICLR as long as any critical concerns are dealt with appropriately.
Critical weaknesses
Weakness 1:
The activations and both have the additional affine symmetry \begin{equation} \sigma(x)=\sigma(-x) + x.\qquad\qquad (1) \end{equation} This symmetry can lead to further gauge symmetries depending on the architecture and parameter configuration. For example, in the case it leads to the following degeneracy (and who knows what else):
Suppose that we have a network defined by
where , . In the case $W=\begin{pmatrix} w&-w \end{pmatrix}$, $U=\begin{pmatrix} u & -u \end{pmatrix}^T$, $V=\begin{pmatrix} v&v \end{pmatrix}^T$, we have that Equation (1) yields
\begin{align*} f(x)&=\begin{pmatrix} w&-w \end{pmatrix}\Bigg(\begin{pmatrix} \mathrm{SiLU}(ux) \\ \mathrm{SiLU}(-ux) \end{pmatrix}\odot \begin{pmatrix} vx \\ vx \end{pmatrix}\Bigg)\\&= \begin{pmatrix} w&-w \end{pmatrix}\Bigg(\begin{pmatrix} \mathrm{SiLU}(ux)vx \\ \mathrm{SiLU}(-ux)vx \end{pmatrix}\Bigg)\\&=w\mathrm{SiLU}(ux)vx-w\mathrm{SiLU}(-ux)vx\\&=w(\mathrm{SiLU}(ux)-\mathrm{SiLU}(-ux))vx=wuvx^2, \end{align*} where Equation (1) was used in the last step. Clearly, there is a symmetry where we can proportionally rescale and or and , not just and . So, there are further gauge symmetries than the ones mentioned in the paper. Fortunately, this oversight does not affect the case since a careful calculation shows that for both kinds of networks considered in the paper (with and without identity skip connection), the additional symmetry of the activation does not yield further symmetries of the network affecting the optimization.
This does not necessarily require a solution in the sense of finding the correct gauge-invariant update for the optimizer, but I believe that this should, at the very least, be mentioned in the paper as a limitation for the case. The literature on parameter symmetry and identifiablity has many works which could be cited for the symmetry in Equation (1), for example, compare with equation (2) in Agrawal and Ostrowski 2022.
Weakness 2:
There are many grammatical errors and difficult-to-understand sentences and paragraphs which I believe need to be fixed in order for this paper to be publishable.
I have listed below the grammar errors and typos I found when reading normally, along with a couple of examples of sentences which I believe are hard to understand. I think that it would be a good idea to carefully read through the paper and rewrite for an even clearer exposition.
Line 127 : "opertion" "operation"
Line 137: "Consider" "Let" and "...be diff..." "...be a diff..."
Line 155: "in" "is"
Footnote 1: "Talor" "Taylor"
Line 174: Consider rewriting the sentence starting with "We call..." I believe it is hard to understand if one does not already know what is meant.
Line 175: "incomming" "incoming"
Line 194: Unclear what is meant by "Muon optimizer solves optimization setting involved matrices."
Line 252: "how the optimization perform" is not correct
Line 282: "To this end" "So far"
Line 288: This sentence needs to be rewritten and perhaps expanded upon. It is not grammatically correct and difficult to understand.
Line 371: The setup could be made clearer. The sentence "We use GPT-2... to evaluate our models." does not seem correct. It is different optimizers being compared, right?
Line 416: "...it falls behind we..." "...it falls behind. We..."
Lines 432-434: Figure 6: Something is weird with the numbers in the figure. They have a different font and are hard to read.
Line 1945 "unitand" "unit and"
Summary: Neither weakness is major. However, I consider both to be severe enough that they merit addressing before the paper can be published.
Weakness 1 is a legitimate scope choice, but it should be made clear in the text that there are further gauge symmetries that are not considered. Alternatively, the issue should be resolved, but this could potentially require a lot of work.
Weakness 2 is simply something that deserves to be fixed. I think that any paper published at this conference should be easy to read. The multitude of grammatical errors, typos, and difficult-to-understand sentences should thus be fixed before publication.
My score would change to at least 3 if both concerns are dealt with appropriately.
Reference used:
Agrawal and Ostrowski, 2022, "A classification of -invariant shallow neural networks".
Questions for authors
Clarification requests:
Am I correct in understanding that the results are for shallow NNs? If so, it should be clarified early in the paper and mentioned in the conclusion. To be clear, I agree that shallow nets merit study, but it should be mentioned.
Am I correct in understanding that in the language task, the new optimizer did not manage to beat Muon in the two 1B parameter experiments with dense networks? If so, this should also be clarified in the conclusion of the paper.
On Line 227 in Theorem 4.2, what is a "side norm"?
Questions relating to future work:
What do you believe is the reason for the "failure" of Gauge-Muon in the dense 1B regime? How likely is it that the gains from Gauge-Muon will persist at scale?