- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why Muon can outperform Adam on equivariant networks. The authors pointed out the reason lies in irrep-dependent parameter sharing, which may cause step size mismatch. Then, the authors propose spectral and RMS block normalization to improve performance. The authors conduct experiments on rMD17, MD22, QM9 to validate their findings.
Critical strengths
This paper proposed a valuable perspective: the effective step sizes of different irrep blocks may differ significantly under Adam, and the experiments show that the proposed block normalization methods do bring some improvements.
Critical weaknesses
- For the comparison between the equivariant model and the matched MLP, the differences are not limited to equivariance. Different nonlinear operations and block structures may also affect the results. The authors could design more controlled experiments to isolate these effects.
- The experiments show that the relative performance of Spectral, RMS, and Muon varies across architectures and tasks. Although authors admit to leave the inner principal as future work, I still think the authors should provide some empirical guidance about when to use Spectral or RMS or Muon.
Questions for authors
- In Figure 4a, spectral eliminates the spectral norm mismatch, but Table 2 shows that it brings little improvement over Adam on several datasets. In contrast RMS does not eliminate the mismatch but improves performance. This is weird. What does this suggest?
- Could authors provide some guidance on how to select between Spectral, RMS and Muon? (W2)
- The paper investigates spectral and RMS normalization, but other normalization strategies may provide additional insights.
5 comments
W2 + Q2: "[...] I still think the authors should provide some empirical guidance about when to use Spectral or RMS or Muon. Could authors provide some guidance on how to select between Spectral, RMS and Muon?"
RW2 + RQ2: We agree, and the new experiments make the guidance more specific. Section 6 now gives this guidance:
- Muon when accuracy matters most. It gives the lowest force error on 22 of the 24 model and dataset pairs of Table 2, and the lowest error on QM9 and particle dynamics (new results).
- Block normalization when training with Adam. It improves Adam at almost no cost, and averaged over the 12 datasets of Table 2 it closes about 70% of the gap between Adam and Muon on NequIP and about 45% on GotenNet. RMS normalization is the simplest choice: it needs no singular values and lowers Adam's force error on all 24 pairs. Spectral normalization gives the same results once each update is normalized to spectral norm , the shape factor of RQ1 below, which becomes the default in the revision.
Q1: "In Figure 4a, spectral eliminates the spectral norm mismatch, but Table 2 shows that it brings little improvement over Adam on several datasets. In contrast RMS does not eliminate the mismatch but improves performance. This is weird. What does this suggest?"
RQ1: It suggests that what matters is the step size each matrix receives, not equal spectral norms. Both variants set the size of each block update to the learning rate, but they differ in the norm that measures the size. Spectral normalization sets the spectral norm of the update to the learning rate, so every matrix gets the same spectral step. RMS normalization sets the root mean square of its entries to the learning rate, so a larger matrix gets a larger spectral step. Figure 4a measures spectral norms, so spectral normalization has a ratio of one by construction, while RMS normalization fixes a different norm and still sets the step size of every block. Sections 4.1 and 5.2 now state this.
This difference matters when the shapes differ. In GotenNet, almost all normalized matrices are , while in NequIP the shapes range from to . Following the Reviewer's question, we tested a variant that accounts for matrix shape, based on the spectral condition of Yang et al. (2024), which keeps the feature updates of layers with different shapes at the same scale. We therefore normalize each update to spectral norm . Results on NequIP:
| dataset | Adam | spectral | spectral with shape factor | RMS |
|---|---|---|---|---|
| ethanol | 0.366 0.006 | 0.375 0.002 | 0.314 0.011 | 0.319 0.009 |
| malonaldehyde | 0.542 0.029 | 0.545 0.010 | 0.475 0.010 | 0.474 0.015 |
| benzene | 0.159 0.006 | 0.135 0.004 | 0.121 0.003 | 0.118 0.002 |
| uracil | 0.487 0.019 | 0.473 0.011 | 0.386 0.011 | 0.382 0.015 |
| toluene | 0.489 0.037 | 0.451 0.007 | 0.398 0.008 | 0.392 0.009 |
| salicylic acid | 0.642 0.006 | 0.637 0.008 | 0.562 0.008 | 0.564 0.007 |
| naphthalene | 0.477 0.023 | 0.421 0.007 | 0.354 0.005 | 0.356 0.009 |
| paracetamol | 0.681 0.012 | 0.690 0.008 | 0.589 0.013 | 0.597 0.013 |
| aspirin | 0.817 0.034 | 0.824 0.002 | 0.734 0.009 | 0.735 0.005 |
| azobenzene | 0.633 0.028 | 0.622 0.009 | 0.529 0.021 | 0.524 0.012 |
| Ac-Ala3-NHMe | 1.008 0.090 | 0.962 0.005 | 0.856 0.016 | 0.867 0.019 |
| stachyose | 1.782 0.055 | 1.478 0.009 | 1.368 0.031 | 1.386 0.016 |
On NequIP the shape factor brings spectral normalization to the level of RMS normalization. On GotenNet, where almost all matrices are square, both forms of spectral normalization give the same results. This is the same step size mismatch: one step size does not suit matrices of different shapes, and the shape factor takes the shape into account. In the revision, spectral normalization uses the shape factor by default (Section 4.1 and Appendix A), and Table 2 is updated accordingly.
Q3: "The paper investigates spectral and RMS normalization, but other normalization strategies may provide additional insights."
RQ3: We tested several more variants on the synthetic task:
| update rule | width 64 | width 128 | width 256 |
|---|---|---|---|
| Adam | 0.449 0.014 | 0.330 0.017 | 0.230 0.008 |
| Frobenius per block | 0.443 0.028 | 0.281 0.016 | 0.161 0.008 |
| Adam, vector block learning rate | 0.363 0.013 | 0.214 0.010 | 0.136 0.008 |
| PolarAdamW (Zhang, 2026) | 0.346 0.013 | 0.251 0.009 | 0.186 0.009 |
On NequIP we also tested LayerNorm on the features in place of block normalization, which is worse than plain Adam on aspirin ( against ), and spectral normalization with the shape factor (RQ1). Two insights follow. Fixing the size of the block updates in the Frobenius norm also helps. And the best step size differs between irrep types (the vector blocks prefer a learning rate about eight times larger) and between matrix shapes (the shape factor on NequIP). These results are in Appendix D.2 and D.3, and Section 6 states the second insight as a direction for future work.
We thank the reviewer again: the questions led to a cleaner test of where the mismatch comes from and to the shape factor, now the default for spectral normalization. We hope our answers address the reviewer's concerns and would appreciate a reconsideration of the score. We are happy to answer any further questions.
References
Maurice Weiler and Gabriele Cesa. General E(2)-Equivariant Steerable CNNs. In Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1911.08251.
Greg Yang, James B. Simon, and Jeremy Bernstein. A Spectral Condition for Feature Learning. arXiv:2310.17813, 2024.
Haozhou Zhang. PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation. arXiv:2605.07067, 2026.
We thank the reviewer for the positive assessment and the constructive questions. We have updated the PDF. The changes for this review are marked R3 in the margin.
W1: "For the comparison between the equivariant model and the matched MLP, the differences are not limited to equivariance. Different nonlinear operations and block structures may also affect the results. The authors could design more controlled experiments to isolate these effects."
RW1: We agree that the dense MLP differs from the equivariant model in several ways at once. We provide a further ablation.
The equivariant model differs from the MLP in two main ways. The first is the weight sharing: the vector block W1 is one matrix applied to all three components of the vectors, as W1⊗I3. The second is the structure of the layer: separate blocks for scalars and vectors, connected by the nonlinearities of the model. The "untied" model removes the first and keeps the second: each matrix shared across the three vector components is replaced by three separate matrices, initialized as copies. Layers, nonlinearities, block structure and the function at initialization are those of the equivariant model. If the mismatch came from the weight sharing, it would disappear in the untied model; if it came from the block structure and the nonlinearities, it would remain. Ratio between the spectral norms of the scalar and vector block updates within a layer under Adam:
The mismatch remains without the sharing and grows with width. This makes our statement more precise: the mismatch does not come from the weight sharing itself, whose gradient scaling Adam compensates, but from the structure of the layer, where blocks of different irrep types play different roles and one learning rate treats them alike. The dense MLP does not have this structure. In our response to Reviewer 1 (W2), we also show that the gap between the equivariant model and the dense MLP holds during training, not only at the first update. The untied model is now in Appendix D (Table 10), and Section 5.1 states this.
Two more comparisons point the same way:
More irrep types, larger mismatch. NequIP on aspirin, Adam's ratio between the largest and smallest block update within a layer:
Another group, with a plain network as a baseline. A small C8-equivariant steerable CNN (Weiler & Cesa, 2019) on CIFAR-10. A steerable convolution expands its kernel in a fixed basis, one basis filter per pair of irreps of C8 and angular frequency, and learns one channel mixing matrix per basis filter. These matrices serve as the irrep blocks. Ratio between the largest and smallest block update within a layer under Adam, at the first update / over the last 100 steps:
The plain CNN shares weights as well, but its blocks are all of one type, and it shows almost no mismatch. In all three comparisons the mismatch follows the typed block structure, not a particular nonlinearity or model. The three comparisons are in Appendix D (Tables 10, 11 and 13) and are referenced in Section 5.1.