acceptodds
Under review as a conference paper at ICLR 2027

Equivariance Breaks the Learning Rate

Abstract

Equivariant networks are commonly trained with Adam, yet recent work reports that matrix structured optimizers such as Muon can perform better, with the reasons for these gains only partly understood. We identify one source of this difference inside equivariant layers. An equivariant layer learns one channel mixing matrix per degree , which we call an irrep block, and shares it across the components, giving the expanded map . This sharing sums gradient contributions across components and can produce different update scales under SGD. Adam's entrywise normalization reduces sensitivity to gradient scale, but neither optimizer directly controls the effective step size of each block. A single learning rate can therefore produce different effective step sizes across blocks. Muon instead controls the effective step size by approximately equalizing the singular values of each momentum matrix. We normalize each irrep block update by a single scalar, preserving its singular value ratios while letting the learning rate control its size. We implement this with spectral normalization or a simpler root-mean-square normalization. We evaluate spectral normalization in a controlled -equivariant model with a matched non-equivariant model. In this setting, the step size mismatch grows with width in the equivariant model but not in the non-equivariant model. We evaluate both variants across molecular force prediction on the rMD17 and MD22 datasets, QM9 molecular property prediction, and charged particle dynamics. Across these applications, block normalization generally improves Adam and closes part of its gap to Muon. These results highlight an overlooked interaction between equivariant architectures and their optimizers. Studying and designing the two together may help explain and address training difficulties often attributed to equivariance itself.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
equivariance, optimization, adam, muon, learning, rate, irrep

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GuFkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why Muon can outperform Adam on equivariant networks. The authors pointed out the reason lies in irrep-dependent parameter sharing, which may cause step size mismatch. Then, the authors propose spectral and RMS block normalization to improve performance. The authors conduct experiments on rMD17, MD22, QM9 to validate their findings.

Critical strengths

This paper proposed a valuable perspective: the effective step sizes of different irrep blocks may differ significantly under Adam, and the experiments show that the proposed block normalization methods do bring some improvements.

Critical weaknesses
  1. For the comparison between the equivariant model and the matched MLP, the differences are not limited to equivariance. Different nonlinear operations and block structures may also affect the results. The authors could design more controlled experiments to isolate these effects.
  2. The experiments show that the relative performance of Spectral, RMS, and Muon varies across architectures and tasks. Although authors admit to leave the inner principal as future work, I still think the authors should provide some empirical guidance about when to use Spectral or RMS or Muon.
Questions for authors
  1. In Figure 4a, spectral eliminates the spectral norm mismatch, but Table 2 shows that it brings little improvement over Adam on several datasets. In contrast RMS does not eliminate the mismatch but improves performance. This is weird. What does this suggest?
  2. Could authors provide some guidance on how to select between Spectral, RMS and Muon? (W2)
  3. The paper investigates spectral and RMS normalization, but other normalization strategies may provide additional insights.
5 comments

We thank the reviewer for the positive assessment and the constructive questions. We have updated the PDF. The changes for this review are marked R3 in the margin.

W1: "For the comparison between the equivariant model and the matched MLP, the differences are not limited to equivariance. Different nonlinear operations and block structures may also affect the results. The authors could design more controlled experiments to isolate these effects."

RW1: We agree that the dense MLP differs from the equivariant model in several ways at once. We provide a further ablation.

The equivariant model differs from the MLP in two main ways. The first is the weight sharing: the vector block W1W_1 is one matrix applied to all three components of the vectors, as W1⊗I3W_1 \otimes I_3. The second is the structure of the layer: separate blocks for scalars and vectors, connected by the nonlinearities of the model. The "untied" model removes the first and keeps the second: each matrix shared across the three vector components is replaced by three separate matrices, initialized as copies. Layers, nonlinearities, block structure and the function at initialization are those of the equivariant model. If the mismatch came from the weight sharing, it would disappear in the untied model; if it came from the block structure and the nonlinearities, it would remain. Ratio between the spectral norms of the scalar and vector block updates within a layer under Adam:

widthfirst update: equivariant / untiedlast 100 steps: equivariant / untied
162.21 ±\pm 0.33 / 2.61 ±\pm 0.351.32 ±\pm 0.19 / 1.41 ±\pm 0.36
643.14 ±\pm 0.28 / 4.02 ±\pm 0.331.68 ±\pm 0.12 / 2.50 ±\pm 0.43
1284.11 ±\pm 0.44 / 5.37 ±\pm 0.541.91 ±\pm 0.39 / 2.22 ±\pm 0.39
2565.23 ±\pm 0.40 / 6.73 ±\pm 0.362.63 ±\pm 0.56 / 3.24 ±\pm 0.67

The mismatch remains without the sharing and grows with width. This makes our statement more precise: the mismatch does not come from the weight sharing itself, whose gradient scaling Adam compensates, but from the structure of the layer, where blocks of different irrep types play different roles and one learning rate treats them alike. The dense MLP does not have this structure. In our response to Reviewer 1 (W2), we also show that the gap between the equivariant model and the dense MLP holds during training, not only at the first update. The untied model is now in Appendix D (Table 10), and Section 5.1 states this.

Two more comparisons point the same way:

More irrep types, larger mismatch. NequIP on aspirin, Adam's ratio between the largest and smallest block update within a layer:

featuresmaximum degree 1maximum degree 2maximum degree 3
ratio within a layer1.472.062.60

Another group, with a plain network as a baseline. A small C8C_8-equivariant steerable CNN (Weiler & Cesa, 2019) on CIFAR-10. A steerable convolution expands its kernel in a fixed basis, one basis filter per pair of irreps of C8C_8 and angular frequency, and learns one channel mixing matrix per basis filter. These matrices serve as the irrep blocks. Ratio between the largest and smallest block update within a layer under Adam, at the first update / over the last 100 steps:

widthC8C_8-equivariant CNNplain CNN
81.76 / 4.491.08 / 1.28
162.06 / 2.411.05 / 1.18
322.02 / 2.161.04 / 1.14
642.51 / 2.101.04 / 1.10

The plain CNN shares weights as well, but its blocks are all of one type, and it shows almost no mismatch. In all three comparisons the mismatch follows the typed block structure, not a particular nonlinearity or model. The three comparisons are in Appendix D (Tables 10, 11 and 13) and are referenced in Section 5.1.

W2 + Q2: "[...] I still think the authors should provide some empirical guidance about when to use Spectral or RMS or Muon. Could authors provide some guidance on how to select between Spectral, RMS and Muon?"

RW2 + RQ2: We agree, and the new experiments make the guidance more specific. Section 6 now gives this guidance:

  • Muon when accuracy matters most. It gives the lowest force error on 22 of the 24 model and dataset pairs of Table 2, and the lowest error on QM9 and particle dynamics (new results).
  • Block normalization when training with Adam. It improves Adam at almost no cost, and averaged over the 12 datasets of Table 2 it closes about 70% of the gap between Adam and Muon on NequIP and about 45% on GotenNet. RMS normalization is the simplest choice: it needs no singular values and lowers Adam's force error on all 24 pairs. Spectral normalization gives the same results once each m×nm \times n update is normalized to spectral norm ηm/n\eta\sqrt{m/n}, the shape factor of RQ1 below, which becomes the default in the revision.

Q1: "In Figure 4a, spectral eliminates the spectral norm mismatch, but Table 2 shows that it brings little improvement over Adam on several datasets. In contrast RMS does not eliminate the mismatch but improves performance. This is weird. What does this suggest?"

RQ1: It suggests that what matters is the step size each matrix receives, not equal spectral norms. Both variants set the size of each block update to the learning rate, but they differ in the norm that measures the size. Spectral normalization sets the spectral norm of the update to the learning rate, so every matrix gets the same spectral step. RMS normalization sets the root mean square of its entries to the learning rate, so a larger matrix gets a larger spectral step. Figure 4a measures spectral norms, so spectral normalization has a ratio of one by construction, while RMS normalization fixes a different norm and still sets the step size of every block. Sections 4.1 and 5.2 now state this.

This difference matters when the shapes differ. In GotenNet, almost all normalized matrices are 64×6464 \times 64, while in NequIP the shapes range from 32×3232 \times 32 to 32×1126432 \times 11264. Following the Reviewer's question, we tested a variant that accounts for matrix shape, based on the spectral condition of Yang et al. (2024), which keeps the feature updates of layers with different shapes at the same scale. We therefore normalize each m×nm \times n update to spectral norm ηm/n\eta\sqrt{m/n}. Results on NequIP:

datasetAdamspectralspectral with shape factorRMS
ethanol0.366 ±\pm 0.0060.375 ±\pm 0.0020.314 ±\pm 0.0110.319 ±\pm 0.009
malonaldehyde0.542 ±\pm 0.0290.545 ±\pm 0.0100.475 ±\pm 0.0100.474 ±\pm 0.015
benzene0.159 ±\pm 0.0060.135 ±\pm 0.0040.121 ±\pm 0.0030.118 ±\pm 0.002
uracil0.487 ±\pm 0.0190.473 ±\pm 0.0110.386 ±\pm 0.0110.382 ±\pm 0.015
toluene0.489 ±\pm 0.0370.451 ±\pm 0.0070.398 ±\pm 0.0080.392 ±\pm 0.009
salicylic acid0.642 ±\pm 0.0060.637 ±\pm 0.0080.562 ±\pm 0.0080.564 ±\pm 0.007
naphthalene0.477 ±\pm 0.0230.421 ±\pm 0.0070.354 ±\pm 0.0050.356 ±\pm 0.009
paracetamol0.681 ±\pm 0.0120.690 ±\pm 0.0080.589 ±\pm 0.0130.597 ±\pm 0.013
aspirin0.817 ±\pm 0.0340.824 ±\pm 0.0020.734 ±\pm 0.0090.735 ±\pm 0.005
azobenzene0.633 ±\pm 0.0280.622 ±\pm 0.0090.529 ±\pm 0.0210.524 ±\pm 0.012
Ac-Ala3-NHMe1.008 ±\pm 0.0900.962 ±\pm 0.0050.856 ±\pm 0.0160.867 ±\pm 0.019
stachyose1.782 ±\pm 0.0551.478 ±\pm 0.0091.368 ±\pm 0.0311.386 ±\pm 0.016

On NequIP the shape factor brings spectral normalization to the level of RMS normalization. On GotenNet, where almost all matrices are square, both forms of spectral normalization give the same results. This is the same step size mismatch: one step size does not suit matrices of different shapes, and the shape factor takes the shape into account. In the revision, spectral normalization uses the shape factor by default (Section 4.1 and Appendix A), and Table 2 is updated accordingly.

Q3: "The paper investigates spectral and RMS normalization, but other normalization strategies may provide additional insights."

RQ3: We tested several more variants on the synthetic task:

update rulewidth 64width 128width 256
Adam0.449 ±\pm 0.0140.330 ±\pm 0.0170.230 ±\pm 0.008
Frobenius per block0.443 ±\pm 0.0280.281 ±\pm 0.0160.161 ±\pm 0.008
Adam, vector block learning rate ×8\times 80.363 ±\pm 0.0130.214 ±\pm 0.0100.136 ±\pm 0.008
PolarAdamW (Zhang, 2026)0.346 ±\pm 0.0130.251 ±\pm 0.0090.186 ±\pm 0.009

On NequIP we also tested LayerNorm on the features in place of block normalization, which is worse than plain Adam on aspirin (1.249±0.1191.249 \pm 0.119 against 0.817±0.0340.817 \pm 0.034), and spectral normalization with the shape factor (RQ1). Two insights follow. Fixing the size of the block updates in the Frobenius norm also helps. And the best step size differs between irrep types (the vector blocks prefer a learning rate about eight times larger) and between matrix shapes (the shape factor on NequIP). These results are in Appendix D.2 and D.3, and Section 6 states the second insight as a direction for future work.

We thank the reviewer again: the questions led to a cleaner test of where the mismatch comes from and to the shape factor, now the default for spectral normalization. We hope our answers address the reviewer's concerns and would appreciate a reconsideration of the score. We are happy to answer any further questions.

References

Maurice Weiler and Gabriele Cesa. General E(2)-Equivariant Steerable CNNs. In Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1911.08251.

Greg Yang, James B. Simon, and Jeremy Bernstein. A Spectral Condition for Feature Learning. arXiv:2310.17813, 2024.

Haozhou Zhang. PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation. arXiv:2605.07067, 2026.

Reviewer_HepZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the interactions between optimizers and equivariant networks that process rotational higher-order tensor features. The analysis covers from toy examples to certain real world tasks, such as interatomic potentials and property predictions. The main results reveal that using a single learning rate across all irreps blocks leads to suboptimal training, as different tensor degrees exhibit different effective step sizes. The paper proposed a simple yet effective fix. That is to divide each Adam’s update block by its own size, ensuring each block update is independent of block size.

Critical strengths
  1. The paper discusses an insightful connection between optimization and equivariant networks. This topic is often overlooked by the community.
  2. The mechanism is clearly explained, and the proposed fix is plausible.
Critical weaknesses
  1. I think the title overclaims a bit. The paper focuses only on SO(3)-equivariant networks, not the general equivariant ones. It would be better to be more specific about this in the paper’s title.
  2. The main issue I found is an inconsistency between the paper’s central claim and the experimental results. The paper claims that the update mismatch across tensor blocks cause suboptimal training and propose to normalize them by either spectral or RMS normalization. While spectral normalization is demonstrated as the right fix for Adam (figure 2) RMS outperforms it in most real world tasks (table 2 and table 4). In table 3, Adam baseline can even beat spectral. Meanwhile, figure 4a shows RMS does not fully equalize the block update sizes as compared to spectral and Muon. RMS’s spectral norm ratio varies a lot through training, yet it still can achieve good results.
  3. Furthermore, Muon also makes the spectral norm ratio of 1 across training, and it beats spectral in all tasks at roughly the same cost (table 5). Hence I suspect the practical utility of the proposed block spectral normalization in the paper.
Questions for authors
  1. Can you elaborate on the architectures used in the toy example? It is unclear to me how scalar and vector features are mixed and how vector features are updated after the mixing step.
6 comments

We thank the reviewer for the careful reading and for the questions on the consistency of the claims. They led to two new experiments and to a clearer statement of what block normalization is for. We have updated the PDF. The changes for this review are marked R2 in the margin.

W1: "I think the title overclaims a bit. The paper focuses only on SO(3)-equivariant networks, not the general equivariant ones. It would be better to be more specific about this in the paper's title."

RW1: The reviewer is right: our experiments use SO(3) and O(3) equivariant models. We chose them because rotations in three dimensions are the symmetry most used in practice. The abstract and Section 6 now state this scope.

We also tested whether the observation is specific to this group, on 2D rotations: a small C8C_8-equivariant steerable CNN (Weiler & Cesa, 2019) on CIFAR-10. A steerable convolution expands its kernel in a fixed basis, one basis filter per pair of irreps of C8C_8 and angular frequency, and learns one channel mixing matrix per basis filter. These matrices serve as the irrep blocks. The mismatch is present: late in training, Adam’s ratio between the largest and smallest block update within a layer is 2.1 to 4.5 in the steerable CNN and 1.1 to 1.3 in a plain CNN of the same size. Spectral block normalization helps, and more so at larger widths. Test accuracy (%) on CIFAR-10 by width:

widthAdamspectralRMS
3281.4 ±\pm 0.382.1 ±\pm 0.881.3 ±\pm 0.5
6481.8 ±\pm 0.384.4 ±\pm 0.282.8 ±\pm 0.4
12883.0 ±\pm 0.986.5 ±\pm 0.782.9 ±\pm 0.4

These results are preliminary (short training, no augmentation, a small learning rate search). On CIFAR-100, where all models overfit early, we see no clear gain yet. Together with the SO(3) and O(3) results, this indicates that the effect is not specific to one group. The plain CNN shares weights as well, but its blocks are all of one type, and it shows almost no mismatch. We therefore expect the effect wherever a layer combines blocks of different types, as in steerable networks for other groups. We would like to keep the title, with the scope stated in the abstract. The experiment is in Appendix D.1 (Tables 13 and 14) and is referenced in Sections 1, 5.1 and 6.

W2: "The main issue I found is an inconsistency between the paper's central claim and the experimental results. [...] While spectral normalization is demonstrated as the right fix for Adam (figure 2) RMS outperforms it in most real world tasks (table 2 and table 4). In table 3, Adam baseline can even beat spectral. Meanwhile, figure 4a shows RMS does not fully equalize the block update sizes as compared to spectral and Muon. RMS's spectral norm ratio varies a lot through training, yet it still can achieve good results."

RW2: We thank the reviewer for raising this point. A new experiment explains the difference between the two variants.

RMS against spectral normalization. Both variants set the size of each block update to the learning rate. However, they differ in the norm that measures the size.

Spectral normalization sets the spectral norm of the update to the learning rate, so every matrix gets the same spectral step.

RMS normalization sets the root mean square of its entries to the learning rate, so a larger matrix gets a larger spectral step.

Regarding Figure 4a. The figure measures spectral norms, so spectral normalization has a ratio of one by construction. RMS normalization fixes a different norm, so its spectral ratio is not one, yet it still sets the step size of every block and lowers Adam’s force error on all 24 model and dataset pairs. The caption of Figure 4 now states that Figure 4a measures Adam’s mismatch and does not rank the normalizations, and Sections 4.1 and 5.2 explain why RMS normalization does not equalize the spectral norms.

Regarding the performance of spectral normalization. In GotenNet, almost all normalized matrices are 64×6464 \times 64, while in NequIP the shapes range from 32×3232 \times 32 to 32×1126432 \times 11264. Following the reviewer’s comment, we tested a variant that accounts for matrix shape, based on the spectral condition of Yang et al. (2024), which keeps the feature updates of layers with different shapes at the same scale. We therefore normalize each m×nm \times n update to spectral norm ηm/n\eta\sqrt{m/n}. Results on all 24 model and dataset pairs:

NequIP:

datasetAdamspectralspectral with shape factorRMS
ethanol0.366 ±\pm 0.0060.375 ±\pm 0.0020.314 ±\pm 0.0110.319 ±\pm 0.009
malonaldehyde0.542 ±\pm 0.0290.545 ±\pm 0.0100.475 ±\pm 0.0100.474 ±\pm 0.015
benzene0.159 ±\pm 0.0060.135 ±\pm 0.0040.121 ±\pm 0.0030.118 ±\pm 0.002
uracil0.487 ±\pm 0.0190.473 ±\pm 0.0110.386 ±\pm 0.0110.382 ±\pm 0.015
toluene0.489 ±\pm 0.0370.451 ±\pm 0.0070.398 ±\pm 0.0080.392 ±\pm 0.009
salicylic acid0.642 ±\pm 0.0060.637 ±\pm 0.0080.562 ±\pm 0.0080.564 ±\pm 0.007
naphthalene0.477 ±\pm 0.0230.421 ±\pm 0.0070.354 ±\pm 0.0050.356 ±\pm 0.009
paracetamol0.681 ±\pm 0.0120.690 ±\pm 0.0080.589 ±\pm 0.0130.597 ±\pm 0.013
aspirin0.817 ±\pm 0.0340.824 ±\pm 0.0020.734 ±\pm 0.0090.735 ±\pm 0.005
azobenzene0.633 ±\pm 0.0280.622 ±\pm 0.0090.529 ±\pm 0.0210.524 ±\pm 0.012
Ac-Ala3-NHMe1.008 ±\pm 0.0900.962 ±\pm 0.0050.856 ±\pm 0.0160.867 ±\pm 0.019
stachyose1.782 ±\pm 0.0551.478 ±\pm 0.0091.368 ±\pm 0.0311.386 ±\pm 0.016

GotenNet:

datasetAdamspectralspectral with shape factorRMS
ethanol0.194 ±\pm 0.0010.152 ±\pm 0.0050.153 ±\pm 0.0080.165 ±\pm 0.004
malonaldehyde0.289 ±\pm 0.0060.232 ±\pm 0.0060.230 ±\pm 0.0040.237 ±\pm 0.003
benzene0.052 ±\pm 0.0040.038 ±\pm 0.0020.036 ±\pm 0.0010.038 ±\pm 0.003
uracil0.205 ±\pm 0.0140.172 ±\pm 0.0040.170 ±\pm 0.0010.175 ±\pm 0.007
toluene0.167 ±\pm 0.0120.134 ±\pm 0.0050.130 ±\pm 0.0020.135 ±\pm 0.004
salicylic acid0.311 ±\pm 0.0080.247 ±\pm 0.0010.248 ±\pm 0.0100.251 ±\pm 0.004
naphthalene0.209 ±\pm 0.0060.141 ±\pm 0.0070.141 ±\pm 0.0040.144 ±\pm 0.003
paracetamol0.388 ±\pm 0.0110.295 ±\pm 0.0030.294 ±\pm 0.0020.304 ±\pm 0.006
aspirin0.432 ±\pm 0.0250.365 ±\pm 0.0120.362 ±\pm 0.0100.376 ±\pm 0.002
azobenzene0.326 ±\pm 0.0230.232 ±\pm 0.0050.230 ±\pm 0.0070.234 ±\pm 0.008
Ac-Ala3-NHMe0.753 ±\pm 0.0180.598 ±\pm 0.0200.585 ±\pm 0.0140.596 ±\pm 0.012
stachyose1.589 ±\pm 0.0331.245 ±\pm 0.0121.249 ±\pm 0.0161.266 ±\pm 0.025

On NequIP the shape factor brings spectral normalization to the level of RMS normalization. On GotenNet, where almost all matrices are square, both forms of spectral normalization give the same results. This is the same step size mismatch: one step size does not suit matrices of different shapes, and the shape factor takes the shape into account. In the revision, spectral normalization uses the shape factor by default: square matrices keep spectral norm η\eta, and every m×nm \times n matrix with m≠nm \neq n is normalized to ηm/n\eta\sqrt{m/n} (Section 4.1 and Appendix A). Table 2 is updated accordingly.

Finally, in Table 3, Adam (0.817 ±\pm 0.034) and spectral normalization (0.824 ±\pm 0.002) are effectively the same on this dataset. With the shape factor, spectral normalization lowers the error to 0.734 ±\pm 0.009, and Table 3 now reports this value.

Reviewer_938kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the why Muon outperform Adam on O(3)-equivariant networks. They argue that sharing each irrep block across 2l+1 components scales the gradient, whcih Adam largely cancels, while Adam still leaves the update norms of the blocks in a layer unequal under one learning rate. The authors experiment with block normalization which rescales each block's adam update by one scalar so taht its norm equals the learning rate. The authors validates their ideas on property prediction dataset (QM9), and molecular dynamics datasets (rMD17 and MD22).

Critical strengths
  1. RMS normalization lowers Adam's force MAE on all 24 model/dataset pairs and spectral normalization on all twelve with QM9. The rule is one scalar per irrep block and leaves Adam's moments untouched, so a usable result at 5000 steps remains even if the explanation is incomplete.
  2. Figure 5 isolates the parameters, with only irrep blocks and the readout trained on GotenNet aspirin. A scalar rescaling recovers ~85% of Muon's margin over Adam there, whcih is the most direct evidence that irrep block step sizes matter.
Critical weaknesses
  1. No experiment isolates equalizing irrep blocks from fixed update norms.
  2. The mismatch grows with width claim in the equivariant model but not in the matched MLP is measured only at Adam's first update. There, the bias corrected update is sign(g) so every block has the same RMS norm and the raito reflect only block shape and the gradients sign pattern at initialization.
Questions for authors
  1. Does Adam with separately tuned blcok and non-block rates match block normalization? If it does, equalizing blocks is not what helps.
  2. Were Muon runs on QM9 and particle dynamics experimented, and which tensors does Muon impact in each model?
  3. The Section 5.1 task and model closely resemble the SO(3) testbed on Zhang (2026), it is cited but not on there. Is the testbed taken from it, and does PolarAdamW also outperform Muon with Table 1's tuned protocol?
5 comments

We thank the reviewer for the careful review. We ran the proposed experiments; they test the central observation more directly than the submission did. We have updated the PDF. The changes for this review are marked R1 in the margin.

W1 + Q1: "No experiment isolates equalizing irrep blocks from fixed update norms. Does Adam with separately tuned block and non-block rates match block normalization? If it does, equalizing blocks is not what helps."

RW1 + RQ1: We ran an experiment where the weights of irreps with degree l≥1l \geq 1 get their own learning rate.

A separately tuned learning rate gives mixed results, and it does not match block normalization in general. On NequIP it gives no gain, and on GotenNet it gives part of the gain on four of five datasets. On the synthetic task it improves Adam at every width, and is better than block normalization at widths 64 and 128 and worse at width 256.

Both techniques address the same step size mismatch, and the paper does not claim that block normalization is the only solution.

New result, NequIP (×k\times k: the learning rate of the l≥1l \geq 1 weights is multiplied by kk; bold: best result; underline: second best):

dataset×0.5\times 0.5×1\times 1 (Adam)×2\times 2×4\times 4×8\times 8spectralRMS
aspirin0.812±0.0220.812 \pm 0.0220.817±0.0340.817 \pm 0.0340.838±0.0400.838 \pm 0.0400.909±0.1150.909 \pm 0.1150.932±0.0490.932 \pm 0.0490.734±0.009\mathbf{0.734 \pm 0.009}0.735±0.005‾\underline{0.735 \pm 0.005}
benzene0.158±0.0040.158 \pm 0.0040.159±0.0060.159 \pm 0.0060.180±0.0120.180 \pm 0.0120.176±0.0180.176 \pm 0.0180.195±0.0150.195 \pm 0.0150.121±0.003‾\underline{0.121 \pm 0.003}0.118±0.002\mathbf{0.118 \pm 0.002}
naphthalene0.480±0.0120.480 \pm 0.0120.477±0.0230.477 \pm 0.0230.517±0.0250.517 \pm 0.0250.552±0.0460.552 \pm 0.0460.563±0.0350.563 \pm 0.0350.354±0.005\mathbf{0.354 \pm 0.005}0.356±0.009‾\underline{0.356 \pm 0.009}
azobenzene0.650±0.0190.650 \pm 0.0190.633±0.0280.633 \pm 0.0280.674±0.0620.674 \pm 0.0620.712±0.0700.712 \pm 0.0700.754±0.0540.754 \pm 0.0540.529±0.021‾\underline{0.529 \pm 0.021}0.524±0.012\mathbf{0.524 \pm 0.012}
stachyose1.795±0.0981.795 \pm 0.0981.782±0.0551.782 \pm 0.0551.754±0.0681.754 \pm 0.0681.726±0.0421.726 \pm 0.0421.769±0.0631.769 \pm 0.0631.368±0.031\mathbf{1.368 \pm 0.031}1.386±0.016‾\underline{1.386 \pm 0.016}

New result, GotenNet:

dataset×0.5\times 0.5×1\times 1 (Adam)×2\times 2×4\times 4×8\times 8spectralRMS
aspirin0.464±0.0400.464 \pm 0.0400.432±0.0250.432 \pm 0.0250.435±0.0300.435 \pm 0.030 (1 of 3 failed)0.615±0.1810.615 \pm 0.181 (1 of 3 failed)failed (3 of 3)0.365±0.012\mathbf{0.365 \pm 0.012}0.376±0.002‾\underline{0.376 \pm 0.002}
benzene0.058±0.0030.058 \pm 0.0030.052±0.0040.052 \pm 0.0040.048±0.0060.048 \pm 0.0060.047±0.0040.047 \pm 0.0040.044±0.0020.044 \pm 0.0020.038±0.002\mathbf{0.038 \pm 0.002}0.038±0.003\mathbf{0.038 \pm 0.003}
naphthalene0.219±0.0040.219 \pm 0.0040.209±0.0060.209 \pm 0.0060.196±0.0040.196 \pm 0.0040.184±0.0070.184 \pm 0.0070.181±0.0080.181 \pm 0.0080.141±0.007\mathbf{0.141 \pm 0.007}0.144±0.003‾\underline{0.144 \pm 0.003}
azobenzene0.372±0.0150.372 \pm 0.0150.326±0.0230.326 \pm 0.0230.288±0.0190.288 \pm 0.0190.276±0.0110.276 \pm 0.0110.261±0.0090.261 \pm 0.0090.232±0.005\mathbf{0.232 \pm 0.005}0.234±0.008‾\underline{0.234 \pm 0.008}
stachyose1.751±0.0541.751 \pm 0.0541.589±0.0331.589 \pm 0.0331.513±0.0811.513 \pm 0.0811.450±0.0581.450 \pm 0.0581.432±0.0611.432 \pm 0.0611.245±0.012\mathbf{1.245 \pm 0.012}1.266±0.025‾\underline{1.266 \pm 0.025}

Synthetic task:

width×1\times 1 (Adam)×2\times 2×4\times 4×8\times 8spectral
640.449 ±\pm 0.0140.421 ±\pm 0.0110.380 ±\pm 0.0280.363 ±\pm 0.0130.376 ±\pm 0.017
1280.330 ±\pm 0.0170.287 ±\pm 0.0140.246 ±\pm 0.0150.214 ±\pm 0.0100.225 ±\pm 0.006
2560.230 ±\pm 0.0080.170 ±\pm 0.0060.146 ±\pm 0.0070.136 ±\pm 0.0080.121 ±\pm 0.004

On the synthetic task and on GotenNet, a separate learning rate alone improves Adam, which is evidence for the mismatch that does not depend on our method. It adds one hyperparameter per irrep type, while block normalization adds none.

On W1. We agree: block normalization fixes both the size of each block update and makes the blocks of a layer equal, and the paper does not separate the two. Equal sizes may also not be optimal, since different irrep types can need different step sizes (on the synthetic task, Adam with an eight times larger learning rate for the vector blocks is best at widths 64 and 128). Where a separate learning rate helps, it confirms the mismatch the paper describes. Section 6 now states this as a limitation and leaves a full analysis, including step sizes per irrep type and their interaction with the nonlinearities, to future work.

We added these experiments to Appendix D.3 (Tables 17-19) and discuss them in Section 5.2. The abstract and Section 4.1 now state that block normalization is one fix, not the only one.

W2: "The mismatch grows with width claim in the equivariant model but not in the matched MLP is measured only at Adam's first update. There, the bias corrected update is sign(g) so every block has the same RMS norm and the ratio reflect only block shape and the gradients sign pattern at initialization."

RW2: The reviewer is right: at the first update Adam's step is sign(g), so the ratio in Figure 2 depends only on the block shapes and the sign pattern of the first gradient, which itself depends on the structure of the layer. We therefore measured the ratio during training, over the last 100 of 1000 steps:

widthfirst update, equivariantfirst update, denselast 100 steps, equivariantlast 100 steps, dense
161.74 ±\pm 0.131.67 ±\pm 0.112.56 ±\pm 0.701.65 ±\pm 0.24
321.88 ±\pm 0.211.70 ±\pm 0.123.65 ±\pm 0.582.30 ±\pm 0.38
642.25 ±\pm 0.121.73 ±\pm 0.156.96 ±\pm 1.252.00 ±\pm 0.24
1282.63 ±\pm 0.241.76 ±\pm 0.115.69 ±\pm 1.312.12 ±\pm 0.16
1922.79 ±\pm 0.181.73 ±\pm 0.088.25 ±\pm 2.221.95 ±\pm 0.14
2562.73 ±\pm 0.341.76 ±\pm 0.079.85 ±\pm 2.461.99 ±\pm 0.28

The trend holds during training and is larger there. In our response to Reviewer 3 (W1), an ablation that removes only the weight sharing shows the same: the mismatch remains and grows with width. This is now Figure 2b, discussed in Section 5.1.

Q2: "Were Muon runs on QM9 and particle dynamics experimented, and which tensors does Muon impact in each model?"

RQ2: Muon gives the lowest error on QM9 and similar results for particle dynamics. We added both results to Appendix D.4 (Tables 20 and 21) and refer to them in Section 5.3.

taskAdamspectralRMSMuon
QM9, GotenNet, average of Table 41.704 ±\pm 0.0441.633 ±\pm 0.0211.619 ±\pm 0.0231.36 ±\pm 0.01
particle dynamics, NequIP, test MSE0.01465 ±\pm 0.000400.01282 ±\pm 0.000460.01340 ±\pm 0.000530.01217 ±\pm 0.00033

Muon and block normalization act on the same tensors, and both treat each irrep block separately; all other parameters use AdamW. We added this table to Appendix C.1 (Table 9):

modelMuon or block normalizationAdamW
NequIPirrep blocks (1.5% of parameters), radial network matrices (98%)embedding, biases, other 1D parameters
GotenNetirrep blocks (63%), ordinary weight matrices (31%)embeddings, biases, other 1D parameters

Q3: "The Section 5.1 task and model closely resemble the SO(3) testbed on Zhang (2026), it is cited but not on there. Is the testbed taken from it, and does PolarAdamW also outperform Muon with Table 1's tuned protocol?"

RQ3: Yes, and we apologize for the missing citation. The task and the model follow the SO(3) testbed of Zhang (2026). Section 5.1 now credits it and states the protocol differences:

Zhang (2026) uses one fixed set of hyperparameters for all optimizers, at widths 16 to 128, and describes it as "not a tuned benchmark". We search the learning rate, the initialization scale and the moment coefficients for every optimizer and width, train for 1000 full batch steps at widths 16 to 256, and report the test loss at the best validation step.

We also ran PolarAdamW:

width163264128192256
PolarAdamW0.614 ±\pm 0.0200.485 ±\pm 0.0260.346 ±\pm 0.0130.251 ±\pm 0.0090.207 ±\pm 0.0090.186 ±\pm 0.009
Muon0.634 ±\pm 0.0220.513 ±\pm 0.0170.394 ±\pm 0.0220.300 ±\pm 0.0180.249 ±\pm 0.0090.224 ±\pm 0.010
spectral0.674 ±\pm 0.0170.537 ±\pm 0.0220.376 ±\pm 0.0170.225 ±\pm 0.0060.156 ±\pm 0.0080.121 ±\pm 0.004

Under our protocol, PolarAdamW beats Muon at every width, while in Table 2 of Zhang (2026) Muon is ahead; the difference in protocol listed above is the likely reason. On GotenNet, Muon stays ahead of PolarAdamW on both datasets where we ran it: aspirin (0.280±0.0040.280 \pm 0.004 against 0.311±0.0030.311 \pm 0.003 force MAE) and QM9 (1.36±0.011.36 \pm 0.01 against 1.43±0.041.43 \pm 0.04, in the units of Table 4). These results are in Appendix D.2 (Table 16).

We thank the reviewer again: the proposed experiments strengthened the paper’s core claim, the step size mismatch, with evidence that does not depend on block normalization. We hope our answers address the reviewer’s concerns and would appreciate a reconsideration of the score. We are happy to answer any further questions.

References

Haozhou Zhang. PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation. arXiv:2605.07067, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reviews. We have uploaded a revised PDF of the submission. The changes are marked in the margin with the reviewer they answer (R1=938k, R2=HepZ, R3=GuFk).

The reviewers found the connection between optimization and equivariant networks insightful and often overlooked (R2), the different step sizes of irrep blocks under Adam a valuable perspective (R3), and the mechanism clearly explained (R2). They noted that RMS normalization lowers Adam's force error on all 24 model and dataset pairs and that Figure 5 gives direct evidence that the step sizes of the irrep blocks matter (R1).

The main questions were where the mismatch comes from (R1, R3), whether a separately tuned learning rate gives the same gain (R1), why RMS normalization often beats spectral normalization (R2, R3), and when to use which method (R2, R3). We ran new experiments for each:

  1. Where the mismatch comes from (R1 W2, R3 W1). The mismatch is present during training, not only at the first update, and it grows with width. An ablation that removes the weight sharing of the vector blocks and keeps everything else still shows it. It also grows with the maximum degree in NequIP, and it appears in a C8C_8-equivariant steerable CNN but not in a plain CNN of the same size. The mismatch follows the structure of blocks of different irrep types. See Figure 2b, Section 5.1 and Appendix D.
  2. A separate learning rate per irrep type (R1 W1 + Q1). On five datasets for each of NequIP and GotenNet and on the synthetic task, it gives mixed results: no gain on NequIP, part of the gain on GotenNet, and a clear gain on the synthetic task. Where it helps, it confirms the mismatch independently of block normalization. See Section 5.2 and Appendix D.3.
  3. Spectral normalization with a shape factor (R2 W2, R3 Q1). Both variants fix the size of each block update. They differ in the norm that measures the size. Following the spectral condition of Yang et al. (2024), we normalize each m×nm \times n update to spectral norm ηm/n\eta\sqrt{m/n}. On NequIP, where the matrix shapes generally differ, this brings spectral normalization to the level of RMS normalization. On GotenNet, where almost all matrices are square, the results do not change. This is a mismatch between matrices of different shapes. See Section 4.1, Table 2 and Appendix A.
  4. Another group (R2 W1). On CIFAR-10, spectral block normalization improves the test accuracy of a C8C_8-equivariant steerable CNN. These results are preliminary, and on CIFAR-100 we see no clear gain yet. See Appendix D.1.
  5. More baselines (R1 Q2 + Q3, R3 Q3). Muon on QM9 and particle dynamics, PolarAdamW (Zhang, 2026) on the synthetic task and with GotenNet, Frobenius normalization per block, and LayerNorm on NequIP. See Appendix D.2-D.4.

Changes in the revision:

  • Spectral normalization uses the shape factor by default. Square matrices keep spectral norm η\eta, and every m×nm \times n matrix with m≠nm \neq n is normalized to ηm/n\eta\sqrt{m/n} (Section 4.1 and Appendix A). Table 2 is updated accordingly.
  • Figure 2b shows the mismatch during training. The ablation without weight sharing is in Appendix D (Table 10) and is discussed in Section 5.1. The caption of Figure 4 describes Figure 4a as a diagnostic of Adam's mismatch, not as a ranking of the variants.
  • The abstract, Section 4.1 and Section 6 state that block normalization is one fix for the mismatch, not the only one, and that blocks of different irrep types may need different step sizes. We state this as a limitation and as future work.
  • Section 6 gives guidance on the choice of method: Muon when accuracy matters most, and block normalization to improve Adam at almost no cost.
  • The abstract and Section 6 state the scope (SO(3) and O(3) equivariant models). Section 5.1 credits the testbed of Zhang (2026) and states the differences in protocol. Appendix C.1 gives the equations of the synthetic model.
  • The new experiments are in Appendix D and are discussed in Section 5. Muon results on QM9 and particle dynamics are in Appendix D.4 (Tables 20 and 21) and are referenced in Section 5.3.

We believe these results strengthen the central claim of the paper: under Adam, one learning rate gives irrep blocks of different types different step sizes, and correcting this improves training. We thank the reviewers again and are happy to answer further questions.

References

Maurice Weiler and Gabriele Cesa. General E(2)-Equivariant Steerable CNNs. In Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1911.08251.

Greg Yang, James B. Simon, and Jeremy Bernstein. A Spectral Condition for Feature Learning. arXiv:2310.17813, 2024.

Haozhou Zhang. PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation. arXiv:2605.07067, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.