- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper uses the fact that a circular convolution is block-diagonalised by the DFT, so the layer splits into one matrix per frequency. The authors define frequency-decoupled optimisers, whose update for each Fourier block depends only on that block’s own gradient and state. They give exact criteria for decoupling and show that SGD satisfies them, while Adam and Muon couple frequencies through their preconditioners. They then propose a generic construction that runs any optimiser independently per frequency, yielding Fourier-Adam, CommMuon and LU-Muon at the same memory cost. Experiments on FNO, ViT, DDPM and ResNet-50 show lower error, most clearly in FNO rollouts, and changed layer spectra.
Critical strengths
-
General, rigorous framework. The theory needs only a finite abelian group acting unitarily on the parameters, so it covers standard and group convolutions as well as FNO spectral layers. Proofs are complete, including the real-parameter case and Newton–Schulz variants.
-
Exact characterisation of existing optimisers. Necessary-and-sufficient criteria classify SGD, Adam, Muon, matrix AdaGrad and clipping precisely. For each non-decoupled method, they also identify the specific coupling mechanism.
-
Constructive, general recipe. Localisation turns any source optimiser into a decoupled one. The compact-state reduction keeps memory equal to the source, so the recipe is practical, not just theoretical.
-
Strong results where the theory is exact. On FNO, where weights are stored per Fourier mode, the localised optimisers consistently beat their sources across seeds. With pushforward training, LU-Muon cuts 20-step rollout error from about 55% to 41%.
-
Clear, reproducible algorithms. Every method comes with pseudocode, initialisation and bias correction. The derivation procedure in Appendix G.7 lets readers localise other optimisers themselves.
Critical weaknesses
-
The theory doesn’t match most experiments. The block-diagonalisation is exact only when the kernel and the feature map share the same cyclic group, as in FNO. For ResNet, DDPM and ViT, the authors take the DFT of the small k×k kernel, not of the feature map. That is a different symmetry from the one the layer actually has, so outside FNO the method is a heuristic.
-
No link between decoupling and better optimisation. Independence requires a loss that splits across frequencies, which never holds in nonlinear networks. Nothing shows why decoupling should speed up training or improve the solution found, so the title’s claim rests entirely on experiments.
-
Weak experimental rigour and missing baselines.
The DDPM comparison is a single, untuned run. The ResNet-50 Adam baseline (66.7% top-1) looks under-tuned. In ViT, the two learning rates differ by about 300×. SGD, which the paper itself proves is decoupled, is never compared, and K-FAC and Shampoo are absent from the main results.
Questions for authors
-
For standard CNNs, why use the DFT of the k×k kernel rather than the feature-map DFT that actually diagonalises the layer? How do results change with the latter?
-
SGD is already decoupled. How does it compare with Fourier-Adam and LU-Muon, and do the gains hold against a well-tuned Adam on ResNet-50?
4 comments
W3/Q2. Weak experimental rigour and missing baselines...
We would like to clarify the goal and design of our experimental evaluation.
SGD is not a baseline. Our objective is to isolate the effect of frequency decoupling by comparing each non-decoupled source optimizer directly against its localized counterpart. We do not claim that a decoupled version of one optimizer family must outperform an existing optimizer from a completely different family. For instance, we do not claim that Fourier-Adam outperforms Muon. Regarding SGD: as proved in Proposition 4.5, standard SGD is already frequency-decoupled. Consequently, applying our localization construction (Theorem 4.8) to SGD simply returns SGD itself.
Experimental rigor across benchmarks. All experimental protocols are deliberately fixed between each source optimizer and its localized counterpart to ensure a fair, controlled comparison:
- DDPM: In the revised paper (Section 5.4, Table 5), we have updated the DDPM evaluation to report multi-seed statistics (), confirming that Fourier-Adam consistently improves test MSE, FID, and stable rank.
- ViT learning rates: Both Adam and Fourier-Adam learning rates were tuned via the exact same grid-search protocol (Appendix A.1.2). The difference in optimal learning rates is a direct theoretical consequence of Proposition G.1 (Appendix G.2): under the orthonormal DFT of an -element kernel, replica localization rescales the effective step size and damping by ().
- ResNet-50: The ResNet-50 experiment was designed as a controlled fixed-budget comparison (200 epochs, identical cosine schedule and hyperparameters, no data augmentation) specifically to test whether replacing Adam with Fourier-Adam under identical conditions improves training dynamics, yielding a top-1 accuracy gain ( vs. ). We also report additional ResNet-50 results on ImageNet for Adam with the learning rate tuned via grid search. Even the best Adam configuration on this task does not reach the final accuracy of Fourier-Adam, and it still exhibits worse training dynamics in the early epochs than our method (please, see Table 6).
References
[1] H. Li et al. Convergence of Adam under relaxed assumptions, NeurIPS 2023.
[2] Riabinin et al. Gluon: Making muon & scion great again!.
[3] Saxe et al. Exact solutions to the nonlinear dynamics, ICLR 2014.
[4] Aarts G et al. Stochastic weight matrix dynamics, Physical Review E. 2025.
[5] Olsen et al. From SGD to spectra.
Dear Reviewer emYW!
Thank you again for your constructive feedback. We have provided detailed responses to your questions and concerns, along with additional clarifications and revisions.
As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain. We would be happy to provide further clarification.
Thank you for your time and consideration!
Thank you for the rebuttal: follow-up on scope, mechanism and protocol
Thank you for the detailed response and the new experiments. The multi-seed DDPM results, the tuned Adam grid and Appendix A.2.6 are useful. My rating is unchanged, for three reasons.
W1, Scope: You agree the method is a heuristic outside the shared-grid setting, but the title, abstract and Section 5 still present a general result for convolutional layers, and the main text never states the limitation. Table 14 shows no gain from localisation on a standard CNN: Fourier-Adam matches Adam (93.02 and 93.11 vs 93.09) and LU-Muon is below Muon (93.70 and 93.93 vs 94.04) on both grids. W2, Mechanism: Appendix B.5 shows the localised methods are no worse in the worst case, under exact gradients and, for the Muon variants, no momentum. It does not show that decoupling helps. Section 5.2 states that Adam is already decoupled on FNO and that its gap to Fourier-Adam reflects geometry, yet Table 2 reports a 6-point pushforward gain for that pair. The negative cases (per-frequency Shampoo in Table 7, LU-Muon on ViT, Fourier-Adam on KdV) deserve discussion in the main text.
W3, Protocol: The full protocol used in the different experiments is unclear. Appendix A.1.1 applies gradient clipping in the Muon runs only and uses different Newton–Schulz settings for Muon and LU-Muon, so the protocols are not identical. Table 6 lacks standard deviations and the selected learning rate, and Adam’s train loss exceeds its validation loss after 200 epochs; please give the full protocol.
I would reconsider if the title and claims were rescoped to spectral or shared-grid layers, with the heuristic, protocol and limitations, stated in the main text, and if a control separated frequency decoupling from a generic change of basis (for example, Adam in a random orthogonal basis of the kernel).
Dear Reviewer emYW!
We thank Reviewer emYW for the high evaluation of our theoretical framework as well as for the constructive comments. Below we address all concerns in detail.
We appreciate this observation. Our formal framework is exact when the kernel and feature map share the same cyclic group action (e.g. applying DFT on the same grid). Outside of this exact setting we view this as a tractable, theory-motivated heuristic for standard spatial convolutions. Furthermore in Appendix A.2.6 we provide experimental evidence that applying DFT on feature-map grid yields similar results to the kernel grid thus demonstrating the broader utility of frequency localization. Moreover, in practice, using the feature-map grid leads to a substantial increase in the required computational resources (please, see Appendix A.2.6), which fully justifies the use of the heuristic approach in our experiments.
We agree that in non-linear networks the loss does not split across frequencies, meaning that the link between frequency decoupling and improved end-to-end optimization is empirical and heuristic. However, in the newly added Appendix B.5, we formally prove that the convergence rates of our decoupled optimizers match those of their coupled source counterparts without requiring the splitting-loss condition. We also note that analyzing exact spectral dynamics in simplified settings (such as linear layers or deep linear networks without non-linear activation functions) and then validating the resulting methods empirically on full non-linear architectures is standard practice in the literature [3, 4, 5].
Furthermore, in the newly added Appendix A.2.5 (Tables 12 and 13), we provide additional empirical results examining how each optimizer recovers a trained network after a random weight perturbation. Specifically, we take trained FNO checkpoints, perturb every trainable tensor with relative noise, and fine-tune the perturbed models from a zero optimizer state using either coupled Muon or decoupled LU-Muon:
We kindly ask to see the continuation of the answer in the next Official Comment.