acceptodds
Under review as a conference paper at ICLR 2027

Enforcing Independent Spectral Dynamics Improves Optimization of Convolutional Layers

Abstract

Commonly used optimizers such as Adam implicitly treat each parameter block as a flat vector, ignoring the multilinear structure of the weights in many modern machine learning models. Recent works has shown that exploiting matrix structure can lead to better optimization dynamics. However, these results are largely confined to matrix-valued parameters. The dynamics of layers whose weights are not naturally matrices as convolutional kernels remain poorly understood and standard optimizers do not account for their structure. In this paper, we study the connection between convolutional kernels and circulant matrices and, exploiting their diagonalization by the Fourier transform, we propose a new class of optimizers with independent spectral evolution. We further analyze existing optimizers within our theoretical framework and propose a method for constructing new decoupled optimizers from the geometry of existing ones. Finally we show that decoupled optimizers provide other great spectral properties allowing layer compression and show overall better performance on machine learning models utilizing convolutional layers (e.g. CNNs, Neural Operators and Diffusion models) at the cost of computational complexity.

Primary area:
optimization
Keywords:
training dynamics, convolutional layers, structure-aware optimization, spectral analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_emYWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper uses the fact that a circular convolution is block-diagonalised by the DFT, so the layer splits into one matrix per frequency. The authors define frequency-decoupled optimisers, whose update for each Fourier block depends only on that block’s own gradient and state. They give exact criteria for decoupling and show that SGD satisfies them, while Adam and Muon couple frequencies through their preconditioners. They then propose a generic construction that runs any optimiser independently per frequency, yielding Fourier-Adam, CommMuon and LU-Muon at the same memory cost. Experiments on FNO, ViT, DDPM and ResNet-50 show lower error, most clearly in FNO rollouts, and changed layer spectra.

Critical strengths
  1. General, rigorous framework. The theory needs only a finite abelian group acting unitarily on the parameters, so it covers standard and group convolutions as well as FNO spectral layers. Proofs are complete, including the real-parameter case and Newton–Schulz variants.

  2. Exact characterisation of existing optimisers. Necessary-and-sufficient criteria classify SGD, Adam, Muon, matrix AdaGrad and clipping precisely. For each non-decoupled method, they also identify the specific coupling mechanism.

  3. Constructive, general recipe. Localisation turns any source optimiser into a decoupled one. The compact-state reduction keeps memory equal to the source, so the recipe is practical, not just theoretical.

  4. Strong results where the theory is exact. On FNO, where weights are stored per Fourier mode, the localised optimisers consistently beat their sources across seeds. With pushforward training, LU-Muon cuts 20-step rollout error from about 55% to 41%.

  5. Clear, reproducible algorithms. Every method comes with pseudocode, initialisation and bias correction. The derivation procedure in Appendix G.7 lets readers localise other optimisers themselves.

Critical weaknesses
  1. The theory doesn’t match most experiments. The block-diagonalisation is exact only when the kernel and the feature map share the same cyclic group, as in FNO. For ResNet, DDPM and ViT, the authors take the DFT of the small k×k kernel, not of the feature map. That is a different symmetry from the one the layer actually has, so outside FNO the method is a heuristic.

  2. No link between decoupling and better optimisation. Independence requires a loss that splits across frequencies, which never holds in nonlinear networks. Nothing shows why decoupling should speed up training or improve the solution found, so the title’s claim rests entirely on experiments.

  3. Weak experimental rigour and missing baselines.

The DDPM comparison is a single, untuned run. The ResNet-50 Adam baseline (66.7% top-1) looks under-tuned. In ViT, the two learning rates differ by about 300×. SGD, which the paper itself proves is decoupled, is never compared, and K-FAC and Shampoo are absent from the main results.

Questions for authors
  1. For standard CNNs, why use the DFT of the k×k kernel rather than the feature-map DFT that actually diagonalises the layer? How do results change with the latter?

  2. SGD is already decoupled. How does it compare with Fourier-Adam and LU-Muon, and do the gains hold against a well-tuned Adam on ResNet-50?

4 comments

Dear Reviewer emYW!

We thank Reviewer emYW for the high evaluation of our theoretical framework as well as for the constructive comments. Below we address all concerns in detail.

W1/Q1. The theory doesn't match most experiments...

We appreciate this observation. Our formal framework is exact when the kernel and feature map share the same cyclic group action (e.g. applying DFT on the same grid). Outside of this exact setting we view this as a tractable, theory-motivated heuristic for standard spatial convolutions. Furthermore in Appendix A.2.6 we provide experimental evidence that applying DFT on feature-map grid yields similar results to the kernel grid thus demonstrating the broader utility of frequency localization. Moreover, in practice, using the feature-map grid leads to a substantial increase in the required computational resources (please, see Appendix A.2.6), which fully justifies the use of the heuristic approach in our experiments.

W2. No link between decoupling and better optimisation...

We agree that in non-linear networks the loss does not split across frequencies, meaning that the link between frequency decoupling and improved end-to-end optimization is empirical and heuristic. However, in the newly added Appendix B.5, we formally prove that the convergence rates of our decoupled optimizers match those of their coupled source counterparts without requiring the splitting-loss condition. We also note that analyzing exact spectral dynamics in simplified settings (such as linear layers or deep linear networks without non-linear activation functions) and then validating the resulting methods empirically on full non-linear architectures is standard practice in the literature [3, 4, 5].

Furthermore, in the newly added Appendix A.2.5 (Tables 12 and 13), we provide additional empirical results examining how each optimizer recovers a trained network after a random weight perturbation. Specifically, we take trained FNO checkpoints, perturb every trainable tensor with relative noise, and fine-tune the perturbed models from a zero optimizer state using either coupled Muon or decoupled LU-Muon:

  1. Consistent recovery across noise levels: After fine-tuning, LU-Muon achieves the lowest mean test rollout error in every noisy configuration across both tables, and stays closest to its unperturbed (clean) control at α=0.3\alpha = 0.3 in all six source–task blocks.
  2. Robustness of the learned solutions: When starting from a perturbed Adam checkpoint, fine-tuning with LU-Muon reaches a noticeably lower rollout error than fine-tuning with coupled Muon. Conversely, the largest gaps relative to the clean control group arise when a checkpoint trained by LU-Muon is fine-tuned with a non-localized optimizer, suggesting that decoupled updates both guide the model toward more stable solutions and preserve them better after perturbations.

We kindly ask to see the continuation of the answer in the next Official Comment.

W3/Q2. Weak experimental rigour and missing baselines...

We would like to clarify the goal and design of our experimental evaluation.

SGD is not a baseline. Our objective is to isolate the effect of frequency decoupling by comparing each non-decoupled source optimizer directly against its localized counterpart. We do not claim that a decoupled version of one optimizer family must outperform an existing optimizer from a completely different family. For instance, we do not claim that Fourier-Adam outperforms Muon. Regarding SGD: as proved in Proposition 4.5, standard SGD is already frequency-decoupled. Consequently, applying our localization construction (Theorem 4.8) to SGD simply returns SGD itself.

Experimental rigor across benchmarks. All experimental protocols are deliberately fixed between each source optimizer and its localized counterpart to ensure a fair, controlled comparison:

  • DDPM: In the revised paper (Section 5.4, Table 5), we have updated the DDPM evaluation to report multi-seed statistics (mean±SD\text{mean} \pm \text{SD}), confirming that Fourier-Adam consistently improves test MSE, FID, and stable rank.
  • ViT learning rates: Both Adam and Fourier-Adam learning rates were tuned via the exact same grid-search protocol (Appendix A.1.2). The difference in optimal learning rates is a direct theoretical consequence of Proposition G.1 (Appendix G.2): under the orthonormal DFT of an NN-element kernel, replica localization rescales the effective step size and damping by N\sqrt{N} (ηtsrc=ηt/N\eta_t^{\mathrm{src}} = \eta_t / \sqrt{N}).
  • ResNet-50: The ResNet-50 experiment was designed as a controlled fixed-budget comparison (200 epochs, identical cosine schedule and hyperparameters, no data augmentation) specifically to test whether replacing Adam with Fourier-Adam under identical conditions improves training dynamics, yielding a +5.73%+5.73\% top-1 accuracy gain (72.41%72.41\% vs. 66.68%66.68\%). We also report additional ResNet-50 results on ImageNet for Adam with the learning rate tuned via grid search. Even the best Adam configuration on this task does not reach the final accuracy of Fourier-Adam, and it still exhibits worse training dynamics in the early epochs than our method (please, see Table 6).

References

[1] H. Li et al. Convergence of Adam under relaxed assumptions, NeurIPS 2023.

[2] Riabinin et al. Gluon: Making muon & scion great again!.

[3] Saxe et al. Exact solutions to the nonlinear dynamics, ICLR 2014.

[4] Aarts G et al. Stochastic weight matrix dynamics, Physical Review E. 2025.

[5] Olsen et al. From SGD to spectra.

Dear Reviewer emYW!

Thank you again for your constructive feedback. We have provided detailed responses to your questions and concerns, along with additional clarifications and revisions.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration!

Reviewer_emYWOpenReview ↗

Thank you for the rebuttal: follow-up on scope, mechanism and protocol

Thank you for the detailed response and the new experiments. The multi-seed DDPM results, the tuned Adam grid and Appendix A.2.6 are useful. My rating is unchanged, for three reasons.

W1, Scope: You agree the method is a heuristic outside the shared-grid setting, but the title, abstract and Section 5 still present a general result for convolutional layers, and the main text never states the limitation. Table 14 shows no gain from localisation on a standard CNN: Fourier-Adam matches Adam (93.02 and 93.11 vs 93.09) and LU-Muon is below Muon (93.70 and 93.93 vs 94.04) on both grids. W2, Mechanism: Appendix B.5 shows the localised methods are no worse in the worst case, under exact gradients and, for the Muon variants, no momentum. It does not show that decoupling helps. Section 5.2 states that Adam is already decoupled on FNO and that its gap to Fourier-Adam reflects geometry, yet Table 2 reports a 6-point pushforward gain for that pair. The negative cases (per-frequency Shampoo in Table 7, LU-Muon on ViT, Fourier-Adam on KdV) deserve discussion in the main text.

W3, Protocol: The full protocol used in the different experiments is unclear. Appendix A.1.1 applies gradient clipping in the Muon runs only and uses different Newton–Schulz settings for Muon and LU-Muon, so the protocols are not identical. Table 6 lacks standard deviations and the selected learning rate, and Adam’s train loss exceeds its validation loss after 200 epochs; please give the full protocol.

I would reconsider if the title and claims were rescoped to spectral or shared-grid layers, with the heuristic, protocol and limitations, stated in the main text, and if a control separated frequency decoupling from a generic change of basis (for example, Adam in a random orthogonal basis of the kernel).

Reviewer_GfdNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new family of optimizers called frequency decoupled optimizers. It characterizes this property through preconditioner structure and constructs localized variants of Adam, Muon, and Shampoo.

Critical strengths
  1. The frequency perspective is under-explored in the literature. The proposed framework clearly distinguishes locality and trajectory, and characterize frequency-separable loss.
  2. The paper gives a clear coupling of Muon.
Critical weaknesses
  1. The locality can be established, but there is critical gap on why localized optimizer have an advantage, both theoretically and experimentally. How can one isolate locality as the source of improvements in the experiments. Some ablations can make the connection more convincing.
  2. The difficult part is to connect the theory to the actual experiments. In the comparison between Adam and Fourier-Adam, both have decoupled between different frequencies, and the benefits of Fourier-Adam cannot be justified using frequency decoupling.
Questions for authors

n.a.

3 comments

We thank Reviewer GfdN for the positive assessment of our theoretical framework and for the constructive comments. Below we address both concerns in detail.

W1. The locality can be established...

We thank the reviewer for this observation. In our original submission, the stated advantage of localized optimizers referred to their empirical performance, as shown across our experiments in Section 5 and Appendix A.2. In the new revision we have added this explicitly in the Abstract. At the same time, we agree that both the theoretical and experimental support for this point needed to be expanded.

In the revised paper, we have elaborated on this aspect in greater detail, fixed several minor typos and inaccuracies.

Theoretical convergence guarantees. We introduce Appendix B.5 where we provide formal convergence rates for the proposed localized optimizers by building on known analyses of their source methods. Specifically, the bound for Fourier-Adam matches the Adam convergence rate, while for the Muon-based methods we prove that each localized step forms the exact linear minimization oracle (LMO) of an induced norm, allowing us to apply the convergence framework of Gluon [1]. Consequently, each localized optimizer converges to a stationary point at the same rate as its source counterpart, with problem constants evaluated in the localized geometry. These bounds establish stationarity rather than a uniform improvement in constants over the source algorithms. Developing a general convergence theory for the entire class of frequency-decoupled optimizers that characterizes when localization strictly improves these constants remains an interesting open problem. It requires techniques beyond reducing to existing source-method analyses, and we leave it for future work.

Extended experimental validation. In the updated Section 5.2 and Appendix A.2 we have also included several additional FNO benchmarks and updated DDPM results with multi-seed statistics (Table 5), providing further evidence for the name of the paper.

W2. The difficult part is to connect the theory to the actual experiments...

We respectfully clarify how our theoretical decoupling results connect to the experimental comparisons between Adam and Fourier-Adam, as well as to our other benchmarks:

Adam on FNO (dedicated exception) vs. non-decoupled Adam in all other experiments. On FNO specifically, we agree with the reviewer: because the FNO architecture stores its spectral weights separately for each Fourier mode, the coordinate-wise update of standard Adam already acts in a frequency-decoupled manner. By construction (Proposition G.1 / Appendix A.1.1), on FNO Adam and Fourier-Adam differ only in how the second moment of a weight is formed. In the revised manuscript, we have explicitly added this as a dedicated exception for Adam in the FNO sections (Sections 5.1 and 5.2). For all other experiments (ViT, DDPM, and ResNet-50), however, standard Adam is not frequency-decoupled, which follows directly from our theory. As formally proven in Proposition 4.6 (Appendix E) and Proposition H.2 for real parameters (Appendix H), spatial Adam is not frequency-decoupled on any convolutional layer with kernel size greater than 1×11 \times 1.

How decoupling is measured and isolated across tasks vs. role of spectral metrics. Empirically, the effect of frequency localization is measured via controlled pairwise ablation: by construction, each localized optimizer is derived directly from its non-decoupled source counterpart via per-frequency replica localization (Section 4.4). The source and localized methods share the same base update rule, initialization, and hyperparameter tuning protocol, and differ only in whether the update acts on each frequency separately. A direct comparison of the non-decoupled baseline with its decoupled counterpart therefore isolates frequency locality as the source of the performance difference.

The only exception to this experimental protocol is the Adam and Fourier-Adam pair when benchmarked on FNO. On the other hand Muon acts on packed blocks of the FNO weights across all frequencies simultaneously Eq. (4) (Appendix E). Thus, comparing Muon with LU-Muon directly isolates locality on FNO (Tables 2–3 and 7–11).

References

[1] - Riabinin et al. Gluon: Making muon & scion great again!

Dear Reviewer GfdN!

We sincerely appreciate your valuable feedback. During the rebuttal period we have provided detailed responses to your questions and concerns, along with additional clarifications and revisions.

Since the discussion period is almost over, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration!

i thank the authors for the replies. 1) thanks for the clarification on the adam and fourier-adam, it makes sense to me. 2) i think my main concern on critical gap on why localized optimizer have an advantage, both theoretically and experimentally still holds after the reply. There is still no theoretical advantages from the added theorems, and experiment wise, for vit (muon vs lu-muon), fno (adam vs fourier-adam), the performance is still suboptimal. although i liked the idea of this localized optimizer, i think with solid improvement theoretically or empirically will make it a stronger case.

Reviewer_az1MOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how optimizers interact with convolutional layers, whose spectra split across Fourier frequencies. It defines frequency-decoupled optimizers whose state and update at each frequency depend only on that frequency. The authors show that when the loss splits by frequency, the state and parameters evolve as independent per-frequency systems. They derive necessary and sufficient criteria for decoupling, prove that Adam, Muon and matrix AdaGrad are coupled, and propose Fourier-Adam, CommMuon and LU-Muon. Optimizers are benchmarked on FNO, ViT, DDPM and ResNet-50.

Critical strengths

The formal framework is rigorous, and the proofs of the main theorems (4.2-4.4, 4.8, 4.9) are sound. Theorem 4.4 gives an exact if-and-only-if criterion at the level of the preconditioner, turning "respects frequency structure" into precise conditions. The paper derives these conditions analytically and, with its corollaries, applies them to SGD, Adam, matrix AdaGrad and Muon. The localization construction then produces explicit algorithms with pseudocode. Its compact forms are correctly derived; for example, Fourier-Adam is exactly localized Adam up to a rescaling of ϵ\epsilon and η\eta.

Critical weaknesses

I have a few problems with the claims made in the abstract, which I thought would be pursued during the rest of the paper:

(1) "Improves optimization" (title). No theorem connects decoupling to convergence, conditioning or any reported spectral metric (C10σC_{10}^{\sigma}, σmax⁡\sigma_{\max}, stable rank rsr_s). The experiments do not test decoupling or measure cross-frequency coupling during training, and spectra are reported only at the selected checkpoint, so independent spectral dynamics are never observed. The claim is therefore purely empirical, and the evidence is mixed: LU-Muon is worse than Muon on ViT, and Fourier-Adam only matches Adam on one-step FNO.

(2) "Allowing layer compression" (abstract). The paper has no compression theorem or experiment, and nothing is pruned, truncated or low-rank approximated. The reported spectra also point in both directions: moving from Adam to Fourier-Adam makes the spectrum more concentrated on ViT (C10σC_{10}^{\sigma} rises from 36.8%36.8\% to 70.5%70.5\%) but less concentrated on DDPM (stable rank rsr_s rises from 48.348.3 to 60.360.3) and ResNet-50 (C10σC_{10}^{\sigma} falls from 65.0%65.0\% to 40.3%40.3\%).

(3) "At the cost of computational complexity" (abstract). This cost is never quantified: there is no complexity analysis or timing, and the "Computational costs" paragraph in Appendix I covers only stored momentum and moment state. Readers therefore cannot weigh the reported gains against the overhead.

Questions for authors

Q1. The paper gives no convergence result. Can you provide a convergence rate theorem showing that decoupled optimizers improve on their source optimizers, or else state clearly that any faster convergence is only an empirical observation?

Q2. Can you show the evolution of per-frequency spectra during training, along with a measure of cross-frequency coupling, for decoupled and source optimizers?

Q3. What evidence supports "allowing layer compression", given that spectra become more concentrated on ViT but less concentrated on DDPM and ResNet-50? Please report the test metric (accuracy, rollout error or FID) after truncating the layers at fixed compression ratios, for each optimizer.

Q4. The abstract says the gains come at a computational cost, so what is the theoretical per-step time complexity of each proposed method compared with its source optimizer? Could you please also report the measured wall-clock time (per training step or per epoch) for every optimizer on at least one experimental setup?

3 comments

Dear Reviewer az1M!

We thank Reviewer az1M for the thoughtful feedback and constructive comments. Below we address each concern in detail.

W1/Q1. Improves optimization. No theorem connects decoupling to convergence...

We appreciate this comment. In the initial submission, we intended our claim regarding advantage of localized optimizers to refer to their practical performance – as evidenced by our empirical results in Section 5 and Appendix A.2. In the new revision of the paper, we have explicitly clarified this in Abstract. We also have paid closer attention to this point by adding a more detailed description and fixing several inaccuracies and typos.

  1. New convergence analysis. We have added Appendix B.5 with convergence guarantees for the localized optimizers. The result for Fourier-Adam follows the analysis of deterministic Adam. For the Muon-like methods we show that each localized update is the exact LMO of a specific norm, and convergence then follows from the framework of Gluon [1]. Each localized method therefore converges to a stationary point at the same rate as the corresponding source method, with the constants measured in the localized geometry. A general convergence theory for the whole family of localized decoupled optimizers, which would explain when localization improves the constants, is an interesting open problem. It requires techniques beyond reducing to existing source-method analyses, and we leave it for future work.

  2. Additional experiments. In the updated Section 5.2 and Appendix A.2, we have also included several additional FNO benchmarks and updated DDPM results with multi-seed statistics (Table 5), providing further evidence for the name of the paper.

W2/Q3. Allowing layer compression. The paper has no compression theorem...

We have removed this claim from the abstract (please see the new version of the paper). We would like to point out that the spectral properties of the weights do not always indicate whether a model can be compressed. For example, the stable rank computes only the ∥W∥F2/∥W∥22\|W\|_F^2 / \|W\|_2^2 ratio and does not describe how the singular values are distributed which is crucial for compression prospects. The options for model compression remain an open question, and it has not been fully understood yet which properties are necessary for successful compression [2, 3]. Frequency decoupling is more likely to give training additional freedom in which local minimum it reaches than to determine the compression properties of that minimum since compression only depends on the final weights which in turn depend not only on the training dynamics but also on the stochasticity of training. In addition, the quality of a compressed model is measured on test data and therefore depends on how the training and test data splits are related. Our framework does not account for these or other such factors. Our results therefore describe only the spectral dynamics of the weights during training, and now we no longer draw any conclusions about weight compression from them.

W3/Q4. At the cost of computational complexity. This cost is never quantified...

We added section I of Appendix, in which we give per-step asymptotical estimates for operation count and optimizer-state memory of every localized method described in the paper (Table 17). We show that most of the methods share these estimates with the source ones e.g. Fourier-Adam keeps the Θ(P)\Theta(P) cost and memory of Adam and adds only the FFTs. However methods like LU-Muon can increase computional cost depending on the chosen unfolding.

In practice, across all our experiments, LU-Muon takes at most 1.6 the time per step of Muon and at most 1.4 times its memory, while Fourier-Adam takes about 2 times the time per step of Adam and about 1.5 times its memory. In both cases the extra memory comes from our unoptimized implementation and not from the methods themselves, whose optimizer state has the same order as that of their sources.

Q2. Can you show the evolution of per-frequency spectra during

Evolution of per-frequency spectra cannot establish independence. A spectral trajectory shows how the singular values of a block change, not whether the change came from its own gradient or from other frequencies. A coupled update can also rotate singular vectors of a block without changing its spectrum. Decoupling is a property of the update rule, and Theorem 4.4 characterizes it exactly. A decoupled optimizer updates each frequency using only its own gradient history, while for the source optimizers the coupling is given in closed form by the off-diagonal blocks of their preconditioners.

References

[1] - Riabinin et al. Gluon: Making muon & scion great again!.

[2] - Martin C., Implicit self-regularization in deep neural networks, Journal of Machine Learning Research.

[3] - Zagitov et al. Rethinking the Role of Tensor Decompositions, EMNLP 2026.

Thank you for the detailed response and the new convergence and complexity results.

  1. Computational cost. Table 17 answers the theoretical part of Q4, but the quoted measurements in rebuttal response (about 2× time and 1.5× memory for Fourier-Adam) are not in the paper, and Table 14 instead shows a 7.7× longer optimizer step (7.7 ms vs 1.0 ms) with equal memory. Please add per-benchmark time and peak-memory measurements, state whether they refer to the optimizer step or the full training step, and explain the 7.7× overhead.

  2. Performance on CNNs. In Table 14 (ResNet-18 on CIFAR-10), Fourier-Adam (93.02±0.16%) does not beat Adam (93.09±0.48%) and LU-Muon (93.70±0.10%) trails Muon (94.04±0.11%), which, together with ViT, conflicts with the abstract's claim of overall better performance on CNNs. Could you explain when decoupling is expected to help?

Thank you for the feedback and the new notions. Below we address both concerns in detail:

  1. Computational cost. Table 17 answers the theoretical part of Q4...
  1. Thank you for pointing out the typo. In our previous answer we meant the total time spent per epoch while the optimizer step takes only a fraction of it. We have already addressed this in Table 14 in the updated version of the article and added Tables 18-19 with practical costs on DDPM and FNO 3D (please see in Appendix I).
  1. Performance on CNNs. In Table 14...
  1. In Table 14 the decoupled optimizers are on a par with their coupled counterparts: Fourier-Adam and Adam are within seed noise (93.02±0.16 vs 93.09±0.48), and LU-Muon on the feature-map grid matches Muon (93.93±0.14 vs 94.04±0.11). On the kernel grid LU-Muon is 0.34 pp lower.

    In ViT, following our protocol, the decoupled optimizers act only on the convolutional patch embedding, so only small differences are expected. Across benchmarks the picture is mixed. Overall greater alignment with the theory leads to improved results (please, see Table 14, where the feature map adds points on average). For example the performance of LU-Muon shown in Table 14 can be explained due to the fact that a zero-padded 3x3 convolution is not diagonal on the kernel grid, whereas the spectral layers of FNO are diagonalized in the Fourier basis. However we do not have a complete explanation for this phenomenon and consider it an interesting open question.

    We believe that another practical key benefit of decoupling is robustness to perturbations of the weights. We provide Appendix A.2.5 (Tables 12 and 13) to prove this claim: we take trained FNO checkpoints, perturb every trainable tensor with relative Gaussian noise, and fine-tune the perturbed models from a fresh optimizer state with either Muon or LU-Muon. LU-Muon reaches the lower error in every setting, and at the largest noise level it stays closest to its noise-free control.

    We must also note that we provide convergence guarantess in Appendix B.5, where we prove that the decoupled methods attain the same convergence rates as their source methods with the only requirement being the loss function smoothness.

Comments on the paper

2 comments

Dear reviewers,

We are now about halfway through the discussion period. Please read through the authors' rebuttal and respond if there are points that remain unclear (or to let them know they have addressed your concerns).

Sincerely,

AC

We thank the reviewers for their thoughtful feedback and constructive suggestions.

We have uploaded a revised version of the article. Below is a summary of the main additions and corrections:

  • Formal Convergence Guarantees (Appendix B.5): We have added a new section establishing convergence rates for the proposed optimization methods.
  • Computational and Memory Complexity Analysis (Appendix I): We have added asymptotic per-step operation counts and optimizer-state memory estimates for every decoupled optimizer alongside its source counterpart, as well as experimental wall-clock time and peak memory measurements across our benchmarks.
  • New FNO Benchmarks (Section 5.2, Appendix A.2): We have expanded our empirical evaluation in the FNO setting by adding 3D compressible Navier-Stokes and four additional PDE benchmarks.
  • Weight Perturbation and Recovery Experiments (Appendix A.2.5): To investigate the stability of the learned minima, we have added noise-recovery experiments on FNO.
  • Feature-Map Grid vs. Kernel Grid DFT Ablation (Appendix A.2.6): To address the distinction between exact layer diagonalization and spatial-kernel localization on standard CNNs, we have added an experimental comparison between applying the DFT on the full feature-map grid and on the k×kk \times k kernel grid.
  • Multi-Seed DDPM Statistics and Tuned ResNet-50 Baselines (Section 5.4, Appendix A.2.1): We have updated the DDPM evaluation to report multi-seed statistics (Table 5) and added grid-search tuned learning-rate results for the Adam baseline on ImageNet ResNet-50 (Table 6).
  • Clarifications of Scope (Abstract): We have updated the abstract to better reflect the scope and contributions of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.