Enforcing Independent Spectral Dynamics Improves Optimization of Convolutional Layers

Commonly used optimizers such as Adam implicitly treat each parameter block as a flat vector, ignoring the multilinear structure of the weights in many modern machine learning models. Recent works has shown that exploiting matrix structure can lead to better optimization dynamics. However, these results are largely confined to matrix-valued parameters. The dynamics of layers whose weights are not naturally matrices as convolutional kernels remain poorly understood and standard optimizers do not account for their structure. In this paper, we study the connection between convolutional kernels and circulant matrices and, exploiting their diagonalization by the Fourier transform, we propose a new class of $\text{frequency-decoupled}$ optimizers with independent spectral evolution. We further analyze existing optimizers within our theoretical framework and propose a method for constructing new decoupled optimizers from the geometry of existing ones. Finally we show that decoupled optimizers provide other great spectral properties allowing layer compression and show overall better performance on machine learning models utilizing convolutional layers (e.g. CNNs, Neural Operators and Diffusion models) at the cost of computational complexity.