The paper asks whether the more expensive curvature information used by Gauss-Newton (GN) methods is always better than the gradient-statistics-based diagonal preconditioning used by Adam, and if not, when it fails. Writing a preconditioner as P=UDpU⊤, the authors separate the choice of diagonal scaling (Adam-style, or GN-style with p∈{−1/2,−1}) from two other factors: the basis U in which the scaling operates (the identity basis or the GN eigenbasis), and the gradient noise from mini-batching (population vs. stochastic gradients).
Taking the optimality of GN⁻¹ for linear regression as a reference point, the authors show how this optimality breaks down as the basis, the noise, and the problem structure change. First, with population gradients but a poorly chosen basis, they construct an example with a block-structured input covariance in which the identity basis hides the true curvature. In this example both GN⁻¹ and GN⁻¹ᐟ² reduce to plain gradient descent, while Adam-style scaling "auto-tunes" to the curvature of each block through gradient-norm normalization and converges faster. They also show that under a misaligned basis, which of GN⁻¹ and GN⁻¹ᐟ² is better depends on the data covariance. Second, for linear regression with Gaussian inputs and single-sample stochastic gradients, Adam-style scaling and GN⁻¹ᐟ² agree up to a loss-dependent constant in any basis, so the auto-tuning advantage disappears. Based on their convergence bound, basis alignment still matters: GN⁻¹ is preferable in the correct eigenbasis but can be worse than GN⁻¹ᐟ² in the wrong one. Third, for a non-convex, squared-parameterized variant of logistic regression, even with population gradients and the correct eigenbasis, when step sizes are non-increasing and the input distribution is highly imbalanced, GN⁻¹ must use a small final step size to converge globally from the given initialization, which causes a slowdown polynomial in the dimension; Adam-style scaling avoids this. The authors conclude that in these settings Adam-style scaling offers genuine advantages rather than serving merely as a cheap proxy for GN. Empirically, the authors verify that Adam-style scaling can outperform GN⁻¹ with full batches on synthetic quadratic and logistic problems, and that the relative performance of GN⁻¹ and GN⁻¹ᐟ² depends on the data covariance. On MLP tasks (including CIFAR-10), Adam and GN⁻¹ᐟ² behave similarly at small batch sizes regardless of the basis, and Adam is close to or better than GN⁻¹ when the basis is misaligned. On a one-layer Transformer, Adam still outperforms GN⁻¹ even with the eigenbasis and full batches.
Thank you very much for your feedback! We’d like to address the concerns below.
Extending the comparison with nonzero β1 (W1, Q2): We provide additional empirical results in Sec D.5 of the appendix, which shows that our conclusions continue to hold with nonzero β1.
Specifically, Figure 12 shows the result on CIFAR10 learned under the identity basis with population updates, with β1=0.9 following standard practices for Adam. We tune β2 and find β2=0.9 to be optimal. Compared to results in our original submission which has β1=0, using momentum with β1=0.9 makes training more stable for both Adam and GN, and it benefits Adam even more, resulting in a wider gap between Adam and the GN variants.
Wall-clock consideration (W3, Q3): We’d first like to clarify that our comparison is across the Adam-style diagonal scaling and the GN-style diagonal scaling under the same basis for different basis choices. Reducing the cost of estimating the Kronecker basis is orthogonal to our contributions. Despite the current high costs, the comparison under the eigenbasis or its Kronecker approximation is worth studying, as it demonstrates the potential of better optimizer design [1,2].
For the actual wall-clock time, here are the numbers on CIFAR experiments as a reference: the full-batch experiments take 1) around 10min for GN and 4min for Adam under the identity basis; 2) around 80min for both GN and Adam under the Kronecker basis.
References:
[1] Vyas et al. 24: SOAP: Improving and Stabilizing Shampoo using Adam.
[2] Abreu et al. 25: The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton.
Please let us know if you have any other concerns or suggestions. Thank you very much again!