acceptodds
Under review as a conference paper at ICLR 2027

Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise

Abstract

Preconditioned methods are central to deep learning optimization. Predominant approaches include computationally light diagonal preconditioners such as Adam, which rely on gradient statistics, and second-order methods such as Gauss-Newton (GN), which capture richer curvature information. Seeking the best of both worlds, we disentangle the preconditioner design space into several factors, separating the choice of diagonal scaling (Adam-style versus GN-style) from 1) the _basis choices_ under which the diagonal scaling operates, and 2) the _gradient noises_ from mini-batching. Our theoretical results show that GN's optimality for linear regression no longer holds under a poorly chosen basis, high gradient noises, or a move from linear regression to a non-convex variant of logistic regression. In these settings, Adam-style methods offer genuine advantages over curvature-inspired preconditioning, rather than serving merely as a tractable proxy. Empirical results on synthetic problems and CIFAR-10 support these findings.

Primary area:
optimization
Keywords:
optimization, theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m8PXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper separates preconditioner comparisons into diagonal scaling, basis choice, and gradient noise. It gives theoretical examples where Adam-style scaling can outperform Gauss--Newton (GN), then tests these findings on synthetic tasks, a shallow MLP, CIFAR-10, and a small synthetic Transformer task.

Critical strengths

1、The paper separates basis quality, diagonal scaling, and gradient noise, revealing when their relative performance changes.

2、It gives a concrete non-convex example where GN$^{-1}$ is suboptimal even with population gradients and the GN eigenbasis.

Critical weaknesses

1、The ``Adam'' analysis is idealized and uses β1=0\beta_1=0, so it does not directly capture common AdamW setups.

2、The Transformer experiment is synthetic, and the CIFAR-10 experiment uses a shallow MLP; evidence on modern deep-network training is limited.

3、The Kronecker eigenbasis experiments are substantially more expensive than identity-basis runs, but there is no full wall-clock comparison.

Questions for authors

1、Does the GN slowdown persist with other non-convex parameterizations?

2、Could you compare against standard AdamW with nonzero β1\beta_1?

3、What are the wall-clock costs when basis estimation is included?

1 comment

Thank you very much for your feedback! We’d like to address the concerns below.

Extending the comparison with nonzero β1\beta_1 (W1, Q2): We provide additional empirical results in Sec D.5 of the appendix, which shows that our conclusions continue to hold with nonzero β1\beta_1.
Specifically, Figure 12 shows the result on CIFAR10 learned under the identity basis with population updates, with β1=0.9\beta_1=0.9 following standard practices for Adam. We tune β2\beta_2 and find β2=0.9\beta_2=0.9 to be optimal. Compared to results in our original submission which has β1=0\beta_1=0, using momentum with β1=0.9\beta_1=0.9 makes training more stable for both Adam and GN, and it benefits Adam even more, resulting in a wider gap between Adam and the GN variants.

Wall-clock consideration (W3, Q3): We’d first like to clarify that our comparison is across the Adam-style diagonal scaling and the GN-style diagonal scaling under the same basis for different basis choices. Reducing the cost of estimating the Kronecker basis is orthogonal to our contributions. Despite the current high costs, the comparison under the eigenbasis or its Kronecker approximation is worth studying, as it demonstrates the potential of better optimizer design [1,2].
For the actual wall-clock time, here are the numbers on CIFAR experiments as a reference: the full-batch experiments take 1) around 10min for GN and 4min for Adam under the identity basis; 2) around 80min for both GN and Adam under the Kronecker basis.

References:

[1] Vyas et al. 24: SOAP: Improving and Stabilizing Shampoo using Adam.

[2] Abreu et al. 25: The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton.

Please let us know if you have any other concerns or suggestions. Thank you very much again!

Reviewer_CMFBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a general form to unify analysis of Gauss-Newton (2nd order) and Adam (1st order) optimizations. And show that under certain conditions, Adam performs better while being cheaper in computation cost.

Critical strengths

The paper gives concrete constructs for comparing GN and Adam, for example formula (4) and section 5.

Critical weaknesses

The empirical validation is performed on toy dataset, and it's not certain whether the shown phenomena are still relevant for more real world cases. This is of particular concern as methods like Muon have been tested on large scale experiments, not to say Adam.

Questions for authors

No particular questions.

Reviewer_ZBkaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether the more expensive curvature information used by Gauss-Newton (GN) methods is always better than the gradient-statistics-based diagonal preconditioning used by Adam, and if not, when it fails. Writing a preconditioner as P=UDpU⊤P=UD^pU^\top, the authors separate the choice of diagonal scaling (Adam-style, or GN-style with p∈{−1/2,−1}p\in\lbrace -1/2,-1\rbrace) from two other factors: the basis UU in which the scaling operates (the identity basis or the GN eigenbasis), and the gradient noise from mini-batching (population vs. stochastic gradients).

Taking the optimality of GN⁻¹ for linear regression as a reference point, the authors show how this optimality breaks down as the basis, the noise, and the problem structure change. First, with population gradients but a poorly chosen basis, they construct an example with a block-structured input covariance in which the identity basis hides the true curvature. In this example both GN⁻¹ and GN⁻¹ᐟ² reduce to plain gradient descent, while Adam-style scaling "auto-tunes" to the curvature of each block through gradient-norm normalization and converges faster. They also show that under a misaligned basis, which of GN⁻¹ and GN⁻¹ᐟ² is better depends on the data covariance. Second, for linear regression with Gaussian inputs and single-sample stochastic gradients, Adam-style scaling and GN⁻¹ᐟ² agree up to a loss-dependent constant in any basis, so the auto-tuning advantage disappears. Based on their convergence bound, basis alignment still matters: GN⁻¹ is preferable in the correct eigenbasis but can be worse than GN⁻¹ᐟ² in the wrong one. Third, for a non-convex, squared-parameterized variant of logistic regression, even with population gradients and the correct eigenbasis, when step sizes are non-increasing and the input distribution is highly imbalanced, GN⁻¹ must use a small final step size to converge globally from the given initialization, which causes a slowdown polynomial in the dimension; Adam-style scaling avoids this. The authors conclude that in these settings Adam-style scaling offers genuine advantages rather than serving merely as a cheap proxy for GN. Empirically, the authors verify that Adam-style scaling can outperform GN⁻¹ with full batches on synthetic quadratic and logistic problems, and that the relative performance of GN⁻¹ and GN⁻¹ᐟ² depends on the data covariance. On MLP tasks (including CIFAR-10), Adam and GN⁻¹ᐟ² behave similarly at small batch sizes regardless of the basis, and Adam is close to or better than GN⁻¹ when the basis is misaligned. On a one-layer Transformer, Adam still outperforms GN⁻¹ even with the eigenbasis and full batches.

Critical strengths
  1. A clear question and a useful decomposition. The paper asks a concrete question, namely whether the curvature information in Gauss-Newton is always better than the gradient statistics used by Adam, and answers it by separating the preconditioner P=UDpU⊤P=UD^pU^\top into the diagonal scaling, the basis it operates in, and the gradient-noise regime. This separation is useful because it explains why existing comparisons between Adam-type and GN-type methods can disagree: the conclusion depends on which of these factors changes. It also gives practical guidance for optimizer design, for example that the benefit of GN-style scaling relies on a well-aligned basis, and that at small batch sizes Adam-style scaling behaves like GN⁻¹ᐟ². This is the main reason I view the paper favorably.

  2. Theory and experiments are closely matched. Each claim in Table 1 is supported by a constructive example or a theorem: the auto-tuning example in Section 3.1, the equivalence between Adam-style scaling and GN⁻¹ᐟ² in Section 4 (Lemma 1, Corollary 1), and the lower bound for GN⁻¹ on the non-convex logistic problem in Section 5 (Theorem 2). The experiments are designed to test these specific predictions, covering synthetic problems, MLPs on the full basis × batch-size grid, basis interpolation, and a Transformer task. The authors are also careful about scope; for example, they explain why Theorem 2 does not contradict the line-search results of Abreu et al. (2025). This makes the conclusions credible within the settings studied.

Critical weaknesses
  1. Several comparisons are not fully rigorous.

    • The comparison between GN⁻¹ and GN⁻¹ᐟ² in Section 4 is based on the upper bound in Lemma 2. Its proof contracts the potential ∑ivi/λi(A(P))\sum_i v_i/\lambda_i(A(P)) rather than the loss, so converting back to E[ℓ]\mathbb{E}[\ell] introduces a factor of order κ(A(P))\kappa(A(P)) that is hidden in O(⋅)O(\cdot). This factor depends on PP, which is exactly what is being compared. Comparing upper bounds also does not establish that one method is actually faster.
    • In Section 5, the O(log⁡(1/ϵ))O(\log(1/\epsilon)) rate for Adam-style scaling is argued informally rather than proved, so the claimed separation from GN⁻¹ is only half established.
    • The last step of the proof of Theorem 2 uses the lower bound λmin⁡≥νmin⁡/c1\lambda_{\min}\ge\nu_{\min}/c_1, but an upper bound on λmin⁡\lambda_{\min} is needed to keep the inequality direction. The result can be fixed, since λmin⁡≤c2νmin⁡\lambda_{\min}\le c_2\nu_{\min} also holds when Pi∈[0.6,0.8]P_i\in[0.6,0.8], but the proof as written is incorrect.

    These points matter because the paper's main contribution is theoretical, and the separations in Table 1 are only as strong as these arguments.

  2. The settings are narrow, so the practical implications are uncertain. The theory covers linear regression with Gaussian inputs and a specially constructed non-convex logistic problem with one-hot inputs, highly imbalanced ν\nu, a specific initialization, and non-increasing step sizes. Common schedules with warmup are not non-increasing, so they fall outside Theorem 2. The experiments use small models (two-layer MLPs, a one-layer Transformer) and full or Kronecker-approximated GN eigenbases, and they do not account for compute cost. The paper's main practical hypothesis, that Adam-style scaling behaves like GN⁻¹ᐟ² in large-scale training because variance dominates, is left untested. It is therefore unclear how much these results should change how optimizers are designed or chosen in practice.

Minor: Claim 1 (Appendix B.3) is an existence statement supported by a numerically found covariance matrix. This is acceptable in principle, but the specific Σx\Sigma_x is not reported, so the example cannot be checked, and there is no characterization of when GN⁻¹ᐟ² is better conditioned than GN⁻¹.

Questions for authors
  1. Were GN and Adam given the same learning-rate schedules? Section 6 states that the learning-rate schedule (constant or step decay) is swept for Adam, but neither the main text nor Appendix D.1 says whether the same schedules were considered for GN⁻¹ and GN⁻¹ᐟ². The auto-tuning argument in Section 3.1 itself relies on decaying the learning rate after a burn-in period. In the MLP and Transformer experiments (Figs. 3–6), which involve stochastic gradients or non-convex losses, a decaying schedule can also benefit GN. Could the authors clarify the schedules used for GN? If GN was run only with a constant learning rate, part of Adam's advantage in these experiments may come from the schedule rather than from the diagonal scaling, and results with matched schedules would be needed.

  2. Do the conclusions extend beyond the idealized Adam-style scaling? The analysis in Sections 3.1 and 5 uses the scaling in (2), which with population gradients reduces to ∣ui⊤∇L∣−1\vert u_i^\top\nabla L\vert^{-1}, i.e., normalized or sign gradient descent with β2=0\beta_2=0, and the authors attribute the auto-tuning effect to this normalization by the mean gradient. In the experiments, however, β2\beta_2 is selected from {0,0.9,0.95,0.99}\lbrace 0, 0.9, 0.95, 0.99\rbrace, and β1\beta_1 is fixed to 0 for all methods. Which values of β2\beta_2 were selected in the reported results, and does the auto-tuning mechanism still hold when the second moment is an EMA over past gradients? It would also help to know whether the conclusions hold with momentum (β1>0\beta_1>0), as Adam is typically used in practice. If not, the claims about "Adam-style" methods should be stated more narrowly.

If the authors can provide reasonable explanations and supporting evidence for these questions, and revise or clarify the points raised in the weaknesses, I would be happy to further raise my score.

5 comments

Dear authors,

I would appreciate a response to my review; if some of my main concerns remain unaddressed at the end, I may not be able to maintain my current score.

Thank you very much for your careful read and constructive feedback! We'd like to address the concerns below, and have highlighted the changes in blue in the revised paper.

Making comparisons fully rigorous (W1): We have added clarifications and corrected typos. Specifically:

  • Sec 4’s comparison of GN−1\text{GN}^{-1} vs GN−1/2\text{GN}^{-1/2}: The reviewer is correct that the comparison is based on the upper bound in Lemma 2, which we find to be qualitatively consistent with the empirical results but is not a separation in the actual convergence rate. We have updated the revision (line 296-298) to make this clear.
  • Sec 5, making Adam-style scaling’s convergence formal: We have updated the revised paper to include a Proposition 4 in Section C.1 of the appendix.
  • In Theorem 2’s proof, λmin⁡\lambda_{\min} needs an upper bound rather than a lower bound: thank you for catching this! We have corrected this typo in the revised paper.

There were also concerns about whether the conclusions hold in more general settings, such as using learning rate schedules (step decay and warmup) and non-zero β1,β2\beta_1, \beta_2. We answer this affirmatively below; the empirical evidence can be found in the newly added Section D.5 of the appendix.

GN with learning rate schedule (Q1) and warmup (W2): The reviewer is correct that we did not use learning rate decay for GN in the original submission. However, we note that this is not to give Adam an unfair advantage, but because step decay does not benefit GN: the challenge with GN is that the learning rate cannot be sufficiently large, hence decreasing the learning rate leads to further slow down. More specifically, for the two examples associated with theory:

  • For the auto-tuning example in Sec 3.1, GN behaves as vanilla gradient descent and requires Θ(dlog⁡(1/ϵ))\Theta(d\log(1/\epsilon))steps, where dd comes from the condition number of Σx\Sigma_x. The reason is that the convergence speed is limited by η=O(1/d)\eta=O(1/d), and decaying η\eta will slow down the convergence further. In contrast, Adam converges in Θ(log⁡(d/ϵ))\Theta(\log(d/\epsilon)) steps.
  • Similarly, for logistic regression considered in Sec 5 (Theorem 2), GN is bottlenecked by insufficiently large learning rate.

We accompany these claims with empirical results. In Figure 11 of the paper revision, we additionally tune GN with 1) step decay, 2) warmup, and 3) warmup followed by step decay, and neither step decay nor warmup help with convergence, consistent with our theory. Fig 11 also includes results on CIFAR and Transformer. Across settings, we see that step decay slows down GN’s convergence. Smaller step decay intervals are slower, due to smaller learning rates.

Extending the conclusions beyond the idealized Adam-style scaling (Q2): We empirically confirm that the advantage of Adam-style scaling holds even with nonzero β1,β2\beta_1, \beta_2.

  • The best β2\beta_2 values: We tune β2\beta_2 for all experiments. Below we report the best β2\beta_2 values for the CIFAR10 results, which generalizes the conclusion on linear regression:

    Adam β2\beta_2Population regimeStochastic regime
    Identity basis0.9 (≈0.95)\approx 0.95)0.9 (≈0.95)\approx 0.95)
    Eigenbasis00.9

    For the identity basis, Adam with β2=0.95\beta_2= 0.95 behaves similarly to β2=0.9\beta_2 = 0.9. Adam outperforms GN$^{-1}$ and GN$^{-1/2}$ in both stochastic and population regime even with non-zero β2\beta_2, suggesting that the auto-tuning effect holds.

  • Auto-tuning with β1≠0\beta_1 \neq 0: We additionally experiment with a non-zero momentum. We use β1=0.9\beta_1=0.9 following standard practices for Adam. We tune β2\beta_2 and find β2=0.9\beta_2=0.9 to be optimal. Figure 12 of the revised paper shows the result on CIFAR10 learned under the identity basis with population updates: using momentum with β1=0.9\beta_1=0.9 makes training more stable for both Adam and GN, and it benefits Adam even more, resulting in a wider gap between Adam and the GN variants.

We hope our response helps address your concerns, and please let us know if you have further questions. Thank you again!

Dear reviewer,

Thank you for engaging in the discussion! We really appreciate your time and effort in your review, and sincerely apologize that it took longer than expected for us to get the response ready.

We just posted a response; please let us know if you have further questions or suggestions. Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.