acceptodds
Under review as a conference paper at ICLR 2027

Fine-grained Analysis of Adam's Implicit Bias Near the Manifold

Abstract

Understanding the implicit biases of different optimizers is one of the key challenges in studying the training dynamics of deep learning. However, the implicit bias of Adam, one of the most widely used adaptive gradient methods (AGMs), is still not fully understood theoretically. li2025adam studied the implicit bias of Adam with a second-moment momentum coefficient , a setting they termed the 2-scheme. They proved that, under the 2-scheme, Adam evolves near the manifold of training loss minimizers and implicitly reduces a form of sharpness different from that associated with SGD over an timescale. However, in practice, is much larger than . This leaves an open theoretical question: what is the implicit bias of Adam in the -regime with ? In this work, we resolve this question. We first extend the convergence results of li2025adam to and provide counterexamples showing that such convergence can fail for . We then characterize Adam's implicit bias for by developing a new framework for analyzing the limiting flows of AGMs, including Adam, over long training horizons. This framework generalizes the ideas in katzenberger1991solutions and also provides insight into a broader class of stochastic processes beyond AGMs.

Primary area:
learning theory
Keywords:
deep learning theory, optimizer, implicit bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bjz3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigated the implicit bias of Adam. Specifically, the authors characterize the implicit bias for the scaling 1−β2=Θ(ηα)1-\beta_2=\Theta(\eta^{\alpha}), where β2\beta_2 is the second-order moment parameter, η\eta is the learning rate, and α∈[1,2)\alpha\in [1,2).

Critical strengths

The main strengths of this paper may be some potentially rigorous mathematical derivation.

Critical weaknesses

The main weakness of this paper is that existing literature has investigated the implicit bias of Adam, which significantly harms the novelty of this paper. This authors missed the significant literature "Cattaneo et al., On the implicit bias of Adam. ICML 2024"

Besides, the authors missed many other related works cited in the paper "Cattaneo et al., On the implicit bias of Adam. ICML 2024". The paper overview has significant flaws.

Also, the assumptions in this paper is too restrictive. They simultaneously assume Lipschitzness and boundedness of gradients. However, in existing analysis of Adam, one only needs Lipschitzness of gradients (see e.g., [1, 2]).

Last but not least, the presentation of this paper should be improved. The notions of sharpness and minimizer manifold should be introduced more detailed.

References

[1] Wang et al., Closing the gap between the upper bound and lower bound of adam’s iteration complexity. Neurips 2023

[2] Hong and Lin, On convergence of adam for stochastic optimization under relaxed assumptions. Neurips 2024

Questions for authors

Can you provide some experiments to illustrate and support the theoretical findings in this paper? This is not the weakness of this paper, while it is just the suggestion of this paper.

10 comments
AuthorsOpenReview ↗

Our work and "Cattaneo et al" are distinct settings and distinct analyses of Adam's implicit bias.

We have read the review. We believe that the main concern about novelty stems from a misunderstanding of the distinction between our work and Cattaneo et al. (2024). [1]

[1] follow the implicit gradient regularization (IGR) approach via backward error analysis, with an ODE approximation guarantee over O(η−1)O(\eta^{-1}) iterations and fixed β2\beta_2. Our work, instead, characterizes the noise-driven slow dynamics near a minimizer manifold over O(η−2)O(\eta^{-2}) iterations, under the coupled scaling 1−β2=O(ηα)1-\beta_2=O(\eta^\alpha), 1≤α<21\leq\alpha<2. These are distinct settings and distinct analyses of Adam’s implicit bias, and their results do not establish ours. Our setting directly answers "what kind of minimizers does Adam ultimately choose", which is our motivation.

W1. Relation to Cattaneo et al. [1], “On the Implicit Bias of Adam,” ICML 2024.

We list two key differences between the IGR approach and our Slow-SDE based approach showing that we pursue different settings.

  1. IGR and Slow-SDE have different timescales. [1]'s Theorem 3.1 establishes an O(η2)O(\eta^2) trajectory error for n≤⌊T/η⌋n\leq\lfloor T/\eta\rfloor, with T fixed. This is not an approximation guarantee on the O(η−2)O(\eta^{-2})-iteration horizon studied in our paper. Our work aims at characterizing the accumulated stochastic effects that result in a slow motion along the minimizer manifold, and a preference among minimizers, since this motion lasts O(η−2)O(\eta^{-2}) time and requires an analyses that lasts such a time period. On the other hand, when η→0\eta \to 0, O(η−1)O(\eta^{-1}) is infinitesimal compared to O(η−2)O(\eta^{-2}), so [1]'s results cannot establish ours.

  2. We give different regimes for 1−β21-\beta_2. [1]'s Section 1.1 explicitly keeps the momentum parameters β,ρ\beta, \rho (our β1,β2\beta_1, \beta_2) fixed as η→0\eta\to0. In our setting, β2→1\beta_2\to1 jointly with η→0\eta\to0. Moreover, their approximation constant depends on both the time horizon and the momentum parameters. Consequently, substituting our β2(η)\beta_2(\eta) and extending the horizon does not yield the uniform approximation needed for our results. Our regime of 1−β21-\beta_2 is critical to extending the analysis to an O(η−2)O(\eta^{-2}) timescale.

We will add the missing citation and clarify this relationship. However, an omission in the related-work discussion does not establish a lack of novelty: the cited work [1] does not cover the scaling regimes of β2\beta_2, the long-horizon (O(η−2)O(\eta^{-2})) dynamics, or the phase transition of the implicit bias happening at the 11-scheme.

W2. Assumptions

Above all, the cited results [3,4] focus on the convergence phase which lasts O~(η−1)\tilde{O}(\eta^{-1}) iterations, but they do not consider the subsequent motion along a minimizer manifold or the implicit bias of Adam. Our analysis instead addresses the latter over O(η−2)O(\eta^{-2}) iterations, by extending Katzenberger’s framework [6]. For comparison, [1, Thm 3.1, Eq. (8)], which also studies implicit bias with a shorter timescale, also assumes uniform bounds on each mini-batch loss's derivatives through order four, including its gradient.

Bounded stochastic gradients is a stronger assumption, but this assumption holds in most practical cases. For any smooth per-sample losses on a finite dataset, gradients are uniformly bounded on a compact neighborhood of Γ\Gamma, and our iterates remain in that neighborhood with high probability. Moreover, a training will be stopped when the gradient explodes.

W3. Presentation

We introduce the the minimizer manifold together with the definition and assumptions of the loss function in Section 2.1. It also explains that a local minimizer manifold exists under suitable regularity and rank conditions, following [2]. These statements provide the geometric setting needed to understand our results.

Given the page limit, we did not include a general math definition of a manifold, since this would require additional background beyond the properties used in our analysis. We would be happy to include this background in the appendix to make the presentation more accessible if needed.

Regarding sharpness, the abstract and introduction describe its role in implicit regularization and emphasize that Adam and SGD are associated with different sharpness measures. In the label noise setting considered here, the relevant quantities are tr(H)\text{tr}(H) for SGD and tr(Diag⁡(H)1/2)\text{tr}\left(\operatorname{Diag}(H)^{1/2}\right) for Adam, where HH is the loss Hessian and Diag⁡(H)\operatorname{Diag}(H) retains the diagonal entries of HH, so the second quantity equals ∑iHii\sum_i\sqrt{H_{ii}}. The Adam expression is the simplified form with zero stabilization parameter, where this specialization is well defined. Thus, “sharpness” refers to a class of measures of local loss curvature rather than a single universal quantity. This is why we describe Adam as reducing “a different form of sharpness.” We will make the distinction between these measures more explicit in the final version.

Q. Experiments

Our submission already includes synthetic experiments illustrating the predicted dynamics. Under label noise, our results for 1<α<21<\alpha<2 yield the same sharpness regularizer Rϵ,τ\mathcal R_{\epsilon,\tau} as the 22-scheme in [5], whereas α=1\alpha=1 introduces an additional drift.

We thank the reviewer for providing a suggestion for our paper. Actually, the numerical experiment illustrated in Figure 1 and 3 is precisely designed to illustrate our results. Under label noise, our results for 1<α<21<\alpha<2 yield the same implicit regularizer Rϵ,τ\mathcal R_{\epsilon,\tau} as the 22-scheme in [5], whereas α=1\alpha=1 introduces an additional drift. Hence [5]'s experiments illustrating this implicit bias therefore remain relevant to 1<α≤21<\alpha\le2.

Regarding α=1\alpha=1, in our attempts with more realistic learning problems, it was difficult to obtain a clear empirical separation between α=2\alpha=2, 1<α<21<\alpha<2 and α=1\alpha=1. We therefore construct an example where the baseline sharpness is constant along the minimizer manifold, isolating this additional drift. We will make these two points clear in the revision.

[1] Cattaneo, M. D., Klusowski, J. M., and Shigida, B. On the Implicit Bias of Adam. ICML, 2024.

[2] Li, Z., Wang, T., and Arora, S. What Happens after SGD Reaches Zero Loss?—A Mathematical Framework. ICLR, 2022.

[3] Wang, B., Fu, J., Zhang, H., Zheng, N., and Chen, W. Closing the Gap between the Upper Bound and Lower Bound of Adam's Iteration Complexity. NeurIPS, 2023.

[4] Hong, Y. and Lin, J. On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions. NeurIPS, 2024.

[5] Li, X., Wen, H., and Lyu, K. Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold. NeurIPS, 2025.

[6] Katzenberger, G. S. Solutions of a Stochastic Differential Equation Forced Onto a Manifold by a Large Drift. The Annals of Probability, 19(4):1587–1628, 1991.

AuthorsOpenReview ↗

Look forward to a feedback from the reviewer

Dear Reviewer bjz3,

Thank you for your feedback on our paper. As the rebuttal deadline approaches, we would appreciate your response to our clarifications, particularly regarding the novelty concern.

As explained in our rebuttal, our work and Cattaneo et al. (2024) [1] address distinct settings and establish different results. Their analysis considers fixed momentum parameters and an ODE approximation over O(η−1)O(\eta^{-1}) iterations. Our work characterizes stochastic motion near a minimizer manifold over O(η−2)O(\eta^{-2}) iterations under the coupled scaling 1−β2=cβηα1-\beta_2=c_\beta\eta^\alpha, with 1≤α<21\le\alpha<2. Consequently, their results do not establish ours.

We would appreciate knowing whether these distinctions address your concern about novelty. If you believe that the cited work already establishes our contribution, could you please identify the specific result and explain how it applies to our setting? We have also addressed your concerns about the assumptions and presentation, and would welcome any remaining questions.

Thank you again for your time and consideration. We look forward to your feedback.

Sincerely,

The Authors

[1] Cattaneo, M. D., Klusowski, J. M., and Shigida, B. On the Implicit Bias of Adam. ICML, 2024.

Reviewer_hK3qOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

(Dear Authors and AC: Please note that I am not a ML theory person, so I am providing my review that is based my read of only the main paper. The proofs are all in the appendix which I tried to skim but did not check thoroughly)

The paper extends convergence analysis of Adam optimizer to a regime that was not considered in prior work by Li et al. (2025). The regime is defined by the exponent of 1−β2=Θ(ηα1 − \beta_2 = \Theta(\eta^\alpha. Previous worked had analyzed α=2\alpha=2 while the current paper derives a slow-moving SDE that describes the dynamics of Adam when 1<α<21 \lt \alpha \lt 2 and α=1\alpha=1 separately. The paper's finds that 1<α<21 \lt \alpha \lt 2 behaves like previous "2-scheme" finding by Li et al. (2025) in terms of a measure of sharpness under label noise while α=1\alpha=1 has a different implicit bias.

Critical strengths
  • Analyzing the convergence properties of Adam optimizer should be of interest to both theorists and practitioners. This paper extends Adam's convergence to a regime when the trajectory is close to loss minimizer manifold for cases not considered in prior work

  • The paper provides SDEs for two cases with the finding for α=1\alpha=1 not being obvious to me even if I carefully read Li et al. (2025) as a general ML researcher/practitioner. A careful read suggests that the

  • Figure 1 provides some intuition on how the SDEs work in different regime.

Critical weaknesses
  • In practice Adam's beta2beta_2 ranges from 0.95 to 0.999 (default as noted in paper in some frameworks). The paper's analysis appears not to completely cover the regime used in practice when it comes to Adam/AdamW optimizer. Would the authors comment on this observation?

  • As a practitioner, this paper is a very difficult read. What would really be helpful is having some numerical examples with matrix factorization of a simple deep linear network setting to check and show SDE behavior vs a network's behavior.

  • Another advantage with having a numerical example would be to check the validity of assumptions made in the paper including behavior as a function of Adam's β2\beta_2

  • While I appreciate that proofs are lengthy and hence parked in the appendix, I wonder if some of the main proofs might benefit by having a sketch to give the reader an appreciation for techniques involved in the proof.

Minor

  • FIgure 3 in the paper is hard to read. I am unable to spot "yellow trajectories" mentioned in text.
Questions for authors

Please see my comments and question in weakness section. The paper is off to a good start but would benefit greatly with numerical examples with "real" networks or settings considered in prior works like Li et al. (2025). Having a few examples would definitely help me understand the paper better and support acceptance.

3 comments

Dear reviewer hK3q,

We sincerely thank you for recognizing the interest of the problem and the distinct behaviour of the 11-scheme. We also appreciate your suggestions for making the presentation more accessible to a broader readership. We would be happy to improve the exposition in the camera ready version. Below, we address your concerns and clarify the intuition behind our results.

W1. Regime in practice

Thank you for raising this point. We agree that our analysis does not cover all practical configurations of Adam and AdamW. We would like to clarify its practical motivation, its contribution beyond existing theory, and the remaining limitations.

1. Our analysis addresses scalings motivated by widely used training settings. PyTorch’s default Adam configuration uses β2=0.999\beta_2=0.999 and η=0.001\eta=0.001, giving 1−β2=η1-\beta_2=\eta [1]. The value β2=0.999\beta_2=0.999 is also used in ViT pretraining, including the ResNet models evaluated in that work [2], and in the official implementations of DDPM [3] and DiT [4]. These examples motivate studying settings where 1−β21-\beta_2 is comparable to η\eta. Our schemes describe how these quantities scale jointly as η→0\eta\to0, so a particular value of β2\beta_2 alone does not determine the applicable scheme. Accordingly, these practical configurations motivate our analysis without implying that our guarantees cover every training setup using them.

2. Extending the analysis beyond the 22-scheme requires addressing different limiting dynamics. Li et al. [5] characterize the 22-scheme, where 1−β2=Θ(η2)1-\beta_2=\Theta(\eta^2). Our work extends this characterization to 1≤α<21\leq\alpha<2, allowing substantially faster adaptation of the preconditioner. At α=1\alpha=1, the parameter and second moment updates must be analyzed jointly, and additional drift terms survive over O(η−2)O(\eta^{-2}) iterations, changing the implicit bias. For 1<α<21<\alpha<2, the analysis must control the accumulated effects of the changing preconditioner on the projected parameters over this entire horizon. Building on Katzenberger’s framework [6], we develop the estimates needed to handle these coupled updates. This establishes previously unresolved limiting dynamics and identifies a qualitative change in implicit bias at α=1\alpha=1.

3. Extending the results to smaller α\alpha, particularly the 00-scheme, requires further analysis. Keeping β2<1\beta_2<1 fixed as η→0\eta\to0 corresponds to the 00-scheme, which is outside our current results. Our convergence guarantee covers 0.5≤α≤20.5\leq\alpha\leq2, while our characterization of the limiting dynamics covers 1≤α<21\leq\alpha<2. Below α=0.5\alpha=0.5, even establishing an appropriate guarantee for approaching and remaining near the minimizer manifold requires further work. Such a guarantee is a prerequisite for deriving the subsequent slow dynamics along the manifold. These questions are therefore important directions for future work. AdamW with nonzero weight decay also introduces an additional drift absent from our setting and requires separate analysis. Although DiT uses the AdamW implementation, its official configuration sets weight decay to zero [4].

W2&Q1. Some numerical examples

We reproduced the diagonal network experiment in [5] and found that schemes (1,2)(1,2) achieve the same performance as the 22-scheme, with both substantially outperforming SGD in this setting. These results are consistent with our theoretical finding that, under label noise, schemes (1,2)(1,2) share the same sharpness related fixed point condition as the 22-scheme. We would be happy to include these experimental results in the camera ready version if the reviewer finds them helpful.

In settings such as those studied in [5], we found it difficult to clearly distinguish the implicit biases of the 1<α<21<\alpha<2 and α=1\alpha=1 regimes empirically. We therefore designed the experiments in Figures 1 and 3 to isolate and illustrate the additional drift predicted for α=1\alpha=1.

W3. A proof sketch would help explain the techniques

Thank you for this helpful suggestion. To convey the intuition behind our results, Section 3.2 provides an illustration of the dynamics, while Appendix C explains our approach and proof in more detail. The latter discussion was placed in the appendix due to space constraints. We agree that making this explanation more accessible would help readers follow the main results, and we provide a brief summary below.

The illustration in Section 3.2 explains the main mechanism underlying our proof. We study how Adam behaves after training loss has nearly converged. In this scenario, Adam still fluctuates around a manifold of minimizers. Gradient noise pushes parameters away, while the full gradient pulls them back. As illustrated in Figure 2, projecting each iterate onto the manifold along the preconditioned gradient flow removes this fast restoring motion: its first-order effect cancels, leaving an O(η2)O(\eta^2) drift along the manifold, observable over O(η−2)O(\eta^{-2}) iterations.

Adam's second-moment estimate vv introduces another timescale. Under 1−β2=Θ(ηα)1-\beta_2=\Theta(\eta^\alpha), the three relevant speeds are O(η)O(\eta) for parameter fluctuations, O(ηα)O(\eta^\alpha) for vv, and O(η2)O(\eta^2) for projected motion. When 1<α<21<\alpha<2, vv is too slow to follow individual parameter fluctuations but fast enough to track their smoothed statistics along the manifold. Thus, the preconditioner is approximately fixed during fast recovery while adapting to the slowly moving minimizer. At α=2\alpha=2, vv instead evolves on the same timescale as the projected motion and cannot fully track its moving target.

At α=1\alpha=1, vv evolves alongside parameter fluctuations, making their coupling essential. The same stochastic gradient updates both the parameters and vv (through its square), producing an additional drift involving third gradient moments. The resulting correction is O(η(1−β2))O(\eta(1-\beta_2)) per step, accumulating to O(ηα−1)O(\eta^{\alpha-1}) over O(η−2)O(\eta^{-2}) steps. It survives at α=1\alpha=1 but vanishes for α>1\alpha>1, explaining why the 1-scheme can exhibit a different implicit bias, whereas 1<α<21<\alpha<2 recovers the 2-scheme's sharpness bias under label noise.

W4. The yellow trajectories mentioned near Figure 3 are difficult to identify

Thank you for pointing out this ambiguity. The yellow trajectories appear in Figure 1, and the passage refers back to that comparison. Figure 3 displays the magnitude of the limiting drift and its zeros with blue trajectories.

[1] PyTorch. torch.optim.Adam documentation.

[2] Dosovitskiy, A., et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR, 2021.

[3] Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Probabilistic Models. NeurIPS, 2020. See also the official optimizer implementation, which uses the TensorFlow 1.15 Adam defaults.

[4] Peebles, W., and Xie, S. Scalable Diffusion Models with Transformers. ICCV, 2023. See the official training configuration.

[5] Li, X., Wen, H., and Lyu, K. Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold. NeurIPS, 2025.

[6] Katzenberger, G. S. Solutions of Stochastic Differential Equations Forced onto a Manifold by a Large Drift. The Annals of Probability, 1991.

I thank the authors for their response. I want to acknowledge their hard work and for patiently helping me understand the paper better. I am already leaning positive, so I will keep my score. Happy to update if my review changes during AC-Reviewers discussion.

Reviewer_vBzkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, we study the Implicit Bias of Adam near the minima manifold, focusing on how the relative update timescales of second-order momentum and model parameters affect optimal solution selection. Study has only covers 1−β2=Θ(η2)1-\beta_2=\Theta(\eta^2) limited, the author will analyze extended to 1−β2=cβηα1-\beta_2=c_\beta\eta^\alpha (1≤α<21\leq\alpha<2), we derive Slow Sdes for different regimes through a multi-timescale stochastic dynamics framework, and establish theoretical convergence guarantees. The results show that in the Label Noise condition, 1<α<21<\alpha<2 when Adam maintains the existing Sharpness-related Implicit Bias, while α=1\alpha=1 when the parameter is coupled to the preconditioner produces additional Drift, which may change the optimal solution selection. Finally, the authors conducted synthetic experiments to verify the differences in optimization behavior under different regimes.

Critical strengths
  1. The research motivation is clear and significant. Aiming at the limitation that the existing Adam Implicit Bias theory only covers specific hyper-parameter scaling conditions, this paper proposes to study the difference of optimization behavior under different time scales, and the research problem is clear and has reasonable motivation.

  2. This paper extends the existing analysis to Scaling Regimes that have not been solved before, and find the existence of additional Drift in 1-scheme, which provides a new theoretical insight into Adam's Implicit Bias and optimal solution selection mechanism.

Critical weaknesses
  1. Mismatch between theoretical assumptions and standard Adam. The analysis assumes that the preconditioner S(v)S(v) is globally smooth and uniformly positive definite. However, the standard Adam preconditioner is non-differentiable at zero and lacks a uniform positive lower bound. It remains unclear whether the theoretical assumptions rigorously cover standard Adam, potentially limiting the applicability of the main conclusions.
  2. Insufficient connection between theoretical scaling regimes and practical Adam configurations. The analysis relies on the asymptotic scaling 1−β2=cβηα1-\beta_2=c_\beta\eta^\alpha, whereas practical Adam typically uses fixed hyperparameters. The correspondence between these theoretical regimes and practical configurations is not sufficiently established, leaving the practical implications of the derived implicit bias unclear.
Questions for authors

See weakness.

2 comments

Dear reviewer vBzk,

We sincerely thank you for the time and effort you devoted to reviewing our paper. We are grateful for your recognition of our motivation and interest of the additional drift identified in the 11-scheme. We address the two concerns in your review below.

W1. Mismatch between theoretical assumptions and standard Adam

Thank you for raising this important question about the applicability of our assumptions. It's reasonable to assume that SS is smooth and uniformly positive definite.

Actually, the uniform positive lower bound holds on the accessible bounded region. For standard Adam [1], we have S(v)=Diag⁡((vi+ϵ)−1)S(v)=\operatorname{Diag}\left(({\sqrt{v_i}+\epsilon})^{-1}\right) with ϵ>0\epsilon>0. We agree that SS has no uniform positive lower bound over the entire unbounded orthant. However, we prove that the parameters stays in a compact neighbourhood of the minimizer manifold Γ\Gamma (See Theorem 3.1), where S(v)S(v) has a uniform positive lower bound.

As for the smoothness of standard Adam's preconditioner, it's true that it is non-differentiable at zero. However, it enters the drift multiplied by quantities that vanish sufficiently quickly and we can extend our results to zero. We will make this statement clear in the camera ready version.

W2. Insufficient connection between theoretical scaling regimes and practical Adam configurations

Thank you for asking us to clarify the connection to practical configurations. Our scaling regimes describe how Adam's behaviour depends on the relative scales of η\eta and β2\beta_2. In our asymptotic family, the hyperparameters remain constant within each run and are related across runs as η→0\eta\to0. They do not change asymptotically during a single run. Accoringly, the results characterize a mechanism relevant to Adam's memory dependence of a family of configurations, rather than analyzing a single default configuration. Moreover, values of β2\beta_2 close to one are relevant to practical training. For example, ViT [2] uses β2=0.999\beta_2=0.999 for pretraining both its Transformer and ResNet models.

[1] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. ICLR, 2015.

[2] Alexey Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR, 2021. See Section 4.1 for the pretraining configuration.

Thank you for clarifying the scaling assumptions and the connection to prior work. I maintain my rating of 3.

Two reservations remain:

  1. localization explains the positive lower bound on the preconditioner over the accessible region, but Lemma G.1 establishes agreement with standard Adam only where all second-moment coordinates are positive. The extension to zero coordinates still needs justification.
  2. the diagonal-network result reported in response to hK3q is consistent with the shared sharpness bias for 1<α<21<\alpha<2 and α=2\alpha=2, but leaves open how pronounced the distinct α=1\alpha=1 effect is in that setting. These limitations narrow the scope of the claims.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Note to the Area Chair on Reviewer bjz3’s Comment

Dear Area Chair,

We respectfully request your independent assessment of Reviewer bjz3’s novelty judgment of this paper and its role in the score of 1. We have provided a detailed response, but the central disagreement remains unresolved. We are concerned that the assessment overlooks the different but complementary mechanisms revealed by our slow SDE analysis and the cited IGR work, and therefore does not accurately reflect our contribution. We would like to highlight two points.

  1. We believe that the reviewer’s conclusion that we “just answer the same question in another restrictive setting” unfairly discounts our contribution. As explicitly explained in our rebuttal, our paper reveals different but complementary mechanisms of Adam’s implicit bias compared with previous IGR work, including Cattaneo et al. We apologize for omitting this important citation and will add it and clarify the relationship. However, we do not believe this omission justifies rejecting our paper on novelty grounds. Please see our detailed rebuttal for the comparison.

  2. The reviewer’s objections concerning the scaling assumptions and comparison with IGR appear to question the slow SDE framework more broadly, rather than establish that our specific results lack novelty. IGR and slow SDE are established, related but distinct lines of research on implicit bias. Our analysis belongs to the latter. This line includes [R4] (ICLR 2022), [R5] (ICLR 2023), [R6–R7] (ICLR 2024), and [R1] (NeurIPS 2025), with related developments and applications in [R8] (JMLR 2026) and [R9] (ICML 2023). These works characterize the accumulated effects of stochasticity near minimizer manifolds over long training horizons. The cited IGR results do not establish the particular dynamics and mechanisms proved in our paper. See our rebuttal and the related work section of R1 for details. We will also clarify this relationship in the revision.

We would therefore appreciate the AC’s assessment of whether the cited results actually imply our main conclusions, and whether the novelty objection supporting such a low score remains justified after these distinctions are taken into account.

[update] By the close of the discussion, we had addressed Reviewer bjz3’s other concerns, and the reviewer continued to express reservations about one assumption. We respectfully maintain that this is a minor issue with limited bearing on the paper’s central question and that it does not diminish the novelty or significance of our main contributions.

Thank you for your time and consideration.

Authors.


[R1] Li, X., Wen, H., and Lyu, K. Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold. NeurIPS, 2025.

[R2] Cattaneo, M. D., Klusowski, J. M., and Shigida, B. On the Implicit Bias of Adam. ICML, 2024.

[R3] Katzenberger, G. S. Solutions of a Stochastic Differential Equation Forced Onto a Manifold by a Large Drift. The Annals of Probability, 19(4):1587–1628, 1991.

[R4] Li, Z., Wang, T., and Arora, S. What Happens after SGD Reaches Zero Loss?—A Mathematical Framework. ICLR, 2022.

[R5] Gu, X., Lyu, K., Huang, L., and Arora, S. Why (and When) Does Local SGD Generalize Better than SGD?. ICLR, 2023.

[R6] Wang, R., Malladi, S., Wang, T., Lyu, K., and Li, Z. The Marginal Value of Momentum for Small Learning Rate SGD. ICLR, 2024.

[R7] Gu, X., Lyu, K., Arora, S., Zhang, J., and Huang, L. A Quadratic Synchronization Rule for Distributed Deep Learning. ICLR, 2024.

[R8] Shalova, A., Schlichting, A., and Peletier, M. Singular-limit Analysis of Gradient Descent with Noise Injection. Journal of Machine Learning Research, 27(146):1–56, 2026.

[R9] Liu, H., Xie, S. M., Li, Z., and Ma, T. Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models. ICML, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.