- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigated the implicit bias of Adam. Specifically, the authors characterize the implicit bias for the scaling , where is the second-order moment parameter, is the learning rate, and .
Critical strengths
The main strengths of this paper may be some potentially rigorous mathematical derivation.
Critical weaknesses
The main weakness of this paper is that existing literature has investigated the implicit bias of Adam, which significantly harms the novelty of this paper. This authors missed the significant literature "Cattaneo et al., On the implicit bias of Adam. ICML 2024"
Besides, the authors missed many other related works cited in the paper "Cattaneo et al., On the implicit bias of Adam. ICML 2024". The paper overview has significant flaws.
Also, the assumptions in this paper is too restrictive. They simultaneously assume Lipschitzness and boundedness of gradients. However, in existing analysis of Adam, one only needs Lipschitzness of gradients (see e.g., [1, 2]).
Last but not least, the presentation of this paper should be improved. The notions of sharpness and minimizer manifold should be introduced more detailed.
References
[1] Wang et al., Closing the gap between the upper bound and lower bound of adam’s iteration complexity. Neurips 2023
[2] Hong and Lin, On convergence of adam for stochastic optimization under relaxed assumptions. Neurips 2024
Questions for authors
Can you provide some experiments to illustrate and support the theoretical findings in this paper? This is not the weakness of this paper, while it is just the suggestion of this paper.
10 comments
Q. Experiments
Our submission already includes synthetic experiments illustrating the predicted dynamics. Under label noise, our results for yield the same sharpness regularizer as the -scheme in [5], whereas introduces an additional drift.
We thank the reviewer for providing a suggestion for our paper. Actually, the numerical experiment illustrated in Figure 1 and 3 is precisely designed to illustrate our results. Under label noise, our results for yield the same implicit regularizer as the -scheme in [5], whereas introduces an additional drift. Hence [5]'s experiments illustrating this implicit bias therefore remain relevant to .
Regarding , in our attempts with more realistic learning problems, it was difficult to obtain a clear empirical separation between , and . We therefore construct an example where the baseline sharpness is constant along the minimizer manifold, isolating this additional drift. We will make these two points clear in the revision.
[1] Cattaneo, M. D., Klusowski, J. M., and Shigida, B. On the Implicit Bias of Adam. ICML, 2024.
[2] Li, Z., Wang, T., and Arora, S. What Happens after SGD Reaches Zero Loss?—A Mathematical Framework. ICLR, 2022.
[3] Wang, B., Fu, J., Zhang, H., Zheng, N., and Chen, W. Closing the Gap between the Upper Bound and Lower Bound of Adam's Iteration Complexity. NeurIPS, 2023.
[4] Hong, Y. and Lin, J. On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions. NeurIPS, 2024.
[5] Li, X., Wen, H., and Lyu, K. Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold. NeurIPS, 2025.
[6] Katzenberger, G. S. Solutions of a Stochastic Differential Equation Forced Onto a Manifold by a Large Drift. The Annals of Probability, 19(4):1587–1628, 1991.
Look forward to a feedback from the reviewer
Dear Reviewer bjz3,
Thank you for your feedback on our paper. As the rebuttal deadline approaches, we would appreciate your response to our clarifications, particularly regarding the novelty concern.
As explained in our rebuttal, our work and Cattaneo et al. (2024) [1] address distinct settings and establish different results. Their analysis considers fixed momentum parameters and an ODE approximation over iterations. Our work characterizes stochastic motion near a minimizer manifold over iterations under the coupled scaling , with . Consequently, their results do not establish ours.
We would appreciate knowing whether these distinctions address your concern about novelty. If you believe that the cited work already establishes our contribution, could you please identify the specific result and explain how it applies to our setting? We have also addressed your concerns about the assumptions and presentation, and would welcome any remaining questions.
Thank you again for your time and consideration. We look forward to your feedback.
Sincerely,
The Authors
[1] Cattaneo, M. D., Klusowski, J. M., and Shigida, B. On the Implicit Bias of Adam. ICML, 2024.
Our work and "Cattaneo et al" are distinct settings and distinct analyses of Adam's implicit bias.
We have read the review. We believe that the main concern about novelty stems from a misunderstanding of the distinction between our work and Cattaneo et al. (2024). [1]
[1] follow the implicit gradient regularization (IGR) approach via backward error analysis, with an ODE approximation guarantee over O(η−1) iterations and fixed β2. Our work, instead, characterizes the noise-driven slow dynamics near a minimizer manifold over O(η−2) iterations, under the coupled scaling 1−β2=O(ηα), 1≤α<2. These are distinct settings and distinct analyses of Adam’s implicit bias, and their results do not establish ours. Our setting directly answers "what kind of minimizers does Adam ultimately choose", which is our motivation.
W1. Relation to Cattaneo et al. [1], “On the Implicit Bias of Adam,” ICML 2024.
We list two key differences between the IGR approach and our Slow-SDE based approach showing that we pursue different settings.
IGR and Slow-SDE have different timescales. [1]'s Theorem 3.1 establishes an O(η2) trajectory error for n≤⌊T/η⌋, with T fixed. This is not an approximation guarantee on the O(η−2)-iteration horizon studied in our paper. Our work aims at characterizing the accumulated stochastic effects that result in a slow motion along the minimizer manifold, and a preference among minimizers, since this motion lasts O(η−2) time and requires an analyses that lasts such a time period. On the other hand, when η→0, O(η−1) is infinitesimal compared to O(η−2), so [1]'s results cannot establish ours.
We give different regimes for 1−β2. [1]'s Section 1.1 explicitly keeps the momentum parameters β,ρ (our β1,β2) fixed as η→0. In our setting, β2→1 jointly with η→0. Moreover, their approximation constant depends on both the time horizon and the momentum parameters. Consequently, substituting our β2(η) and extending the horizon does not yield the uniform approximation needed for our results. Our regime of 1−β2 is critical to extending the analysis to an O(η−2) timescale.
We will add the missing citation and clarify this relationship. However, an omission in the related-work discussion does not establish a lack of novelty: the cited work [1] does not cover the scaling regimes of β2, the long-horizon (O(η−2)) dynamics, or the phase transition of the implicit bias happening at the 1-scheme.
W2. Assumptions
Above all, the cited results [3,4] focus on the convergence phase which lasts O~(η−1) iterations, but they do not consider the subsequent motion along a minimizer manifold or the implicit bias of Adam. Our analysis instead addresses the latter over O(η−2) iterations, by extending Katzenberger’s framework [6]. For comparison, [1, Thm 3.1, Eq. (8)], which also studies implicit bias with a shorter timescale, also assumes uniform bounds on each mini-batch loss's derivatives through order four, including its gradient.
Bounded stochastic gradients is a stronger assumption, but this assumption holds in most practical cases. For any smooth per-sample losses on a finite dataset, gradients are uniformly bounded on a compact neighborhood of Γ, and our iterates remain in that neighborhood with high probability. Moreover, a training will be stopped when the gradient explodes.
W3. Presentation
We introduce the the minimizer manifold together with the definition and assumptions of the loss function in Section 2.1. It also explains that a local minimizer manifold exists under suitable regularity and rank conditions, following [2]. These statements provide the geometric setting needed to understand our results.
Given the page limit, we did not include a general math definition of a manifold, since this would require additional background beyond the properties used in our analysis. We would be happy to include this background in the appendix to make the presentation more accessible if needed.
Regarding sharpness, the abstract and introduction describe its role in implicit regularization and emphasize that Adam and SGD are associated with different sharpness measures. In the label noise setting considered here, the relevant quantities are tr(H) for SGD and tr(Diag(H)1/2) for Adam, where H is the loss Hessian and Diag(H) retains the diagonal entries of H, so the second quantity equals ∑iHii. The Adam expression is the simplified form with zero stabilization parameter, where this specialization is well defined. Thus, “sharpness” refers to a class of measures of local loss curvature rather than a single universal quantity. This is why we describe Adam as reducing “a different form of sharpness.” We will make the distinction between these measures more explicit in the final version.