acceptodds
Under review as a conference paper at ICLR 2027

On the Convergence of Adam, Revisited

Abstract

We show that projected Adam for online optimization with arbitrary moment decay parameters can have average regret bounded away from zero. A similar result of Reddi-Kale-Kumar from 2018 required . Similar to their result, we use a three-periodic sequence of linear functions on with slopes , though we use slightly larger than . This nonzero average regret result extends to Adam variants such as AdamW, RMSProp, NAdam, Adan, AdaMax, Muon, and to an i.i.d. variant of the three-periodic sequence of slopes for Adam.

Primary area:
learning theory
Keywords:
Adam, convergence, regret

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VebDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the failure cases of the popular optimization method Adam and demonstrates that in certain conditions Adam fails to converge in the sense of online learning regret.

The main improvement is that it extends the non-convergence results of Reddi et al. (2018) to arbitrary β1,β2∈[0,1)\beta_1,\beta_2\in[0,1), using a sequence of linear functions with uniformly bounded gradients. That said, the refinement is hyper-parameter level: β1≤β2→β1,β2∈[0,1)\beta_1 \le \sqrt{\beta_2} \rightarrow \beta_{1},\beta_{2} \in [0, 1).

The authors also claimed that their results can be extended to several Adam variants such as AdamW and Muon.

Critical strengths
  • The extensions to multiple Adam variants and stochastic settings improve the generalizability of the negative results, which may infer the uniform failure condition of the EMA-type first moment update.
  • The simplification and self-contained proof of the non-convergence analysis.
Critical weaknesses

Significance of the work

  • The main improvement over existing results appears to be extending the parameter range from β1≤β2\beta_1 \leq \sqrt{\beta_2} to arbitrary β1,β2∈[0,1)\beta_1, \beta_2 \in [0,1). However, the condition β1≤β2\beta_1 \leq \sqrt{\beta_2} is widely adopted in convergence analyses and is satisfied by commonly used practical settings. Therefore, the existing negative results already seem sufficient to demonstrate potential convergence failures in practically relevant regimes. I am not convinced that further extending the parameter range provides substantial practical value or new theoretical insight.
  • From an intuitive perspective, Adam's failure to converge under adversarial gradient sequences is not particularly surprising. Since Adam's updates depend heavily on historical gradient information, one can imagine adversarial future gradients that conflict with this history and lead to undesirable updates. In this sense, the existence of such failure cases seems relatively natural. I would argue that a more interesting question is why Adam performs so well in practice despite these potential failures, rather than identifying rarely additional worst-case scenarios in which it fails.

Limited technical contribution

  • The main result, Theorem 1.5, appears to introduce several nontrivial restrictions in exchange for covering a broader range of β1\beta_1 and β2\beta_2. It is unclear whether this trade-off represents a meaningful improvement over the existing result (Theorem 1.4). In particular:
    • The function class is not explicitly defined in this paper. Moreover, the analysis appears to narrow the function class from general convex functions, as considered in Reddi et al. (2018), to linear functions.
    • Additional restrictions on αt\alpha_t: Theorem 1.5 imposes a series of restrictions on the step-size sequence that are absent from Theorem 1.4.
    • Additional restrictions on ft′(x)f_t'(x): Theorem 1.5 also introduces new restrictions on the gradients, which I consider a more serious concern. Any additional assumptions on the function class should be carefully justified, especially when the goal is to strengthen an existing negative result. Otherwise, it becomes difficult to determine whether the improvement comes from a genuinely stronger analysis or simply from imposing more favorable conditions on the problem.
  • The proof of the main theorem appears relatively standard and relies heavily on the additional restrictions mentioned above. In particular:
    • The central construction still relies on periodic adversarial gradient sequences, where carefully designed gradients exploit the influence of historical information on Adam's updates. This is essentially the same mechanism underlying the counterexample in Reddi et al. (2018). I do not see a fundamentally new construction or theoretical insight beyond extending the applicable parameter range.
  • Finally, the analysis simplifies Adam by omitting the bias-correction terms.
Questions for authors
  • Since the parameter regime studied by Reddi et al. (2018) already covers commonly used practical settings, what additional theoretical insights or practical implications does the extension to arbitrary parameter choices offer?
  • Could the authors explicitly specify the function class considered and clarify whether the construction extends beyond linear functions?
  • Could the authors explain the necessity of the additional step-size conditions in Theorem 1.5 and how these conditions affect its comparison with Theorem 1.4?
Reviewer_wLhrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors proved the non-zero regret for projected Adam for online optimization with arbitrary moment decay parameters β1,β2∈[0,1)\beta_1, \beta_2 \in [0,1).

Their results extend the previous results from β1<β2\beta_1<\sqrt{\beta_2} to general β1,β2∈[0,1)\beta_1, \beta_2 \in [0,1).

Critical strengths

The main results and contributions are clearly presented in this paper.

Critical weaknesses

The contribution of this paper is quite restrictive and incremental. Specifically, the non-zero regret heavily relies on the learning rate αt=αt\alpha_t=\frac{\alpha}{\sqrt{t}}. This learning rate significantly weaken the effect of adaptivity of the past cumulative gradients used in Adam. However, the more important learning rate of Adam is the constant learning rates, which preserves the adaptivity effect of adaptive stepsizes. The paper cannot provide the results for this more important learning rate.

On the other hand, Adam is mainly used in non-convex optimization, rather than simple linear and convex function. This paper does not investigate the non-convex setting.

Questions for authors

Can you get similar results for the constant learning rate?

Reviewer_6gQAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is a revisit and a refinement of the previous nonconvergence result of Projected Adam by extending the counterexample of Reddi-Kale-Kumar ICLR'18 to a less restricted set of parameter choices, while maintaining uniformly bounded gradients. The major contributions are theoretical. While I do not see any significant new proof or analysis technique introduced in this paper, I did find some clever uses of established techniques, especially in the stochastic extension. The paper also extended the counterexample to AdamW, Muon, etc. The paper is an incremental strengthening of an existing line of analysis. There is no methodological breakthrough.

Critical strengths
  • There are new counterexamples that extend the previous results to unrestricted parameter choices.
  • The construction is a modest refinement or improvement over the previous ones. It makes clever use of existing proof techniques.
  • I checked the mathematical steps and didn't spot any major errors.
  • The writing is good.
Critical weaknesses
  • Since commonly used hyperparameter settings are covered by the previous results, I don't find removing the parameter restriction to be of practical significance.
  • I understand that the pursuit of weaker conditions for a theoretical conclusion requires a specific counterexample construction, but I hope the author(s) can help build some connection between their counterexample and a realistic optimization landscape.

Minor issues:

  • A student of mine finds the AdamW extension somewhat a statement–proof mismatch. Thm 1.6 states that Thm 1.5 also holds for AdamW, but Appendix A changes the feasible domain from [−1,1][-1,1] to [−4,−2][-4,-2], so this doesn't establish an apple-to-apple result, and one potential side-effect is that weight decay can push the iteration toward the right endpoint, which opposes the minimizer of the unregularized cumulative loss.
  • The conceptual advance over existing counterexamples is limited, which basically improved from previous works by covering all fixed moment parameters.
  • Some typos (e.g., line 138): "As in (Reddi et al., 2018, Theorem 6) in the β1=0\beta_1=0 case of Adam", where there are two "in".
Questions for authors

Please see the major and minor weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.