acceptodds
Desk rejected at ICLR 2027

Adam under Bounded Variance: Polylogarithmic Confidence without Clipping

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

High-probability guarantees quantify the reliability of Adam and other adaptive optimizers. Under bounded noise variance, jin2026sharper show that Adam's second moment normalization already yields polylogarithmic confidence for the preconditioned gradient energy, while their ordinary squared gradient guarantee retains polynomial dependence on . This motivates asking whether Adam can preserve that confidence benefit in ordinary stationarity. We show that adaptive normalization coupled with second moment memory of order achieves this without clipping or stronger noise assumptions. For smooth objectives bounded below and conditionally unbiased noise with bounded variance, we combine with a suitably chosen constant learning rate, any fixed first moment coefficient, and positive second moment initialization. For any fixed positive offset, , and , the average squared gradient norm is with probability at least . When the target confidence is specified before the run, adjusting the memory and learning rate sharpens the guarantee to under the same assumptions. Descent localized by objective height and a potential coupling accumulation with displacement control the cumulative gradient cost of large denominators and give both guarantees through a common analysis.

Primary area:
optimization
Keywords:
Adam, High-probability convergence, Polylogarithmic dependence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xghTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the high-probability guarantees of the popular optimizer Adam under the general assumptions: smooth nonconvex function, unbiased gradient, and bounded variance. This work is motivated by studying the superiority of Adam over SGD through the facts that Adam achieving a better dependence on the failure probability δ\delta, in terms of the convergence upper bound.

Building on existing study Jin et al. (2026), which established polynomial dependence on 1/δ1/\delta, this work refines the analysis through better choice of β2\beta_{2} and learning rate, and achieve polylog(1/δ)\text{polylog}(1/\delta) dependence.

Critical strengths
  • This work is well-motivated, studying the superiority of Adam over SGD through convergence dependence on the failure probability δ\delta (but sadly, the comparison is between upper bounds).
  • The techniques is somewhat convincing, it uses the accumulation control rather than bounding the trajectory-wise terms, which make sense to avoid extra costs.
Critical weaknesses

Nontrivial technical issues

  • The convergence result hide some nontrivial dependences:
    • The polynomial dimension dependence of dd. It is unclear how to interpret this dependence, which usually appears for ℓ1\ell_{1}-norm criterion.
    • The convergence result hide parameter ϵ\epsilon. I believe ϵ\epsilon is necessary here, and the refinement may come from a ϵ\epsilon dependent tuning, a common strategy to achieve stronger result but is impractical.
    • The constant CC is implicitly dependent on dd.
  • The algorithm 1 is not plain Adam, it omits the bias correction terms.

Significance of the work

  • Compared with existing study, the claimed improvement comes from the a different choice of β1,γ\beta_{1}, \gamma and some analysis refinements, the technical contribution is somewhat weak and not well-presented as well.
  • Meanwhile, the "established advantage" is all about comparison between upper bound; I don't see separation result to really support the claimed advantage over SGD.

Questions for authors
  • how does the dimension dd enter the convergence bound?
Reviewer_8LGJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proves that Adam can have ordinary stationary gradient bounds with polylog dependence on \delta (w.p. at least 1 - \delta) under standard technical assumptions, e.g., smooth objectives bounded below and con- ditionally unbiased stochastic gradients with bounded variance. The previous work Jin et.al 2026 showed polynomial dependence when convert from cumulative weighted squared gradient to the ordinary gradient bound. The technical argument follows from stopped descent and a conversion from preconditioned gradient to ordinary squared gradients under the assumptions.

Critical strengths

This paper is a follow-up work of Jin et.al 2026, and proves that Adam can have ordinary stationary gradient bounds with polylog dependence on \delta (w.p. at least 1 - \delta) under standard technical assumptions, e.g., smooth objectives bounded below and con- ditionally unbiased stochastic gradients with bounded variance.

Critical weaknesses

The paper establishes an interesting and improved confidence bound for Adam. However, the overall contribution is limited to improving an existing high-probability bound on Adam, and it does not provide new insights into the understanding of the algorithm. The theory also relies on selective parameter choices, which make it hard for the theoretical improvement translates into useful guidance for practical Adam configurations.

Questions for authors

Is it possible to show, via lower bound or counter-example, that the horizon-length memory, (1-\beta_2=\Theta(1/T)), necessarily prevents polylogarithmic confidence dependence for ordinary stationarity?

Reviewer_vYokOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the high-probability convergence of Adam under bounded conditional variance, without gradient clipping or stronger noise assumptions. The main result improves the confidence dependence of the ordinary squared-gradient stationarity bound from polynomial to polylogarithmic in 1/δ1/\delta, compared with Jin et al. (2026). With a second-moment memory length of order T\sqrt{T}, the paper establishes an O(log⁡2(d/δ)/T)O(\log^2(d/\delta)/\sqrt{T}) bound in the relevant confidence regime. When the parameters are allowed to depend on δ\delta, the bound improves to O(log⁡(d/δ)/T)O(\sqrt{\log(d/\delta)/T}). The analysis combines stopping-time localization, normalized gradient energy bounds, and a joint potential argument to control intervals with large second-moment accumulators.

Critical strengths
  1. The paper addresses a well-defined limitation of existing high-probability analyses of Adam, particularly the polynomial confidence dependence in Jin et al. (2026).

  2. The results require only bounded conditional variance, without assuming bounded stochastic gradients, higher-order moments, or gradient clipping.

  3. The analysis of high-accumulator intervals provides a potentially useful technique for converting preconditioned gradient bounds into ordinary stationarity guarantees.

Critical weaknesses
  1. Insufficient acknowledgment of prior results and unclear novelty. Several key technical results and proof techniques have appeared in earlier studies but are not adequately acknowledged. For example, the logarithmic bound on accumulated normalized gradients (Lemma B.5) is closely related to Défossez et al. (2022), the normalized momentum bound in Lemma B.4 was also established in Simple Convergence Proof of Adam From a Sign-like Descent Perspective (2025), and stopping-time localization with martingale arguments was developed in Li et al. (2023). Although some of these works are cited elsewhere, their connections to the present analysis are not clearly discussed, and the sign-like Adam paper appears to be missing from the references. The authors should better acknowledge these existing results and clarify which parts of their analysis constitute genuinely new contributions.

  2. Unfavorable dimension dependence and overly conservative constants. The convergence bounds have an explicit linear dependence on dd, up to logarithmic factors, and the prescribed learning-rate coefficient scales as d−1d^{-1} when other problem parameters are fixed. The constants are extremely conservative. For example, even in a one-dimensional deterministic setting with d=L=Δ0=v=ϵ=1d=L=\Delta_0=v=\epsilon=1, σ=0\sigma=0, and β1=0.9\beta_1=0.9, the prescribed learning-rate coefficient is only cγ≈8.2×10−17c_\gamma\approx 8.2\times10^{-17}, while the convergence-bound constant is C≈2.2×1028C\approx 2.2\times10^{28}. For T=106T=10^6, this gives a learning rate of approximately 8.2×10−208.2\times10^{-20}. Such constants make the theoretical guarantees practically uninformative. The bounds also retain substantial dependence on ϵ\epsilon.

  3. Limited connection to practical Adam training. The analysis requires horizon-dependent second-moment parameters and does not establish the same guarantees for commonly used fixed-β2\beta_2 AdamW. No experiments are provided to support the proposed memory scaling or demonstrate improved training reliability.

Questions for authors
  1. Several important proof ingredients appear closely related to existing results, particularly the normalized gradient bounds in Défossez et al. (2022), the normalized momentum bounds in Simple Convergence Proof of Adam From a Sign-like Descent Perspective (2025), and the stopping-time arguments in Li et al. (2023). Could the authors acknowledge these connections more explicitly, include the missing relevant references, and clarify precisely which theoretical results and proof techniques are new compared with prior work?

  2. Although the confidence dependence improves over Jin et al. (2026), the bounds retain explicit dimension dependence. Could the authors provide a comparison accounting for the full dependence on dd, δ\delta, and TT, including the learning-rate restrictions and hidden constants in both works?

  3. Beyond improving the confidence dependence, does the analysis establish any new qualitative property of Adam? Is there theoretical or empirical evidence that the proposed shorter memory improves robustness or recovery from unusually large stochastic gradients?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.