acceptodds
Under review as a conference paper at ICLR 2027

AdamE: Dynamic Exponential Decay Rates for the Adam Optimizer

Abstract

Adaptive optimizers such as Adam are widely used in deep learning. Their gradient statistics are governed by two fixed decay rates, most commonly 0.9 and 0.999. The best pair differs across tasks, and searching for it is costly. We show that under any fixed second-moment rate the effective per-step gradient scale can shrink over training, the failure mode behind known counterexamples where Adam fails to converge. We address this with AdamE, a drop-in variant of Adam whose decay rates follow dynamic, closed-form schedules: the first rate starts at one half and decays smoothly toward zero, while the second rises toward one, with both rates determined by the iteration index. The schedules introduce no new hyperparameters and provably restore the monotonicity that fixed rates lack, recovering convergence without altering the update rule as AMSGrad does. We prove that AdamE attains near-optimal regret in the online convex setting, converges to a stationary point on smooth non-convex objectives, and admits an unconditional error bound on least squares, where the corresponding Adam guarantee holds only in a restricted regime. Experiments on language modeling, node classification, graph clustering, image classification, and image recognition show that AdamE achieves competitive convergence and final performance relative to well-tuned Adam-type baselines, with no manual tuning of the decay rates.

Primary area:
optimization
Keywords:
Adaptive gradient methods, Adam optimizer, Convergence analysis, Dynamic decay-rate schedules, Stochastic optimization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ci8hOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes AdamE, an Adam variant with coupled decay schedules alpha_t = t/(1+t^2) and beta_t = 1-alpha_t. These schedules ensure monotonicity of the effective step sizes without AMSGrad's elementwise maximum. The paper claims guarantees for least squares, online convex optimization, and nonconvex optimization, and compares against seven tuned Adam-type baselines on seven datasets covering language modeling, node classification, graph clustering, image classification, and image recognition. Lemmas A.1-A.4 and Theorem 2 are valid, but Theorem 3 does not establish the claimed regret rate, and the substantial overlap with NosAdam (IJCAI 2019) is not acknowledged.

Critical strengths
  1. The mechanism enforcing monotonicity is sound and nontrivial: the argument for H_{t+1} > 0 and Lemmas A.1-A.4 are valid. The method couples a first moment with short memory, assigning at least half the weight to the current gradient and factorially decaying weights to past gradients, with a second moment that averages the full history. This coupling is distinctive and establishes the claimed structural monotonicity without an additional assumption.

  2. The work is comprehensive: it introduces no additional hyperparameters, can be used as a drop-in replacement, provides analyses in three settings, and evaluates seven datasets. The motivating experiments also convincingly show that the preferred decay-rate settings can reverse across tasks.

Critical weaknesses
  1. Theorem 3 does not establish the claimed regret rate. Equation (13) gives sqrt(T)(1+log T)(2D^2/lambda + 12 sqrt(2) lambda) sum_i ||g_{1:T,i}||2. The gradient-history factor can scale as Theta(sqrt(T)) under bounded gradients, so the right-hand side can grow as Theta(T log T). This does not establish sublinear regret or support the near-optimal rate claimed in the main text and abstract. The proof of Theorem 4 also makes an invalid pointwise replacement of f(theta{T+1}; xi) by f(theta*): theta* minimizes the expected loss, not each sample loss. These issues may be repairable, but the central theoretical claims are not established in the current version.

  2. The paper does not establish its novelty relative to NosAdam (Huang, Wang, and Dong, IJCAI 2019). NosAdam's v_t and AdamE's s_t are both normalized weighted averages over the full gradient history, and NosAdam's condition that B_t/t is nonincreasing is equivalent after reparameterization to t beta_t >= t-1 in Lemma A.3. Both methods address the failure of Gamma_t identified by Reddi et al. and restore convergence without an AMSGrad-style modification. Yet NosAdam is not cited or compared, and there is no AdaGrad comparison or alternative-schedule ablation to isolate the contribution of the proposed coupling.

Questions for authors
  1. Equation (13) contains an explicit sqrt(T) prefactor and a gradient-history factor that can also scale as sqrt(T). How can the claimed near-optimal regret rate be established once both factors are accounted for?

  2. Please clarify the relationship to NosAdam, including the equivalence of the second-moment families and monotonicity conditions. What theoretical distinction or matched empirical comparison establishes the contribution of AdamE's particular coupling?

1 comment

This paper needs to be carefully revised and reorganized.

Reviewer_nWYEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study investigates the limit of the decaying parameters of first and second momentum and proposes AdamE with adapt first and second momentum decay hyperparameters. αt=t1+t2\alpha_t=\frac{t}{1+t^2}, and βt=1−αt\beta_t=1-\alpha_t. The underlying mechanism is that it introduces an attenuated historical impact on the first momentum and an enhanced history impact on the second momentum. The authors utilize the structure to construct a monotonicity property and claim near-optimal regret for online convex optimization, convergence to stationary points for stochastic nonconvex optimization, and an error bound for least-squares problems. Experiments cover language modelling, graph learning, and image classification.

Critical strengths
  1. The study helps understand how time-varying moment decay rates affect optimization and that it reduces the pain of tuning these rates.
  2. It is simply designed. Both the decay rates are computed directly from iteration step without introducing additional tunable schedule params.
  3. I appreciate the summplementary lemmas that provide a lower bound on the second momentum and establish monotonicity of time-weighted second moments and effective step sizes under the specified learning-rate schedule.
  4. The evaluation extends beyong synthetic examples to language modeling, graph learning, and vision.
Critical weaknesses
  • The major issue lies in Eq 13. The paper claims that "Every term in equation 13 is nonnegative and finite, and the regret of AdamE is upper bounded by O(√T log T)". However, $ ||g_{1:T,i}|| $ is the norm of the entire gradient sequence for coordinate ii, not the norm of a single-step gradient. Therefore, bounded gradients do not make this quantity independent of T. For example, the scalar sequence gi=1g_i=1 has uniformly bounded per-step gradients but satisfies ∥∥g1:T,i∥∥2=T\|\|g_{1:T,i}\|\|_2=\sqrt T. Therefore, the upper bound is O(T log T) instead of O(√T log T).
  • The experiments do not sufficiently isolate the contribution of the proposed schedules. Although the evaluation covers several tasks, the current presentation leaves important questions about tuning fairness and robustness. The paper should report the full learning-rate search spaces, search budgets, validation-based selection rules for each optimizer for fair comparison.
Questions for authors

As a study in optimization theory, the flaws in the upper bound derivation in Eq. 13 is critical and directly falsifies other following derivations. Can the authors provide a corrected derivation that explicitly accounts for the growth of the gradient-sequence norms?

Reviewer_XAuNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a variant of Adam by setting a particular schedule for moment momentum parameters of the first-order moment and second-order moments.

The contribution include some analysis and experimental results on language modeling and image classification tasks.

Critical strengths
  1. the decreasing schedule of \alpha_t looks interesting and novel.

  2. the experiments on language modeling tasks look promising

Critical weaknesses
  1. the presentation and analysis is poorly done. The analysis in sec. 4.2 should be presented in a better way so that readers can understand the points. The authors try to say that H_t >0, which helps solves the failing mode of Adam or addresses the mistakes in the original proof of Adam. It should be stated more clearly.

  2. No proof is given at all. No way to verify the correctness of the theoretical results.

Questions for authors

You forgot to include the proof.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.