- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes AdamE, an Adam variant with coupled decay schedules alpha_t = t/(1+t^2) and beta_t = 1-alpha_t. These schedules ensure monotonicity of the effective step sizes without AMSGrad's elementwise maximum. The paper claims guarantees for least squares, online convex optimization, and nonconvex optimization, and compares against seven tuned Adam-type baselines on seven datasets covering language modeling, node classification, graph clustering, image classification, and image recognition. Lemmas A.1-A.4 and Theorem 2 are valid, but Theorem 3 does not establish the claimed regret rate, and the substantial overlap with NosAdam (IJCAI 2019) is not acknowledged.
Critical strengths
-
The mechanism enforcing monotonicity is sound and nontrivial: the argument for H_{t+1} > 0 and Lemmas A.1-A.4 are valid. The method couples a first moment with short memory, assigning at least half the weight to the current gradient and factorially decaying weights to past gradients, with a second moment that averages the full history. This coupling is distinctive and establishes the claimed structural monotonicity without an additional assumption.
-
The work is comprehensive: it introduces no additional hyperparameters, can be used as a drop-in replacement, provides analyses in three settings, and evaluates seven datasets. The motivating experiments also convincingly show that the preferred decay-rate settings can reverse across tasks.
Critical weaknesses
-
Theorem 3 does not establish the claimed regret rate. Equation (13) gives sqrt(T)(1+log T)(2D^2/lambda + 12 sqrt(2) lambda) sum_i ||g_{1:T,i}||2. The gradient-history factor can scale as Theta(sqrt(T)) under bounded gradients, so the right-hand side can grow as Theta(T log T). This does not establish sublinear regret or support the near-optimal rate claimed in the main text and abstract. The proof of Theorem 4 also makes an invalid pointwise replacement of f(theta{T+1}; xi) by f(theta*): theta* minimizes the expected loss, not each sample loss. These issues may be repairable, but the central theoretical claims are not established in the current version.
-
The paper does not establish its novelty relative to NosAdam (Huang, Wang, and Dong, IJCAI 2019). NosAdam's v_t and AdamE's s_t are both normalized weighted averages over the full gradient history, and NosAdam's condition that B_t/t is nonincreasing is equivalent after reparameterization to t beta_t >= t-1 in Lemma A.3. Both methods address the failure of Gamma_t identified by Reddi et al. and restore convergence without an AMSGrad-style modification. Yet NosAdam is not cited or compared, and there is no AdaGrad comparison or alternative-schedule ablation to isolate the contribution of the proposed coupling.
Questions for authors
-
Equation (13) contains an explicit sqrt(T) prefactor and a gradient-history factor that can also scale as sqrt(T). How can the claimed near-optimal regret rate be established once both factors are accounted for?
-
Please clarify the relationship to NosAdam, including the equivalence of the second-moment families and monotonicity conditions. What theoretical distinction or matched empirical comparison establishes the contribution of AdamE's particular coupling?
This paper needs to be carefully revised and reorganized.