- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies discounted regret for Adam while retaining its original bias-correction factors. Under clipped updates and , it establishes monotonicity of the effective step sizes, characterizes the single-valley structure of the Abel-summation coefficients, and derives pointwise and cumulative discounted-regret bounds once explicit thresholds are reached. Under fixed proportional coupling , these bounds retain the existing parameter dependence. The paper further uses the existing discounted-to-nonconvex conversion to derive nonconvex stationarity guarantees and a corollary for coordinate-wise clipped Adam, with random step-size scaling and model EMA.
Critical strengths
-
The research objective is specific, and retaining bias correction has technical value. The paper directly handles the original correction factors under a fixed external learning rate. It distinguishes the monotonicity of the effective step sizes from the nonmonotonicity of the subsequent summation coefficients and addresses them separately.
-
The core regret analysis is largely complete. The single-valley structure, terminal-term control, and absorption of the cumulative remainder are supported by detailed proofs, with explicit constants and thresholds. Proposition 6.4 provides additional insight by characterizing the coefficient required for terminal-term control, while explicitly limiting this characterization to the proof framework. The separate treatment of early rounds in the stationarity derivation appropriately avoids invoking the larger cumulative threshold.
Critical weaknesses
-
The scope and substantive gains of the new guarantees remain limited. The paper primarily recovers existing leading-order rates while retaining bias correction. Theorem 7.1 requires , which is stricter than (Theorem 1, Nguyen 2025), and also excludes the choice used by Ahn & Cutkosky. Moreover, the matching pointwise and cumulative bounds apply only once and are reached, respectively. Let . Under fixed proportional coupling as , , a restriction that cannot be ignored when is close to . The tightness established in Proposition 6.4 concerns only the terminal-term Abel envelope; it does not characterize the actual early-round regret cost for realizable gradient sequences. Although the paper acknowledges these limitations, its explanation of the actual effect of bias correction remains limited, and the significance of the new guarantees requires fuller justification.
-
The stationarity complexity still incurs an additional logarithmic loss in some parameter regimes. Let . For any and , Theorem 7.2 displays only , whereas the sufficient condition actually given in Appendix I is
Questions for authors
- Does the stronger restriction reflect a genuine obstacle caused by retaining bias correction, or a limitation of the current growth-event and Abel-summation analysis? Please clarify how the covered parameter range differs from that of prior results.
- When , can the current analysis eliminate the additional logarithmic term? If not, to which parameter regimes should the claim of “optimal complexity” be restricted?
3 comments
Corrections to minor issues
Shared learning rate. Appendix J now sums the unoptimized scalar bounds before selecting the common
Recovery threshold. Remark 4.2 now describes as a sufficient threshold.
Denominator comparison. We have corrected the explanation to retain the time-varying bias correction.
Dear Reviewer,
Thank you again for taking the time to review our paper and share your feedback. We have provided a detailed response addressing your comments and concerns.
When you have a chance, we would greatly appreciate your thoughts on our response and would be happy to discuss any remaining questions or concerns.
Thank you again for your time and consideration.
We thank the reviewer for the careful assessment. Your comments helped us improve the analysis. The revision removes the stronger coupling restriction and the additional logarithmic iteration factor.
W1. Parameter restriction
The restriction β1<β2 was a limitation of our proof. The revised Theorem 7.1 covers 0<β1,β2<1 with β12≤β2.
The extension uses two refinements of the existing analysis:
-
\qquad \phi_k(b):=\frac{b(1-b^k)}{1-b^{k+1}}.$$ Previously, we used the stronger comparison $\phi_k(\beta_1)\le\phi_k(\beta_2)$. The sharper inequality $$\phi_k(\beta_1)^2\le\phi_k(\beta_1^2)\le\phi_k(\beta_2)$$ establishes the required result under $\beta_1^2\le\beta_2$, since $\phi_k$ is increasing. * > **Movement bound (Proposition 5.1).** We retain the existing geometric-growth argument under clipping, but define growth event using $\beta_2^{-s}\|v_s\|^2$ to match the second-moment weights. Taking square roots changes the comparison factor from $(\beta_2/\beta_1)^s$ to $(\sqrt{\beta_2}/\beta_1)^s$. The latter is nondecreasing under $\beta_1^2\le\beta_2$, preserving the $\sqrt2$ growth that bounds the selected sum by a constant times its largest term, including at equality.The next response explains why this logarithmic transient does not introduce an additional logarithmic factor in the stationarity complexity.
W2. Complete complexity and the additional logarithm