- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes two parameter-free optimization algorithms: Adagrad++and Adam++, as simple variants of Adagrad and Adam, respectively. The core idea is to introduce an automatic learning rate based on the distance between the iteration point and the initial point, thereby eliminating the need for manual adjustment of the learning rate.
Critical strengths
1.The paper has diverse experimental scenarios. 2.Adam++has low computational and memory overhead.
Critical weaknesses
1.I don't understand why the learning rate has to take this form. The paper claims to be inspired by DOG, which actually weakens its contribution, and it doesn't provide any theoretical guarantees that are better than Adam's. 2.This paper does not discuss whether this learning rate modification strategy can be applied to SGD-based and second-order optimizers such as AdaHessian, and the abstract directly limits it to Adam-based optimizers.
Questions for authors
1.The paper decomposes the global stride of DoG and directly applies it to Adam, but does not demonstrate why this approach holds true for Adam-based optimizers. Is there any theoretical or experimental basis for this? 2.Does the additional assumption that the iterative point of the theorem is far away from the initial point still hold in non-convex or pathological problems?
We thank the reviewer for prompting a clearer articulation of the method's theoretical motivation and novelty. Our aim is not to claim that the distance idea originates here, or that we improve the best tuned Adam rate. Rather, it is to obtain a comparably simple, coordinate-wise adaptive algorithm with an automatically determined global scale and an accompanying analysis.
A1: DoG's effective scalar step is ∑i≤t∥gi∥22maxi≤t∥xi−x0∥2, appropriate to a globally normalized SGD update. Adam and AdaGrad already normalize gradients coordinate-wise. Copying DoG's global gradient denominator into Adam would double-count gradient normalization and would no longer preserve Adam's familiar per-coordinate preconditioning. We instead transfer only the displacement numerator and normalize it by d, yielding the root-mean-square displacement per coordinate. The difference is structural, not cosmetic: the global scale and adaptive coordinate-wise denominator perform different tasks. Case 2 additionally uses t+1 so that the EMA second-moment scale and AdaGrad-style accumulated-gradient scale are compatible in order of magnitude. Theorems 3.2 and 4.1 analyze the resulting new recursions, rather than borrowing DoG's theorem. The novelty is a simple parameter-free adaptation within the AdaGrad/Adam family supported by direct convergence analysis, instead of simply faster convergence rate than Adam.
A2: The modification strategy is general: a global displacement-based scale can be paired with an optimizer-specific preconditioner. For SGD, DoG already provides the natural global-normalization counterpart, so simply duplicating our Adam-specific normalization is not the right extension. For AdaHessian or other curvature-based methods, a similar external scale is structurally possible in xt+1=xt−ηtPtgt, but the proof would need assumptions on the curvature preconditioner, its spectrum, estimation error, and temporal variability. We do not claim those methods are covered by our current Adam/AdaGrad results. The abstract focuses on the algorithms for which we provide concrete update rules and analysis; limiting the scope of a proved claim is preferable to speculative generality.
A3: Yes to the first question, with the exact assumptions and algorithmic setting stated by the theorems; a uniformly faster rate than tuned Adam is not the claim. The relevant comparison is whether an Adam-family, coordinate-wise adaptive method can achieve a comparable standard convergence order without a tuned exogenous global LR magnitude. The leading rate is of the familiar O(T−1/2) type under the rate-level conditions discussed in the paper. The gradient-accumulation-sensitive form can be sharper on favorable trajectories. These are complementary statements: the data-dependent inequality is the primary theorem, while a worst-case rate is one specialization.
A4: No universal distance-to-comparator bound is assumed to hold for arbitrary pathological objectives; such a claim would be unreasonable even for many standard stochastic optimizers. The main convex theorems retain the realized trajectory distances rather than claiming universal stability. Uniform control is used only when summarizing those trajectory-dependent bounds as rates with fixed constants. Appendix E analyzes nonconvex stationarity under additional smoothness and stochastic assumptions; the expectation-level presentation of its trajectory maximum will be made rigorous as detailed in our response to Reviewer 1. Importantly, the algorithm never requires the comparator or its distance as input. The ridge-regression evidence in Appendix H.3.4 supports practical plausibility in its tested regime, but is not presented as a theorem about all nonconvex landscapes. This distinction answers the pathological-case objection without altering the core algorithm.