acceptodds
Under review as a conference paper at ICLR 2027

Towards Simple and Provable Parameter-Free Adaptive Gradient Methods

Abstract

Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process. However, ad-hoc tuning of learning rates poses a challenge, leading to inefficiencies in practice. To address this issue, recent research has focused on developing ”parameter-free” algorithms that operate effectively without the need for learning rate tuning. Despite these efforts, existing parameter-free variants of AdaGrad and Adam tend to be overly complex and/or lack formal convergence guarantees. In this paper, we present AdaGrad++ and Adam++, novel and simple parameter-free variants of AdaGrad and Adam with convergence guarantees. We prove that AdaGrad++ achieves comparable convergence rates to AdaGrad in convex optimization without predefined learning rate assumptions. Similarly, Adam++ matches the convergence rate of Adam without relying on any conditions on the learning rates. Experimental results across various deep learning tasks validate the competitive performance of Adam++.

Primary area:
optimization
Keywords:
optimizer, parameter-free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YnXmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes two parameter-free optimization algorithms: Adagrad++and Adam++, as simple variants of Adagrad and Adam, respectively. The core idea is to introduce an automatic learning rate based on the distance between the iteration point and the initial point, thereby eliminating the need for manual adjustment of the learning rate.

Critical strengths

1.The paper has diverse experimental scenarios. 2.Adam++has low computational and memory overhead.

Critical weaknesses

1.I don't understand why the learning rate has to take this form. The paper claims to be inspired by DOG, which actually weakens its contribution, and it doesn't provide any theoretical guarantees that are better than Adam's. 2.This paper does not discuss whether this learning rate modification strategy can be applied to SGD-based and second-order optimizers such as AdaHessian, and the abstract directly limits it to Adam-based optimizers.

Questions for authors

1.The paper decomposes the global stride of DoG and directly applies it to Adam, but does not demonstrate why this approach holds true for Adam-based optimizers. Is there any theoretical or experimental basis for this? 2.Does the additional assumption that the iterative point of the theorem is far away from the initial point still hold in non-convex or pathological problems?

1 comment

We thank the reviewer for prompting a clearer articulation of the method's theoretical motivation and novelty. Our aim is not to claim that the distance idea originates here, or that we improve the best tuned Adam rate. Rather, it is to obtain a comparably simple, coordinate-wise adaptive algorithm with an automatically determined global scale and an accompanying analysis.

Q1: Why use this learning-rate form, and what is new beyond DoG?

A1: DoG's effective scalar step is max⁡i≤t∥xi−x0∥2∑i≤t∥gi∥22\frac{\max_{i\le t}\|x_i-x_0\|_2}{\sqrt{\sum_{i\le t}\|g_i\|_2^2}}, appropriate to a globally normalized SGD update. Adam and AdaGrad already normalize gradients coordinate-wise. Copying DoG's global gradient denominator into Adam would double-count gradient normalization and would no longer preserve Adam's familiar per-coordinate preconditioning. We instead transfer only the displacement numerator and normalize it by d\sqrt d, yielding the root-mean-square displacement per coordinate. The difference is structural, not cosmetic: the global scale and adaptive coordinate-wise denominator perform different tasks. Case 2 additionally uses t+1\sqrt{t+1} so that the EMA second-moment scale and AdaGrad-style accumulated-gradient scale are compatible in order of magnitude. Theorems 3.2 and 4.1 analyze the resulting new recursions, rather than borrowing DoG's theorem. The novelty is a simple parameter-free adaptation within the AdaGrad/Adam family supported by direct convergence analysis, instead of simply faster convergence rate than Adam.

Q2: Could the scaling apply to SGD or AdaHessian?

A2: The modification strategy is general: a global displacement-based scale can be paired with an optimizer-specific preconditioner. For SGD, DoG already provides the natural global-normalization counterpart, so simply duplicating our Adam-specific normalization is not the right extension. For AdaHessian or other curvature-based methods, a similar external scale is structurally possible in xt+1=xt−ηtPtgtx_{t+1}=x_t-\eta_t P_tg_t, but the proof would need assumptions on the curvature preconditioner, its spectrum, estimation error, and temporal variability. We do not claim those methods are covered by our current Adam/AdaGrad results. The abstract focuses on the algorithms for which we provide concrete update rules and analysis; limiting the scope of a proved claim is preferable to speculative generality.

Q3: Is there rigorous evidence for Adam++? are the bounds better than Adam's?

A3: Yes to the first question, with the exact assumptions and algorithmic setting stated by the theorems; a uniformly faster rate than tuned Adam is not the claim. The relevant comparison is whether an Adam-family, coordinate-wise adaptive method can achieve a comparable standard convergence order without a tuned exogenous global LR magnitude. The leading rate is of the familiar O~(T−1/2)\widetilde O(T^{-1/2}) type under the rate-level conditions discussed in the paper. The gradient-accumulation-sensitive form can be sharper on favorable trajectories. These are complementary statements: the data-dependent inequality is the primary theorem, while a worst-case rate is one specialization.

Q4.Does bounded distance to initialization hold in pathological nonconvex settings?

A4: No universal distance-to-comparator bound is assumed to hold for arbitrary pathological objectives; such a claim would be unreasonable even for many standard stochastic optimizers. The main convex theorems retain the realized trajectory distances rather than claiming universal stability. Uniform control is used only when summarizing those trajectory-dependent bounds as rates with fixed constants. Appendix E analyzes nonconvex stationarity under additional smoothness and stochastic assumptions; the expectation-level presentation of its trajectory maximum will be made rigorous as detailed in our response to Reviewer 1. Importantly, the algorithm never requires the comparator or its distance as input. The ridge-regression evidence in Appendix H.3.4 supports practical plausibility in its tested regime, but is not presented as a theorem about all nonconvex landscapes. This distinction answers the pathological-case objection without altering the core algorithm.

Reviewer_UtbNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaGrad++ and Adam++. These two claimed parameter-free optimizers serve as variants of AdaGrad and Adam and are inspired by the DoG framework. The authors also demonstrate that AdaGrad++ and Adam++ achieve an similar convergence rate in convex optimization as AdaGrad and Adam respectively. Experiments are conducted on image classification and LLM pretraining tasks to evaluate the performance of the proposed algorithms.

Critical strengths
  1. The proposed algorithms are a natural extension from AdaGrad and Adam after applying the inspiration from DoG framework. The algorithmic modification is straightforward and avoids the inner-product accumulation states required by D-Adaptation and Prodigy.
  2. The experiments cover standard vision benchmarks as well as GPT-2 pretraining tasks on OpenWebText.
Critical weaknesses
  1. The trajectory-dependent bounds require a more precise interpretation and justification. Theorems 3.2 and 4.1 depend on the maximum distance between the iterates and a comparator. The discussion following Theorem 3.2 on page 5 explicitly assumes that these quantities remain bounded by constant multiples of the initialization distance. However, this uniform control is not established for the original algorithms. Consequently, the manuscript should distinguish a trajectory-dependent finite-time bound from a convergence rate conditional on trajectory stability.

  2. The submitted implementations differ from the analyzed algorithms. The analyzed Adam++ uses exponentially decaying momentum. However in the submitted code optimizers/adampp.py:71-76, the provided CV entry point passes fixed values through betas (train_cv.py:85,100). Thus, the execution path uses constant momentum. The authors should explicitly identify which results cover the implementation used in practice.

  3. Practical advantages should also include tuning cost, runtime, and memory. The GPT-2 experiments report approximately 0.02 lower loss, but do not fully establish the practical advantage of reducing tuning. Stronger evidence would include fixed-default performance, matched search budgets, time-to-target loss, and peak memory. For example, the memory concern is supported by the implementation optimizers/adampp.py:19 that stores a full copy of the initial parameters and that lines 60-62 allocate the first and second momentu tensors and that lines 63-64 additionally allocate the historical maximum while AMSGrad is enabled.

  4. Initialization formula in the code and the paper are different. The code implemented in optimizers/adampp.py corresponds to 1+||x_0||_2 instead of 1+||x_0||_2^2 as referred in the manuscript (line 1500).

Questions for authors
  1. GPT-2 training code and configurations (e.g. initialization discrepancies).
  2. What benefits remain under fixed defaults and matched tuning budgets.
2 comments

We appreciate the detailed analysis of both the paper and the code. And the responses to your questions are shown below.

Q1: The trajectory-dependent bounds require a more precise interpretation and justification. Theorems 3.2 and 4.1 depend on the maximum distance between the iterates and a comparator. The discussion following Theorem 3.2 on page 5 explicitly assumes that these quantities remain bounded by constant multiples of the initialization distance. However, this uniform control is not established for the original algorithms. Consequently, the manuscript should distinguish a trajectory-dependent finite-time bound from a convergence rate conditional on trajectory stability.

A1: We would like to clarify a fundamental distinction: trajectory stability is not an assumption required by Theorems 3.2 and 4.1; it is a sufficient condition for obtaining simplified asymptotic convergence rates from these theorems. Specifically, Theorems 3.2 and 4.1 establish high-probability, trajectory-dependent finite-time bounds for the original AdaGrad++ and Adam++ algorithms. These bounds explicitly retain the maximum iterate-comparator distances rather than assuming them to be bounded by constants. Therefore, the validity of the finite-time guarantees does not rely on proving uniform trajectory stability. If the trajectory grows, the bounds reflect this growth; the theorems do not silently assume otherwise.

Moreover, the trajectory-dependent quantities are closely related through the proposed adaptive scaling rule. By definition, ηT=max⁡η0,max⁡t≤T∥xt−x0∥2d\eta_T=\max{\eta_0,\max_{t\leq T}\frac{\|x_t-x_0\|_2}{\sqrt d}}. Consequently, the triangle inequality immediately gives max⁡t≤T(∥xt−x∗∥2≤∥x0−x∗∥2+d,ηT)\max_{t\leq T}(\|x_t-x^*\|_2\leq \|x_0-x^*\|_2+\sqrt d,\eta_T). Thus, controlling the adaptive scale at the initialization-distance level, i.e., ηT=O(∣x0−x∗∣2/d)\eta_T=O(|x_0-x^*|_2/\sqrt d), is sufficient to control the maximum iterate-comparator distance. This shows that the distance quantities appearing in our bounds are directly connected to the algorithm's adaptive scaling mechanism, rather than being unrelated stability assumptions. We agree that this sufficient stability condition has not been established universally for the original algorithms. However, this does not invalidate the trajectory-dependent finite-time results or require modifying the algorithms. It only means that the simplified O~(1/T)\widetilde O(1/\sqrt T) rate should be understood under the additional uniform control of the trajectory-dependent factors. This distinction is already motivated in the discussion following Theorem 3.2, with empirical evidence provided in Appendix H.3.4.

To avoid ambiguity, we made this distinction explicit in Appendix J.1 in the revised manuscript: Theorems 3.2 and 4.1 remain trajectory-dependent finite-time guarantees, while the simplified convergence rates are stated under sufficient trajectory-stability conditions. Importantly, the purpose of our theoretical contribution is precisely to provide convergence bounds for distance-adaptive, coordinate-wise preconditioned methods without prescribing a conventional learning-rate schedule. The trajectory-dependent formulation is central to this contribution, not a substitute for an unproved stability theorem.

Q2: The submitted implementations differ from the analyzed algorithms. The analyzed Adam++ uses exponentially decaying momentum. However in the submitted code optimizers/adampp.py:71-76, the provided CV entry point passes fixed values through betas (train_cv.py:85,100). Thus, the execution path uses constant momentum. The authors should explicitly identify which results cover the implementation used in practice.

A2: Thank you for noting this. The parameter λ\lambda affects only the first-moment schedule, β1,t=β1λt\beta_{1,t}=\beta_1\lambda^t; it does not define the distance-adaptive learning-rate mechanism. The current manuscript already states in Appendix G that λ=1\lambda=1 gives conventional constant momentum as we used in the experiments, whereas the theorem employs 0<λ<10<\lambda<1 to bound the additional momentum terms using a geometric series. And Theorem 4.1 proves the decaying-momentum variant. The requirement λ<1\lambda<1 is a sufficient condition for this proof, not a demonstration that λ=1\lambda=1 is unstable or nonconvergent. To make the paper more rigorous, we have added the proofs for the scenarios when λ=1\lambda=1 in Appendix K.

Q3: Practical advantages should also include tuning cost, runtime, and memory. The GPT-2 experiments report approximately 0.02 lower loss, but do not fully establish the practical advantage of reducing tuning. Stronger evidence would include fixed-default performance, matched search budgets, time-to-target loss, and peak memory. For example, the memory concern is supported by the implementation optimizers/adampp.py:19 that stores a full copy of the initial parameters and that lines 60-62 allocate the first and second momentu tensors and that lines 63-64 additionally allocate the historical maximum while AMSGrad is enabled.

A3: The practical advantage targeted by Adam++ is removing a task-specific search for the global learning-rate magnitude, not reducing every element-wise operation or optimizer-state tensor. In Section 5.1, AdamW's rate was selected from a grid search, requires around 10 times more experiments than Adam++. Therefore the current comparison supports competitiveness with less conventional LR tuning. However, it is not presented as a benchmark of universal wall-clock superiority. Adam++ requires more computation and additional memory traffic. For example, the time for experiments of training ResNet-18 on CIFAR10 dataset with Adam, Adam++ (Case 1) and Prodigy requires 42min, 50min and 56min; and, AdamW uses 3.636GB for GPU memory while Adam++ (Case 1) uses 3.682GB for GPU memory as shown in wandb log, as there are a great part of storage for activation and other usage. The difference matches 42.63MB, as the theoretical increment for Adam++.

Q4: Initialization formula in the code and the paper are different. The code implemented in optimizers/adampp.py corresponds to 1+||x_0||_2 instead of 1+||x_0||_2^2 as referred in the manuscript (line 1500).

A4: Thank you for noting this mistake. We actually use 1×10−61\times 10^{-6} for Adam++ as the initial learning rate of Adam++ for image classification tasks, and for GPT-2 small and GPT-2 medium tasks, the initial learning rates of AdamW++ are actually 6×10−46\times 10^{-4} and 3×10−43\times 10^{-4}, as the code submitted. There is neither 1+∥∥x0∥∥21+\|\|x_0\|\|_2 factor nor 1+∥∥x0∥∥221+\|\|x_0\|\|_2^2 factor. We have corrected this mistake in the revised paper.

Q5: GPT-2 training code and configurations (e.g. initialization discrepancies).

A5: Thank you for your attention to our code for GPT experiments. Our GPT code is based on nanoGPT and Sophia (Liu et. al., 2023) as stated in Section 5. We have added sample code for GPT-2 experiments.

Q6: What benefits remain under fixed defaults and matched tuning budgets.

A6: The main benefit of Adam++ is that it achieves competitive performance without a task-specific search for the global learning rate. This advantage is already demonstrated by our existing experiments. In Section 5.1, AdamW's learning rate is selected through a search over [10−4,10−2][10^{-4},10^{-2}], while Adam++ uses the same initialization ϵ=10−6\epsilon=10^{-6} across ResNet-18, VGG16, and DenseNet-121. Despite not performing a comparable learning-rate search, Adam++ achieves comparable or better performance than the tuned AdamW baseline in most evaluated settings. Thus, our results compare Adam++ with a stronger, learning-rate-tuned baseline, rather than an untuned AdamW baseline. For fixed-default performance, the image classification experiments already demonstrate that a single Adam++ initialization works across different architectures without task-specific learning-rate tuning. For matched tuning budgets, we do not claim that Adam++ always outperforms a fully tuned AdamW. Our claim is that Adam++ reduces the need for such tuning while maintaining competitive performance. Moreover, Appendix H.4 (Figure 11) already reports performance against wall-clock time, providing additional evidence of practical efficiency. The additional memory cost is also discussed in Appendix I. We will make these existing comparisons more explicit in the revised manuscript. The practical advantage we emphasize is reduced learning-rate tuning effort, not guaranteed superiority in runtime, memory, or performance under every tuning budget.

Reviewer_oauvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaGradand Adam to reduce learning-rate tuning while preserving coordinate-wise adaptive updates. Both methods use the maximum root-mean-square displacement from initialization to set the global update scale, combined with accumulated squared gradients or an AMSGrad-style second-moment estimate. The authors provide convergence analyses for convex and nonconvex optimization, including stochastic settings. Experiments show competitive image classification performance and approximately 0.02 lower training and validation losses than AdamW on both GPT-2 models.

Critical strengths

1.The paper addresses a practical problem: coordinate-wise adaptation does not remove the need to tune Adam’s global learning rate. Reducing this tuning burden while retaining adaptive updates is valuable, and the comparisons with D-Adaptation and Prodigy are relevant. 2.Normalizing the displacement by d\sqrt{d} converts DoG’s total distance into a per-coordinate scale that naturally complements coordinate-wise gradient adaptation. The resulting algorithms are simple and remain close to familiar adaptive update rules.

Critical weaknesses

1.In Theorems 3.2 and 4.1, M1M_1 explicitly contains 1/η01/\eta_0, but this dependence disappears from the subsequent corollaries without explanation. The authors should explain whether it is eliminated or hidden in a constant, as this affects the claimed comparison with existing convergence bounds. 2.Theorems E.6 and E.7 define DD using the maximum displacement along a stochastic trajectory, yet place it outside expectations. In particular, Eq. (E.16) bounds an expectation involving DD by an expression containing DD without an expectation. The authors should justify this step or introduce an appropriate deterministic bound or moment assumption. 3.The language modeling experiments use model-specific initialization coefficients. AdamW++ uses 6\times10^{-4}(1+x_0_2^2) and 3\times10^{-4}(1+x_0_2^2) for GPT-2 Small and Medium, respectively, borrowing coefficients from their default AdamW learning rates. Other parameter-free baselines use their default initializations. This makes it unclear how much the results depend on prior learning-rate knowledge rather than automatic adaptation. 4.Adam++ Case 2 combines distance-based scaling, a t+1\sqrt{t+1} factor, a running maximum of second moments, and, in the theoretical formulation, decaying momentum. Comparisons with AdamW, D-Adaptation, and Prodigy do not separate these effects. Matched controls and targeted ablations would help establish which changes drive the gains.

Questions for authors

1.How is the 1/η01/\eta_0 dependence removed from the simplified corollaries? In Theorems E.6 and E.7, is DD intended to be random, or is a deterministic upper bound assumed? 2.What value of λ\lambda is used in each main experiment? If λ=1\lambda=1, do the convergence guarantees extend to this setting? 3.Does AdamW++ retain its advantage when both GPT-2 models use the same initialization coefficient, without borrowing their respective default AdamW learning rates? 4.Could the displayed equations following Eq. (2.4) in the main text be numbered for easier reference?

1 comment

We thank the reviewer for the close reading. And here are our responses.

Q1: How is the 1/η01/\eta_0 dependence removed from the simplified corollaries? In Theorems E.6 and E.7, is D intended to be random, or is a deterministic upper bound assumed?

A1: It is not algebraically canceled. It is absorbed into the constants when discussing asymptotic dependence on the iteration horizon, with η0 0\eta_0\>0 fixed independently of TT. In particular, the theorem gives a more informative, parameter-explicit estimate, while the corollaries emphasize the rate in TT under the bounded-trajectory quantities used in the discussion after Theorem 3.2. The distinction matters: we do not assert that the constant is independent of η0\eta_0, and the algorithms are learning-rate-free in the sense of avoiding a tuned external learning-rate schedule, not initialization-free in the stronger sense of uniform constants over all positive η0\eta_0. A fixed positive initialization is also required to break the zero-displacement symmetry at the first step. We will make the hidden dependence explicit, e.g., by writing O~η0,trajectory(T−1/2)\widetilde O_{\eta_0,\mathrm{trajectory}}(T^{-1/2}), with the dependence of the logarithmic terms on ηT/η0\eta_T/\eta_0 retained. This is a clarification of the comparison's scope, not a change to the update rule or the theorem's explicit bound.

Q2: Theorems E.6 and E.7 define D using the maximum displacement along a stochastic trajectory, yet place it outside expectations. In particular, Eq. (E.16) bounds an expectation involving D by an expression containing D without an expectation. The authors should justify this step or introduce an appropriate deterministic bound or moment assumption.

A2: Thank you for noting this. Actually, we assume there is a D that D=max⁡(max⁡t≤T∣xt−x0∣/d,ϵ)D=\max(\max_{t\le T}\\|x_t-x_0\\|/\sqrt d, \epsilon) for all the possibilities so that D is a deterministic upper bound of the optimization progress. We rephrased our expressions in the Appendix J.2 to make it explict.

Q3: Does AdamW++ retain its advantage when both GPT-2 models use the same initialization coefficient, without borrowing their respective default AdamW learning rates?

A3: We agree this is not established by the current GPT-2 experiments. The actual adaptation is governed by ηt=max⁡{η0,max⁡i≤t∣xi−x0∣2/d}\eta_t=\max\{\eta_0,\max_{i\le t}\\|x_i-x_0\\|_2/\sqrt d\}. Once the displacement term dominates, the scale is no longer set by the initialization coefficient. This is a concrete reason to expect reduced long-run initialization sensitivity, although it is not a substitute for direct GPT-2 evidence. As shown in Figure 5, it implement ablation study on the sensitivity evidence of η0\eta_0 on a CV task. We re-implemented GPT-2 experiments with Adam++, and the final validation losses are 2.98 for GPT-2 small and 2.78 for GPT-2 medium, identical to the results in the paper. The curves are almost the same after the initial stages since the ηt\eta_t goes saturated quickly. We have added the results to the paper.

Q4: Adam++ Case 2 combines distance-based scaling, a t+1\sqrt{t+1} factor, a running maximum of second moments, and, in the theoretical formulation, decaying momentum. Comparisons with AdamW, D-Adaptation, and Prodigy do not separate these effects. Matched controls and targeted ablations would help establish which changes drive the gains.

A4: We agree the present full-optimizer comparisons cannot attribute performance to one factor. Each component has a separate function: the distance maximum supplies the global adaptive scale; the t+1\sqrt{t+1} factor makes the EMA-based preconditioner temporally comparable to cumulative-gradient scaling; the running maximum is the AMSGrad-style monotonicity device used in the analysis; and decaying momentum is a technical condition of the present proof. Importantly, Appendix H.3.3 already compares Case 2 with and without the running maximum on ResNet-18/CIFAR-10: the simplified version has lower test loss but slightly lower accuracy.

Q5: What value of λ\lambda is used in each main experiment? If λ=1\lambda=1, do the convergence guarantees extend to this setting?

A5: Thank you for noting this. The parameter λ affects only the first-moment schedule, β1,t=β1λt\beta_{1,t}=\beta_1\lambda^t; it does not define the distance-adaptive learning-rate mechanism. The current manuscript already states in Appendix G that λ=1 gives conventional constant momentum as we used in the experiments, whereas the theorem employs 0<λ<1 to bound the additional momentum terms using a geometric series. And Theorem 4.1 proves the decaying-momentum variant. The requirement λ<1 is a sufficient condition for this proof, not a demonstration that λ=1 is unstable or nonconvergent. To make the paper more rigorous, we have added the proofs for the scenarios when λ=1 in Appendix K.

Q6: Could the displayed equations following Eq. (2.4) in the main text be numbered for easier reference?

A6: Thank you for your notation. We have numbered the step-size equations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in the revised PDF

We have uploaded a revised PDF. New and changed text is in red. The title, the section structure and the theorem numbering are unchanged, and the main text stays within 9 pages.

Main text

  • Sec. 3.2: add a sentence to explain that the convergence rate is built on data-dependent constancy of M1M_1, M2M_2 and M3M_3.
  • Sec. 4.2: add reference to Appendix H.3.4. to validate the assumption of the theorem; add explanation that the convergence rate is built on constant bound of some value.

Appendix

  • Sec. C: add Eq. (C.6) label for reference
  • Sec. G and H.2.: correct the initial learning rate of Adam++
  • Sec. H.3.4: rephrase the result analysis
  • Sec. J: add the investigation on Theorem 3.2 for convergence
  • Sec. K: add convex convergence of Adam++ with constant momentum
  • Sec. L: add a quantitative scale adaptation example for Adam++
  • Sec. M: add analysis on quantitative scale adaptation on smooth nonconvex objectives
  • We have updated AI Use Statement

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.