acceptodds
Under review as a conference paper at ICLR 2027

Anon: Extrapolating Adaptivity Beyond SGD and Adam

Abstract

Adaptive optimizers such as Adam and non-adaptive methods like SGD, the dominant first-order element-wise optimizers, exhibit distinct generalization capabilities across different architectures. Prior tunable optimizers attempt to bridge this gap by strictly interpolating between SGD and Adam, effectively confining adaptivity within the 0-to-1 bound. However, this restricted interpolation leaves performance on the table: the optimal adaptivity of some tasks lies outside this interval, and no existing tunable optimizer can reach it. Extrapolating adaptivity violates the strict non-decreasing pre-conditioner assumption underlying the convergence guarantees of existing adaptive methods, so standard guarantees no longer apply. To break this barrier, we propose Anon, an optimizer that achieves fully continuous adaptivity extrapolation across the entire real-number spectrum. To guarantee provable stability in these out-of-bound regimes, we introduce Incremental Delay Update (IDU), a novel mechanism that bypasses hard max-tracking strategies. We theoretically establish Anon's convergence in both convex and non-convex settings. Empirically, Anon demonstrates highly competitive and scalable performance among state-of-the-art element-wise optimizers on representative image classification, diffusion, and large language modeling tasks.

Primary area:
optimization
Keywords:
optimizer, Adam, SGD

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ujt8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper defines adaptivity through a preconditioner’s response to scaling the gradient history and proposes Anon, an element-wise optimizer with a real-valued adaptivity parameter. Its Incremental Delay Update mechanism updates the preconditioner at geometrically spaced steps, supporting convergence analysis without requiring monotonic preconditioners. Experiments on image classification, diffusion, and language modeling suggest that modest extrapolation beyond SGD- and Adam-level adaptivity can improve performance.

Critical strengths
  1. The adaptivity formulation provides a coherent perspective on diagonal optimizers. Characterizing gradient-scale response gives a precise interpretation of the SGD and Adam endpoints and motivates examining values outside their usual interpolation range. This offers a useful framework for studying task-dependent optimization behavior.

  2. IDU is a concrete algorithmic contribution accompanied by theoretical results. The paper provides convex regret and non-convex stationarity bounds and acknowledges that their constants deteriorate with the extrapolation parameter. Experiments span several model families, and the diffusion comparison at fixed γ=1\gamma=1 provides evidence that IDU can contribute independently of extrapolation.

Critical weaknesses
  1. The fairness of the main baseline comparisons is not sufficiently established. Section 4 states that baseline metrics outside the diffusion study are sourced from prior publications. Even when these are strong published results, differences in implementations, training recipes, and tuning protocols can confound optimizer comparisons. The controlled CIFAR-10 experiment in the appendix is useful, but does not establish that the principal ImageNet and language-model comparisons are equally controlled.

  2. The statistical reliability of the reported improvements is unclear. Tables 2–4 report single runs without uncertainty estimates. Some improvements are small, including 0.12 percentage points over SGDM on ResNet18 and 0.03 points over HVAdam on ResNet50. Likewise, the diffusion improvement from FID 8.03 at γ=1\gamma=1 to 7.75 at γ=1.01\gamma=1.01 needs repeated runs to establish its robustness. This limits the evidence for a consistently beneficial optimizer modification.

  3. The incremental benefit of extrapolation is not convincingly separated from IDU and hyperparameter selection. The central motivation concerns exceeding the capabilities of interpolation-based methods, yet the main experiments do not directly compare with the cited tunable alternatives Padam and MADA. Moreover, GPT2-medium uses γ=1\gamma=1, so its improvement supports IDU rather than extrapolation. The strongest evidence would compare the best in-range and extrapolated configurations under matched joint learning-rate/adaptivity searches.

Questions for authors
  1. Which ImageNet and language-model baselines were rerun in the same implementation, and can the principal comparisons be established under identical training and tuning protocols?

  2. What is the variation across seeds for the main improvements, particularly the modest ImageNet margins and the diffusion comparison between γ=1\gamma=1 and 1.011.01?

  3. Does extrapolation consistently outperform the best γ∈[0,1]\gamma\in[0,1] configuration under an equal tuning budget, and how does it compare with a directly tunable interpolation method?

2 comments

W1: which baselines were rerun in the same implementation.

R18: SGDM and AdaBelief are reproduced by us, AMSGrad is searched by us; R50: SGDM and Lookaround reproduced (lr=0.1\mathrm{lr}=0.1, momentum 0.90.9); LLM: the official Sophia codebase under its original protocol; the full diffusion table is self-trained. The fairness is checkable: every reproduced entry uses the original authors' official repository and recommended configuration, and every other entry is the original authors' best value from their paper. Control is further backed by C.6: our AGD reproduction matches the published value within one standard deviation on both architectures (92.2592.25 vs 92.3592.35; 93.0293.02 vs 93.1293.12).

W2 / Q2: statistical reliability.

We answer this concern with measurement rather than wording. Appendix A (red) reports the controlled three-seed distributions of the diffusion anchors: 8.97±0.188.97\pm0.18 / 8.26±0.238.26\pm0.23 / 7.90±0.177.90\pm0.17; the originally reported single-run values (9.11/8.03/7.759.11/8.03/7.75) all fall inside these distributions, which calibrates how much a single run can deviate. In the same table, Anon's seed-paired differences from Adam are in the same direction on all three seeds (IDU mean +0.72+0.72 FID, extrapolation mean +0.36+0.36 FID), which is exactly the signal the paired design isolates from that noise. Small margins in the large-scale single-run results are no longer presented as statistically established. Full ImageNet multi-seed is beyond this rebuttal's compute budget; we state it as a compute bound and make no claims beyond the data.

W3 / Q3: equal-budget comparison with a directly tunable interpolation method, and decomposition of the extrapolation gain.

Direct comparison is added for Padam, with the opponent tuned to its strongest configuration: C.6 now includes a Padam row searched on a budget no smaller than Anon's (p∈0.1,0.25,0.5×lr∈10−3,…,1p\in\\{0.1,0.25,0.5\\}\times\mathrm{lr}\in\\{10^{-3},\ldots,1\\}, configuration fixed on seed 0, confirmed over three seeds). Its best configuration (p=0.25p=0.25, lr=0.1\mathrm{lr}=0.1) reaches 92.14±0.1092.14\pm0.10 / 93.13±0.1593.13\pm0.15. On ResNet32 the best interpolation configuration passes all endpoint baselines (93.1393.13 vs SGDM/AdamW/AMSGradW at 92.95/92.62/92.8492.95/92.62/92.84); on ResNet20 it lands between them (92.1492.14, above AdamW and AMSGradW but below SGDM at 92.3392.33). In both cases Padam still ranks behind Anon (92.47±0.0592.47\pm0.05 / 93.20±0.0893.20\pm0.08). The decomposition of the extrapolation gain is guaranteed by the anchor design: the γ=1\gamma=1 row isolates IDU (seed-paired differences positive on all three seeds), and the γ=1.01\gamma=1.01-versus-γ=1\gamma=1 paired improvements (0.25/0.64/0.180.25/0.64/0.18) are pure adaptivity extrapolation with IDU held fixed. The γ=1\gamma=1 row of GPT2-medium supports IDU rather than extrapolation; that is a feature of the decomposition design, not a defect.

Please engage with the authors during the discussion period.

Reviewer_cq3KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines adaptivity as the sensitivity of an optimizer's preconditioner to gradient rescaling. It uses this view to motivate Anon, which has a fixed parameter gamma that can move beyond the usual range between SGD-like and Adam-like adaptivity. Anon also uses Incremental Delay Update (IDU), which updates the preconditioner at geometrically spaced steps and holds it fixed between updates. The paper gives convergence guarantees under stated assumptions and evaluates the method on image classification, diffusion models, and language modeling. Its main empirical claim is that extrapolating adaptivity can improve performance beyond the conventional range.

Critical strengths
  1. The adaptivity viewpoint gives a useful way to organize optimizer designs. Making this quantity explicit helps separate the effect of coordinate scaling from other design choices. Allowing gamma to move outside the usual range is a reasonable direction to explore.

  2. The paper combines convergence analysis with experiments across several types of models. IDU provides a way to control preconditioner variation while allowing extrapolated adaptivity. The experiments show competitive results, and the paper includes comparisons with and without IDU, as well as repeated runs on CIFAR-10. These give the proposal more support than toy examples alone.

Critical weaknesses

1. My main concern is whether the comparisons are fair.

In particular, have the hyperparameters of each optimizer been tuned carefully enough to get close to its best performance? For example, Appendix C.1 uses a coarse, tenfold-spaced learning-rate grid for SGDM, while Appendix C.6 searches 35 configurations for Anon but uses previously reported settings for the baselines. Could these tuning differences explain some of the reported gains?

2. The explanation of when and why extrapolation helps seems incomplete.

The toy examples show different trajectories, but how do these connect to better performance or a preference for flatter minima? Given the coupling between gamma and the learning rate noted in Appendix A, would extrapolation still outperform interpolation after careful tuning for each gamma? Moreover, Figure 4's best result is at gamma = 0. Why was a negative gamma selected instead in Appendix C.6?

3. The practical motivation for IDU needs a clearer explanation.

The authors seem to motivate IDU mainly through its convergence guarantee, but the bound does not give me a clear intuition for why IDU is needed in practice. Could the authors give a more intuitive explanation of when and why IDU is effective or necessary?

Questions for authors
  1. How should we reconcile Figure 4 with the selected configuration in Appendix C.6? Why was gamma = -0.08 with learning rate 1 selected when the figure's highest value is at gamma = 0 with learning rate 0.5? How do the best zero-gamma and negative-gamma configurations compare over repeated runs using the same selection rule?

  2. Does extrapolation still help when the learning rate is tuned carefully for each gamma? A comparison covering representative values inside [0, 1], both endpoints, and values outside the interval would directly address this. Beyond the suggested search directions for different model families, what evidence explains when extrapolation helps or hurts?

  3. What practical behavior does IDU correct in the main experiments? Can the authors connect the changes in the preconditioner to the observed gains and losses, including the comparisons in Table 12? This would help explain why progressively longer freezing intervals are a useful design choice.

  4. How does Anon relate to earlier gradient-power methods, especially GradPower [1]? The paper discusses Padam and MADA, but I could not find a discussion of GradPower or the earlier Power Gradient Descent [2]. GradPower transforms the gradient before passing it to the base optimizer, so it is not generally the same as powering Anon's second-moment-based preconditioner. A clear comparison of the ideas and a relevant empirical baseline would help establish what Anon adds.

[1] Jinbo Wang, Mingze Wang, Jiaqi Zhang, Wei Wang, Peng Pei, Xunliang Cai, Weinan E, and Lei Wu. GradPower: Powering Gradients for Faster Language Model Pre-Training. ICML, 2026. arXiv:2505.24275.

[2] Marco Baiesi. Power Gradient Descent. arXiv preprint arXiv:1906.04787, 2019.

5 comments

Q1: why Figure 4's argmax (γ=0,η=0.5)(\gamma=0,\eta=0.5) differs from the configuration selected in C.6 (γ=−0.08,η=1)(\gamma=-0.08,\eta=1).

The two are different statistical estimators, and there is no contradiction: Figure 4 reports a single run per cell (now stated in its caption), while C.6 selects by the three-seed mean over the same grid (rule now explicit, red). Under that rule the single-run argmax cell (γ=0,η=0.5)(\gamma=0,\eta=0.5) has a three-seed mean of 92.41±0.1192.41\pm0.11, below the selected cell's 92.47±0.0592.47\pm0.05: the figure's argmax is seed noise, which is exactly why C.6 uses the mean rule. Under the same rule, the zero-γ\gamma cell with the best single-run value, (γ=0,η=0.5)(\gamma=0,\eta=0.5), reaches 92.41±0.1192.41\pm0.11 when re-run over three seeds, and the negative-γ\gamma cell (γ=−0.08,η=1)(\gamma=-0.08,\eta=1) reaches 92.47±0.0592.47\pm0.05, so negative-γ\gamma wins, with about half the standard deviation.

Q2: after per-γ\gamma learning-rate tuning, does extrapolation still beat interpolation?

Yes, on both tasks. CIFAR: on the 35-point joint grid of Figure 4, the best three-seed mean lies at γ=−0.08\gamma=-0.08 (extrapolated), and the in-range cell with the best single-run value, (0,0.5)(0,0.5), loses at 92.41±0.1192.41\pm0.11 when re-run over three seeds. Diffusion (new runs, seed-paired): γ=1\gamma=1 averages 8.26±0.238.26\pm0.23 and γ=1.01\gamma=1.01 averages 7.90±0.177.90\pm0.17, with paired gains of 0.25/0.64/0.180.25/0.64/0.18 FID on the three seeds, all in the same direction.

Q3 / W3: practical intuition and evidence for IDU.

IDU is a low-pass filter on the pre-conditioner path, and it acts where that path varies most. Evidence: (i) the γ=1\gamma=1 diffusion row is Adam+IDU, and its seed-paired difference from Adam is positive on all three seeds (1.06/0.28/0.811.06/0.28/0.81, mean 0.720.72 FID), with Anon the only changed factor; (ii) shortening the second-moment window (β2\beta_2 0.99→0.750.99\to0.75) makes IDU consistently +0.12∼0.32+0.12\sim0.32 with lower variance (C.7, Table 12), while at standard β2\beta_2 its effect is small and mixed (−0.10-0.10 to +0.16+0.16, Table 13), and at γ=1.1\gamma=1.1 with standard β2\beta_2 the difference is −0.04-0.04: where the pre-conditioner path is already smooth, IDU neither helps nor hurts, which matches the prediction that it acts where variation is large. The counter-case is stated as openly: low-noise CIFAR-10 classification does not engage the mechanism (AMSGradW$-$AdamW is only +0.17/+0.22+0.17/+0.22, Table 11), which is why Table 2's CNN rows show no IDU effect. The mechanism's benefit is task-dependent, and we do not claim otherwise.

W1: tuning fairness (coarse grid in C.1 vs 35 configurations for Anon in C.6).

Our protocol withstands line-by-line inspection (Appendices C.1/C.2, red). Classification baselines are taken from the original inventing papers at their authors' best-tuned configurations; the entries we reproduce ourselves (SGDM/AdaBelief on R18, SGDM/Lookaround on R50) use the original authors' official repositories and recommended settings; where an original configuration is absent (e.g., AMSGrad), we extend the search on a {0.1,0.01,0.001}\{0.1,0.01,0.001\} grid rather than narrowing anyone's. On the diffusion side, Anon's learning rate is searched per our stated protocol over {0.1,…,10−5}\{0.1,\ldots,10^{-5}\}, and γ\gamma adds only 2-4 trials. The added Padam comparison (see our reply to R3) receives a search budget (p×lrp\times\mathrm{lr}, 24 configurations) no smaller than Anon's. The tenfold grid in C.1 is itself the AdaBelief paper's protocol, under which SGDM is the most fully tuned method in the table.

Q4 / W2: relation to GradPower and Power Gradient Descent.

A dedicated discussion is added in Appendix A (red). Both works transform the raw gradient (ϕp(g)=sign(g)∣g∣p\phi_p(g)=\mathrm{sign}(g)|g|^p, power applied per sample before averaging), whereas Anon powers the pre-conditioner statistic (EMA(g2)\mathrm{EMA}(g^2), average then power); these are different functionals of the gradient history with different noise profiles. Their exponent is a fixed convergence-speed transformation, not an extrapolable adaptivity axis; neither addresses the [0,1][0,1] confinement, and neither provides convergence guarantees for the powered dynamics. The directions are complementary (GradPower can wrap any base optimizer, including Anon). Our work was developed in 2025 and its initial submission predates GradPower's ICML 2026 publication; the two lines are independent (now noted in Appendix A). If an empirical counterpart is desired, we will add an AdamWPower row in the camera ready.

Thanks for the clarifications and extra experiments. My concerns have been partially addressed. I still have questions about tuning fairness and the practical motivation for IDU. I'll keep my score at 2.

Thank you for the follow-up. To make the two remaining points actionable: on tuning fairness, the Padam row in C.6 is a directly tunable interpolation baseline tuned on a budget no smaller than Anon's, and it still ranks behind Anon on both architectures; what concrete control beyond this would you require? On the practical motivation for IDU, the γ=1\gamma=1 configuration is Adam+IDU with every other factor identical, and its seed-paired FID difference from Adam is positive on all three seeds; what direct measurement further would you consider sufficient? Both points, with full protocols and data, are in the uploaded revision; if a specific experiment is still missing, naming it would let us add it in the camera ready. On tuning fairness specifically: which experiment's protocol do you consider unfair, and in what respect?

Reviewer_oiMqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a first-order optimizer that extends the degree of gradient adaptivity beyond the conventional range between SGD (adaptivity 0) and Adam (adaptivity$\approx$1). It evaluates Anon on ImageNet classification, diffusion models, and GPT-2 language modeling. The method reports competitive or improved performance over several established optimizers.

Critical strengths

The paper offers a novel stabilization mechanism (IDU) to control preconditioner variation without enforcing the monotonicity constraint used by AMSGrad. And it provides convergence guarantees in both convex and nonconvex settings, supporting the proposed mechanism beyond empirical observations.

Critical weaknesses
  1. Anon introduces γ\gamma, whose optimal value varies by task and interacts with learning rate. The paper does not establish that the benefits outweigh the additional tuning cost.
  2. Language-model experiments are restricted mainly to GPT-2 small/medium, with single-run results. This is insufficient to establish scalability to modern billion-parameter LLM training.
Questions for authors

How sensitive is performance to the joint choice of γ\gamma and learning rate? Does Anon remain advantageous when all methods receive the same tuning budget?

2 comments

W1 / Q: tuning cost of γ\gamma, joint (γ,lr)(\gamma,\mathrm{lr}) sensitivity, and whether Anon remains advantageous under an equal tuning budget.

The search over γ\gamma does not have to be a joint scan; it can be done sequentially: fix γ\gamma at the anchor of the stronger base recipe (γ=1\gamma=1 for Adam-type pipelines, γ=0\gamma=0 for SGD-type ones), tune the remaining hyperparameters exactly as one would for that base optimizer (Anon coincides with the Adam and SGD families at the anchors, so all existing tuning knowledge carries over), and then adjust γ\gamma alone. In our production-project experience this protocol typically suffices to surpass the default baselines, so the added cost is O(n+m)O(n+m) rather than the O(nm)O(nm) of jointly searching an extra hyperparameter (protocol and directional prior in Appendix A, red). The joint sensitivity itself is measured: Figure 4 is a complete γ×η\gamma\times\eta grid, and the optimal region is a diagonal band rather than an isolated point.

One environment-sensitivity finding, reported as-is since it bears on this question: when reproducing the diffusion recipe (γ=1.01\gamma=1.01, lr=10−4\mathrm{lr}=10^{-4}) with fixed seeds in a refreshed software stack (newer PyTorch/CUDA), training diverged reproducibly within the first 55k steps under the default base-22 blocks. The original base-22 results were trained healthily to 500500k steps on the earlier stack, so we attribute this to numerical sensitivity of the long frozen blocks rather than to the update rule itself. Reducing the block base to 1.51.5 (as we prove, the O(T)O(\sqrt{T}) bound holds for any geometric base) fully restores robustness, and the FID matches the original base-22 runs. Algorithm 2, §3.3 and the γ=1.01\gamma=1.01 row of Table 3 are updated accordingly (red; case study in Appendix A).

W2: LLM experiments cover only GPT-2 small/medium, single run.

The controlled three-seed replication in Appendix A (reproduction table) reports 8.97±0.188.97\pm0.18 / 8.26±0.238.26\pm0.23 / 7.90±0.177.90\pm0.17 for the adam / Anon γ=1\gamma=1 / Anon γ=1.01\gamma=1.01 anchors; the originally reported single-run values (9.11/8.03/7.759.11/8.03/7.75) all lie within these distributions, and the seed-paired differences are uniformly favorable to Anon (IDU +0.72+0.72 FID on average, extrapolation +0.36+0.36). The Qwen2-1B+Muon study (Table 9) already covers the billion-parameter scale as a proof of concept; what remains future work is multi-seed, larger-scale validation, and we do not argue otherwise. Note that the evaluation already spans three task families with different architectures and data regimes (image classification, diffusion, and language modeling), each with an evaluation protocol documented in Appendices C.1–C.3. For GPT-2, the training curves preserve the same ordering (γ=1.1\gamma=1.1 best) at every reported checkpoint (appendix). We believe these results have demonstrated the advantages of Anon as a general-purpose optimizer across task families.

Please engage with the authors during the discussion period.

Comments on the paper

1 comment

We thank all reviewers. A revised PDF has been uploaded; all changes are marked in red:

  1. Algorithm 2 + §3.3: IDU block schedule generalized to a geometric base rr (default 22; r=1.5r=1.5 in the γ=1.01\gamma=1.01 diffusion re-runs); the O(T)O(\sqrt{T}) bound holds for any r>1r>1;
  2. Appendix A: case study on the numerical-environment sensitivity of the default base-22 blocks observed when re-running in a refreshed software stack (the original base-22 runs trained healthily), resolved by r=1.5r=1.5 + a three-seed reproduction table of the diffusion anchors;
  3. Table 3 caption + §4.2: base-1.51.5 blocks noted for the γ=1.01\gamma=1.01 row;
  4. Appendix C.6: direct Padam benchmark with full search protocol (p×ηp\times\eta grid, three-seed confirmation) and the explicit three-seed-mean selection rule;
  5. Appendices C.1/C.2: exact list of baselines rerun in our implementation, all from the original authors' official repositories and recommended configurations; our own searches only extend where originals are absent;
  6. Appendix A: discussion of GradPower and Power Gradient Descent, including the timeline relative to our initial submission; a sequentially separable γ\gamma-tuning protocol (O(n+m)O(n+m) added cost);
  7. Figure 4 caption: single run per cell noted;
  8. Small margins in single-run large-scale results are no longer presented as statistically established; the controlled multi-seed evidence is in Appendix A and C.6.

Requests that are compute-bound rather than method-bound (multi-seed validation at 1B and beyond, full ImageNet multi-seed) are stated as such in the revision; everything testable within the rebuttal window has been tested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.