- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper defines adaptivity through a preconditioner’s response to scaling the gradient history and proposes Anon, an element-wise optimizer with a real-valued adaptivity parameter. Its Incremental Delay Update mechanism updates the preconditioner at geometrically spaced steps, supporting convergence analysis without requiring monotonic preconditioners. Experiments on image classification, diffusion, and language modeling suggest that modest extrapolation beyond SGD- and Adam-level adaptivity can improve performance.
Critical strengths
-
The adaptivity formulation provides a coherent perspective on diagonal optimizers. Characterizing gradient-scale response gives a precise interpretation of the SGD and Adam endpoints and motivates examining values outside their usual interpolation range. This offers a useful framework for studying task-dependent optimization behavior.
-
IDU is a concrete algorithmic contribution accompanied by theoretical results. The paper provides convex regret and non-convex stationarity bounds and acknowledges that their constants deteriorate with the extrapolation parameter. Experiments span several model families, and the diffusion comparison at fixed provides evidence that IDU can contribute independently of extrapolation.
Critical weaknesses
-
The fairness of the main baseline comparisons is not sufficiently established. Section 4 states that baseline metrics outside the diffusion study are sourced from prior publications. Even when these are strong published results, differences in implementations, training recipes, and tuning protocols can confound optimizer comparisons. The controlled CIFAR-10 experiment in the appendix is useful, but does not establish that the principal ImageNet and language-model comparisons are equally controlled.
-
The statistical reliability of the reported improvements is unclear. Tables 2–4 report single runs without uncertainty estimates. Some improvements are small, including 0.12 percentage points over SGDM on ResNet18 and 0.03 points over HVAdam on ResNet50. Likewise, the diffusion improvement from FID 8.03 at to 7.75 at needs repeated runs to establish its robustness. This limits the evidence for a consistently beneficial optimizer modification.
-
The incremental benefit of extrapolation is not convincingly separated from IDU and hyperparameter selection. The central motivation concerns exceeding the capabilities of interpolation-based methods, yet the main experiments do not directly compare with the cited tunable alternatives Padam and MADA. Moreover, GPT2-medium uses , so its improvement supports IDU rather than extrapolation. The strongest evidence would compare the best in-range and extrapolated configurations under matched joint learning-rate/adaptivity searches.
Questions for authors
-
Which ImageNet and language-model baselines were rerun in the same implementation, and can the principal comparisons be established under identical training and tuning protocols?
-
What is the variation across seeds for the main improvements, particularly the modest ImageNet margins and the diffusion comparison between and ?
-
Does extrapolation consistently outperform the best configuration under an equal tuning budget, and how does it compare with a directly tunable interpolation method?
2 comments
Please engage with the authors during the discussion period.
W1: which baselines were rerun in the same implementation.
R18: SGDM and AdaBelief are reproduced by us, AMSGrad is searched by us; R50: SGDM and Lookaround reproduced (lr=0.1, momentum 0.9); LLM: the official Sophia codebase under its original protocol; the full diffusion table is self-trained. The fairness is checkable: every reproduced entry uses the original authors' official repository and recommended configuration, and every other entry is the original authors' best value from their paper. Control is further backed by C.6: our AGD reproduction matches the published value within one standard deviation on both architectures (92.25 vs 92.35; 93.02 vs 93.12).
W2 / Q2: statistical reliability.
We answer this concern with measurement rather than wording. Appendix A (red) reports the controlled three-seed distributions of the diffusion anchors: 8.97±0.18 / 8.26±0.23 / 7.90±0.17; the originally reported single-run values (9.11/8.03/7.75) all fall inside these distributions, which calibrates how much a single run can deviate. In the same table, Anon's seed-paired differences from Adam are in the same direction on all three seeds (IDU mean +0.72 FID, extrapolation mean +0.36 FID), which is exactly the signal the paired design isolates from that noise. Small margins in the large-scale single-run results are no longer presented as statistically established. Full ImageNet multi-seed is beyond this rebuttal's compute budget; we state it as a compute bound and make no claims beyond the data.
W3 / Q3: equal-budget comparison with a directly tunable interpolation method, and decomposition of the extrapolation gain.
Direct comparison is added for Padam, with the opponent tuned to its strongest configuration: C.6 now includes a Padam row searched on a budget no smaller than Anon's (p∈0.1,0.25,0.5×lr∈10−3,…,1, configuration fixed on seed 0, confirmed over three seeds). Its best configuration (p=0.25, lr=0.1) reaches 92.14±0.10 / 93.13±0.15. On ResNet32 the best interpolation configuration passes all endpoint baselines (93.13 vs SGDM/AdamW/AMSGradW at 92.95/92.62/92.84); on ResNet20 it lands between them (92.14, above AdamW and AMSGradW but below SGDM at 92.33). In both cases Padam still ranks behind Anon (92.47±0.05 / 93.20±0.08). The decomposition of the extrapolation gain is guaranteed by the anchor design: the γ=1 row isolates IDU (seed-paired differences positive on all three seeds), and the γ=1.01-versus-γ=1 paired improvements (0.25/0.64/0.18) are pure adaptivity extrapolation with IDU held fixed. The γ=1 row of GPT2-medium supports IDU rather than extrapolation; that is a feature of the decomposition design, not a defect.