acceptodds
Withdrawn from ICLR 2027

Minimal-State Optimization: How Little Optimizer Memory Does AdamW’s Accuracy Need?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adaptive optimizers such as AdamW store two extra tensors per parameter (first and second moments), doubling the model’s memory footprint. How much of this optimizer state is actually necessary to match AdamW’s accuracy? We study this through A∗Grad, a learning-rate-free optimizer whose step size is set by a Polyak rule with a trust re- gion, and whose only tunable hyperparameter is a dimensionless ratio ρ. By varying only the geometry of the update direction, A∗Grad spans a spectrum of optimizer-state sizes. We find the required state is task-dependent, and a sharpness diagnosis predicts it: the sign-based step (zero state) discards gradient magnitude and converges to min- ima 5–10× sharper than SGD’s (filter-normalized perturbation), which hurts generaliza- tion. Restoring magnitude resolves this issue, though the optimal restoration mecha- nism is strictly task-dependent. For from-scratch vision (CIFAR-100/WRN), a global L2- normalized step with weight decay suffices at zero optimizer state, outperforming AdamW by +2.4%p. For Transformer fine-tuning (BERT/MNLI), per-coordinate RMS is needed; at half AdamW’s state, it outperforms it by +0.5%p with the smallest seed variance among all methods. We place A∗Grad against two recent low-memory optimizers, MicroAdam and NanoAdam, on an accuracy–memory Pareto front, and delineate the operational regimes where A∗Grad demonstrates clear advantages (stability, zero-state vision, hyperparameter- free operation) versus scenarios where its benefits are attenuated (certain fine-tuning tasks, peak accuracy at scale). The code of the proposed optimizer is publicly available at Github (https://anonymous.4open.science/r/astar_grad-D001).

Primary area:
optimization
Keywords:
Optimizer, Learning-Rate-Free, Sharpness-guided

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_x46dOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary
  • Learning-rate-free optimizer: Polyak step + trust cap ρ∥w∥2/∥d∥2\rho\|w\|_2/\|d\|_2
  • Directions: sign, global L2L_2, per-coordinate RMS
  • Sharpness diagnosis → vision needs 0 state, Transformers need ½ of AdamW's
  • Claims to match or beat AdamW on CIFAR-100 and BERT fine-tuning; weight-norm bound as non-divergence guarantee
Critical strengths
  • Useful question (how much optimizer state is needed) + decent writing
Critical weaknesses
  • Weight-decay experimental issue:
    • only A$^*$Grad gets wd; AdamW and SGD at wd 00
    • ablation says: L2L_2 without wd =71.8≈= 71.8 \approx AdamW 71.6571.65; with wd →74.1\to 74.1
    • single-seed ablation (pg 3) stated as confirming the sharpness diagnostics, does not make sense.
    • baseline settings only in Appendix A, Table 5 (pg 8) + no CIFAR lr sweep described
  • Rest doesn't compensate:
    • BERT +0.52+0.52 < AdamW seed SD (0.640.64); AdamW learning rate swept over 2 values
    • Corollary 1 vacuous: (1.0015)36,800≈1023(1.0015)^{36{,}800} \approx 10^{23}
    • paper is ≈ ALI-G + LARS trust ratio, neither are cited in related works, seems like an oversight.
Questions for authors
  • (Clarification) rerun CIFAR with wd, momentum, and a schedule for all baselines
  • (Clarification) put baseline settings in the main text; report CIFAR sweeps; 3-seed ablation
  • (Clarification) differences from ALI-G and LARS? (normalized-L2L_2 uncapped step = ALI-G with L∗=0L^* = 0)
  • (Clarification) was sharpness measured on BERT?
  • (Clarification) number of AdamW seeds in Table 1 (no std shown); wider learning-rate sweep
  • Suggestions: final vs. best test accuracy; fix Corollary 1; add Adafactor, Adam-mini, D-Adaptation, Prodigy, Schedule-Free
  • Typos: missing years (Li et al., Polyak, Modoranu et al.); "Generative AI AI tools" (pg 7); incomplete references (pg 7)
Reviewer_Ks6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether reduced optimizer state can retain AdamW-level accuracy by combining a smoothed Polyak step and a relative-update cap with sign, L2-normalized, and RMS directions. Reducing memory is a clear practical goal. The more distinctive claim, however, is that sharpness can predict the required optimizer state, and the limited configurations evaluated do not yet establish this predictive ability. Experiments compare accuracy and optimizer memory on vision and BERT tasks, including comparisons with low-memory alternatives.

Critical strengths
  1. Section 2 compares three update directions within a common step-size framework and clearly accounts for the difference between storing no per-parameter state and storing one second-moment tensor.

  2. Tables 1-2 demonstrate that competitive accuracy can be retained with less optimizer state on the evaluated vision and BERT tasks.

  3. Sections 5, 6.2, and 8 candidly report performance degradation on some tasks, the lack of a speed advantage, and hardware limitations. These negative results help clarify the method's practical scope.

Critical weaknesses
  1. The central claims about minimum state requirements and sharpness-based prediction exceed the evidence. Table 1 shows that stateless SGD already achieves vision accuracy comparable to A*Grad-L2+wd, while NanoAdam offers a different low-state tradeoff. Section 3 measures perturbation losses only for sign and SGD at CIFAR checkpoints, without testing the proposed remedies or extending the diagnosis to BERT. These results establish the sufficiency of selected configurations, but not minimum state requirements across methods or a validated rule for predicting them.

  2. The baseline configurations leave the accuracy comparisons insufficiently controlled. The proposed vision configuration uses weight decay, whereas Appendix Table 5 sets AdamW's weight decay to zero, and the BERT baselines use neither warmup nor learning-rate schedules. The reported gains therefore do not adequately separate the method's contribution from the effects of baseline configuration.

  3. The bound ∣∣wt∣∣≤(1+rho)t∣∣w0∣∣||w_t|| \leq(1+rho)^t ||w_0|| in Section 6.1 permits exponential growth of the parameters. It bounds update magnitudes over a finite number of steps but does not establish the claimed non-divergence guarantee or exclude losses as large as 4×10294 \times 10^{29}.

  4. Appendix A reports the best test accuracy observed during CIFAR training, so the test set participates in checkpoint selection. The resulting comparisons do not adequately establish generalization performance under an independent checkpoint-selection rule.

Questions for authors
  1. To which candidate optimizer family does the minimum-state claim apply? Can sharpness measurements for sign, L2, L2 with weight decay, and RMS at matched checkpoints support a prespecified rule that predicts the effective direction on a new task?

  2. Do the gains in Tables 1-2 persist when AdamW is tuned under a comparable budget for weight decay, warmup, and learning-rate schedules, and checkpoints are selected independently of the test set? Please report the resulting accuracy differences and intervals across seeds.

  3. Can you restrict the conclusion in Section 6.1 to a bound on relative update size, or provide additional assumptions that rule out long-term divergence of the parameters and loss?

Reviewer_AFyROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how much optimizer state is needed to match AdamW’s accuracy through A*Grad, which combines a Polyak-type step size with a cap on the relative update magnitude. It compares sign-based, L2-normalized, and coordinate-wise RMS-normalized directions, reporting favorable results with an approximately zero-state L2 variant for image classification and an RMS variant using half of AdamW’s state for BERT fine-tuning. The paper further presents sharpness as a diagnostic for the required state size and provides a theoretical guarantee concerning update stability.

Critical strengths

Studying the relationship between update geometry and optimizer state within a common update framework is interesting. The paper evaluates some main results across multiple seeds and reports task-dependent performance reversals, the absence of a speed advantage, and hardware limitations affecting the comparisons. The different empirical trends observed in image classification and BERT fine-tuning warrant further investigation.

Critical weaknesses
  1. The central claims lack sufficient explanation and validation despite substantial room within the page limit.

    The research content occupies approximately six pages, leaving substantial room within the main-text page limit. Nevertheless, the claim that sharpness predicts the required optimizer state is repeated in the Abstract, Introduction, experimental sections, and Conclusion without directly measuring how sharpness changes under the L2 and RMS variants. Success or failure among a small set of update directions also does not establish the minimum state required for each task, and the analysis is insufficient to support this central claim.

  2. The evaluation covers too few models, datasets, and training settings to support the generality of the claims.

    The main vision evaluation uses WRN-16-8 trained from scratch on CIFAR-100, whereas the language evaluation uses BERT-family fine-tuning, simultaneously changing the domain, architecture, and training stage. This design cannot distinguish which factor explains the observed differences in required state, and scaling BERT or adding GLUE tasks within the same model family does not resolve this confounding. As a useful example of evaluation design, Sharpness-Aware Minimization by Foret et al. (2021) evaluates multiple models and datasets, including CIFAR-10/100, ImageNet, and fine-tuning tasks, while also analyzing Hessian spectra and the relationship between sharpness and generalization. I am not requesting an identical computational budget, but this paper needs comparisons across model families on the same dataset, across datasets within the same model family, and under controlled training stages. The current evaluation does not support general conclusions about task-dependent minimum state or a diagnostic principle for identifying it.

Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-Aware Minimization for Efficiently Improving Generalization. ICLR, 2021.

  1. The main AdamW comparison does not adequately establish the superiority of the update rule.

    On CIFAR-100, weight decay is applied to the proposed method but set to zero for AdamW, making the baseline update equivalent to Adam when the remaining settings are identical. The paper’s own ablation reports an improvement of approximately 2.3 percentage points from adding weight decay, suggesting that the reported 2.4-point advantage over AdamW may substantially reflect different regularization conditions. An advantage over AdamW should not be emphasized without comparison against AdamW with an appropriately tuned weight-decay coefficient.

  2. The “non-divergence guarantee” overstates what is proved.

    Section 6.1 bounds the relative update magnitude and establishes the exponential bound ∥wt∥2≤(1+ρ)t∥w0∥2\|w_t\|_2\le(1+\rho)^t\|w_0\|_2. This bound guarantees neither uniform boundedness over time nor convergence of the loss, and it permits the parameter norm to grow without bound as the number of iterations increases. In particular, the statement that divergence is “strictly precluded” for any finite ρ\rho does not follow from this corollary; without a stronger result, the theoretical contribution should be characterized as a bound on relative update magnitude.

  3. Inconsistencies between experimental settings and reported results prevent the main comparisons from being traced reliably.

    Appendix Table 4 lists a NanoAdam learning rate of 7×10−57\times10^{-5}, whereas the accompanying text identifies 3×10−43\times10^{-4} as the full-training optimum. Since the paper also states that each method uses its best setting, it is unclear which learning rate produced the main results. Furthermore, Table 1 is described as reporting means and standard deviations over three seeds, but the BERT-base AdamW result lacks a standard deviation, preventing adequate assessment of baseline variability and the robustness of the reported improvement.

  4. The presentation mixes research findings with development history and self-assessments of the authors’ reporting practices.

    The NanoAdam implementation correction in Section 5 and the discovery of missing bias correction in Section 6.2 are useful reproducibility details, but the Introduction elevates them to independent methodological contributions. Statements such as “Evaluating on a single seed would have published a fragile algorithm” (Section 6.2), “and we say so rather than hide it” (Reproducibility Statement), and “the claims are neither over- nor under-stated” (Ethics Statement) concern hypothetical development outcomes or the authors’ assessments of their own transparency and calibration, rather than validated research findings. These statements do not provide scientific support; given the available space, the manuscript should prioritize final comparison conditions, information needed for reproduction, and direct evidence for its central claims.

  5. The references are insufficiently prepared.

    Several in-text citations, including “Li et al.,” “Modoranu et al.,” and “Polyak,” lack publication years, while bibliography entries for Li et al., MicroAdam, Polyak, and NanoAdam omit years or venue information. The surname “zhou” is also left in lowercase, indicating a need to verify and standardize the bibliographic information. These issues alone would not justify rejection, but together with the experimental inconsistencies, they raise concerns about the manuscript’s readiness and verifiability.

Questions for authors
  1. Can you directly validate the sharpness-based diagnosis across update variants and separate the effects of model architecture, dataset, and training stage?

  2. Do the main conclusions hold against AdamW with appropriately tuned weight decay and a comparable hyperparameter search budget?

  3. How does the exponential norm bound in Section 6.1 establish non-divergence when it permits unbounded parameter growth?

  4. Can you reconcile the conflicting NanoAdam learning rates and provide the missing standard deviation for BERT-base AdamW?

Even if these questions are satisfactorily answered, addressing the broader concerns would require substantial additional validation and a comprehensive rewrite of the manuscript; I therefore do not consider it ready for acceptance in its current form.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.