acceptodds
Under review as a conference paper at ICLR 2027

PowerStep: Memory-Efficient Adaptive Optimization via -Norm Steepest Descent

Abstract

Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by -norm steepest descent, PowerStep applies a signed-power transform directly to one momentum buffer. We establish a finite-horizon stationarity bound for exact, unregularized updates, with an term and a noise-dependent residual. Experiments on Transformers from 124M to 235B parameters show competitive validation quality while halving optimizer-state memory. Combined with uniform quantization, PowerStep remains numerically stable and reduces optimizer-state memory by compared to AdamW. PowerStep thus provides a simple, memory-efficient alternative for large-scale training.

Primary area:
optimization
Keywords:
Adam, AdamW, adaptive optimization, l_p norm

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FtncOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to power transform the Polyak momentum. With this transform, the optimizer

Critical strengths
  1. The design and implementation are hard-efficient
  2. The empirical experiments are comprehensive
Critical weaknesses
  1. Novelty concern without citation: This paper's proposal is almost identical to an older paper

Chen, Lizhang, et al. "Lion secretly solves a constrained optimization: As lyapunov predicts." International Conference on Learning Representations. Vol. 2024. 2024.

  1. The theoretical results are also substantially weaker and narrower than the above paper, with constraints and assumption of globally bounded true gradients, plus smoothness and excluding anaylsis of weight decay (though practically it needs weight decay to show meaningful results)

  2. The experiment does not show substantial gain in terms of model quality

  3. Efficiency claim is not well established. It handicaps 8-bit Adam with int8 optimizer states but have not compared with more widely used baselines with fp8. In addition, When comparing against optimizers with a single copy of optimizer states, no comparison is against Muon and Adafactor.

  4. Larger experiments are not trained to Chinchilla optimal. But smaller models are extremely overtrained.

Questions for authors

N/A

Reviewer_vugdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PowerStep, a memory-efficient optimizer that applies an elementwise signed-power transformation to accumulated momentum. Motivated by steepest descent under the ℓp\ell_p norm, the method performs coordinate-wise scaling with a single momentum buffer, eliminating the need to store second-moment estimates. PowerStep also offers improved compatibility with uniform int8 state quantization, further reducing optimizer-state memory requirements.

Critical strengths
  1. This paper addresses the important problem of optimizer-state memory overhead, which can become a bottleneck in large-scale model training.

  2. PowerStep is supported by clear technical motivation, convergence analysis, and experiments on models ranging from 123M to 235B parameters. Together, these contributions constitute a comprehensive study.

Critical weaknesses
  1. The connection between general normed spaces and the memory efficiency and quantization robustness of optimizer states is unclear to me. In particular, the paper lacks an intuitive explanation of why PowerStep can outperform Adam in these respects.

  2. Despite covering model sizes up to 235B parameters, all experiments focus on Transformer language models, leaving generalization to other architectures and tasks untested.

Questions for authors

Do Eqs. (7) and (8) explain why PowerStep performs well? In my view, the first-moment estimate may have a small magnitude due to cancellation between positive and negative gradients and may fluctuate frequently. This behavior seems inconsistent with the intended role of preconditioned momentum.

Reviewer_aqCQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PowerStep, a memory-efficient optimizer that replaces AdamW's second-moment state with a signed-power transform applied to a single momentum buffer. The method is motivated by ℓp\ell_p-norm steepest descent and aims to retain coordinate-wise adaptivity while reducing optimizer-state memory; the paper also provides a finite-horizon convergence analysis for the exact unregularized update. Experiments on dense and MoE Transformers from 124M to 235B parameters show validation losses close to AdamW while halving FP32 optimizer-state storage, and an int8 version further reduces optimizer-state memory by about 8× relative to FP32 AdamW.

Critical strengths
  1. PowerStep obtains coordinate-wise adaptive updates from a single momentum buffer through a signed-power transform. This directly reduces persistent optimizer memory with very little algorithmic complexity.

  2. The signed-power rule is derived from ℓp\ell_p-norm steepest descent, and the paper provides a finite-horizon stationarity analysis for the resulting optimizer.

Critical weaknesses
  1. The authors train models up to 235B parameters for only 20K steps, or about 10.5B tokens, and evaluate these pretraining checkpoints mainly with validation loss and only one task. They do not show whether PowerStep can still match AdamW after long-horizon training or on other downstream tasks.

  2. The main experiments largely compare PowerStep with full-state AdamW, while methods such as Adam-mini [1], GaLore [2], and LDAdam [3] are only discussed rather than evaluated under matched training settings. As a result, it is unclear whether PowerStep offers a better quality–memory–throughput trade-off than existing memory-efficient alternatives.

[1] Zhang, Yushun, et al. "Adam-mini: Use fewer learning rates to gain more." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Zhao, Jiawei, et al. "Galore: Memory-efficient llm training by gradient low-rank projection." arXiv preprint arXiv:2403.03507 (2024).

[3] Robert, Thomas, et al. "Ldadam: Adaptive optimization from low-dimensional gradient statistics." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors
  1. Can the authors report end-to-end peak training memory and throughput for the large-scale experiments, rather than only optimizer-state storage?

  2. Can the authors compare PowerStep with strong memory-efficient optimizers such as Adam-mini [1], GaLore [2], or LDAdam [3] under matched model, training budget, and hardware settings?

[1] Zhang, Yushun, et al. "Adam-mini: Use fewer learning rates to gain more." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Zhao, Jiawei, et al. "Galore: Memory-efficient llm training by gradient low-rank projection." arXiv preprint arXiv:2403.03507 (2024).

[3] Robert, Thomas, et al. "Ldadam: Adaptive optimization from low-dimensional gradient statistics." International Conference on Learning Representations. Vol. 2025. 2025.

Comments on the paper

1 comment

We thank the Area Chairs and reviewers for their careful consideration and constructive feedback. We would like to summarize the scope of our claims and the contributions we hope the reviewers will consider.

1. Comparable loss with reduced optimizer-state memory. We do not claim that PowerStep consistently achieves lower loss than AdamW or its variants. Our goal is to investigate whether competitive, comparable loss can be achieved with less optimizer-state memory. For example, in the 50K-step experiments, the reported PowerStep and Adam-mini losses are 2.9568 vs. 2.9547 on GPT-2-Small and 2.7214 vs. 2.7207 on GPT-2-Medium. These numbers are close, although they come from historical multi-seed PowerStep runs and single-seed Adam-mini runs with differences in validation samples and protocol; they do not establish statistical equivalence or a definitive ranking.

PowerStep stores one momentum buffer and no second-moment state. With FP32 states, its measured optimizer-state size is close to Adam-mini's: both are approximately 1.32 GiB/device in our GPT-2-Medium measurements. With our uniform INT8 momentum recipe, PowerStep uses 0.341 GiB/device, compared with approximately 1.322 GiB/device for the measured FP32 Adam-mini configuration. This comparison uses different state precisions and does not establish an advantage over a quantized Adam-mini implementation.

2. Removing the second moment and connecting geometry with adaptive optimization. Adafactor and Adam-mini reduce memory by approximating or sharing second-moment statistics, while retaining a second-moment-based scaling mechanism. Our results provide empirical evidence that a separate second-moment state can be removed in the tested Transformer settings while maintaining competitive loss. This is a different design point for memory-efficient adaptive optimization, rather than a claim that a second moment is unnecessary for every task or training regime.

The paper also develops a theoretical connection between ℓp\ell_p steepest-descent geometry and momentum-dependent coordinate scaling. The signed-power transformation can be interpreted as coordinate-wise scaling of the same momentum buffer that supplies the update direction. This connection helps explain how a single-state update can exhibit coordinate-dependent behavior without reconstructing Adam's second moment. We acknowledge that Powerball previously connected ℓp\ell_p geometry to signed-power updates, and that the momentum update is contained in the Lion-K family. We will cite and distinguish these works clearly; our contribution is the interpretation and study of this design in the practical setting, not the invention of the underlying update family.

3. Empirical evidence at large scale. Our experiments extend the study from GPT-2 models to dense and MoE Transformers, including a 235B-parameter MoE model. We believe that evaluating this single-state optimizer and its quantized version at that scale contributes empirical knowledge relevant to large-model training. At the same time, the 235B experiment covers 20K steps, or approximately 10.49B token positions, and does not establish long-horizon equivalence or superiority over other memory-efficient optimizers at that scale.

We appreciate the reviewers' feedback and will revise the paper to distinguish these findings from stronger claims that our current evidence does not support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.