- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to power transform the Polyak momentum. With this transform, the optimizer
Critical strengths
- The design and implementation are hard-efficient
- The empirical experiments are comprehensive
Critical weaknesses
- Novelty concern without citation: This paper's proposal is almost identical to an older paper
Chen, Lizhang, et al. "Lion secretly solves a constrained optimization: As lyapunov predicts." International Conference on Learning Representations. Vol. 2024. 2024.
-
The theoretical results are also substantially weaker and narrower than the above paper, with constraints and assumption of globally bounded true gradients, plus smoothness and excluding anaylsis of weight decay (though practically it needs weight decay to show meaningful results)
-
The experiment does not show substantial gain in terms of model quality
-
Efficiency claim is not well established. It handicaps 8-bit Adam with int8 optimizer states but have not compared with more widely used baselines with fp8. In addition, When comparing against optimizers with a single copy of optimizer states, no comparison is against Muon and Adafactor.
-
Larger experiments are not trained to Chinchilla optimal. But smaller models are extremely overtrained.
Questions for authors
N/A
We thank the Area Chairs and reviewers for their careful consideration and constructive feedback. We would like to summarize the scope of our claims and the contributions we hope the reviewers will consider.
1. Comparable loss with reduced optimizer-state memory. We do not claim that PowerStep consistently achieves lower loss than AdamW or its variants. Our goal is to investigate whether competitive, comparable loss can be achieved with less optimizer-state memory. For example, in the 50K-step experiments, the reported PowerStep and Adam-mini losses are 2.9568 vs. 2.9547 on GPT-2-Small and 2.7214 vs. 2.7207 on GPT-2-Medium. These numbers are close, although they come from historical multi-seed PowerStep runs and single-seed Adam-mini runs with differences in validation samples and protocol; they do not establish statistical equivalence or a definitive ranking.
PowerStep stores one momentum buffer and no second-moment state. With FP32 states, its measured optimizer-state size is close to Adam-mini's: both are approximately 1.32 GiB/device in our GPT-2-Medium measurements. With our uniform INT8 momentum recipe, PowerStep uses 0.341 GiB/device, compared with approximately 1.322 GiB/device for the measured FP32 Adam-mini configuration. This comparison uses different state precisions and does not establish an advantage over a quantized Adam-mini implementation.
2. Removing the second moment and connecting geometry with adaptive optimization. Adafactor and Adam-mini reduce memory by approximating or sharing second-moment statistics, while retaining a second-moment-based scaling mechanism. Our results provide empirical evidence that a separate second-moment state can be removed in the tested Transformer settings while maintaining competitive loss. This is a different design point for memory-efficient adaptive optimization, rather than a claim that a second moment is unnecessary for every task or training regime.
The paper also develops a theoretical connection between ℓp steepest-descent geometry and momentum-dependent coordinate scaling. The signed-power transformation can be interpreted as coordinate-wise scaling of the same momentum buffer that supplies the update direction. This connection helps explain how a single-state update can exhibit coordinate-dependent behavior without reconstructing Adam's second moment. We acknowledge that Powerball previously connected ℓp geometry to signed-power updates, and that the momentum update is contained in the Lion-K family. We will cite and distinguish these works clearly; our contribution is the interpretation and study of this design in the practical setting, not the invention of the underlying update family.
3. Empirical evidence at large scale. Our experiments extend the study from GPT-2 models to dense and MoE Transformers, including a 235B-parameter MoE model. We believe that evaluating this single-state optimizer and its quantized version at that scale contributes empirical knowledge relevant to large-model training. At the same time, the 235B experiment covers 20K steps, or approximately 10.49B token positions, and does not establish long-horizon equivalence or superiority over other memory-efficient optimizers at that scale.
We appreciate the reviewers' feedback and will revise the paper to distinguish these findings from stronger claims that our current evidence does not support.