- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AdaPM, an optimizer that assigns different amounts of momentum memory to different Transformer blocks. It combines this allocation with low-rank momentum estimation and a residual correction term. The experiments report substantial reductions in optimizer-state memory while maintaining competitive training performance. The paper also evaluates combinations with Adam-mini and Muon.
The main questions are whether the proposed allocation is sufficiently justified, how the method differs from earlier work, and whether its memory savings lead to consistent end-to-end training benefits.
Critical strengths
-
The paper addresses a practical training bottleneck. Optimizer states account for a substantial part of training memory, and assigning momentum memory non-uniformly across blocks is a reasonable approach. The reported memory savings, together with competitive training loss across several model sizes and settings, make the method worth investigating.
-
The residual correction appears useful in the reported ablation. It addresses a concrete weakness of low-rank momentum estimation rather than relying on compression alone. The experiments combining AdaPM with Adam-mini and Muon are also encouraging, although they do not yet establish how broadly the method applies.
Critical weaknesses
-
The justification and novelty of the allocation rule need to be clearer. Low-rank momentum and optimizer-state compression have been studied before, so the paper should identify more precisely what AdaPM adds. The connection to “The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training” also needs discussion. More importantly, the curvature analysis does not directly establish why the proposed block-wise momentum allocation is appropriate. There also appears to be a discrepancy between the variance expression in Theorem 3.1 and the derivation in Appendix A.1. This should be resolved before the theoretical argument can be assessed with confidence.
-
The experiments do not yet establish the source and reliability of the efficiency gains. The paper reports memory savings and GPU-hour reductions, but it is unclear how much of the benefit comes from AdaPM itself rather than Adam-mini. The cost of optimizing the low-rank factors is also not sufficiently quantified. Without clear multi-seed results, it is difficult to judge whether the reported performance differences are consistent. These gaps matter because the practical value of the method depends on saving memory without introducing enough extra computation or training variability to offset the benefit.
Questions for authors
-
What specifically distinguishes AdaPM’s block-wise allocation from existing optimizer-state compression methods and from the allocation motivation in “The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training”? Which part of the curvature analysis supports the particular allocation rule used?
-
Could you reconcile the variance expression in Theorem 3.1 with Appendix A.1? Is the discrepancy a notation or presentation issue, or does correcting it change the theorem or any subsequent conclusions?
-
Under matched training settings, what additional memory savings and wall-clock benefits does AdaPM provide over Adam-mini alone? Please include the cost of low-rank factor optimization and residual correction, and clarify whether the reported performance differences are consistent across seeds.
-
Please specify the clipping rule and the update frequencies for the allocation and low-rank factors. Were these settings fixed across the reported experiments, or tuned separately for different models and optimizers?
2 comments
Question 3:
We thank the reviewer for asking for a breakdown of the wall-clock overhead. Table R2 reports the per-iteration time decomposition for GPT-2 124M under matched settings.
| Component | Fraction of iteration time |
|---|---|
| Forward + backward | 82.2% |
| Gradient clipping | 0.5% |
| Optimizer step (total) | 17.1% |
| — low-rank factor optimization | 5.5% |
| — residual correction | 1.3% |
| — remaining optimizer ops | 10.3% |
The low-rank factorization (5.5%) and residual correction (1.3%) together account for only 6.8% of iteration time. Despite this per-iteration overhead, AdaPM still yields end-to-end wall-clock speedup because of its lower memory footprint: the reduced optimizer state allows larger effective batch sizes and higher GPU utilization, which more than offsets the extra factorization cost. Compared with Adam-mini alone, AdaPM additionally reduces first-order momentum memory (to ~5% of full momentum) while Adam-mini only compresses second-order statistics, so the combination AdaPM+Adam-mini lowers total optimizer-state memory beyond either method alone; the corresponding peak-memory and GPU-hour numbers are reported in Appendix G, Table 3, together with the 5.5%/1.3% overhead figures above so that the net wall-clock gain is transparent.
On seed consistency: the reported performance differences (AdaPM vs. AdamW and vs. baselines) are stable across the seeds we have run; we are running an additional independent seed and will report the mean and variance of final validation loss across seeds in the final version to confirm that the ordering is not seed-dependent.
Question 4:
We thank the reviewer for asking us to specify these settings and whether they are fixed or tuned. All three are fixed across the reported experiments and are not per-model tuned.
(1) Allocation rule. Following the spectral-ratio measurement in Fig. 8(a) (the same top-10% cumulative sharpness ratio as in Fig. 4), each block is assigned as: low-rank if with early saturation (Query, Key, MLP , MLP ); low-rank with residual correction if ; and full momentum if or the block exhibits an early-low/late-high profile with a heavy tail (Value). Embedding and Attention-Output blocks are assigned no momentum, as their momentum ablation shows <0.01 loss change. This yields the three-regime partition of Section 4.1 and is applied unchanged to both GPT-2 and LLaMA.
(2) Low-rank factor clipping rule. After bias correction produces , we apply element-wise clipping with a fixed threshold , as stated in Section 4.1. This is a global constant, not adapted per model or per optimizer, and is used only to bound outliers from the low-rank factorization.
(3) Update frequency. The low-rank factors are updated every iteration via warm-started gradient descent (typically stabilizing within 5 steps, Appendix B). We additionally ablate updating the factors every iterations: as added to Fig. 5(b), does not degrade performance, indicating the factorization can be refreshed sparsely without loss. We report the every-iteration setting as the default in all main experiments and note as a cheaper option.
Question 1:
We thank the reviewer for asking us to sharpen the motivation and novelty. The paper's core finding is not that low-rank optimizer states are possible—GaLore, APOLLO, and SWAN already show that—but that momentum is not uniformly necessary across Transformer blocks, and sharpness tells us where it is unnecessary. Concretely, we observe that momentum acceleration yields little benefit in low-sharpness regions: in blocks whose local loss landscape is flat (Embedding and Attention-Output), removing momentum changes the loss by <0.01, whereas in high-sharpness blocks (Value) even a low-rank approximation of momentum degrades performance. This "sharpness disparity" is the motivation: since momentum's benefit is curvature-dependent (Theorem 3.1: the steady-state variance term scales as λi(1+β)/(1−β)), the optimizer should not spend full-rank momentum on flat blocks. AdaPM operationalizes this by allocating momentum capacity to match local sharpness, rather than applying a uniform low-rank compression as prior work does.
Relative to prior low-rank and optimizer-state compression methods, AdaPM's specific contributions are: (i) a block-wise, non-uniform allocation (full / low-rank / none) derived from measured spectral sharpness, instead of a single global rank or uniform treatment of all blocks; (ii) a residual-corrected low-rank momentum estimator that compensates for rank-truncation loss via β1rt/(1−β1), which is what allows low-rank momentum to match full momentum in Query/Key/MLP rather than degrading as in GaLore/APOLLO; and (iii) an explicit, transferable allocation rule based on R(10%), validated on both GPT and LLaMA, so the partition does not require per-model manual tuning. The connection to the sharpness-disparity principle is thus direct: the principle identifies where momentum can be reduced, and AdaPM turns that principle into an optimizer that reduces momentum state by ~95% (and total optimizer state by up to 95% with Adam-mini) without loss, whereas uniform compression methods must either keep full rank everywhere or accept degradation.
Question 2:
Thank you for requesting clarification. The previous presentation did not clearly distinguish the order-wise scaling from the leading coefficient in the small-step-size approximation. The factor 1+β is bounded between 1 and 2 and therefore does not affect the scaling. We will make the coefficient explicit and consistently use
\approx \frac{\eta\sigma_i^2}{2\lambda_i(1-\beta)}, \qquad \frac{A_{\mathrm{mom}}}{A_{\mathrm{no\text{-}mom}}} \approx \frac{1}{1-\beta}.$$ We will include the covariance calculation in Appendix A.1 and update the related expressions. This revision changes the numerical amplification factor while preserving its scaling and growth as $\beta$ approaches 1 under the stated fixed-learning-rate comparison. Theorem 4.2 addresses the additional mean error introduced by low-rank compression relative to the full-rank momentum. Its correction removes the accumulated mean residual under Assumption 4.1. Since the full-rank reference is itself driven by noisy gradients, matching its mean permits nonzero parameter fluctuations. Thus, the mean-bias result in Theorem 4.2 and the variance analysis in Theorem 3.1 are compatible.