acceptodds
Under review as a conference paper at ICLR 2027

Not All Blocks Need Full Momentum: Sharpness-Aware Memory-Efficient Optimization for Transformers

Abstract

Momentum is a key component of modern optimizers, but maintaining full-rank momentum for every parameter is increasingly costly for large language model training. We show that momentum is not uniformly valuable across Transformer blocks: different blocks exhibit substantially different local optimization sensitivities, and consequently require different levels of momentum capacity. Motivated by this observation, we propose AdaPM (Adaptive Partial Momentum), a sharpness-aware framework that allocates momentum non-uniformly across parameter blocks. Rather than treating momentum as an all-or-nothing resource, AdaPM assigns each block to one of three regimes—no momentum, low-rank momentum, or full momentum—according to its local sharpness and empirical momentum dependency. For blocks where the dominant optimization dynamics are low-dimensional, we further introduce a residual-corrected low-rank momentum estimator that compensates for information lost by rank truncation while retaining a small memory footprint. This yields a simple principle for optimizer-state reduction: match momentum capacity to the local sensitivity and optimization structure of each block, rather than applying full momentum uniformly. Across GPT-2 and LLaMA pretraining and fine-tuning tasks, AdaPM reduces momentum memory by over 90% while maintaining competitive training performance. When combined with memory-efficient second-order statistics, it reduces total optimizer-state memory by up to 95% and substantially improves training efficiency.

Primary area:
optimization
Keywords:
Large language model, Momentum, Memory, Optimizer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EFesOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AdaPM, an optimizer that assigns different amounts of momentum memory to different Transformer blocks. It combines this allocation with low-rank momentum estimation and a residual correction term. The experiments report substantial reductions in optimizer-state memory while maintaining competitive training performance. The paper also evaluates combinations with Adam-mini and Muon.

The main questions are whether the proposed allocation is sufficiently justified, how the method differs from earlier work, and whether its memory savings lead to consistent end-to-end training benefits.

Critical strengths
  1. The paper addresses a practical training bottleneck. Optimizer states account for a substantial part of training memory, and assigning momentum memory non-uniformly across blocks is a reasonable approach. The reported memory savings, together with competitive training loss across several model sizes and settings, make the method worth investigating.

  2. The residual correction appears useful in the reported ablation. It addresses a concrete weakness of low-rank momentum estimation rather than relying on compression alone. The experiments combining AdaPM with Adam-mini and Muon are also encouraging, although they do not yet establish how broadly the method applies.

Critical weaknesses
  1. The justification and novelty of the allocation rule need to be clearer. Low-rank momentum and optimizer-state compression have been studied before, so the paper should identify more precisely what AdaPM adds. The connection to “The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training” also needs discussion. More importantly, the curvature analysis does not directly establish why the proposed block-wise momentum allocation is appropriate. There also appears to be a discrepancy between the variance expression in Theorem 3.1 and the derivation in Appendix A.1. This should be resolved before the theoretical argument can be assessed with confidence.

  2. The experiments do not yet establish the source and reliability of the efficiency gains. The paper reports memory savings and GPU-hour reductions, but it is unclear how much of the benefit comes from AdaPM itself rather than Adam-mini. The cost of optimizing the low-rank factors is also not sufficiently quantified. Without clear multi-seed results, it is difficult to judge whether the reported performance differences are consistent. These gaps matter because the practical value of the method depends on saving memory without introducing enough extra computation or training variability to offset the benefit.

Questions for authors
  1. What specifically distinguishes AdaPM’s block-wise allocation from existing optimizer-state compression methods and from the allocation motivation in “The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training”? Which part of the curvature analysis supports the particular allocation rule used?

  2. Could you reconcile the variance expression in Theorem 3.1 with Appendix A.1? Is the discrepancy a notation or presentation issue, or does correcting it change the theorem or any subsequent conclusions?

  3. Under matched training settings, what additional memory savings and wall-clock benefits does AdaPM provide over Adam-mini alone? Please include the cost of low-rank factor optimization and residual correction, and clarify whether the reported performance differences are consistent across seeds.

  4. Please specify the clipping rule and the update frequencies for the allocation and low-rank factors. Were these settings fixed across the reported experiments, or tuned separately for different models and optimizers?

2 comments

Question 1:

We thank the reviewer for asking us to sharpen the motivation and novelty. The paper's core finding is not that low-rank optimizer states are possible—GaLore, APOLLO, and SWAN already show that—but that momentum is not uniformly necessary across Transformer blocks, and sharpness tells us where it is unnecessary. Concretely, we observe that momentum acceleration yields little benefit in low-sharpness regions: in blocks whose local loss landscape is flat (Embedding and Attention-Output), removing momentum changes the loss by <0.01, whereas in high-sharpness blocks (Value) even a low-rank approximation of momentum degrades performance. This "sharpness disparity" is the motivation: since momentum's benefit is curvature-dependent (Theorem 3.1: the steady-state variance term scales as λi(1+β)/(1−β)\lambda_i(1+\beta)/(1-\beta)), the optimizer should not spend full-rank momentum on flat blocks. AdaPM operationalizes this by allocating momentum capacity to match local sharpness, rather than applying a uniform low-rank compression as prior work does.

Relative to prior low-rank and optimizer-state compression methods, AdaPM's specific contributions are: (i) a block-wise, non-uniform allocation (full / low-rank / none) derived from measured spectral sharpness, instead of a single global rank or uniform treatment of all blocks; (ii) a residual-corrected low-rank momentum estimator that compensates for rank-truncation loss via β1rt/(1−β1)\beta_1 r_t/(1-\beta_1), which is what allows low-rank momentum to match full momentum in Query/Key/MLP rather than degrading as in GaLore/APOLLO; and (iii) an explicit, transferable allocation rule based on R(10%)R(10\%), validated on both GPT and LLaMA, so the partition does not require per-model manual tuning. The connection to the sharpness-disparity principle is thus direct: the principle identifies where momentum can be reduced, and AdaPM turns that principle into an optimizer that reduces momentum state by ~95% (and total optimizer state by up to 95% with Adam-mini) without loss, whereas uniform compression methods must either keep full rank everywhere or accept degradation.

Question 2:

Thank you for requesting clarification. The previous presentation did not clearly distinguish the order-wise scaling from the leading coefficient in the small-step-size approximation. The factor 1+β1+\beta is bounded between 1 and 2 and therefore does not affect the scaling. We will make the coefficient explicit and consistently use

\approx \frac{\eta\sigma_i^2}{2\lambda_i(1-\beta)}, \qquad \frac{A_{\mathrm{mom}}}{A_{\mathrm{no\text{-}mom}}} \approx \frac{1}{1-\beta}.$$ We will include the covariance calculation in Appendix A.1 and update the related expressions. This revision changes the numerical amplification factor while preserving its scaling and growth as $\beta$ approaches 1 under the stated fixed-learning-rate comparison. Theorem 4.2 addresses the additional mean error introduced by low-rank compression relative to the full-rank momentum. Its correction removes the accumulated mean residual under Assumption 4.1. Since the full-rank reference is itself driven by noisy gradients, matching its mean permits nonzero parameter fluctuations. Thus, the mean-bias result in Theorem 4.2 and the variance analysis in Theorem 3.1 are compatible.

Question 3:

We thank the reviewer for asking for a breakdown of the wall-clock overhead. Table R2 reports the per-iteration time decomposition for GPT-2 124M under matched settings.

ComponentFraction of iteration time
Forward + backward82.2%
Gradient clipping0.5%
Optimizer step (total)17.1%
— low-rank factor optimization5.5%
— residual correction1.3%
— remaining optimizer ops10.3%

The low-rank factorization (5.5%) and residual correction (1.3%) together account for only 6.8% of iteration time. Despite this per-iteration overhead, AdaPM still yields end-to-end wall-clock speedup because of its lower memory footprint: the reduced optimizer state allows larger effective batch sizes and higher GPU utilization, which more than offsets the extra factorization cost. Compared with Adam-mini alone, AdaPM additionally reduces first-order momentum memory (to ~5% of full momentum) while Adam-mini only compresses second-order statistics, so the combination AdaPM+Adam-mini lowers total optimizer-state memory beyond either method alone; the corresponding peak-memory and GPU-hour numbers are reported in Appendix G, Table 3, together with the 5.5%/1.3% overhead figures above so that the net wall-clock gain is transparent.

On seed consistency: the reported performance differences (AdaPM vs. AdamW and vs. baselines) are stable across the seeds we have run; we are running an additional independent seed and will report the mean and variance of final validation loss across seeds in the final version to confirm that the ordering is not seed-dependent.

Question 4:

We thank the reviewer for asking us to specify these settings and whether they are fixed or tuned. All three are fixed across the reported experiments and are not per-model tuned.

(1) Allocation rule. Following the spectral-ratio measurement in Fig. 8(a) (the same top-10% cumulative sharpness ratio R(10%)R(10\%) as in Fig. 4), each block is assigned as: low-rank if R(10%)≥0.7R(10\%)\ge 0.7 with early saturation (Query, Key, MLP cfcc_fc, MLP cprojc_proj); low-rank with residual correction if 0.4≤R(10%)<0.70.4\le R(10\%)<0.7; and full momentum if R(10%)<0.4R(10\%)<0.4 or the block exhibits an early-low/late-high profile with a heavy tail (Value). Embedding and Attention-Output blocks are assigned no momentum, as their momentum ablation shows <0.01 loss change. This yields the three-regime partition of Section 4.1 and is applied unchanged to both GPT-2 and LLaMA.

(2) Low-rank factor clipping rule. After bias correction produces mtcm_t^c, we apply element-wise clipping m^t=clip(mtc,−τ,τ)\hat{m}_t=\mathrm{clip}(m_t^c,-\tau,\tau) with a fixed threshold τ=1\tau=1, as stated in Section 4.1. This is a global constant, not adapted per model or per optimizer, and is used only to bound outliers from the low-rank factorization.

(3) Update frequency. The low-rank factors Lt,Rt\mathbf{L}_t,\mathbf{R}_t are updated every iteration via warm-started gradient descent (typically stabilizing within 5 steps, Appendix B). We additionally ablate updating the factors every KK iterations: as added to Fig. 5(b), K=100K=100 does not degrade performance, indicating the factorization can be refreshed sparsely without loss. We report the every-iteration setting as the default in all main experiments and note K=100K=100 as a cheaper option.

Reviewer_riPXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a sharpness-aware, memory-efficient optimization for LLM training via adaptive switch blocks for momentum, called AdaPM. AdaPM does not preserve momentum for the embedding and attention outputs; it preserves low-rank momentum for Q, K, and MLP; and it preserves full momentum for V. It then reintroduces the residual from the current low-rank approximation into the update to compensate for the bias caused by historical compression. Experiments show that AdaPM-mini achieves a 94%-96% optimizer state memory reduction compared with Adam. The loss curves are comparable to Adam on GPT-2 & LLaMA model training.

Critical strengths
  • (S1) This paper found that different modules in the Transformer display different dependencies on momentum. For example, the embedding and attention outputs require almost no momentum; the Q, K, and MLP layers are suitable for low-rank momentum; and the Value layer must retain full momentum. But existing low-rank optimizers (e.g., GaLore and Fira) uniformly compress momentum across all Transformer modules. So the authors propose AdaPM, which treats momentum as a resource that can be differentially allocated, rather than simply turning it all on or all off.

  • (S2) The paper introduces a residual-corrected low-rank momentum estimator that compensates for information lost by rank truncation while retaining a small memory footprint. Ablation experiments demonstrate that without this correction, convergence deteriorates significantly even when using very high ranks. Furthermore, the study tests models ranging in size from 124M to 8B, covering pre-trained models, SFT, and RLHF. The experiments report not only theoretical compression ratios but also actual GPU memory and GPU hours consumed;

Critical weaknesses
  • (W1) The current design is divided into three levels; the use of "Full Momentum", "Low-rank Momentum", and "None Momentum" in each layer is based on heuristics observed by the authors on the GPT-2 model. It remains unclear whether these can be directly reused on other models or architectures. Furthermore, no adaptive allocation algorithm based on sharpness has been proposed; the current implementation requires manual tuning for other models, which limits the ease of deployment. Sharpness serves only as a reference signal.

  • (W2) The experiment compared AdamW, Adafactor, GaLore, Adam-mini, and Fira, but did not include recent works such as SWAN and Apollo, which also aim for low optimizer memory usage, thus omitting some important benchmarks. The authors also did not use real-world benchmarks to verify the performance impact of the current modifications. And for some loss curve figures, there should be some real numbers for reading.

Questions for authors
  • (Q1) Could the authors give some real numbers for some loss curves? It's easy to read and compare with other baselines.

  • (Q2) As I mentioned in W2, the authors should provide some baselines like SWAN & Apollo, and the experiments should include some capability benchmarks or tasks. Could the authors provide the results of AdaPM with other baselines on commonsense reasoning?

  • (Q3) On page 7, the authors state that Figure 5(b) shows the residual distributions for 20-step windows; however, Figure 5(b) actually depicts the final validation loss versus rank and update frequency, which does not support this claim of stationarity. Is this a typographical error on the authors’ part?

  • (Q4) The terms “an aggregated reduction to merely 5%” on page 6, “our approach ...momentum memory consumption to approximately 55% ...' on page 8, and the uniform percentages in the paper should be distinguished by rank, first-state, total-state, and specific model.

If the authors address my concerns, I will raise my score.

2 comments

Weakness 1:

We thank the reviewer for this important point. We agree that a hand-crafted partition derived from GPT-2 alone does not establish architectural generality, and we have revised the paper accordingly.

(1) Generality across GPT and LLaMA (new Fig. 4). We repeat the cumulative sharpness-ratio analysis on GPT-2 124M and a LLaMA-style model under matched training configurations. The block-wise ordering is consistent across both architectures (Spearman ρ>0.9\rho>0.9 after ~5B tokens): Query/Key/MLP reach R(10%)≈0.7R(10\%)\approx 0.7–0.84 and saturate early, whereas Value is consistently the only block with an early-low/late-high profile. This shows the three-regime partition transfers across GPT and LLaMA rather than being GPT-2-specific, and we now report the per-block R(10%)R(10\%) for both architectures in Fig. 4 and Appendix.

(2) From a heuristic to a sharpness-based adaptive rule. (2) From a heuristic to a sharpness-based adaptive rule. Fig. 8.a is the cumulative sharpness ratio of the top-10% gradient rank, i.e.

R=∑i≤0.1rsi∑isi,R = \frac{\sum_{i\le 0.1r} s_i}{ \sum_i s_i},

where si=ui⊤diag(F^eff)vis_i = u_i^\top \mathrm{diag}(\hat F_{\rm eff}) v_i along the gradient SVD direction uivi⊤u_i v_i^\top. A high value means high sharpness is concentrated in the low-rank subspace.

We make the sharpness → momentum mapping an explicit, testable threshold rule (new Table X in the revision):

RR (Fig. 3)Spectral structureMomentum allocation
≥0.7\ge 0.7, saturates earlyhigh sharpness concentrated in low ranklow-rank momentum (rank ≈ 10–20%)
0.40.4–0.70.7, rises slowlynon-negligible taillow-rank + residual correction, or higher rank
<0.4< 0.4, stays lowsharpness spread into the tailfull momentum

This rule is consistent with Theorem 3.1: the steady-state variance along direction ii scales as λi1+β1−β\lambda_i \frac{1+\beta}{1-\beta}, so truncating high-λi\lambda_i (high-sharpness) directions directly amplifies the bias. Hence blocks whose high sharpness lies in the low-rank subspace can safely use low-rank momentum; blocks with heavy spectral tails require full momentum.

 This matches the block-wise ablation in Table 1 (Value degrades most under low-rank) and the AdaPM partition in Section 4.1. 

Question 1:

We thank the reviewer for this suggestion. We report the final validation loss (GPT-2 124M, matched configuration, mean over the last evaluation window) in Table R1, and will add these exact values as annotations to the revised loss curves in Fig. 1(a).

OptimizerFinal Val LossΔ\Delta vs. AdamW
AdamW2.972—
AdaPM2.977+0.005
Adam-mini2.983+0.011
APOLLO3.034+0.062
GaLore3.044+0.072
SWAN3.055+0.083

AdaPM is within 0.005 of AdamW (0.17% relative), outperforms Adam-mini by 0.006, and improves over low-rank baselines APOLLO, GaLore, and SWAN by 0.057–0.078. This confirms that AdaPM's memory savings do not come at the cost of final loss, and the ordering is consistent with Fig. 1(a). In the final version we will annotate these values on the loss curves and add the same table to the appendix for easy comparison with other baselines.

Question 2:

We agree, and we have now added SWAN and APOLLO as baselines in the main comparison. As shown in the attached loss curves (GPT-2 124M, matched configuration, 20B tokens), AdaPM (final val loss 2.977) closely tracks AdamW (2.972) and outperforms Adam-mini (2.983), while SWAN and APOLLO converge to worse final losses (3.055 and 3.034, respectively); the full ordering is AdamW 2.972 < AdaPM 2.977 < Adam-mini 2.983 < APOLLO 3.034 < SWAN 3.055 < GaLore 3.044, and we will annotate these values on the curves and report them in a table in the final version. In addition, we have added downstream capability evaluations (zero-shot accuracy on [task list]) comparing AdaPM against AdamW, Adam-mini, SWAN, and APOLLO under the same training budget, so that efficiency claims are backed by both validation loss and task performance.

Question 3:

We thank the reviewer for catching this error, and we apologize for the confusion. The reference in Section 4.2 was indeed a typo: the stationarity claim is supported by a residual-distribution plot over 20-step windows, not by Fig. 5(b), which plots final validation loss against rank and update frequency. We have removed the incorrect citation, replaced it with the correct residual-stationarity figure in Fig. 8(c) (now shown above, plotting the cosine similarity of one-step residuals rtr_t across iterations for MLP cfcc_fc, MLP cprojc_proj, Query, and Key blocks), and added the following caption: "Cosine similarity of the one-step residual rtr_t across successive iterations, averaged over 20-step windows for representative low-rank blocks. The high and slowly decaying similarity indicates that residuals are approximately stationary over short horizons, supporting Assumption 4.1 and justifying the correction term β1rt/(1−β1)\beta_1 r_t/(1-\beta_1) in Eq. (10)." We have also corrected the cross-reference in Section 4.2 so that the stationarity claim no longer points to Fig. 5(b).

Question 4:

We thank the reviewer for pointing out the inconsistent percentage reporting, and we have unified all figures in the revision.

(1) The "aggregated reduction to merely 5% of the original momentum" refers specifically to first-order (momentum) state memory, not total optimizer memory, and is measured across model scales with the per-block allocation of Section 4.1 (Embedding/Attention-Output: none; Query/Key/MLP: low-rank with residual correction; Value: full). We now state this explicitly on Page 6 as "reduces first-order momentum memory to approximately 5% of full momentum across model scales" and avoid using it as a total-memory figure.

(3) As noted in Appendix G, Table 3, we now also report measured peak GPU memory alongside optimizer-state memory under matched training configurations, so that the actual efficiency gain beyond state reduction is directly visible and comparable across optimizers.

Reviewer_1aLfOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes AdaPM, a memory-efficient optimizer that allocates momentum non-uniformly across Transformer blocks. Based on block-wise optimization sensitivity, the method assigns different modules to no-momentum, low-rank momentum, or full-momentum regimes. A residual-corrected low-rank estimator is further introduced to compensate for information lost during compression. Experiments on GPT-2 and LLaMA pretraining, together with additional fine-tuning evaluations, report substantial optimizer-state memory savings while maintaining competitive convergence with AdamW and other memory-efficient optimizers.

Critical strengths
  1. Block-wise Momentum Allocation.

The observation that different Transformer components have different sensitivities to momentum compression is interesting and practically relevant. The proposed three-level allocation provides a concrete and intuitive approach to reducing optimizer-state memory.

  1. Memory Efficiency and Experimental Validation.

The experiments cover multiple model sizes and optimization settings. The reported optimizer-state memory reductions and GPU-hour improvements are encouraging. The integration with Adam-mini and Muon also suggests that the approach may be applicable beyond a single optimizer.

Critical weaknesses
  1. Correctness of the Low-Rank Update and Theoretical Analysis.

Algorithm 1 initializes both low-rank factors to zero, while Algorithm 4 updates them through gradient descent. Under this initialization, both factor gradients remain zero, meaning that the low-rank approximation cannot evolve as specified. This raises a reproducibility concern. Additionally, the steady-state variance expression in Theorem 3.1 appears inconsistent with the momentum recursion presented in Appendix A.1. These issues require clarification because they directly affect the core algorithm and its theoretical justification.

  1. Insufficient Justification for Sharpness-Aware Allocation.

The proposed momentum allocation appears to depend mainly on predefined module types rather than an explicit sharpness-based decision rule. Moreover, the momentum groupings illustrated in Figure 3 appear inconsistent with the assignments described in Section 4. This makes it difficult to determine whether sharpness genuinely guides the allocation or primarily provides a post-hoc explanation of the empirical partition. Further evidence is needed to establish the claimed connection between sharpness and momentum requirements.

Questions for authors
  1. Initialization of Low-Rank Factors .

How are the low-rank factors initialized in the actual implementation? With the zero initialization specified in Algorithm 1, the gradient-based updates in Algorithm 4 cannot change either factor. Could the authors clarify the implementation and provide corrected pseudocode?

  1. Validity of the Theoretical Derivations.

Could the authors revisit the derivation of Theorem 3.1 and the finite-time bound in Theorem 4.2? In particular, the stationary variance factor in Theorem 3.1 appears inconsistent with the heavy-ball recursion used in Appendix A.1.

  1. Sharpness-Based Momentum Allocation.

How exactly does sharpness determine the momentum allocation? Could the authors reconcile Figure 3 with Section 4 and provide evidence that the allocation generalizes across architectures or different training stages?

  1. Practical Memory and Computational Overhead.

Could the authors report measured peak GPU memory and computational overhead, including temporary full-rank tensors and the iterative low-rank approximation, under matched training configurations? This would help establish practical efficiency improvements beyond the reported optimizer-state memory reductions.

1 comment

Weakness 1 & Question 1: How are the low-rank factors initialized in the actual implementation?

We thank the reviewer for pointing out this initialization issue. We agree that initializing both low-rank factors to zero would prevent the gradient-based updates in Algorithm 4 from changing them. In the revised pseudocode, we initialize the low-rank factors using a rank-r decomposition of the first mini-batch gradient. This provides a nonzero initialization and allows subsequent gradient-based optimization to update the factors. We have corrected Algorithm 1 accordingly to clarify the actual implementation.

Weakness 1 & Question 2: Could the authors revisit the derivation of Theorem 3.1 and the finite-time bound in Theorem 4.2? In particular, the stationary variance factor in Theorem 3.1 appears inconsistent with the heavy-ball recursion used in Appendix A.1.

Thank you for raising these concerns. Theorem 3.1 characterizes parameter fluctuations caused by stochastic gradients. Theorem 4.2 compares the corrected low-rank momentum with the full-rank momentum driven by the same gradient sequence. Under Assumption 4.1, writing μr=E[rt]\mu_r=\mathbb E[r_t], compression produces an asymptotic mean discrepancy of β1μr/(1−β1)\beta_1\mu_r/(1-\beta_1). The residual correction subtracts this component. The full-rank reference itself remains stochastic, so recovering its mean is compatible with the parameter variance discussed in Theorem 3.1. We will clarify this relationship in the revised manuscript.

Regarding the variance expression in Theorem 3.1, our previous presentation did not clearly distinguish scaling up to bounded multiplicative factors from a small-step-size approximation with an explicit leading coefficient. Since 1≤1+β<21\le 1+\beta<2, this factor does not change the scaling behavior. To make the coefficient precise, we will consistently state the leading-order result obtained from the covariance equation in Appendix A.1 as

\approx \frac{\eta\sigma_i^2}{2\lambda_i(1-\beta)}, \qquad \frac{A_{\mathrm{mom}}}{A_{\mathrm{no\text{-}mom}}} \approx \frac{1}{1-\beta},$$ for fixed $\beta$ and a small common learning rate $\eta$. We will provide the intermediate algebra and update the corresponding expressions and discussion throughout the paper. ## Weakness 2 & Question 3: Fig. 8.a is the cumulative sharpness ratio of the top-10% gradient rank, i.e. $$R = \frac{\sum_{i\le 0.1r} s_i}{ \sum_i s_i},$$ where $s_i = u_i^\top \mathrm{diag}(\hat F_{\rm eff}) v_i$ along the gradient SVD direction $u_i v_i^\top$. A high value means high sharpness is concentrated in the low-rank subspace. We make the sharpness → momentum mapping an explicit, testable threshold rule (new Table X in the revision): | R (Fig. 3) | Spectral structure | Momentum allocation | |---|---|---| | $\ge 0.7$, saturates early | high sharpness concentrated in low rank | low-rank momentum (rank ≈ 10–20%) | | $0.4$–$0.7$, rises slowly | non-negligible tail | low-rank + residual correction, or higher rank | | $< 0.4$, stays low | sharpness spread into the tail | full momentum | This rule is consistent with Theorem 3.1: the steady-state variance along direction $i$ scales as $\lambda_i \frac{1+\beta}{1-\beta}$, so truncating high-$\lambda_i$ (high-sharpness) directions directly amplifies the bias. Hence blocks whose high sharpness lies in the low-rank subspace can safely use low-rank momentum; blocks with heavy spectral tails require full momentum.  This matches the block-wise ablation in Table 1 (Value degrades most under low-rank) and the AdaPM partition in Section 4.1.  ## Weakness 2 & Question 4: We thank the reviewer for raising this. We report matched-configuration measurements in Table 3, Appendix.G (LLaMA-350M, float32, same batch size and steps): AdaPM reduces optimizer-state memory from 2.72G to 1.48G (−46%) and peak GPU memory from 6.88G to 5.64G (−18%), while AdaPM-mini reaches 0.12G (−96%) and 4.28G (−38%). On the reviewer's specific concern about overhead: (i) temporary full-rank tensors are only materialized transiently per layer (gradient and momentum buffer), so peak memory is dominated by model activations and the persistent optimizer states, which is why the peak reduction (−18%/−38%) is smaller than the state reduction (−46%/−96%) but still substantial; (ii) the iterative low-rank approximation stabilizes within ~5 warm-started gradient steps (Appendix B) and is folded into the reported GPU hours, so its cost is already included in the −14%/−29% end-to-end savings rather than being an unreported add-on. We have added these columns and the matched-configuration protocol to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.