acceptodds
Withdrawn from ICLR 2027

AdamPM: Memory-Efficient LLM Optimization via Moment Pooling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adam is widely used for training large language models (LLMs) owing to its strong optimization performance and training stability, but its element-wise estimation for first and second moments incur substantial memory overhead. Although numerous memory-efficient optimizers have been proposed, many incur additional computational overhead or require extra tuning in hyperparameters. Consequently, Adam remains a dominant and reliable choice for LLM pre-training. We propose AdamPM (Adam with Pooled Moments), a simple and memory-efficient variant of Adam that groups adjacent moment entries along the row or column dimension and represents each group by a single shared mean. AdamPM can well inherit Adam's hyperparameter settings and is readily integrated into existing Adam optimizer implementations, with less memory overhead. We systematically investigate pooling of both first and second moment estimates and identify second moment pooling as the effective design. With a group size of , AdamPM reduces second moment memory to of the original requirement. Extensive experiments on language model pretraining, fine-tuning and multimodal learning demonstrate that AdamPM maintains model performance and training speed while substantially reducing optimizer-state memory.

Primary area:
optimization
Keywords:
Memory-Efficient Optimizer, Large Language Models Training, Adaptive Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RokkOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AdamPM, a memory-efficient Adam variant that compresses second-moment states by averaging adjacent elements while preserving full first-moment states. Experiments on LLM pretraining and fine-tuning demonstrate comparable performance to Adam with reduced optimizer memory and negligible computational overhead.

Critical strengths

1. Simplicity and practicality. AdamPM is easy to implement and can directly reuse Adam's hyperparameters without introducing significant computational overhead.

2. Extensive empirical validation. The paper evaluates different pooling strategies across language model pretraining, fine-tuning, and multimodal tasks, demonstrating a favorable memory-performance trade-off.

Critical weaknesses

1. Limited algorithmic novelty. AdamPM closely resembles Adam-mini, as both retain full first-moment states while sharing second-moment statistics across parameter groups. The main difference lies in the grouping strategy, whose novelty and advantages are insufficiently justified.

2. Insufficiently controlled comparisons. AdamPM generally uses more optimizer memory than Adam-mini, while Adam-mini is evaluated using Adam's hyperparameters without independent tuning. The experiments do not convincingly establish AdamPM's superiority under matched memory budgets and properly tuned baselines.

Questions for authors
  1. What is the fundamental advantage of pooling adjacent elements over the structured grouping used in Adam-mini? Could the authors compare against random grouping under identical memory budgets?

  2. Could the authors provide memory-matched comparisons with independently tuned Adam-mini and FOAM baselines?

  3. How sensitive is AdamPM to the second-moment decay rate β2? Does the method remain effective with β2 = 0.99 or 0.999 rather than 0.95?

Reviewer_EFeAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

AdamPM keeps Adam's element-wise first moments but shares second-moment estimates across small groups of adjacent parameters. The paper studies group sizes and pooling directions, gives a stationarity analysis, and reports pretraining and fine-tuning results with learning rates transferred from Adam.

Critical strengths
  • This is a small, understandable change to Adam, with a clear storage trade-off controlled by group size.
  • Adam is tuned before its learning rates are transferred, which makes the comparison useful. The first-moment and pooling-direction ablations also help explain the chosen design.
  • The paper measures actual training memory and runtime. The nearly identical 1.3B runtimes support the claim that pooling adds little overhead in that setup.
Critical weaknesses
  • FOAM [1] seems an essential comparison here. It also compresses adjacent blocks, yet it is absent from the experiments. The discussion of first-moment handling and residual correction does not establish whether AdamPM offers a better accuracy–memory trade-off.
  • The experimental coverage is limited. Pretraining stops at 1.3B with 256-token sequences and zero weight decay; the 7B result only measures static memory. Larger-scale runs, longer contexts, and nonzero weight decay are needed to assess whether the reported stability and hyperparameter transfer hold more broadly.
  • The actual memory reduction is more modest than the second-moment figures suggest: AdamPM-8 saves about 7% in Table A2. The measurement protocol is unclear. Pretraining also stops at 1.3B, uses sequence length 256 and zero weight decay, and provides no variation across seeds.
  • The convergence result has limited practical force. Eq. (2) retains (3+4C/ϵ)σ2(3+4C/\epsilon)\sigma^2, while Eq. (3) requires a very small relative-noise bound when ϵ=10−8\epsilon=10^{-8}. I do not see evidence that this condition holds in the experiments, or that the bound explains which group sizes should work.

[1] FOAM: Blocked State Folding for Memory-Efficient LLM Training

Questions for authors
  • Can you compare directly with FOAM using matched token budgets, state dtypes, and comparable learning-rate tuning? A control retaining full first moments would help separate the relevant design choices.
  • What exactly does “Memory Cost” in Table A2 measure? Are Algorithm 1's full-size squared-gradient and broadcast temporaries included, and what parameter, state, and master-weight dtypes are used?
  • What empirical estimates of CC and gradient noise support κ<ϵ/(16C+12ϵ)\kappa<\epsilon/(16C+12\epsilon) in Eq. (3)? If it fails, how informative is Eq. (2) under the reported settings?
Reviewer_CVS1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AdamPM retains Adam’s full first moment while pooling adjacent second-moment entries. It reduces optimizer-state storage and reports comparable performance across language-model pretraining, GLUE fine-tuning, and multimodal fine-tuning, while reusing Adam’s hyperparameters.

Critical strengths
  1. The method is simple, architecture-independent, and supported by learning-rate-transfer and pooling ablations.

  2. Measured 1.3B training results demonstrate actual memory savings with nearly unchanged training time, supporting its practical value.

Critical weaknesses
  1. The incremental contribution over existing moment-sharing methods is insufficiently established. FOAM is discussed extensively but not evaluated directly. Adam-mini mainly reuses Adam’s hyperparameters, which tests transferability but does not establish the best achievable accuracy–memory trade-off of that baseline.

  2. The performance-preservation claim lacks uncertainty estimates. Tables 1, 3, and 4 report no variability across seeds, making small improvements and apparent equivalence to Adam difficult to assess reliably.

Questions for authors
  1. How does AdamPM compare with FOAM, including a variant retaining the full first moment without residual correction, and with independently tuned Adam-mini under comparable tuning budgets?

  2. Can representative pretraining and fine-tuning results be repeated across multiple seeds to establish whether performance is consistently preserved?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.