- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes AdamPM, a memory-efficient Adam variant that compresses second-moment states by averaging adjacent elements while preserving full first-moment states. Experiments on LLM pretraining and fine-tuning demonstrate comparable performance to Adam with reduced optimizer memory and negligible computational overhead.
Critical strengths
1. Simplicity and practicality. AdamPM is easy to implement and can directly reuse Adam's hyperparameters without introducing significant computational overhead.
2. Extensive empirical validation. The paper evaluates different pooling strategies across language model pretraining, fine-tuning, and multimodal tasks, demonstrating a favorable memory-performance trade-off.
Critical weaknesses
1. Limited algorithmic novelty. AdamPM closely resembles Adam-mini, as both retain full first-moment states while sharing second-moment statistics across parameter groups. The main difference lies in the grouping strategy, whose novelty and advantages are insufficiently justified.
2. Insufficiently controlled comparisons. AdamPM generally uses more optimizer memory than Adam-mini, while Adam-mini is evaluated using Adam's hyperparameters without independent tuning. The experiments do not convincingly establish AdamPM's superiority under matched memory budgets and properly tuned baselines.
Questions for authors
-
What is the fundamental advantage of pooling adjacent elements over the structured grouping used in Adam-mini? Could the authors compare against random grouping under identical memory budgets?
-
Could the authors provide memory-matched comparisons with independently tuned Adam-mini and FOAM baselines?
-
How sensitive is AdamPM to the second-moment decay rate β2? Does the method remain effective with β2 = 0.99 or 0.999 rather than 0.95?