- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Gefen, a memory-efficient optimizer based on AdamW. It splits parameter tensors into blocks and shares a single second-moment estimate within each block, and compresses first-moment states with a learned codebook. Gefen decides the block separation from initial gradients. The authors relate Hessian coupling to squared-gradient similarity as a motivation for these blocks. These blocks are also reused for first-moment quantization. Experiments across GPT, CNN, and diffusion models show memory reduction while maintaining performance comparable to AdamW.
Critical strengths
- The paper reduces the memory overhead introduced by optimizer states, which is an important challenge in large-scale experiments.
- Gefen automatically identifies parameter blocks from gradient information. This makes it easy to apply across different model architectures.
- Evaluations are across different architectures and tasks, including language model pretraining, image classification, and diffusion models. The results consistently show less memory and performance comparable to AdamW without requiring learning-rate retuning. Moreover, ablation studies partition stability, partition quality, and quantization strategies.
Critical weaknesses
- The paper connects Hessian to squared-gradient similarity but doesn't link to the proposed method directly. Gefen selects the block size by maximizing the difference between adjacent blocks' mean log-squared gradients, which doesn't mean that squared gradients within a block are sufficiently similar.
- Gefen uses initial gradients to split the blocks, but it doesn't mean square gradients within a block keep similar during the whole training process. It's necessary to report how accurately the shared second moments approximate AdamW's original per-parameter second-moment estimates throughout training.
- The blocks of Gefen are contiguous and equal size. Why not clustering them into different groups allowing non-adjacent and different group sizes?
- Gefen reuses blocks based on 2nd moment for 1st moment quantization, lacking of motivation and explanation. It is unclear whether these blocks are also appropriate for minimizing first-moment quantization error.
- The figures are a bit difficult to read (e.g., Figure 3). Although a magnified view of a portion is provided, it is not the final performance.
Questions for authors
- How accurate of Gefen approximation of 2nd moment compared with original AdamW?
- Is it necessary to design contiguous and equal-size blocks?
- Why reuse blocks based on the 2nd moment directly for 1st moment quantization?
- Why is Muon in Table 2 much worse than AdamW? Do the authors tune the hyperparameters for different optimizers independently?
- Gefen-Muon in the appendix looks interesting. How do the authors decide the blocks for it, considering it doesn't have 2nd moment? And in Table 4, why is Gefen's peak memory larger than Muon's when training CNN?
2 comments
Thanks gdv1!
Following your feedback, we revised the wording in the PDF to describe the theoretical results as “motivation” for moment sharing. We also added an explicit statement in Appendix N that comparable training quality can coexist with substantial coordinatewise second-moment errors in this setting.
We think our previous response did not clearly separate the two contributions.
The primary goal of the two theorems is to help explain why the Hessian-based grouping used in previous work, particularly Adam-mini, has worked well empirically. Under their assumptions, the theorems connect strong Hessian coupling to similarity in gradient magnitudes, providing a reason why sharing second moments among such parameters can be a good idea. This addresses an explanatory gap in previous work. The theorems do not establish that Gefen's partitioning algorithm recovers Hessian blocks accurately.
Our partitioning algorithm is a separate contribution. It addresses the practical question of how to find useful sharing blocks without computing the Hessian or requiring architectural information. It operates directly on the observed gradients, without measuring Hessian strength.
The intuition is to find large blocks that save memory while retaining boundaries between regions with different characteristic squared-gradient scales. Algorithm 2 searches for pronounced contrast between neighboring blocks' mean log-squared gradients. Boundaries inside a relatively homogeneous region tend to produce smaller contrast, while merging different regions can average away their differences. This motivates the criterion intuitively, although it does not guarantee an optimal partition or uniformly small within-block variation.
Your observation about Figure 16 helps clarify another distinction. Similar squared gradients over time provide a sufficient condition for similar second moments. However, accurate coordinatewise approximation is not established as a necessary condition for comparable training quality. The figure shows that substantial discrepancies in some layers can coexist with comparable performance in this experiment.
More precisely, let denote the shadow AdamW second moment computed using the same gradients that Gefen sees. With zero initialization:
\tag{1}$$ Because averaging commutes with the EMA update, Gefen retains the block's average second-moment scale exactly. Figure 16 measures the coordinatewise differences **discarded by averaging**. Gefen retains adaptive scaling between blocks and coordinate-specific first moments, while discarding variation in second moments within each block. > "Figure 16 shows second-moment approximation errors" Figure 16 shows that the squared gradients inside a partition are **not exactly identical**, and this is okay. They do not need to be identical. What Gefen empirically shows is that holding an accurate **mean** (accurate by definition) of an accurate partition (as shown in ablations) - works well. It does not need the **variance**, inside a partition, to be zero. Essentially what figure 16 shows is the variability inside a block, and it is perfectly fine that the variability is not zero. There is no contradiction between what figure 16 shows and the success of Gefen. It shows that even when there is a significant variablity (AdamW-wise), the individual second moments AdamW holds are somewhat redundant, and that they can be replaced by the average. Inside a partition, that level of variability as figure 16 shows, it fine. If we use inaccurate partitions (see ablations) the variability becomes detrimental. We hope our explanation is better now. We agree that the theory does not fully explain which discrepancies training can tolerate. The theorems are a motivation starting point, showing that if we know that hessian has a block-diagonal structure, the squared-gradients should have somewhat similar structure, and since they have similar structure they can be partitioned, and that's why our algorithm tries to find those partitions, and it turns out to work well. We also believe the breadth of the empirical evidence matters when assessing the contribution. We evaluated Gefen across different tasks, architectures, benchmarks, and model sizes, including CNNs, language models, diffusion models, and mixture-of-experts models, with additional longer training runs. We consistently observe comparable quality to AdamW alongside substantial memory savings, without learning-rate retuning. Our code is public, allowing others to reproduce the results. We believe that explaining an existing design principle, providing automatic partitioning, and releasing an extensively evaluated implementation offer a useful contribution to the research community. We hope these clarifications and PDF revisions address your concerns and help you reconsider your assessment!
Thank you for the detailed rebuttal and additional experiments.
However, my main concerns remain. The theoretical results do not directly justify the proposed partitioning method. Additional Figure 16 in Appendix N shows substantial second-moment approximation errors in some layers, yet Gefen still achieves performance comparable to AdamW. This raises the question of whether accurate second-moment approximation is necessary for Gefen's success, potentially undermining its central motivation.
More broadly, the paper's motivation, methodological justification, and interpretation of results remain insufficiently clear. I think clearer explanations and insights, rather than additional experiments, would be more valuable.