acceptodds
Under review as a conference paper at ICLR 2027

Gefen: Optimized Stochastic Optimizer

Abstract

AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory, increasing the already substantial cost of large-scale pretraining. We propose Gefen, a memory-efficient optimizer that automatically shares second-moment estimates across parameter blocks and quantizes the first moment using a learned codebook, thereby reducing AdamW’s memory footprint by up to 8× while maintaining the same performance, corresponding to a reduction of 6.5 GiB per billion parameters. Prior work reduces optimizer memory by sharing second moments across parameters grouped along the Hessian’s block-diagonal structure, but relies on hand-specified architectural rules and leaves unexplained why such grouping works. We prove that large mixed Hessian entries constrain the ratio of squared gradients toward one. This supplies the missing explanation, since a shared second moment is accurate when the squared gradients it pools are similar. It also shows the Hessian need not be computed: a block-structured Hessian forces the squared gradients to inherit that structure, so the blocks can be found in the squared gradients themselves. Gefen therefore infers block structure from the initial squared gradients, requiring no architecture-specific metadata or user-tuned hyperparameters beyond AdamW defaults. Gefen learns an exact histogram-based dynamic-programming quantization codebook and reuses the same blocks for first-moment scaling. Across diverse pretraining experiments, Gefen achieves the lowest peak optimizer memory among the compared methods that maintain AdamW-level performance. In single-machine and distributed training, the reduced footprint enables larger microbatches and substantially improves throughput over AdamW, making Gefen a drop-in replacement that can train larger models or use larger global batch sizes. We provide the complete Python implementation, including fused CUDA kernels.

Primary area:
optimization
Keywords:
optimizer, memory-efficient optimization, adaptive gradient methods, optimizer-state quantization, deep learning optimizers, large-scale model training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gdv1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Gefen, a memory-efficient optimizer based on AdamW. It splits parameter tensors into blocks and shares a single second-moment estimate within each block, and compresses first-moment states with a learned codebook. Gefen decides the block separation from initial gradients. The authors relate Hessian coupling to squared-gradient similarity as a motivation for these blocks. These blocks are also reused for first-moment quantization. Experiments across GPT, CNN, and diffusion models show memory reduction while maintaining performance comparable to AdamW.

Critical strengths
  1. The paper reduces the memory overhead introduced by optimizer states, which is an important challenge in large-scale experiments.
  2. Gefen automatically identifies parameter blocks from gradient information. This makes it easy to apply across different model architectures.
  3. Evaluations are across different architectures and tasks, including language model pretraining, image classification, and diffusion models. The results consistently show less memory and performance comparable to AdamW without requiring learning-rate retuning. Moreover, ablation studies partition stability, partition quality, and quantization strategies.
Critical weaknesses
  1. The paper connects Hessian to squared-gradient similarity but doesn't link to the proposed method directly. Gefen selects the block size by maximizing the difference between adjacent blocks' mean log-squared gradients, which doesn't mean that squared gradients within a block are sufficiently similar.
  2. Gefen uses initial gradients to split the blocks, but it doesn't mean square gradients within a block keep similar during the whole training process. It's necessary to report how accurately the shared second moments approximate AdamW's original per-parameter second-moment estimates throughout training.
  3. The blocks of Gefen are contiguous and equal size. Why not clustering them into different groups allowing non-adjacent and different group sizes?
  4. Gefen reuses blocks based on 2nd moment for 1st moment quantization, lacking of motivation and explanation. It is unclear whether these blocks are also appropriate for minimizing first-moment quantization error.
  5. The figures are a bit difficult to read (e.g., Figure 3). Although a magnified view of a portion is provided, it is not the final performance.
Questions for authors
  1. How accurate of Gefen approximation of 2nd moment compared with original AdamW?
  2. Is it necessary to design contiguous and equal-size blocks?
  3. Why reuse blocks based on the 2nd moment directly for 1st moment quantization?
  4. Why is Muon in Table 2 much worse than AdamW? Do the authors tune the hyperparameters for different optimizers independently?
  5. Gefen-Muon in the appendix looks interesting. How do the authors decide the blocks for it, considering it doesn't have 2nd moment? And in Table 4, why is Gefen's peak memory larger than Muon's when training CNN?
2 comments

Thank you for the detailed rebuttal and additional experiments.

However, my main concerns remain. The theoretical results do not directly justify the proposed partitioning method. Additional Figure 16 in Appendix N shows substantial second-moment approximation errors in some layers, yet Gefen still achieves performance comparable to AdamW. This raises the question of whether accurate second-moment approximation is necessary for Gefen's success, potentially undermining its central motivation.

More broadly, the paper's motivation, methodological justification, and interpretation of results remain insufficiently clear. I think clearer explanations and insights, rather than additional experiments, would be more valuable.

Thanks gdv1!

Following your feedback, we revised the wording in the PDF to describe the theoretical results as “motivation” for moment sharing. We also added an explicit statement in Appendix N that comparable training quality can coexist with substantial coordinatewise second-moment errors in this setting.

We think our previous response did not clearly separate the two contributions.

The primary goal of the two theorems is to help explain why the Hessian-based grouping used in previous work, particularly Adam-mini, has worked well empirically. Under their assumptions, the theorems connect strong Hessian coupling to similarity in gradient magnitudes, providing a reason why sharing second moments among such parameters can be a good idea. This addresses an explanatory gap in previous work. The theorems do not establish that Gefen's partitioning algorithm recovers Hessian blocks accurately.

Our partitioning algorithm is a separate contribution. It addresses the practical question of how to find useful sharing blocks without computing the Hessian or requiring architectural information. It operates directly on the observed gradients, without measuring Hessian strength.

The intuition is to find large blocks that save memory while retaining boundaries between regions with different characteristic squared-gradient scales. Algorithm 2 searches for pronounced contrast between neighboring blocks' mean log-squared gradients. Boundaries inside a relatively homogeneous region tend to produce smaller contrast, while merging different regions can average away their differences. This motivates the criterion intuitively, although it does not guarantee an optimal partition or uniformly small within-block variation.

Your observation about Figure 16 helps clarify another distinction. Similar squared gradients over time provide a sufficient condition for similar second moments. However, accurate coordinatewise approximation is not established as a necessary condition for comparable training quality. The figure shows that substantial discrepancies in some layers can coexist with comparable performance in this experiment.

More precisely, let vi,tAv^A_{i,t} denote the shadow AdamW second moment computed using the same gradients that Gefen sees. With zero initialization:

\tag{1}$$ Because averaging commutes with the EMA update, Gefen retains the block's average second-moment scale exactly. Figure 16 measures the coordinatewise differences **discarded by averaging**. Gefen retains adaptive scaling between blocks and coordinate-specific first moments, while discarding variation in second moments within each block. > "Figure 16 shows second-moment approximation errors" Figure 16 shows that the squared gradients inside a partition are **not exactly identical**, and this is okay. They do not need to be identical. What Gefen empirically shows is that holding an accurate **mean** (accurate by definition) of an accurate partition (as shown in ablations) - works well. It does not need the **variance**, inside a partition, to be zero. Essentially what figure 16 shows is the variability inside a block, and it is perfectly fine that the variability is not zero. There is no contradiction between what figure 16 shows and the success of Gefen. It shows that even when there is a significant variablity (AdamW-wise), the individual second moments AdamW holds are somewhat redundant, and that they can be replaced by the average. Inside a partition, that level of variability as figure 16 shows, it fine. If we use inaccurate partitions (see ablations) the variability becomes detrimental. We hope our explanation is better now. We agree that the theory does not fully explain which discrepancies training can tolerate. The theorems are a motivation starting point, showing that if we know that hessian has a block-diagonal structure, the squared-gradients should have somewhat similar structure, and since they have similar structure they can be partitioned, and that's why our algorithm tries to find those partitions, and it turns out to work well. We also believe the breadth of the empirical evidence matters when assessing the contribution. We evaluated Gefen across different tasks, architectures, benchmarks, and model sizes, including CNNs, language models, diffusion models, and mixture-of-experts models, with additional longer training runs. We consistently observe comparable quality to AdamW alongside substantial memory savings, without learning-rate retuning. Our code is public, allowing others to reproduce the results. We believe that explaining an existing design principle, providing automatic partitioning, and releasing an extensively evaluated implementation offer a useful contribution to the research community. We hope these clarifications and PDF revisions address your concerns and help you reconsider your assessment!
Reviewer_yFtqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes Gefen, an AdamW replacement for memory-constrained pretraining. Gefen automatically determines a parameter partition from the first-step gradient, shares the second moment within each block, and quantizes the momentum to 8 bits using a codebook learned by histogram-based dynamic programming. The authors further prove that large mixed Hessian entries constrain the ratio of squared gradients toward one, and show that the block structure can therefore be recovered directly from squared gradients without computing the Hessian. According to the experiments, Gefen achieves training performance close to that of AdamW while reducing optimizer-state memory by up to roughly 8×8\times.

Critical strengths
  1. The paper is clearly written and easy to follow. The problem is important: optimizer states account for a large share of memory in large-model training. The paper considers both persistent and peak memory, as well as throughput gains in memory-constrained settings, which gives the method clear practical value.

  2. Gefen is effective and easy to apply. The authors also provide theoretical results that motivate sharing the second moment within Hessian-aligned blocks. The paper gives a thorough idea on compressing AdamW's states: the second moment is shared within blocks, and the momentum is quantized to 8 bits.

  3. The experiments cover language models, image classification and diffusion models. Across these tasks, Gefen performs close to AdamW while reducing optimizer-state memory by up to about 8×8\times relative to fp32 states.

Critical weaknesses
  1. The first-step partition is not robust. With zero-initialized methods such as LoRA, some tensors receive exactly zero gradient at step 0, which gives S(p)≡0S(p)\equiv0 and p=1p=1.

  2. The components are not ablated separately. There is no run with second-moment sharing but fp32 momentum, or with 8-bit momentum but a per-coordinate second moment. The loss and memory contribution of each part is therefore unclear.

  3. As acknowledged in the Limitations section, the partition is inferred only from the first-step gradient and then kept fixed, so distribution shift or a change in the training regime could make it invalid. The paper lack experiments discussing Gefen's performance in such settings, as it's important in practical scenarios.

Questions for authors

The GPT-2-125M run with about 20B tokens is strong evidence, but the larger Llama 8B and Qwen3-4B runs are shown for only about 0.8B and 0.5B tokens. Could the authors train these larger models on a larger token budget and report whether Gefen stays close to AdamW as training continues?

Reviewer_ki2ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The reported optimizer-state memory savings and fused implementation have practical value. Agreement with AdamW across several evaluated architectures, while retaining its learning rate, provides useful preliminary evidence of drop-in compatibility.

Critical strengths

The optimizer-state memory savings and fused implementation offer practical value. The reported agreement with AdamW across several architectures without learning-rate retuning is encouraging, and the ability to accommodate larger microbatches demonstrates a useful benefit in memory-constrained settings.

Critical weaknesses

The contribution of automatic partitioning is not adequately established. Appendix C.3 compares against shifted, randomized, and doubled versions of Gefen’s partitions, but not Adam-mini’s rules or simple row-wise partitions with the same first-moment quantization procedure. Showing that deliberately perturbed partitions degrade performance does not establish the value of the proposed detector relative to reasonable alternatives. The central memory–quality comparison is missing.

The theoretical justification is overstated. Theorems 3.1–3.2 establish conditional relationships between Hessian coupling and gradient magnitudes, not recovery of Hessian blocks from a single gradient observation. Within-block similarity does not imply separation between blocks or justify Algorithm 2’s adjacent-block-contrast criterion. The claim that Hessian structure can therefore be recovered from initial squared gradients exceeds what the analysis establishes. This is a gap in the justification of the proposed mechanism, not merely the absence of a general convergence theorem.

The larger-model experiments are insufficient to substantiate performance preservation. Figure 3 shows approximately 0.8B training tokens for Llama-8B and 0.5B for Qwen3-4B. These short runs provide limited evidence about final quality or long-horizon stability when both the partition and codebook are fixed at initialization. The longer GPT-2-125M experiment and small-model stability analyses do not close this gap at larger scales. Qwen3 is also evaluated at only a 2,048-token sequence length, further limiting the tested regime.

The baseline evaluation does not adequately establish competitiveness. Figure 2 omits Adam4bit and Adam8bit even in the DDP setting, although Table 3 reports both as DDP-compatible. Consequently, the throughput results do not establish an advantage over relevant low-bit optimizers. Furthermore, evaluating alternatives primarily with AdamW settings supports a compatibility comparison, not a conclusion that their achievable memory–quality trade-offs are inferior. The all-parameter Muon configuration obtained through tensor reshaping further limits the representativeness of that baseline.

Questions for authors

Clarification: Does Figure 2 use ordinary FSDP or FSDP2 sharded initialization? Why were Adam4bit and Adam8bit also omitted from the DDP throughput comparison?

Suggested additional experiments: Compare automatic, Adam-mini, and row-wise partitioning using the same first-moment quantization procedure over substantially longer large-model training. Include low-bit optimizers in compatible end-to-end throughput benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.