- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors present ReAdam, an exact reformulation of Adam that stores, instead of the first and second moments, the effective update U and a scale S, with recurrences derived for both. Same memory footprint as Adam. Because U does not depend on the gradient scale and mostly lies in [−1,1], it is quantized to 4 bits with a fixed codebook and no per-block scales, while S is compressed with an Adafactor rank-1 factorization of V. 4-bit ReAdam matches the quality of 4-bit LPMM with Adafactor, with 5.3% less optimizer-state memory on average.
Critical strengths
The paper proposes an interesting, exact reformulation of Adam that stores the effective update U instead of the first moment. U is scale invariant and mostly lies in [−1,1], so it can be quantized with a single fixed codebook, removing the need to store a scale factor per block as in prior 4-bit optimizers. The derivation is clean and easy to follow (Appendix A).
Critical weaknesses
W1 Scope of evaluation. The paper presents ReAdam as a general memory-efficient replacement for Adam (e.g., Table 3 projects savings for training 1B to 684B models), yet 4 of the 5 experiments are fine-tuning. In Table 1, 4-bit ReAdam outperforms full precision AdamW on all three LLM fine-tuning tasks but underperforms it on the only from-scratch task (LibriSpeech, 38M; 16.6 vs 16.2 WER). There is no LLM pretraining, which is where optimizer memory matters most, and no loss curves are shown. Please either add LLM pretraining at a reasonable model size, with training and validation loss curves against AdamW and LPMM, or state explicitly in the Limitations that the method is validated mainly for fine-tuning. I suggest also comparing it against the new Gefen optimizer, or to try to combine ReAdam with Gefen (successor to Adam-Mini that you mentioned).
W2. Table 3. The table adds little: the relative saving over LPMM (AF) is a constant ~5.9% across all models, which follows from removing LPMM's per-block scales (0.25 of 4.25 bits per parameter). The claim that savings grow with model size (line 424) holds only in absolute GB, by construction. Since these are analytical estimates and no such models were trained, I suggest replacing the table with this one-line calculation and moving it to the appendix.
W3. Efficiency. The paper reports only optimizer-state memory. Please add per-step time or throughput against AdamW and LPMM.
Questions for authors
Abstract, lines 013 to 014. The sentence "storing these moments substantially increases Adam's memory footprint" is slightly imprecise. The first and second moments are not an addition on top of Adam; they are Adam's optimizer state, so they constitute its memory footprint rather than increase it. I suggest rephrasing.
Lines 130, 138, 179. U_t is called "theoretically bounded", but no bound is derived or stated, and Kingma & Ba (2015) only give an informal approximation under a different condition. Please provide the bound and its value.
Related Work - Another method that deals with quantization and sharing moments of Adam is the Gefen optimizer, and may be relevant to mention.
Algorithm 1. The algorithm is given only for matrix parameters, since the factored scale needs row and column statistics. Please state how 1D parameters (biases, norm weights) and higher order tensors (e.g., conv kernels) are handled, and how this affects the reported memory.
Abstract, line 027. Please say what the 6% is relative to. As written it reads as relative to Adam, but it is relative to 4-bit LPMM
Table 1, subscript should be % and not x
5 comments
Thanks for the additional experiments and clarifications.
The throughput measurements address W3, and the proposed revision to Table 3 addresses W2. The GPT-2 Large pretraining experiment also addresses the model-scale aspect of W1. However, the pretraining quality comparison remains unclear. In figure 2, why don't you put for each optimizer, the memory footprint and total clock time? You attribute the gap to clipping updates to the interval [−1,1]. Is this supported by an ablation? The response strengthens the paper, but I currently retain my rating. The idea of the paper is very promising - reducing AdamW memory footprint by x16 while retaining quality. But - you did not supply any code, making the reproducibility more difficult, and in the paper there are not enough pretraining tasks. You don't have to have huge models - look at other optimizers papers, such as Adam-Mini, Gefen, and others, take a few pretraining tasks 5-10 tasks, and show the training curve of your method against AdamW. Right now I see only a single curve at the very last page of the paper - the training curves should be the main thing. If they are similar, reproducible, and with lower memory footprint, that's huge. Thanks.
We thank the reviewer for their speedy response.
Right now I see only a single curve at the very last page of the paper - the training curves should be the main thing.
We apologize for any confusion regarding the placement of Figure 2 in the revised manuscript. Our intention in temporarily placing it in the appendix was solely to make the figure accessible to reviewers without disrupting the main text during the discussion period. Once all the reviewers’ feedback is finalized, we will incorporate these recommended changes and new experimental results into the main draft, and the pretraining loss curves will be incorporated into Section 4.
Additional Pretraining experiments and runtime metrics
We did not include pretraining results earlier in our original submission since we thought pretraining results needed to be shown for models of at least 4B, for which we lacked the compute budget. Now, in response to the reviewers’ suggestion of showing pretraining for smaller models, we conducted the following experiments (along with the GPT-2 Large pretraining that we reported earlier): (1) Pretraining of LLama-2-271M and LLama-3-490M on the C4 dataset using Adam-mini’s official codebase (2) Training of Resnet-18 on the Imagenet Dataset using the popular PyTorch-image-models (timm) codebase. Please find the loss and validation accuracy curves in Appendix B.5 of our updated manuscript; the image is placed in the appendix only to follow ICLR guidelines. We will add it to the main draft at the end of the rebuttal phase. As the training loss curves demonstrate, all compressed optimizers closely track the full-precision AdamW baseline with ReAdam achieving the lowest memory footprint while maintaining highly competitive per-step execution times. Detailed runtime metrics are below:
| Pretraining Task | Number of Parameters | Training Steps | Optimizer | Total Time taken (in hours) | Peak Allocated Memory (in GB) | Optimizer State size (in MB) |
|---|---|---|---|---|---|---|
| GPT2-Large | 774M | 50k | AdamW | 51.6 | 46.2123 | 5905.4 |
| 8-bit AdamW | 52.38 | 41.9125 | 1501.8 | |||
| 4-bit LPMM w/ Adafactor | 53.98 | 40.8350 | 398.3 | |||
| 4-bit ReAdam | 52.17 | 40.8126 | 375.3 | |||
| Llama2 | 291M | 50k | AdamW | 5.64 | 129.663 | 2168.7 |
| 8-bit AdamW | 6.28 | 128.045 | 550.8 | |||
| 4-bit LPMM w/ Adafactor | 6.56 | 127.640 | 145.8 | |||
| 4-bit ReAdam | 6.20 | 127.631 | 137.3 | |||
| Llama3 | 490M | 10k | AdamW | 1.50 | 80.3043 | 3913.6 |
| 8-bit AdamW | 1.56 | 77.3847 | 993.9 | |||
| 4-bit LPMM w/ Adafactor | 1.60 | 76.6533 | 262.5 | |||
| 4-bit ReAdam | 1.53 | 76.6386 | 247.2 | |||
| Resnet | 12M | 150k | AdamW | 10.0 | 13.4127 | 89.2 |
| 8-bit AdamW | 9.97 | 13.3057 | 22.7 | |||
| 4-bit LPMM w/ Adafactor | 10.96 | 12.9471 | 35.3 | |||
| 4-bit ReAdam | 10.93 | 12.5906 | 5.8 |
Due to the paucity of time, the Llama3 reported numbers above are only for 10k steps. We will make sure to update the plot once the experiments are complete. Additionally, we are also running experiments on resnet101, but we may not have the results before the rebuttal window closes.
You did not supply any code, making the reproducibility more difficult.
We have uploaded a zip of our complete codebase, including detailed instructions to reproduce our throughput analysis and the GPT-2 Large pretraining runs as supplementary material on OpenReview. Please note that the current release supports only GPU-based training via CUDA. We are in the process of including CPU and Triton-based implementations of the same.
If they are similar, reproducible, and with lower memory footprint, that's huge.
We hope that with our additional pretraining experiments and supplementary code, it is now more convincing to the reviewer that ReAdam does indeed produce similar, reproducible training curves with lower memory overhead.
Thanks for engaging with us so far! We look forward to hearing from you if there are any pending concerns.
We thank the reviewer for their review and address their specific concerns below.
Our experiments span three modalities (speech, text, vision) and trainable parameter counts ranging from 38M to 7B, with evaluations across three different training conditions (LoRA, full fine-tuning, and training from scratch). Since fine-tuning large models on resource-constrained hardware is an important use case for low-bit optimizers, prior work (such as 8-bit Adam, 4-bit LPMM, and SOLO) also focuses on fine-tuning benchmarks for their evaluation.
Pretraining of models with 7B or more parameters is beyond our compute budget. Motivated by the reviewers’ feedback, we have now expanded our evaluation to also report preliminary pretraining experiments (with 50k training steps) from pretraining GPT-2 large (774M parameters) on OpenWebText in our general comment "Preliminary Pretraining Results on GPT-2" above.
We thank the reviewer for pointing us to Gefen, a new Hessian-grounded efficient optimizer which is a successor to Adam-mini. Since these ideas are complementary to ReAdam, they can be combined as highlighted in Section 5; we are exploring these avenues as part of future work.
We thank the reviewer for this suggestion. Table 3 was included to bridge the gap imposed by our computational constraints: although training models beyond 7B parameters from scratch exceeds our compute budget, we wanted to give practitioners a practical estimate of how much memory ReAdam can save as models scale to hundreds of billions of parameters. We agree with the reviewer that Table 3 can also be effectively summarized via a one-line calculation, with the table moved to the appendix. We will incorporate this change in the revised version.
We conducted a detailed time, throughput, FLOPs, and memory profiling analysis across diverse tensor shapes and models to demonstrate how ReAdam compares computationally against AdamW, 8-bit AdamW, and 4-bit LPMM. The details are provided in our general comment "Detailed Computation and Throughput analysis" above.
Recommended edits to lines 13-14, 27, Table 1 subscripts, and adding gefen to related work: We thank the reviewer for catching these typos and omissions. We will incorporate these changes in the updated manuscript.
The derivation for the theoretical bound is missing: For the general case of β12<β2, using Cauchy’s inequality, one can get an upper bound on the update ∣Ut∣ as (1−β2)(1−β12/β2)1−β1. For standard hyperparameters (β1=0.9, β2=0.999), this evaluates to ≈7.27. The complete step-by-step derivation of this bound is available in Appendix 3 and Appendix 4 of "Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL" by Miahi et al. (https://arxiv.org/abs/2602.03839). We will point to these details in an updated version. While the value for ∣Ut∣ can theoretically reach as high as 7.27, our empirical bucket analysis shows that a very tiny fraction of total weight parameters go beyond $ \mid \mathbf{U}_t \mid> 1$; therefore, we chose to have quantization levels in the range [−1,1].
Algorithm 1 does not mention how 1D and higher-order tensors are handled: We thank the reviewer for highlighting this. For 1D parameters where Adafactor cannot operate, we directly perform 8-bit compression for the scale (St in equation 4). We use 8-bit compression here because our code is integrated within bitsandbytes, which readily offers 8 bit compression for arbitrary-range tensors. Given these 1D parameters are often quite small, using 4 bit vs 8 bit offers little benefit. For higher-order parameters, we stick to the original ReAdam, and the Adafactor compression is employed on the last 2 axes. This is consistent with how PyTorch’s Adafactor is implemented (lines 100-109 in https://github.com/pytorch/pytorch/blob/v2.14.0/torch/optim/_adafactor.py). We will include these details in the revised version.
We thank the reviewer again for their thoughtful review, and would be happy to provide any further clarifications.