acceptodds
Under review as a conference paper at ICLR 2027

ReAdam: Reformulating Adam for Low-Bit Quantization

Abstract

Adaptive optimizers like Adam are very effective for model training, adaptively adjusting the learning rate for each learnable parameter using first- and second-order gradient statistics. However, storing these moments substantially increases Adam’s memory footprint. Existing memory-efficient optimizers aim at preserving Adam’s optimizer state representation and quantizing the first- and second-order moments independently. However, these raw moments are unbounded and prone to outliers, thus requiring more elaborate and memory-intensive quantization schemes. We pose a different question: instead of improving how Adam’s existing optimizer state can be quantized, can the optimizer state itself be reparameterized to be more amenable to quantization? To this end, we derive a new scale-normalized reformulation of Adam in READAM by separating the optimizer state into (i) a gradient scale and (ii) a normalized quantity corresponding directly to Adam’s effective update. We establish that the square root of the second moment is a particularly effective choice for the scale. READAM is an equivalent reformulation of Adam using these two new optimizer coordinates, that are much more compression-friendly compared to the raw moments of Adam. READAM performs on par with or better than existing low-bit optimizers across multiple benchmarks and modalities (text, speech and vision), while also reducing optimizer-state memory usage by 6% (relative).

Primary area:
optimization
Keywords:
optimizer, Adam, quantization, memory efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2fCAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present ReAdam, an exact reformulation of Adam that stores, instead of the first and second moments, the effective update U and a scale S, with recurrences derived for both. Same memory footprint as Adam. Because U does not depend on the gradient scale and mostly lies in [−1,1], it is quantized to 4 bits with a fixed codebook and no per-block scales, while S is compressed with an Adafactor rank-1 factorization of V. 4-bit ReAdam matches the quality of 4-bit LPMM with Adafactor, with 5.3% less optimizer-state memory on average.

Critical strengths

The paper proposes an interesting, exact reformulation of Adam that stores the effective update U instead of the first moment. U is scale invariant and mostly lies in [−1,1], so it can be quantized with a single fixed codebook, removing the need to store a scale factor per block as in prior 4-bit optimizers. The derivation is clean and easy to follow (Appendix A).

Critical weaknesses

W1 Scope of evaluation. The paper presents ReAdam as a general memory-efficient replacement for Adam (e.g., Table 3 projects savings for training 1B to 684B models), yet 4 of the 5 experiments are fine-tuning. In Table 1, 4-bit ReAdam outperforms full precision AdamW on all three LLM fine-tuning tasks but underperforms it on the only from-scratch task (LibriSpeech, 38M; 16.6 vs 16.2 WER). There is no LLM pretraining, which is where optimizer memory matters most, and no loss curves are shown. Please either add LLM pretraining at a reasonable model size, with training and validation loss curves against AdamW and LPMM, or state explicitly in the Limitations that the method is validated mainly for fine-tuning. I suggest also comparing it against the new Gefen optimizer, or to try to combine ReAdam with Gefen (successor to Adam-Mini that you mentioned).

W2. Table 3. The table adds little: the relative saving over LPMM (AF) is a constant ~5.9% across all models, which follows from removing LPMM's per-block scales (0.25 of 4.25 bits per parameter). The claim that savings grow with model size (line 424) holds only in absolute GB, by construction. Since these are analytical estimates and no such models were trained, I suggest replacing the table with this one-line calculation and moving it to the appendix.

W3. Efficiency. The paper reports only optimizer-state memory. Please add per-step time or throughput against AdamW and LPMM.

Questions for authors

Abstract, lines 013 to 014. The sentence "storing these moments substantially increases Adam's memory footprint" is slightly imprecise. The first and second moments are not an addition on top of Adam; they are Adam's optimizer state, so they constitute its memory footprint rather than increase it. I suggest rephrasing.

Lines 130, 138, 179. U_t is called "theoretically bounded", but no bound is derived or stated, and Kingma & Ba (2015) only give an informal approximation under a different condition. Please provide the bound and its value.

Related Work - Another method that deals with quantization and sharing moments of Adam is the Gefen optimizer, and may be relevant to mention.

Algorithm 1. The algorithm is given only for matrix parameters, since the factored scale needs row and column statistics. Please state how 1D parameters (biases, norm weights) and higher order tensors (e.g., conv kernels) are handled, and how this affects the reported memory.

Abstract, line 027. Please say what the 6% is relative to. As written it reads as relative to Adam, but it is relative to 4-bit LPMM

Table 1, subscript should be % and not x

5 comments

We thank the reviewer for their review and address their specific concerns below.

W1: Scope of evaluation.

Our experiments span three modalities (speech, text, vision) and trainable parameter counts ranging from 38M to 7B, with evaluations across three different training conditions (LoRA, full fine-tuning, and training from scratch). Since fine-tuning large models on resource-constrained hardware is an important use case for low-bit optimizers, prior work (such as 8-bit Adam, 4-bit LPMM, and SOLO) also focuses on fine-tuning benchmarks for their evaluation.

Pretraining of models with 7B or more parameters is beyond our compute budget. Motivated by the reviewers’ feedback, we have now expanded our evaluation to also report preliminary pretraining experiments (with 50k training steps) from pretraining GPT-2 large (774M parameters) on OpenWebText in our general comment "Preliminary Pretraining Results on GPT-2" above.

We thank the reviewer for pointing us to Gefen, a new Hessian-grounded efficient optimizer which is a successor to Adam-mini. Since these ideas are complementary to ReAdam, they can be combined as highlighted in Section 5; we are exploring these avenues as part of future work.

W2: The significance of Table 3

We thank the reviewer for this suggestion. Table 3 was included to bridge the gap imposed by our computational constraints: although training models beyond 7B parameters from scratch exceeds our compute budget, we wanted to give practitioners a practical estimate of how much memory ReAdam can save as models scale to hundreds of billions of parameters. We agree with the reviewer that Table 3 can also be effectively summarized via a one-line calculation, with the table moved to the appendix. We will incorporate this change in the revised version.

W3: Additional throughput numbers

We conducted a detailed time, throughput, FLOPs, and memory profiling analysis across diverse tensor shapes and models to demonstrate how ReAdam compares computationally against AdamW, 8-bit AdamW, and 4-bit LPMM. The details are provided in our general comment "Detailed Computation and Throughput analysis" above.

Other comments:

  • Recommended edits to lines 13-14, 27, Table 1 subscripts, and adding gefen to related work: We thank the reviewer for catching these typos and omissions. We will incorporate these changes in the updated manuscript.

  • The derivation for the theoretical bound is missing: For the general case of β12<β2\beta_1^2 < \beta_2, using Cauchy’s inequality, one can get an upper bound on the update ∣Ut∣\mid \mathbf{U_t} \mid as 1−β1(1−β2)(1−β12/β2)\frac{1 - \beta_1}{\sqrt{(1-\beta_2) (1 - \beta_1^2 /\beta_2)}}. For standard hyperparameters (β1\beta_1=0.90.9, β2\beta_2=0.9990.999), this evaluates to ≈7.27\approx 7.27. The complete step-by-step derivation of this bound is available in Appendix 3 and Appendix 4 of "Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL" by Miahi et al. (https://arxiv.org/abs/2602.03839). We will point to these details in an updated version. While the value for ∣Ut∣\mid \mathbf{U}_t \mid can theoretically reach as high as 7.277.27, our empirical bucket analysis shows that a very tiny fraction of total weight parameters go beyond $ \mid \mathbf{U}_t \mid> 1$; therefore, we chose to have quantization levels in the range [−1,1][-1, 1].

  • Algorithm 1 does not mention how 1D and higher-order tensors are handled: We thank the reviewer for highlighting this. For 1D parameters where Adafactor cannot operate, we directly perform 8-bit compression for the scale (St\mathbf{S}_t in equation 4). We use 8-bit compression here because our code is integrated within bitsandbytes, which readily offers 8 bit compression for arbitrary-range tensors. Given these 1D parameters are often quite small, using 4 bit vs 8 bit offers little benefit. For higher-order parameters, we stick to the original ReAdam, and the Adafactor compression is employed on the last 2 axes. This is consistent with how PyTorch’s Adafactor is implemented (lines 100-109 in https://github.com/pytorch/pytorch/blob/v2.14.0/torch/optim/_adafactor.py). We will include these details in the revised version.

We thank the reviewer again for their thoughtful review, and would be happy to provide any further clarifications.

Thanks for the additional experiments and clarifications.

The throughput measurements address W3, and the proposed revision to Table 3 addresses W2. The GPT-2 Large pretraining experiment also addresses the model-scale aspect of W1. However, the pretraining quality comparison remains unclear. In figure 2, why don't you put for each optimizer, the memory footprint and total clock time? You attribute the gap to clipping updates to the interval [−1,1]. Is this supported by an ablation? The response strengthens the paper, but I currently retain my rating. The idea of the paper is very promising - reducing AdamW memory footprint by x16 while retaining quality. But - you did not supply any code, making the reproducibility more difficult, and in the paper there are not enough pretraining tasks. You don't have to have huge models - look at other optimizers papers, such as Adam-Mini, Gefen, and others, take a few pretraining tasks 5-10 tasks, and show the training curve of your method against AdamW. Right now I see only a single curve at the very last page of the paper - the training curves should be the main thing. If they are similar, reproducible, and with lower memory footprint, that's huge. Thanks.

We thank the reviewer for their speedy response.

Right now I see only a single curve at the very last page of the paper - the training curves should be the main thing.

We apologize for any confusion regarding the placement of Figure 2 in the revised manuscript. Our intention in temporarily placing it in the appendix was solely to make the figure accessible to reviewers without disrupting the main text during the discussion period. Once all the reviewers’ feedback is finalized, we will incorporate these recommended changes and new experimental results into the main draft, and the pretraining loss curves will be incorporated into Section 4.

Additional Pretraining experiments and runtime metrics

We did not include pretraining results earlier in our original submission since we thought pretraining results needed to be shown for models of at least 4B, for which we lacked the compute budget. Now, in response to the reviewers’ suggestion of showing pretraining for smaller models, we conducted the following experiments (along with the GPT-2 Large pretraining that we reported earlier): (1) Pretraining of LLama-2-271M and LLama-3-490M on the C4 dataset using Adam-mini’s official codebase (2) Training of Resnet-18 on the Imagenet Dataset using the popular PyTorch-image-models (timm) codebase. Please find the loss and validation accuracy curves in Appendix B.5 of our updated manuscript; the image is placed in the appendix only to follow ICLR guidelines. We will add it to the main draft at the end of the rebuttal phase. As the training loss curves demonstrate, all compressed optimizers closely track the full-precision AdamW baseline with ReAdam achieving the lowest memory footprint while maintaining highly competitive per-step execution times. Detailed runtime metrics are below:

Pretraining TaskNumber of ParametersTraining StepsOptimizerTotal Time taken (in hours)Peak Allocated Memory (in GB)Optimizer State size (in MB)
GPT2-Large774M50kAdamW51.646.21235905.4
8-bit AdamW52.3841.91251501.8
4-bit LPMM w/ Adafactor53.9840.8350398.3
4-bit ReAdam52.1740.8126375.3
Llama2291M50kAdamW5.64129.6632168.7
8-bit AdamW6.28128.045550.8
4-bit LPMM w/ Adafactor6.56127.640145.8
4-bit ReAdam6.20127.631137.3
Llama3490M10kAdamW1.5080.30433913.6
8-bit AdamW1.5677.3847993.9
4-bit LPMM w/ Adafactor1.6076.6533262.5
4-bit ReAdam1.5376.6386247.2
Resnet12M150kAdamW10.013.412789.2
8-bit AdamW9.9713.305722.7
4-bit LPMM w/ Adafactor10.9612.947135.3
4-bit ReAdam10.9312.59065.8

Due to the paucity of time, the Llama3 reported numbers above are only for 10k steps. We will make sure to update the plot once the experiments are complete. Additionally, we are also running experiments on resnet101, but we may not have the results before the rebuttal window closes.

You did not supply any code, making the reproducibility more difficult.

We have uploaded a zip of our complete codebase, including detailed instructions to reproduce our throughput analysis and the GPT-2 Large pretraining runs as supplementary material on OpenReview. Please note that the current release supports only GPU-based training via CUDA. We are in the process of including CPU and Triton-based implementations of the same.

If they are similar, reproducible, and with lower memory footprint, that's huge.

We hope that with our additional pretraining experiments and supplementary code, it is now more convincing to the reviewer that ReAdam does indeed produce similar, reproducible training curves with lower memory overhead.

Thanks for engaging with us so far! We look forward to hearing from you if there are any pending concerns.

Reviewer_zMrAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

ReAdam addresses the difficulty of low bit quantizing AdamW by reformulating Adam’s first and second moment state into two quantization friendly coordinates: a coordinate wise gradient scale (from the bias corrected second moment) and a bounded effective update. Recursive updates preserve gradient magnitude in the scale and keep the effective update stable under standard decay, enabling low bit storage. Experiments show the optimizer state can be represented with 4 bits and achieves competitive or better performance than existing low-bit optimizers across text, speech, and vision, with modest memory savings.

Critical strengths
  • This idea of modifying AdamW so the optimizer state becomes more suitable for quantization is very interesting.
  • Experiments show that the resulting optimizer state can be represented with 4 bits, performs on par with or better than existing low bit optimizers across text, speech, and vision benchmarks, and achieves a modest reduction of optimizer state memory usage compared with prior work.
Critical weaknesses

Overall, the experimental validation feels a bit weak, so it is hard to conclude whether the proposed solution is truly effective.

  • In Table 1, all comparisons should ideally be carried out on pretraining rather than relying on fine-tuning, since fine-tuning results do not convincingly demonstrate the effectiveness of quantization.
  • The observation that the 4-bit version performs even better than both the full precision baseline and the 8-bit version is quite surprising, because quantization is usually expected to cause little or moderate degradation relative to high precision while primarily reducing memory usage. It would be helpful to clarify how such an improvement is possible, for example whether it is due to implementation details, different training settings, evaluation randomness, or any regularization effect.
  • It would be more convincing to report and compare memory consumption and running time as well, so we can judge whether the performance gain justifies any additional computational overhead.
Questions for authors

Please see the weaknesses listed above.

3 comments

We thank the reviewer for their review and address their specific concerns below.

Comparisons should have been carried out on pretraining and not finetuning.

From an optimizer-state compression perspective, we do not think fine-tuning is inherently less challenging than pretraining. During pretraining, quantization must correctly manage variations in gradient magnitudes and directions, along with dynamically changing step sizes. During fine-tuning, models must also preserve small changes in optimizer state that are necessary to effectively adapt a pretrained model. Coarse quantization can introduce instability or cause gradient stagnation, thus making fine-tuning a rigorous setting for testing a compressed optimizer's stability and numerical robustness. Moreover, since fine-tuning large models on resource-constrained hardware is an important practical use case for low-bit optimizers, prior work (8-bit Adam, 4-bit LPMM and SOLO) also focuses on fine-tuning benchmarks for their evaluation.

Furthermore, our evaluation suite already includes a from-scratch speech recognition task: the E-Branchformer model (38M parameters) is trained end-to-end on the 100-hour LibriSpeech corpus (Table 1). Although the size of the model is relatively small, it still tests ReAdam on a setting (akin to pretraining) where all the parameters are learned from scratch.

Pretraining of models with 7B or more parameters is unfortunately beyond our compute budget. Motivated by the reviewers’ feedback, we have now expanded our evaluation to also report preliminary pretraining experiments (with 50k training steps) from pretraining GPT-2 large (774M parameters) in our general comment "Preliminary Pretraining Results on GPT-2" above.

Claim that 4-bit ReAdam outperforms full-precision Adam

We want to unequivocally state that ReAdam performs comparably to the full-precision baseline while being highly memory-efficient. Although we see slight gains in a few experiments, we attribute these performance fluctuations to training stochasticity, and they are not statistically significant. We have explicitly stated this in lines 386–388 of the manuscript as well. We will make sure there are no statements that suggest otherwise in a revised version.

Experiment settings: To ensure fairness, for a given benchmark, we’ve maintained identical experimental conditions across all our experiments, including the hardware environment, GPU type, and training hyperparameters. We substitute only the optimizer, as already detailed in footnote 3.

Why ReAdam performs no worse than full-precision Adam: We hypothesize that because quantization levels at low bit-widths are far apart, the states remain at their current levels until the incoming gradient signal induces a sufficiently large change, producing larger-than-desired updates. For fine-tuning, this may not be very disruptive since training starts near a local minimum with more stable gradients and scope for recovery. We have discussed this phenomenon in lines 406–413.

Additional memory and computational throughput results

We conducted a detailed time, throughput, FLOPs, and memory profiling analysis across diverse tensor shapes and models to demonstrate how ReAdam compares against AdamW, 8-bit AdamW, and 4-bit LPMM. The details are provided in our general comment "Detailed Computation and Throughput analysis" above.

We thank the reviewer again for their thoughtful review, and would be happy to provide any further clarifications.

We appreciate the reviewer’s feedback. In addition to the results and clarifications posted earlier, we have some additional pretraining results in response to Reviewer 2fCA’s follow-up comment. In total, we now have results on 4 additional pretraining tasks: (1) GPT-2 Large on OpenWebText, (2) LLama-2-271M on the C4 dataset, (3) LLama-3-490M on the C4 dataset, and (4) Resnet-18 on the Imagenet dataset. Please find the loss curve in Appendix B.5 of our updated manuscript; the image is placed in the appendix ONLY to ensure reviewers can access it without breaking ICLR guidelines. We will add it to the main content at the end of the rebuttal phase. As these training loss curves demonstrate, all compressed optimizers closely track the full-precision AdamW baseline with ReAdam achieving the lowest memory footprint and maintaining highly competitive per-step execution times. Detailed runtime metrics are below:

Pretraining TaskNumber of ParametersTraining StepsOptimizerTotal Time taken (in hours)Peak Allocated Memory (in GB)Optimizer State size (in MB)
GPT2-Large774M50kAdamW51.646.21235905.4
8-bit AdamW52.3841.91251501.8
4-bit LPMM w/ Adafactor53.9840.8350398.3
4-bit ReAdam52.1740.8126375.3
Llama2291M50kAdamW5.64129.6632168.7
8-bit AdamW6.28128.045550.8
4-bit LPMM w/ Adafactor6.56127.640145.8
4-bit ReAdam6.20127.631137.3
Llama3490M10kAdamW1.5080.30433913.6
8-bit AdamW1.5677.3847993.9
4-bit LPMM w/ Adafactor1.6076.6533262.5
4-bit ReAdam1.5376.6386247.2
Resnet12M150kAdamW10.013.412789.2
8-bit AdamW9.9713.305722.7
4-bit LPMM w/ Adafactor10.9612.947135.3
4-bit ReAdam10.9312.59065.8

Additionally, we have uploaded our complete codebase as supplementary material along with detailed reproducibility instructions. Please let us know if we can help with any additional clarifications before the discussion period ends. We look forward to hearing from you.

We hope our additional pretraining experiments and throughput analyses address the reviewer's key concerns. Since the official discussion period is soon drawing to a close, we would greatly appreciate hearing from you.

Thanks in advance!

Reviewer_M3rcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a reparameterization of Adam's moment tensors that make them more amenable to quantization, allowing large memory savings through 4-bit quantization, whilst preserving the performance in experiments.

Critical strengths

The new parameterization is well motivated, and easy to follow in the paper. To the best of my knowledge, this is a new idea that's a nice contribution. It appears to do work as intended in experiments, in saving memory without costing performance.

Critical weaknesses

The memory use of Adam was a major practical problem several years ago, but my impression was that it had largely been resolved in practice by optimizer state sharding (Zero, etc.). The paper does mention Zero, but just says that they are taking a "different line of attack" instead of motivating what the advantage of their approach is.

Most of the experiments are in Lora settings, with few parameters, so the memory cost is small even in full precision. It's unclear how well the results would generalize when optimizing larger numbers of parameters.

It seems a bit surprising that they claim their method outperforms full precision Adam in most of their experiments.

Questions for authors
3 comments

We thank the reviewer for their constructive review and address their questions/comments below.

1. Sharding techniques like ZeRO already solve the memory problem in training using Adam, so what is the advantage of using ReAdam?

Sharding and compression address two distinct sources of memory overhead during training. The main goal of optimizer-state sharding is to eliminate the memory overhead introduced by the duplication of optimizer states in a distributed training setup (where multiple GPUs require access to optimizer states, resulting in state duplication). Theoretically, even if sharding completely eliminates cross-device duplication, the underlying optimizer-state memory requirement per parameter (summed across devices) remains identical to that in a single-GPU training setup. That is, sharding minimizes only the redundancy of the states across devices; the actual size of the states themselves is unchanged. This is where compression comes into play. The goal of compression is to reduce the memory footprint of the optimizer states themselves, irrespective of whether training happens on a single GPU or in a distributed cluster. In principle, both techniques are complementary and can be combined: ZeRO minimizes cross-device optimizer state redundancy, while ReAdam reduces the underlying memory required to store the optimizer states on each device.

2. Experiments largely focus on LoRA-based training where parameter counts are lower.

Our evaluations extend well beyond LoRA-based training: three of the five experimental settings in Table 1 correspond to full fine-tuning or training from scratch. Only two use PEFT (specifically LoRA), and they differ in rank and architecture. Altogether, the experiments span three modalities (speech, text, vision) and trainable parameter counts ranging from 38M to 7B, with evaluations across different training conditions (LoRA, full fine-tuning, and training from scratch).

Pretraining of models with 7B or more parameters is unfortunately beyond our compute budget. Motivated by the reviewers’ feedback, we have now expanded our evaluation to also report preliminary pretraining experiments (with 50k training steps) from pretraining GPT-2 large (774M parameters) in our general comment "Preliminary Pretraining Results on GPT-2" above.

3. Claim that ReAdam outperforms full-precision Adam

We want to unequivocally state that ReAdam performs comparably to the full-precision baseline while being highly memory-efficient. Although we see slight gains in some experiments, we attribute these performance fluctuations to training stochasticity and they are not statistically significant. We have explicitly stated this in lines 386–388 as well. We will make sure there are no statements that suggest otherwise in a revised version.

4. Additional throughput analysis

We additionally conducted a detailed time, throughput, FLOPs, and memory profiling analysis across diverse tensor shapes and models to demonstrate how ReAdam compares against AdamW, 8-bit AdamW, and 4-bit LPMM. The details are provided in our general comment "Detailed Computation and Throughput analysis" above.

We thank the reviewer again for their thoughtful review and would be happy to provide any further clarifications.

We appreciate the reviewer’s feedback. In addition to the results and clarifications posted earlier, we have some additional pretraining results in response to Reviewer 2fCA’s follow-up comment. In total, we now have results on 4 additional pretraining tasks: (1) GPT-2 Large on OpenWebText, (2) LLama-2-271M on the C4 dataset, (3) LLama-3-490M on the C4 dataset, and (4) Resnet-18 on the Imagenet dataset. Please find the loss curve in Appendix B.5 of our updated manuscript; the image is placed in the appendix ONLY to ensure reviewers can access it without breaking ICLR guidelines. We will add it to the main content at the end of the rebuttal phase. As these training loss curves demonstrate, all compressed optimizers closely track the full-precision AdamW baseline with ReAdam achieving the lowest memory footprint and maintaining highly competitive per-step execution times. Detailed runtime metrics are below:

Pretraining TaskNumber of ParametersTraining StepsOptimizerTotal Time taken (in hours)Peak Allocated Memory (in GB)Optimizer State size (in MB)
GPT2-Large774M50kAdamW51.646.21235905.4
8-bit AdamW52.3841.91251501.8
4-bit LPMM w/ Adafactor53.9840.8350398.3
4-bit ReAdam52.1740.8126375.3
Llama2291M50kAdamW5.64129.6632168.7
8-bit AdamW6.28128.045550.8
4-bit LPMM w/ Adafactor6.56127.640145.8
4-bit ReAdam6.20127.631137.3
Llama3490M10kAdamW1.5080.30433913.6
8-bit AdamW1.5677.3847993.9
4-bit LPMM w/ Adafactor1.6076.6533262.5
4-bit ReAdam1.5376.6386247.2
Resnet12M150kAdamW10.013.412789.2
8-bit AdamW9.9713.305722.7
4-bit LPMM w/ Adafactor10.9612.947135.3
4-bit ReAdam10.9312.59065.8

Additionally, we have uploaded our complete codebase as supplementary material along with detailed reproducibility instructions. Please let us know if we can help with any additional clarifications before the discussion period ends. We look forward to hearing from you.

We hope we were able to address the reviewer's key concerns. Since the official discussion period is soon drawing to a close, we would greatly appreciate hearing from you.

Thanks in advance!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Preliminary Pretraining Results on GPT-2

We thank all the reviewers for their constructive feedback. In response to the reviewers’ suggestion to show pretraining experiments, we report preliminary pretraining results on GPT-2 Large (774M parameters) on the OpenWebText dataset for 50k training steps. These experiments were conducted on a single 180 GB NVIDIA B200 GPU with CUDA 12.8 and PyTorch 2.11.0. Please find the loss curve in Appendix B.5 of our updated manuscript. As the training loss curves demonstrate, all compressed optimizers closely track the full-precision AdamW baseline. In terms of optimizer state memory, ReAdam achieves the lowest memory footprint at 375.3 MB (compared to 398.3 MB for LPMM and 1.47 GB for 8-bit Adam) while maintaining highly competitive per-step execution times of ~370 ms (matching 8-bit Adam at ~372 ms and outperforming LPMM at ~395 ms). The small offset between ReAdam and LPMM is due to the conservative lag introduced by clipping updates to [-1,1]; this restriction keeps optimization steps constrained without causing any training divergence. We will incorporate these pretraining results, memory metrics, and loss trajectories into the revised version.

AuthorsOpenReview ↗

Detailed Computation and Throughput analysis

We thank all the reviewers for their constructive feedback. In response to the reviewers’ questions about throughput, we report wall-clock time, FLOPs, and throughput of ReAdam against AdamW, 8-bit AdamW, and the official implementation of 4-bit LPMM across diverse tensor shapes and models.

Implementation details: ReAdam is integrated directly into the bitsandbytes library; we will make the code publicly available upon acceptance. The entire computation of ReAdam, from quantization/dequantization and state updates (for both U and Adafactor statistics) to parameter updates, is implemented as a single fused CUDA kernel. To accommodate Adafactor-style state updates within this fused kernel (which require row and column reductions), the kernel operates on tiles of the weight matrix and the partial row and column sums associated with the tile are added into global memory using atomic operations.

All benchmarks were measured on a single 48 GB NVIDIA RTX A6000 Ada Generation GPU using CUDA 12.6 and PyTorch 2.6.0. Reported numbers represent the median across 5 independent runs after 10 warmup iterations, with cache clearing and explicit CUDA synchronization enforced prior to recording.

Wall time, FLOPs, and Optimizer State Size across common weight shapes for 1 optimization step

Weight TypeOptimizerTime taken for 1 step (in ms)FLOPs for 1 step (in GFLOP)Optimizer State size (in MB)
1-D tensor of shape (4096,) (eg: Bias)AdamW0.10850.00010.03
8-bit AdamW0.11950.00010.01
4-bit LPMM w/ Adafactor0.18920.00010.01
4-bit ReAdam0.11520.00010.01
1-D tensor of shape (8192,) (eg: Bias)AdamW0.10730.00020.06
8-bit AdamW0.11940.00030.02
4-bit LPMM w/ Adafactor0.58670.00050.01
4-bit ReAdam0.11530.00030.01
1-D tensor of shape (1048576,) (eg: Bias)AdamW0.27270.02948.00
8-bit AdamW0.11850.03362.03
4-bit LPMM w/ Adafactor0.58830.05861.06
4-bit ReAdam0.11600.03411.01
2-D tensor of shape (2048, 2048) (eg: Attention)AdamW0.24780.117532.0
8-bit AdamW0.16200.13428.13
4-bit LPMM w/ Adafactor0.56700.18182.14
4-bit ReAdam0.14870.14112.02
2-D tensor of shape (4096, 16) (eg: LoRA)AdamW0.24880.00180.50
8-bit AdamW0.12010.00210.13
4-bit LPMM w/ Adafactor0.56270.00330.05
4-bit ReAdam0.15090.00270.05
2-D tensor of shape (4096, 14336) (eg: MLP Up projection)AdamW2.46441.6451448.0
8-bit AdamW1.61021.8790113.75
4-bit LPMM w/ Adafactor8.74802.545429.82
4-bit ReAdam1.05021.976028.07
3-D tensor of shape (8, 1024, 4096) (eg: MoE)AdamW1.48650.9400256.0
8-bit AdamW0.94491.073765.0
4-bit LPMM w/ Adafactor4.87631.447717.16
4-bit ReAdam0.65431.129316.16
4-D tensor of shape (512, 512, 3, 3) (eg: Convolution)AdamW0.24970.066118.00
8-bit AdamW0.13630.07554.57
4-bit LPMM w/ Adafactor0.56040.12737.20
4-bit ReAdam0.29250.11788.50

Key Takeaways:

  • Runtime & Memory: Overall, ReAdam performs comparably to or faster than 8-bit Adam in most cases, while maintaining superior memory efficiency. For large-scale Transformer blocks (such as MLP, up-projections, and MoE tensors), ReAdam achieves up to 35% faster runtime than 8-bit Adam and ~80% faster runtime relative to LPMM.
  • Edge cases: (1) For small convolutional kernels like (512, 512, 3, 3), storing factored Adafactor row/column statistics across the last 2 axes (3 x 3) adds overhead where simple 8-bit second-moment compression proves slightly more memory-efficient. This could be mitigated by reshaping the 4D tensor into a 2D matrix of shape (512, 512 * 3 * 3), something we are yet to explore as it requires careful analysis from an Adafactor scaling perspective. (2) For heavily constrained narrow shapes like (4096, 16) (e.g., LoRA B), tiling gets restricted, resulting in slight time degradation.

Wall time, Throughput, FLOPs, and Memory on GPT2-Large for 1 model step (forwawrd + backward)

ModelOptimizerTime taken for 1 step (in ms)ThroughputFLOPs for 1 step (in GFLOP)Optimizer State size (in MB)
GPT2-Large-770MAdamW385.1110,636 tokens/s21.68515905.4
8-bit AdamW400.1010,238 tokens/s24.77301501.8
4-bit LPMM w/ Adafactor467.978,753 tokens/s33.5445398.3
4-bit ReAdam399.5710,251 tokens/s27.7754375.3

Key Takeaways: ReAdam takes comparable time and has throughput similar to 8-bit AdamW when performing one step (forward + backward) through the model, while requiring only 25% of the memory for 8-bit AdamW. Across all the metrics, we find ReAdam to perform significantly better than LPMM while also requiring the least amount of memory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.