- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an optimizer, coined TACO, that reduces memory usage for LLM fine-tuning by keeping a sparse gradient history instead of two dense moment matrices.
It retains 16 history entries per column and updates only the coordinate with the largest absolute history value, using its sign to determine the update direction.
The authors derive this update from a normalized operator-norm constraint and provide theoretical analysis under simplified assumptions.
The experiments show substantial peak-memory savings and enable fine-tuning models up to 32B parameters on one H100, although accuracy is lower than several baselines on some tasks.
Critical strengths
1, The authors derive the column-wise Top-1 update as the exact steepest-descent direction under their chosen operator-norm constraint. This gives a clear explanation for both the coordinate selection and the matrix-dimension scaling.
2, Since the update only needs the dominant coordinate in each column, TACO stores a small set of candidates in FP8 instead of maintaining a dense history. The ablations help explain the choice of 16 retained entries and its memory–accuracy trade-off.
3, On OPT-13B, TACO reduces peak memory from 80.6 GB to 27.5 GB compared with AdamW8bit. It also enables fine-tuning models up to 32B parameters on one H100, with experiments covering several model families.
Critical weaknesses
1, The choice of 16 retained entries is supported by an ablation on OPT-1.3B, but the paper does not measure how often this compressed history preserves the winner selected by dense history. The convergence result covers vanilla TACO, while the winner-preservation theorem assumes a sufficiently small approximation error. These results do not establish that the practical FP8 implementation satisfies those conditions.
2, In Table 3, TACO trails Adafactor by 10.5 accuracy points on BoolQ, 13.9 F1A points on MultiRC, and 6.0 F1 points on SQuAD. These are meaningful differences. The claim of reducing memory without sacrificing accuracy should be narrowed, and comparisons at matched accuracy would better show the practical benefit.
3, TACO processes and releases gradients during backpropagation, which contributes to its low peak memory. However, Adafactor already uses less persistent optimizer state: 0.013 GB versus TACO’s 0.159 GB in Table 4. A comparison using the same gradient-release strategy would clarify how much of the peak-memory advantage comes from TACO itself.
Questions for authors
-
Could the authors reconcile the repeated accuracy results? For OPT-1.3B, Table 5 reports 63.7 on BoolQ and 69.6 on RTE, while the 2K-step results in Table 6 are 68.6 and 72.6. Qwen3-32B’s SST-2 accuracy is also 94.1 in Table 5 but 93.1 in Figure 6. Were these experiments run with different seeds, data, or checkpoint-selection settings? Please state the differences or correct the entries.
-
How are the gradient-history values scaled before FP8 storage? Under standard nearest rounding, unscaled E4M3 values below approximately 0.0009766 become zero. With an initially empty history and an EMA coefficient of 0.95, a gradient magnitude below approximately 0.01953 would therefore disappear when its contribution is directly cast to FP8. Does the implementation use scaling, and how often does quantization change the selected coordinate or its sign?
-
How does the implementation prevent small updates from disappearing in BF16? For a square matrix, the default learning rate gives a selected-coordinate update of 0.00003. Around a BF16 weight magnitude of 0.02, this update can round away entirely under nearest rounding. Does TACO use stochastic rounding, a correction buffer, or another mechanism? Please report the fraction of selected updates that actually change the stored weights.
-
How much of the peak-memory reduction comes from sparse history, and how much comes from releasing gradients during backpropagation? Table 4 shows that Adafactor already has less persistent optimizer state than TACO: 0.013 GB versus 0.159 GB. Could the authors compare them using the same gradient-release strategy, checkpointing, and precision settings? This would make the contribution of the optimizer itself clearer.
-
How was the 254.8 GB Adam memory value in Figure 1 obtained? The caption describes an experiment on an H100 80 GB GPU, while the hardware section reports approximately 85.5 decimal GB of device capacity. Was the Adam value estimated, measured across multiple GPUs, or obtained under another configuration? Please distinguish measured values from estimates and use consistent capacity units in the figure.
8 comments
Official Comment 2
W3: Contribution of gradient release to peak memory
Answer: We note that Adafactor and AdamW8bit do not natively support the autograd hook. We non-trivially implemented the exact same post accumulate autograd hook gradient-release mechanism for Adafactor and AdamW8bit (retaining their upstream numerical update routines) and compared them alongside AdaLOMO and GaLore under identical OPT-13B settings.
The following table isolates the peak memory impact of backward gradient release from persistent optimizer state.
| Optimizer | State (GB) | Peak (GB) | Runtime (min) | Throughput (tokens/s) | DROP F1 |
|---|---|---|---|---|---|
| TACO | 0.1590 | 30.99 ± 0.06 | 24.07 ± 0.05 | 1776.5 ± 3.0 | 32.06 ± 0.51 |
| TACO + stochastic rounding | 0.1590 | 31.01 ± 0.03 | 26.54 ± 0.08 | 1610.90 ± 4.66 | 31.61 ± 0.71 |
| TACO-FP16 | 0.1886 | 31.02 ± 0.06 | 24.16 ± 0.23 | 1769.8 ± 17.2 | 38.92 ± 0.52 |
| TACO-FP16 + stochastic rounding | 0.1886 | 31.03 ± 0.03 | 26.59 ± 0.06 | 1607.87 ± 4.49 | 38.38 ± 3.18 |
| AdaLOMO | 0.0257 | 30.91 ± 0.06 | 42.37 ± 0.61 | 1009.2 ± 15.3 | 42.55 ± 2.60 |
| GaLore | 2.6551 | 33.85 ± 0.29 | 132.47 ± 0.51 | 322.8 ± 1.3 | 34.92 ± 0.91 |
| Adafactor | 0.0128 | 54.23 ± 0.04 | 20.54 ± 0.11 | 2081.5 ± 9.8 | 37.42 ± 2.77 |
| Adafactor + autograd hook | 0.0128 | 30.84 ± 0.06 | 22.17 ± 0.26 | 1929.0 ± 21.1 | 36.95 ± 2.14 |
| AdamW8bit | - | OOM | OOM | OOM | OOM |
| AdamW8bit + autograd hook | 27.7076 | 58.61 ± 0.05 | 26.52 ± 1.12 | 1614.1 ± 65.5 | 34.44 ± 0.73 |
With the same gradient release strategy, Adafactor reaches peak memory comparable to TACO-FP16: 30.84 GB for Adafactor with autograd hook and 31.02 GB for TACO-FP16. Their optimizer states are 0.0128 GB and 0.1886 GB, respectively. AdamW8bit with autograd hook still requires 27.7076 GB of optimizer state and 58.61 GB peak memory.
Although peak memory of Adafactor + autograd hook versus TACO-FP16 is 30.84 GB versus 31.02 GB, which differs by only about 0.6%, there are algorithmic limitations of Adafactor compared to TACO: (1) TACO retains sparse directional history at low memory cost (0.1886 GB), whereas standard Adafactor retains factored magnitude history and discards first-moment momentum; (2) Adafactor approximates the matrix and can be prone to distortion and training instability, whereas TACO avoids this particular approximation error because it does not reconstruct and invert a second-moment matrix; (3) TACO’s Top-1 direction changes at most entries per matrix step, so the update count is , compared with for a dense Adafactor or AdaLOMO update (excluding auxiliary parameters and any dense weight-decay operation), which we postulate makes TACO effective in distributed training over multiple machines; and (4) TACO-FP16 maintains uniform learning-rate stability (1e-5 to 3e-5) across tasks and scales, whereas Adafactor + hook varies by 10x and AdamW8bit + hook spans 100x. We have added this to the revised version of the paper.
Q1: Repeated accuracy results
Answer: The repeated entries were produced by experiments run on different H100 systems. Due to ICLR rebuttal period and computational resource limitation, these set of experiments are still runing. We will report these results as soon as they are available.
Q2: FP8 quantization on gradient history
Answer: The submitted implementation stores the gradient history directly in unscaled E4M3.
In the following experiment, we measure (i) the fraction of Top-1 selections whose stored history value is nonzero, and (ii) the fraction of effective update that actually change the stored BF16 parameter.
| Task | E4M3 Effective Top-1 Selection (%) | E4M3 Effective weights update (%) | FP16 Effective Top-1 Selection (%) | FP16 Effective weights update (%) |
|---|---|---|---|---|
| SST-2 | 55.06 | 21.98 | 93.25 | 34.75 |
| BoolQ | 64.98 | 23.18 | 94.85 | 37.83 |
| MultiRC | 47.79 | 19.28 | 91.57 | 40.90 |
FP16 substantially increases both quantities across SST-2, BoolQ, and MultiRC while adding only 29.7 MB of optimizer state. We evaluate FP16 history as a higher precision alternative.
Official Comment 3
Q3: BF16 effective parameter updates
Answer: The submitted implementation does not use stochastic rounding or a correction buffer for the BF16 parameter update.
In the next experiment, we directly measure (i) the fraction of Top-1 selections whose stored history value is nonzero, and (ii) the fraction of effective update that actually change the stored BF16 parameter.
| Task | E4M3 Effective Top-1 Selection (%) | E4M3 Effective weights update (%) | FP16 Effective Top-1 Selection (%) | FP16 Effective weights update (%) |
|---|---|---|---|---|
| SST-2 | 55.06 | 21.98 | 93.25 | 34.75 |
| BoolQ | 64.98 | 23.18 | 94.85 | 37.83 |
| MultiRC | 47.79 | 19.28 | 91.57 | 40.90 |
We additionally report the same diagnostic for representative transformer layers 0, 12, and 23 and for each major projection type. Each cell reports nonzero Top-1 selection (%) / effective BF16 weight update (%).
| Layer | Projection type | SST-2 E4M3 | SST-2 FP16 | BoolQ E4M3 | BoolQ FP16 | MultiRC E4M3 | MultiRC FP16 |
|---|---|---|---|---|---|---|---|
| 0 | Query projection | 90.37 / 32.33 | 100.00 / 42.26 | 98.52 / 21.76 | 100.00 / 37.12 | 89.28 / 21.80 | 100.00 / 37.00 |
| 0 | Key projection | 84.48 / 28.69 | 100.00 / 41.70 | 99.33 / 23.74 | 100.00 / 38.57 | 99.54 / 26.49 | 100.00 / 38.82 |
| 0 | Value projection | 99.97 / 74.47 | 100.00 / 78.79 | 100.00 / 62.26 | 100.00 / 79.43 | 100.00 / 70.46 | 100.00 / 80.22 |
| 0 | Attention output | 97.29 / 84.87 | 100.00 / 94.58 | 99.77 / 71.29 | 100.00 / 94.78 | 95.98 / 54.87 | 100.00 / 94.93 |
| 0 | FFN expansion | 99.64 / 84.43 | 100.00 / 90.36 | 99.99 / 73.05 | 100.00 / 94.93 | 99.95 / 78.75 | 100.00 / 94.64 |
| 0 | FFN contraction | 47.88 / 3.58 | 98.21 / 9.92 | 34.49 / 2.66 | 98.69 / 10.25 | 9.99 / 0.74 | 98.71 / 10.29 |
| 12 | Query projection | 75.93 / 19.20 | 100.00 / 30.26 | 53.46 / 17.52 | 100.00 / 34.34 | 52.56 / 20.31 | 100.00 / 55.76 |
| 12 | Key projection | 69.08 / 17.08 | 100.00 / 37.20 | 91.12 / 26.46 | 100.00 / 40.00 | 89.41 / 23.40 | 100.00 / 50.58 |
| 12 | Value projection | 98.01 / 38.11 | 100.00 / 47.01 | 99.40 / 25.73 | 100.00 / 51.63 | 98.33 / 23.57 | 100.00 / 50.33 |
| 12 | Attention output | 61.27 / 26.65 | 100.00 / 52.42 | 84.08 / 43.53 | 100.00 / 64.54 | 69.62 / 29.40 | 100.00 / 76.67 |
| 12 | FFN expansion | 99.90 / 82.10 | 100.00 / 89.93 | 99.90 / 52.98 | 100.00 / 89.98 | 99.90 / 46.64 | 100.00 / 90.98 |
| 12 | FFN contraction | 42.32 / 6.88 | 99.52 / 26.08 | 40.82 / 15.47 | 99.86 / 41.83 | 24.76 / 7.27 | 99.87 / 52.19 |
| 23 | Query projection | 25.61 / 5.28 | 100.00 / 26.22 | 98.46 / 17.85 | 100.00 / 23.34 | 21.34 / 3.54 | 100.00 / 24.38 |
| 23 | Key projection | 99.78 / 3.84 | 100.00 / 4.15 | 100.00 / 5.61 | 100.00 / 6.31 | 99.71 / 8.90 | 100.00 / 7.66 |
| 23 | Value projection | 3.52 / 1.30 | 100.00 / 34.38 | 99.54 / 24.02 | 100.00 / 33.87 | 18.90 / 4.93 | 100.00 / 32.29 |
| 23 | Attention output | 1.75 / 0.74 | 100.00 / 28.31 | 68.64 / 12.10 | 100.00 / 28.39 | 5.11 / 1.43 | 100.00 / 29.09 |
| 23 | FFN expansion | 99.97 / 51.52 | 100.00 / 82.76 | 99.99 / 72.46 | 100.00 / 80.19 | 99.83 / 73.24 | 100.00 / 79.85 |
| 23 | FFN contraction | 4.84 / 0.23 | 56.71 / 3.46 | 15.25 / 0.74 | 66.66 / 3.91 | 2.95 / 0.23 | 44.53 / 2.80 |
For E4M3 history, the fractions of selected updates that change the BF16 weight are 21.98% on SST-2, 23.18% on BoolQ, and 19.28% on MultiRC. With FP16 history, they increase to 34.75%, 37.83%, and 40.90%, respectively
Official Comment
We thank the reviewer for their overall positive assessment of our paper. Below, we answer the questions/concerns the reviewer had.
W1.1: Retained history size k=16
Answer: The retained history size k is a tunable implementation parameter. The principle is to choose the smallest k so that TACO's Top-1 selector can identifying the dominant coordinate in each column. In the submitted OPT-1.3B ablation, k=16 provides a favorable memory and accuracy operating point. Increasing k further gives limited additional downstream improvement while increasing optimizer state. We therefore use k=16 throughout our experiments.
W1.2: Relation between the winner preservation condition and practical TACO
Answer: Theorem 5 gives a sufficient condition for preserving the dominant entry under compressed history To assess winner preservation in practice, we measure the agreement between the winner selected from dense history and the winner selected from the k=16 compressed history.
Empirically, our E4M3 vs. FP16 precision diagnostics directly isolate the impact of history approximation error ϵt,j on OPT-13B: eliminating E4M3 underflow via FP16 history (+29.7 MB state) increases nonzero Top-1 selections from 47.8% - 65.0% to 91.6% - 94.9% and recovers downstream accuracy across seven of eight tasks.
W2: Accuracy compared with Adafactor
Answer: To clarify the empirical gap, we reran the OPT-13B experiments with three seeds and additionally evaluated TACO with FP16 gradient history. The history precision is an implementation choice and does not change the TACO update rule.
The following experiment reports the new 2000 steps OPT-13B downstream results, including the tasks highlighted by the reviewer, and directly compares TACO, TACO-FP16, and the baselines under the same training protocol.
TACO-FP16 improves the TACO-FP8 configuration on seven out of eight tasks while adding only 29.7 MB of optimizer state. On the tasks highlighted by the reviewer, BoolQ improves from 74.45 to 80.93, MultiRC from 57.32 to 68.74, and SQuAD from 83.77 to 88.48; DROP improves from 32.06 to 38.92. Thus, increasing the history precision substantially reduces the performance gap while retaining TACO's low memory behavior.