acceptodds
Under review as a conference paper at ICLR 2027

TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

Abstract

Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state. The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates. Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models. To reduce optimizer memory without sacrificing accuracy or computational efficiency in LLM fine-tuning, we propose **T**ernary **A**bsolute-max **C**olumn-wise **O**ne-sparse optimizer, or **TACO**, which follows Muon's operator-norm steepest-descent view but takes the geometric route further. TACO computes the exact steepest-descent direction under a dimension-normalized operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices. This retains first-order gradients while making optimizer state memory nearly negligible. Our practical TACO optimizer maintains only a small set of low precision gradient components per column, reducing persistent optimizer state by relative to AdamW8bit (from GB to GB) and peak training memory by (from GB to GB) on OPT-13B, while achieving comparable accuracy and runtime. TACO further enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100 GPU across multiple model families and tasks.

Primary area:
optimization
Keywords:
LLMs Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZgU1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an optimizer, coined TACO, that reduces memory usage for LLM fine-tuning by keeping a sparse gradient history instead of two dense moment matrices.

It retains 16 history entries per column and updates only the coordinate with the largest absolute history value, using its sign to determine the update direction.

The authors derive this update from a normalized operator-norm constraint and provide theoretical analysis under simplified assumptions.

The experiments show substantial peak-memory savings and enable fine-tuning models up to 32B parameters on one H100, although accuracy is lower than several baselines on some tasks.

Critical strengths

1, The authors derive the column-wise Top-1 update as the exact steepest-descent direction under their chosen operator-norm constraint. This gives a clear explanation for both the coordinate selection and the matrix-dimension scaling.

2, Since the update only needs the dominant coordinate in each column, TACO stores a small set of candidates in FP8 instead of maintaining a dense history. The ablations help explain the choice of 16 retained entries and its memory–accuracy trade-off.

3, On OPT-13B, TACO reduces peak memory from 80.6 GB to 27.5 GB compared with AdamW8bit. It also enables fine-tuning models up to 32B parameters on one H100, with experiments covering several model families.

Critical weaknesses

1, The choice of 16 retained entries is supported by an ablation on OPT-1.3B, but the paper does not measure how often this compressed history preserves the winner selected by dense history. The convergence result covers vanilla TACO, while the winner-preservation theorem assumes a sufficiently small approximation error. These results do not establish that the practical FP8 implementation satisfies those conditions.

2, In Table 3, TACO trails Adafactor by 10.5 accuracy points on BoolQ, 13.9 F1A points on MultiRC, and 6.0 F1 points on SQuAD. These are meaningful differences. The claim of reducing memory without sacrificing accuracy should be narrowed, and comparisons at matched accuracy would better show the practical benefit.

3, TACO processes and releases gradients during backpropagation, which contributes to its low peak memory. However, Adafactor already uses less persistent optimizer state: 0.013 GB versus TACO’s 0.159 GB in Table 4. A comparison using the same gradient-release strategy would clarify how much of the peak-memory advantage comes from TACO itself.

Questions for authors
  1. Could the authors reconcile the repeated accuracy results? For OPT-1.3B, Table 5 reports 63.7 on BoolQ and 69.6 on RTE, while the 2K-step results in Table 6 are 68.6 and 72.6. Qwen3-32B’s SST-2 accuracy is also 94.1 in Table 5 but 93.1 in Figure 6. Were these experiments run with different seeds, data, or checkpoint-selection settings? Please state the differences or correct the entries.

  2. How are the gradient-history values scaled before FP8 storage? Under standard nearest rounding, unscaled E4M3 values below approximately 0.0009766 become zero. With an initially empty history and an EMA coefficient of 0.95, a gradient magnitude below approximately 0.01953 would therefore disappear when its contribution is directly cast to FP8. Does the implementation use scaling, and how often does quantization change the selected coordinate or its sign?

  3. How does the implementation prevent small updates from disappearing in BF16? For a square matrix, the default learning rate gives a selected-coordinate update of 0.00003. Around a BF16 weight magnitude of 0.02, this update can round away entirely under nearest rounding. Does TACO use stochastic rounding, a correction buffer, or another mechanism? Please report the fraction of selected updates that actually change the stored weights.

  4. How much of the peak-memory reduction comes from sparse history, and how much comes from releasing gradients during backpropagation? Table 4 shows that Adafactor already has less persistent optimizer state than TACO: 0.013 GB versus 0.159 GB. Could the authors compare them using the same gradient-release strategy, checkpointing, and precision settings? This would make the contribution of the optimizer itself clearer.

  5. How was the 254.8 GB Adam memory value in Figure 1 obtained? The caption describes an experiment on an H100 80 GB GPU, while the hardware section reports approximately 85.5 decimal GB of device capacity. Was the Adam value estimated, measured across multiple GPUs, or obtained under another configuration? Please distinguish measured values from estimates and use consistent capacity units in the figure.

8 comments
AuthorsOpenReview ↗

Official Comment

We thank the reviewer for their overall positive assessment of our paper. Below, we answer the questions/concerns the reviewer had.

W1.1: Retained history size k=16k=16

Answer: The retained history size kk is a tunable implementation parameter. The principle is to choose the smallest kk so that TACO's Top-1 selector can identifying the dominant coordinate in each column. In the submitted OPT-1.3B ablation, k=16k=16 provides a favorable memory and accuracy operating point. Increasing kk further gives limited additional downstream improvement while increasing optimizer state. We therefore use k=16k=16 throughout our experiments.

W1.2: Relation between the winner preservation condition and practical TACO

Answer: Theorem 5 gives a sufficient condition for preserving the dominant entry under compressed history To assess winner preservation in practice, we measure the agreement between the winner selected from dense history and the winner selected from the k=16k=16 compressed history.

Empirically, our E4M3 vs. FP16 precision diagnostics directly isolate the impact of history approximation error ϵt,j\epsilon_{t,j} on OPT-13B: eliminating E4M3 underflow via FP16 history (+29.7 MB state) increases nonzero Top-1 selections from 47.8% - 65.0% to 91.6% - 94.9% and recovers downstream accuracy across seven of eight tasks.

W2: Accuracy compared with Adafactor

Answer: To clarify the empirical gap, we reran the OPT-13B experiments with three seeds and additionally evaluated TACO with FP16 gradient history. The history precision is an implementation choice and does not change the TACO update rule.

The following experiment reports the new 2000 steps OPT-13B downstream results, including the tasks highlighted by the reviewer, and directly compares TACO, TACO-FP16, and the baselines under the same training protocol.

OptimizerSST-2RTEBoolQCOPACBMultiRCSQuADDROP
TACO94.50 ± 0.3081.11 ± 1.7874.45 ± 0.9088.00 ± 1.0080.36 ± 3.0957.32 ± 1.0283.77 ± 1.5032.06 ± 0.51
TACO-FP1695.60 ± 0.5884.60 ± 0.5580.93 ± 0.5187.67 ± 2.0888.69 ± 1.0368.74 ± 1.0688.48 ± 0.4538.92 ± 0.52
AdaLOMO96.33 ± 0.3087.24 ± 0.5585.32 ± 0.6589.67 ± 2.5294.05 ± 1.0375.60 ± 1.3489.35 ± 0.2542.55 ± 2.60
GaLore95.34 ± 0.1384.36 ± 4.1181.55 ± 0.5189.33 ± 3.2189.88 ± 4.4965.61 ± 3.1387.88 ± 0.3234.92 ± 0.91
Adafactor95.53 ± 0.6084.84 ± 1.0884.40 ± 0.4491.00 ± 1.0091.07 ± 0.0077.57 ± 1.5689.42 ± 0.4637.42 ± 2.77
Adafactor + autograd hook95.53 ± 0.6084.84 ± 1.0884.40 ± 0.4488.00 ± 1.7392.86 ± 1.7977.57 ± 1.5689.42 ± 0.4636.95 ± 2.14
AdamW8bit96.10 ± 0.3084.00 ± 0.42OOM92.67 ± 1.5376.19 ± 16.59OOM88.23 ± 0.40OOM
AdamW8bit + autograd hook96.10 ± 0.3084.00 ± 0.4263.43 ± 0.7587.00 ± 7.5573.81 ± 12.1656.90 ± 0.4088.23 ± 0.4034.44 ± 0.73

TACO-FP16 improves the TACO-FP8 configuration on seven out of eight tasks while adding only 29.7 MB of optimizer state. On the tasks highlighted by the reviewer, BoolQ improves from 74.45 to 80.93, MultiRC from 57.32 to 68.74, and SQuAD from 83.77 to 88.48; DROP improves from 32.06 to 38.92. Thus, increasing the history precision substantially reduces the performance gap while retaining TACO's low memory behavior.

AuthorsOpenReview ↗

Official Comment 2

W3: Contribution of gradient release to peak memory

Answer: We note that Adafactor and AdamW8bit do not natively support the autograd hook. We non-trivially implemented the exact same post accumulate autograd hook gradient-release mechanism for Adafactor and AdamW8bit (retaining their upstream numerical update routines) and compared them alongside AdaLOMO and GaLore under identical OPT-13B settings.

The following table isolates the peak memory impact of backward gradient release from persistent optimizer state.

OptimizerState (GB)Peak (GB)Runtime (min)Throughput (tokens/s)DROP F1
TACO0.159030.99 ± 0.0624.07 ± 0.051776.5 ± 3.032.06 ± 0.51
TACO + stochastic rounding0.159031.01 ± 0.0326.54 ± 0.081610.90 ± 4.6631.61 ± 0.71
TACO-FP160.188631.02 ± 0.0624.16 ± 0.231769.8 ± 17.238.92 ± 0.52
TACO-FP16 + stochastic rounding0.188631.03 ± 0.0326.59 ± 0.061607.87 ± 4.4938.38 ± 3.18
AdaLOMO0.025730.91 ± 0.0642.37 ± 0.611009.2 ± 15.342.55 ± 2.60
GaLore2.655133.85 ± 0.29132.47 ± 0.51322.8 ± 1.334.92 ± 0.91
Adafactor0.012854.23 ± 0.0420.54 ± 0.112081.5 ± 9.837.42 ± 2.77
Adafactor + autograd hook0.012830.84 ± 0.0622.17 ± 0.261929.0 ± 21.136.95 ± 2.14
AdamW8bit-OOMOOMOOMOOM
AdamW8bit + autograd hook27.707658.61 ± 0.0526.52 ± 1.121614.1 ± 65.534.44 ± 0.73

With the same gradient release strategy, Adafactor reaches peak memory comparable to TACO-FP16: 30.84 GB for Adafactor with autograd hook and 31.02 GB for TACO-FP16. Their optimizer states are 0.0128 GB and 0.1886 GB, respectively. AdamW8bit with autograd hook still requires 27.7076 GB of optimizer state and 58.61 GB peak memory.

Although peak memory of Adafactor + autograd hook versus TACO-FP16 is 30.84 GB versus 31.02 GB, which differs by only about 0.6%, there are algorithmic limitations of Adafactor compared to TACO: (1) TACO retains sparse directional history at low memory cost (0.1886 GB), whereas standard Adafactor retains factored magnitude history and discards first-moment momentum; (2) Adafactor approximates the matrix VV and can be prone to distortion and training instability, whereas TACO avoids this particular approximation error because it does not reconstruct and invert a second-moment matrix; (3) TACO’s Top-1 direction changes at most nn entries per matrix step, so the update count is O(n)O(n), compared with O(mn)O(mn) for a dense Adafactor or AdaLOMO update (excluding auxiliary parameters and any dense weight-decay operation), which we postulate makes TACO effective in distributed training over multiple machines; and (4) TACO-FP16 maintains uniform learning-rate stability (1e-5 to 3e-5) across tasks and scales, whereas Adafactor + hook varies by 10x and AdamW8bit + hook spans 100x. We have added this to the revised version of the paper.

Q1: Repeated accuracy results

Answer: The repeated entries were produced by experiments run on different H100 systems. Due to ICLR rebuttal period and computational resource limitation, these set of experiments are still runing. We will report these results as soon as they are available.

Q2: FP8 quantization on gradient history

Answer: The submitted implementation stores the gradient history directly in unscaled E4M3.

In the following experiment, we measure (i) the fraction of Top-1 selections whose stored history value is nonzero, and (ii) the fraction of effective update that actually change the stored BF16 parameter.

TaskE4M3 Effective Top-1 Selection (%)E4M3 Effective weights update (%)FP16 Effective Top-1 Selection (%)FP16 Effective weights update (%)
SST-255.0621.9893.2534.75
BoolQ64.9823.1894.8537.83
MultiRC47.7919.2891.5740.90

FP16 substantially increases both quantities across SST-2, BoolQ, and MultiRC while adding only 29.7 MB of optimizer state. We evaluate FP16 history as a higher precision alternative.

AuthorsOpenReview ↗

Official Comment 3

Q3: BF16 effective parameter updates

Answer: The submitted implementation does not use stochastic rounding or a correction buffer for the BF16 parameter update.

In the next experiment, we directly measure (i) the fraction of Top-1 selections whose stored history value is nonzero, and (ii) the fraction of effective update that actually change the stored BF16 parameter.

TaskE4M3 Effective Top-1 Selection (%)E4M3 Effective weights update (%)FP16 Effective Top-1 Selection (%)FP16 Effective weights update (%)
SST-255.0621.9893.2534.75
BoolQ64.9823.1894.8537.83
MultiRC47.7919.2891.5740.90

We additionally report the same diagnostic for representative transformer layers 0, 12, and 23 and for each major projection type. Each cell reports nonzero Top-1 selection (%) / effective BF16 weight update (%).

LayerProjection typeSST-2 E4M3SST-2 FP16BoolQ E4M3BoolQ FP16MultiRC E4M3MultiRC FP16
0Query projection90.37 / 32.33100.00 / 42.2698.52 / 21.76100.00 / 37.1289.28 / 21.80100.00 / 37.00
0Key projection84.48 / 28.69100.00 / 41.7099.33 / 23.74100.00 / 38.5799.54 / 26.49100.00 / 38.82
0Value projection99.97 / 74.47100.00 / 78.79100.00 / 62.26100.00 / 79.43100.00 / 70.46100.00 / 80.22
0Attention output97.29 / 84.87100.00 / 94.5899.77 / 71.29100.00 / 94.7895.98 / 54.87100.00 / 94.93
0FFN expansion99.64 / 84.43100.00 / 90.3699.99 / 73.05100.00 / 94.9399.95 / 78.75100.00 / 94.64
0FFN contraction47.88 / 3.5898.21 / 9.9234.49 / 2.6698.69 / 10.259.99 / 0.7498.71 / 10.29
12Query projection75.93 / 19.20100.00 / 30.2653.46 / 17.52100.00 / 34.3452.56 / 20.31100.00 / 55.76
12Key projection69.08 / 17.08100.00 / 37.2091.12 / 26.46100.00 / 40.0089.41 / 23.40100.00 / 50.58
12Value projection98.01 / 38.11100.00 / 47.0199.40 / 25.73100.00 / 51.6398.33 / 23.57100.00 / 50.33
12Attention output61.27 / 26.65100.00 / 52.4284.08 / 43.53100.00 / 64.5469.62 / 29.40100.00 / 76.67
12FFN expansion99.90 / 82.10100.00 / 89.9399.90 / 52.98100.00 / 89.9899.90 / 46.64100.00 / 90.98
12FFN contraction42.32 / 6.8899.52 / 26.0840.82 / 15.4799.86 / 41.8324.76 / 7.2799.87 / 52.19
23Query projection25.61 / 5.28100.00 / 26.2298.46 / 17.85100.00 / 23.3421.34 / 3.54100.00 / 24.38
23Key projection99.78 / 3.84100.00 / 4.15100.00 / 5.61100.00 / 6.3199.71 / 8.90100.00 / 7.66
23Value projection3.52 / 1.30100.00 / 34.3899.54 / 24.02100.00 / 33.8718.90 / 4.93100.00 / 32.29
23Attention output1.75 / 0.74100.00 / 28.3168.64 / 12.10100.00 / 28.395.11 / 1.43100.00 / 29.09
23FFN expansion99.97 / 51.52100.00 / 82.7699.99 / 72.46100.00 / 80.1999.83 / 73.24100.00 / 79.85
23FFN contraction4.84 / 0.2356.71 / 3.4615.25 / 0.7466.66 / 3.912.95 / 0.2344.53 / 2.80

For E4M3 history, the fractions of selected updates that change the BF16 weight are 21.98% on SST-2, 23.18% on BoolQ, and 19.28% on MultiRC. With FP16 history, they increase to 34.75%, 37.83%, and 40.90%, respectively

Reviewer_3vqcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TACO, a low-memory first-order optimizer for full-parameter fine-tuning of LLMs. Starting from the dimension-normalized 1→1 operator norm, the authors prove that the linearized steepest-descent direction can take the coordinate with the largest gradient magnitude in each column and its sign, resulting in only one nonzero update per column. The practical implementation maintains a sparse exponential moving average of the k=16 heavy gradient elements in each column, represents states with low-precision values and indices, and uses autograd hooks to process and release gradients during backpropagation. The theoretical discussion covers nonconvex convergence of a version using exact gradients without history, relationships between the paths of Adam and Muon in simplified problems, and spectral-norm scaling conditional on a winner-distribution assumption.

The authors conduct experiments across multiple model families, model scales, and eight NLP tasks. For OPT-13B/SST-2, TACO has 0.159 GB of persistent state and 27.54 GB of peak GPU memory; the corresponding figures for AdamW8bit are 27.708 GB and 80.60 GB. The authors further demonstrate the feasibility of full-parameter fine-tuning of OPT-30B and Qwen3-32B on a single 80 GB H100.

Critical strengths
  1. The method has a testable geometric starting point. The normalized 1→1 operator-norm constraint in Section 3 leads to a sign update at the maximum-gradient coordinate of each column, rather than merely adding a Top-1 sparsity mask afterward. This structure directly supports an implementation storing only a small number of indices and gradient histories, and is the paper's clearest technical contribution.

  2. The large-scale memory evidence has potential engineering value. Table 4 (p.22) shows approximately 174-fold state compression and approximately 2.93-fold improvement in peak GPU memory compared with AdamW8bit; Table 5 shows single-GPU fine-tuning of 30–32B models. The authors also report throughput, training time, hyperparameters, and several ablations, so the contribution is not limited to mathematical motivation.

Critical weaknesses
  1. The peak-memory advantage has not been disentangled from general gradient-lifecycle techniques, and the quality–speed trade-off is not sufficiently favorable. TACO uses autograd hooks in Section 5 (p.7) to update immediately and release dense gradients, but results are not provided for baselines such as Adafactor and AdamW8bit under equivalent hooks/release mechanisms. In Table 4, Adafactor has only 0.013 GB of persistent state, less than TACO's 0.159 GB, but its peak memory is 52.59 GB, which is 25.05 GB more than TACO's 27.54 GB, close to the size of one BF16 dense gradient for OPT-13B. This creates a clear alternative explanation that has not been ruled out: the peak-memory gap may mainly reflect whether gradients are released promptly, rather than a unique advantage of optimizer states or update geometry. Meanwhile, TACO underperforms Adafactor on seven of eight tasks in Table 3: RTE 80.8 versus 87.3, BoolQ 71.5 versus 82.0, MultiRC F1A 61.2 versus 75.1, and SQuAD F1 82.6 versus 88.6. SST-2 throughput is also lower: 275.8 versus 349.7 tokens/s (Adafactor). Therefore, the paper has not yet established a convincing accuracy–total-memory–time Pareto advantage.

  2. The theoretical conditions are not yet sufficient to explain key properties of practical training. Theorem 1 is a nonconvex result for a history-free, exact-gradient version and does not cover stochastic gradients, Top-k EMA truncation, or FP8 quantization. Theorem 2 considers an idealized problem in which the objective depends on W only through Wz for a fixed vector z, and the compared Adam proxy does not contain all mechanisms of actual AdamW. Theorem 4's μP spectral scaling requires the winner counts per row to satisfy max_i c_i = Θ(n/m), but the main text does not report this distribution in actual layers. A fixed-input outer-product gradient G=gzᵀ may instead cause all nonzero columns to select the same row, creating an obvious boundary case for the balance assumption. The theoretical conclusions are valuable as conditional mechanisms, but do not support unconditional claims of Adam-like behavior or width-stable generalization.

Questions for authors
  1. Could Adafactor and AdamW8bit (preferably also GaLore) use the same optimizer-in-backward/hook and immediate gradient-release strategy as TACO, with peak memory, memory breakdown, accuracy, and throughput remeasured under the same batch, activation checkpointing, training steps, and hardware? This is the most important experiment for deciding whether TACO's unique contribution holds.

  2. Please provide accuracy–memory–throughput Pareto curves after retuning under the final training configurations and matching wall-clock time or number of training tokens, covering at least BoolQ, RTE, MultiRC, and SQuAD, with per-seed variance. The appendix's learning-rate search uses batch size 1 and 500 steps, whereas the main experiments default to batch size 4 and 2000 steps. Are the baselines' hyperparameters still appropriate under the main training conditions?

  3. Could the authors report the distribution of winner counts max_i c_i across actual LLM layers as a function of width and training progress, to test the premise of Theorem 4? If certain layers have highly concentrated winners, can the observed learning-rate transfer be explained by other mechanisms?

7 comments
AuthorsOpenReview ↗

Official Comment 1

We thank the reviewer for their insights. We provide responses to their review below.

W1: Peak memory and the quality-speed tradeoff

Answer: We note that Adafactor and AdamW8bit do not natively support the autograd hook. During rebuttal, we non-trivially implemented post-accumulate autograd hook wrappers for Adafactor and AdamW8bit that update parameters and release gradients during backpropagation while retaining their upstream numerical update routines. We reran the OPT-13B experiments under a common 2000-step configuration with three seeds alongside AdaLOMO and GaLore, which already support autograd hook.

The following table reports downstream task performance across all eight tasks.

OptimizerSST-2RTEBoolQCOPACBMultiRCSQuADDROP
TACO94.50 ± 0.3081.11 ± 1.7874.45 ± 0.9088.00 ± 1.0080.36 ± 3.0957.32 ± 1.0283.77 ± 1.5032.06 ± 0.51
TACO-FP1695.60 ± 0.5884.60 ± 0.5580.93 ± 0.5187.67 ± 2.0888.69 ± 1.0368.74 ± 1.0688.48 ± 0.4538.92 ± 0.52
AdaLOMO96.33 ± 0.3087.24 ± 0.5585.32 ± 0.6589.67 ± 2.5294.05 ± 1.0375.60 ± 1.3489.35 ± 0.2542.55 ± 2.60
GaLore95.34 ± 0.1384.36 ± 4.1181.55 ± 0.5189.33 ± 3.2189.88 ± 4.4965.61 ± 3.1387.88 ± 0.3234.92 ± 0.91
Adafactor95.53 ± 0.6084.84 ± 1.0884.40 ± 0.4491.00 ± 1.0091.07 ± 0.0077.57 ± 1.5689.42 ± 0.4637.42 ± 2.77
Adafactor + autograd hook95.53 ± 0.6084.84 ± 1.0884.40 ± 0.4488.00 ± 1.7392.86 ± 1.7977.57 ± 1.5689.42 ± 0.4636.95 ± 2.14
AdamW8bit96.10 ± 0.3084.00 ± 0.42OOM92.67 ± 1.5376.19 ± 16.59OOM88.23 ± 0.40OOM
AdamW8bit + autograd hook96.10 ± 0.3084.00 ± 0.4263.43 ± 0.7587.00 ± 7.5573.81 ± 12.1656.90 ± 0.4088.23 ± 0.4034.44 ± 0.73

We also report the corresponding efficiency comparison on DROP dataset.

OptimizerState (GB)Peak (GB)Runtime (min)Throughput (tokens/s)DROP F1
TACO0.159030.99 ± 0.0624.07 ± 0.051776.5 ± 3.032.06 ± 0.51
TACO-FP160.188631.02 ± 0.0624.16 ± 0.231769.8 ± 17.238.92 ± 0.52
AdaLOMO0.025730.91 ± 0.0642.37 ± 0.611009.2 ± 15.342.55 ± 2.60
GaLore2.655133.85 ± 0.29132.47 ± 0.51322.8 ± 1.334.92 ± 0.91
Adafactor0.012854.23 ± 0.0420.54 ± 0.112081.5 ± 9.837.42 ± 2.77
Adafactor + autograd hook0.012830.84 ± 0.0622.17 ± 0.261929.0 ± 21.136.95 ± 2.14
AdamW8bit-OOMOOMOOMOOM
AdamW8bit + autograd hook27.707658.61 ± 0.0526.52 ± 1.121614.1 ± 65.534.44 ± 0.73

Backward gradient release reduces Adafactor peak memory by 23.39 GB (54.23 -> 30.84 GB), placing Adafactor + hook (30.84 GB), AdaLOMO (30.91 GB), and TACO-FP16 (31.02 GB) at the same ~31 GB VRAM floor, whereas AdamW8bit + hook remains at 58.61 GB due to its 27.7076 GB persistent state.

Regarding throughput, TACO-FP16 achieves 1769.8 tokens/s, within 8% of Adafactor + hook (1929.0 tokens/s), 1.75x faster than AdaLOMO (1009.2 tokens/s), and 5.5x faster than GaLore (322.8 tokens/s) on long sequences DROP dataset.

For downstream performance, TACO-FP16 improves over the submitted TACO configuration (+11.42 F1A on MultiRC, +6.48 on BoolQ, +4.71 on SQuAD, +6.86 on DROP) while adding only 29.7 MB of optimizer state.

Although peak memory of Adafactor + autograd hook versus TACO-FP16 is 30.84 GB versus 31.02 GB, which differs by only about 0.6%, there are algorithmic limitations of Adafactor compared to TACO: (1) TACO retains sparse directional history at low memory cost (0.1886 GB), whereas standard Adafactor retains factored magnitude history and discards first-moment momentum; (2) Adafactor approximates the matrix VV and can be prone to distortion and training instability, whereas TACO avoids this particular approximation error because it does not reconstruct and invert a second-moment matrix; (3) TACO’s Top-1 direction changes at most nn entries per matrix step, so the update count is O(n)O(n), compared with O(mn)O(mn) for a dense Adafactor or AdaLOMO update (excluding auxiliary parameters and any dense weight-decay operation), which we postulate makes TACO effective in distributed training over multiple machines; and (4) TACO-FP16 maintains uniform learning rate stability (1e-5 to 3e-5) across tasks and scales, whereas Adafactor + hook varies by 10x and AdamW8bit + hook spans 100x. We have added this to the revised version of the paper.

AuthorsOpenReview ↗

Official Comment 2

W2: Scope of the theoretical analysis

Answer: We address the three theoretical points raised by the reviewer separately.

  1. Stochastic Convergence (Theorem 1): We have extended the convergence analysis from deterministic gradients to stochastic minibatch gradients and added it as Theorem 3 to the revised paper.

  2. Continuation Geometry (Theorem 2): Under Assumptions 2 and 3, the rank-one gradient structure Gt=gtz⊤G_t=g_tz^\top implies that TACO and the Adam proxy preserve the right factor sign⁡(z)\operatorname{sign}(z), whereas Muon preserves zz. Consequently, TACO and the Adam proxy share the same continuation geometry and limiting solution within this theoretical setting, without establishing unconditional equivalence to practical AdamW.

  3. Empirical Winner Concentration and μP\mu P Scaling (Theorem 4): we directly measure winner counts in actual OPT layers across model widths and throughout training.

To test the winner-balance premise in Theorem 4, we measure winner concentration in actual OPT layers. For each matrix update, let cic_i be the number of active columns selecting row ii, and let cmax=max⁡icic_{\text{max}} = \max_i c_i be the number selecting the busiest row. Each entry a[b]a[b] reports the median [95th percentile] of cmaxc_{\text{max}} over the corresponding transformer blocks and sampled training steps. For example, 430[1709]430[1709] means a median maximum winner count of 430 and a 95th percentile of 1709.

Projection typeOPT-1.3BOPT-6.7BOPT-13BOPT-30B
FFN expansion430 [1,709]953 [2,782]1,016 [3,692]1,864 [4,617]
FFN contraction282 [1,366]206 [900]78 [522]47 [328]
Attention output54 [213]53 [640]19 [111]20 [678]
Value projection226 [572]332 [1,527]421 [1,456]464 [2,445]
Query projection400 [848]289 [1,238]66 [1,033]398 [2,265]
Key projection338 [925]162 [1,601]8 [685]256 [1,990]

We also report the normalized spectral ratio ρt=∣∣Dt∣∣2/m/n\rho_t=||D_t||_2/\sqrt{m/n}, where DtD_t is the TACO direction before multiplication by the learning rate. Each entry a→ba \to b reports the median ρt\rho_t early to late training, where early and late are the first and last thirds of the 2000 steps budget. For example, 42.6$\to$41.1 means the median normalized spectral ratio changes from 42.6 early in training to 41.1 late in training.

Projection typeOPT-1.3BOPT-6.7BOPT-13BOPT-30B
FFN expansion42.6 → 41.162.3 → 61.166.2 → 62.384.0 → 87.4
FFN contraction7.4 → 9.05.6 → 9.13.8 → 5.12.4 → 4.2
Attention output7.0 → 7.64.0 → 8.93.5 → 4.92.7 → 6.0
Value projection15.8 → 14.413.2 → 21.117.1 → 22.111.6 → 25.4
Query projection20.2 → 19.89.7 → 18.85.3 → 9.59.6 → 22.5
Key projection16.4 → 19.15.8 → 14.52.4 → 2.94.2 → 18.3

The measurements show differences across projection types. Under conditional isotropy, the concentration independent total squared update magnitude suggests a possible average-case mechanism for learning rate transfer.

AuthorsOpenReview ↗

Official Comment 3

Q1: Equivalent autograd hook baselines

Answer: Yes. We note that Adafactor and AdamW8bit do not natively support the autograd hook. We non-trivially implemented post accumulate autograd hook wrappers for Adafactor and AdamW8bit while retaining their upstream numerical update routines.

The following table reports downstream task performance across all eight tasks under this controlled configuration along with GaLore and AdaLomo.

OptimizerSST-2RTEBoolQCOPACBMultiRCSQuADDROP
TACO94.50 ± 0.3081.11 ± 1.7874.45 ± 0.9088.00 ± 1.0080.36 ± 3.0957.32 ± 1.0283.77 ± 1.5032.06 ± 0.51
TACO-FP1695.60 ± 0.5884.60 ± 0.5580.93 ± 0.5187.67 ± 2.0888.69 ± 1.0368.74 ± 1.0688.48 ± 0.4538.92 ± 0.52
AdaLOMO96.33 ± 0.3087.24 ± 0.5585.32 ± 0.6589.67 ± 2.5294.05 ± 1.0375.60 ± 1.3489.35 ± 0.2542.55 ± 2.60
GaLore95.34 ± 0.1384.36 ± 4.1181.55 ± 0.5189.33 ± 3.2189.88 ± 4.4965.61 ± 3.1387.88 ± 0.3234.92 ± 0.91
Adafactor95.53 ± 0.6084.84 ± 1.0884.40 ± 0.4491.00 ± 1.0091.07 ± 0.0077.57 ± 1.5689.42 ± 0.4637.42 ± 2.77
Adafactor + autograd hook95.53 ± 0.6084.84 ± 1.0884.40 ± 0.4488.00 ± 1.7392.86 ± 1.7977.57 ± 1.5689.42 ± 0.4636.95 ± 2.14
AdamW8bit96.10 ± 0.3084.00 ± 0.42OOM92.67 ± 1.5376.19 ± 16.59OOM88.23 ± 0.40OOM
AdamW8bit + autograd hook96.10 ± 0.3084.00 ± 0.4263.43 ± 0.7587.00 ± 7.5573.81 ± 12.1656.90 ± 0.4088.23 ± 0.4034.44 ± 0.73

We also report the corresponding efficiency comparison on DROP dataset.

OptimizerState (GB)Peak (GB)Runtime (min)Throughput (tokens/s)DROP F1
TACO0.159030.99 ± 0.0624.07 ± 0.051776.5 ± 3.032.06 ± 0.51
TACO-FP160.188631.02 ± 0.0624.16 ± 0.231769.8 ± 17.238.92 ± 0.52
AdaLOMO0.025730.91 ± 0.0642.37 ± 0.611009.2 ± 15.342.55 ± 2.60
GaLore2.655133.85 ± 0.29132.47 ± 0.51322.8 ± 1.334.92 ± 0.91
Adafactor0.012854.23 ± 0.0420.54 ± 0.112081.5 ± 9.837.42 ± 2.77
Adafactor + autograd hook0.012830.84 ± 0.0622.17 ± 0.261929.0 ± 21.136.95 ± 2.14
AdamW8bit-OOMOOMOOMOOM
AdamW8bit + autograd hook27.707658.61 ± 0.0526.52 ± 1.121614.1 ± 65.534.44 ± 0.73

Backward gradient release reduces Adafactor peak memory by 23.39 GB (54.23$\to$30.84 GB), placing Adafactor + hook (30.84 GB), AdaLOMO (30.91 GB), and TACO-FP16 (31.02 GB) at the same ~31 GB VRAM floor, whereas AdamW8bit + hook remains at 58.61 GB due to its 27.7076 GB persistent state.

For downstream performance, TACO-FP16 improves over the submitted TACO configuration (+11.42 F1A on MultiRC, +6.48 on BoolQ, +4.71 on SQuAD, +6.86 on DROP) while adding only 29.7 MB of optimizer state.

TACO-FP16 offers three distinct structural advantages over existing low-state baselines: (1) 1.75x higher throughput than AdaLOMO (1769.8 vs. 1009.2 tokens/s, cutting DROP runtime from 42.37 to 24.16 min) at near peak memory; (2) uniform learning-rate stability across tasks and scales (1e-5 to 3e-5, whereas Adafactor + hook varies by 10x and AdamW8bit + hook spans 100x); and (3) intrinsic column-wise 1-sparse ternary updates, whereas Adafactor and AdaLOMO apply dense updates across all parameters.

Although peak memory of Adafactor + autograd hook versus TACO-FP16 is 30.84 GB versus 31.02 GB, which differs by only about 0.6%, there are algorithmic limitations of Adafactor compared to TACO: (1) TACO retains sparse directional history at low memory cost (0.1886 GB), whereas standard Adafactor retains factored magnitude history and discards first-moment momentum; (2) Adafactor approximates the matrix VV and can be prone to distortion and training instability, whereas TACO avoids this particular approximation error because it does not reconstruct and invert a second-moment matrix; (3) TACO’s Top-1 direction changes at most nn entries per matrix step, so the update count is O(n)O(n), compared with O(mn)O(mn) for a dense Adafactor or AdaLOMO update (excluding auxiliary parameters and any dense weight-decay operation), which we postulate makes TACO effective in distributed training over multiple machines; and (4) TACO-FP16 maintains uniform learning-rate stability (1e-5 to 3e-5) across tasks and scales, whereas Adafactor + hook varies by 10x and AdamW8bit + hook spans 100x. We have added this to the revised version of the paper.

Reviewer_bkwwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TACO is an optimizer for full-parameter LLM fine-tuning, derived as the exact steepest-descent direction under a dimension-normalized 1→1 operator norm: in each column of a weight matrix, only the entry with the largest-magnitude gradient (or gradient history) is updated, by its sign scaled by m/n. The practical version keeps an EMA gradient history truncated to the top 16 entries per column (FP8 values, int32 indices), consumes and frees each dense matrix gradient through autograd hooks, and updates scalar and vector parameters with a small auxiliary AdamW. The theory gives an O(ε^-2) rate for deterministic, history-free TACO, a relation to an Adam proxy on a fixed-input objective, and μP spectral scaling under balanced winner counts. Experiments compare TACO with six first-order optimizers on OPT-13B over eight tasks and run it on five model families up to 32B, reporting 0.159 GB of optimizer state and 27.54 GB of peak memory on OPT-13B.

Critical strengths
  1. A principled source of sparsity. The ternary one-sparse update is the exact solution of the local steepest-descent problem (Eqs. 3 and 4), not a heuristic, and Theorem 5 gives a useful design principle for low-state optimizers: a history only needs to preserve each column's winning entry, which a margin condition guarantees.

  2. A large, verifiable cut in optimizer state, and single-GPU fine-tuning at 30B scale. The reported state, 0.159 GB on OPT-13B (Table 4) and 38.3 MB on OPT-1.3B (Table 8), follows from the 5kn-byte formula (p. 6) plus the small auxiliary AdamW state; I recomputed both from the OPT configurations. That is 174x less than AdamW8bit. With the default configuration, fine-tuning OPT-30B and Qwen3-32B fits on one 80 GB H100 for every reported task (Table 5, Fig. 1).

Critical weaknesses
  1. The memory comparison does not isolate what TACO adds over existing low-state optimizers. The state reduction relative to AdamW8bit is real: of the 53.1 GB peak gap in Table 4, 27.5 GB is the difference in optimizer state. The remaining gap, and nearly all of the gap to the low-state baselines, is consistent with gradient lifetime rather than the update rule: peak memory minus weights (about 25.7 GB) minus state is 1.7 GB for TACO and 1.8 GB for FlashAdamW, both of which release gradients (pp. 7 and 20), but 26.9 to 28.3 GB for the other five baselines, about one BF16 gradient copy. Adafactor already keeps less state than TACO (0.013 vs 0.159 GB) yet peaks at 52.59 vs 27.54 GB. The paper calls autograd hooks a standard technique (p. 7), and LOMO (Lv et al., ACL 2024) and AdaLomo (Lv et al., Findings of ACL 2024) also consume gradients during backward in full fine-tuning. Without a strong low-state baseline run with compatible gradient release, Adafactor in particular, the 2.9x peak reduction and the Pareto position in Fig. 1 do not show TACO's own practical advantage.

  2. Accuracy trails the strongest low-state baseline on most tasks, and the tradeoff is not quantified. Under the 2K-step OPT-13B protocol (Table 3), TACO is below Adafactor on seven of eight tasks, by 6.5 (RTE), 10.5 (BoolQ), 13.9 (MultiRC), 6.0 (SQuAD) and 6.9 (DROP) points, and below the best baseline on all eight. It does compare well with some baselines: it is ahead of AdamW8bit on BoolQ, COPA and CB and completes MultiRC and DROP, where AdamW8bit runs out of memory. Table 3 reports no seed variability for these runs, so the gaps cannot be judged against seed noise. Longer training helps TACO on OPT-1.3B (BoolQ 68.6 to 75.1 from 2K to 16K steps, Table 6), but no baseline is reported at longer budgets or matched wall-clock, so it is unclear whether the OPT-13B gaps close at comparable cost. The abstract's "comparable accuracy" therefore depends on the comparator, and a matched-resource comparison is needed to judge the tradeoff.

Questions for authors
  1. Could you compare TACO with Adafactor using compatible gradient release that preserves Adafactor's update semantics, under the same BF16 weights and checkpointing, and report peak memory, throughput and task quality? Positioning against LOMO and AdaLomo, which also consume gradients during backward, would help. A clear TACO advantage in this matched setting would raise my score.

  2. On a few OPT-13B tasks with large gaps (for example BoolQ and MultiRC), how do TACO and Adafactor compare at matched training time, with per-seed scores or uncertainty for those runs?

  3. How often do selected updates actually change the stored weights? The weights are BF16 (Table 1) and there is no room for an FP32 master copy at 27.54 GB; by my calculation, with round-to-nearest a 3e-5 step leaves a BF16 weight unchanged whenever |w| > 2^-7 (about 0.0078). If the history is stored in E4M3 without a scale, entries of magnitude at most 2^-10 (about 0.001) round to zero, so a selected winner can yield a zero update. Please state whether stochastic rounding, compensation or scaled FP8 is used, and report per layer the fraction of selected coordinates with a nonzero stored sign and the fraction whose stored weight changes.

5 comments
AuthorsOpenReview ↗

Official Comment

We thank the reviewer for their review and respond below.

W1: Gradient release and peak memory

Answer: We note that Adafactor and AdamW8bit do not natively support the autograd hook. During rebuttal, we non-trivially implemented these post-accumulate autograd hook wrappers for Adafactor and AdamW8bit that update parameters and release gradients during backpropagation while retaining their upstream numerical update routines. We reran the OPT-13B experiments under a common 2000-step configuration with three seeds alongside AdaLOMO and GaLore, which already support autograd hooks.

The following experiment compares optimizer state, peak memory, runtime, throughput, and DROP performance under the same OPT-13B configuration. For Adafactor, the paired comparison measures the peak memory reduction from the hook integration, with unchanged optimizer state storage.

OptimizerState (GB)Peak (GB)Runtime (min)Throughput (tokens/s)DROP F1
TACO0.159030.99 ± 0.0624.07 ± 0.051776.5 ± 3.032.06 ± 0.51
TACO + stochastic rounding0.159031.01 ± 0.0326.54 ± 0.081610.90 ± 4.6631.61 ± 0.71
TACO-FP160.188631.02 ± 0.0624.16 ± 0.231769.8 ± 17.238.92 ± 0.52
TACO-FP16 + stochastic rounding0.188631.03 ± 0.0326.59 ± 0.061607.87 ± 4.4938.38 ± 3.18
AdaLOMO0.025730.91 ± 0.0642.37 ± 0.611009.2 ± 15.342.55 ± 2.60
GaLore2.655133.85 ± 0.29132.47 ± 0.51322.8 ± 1.334.92 ± 0.91
Adafactor0.012854.23 ± 0.0420.54 ± 0.112081.5 ± 9.837.42 ± 2.77
Adafactor + autograd hook0.012830.84 ± 0.0622.17 ± 0.261929.0 ± 21.136.95 ± 2.14
AdamW8bit-OOMOOMOOMOOM
AdamW8bit + autograd hook27.707658.61 ± 0.0526.52 ± 1.121614.1 ± 65.534.44 ± 0.73

With autograd hook, Adafactor peak memory decreases from 54.23 GB to 30.84 GB, comparable to 31.02 GB for TACO-FP16. AdamW8bit with autograd hook still uses 58.61 GB peak memory because its optimizer state remains 27.7076 GB.

At this 31GB VRAM floor, TACO-FP16 offers three distinct structural advantages over existing low-state baselines: (1) 1.75x higher throughput than AdaLOMO (1769.8 vs. 1009.2 tokens/s, cutting DROP runtime from 42.37 to 24.16 min) at near peak memory; (2) uniform learning-rate stability across tasks and scales (1e-5 to 3e-5, whereas Adafactor + hook varies by 10x and AdamW8bit + hook spans 100x); and (3) intrinsic column-wise 1-sparse ternary updates, whereas Adafactor and AdaLOMO apply dense updates across all parameters.

Although the peak memory of Adafactor + autograd hook versus TACO-FP16 is 30.84 GB versus 31.02 GB, which differs by only about 0.6%, there are algorithmic limitations of Adafactor compared to TACO. TACO achieves competitive adaptation while retaining selected signed directional history (0.1886 GB) and avoiding a factored second-moment preconditioner VV (which requires reconstructing and inverting a rank-1 approximation and can be prone to distortion and training instability). Additionally, TACO’s Top-1 direction changes at most nn entries per matrix step. The update count is therefore O(n)O(n) vs O(mn)O(mn) for a dense Adafactor update, excluding auxiliary parameters and any dense weight decay operation. We postulate this makes TACO effective in distributed training over multiple machines. We have added this to the revised version of the paper.

As a proof of concept, we evaluated TACO’s communication volume during synchronous distributed fine-tuning on four GPUs. We fine-tuned OPT-1.3B on SST-2 for 500 optimizer steps, using a local batch size of 4 per GPU and a global batch size of 16. We compared TACO, TACO-FP16, Adafactor + autograd hook, and AdaLOMO using three training seeds.

The table reports the estimated inter-GPU data volume, averaged per GPU. The cumulative column sums estimated data sent across all four GPUs over 500 steps.

OptimizerEstimated sent per GPU per stepEstimated received per GPU per stepEstimated total sent across 4 GPUs over 500 steps
TACO, E4M3 EMA8.63 MB8.63 MB17.26 GB
TACO, FP16 EMA8.63 MB8.63 MB17.26 GB
Adafactor + hook3.95 GB3.95 GB7,894.55 GB
AdaLOMO + hook3.95 GB3.95 GB7,894.55 GB

TACO communicates sparse updates using row indices and signs, together with communication for auxiliary AdamW gradients. The dense baselines communicate uncompressed BF16 parameter changes. Under the accounting used here, TACO’s estimated sent volume was approximately 458 times smaller than that of either dense baseline. The two TACO variants had the same communication volume because their different EMA storage formats did not change the transmitted index and sign representation. These results quantify the estimated communication volume under the specified update synchronization schemes.

AuthorsOpenReview ↗

Official Comment 2

W2: Accuracy and the memory-accuracy tradeoff

Answer: We reran the OPT-13B experiments under a common 2000 steps configuration with three seeds. We also evaluate TACO-FP16, which stores the sparse gradient history in FP16 instead of E4M3.

The following experiment compares downstream performance across TACO, TACO-FP16, and the low memory baselines under the same 2000 steps OPT-13B configuration.

OptimizerSST-2RTEBoolQCOPACBMultiRCSQuADDROP
TACO94.50 ± 0.3081.11 ± 1.7874.45 ± 0.9088.00 ± 1.0080.36 ± 3.0957.32 ± 1.0283.77 ± 1.5032.06 ± 0.51
TACO-FP1695.60 ± 0.5884.60 ± 0.5580.93 ± 0.5187.67 ± 2.0888.69 ± 1.0368.74 ± 1.0688.48 ± 0.4538.92 ± 0.52
AdaLOMO96.33 ± 0.3087.24 ± 0.5585.32 ± 0.6589.67 ± 2.5294.05 ± 1.0375.60 ± 1.3489.35 ± 0.2542.55 ± 2.60
GaLore95.34 ± 0.1384.36 ± 4.1181.55 ± 0.5189.33 ± 3.2189.88 ± 4.4965.61 ± 3.1387.88 ± 0.3234.92 ± 0.91
Adafactor95.53 ± 0.6084.84 ± 1.0884.40 ± 0.4491.00 ± 1.0091.07 ± 0.0077.57 ± 1.5689.42 ± 0.4637.42 ± 2.77
Adafactor + autograd hook95.53 ± 0.6084.84 ± 1.0884.40 ± 0.4488.00 ± 1.7392.86 ± 1.7977.57 ± 1.5689.42 ± 0.4636.95 ± 2.14
AdamW8bit96.10 ± 0.3084.00 ± 0.42OOM92.67 ± 1.5376.19 ± 16.59OOM88.23 ± 0.40OOM
AdamW8bit + autograd hook96.10 ± 0.3084.00 ± 0.4263.43 ± 0.7587.00 ± 7.5573.81 ± 12.1656.90 ± 0.4088.23 ± 0.4034.44 ± 0.73

TACO-FP16 improves over unscaled E4M3 TACO on seven of eight tasks while adding only 29.7 MB of optimizer state: BoolQ improves from 74.45 to 80.93 (+6.48), MultiRC from 57.32 to 68.74 (+11.42), SQuAD from 83.77 to 88.48 (+4.71), and DROP from 32.06 to 38.92 (+6.86). TACO-FP16 matches or outperforms GaLore and AdamW8bit + hook across most tasks within ~31.0 GB peak memory.

Q1: Adafactor with compatible gradient release

Answer: Yes. We note that Adafactor and AdamW8bit do not natively support the autograd hook. We non-trivially implemented Adafactor with the same post accumulate autograd hook strategy used by TACO while retaining Adafactor's upstream numerical update routine.

The following table reports the comparison on OPT-13B (DROP) with Adafactor and AdamW8bit equipped with gradient release under the same training configuration.

OptimizerState (GB)Peak (GB)Runtime (min)Throughput (tokens/s)DROP F1
TACO0.159030.99 ± 0.0624.07 ± 0.051776.5 ± 3.032.06 ± 0.51
TACO + stochastic rounding0.159031.01 ± 0.0326.54 ± 0.081610.90 ± 4.6631.61 ± 0.71
TACO-FP160.188631.02 ± 0.0624.16 ± 0.231769.8 ± 17.238.92 ± 0.52
TACO-FP16 + stochastic rounding0.188631.03 ± 0.0326.59 ± 0.061607.87 ± 4.4938.38 ± 3.18
AdaLOMO0.025730.91 ± 0.0642.37 ± 0.611009.2 ± 15.342.55 ± 2.60
GaLore2.655133.85 ± 0.29132.47 ± 0.51322.8 ± 1.334.92 ± 0.91
Adafactor0.012854.23 ± 0.0420.54 ± 0.112081.5 ± 9.837.42 ± 2.77
Adafactor + autograd hook0.012830.84 ± 0.0622.17 ± 0.261929.0 ± 21.136.95 ± 2.14
AdamW8bit-OOMOOMOOMOOM
AdamW8bit + autograd hook27.707658.61 ± 0.0526.52 ± 1.121614.1 ± 65.534.44 ± 0.73

Adafactor with autograd hook uses 30.84 GB peak memory, 0.0128 GB optimizer state, and reaches 1929.0 ± 21.1 tokens/s on DROP dataset. TACO-FP16 uses 31.02 GB peak memory, 0.1886 GB optimizer state, and reaches 1769.8 ± 17.2 tokens/s. We additionally report AdaLOMO and GaLore under the same comparison.

However, we note that, by design, the following are algorithmic limitations of AdaFactor compared to TACO, but they should be distinguished from performance failures.

  1. TACO retains sparse directional history at low memory cost; standard Adafactor retains factored magnitude history and discards first-moment momentum.

  2. Adafactor approximates the matrix VV and can be prone to distortion and training instability. TACO avoids this particular approximation error because it does not reconstruct and invert a second-moment matrix.

  3. TACO’s Top-1 direction changes at most nn entries per matrix step. The update count is therefore O(n)O(n), compared with O(mn)O(mn) for a dense Adafactor update, excluding auxiliary parameters and any dense weight-decay operation. We postulate this makes TACO effective in distributed training over multiple machines. We have added this to the revised version of the paper.

AuthorsOpenReview ↗

Official Comment 3

Q2: Matched training time

Answer: We compare TACO-FP16, AdaLOMO, and Adafactor with autograd hook at fixed wall-clock optimization budgets (0, 5, 10, 20, 30, 45, and 60 min).

The following table compares the three optimizers at matched accumulated optimization time across all eight tasks.

DatasetOptimizer0 min5 min10 min20 min30 min45 min60 min
SST-2TACO-FP1661.01 ± 0.0094.84 ± 0.6494.30 ± 1.0493.73 ± 0.4093.73 ± 0.4094.76 ± 1.8094.76 ± 1.80
AdaLOMO61.01 ± 0.0094.99 ± 0.2495.15 ± 0.2995.11 ± 0.5895.80 ± 0.1395.45 ± 0.7395.26 ± 0.63
Adafactor + autograd hook61.01 ± 0.0095.30 ± 0.2395.49 ± 0.1895.64 ± 0.1195.83 ± 0.4395.83 ± 0.4395.80 ± 0.37
RTETACO-FP1657.40 ± 0.0079.54 ± 2.2979.54 ± 2.2982.19 ± 1.9983.03 ± 1.9182.67 ± 1.5782.67 ± 1.57
AdaLOMO57.40 ± 0.0080.02 ± 2.4081.95 ± 1.2586.16 ± 0.9186.52 ± 1.4686.52 ± 1.4686.52 ± 1.46
Adafactor + autograd hook57.40 ± 0.0085.56 ± 2.1786.76 ± 1.1086.76 ± 1.1085.92 ± 0.6385.56 ± 0.0085.32 ± 0.75
BoolQTACO-FP1659.25 ± 0.0075.93 ± 1.3479.32 ± 0.5179.78 ± 1.2380.98 ± 1.4680.93 ± 1.3980.65 ± 1.04
AdaLOMO59.25 ± 0.0075.18 ± 3.0077.92 ± 0.2880.08 ± 2.7081.70 ± 1.6383.40 ± 0.9883.75 ± 2.81
Adafactor + autograd hook59.25 ± 0.0076.55 ± 3.8080.80 ± 3.5380.93 ± 3.6381.93 ± 4.6882.47 ± 4.9682.18 ± 4.71
COPATACO-FP1679.00 ± 0.0086.00 ± 1.7387.67 ± 2.5288.00 ± 2.6588.00 ± 2.6590.00 ± 1.0090.00 ± 1.00
AdaLOMO79.00 ± 0.0087.33 ± 0.5888.33 ± 1.1588.33 ± 0.5889.00 ± 1.0089.00 ± 1.0089.00 ± 1.00
Adafactor + autograd hook79.00 ± 0.0089.33 ± 1.5388.67 ± 0.5889.00 ± 0.0089.00 ± 0.0089.00 ± 0.0089.00 ± 0.00
CBTACO-FP1644.64 ± 0.0086.90 ± 1.0387.50 ± 1.7986.90 ± 6.2789.29 ± 3.0989.29 ± 3.0989.29 ± 3.09
AdaLOMO44.64 ± 0.0091.67 ± 8.9993.45 ± 6.2793.45 ± 6.2793.45 ± 6.2793.45 ± 6.2793.45 ± 6.27
Adafactor + autograd hook44.64 ± 0.0094.05 ± 4.1294.05 ± 4.1294.05 ± 4.1294.64 ± 3.0994.64 ± 3.0994.64 ± 3.09
MultiRCTACO-FP1656.67 ± 0.0056.67 ± 0.0060.64 ± 3.4764.78 ± 3.7868.23 ± 1.6970.19 ± 1.7172.58 ± 0.48
AdaLOMO56.67 ± 0.0056.67 ± 0.0058.84 ± 4.6970.98 ± 0.8073.29 ± 2.6073.29 ± 2.6076.17 ± 1.82
Adafactor + autograd hook56.67 ± 0.0062.55 ± 9.8862.55 ± 9.8867.61 ± 1.3774.21 ± 1.4174.77 ± 1.7075.59 ± 2.94
SQuADTACO-FP1644.86 ± 0.0087.03 ± 0.2487.26 ± 0.3987.74 ± 0.4487.84 ± 0.3888.38 ± 0.6788.68 ± 0.74
AdaLOMO44.86 ± 0.0086.37 ± 0.4987.01 ± 1.0087.88 ± 0.4688.37 ± 0.5488.71 ± 0.7688.71 ± 0.76
Adafactor + autograd hook44.86 ± 0.0087.67 ± 0.3386.93 ± 0.9689.24 ± 0.7890.06 ± 0.5090.22 ± 1.0289.98 ± 0.21
DROPTACO-FP1628.34 ± 0.0032.69 ± 0.3034.86 ± 0.7937.01 ± 0.1938.55 ± 1.1839.45 ± 0.4840.69 ± 0.94
AdaLOMO28.34 ± 0.0034.50 ± 1.1535.43 ± 1.1238.08 ± 1.1638.61 ± 0.2341.25 ± 2.1641.74 ± 2.79
Adafactor + autograd hook28.34 ± 0.0034.72 ± 0.3236.03 ± 1.0736.03 ± 1.0737.83 ± 3.9640.06 ± 3.2542.95 ± 1.79

On BoolQ, TACO-FP16 reaches 80.98 ± 1.46 at 30 minutes, matching AdaLOMO (81.70 ± 1.63) and Adafactor + autograd hook (81.93 ± 4.68) within one standard deviation. On COPA, TACO-FP16 surpasses both baselines at 45–60 minutes (90.00 ± 1.00 vs. 89.00). On MultiRC (56.67→72.58)(56.67 \to 72.58) and DROP (32.69→40.69)(32.69 \to 40.69), TACO-FP16 exhibits steady monotonic improvement across the full 60-minute window, narrowing the initial 2000 step gap (from 10.8 points at 20 min to 3.0 points at 60 min on MultiRC). We have added this to the Appendix of the revised version of the paper.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General message to AC, SACs, PCs, and the reviewers

We thank the Area Chair (AC) and Senior Area Chairs (SACs) for handling the paper, and all three reviewers (bkww, 3vqc, and ZgU1) for their constructive feedback and careful reading of our manuscript. During the rebuttal period, we were able to address all questions and issues raised. We derived the new theoretical convergence of TACO using minibatch stochastic gradients, and implemented the requested empirical modification of Adafactor and AdamW8bit. All major modifications in the revised manuscript are highlighted in blue. Below, we provide a summary of the major revisions.

Summary of Major Revisions:

  1. Controlled Gradient Release Baselines & TACO-FP16 Evaluation (Appendix C.1, Tables 11–12; Sections 1, 2, 5, 6)

  2. Matched Wallclock Optimization Budget Comparison (Appendix C.1, Table 13)

  3. Two Stage Learning Rate Search (Appendix C.1.1, Table 14)

  4. History Precision & Effective BF16 Update Diagnostics (Appendix C.2, Tables 15-16)

  5. Stochastic Convergence Analysis and Clarified Scope of Continuation Geometry (Section 3, Theorem 3; Section 4; Appendix A.1)

  6. Empirical Row-Winner Concentration & Normalized Spectral Ratio Diagnostics (Appendix C.3, Tables 17–18; Section 4)

  7. Multi-GPU Distributed Communication Experiment as a Proof of Concept (Appendix C.4, Tables 19-20; Abstract, Sections 1, 5)

  8. Figure 1 & Related Work Updates (Figure 1, Sections 1-2)

Overall, we believe these extensive new experiments, diagnostics, and theoretical extensions directly resolve the reviewers' concerns and substantially strengthen the manuscript. We would appreciate it if Reviewer ZgU1 could highlight TACO's geometric simplicity, communication efficiency, and strong TACO-FP16 empirical gains during the discussion, and we hope our controlled hook-based baselines, matched wall-clock comparisons, and precision/winner diagnostics will encourage Reviewers bkww and 3vqc to reconsider their evaluations favorably.

AuthorsOpenReview ↗

Can Reviewer bkww kindly let us know what else we need to do to get the reviewer’s attention to our rebuttal and the revised version?

Dear Reviewer bkww,

We would greatly appreciate it if the reviewer could provide any comments on our work after we submitted the rebuttal and the revised version of the paper, as the author–reviewer discussion period ends in less than 24 hours.

We believe we have clarified all questions related to isolating TACO's sparse optimizer state from backward gradient release, quantifying the memory–accuracy–throughput trade-off, and analyzing E4M3/FP16 history and BF16 updates. The specific issues that the reviewer asked for further clarification on are:

  1. Controlled gradient release baselines (Adafactor, AdamW8bit, AdaLOMO, GaLore) and 4-GPU distributed communication results (W1 & Q1)
  2. Multi-seed OPT-13B evaluation and TACO-FP16 results across all eight tasks (W2)
  3. Matched wallclock optimization budget comparisons with three seed variance (Q2)
  4. Unscaled E4M3 vs. FP16 history precision and per-layer effective BF16 parameter update diagnostics (Q3)

are provided with objective details in our rebuttal and revised manuscript with major additions highlighted in blue.

We would again like to reiterate that TACO's scientific contribution lies in deriving column-wise one-sparse updates directly from the dimension-normalized 1→11\to1 operator-norm geometry, enabling O(n)\mathcal{O}(n) persistent state, O(n)\mathcal{O}(n) per-matrix parameter updates with competitive downstream performance (accuracy/F1 score), practical throughput, and a low memory footprint in LLM fine-tuning, rather than relying solely on backward gradient release.

If these responses address the reviewer’s concerns, we kindly ask the reviewer to reconsider their score.

Sincerely,

The Authors

AuthorsOpenReview ↗

Can Reviewer 3vqc kindly let us know what else we need to do to get the reviewer’s attention to our rebuttal and the revised version?

Dear Reviewer 3vqc,

We would greatly appreciate it if the reviewer could provide any comments on our work after we submitted the rebuttal and the revised version of the paper, as the author–reviewer discussion period ends in less than 24 hours.

We believe we have clarified all questions related to disentangling peak memory from gradient-lifecycle techniques, reporting direct efficiency comparisons across performance, memory, and throughput in tabular form, and grounding our theoretical claims for TACO with minibatch stochastic gradients and empirical layer diagnostics. The specific issues that the reviewer asked for further clarification on are:

  1. Controlled autograd hook baselines (Adafactor, AdamW8bit, AdaLOMO, GaLore) and matched optimization time comparisons (W1 & Q1)
  2. Stochastic minibatch convergence analysis (Theorem 3) and conditional scope of continuation geometry (Theorems 4 and 5) (W2)
  3. Empirical row-winner concentration cmax⁡c_{\max} and normalized spectral ratio ρt\rho_t diagnostics across model widths (W2 & Q3)
  4. Direct tabular efficiency comparisons (state memory, peak memory, runtime, throughput, and task performance), matched optimization time comparisons with per-seed variance, and learning rate selection under the final configuration (Q2.1, Q2.2, Q2.3)

are provided with objective details in our rebuttal and revised manuscript with major additions highlighted in blue.

We would again like to reiterate that TACO's scientific contribution lies in deriving column-wise one-sparse updates directly from the dimension-normalized 1→11\to1 steepest-descent geometry and validating its conditional theoretical mechanisms, wallclock efficiency, and O(n)\mathcal{O}(n) update sparsity at scale, rather than claiming unconditional equivalence to AdamW or attributing peak memory solely to autograd hooks.

If these responses address the reviewer’s concerns, we kindly ask the reviewer to reconsider their score.

Sincerely,

The Authors

AuthorsOpenReview ↗

Official Comment to the AC—The author–reviewer discussion period ends in less than 24 hours—Please initiate the discussion

Dear Area Chair (CC: SACs and PCs),

In our rebuttal, we have carefully addressed each of the concerns raised and have made every effort to provide clear and comprehensive responses. As the discussion period draws to a close, we would greatly appreciate any additional feedback on our responses. We addressed every concern the reviewers raised, and these issues are resolved after the rebuttal. Please see Reviewer ZgU1's (confidence 4 and gave us a score of 3) latest comment, which says, Thanks for your clarification. I am satisfied with the rebuttal now. I am waiting for other reviewers' response. And after that, I do the final recommendation. However, we expect your active engagement.

At this stage, we request that the AC uphold the responsibilities of the review process by encouraging reviewers to engage with the authors during the discussion period. A lack of engagement undermines the purpose of the rebuttal process and falls short of the collaborative and constructive review standards that conferences such as ICLR strive to uphold. We request that the Area Chair and their team show basic competence and courtesy by engaging in meaningful discussion with the authors, which at this point is a fundamental component of a fair and rigorous evaluation, and is in line with the ICLR guidelines.

Sadly, we have to insist that the AC initiate this discussion at a top venue like ICLR, where author-reviewer discussion is the norm and historically decides the fate of the paper; the author–reviewer discussion period ends in less than 24 hours.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.