acceptodds
Under review as a conference paper at ICLR 2027

GWT: Scalable Optimizer State Compression for Large Language Model Training

Abstract

Training large language models (LLMs) requires substantial memory, a significant fraction of which is consumed by the moment states maintained by adaptive optimizers such as Adam. Existing memory-efficient approaches commonly compress parameters or gradients through low-rank projections, which may discard information outside the selected subspace or incur additional costs from subspace construction and singular value decomposition (SVD). We introduce Gradient Wavelet Transform (GWT), an SVD-free framework that applies a multilevel Haar transform to matrix gradients. GWT maintains first- and second-moment states only for the compact approximation coefficients, while retaining all detail coefficients transiently in each update. We show that the orthogonal transform preserves the complete current gradient and establish gradient alignment and a stationary-point guarantee for smooth objectives under bounded preconditioning and a directly verifiable coarse-momentum coherence condition. Experiments on language-model pre-training and downstream fine-tuning show that GWT reduces estimated model-and-optimizer-state memory by up to 52% while maintaining competitive or improved model quality. On LLaMA-3B pre-training, GWT achieves the throughput of 8-bit Adam and is compatible with multiple optimizer families, including Adam, Adam-mini, and Muon.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Wavelet transform, large language model, memory efficient

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YmrTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a optimizer called GWT, a memory efficient-based optimizer for LLM training. Unlike low-rank projection methods, e.g., GaLore, GWT applies a multi-level Harr wavelet transform to matrix gradients and maintains optimizer states only for coarse approximatio coefficients, while preserving the instantaneous detail coefficients during gradient reconstruction. This paper also provides theoretical analisys of gradient information preservation, directional alignment, and convergence under bounded preconditioning and coarse-momentum coherence assumptions. Experiments demonstrate competitive performance compared with existing memory-efficient optimizers.

Critical strengths
  • (S1) The paper proposes a novel optimizer-state compression approach that differs from conventional low-rank projection methods, demonstrating clear innovation and research value.

  • (S2) The proposed technique incurs low computational overhead and retains competitive training performance across multiple model scales and tasks, with straightforward integration into existing optimization frameworks.

Critical weaknesses
  • (W1) In Appendix E.6 (Figure 11), the authors found that when Full Adam uses the same module-wise learning rate, its convergence performance improves significantly, and its performance approaches or, on small-scale datasets, exceeds that of GWT. Therefore, it is currently unclear to what extent GWT’s performance improvement actually stems from the wavelet transform itself.

  • (W2) Table 12 shows that the Detail coefficient accounts for 93.7%–96.7% of the inner product between the gradient and the update direction. This indicates that Detail plays an important role in the update direction, but the actual contribution of Coarse’s historical momentum remains unclear. And some typos, e.g., "GWT-2 achieves 0.532K tokens ...... about 6% higher throughput than GaLore-1/4...", looks like not 6%

Questions for authors
  • (Q1) Could the authors provided some experiments with Fira & SWAN baseline?

  • (Q2) Could the authors provide more independent validation or ablation experiments for GWT's performance from which methods?

  • (Q3) The authors use Haar wavelets to decompose the gradient into coarse and detail components, but do not fully explain why Haar wavelets must be chosen over other orthogonal transforms. Could the authors add comparative experiments with other methods?

  • (Q4) Based on W2. Could the authors provide ablation experiments showing the necessity of Coarse?

4 comments
AuthorsOpenReview ↗

Response to Reviewer YmrT: Weaknesses (W1–W2)

We sincerely thank the reviewer for recognizing GWT's novelty, low computational overhead, and competitive performance. We appreciate the constructive suggestions and address the identified weaknesses (W) and questions (Q) below.

W1: Does the Wavelet Transform Explain the Performance Gains?

We agree that the performance gap between GWT and vanilla Adam cannot be attributed solely to the wavelet transform. Module-wise learning rates substantially improve Adam, as our experiments confirm. We have therefore refined the performance claims in Section 4.1 and Figure 2 to emphasize competitive training quality with lower optimizer-state memory.

Building on our existing module-wise Adam comparisons on LLaMA-60M and 130M, we extend the evaluation to 350M and 1.3B using the same learning-rate strategy as GWT:

lr_{\mathrm{others}}=lr.$$ We further refine the scaling-factor search for **60M–350M**, additionally evaluating $\alpha\in\{0.1,0.2,0.3\}$ with the base learning rate fixed at 0.01. The expanded search selects $\alpha=0.20$ for 60M and $\alpha=0.25$ for 130M and 350M. The results below report PPL, with estimated model-and-optimizer-state memory in parentheses. | Method | 60M | 130M | 350M | 1.3B | |---|---:|---:|---:|---:| | Adam (Vanilla) | 30.60 (0.34G) | 25.08 (0.80G) | 18.75 (2.20G) | 16.10 (8.03G) | | Adam (Module-wise) | **29.20** (0.34G) | 22.74 (0.80G) | 16.46 (2.20G) | 14.17 (8.03G) | | GWT-2 | 29.35 (0.27G) | **22.47** (0.55G) | **16.29** (1.24G) | **13.50** (4.41G) | These results show that module-wise learning rates account for much of the original performance gap with vanilla Adam. Module-wise Adam slightly outperforms GWT at 60M, while GWT achieves lower PPL at larger scales. Importantly, GWT's primary contribution lies in **wavelet-based optimizer-state compression**. By maintaining historical adaptive states only for the coarse coefficients while retaining the current detail coefficients, GWT substantially reduces memory without sacrificing training quality. For example, at 350M and 1.3B, GWT reduces estimated model-and-optimizer-state memory from 2.20G to 1.24G and from 8.03G to 4.41G, respectively, while achieving lower PPL than module-wise Adam. Detailed settings and results appear in **Table 2 and Appendix E.6**. ### **W2: Coarse-Momentum Contribution and Throughput Correction** We agree that the large contribution of detail coefficients to gradient-update alignment does not establish whether historical coarse momentum is necessary. To investigate this, we added ablations of coarse momentum, the coarse update direction, and detail preconditioning. All three variants increase validation PPL. The results are reported in **Appendix E.8, Table 19 (p. 33)** and discussed under Q4. We also thank the reviewer for identifying the throughput error. The difference between 0.532K and 0.526K tokens/s/GPU is approximately **1.1%, not 6%**. We have removed the incorrect claim from **Section 4.1 (p. 7)**.
AuthorsOpenReview ↗

Response to Reviewer YmrT: Questions (Q1–Q4)

Q1: Can You Compare GWT with Fira and SWAN?

We have added Fira to our main experiments, evaluating two compression levels across LLaMA-60M to 1.3B. We have also incorporated the published results of SWAN into Table 2, with its memory re-estimated under our BF16 model configurations.

The results below report PPL, with estimated model-and-optimizer-state memory in parentheses. Fira results are from our reproduced experiments, while SWAN results are taken from the cited publication.

Method60M130M350M1.3B
Fira-1/430.43 (0.28G)23.18 (0.58G)16.72 (1.38G)13.89 (4.76G)
GWT-229.35 (0.27G)22.47 (0.55G)16.29 (1.24G)13.50 (4.41G)
Fira-1/830.71 (0.26G)23.31 (0.53G)16.74 (1.23G)14.11 (4.15G)
GWT-329.81 (0.26G)22.63 (0.51G)16.35 (1.09G)13.48 (3.80G)
SWAN‡ (literature)30.59 (0.25G)22.61 (0.46G)16.63 (1.00G)13.56 (3.20G)

GWT achieves lower PPL than Fira at both compression levels, with comparable or lower estimated memory. Compared with SWAN, GWT-2 achieves lower reported PPL across all four model scales, while SWAN requires less estimated memory.

Following the reviewers' suggestions, we have also included SinkGD and SCALE in Table 2, with their published PPL and consistently re-estimated memory. We discuss these comparisons in Section 4.1, highlighting the differences in training quality and memory efficiency. We have also expanded Related Work to clarify their methodological distinctions from GWT, particularly its use of compressed historical adaptive states and instantaneous detail coefficients.

Q2: Can You Provide Independent Validation and Ablations?

We have added component ablations and wavelet-basis comparisons on LLaMA-130M. They test the contributions of coarse momentum, coarse updates, detail preconditioning, and the choice of wavelet basis. The results appear in Appendix E.8 (pp. 32–34) and are detailed under Q3 and Q4.

Q3: Why Haar over Other Orthogonal Transforms?

We choose Haar as the default wavelet for theoretical simplicity and computational efficiency. Its approximation coefficients correspond to normalized block means. This structure supports the detail-energy bound in Lemma 3.1, while the two-tap filter enables efficient decomposition and reconstruction.

We additionally compare Haar with Daubechies-2 (db2) at decomposition levels 2 and 3. Unlike Haar, db2 uses a four-tap filter with nonuniform weights and captures overlapping local information.

The results are reported in Appendix E.7, Table 19 (p. 34).

WaveletPPL ↓Time ↓Mem. ↓
Haar-222.474.03h0.55G
Haar-322.634.06h0.51G
db2-222.394.91h0.55G
db2-322.485.03h0.51G

Both db2 configurations achieve slightly lower PPL than Haar at the corresponding levels but require more training time. Thus, GWT is not restricted to Haar, while Haar provides a favorable balance between theoretical tractability, training quality, and computational efficiency.

Q4: Is the Coarse Component Necessary?

To directly evaluate the role of the coarse component, we conduct three targeted ablations on LLaMA-130M:

  • w/o Coarse-M: Replaces historical coarse momentum with current coarse coefficients while retaining the second-moment estimates.
  • w/o Coarse: Removes the coarse update direction while retaining coarse statistics for detail preconditioning.
  • Res-RMS: Replaces shared coarse second-moment preconditioning with instantaneous block-wise residual RMS.

The results are summarized below.

MethodPPL ↓
GWT-222.47
w/o Coarse-M23.03
w/o Coarse24.28
Res-RMS23.67

All three ablations degrade validation PPL. Removing the coarse update causes the largest degradation, confirming its importance even though the detail coefficients contribute substantially to gradient-update alignment.

These results support the complementary roles of historical coarse momentum, the coarse update direction, and shared detail preconditioning. Detailed settings and analysis are provided in Appendix E.7, Table 18 (p. 33).

Thanks to the author's detailed response, my concerns have been solved. I maintain my initial score.

We are glad that the additional experiments addressed your concerns. Your suggestions helped us make the practical implications of the framework clearer. We appreciate your time and constructive feedback.

Reviewer_Z5xBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes Gradient Wavelet Transform (GWT), a method for optimizer state compression for LLM pretraining and finetuning tasks. It achieves the compression by applying multilevel Haar transforms to the gradients and maintaining states only for the resulting approximation coefficients, saving about 75% of the Adam state memory. The main evaluations include LLaMA 60M-3B pretrained on C4 as well as finetuning tasks on MMLU and GLUE, suggesting GWT obtains competitive performance.

Critical strengths

Novel application of Wavelet Transform. The proposed approach using Wavelet Transform on the gradients to achieve model compression appears promising and different from most prior works that that rely on the computationally heavy SVD.

Critical weaknesses

Unusual empirical results. It seems unusual to me that GWT, a method based on compressing the Adam states, performs significantly better than the full Adam in the LLM pretraining experiments of Table 2. Shouldn't the compressed version of Adam be at most as good (in terms of validation perplexity achieved) as the full version, not better? It is currently not justified why compressing the Adam states would provide improved perplexity (at the same token budget, i.e., better optimization dynamics). I understand that the benchmark setting is adapted from the GaLore paper and is aimed at testing optimizers for memory-efficient pretraining, however it should be clear to the reader what are its limitations and why it might favor state-compressed versions of Adam.

Incomplete comparison to prior works. The paper mentions SWAN, SinkGD and SCALE as related memory-efficient optimizers, yet the main experimental results in Table 2 only compare to the older GaLore and APOLLO. I understand that rerunning all those extra methods might not be possible during the rebuttal, however as their papers also contain results from the same benchmark, they can be included in the table as is (perhaps with a note that those results are taken from the literature), along with a paragraph containing a comparison.

Questions for authors

My main question is related to weakness 1., i.e., how is it possible that a compressed version of Adam to perform better than the full Adam? If this is due to the specific benchmarking setup this needs to be explained in the paper.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Z5xB: Weaknesses (W1)

We sincerely thank the reviewer for recognizing the novelty of GWT's wavelet-based optimizer-state compression approach. We have expanded the empirical comparisons and clarified the benchmarking configuration to address the identified weaknesses (W) and question (Q) below.

W1: Why Does Compressed Adam Outperform Full Adam?

We agree that outperforming vanilla Adam does not necessarily imply that optimizer-state compression itself improves optimization. To better understand this phenomenon, we consider observations from prior work, relevant theoretical and empirical insights, and the effects of learning-rate configurations.

Observations from prior studies. Several memory-efficient optimizers have reported better PPL than full Adam(W). Their published LLaMA/C4 pre-training results are summarized below.

Method60M130M350M1.3B
Fira31.06 (−3.00)22.73 (−2.35)16.85 (−1.95)14.31 (−1.25)
APOLLO31.55 (−2.51)22.94 (−2.14)16.85 (−1.95)14.20 (−1.36)
LDAdam—22.65 (−1.99)17.30 (−0.78)14.09 (−0.77)

Results are taken from the original Fira (NeurIPS 2025), APOLLO (MLSys 2025), and LDAdam (ICLR 2025) papers. Entries report method PPL (difference from the corresponding full Adam(W) baseline).

These observations show that outperforming vanilla Adam is not unique to GWT. The improvements may reflect both optimization mechanisms and experimental configurations.

Theoretical and empirical insights. Prior work provides several relevant perspectives. Hardt et al. (ICML 2016, Train Faster, Generalize Better) connect algorithmic stability with generalization, while Shen et al. (2025, Efficient Deep Neural Network Training...) discuss regularization effects from reduced precision. More directly, APOLLO (Appendix A.4) offers preliminary insights into how coarser, SGD-like updates may improve generalization through increased update stochasticity. Fira (Appendix K) further shows that its Fira-only-scaling variant is more robust to injected gradient noise than Adam.

For GWT, Theorem 3.4 establishes gradient alignment and convergence under the stated conditions. Empirically, 98.24%–99.05% of sampled matrix–step pairs satisfy the sufficient coherence condition (Appendix E.1), and all sampled cases exhibit positive alignment (Appendix E.2). Component ablations (Appendix E.8, Table 19) further support the roles of coarse momentum, coarse updates, and detail preconditioning.

Fair comparison with Adam. One important factor is the module-wise learning-rate strategy used by GaLore-style optimizers. Building on our original analysis of LLaMA-60M and 130M (original Appendix E.6), we extend the module-wise Adam comparison to 350M and 1.3B. We use the same parameter grouping as GWT:

lr_{\mathrm{others}}=lr.$$ We search six configurations with base learning rates of 0.005 and 0.01 and scaling factors of 0.25, 0.5, and 0.75. The selected configuration is (0.01, 0.25) for 60M–350M and (0.005, 0.25) for 1.3B. The higher base rate causes divergence at 1.3B. The results below report PPL, with estimated model-and-optimizer-state memory in parentheses. | Method | 60M | 130M | 350M | 1.3B | |---|---:|---:|---:|---:| | Adam (Vanilla) | 30.60 (0.34G) | 25.08 (0.80G) | 18.75 (2.20G) | 16.10 (8.03G) | | Adam (Module-wise) | **29.29** (0.34G) | 22.74 (0.80G) | 16.46 (2.20G) | 14.17 (8.03G) | | GWT-2 | 29.35 (0.27G) | **22.47** (0.55G) | **16.29** (1.24G) | **13.50** (4.41G) | Module-wise learning rates substantially improve Adam and narrow the original performance gap. At 60M, tuned Adam slightly outperforms GWT. At larger scales, GWT achieves lower PPL while reducing optimizer-state memory. Accordingly, we have refined the performance claims in **Section 4.1 (p. 7) and Figure 2 (p. 8)**. The revised discussion emphasizes competitive training quality with reduced memory, rather than attributing the original gap solely to state compression. The tuned Adam results appear in **Table 2**, with detailed settings in **Appendix E.7 (pp. 31–32)**.
AuthorsOpenReview ↗

Response to Reviewer Z5xB: Weakness 2 (W2) and Question1 (Q1)

W2: Missing Comparisons with SWAN, SinkGD, and SCALE

We thank the reviewer for suggesting comparisons with SWAN, SinkGD, and SCALE. We also include Fira because it is closely related to GWT. Both methods preserve residual gradient information while maintaining compressed adaptive states.

We evaluate Fira across LLaMA-60M to 1.3B at two compression levels. The results below report PPL, with estimated memory in parentheses.

Method60M130M350M1.3B
Fira-1/430.43 (0.28G)23.18 (0.58G)16.72 (1.38G)13.89 (4.76G)
GWT-229.35 (0.27G)22.47 (0.55G)16.29 (1.24G)13.50 (4.41G)
Fira-1/830.71 (0.26G)23.31 (0.53G)16.74 (1.23G)14.11 (4.15G)
GWT-329.81 (0.26G)22.63 (0.51G)16.35 (1.09G)13.48 (3.80G)

GWT achieves lower PPL than Fira across all evaluated scales, with comparable or lower estimated memory. These results have been added to Table 2 (p. 7).

For SWAN, SinkGD, and SCALE, independently reproducing every method under a unified setting would require additional computational resources. We therefore supplement the direct Fira experiments with published results on LLaMA/C4.

Method130M350M
SWAN‡22.61 (0.43G)16.63 (0.93G)
SinkGD22.75 (0.43G)16.51 (0.93G)
SCALE22.57 (0.32G)16.32 (0.80G)
GWT-222.47 (0.55G)16.29 (1.24G)

These published results suggest competitive training quality for GWT, while the other methods report lower estimated memory. Because the experiments were conducted independently, differences in training settings, evaluation, and memory accounting limit direct numerical comparisons. We also avoid comparing measured peak memory and throughput across different environments.

The comparison and its limitations are documented in Appendix E.4, Table 13 (p. 29).

Q1: Why Can Compressed Adam Perform Better than Full Adam?

As detailed under W1, we address this question from three perspectives. Prior studies such as Fira, APOLLO, and LDAdam report similar improvements over full Adam. Theoretical and empirical analyses, including GWT's gradient-alignment results and component ablations, offer insights into why compressed adaptive states can remain effective. Our fair Adam comparison further shows that module-wise learning rates explain a substantial part of the original performance gap, while GWT maintains competitive PPL with reduced optimizer-state memory.

The corresponding results and discussions are provided in Section 4.1 (p. 7), Table 2, and Appendices E.1–E.2 and E.7–E.8.

Thank you for your response.

I appreciate the additional Adam experiments with module-wise learning rates. They support my concern that the original Adam baseline was unusually weak and show that the learning-rate configuration accounts for much of the gap. However the selected configurations always use a scaling factor of 0.25 which is the lowest value tested. This suggests that testing smaller values could further improve Adam's performance.

Regarding the remaining gap, I still think that the theoretical analysis and empirical results do not fully explain GWT's advantage over Adam. The results may depend on the particular benchmark settings, although I recognize that the GaLore setup is commonly used to compare memory-efficient optimizers, so I do not consider this a major weakness of the paper.

Regarding the comparisons with SWAN, SinkGD, and SCALE, I find the current response not entirely satisfactory. Adding Fira, an older method, does not replace a discussion of these recent methods, which achieve competitive perplexity with lower memory. At least a discussion of these comparisons should be included in the main experiment section and not just in the appendix (in a section that is not refereed in the main paper at all). Published results could also be included in the main table and clearly marked as cited, as APOLLO and Fira do in their tables.

The concerns raised in the appendix do not adequately justify leaving these comparisons out of the main discussion. The memory results in parentheses are theoretical estimates of parameter and optimizer-state storage, not measured peak memory, so hardware differences should not prevent this comparison. Also SCALE particularly does not use computationally heavy operations, so the general concern about training efficiency does not seem to apply. The evaluation procedures also appear to follow the same protocol.

As noted in my review, I still find the proposed approach interesting, but the main paper should discuss how it compares with recent and directly relevant methods that follow the same standard GaLore benchmark. This would help readers fairly assess the method's advantages.

Reviewer_zqgqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

GWT applies an l-level Haar transform to the gradients of the attention and MLP matrices (other parameters use Adam), keeps Adam moments only for the coarsest approximation band (2^-l of the entries), and rescales the current detail coefficients with the broadcast coarse second moment, without momentum, before the inverse transform (Eqs. 6 to 9). A norm-growth limiter is added. Theorem 3.4 gives a deterministic stationarity bound for the update direction before the limiter, under bounded preconditioning and a coarse-momentum coherence condition. Experiments pre-train LLaMA models from 60M to 3B on C4 and fine-tune on MMLU and GLUE. The paper reports up to 52% lower estimated weight-plus-optimizer-state memory, lower perplexity than full Adam and than GaLore and APOLLO at matched state, and 1.9x the throughput of 8-bit Adam at 3B.

Critical strengths
  1. Simple, SVD-free, and strong at matched optimizer state. GWT needs no projection matrices or SVD, and its update cost is linear in the matrix size. Within the paper's protocol it beats GaLore and APOLLO at matched optimizer-state compression at every scale, for example 13.50 vs 15.66 and 14.10 perplexity at 1.3B (Table 2) and 13.21 vs 14.73 and 13.75 at 3B (Table 3). A cheap drop-in method with this profile would be useful in practice.

  2. The paper tests its own assumptions and controls. It measures the coherence condition, energy concentration and update alignment (App. E, Tables 10 to 12), and it includes a module-wise Adam control (App. E.6). I checked Lemmas 3.1 and 3.2 and Theorem 3.4 and found them correct. This makes the claims checkable, which I appreciate.

Critical weaknesses
  1. The main comparison with full Adam does not separate the optimizer from the learning-rate strategy. In Table 2, Adam uses one uniform rate per scale (Table 8), while GWT, GaLore and APOLLO use 0.01 on embeddings, norms and the head and 0.01·α on the matrices, and only GWT's α was swept for the main runs (p. 22). Appendix E.6 adds a uniform-rate Adam sweep and module-wise Adam controls at 60M and 130M. In Fig. 11 the best module-wise Adam runs finish close to GWT (about 29.3 vs 29.3 and 22.8 vs 22.5, read from the plot), against gaps of 4.02 and 2.61 PPL in Table 2, and the paper itself calls GWT "close to" module-wise Adam; even the uniform-rate Adam shown there ends near 30.6 at 60M, against 33.37 in Table 2. At 60M, Muon, Adam-mini and LDAdamW also beat Table 2's Adam by about 4 PPL. This supports GWT's ability to keep Adam-like quality with much less optimizer state, but not the stronger claim that it beats full Adam (Sec. 4.1, Fig. 2), and no comparable control is reported at 350M or above.

  2. The mechanism is simpler than the wavelet framing suggests, and its relation to residual-preserving methods needs sharper positioning. Because the detail coefficients carry no state and share the broadcast coarse preconditioner (Eq. 7), the multilevel transform collapses: within each block of b = 2^l entries the update is α(g − ḡ + m̂/√b)/(√v̂ + ε), where ḡ is the current block mean and m̂, v̂ are the Adam moments of the normalized coarse coefficient √b·ḡ. As a check, the row [3, 1, 2, 2] with l = 2, m̂ = 2 and √v̂ = 4 gives α[2, 0, 1, 1]/(4 + ε) both through Eqs. 6 to 9 and through this formula. So GWT keeps moments for normalized block means and adds a stateless zero-mean residual scaled by the same coarse second moment; in Table 12 the residual accounts for 93.7% to 96.7% of the instantaneous gradient-update inner product. This is close to the stateful-subspace plus state-free-residual design of FRUGAL (Zmushko et al., ICML 2025), and Fira, which the paper cites for its limiter, also uses information from the compressed update to scale the residual; the introduction's statement that earlier methods leave outside-subspace components unused is too broad for these methods. Trion (ICLR 2026), which replaces SVD with a predefined DCT basis, is also relevant. Stating this reduction and distinguishing GWT's fixed Haar subspace and shared preconditioner from these designs would clarify the contribution.

Questions for authors
  1. Please give the exact endpoints, selected rates and module grouping for the Fig. 11 controls and reconcile them with Table 2, and add a comparison with comparable tuning effort for Adam and GWT at one larger scale (for example 350M, plus any existing 1.3B control). Please also list the Muon, Adam-mini and LDAdamW settings. If GWT matches tuned Adam at a larger scale with this memory saving, I would raise my score.

  2. Do you agree that the update reduces to moments on normalized block means plus a stateless residual scaled by the coarse second moment? Please clarify the contribution relative to FRUGAL and Fira; a targeted test of the coarse momentum, or of the residual scaling (for example by the residual's own block RMS or by its sign), would show what the chosen statistics and the Haar structure add.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zqgq: Weaknesses (W1–W2)

We sincerely thank the reviewer for recognizing GWT's SVD-free design, memory efficiency, and competitive performance. We address the weaknesses (W) and questions (Q) below.

W1: Fairness of Adam Comparisons under Module-wise Learning Rates

We agree that module-wise learning rates contribute substantially to the performance gap between GWT and vanilla Adam. We therefore extend the module-wise Adam comparison to LLaMA-350M and 1.3B, using the same parameter grouping as GWT:

lr_{others}=lr.$$ We first search 6 configurations with base learning rates of 0.005 and 0.01 and $\alpha\in\\{0.25,0.5,0.75\\}$. For 60M–350M, the best initial configuration is $(0.01,0.25)$. Motivated by this result, we conduct a finer search over $\alpha\in\\{0.1,0.2,0.3\\}$ for these three models, keeping the base learning rate fixed at 0.01. The expanded search selects $\alpha=0.2$ for 60M and $\alpha=0.25$ for 130M and 350M. For 1.3B, the selected configuration remains $(0.005,0.25)$. The vanilla Adam learning rates are 0.0025, 0.001, 0.001, and 0.0005, respectively. The results below report PPL, with estimated memory in parentheses. | Method | 60M | 130M | 350M | 1.3B | |---|---:|---:|---:|---:| | Adam (Vanilla) | 30.60 (0.34G) | 25.08 (0.80G) | 18.75 (2.20G) | 16.10 (8.03G) | | Adam (Module-wise) | **29.20** (0.34G) | 22.74 (0.80G) | 16.46 (2.20G) | 14.17 (8.03G) | | GWT-2 | 29.35 (0.27G) | **22.47** (0.55G) | **16.29** (1.24G) | **13.50** (4.41G) | Module-wise learning rates substantially improve vanilla Adam. With the expanded search, module-wise Adam slightly outperforms GWT at 60M, while GWT achieves lower PPL at 130M, 350M, and 1.3B with reduced optimizer-state memory. We have refined the performance claims in **Section 4.1 and Figure 2**, with detailed settings and results provided in **Appendix E.6**. **Correction of the 60M Adam result.** The original Table 2 contained a transcription error. The correct vanilla Adam PPL is 30.60 (lr = 0.0025), rather than 33.37 (lr = 0.001). We have corrected the value and apologize for the oversight. ### **W2: Haar Block Averaging and Distinctions from Related Work** **Haar interpretation and wavelet generality.** We agree with the reviewer's interpretation. With Haar, GWT maintains historical moments of normalized block means. It uses the shared coarse second moments to precondition the current zero-mean residual. We now state this explicitly in **Section 3.2 (p. 4)**. To examine whether GWT depends on Haar-specific block averaging, we additionally evaluate the Daubechies-2 (db2) wavelet at 2 and 3 decomposition levels. For the two-level case, away from boundaries, the approximation coefficients are $$a_q^{Haar} =\frac{1}{2}\sum_{k=1}^{4}g_{4(q-1)+k},\quad a_q^{db2} =\sum_{i,j=0}^{3}h_i h_j\,g_{4(q-1)+2i+j+\delta}.$$ Here, $h\approx(-0.1294,\,0.2241,\,0.8365,\,0.4830)$ denotes the db2 low-pass filter, and $\delta$ accounts for filter alignment. Unlike Haar's equally weighted entries, db2 combines up to 10 neighboring entries with nonuniform weights. Thus, db2 captures overlapping local information rather than non-overlapping block means. The results are reported in **Appendix E.7, Table 19 (p. 34)**. | Wavelet | PPL ↓ | Time ↓ | Mem. ↓ | |---|---:|---:|---:| | Haar-2 | 22.47 | 4.03h | 0.55G | | Haar-3 | 22.63 | 4.06h | 0.51G | | db2-2 | **22.39** | 4.91h | 0.55G | | db2-3 | **22.48** | 5.03h | 0.51G | These results show that GWT remains effective beyond Haar-specific block averaging. Haar provides a favorable balance between training quality and computational efficiency. **Distinctions from related work and Fira comparison.** Related methods also reduce optimizer-state memory, but adopt different designs. FRUGAL combines stateful and state-free updates, while Fira uses SVD-based low-rank projection with norm-based residual scaling. Trion dynamically selects subspaces from a predefined DCT basis. In contrast, GWT maintains historical moments in a fixed wavelet approximation band and uses shared coarse second moments to precondition the current detail coefficients. It requires neither SVD nor dynamic subspace selection. We have clarified these distinctions in the **Introduction and Related Work (pp. 1–3)**. We further compare GWT directly with **Fira** at two nominal compression levels. Results appear in Table 2 (p. 7), with estimated memory in parentheses. | Method | 60M | 130M | 350M | 1.3B | |---|---:|---:|---:|---:| | Fira-1/4 | 30.43 (0.28G) | 23.18 (0.58G) | 16.72 (1.38G) | 13.89 (4.76G) | | **GWT-2** | **29.35** (0.27G) | **22.47** (0.55G) | **16.29** (1.24G) | **13.50** (4.41G) | | Fira-1/8 | 30.71 (0.26G) | 23.31 (0.53G) | 16.74 (1.23G) | 14.11 (4.15G) | | **GWT-3** | **29.81** (0.26G) | **22.63** (0.51G) | **16.35** (1.09G) | **13.48** (3.80G) | GWT achieves lower PPL than Fira across the evaluated scales, with comparable or lower estimated memory consumption.
AuthorsOpenReview ↗

Response to Reviewer zqgq: Questions (Q1–Q2)

Q1: Exact Adam Settings, Larger-Scale Controls, and Baseline Hyperparameters

As detailed in W1, we report the final PPL values, selected learning rates, and module groupings for the Adam controls. The comparison now extends to 350M and 1.3B, with a finer search around the initially selected scaling factor for 60M, 130M, and 350M. The final scaling factors are 0.20 for 60M and 0.25 for 130M and 350M. We have also supplemented the Muon, Adam-mini, and LDAdamW hyperparameters in Appendix D.2 and Table 8. We hope these additional experiments and clarifications address your concerns.

Q2: Block-Mean Interpretation and Coarse/Residual Ablations

We agree with the proposed block-mean interpretation, which is now explicitly stated in Section 3.2 (p. 4).

To directly evaluate the contributions of historical coarse information and residual preconditioning, we conduct three additional ablations on LLaMA-130M:

  • w/o Coarse-M: Replaces historical coarse momentum with current coarse coefficients while retaining the second-moment estimates.
  • w/o Coarse: Removes the coarse update direction while retaining coarse statistics for detail preconditioning.
  • Res-RMS: Replaces shared coarse second-moment preconditioning with instantaneous block-wise residual RMS.

The results are summarized below.

MethodPPL ↓
GWT-222.47
w/o Coarse-M23.03
w/o Coarse24.28
Res-RMS23.67

All three ablations degrade validation perplexity, supporting the contributions of historical coarse momentum, the coarse update direction, and shared second-moment preconditioning. Detailed settings and analysis are provided in Appendix E.7, Table 18 (p. 33).

Thank you for the detailed response and the new experiments. They address my main concerns. The module-wise Adam controls at 350M and 1.3B show GWT matching or slightly improving on tuned Adam with about 44% to 45% less memory. The block-mean reduction and the new ablations, especially Res-RMS, show what the coarse momentum and shared preconditioner add. Two small suggestions for the final version. At 1.3B the selected α of 0.25 is still at the edge of the grid, and the 0.15 to 0.27 PPL gaps from 60M to 350M come from single runs, so I suggest describing them as comparable. Please also state in Related Work how GWT’s residual scaling differs from FRUGAL and Fira. I have raised my score to 3.

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. We are pleased that our additional experiments and clarifications have addressed your main concerns. Following your suggestions, we will incorporate the relevant experimental results and comparisons into the main paper and further clarify the presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.