- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a optimizer called GWT, a memory efficient-based optimizer for LLM training. Unlike low-rank projection methods, e.g., GaLore, GWT applies a multi-level Harr wavelet transform to matrix gradients and maintains optimizer states only for coarse approximatio coefficients, while preserving the instantaneous detail coefficients during gradient reconstruction. This paper also provides theoretical analisys of gradient information preservation, directional alignment, and convergence under bounded preconditioning and coarse-momentum coherence assumptions. Experiments demonstrate competitive performance compared with existing memory-efficient optimizers.
Critical strengths
-
(S1) The paper proposes a novel optimizer-state compression approach that differs from conventional low-rank projection methods, demonstrating clear innovation and research value.
-
(S2) The proposed technique incurs low computational overhead and retains competitive training performance across multiple model scales and tasks, with straightforward integration into existing optimization frameworks.
Critical weaknesses
-
(W1) In Appendix E.6 (Figure 11), the authors found that when Full Adam uses the same module-wise learning rate, its convergence performance improves significantly, and its performance approaches or, on small-scale datasets, exceeds that of GWT. Therefore, it is currently unclear to what extent GWT’s performance improvement actually stems from the wavelet transform itself.
-
(W2) Table 12 shows that the Detail coefficient accounts for 93.7%–96.7% of the inner product between the gradient and the update direction. This indicates that Detail plays an important role in the update direction, but the actual contribution of Coarse’s historical momentum remains unclear. And some typos, e.g., "GWT-2 achieves 0.532K tokens ...... about 6% higher throughput than GaLore-1/4...", looks like not 6%
Questions for authors
-
(Q1) Could the authors provided some experiments with Fira & SWAN baseline?
-
(Q2) Could the authors provide more independent validation or ablation experiments for GWT's performance from which methods?
-
(Q3) The authors use Haar wavelets to decompose the gradient into coarse and detail components, but do not fully explain why Haar wavelets must be chosen over other orthogonal transforms. Could the authors add comparative experiments with other methods?
-
(Q4) Based on W2. Could the authors provide ablation experiments showing the necessity of Coarse?
4 comments
Response to Reviewer YmrT: Questions (Q1–Q4)
Q1: Can You Compare GWT with Fira and SWAN?
We have added Fira to our main experiments, evaluating two compression levels across LLaMA-60M to 1.3B. We have also incorporated the published results of SWAN into Table 2, with its memory re-estimated under our BF16 model configurations.
The results below report PPL, with estimated model-and-optimizer-state memory in parentheses. Fira results are from our reproduced experiments, while SWAN results are taken from the cited publication.
| Method | 60M | 130M | 350M | 1.3B |
|---|---|---|---|---|
| Fira-1/4 | 30.43 (0.28G) | 23.18 (0.58G) | 16.72 (1.38G) | 13.89 (4.76G) |
| GWT-2 | 29.35 (0.27G) | 22.47 (0.55G) | 16.29 (1.24G) | 13.50 (4.41G) |
| Fira-1/8 | 30.71 (0.26G) | 23.31 (0.53G) | 16.74 (1.23G) | 14.11 (4.15G) |
| GWT-3 | 29.81 (0.26G) | 22.63 (0.51G) | 16.35 (1.09G) | 13.48 (3.80G) |
| SWAN‡ (literature) | 30.59 (0.25G) | 22.61 (0.46G) | 16.63 (1.00G) | 13.56 (3.20G) |
GWT achieves lower PPL than Fira at both compression levels, with comparable or lower estimated memory. Compared with SWAN, GWT-2 achieves lower reported PPL across all four model scales, while SWAN requires less estimated memory.
Following the reviewers' suggestions, we have also included SinkGD and SCALE in Table 2, with their published PPL and consistently re-estimated memory. We discuss these comparisons in Section 4.1, highlighting the differences in training quality and memory efficiency. We have also expanded Related Work to clarify their methodological distinctions from GWT, particularly its use of compressed historical adaptive states and instantaneous detail coefficients.
Q2: Can You Provide Independent Validation and Ablations?
We have added component ablations and wavelet-basis comparisons on LLaMA-130M. They test the contributions of coarse momentum, coarse updates, detail preconditioning, and the choice of wavelet basis. The results appear in Appendix E.8 (pp. 32–34) and are detailed under Q3 and Q4.
Q3: Why Haar over Other Orthogonal Transforms?
We choose Haar as the default wavelet for theoretical simplicity and computational efficiency. Its approximation coefficients correspond to normalized block means. This structure supports the detail-energy bound in Lemma 3.1, while the two-tap filter enables efficient decomposition and reconstruction.
We additionally compare Haar with Daubechies-2 (db2) at decomposition levels 2 and 3. Unlike Haar, db2 uses a four-tap filter with nonuniform weights and captures overlapping local information.
The results are reported in Appendix E.7, Table 19 (p. 34).
| Wavelet | PPL ↓ | Time ↓ | Mem. ↓ |
|---|---|---|---|
| Haar-2 | 22.47 | 4.03h | 0.55G |
| Haar-3 | 22.63 | 4.06h | 0.51G |
| db2-2 | 22.39 | 4.91h | 0.55G |
| db2-3 | 22.48 | 5.03h | 0.51G |
Both db2 configurations achieve slightly lower PPL than Haar at the corresponding levels but require more training time. Thus, GWT is not restricted to Haar, while Haar provides a favorable balance between theoretical tractability, training quality, and computational efficiency.
Q4: Is the Coarse Component Necessary?
To directly evaluate the role of the coarse component, we conduct three targeted ablations on LLaMA-130M:
- w/o Coarse-M: Replaces historical coarse momentum with current coarse coefficients while retaining the second-moment estimates.
- w/o Coarse: Removes the coarse update direction while retaining coarse statistics for detail preconditioning.
- Res-RMS: Replaces shared coarse second-moment preconditioning with instantaneous block-wise residual RMS.
The results are summarized below.
| Method | PPL ↓ |
|---|---|
| GWT-2 | 22.47 |
| w/o Coarse-M | 23.03 |
| w/o Coarse | 24.28 |
| Res-RMS | 23.67 |
All three ablations degrade validation PPL. Removing the coarse update causes the largest degradation, confirming its importance even though the detail coefficients contribute substantially to gradient-update alignment.
These results support the complementary roles of historical coarse momentum, the coarse update direction, and shared detail preconditioning. Detailed settings and analysis are provided in Appendix E.7, Table 18 (p. 33).
Thanks to the author's detailed response, my concerns have been solved. I maintain my initial score.
We are glad that the additional experiments addressed your concerns. Your suggestions helped us make the practical implications of the framework clearer. We appreciate your time and constructive feedback.
Response to Reviewer YmrT: Weaknesses (W1–W2)
We sincerely thank the reviewer for recognizing GWT's novelty, low computational overhead, and competitive performance. We appreciate the constructive suggestions and address the identified weaknesses (W) and questions (Q) below.
W1: Does the Wavelet Transform Explain the Performance Gains?
We agree that the performance gap between GWT and vanilla Adam cannot be attributed solely to the wavelet transform. Module-wise learning rates substantially improve Adam, as our experiments confirm. We have therefore refined the performance claims in Section 4.1 and Figure 2 to emphasize competitive training quality with lower optimizer-state memory.
Building on our existing module-wise Adam comparisons on LLaMA-60M and 130M, we extend the evaluation to 350M and 1.3B using the same learning-rate strategy as GWT:
lr_{\mathrm{others}}=lr.$$ We further refine the scaling-factor search for **60M–350M**, additionally evaluating $\alpha\in\{0.1,0.2,0.3\}$ with the base learning rate fixed at 0.01. The expanded search selects $\alpha=0.20$ for 60M and $\alpha=0.25$ for 130M and 350M. The results below report PPL, with estimated model-and-optimizer-state memory in parentheses. | Method | 60M | 130M | 350M | 1.3B | |---|---:|---:|---:|---:| | Adam (Vanilla) | 30.60 (0.34G) | 25.08 (0.80G) | 18.75 (2.20G) | 16.10 (8.03G) | | Adam (Module-wise) | **29.20** (0.34G) | 22.74 (0.80G) | 16.46 (2.20G) | 14.17 (8.03G) | | GWT-2 | 29.35 (0.27G) | **22.47** (0.55G) | **16.29** (1.24G) | **13.50** (4.41G) | These results show that module-wise learning rates account for much of the original performance gap with vanilla Adam. Module-wise Adam slightly outperforms GWT at 60M, while GWT achieves lower PPL at larger scales. Importantly, GWT's primary contribution lies in **wavelet-based optimizer-state compression**. By maintaining historical adaptive states only for the coarse coefficients while retaining the current detail coefficients, GWT substantially reduces memory without sacrificing training quality. For example, at 350M and 1.3B, GWT reduces estimated model-and-optimizer-state memory from 2.20G to 1.24G and from 8.03G to 4.41G, respectively, while achieving lower PPL than module-wise Adam. Detailed settings and results appear in **Table 2 and Appendix E.6**. ### **W2: Coarse-Momentum Contribution and Throughput Correction** We agree that the large contribution of detail coefficients to gradient-update alignment does not establish whether historical coarse momentum is necessary. To investigate this, we added ablations of coarse momentum, the coarse update direction, and detail preconditioning. All three variants increase validation PPL. The results are reported in **Appendix E.8, Table 19 (p. 33)** and discussed under Q4. We also thank the reviewer for identifying the throughput error. The difference between 0.532K and 0.526K tokens/s/GPU is approximately **1.1%, not 6%**. We have removed the incorrect claim from **Section 4.1 (p. 7)**.