- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies low-rank gradient projection for memory-efficient LLM training and proposes a new method FLAME. The method removes the first-moment state, uses a fresh Gaussian projection at every step, continues to use the previous second-moment estimate, and computes projected gradients directly from compressed input activations. The paper provides a convergence guarantee and evaluates performance of the proposed method in pretraining and SFT tasks.
Critical strengths
- FLAME combines low-rank optimization with activation compression in a simple way. The method looks interesting.
- The paper analyzes how optimizer moments behave under random projections and provides a standard convergence guarantee for the proposed discrete-time update. It is nice to have a theory here, although the assumption may be a little bit strong (requiring the a.s. bound of full gradient).
Critical weaknesses
- The experimental evidence is limited, in terms of experiment scale and settings, and important settings are missing. Pretraining is evaluated only on a 1B model for 10,000 steps, with relatively high final perplexity (Figure 1b), suggesting a very under-trained result. This is insufficient to establish effectiveness over longer training runs. Fine-tuning is limited to task-specific benchmarks, whose scales are generally small and are usually used only for evaluation. More importantly, for an optimizer paper, the authors report no effort for ablating the hyper parameters, not even how the learning rate is chosen, which can largely affect the results. Key pretraining details, including sequence length, total training tokens, and the learning-rate schedule, and the hyperparmeters such as , are also missing from the manuscript, which affects the reliability of the experiment results.
- The method design and the analysis seems not consistent enough to me. The analysis motivates removing first-order momentum, but Theorem 2(b) only shows that the expected row-wise sum of the second-moment state accumulates historical gradient norms. There is still clear gap from reusing coordinate-wise second-moment estimates remains effective. In addition, activation compression is adapted from AGO-CC, but its usage here worths more explanation and understanding. From the paper side, it is like the activation matrix just suddenly appears, which looks confusing.
Questions for authors
- Can the authors explain or provide evidence for why the previous second-moment state remains useful across independent Gaussian projections? Theorem 2(b) does not seem sufficient to justify this step.
- Could the authors provide the missing pretraining settings and controlled ablations, especially learning-rate and rank sweeps, and comparisons that isolate first-moment removal from activation compression under otherwise matched settings, which can give a more comprehensive ablation to the method?
6 comments
Response to Reviewer 6Q61 Part(2/3)
Controlled Ablations of Momentum Removal and Activation Compression. Thank you for this helpful suggestion. In our pretraining setup, parameter memory accounts for a relatively small fraction of total GPU memory, so we conduct controlled ablations on fine-tuning tasks to better quantify the individual contributions of FLAME's optimizer design and AGO-CC. In our response to Reviewer RMoA (Weakness 1 & Question 1), we provide detailed ablation analyses of training throughput and memory savings. We also explain why activation–gradient co-compression in FLAME is an integral part of our theory-guided method design. We would greatly appreciate it if you could take the time to read that response.
Weakness 2 & Question 1: Reusing Second-Moment States and Activation Compression
Why first- and second-moment states are treated differently. Thank you for highlighting the need to explain this connection more clearly. We analyze the two states separately: under its fixed-gradient setting, Theorem 2(a) shows that reconstructed first-moment contributions from previous periods vanish in expectation, motivating removal of directional history across random subspaces. Theorem 2(b), in contrast, shows that second-moment states retain historical row energy in expectation, motivating retention of scale information. These are distinct roles, so removing the first moment while retaining the second is consistent with the analysis. We agree that the row-sum identity alone does not fully justify coordinatewise reuse; below we explain its statistical meaning and characterize the actual update.
We follow the notation of Appendix B.6, suppressing the layer index for a single linear layer with weight and projection rank . Let contain all randomness through step , let be the current stochastic gradient, and let be the full true gradient. We write for the corresponding layer’s -th gradient row and for column of .
Why historical second moments remain meaningful. Theorem 2(b) motivates retaining historical energy information, but is not the sole justification for coordinatewise reuse. Each fresh Gaussian projection column satisfies
Consequently, every coordinate of has the same unconditional expectation within a row, equal to the exponentially weighted expected historical row energy divided by . These coordinates record a common energy scale rather than fixed parameter directions; this does not assume that their realized values are equal.
The actual coordinatewise update. As in Appendix B.6, define the historical normalization used at step as
For , this is the direction used in the actual parameter update. Conditional unbiasedness and the fresh projector’s independence from the current gradient and history give
Thus, reuse requires no alignment between old and new projection coordinates: the actual elementwise normalization yields a positive row-wise scale in conditional expectation. Elementwise differences are also retained in the exact second moment:
These identities follow from the Gaussian-moment and conditional-descent lemmas in Appendix B.6. They directly characterize the coordinatewise update, rather than inferring its behavior from the row sum alone. Theorem 6’s proof further controls the historical scale’s dependence on the current true gradient. We will clarify this connection.
Response to Reviewer 6Q61 Part(3/3)
How activation compression fits. Thank you for pointing out that this connection needs clearer explanation. Activation–gradient co-compression is an integral part of FLAME’s theory-guided design: our analysis motivates gradient-independent Gaussian projections, which allow us to move projection from the optimizer stage to forward-pass activations while preserving the adaptive update.
For a linear layer , denotes its input activation. With optimizer-only projection, we save the full , compute the full weight gradient during backpropagation, and then form for historical second-moment normalization and mapping back to parameter space.
In FLAME’s integrated pipeline, the gradient-independent projector is available during the forward pass. We save and directly compute
For the same projector, both implementations therefore produce exactly the same projected gradient and optimizer update. The integrated pipeline avoids saving full activations for the weight-gradient computation and materializing the full weight gradient, while preserving the forward and input-gradient computations. Relative to optimizer-only projection, it introduces no additional approximation. Building on the co-compression strategy established by AGO-CC, our contribution is to integrate it with a theory-guided adaptive optimizer and characterize the resulting actual adaptive updates. We will clarify this design connection and the relationship to AGO-CC.
Gradient-Boundedness Assumption
Related boundedness conditions already appear in Adam-type convergence analyses. ADOPT [1, Assumption 2.5] assumes uniformly bounded stochastic-gradient second moments and explicitly notes that this condition bounds both true gradients and noise variance. Défossez et al. [2, Section 2.3, Eq. (7)] impose the stronger requirement that stochastic-gradient samples are uniformly almost surely bounded over all parameter values. We instead bound true gradients along the iterates and conditional noise variance, without requiring almost-sure bounds on individual stochastic or projected gradients.
Thus, our assumptions provide the same type of second-moment control as ADOPT while relaxing Défossez et al.’s samplewise boundedness requirement and permitting unbounded noise with finite conditional variance. We will clarify these distinctions.
References
[1] Shohei Taniguchi et al. ADOPT: Modified Adam Can Converge with Any with the Optimal Rate. NeurIPS, 2024.
[2] Alexandre Défossez et al. A Simple Convergence Proof of Adam and Adagrad. TMLR, 2022.
Response to Reviewer 6Q61 Part(1/3)
We sincerely thank you for your constructive feedback and for recognizing our theoretical results. We address your concerns as follows.
Weakness 1 & Question 2: Experimental Clarifications and Hyperparameter Ablations
Experimental Settings. Thanks for your feedback. We sincerely apologize for the incomplete reporting of our hyperparameters and experimental settings. For pretraining, we use a maximum sequence length of 256, following the pretraining setups in GaLore [1] and LDAdam [5]. A per-device batch size of 64 with 8 gradient accumulation steps yields an effective batch size of 512. We train for 10,000 optimizer updates, corresponding to approximately 1.31B tokens. We set the initial learning rate to 5×10−4 with linear decay. For FLAME, we set β2=0.999 and ϵ=10−6. For the baseline optimizers, we additionally use β1=0.9, with the same β2 and ϵ settings. We will include these details in the next version. We will discuss hyperparameter selection and ablations below.
Pretraining Evaluation. Thank you for your suggestion. We agree that longer training runs and larger models would provide additional evidence of FLAME’s effectiveness. Evaluating larger models over longer training runs across multiple methods requires substantial computational resources. Within our current computational budget, we evaluate FLAME on a 1B model for 10,000 optimizer updates. For context, GaLore [1] and Online Subspace Descent [2] also report evaluations on LLaMA 1B for 10,000 steps. Other studies, such as GoLore [3] and FLORA [4], report pretraining results on smaller models (60M and 350M). In our evaluation, FLAME’s validation perplexity (red curve in Figure 1b) rapidly approaches that of Adam and GaLore, and the curves flatten toward the end of training. We aim to extend our pretraining evaluation to larger scales in the next version. We hope these comparisons provide helpful context for current pretraining results.
Fine-Tuning Benchmarks. Thank you for raising this concern. For fine-tuning, most prior studies follow GaLore [1] in using GLUE, while some also evaluate additional tasks, such as GSM8K in LDAdam [5] and commonsense reasoning tasks, including SIQA, in Fira [6]. Our evaluation covers all GLUE subtasks and additionally includes GSM8K, MATH, and CommonsenseQA, datasets from other domains with higher reasoning demands. This provides a broad and challenging assessment of FLAME’s fine-tuning performance. MATH also continues to be used as a training resource in recent research, including SOAR [7]. In the context of prior optimizer studies, we believe our fine-tuning evaluation is reasonably comprehensive. We also provide additional experiments across multiple learning rates and random seeds in our response to Reviewer RMoA (Weakness 2 & Question 2). We would appreciate it if you could take the time to read that response and reassess the adequacy of our fine-tuning evaluation.
Learning-Rate and Rank Sweeps for Pretraining. Following prior work such as LDAdam [5], we used an initial learning rate of 5×10−4 for pretraining. To examine this choice, we compared the full-rank Adam baseline at this learning rate and two alternatives, 10−3 and 10−5. The resulting perplexities are presented below.
Among the tested learning rates, 5×10−4 yields the lowest perplexity, supporting our choice of the initial learning rate.
Most prior studies use a projection rank of 512 for pretraining 1B models [1, 2, 6]. GaLore [1] also reports results with a rank of 1024. We conducted experiments at rank 1024 and found that the perplexity curves closely matched those obtained at rank 512. For example, GaLore achieved perplexities of 35.19 and 35.06 at ranks 512 and 1024, respectively. We therefore chose rank 512, consistent with prior studies. We will include these additional ablation results and clarify our hyperparameter choices in the next version.
References
[1] Jiawei Zhao et al. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML, 2024.
[2] Kaizhao Liang et al. Memory-Efficient LLM Training with Online Subspace Descent. NeurIPS, 2024.
[3] Yutong He et al. Subspace Optimization for Large Language Models with Convergence Guarantees. ICML, 2025.
[4] Yongchang Hao et al. FLORA: Low-Rank Adapters Are Secretly Gradient Compressors. ICML, 2024.
[5] Thomas Robert et al. LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. ICLR, 2025.
[6] Xi Chen et al. Fira: Can We Achieve Full-rank Training of LLMs under Low-rank Constraint? NeurIPS, 2025.
[7] Shobhita Sundaram et al. Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability. ICML, 2026.