acceptodds
Under review as a conference paper at ICLR 2027

FLAME: First-Moment-Free Low-Rank Gradient Projection with Adam-Style Convergence Guarantees

Abstract

Low-rank gradient projection has emerged as a promising approach to optimizer-state compression, but existing methods still suffer from unstable optimization and high computational overhead. To understand these limitations, we develop a unified analytical framework and prove that, under independent random projections, first-order momentum retains only within-period history whereas the second-order state contains information from all periods, and that the induced adaptive updates effectively preserve direction in expectation. Guided by these findings, we propose FLAME (First-Moment-Free Low-Rank Adaptive Moment Estimation), which updates parameters using the current Gaussian-projected gradient normalized by the previous low-rank second-order estimate. It enables effective optimization while halving optimizer-state storage relative to existing Adam-based low-rank gradient projection methods and substantially reducing computational overhead. We prove an convergence rate for FLAME under discrete-time Adam-style optimization. FLAME further incorporates activation compression to reduce training memory. Experiments on LLaMA-3 fine-tuning and pretraining demonstrate competitive task performance and favorable convergence behavior. In terms of system efficiency, FLAME achieves the lowest peak memory among competing low-rank gradient-projection methods, reducing memory by 32-53% relative to full-rank Adam, while improving throughput by 23-45% over the fastest competitor.

Primary area:
optimization
Keywords:
Low-rank gradient projection, Memory-efficient training, Large language models, Adaptive optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6Q61OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies low-rank gradient projection for memory-efficient LLM training and proposes a new method FLAME. The method removes the first-moment state, uses a fresh Gaussian projection at every step, continues to use the previous second-moment estimate, and computes projected gradients directly from compressed input activations. The paper provides a convergence guarantee and evaluates performance of the proposed method in pretraining and SFT tasks.

Critical strengths
  1. FLAME combines low-rank optimization with activation compression in a simple way. The method looks interesting.
  2. The paper analyzes how optimizer moments behave under random projections and provides a standard O(T−1/2)O(T^{-1/2}) convergence guarantee for the proposed discrete-time update. It is nice to have a theory here, although the assumption may be a little bit strong (requiring the a.s. bound of full gradient).
Critical weaknesses
  1. The experimental evidence is limited, in terms of experiment scale and settings, and important settings are missing. Pretraining is evaluated only on a 1B model for 10,000 steps, with relatively high final perplexity (Figure 1b), suggesting a very under-trained result. This is insufficient to establish effectiveness over longer training runs. Fine-tuning is limited to task-specific benchmarks, whose scales are generally small and are usually used only for evaluation. More importantly, for an optimizer paper, the authors report no effort for ablating the hyper parameters, not even how the learning rate is chosen, which can largely affect the results. Key pretraining details, including sequence length, total training tokens, and the learning-rate schedule, and the hyperparmeters such as β2\beta_2, are also missing from the manuscript, which affects the reliability of the experiment results.
  2. The method design and the analysis seems not consistent enough to me. The analysis motivates removing first-order momentum, but Theorem 2(b) only shows that the expected row-wise sum of the second-moment state accumulates historical gradient norms. There is still clear gap from reusing coordinate-wise second-moment estimates remains effective. In addition, activation compression is adapted from AGO-CC, but its usage here worths more explanation and understanding. From the paper side, it is like the activation matrix just suddenly appears, which looks confusing.
Questions for authors
  1. Can the authors explain or provide evidence for why the previous second-moment state remains useful across independent Gaussian projections? Theorem 2(b) does not seem sufficient to justify this step.
  2. Could the authors provide the missing pretraining settings and controlled ablations, especially learning-rate and rank sweeps, and comparisons that isolate first-moment removal from activation compression under otherwise matched settings, which can give a more comprehensive ablation to the method?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6Q61 Part(1/3)

We sincerely thank you for your constructive feedback and for recognizing our theoretical results. We address your concerns as follows.

Weakness 1 & Question 2: Experimental Clarifications and Hyperparameter Ablations

Experimental Settings. Thanks for your feedback. We sincerely apologize for the incomplete reporting of our hyperparameters and experimental settings. For pretraining, we use a maximum sequence length of 256, following the pretraining setups in GaLore [1] and LDAdam [5]. A per-device batch size of 64 with 8 gradient accumulation steps yields an effective batch size of 512. We train for 10,000 optimizer updates, corresponding to approximately 1.31B tokens. We set the initial learning rate to 5×10−45\times10^{-4} with linear decay. For FLAME, we set β2=0.999\beta_2=0.999 and ϵ=10−6\epsilon=10^{-6}. For the baseline optimizers, we additionally use β1=0.9\beta_1=0.9, with the same β2\beta_2 and ϵ\epsilon settings. We will include these details in the next version. We will discuss hyperparameter selection and ablations below.

Pretraining Evaluation. Thank you for your suggestion. We agree that longer training runs and larger models would provide additional evidence of FLAME’s effectiveness. Evaluating larger models over longer training runs across multiple methods requires substantial computational resources. Within our current computational budget, we evaluate FLAME on a 1B model for 10,000 optimizer updates. For context, GaLore [1] and Online Subspace Descent [2] also report evaluations on LLaMA 1B for 10,000 steps. Other studies, such as GoLore [3] and FLORA [4], report pretraining results on smaller models (60M and 350M). In our evaluation, FLAME’s validation perplexity (red curve in Figure 1b) rapidly approaches that of Adam and GaLore, and the curves flatten toward the end of training. We aim to extend our pretraining evaluation to larger scales in the next version. We hope these comparisons provide helpful context for current pretraining results.

Fine-Tuning Benchmarks. Thank you for raising this concern. For fine-tuning, most prior studies follow GaLore [1] in using GLUE, while some also evaluate additional tasks, such as GSM8K in LDAdam [5] and commonsense reasoning tasks, including SIQA, in Fira [6]. Our evaluation covers all GLUE subtasks and additionally includes GSM8K, MATH, and CommonsenseQA, datasets from other domains with higher reasoning demands. This provides a broad and challenging assessment of FLAME’s fine-tuning performance. MATH also continues to be used as a training resource in recent research, including SOAR [7]. In the context of prior optimizer studies, we believe our fine-tuning evaluation is reasonably comprehensive. We also provide additional experiments across multiple learning rates and random seeds in our response to Reviewer RMoA (Weakness 2 & Question 2). We would appreciate it if you could take the time to read that response and reassess the adequacy of our fine-tuning evaluation.

Learning-Rate and Rank Sweeps for Pretraining. Following prior work such as LDAdam [5], we used an initial learning rate of 5×10−45\times10^{-4} for pretraining. To examine this choice, we compared the full-rank Adam baseline at this learning rate and two alternatives, 10−310^{-3} and 10−510^{-5}. The resulting perplexities are presented below.

MethodLR = 10−310^{-3}LR = 5×10−45\times10^{-4}LR = 10−510^{-5}
Adam75.2733.582316.81

Among the tested learning rates, 5×10−45\times10^{-4} yields the lowest perplexity, supporting our choice of the initial learning rate.

Most prior studies use a projection rank of 512 for pretraining 1B models [1, 2, 6]. GaLore [1] also reports results with a rank of 1024. We conducted experiments at rank 1024 and found that the perplexity curves closely matched those obtained at rank 512. For example, GaLore achieved perplexities of 35.19 and 35.06 at ranks 512 and 1024, respectively. We therefore chose rank 512, consistent with prior studies. We will include these additional ablation results and clarify our hyperparameter choices in the next version.

References

[1] Jiawei Zhao et al. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML, 2024.

[2] Kaizhao Liang et al. Memory-Efficient LLM Training with Online Subspace Descent. NeurIPS, 2024.

[3] Yutong He et al. Subspace Optimization for Large Language Models with Convergence Guarantees. ICML, 2025.

[4] Yongchang Hao et al. FLORA: Low-Rank Adapters Are Secretly Gradient Compressors. ICML, 2024.

[5] Thomas Robert et al. LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. ICLR, 2025.

[6] Xi Chen et al. Fira: Can We Achieve Full-rank Training of LLMs under Low-rank Constraint? NeurIPS, 2025.

[7] Shobhita Sundaram et al. Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability. ICML, 2026.

AuthorsOpenReview ↗

Response to Reviewer 6Q61 Part(2/3)

Controlled Ablations of Momentum Removal and Activation Compression. Thank you for this helpful suggestion. In our pretraining setup, parameter memory accounts for a relatively small fraction of total GPU memory, so we conduct controlled ablations on fine-tuning tasks to better quantify the individual contributions of FLAME's optimizer design and AGO-CC. In our response to Reviewer RMoA (Weakness 1 & Question 1), we provide detailed ablation analyses of training throughput and memory savings. We also explain why activation–gradient co-compression in FLAME is an integral part of our theory-guided method design. We would greatly appreciate it if you could take the time to read that response.

Weakness 2 & Question 1: Reusing Second-Moment States and Activation Compression

Why first- and second-moment states are treated differently. Thank you for highlighting the need to explain this connection more clearly. We analyze the two states separately: under its fixed-gradient setting, Theorem 2(a) shows that reconstructed first-moment contributions from previous periods vanish in expectation, motivating removal of directional history across random subspaces. Theorem 2(b), in contrast, shows that second-moment states retain historical row energy in expectation, motivating retention of scale information. These are distinct roles, so removing the first moment while retaining the second is consistent with the analysis. We agree that the row-sum identity alone does not fully justify coordinatewise reuse; below we explain its statistical meaning and characterize the actual update.

We follow the notation of Appendix B.6, suppressing the layer index ll for a single linear layer with weight Xt∈Rm×n\mathbf{X}_t\in\mathbb{R}^{m\times n} and projection rank rr. Let Ft−1\mathcal{F}_{t-1} contain all randomness through step t−1t-1, let Gt\mathbf{G}_t be the current stochastic gradient, and let gt=∇f(θt)\mathbf{g}_t=\nabla f(\boldsymbol{\theta}_t) be the full true gradient. We write gt,a\mathbf{g}_{t,a} for the corresponding layer’s aa-th gradient row and ωt,j\boldsymbol{\omega}_{t,j} for column jj of Ωt\boldsymbol{\Omega}_t.

Why historical second moments remain meaningful. Theorem 2(b) motivates retaining historical energy information, but is not the sole justification for coordinatewise reuse. Each fresh Gaussian projection column satisfies

E[(Gt,aωt,j)2∣Ft−1,Gt]=∥Gt,a∥22/r.\mathbb{E}[(\mathbf{G}_{t,a}\boldsymbol{\omega}_{t,j})^2\mid\mathcal{F}_{t-1},\mathbf{G}_t]=\lVert\mathbf{G}_{t,a}\rVert_2^2/r.

Consequently, every coordinate of V^t−1\widehat{\mathbf{V}}_{t-1} has the same unconditional expectation within a row, equal to the exponentially weighted expected historical row energy divided by rr. These coordinates record a common energy scale rather than fixed parameter directions; this does not assume that their realized values are equal.

The actual coordinatewise update. As in Appendix B.6, define the historical normalization used at step tt as

Λt,a=diag⁡j!(((V^t−1)aj+ϵ)−1/2),Dt,a=Gt,aΩtΛt,aΩt⊤.\boldsymbol{\Lambda}_{t,a}=\operatorname{diag}_j\\!\left(((\widehat{\mathbf{V}}_{t-1})_{aj}+\epsilon)^{-1/2}\right), \qquad \mathbf{D}_{t,a}=\mathbf{G}_{t,a}\boldsymbol{\Omega}_t\boldsymbol{\Lambda}_{t,a}\boldsymbol{\Omega}_t^\top.

For t≥2t\ge 2, this is the direction used in the actual parameter update. Conditional unbiasedness and the fresh projector’s independence from the current gradient and history give

E[Dt,a∣Ft−1]=ζt,agt,a,ζt,a:=tr⁡Λt,ar>0.\mathbb{E}[\mathbf{D}_{t,a}\mid\mathcal{F}_{t-1}]=\zeta_{t,a}\mathbf{g}_{t,a}, \qquad \zeta_{t,a}:=\frac{\operatorname{tr}\boldsymbol{\Lambda}_{t,a}}{r}>0.

Thus, reuse requires no alignment between old and new projection coordinates: the actual elementwise normalization yields a positive row-wise scale in conditional expectation. Elementwise differences are also retained in the exact second moment:

E[∥Dt,a∥22∣Ft−1,Gt]=(tr⁡Λt,a)2+(n+1)tr⁡(Λt,a2)r2∥Gt,a∥22.\mathbb{E}[\lVert\mathbf{D}_{t,a}\rVert_2^2\mid\mathcal{F}_{t-1},\mathbf{G}_t]=\frac{(\operatorname{tr}\boldsymbol{\Lambda}_{t,a})^2+(n+1)\operatorname{tr}(\boldsymbol{\Lambda}_{t,a}^2)}{r^2}\lVert\mathbf{G}_{t,a}\rVert_2^2.

These identities follow from the Gaussian-moment and conditional-descent lemmas in Appendix B.6. They directly characterize the coordinatewise update, rather than inferring its behavior from the row sum alone. Theorem 6’s proof further controls the historical scale’s dependence on the current true gradient. We will clarify this connection.

AuthorsOpenReview ↗

Response to Reviewer 6Q61 Part(3/3)

How activation compression fits. Thank you for pointing out that this connection needs clearer explanation. Activation–gradient co-compression is an integral part of FLAME’s theory-guided design: our analysis motivates gradient-independent Gaussian projections, which allow us to move projection from the optimizer stage to forward-pass activations while preserving the adaptive update.

For a linear layer Zt=AtXt⊤\mathbf{Z}_t=\mathbf{A}_t\mathbf{X}_t^\top, At\mathbf{A}_t denotes its input activation. With optimizer-only projection, we save the full At\mathbf{A}_t, compute the full weight gradient Gt=(∇ZtLt)⊤At\mathbf{G}_t=(\nabla_{\mathbf{Z}_t}\mathcal{L}_t)^\top\mathbf{A}_t during backpropagation, and then form Ut=GtΩt\mathbf{U}_t=\mathbf{G}_t\boldsymbol{\Omega}_t for historical second-moment normalization and mapping back to parameter space.

In FLAME’s integrated pipeline, the gradient-independent projector is available during the forward pass. We save A~t=AtΩt\widetilde{\mathbf{A}}_t=\mathbf{A}_t\boldsymbol{\Omega}_t and directly compute

Ut=(∇ZtLt)⊤A~t=(∇ZtLt)⊤AtΩt=GtΩt.\mathbf{U}_t=(\nabla_{\mathbf{Z}_t}\mathcal{L}_t)^\top\widetilde{\mathbf{A}}_t=(\nabla_{\mathbf{Z}_t}\mathcal{L}_t)^\top\mathbf{A}_t\boldsymbol{\Omega}_t=\mathbf{G}_t\boldsymbol{\Omega}_t.

For the same projector, both implementations therefore produce exactly the same projected gradient and optimizer update. The integrated pipeline avoids saving full activations for the weight-gradient computation and materializing the full weight gradient, while preserving the forward and input-gradient computations. Relative to optimizer-only projection, it introduces no additional approximation. Building on the co-compression strategy established by AGO-CC, our contribution is to integrate it with a theory-guided adaptive optimizer and characterize the resulting actual adaptive updates. We will clarify this design connection and the relationship to AGO-CC.

Gradient-Boundedness Assumption

Related boundedness conditions already appear in Adam-type convergence analyses. ADOPT [1, Assumption 2.5] assumes uniformly bounded stochastic-gradient second moments and explicitly notes that this condition bounds both true gradients and noise variance. Défossez et al. [2, Section 2.3, Eq. (7)] impose the stronger requirement that stochastic-gradient samples are uniformly almost surely bounded over all parameter values. We instead bound true gradients along the iterates and conditional noise variance, without requiring almost-sure bounds on individual stochastic or projected gradients.

Thus, our assumptions provide the same type of second-moment control as ADOPT while relaxing Défossez et al.’s samplewise boundedness requirement and permitting unbounded noise with finite conditional variance. We will clarify these distinctions.

References

[1] Shohei Taniguchi et al. ADOPT: Modified Adam Can Converge with Any β2\beta_2 with the Optimal Rate. NeurIPS, 2024.

[2] Alexandre Défossez et al. A Simple Convergence Proof of Adam and Adagrad. TMLR, 2022.

Reviewer_i6maOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

FLAME is a low-rank gradient-projection optimizer that drops the first moment, samples a fresh Gaussian projection each step, and normalizes by the previous second-moment estimate, as in ADOPT. Activation co-compression, adopted from AGO-CC, avoids materializing full weight gradients. The paper analyzes moments under random projection, proves an O(1/T)O(1/\sqrt{T}) rate, and evaluates LLaMA-3 fine-tuning and pretraining.

Critical strengths

The design follows from the analysis: Theorem 2 motivates dropping cross-period momentum, and previous-state normalization makes the expected update a per-row rescaling of the gradient. The convergence proof is correct and handles gradient-history dependence explicitly. Among projection methods, FLAME has the lowest peak memory and higher throughput than APOLLO.

Critical weaknesses
  1. The normalization comes from ADOPT and the co-compression from AGO-CC, and co-compression drives most of the memory advantage: gradient memory falls from 6.03 to 0.75 GB (3B) and from 14.96 to 2.07 GB (8B), while halving the optimizer state saves at most 0.94 and 2.51 GB. Running a random-projection baseline such as GoLore with co-compression would isolate FLAME's optimizer.
  2. The theory is stated more broadly than proved. The small-remainder reading of Theorem 3 needs Lemma 9's coordinatewise condition, shown only with C=1C = 1 and unattainable with moderate CC in the first period, where V0=0V_0 = 0. Theorems 2-3 fix gradients that depend on the projector when τ>1\tau > 1. Theorem 6's bound grows like n/r/ϵ\sqrt{n/r}/\sqrt{\epsilon}, so it matches Adam only in TT.
Questions for authors
  1. How are unprojected parameters (embedding, LM head) optimized in FLAME and in each baseline? FLAME's optimizer state (0.75/2.07 GB) is about one bf16 copy of those weights.
  2. What does momentum add at matched τ\tau? Table 5's FLAME-Momentum also changes τ\tau (200 vs. 1).
3 comments
AuthorsOpenReview ↗

Response to Reviewer i6ma Part(1/3)

We sincerely thank you for your constructive feedback and for recognizing our theoretical results and FLAME’s efficiency. We address your concerns as follows.

Weakness 1: Ablation Study of FLAME’s Memory Savings

Thanks for your suggestion. To isolate the memory savings attributable to FLAME’s optimizer design, we evaluate FLAME-OptOnly with activation–gradient co-compression disabled on LLaMA-3 8B/GSM8K.

MethodPeak GPU Memory (GB)
Adam62.89
GaLore36.52
FLAME-OptOnly33.98
FLAME29.25
GoLore with AGO-CC32.13

As shown, FLAME-OptOnly reduces peak GPU memory by 46.0% relative to Adam and 7.0% relative to GaLore, demonstrating the independent memory savings from FLAME’s optimizer design. Activation–gradient co-compression further increases the memory reduction relative to Adam by 7.5 percentage points, bringing the overall reduction to 53.5%. With co-compression enabled for both methods, FLAME also reduces peak GPU memory by 2.88 GB (9.0%) relative to GoLore with AGO-CC, further demonstrating the advantages of its optimizer design.

Theoretical and Methodological Contributions. FLAME follows ADOPT in using the previous second-moment estimate for normalization. Beyond this, our contributions include an analysis of optimizer states and actual adaptive updates that motivates first-moment removal and provides a theoretical justification for FLAME’s use of random projections. Moreover, AGO-CC in FLAME is an integral part of this theory-guided method design. FLAME uses a Gaussian random projection matrix Ω\Omega that is independent of the current gradient, allowing activation compression during the forward pass. By contrast, baseline low-rank gradient-projection methods commonly derive their projection matrices from the current gradient using SVD-based methods, making it difficult to apply AGO-CC. We discuss these contributions in detail in our response to Reviewer RMoA (Weakness 1 & Question 1). We would greatly appreciate it if you could take the time to read it.

Weakness 2: Scope of the Theoretical Results

Theorem 3 and the small remainder. Theorem 3 aims to explain why random projections can yield effective adaptive updates despite larger gradient-reconstruction errors than SVD-based projections. Crucially, normalization in the projected space followed by mapping back to the parameter space generally differs from reconstructing the gradient first and then normalizing in the parameter space. Gradient-reconstruction accuracy therefore cannot directly measure the quality of the final update. Theorem 3 characterizes this distinction by decomposing the actual update into a reference term and a remainder. Under the fixed-gradient setting, the expected reference term preserves the direction of each row of the current period's momentum, changing only its magnitude through scaling determined by the pre-period second-moment state. The remainder captures the correction caused by current-period gradients entering the normalization denominator. Lemma 9 provides a sufficient condition for controlling this correction, rather than a prerequisite for Theorem 3. It holds for any C>0C>0 satisfying the comparison condition; C=1C=1 is only a numerical illustration.

Empirical statistics. During LLaMA-3 8B training on GSM8K, we collected training snapshots and computed the coordinatewise ratios

ct,aj=(Ut,a)j2(Vt−1)aj+ϵ,Cp(t)=Qp!(ct,ajall projected layers,,a,j).c_{t,aj}=\frac{(\mathbf{U}_{t,a})_j^2}{(\mathbf{V}_{t-1})_{aj}+\epsilon}, \qquad C_p(t)=Q_p\\!\left(\\{c_{t,aj}\\}_{\text{all projected layers},\\,a,j}\right).

Here Vt−1\mathbf{V}_{t-1} is the pre-update second-moment state without bias correction. Each snapshot contains 44,040,192 ratios. We computed C0.95(t)C_{0.95}(t) for each snapshot and found max⁡tC0.95(t)=1.2111\max_t C_{0.95}(t)=1.2111. Thus, in the collected snapshots, the current projected-gradient energy does not substantially exceed the historical second-moment scale for the vast majority of coordinates, providing empirical support for the comparison condition's plausibility at typical coordinates.

The initial period. We agree that, when V0=0\mathbf{V}_0=0, Lemma 9's condition reduces to (Uq,a)j2≤Cϵ(\mathbf{U}_{q,a})_j^2\le C\epsilon, so a small remainder cannot generally be guaranteed. The first period builds the second-moment normalization scale from zero, whereas the reference update uses only ϵ\sqrt{\epsilon} in its denominator. A substantial adjustment of the actual normalization relative to this reference scale is therefore an expected initialization effect.

AuthorsOpenReview ↗

Response to Reviewer i6ma Part(2/3)

Moreover, the initial period contains no cross-subspace historical-state mixing. Let ht,a=(1−β1)∑s=1tβ1t−sGs,a\mathbf{h}_{t,a}=(1-\beta_1)\sum_{s=1}^{t}\beta_1^{t-s}\mathbf{G}_{s,a} denote the accumulated within-period momentum. Since the projector Ω\boldsymbol{\Omega} is fixed within the period, Mt,a=ht,aΩ\mathbf{M}_{t,a}=\mathbf{h}_{t,a}\boldsymbol{\Omega}, and the actual update direction satisfies

⟨ht,a,Mt,aΛtaΩ⊤⟩=∑j=1r(Λta)jj(ht,aωj)2≥0,\left\langle\mathbf{h}_{t,a},\mathbf{M}_{t,a}\boldsymbol{\Lambda}_t^a\boldsymbol{\Omega}^\top\right\rangle=\sum_{j=1}^{r}(\boldsymbol{\Lambda}_t^a)_{jj}(\mathbf{h}_{t,a}\boldsymbol{\omega}_j)^2\ge 0,

where ωj\boldsymbol{\omega}_j is column jj of Ω\boldsymbol{\Omega}. Thus, even with substantial initialization-scale corrections, the actual update maintains a nonnegative inner product with within-period momentum, partially explaining the optimizer's behavior during the initial period. We also emphasize that FLAME's main convergence proof does not require the small-remainder condition; Theorem 3 serves to elucidate the properties of random projection. Fixed-gradient analysis. Theorems 2–3 treat the gradient sequence as a given input to isolate subspace-transition and adaptive-normalization effects. Nevertheless, the key behavior of the two states at refresh can also be derived directly for adaptive training. Using the history Ft−1\mathcal{F}_{t-1} and conditional unbiasedness of Appendix B.6, let tt be a projection-refresh step. Given the history and current stochastic gradient, the fresh projector remains independent and follows its prescribed distribution, while previous states are determined by the history. A supplementary derivation therefore gives

E[MtΩt⊤∣Ft−1]=(1−β1)∇f(θt),\mathbb{E}[\mathbf{M}_t\boldsymbol{\Omega}_t^\top\mid\mathcal{F}_{t-1}]=(1-\beta_1)\nabla f(\boldsymbol{\theta}_t),

E[∥Vt,a∥1∣Ft−1]=β2∥Vt−1,a∥1+(1−β2)E[∥Gt,a∥22∣Ft−1].\mathbb{E}[\lVert\mathbf{V}_{t,a}\rVert_1\mid\mathcal{F}_{t-1}]=\beta_2\lVert\mathbf{V}_{t-1,a}\rVert_1+(1-\beta_2)\mathbb{E}[\lVert\mathbf{G}_{t,a}\rVert_2^2\mid\mathcal{F}_{t-1}].

We suppress the layer index; the gradient in the first identity denotes the corresponding parameter block. Thus, even when the current gradient depends on all past projections, the reconstructed historical first-moment contribution still vanishes in conditional expectation at refresh, whereas the second-moment state continues accumulating historical energy. The fixed-gradient setting further enables explicit whole-period characterizations in Theorems 2–3. We agree that these do not directly cover the dependence of subsequent within-period gradients on a reused projector when τ>1\tau > 1. We will clarify this boundary and include the refresh-step derivation. FLAME refreshes every step, and Theorem 6 directly handles the history dependence of its actual iterates.

Convergence rate and constants. We agree that Theorem 6 matches full-rank Adam in the O(T−1/2)O(T^{-1/2}) rate with respect to the step count TT, while carrying an O(n/(rϵ))O(\sqrt{n/(r\epsilon)}) prefactor. The dependence on dimension nn and rank rr arises from the exact second-moment identity for Gaussian back-projection (Appendix B.6, Lemma 10). This factor captures the trade-off between compression and the update's second moment: a larger projection rank reduces this cost; for a fixed model and rank, the factor does not grow with the number of training steps.

Moreover, stabilizing parameters appearing in convergence-bound denominators are not uncommon in adaptive optimization: ADOPT [1, Appendix F, proof of Theorem 4.1] explicitly contains an O(α(1+β1)G2L/[(1−β1)ϵ2])O(\alpha(1+\beta_1)G^2L/[(1-\beta_1)\epsilon^2]) term; for standard Adam, Li et al. [2, discussion following Theorem 4.1] also explicitly report an O(λ−2)O(\lambda^{-2}) iteration-complexity dependence on their stabilizer λ\lambda. LDAdam [3, Theorem 1] directly includes 2C0Lσ2/(ϵT)2C_0L\sigma^2/(\epsilon\sqrt{T}). Thus, having ϵ\sqrt{\epsilon} in the denominator is consistent with existing adaptive-optimization analyses, and our leading coefficient's ϵ−1/2\epsilon^{-1/2} dependence is milder than the ϵ−1\epsilon^{-1} dependence of that LDAdam term.

Question 1: Optimization of Unprojected Parameters

Thank you for your question. In the baseline methods, the embedding and LM-head parameters are optimized with Adam without low-rank projection. FLAME also updates these parameters without low-rank projection, using first-moment-free adaptive updates. In the 3B model, the embedding and LM head share weights. In the 8B model, these weights are not shared, so separate optimizer states are maintained for each parameter matrix. This difference is determined by the architectural configurations of the models.

AuthorsOpenReview ↗

Response to Reviewer i6ma Part(3/3)

Question 2: What Momentum Adds at Matched Refresh Periods

Thanks for your question. At a fixed refresh period τ\tau, the historical information that first-order momentum adds beyond the current gradient decomposes into a within-period term (historical gradients from the current period sharing the same projector) and an out-of-period term (historical gradients retained in previous projection coordinates). The within-period term accumulates and smooths gradients in shared low-rank coordinates, whereas the out-of-period term stems from independent past subspaces: in a matched one-step comparison at refresh using the same historical normalization and effective scaling of the current-gradient contribution, it has zero conditional mean and adds only a nonnegative variance term. As a special case, when τ=1\tau=1, every step refreshes the projector, so no historical within-period term exists and retaining momentum introduces only this out-of-period term; removing the first moment therefore preserves the current update's conditional mean while eliminating this additional noise.

Meanwhile, smaller τ\tau provides more independent subspaces, improving cumulative full-space projection coverage. Proposition 4 quantifies this property: at a fixed training horizon, more frequent refreshes yield a tighter high-probability bound on cumulative coverage error. Removing the first moment allows FLAME to refresh every step, benefiting from this coverage property while avoiding the additional noise introduced by directly reused old-coordinate momentum.

Empirically, we compare FLAME with its momentum-retaining variant(FLAME with Momentum) at the matched refresh period τ=1\tau=1 on LLaMA-3 8B/GSM8K.

Each column is labeled as seed / learning rate.

Method42 / 2×10−52\times10^{-5}42 / 1×10−41\times10^{-4}42 / 1×10−61\times10^{-6}43 / 2×10−52\times10^{-5}
FLAME0.6540.4320.5760.652
FLAME with Momentum0.5750.5850.1870.582

These results provide evidence for the performance benefits of removing the first moment at a matched projection refresh period.

In Table 5, FLAME-Momentum retains only within-period first-moment history at τ=200\tau=200, resetting both the momentum history and its bias-correction counter at each subspace refresh. This differs from FLAME with Momentum in the matched-period experiment above. The performance degradation in Table 5 is consistent with our concern that these periodic resets may disrupt optimization. The comparison was intended to assess this within-period momentum strategy and support our choice of using the current projected gradient directly to avoid these resets, rather than to study the effect of the refresh period itself.

References

[1] Taniguchi et al. ADOPT: Modified Adam Can Converge with Any β2\beta_2 with the Optimal Rate. NeurIPS, 2024.

[2] Li, Jadbabaie, and Rakhlin. Convergence of Adam Under Relaxed Assumptions. NeurIPS, 2023.

[3] Robert et al. LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. ICLR, 2025.

Reviewer_RMoAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FLAME combines momentum-free low-rank updates, previous-step second-moment normalization, and activation compression. The paper provides a discrete-time convergence guarantee and reports competitive performance with reduced memory and improved throughput.

Critical strengths
  1. The analysis of projected optimizer states provides useful insight, and the discrete-time convergence guarantee strengthens the methodological contribution.

  2. The method is simple and achieves promising memory and throughput improvements across the evaluated fine-tuning settings.

Critical weaknesses
  1. The system gains are not sufficiently isolated. FLAME adopts AGO-CC’s co-compression strategy, while the baselines retain full activations and gradients (Tables 6–7). Without a matched pipeline comparison, the additional benefit of FLAME’s optimizer design remains unclear.

  2. The evidence supporting momentum removal is limited. Table 5 compares configurations with different refresh periods and bias-correction behavior. Experiments also use one seed and a shared learning rate, leaving the robustness of the performance differences uncertain.

Questions for authors
  1. How much of the memory and throughput advantage remains against AGO-CC or a momentum baseline using the same activation/gradient compression pipeline?

  2. On a representative benchmark, does removing momentum still improve performance with matched refresh periods and normalization, separately tuned learning rates, and multiple seeds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RMoA Part(1/3)

We sincerely thank the reviewer for the constructive feedback and for recognizing the value of our theory and method. We address your concerns as follows.

Weakness 1 & Question 1: Independent System-Level Benefits of FLAME’s Optimizer Design

Thanks for the comment. Below, we separately explain the contributions and advantages of FLAME’s optimizer design in terms of throughput improvements and memory savings.

Throughput Improvements. Compared with other low-rank gradient-projection methods, FLAME reduces computational overhead and increases training throughput primarily through inexpensive Gaussian projections and first-moment-free updates. AGO-CC provides additional throughput gains by directly computing projected weight gradients during backpropagation, avoiding the computation of full weight gradients. To empirically isolate the benefits of FLAME’s optimizer design, we evaluate FLAME-OptOnly, which disables activation–gradient co-compression while keeping all other experimental settings identical. We conduct this experiment using LLaMA-3 8B on GSM8K, and the results are presented below.

MethodThroughput (tokens/s)
Adam1,875
GaLore754
APOLLO2,071
FLAME-OptOnly2,822
FLAME3,004

As shown, FLAME-OptOnly improves throughput over Adam by 50.5%, and AGO-CC contributes an additional 9.7 percentage points, bringing the overall throughput improvement over Adam to 60.2%. These results show that most of FLAME’s throughput gains come from its optimizer design. FLAME-OptOnly also achieves 36.3% higher throughput than APOLLO and 3.74 times the throughput of GaLore. Its throughput advantage over GaLore highlights the computational benefits of inexpensive random projections, while its advantage over APOLLO highlights the additional efficiency gains from first-moment-free updates.

AuthorsOpenReview ↗

Response to Reviewer RMoA Part(2/3)

Memory Savings. In addition to the memory savings from activation–gradient co-compression, FLAME reduces optimizer-state memory by removing the first-moment state and maintaining the second-moment state in a low-rank space. To quantify these contributions, we report peak GPU memory usage under the same settings as the throughput evaluation.

MethodPeak GPU Memory (GB)
Adam62.89
GaLore36.52
FLAME-OptOnly33.98
FLAME29.25
GoLore with AGO-CC32.13

As shown, FLAME-OptOnly reduces peak GPU memory by 46.0% relative to Adam and 7.0% relative to GaLore, demonstrating its independent savings from FLAME’s optimizer design. AGO-CC further increases the memory reduction relative to Adam by 7.5 percentage points, bringing the overall peak-memory reduction to 53.5%. These results demonstrate that both contribute to FLAME’s overall memory savings. With co-compression enabled for both methods, FLAME also reduces peak GPU memory by 2.88 GB (9.0%) relative to GoLore with AGO-CC, further demonstrating the advantages of its optimizer design. We will include these ablation results in the next version.

Moreover, we emphasize that integrating activation–gradient co-compression into FLAME is an integral part of our theory-guided method design. FLAME uses a Gaussian random projection matrix Ω\Omega that is independent of the current gradient, allowing activation compression during the forward pass. Our analysis of the actual adaptive update provides a theoretical justification for this choice of projection. By contrast, baseline low-rank gradient-projection methods commonly derive their projection matrices from the current gradient using SVD-based methods. This makes AGO-CC difficult to apply, because the projection matrix cannot be computed during the forward pass, when the current gradient has not yet been obtained.

Specifically, low-rank gradient projection is applied to the matrix-valued weights of linear layers. With optimizer-only projection, we retain the full activations A\mathbf{A} during the forward pass and compute the full weight gradient G=(∇ZL)⊤A\mathbf{G}=(\nabla_{\mathbf{Z}}\mathcal{L})^\top\mathbf{A} during backpropagation. At the optimizer stage, we project it as U=GΩ\mathbf{U}=\mathbf{G}\Omega for the optimizer update. With activation–gradient co-compression, we instead store compressed activations A~=AΩ\widetilde{\mathbf{A}}=\mathbf{A}\Omega during the forward pass and directly compute the projected gradient U=(∇ZL)⊤A~=(∇ZL)⊤AΩ=GΩ\mathbf{U}=(\nabla_{\mathbf{Z}}\mathcal{L})^\top\widetilde{\mathbf{A}}=(\nabla_{\mathbf{Z}}\mathcal{L})^\top\mathbf{A}\Omega=\mathbf{G}\Omega during backpropagation. This avoids materializing the full weight gradient without introducing additional approximation error relative to applying the same projection at the optimizer stage.

This optimizer-level perspective also distinguishes our work from AGO-CC. AGO-CC analyzes how activation compression affects gradients in an SGD setting, but does not characterize how the compressed gradients affect Adam’s moment estimates and actual adaptive updates. Our work starts from an analysis of optimizer states and actual adaptive updates to guide FLAME’s design, and then extends its gradient-independent projection to activation compression.

AuthorsOpenReview ↗

Response to Reviewer RMoA Part(3/3)

Weakness 2 & Question 2: Additional Evidence for Momentum Removal

Thank you for your feedback and suggestions. Below, we provide additional experimental evidence to address these concerns.

Clarification of Table 5. We apologize for not clearly explaining the purpose of Table 5. The comparison between FLAME and FLAME-Momentum in Table 5 shows the performance degradation associated with retaining only within-period gradients in the first-order moment when τ>1\tau>1. This degradation is consistent with our concern that each subspace refresh abruptly resets both the momentum history and its bias-correction counter, potentially disrupting optimization. The comparison was intended to support our choice of using the current projected gradient directly to avoid these resets, rather than to evaluate the effect of momentum removal under otherwise identical settings. Accordingly, we used different projection refresh periods and bias-correction schemes for the two configurations. To further assess this design choice, we compare the two configurations using additional learning rates and random seeds. The results are presented below.

Each column is labeled as seed / learning rate.

Method42 / 2×10−52\times10^{-5}42 / 1×10−41\times10^{-4}42 / 1×10−61\times10^{-6}43 / 2×10−52\times10^{-5}
FLAME0.6540.4320.5760.652
FLAME-Momentum0.6210.5010.4510.612

Across the tested learning rates, FLAME achieves a higher best accuracy than FLAME-Momentum. At a learning rate of 2×10−52\times10^{-5}, FLAME also outperforms FLAME-Momentum with seed 43. These results further support our choice.

Additional Experimental Evidence for Momentum Removal. To further evaluate the effect of momentum removal, we compare FLAME with FLAME with Momentum, GaLore, and LDAdam. FLAME with Momentum retains the first-moment state while keeping the projection refresh period, normalization, and all other settings identical to FLAME. We use LLaMA-3 8B on GSM8K across multiple learning rates and random seeds.

Each column is labeled as seed / learning rate.

Method42 / 2×10−52\times10^{-5}42 / 1×10−41\times10^{-4}42 / 1×10−61\times10^{-6}43 / 2×10−52\times10^{-5}
FLAME0.6540.4320.5760.652
GaLore0.6500.4610.3170.645
LDAdam0.6280.3910.1670.621
FLAME with Momentum0.5750.5850.1870.582

These results provide more comprehensive support for our choice of first-moment-free updates. After selecting the best tested learning rate for each method, FLAME achieves the highest accuracy. At a learning rate of 2×10−52\times10^{-5}, this advantage is also observed with seed 43. Following prior work such as GaLore [1] and COAP [2], our original fine-tuning experiments used a shared learning rate of 2×10−52\times10^{-5}. The learning-rate sweep shows that this value yields the best accuracy for FLAME and most of the evaluated baselines, supporting our original choice. We will include these additional experimental results in the next version.

References

[1] Jiawei Zhao et al. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML, 2024.

[2] Jinqi Xiao et al. COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection. CVPR, 2025.

Comments on the paper

1 comment
Area_Chair_eBNkOpenReview ↗

Please respond to the rebuttal

Dear Reviewers,

The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.