acceptodds
Under review as a conference paper at ICLR 2027

Low Rank Adaptive Friction: Memory Efficient Optimization without Linear Damping

Abstract

iKFAD is a recently proposed optimiser that replaces adaptive learning rates with adaptive friction in the momentum dynamics, yet performs as well as Adam. Its limitation is that the full friction tensor carries the same memory overhead per layer as Adam's second-moment buffer. Here we replace iKFAD's friction tensor with a rank-1 outer-product factorisation built from row and column momentum statistics, resulting in Rank-1 iKFAD (R-iKFAD). This reduces the friction memory footprint from to per layer, which approximately halves iKFAD's total optimiser state. Despite this reduction, R-iKFAD maintains parity in performance with iKFAD: experiments on GPT2-Nano, TinyViT, DistilBERT and GPT2-S confirm that it matches or exceeds iKFAD while nearly halving the memory footprint and remaining comparably robust to hyperparameters. We analyse the continuous-time dynamics in two damping regimes. For linear damping () we prove exponential convergence under strong convexity. For , the preferred option in our experiments, the friction is generated entirely from past momentum and switches off as the momentum vanishes, so geometric convergence cannot be shown. We nonetheless prove convergence to the minimiser, together with matching upper and lower bounds on the energy: of order when the regularisation scale is zero, and of order when it is positive. To our knowledge this is the first convergence rate for a rank-1 factored optimiser in continuous time, and the first such result that does not require positive damping.

Primary area:
optimization
Keywords:
Optimization, Low Rank, LLM Pretraining, Dynamical Systems, Lyapunov Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_umWnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies Adafctor's row/column compression to an adaptative momentum damping mechanism. They call it R-iKFAD, because it's rank-1 realization of the iKFAD's friction matrix.

Critical strengths

1.The comparison against Adafactor is fair and square, with similar memory budget

  1. the continuous time analysis provides certain level of convergence guarantee
Critical weaknesses
  1. No scaling ladder results

  2. All comparisons use constant learning rates and zero weight decay, this rather unconventional

  3. It feels a bit incremental, the discrete time analysis is also missing

Questions for authors

Looking in the charts, the experiments do not seem to have converged at all? Do you have longer runs that show momentums magnitudes become small enough, would this rank-1 formulation cause practical issues similar to the Adafactor failure mode?

2 comments

We thank the reviewer for their critical feedback. We address the above stated concerns herewith:

  1. Scaling Ladder: We are still running a model-size ladder from 30M to 355M parameters and will post the results here before the discussion ends.

  2. Constant learning rate and no weight decay: We agree that this is not the practice of most SOTA implementations. We opted for this setup to isolate the optimizer dynamics as done in (Karoni et al., 2026). To address this concern, we ran repeats of GPT2-Nano, TinyViT and DistilBERT experiments with a 2% linear warmup, cosine decay to 10% and decoupled weight decay and re-tuning only the peak step size (5 seeds). We included the results in new Appendix J. R-iKFAD0 remains within noise of or superior to iKFAD0 on all three. On DistilBERT, it is the best optimizer and beats Adam and Adafactor-m on all five seeds. On TinyViT, Adam and Adafactor-m remain ahead (just like the fixed LR result) and on GPT2-Nano all four optimizers end within 0.003 of one another. This indicates that our results with fixed learning LR and no WD are still conclusive. We were unable to run this sweep for GPT2-S due to time and computation constraints but will aim to include it in the camera-ready version.

  3. Incremental contribution: We agree that the methodology alone could look incremental due to the reuse of Adafactor's rank-1 trick. However, the paper does more than simply apply Adafactor's factorization to iKFAD. Firstly, the factorization is not a direct transfer. Adafactor factors a second-moment estimate that divides the step, whereas R-iKFAD factors a friction built from momentum statistics that only damps the momentum. The factored friction therefore stays bounded however small the statistics become and can never enlarge a step. In practice this halves the auxiliary memory of iKFAD and Adam while matching iKFAD across four benchmarks, and on GPT2-S R-iKFAD0 reaches Adam's final loss in 73% of the steps. Secondly, we prove a continuous-time convergence analysis, confirmed numerically, that applies not only to R-iKFAD but also to KFAD and iKFAD. To our knowledge, it is the first to cover the undamped case (γ=0), with convergence and matching upper and lower rate bounds. We believe that the continuous-time analysis will help motivate and aid the undamped stochastic gradients discrete area and would like to stress its novelty and verifiability.

  4. Discrete-time analysis: New Theorem 4 (proof in Appendix G.4) extends the case γ>0\gamma>0 to the BACD discretisation we implement. The iterates converge exponentially for every step size h≤γ2γ2+Lfh \leq \frac{\gamma}{2\gamma^2 + L_f}, a bound independent of the initial data. The discrete case γ=0\gamma=0 is substantially harder (even with exact gradients). The only dissipation is the friction which vanishes as the iterates approach the minimizer, while the splitting adds energy of order h² per step. This means that the discrete energy need not decrease. Since the γ=0\gamma=0 and γ>0\gamma>0 variants perform very similarly in practice (Section 4.2), we regard the γ>0\gamma>0 result as sufficient for now and leave γ=0\gamma=0 and stochastic gradients to future work.

  5. Convergence and Adafactor failure mode: We used the same training setup as (Karoni et al. (2026)) with the same number of epochs. GPT2-S trains for 200k steps which ends up being 2.46B tokens and is the Chinchilla compute-optimal budget (Hoffmann et al., 2022) of 20 tokens per parameter. TinyViT and DistilBERT converge within their budgets: the TinyViT test loss reaches its minimum around epoch 30 and then increases and the DistilBERT training loss levels off after epoch 1. For GPT2-Nano, a run ten times longer (50k steps, 3 seeds) reaches its lowest validation loss between 16k and 19k steps and then overfits. R-iKFAD0 ties Adam for the lowest loss. The momentum does not become small. Its RMS levels off at about 5×10−45\times 10^{-4} through 50k steps while the rank-1 normalizer 1TR\mathbf{1}^T \mathbf{R} stays above 230 in every layer. The mechanism that causes Adafactor's failure mode is completely absent in RiKFAD0: The friction is bounded by Cj\mathbf{C}_j however small R\mathbf{R} becomes and it can enter only through a damping factor exp⁡(−hξ~2)≤1\exp\left(-\frac{h\tilde{\xi}}{2}\right) \leq 1 which may weaken the damping but never enlarge the step. On GPT2-S, all 5 R-iKFAD0 seeds train without instability unlike one of the Adafactor-m seeds (Figure 3). This theoretically and empirically demonstrates that RiKFAD0 does not suffer from Adafactor's failure mode.

As promised, the new Appendix K adds a model-size ladder. We trained GPT2 models with 30M, 51M, 124M and 355M parameters on the same 2.46B tokens. Each optimizer reuses its GPT2-S hyperparameters, with the step size rescaled by the best power of two at each size, and each run uses one seed. Given computational and time constraints, the ladder is a bit minimal but we hope it demonstrates the key conclusion. R-iKFAD0 reaches a lower test loss than iKFAD0 and Adam at every size with half their auxiliary memory. Its margin over Adam grows with model size, from 0.010 at 30M to 0.070 at 355M, and its peak memory saving grows from 114 MiB to 1350 MiB. Against Adafactor-m (same memory baseline) it is behind only at 30M. At 355M, R-iKFAD0 reaches Adam’s final test loss in 74% of the steps and 81% of the wall-clock time (same as 124M). In the camera-ready version we will add more seeds, larger models and training budgets that grow with model size.

Reviewer_T8ScOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper replaces iKFAD's full friction matrix with row and column statistics of squared momentum. The resulting rank-one friction keeps a full momentum buffer. It reduces the calculated auxiliary optimizer state from about two scalars per parameter to about one. The paper also studies the corresponding continuous-time system. It proves convergence under strong convexity with and without linear damping. At zero linear damping, it gives two-sided energy bounds with different exponents for zero and positive stabilization. Experiments compare R-iKFAD with iKFAD, Adam, and Adafactor with momentum on four tasks.

Critical strengths
  • The proposed factorization substantially reduces optimizer state with little loss of performance in the reported tasks.
  • The paper provides convergence results for dynamics without linear damping.
  • The experiments cover several model types and include relevant optimizer comparisons.
Critical weaknesses
  • The theory does not cover the training algorithm. Theorems 1–3 concern exact-gradient continuous-time dynamics under strong convexity. Section 4 instead uses stochastic gradients and finite BACD steps on nonconvex networks. The paper acknowledges this gap.
  • The lowest-memory alternative is not evaluated. Section 4 uses Adafactor-m to compare methods with similar auxiliary state. This is a fair control for the paper's main claim. Appendix C, Table 5 also shows that momentum-free Adafactor uses much less state, but Table 2 gives no loss for it. The paper therefore leaves the broader performance–memory tradeoff open.
  • Performance parity lacks a direct uncertainty test. Table 2 shows small R-iKFAD versus iKFAD differences on three replicated tasks, but gives no paired seed differences or practical equivalence margin. GPT2-S has one seed. The results suggest parity, but do not yet establish its precision.
Questions for authors
  • Can the authors report paired R-iKFAD versus iKFAD test-loss differences with confidence intervals and a practical equivalence margin? Can GPT2-S be repeated with more than one seed?
  • How does tuned momentum-free Adafactor perform on the same four tasks?
  • Does the 25-epoch TinyViT setting in Appendix B describe only the sweep? What was the final-run length behind Figures 3, 4, and 7?
3 comments

We thank the reviewer for their insightful feedback. Below we address the concerns delineated above:

  1. Theory and the training algorithm: Our continuous-time results address a regime that prior analyses exclude. KFAD, iKFAD and H-Fac are analysed only with positive linear damping, whereas we prove convergence when the only damping is an adaptive friction that vanishes with the momentum (γ=0\gamma=0). We give matching upper and lower bounds on the energy, of order t−1t^{-1} without and t−1/2t^{−1/2} with the stabilizer, and numerical integration confirms both exponents. These γ=0\gamma=0 results also carry over to KFAD and iKFAD. We believe that the continuous-time analysis will help motivate and aid the undamped stochastic gradients discrete area and would like to stress its novelty and verifiability. New Theorem 4 (proof in Appendix G.4) extends the case γ>0\gamma>0 to the BACD discretisation we implement. The iterates converge exponentially for every step size h≤γ2γ2+Lfh \leq \frac{\gamma}{2\gamma^2 + L_f}, a bound independent of the initial data. The discrete case γ=0\gamma=0 is substantially harder (even with exact gradients). The only dissipation is the friction which vanishes as the iterates approach the minimizer, while the splitting adds energy of order h² per step. This means that the discrete energy need not decrease. Since the γ=0\gamma=0 and γ>0\gamma>0 variants perform very similarly in practice (Section 4.2), we regard the γ>0\gamma>0 result as sufficient for now and leave γ=0\gamma=0 and stochastic gradients to future work.

  2. Momentum-free Adafactor: Table 2 now includes tuned momentum-free Adafactor on all four tasks (new Appendix I) with its step size swept over the same range and budget as Adafactor-m. R-iKFAD0 reaches a lower loss on every task. On GPT2-S, one of its five seeds also diverged late in training which highlights an instability that we do not encounter with R-iKFAD0.

  3. Paired differences and GPT2-S seeds: GPT2-S is now repeated over 5 seeds (Table 2/Figure 3) and new Appendix H reports paired per-seed differences with 95% intervals and a margin of 1% of the iKFAD0 loss. R-iKFAD0 is comparable to iKFAD0 on GPT2-Nano, DistilBERT and GPT2-S and better on all five GPT2-S seeds by a significant margin. The extra seeds also show that Adafactor-m is the least stable on GPT2-S: its seed standard deviation is about four times that of R-iKFAD0 and one seed has a transient loss spike (Figure 3).

  4. TinyViT budget: We apologize for not communicating this more clearly. 25 epochs was the budget of each sweep trial. All runs converged early and then the test loss started increasing which is why we opted for 25 epochs in the sweep rather than waste the budget on 50 epochs. The final runs behind Figures 3, 4 and 7 train for 50 epochs. Appendix B now states both.

Dear Authors,

Thank you for your responses which address several of my concerns. However, I will keep my initial score for several reasons. The new Theorem 4 covers only γ > 0, while the paper recommends γ = 0, so the discrete-time theory still does not cover the method used in the experiments. R-iKFAD0 is better than iKFAD0 only on GPT2-S with a constant step size and no weight decay. On GPT2-Nano and DistilBERT the difference from iKFAD0 is not significant, and on TinyViT parity is not shown. The main benefit of the method is memory saving which is small in practice (3.6% of peak GPU memory on GPT2-S). Finally, on TinyViT R-iKFAD0 is worse than Adafactor-m which uses the same memory.

Thank you for your reply. Two additions to the revised PDF might bear on your remaining concerns.

Appendix K adds a model-size ladder with GPT2 models of 30M, 51M, 124M and 355M parameters (one seed per run). R-iKFAD0 reaches a lower test loss than iKFAD0 at every size, so its lead on GPT2-S is not specific to one model. It also beats Adam at every size, by a margin that grows from 0.010 at 30M to 0.070 at 355M. Against Adafactor-m (same memory baseline), it is behind at 30M, level at 51M and ahead at 124M and 355M. Adafactor-m also diverged at step-size multiplier 2 at 51M and in the 355M screen, whereas no R-iKFAD0 run diverged highlighting its robustness/stability. The memory saving is about one float32 copy of the parameters, so it grows in. proportion to model size, from 114 MiB at 30M to 1350 MiB at 355MAt 355M, trained with micro-batches of 6 sequences, this is 8.7% of Adam’s peak GPU memory against 3.6% at GPT2-S.

Appendix J repeats {GPT2-Nano, TinyViT and DistilBERT} with warmup, cosine decay and weight decay (5 seeds). R-iKFAD0 remains within noise of or better than iKFAD0 on all three. On DistilBERT it is the best of the four optimizers.

Reviewer_QkULOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces R-iKFAD, which replaces iKFAD’s full adaptive-friction matrix with row and column momentum statistics. This reduces the memory footprint.

Critical strengths
  • Empirical tests are comprehensive for the memory footprint.
  • Table 5 and 2 provide useful benchmarks and baselines.
  • Theory is well substantiated.
Critical weaknesses
  • The robustness and damping seem to be conditional.
  • Can the authors compare independently optimized damping regimes.
  • Seems like some related work citations such as VRAdam(ICLR 2026), etc... seem to be missing.
Questions for authors
  • Please look at the weaknesses.
  • Can you reconcile the TinyViT training budgets and provide peak GPU memory and time-to-quality measurements?
1 comment

We thank the reviewer for their detailed feedback. We address their concerns below:

  1. Conditional robustness and damping: We believe this refers to the γ−h\gamma - h sweeps of Figure 5 and that there is a misunderstanding about what it represents. Each cell is a single run at a fixed (γ,h)(\gamma, h). We varied only \gamma and hh and kept α,μ\alpha, \mu and all other hyperparameters fixed, without re-tuning them for any (γ,h)(\gamma, h) configuration. "Best performance" in the caption means the best value reached during that run, not the best over further tuning. Even so, both iKFAD and R-iKFAD keep a broad high-performance plateau across γ\gamma from 0 up to 10 and two decades of h. Any optimizer will have some conditional robustness as none are robust over every hyperparameter. Fig 5 shows that R-iKFAD retains the robustness of iKFAD despite using half the memory. We have reworded the Figure 5 caption to make this explicit. Please let us know if this clarifies your doubt.

  2. Independently optimized damping regimes: The ablation in Section 4.2 already compares independently optimized damping regimes. R-iKFAD0 and R-iKFAD with γ>0\gamma>0 were tuned separately, and for γ>0\gamma>0 Optuna tuned (h,α,μ,γ)(h, α, μ, γ) jointly. The submitted text wrongly said that γ>0\gamma>0 reused the (h,α,μ)(h, α, μ) of R-iKFAD0, which we have now corrected. Linear damping gives no meaningful gain as was shown in (Karoni et al. 2026) for iKFAD. The lowest losses with and without γ are 1.632 and 1.633 on GPT2-Nano respectively and 0.268 for both on DistilBERT, and on TinyViT γ>0\gamma>0 is worse (0.615 against 0.606). The γ−h\gamma - h sweeps of Figure 5 show broad plateaus for both iKFAD and R-iKFAD. While preparing the revision we found a plotting-script error in Figure 5. The revised figure has been added and the conclusion is unchanged.

  3. VRAdam citation: Thanks for the note. The citation has been added to the PDF.

  4. TinyViT training budgets: The 25 epochs in Appendix B was the budget of each sweep trial. All runs converged early and then the test loss started increasing, which is why we opted for 25 epochs in the sweep rather than waste the budget on 50 epochs. The final TinyViT runs train for 50 epochs and Appendix B now states both.

  5. Peak GPU memory and time-to-quality: The new Section 4.5 reports measured peak GPU memory, runtime, and time-to-quality. Peak memory tracks the auxiliary state of Table 7 (Table 5 in the submission), so on GPT2-S, R-iKFAD0 uses (474 MiB) less than Adam and iKFAD0. R-iKFAD0 takes (13%) longer per step than Adam on GPT2-S, but reaches Adam's final loss in (73%) of the steps, which is (83%) of Adam's wall-clock time. This per-step overhead is inherited from iKFAD (Karoni et al., 2026) whose plain PyTorch implementation R-iKFAD builds on. Each step loops over the parameter tensors once per sub-step of the splitting and launches several small elementwise kernels per tensor. R-iKFAD0 also forms the rank-1 friction as a dense temporary matrix. PyTorch's Adam instead uses fused multi-tensor kernels. The overhead is therefore per tensor rather than per parameter, which is why it falls from 2.3×2.3\times Adam's step time on GPT2-Nano to 1.13×1.13\times on GPT2-S. We have not optimized this. A fused kernel could remove most of it. It would compute the damping factor exp⁡(−hξ~ij/2)\exp(-h\widetilde{\xi}_{ij}/2) on the fly from RiR_i and CjC_j instead of forming ξ~\widetilde{\xi}, apply the sub-steps in a single pass over the momentum, and accumulate the new row and column sums in that pass. Batching all tensors into one launch (like PyTorch's fused Adam) would cut it further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.