- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper applies Adafctor's row/column compression to an adaptative momentum damping mechanism. They call it R-iKFAD, because it's rank-1 realization of the iKFAD's friction matrix.
Critical strengths
1.The comparison against Adafactor is fair and square, with similar memory budget
- the continuous time analysis provides certain level of convergence guarantee
Critical weaknesses
-
No scaling ladder results
-
All comparisons use constant learning rates and zero weight decay, this rather unconventional
-
It feels a bit incremental, the discrete time analysis is also missing
Questions for authors
Looking in the charts, the experiments do not seem to have converged at all? Do you have longer runs that show momentums magnitudes become small enough, would this rank-1 formulation cause practical issues similar to the Adafactor failure mode?
2 comments
As promised, the new Appendix K adds a model-size ladder. We trained GPT2 models with 30M, 51M, 124M and 355M parameters on the same 2.46B tokens. Each optimizer reuses its GPT2-S hyperparameters, with the step size rescaled by the best power of two at each size, and each run uses one seed. Given computational and time constraints, the ladder is a bit minimal but we hope it demonstrates the key conclusion. R-iKFAD0 reaches a lower test loss than iKFAD0 and Adam at every size with half their auxiliary memory. Its margin over Adam grows with model size, from 0.010 at 30M to 0.070 at 355M, and its peak memory saving grows from 114 MiB to 1350 MiB. Against Adafactor-m (same memory baseline) it is behind only at 30M. At 355M, R-iKFAD0 reaches Adam’s final test loss in 74% of the steps and 81% of the wall-clock time (same as 124M). In the camera-ready version we will add more seeds, larger models and training budgets that grow with model size.
We thank the reviewer for their critical feedback. We address the above stated concerns herewith:
Scaling Ladder: We are still running a model-size ladder from 30M to 355M parameters and will post the results here before the discussion ends.
Constant learning rate and no weight decay: We agree that this is not the practice of most SOTA implementations. We opted for this setup to isolate the optimizer dynamics as done in (Karoni et al., 2026). To address this concern, we ran repeats of GPT2-Nano, TinyViT and DistilBERT experiments with a 2% linear warmup, cosine decay to 10% and decoupled weight decay and re-tuning only the peak step size (5 seeds). We included the results in new Appendix J. R-iKFAD0 remains within noise of or superior to iKFAD0 on all three. On DistilBERT, it is the best optimizer and beats Adam and Adafactor-m on all five seeds. On TinyViT, Adam and Adafactor-m remain ahead (just like the fixed LR result) and on GPT2-Nano all four optimizers end within 0.003 of one another. This indicates that our results with fixed learning LR and no WD are still conclusive. We were unable to run this sweep for GPT2-S due to time and computation constraints but will aim to include it in the camera-ready version.
Incremental contribution: We agree that the methodology alone could look incremental due to the reuse of Adafactor's rank-1 trick. However, the paper does more than simply apply Adafactor's factorization to iKFAD. Firstly, the factorization is not a direct transfer. Adafactor factors a second-moment estimate that divides the step, whereas R-iKFAD factors a friction built from momentum statistics that only damps the momentum. The factored friction therefore stays bounded however small the statistics become and can never enlarge a step. In practice this halves the auxiliary memory of iKFAD and Adam while matching iKFAD across four benchmarks, and on GPT2-S R-iKFAD0 reaches Adam's final loss in 73% of the steps. Secondly, we prove a continuous-time convergence analysis, confirmed numerically, that applies not only to R-iKFAD but also to KFAD and iKFAD. To our knowledge, it is the first to cover the undamped case (γ=0), with convergence and matching upper and lower rate bounds. We believe that the continuous-time analysis will help motivate and aid the undamped stochastic gradients discrete area and would like to stress its novelty and verifiability.
Discrete-time analysis: New Theorem 4 (proof in Appendix G.4) extends the case γ>0 to the BACD discretisation we implement. The iterates converge exponentially for every step size h≤2γ2+Lfγ, a bound independent of the initial data. The discrete case γ=0 is substantially harder (even with exact gradients). The only dissipation is the friction which vanishes as the iterates approach the minimizer, while the splitting adds energy of order h² per step. This means that the discrete energy need not decrease. Since the γ=0 and γ>0 variants perform very similarly in practice (Section 4.2), we regard the γ>0 result as sufficient for now and leave γ=0 and stochastic gradients to future work.
Convergence and Adafactor failure mode: We used the same training setup as (Karoni et al. (2026)) with the same number of epochs. GPT2-S trains for 200k steps which ends up being 2.46B tokens and is the Chinchilla compute-optimal budget (Hoffmann et al., 2022) of 20 tokens per parameter. TinyViT and DistilBERT converge within their budgets: the TinyViT test loss reaches its minimum around epoch 30 and then increases and the DistilBERT training loss levels off after epoch 1. For GPT2-Nano, a run ten times longer (50k steps, 3 seeds) reaches its lowest validation loss between 16k and 19k steps and then overfits. R-iKFAD0 ties Adam for the lowest loss. The momentum does not become small. Its RMS levels off at about 5×10−4 through 50k steps while the rank-1 normalizer 1TR stays above 230 in every layer. The mechanism that causes Adafactor's failure mode is completely absent in RiKFAD0: The friction is bounded by Cj however small R becomes and it can enter only through a damping factor exp(−2hξ~)≤1 which may weaken the damping but never enlarge the step. On GPT2-S, all 5 R-iKFAD0 seeds train without instability unlike one of the Adafactor-m seeds (Figure 3). This theoretically and empirically demonstrates that RiKFAD0 does not suffer from Adafactor's failure mode.