acceptodds
Under review as a conference paper at ICLR 2027

Temper: Reading the Second Moment from the Riemannian Gradient in LoRA

Abstract

Low-rank adaptation trains a weight update as the product of two factors, and one update has many factorizations. Recent LoRA ptimizers are therefore built to be transformation-invariant: the step should not depend on which one is stored. One line secures that by stepping along the Riemannian gradient, the projection of the Euclidean gradient onto the manifold's tangent space and a function of the update alone. However, invariance fixes the direction of the step, not how far it moves along each coordinate. That distance is what a second moment sets, and the methods that step along the Riemannian gradient either carry none or read one from the Euclidean gradient, almost all of which lies where no rank- step can follow. To this end, we propose Temper, which accumulates one second moment per row and one per column of the Riemannian gradient itself, and never forms the dense matrix that reading it there is known to cost: a tangent vector is low rank, so both marginals follow exactly from two small Gram matrices.Temper is provably invariant and costs no more per step than an unpreconditioned one. On Llama-3-8B and Qwen-2.5-7B it leads the strongest baseline by to points on the commonsense average, GSM8K and all three HumanEval pass rates.

Primary area:
optimization
Keywords:
low-rank adaptation, parameter-efficient fine-tuning, large language model fine-tuning, Riemannian optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eZgjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Temper, a LoRA optimizer that treats the update Z=BAZ=BA as a point on the fixed-rank manifold and steps along the Riemannian gradient G=PZ(∇Z)G=P_Z(\nabla Z). It keeps moving averages of the row and column sums ρ(G)i=∑jGij2\rho(G)_i=\sum_jG_{ij}^2 and γ(G)j=∑iGij2\gamma(G)_j=\sum_iG_{ij}^2, combines them with the rank-one reconstruction of Adafactor, and computes both exactly from the factored form G=PQ⊤G=PQ^\top as diag(P(Q⊤Q)P⊤)diag(P(Q^\top Q)P^\top) and diag(Q(P⊤P)Q⊤)diag(Q(P^\top P)Q^\top), at cost O((m+n)r2)O((m+n)r^2) and without forming an m×nm\times n matrix. The momentum is transported by projection, the scaled direction is projected back onto the tangent space, and the iterate is retracted by a truncated SVD. The motivation is that invariance fixes the direction of a LoRA step but not its size per coordinate, and that the statistic must come from GG. The theory gives invariance and bounds on the row norms of the scaled matrix. Temper is first in 23 of 24 cells on Commonsense-170K, GSM8K and HumanEval, with gains of 0.9 to 3.2 points over the strongest baseline on the commonsense average and on the math and code metrics.

Critical strengths

∙\bullet Section 3.1 explains clearly why an elementwise statistic in factor or frame coordinates is not invariant (eq. 9) while any function of GG is.

∙\bullet The experiments use three seeds with standard deviations and a separate learning-rate grid per optimizer with the selected value inside it, and the authors say openly that only two of the sixteen commonsense gaps exceed the seed spread. The supplementary code documents in detail how the baselines were reimplemented.

Critical weaknesses

W1. I am not sure that the reported results come from the proposed algorithm.

First, it seems that the Qwen-2.5-7B numbers on GSM8K and HumanEval lie well below those that the Qwen2.5 Technical Report [1] gives for the base model. The setups differ. The report evaluates the pre-trained model with few-shot prompts, while the paper evaluates fine-tuned models and states neither the number of shots, the prompt format and the decoding, nor the sampling temperature behind Pass@k from 20 samples, and it does not say whether the base or the instruct model was used. The gap therefore does not by itself show that fine-tuning hurts. Without the metric values of the models before fine-tuning, however, it is hard to tell whether fine-tuning helps at all and how well each optimizer performs.

Second, when I checked the attached repository and tried to work out how to run the method, I found that the class Temper, which the package exports as the entry point, runs AdamW on the LoRA factors and not a Riemannian step on the manifold. With the default settings its first phase is AdamW applied to AA and BB separately, and the later phase, which a controller switches on, also works on the factors. This class has no tangent projection and no SVD retraction. The method of Section 3 seems to be implemented instead by the class Riemannion. When I read this class in detail, it seemed to me to implement heavy-ball momentum without the factor (1−β1)(1-\beta_1) and without the bias correction 1/(1−β1t)1/(1-\beta_1^t), while eq. (16) and Table 10 describe a moving average with this correction. If so, and if the second moment is read from GtG_t as Appendix E states, Temper effectively runs with a warm-up in the code, since its step grows by up to a factor of 10 over the first steps, while the other methods run at the constant rate stated in Table 9.

W2. The motivation of the method is overstated.

First, Riemannian methods based on Muon also normalize the step. The paper motivates the second moment by saying that invariance fixes only the direction of the step, while its size along each coordinate has to be set by an adaptive second moment, and that "Riemannion and LoRA-Muon carry no adaptive scales at all" (Section 1). The second claim is too strong. Any optimizer can be viewed as a preconditioner applied to the gradient, followed by a step along the resulting update. For Muon this preconditioner can be written explicitly. Exact Muon takes the polar factor of the momentum as the update, and the polar factor is the momentum multiplied by matrices on the left and on the right. For M=UΣV⊤M=U\Sigma V^\top, Ortho(M)=UV⊤=(MM⊤)−1/4M(M⊤M)−1/4\mathrm{Ortho}(M)=UV^\top=(MM^\top)^{-1/4}M(M^\top M)^{-1/4}, with pseudo-inverse powers when MM is rank-deficient, which rescales each singular direction of the momentum by 1/σi(M)1/\sigma_i(M). Riemannian methods that transform the Riemannian gradient in this way, such as Riemannion and LoRA-Muon [2], therefore also apply a non-diagonal matrix preconditioner.

Second, the claim that most of the norm of the Euclidean gradient lies in the normal part N=∇Z−GN=\nabla Z-G depends on the initialization. The paper presents the method as Riemannian, so the starting point has to lie on the manifold of rank-rr matrices. This excludes the usual start B0=0B_0=0 and calls for another initialization, a random one or one of those proposed in other work, such as LoRA-GA [3] or LOI from the Riemannion paper. The ratio ∥N∥F2/∥∇Z∥F2\|N\|_F^2/\|\nabla Z\|_F^2 can depend on this choice. The paper does not study this question. Figure 1(a) starts at the value for random subspaces, which suggests a random initialization, but the paper neither states its initialization nor shows how much the share drops under gradient-based ones.

Third, RAdamW and AdaPreLoRA read their statistic from ∇Z\nabla Z by design. Reading it from GG is a modeling choice, and the ablation of Table 4 is not enough to support it: the table reports no standard deviations and does not say whether the learning rate was tuned separately for each row.

W3. The exposition is hard to follow.

Notation is often used before it is defined, or is not defined in the main text at all. The introduction uses ρ(∇Z)i\rho(\nabla Z)_i and the unmagnified gradients before eqs. (10) and (2) define them, and Section 4.2 speaks of "removing the magnification" without saying what the magnification is. Table 1 lists LoRA-Pro as reading its statistic from g~\tilde g, a symbol that is defined nowhere. Section 3.3 uses the previous frames U~A\tilde U_A and U~B\tilde U_B, which only Appendix A.8 defines, and SVDr\mathrm{SVD}_r in eq. (17) is never defined as the rank-rr truncated SVD. Table 4 reports a "Trained ppl" column without saying on which data or at which step it is measured. The main text also relies on Propositions 3 and 5, which are stated only in the appendix. The text has many stylistic and grammatical slips and a few missing references. Examples are "and refer to Appendix D" (Section 3.3), "and merges it afterwards" (Section 5), "the definitions gives" (Appendix B) and textual citations such as "AdaLoRA Zhang et al. (2023)" and "CodeAlpaca Chaudhary (2023)" where parenthetical ones are meant. The proof of Proposition 6 has a broken reference "(??)", and Llama-3.2-3B, which Figure 1 and Table 5 use, is never cited.

The figures need fuller captions. The caption of Figure 1 does not say which model and task are shown, how the share in panel (a) is aggregated over the layers of each parameter type, which layer and training step panel (b) uses, or how the Spearman correlation and the log-log slope are computed. In Figure 3 the y-axis labelled training loss runs from 20 to 140 with no unit, while Table 4 implies a loss of 0.23 to 0.39 per token, and the caption does not say which seed is plotted or how the moving average is smoothed.

[1] - Qwen Team, "Qwen2.5 Technical Report"

[2] - Cesista, Franz Louis, et al. "LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold." arXiv preprint arXiv:2606.12921 (2026).

[3] - Wang, Shaowen, Linxi Yu, and Jian Li. "Lora-ga: Low-rank adaptation with gradient approximation." Advances in Neural Information Processing Systems 37 (2024): 54905-54931.

Questions for authors

Please see the Weaknesses section for my questions.

Reviewer_dsykOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors study LoRA optimizers that are transformation invariant. The authors propose Temper, which accumulates per-row and per-column second moments for preconditioning. Their experiments show strong improvements over the baselines.

Critical strengths
  1. The problem the author try to solve is clear and easy to understand. The proposed solution is also clear.
  2. Despite the proposed method seems to be a bit straightforward, the empirical performance is strong compared to the baseline.
Critical weaknesses
  1. The novelty/contribution of the method feels slightly incremental.
Questions for authors

Could the author conduct experiments on other ranks? It seems like for smaller rank, the memory overhead of the proposed method might be more significant, while for larger rank the smaller relative overhead might shrinks the advantage of the proposed method.

Reviewer_SFLvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a LoRA optimizer that runs Adam-style updates directly on the low-rank matrix. Based on existing work which fixes the update direction, the current writeup also fixes the update distance.

Critical strengths

Clear, well-argued motivation. Invariance fixes the direction but not the per-coordinate scale, so the proposed method is well-motivated.

The introduced method is efficient and faster than existing method.

Critical weaknesses

Limited novelty relative to prior Riemannian adaptive methods. The proposed method already exists in matrix manifold literature, and the main novelty is computing diag(PQᵀQPᵀ) via Gram matrices and use Adafactor’s rank-one reconstruction, plus applying that to LoRA setting.

Most gaps are within seed noise. On commonsense tasks, the authors themselves note that only 2 of 16 gaps exceed the combined seed spread. The averages differ by 0.9–1.2 points with per-task stds of 1–2.

Questions for authors
  1. How were final learning rates selected: on a validation split, or on the reported test benchmarks?

  2. How does Temper compare in accuracy with a RASA-style implementation that forms G explicitly? If they match, is the contribution purely computational?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.