- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes Temper, a LoRA optimizer that treats the update as a point on the fixed-rank manifold and steps along the Riemannian gradient . It keeps moving averages of the row and column sums and , combines them with the rank-one reconstruction of Adafactor, and computes both exactly from the factored form as and , at cost and without forming an matrix. The momentum is transported by projection, the scaled direction is projected back onto the tangent space, and the iterate is retracted by a truncated SVD. The motivation is that invariance fixes the direction of a LoRA step but not its size per coordinate, and that the statistic must come from . The theory gives invariance and bounds on the row norms of the scaled matrix. Temper is first in 23 of 24 cells on Commonsense-170K, GSM8K and HumanEval, with gains of 0.9 to 3.2 points over the strongest baseline on the commonsense average and on the math and code metrics.
Critical strengths
Section 3.1 explains clearly why an elementwise statistic in factor or frame coordinates is not invariant (eq. 9) while any function of is.
The experiments use three seeds with standard deviations and a separate learning-rate grid per optimizer with the selected value inside it, and the authors say openly that only two of the sixteen commonsense gaps exceed the seed spread. The supplementary code documents in detail how the baselines were reimplemented.
Critical weaknesses
W1. I am not sure that the reported results come from the proposed algorithm.
First, it seems that the Qwen-2.5-7B numbers on GSM8K and HumanEval lie well below those that the Qwen2.5 Technical Report [1] gives for the base model. The setups differ. The report evaluates the pre-trained model with few-shot prompts, while the paper evaluates fine-tuned models and states neither the number of shots, the prompt format and the decoding, nor the sampling temperature behind Pass@k from 20 samples, and it does not say whether the base or the instruct model was used. The gap therefore does not by itself show that fine-tuning hurts. Without the metric values of the models before fine-tuning, however, it is hard to tell whether fine-tuning helps at all and how well each optimizer performs.
Second, when I checked the attached repository and tried to work out how to run the method, I found that the class Temper, which the package exports as the entry point, runs AdamW on the LoRA factors and not a Riemannian step on the manifold. With the default settings its first phase is AdamW applied to and separately, and the later phase, which a controller switches on, also works on the factors. This class has no tangent projection and no SVD retraction. The method of Section 3 seems to be implemented instead by the class Riemannion. When I read this class in detail, it seemed to me to implement heavy-ball momentum without the factor and without the bias correction , while eq. (16) and Table 10 describe a moving average with this correction. If so, and if the second moment is read from as Appendix E states, Temper effectively runs with a warm-up in the code, since its step grows by up to a factor of 10 over the first steps, while the other methods run at the constant rate stated in Table 9.
W2. The motivation of the method is overstated.
First, Riemannian methods based on Muon also normalize the step. The paper motivates the second moment by saying that invariance fixes only the direction of the step, while its size along each coordinate has to be set by an adaptive second moment, and that "Riemannion and LoRA-Muon carry no adaptive scales at all" (Section 1). The second claim is too strong. Any optimizer can be viewed as a preconditioner applied to the gradient, followed by a step along the resulting update. For Muon this preconditioner can be written explicitly. Exact Muon takes the polar factor of the momentum as the update, and the polar factor is the momentum multiplied by matrices on the left and on the right. For , , with pseudo-inverse powers when is rank-deficient, which rescales each singular direction of the momentum by . Riemannian methods that transform the Riemannian gradient in this way, such as Riemannion and LoRA-Muon [2], therefore also apply a non-diagonal matrix preconditioner.
Second, the claim that most of the norm of the Euclidean gradient lies in the normal part depends on the initialization. The paper presents the method as Riemannian, so the starting point has to lie on the manifold of rank- matrices. This excludes the usual start and calls for another initialization, a random one or one of those proposed in other work, such as LoRA-GA [3] or LOI from the Riemannion paper. The ratio can depend on this choice. The paper does not study this question. Figure 1(a) starts at the value for random subspaces, which suggests a random initialization, but the paper neither states its initialization nor shows how much the share drops under gradient-based ones.
Third, RAdamW and AdaPreLoRA read their statistic from by design. Reading it from is a modeling choice, and the ablation of Table 4 is not enough to support it: the table reports no standard deviations and does not say whether the learning rate was tuned separately for each row.
W3. The exposition is hard to follow.
Notation is often used before it is defined, or is not defined in the main text at all. The introduction uses and the unmagnified gradients before eqs. (10) and (2) define them, and Section 4.2 speaks of "removing the magnification" without saying what the magnification is. Table 1 lists LoRA-Pro as reading its statistic from , a symbol that is defined nowhere. Section 3.3 uses the previous frames and , which only Appendix A.8 defines, and in eq. (17) is never defined as the rank- truncated SVD. Table 4 reports a "Trained ppl" column without saying on which data or at which step it is measured. The main text also relies on Propositions 3 and 5, which are stated only in the appendix. The text has many stylistic and grammatical slips and a few missing references. Examples are "and refer to Appendix D" (Section 3.3), "and merges it afterwards" (Section 5), "the definitions gives" (Appendix B) and textual citations such as "AdaLoRA Zhang et al. (2023)" and "CodeAlpaca Chaudhary (2023)" where parenthetical ones are meant. The proof of Proposition 6 has a broken reference "(??)", and Llama-3.2-3B, which Figure 1 and Table 5 use, is never cited.
The figures need fuller captions. The caption of Figure 1 does not say which model and task are shown, how the share in panel (a) is aggregated over the layers of each parameter type, which layer and training step panel (b) uses, or how the Spearman correlation and the log-log slope are computed. In Figure 3 the y-axis labelled training loss runs from 20 to 140 with no unit, while Table 4 implies a loss of 0.23 to 0.39 per token, and the caption does not say which seed is plotted or how the moving average is smoothed.
[1] - Qwen Team, "Qwen2.5 Technical Report"
[2] - Cesista, Franz Louis, et al. "LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold." arXiv preprint arXiv:2606.12921 (2026).
[3] - Wang, Shaowen, Linxi Yu, and Jian Li. "Lora-ga: Low-rank adaptation with gradient approximation." Advances in Neural Information Processing Systems 37 (2024): 54905-54931.
Questions for authors
Please see the Weaknesses section for my questions.