- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ProLand, a retraction-free optimization method for LoRA with an orthogonality constraint on one factor. It constructs a search direction based on Muon for the low-rank matrix and computes coordinated updates for and . A Landing correction reduces orthogonality violations without requiring a retraction at each iteration. The method also has an efficient implementation that avoids storing full gradient and momentum matrices.
Under smoothness and stochastic-gradient assumptions, with the singular values of uniformly bounded above and away from zero, the authors establish deterministic and stochastic convergence bounds for stationarity and constraint violation. Experiments evaluate ProLand on language understanding, commonsense reasoning, and code generation, with additional ablations examining the Landing correction.
Critical strengths
- The factor updates and Landing correction are designed to work together. This gives a clear way to optimize the low-rank matrix while controlling orthogonality violations.
- The analysis provides deterministic and stochastic stationarity bounds together with feasibility guarantees. This supports the method under the stated assumptions.
- The low-rank implementation avoids storing full gradient and momentum matrices. This is useful for keeping the optimizer practical for LoRA.
- The method performs well across several models and tasks. The controlled ablation in Table 5, p. 9, also supports the benefit of the Landing correction.
Critical weaknesses
- Limited coverage of the practical algorithm. Assumption 4.2, p. 5, requires the singular values of to remain uniformly bounded above and away from zero, but entry into this regime from zero initialization is not established, as noted on p. 28. The analysis also does not cover all numerical approximations used in practice on p. 40.
- Guidance for choosing the Landing coefficient. The paper includes diagnostics with several Landing coefficients, but it is not yet clear how to choose this parameter when the rank or learning rate changes. Practical guidance based on the existing results would make the method easier to use.
Questions for authors
- It would be better if the authors clarify how the theoretical assumptions relate to the initialization and numerical approximations used in the experiments, and what the existing diagnostics suggest about their effect.
- Based on the existing diagnostics, how should a practitioner choose the Landing coefficient? Does the choice need to change with the adapter rank or learning rate?
5 comments
Author Response to Reviewer 7SJM (2/2: [W2, Q2])
[W2, Q2] Practical selection of the Landing coefficient
We thank the reviewer for this practical suggestion. We recommend (applied directly as , unscaled by ) as the default across adapter ranks and learning rates.
If larger step sizes or weaker conditioning elevate monitored (e.g., above ), can be increased within (such as ).
Dependence on the learning rate. In , Lemma D.1 guarantees safe-region contraction for any with local rate independent of , while Proposition 4.1 bounds the steady-state residual proportionally to .
Across 18 Qwen2.5-3B runs (CodeAlpaca, ; Figure 5), suppresses median Stiefel drift by two orders of magnitude relative to under both linear-decay and constant schedules while leaving validation loss essentially unchanged.
A new 3-seed sweep on Llama-3-8B (, GSM8K; Table 12) further confirms that decouples feasibility control from step-size selection across a learning-rate span:
| Base LR (, GSM8K) | (Def.) | Med. () | ||||
|---|---|---|---|---|---|---|
| ( opt.) | 63.89 ± 0.66 | 63.48 ± 0.74 | 63.53 ± 0.27 | 63.61 ± 1.62 | 64.14 ± 0.82 | |
| (val.-selected) | 65.35 ± 0.13 | 65.50 ± 0.80 | 66.84 ± 0.90 | 66.01 ± 0.61 | ||
| ( opt.) | 63.96 ± 0.24 | 63.36 ± 1.10 | 63.33 ± 0.86 | 63.96 ± 0.16 | 63.63 ± 0.84 |
Without Landing (), Stiefel drift grows sharply as increases, whereas every keeps median near across the entire step-size span while improving peak validation-selected accuracy over .
Dependence on the adapter rank. In Proposition 4.1, has no explicit dependence on , and RMS step scaling () offsets the Frobenius factor.
Empirically, holding the default strictly fixed achieves the highest multi-task average at every evaluated rank across all main benchmarks—DeBERTa-v3-base (, Table 1 and Table 10 preset), Qwen2.5-3B and Llama-3.2-3B (, Tables 2–3), Moonlight-16B-A3B (, Table 4), Llama-3-8B (, Table 7), and Llama-3-70B (, Table 7).
Separately, in matched single-component ablations—both at fixed on Qwen2.5-3B (, Table 11) and across shared configurations () on Llama-3-8B (, Table 5)—infeasible Landing consistently outperforms and per-step polar retraction while keeping Stiefel residuals bounded.
We actively conducted extensive multi-rank and sensitivity experiments during the rebuttal period; given the limited rebuttal window, denser cross-rank sensitivity grids will be completed in the subsequent revision.
Revisions in the manuscript. Practitioner guidelines and the sensitivity grid are included in Appendix I.1, Table 12 (p. 39) and Appendix L.3 (p. 45), complementing the trajectory diagnostics in Appendix D.2, Figure 5 (p. 24) and multi-rank ablations in Section 5.4, Table 5 (p. 10).
Follow-up Author Response
Dear Reviewer 7SJM,
Following up on our initial response regarding [W2, Q2] (using the fixed default Landing coefficient across tasks, ranks, and model scales without per-task retuning), we share newly completed baseline and multi-seed evaluations across code generation, 3B language benchmarks, and Llama-3-70B scaling. Across these setups, ProLand—holding fixed throughout—consistently outperforms the newly added baselines:
-
Code generation on 3B () and 16B MoE (; Appendix K, Tables 14–15): Under the 3-point CodeAlpaca-holdout selection protocol on HumanEval, HumanEval+, MBPP, and MBPP+ (
pass@1), Qwen2.5-3B four-metric averages are60.70%(ProLand),60.40%(PoLAR),59.94%(Riemannion),59.81%(Stiefel-LoRA),58.48%(LoRA-Pro), and57.52%(DoRA); Llama-3.2-3B averages are37.94%(sMuon),37.45%(ProLand),37.39%(Stiefel-LoRA),37.35%(LoRA-Pro),35.63%(DoRA), and34.83%(PoLAR). On Moonlight-16B-A3B (), ProLand reaches59.89%compared with59.37%for sMuon and56.31%for LoRA-Pro (56.10 / 48.78 / 66.67 / 53.70). -
3-seed 3B language benchmarks at (Appendix G.5): Under the 3-seed (
2026, 1337, 3407), 3-point PIQA-holdout protocol across all seven language tasks (63 newly trained task-specific adapters), seven-task averages on Qwen2.5-3B are67.40 ± 0.45%(ProLand),67.13 ± 0.12%(LoRA-Pro), and66.31 ± 0.19%(PoLAR); on Llama-3.2-3B, seven-task averages are65.72 ± 0.52%(ProLand),65.59 ± 0.14%(LoRA-Pro),64.37 ± 0.06%(DoRA), and63.91 ± 0.28%(PoLAR). -
Llama-3-70B () multi-task scaling with DoRA (Appendix G.3, Table 7): Completing 3-seed training and ARC-Challenge evaluation for DoRA (
64.90 ± 0.40%) alongsideseed=2026CoT GSM8K decoding (85.60%,75.48%two-task average), DoRA runs at0.23×relative throughput and1.14×relative peak allocated memory, compared with65.19 ± 1.34%(ARC-C),86.40 ± 0.48%(GSM8K;75.80 ± 0.75%average),1.02×throughput, and0.97×memory for ProLand.
Revisions in the manuscript. Updated in Section 5.1 (p. 7), Appendix G.3, Table 7 (p. 36), Appendix G.5 (p. 38), Appendix K, Tables 14–15 (pp. 41–42), and Appendix L, Tables 17–20 (pp. 42, 44–46).
Author Response to Reviewer 7SJM (1/2: [W1, Q1])
We sincerely thank Reviewer 7SJM for the clear, constructive review and for recognizing that ProLand's "factor updates and Landing correction are designed to work together", that our analysis "provides deterministic and stochastic stationarity bounds together with feasibility guarantees", that our "low-rank implementation avoids storing full gradient and momentum matrices", and that ProLand "performs well across several models and tasks."
[W1, Q1] Theoretical assumptions, initialization, and numerical approximations
We thank the reviewer for highlighting the relationship between the theoretical assumptions and the practical implementation. Our recorded post-warmup trajectories exhibit a positive lower bound on the smallest singular value of At, while new Llama-3-8B experiments demonstrate rapid rank recovery from zero initialization. The finite-step numerical directions also closely agree with FP64 references.
Initialization and factor regularity. As specified in Section 4, Assumption 4.2 applies after a finite initialization phase. This regularity is supported by our original Qwen2.5-3B CodeAlpaca diagnostics: across 18 runs, each monitoring 252 adapters, all recorded post-warmup states satisfy σr(At)≥3.65×10−4 and σ1(At)≤8.06×10−2, with a maximum condition number below 4.45 (Figure 6).
To further examine the transition from standard zero initialization, we added a Llama-3-8B ablation (r=64, three seeds). All 224 adapter pairs per seed recover full row rank after the first update (σr(A1)>10−10).
Moreover, zero and tiny full-rank initializations yield nearly identical final median smallest singular values (0.0181 vs. 0.0180) and condition numbers (1.510 vs. 1.512), respectively (Table 13). Together, these observations support favorable factor conditioning following the initialization phase, consistent with Assumption 4.2.
Numerical approximations. Our convergence analysis in Section 4 uses exact polar factors and pseudoinverses. In Qwen2.5-3B direction replay, the iterative and FP64 reference directions reach a median cosine similarity of 0.99925 across 2,268 states (Figure 8).
On Llama-3-8B, the default 12/8-step implementation and direct small-core SVD/eigensolves achieve 66.84±0.90% and 66.24±0.84% GSM8K accuracy, with identical median final Gram residuals of 1.42×10−5.
BF16 storage increases the median residual to 1.75×10−3, while retaining 65.98±0.12% accuracy; excessive damping (ε=10−2) reduces accuracy to 61.01% (Table 13). Extending the convergence analysis to these numerical approximations is left for future work.
Revisions in the manuscript. Section 4 (pp. 5–6) specifies the post-initialization analysis window. Appendix D.5 and Figure 6 (pp. 26–27) report the original factor-spectrum diagnostics, while Appendix J.1, Table 13 (p. 40) provides the new initialization and numerical ablations. The direction-replay results appear in Appendix J, Figure 8 (p. 40).