acceptodds
Under review as a conference paper at ICLR 2027

ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-tuning

21 views

Abstract

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning but can suffer from redundant basis directions. Constraining the basis factor to the Stiefel manifold addresses this redundancy through orthonormality. Landing methods allow temporarily infeasible iterates to explore a neighborhood of the manifold while progressively restoring orthogonality without per-step retractions. To improve optimization efficiency while retaining this flexibility, we introduce ProLand (Product-space Landing), a retraction-free optimizer that combines coordinated factor updates with Muon-style spectral directions. By designing the updates in the space of the represented weight increment, ProLand separates objective optimization from orthogonality restoration, supporting efficient feasibility recovery. This coordinated construction also enables Muon-style directions to guide objective optimization while preserving that separation. Experiments cover four GLUE validation tasks on DeBERTa-v3-base and seven language benchmarks on Qwen2.5-3B, Llama-3.2-3B, and Moonlight-16B-A3B across multiple adapter ranks. Across these 11 tasks, ProLand achieves the highest average score among the compared methods in every tested encoder and decoder model–rank configuration. Code is available at https://anonymous.4open.science/r/proland-supplement-7A24/.

Primary area:
optimization
Keywords:
Low-rank adaptation, Muon, Spectral optimization, Stiefel manifold, Retraction-free optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7SJMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProLand, a retraction-free optimization method for LoRA with an orthogonality constraint on one factor. It constructs a search direction based on Muon for the low-rank matrix BABA and computes coordinated updates for AA and BB. A Landing correction reduces orthogonality violations without requiring a retraction at each iteration. The method also has an efficient implementation that avoids storing full gradient and momentum matrices.

Under smoothness and stochastic-gradient assumptions, with the singular values of AtA_t uniformly bounded above and away from zero, the authors establish deterministic and stochastic convergence bounds for stationarity and constraint violation. Experiments evaluate ProLand on language understanding, commonsense reasoning, and code generation, with additional ablations examining the Landing correction.

Critical strengths
  1. The factor updates and Landing correction are designed to work together. This gives a clear way to optimize the low-rank matrix while controlling orthogonality violations.
  2. The analysis provides deterministic and stochastic stationarity bounds together with feasibility guarantees. This supports the method under the stated assumptions.
  3. The low-rank implementation avoids storing full gradient and momentum matrices. This is useful for keeping the optimizer practical for LoRA.
  4. The method performs well across several models and tasks. The controlled ablation in Table 5, p. 9, also supports the benefit of the Landing correction.
Critical weaknesses
  1. Limited coverage of the practical algorithm. Assumption 4.2, p. 5, requires the singular values of AtA_t to remain uniformly bounded above and away from zero, but entry into this regime from zero initialization is not established, as noted on p. 28. The analysis also does not cover all numerical approximations used in practice on p. 40.
  2. Guidance for choosing the Landing coefficient. The paper includes diagnostics with several Landing coefficients, but it is not yet clear how to choose this parameter when the rank or learning rate changes. Practical guidance based on the existing results would make the method easier to use.
Questions for authors
  1. It would be better if the authors clarify how the theoretical assumptions relate to the initialization and numerical approximations used in the experiments, and what the existing diagnostics suggest about their effect.
  2. Based on the existing diagnostics, how should a practitioner choose the Landing coefficient? Does the choice need to change with the adapter rank or learning rate?
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer 7SJM (1/2: [W1, Q1])

We sincerely thank Reviewer 7SJM for the clear, constructive review and for recognizing that ProLand's "factor updates and Landing correction are designed to work together", that our analysis "provides deterministic and stochastic stationarity bounds together with feasibility guarantees", that our "low-rank implementation avoids storing full gradient and momentum matrices", and that ProLand "performs well across several models and tasks."

[W1, Q1] Theoretical assumptions, initialization, and numerical approximations

We thank the reviewer for highlighting the relationship between the theoretical assumptions and the practical implementation. Our recorded post-warmup trajectories exhibit a positive lower bound on the smallest singular value of AtA_t, while new Llama-3-8B experiments demonstrate rapid rank recovery from zero initialization. The finite-step numerical directions also closely agree with FP64 references.

Initialization and factor regularity. As specified in Section 4, Assumption 4.2 applies after a finite initialization phase. This regularity is supported by our original Qwen2.5-3B CodeAlpaca diagnostics: across 18 runs, each monitoring 252 adapters, all recorded post-warmup states satisfy σr(At)≥3.65×10−4\sigma_r(A_t) \ge 3.65 \times 10^{-4} and σ1(At)≤8.06×10−2\sigma_1(A_t) \le 8.06 \times 10^{-2}, with a maximum condition number below 4.454.45 (Figure 6).

To further examine the transition from standard zero initialization, we added a Llama-3-8B ablation (r=64r=64, three seeds). All 224 adapter pairs per seed recover full row rank after the first update (σr(A1)>10−10\sigma_r(A_1) > 10^{-10}).

Moreover, zero and tiny full-rank initializations yield nearly identical final median smallest singular values (0.01810.0181 vs. 0.01800.0180) and condition numbers (1.5101.510 vs. 1.5121.512), respectively (Table 13). Together, these observations support favorable factor conditioning following the initialization phase, consistent with Assumption 4.2.

Numerical approximations. Our convergence analysis in Section 4 uses exact polar factors and pseudoinverses. In Qwen2.5-3B direction replay, the iterative and FP64 reference directions reach a median cosine similarity of 0.999250.99925 across 2,268 states (Figure 8).

On Llama-3-8B, the default 12/8-step implementation and direct small-core SVD/eigensolves achieve 66.84±0.90%66.84 \pm 0.90\% and 66.24±0.84%66.24 \pm 0.84\% GSM8K accuracy, with identical median final Gram residuals of 1.42×10−51.42 \times 10^{-5}.

BF16 storage increases the median residual to 1.75×10−31.75 \times 10^{-3}, while retaining 65.98±0.12%65.98 \pm 0.12\% accuracy; excessive damping (ε=10−2\varepsilon = 10^{-2}) reduces accuracy to 61.01%61.01\% (Table 13). Extending the convergence analysis to these numerical approximations is left for future work.

Revisions in the manuscript. Section 4 (pp. 5–6) specifies the post-initialization analysis window. Appendix D.5 and Figure 6 (pp. 26–27) report the original factor-spectrum diagnostics, while Appendix J.1, Table 13 (p. 40) provides the new initialization and numerical ablations. The direction-replay results appear in Appendix J, Figure 8 (p. 40).

AuthorsOpenReview ↗

Author Response to Reviewer 7SJM (2/2: [W2, Q2])

[W2, Q2] Practical selection of the Landing coefficient

We thank the reviewer for this practical suggestion. We recommend ρ=0.05\rho = 0.05 (applied directly as −ρBtCt-\rho B_t C_t, unscaled by γt\gamma_t) as the default across adapter ranks and learning rates.

If larger step sizes or weaker AtA_t conditioning elevate monitored ∥Bt⊤Bt−Ir∥op\|B_t^\top B_t - I_r\|_{\mathrm{op}} (e.g., above 0.050.05), ρ\rho can be increased within (0,1/2](0, 1/2] (such as ρ=0.20\rho = 0.20).

Dependence on the learning rate. In Ct+1=pρ(Ct)+γt2DB,t⊤DB,tC_{t+1} = p_\rho(C_t) + \gamma_t^2 D_{B,t}^\top D_{B,t}, Lemma D.1 guarantees safe-region contraction for any ρ∈(0,1/2]\rho \in (0, 1/2] with local rate 1−2ρ1 - 2\rho independent of γt\gamma_t, while Proposition 4.1 bounds the steady-state residual proportionally to γt2/a−2\gamma_t^2 / a_-^2.

Across 18 Qwen2.5-3B runs (CodeAlpaca, r=16r=16; Figure 5), ρ∈{0.05,0.45}\rho \in \{0.05, 0.45\} suppresses median Stiefel drift by two orders of magnitude relative to ρ=0\rho = 0 under both linear-decay and constant schedules while leaving validation loss essentially unchanged.

A new 3-seed ρ×γ\rho \times \gamma sweep on Llama-3-8B (r=64r=64, GSM8K; Table 12) further confirms that ρ=0.05\rho = 0.05 decouples feasibility control from step-size selection across a 6×6\times learning-rate span:

Base LR γ\gamma (r=64r=64, GSM8K)ρ=0\rho = 0ρ=0.005\rho = 0.005ρ=0.05\rho = 0.05 (Def.)ρ=0.20\rho = 0.20ρ=0.45\rho = 0.45Med. ∥CT∥op\Vert C_T\Vert_{\mathrm{op}} (0→0.050 \to 0.05)
3.0×10−63.0 \times 10^{-6} (0.5×0.5\times opt.)63.89 ± 0.6663.48 ± 0.7463.53 ± 0.2763.61 ± 1.6264.14 ± 0.821.89e-2→5.07e-61.89\text{e-}2 \to 5.07\text{e-}6
6.0×10−66.0 \times 10^{-6} (val.-selected)65.35 ± 0.1365.50 ± 0.8066.84 ± 0.9066.54±0.88‾\underline{66.54 \pm 0.88}66.01 ± 0.617.96e-2→1.42e-57.96\text{e-}2 \to 1.42\text{e-}5
1.8×10−51.8 \times 10^{-5} (3.0×3.0\times opt.)63.96 ± 0.2463.36 ± 1.1063.33 ± 0.8663.96 ± 0.1663.63 ± 0.843.51e-1→1.92e-53.51\text{e-}1 \to 1.92\text{e-}5

Without Landing (ρ=0\rho = 0), Stiefel drift grows sharply as γ\gamma increases, whereas every ρ∈{0.05,0.20,0.45}\rho \in \{0.05, 0.20, 0.45\} keeps median ∥CT∥op\|C_T\|_{\mathrm{op}} near 10−510^{-5} across the entire 6×6\times step-size span while improving peak validation-selected accuracy over ρ=0\rho = 0.

Dependence on the adapter rank. In Proposition 4.1, qCq_C has no explicit dependence on rr, and RMS step scaling (γeff∝1/r\gamma_{\mathrm{eff}} \propto 1/\sqrt{r}) offsets the r\sqrt{r} Frobenius factor.

Empirically, holding the default ρ=0.05\rho = 0.05 strictly fixed achieves the highest multi-task average at every evaluated rank across all main benchmarks—DeBERTa-v3-base (r=16r=16, Table 1 and Table 10 preset), Qwen2.5-3B and Llama-3.2-3B (r∈{4,8,16}r \in \{4, 8, 16\}, Tables 2–3), Moonlight-16B-A3B (r=16r=16, Table 4), Llama-3-8B (r∈{16,32,64}r \in \{16, 32, 64\}, Table 7), and Llama-3-70B (r=32r=32, Table 7).

Separately, in matched single-component ablations—both at fixed ρ=0.05\rho = 0.05 on Qwen2.5-3B (r=16r=16, Table 11) and across shared (γ,ρ)(\gamma, \rho) configurations (ρ∈{0.005,0.05,0.20}\rho \in \{0.005, 0.05, 0.20\}) on Llama-3-8B (r∈{16,32,64}r \in \{16, 32, 64\}, Table 5)—infeasible Landing consistently outperforms ρ=0\rho = 0 and per-step polar retraction while keeping Stiefel residuals bounded.

We actively conducted extensive multi-rank and ρ×γ\rho \times \gamma sensitivity experiments during the rebuttal period; given the limited rebuttal window, denser cross-rank sensitivity grids will be completed in the subsequent revision.

Revisions in the manuscript. Practitioner guidelines and the ρ×γ\rho \times \gamma sensitivity grid are included in Appendix I.1, Table 12 (p. 39) and Appendix L.3 (p. 45), complementing the trajectory diagnostics in Appendix D.2, Figure 5 (p. 24) and multi-rank ablations in Section 5.4, Table 5 (p. 10).

AuthorsOpenReview ↗

Follow-up Author Response

Dear Reviewer 7SJM,

Following up on our initial response regarding [W2, Q2] (using the fixed default Landing coefficient ρ=0.05\rho = 0.05 across tasks, ranks, and model scales without per-task retuning), we share newly completed baseline and multi-seed evaluations across code generation, 3B language benchmarks, and Llama-3-70B scaling. Across these setups, ProLand—holding ρ=0.05\rho = 0.05 fixed throughout—consistently outperforms the newly added baselines:

  1. Code generation on 3B (r=8r=8) and 16B MoE (r=16r=16; Appendix K, Tables 14–15): Under the 3-point CodeAlpaca-holdout selection protocol on HumanEval, HumanEval+, MBPP, and MBPP+ (pass@1), Qwen2.5-3B four-metric averages are 60.70% (ProLand), 60.40% (PoLAR), 59.94% (Riemannion), 59.81% (Stiefel-LoRA), 58.48% (LoRA-Pro), and 57.52% (DoRA); Llama-3.2-3B averages are 37.94% (sMuon), 37.45% (ProLand), 37.39% (Stiefel-LoRA), 37.35% (LoRA-Pro), 35.63% (DoRA), and 34.83% (PoLAR). On Moonlight-16B-A3B (r=16r=16), ProLand reaches 59.89% compared with 59.37% for sMuon and 56.31% for LoRA-Pro (56.10 / 48.78 / 66.67 / 53.70).

  2. 3-seed 3B language benchmarks at r=16r=16 (Appendix G.5): Under the 3-seed (2026, 1337, 3407), 3-point PIQA-holdout protocol across all seven language tasks (63 newly trained task-specific adapters), seven-task averages on Qwen2.5-3B are 67.40 ± 0.45% (ProLand), 67.13 ± 0.12% (LoRA-Pro), and 66.31 ± 0.19% (PoLAR); on Llama-3.2-3B, seven-task averages are 65.72 ± 0.52% (ProLand), 65.59 ± 0.14% (LoRA-Pro), 64.37 ± 0.06% (DoRA), and 63.91 ± 0.28% (PoLAR).

  3. Llama-3-70B (r=32r=32) multi-task scaling with DoRA (Appendix G.3, Table 7): Completing 3-seed training and ARC-Challenge evaluation for DoRA (64.90 ± 0.40%) alongside seed=2026 CoT GSM8K decoding (85.60%, 75.48% two-task average), DoRA runs at 0.23× relative throughput and 1.14× relative peak allocated memory, compared with 65.19 ± 1.34% (ARC-C), 86.40 ± 0.48% (GSM8K; 75.80 ± 0.75% average), 1.02× throughput, and 0.97× memory for ProLand.

Revisions in the manuscript. Updated in Section 5.1 (p. 7), Appendix G.3, Table 7 (p. 36), Appendix G.5 (p. 38), Appendix K, Tables 14–15 (pp. 41–42), and Appendix L, Tables 17–20 (pp. 42, 44–46).

Reviewer_zVc3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This paper advocates ProLand, a retraction-free optimizer for Stiefel-LoRA that combines Muon-style spectral updates with landing. The key idea is to construct optimization directions at the product level W=BAW=BA, then realize them through factor updates satisfying B⊤DB=0B^\top D_B=0. This allows the optimization step to have no first-order effect on the orthogonality constraint, while a separate landing correction restores feasibility. Convergence guarantees are provided for both product-space stationarity (∣∣R∣∣∗|| R ||_*) and constraint violation (∣∣C∣∣op|| C || _{\text{op}}). Competitive performance is reported over several Muon- and manifold-based alternatives on GLUE and reasoning tasks.

Critical strengths
  • Interesting attempt to combine Muon updates with Landing for Stiefel-LoRA.
  • The feasibility analysis is technically solid; the B⊤DB=0B^\top D_B=0 construction yields a clean constraint-residual recursion and enables separate control of stationarity and feasibility.
  • Both convergence guarantees and an exact low-rank implementation are provided.
  • Extensive numerical results on GLUE, language understanding, and code-generation benchmarks.
Critical weaknesses
  • The paper is difficult to follow. Section 3 starts directly with Algorithm 1, while the motivation behind the product-space construction, the role of YtY_t, and the rationale for the factor realization are explained only later.
  • While the paper combines several ideas in a technically coherent way, the novel contribution relative to prior work (landing methods, Muon-style updates, and product-aware LoRA optimization) could be articulated more clearly.
  • The theoretical results rely on a uniform lower bound on σr(At)\sigma_r(A_t). While it's reasonable to assume full-rank, the uniform lower bound on the smallest singular value needs further justification.
  • The experiments are restricted to relatively small models (mainly 3B-scale models, with one 16B MoE model). It remains unclear whether the observed gains persist at larger scales where LoRA is most commonly deployed.
  • Although many benchmarks are reported, most decoder-language results use a single training seed, making it difficult to assess the statistical significance and robustness of the improvements.
Questions for authors
  • The paper relies on the stronger sufficient condition B⊤DB=0B^\top D_B=0, which plays a central role in the feasibility analysis. Beyond the resulting Gram-recursion simplification, is there a more geometric or optimization-based intuition for why this particular realization should be preferred over other valid realizations of the same product-space direction?
  • The convergence analysis assumes a uniform lower bound on σr(At)\sigma_r(A_t). How sensitive is ProLand to violations of this assumption in practice, and how often is it observed empirically during training?
  • The paper motivates the use of Landing over per-step retractions, but the empirical comparison between Landing and retraction is relatively limited. Could you provide broader evidence that the benefits consistently come from the infeasible optimization trajectory rather than implementation or tuning differences?
  • Can you provide results on larger-scale models (e.g., 8B dense models) to better assess whether the observed gains persist at scales?

I'm currently borderline about this submission yet leaning to accept if the concerns can be addressed.

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer zVc3 (1/3: [W1, W2, W3, Q2])

We sincerely thank Reviewer zVc3 for the thoughtful, expert review, and for highlighting that our feasibility analysis is "technically solid" and that the B⊤DB=0B^\top D_B = 0 construction "yields a clean constraint-residual recursion and enables separate control of stationarity and feasibility."

[W1] Organization of Section 3

We thank the reviewer for pointing out the organizational and presentation issues in Section 3. We have reorganized Section 3 to present the motivation and mathematical rationale for each construction before Algorithm 1.

Specifically, we first explain why the product-dependent objective motivates spectral updates in the product space rather than independently in the two factor spaces. We then introduce momentum reprojection into the evolving first-order product space, the construction of the polar factor YtY_t and its subsequent projection to ensure realizability, and the compatible factor updates combined with Landing for Stiefel feasibility restoration. Algorithm 1 now consolidates these constructions, followed by a separate description of the exact low-rank implementation.

This revised organization (Section 3, p. 4) establishes a clearer progression from design motivation and update-direction construction to factor realization, feasibility restoration, and efficient implementation.


[W2] Positioning of the novel contribution

We thank the reviewer for this constructive comment. We have revised the Introduction and Section 1.1 to clarify ProLand's distinguishing contribution: realizing product-space spectral directions while preserving controlled Stiefel feasibility restoration along infeasible optimization trajectories.

ProLand addresses the interaction between product-space spectral optimization and infeasible Stiefel restoration through a horizontal factor realization that preserves the prescribed product-space direction even at infeasible iterates. This construction eliminates the objective update's first-order contribution to the Gram variation and the optimization–Landing cross terms, yielding a second-order feasibility perturbation and joint stationarity–feasibility guarantees. ProLand also admits an exact low-rank implementation.

Revisions in the manuscript. Section 1.1 clarifies the distinction from prior methods, while Appendix A (Table 6) provides a detailed structural comparison.


[W3, Q2] Uniform lower bound on the smallest singular value of AtA_t

We thank the reviewer for raising this important question regarding the uniform lower bound on the smallest singular value of AtA_t. Our experiments show that ProLand handles the rank-deficient standard zero initialization and rapidly recovers full-rank factors, while no rank collapse is observed in the recorded post-warmup training trajectories.

Empirical sensitivity to initialization. We added a Llama-3-8B initialization ablation (r=64r=64, three seeds). Starting from A0=0A_0 = 0, all 224 adapter pairs per seed recover full row rank after the first update. Compared with a small full-rank initialization, standard zero initialization yields nearly identical final median smallest singular values (0.01810.0181 vs. 0.01800.0180) and condition numbers (1.5101.510 vs. 1.5121.512), while achieving GSM8K accuracy of 66.84±0.90%66.84 \pm 0.90\% versus 65.43±0.69%65.43 \pm 0.69\% (Table 13).

Observed frequency during training. Our original Qwen2.5-3B CodeAlpaca diagnostics cover 18 runs, each monitoring 252 adapters. Across all recorded post-warmup states, we observe σr(At)≥3.65×10−4\sigma_r(A_t) \ge 3.65 \times 10^{-4} and σ1(At)≤8.06×10−2\sigma_1(A_t) \le 8.06 \times 10^{-2}, with a maximum condition number below 4.454.45 (Figure 6). These observations support favorable factor conditioning after a finite initialization phase, consistent with Assumption 4.2.

Revisions in the manuscript. As specified in Section 4 (pp. 5–6), Assumption 4.2 applies after a finite initialization phase. The original factor-spectrum diagnostics are provided in Appendix D.5 and Figure 6 (pp. 26–27), while the new initialization ablation is reported in Appendix J.1, Table 13 (p. 40).

AuthorsOpenReview ↗

Author Response to Reviewer zVc3 (2/3: [W4, Q4, W5])

[W4, Q4] Scaling to larger dense models

ProLand achieves the highest two-task average across all evaluated ranks and model scales on Llama-3-8B (r∈{16,32,64}r \in \{16, 32, 64\}) and Llama-3-70B (r=32r=32).

All runs use three independent training seeds (2026, 1337, 3407) on ARC-Challenge and chain-of-thought GSM8K (per-task 3-epoch adapters on 8B; shared 1-epoch multi-task adapter on 70B), under a matched 8-candidate validation grid on dev seed=4242 with uniform zero weight decay across all methods and fixed ρ=0.05\rho = 0.05 for ProLand.

1. Llama-3-8B (r∈{16,32,64}r \in \{16, 32, 64\}, per-task adapters):

Method (3 Seeds, Mean ± SD)r=16r=16 Avg. (%)r=32r=32 Avg. (%)r=64r=64 Avg. (%)
AdamW (LoRA)59.60 ± 0.8059.43 ± 0.3859.02 ± 0.47
sMuon58.49 ± 0.1458.40 ± 0.5958.46 ± 0.32
Riemannion58.26 ± 0.1559.28 ± 0.5059.01 ± 0.15
Stiefel-LoRA58.10 ± 0.3658.68 ± 0.7358.55 ± 0.37
PoLAR58.52 ± 0.5858.94 ± 1.0360.74 ± 0.50
DoRA59.45 ± 0.5160.37 ± 0.7761.51 ± 0.61
LoRA-Pro62.01±0.23‾\underline{62.01 \pm 0.23}62.03±0.07‾\underline{62.03 \pm 0.07}61.54±0.88‾\underline{61.54 \pm 0.88}
ProLand (Ours)62.60 ± 1.1862.23 ± 0.7063.13 ± 0.10

ProLand leads the two-task average at every rank, ranking first on ARC-C across all ranks and on both tasks at r=64r=64.

2. Llama-3-70B (r=32r=32, shared multi-task adapter):

Method (3 Seeds, Mean ± SD)ARC-C (%)GSM8K (CoT, %)Avg. (%)
AdamW (LoRA)64.85 ± 0.5384.48 ± 0.2374.66 ± 0.24
sMuon65.42±0.57‾\underline{65.42 \pm 0.57}84.51±0.66‾\underline{84.51 \pm 0.66}74.96±0.21‾\underline{74.96 \pm 0.21}
Stiefel-LoRA65.87 ± 1.3783.27 ± 2.3974.57 ± 1.42
ProLand (Ours)65.19 ± 1.3486.40 ± 0.4875.80 ± 0.75

Methods are comparable on ARC-C, whereas ProLand delivers a clear gain on multi-step GSM8K reasoning and the highest overall average while matching AdamW throughput and allocated memory.

Revisions in the manuscript. Full per-task scaling results appear in Section 5.3, Figure 2 (p. 9) and Appendix G.3, Table 7 (p. 36), with selection settings in Appendix L.3, Tables 17 and 20 (pp. 43, 45).


[W5] Multi-seed robustness of decoder-language results

We upgraded all 3B decoder language benchmarks and all Llama-3-8B/70B benchmarks to three independent training seeds (2026, 1337, 3407).

Across all 6 multi-seed 3B configurations (Qwen2.5-3B and Llama-3.2-3B across r∈{4,8,16}r \in \{4, 8, 16\} on 7 tasks, now including LoRA-Pro across all 6 settings; 126 ProLand and 882 baseline task-specific adapters), ProLand achieves the highest 3-seed 7-task mean in every configuration:

ModelRankProLand (Ours)Strongest Baseline (3-Seed Mean ± SD)Δ\Delta (pp)
Qwen2.5-3Br=4r=467.33 ± 0.3666.77 ± 0.05 (LoRA-Pro)+0.56
Qwen2.5-3Br=8r=867.37 ± 0.6566.67 ± 0.08 (AdamW) / 66.63 ± 0.38 (LoRA-Pro)+0.70
Qwen2.5-3Br=16r=1667.40 ± 0.4567.13 ± 0.12 (LoRA-Pro)+0.27
Llama-3.2-3Br=4r=465.05 ± 0.5664.90 ± 0.18 (Per-factor Muon)+0.15
Llama-3.2-3Br=8r=865.39 ± 0.5865.26 ± 0.24 (Riemannion)+0.13
Llama-3.2-3Br=16r=1665.72 ± 0.5265.59 ± 0.14 (LoRA-Pro)+0.13

Across the 42 (model, rank, task) cells in Tables 2–3, ProLand wins a clear majority (29–38 cells) against each baseline, with both two-sided paired tt-tests and Wilcoxon signed-rank tests yielding p<10−3p < 10^{-3} against all seven baselines (including LoRA-Pro).

We actively conducted extensive multi-seed experiments during the rebuttal period; given the limited rebuttal window, additional multi-seed data for the remaining setups will be completed in the subsequent revision.

Revisions in the manuscript. Full per-task 3-seed results are reported in Section 5.3, Tables 2–3 (p. 8) and Appendix G.3, Table 7 (p. 36).

AuthorsOpenReview ↗

Author Response to Reviewer zVc3 (3/3: [Q1, Q3])

[Q1] Geometric intuition for the horizontal realization

We thank the reviewer for this insightful question. The horizontal realization (ProLand), characterized by Bt⊤DB,t=0B_t^\top D_{B,t} = 0, uniquely minimizes the BB-factor update norm among compatible realizations of the same first-order product direction. For full-column-rank BtB_t and full-row-rank AtA_t, let Kt=Bt⊤DB,tK_t = B_t^\top D_{B,t}. Every compatible realization satisfies

∥DB,t(Kt)∥F2=∥DB,t(0)∥F2+tr⁡ ⁣(Kt⊤(Bt⊤Bt)−1Kt).\|D_{B,t}(K_t)\|_F^2 = \|D_{B,t}(0)\|_F^2 + \operatorname{tr}\!\left(K_t^\top (B_t^\top B_t)^{-1} K_t\right).

Thus, ProLand (Kt=0K_t = 0) uniquely minimizes the Frobenius norm of the BB-factor update among all compatible realizations of the same first-order product direction.

The distinction also extends to infeasible iterates. In the Coordinate-skew + Landing control, Kt+Kt⊤K_t + K_t^\top is generally nonzero away from the Stiefel manifold. Consequently, the exact Gram recursion retains the first-order optimization term −γt(Kt+Kt⊤)-\gamma_t(K_t + K_t^\top) and the optimization–Landing mixed term γtρ(CtKt+Kt⊤Ct)\gamma_t \rho(C_t K_t + K_t^\top C_t), where Ct=Bt⊤Bt−IrC_t = B_t^\top B_t - I_r. In contrast, ProLand enforces Kt=0K_t = 0 even at infeasible iterates, eliminating both terms exactly and yielding

Ct+1=pρ(Ct)+γt2DB,t⊤DB,t.C_{t+1} = p_\rho(C_t) + \gamma_t^2 D_{B,t}^\top D_{B,t}.

This separates the Landing contraction from the optimization-induced second-order perturbation and enables the tighter feasibility analysis.

Revisions in the manuscript. The geometric interpretation is provided in Section 3 (p. 4), the minimum-norm proof in Appendix B.3 (p. 19), and the Gram-recursion comparison in Appendix D.1 (p. 22). The experimental comparison is reported separately in our response to Reviewer bEcK ([W6]).


[Q3] Controlled comparisons between Landing and retraction

Both code-identical single-component ablations and independently tuned baseline comparisons show consistent gains from infeasible Landing over per-step retraction across models, tasks, and ranks.

1. Shared-configuration single-component ablation. Within the exact same codebase, byte-identical (B0,A0)(B_0, A_0), identical mini-batch order, shared (γ,ρ)(\gamma, \rho) configurations, and the same product-direction rule, we compare ProLand (Bt+1=Bt−γtDB,t−ρBtCtB_{t+1} = B_t - \gamma_t D_{B,t} - \rho B_t C_t), per-step polar retraction (B~t(B~t⊤B~t)−1/2\widetilde{B}_t(\widetilde{B}_t^\top \widetilde{B}_t)^{-1/2} with ρ=0\rho = 0), and unconstrained updates (ρ=0\rho = 0):

Feasibility Variant (3 Seeds)OBQA (r=16r=16)8B r=16r=16 Avg.8B r=32r=32 Avg.8B r=64r=64 Avg.8B GSM8K Med. ∥CT∥op\Vert C_T\Vert_{\mathrm{op}}
ProLand (Landing)40.73 ± 0.6963.38 ± 0.4763.05 ± 0.1763.35 ± 0.771.76e-21.76\text{e-}2 / 1.59e-51.59\text{e-}5 / 1.31e-61.31\text{e-}6
With polar retraction39.67±0.76‾\underline{39.67 \pm 0.76}61.87±0.46‾\underline{61.87 \pm 0.46}61.72 ± 0.2362.17±0.27‾\underline{62.17 \pm 0.27}7.54e-87.54\text{e-}8 / 8.10e-88.10\text{e-}8 / 9.16e-89.16\text{e-}8
Without Landing (ρ=0\rho=0)38.93 ± 0.6161.77 ± 1.0062.25±0.75‾\underline{62.25 \pm 0.75}62.03 ± 0.842.15e-12.15\text{e-}1 / 1.40e-11.40\text{e-}1 / 7.96e-27.96\text{e-}2

Landing outperforms exact polar retraction on every rank and task (per-task scores in Tables 5 and 11) while keeping Stiefel residuals tightly bounded and reducing full optimizer-step latency on Llama-3-8B (Table 8).

2. Independently tuned manifold baselines. Under matched validation budgets, ProLand (ρ=0.05\rho = 0.05) likewise consistently outperforms external manifold baselines across scales:

  • Llama-3-8B and Llama-3-70B: Under independent 8-candidate learning-rate selection with zero weight decay (see [W4, Q4]), ProLand outperforms retraction-based Riemannion and Stiefel-LoRA as well as two-sided factor-Landing PoLAR across all ranks on Llama-3-8B, and outperforms Stiefel-LoRA on Llama-3-70B.

  • DeBERTa-v3-base (r=16r=16): ProLand similarly outperforms Riemannion, Manifold-LoRA, Stiefel-LoRA, StelLA, and PoLAR under both the shared preset (Table 1) and matched 9-trial internal-holdout tuning (Table 10).

Revisions in the manuscript. Controlled ablations appear in Section 5.4, Table 5 (p. 10) and Appendix I, Table 11 (p. 39), with protocol details in Appendix L.4 (pp. 45–47) and external manifold benchmarks in Section 5.2, Table 1 (p. 7), Appendix G.3, Table 7 (p. 36), and Appendix G.4, Table 10 (pp. 37–38).

Reviewer_bEcKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents another variant of LoRA, ProLand. ProLAnd is a retraction-free optimizer for LoRA that keeps the basis factor B near the Stiefel manifold with a Landing correction while choosing Muon-style spectral directions in the product space of BA.

Critical strengths

The paper is well written, clean and principled. Enforcing BᵀD_B = 0 off the manifold provably cancels all terms linear in γ in the Gram recursion, giving a clear mechanism for separating optimization from feasibility. The exact low-rank implementation makes it practical for wide application cases.

Critical weaknesses

The largest dense model is 3B, and Moonlight (MoE) is tested only at rank 16. The paper needs results on 7B–70B models (e.g., Llama-3.1-8B/70B, Qwen2.5-7B/14B), higher ranks (32/64), and harder generative tasks (GSM8K/MATH, MT-Bench/AlpacaEval, longer-horizon instruction tuning). Zero-shot multiple-choice likelihood benchmarks with 3-epoch training are a narrow test of an optimizer. Multi-seed runs with significance tests are needed. The most directly related methods are also missing: Stiefel-LoRA (the retraction-based method that motivates the work), StelLA, PoLAR, LoRA-Pro, and DoRA. Baseline learning rates are borrowed from sMuon while ProLand uses its own settings (e.g., different weight decay on DeBERTa), which raises tuning-fairness concerns. The efficiency motivation (avoiding per-step retractions) is never measured: there is no wall-clock, throughput, or memory comparison against retraction or the baselines. Can we compare the central design choice, BᵀD_B = 0 versus the standard skew-symmetric tangent realization? The claim that infeasible exploration reaches better solutions rests on a 2D toy example and a single OpenBookQA retraction ablation, where absolute accuracy (~40%) is far below the main tables. This should be repeated on several tasks and models.

Questions for authors

Follow the Weakness section.

9 comments
AuthorsOpenReview ↗

Author Response to Reviewer bEcK (1/5: [W1, W2])

We sincerely thank Reviewer bEcK for the constructive review and for recognizing that our paper is "well written, clean and principled", that enforcing B⊤DB=0B^\top D_B = 0 off the manifold "provably cancels all terms linear in γ\gamma in the Gram recursion, giving a clear mechanism for separating optimization from feasibility", and that our "exact low-rank implementation makes it practical for wide application cases."

[W1] Model scale, adapter rank, and task coverage

We thank the reviewer for this recommendation. ProLand achieves the highest overall accuracy across all evaluated dense scales and ranks on Llama-3-8B (r∈{16,32,64}r \in \{16, 32, 64\}) and Llama-3-70B (r=32r=32).

All runs use three training seeds (2026, 1337, 3407) on ARC-Challenge and multi-step chain-of-thought GSM8K generation (strict exact match; per-task 3-epoch adapters on 8B and a shared 1-epoch / 490-update multi-task adapter on 70B). Open-ended code generation on HumanEval/HumanEval+ and MBPP/MBPP+ is also evaluated in Appendix K (Tables 14–15).

1. Llama-3-8B (r∈{16,32,64}r \in \{16, 32, 64\}, per-task adapters):

Method (3 Seeds, Mean ± SD)r=16r=16 Avg. (%)r=32r=32 Avg. (%)r=64r=64 Avg. (%)
AdamW (LoRA)59.60 ± 0.8059.43 ± 0.3859.02 ± 0.47
sMuon58.49 ± 0.1458.40 ± 0.5958.46 ± 0.32
Riemannion58.26 ± 0.1559.28 ± 0.5059.01 ± 0.15
Stiefel-LoRA58.10 ± 0.3658.68 ± 0.7358.55 ± 0.37
PoLAR58.52 ± 0.5858.94 ± 1.0360.74 ± 0.50
DoRA59.45 ± 0.5160.37 ± 0.7761.51 ± 0.61
LoRA-Pro62.01±0.23‾\underline{62.01 \pm 0.23}62.03±0.07‾\underline{62.03 \pm 0.07}61.54±0.88‾\underline{61.54 \pm 0.88}
ProLand (Ours)62.60 ± 1.1862.23 ± 0.7063.13 ± 0.10

ProLand leads the two-task average at every rank, ranking first on ARC-C across all ranks and on both tasks at r=64r=64 while avoiding LoRA-Pro's dense-moment memory and wall-clock overhead ([W5]).

2. Llama-3-70B (r=32r=32, shared multi-task adapter):

Method (3 Seeds, Mean ± SD)ARC-C (%)GSM8K (CoT, %)Avg. (%)
AdamW (LoRA)64.85 ± 0.5384.48 ± 0.2374.66 ± 0.24
sMuon65.42±0.57‾\underline{65.42 \pm 0.57}84.51±0.66‾\underline{84.51 \pm 0.66}74.96±0.21‾\underline{74.96 \pm 0.21}
Stiefel-LoRA65.87 ± 1.3783.27 ± 2.3974.57 ± 1.42
ProLand (Ours)65.19 ± 1.3486.40 ± 0.4875.80 ± 0.75

Methods are comparable on ARC-C, whereas ProLand delivers a clear gain on multi-step GSM8K reasoning and the highest overall average while matching AdamW throughput and allocated memory.

We actively conducted extensive large-scale experiments during the rebuttal period; given the limited rebuttal window, additional tasks and longer-horizon evaluations will be completed in the subsequent revision.

Revisions in the manuscript. Full per-task scaling and efficiency tables appear in Section 5.3, Figures 2–3 (p. 9) and Appendix G.3, Tables 7–9 (pp. 36–37).


[W2] Multi-seed evaluation and statistical significance

We upgraded all 3B decoder language suites and all Llama-3-8B/70B benchmarks to three independent training seeds (2026, 1337, 3407), matching the 3-seed protocol on DeBERTa-v3-base (Table 1).

On Qwen2.5-3B and Llama-3.2-3B (r∈{4,8,16}r \in \{4, 8, 16\} across 7 tasks), this 3-seed evaluation now also includes LoRA-Pro across all 6 model–rank settings:

ModelRankProLand (Ours)Strongest Baseline (3-Seed Mean ± SD)Δ\Delta (pp)
Qwen2.5-3Br=4r=467.33 ± 0.3666.77 ± 0.05 (LoRA-Pro)+0.56
Qwen2.5-3Br=8r=867.37 ± 0.6566.67 ± 0.08 (AdamW) / 66.63 ± 0.38 (LoRA-Pro)+0.70
Qwen2.5-3Br=16r=1667.40 ± 0.4567.13 ± 0.12 (LoRA-Pro)+0.27
Llama-3.2-3Br=4r=465.05 ± 0.5664.90 ± 0.18 (Per-factor Muon)+0.15
Llama-3.2-3Br=8r=865.39 ± 0.5865.26 ± 0.24 (Riemannion)+0.13
Llama-3.2-3Br=16r=1665.72 ± 0.5265.59 ± 0.14 (LoRA-Pro)+0.13

Across all 1,008 task-specific adapters in Tables 2–3, ProLand ranks first in every model–rank configuration and wins a clear majority (29–38 of 42 (model, rank, task) cells) against each baseline, with both two-sided paired tt-tests and Wilcoxon signed-rank tests yielding p<10−3p < 10^{-3} against all seven baselines (including LoRA-Pro).

We actively conducted extensive multi-seed experiments during the rebuttal period; given the limited rebuttal window, additional multi-seed data for the remaining setups will be completed in the subsequent revision.

Revisions in the manuscript. Full per-task 3-seed results are reported in Section 5.3, Tables 2–3 (p. 8) and Appendix G.3, Table 7 (p. 36).

AuthorsOpenReview ↗

Author Response to Reviewer bEcK (2/5: [W3])

[W3] Comparisons with directly related baselines

We added 3-seed empirical comparisons against all five methods alongside an expanded structural taxonomy.

Specifically, our comparisons cover all five methods on DeBERTa-v3-base (r=16r=16, Table 1 preset and Table 10 matched 9-trial tuning), Stiefel-LoRA, LoRA-Pro, DoRA, and PoLAR across r∈{16,32,64}r \in \{16, 32, 64\} on Llama-3-8B, LoRA-Pro across all six 3B configurations (r∈{4,8,16}r \in \{4, 8, 16\} on Qwen2.5-3B and Llama-3.2-3B, Tables 2–3), and Stiefel-LoRA (accuracy and profiling) as well as DoRA and LoRA-Pro (efficiency profiling) on Llama-3-70B ([W1], [W4], [W5]):

  1. Stiefel-LoRA: Optimizes one factor on St⁡(m,r)\operatorname{St}(m,r) in factor space with per-step QR/polar retraction, whereas ProLand realizes a product-space polar direction horizontally (B⊤DB=0B^\top D_B = 0) with infeasible Landing. Across DeBERTa-v3-base (preset and matched 9-trial tuning), all three Llama-3-8B ranks, and Llama-3-70B, ProLand consistently outperforms Stiefel-LoRA in accuracy while avoiding per-step retraction latency ([W1], [W4], [W5]).

  2. LoRA-Pro: Matches full-matrix AdamW gradients via a linear solve without Stiefel conditioning and requires dense m×nm \times n moments. ProLand outperforms LoRA-Pro across all 10 evaluated settings from 184M to 8B—on DeBERTa-v3-base (preset and matched 9-trial tuning), all six Qwen2.5-3B and Llama-3.2-3B configurations, and all three Llama-3-8B ranks—while eliminating dense-moment memory and step-time bottlenecks ([W1], [W2], [W5]).

  3. DoRA and PoLAR: Unlike magnitude–direction rescaling (DoRA) or three-factor two-sided Stiefel Landing (PoLAR), ProLand retains the standard two-factor product BABA with a single horizontal Stiefel factor. Empirically, ProLand outperforms both methods on DeBERTa-v3-base (preset and matched 9-trial tuning, Tables 1 and 10) and across all evaluated ranks on Llama-3-8B while avoiding DoRA's per-forward norm-recomputation overhead ([W1], [W5]).

  4. StelLA: Uses a three-factor USV⊤USV^\top parameterization with per-step polar retractions on both UU and VV. ProLand outperforms StelLA on DeBERTa-v3-base under both preset and matched 9-trial tuning (Tables 1 and 10).

    We also conducted numerical experiments for StelLA on Llama-3-8B, where we observed numerical issues and subpar performance; we suspect a bug in our LLM implementation and are actively conducting internal checks, so we withhold StelLA's 8B results for now.

Revisions in the manuscript. Structural comparisons appear in Section 1.1 (p. 2) and Appendix A, Table 6 (p. 16); benchmark results appear in Section 5.2, Table 1 (p. 7), Section 5.3, Tables 2–3 and Figures 2–3 (pp. 8–9), and Appendix G.3–G.4, Tables 7–10 (pp. 36–38).

AuthorsOpenReview ↗

Author Response to Reviewer bEcK (3/5: [W4])

[W4] Hyperparameter-tuning fairness

Every benchmark enforces a strictly matched validation-search budget across ProLand and all baselines, with fixed ρ=0.05\rho = 0.05 throughout the main tables:

  1. 3B decoder suites (Tables 2–3): On Qwen2.5-3B and Llama-3.2-3B (r∈{4,8,16}r \in \{4, 8, 16\}), every baseline (including LoRA-Pro around η0=10−4\eta_0 = 10^{-4}) and ProLand searches the identical 3-candidate grid {0.3,1,3}η0\{0.3, 1, 3\}\eta_0 on seed=2026 PIQA holdouts (using sMuon's reference centers η0\eta_0 and RMS factor scaling 0.2doutdin/r0.2\sqrt{d_{\mathrm{out}}d_{\mathrm{in}}/r}, with η0=6×10−6\eta_0 = 6 \times 10^{-6} and fixed ρ=0.05\rho = 0.05 for ProLand), reusing the selected rate across all tasks and seeds.

  2. Moonlight-16B-A3B (Table 4): No hyperparameter search is run for any method; ProLand directly reuses sMuon's published Moonlight rates (6e-5 language, 7e-6 code; Table 18).

  3. Llama-3-8B and Llama-3-70B (Table 7): All 8 methods on 8B (r∈{16,32,64}r \in \{16, 32, 64\}) and all 4 accuracy-evaluated methods on 70B (r=32r=32) search the identical 8-candidate half-decade grid {0.01,0.03,0.1,0.3,1,3,10,30}η0\{0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30\}\eta_0 on held-out dev seed=4242 with uniform zero weight decay (λ=0\lambda = 0). Every selected configuration lies strictly inside {0.03,0.1,0.3,1,3}η0\{0.03, 0.1, 0.3, 1, 3\}\eta_0 (Table 20).

  4. DeBERTa-v3-base weight-decay parity and matched 9-trial holdout tuning (1,236 runs across all 12 methods, Table 10): Because −ρBt(Bt⊤Bt−Ir)-\rho B_t(B_t^\top B_t - I_r) regulates BtB_t toward unit-norm columns, Euclidean decay on BtB_t conflicts with Stiefel feasibility (λB=0\lambda_B = 0). On DeBERTa-v3-base (r=16r=16), we ran two controlled studies:

    • Preset-rate weight-decay parity (λ∈{0,0.001,0.01}\lambda \in \{0, 0.001, 0.01\}): Varying λ\lambda at the preset rate barely shifts any baseline. Under strict λ=0\lambda = 0 across all 12 methods, ProLand (λ=0\lambda = 0) ranks second only to sMuon within 0.11 pp, and ties sMuon for first at matched λA=0.01\lambda_A = 0.01 (λB=0\lambda_B = 0).

    • Matched 9-candidate internal-holdout tuning: Using an 80/20 split of official train on seed=2026 (zero official dev access during selection, followed by 3-seed evaluation on official dev), ProLand and ProLand (λ=0\lambda = 0) rank #1 and #2 among all 12 methods under both uniform cross-task selection and per-task selection (Table 10).

Revisions in the manuscript. Search budgets, weight-decay parity, and selected configurations are documented in Section 5.1–5.2 (p. 7), Appendix G.4, Table 10 (pp. 37–38), and Appendix L.3, Tables 17–20 (pp. 43–45).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.