acceptodds
Under review as a conference paper at ICLR 2027

From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA

Abstract

Parameter-efficient fine-tuning with LoRA is governed by rank, which controls parameter budget and locally accessible task updates. Existing dynamic-rank methods mainly focus on capacity allocation, leaving unresolved whether the current adaptation space is sufficient for the task. We characterize the first-order information boundary of standard factor gradients, showing that their unobservable subspace coincides with the normal space of the low-rank manifold. Thus, factor gradients alone cannot distinguish under-optimization within the current space from task-relevant directions beyond its first-order reach. This motivates a geometric view of rank evolution, where tangent information optimizes existing degrees of freedom and the normal spectrum quantifies the value of new ones. Under a local quadratic model, we show that leading normal singular directions yield the locally optimal rank expansion, with spectral energy determining its gain. Building on this characterization, we propose GEAR, which turns the tangent–normal decomposition into rank evolution through tangent-space optimization, matrix-free normal-spectrum estimation for rank expansion, and task-induced geometric initialization. Across language and vision benchmarks, GEAR achieves the best average performance in all evaluated settings with substantially fewer trainable parameters. On LLaMA3-8B, it raises the average commonsense score from 86.61 to 88.45 while reducing trainable parameters from 42.0M to 14.4M.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Dynamic Rank, Riemannian Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CXuYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GEAR, which turns the tangent-normal decomposition into rank evolution through tangent-space optimization, matrix-free normal-spectrum estimation for rank expansion, and task-induced geometric initialization. The method can effectively address the current issue, i.e., existing dynamic-rank methods mainly focus on capacity allocation, leaving unresolved whether the current adaptation space is sufficient for the task. The author conducted sufficient experiments and comparisons across language and vision benchmarks. The results denote that the proposed GEAR achieves the best average performance in all evaluated settings with substantially fewer trainable parameters, which verifies the effectiveness of the proposed GEAR.

Critical strengths

S1: This paper addresses a fundamental issue: existing dynamic-rank methods primarily focus on capacity allocation, leaving unresolved whether the current adaptation space is sufficient for the task. I think the solved problem is meaningful.

S2: The author proposes an effective method, i.e., GEAR. The method can effectively solve the above issues.

S3: This paper provides sufficient experiments and comparisons and finally verifies the effectiveness of the proposed method.

Critical weaknesses

W1: In Theorem 2, the author proposed the Local Optimality of Rank Expansion. However, I think since the loss functions of actual neural networks have complex nonlinear structures, the authors need to further explain the applicability of this theoretical conclusion in practical optimization processes.

W2: The proposed GEAR uses randomized probing to estimate the dominant normal-gradient directions. Although Theorem 3 provides an error bound, the authors have not yet fully demonstrated whether the actual estimation error is small enough to ensure the reliability of rank-extended decisions.

W3: I believe the authors should include some experiments on hyperparameter sensitivity analysis. I think the performance of the proposed GEAR method is influenced by several parameters, including the Probe Width hh and the prescribed acceptance threshold ρ\rho.

W4: Some of the efficiency analyses are insufficient. In Table 4 of the appendix, GEAR’s training time on DTD is longer than that of LoRA and FlexLoRA. I suggest that the authors provide additional results to support the effectiveness of their method. The authors could present training times and GPU memory usage on large models, e.g., LLaMA3-8B.

Questions for authors

In this paper, the authors proposed an effective method, i.e., GEAR. However, I still have some concerns. I think these concerns should be explained and clarified. Besides, I think the authors should add some extra experiments. Please refer to W1-W4.

Reviewer_kWH3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GEAR asks when a LoRA adapter should increase its rank instead of continuing to optimize its current degrees of freedom. It characterizes gradient information unavailable from standard factor gradients, estimates the missing normal component through additional probes, and proposes rank expansions from its leading singular directions. Candidates are allocated by estimated gain per added parameter and accepted through loss evaluation on separate training microbatches. The procedure also includes task-induced geometric initialization and fixed-rank manifold optimization. Experiments cover commonsense reasoning, GLUE, and VTAB.

Critical strengths

The procedure addresses a useful decision coherently. Additional gradient probes measure directions absent from the current first-order optimization space. Unlike expansion based only on current factor gradients, these probes can propose descent when those gradients vanish but the normal gradient does not. Their estimated spectral energy motivates a cost-aware allocation, while candidate loss evaluation checks finite-step benefit. Table 10 supports the usefulness of the growth procedure: adding it improves 86.23 to 88.45 with initialization and manifold optimization, and 85.65 to 87.32 with manifold optimization alone.

The reported performance and parameter gains are substantial. On LLaMA3-8B, GEAR reaches 88.45 with 14.4M parameters, versus 85.82/41.9M for Riemannion and 85.63/41.3M for GeoLoRA. On Qwen2.5-7B, the corresponding gains exceed two points with about one third as many parameters. GLUE and VTAB also show higher average performance with fewer reported parameters. This breadth supports practical interest in the implementation.

Critical weaknesses

The central methodological principle is already supplied by the closest antecedents. Uschmajew and Vandereycken [1] derive tangent-cone descent by adding a truncated normal residual to tangent optimization, discuss randomized computation, and alternate rank growth with fixed-rank descent. Riemannion [2] supplies arbitrary matrix-free gradient products and task-gradient initialization for LoRA. GeoLoRA [3] supplies gradient-driven expansion and spectral truncation with a global budget option. GEAR's independent normal measurement differs from GeoLoRA's current-factor contractions, but implementing it uses the gradient-product machinery of [2] to compute the established correction in [1]. The observability theorem expresses the kernel of the factorization differential's adjoint; the expansion theorem applies the established tangent-cone projection to a quadratic model. Neither establishes a new rank-selection principle. What remains is periodic probing, spectral gain divided by parameter cost, candidate-prefix and step-size evaluation, and a normalized initialization recipe. These are sensible adaptations, but the paper does not identify a non-obvious obstacle they resolve beyond making known geometric rank growth executable in this setting.

The experiments do not establish a substantive advance in the remaining decision rules. Table 10 switches normal probing, additional capacity, and candidate evaluation together, without reporting each variant's final parameter count. It supports the complete growth procedure, but does not show what is learned specifically from its allocation and acceptance rules beyond familiar rank-growth controls. Section 4 and Appendix D also leave the Riemannion initialization/update and GeoLoRA truncation/budget choices unclear. The favorable benchmark results therefore establish useful configurations, while providing limited evidence that this practical transfer advances decisions about when and where to add rank beyond the known normal-correction principle.

References

[1] André Uschmajew and Bart Vandereycken, "Greedy rank updates combined with Riemannian descent methods for low-rank optimization," International Conference on Sampling Theory and Applications (SampTA), 2015. Paper

[2] Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko, Denis Bobkov, Vera Soboleva, Aibek Alanov, and Maxim Rakhuba, "LoRA Meets Riemannion: Muon Optimizer for Parametrization-Independent Low-Rank Adapters," ICLR, 2026. Paper

[3] Steffen Schotthöfer, Emanuele Zangrando, Gianluca Ceruti, Francesco Tudisco, and Jonas Kusch, "GeoLoRA: Geometric Integration for Parameter Efficient Fine-Tuning," ICLR, 2025. Paper

Questions for authors
  1. Clarification: In Table 10, what were the final trainable-parameter counts for TGI plus manifold optimization with and without rank expansion (88.45 versus 86.23)? Given the candidate-prefix and birth-scale search in Appendix B.2, what evidence already available distinguishes the benefit of normal-spectrum selection from the accompanying capacity and candidate evaluation? This would clarify how specifically the ablation supports the proposed missing-information explanation.

  2. Clarification: For the LLaMA3-8B comparison in Table 1, which Riemannion update and initialization, and which GeoLoRA truncation and budget policy, produced the reported results and parameter counts? Please explain how these settings were selected relative to GEAR's rank budget. This would clarify which geometric optimization and rank-adaptation capabilities the quality–parameter comparison evaluates, and how strongly it supports the claimed practical advance.

Reviewer_aTM3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a dynamic-rank LoRA method called GEAR, motivated by a first-order observability barrier in standard factor gradients. Starting with a task-induced geometric initialization, GEAR uses tangent gradients to optimize adapters on the current manifold and normal gradients to identify and expand into missing task-relevant directions. Experiments on commonsense reasoning, GLUE, and VTAB-1K show that GEAR consistently improves performance while requiring significantly fewer trainable parameters.

Critical strengths
  1. Dynamic rank adaptation is an interesting problem that may attract attention from the PEFT community, and the paper is generally well written, with clear motivation and a coherent presentation.

  2. Practical usage is considered. Randomized gradient probing and compact SVD operations avoid using full gradients in the language experiments.

  3. Experiments on commonsense reasoning, GLUE, and VTAB-1K show that GEAR improves performance with fewer trainable parameters.

Critical weaknesses
  1. Using normal-gradient to determine rank expansion has precedents in Riemannian optimization, particularly in [1] and [2]. Theorem 2 also appears to follow directly from established tangent-cone projection results.

  2. The paper does not show how layer-wise ranks evolve and how frequently expansions are accepted.

  3. The paper reports three-seed means without variability; GEAR uses task-specific hyperparameters, but the baseline tuning procedures remain underspecified. It is unclear whether all methods receive comparable tuning budgets; Runtime and memory measurements are limited to one vision task.

References:

[1] M. Tan et.al. Riemannian Pursuit for Big Matrix Recovery. ICML 2014.

[2] B. Gao and P. A. Absil. A Riemannian rank-adaptive method for low-rank matrix completion. Arxiv:2103.14768

Questions for authors
  1. I would be interested to see how GEAR compares with two simpler alternatives: fixed-rank configurations matched to its final parameter count, and random-direction expansion using the same parameter budget and validation procedure. Could these comparisons help clarify when GEAR’s rank-evolution mechanism is particularly beneficial?

  2. To better connect the theoretical analysis with the practical algorithm, do candidates with larger estimated normal-spectrum gains actually yield larger loss reductions? A plot of predicted versus observed gains would be informative.

  3. Since expansion decisions rely on gradient probes, how stable are the learned rank allocations when using different calibration samples or random probing seeds? Do similar layer-wise rank patterns emerge across runs?

  4. Since the tangent space evolves during fixed-rank training, directions that are currently normal may become accessible later without increasing rank. How does GEAR distinguish a need for additional capacity from a temporary mismatch of the current subspace? Could the authors discuss when rank expansion is preferable to further fixed-rank optimization?

Comments on the paper

1 comment

Thank you for the rebuttal and the additional comparisons. The results show that GEAR is practical in the reported 8B setting and that its rank-allocation decisions matter. I am keeping my rating of 2 because normal-residual rank growth is established, and the added value of GEAR’s allocation rule over GeoLoRA remains unresolved. At comparable final adapter sizes, does GEAR retain its accuracy advantage when GeoLoRA uses its published global budget allocation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.