acceptodds
Under review as a conference paper at ICLR 2027

Rank-Adaptive Parameter-Efficient Fine-Tuning

21 views

Abstract

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bias by optimizing in a low-dimensional subspace. In this work, we question whether a fixed-rank constraint is the most effective inductive bias for parameter-efficient fine-tuning. We introduce Learnable Rank LoRA (LR-LoRA), a PEFT method in which the adapter rank is learned during the training process. Instead of prescribing a uniform rank for all adapter layers, LR-LoRA allows the optimizer to determine the appropriate rank for each layer. Using this approach, we find substantial layer-wise variation in the learned ranks, with the attention and MLP layers in the transformer models exhibiting systematically different rank preferences. Across a range of language understanding and commonsense reasoning benchmarks, LR-LoRA achieves state-of-the-art performance in most settings and consistently outperforms strong PEFT baselines, demonstrating that a learnable rank provides a more flexible and effective inductive bias than fixed-rank adaptations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient, Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZPoFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper advocates learnable rank LoRA (LR-LoRA), which enables adaptive rank allocation by learning a nonlinear function over the LoRA update. Motivated by Nyquist-Shannon sampling theorem, LR-LoRA advocates adopting sinc as the basis, and learning the bandwidth and amplitude of each basis to reconstruct the nonlinear function. Experiments are conducted on commonsense reasoning and GLUE transfer tasks using multiple LLMs, showing higher stable rank over LoRA and an improved empirical performance relative to LoRA variants.

Critical strengths
  1. The motivation is clear: a uniform fixed rank may be unnecessarily restrictive.
  2. The writing is easy to understand and follow. Paper is well-organized.
  3. Using Nyquist-Shannon sampling theorem to approximate nonlinear activation is interesting, which has documented history in signal processing.
  4. The numerical test covers multiple architectures tasks, and results are promising.
Critical weaknesses
  1. Rank is not directly learned or optimized. Instead, ϕ\phi is learned and stable rank is measured afterwards. The interpretation as "learnable rank" remains indirect and unclear. There lacks an explicit control of the rank.
  2. The Nyquist-Shannon sampling-theory motivation seems stronger than necessary. Alternative learned bases also perform reasonably well, suggesting that the main contribution may be the layer-wise learnable nonlinear transformation rather than sinc specifically.
  3. Experiments establish that layer-specific nonlinear transformations are beneficial, but do not fully isolate whether stable-rank adaptation itself is responsible for the gains.
  4. Relevant nonlinear baselines are not sufficiently covered in the main comparisons. For example, baselines including LoRAN and AuroRA are compared only on Qwen2-0.5B in the Appendix, but not on Phi-3-8B or LLaMA3-8B in the main paper, making it difficult to assess LR-LoRA against its competitors at meaningful scale. In addition, key mechanistic analyses are restricted to Qwen2-0.5B, making it unclear whether the observed rank heterogeneity itself persists at larger scales.
Questions for authors
  1. In what sense is LR-LoRA learning rank, considering that rank is not explicitly parameterized or optimized? How can we control the rank or stable rank?
  2. What's the theoretical benefits (e.g., better approximation error or optimization convergence) of using sinc-based Whittaker–Shannon interpolation (2), which is known to be good for bandlimited signals, over other alternative learned/fixed orthogonal bases, particularly for low-rank adaptation of LLMs?
  3. Can your directly separate the effects of layer-wise nonlinear adaptation and effective-rank adaptation?
  4. Given that transferring the learned rank profile to linear LoRA recovers only part of the gain, how much of the improvement can actually be attributed to rank allocation?
  5. Can you compare LR-LoRA with baselines on at least one 8B model, rather than only 0.5B?
  6. What conditions on ϕ\phi are required for rank(ϕ(BA))>r\text{rank}(\phi(BA)) > r? The current "any differentiable nonlinearity" is inaccurate.
  7. Does the layer-wise stable-rank heterogeneity observed on Qwen2-0.5B also hold for 8B models?
6 comments

We thank the reviewer for seven concrete questions. Several of them (Q1, Q2, Q6) have sharper answers than the submission gave, and we have corrected the paper where the reviewer found it imprecise.

1. Rank is not directly learned or optimized. Instead, ϕ\phi is learned and stable rank is measured afterwards. The interpretation as "learnable rank" remains indirect and unclear. There lacks an explicit control of the rank / In what sense is LR-LoRA learning rank, considering that rank is not explicitly parameterized or optimized? How can we control the rank or stable rank?

LR-LoRA learns a separate elementwise map ϕ\phi for each adapter, using the downstream task loss. The rank of each update follows from the learned parameters. This is what we meant by learning the rank. We agree this is not the best terminology and have thus included the term implicit rank adaptation in the abstract and introduction to show the reader that it is the architecture itself that is changing the rank through the added nonlinearity. We have added a paragraph in Section 3.3 that clearly states this, see lines 259-263.

If an application requires a hard rank cap kk, retaining the top kk singular components of the update enforces that cap. To encourage a target stable rank during training, one could add a penalty for deviation from that target. Table 25 demonstrates how the learned profile can inform explicit rank allocation. Proposition K.1 (Appendix K) makes the control explicit: the learned bandwidth ωℓ\omega_\ell, together with the entry scale of BℓAℓB_\ell A_\ell, bounds how much spectral mass ϕℓ(BℓAℓ)\phi_\ell(B_\ell A_\ell) can place beyond its leading (r+dd)\binom{r+d}{d} directions. It is an upper bound, not a setting of stable rank, so clamping ωℓ≤ωˉ\omega_\ell \le \bar\omega during training would bound each layer's spectral tail (not evaluated as a training variant). The mechanism is visible in the paper: at Qwen2 layer 1 the learned ϕ\phi converges to a near-linear map (Figure 5), which keeps that layer's update close to rank rr, while deeper layers develop increasingly nonlinear maps (Figures 6-7).

2. The Nyquist-Shannon sampling-theory motivation seems stronger than necessary. Alternative learned bases also perform reasonably well, suggesting that the main contribution may be the layer-wise learnable nonlinear transformation rather than sinc specifically. / What's the theoretical benefits (e.g., better approximation error or optimization convergence) of using sinc-based Whittaker–Shannon interpolation (2), which is known to be good for bandlimited signals, over other alternative learned/fixed orthogonal bases, particularly for low-rank adaptation of LLMs?

We agree with the reviewer's reading. The main contribution is a separately learned elementwise transformation per adapter, and the submission says sinc is "a well-motivated choice rather than a unique requirement" (Appendix F.1.2). We do not claim a faster approximation rate or a convergence guarantee for sinc over other bases. The finite expansion with learned per-basis bandwidths does not inherit the Whittaker–Shannon reconstruction guarantee, and the revised Section 3.2 presents sampling theory as motivation only.

Sinc does have two properties that matter here, and they are now stated precisely:

Basis (Table 13)Meets Lemma 3.1's sufficient conditionOne bandwidth bounds the tailAcc.
Sincyes (decays)yes (exponential type πω\pi\omega)61.1
Gaussian RBFyes (decays)partly (width; not finite type)60.4
B-splineyes (compact support)no (piecewise cubic)59.8
Fourier cos⁡(ωx+φ)\cos(\omega x+\varphi)no (periodic)yes59.2
Polynomial (1+ωx)3(1+\omega x)^3nohard cap (r+33)\binom{r+3}{3}58.1

Sinc is the only basis in the comparison that has both properties as stated: it satisfies the sufficient condition of Lemma 3.1, and a single learned parameter, ω\omega, provably bounds (Proposition K.1, Appendix K) how much spectral mass lies beyond the leading directions. This makes the learned bandwidth an interpretable per-layer control, and it is the component with the largest measured effect. Fixing ω\omega costs 1.28 points over three seeds (Table 26), more than switching the basis to Gaussian RBF (0.68) or to a learned elementwise MLP (0.98). We present this as a consistent ordering, not a proof that sinc must win.

3. Experiments establish that layer-specific nonlinear transformations are beneficial, but do not fully isolate whether stable-rank adaptation itself is responsible for the gains / Can your directly separate the effects of layer-wise nonlinear adaptation and effective-rank adaptation? / Given that transferring the learned rank profile to linear LoRA recovers only part of the gain, how much of the improvement can actually be attributed to rank allocation?

The submission has four controlled contrasts, each removing one ingredient:

KeptRemovedGain over matched LoRA
Nonlinearity, shared across layers (Table 20)per-layer freedom0.0 of 3.6
Fixed nonlinearity, uniform stable rank (SineLoRA, Tables 1, 4)learning and heterogeneity0.17 of 3.54 (Qwen2 15k)
Learned allocation, linear adapter (Table 25)nonlinearity0.61 of 3.54
Everything except learned bandwidth (Tables 3, 26)ω\omega learning2.3 of 3.6
Full LR-LoRA--3.6 / 3.54

(Table 20 and Table 3 use the single-seed ablation harness, whose LoRA baseline is 67.8. Tables 1, 25 and 26 use the three-seed main protocol, whose LoRA baseline is 52.34.)

Two conclusions follow. First, a nonlinearity without per-layer freedom gives nothing: tying the transfer function across layers returns exactly to LoRA's accuracy, and a fixed nonlinearity that raises stable rank uniformly (SineLoRA) gives little. Second, the learned allocation carries real information. Placed into a linear adapter, it beats uniform ranks and all three permutations of the same rank multiset at an identical parameter count. Allocation alone recovers about 17% of the gain, so most of the gain comes from the interaction: per-layer learning of the map that produces the allocation. The 0.61 measures the allocation's value inside a linear adapter, where integer ranks cannot reproduce the spectral shape that ϕ\phi creates inside the nominal rank (LR-LoRA's mean stable rank is 9.4-10.7 at r=32r=32); we do not claim it is the allocation's full contribution within LR-LoRA.

Within LR-LoRA's own parameterization, which the reviewer asks about, two existing interventions act on the control of the effective-rank profile while leaving the rest unchanged. (i) Fixing the bandwidth (Tables 3 and 26) keeps the per-layer amplitudes, the factors and the sinc nonlinearity, and removes only the learning of ωℓ\omega_\ell. By the bandwidth bound (Proposition K.1, Appendix K), ωℓ\omega_\ell is the parameter that limits how much spectral mass each layer's update can place beyond its leading directions, so this removes each layer's learned control over its spectral spread. It costs 1.3 points (Table 3) and 1.28 points over three seeds (Table 26), the largest effect among the design ablations of Tables 3 and 26. (ii) Sharing only ω\omega across layers, with amplitudes still per layer, costs 0.6 points (71.4→70.871.4\to70.8, Table 20), so the heterogeneity of the bandwidth across layers, not only its being learned, contributes. On LLaMA3-8B the learned bandwidths are indeed heterogeneous, from ω≈0.5\omega\approx0.5 in early layers to 1.51.5 in deep layers (Appendix J.4). We note that ω\omega also changes the shape of the map, so neither intervention changes effective rank alone. In a jointly learned model no intervention does, and we do not claim a clean split. One more piece of evidence bears on attribution: LR-LoRA's gain over LoRA shrinks monotonically as the nominal rank grows, from +5.3 at r=4r=4 to +2.7 at r=64r=64 (Table 12). The benefit is largest where the fixed-rank constraint binds hardest, as a rank-related mechanism predicts, and Figure 3 shows the corresponding slower spectral decay of ϕ(BA)\phi(BA). What the evidence supports is an ordering: no per-layer freedom gives nothing, per-layer amplitudes with a shared or fixed bandwidth give most of the gain, and learning a per-layer bandwidth, the parameter that bounds spectral spread, adds a further 0.6 to 1.3 points.

4. Relevant nonlinear baselines are not sufficiently covered in the main comparisons. For example, baselines including LoRAN and AuroRA are compared only on Qwen2-0.5B in the Appendix, but not on Phi-3-8B or LLaMA3-8B in the main paper, making it difficult to assess LR-LoRA against its competitors at meaningful scale. In addition, key mechanistic analyses are restricted to Qwen2-0.5B, making it unclear whether the observed rank heterogeneity itself persists at larger scales.

Table 1 compares LR-LoRA on Phi-3-mini (3.8B) and LLaMA3-8B, in both data regimes, against LoRA, VeRA, NoLA, RandLoRA, the fixed-nonlinearity baseline SineLoRA, and the adaptive-rank baselines AdaLoRA, DyLoRA and ElaLoRA. LR-LoRA is best in all four Phi-3-mini and LLaMA3-8B cells. It leads SineLoRA by 2.78/2.29 (Phi-3-mini, 15k/170k) and 2.80-4.09 (LLaMA3-8B), and the strongest adaptive-rank baseline, ElaLoRA, by 1.10-1.44. On LLaMA3-8B instruction tuning, LR-LoRA scores 6.55 on MT-Bench against 6.22 (SineLoRA) and 6.31 (RandLoRA) (Table 5). The reviewer is right that LoRAN, AuroRA and ABBA appear only in the Qwen2-0.5B 15k comparison (Table 24, where LR-LoRA leads by 3.03, 1.43 and 1.03). We acknowledge this as a limitation of the larger-scale comparison. Two facts in the submission indicate how it bears on the conclusion. First, LoRAN's family, a fixed elementwise map on the update, is present at larger scale through SineLoRA. The two are close at 0.5B (LoRAN 52.85, SineLoRA 52.51, LoRA 52.34; Tables 1 and 24), and at 8B the fixed map does not scale up its advantage: on LLaMA3-8B 15k, SineLoRA falls below LoRA (81.90 against 83.09) while LR-LoRA reaches 85.99. Second, at 0.5B AuroRA and ABBA (54.45, 54.85) lie within 0.83 points of ElaLoRA (54.02), and LR-LoRA leads all three. ElaLoRA is in every Phi-3-mini and LLaMA3-8B cell of Table 1, and LR-LoRA leads it there by 1.10 to 1.44. LR-LoRA's gain over LoRA also persists at scale (+3.54 on Qwen2 15k, +2.34 to +3.56 at 3.8B and 8B). We state that AuroRA and ABBA are compared only at 0.5B, and we make no claim about their ranking at larger scale beyond this evidence.

The second part of this point (mechanistic analyses at 8B) is answered under Q7 below.

5. Can you compare LR-LoRA with baselines on at least one 8B model, rather than only 0.5B?

Yes. Table 1 in the original submission already did this. It compares LR-LoRA with eight PEFT baselines on LLaMA3-8B and Phi-3-mini (3.8B) in both data regimes, and LR-LoRA is best in all four cells. Details, and how the 0.5B-only AuroRA/ABBA comparison bears on the larger-scale conclusion, are in our answer to W4 above.

6. What conditions on ϕ\phi are required for rank$(\phi(BA)) > r$? The current "any differentiable nonlinearity" is inaccurate.

We agree "any differentiable nonlinearity" was inaccurate and have removed it everywhere: contributions (lines 82-84), Section 3.3 (lines 195-201), the basis-agnostic paragraph (lines 248-254), and the conclusion (lines 508-510). The precise statements are:

  • Sufficient (Lemma 3.1, lines 221-241). If ϕ\phi is continuous, nonzero somewhere and ϕ(x)→0\phi(x)\to0 as ∣x∣→∞|x|\to\infty, then for any 1≤r<min⁡(m,n)1\le r<\min(m,n) there is a width-rr product with rank⁡(BA)=1\operatorname{rank}(BA)=1 and rank⁡ϕ(BA)=min⁡(m,n)\operatorname{rank}\phi(BA)=\min(m,n).
  • Generic. If in addition ϕ\phi is real-analytic, which holds for every nonzero map in our sinc family, then rank⁡ϕ(BA)=min⁡(m,n)\operatorname{rank}\phi(BA)=\min(m,n) for Lebesgue-almost every (B,A)(B,A). The leading minor is a real-analytic function of the factors that Lemma 3.1 shows is not identically zero, so its zero set has measure zero.
  • Not sufficient. Differentiability alone: a polynomial of degree dd gives rank⁡ϕ(BA)≤(r+dd)\operatorname{rank}\phi(BA)\le\binom{r+d}{d} for every (B,A)(B,A), and a linear map gives at most rr.

All three statements are proved in Appendix K. For a specific trained update, the rank depends on BABA as well as ϕ\phi (for example, a product with only rr distinct columns keeps at most rr after any entrywise map). Because exact rank is generically full, we report stable rank, whose distribution the learned bandwidth controls (Proposition K.1).

Reviewer_gQ3fOpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LR-LoRA, which replaces the standard LoRA update with a learned element-wise transformation of the low-rank update. This allows the effective or stable rank of the learned update to vary across layers despite a fixed nominal LoRA rank.

The method is evaluated across commonsense reasoning, GLUE, vision transfer, and instruction tuning. The paper also provides analyses of layer-wise rank variation, transfer-function sharing, alternative nonlinear bases, efficiency, and transplantation of learned layer-wise rank profiles into standard LoRA.

Critical strengths
  1. The empirical evidence is broad and well aligned with the main hypothesis. In particular, the layer-sharing and rank-profile transplantation experiments provide useful evidence that the gains are related to layer-dependent adaptation rather than only increased expressivity.

  2. The method achieves consistent improvements with limited additional parameter and computational overhead.

Critical weaknesses
  1. The claim that “any differentiable element-wise nonlinearity” removes the rank-r constraint needs further qualification. Differentiability alone does not specify when the transformed parameterization can realize updates outside the original rank-r family, and the current statement does not clearly distinguish this representational possibility from the realized rank of a particular transformed update. The paper should provide precise sufficient conditions under which the rank-r ceiling is relaxed.

  2. The term “learning the rank” is somewhat imprecise. The method learns a nonlinear transformation and characterizes the resulting update using stable rank, rather than directly optimizing matrix rank or an explicit rank-allocation variable.

  3. LR-LoRA introduces several additional design choices and hyperparameters for the learned transformation, including the basis size, grid interval, and initialization. The reported robustness studies are mainly one-factor-at-a-time, so sensitivity to their joint configuration and the associated tuning burden remain less clear.

Questions for authors

Can the authors provide precise sufficient conditions on the element-wise transformation under which the transformed parameterization can realize updates outside the original rank-r family? Clarifying the current “any differentiable element-wise nonlinearity” statement would address my main soundness concern.

4 comments

We thank the reviewer for a precise and constructive review. The main soundness question, sufficient conditions for leaving the rank-rr family, led us to a sharper statement than the one in the submission, and we have corrected the paper accordingly.

1. Can the authors provide precise sufficient conditions on the element-wise transformation under which the transformed parameterization can realize updates outside the original rank-r family?

We thank the reviewer for this interesting question. A simple sufficient condition is that ϕ\phi is continuous, nonzero somewhere, and tends to zero as ∣x∣→∞|x|\to\infty. Every nonzero shifted sinc map in LR-LoRA satisfies this condition.

Choose a≠0a\ne0 with ϕ(a)≠0\phi(a)\ne0. For q>1q>1, take the (r+1)×(r+1)(r+1)\times(r+1) matrix with entries Zij=aqi−jZ_{ij}=a q^{i-j}. This matrix has rank one because Zij=(aqi)q−jZ_{ij}=(a q^i)q^{-j}. After applying ϕ\phi entrywise, the diagonal equals ϕ(a)\phi(a). As q→∞q\to\infty, entries below the diagonal tend to zero and entries above it tend to ϕ(0)\phi(0). The limit is triangular with nonzero diagonal, so the transformed matrix is invertible for sufficiently large finite qq.

Embedding this rank-one block in BABA gives an update with rank at least r+1r+1. This establishes that the parameterization can exceed the rank-rr ceiling. The rank of a particular trained update depends on the learned parameters.

We have now included this as Lemma 3.1 in the main paper, see lines 221-241.

2. Clarifying the current "any differentiable element-wise nonlinearity" statement would address my main soundness concern.

Thank you for pointing this out. We agree that the original statement was too broad.

Corrections in the revision: the overbroad statement is removed from the contribution list (lines 82-84), from the paragraph motivating the nonlinearity (now "Why a nonlinearity can lift the rank-rr ceiling", lines 195-201, which states that a linear ϕ\phi never exceeds rr), from the basis-agnostic paragraph (lines 248-254), and from the conclusion (lines 508-510). The last three now refer to the conditions of Lemma 3.1. Appendix K gives the full statements with proofs: genericity of full rank for our sinc maps, the polynomial rank cap (r+dd)\binom{r+d}{d}, and a bound on the spectral tail in terms of the learned bandwidth (Proposition K.1): for ϕ(x)=∑iαisinc(ωi(x−xi))\phi(x)=\sum_i\alpha_i \mathrm{sinc}(\omega_i(x-x_i)), with M=sup⁡x∣ϕ(x)∣M=\sup_x|\phi(x)|, RR the largest absolute entry of BABA, and k=(r+dd)k=\binom{r+d}{d}, we have σk+1(ϕ(BA))≤mn⋅M⋅(πωmax⁡R)d+1/(d+1)!\sigma_{k+1}(\phi(BA))\le\sqrt{mn}\cdot M\cdot(\pi\omega_{\max}R)^{d+1}/(d+1)! for every d≥0d\ge0.

3. The term "learning the rank" is somewhat imprecise. The method learns a nonlinear transformation and characterizes the resulting update using stable rank, rather than directly optimizing matrix rank or an explicit rank-allocation variable.

We agree. LR-LoRA does not optimize matrix rank or an explicit integer allocation. Each adapter learns its own transfer function ϕℓ\phi_\ell jointly with (Aℓ,Bℓ)(A_\ell,B_\ell) from the task loss, and the spectrum of ϕℓ(BℓAℓ)\phi_\ell(B_\ell A_\ell) follows. The revision calls this implicit rank adaptation (abstract, introduction, and a new paragraph in Section 3.3, lines 259-263). In the sense of the bandwidth bound (Proposition K.1, Appendix K), the per-layer bandwidth ωℓ\omega_\ell acts as a continuous, differentiable, learned limit on the spectral tail of each update. It is not an explicit allocation variable. Empirically, what is learned differs across layers: the per-layer standard deviation of stable rank is 3.8/4.2 (attention/MLP) for LR-LoRA against 1.1-1.6 for LoRA, SineLoRA and RandLoRA (Table 4).

4. LR-LoRA introduces several additional design choices and hyperparameters for the learned transformation, including the basis size, grid interval, and initialization. The reported robustness studies are mainly one-factor-at-a-time, so sensitivity to their joint configuration and the associated tuning burden remain less clear.

On tuning burden: none of the reported results used per-task or per-model tuning of the transfer function. One configuration (N=50N=50 grid points on [−3,3][-3,3], αi=0\alpha_i=0, ω0=1\omega_0=1) is used unchanged across all architectures and tasks, including the GLUE and vision runs, with each baseline's own optimization settings. LR-LoRA adds NN, II and two initialization constants over LoRA; the softplus is a fixed positivity map, not a tuned scalar (Appendix F.4.2).

On joint sensitivity, the reviewer is correct that the robustness studies vary one factor at a time, and the revision says so (Appendix F, lines 1166-1169). Within that limit, the existing sweeps cover the main choices around the default, and every point stays above LoRA (67.8 on this harness), including the extreme bandwidth initializations ω0=0.1\omega_0=0.1 and 55 (68.4 and 69.2, Table 17):

ChoiceValuesAccuracyWorst drop
Basis size NN (Table 3)32/50/6470.8/71.4/71.20.6
Bandwidth init ω0\omega_0 (Table 17)0.5/1/270.8/71.4/71.10.6
Amplitude init (Table 3)zero / random71.4/70.90.5
Grid points ±10%, α\alpha / ω\omega init ±50% (Table 21)worst 71.2/70.9/69.81.6

Across the near-default values of NN, ω0\omega_0 and amplitude initialization above, the largest drop is 0.6 points against a 3.6-point gain. Random ±50% perturbations of the bandwidth initialization cost more (worst case 69.8), still 2.0 points above LoRA. The grid interval II is not swept on its own; perturbing the grid points by ±10% changes accuracy by at most 0.2. Two observations bear on interaction, though neither rules it out. First, ω\omega is learned, and training moves it toward the default from both sides (0.5→0.70.5\to0.7, 2.0→1.82.0\to1.8, Table 17), which reduces sensitivity to its initial value. Second, the factors that do cross in the paper show no interaction that erases the gain: Table 12 crosses the default transfer function with base rank r∈{4,…,64}r\in\lbrace 4,\dots,64 \rbrace (gain +5.3+5.3 to +2.7+2.7), Table 18 crosses amplitude initialization with model scale (0.5B and 8B), and Table 21 crosses the default with learning-rate and batch-size changes (×0.5 to ×2, worst case 70.6). Even if the worst drops on the NN, ω0\omega_0 and amplitude axes simply added up, the result would be about 69.7, still above LoRA; this is an extrapolation, not a measurement, and we state it as such. Finally, the default was fixed once and never retuned for any backbone or task, which is the practical measure of tuning burden. Configurations better than the default appear in the paper and were not adopted: a Chebyshev or learned grid (71.8 / 72.1 vs. 71.4, Table 14), a depth-dependent ω\omega initialization (71.6, Table 23), and a small nonzero α\alpha initialization (Table 18). The reported results therefore use an untuned default rather than a selected one. The caption of Table 15 now states that its NN sweep belongs to the N=32N=32 grid sub-study and is not directly comparable with Table 3.

Thank you for the detailed response. The new lemma addresses my main theoretical concern, and the clarifications are helpful.

I still have some concerns about joint hyperparameter sensitivity, but I appreciate the improvements. I will maintain my current rating.

Thank you for your follow-up, and for confirming that Lemma 3.1 resolves the main theoretical concern.

On joint sensitivity, we agree that our sweeps change one factor at a time. The revision states this as a limitation (Appendix F, lines 1166–1169). What the current results show is that every point in these sweeps stays above LoRA, and that the default setting was never retuned for any model or task.

We are grateful for the careful and constructive review.

Reviewer_usXXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary
  • Proposes Learnable Rank LoRA, an extension of LoRA where the method learn what rank to use for each layer instead of fixing the rank,
  • LR-LoRA formulates the residual as ΔW=ϕ(BA)\Delta W=\phi(BA), where ϕ\phi is a non-linearity.
  • ϕ\phi is constructed using shifted sinc functions, such that ϕ(x)=∑i=1Nαi sinc⁡(ωi(x−xi))\phi(x) = \sum_{i=1}^{N} \alpha_i \, \operatorname{sinc}\big(\omega_i (x - x_i)\big), with (ω1,α1),..,(ωN,αN)(\omega_1,\alpha_1),..,(\omega_N,\alpha_N) being learnable, and {x1,x2,..,xN}\{x_1,x_2,..,x_N\} being uniformly spaced grid points.
  • The effetive rank of ΔW\Delta W is evlauated using the stable rank
Critical strengths

The analysis of the stable rank across layers is intersting. This might help understand layers from an interpretability perspective. If studied across model sizes and families, it may help identify common patterns across models, or places where models differ.

However, since this is not studied in depth, this has limited impact on my assessment.

Critical weaknesses
  • The major weakness of the method is that it requires materializing the entire ΔW\Delta W matrix, which is the same size as the weight of the pretrained layer. While LoRA can decompose this as smaller matrix multiplications, LR-LoRA cannot because of the added non-linearity.
  • This will make the method slower to train as compared to LoRA (comparable to full fine-tuning), and will consume more GPU memory. However, the paper does not report the wall time or GPU memory consumption.
  • LoRA is prefered precisely because it allows efficient training, despite sometimes performing lower than full fine-tuning [1]. Hence, performing better than LoRA is not sufficient.
  • The use of sinc function isn't justified. While Section F.1.2 compares with Fourier, spline, RBF and polynimial bases, standard non-linearities line tanh are not evaluated.
Questions for authors
  • How are the hyperparameters of all the baselines tuned?
5 comments

We thank the reviewer for the careful reading. The two concerns we take most seriously are training cost and the justification for sinc. We answer both with numbers that are already in the submission and point to the exact tables.

1. The major weakness of the method is that it requires materializing the entire ΔW\Delta W matrix, which is the same size as the weight of the pretrained layer. While LoRA can decompose this as smaller matrix multiplications, LR-LoRA cannot because of the added non-linearity / This will make the method slower to train as compared to LoRA (comparable to full fine-tuning), and will consume more GPU memory. However, the paper does not report the wall time or GPU memory consumption.

We thank the reviewer for their comment. The trainable state of LR-LoRA is as small as LoRA's: AA is r×nr \times n and BB is m×rm \times r, so only the r(m+n)r(m+n) factor parameters, plus 2N2N transfer scalars per matrix, carry gradients and optimizer state, against mnmn entries for the dense pretrained weight. The update ϕ(BA)\phi(BA) itself is a dense m×nm \times n matrix, as discussed next.

Furthermore, we have given a complete throughput analysis of our approach in section H.1 in the appendix. In that section we analyze the training efficiency of our method precisely because we have to materialize the full ΔW\Delta W matrix. The reviewer is right that ϕ(BA)\phi(BA) is an m×nm \times n matrix of the same size as WW, and that the nonlinearity rules out LoRA's factorized path of first computing AxAx for an input xx and then computing BAxBAx, whereas LR-LoRA has to compute BABA, apply the nonlinearity ϕ\phi, and then compute ϕ(BA)x\phi(BA)x. The question is what that costs in practice.

Wall time and memory are reported. They appear in the main text (Section 4.2, "Training efficiency"), in Table 27 and Table 29 (Appendix H.1), and in the cost derivation in Appendix C. Table 27 also contains a full fine-tuning row measured under the same protocol (sequence length 512, gradient checkpointing). That row answers the comparison with full fine-tuning directly:

MethodQwen2-0.5B Mem (GB)Qwen2-0.5B tok/sPhi-3-mini (3.8B) Mem (GB)Phi-3-mini (3.8B) tok/s
LoRA (r=16r=16)2.1318016.81140
LR-LoRA (r=16r=16)2.1316516.91132
LoRA (r=32r=32)2.4298018.21050
Full fine-tuning8.9182042.1480

LR-LoRA uses 4.2× less memory than full fine-tuning on Qwen2-0.5B and 2.5× less on Phi-3-mini, and trains 1.7× and 2.4× faster. Its memory and throughput are within 1% of LoRA at the same rank; on LLaMA3-8B the overhead is +1.1% memory and −1.0% throughput (Table 29).

The cost stays low because of how the dense matrix is used (Appendix C). Per step we form Weff=W+ϕ(BA)W_{\mathrm{eff}} = W + \phi(BA) once and run one backbone-shaped matmul WeffXW_{\mathrm{eff}}X. Forming ϕ(BA)\phi(BA) costs O(mn(r+N))O(mn(r+N)) per step, independent of batch size bb and sequence length ss, against O(mn⋅bs)O(mn \cdot bs) for the layer's own matmul. The matrix is transient: it is freed after the merge, so only one such matrix is alive at a time. In the backward pass the gradient through ϕ\phi needs the dense product (∇YL)X⊤(\nabla_Y L)X^\top once per adapted matrix (Appendix C states this explicitly). What LR-LoRA does not carry is the source of full fine-tuning's memory: gradients, AdamW moments and master weights for all mnmn entries. Optimizer state is kept only for the r(m+n)r(m+n) factor parameters and 2N2N transfer scalars per matrix (memory breakdown under Table 27). At inference ϕ(BA)\phi(BA) is merged into WW, so deployment cost equals the backbone's, as with LoRA. The stored adapter is 0.10% larger than LoRA's (12,000 transfer scalars on 11,796,480 factor parameters for Qwen2-0.5B at r=32r=32, with adapters on the q, k, v, up and down projections; Appendix G.4).

2. LoRA is prefered precisely because it allows efficient training, despite sometimes performing lower than full fine-tuning [1]. Hence, performing better than LoRA is not sufficient.

We agree that accuracy alone is not the right criterion, so here is the accuracy-cost trade-off. LR-LoRA at rank 16 costs less than LoRA at rank 32 (Table 27: 2.1 vs 2.4 GB and 3165 vs 2980 tok/s on Qwen2-0.5B; 16.9 vs 18.2 GB and 1132 vs 1050 tok/s on Phi-3-mini). It is also more accurate in all six backbone and data settings (Table 6):

Qwen2-0.5B 15kQwen2-0.5B 170kPhi-3-mini 15kPhi-3-mini 170kLLaMA3-8B 15kLLaMA3-8B 170k
LoRA r=32r=3252.3457.3480.3484.9683.0985.24
LoRA r=64r=64----81.3784.2281.2085.33
LR-LoRA r=16r=1655.8060.9983.9886.9185.1587.24

On Phi-3-mini and LLaMA3-8B, LR-LoRA at rank 16 also beats LoRA at rank 64. The same pattern holds across ranks on Qwen2-0.5B (Table 12): LR-LoRA at r=16r=16 (61.1) exceeds LoRA at r=64r=64 (60.1). Among LoRA variants, LR-LoRA is cheaper to train than DoRA (2.2 GB, 3090 tok/s) and RandLoRA (2.3 GB, 3050 tok/s) in Table 27. It also leads AdaLoRA and ElaLoRA in all six cells of Table 1, even though those methods start at rank 48, which is 1.5× LoRA's factor storage (Appendix G.4). At equal rank r=16r=16, where memory and throughput match LoRA's (Table 27), LR-LoRA gains +2.18 to +3.62 points in all six cells and improves on every one of the eight tasks in every cell (48/48, Table 6). The same holds at r=32r=32 (48/48, Appendix B.1), where the gains over LoRA in Table 1 are significant after Holm-Bonferroni correction in all six settings (Appendix B.2). So the claim is about the frontier: at equal or lower training cost and identical inference cost, LR-LoRA is more accurate.

3. The use of sinc function isn't justified. While Section F.1.2 compares with Fourier, spline, RBF and polynimial bases, standard non-linearities line tanh are not evaluated.

A fixed elementwise nonlinearity such as tanh⁡\tanh can be applied to BABA, and it is a reasonable baseline. The submission evaluates this class in three ways:

  • Fixed standard nonlinearity on BABA. SineLoRA applies sin⁡\sin elementwise to the adapter update. It appears in every cell of Table 1, in Table 2 and in Table 5. LR-LoRA leads it by 2.29 to 4.09 points on Phi-3-mini and LLaMA3-8B. LoRAN applies a fixed activation to the low-rank update, shared across layers, and trails LR-LoRA by 3.03 points (Table 24).
  • One learned nonlinearity shared by all layers. In Table 20 we keep the sinc nonlinearity but tie its parameters across layers. Accuracy falls from 71.4 to 67.8, which is exactly the matched LoRA baseline (Table 3). A nonlinearity on its own adds nothing; the gain comes from each layer learning its own map.
  • A learned tanh transfer function. Table 26 replaces the sinc basis with an elementwise 1→8→11 \to 8 \to 1 MLP with tanh⁡\tanh hidden units, learned per layer, i.e. ϕℓ(x)=∑k=18cℓktanh⁡(aℓkx+bℓk)+dℓ\phi_\ell(x)=\sum_{k=1}^{8}c_{\ell k}\tanh(a_{\ell k}x+b_{\ell k})+d_\ell. It reaches 54.90, against 55.88 for sinc and 52.34 for LoRA. A learned tanh map therefore recovers most of the gain over LoRA, and sinc adds a further 0.98 points.

Across the nonlinearities we test, the pattern is consistent. A single map applied at every layer, whether fixed (sin⁡\sin in SineLoRA, +0.17+0.17 over LoRA on Qwen2 15k, Table 1) or learned and tied (Table 20, +0.0+0.0), gives little or nothing. Learned per layer, tanh⁡\tanh units recover 2.56 of the 3.54 points (Table 26) and sinc recovers all of them. Per-layer learning of the map accounts for most of the gain; the remaining 0.98 points separate two learned maps that differ in basis and in parameter count, so we do not attribute them to the basis alone.

Why sinc in particular: each shifted sinc is continuous and decays to zero, which is the sufficient condition of our new Lemma 3.1 guaranteeing that the width-rr parameterization can produce full-rank updates. tanh⁡\tanh does not decay, so it falls outside that sufficient condition (the condition is sufficient, not necessary). Sinc also has an explicit bandwidth ω\omega, and ω\omega provably bounds how much spectral mass ϕ(BA)\phi(BA) can place beyond the leading directions (Proposition K.1, Appendix K). Empirically, learning ω\omega is the largest single design factor: fixing it costs 1.28 points over three seeds (Table 26) and 1.3 points in Table 3. We do not claim sinc is the only workable basis. Gaussian RBFs come within 0.68 points (Table 26).

4. How are the hyperparameters of all the baselines tuned?

For the baseline hyperparameters, all decoder-only experiments reuse the released training and evaluation harness of RandLoRA (Albert et al., 2025a) and change only the adapter parameterization (Section 4). The NoLA, VeRA, LoRA and RandLoRA results are the published results of Albert et al. (2025a), produced with this same harness; all other methods were run by us in it (now stated in Section 4.1 and the table captions). Every method shares the backbone, adapted module set (q, k, v, up and down), data pipeline, schedule (3 epochs, AdamW, learning rate 10−410^{-4}, adapter dropout 0.05), batch size and evaluation scripts. RandLoRA uses the basis ranks and number of bases from Table 9 of Albert et al. The adaptive-rank baselines (AdaLoRA, DyLoRA, ElaLoRA) are matched to an average target rank of 32 (Appendix G.4). The Table 24 baselines were run from their released code at the same module set and schedule, or reimplemented from the paper where no code exists (marked ∗^*). On GLUE, VeRA, LoRA and RandLoRA are taken from Albert et al. (Table 4); SineLoRA and LR-LoRA use the learning-rate grid {5×10−5,10−4,2×10−4}\lbrace 5\times10^{-5}, 10^{-4}, 2\times10^{-4} \rbrace and report the best value. Vision runs use RandLoRA's learning rate and per-dataset epoch counts for all methods (Appendix D). LR-LoRA received no extra tuning: one transfer-function configuration (N=50N=50, I=3I=3, αi=0\alpha_i=0, ω0=1\omega_0=1) is used unchanged for every model and task. Table 10 lists these settings. The transfer-function settings were never tuned per task or per model.

5. On the stable-rank analysis across model sizes (Critical Strengths).

We thank the reviewer for noting this. The direct stable-rank statistic (Table 4, Figure 2) is measured on Qwen2-0.5B. At 8B the paper gives two further pieces of evidence. First, the learned transfer functions on LLaMA3-8B differ across depth (layers 1, 16, 32) and between attention and MLP (Appendix J.3, Figures 11-16). Second, the learned bandwidths on LLaMA3-8B are stratified by depth, ω≈0.5\omega\approx0.5 in early layers, 0.80.8 in middle layers and 1.51.5 in deep layers, while Qwen2-0.5B stays within [0.8,1.2][0.8,1.2] (Appendix J.4). The bandwidth bound (Proposition K.1, Appendix K) ties ωℓ\omega_\ell, together with the entry scale of BℓAℓB_\ell A_\ell, to how much spectral mass a layer's update can place beyond its leading directions, so smaller ω\omega in early layers and larger ω\omega in deep layers are consistent with a depth-dependent spectral profile. The direction agrees with the measured 0.5B profile, where early layers have lower stable rank and later layers higher (Figure 2). The bound also depends on sup⁡∣ϕ∣\sup|\phi| and the entry scale, so this is supporting evidence, not a measurement of stable rank at 8B. The accuracy gain over LoRA also holds at larger scale: +2.34+2.34 to +3.56+3.56 on Phi-3-mini and LLaMA3-8B (Table 1), against +3.54+3.54/+3.71+3.71 on Qwen2-0.5B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.