- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper advocates learnable rank LoRA (LR-LoRA), which enables adaptive rank allocation by learning a nonlinear function over the LoRA update. Motivated by Nyquist-Shannon sampling theorem, LR-LoRA advocates adopting sinc as the basis, and learning the bandwidth and amplitude of each basis to reconstruct the nonlinear function. Experiments are conducted on commonsense reasoning and GLUE transfer tasks using multiple LLMs, showing higher stable rank over LoRA and an improved empirical performance relative to LoRA variants.
Critical strengths
- The motivation is clear: a uniform fixed rank may be unnecessarily restrictive.
- The writing is easy to understand and follow. Paper is well-organized.
- Using Nyquist-Shannon sampling theorem to approximate nonlinear activation is interesting, which has documented history in signal processing.
- The numerical test covers multiple architectures tasks, and results are promising.
Critical weaknesses
- Rank is not directly learned or optimized. Instead, is learned and stable rank is measured afterwards. The interpretation as "learnable rank" remains indirect and unclear. There lacks an explicit control of the rank.
- The Nyquist-Shannon sampling-theory motivation seems stronger than necessary. Alternative learned bases also perform reasonably well, suggesting that the main contribution may be the layer-wise learnable nonlinear transformation rather than sinc specifically.
- Experiments establish that layer-specific nonlinear transformations are beneficial, but do not fully isolate whether stable-rank adaptation itself is responsible for the gains.
- Relevant nonlinear baselines are not sufficiently covered in the main comparisons. For example, baselines including LoRAN and AuroRA are compared only on Qwen2-0.5B in the Appendix, but not on Phi-3-8B or LLaMA3-8B in the main paper, making it difficult to assess LR-LoRA against its competitors at meaningful scale. In addition, key mechanistic analyses are restricted to Qwen2-0.5B, making it unclear whether the observed rank heterogeneity itself persists at larger scales.
Questions for authors
- In what sense is LR-LoRA learning rank, considering that rank is not explicitly parameterized or optimized? How can we control the rank or stable rank?
- What's the theoretical benefits (e.g., better approximation error or optimization convergence) of using sinc-based Whittaker–Shannon interpolation (2), which is known to be good for bandlimited signals, over other alternative learned/fixed orthogonal bases, particularly for low-rank adaptation of LLMs?
- Can your directly separate the effects of layer-wise nonlinear adaptation and effective-rank adaptation?
- Given that transferring the learned rank profile to linear LoRA recovers only part of the gain, how much of the improvement can actually be attributed to rank allocation?
- Can you compare LR-LoRA with baselines on at least one 8B model, rather than only 0.5B?
- What conditions on are required for ? The current "any differentiable nonlinearity" is inaccurate.
- Does the layer-wise stable-rank heterogeneity observed on Qwen2-0.5B also hold for 8B models?
6 comments
3. Experiments establish that layer-specific nonlinear transformations are beneficial, but do not fully isolate whether stable-rank adaptation itself is responsible for the gains / Can your directly separate the effects of layer-wise nonlinear adaptation and effective-rank adaptation? / Given that transferring the learned rank profile to linear LoRA recovers only part of the gain, how much of the improvement can actually be attributed to rank allocation?
The submission has four controlled contrasts, each removing one ingredient:
| Kept | Removed | Gain over matched LoRA |
|---|---|---|
| Nonlinearity, shared across layers (Table 20) | per-layer freedom | 0.0 of 3.6 |
| Fixed nonlinearity, uniform stable rank (SineLoRA, Tables 1, 4) | learning and heterogeneity | 0.17 of 3.54 (Qwen2 15k) |
| Learned allocation, linear adapter (Table 25) | nonlinearity | 0.61 of 3.54 |
| Everything except learned bandwidth (Tables 3, 26) | learning | 2.3 of 3.6 |
| Full LR-LoRA | -- | 3.6 / 3.54 |
(Table 20 and Table 3 use the single-seed ablation harness, whose LoRA baseline is 67.8. Tables 1, 25 and 26 use the three-seed main protocol, whose LoRA baseline is 52.34.)
Two conclusions follow. First, a nonlinearity without per-layer freedom gives nothing: tying the transfer function across layers returns exactly to LoRA's accuracy, and a fixed nonlinearity that raises stable rank uniformly (SineLoRA) gives little. Second, the learned allocation carries real information. Placed into a linear adapter, it beats uniform ranks and all three permutations of the same rank multiset at an identical parameter count. Allocation alone recovers about 17% of the gain, so most of the gain comes from the interaction: per-layer learning of the map that produces the allocation. The 0.61 measures the allocation's value inside a linear adapter, where integer ranks cannot reproduce the spectral shape that creates inside the nominal rank (LR-LoRA's mean stable rank is 9.4-10.7 at ); we do not claim it is the allocation's full contribution within LR-LoRA.
Within LR-LoRA's own parameterization, which the reviewer asks about, two existing interventions act on the control of the effective-rank profile while leaving the rest unchanged. (i) Fixing the bandwidth (Tables 3 and 26) keeps the per-layer amplitudes, the factors and the sinc nonlinearity, and removes only the learning of . By the bandwidth bound (Proposition K.1, Appendix K), is the parameter that limits how much spectral mass each layer's update can place beyond its leading directions, so this removes each layer's learned control over its spectral spread. It costs 1.3 points (Table 3) and 1.28 points over three seeds (Table 26), the largest effect among the design ablations of Tables 3 and 26. (ii) Sharing only across layers, with amplitudes still per layer, costs 0.6 points (, Table 20), so the heterogeneity of the bandwidth across layers, not only its being learned, contributes. On LLaMA3-8B the learned bandwidths are indeed heterogeneous, from in early layers to in deep layers (Appendix J.4). We note that also changes the shape of the map, so neither intervention changes effective rank alone. In a jointly learned model no intervention does, and we do not claim a clean split. One more piece of evidence bears on attribution: LR-LoRA's gain over LoRA shrinks monotonically as the nominal rank grows, from +5.3 at to +2.7 at (Table 12). The benefit is largest where the fixed-rank constraint binds hardest, as a rank-related mechanism predicts, and Figure 3 shows the corresponding slower spectral decay of . What the evidence supports is an ordering: no per-layer freedom gives nothing, per-layer amplitudes with a shared or fixed bandwidth give most of the gain, and learning a per-layer bandwidth, the parameter that bounds spectral spread, adds a further 0.6 to 1.3 points.
4. Relevant nonlinear baselines are not sufficiently covered in the main comparisons. For example, baselines including LoRAN and AuroRA are compared only on Qwen2-0.5B in the Appendix, but not on Phi-3-8B or LLaMA3-8B in the main paper, making it difficult to assess LR-LoRA against its competitors at meaningful scale. In addition, key mechanistic analyses are restricted to Qwen2-0.5B, making it unclear whether the observed rank heterogeneity itself persists at larger scales.
Table 1 compares LR-LoRA on Phi-3-mini (3.8B) and LLaMA3-8B, in both data regimes, against LoRA, VeRA, NoLA, RandLoRA, the fixed-nonlinearity baseline SineLoRA, and the adaptive-rank baselines AdaLoRA, DyLoRA and ElaLoRA. LR-LoRA is best in all four Phi-3-mini and LLaMA3-8B cells. It leads SineLoRA by 2.78/2.29 (Phi-3-mini, 15k/170k) and 2.80-4.09 (LLaMA3-8B), and the strongest adaptive-rank baseline, ElaLoRA, by 1.10-1.44. On LLaMA3-8B instruction tuning, LR-LoRA scores 6.55 on MT-Bench against 6.22 (SineLoRA) and 6.31 (RandLoRA) (Table 5). The reviewer is right that LoRAN, AuroRA and ABBA appear only in the Qwen2-0.5B 15k comparison (Table 24, where LR-LoRA leads by 3.03, 1.43 and 1.03). We acknowledge this as a limitation of the larger-scale comparison. Two facts in the submission indicate how it bears on the conclusion. First, LoRAN's family, a fixed elementwise map on the update, is present at larger scale through SineLoRA. The two are close at 0.5B (LoRAN 52.85, SineLoRA 52.51, LoRA 52.34; Tables 1 and 24), and at 8B the fixed map does not scale up its advantage: on LLaMA3-8B 15k, SineLoRA falls below LoRA (81.90 against 83.09) while LR-LoRA reaches 85.99. Second, at 0.5B AuroRA and ABBA (54.45, 54.85) lie within 0.83 points of ElaLoRA (54.02), and LR-LoRA leads all three. ElaLoRA is in every Phi-3-mini and LLaMA3-8B cell of Table 1, and LR-LoRA leads it there by 1.10 to 1.44. LR-LoRA's gain over LoRA also persists at scale (+3.54 on Qwen2 15k, +2.34 to +3.56 at 3.8B and 8B). We state that AuroRA and ABBA are compared only at 0.5B, and we make no claim about their ranking at larger scale beyond this evidence.
The second part of this point (mechanistic analyses at 8B) is answered under Q7 below.
5. Can you compare LR-LoRA with baselines on at least one 8B model, rather than only 0.5B?
Yes. Table 1 in the original submission already did this. It compares LR-LoRA with eight PEFT baselines on LLaMA3-8B and Phi-3-mini (3.8B) in both data regimes, and LR-LoRA is best in all four cells. Details, and how the 0.5B-only AuroRA/ABBA comparison bears on the larger-scale conclusion, are in our answer to W4 above.
6. What conditions on are required for rank$(\phi(BA)) > r$? The current "any differentiable nonlinearity" is inaccurate.
We agree "any differentiable nonlinearity" was inaccurate and have removed it everywhere: contributions (lines 82-84), Section 3.3 (lines 195-201), the basis-agnostic paragraph (lines 248-254), and the conclusion (lines 508-510). The precise statements are:
- Sufficient (Lemma 3.1, lines 221-241). If is continuous, nonzero somewhere and as , then for any there is a width- product with and .
- Generic. If in addition is real-analytic, which holds for every nonzero map in our sinc family, then for Lebesgue-almost every . The leading minor is a real-analytic function of the factors that Lemma 3.1 shows is not identically zero, so its zero set has measure zero.
- Not sufficient. Differentiability alone: a polynomial of degree gives for every , and a linear map gives at most .
All three statements are proved in Appendix K. For a specific trained update, the rank depends on as well as (for example, a product with only distinct columns keeps at most after any entrywise map). Because exact rank is generically full, we report stable rank, whose distribution the learned bandwidth controls (Proposition K.1).
We thank the reviewer for seven concrete questions. Several of them (Q1, Q2, Q6) have sharper answers than the submission gave, and we have corrected the paper where the reviewer found it imprecise.
1. Rank is not directly learned or optimized. Instead, ϕ is learned and stable rank is measured afterwards. The interpretation as "learnable rank" remains indirect and unclear. There lacks an explicit control of the rank / In what sense is LR-LoRA learning rank, considering that rank is not explicitly parameterized or optimized? How can we control the rank or stable rank?
LR-LoRA learns a separate elementwise map ϕ for each adapter, using the downstream task loss. The rank of each update follows from the learned parameters. This is what we meant by learning the rank. We agree this is not the best terminology and have thus included the term implicit rank adaptation in the abstract and introduction to show the reader that it is the architecture itself that is changing the rank through the added nonlinearity. We have added a paragraph in Section 3.3 that clearly states this, see lines 259-263.
If an application requires a hard rank cap k, retaining the top k singular components of the update enforces that cap. To encourage a target stable rank during training, one could add a penalty for deviation from that target. Table 25 demonstrates how the learned profile can inform explicit rank allocation. Proposition K.1 (Appendix K) makes the control explicit: the learned bandwidth ωℓ, together with the entry scale of BℓAℓ, bounds how much spectral mass ϕℓ(BℓAℓ) can place beyond its leading (dr+d) directions. It is an upper bound, not a setting of stable rank, so clamping ωℓ≤ωˉ during training would bound each layer's spectral tail (not evaluated as a training variant). The mechanism is visible in the paper: at Qwen2 layer 1 the learned ϕ converges to a near-linear map (Figure 5), which keeps that layer's update close to rank r, while deeper layers develop increasingly nonlinear maps (Figures 6-7).
2. The Nyquist-Shannon sampling-theory motivation seems stronger than necessary. Alternative learned bases also perform reasonably well, suggesting that the main contribution may be the layer-wise learnable nonlinear transformation rather than sinc specifically. / What's the theoretical benefits (e.g., better approximation error or optimization convergence) of using sinc-based Whittaker–Shannon interpolation (2), which is known to be good for bandlimited signals, over other alternative learned/fixed orthogonal bases, particularly for low-rank adaptation of LLMs?
We agree with the reviewer's reading. The main contribution is a separately learned elementwise transformation per adapter, and the submission says sinc is "a well-motivated choice rather than a unique requirement" (Appendix F.1.2). We do not claim a faster approximation rate or a convergence guarantee for sinc over other bases. The finite expansion with learned per-basis bandwidths does not inherit the Whittaker–Shannon reconstruction guarantee, and the revised Section 3.2 presents sampling theory as motivation only.
Sinc does have two properties that matter here, and they are now stated precisely:
Sinc is the only basis in the comparison that has both properties as stated: it satisfies the sufficient condition of Lemma 3.1, and a single learned parameter, ω, provably bounds (Proposition K.1, Appendix K) how much spectral mass lies beyond the leading directions. This makes the learned bandwidth an interpretable per-layer control, and it is the component with the largest measured effect. Fixing ω costs 1.28 points over three seeds (Table 26), more than switching the basis to Gaussian RBF (0.68) or to a learned elementwise MLP (0.98). We present this as a consistent ordering, not a proof that sinc must win.