acceptodds
Under review as a conference paper at ICLR 2027

Are Learned LoRA Rank Allocations Reliable? A Controlled Audit on Llama-2-7B

Abstract

Most learning-based and heuristic LoRA rank-allocation methods rest on an uncontrolled comparison: in the coupled-seed protocol, each candidate is evaluated on its random initialization, so allocation and seed are inseparable in one number. We propose a seed-decoupled protocol: one master seed derives three independent streams fixing scoring-head initialization, LoRA initialization, and data order/dropout; each candidate is compared pairwise with a shared equal-budget uniform_r8 reference. Within a master seed only the allocation varies, structurally removing the shared component that accounts for 94.1% of run-to-run variance. Cost: one extra reference run per master seed; gain: 4.7–6.6× tighter comparison error at equal runs. On the same panel and budget, the two protocols disagree: the coupled one puts all four search methods ∼1 point above uniform rank 8; under ours, the advantage disappears (the sole equal-budget search output retested:  pp, Holm ). The divergence comes from the training protocol, not the search methods. Fixing the parameter budget and rank multiset while permuting only internal positions shifts accuracy by 4.3 points, four times any method difference this field seeks to detect, and the search output we retest ranks below its own three hand permutations: reported gains thus fall within the variance the literature leaves uncontrolled. Our protocol lets a new rank-allocation method be audited before it claims superiority over uniform rank; fair comparison needs at least three conditions: equal parameter budget, fixed rank multiset, shared reference. Without all three, a comparison may measure placement alone.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, LoRA, rank allocation, reproducibility, empirical audit

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iUJfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a seed-decoupled evaluation protocol for LoRA rank allocation and uses it to audit whether learned/heuristic allocations genuinely outperform uniform rank. The core concern is valid: under coupled seeds, allocation and initialization are confounded. By fixing head, LoRA, and data streams per master seed and pairing each candidate against a shared equal-budget uniform r8 reference, the protocol removes 94.1% shared variance and tightens comparison error 4.7–6.6× at the cost of one extra reference run per seed. Empirically, a ~1-point advantage under coupled seeds vanishes (−0.63 pp, Holm p = 0.578), while permuting positions at fixed budget and rank multiset shifts accuracy by 4.3 pp—larger than typical claimed effects. The conclusion that fair comparison requires equal budget, fixed rank multiset, and a shared reference is well supported.

Critical strengths

Consider other factors affecting fine-tuning performance.

Critical weaknesses

Evaluating only on Llama-2-7B is not sufficiently convincing. In addition, there is a lack of extensive ablation experiments.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Reply to Reviewer iUJf

Based on your critique and suggestions, we have made further additions: the Llama-3-8B experiments promised in our first reply (noise decomposition and 8-seed paired panels), a 7B Qwen model, the generative reasoning task GSM8K, a q,k,v,o_proj adapter variant, and a full set of ablations that the revised paper now reports in the main text. The positional preference is consistent across three model families and three tasks, and the same permutation ordering holds in all five model–task cells (Kendall's W=1.000W = 1.000). This supports our central claim: rank allocation has a real positional effect, it can be measured only under paired, equal-budget evaluation, and Profile-then-Retrain (PtR) recovers it (+0.88+0.88 pp, 24/24 seed pairs).

1. Beyond Llama-2-7B: delivered as promised.

From §6.2 of the revised paper:

The ordering of its three common permutations is identical in all five cells (Kendall's W=1.000W = 1.000; Friedman χ2(2)=10.0\chi^2(2) = 10.0, p=0.007p = 0.007). The permutations ranked last and first differ by +2.31+2.31 pp (sd 1.01, same sign in 5/5 cells); this contrast holds within each cell on its own.

additionresultpaper
Llama-3-8B noise decompositionhead / data order / LoRA init: 45.5% / 50.3% / 4.2%, so pairing is needed in both families§3.2
Llama-3-8B pre-registered test, 8 seedsv_heavy +0.59+0.59 / +0.52+0.52 (7/8, 8/8); q_heavy −1.81-1.81 / −1.62-1.62 (0/8, 0/8)§6.1, Tab. 3
five cells: L2/MNLI, L3/MNLI, L2/GSM8K, L3/QNLI, Qwen/MNLIidentical ordering, W=1.000W = 1.000; spans 0.86–3.26 pp§6.2, Tab. 4
GSM8K (generative, no new head)span 3.26 pp, the largest of the five cells; best permutation +0.83+0.83 pp, 6/6§6.2–6.3
q,k,v,o_proj adapterssame ordering as the q,v cells; span 3.19 ppTab. 4

On the pre-registered test, §6.1 states:

the predicted direction holds in all eight cells ... its magnitude differs threefold between the two model families (7.11 vs. 2.40 pp) and its direction is the same.

Llama-3 and Qwen use grouped-query attention; there the winning allocations carry fewer parameters than uniform (§3.1), so the gains do not come from capacity.

2. Ablations. Each row below is a controlled comparison at a fixed parameter budget with paired seeds.

ablationfindingpaper
seed streams (S/U/I), fixed scalingσΔ\sigma_\Delta = 0.76 / 2.63 / 3.01 pp; the unpaired head gives 75% of the excess variance§3.2
independent search repetitions (4 procedures × 3 seeds)12/12 optima negative; mean −1.09-1.09 [−1.52,−0.67][-1.52, -0.67], p=0.0002p = 0.0002§4, Tab. 1
PtR vs end-to-end AdaLoRA+0.18+0.18 → +0.88+0.88 pp (24/24); runtime dilution +0.70+0.70§5.2
PtR: direction vs layer structuredirection +1.40+1.40 (8/8); layers +0.45+0.45 (8/8)§5.3, Tab. 2
PtR robustness (3 maps × 8 seeds)F=0.10F = 0.10, p=0.90p = 0.90; each map positive on 21/21 unseen seeds§5.3
permutation vs multiset (6 multisets, 35 points)permutation 61.1% of variance, multiset 38.9%§6.2
adapter scaling, constant2 → constant8span 4.39 → 1.41 pp (3.1×3.1\times); τb\tau_b 0.663 → 0.231§6.3
task transferMNLI→QNLI τ=1.0\tau = 1.0; CoLA τ=0.47\tau = 0.47§6.3
first-order rank statisticsall ∣ρ∣<0.35\lvert\rho\rvert < 0.35, none significant§6.3
best-of-mm selectionranking ρ=0.90\rho = 0.90–0.970.97; inflation 0.9–1.5 pp§7, Tab. 5
collapse: head × data order, layer freezing3/4 arms normal; freezing layers 0–1 restores 84.33%§8

The ablations agree. Searched optima fall below uniform; they looked positive because picking the best single reading inflates it by about as much as the gains reported in the literature. The rank map that AdaLoRA learns carries real positional information. PtR keeps that information and removes the max step: from §5.1, "§4 and this section use the same protocol, budget and seed pairing; §4 gives a negative verdict and this section a positive one (24/24)."

Thank you again for pushing us toward broader evidence. We hope the additional models, the generative task and the ablations address your concerns, and we would be grateful if you reconsidered your assessment.

I appreciate the substantial additions, but the following concerns remain unresolved:

1 The q/v contrast is 7.11 pp on Llama-2 but only 2.40 pp on Llama-3. A threefold gap means the deployable gain cannot be predicted from the paper's characterization. → Add a limitation statement, or analyze how effect size scales with model width / GQA grouping.

2 Evaluation surface is narrow. The main conclusion rests almost entirely on MNLI-m, with QNLI, CoLA, and GSM8K serving only as consistency checks. → Add at least one long-form generation or instruction-following benchmark under the same paired protocol, to rule out that the positional effect is specific to classification heads and short sequences.

3 No comparison with scaling variants (rsLoRA, DoRA). Scaling and rank position may interact, and this omission weakens the "position rather than magnitude" conclusion. → Clarify whether scaling is orthogonal to the positional effect, and if not, test it.

Therefore, I keep my original score unchanged.

Reviewer_LTW4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether learned LoRA rank allocations remain reliable across random seeds. The authors propose an evaluation protocol that separately controls classification-head initialization, LoRA initialization, and data order/dropout, and compares candidate allocations against an equal-parameter-budget uniform-r8 baseline through pairing under shared master seeds. Experiments on Llama-2-7B across MNLI, QNLI, and CoLA examine placement, scaling policies, and cross-task behavior, alongside adapter-update diagnostics for two failed training runs. The results show that pairing substantially reduces cross-seed variability; the sole TPE search output included in the equal-budget main analysis does not significantly outperform uniform-r8; and permuting a fixed rank multiset produces a 4.27-percentage-point performance span on MNLI under constant2. Allocation effects also vary with scaling policy and task. The authors conclude that, within the examined settings, search outputs have not demonstrated reliable superiority over uniform allocation, and that claims of improvement require more rigorous controlled, multi-seed evaluation because placement and training protocols materially affect the results.

Critical strengths

The authors report paired differences, standard deviations, win counts, confidence intervals, and multiple-comparison corrections, helping readers distinguish effect magnitude from cross-seed variability. They also explicitly acknowledge that two failure cases cannot support a causal explanation of the failure mechanism.

Critical weaknesses
  1. Limited methodological novelty: The paper does not establish what seed-stream decoupling adds beyond standard shared-seed blocking and paired analysis. Moreover, auditing only one equal-budget search output limits broader conclusions about learned rank-allocation methods.

  2. Unresolved attribution and selection bias: Scaling and seed protocols change simultaneously between the native and main settings, preventing attribution to seed decoupling alone.

Questions for authors
  1. Under identical scaling and training settings, what additional benefit does seed-stream decoupling provide over ordinary shared-seed pairing?

2.Do independent search repetitions support the same findings?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer LTW4

Based on your critique and suggestions, we have made further additions: we extended the equal-budget audit to all 12 search optima (4 search procedures × 3 independent search seeds), ran a three-arm factorial that changes only the seed protocol at identical scaling and training settings, and measured the selection bias directly. All 12 optima fall below same-budget uniform allocation. Pairing the head stream, which ordinary same-seed runs leave unpaired, removes 75% of the excess variance. Picking the best of mm noisy readings inflates the reported number by 0.9–1.5 pp, enough to make negative optima look on par with or better than uniform. This supports our central claim: the learned rank profile carries real signal (PtR: +0.88+0.88 pp, 24/24 seed pairs), and the failure of search lies in the final step of picking one noisy reading.

1. Q2 / W1: Do independent search repetitions support the findings? Yes, 12 of 12.

From §4 of the revised paper:

We take back all 12 optima of the search phase, from 4 search procedures × 3 search seeds ... None of the 10 newly re-evaluated optima is positive. Their Δ\Delta ranges from −0.29-0.29 to −2.37-2.37 pp, 38 of 40 seed pairs are negative, and the mean is −1.09-1.09 pp (95% CI [−1.52,−0.67][-1.52, -0.67]; one-sample tt over the 10 optima, p=0.0002p = 0.0002). The two from the original panel are negative as well (−0.63-0.63 and −3.91-3.91).

Paired Δ\Delta (pp) against the same-budget uniform reference, Llama-2-7B / MNLI-m (Table 1):

proceduresearch seed 427772025
GA−0.29−0.51−1.31
TPE−0.55−1.02−0.63†
SAGA-RA−1.17−2.37−3.91†
SAGAorig−1.00−1.22−1.49

†original 8-seed panel; all other cells are re-run with 4 seeds (E2a).

The result holds for every procedure and every search seed. §4 also shows where the search scores came from:

The re-evaluated score has a Spearman correlation of ρ=0.94\rho = 0.94 with the optimum's own budget ... After the budget effect is removed, the search-phase score has no predictive power for the re-evaluated score (ρ=−0.20\rho = -0.20, p=0.58p = 0.58).

2. W2 / Q1: Attribution under identical settings. The three arms below use the same candidates, the same master seeds, constant2 scaling and the same training recipe. Only the seed protocol changes. From §3.2:

For the same candidate and the same seeds, the SD of the paired difference is 0.76, 2.63 and 3.01 pp under S, U and I (mean over 4 candidates; E1). Releasing only the head raises the noise 3.5-fold, and the unpaired head accounts for about 75% of the excess variance ... a candidate whose true effect is −3.18-3.18 pp looks more than 1 pp better than uniform in 2 of 8 single-seed comparisons under arm U.

armstreams sharedσΔ\sigma_\Delta (pp)
S: decoupled pairing (ours)head, data order, LoRA init0.76
U: ordinary "same seed"data order, LoRA init2.63
I: independentnone3.01

Arm U is ordinary shared-seed pairing as the standard pipeline runs it. Decoupling adds a pairing that actually holds. It gives a resolution of ±0.6\pm0.6 pp with 8 seeds, against ±1.5\pm1.5 pp for a single seed. The head share is large on Llama-3-8B too (45.5%, §3.2). As promised, C1 is now framed as a measurement protocol (contribution (i)), and we do not claim it as a new blocking scheme.

3. Selection bias: measured and decomposed (§7, Table 5). Single readings rank candidates well (leave-one-out ρ=0.90\rho = 0.90–0.970.97 over four panels). The bias comes from reporting the maximum:

termΔ\Delta (pp)
re-evaluated value (10 optima)−1.09-1.09 [−1.52,−0.67][-1.52, -0.67]
best-of-mm inflation, σinter E[max⁡m]\sigma_{\text{inter}}\,\mathbb{E}[\max_m], m=13,35m = 13, 35+0.9+0.9 to +1.5+1.5
reported reading under single evaluation−0.22-0.22 to +0.39+0.39
PtR, 3 AdaLoRA rank maps (no max step)+0.88+0.88 [+0.41,+1.35][+0.41, +1.35], 24/24

4. Novelty beyond the audit. From §5.1:

§4 and this section use the same protocol, budget and seed pairing; §4 gives a negative verdict and this section a positive one (24/24). The difference lies in whether there is a max step.

PtR freezes the rank map AdaLoRA learns (64 modules, rank sum 512) and retrains it with standard LoRA via rank_pattern, at zero inference cost. Its gain splits into a direction term (+1.40+1.40 pp, 8/8) and a layer term (+0.45+0.45 pp, 8/8), and each map stays positive on 21/21 unseen seeds.

We thank you for the two questions; they led directly to the 12-optimum audit and the S/U/I factorial. We hope these results fully address your concerns.

Reviewer_qJr2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents an audit on LoRA rank allocation. The authors argue that existing methods often claim superiority based on coupled-seed evaluations, where the effects of the rank allocation strategy are indistinguishable from the random noise of the training process. To support this argument, the authors tested various rank-allocation and permutation methods and found that reported gains often vanish when accounting for noise. This study concluded that the placement of rank is a dominant factor, and rank allocation methods provide negligible improvement over random or hand-tuned methods.

Critical strengths
  1. The introduction of the seed-decoupled protocol, and this paper provides a framework for isolating structural effects from initialization noise.
  2. The authors provide an explanation of "adapter collapse." By identifying how energy concentrates in shallow layers during failed runs, they provide a diagnostic tool that helps researchers understand why certain fine-tuning attempts underperform.
  3. This paper conducted different experiments over different rank allocation and rank placement.
Critical weaknesses
  1. This paper only evaluated rank allocation on one model, less convincing for their conclusions.
  2. The authors solely focus on simpler NLP classification tasks, such as MNLI and QNLI, without considering generative tasks or tasks that need a reasoning process.
  3. While the paper describes what collapse looks like, it does not fully resolve the cause of why. Whether the shallow concentration is the trigger for the collapse or merely a downstream symptom of it.
Questions for authors
  1. Do you expect the dominance of seed variance over allocation strategy to remain constant as the base model scales up or the base model family changes, or might different models exhibit more distinct structural preferences?
  2. How do you think the impact of other hyperparameter choices, such as learning rates, layer-wise ranks, or different modules to be LoRA adapted?
  3. Could you provide more evidence on different series of models and tasks, rather than the GLUE task on Llama2-7B?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.