acceptodds
Under review as a conference paper at ICLR 2027

BARS-LoRA: Budget-Aware Rank Stabilization for Adaptive Low-Rank Adaptation

Abstract

Adaptive-rank LoRA decides during training where to spend a limited rank budget, and each decision is only as reliable as the evidence behind it. This evidence degrades in two distinct ways. Hard masks, as in AdaLoRA, commit rank from mini-batch salience estimates that remain noisy under short schedules and can leave entire modules without rank. Soft gates, as in SoRA, read their sparsity pattern from a rank-one dictionary that keeps drifting as training continues. We propose BARS-LoRA (Budget-Aware Rank Stabilization), which applies one principle, stabilize the evidence before committing rank, through two instantiations. SCALP-LoRA scores each hard channel by its first-order deletion effect, which we show is the common signed quantity underlying all three terms of the AdaLoRA triplet score; it discounts this score by its relative dispersion and allocates rank per Linear module with a guaranteed minimum rank. SAGE-SoRA orthogonalizes the gated dictionary and consolidates its basis before the sparse gate pattern is read out. The two branches specialize by training budget. On GLUE with Qwen3-0.6B, SCALP reaches 84.75% six-task accuracy under a four-epoch schedule, 1.15 points above AdaLoRA. At 800 updates, SAGE reaches 82.55% four-task accuracy, 1.64 points above SoRA. On Qwen3-1.7B, SAGE ranks first among seven methods on every task and exceeds DoRA by 1.74 points.

Primary area:
optimization
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, rank stabilization, training budget, LoRA, GLUE, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EBNgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BARS-LoRA studies instability in adaptive-rank LoRA and proposes two budget-dependent variants. SCALP-LoRA stabilizes hard rank allocation using channel-level Taylor salience, dispersion-aware discounting, minimum-rank module allocation, and final consolidation. SAGE-SoRA stabilizes soft gating through basis orthogonalization and consolidation.

Critical strengths
  • The paper provides a coherent diagnosis of two different rank-allocation failure modes and develops corresponding solutions.

  • The evaluation includes three seeds, tuned fixed-rank controls, component studies, and a second backbone. The strong Qwen3-1.7B result supports the practical value of SAGE-SoRA.

Critical weaknesses
  • The SCALP ablation does not convincingly validate its main scoring components. Taylor scoring, dispersion discounting, and Linear-level allocation do not individually improve the four-task mean over AdaLoRA; most of the final gain appears to come from consolidation.

  • The “budget crossover” is not a fully controlled crossover: the final four-epoch point uses task-dependent numbers of updates, whereas the earlier points use uniform 800/1600/3000-update schedules.

Questions for authors
  • How sensitive are the methods to the newly introduced hyperparameters? SCALP introduces several parameters for the dispersion discount and minimum-rank allocation, while SAGE introduces the orthogonality coefficient and consolidation schedule. Were these values selected on the same tasks used for reporting the final results?

  • What is the additional training cost of the two methods? Please report wall-clock time, peak memory, or FLOP overhead relative to AdaLoRA and SoRA.

  • This is partly a scope clarification rather than a request for a large new experiment. The current validation covers GLUE classification. Do the authors expect the same crossover to hold outside prompt classification?

Reviewer_NiK4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BARS-LoRA, a framework for improving adaptive low-rank fine-tuning by stabilizing the evidence used for rank allocation. It consists of two branches: SCALP-LoRA improves hard-mask allocation through channel-level Taylor scoring, relative-dispersion discounting, and minimum-rank constraints; SAGE-SoRA improves soft gating through dictionary orthogonalization and late-stage decay of basis updates. Experiments on GLUE compare different training budgets and report advantages for the two branches under different settings.

Critical strengths
  1. The paper addresses noisy importance estimates in hard masking and basis drift in soft gating. Its shared principle of stabilizing allocation evidence before committing rank provides a meaningful perspective on improving adaptive rank allocation.

  2. Equation (10) connects the first-order channel-deletion derivative to the signed gradient contractions of AdaLoRA’s three parameter groups, providing a rationale for simplifying the score. The channel score also reuses existing gradients without requiring an additional backward pass.

  3. The experiments include multiple PEFT baselines, fixed-rank and learning-rate sweeps, and two model scales. On Qwen3-1.7B, SAGE-SoRA achieves a four-task average accuracy of 86.52%, exceeding DoRA, the strongest comparator in that table, by 1.74 percentage points.

Critical weaknesses
  1. In Table 2, consolidation alone accounts for most of the observed improvement. Full SCALP exceeds this control by only 0.10 percentage points, and full SAGE by 0.17 percentage points, while orthogonalization alone reduces average accuracy. The reliable incremental benefits of the new scoring, dispersion discounting, and orthogonalization beyond the training schedule therefore remain insufficiently established.

  2. SAGE leads at every tested uniform update budget, whereas SCALP leads only at the task-scaled four-epoch endpoint. Although this distinction is explicitly disclosed, four epochs correspond to only 1,200 updates for RTE and are not uniformly longer than 3,000 updates. The observed ranking change therefore cannot be attributed solely to increased training duration.

  3. SCALP constrains total rank, but equal total rank does not automatically imply equal parameter counts or computational costs. SAGE’s final nonzero gate counts, actual parameter counts, and runtime overhead are also insufficiently reported. Moreover, the 1.7B comparison omits its direct predecessor, SoRA, limiting the assessment of SAGE’s incremental contribution at that scale.

Questions for authors
  1. Could the authors discuss recent related work such as IGU-LoRA and clarify SCALP-LoRA’s key differences in importance estimation, uncertainty handling, and computational overhead?

  2. Could the authors report paired confidence intervals relative to the consolidation-only controls and provide leave-one-component-out ablations to establish the incremental benefits of the additional stabilization mechanisms?

  3. Could the authors extend the uniform-budget experiments beyond 3,000 updates, using a fixed task set and comparable learning-rate schedules, to verify whether a reproducible crossover between hard masking and soft gating occurs?

  4. Could the authors report final effective ranks, actual adapter parameter counts, training time, and peak memory, and include a budget-matched comparison with SoRA on Qwen3-1.7B?

Reviewer_M6ioOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies adaptive-rank LoRA methods that allocate a rank budget across layers during training. Authors argue existing methods fail in two ways: hard-mask methods (AdaLoRA) prune ranks from noisy mini-batch scores, while soft-gate methods (SoRA) learn sparsity on a drifting rank-one dictionary. Under the principle of "stabilize evidence before committing rank", they propose two variants. SCALP-LoRA scores each channel by first-order deletion sensitivity, discounts noisy channels via a coefficient-of-variation (CV) factor, guarantees a minimum rank of 1 per layer, and applies a final-phase cosine learning-rate decay ("readout consolidation", Eq. 23). SAGE-SoRA adds an orthogonality regularizer to SoRA (Eq. 22) plus the same final-phase LR decay. On 6 GLUE tasks with Qwen3-0.6B (3 seeds), SCALP gets the highest 4-epoch mean (84.75, +1.15 over AdaLoRA) and SAGE gets the highest 800-step mean on 4 tasks (82.55, +1.64 over SoRA; and +1.74 over DoRA on Qwen3-1.7B). Authors report a budget crossover (SAGE wins at short budgets, SCALP at 4 epochs) and argue training budget should guide adapter choice.

Critical strengths
  1. Solid short-budget gain for SAGE-SoRA. At 800 steps, SAGE beats SoRA by 1.64 pts (82.55±0.2282.55 \pm 0.22 vs 80.91±0.6380.91 \pm 0.63, Welch pp ~ 0.03) and keeps a +2.01 pt lead at 1600 steps (Tab. 4). On Qwen3-1.7B at 800 steps, SAGE beats DoRA by 1.74 pts (86.52±0.1486.52 \pm 0.14 vs 84.78±0.4784.78 \pm 0.47, Tab. 3).
  2. Clean and honest ablations. Authors report 3 seeds with per-task SDs, run a 4-rank by 4-LR sweep for fixed-rank LoRA (Tabs. 5-6), and include "consolidation-only" rows in Tab. 2.
  3. Interesting empirical lesson in Tab. 2. The ablations suggest late-phase step-size annealing matters much more than importance-score design in adaptive-rank LoRA. If confirmed with schedule-matched baselines, this is a useful takeaway.
Critical weaknesses

Paper's own ablations do not support its main story of "stabilizing rank evidence" or "a budget-dependent crossover". Almost all of the accuracy gain comes from a standard final-phase cosine LR decay (Eq. 23), and paper does not state whether baselines used the same LR schedule.

W1. Accuracy gains come from a final-phase cosine LR decay, not from evidence stabilization (Tab. 2, Eq. 23).

  1. "Readout consolidation" (Eq. 23) is simply a cosine LR decay from 3e-4 to 3e-5 over the final phase (the SGDR schedule [R5]).
  2. In Tab. 2a, adding Taylor scoring, CV discount, Linear-module pooling, and minimum rank 1 to AdaLoRA gives 81.83-81.91 (below AdaLoRA's 82.01). By contrast, adding consolidation alone to AdaLoRA jumps to 82.48, and full SCALP gets 82.58. Thus, the cosine LR tail accounts for ~82% of SCALP's gain, while the stabilization rules add only +0.10 (within 3-seed noise).
  3. In Tab. 2b, adding orthogonality alone to SoRA lowers accuracy (80.58 vs 80.91). Adding consolidation alone jumps to 82.38, and full SAGE gets 82.55. So the cosine LR tail accounts for ~90% of SAGE's +1.64 pt gain.
  4. In SCALP, this LR decay runs after the rank mask is already frozen (p5 L257-259). This contradicts the core principle of stabilizing evidence "before it is committed" (p3 L151-153).
  5. Crucially, the AdaLoRA and SoRA rows in Tabs. 1 and 4 match the "no-consolidation" rows in Tab. 2, and paper never states the LR schedule used for any baseline. If baselines ran without a comparable LR decay, the reported margins may simply reflect the LR schedule rather than BARS-LoRA.

W2. The claimed budget crossover does not hold in Tab. 4. At the 4-epoch endpoint where SCALP is claimed to overtake SAGE, the 4-task means are a statistical tie: SCALP gets 82.58±0.6982.58 \pm 0.69, SoRA gets 82.48±0.5382.48 \pm 0.53, and SAGE gets 82.32±0.2082.32 \pm 0.20 (Welch pp ~ 0.85 and 0.59, Tab. 4). In fact, SAGE has the highest mean at every uniform step budget (800, 1600, and 3000 steps), and SAGE at 1600 steps (83.75) beats every 4-epoch score. Also, the 4-epoch budget scales with dataset size (RTE gets only ~1200 steps, which is shorter than the 1600-step budget), so it is not a clean long-budget test.

W3. The testbed is insensitive to rank, and the claimed failure modes are not measured (Tabs. 3, 5, 6, Fig. 7).

  1. Across r∈{4,8,12,16}r \in \{4, 8, 12, 16\}, tuned fixed-rank LoRA varies by only 0.31 pts at 4 epochs (where r=4r=4 is best) and 0.56 pts at 800 steps (Tabs. 5-6). When rank barely affects accuracy, rank allocation has little room to help. Indeed, SCALP vs LoRA r=4r=4 (+0.57, pp ~ 0.31) and SAGE vs LoRA r=16r=16 at 800 steps (+0.40, pp ~ 0.18) are statistical ties.
  2. Paper never directly measures either diagnosed failure mode (mask turnover across pruning steps, or subspace drift of A,BA, B in SoRA vs SAGE). Fig. 7 lacks a SoRA curve and averages gates across unaligned seeds, and SAGE's final active rank is not reported. Also, the 1.7B table (Tab. 3) omits SoRA and LoRA r=16r=16.

W4. Most technical components come from prior work without full credit (Eqs. 10-13, 22, 23). Eq. 10 follows directly from Euler's identity for multilinear forms and matches Molchanov et al. [R4, Eq. 9]. SCALP's importance score (Eq. 11) is the exact s(λ)s(\lambda) term of AdaLoRA [R1]. The EMA mean and deviation trackers (Eqs. 12-13, with β=0.85\beta=0.85) are identical to PLATON [R3] (uncited) and AdaLoRA [R1], and a mean-over-deviation rank score was also published in IGU-LoRA (ICLR 2026) [R6]. Eq. 22 is AdaLoRA's orthogonality regularizer (weight 0.1), which SoRA [R2] explicitly chose not to use. Finally, Eq. 23 is SGDR [R5]. The genuinely new parts (bounded CV discount and minimum rank 1) add only +0.04 and +0.05 in Tab. 2a.

References

[R1] Q. Zhang, M. Chen, A. Bukharin, et al. AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR 2023. arXiv:2303.10512.

[R2] N. Ding, X. Lv, Q. Wang, et al. Sparse Low-rank Adaptation of Pre-trained Language Models. EMNLP 2023. arXiv:2311.11696.

[R3] Q. Zhang, S. Zuo, C. Liang, et al. PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance. ICML 2022. arXiv:2206.12562.

[R4] P. Molchanov, A. Mallya, S. Tyree, et al. Importance Estimation for Neural Network Pruning. CVPR 2019. arXiv:1906.10771.

[R5] I. Loshchilov, F. Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR 2017. arXiv:1608.03983.

[R6] X. Cui, H. Li, R. Zeng, et al. IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring. ICLR 2026. arXiv:2603.13792.

Questions for authors
  1. Schedule-matched baselines (W1). What LR schedule was used for each baseline in Tabs. 1, 3, 4, 5, and 6? If you run LoRA (r=4,16r=4, 16), DoRA, AdaLoRA, and SoRA with the exact same Eq. 23 cosine LR tail at 800 steps and 4 epochs (and include SoRA and LoRA r=16r=16 at 1.7B), how much margin remains for SAGE and SCALP?
  2. Uniform long budget & crossover (W2). What happens at a uniform long step budget (e.g., 5000 steps for all 4 tasks)? Given the 4-epoch tie (pp ~ 0.59-0.85), will you narrow the "budget crossover" claim?
  3. Direct mechanism checks & active rank (W3, W4). Can you report (a) SAGE's and SoRA's final active rank and parameter count, (b) a direct measure of mask turnover (AdaLoRA vs SCALP) and subspace drift (SoRA vs SAGE), and (c) how SCALP's CV discount compares with PLATON [R3] and IGU-LoRA [R6]?
1 comment

As authors didn't post any reply during the rebuttal period, I have no further updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.