acceptodds
Under review as a conference paper at ICLR 2027

When Does Adaptive LoRA Rank Allocation Help? Signal Reliability and Resource Accounting

Abstract

Adaptive LoRA redistributes a limited rank budget, yet final accuracy alone does not isolate the benefit of rank placement. Allocation scores depend on existing ranks, while rank transfers can change parameter cost and model state. We analyze the scores and test transfers at fixed active parameter count. For a score based on loss progress, a sign change shared by all modules can reverse their rankings even when the ordering of score magnitudes persists. We separate the immediate loss change caused by a transfer from the loss change during continued training. Experiments cover eight GLUE tasks and quantized Llama commonsense workloads. On commonsense, Spearman correlations between consecutive measurements average 0.944 for score magnitudes and -0.155 for signed scores. At selected RTE checkpoints, online allocation activates 16.4% more parameters than uniform at equal total rank. At a shared RTE checkpoint, one of three module pairs yields lower final loss after moving rank to the higher-scored module, but a less favorable loss change over 20 updates than the reverse transfer. Synthetic controls show that concentrating rank resolves local capacity shortages, while repeated adjustment helps after demand shifts.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, adaptive rank allocation, signal reliability, resource accounting

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_xcs2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an analytical and experimental study that takes a step back to ask a practical question about adaptive LoRA methods: when rank gets shuffled around between layers, does it actually help because the capacity is placed in a smarter spot, or is it just accidentally throwing more parameters and training compute at the model? Specifically, the paper reveals two key factors: signal reliability and resource accounting. On the resource side, it points out that matching "total rank" is a trap because one unit of rank in a feed-forward layer costs way more parameters than a unit in an attention layer, meaning adaptive methods often sneakily activate 16% more parameters than uniform baselines. On the signal side, it builds a learning-progress score and derives a clean Spearman rank correlation identity showing that whenever global calibration loss randomly ticks up instead of down, the sign flip completely reverses the module rankings, even though the underlying gradient magnitudes stay rock-steady. By testing controlled rank transfers from a shared checkpoint and running synthetic toy models, the paper shows that immediate loss drops from changing LoRA scaling factors can masquerade as better learning, and that online reallocation really only shines when task demands actively shift mid-training.

Critical strengths
  • The paper identifies an interesting question that seems to be practical and important.

  • A lot of papers in the adaptive LoRA space just match the terminal total rank and call it a fair fight. This work does a great job pointing out that one rank unit in an attention projection costs way fewer parameters than one in a feed-forward projection.

  • The synthetic setups in Section 5 do a nice job illustrating when adaptive allocation actually matters in theory.

Critical weaknesses
  • The presentation of the entire paper feels pretty disjointed and jumps into heavy concepts without enough setup. I will report two examples here and there are many other similar inconsistencies in the paper. In Section 1, the third paragraph suddenly brings up "resource accounting" and dives straight into exact parameter counts for DeBERTa and training trajectories without properly motivating why the community has been getting this wrong or giving a smooth conceptual transition from the previous paragraph on signal reliability. Similarly, in Section 3.2, the term "calibration loss" is dropped right into the text out of nowhere. There is no clear upfront explanation in the main text of what this calibration loss actually is, why a separate calibration subset is needed instead of just using the training batch loss, or what its exact functionality is, leaving the reader hanging until tiny implementation details show up way back in Appendix B. Therefore, I feel that the paper is very hard to follow and I can’t get the main point of the paper.

  • There is a big disconnect between the real-task online score and the synthetic online score. In the main GLUE and Llama experiments, the online policy uses the learning-progress score Gi,kG_{i,k} (Equation 5), which distributes global loss changes across modules using current gradient norms. And a huge chunk of the paper is spent showing that this exact score suffers from sign flips. But then, when moving to the synthetic controls in Section 5 to show that online allocation actually helps under shifting demand, the paper ditches Gi,kG_{i,k} and swaps in a totally different "direct module probe" G^i\widehat{G}_i that perturbs one module at a time. This makes the narrative feel a bit disconnected: if the main online score has this huge sign-flip flaw, testing a completely different (and much more expensive) per-module probing score in the synthetic benchmark doesn't really show whether the main online method can actually track demand shifts.

  • The main experiments don't actually enforce the paper's own fair-accounting rule. Even though the paper makes a really strong case that matching total rank (BTB_T) is unfair because it changes PactiveP_{\text{active}} by up to 16.4%, the main benchmark tables (Tables 2 and 4) still only match total rank BTB_T rather than matching active parameter budgets PactiveP_{\text{active}}. Aside from the small 20-step intervention on three module pairs in RTE and the GoRA component study (which accidentally uses fewer parameters due to rounding/clipping), there isn't a full end-to-end training baseline where uniform, one-shot, and online policies are strictly constrained to the exact same PactiveP_{\text{active}} budget.

Questions for authors
  • Why not fix the sign-flip issue in Gi,kG_{i,k} and test a corrected online policy? Since Equation 6 shows that the negative Spearman correlation comes entirely from the shared global loss-difference sign sgn(akak+1)\text{sgn}(a_k a_{k+1}) integrating into the EMA zi,kz_{i,k}, what happens if the online allocator just uses the magnitude ∣Gi,k∣\vert{}G_{i,k}\vert{} or drops the ℓk−1−ℓk\ell_{k-1} - \ell_k factor altogether? Does training with a sign-corrected online score actually beat uniform allocation on GLUE and Commonsense?

  • How does the real-task learning-progress score (Gi,kG_{i,k}) perform on the synthetic shifting-demand benchmark? In Section 5, the synthetic control swaps out Gi,kG_{i,k} for a direct per-module perturbation probe G^i\widehat{G}_i (which requires m+1m+1 forward passes per event). If the standard online score Gi,kG_{i,k} from Section 3.2 is run on the synthetic phase-shift benchmark instead of the direct probe, does it still manage to track the demand shift at step 800, or do the sign flips wreck its ability to reallocate rank?

Reviewer_PsfBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes when adaptive LoRA rank allocation helps, focusing on signal reliability and resource accounting. It shows that equal total rank can correspond to different active parameter counts, and derives a Spearman identity showing that, for a learning-progress allocation score, a shared sign change in loss progress reverses module rankings while magnitude rankings persist. It further tests matched-cost rank transfers from a shared RTE checkpoint, separating the immediate effect of a transfer from subsequent learning, and uses synthetic controls to distinguish local capacity shortages from demand shifts. Experiments cover DeBERTa-v3-base on GLUE and Llama-3.1-8B (QLoRA) on commonsense tasks and GSM8K.

Critical strengths
  1. The paper raises an important and under-examined question: whether gains attributed to adaptive rank allocation come from better placement of capacity or from confounds such as additional active parameters, rank trajectories, or changed initialization.
  2. The resource accounting is careful and transparent, reporting active and stored parameters, rank-steps, and training time, and the component study usefully separates GoRA's allocation from its initialization/scaling.
  3. The intervention design (matched-cost transfers from a shared checkpoint with deterministic replay, and the decomposition F=(L0−−L0+)+DF = (L_0^- - L_0^+) + D) is a clean methodology for isolating the effect of rank placement.
Critical weaknesses
  1. Limited scope of the signal-reliability analysis. The sign-reversal analysis applies to a learning-progress score constructed by the authors, in which every module's score is multiplied by a shared loss difference, so Equation 6 follows directly from multiplying all scores by a common scalar. Scores used in practice (e.g., AdaLoRA and GoRA) do not have this shared-sign structure and are not analyzed, so it is unclear how much the finding informs the reliability of existing allocation methods.

  2. Limited evidence from the rank-transfer intervention. The intervention uses a single RTE checkpoint, three module pairs (all feed-forward input projections), 20 continuation updates, and one seed, with loss differences of about 0.005–0.02 nats and no variability estimate. In pair 1, validation loss increases in both branches (0.677 → 0.777 and 0.700 → 0.795), suggesting an overfitting regime. The claim that final loss and subsequent learning can favor opposite directions rests on one of these three pairs.

  3. The resource-accounting principle is not applied to the main comparisons. That equal total rank can correspond to different parameter counts follows directly from Equation 2 and is already reflected in GoRA's dimension-aware budgeting. Although the paper recommends comparing at matched active parameters, the main comparisons are matched by total rank (e.g., online activates 16.4% more parameters on RTE), and most of the 46 paired intervals include zero. Comparisons at matched active parameter count would strengthen the conclusions.

Questions for authors
  1. Does the sign-reversal analysis extend to the scores used by AdaLoRA or GoRA? If not, what evidence suggests that existing allocation signals are unreliable?
  2. How robust are the transfer results across checkpoints, seeds, module types, and longer continuation horizons? Is the RTE checkpoint at step 1,200 in an overfitting regime, given that validation loss increases in both branches?
Reviewer_ydS9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper examines adaptive LoRA rank allocation from two angles. For signal reliability, it introduces a learning-progress score G_i,k = (ℓ_k−1 − ℓ_k) w_i,k / (r_i K) (Eq. 5), which distributes the change in calibration loss across modules in proportion to their squared gradient norms, and shows that a sign change in the shared loss difference reverses Spearman rankings while magnitude rankings persist (Eq. 6, Table 5). For resource accounting, it distinguishes total rank, active parameters, rank-steps and stored parameters (Eqs. 2–3), and reports that one-shot and online allocation at equal total rank activate up to 16.4% more parameters than uniform LoRA on RTE, while AdaLoRA uses about 1.79× the rank-steps. Experiments on DeBERTa-v3-base (eight GLUE tasks) and Llama-3.1-8B with QLoRA (eight commonsense tasks and GSM8K) find differences of a few tenths of a point between uniform, one-shot, online and AdaLoRA allocation. A 2 × 2 study separates GoRA’s allocation from its initialisation and scaling. A rank-transfer intervention at one RTE checkpoint compares moving one rank unit towards or away from the higher-scored module for three module pairs. Synthetic controls show that concentrating rank fixes a local capacity shortage and that repeated reallocation helps after a demand shift.

Critical strengths
  1. The paper reports per-seed results and leave-one-out analyses on multiple tasks.

  2. The motivation of adaptive LoRA is reasonable, and some related works are discussed in the beginning of the manuscript.

Critical weaknesses
  1. The paper is unlike a research paper. The main reason lies in the fact that 1) Tasks are not clear. I don't find any task description in the manuscript. I can't follow the experiments and analysis. 2) Total rank is abstract. Just one sentence, "Total rank sums the active ranks across modules, but each rank unit has a parameter cost determined by the projection dimensions," is presented. I think the reader can't follow it. What are active ranks? 3)

  2. The compared methods are limited. At the beginning of the manuscript, many LoRA methods are introduced, and the proposed methods seem to be general, but no comparison is conducted.

  3. Many of the terms used were not carefully considered. For example, Oralce, arm. These terms are not defined before being introduced; they appear directly.

  4. Supporting conclusions should be highlighted in the main text. Without the help of AI, I would not be able to find evidence to support the paper's arguments.

In summary, the present and the informativeness are so terrible. Furthermore, the soundness looks not good. I suggest the paper should be revised under the professor's supervision.

Questions for authors
  1. WHAT are RTE, MPRC, CoLA, and STS-B? WHAT are SST-2, QNLI, MNLI, and QQP? NO description and citation in the manuscript.

  2. In the Sect. Introduction, the contextual relevance is very weak. Adaptive allocation can help shorten task-specific demand within a limited budget. Then you use the promise to denote the work (not good), and introduce signal reliability and resource accounting as the core. All analysis are surrounded by two points. However, signal reliability and resource accounting are introduced by other works. I don't know what contributions or difficulties you have. I just know allocation is limited. Then you raise another problem. I am very disappointed with your writing. Finally, "Signal Reliability and Resource Accounting" is used as your title, but judging from the introduction, it is not your original content.

  3. Uniform ranks and one-shot, online rank revision should be cited at the first time. Following this, it should be explained why none of these three methods worked. The overall paper has many similar issues.

  4. The main comparison content is mismatched. Only performance improvements were reported. How did the ranking distribution change?

Reviewer_8DpPOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1(−1)
Soundness
3
Read review
Paper summary

This paper investigates adaptive rank allocation in two dimensions, signal reliability and resource accounting. Specifically, signal reliability tackles whether allocation scores indeed identify modules that benefit from higher rank, while resource accounting studies the source of improvement, which one among more capacity, various optimization and better placement. The authors compare uniform, one-shot (gradient-sensitivity), and online (a learning-progress score they define) allocation along with AdaLoRA. Experiments include DeBERTa on GLUE, quantized Llama on commonsense tasks and GSM8K. A checkpoint intervention further separates the immediate effect of transferring rank from the loss change during subsequent training. And main contribution is a evaluation framework for understanding allocation decisions in LoRA.

Critical strengths
  1. Authors highlight an important, but under-explored question: many adaptive-rank papers claim accuracy gains without separating placement from extra capacity, longer rank trajectories, or changed initialization. The paper makes this confound explicit and can compare directly.

  2. It's good to carefully control local interventions. Restoring a common model and optimizer state, matching continuation batches, and transferring rank between equal-cost modules are sensible controls. Separating immediate loss changes from subsequent loss changes reveals behavior that final-loss comparisons alone would miss.

  3. The experiments and reports are solid and transparent. The paper reports results and studies on task benchmarks, component comparison, signal diagnosis, and synthetic controls. Appendices also cover comprehensive details and results.

Critical weaknesses
  1. The main signal analysis does not generalize broadly. The sign-reversal result is specific to the papers constructed learning-progress score, which is defined by multiplying every modules score by the same signed quantity. Equation 6 is therefore a result of rank-correlation invariance. Even though it is useful for diagnosis it does not prove that AdaLoRA or GoRA has a problem. Also having scores over time does not show that a score predicts the extra benefit of more rank. Comparisons with magnitude-only scores and established allocation signals would strengthen the contribution.

  2. Resource accounting is stronger than resource matching. The paper clearly documents budget differences. Most end‑to‑end comparisons still match total rank rather than active parameter cost. In the GoRA component study allocation‑enabled arms use 453k–534k active parameters versus 664k for uniform allocation so the allocation contrast also changes capacity. Similarly the shifting‑demand experiment uses rank eight for uniform. Rank ten for the other methods. The matched online‑versus‑one‑shot comparison remains informative. These mismatches limit conclusions about allocation relative, to uniform baselines.

Questions for authors

For experiments, does controlling total rank have different result as controlling actual parameter counts? It seems many controlling experiment is still controlling rank.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.