acceptodds
Under review as a conference paper at ICLR 2027

CovRA: LoRA Rank Allocation via Conditional Marginal Gain

Abstract

Low-rank adaptation (LoRA) is a widely used method for parameter-efficient fine-tuning, and its effectiveness is shaped by rank allocation and weight initialization. Existing pre-fine-tuning rank-allocation methods typically determine module ranks from importance scores of modules or candidate rank-one updates, but these scores are computed once and not updated as candidates are selected. When different candidates within a module exhibit similar first-order loss-sensitivity patterns across calibration samples, independent scores may count this overlap repeatedly, overstating the additional value of candidates similar to those already selected. We propose Coverage-based Rank Allocation (CovRA). Using a small calibration set on a frozen model, CovRA constructs candidate rank-one updates for each module and takes their per-sample first-order loss sensitivities as responses. It then selects candidates sequentially within each module: at each step, it reevaluates all unselected candidates, treats the part of their responses explained by selected candidates as already covered, and defines the residual response energy as their additional value, yielding a conditional marginal gain. These gains are standardized across modules and used to assign integer ranks under a fixed parameter budget. CovRA additionally reuses the input directions retained during allocation to initialize LoRA. Across two base models and three tasks, CovRA outperforms standard LoRA at the same parameter budget in every setting. On Llama-3.1-8B-Base, it improves GSM8K accuracy over standard LoRA by 5.16 percentage points. In a controlled comparison in the same setting with random initialization and all else fixed, replacing independent valuation with conditional marginal valuation alone improves accuracy by 1.31 points, indicating that updating valuation with the selected set can yield more effective cross-module rank configurations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LoRA, parameter-efficient fine-tuning, rank allocation, conditional marginal gain, calibration gradients, per-sample responses, low-rank adaptation, parameter efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_deyVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CovRA determines module-specific LoRA ranks before fine-tuning. It constructs candidate rank-one updates from calibration gradients on a frozen model and computes their per-sample first-order loss responses. Within each module, candidates are selected sequentially using residual response energy after projection onto the span of previously selected responses. The resulting gains are standardized and aggregated into module demands, which guide integer rank allocation under a parameter budget. SelInit additionally reuses retained input directions for initialization. Experiments on two Llama base models and three tasks demonstrate improvements over standard LoRA at matched parameter budgets. Internal ablations and permutation diagnostics further support the usefulness of conditional valuation and the existence of response redundancy.

Critical strengths

The ablations directly target the central valuation mechanism. Table 3 compares valuation strategies using a shared candidate pool, budget, downstream allocation procedure, and random initialization. CovRA-RI improves GSM8K accuracy by 1.31 percentage points over independent valuation and by 0.99 points over shared rank-position decay, with positive differences across all three paired seeds in Table F3. This provides direct evidence for conditional valuation without attributing the full improvement over LoRA to this component.

Critical weaknesses
  1. The connection between response de-redundancy and final capacity allocation is insufficiently validated. Conditional gains describe uncovered response energy over a finite candidate pool on calibration data. Final ranks additionally depend on within-type standardization, score aggregation, square-root compression, dynamic priorities, and soft quotas, and may exceed the number of scored candidates. Table 3 supports conditional valuation as a useful input to this pipeline, but does not establish whether aggregated demand DmD_m identifies modules that actually benefit more from additional rank. The discussion acknowledges this missing validation. Furthermore, fixed-α, α/r scaling changes module scaling alongside capacity when allocations change. This does not invalidate the practical gains, but limits their interpretation as evidence of improved capacity allocation.

  2. The incremental value over competing methods and the specific initialization contribution are not sufficiently established. Compared with GoRA, the GSM8K and HumanEval improvements are approximately 0.22 and 0.20 percentage points in Table 1, and 0.18 and 0.41 points in Table 2, without a consistent advantage on MT-Bench. Given variability across three runs, these differences do not establish a reliable advantage, and GoRA and CovRA have unmatched budgets on Llama-3.1. The directly relevant EVA baseline is also absent. Meanwhile, SelInit changes input directions, orthogonalization, and norm handling simultaneously relative to random initialization, without a random-subspace control matching the latter two factors. Consequently, the evidence does not sufficiently isolate the value of the specific design from existing data-driven allocation or more general initialization improvements.

Questions for authors
  1. Can the authors validate whether module demand DmD_m predicts the actual benefit of additional rank, or compare against budget-preserving random allocations? This would help establish whether the proxy identifies modules that benefit from additional capacity.

  2. Do the advantages of CovRA-RI over independent valuation and shared decay persist when module scaling coefficients are independent of rank?

  3. How does CovRA compare with GoRA and EVA under exactly matched parameter budgets? Could additional runs provide more precise estimates and uncertainty intervals for its differences against GoRA?

4 comments
AuthorsOpenReview ↗

Response to Reviewer deyV (1/2): Capacity Allocation and Fixed-Scaling Evidence

We thank the reviewer for recognizing that our controlled ablations directly support conditional valuation. We address the concerns about capacity allocation, scaling confounds, baseline fairness, and initialization attribution through four additional analyses, presented in two replies.

The constant-scaling mechanism experiments use five paired seeds (42–46), while the other experiments use three paired seeds (42–44). Results are reported as mean ± sample standard deviation (SD). Confidence intervals (CIs) and tests are computed from unrounded per-seed differences using two-sided paired tt procedures, without correction for multiple comparisons. Here, pp denotes percentage points.

1. Task-Informed Capacity Allocation Outperforms the Tested Random Layouts (W1)

To address the concern about whether the final rank allocation is meaningful, we randomly permute the allocated ranks among modules with identical per-rank parameter costs. We preserve the total parameter budget, rank distribution, random-initialization protocol, and effective scaling, changing only the assignment of capacity to modules.

Experiments are conducted on Llama-3.1-8B-Base/GSM8K with constant effective scaling of 2.

ConfigurationGSM8K (%, mean ± SD)
CovRA-RI72.33 ± 0.42
Randomly shuffled ranks (matched budget)71.29 ± 0.56
Mean paired improvement+1.04 pp
95% paired CI[0.64, 1.43] pp

Each seed is paired with one budget-preserving random rank configuration. CovRA outperforms all three tested random layouts, providing direct evidence that its final capacity allocation contains useful task-specific information.

This experiment validates the effectiveness of the overall allocation layout. It does not establish that the aggregated module demand DmD_m precisely predicts the actual marginal benefit of adding rank to each module.

The allocation procedure includes base ranks, scored-candidate increments, soft-quota capacity supplementation, and exact-budget repair. In the default setting, K=16K=16 limits the number of directly evaluated candidates, while the maximum allowed final rank is 32. Capacity beyond the directly scored candidates is supplied through quota increments and budget repair, without extrapolating additional conditional marginal gains.

2. Conditional-Valuation Gains Persist under Constant Effective Scaling (W1)

To isolate the effect of rank allocation from rank-dependent scaling, we set

\frac{\alpha_m}{r_m}=2,$$ while keeping the actual parameter budget, random initialization, and other training settings fixed. On Llama-3.1-8B-Base/GSM8K, the results over five paired seeds (42–46) are: | Method | GSM8K (%, mean ± SD) | CovRA-RI gain and 95% paired CI (pp) | Two-sided $p$ | |---|---:|---:|---:| | CovRA-I-RI | 71.07 ± 0.58 | **+1.23 [0.89, 1.56]** | **0.00053** | | CovRA-SD-RI | 71.65 ± 0.44 | **+0.65 [0.48, 0.82]** | **0.00045** | | GoRA-rank-RI | 72.02 ± 0.23 | +0.27 [−0.24, 0.78] | 0.2112 | | **CovRA-RI** | **72.30 ± 0.34** | — | — | **Relative to independent valuation and shared rank-position decay, the paired differences are positive for all five seeds, and both 95% paired CIs lie above zero.** These results support the benefit of conditional valuation when effective scaling is held constant. **GoRA rank transfer.** GoRA-rank-RI retains GoRA's rank-generation bounds, $r_{\min}=4$ and $r_{\max}=32$. After budget alignment, it uses the same random initialization, constant effective scaling, and optimization settings as CovRA-RI. CovRA-RI achieves a mean improvement of **0.27 pp**, but the 95% paired CI **[−0.24, 0.78]** includes zero ($p=0.2112$). Therefore, the current evidence does not establish a statistically reliable advantage over GoRA's rank configuration. Together, the rank-shuffling and constant-scaling experiments support the usefulness of CovRA's task-informed capacity allocation, while distinguishing this evidence from a direct quantitative validation of module-level demand estimates.
AuthorsOpenReview ↗

Response to Reviewer deyV (2/2): Matched-Budget Baselines and SelInit Attribution

We further address the reviewer's concerns regarding comparisons with competing methods and the independent contribution of SelInit.

Unless otherwise specified, the following experiments use three paired seeds (42–44). Results are reported as mean ± sample SD, with 95% paired CIs computed from unrounded per-seed differences using the two-sided paired tt procedure, without correction for multiple comparisons.

3. Full-Method Comparisons at Strictly Matched Parameter Budgets (W2)

The original experiments preserved method-specific initialization and optimization configurations to evaluate complete methods, while reporting their actual trainable-parameter budgets. We now additionally conduct strictly budget-matched comparisons and include EVA and GeLoRA.

On Llama-3.1-8B-Base/GSM8K, all methods use exactly 6,815,744 trainable parameters:

MethodGSM8K (%, mean ± SD)95% paired CI for CovRA - baseline (pp)Trainable parameters
CovRA (full)72.93 ± 0.50—6,815,744
GoRA (matched budget)72.43 ± 0.61[0.22, 0.79]6,815,744
EVA (matched budget)72.25 ± 0.65[0.31, 1.06]6,815,744
GeLoRA (matched budget)72.20 ± 0.68[0.26, 1.21]6,815,744

CovRA improves over GoRA, EVA, and GeLoRA by approximately 0.50, 0.68, and 0.73 pp, respectively. All three paired CIs lie above zero, supporting the performance advantage of the full method under this strictly matched-budget setting for the tested model and task.

Budget alignment. Starting from each method's original rank configuration, we apply a deterministic integer adjustment within its permitted rank bounds. The adjustment exactly matches the target trainable-parameter budget while minimizing the total absolute rank deviation from the original configuration.

Ties between optimal adjustments are resolved using a fixed module order. Test-set performance is never used to guide budget alignment.

Experimental fairness. Full-method comparisons preserve the essential method-specific settings, including initialization, optimization, scaling, and GoRA's 16:1 learning-rate ratio for B/AB/A. In contrast, the rank-only comparison in Reply 1 standardizes these training components.

These complementary experiments separately evaluate full-method performance and the contribution of the rank configurations.

4. Norm-Matched Random Orthogonal Control for SelInit (W2)

To isolate the contribution of calibration-selected input directions, we fix the CovRA rank configuration and training conditions, and compare SelInit against a random orthogonal initialization control.

Both use identical effective scaling, orthogonalization procedures, direction-completion rules, and target norms. The only difference is whether the input directions are selected using calibration data.

Both initializations match the expected squared Frobenius norm of standard random initialization at the same rank:

=\frac{r_m}{3}.$$ For a row-orthonormal input basis $Q_m$, both use $$A_{m,0}=\frac{Q_m}{\sqrt{3}}, \qquad B_{m,0}=0.$$ The results over three paired seeds are: | Initialization | GSM8K (%) | HumanEval (%) | |---|---:|---:| | Random orthogonal directions + norm matching | 72.73 ± 0.31 | 47.76 ± 1.86 | | **SelInit** | **72.93 ± 0.50** | **48.98 ± 1.96** | | Mean paired improvement | +0.20 pp | +1.22 pp | | 95% paired CI | [−0.27, 0.68] | [−1.40, 3.84] | SelInit achieves higher mean performance on both tasks. However, both paired CIs include zero. **With the current sample size, these results do not establish a statistically reliable advantage for calibration-selected directions over a matched random orthogonal subspace.** We will therefore clarify the scope of the direction-selection contribution and present SelInit as an optional initialization extension. Importantly, **the rank-shuffling experiment and the constant-scaling comparisons against independent valuation and shared rank-position decay use random initialization throughout**. Thus, the evidence for the core capacity-allocation contribution does not depend on SelInit. We also acknowledge that the paired CI for the GoRA rank-transfer comparison includes zero, and we will interpret that result within its supported statistical scope.

Thanks for your detailed response, which resolve most of my concerns. I would like to maintain my score.

Dear Reviewer deyV,

Thank you for taking the time to read our response and for your constructive feedback.

Best regards,
The Authors

Reviewer_X4ctOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the problem of evaluating which parts of a model are most important to update under a fixed budget. To resolve this, the authors propose CovRA. CovRA uses a sequential selection process via a calibration dataset. At each step, it measures the "Conditional Marginal Gain" of a candidate by asking how much new information this update provides beyond what has already been covered by previously selected ones, using multiple chained formulations starting from two SVD gradient decompositions derived from calibration sample groups. After the chained computation, CovRA introduces a specialized initialization that reuses the directions identified during the allocation process to give the model a better starting point for training.

Critical strengths
  1. The use of conditional valuation is a logically sound approach to maximizing the diversity of the learned updates.
  2. The method is a framework that integrates gradient-based analysis, rank distribution, and weight initialization.
  3. CovRA provides comprehensive experimental results and shows performance improvement over its baseline methods.
Critical weaknesses
  1. The process of sequential re-evaluation and orthogonal projection is more computationally demanding than standard allocation methods. This introduces additional preprocessing time before the actual fine-tuning can begin.
  2. Since the allocation is determined entirely before training based on a small sample set, the method’s effectiveness is highly dependent on how well those samples represent the overall task.
  3. CovRA appears over-engineered for "response overlap," while the proposed conditional marginal gain is theoretically sophisticated, ablation studies show it only provides a minor boost over independent valuation. This suggests that the core problem CovRA aims to solve may not be a primary bottleneck for LoRA performance, making the complex design less justifiable.
Questions for authors
  1. The evaluation is currently limited to two Llama-family base models. It remains unclear whether CovRA's performance gains generalize to modern instruction-tuned models or non-Llama architectures.
  2. The calibration phase introduces non-negligible overhead due to gradient computation and SVD operations. A more detailed cost-benefit analysis is needed to determine whether the performance improvement justifies the additional wall-clock time and memory consumption incurred before fine-tuning begins.
  3. At very high total parameter budgets, the performance gap between methods seems to narrow. Is there a point where the benefits of smart allocation are outweighed by the sheer capacity of the model?
  4. How would CovRA handle models with different structural properties, such as MoE models, where the "importance" of a module might change dynamically during inference?
5 comments
AuthorsOpenReview ↗

Response to Reviewer X4ct (1/3): Controlled Gains and Computational Complexity

We thank the reviewer for recognizing the rationale behind conditional valuation and the experimental results. We address the concerns about the benefits and computational costs of conditional valuation, calibration robustness, and generalization in three separate responses.

1. Benefits and Computational Complexity of Conditional Valuation (W1, W3, Q2)

The benefits of conditional valuation persist after controlling for rank-dependent scaling. Table 4 shows that actual response redundancy exceeds the matched-permutation control in 125 of 128 modules. We further fix the effective scaling at

αm=2rm,αmrm=2,\alpha_m = 2r_m, \qquad \frac{\alpha_m}{r_m} = 2,

while keeping the parameter budget, random-initialization protocol, and training settings unchanged. The results over five paired seeds (42–46) are reported as mean ± sample standard deviation (SD):

MethodGSM8K accuracy (%)CovRA-RI gain and 95% paired CI (pp)Two-sided pp
CovRA-I-RI71.07 ± 0.58+1.23 [0.89, 1.56]0.00053
CovRA-SD-RI71.65 ± 0.44+0.65 [0.48, 0.82]0.00045
CovRA-RI72.30 ± 0.34——

Paired statistics are computed from unrounded per-seed scores using two-sided paired tt tests and tt-distribution confidence intervals (CIs), without correction for multiple comparisons. Compared with independent valuation and shared rank-position decay, all five paired differences are positive, and both 95% paired CIs lie above zero. This supports the benefit of conditional valuation under constant effective scaling.

For the Uniform control, we use the three shared seeds (42–44). Uniform and the corresponding CovRA-RI runs achieve 67.78 ± 1.19% and 72.33 ± 0.42%, respectively, yielding a mean paired gain of 4.55 percentage points (95% CI [1.61, 7.48]).

Computational Complexity

Let MM, NN, and KK denote the number of modules, calibration samples, and candidates per module, respectively. After separating positive and negative responses, the response matrix has size 2N×K2N \times K.

Independent valuation has time complexity O(MNK)O(MNK). At selection step jj, projecting the remaining candidates onto the selected orthonormal basis costs

O(Nj(K−j))O\left(Nj(K-j)\right)

per module. Therefore, a conservative upper bound for conditional valuation is

\qquad S_{\mathrm{cond}} = O(MNK).$$ The projection is implemented using an orthonormal basis without explicitly constructing a $2N \times 2N$ projection matrix. Incremental residual updates can reduce the time complexity to $O(MNK^2)$. Including other preprocessing stages, the total computational costs are $$T_{\mathrm{pre}} = C_{\mathrm{cand}} + O(MNK^3) + C_{\mathrm{alloc}} + C_{\mathrm{init}},$$ $$S_{\mathrm{aux}} = O(U + MNK).$$ Here, $C_{\mathrm{cand}}$ includes calibration gradients, sketch/SVD, and response construction; $C_{\mathrm{alloc}}$ covers integer allocation and budget repair; $C_{\mathrm{init}}$ is the initialization cost; and $U$ denotes auxiliary storage used by other preprocessing stages. The auxiliary-space bound excludes the frozen model itself. For $P \leq MK$ scored candidates and $R$ quota increments, the direct-scan allocator has complexity $$C_{\mathrm{alloc}} = O\left(P\log P + P^2 + MR\right) + C_{\mathrm{repair}},$$ where bounded cost-state search for budget repair has pseudopolynomial complexity. **Conditional valuation reuses the constructed responses and requires no additional model forward/backward passes or parameter-space SVDs.** Under the default setting, $$M=128,\qquad N=256,\qquad K=16,$$ the sign-separated FP32 response arrays occupy only **4 MiB**, excluding other workspace.
AuthorsOpenReview ↗

Response to Reviewer X4ct (2/3): Measured Overhead and Calibration Robustness

1. Measured Runtime and Memory Costs (W1, W3, Q2)

We supplement the complexity analysis in Reply 1 with empirical resource measurements under the hardware and software environment described in Appendix E and Table H4. All methods use a consistent training workload and measurement protocol; reported times exclude evaluation.

The reference configuration is uniform rank rref=8r_{\mathrm{ref}}=8, corresponding to 6,815,744 trainable parameters. Actual parameter budgets and scaling configurations follow the corresponding performance experiments. The controlled valuation experiments use constant effective scaling of 2, whereas full-method comparisons retain each method's native scaling, initialization, and optimization settings.

MethodPreprocessing + training time (s)End-to-end peak GPU memory (GiB)
LoRA9,50049.6
GoRA10,73061.2
EVA12,60860.8
GeLoRA11,04051.4
CovRA-I-RI9,87651.7
CovRA-RI10,27551.9
CovRA (full)10,57652.1

Times are averaged over three runs. End-to-end peak GPU memory covers both preprocessing and training: we record the maximum GPU memory usage within each run and then average these maxima across runs.

Compared with LoRA, full CovRA increases preprocessing-plus-training time by 11.3% and peak GPU memory by 2.5 GiB, while improving GSM8K accuracy by 5.16 percentage points under the tested configuration. This quantifies the performance-resource trade-off. Table H4 further separates 1,011 s of preprocessing from 9,565 s of training.

The total runtime difference between CovRA-RI and CovRA-I-RI is 399 s (approximately 4.0%). This difference includes training time and therefore cannot be attributed entirely to the conditional-valuation routine.

Appendix H.1 also reports a lower-cost configuration. Reducing KK from 16 to 8 decreases allocation time from 985 s to 781 s (20.7%), while mean GSM8K accuracy changes only from 72.93% to 72.91%. Thus, the candidate-pool size provides a practical way to trade computation for performance while retaining comparable average accuracy.

2. Robustness to Independently Resampled Calibration Subsets (W2)

We independently sample eight calibration subsets of 256 examples each from the training set and compare independent and conditional valuation under matched settings.

For each subset, we conduct multiple training runs, first averaging the accuracy of each method within that subset and then computing a paired difference. Thus, the eight calibration subsets are the independent statistical units; repeated training runs within a subset are not counted as additional independent observations.

MetricResult
Independent calibration subsets8
Samples per subset256
CovRA-I-RI GSM8K accuracy (%)71.3 ± 0.8
CovRA-RI GSM8K accuracy (%)72.2 ± 0.7
Mean paired improvement+0.9 pp
95% paired CI[0.2, 1.6] pp
Cross-subset rank Spearman correlation0.84 ± 0.05

Accuracy means and sample SDs are calculated across the eight subset-level mean scores.

Let aˉsRI\bar a_s^{\mathrm{RI}} and aˉsI\bar a_s^{\mathrm{I}} denote the within-subset mean accuracies of CovRA-RI and CovRA-I-RI for calibration subset ss. Their paired difference is

\bar a_s^{\mathrm{I}}, \qquad s=1,\ldots,8.$$ The mean paired improvement is $$\bar d = \frac{1}{8}\sum_{s=1}^{8}d_s.$$ The 95% CI is calculated from the eight subset-level differences as $$\bar d \pm t_{0.975,7}\frac{s_d}{\sqrt{8}},$$ where $s_d$ is the sample SD of the paired differences. Rank correlations are reported separately as descriptive statistics. **Conditional valuation retains a positive mean gain of 0.9 pp across independently resampled calibration subsets, with a 95% CI of [0.2, 1.6].** The rank allocations also exhibit relatively high cross-subset correlation. These results support robustness to the tested calibration-sampling variations within the training distribution.
AuthorsOpenReview ↗

Response to Reviewer X4ct (3/3): Generalization, Budget Scaling, and Applicability

3. Generalization Beyond the Llama Model Family (Q1)

We evaluate full CovRA against LoRA and GoRA on Qwen2.5-7B-Base, a non-Llama model architecture. All methods adapt the q/k/v/o projections across all 28 layers, counting only LoRA A/B matrices. The actual number of trainable parameters is strictly matched at 5,046,272 for all methods.

The methods share training data, training steps, and evaluation protocols, while retaining their respective initialization and optimization configurations.

TaskLoRA (%)GoRA (%)CovRA (full, %)
GSM8K72.10 ± 0.6273.16 ± 0.5375.89 ± 0.79
HumanEval59.96 ± 0.7063.01 ± 0.7064.43 ± 0.70

Results are reported as mean ± sample SD over three seeds.

Under strictly matched trainable-parameter budgets, CovRA outperforms GoRA by 2.73 percentage points on GSM8K and 1.42 percentage points on HumanEval. These results support the applicability of the full CovRA method beyond the Llama model family.

4. Diminishing Gains at Higher Parameter Budgets (Q3)

We further evaluate Llama-3.1-8B-Base on GSM8K at three parameter-budget levels. The actual number of trainable parameters is strictly matched among methods within each budget level.

Reference rankParameter budgetLoRA (%)GoRA (%)CovRA (%)
86,815,74467.78 ± 1.1972.43 ± 0.6172.93 ± 0.50
3227,262,97673.16 ± 0.4774.75 ± 0.5075.66 ± 0.38
128109,051,90473.69 ± 0.3675.08 ± 0.4475.82 ± 0.38

CovRA's mean improvements over LoRA are approximately 5.16, 2.50, and 2.13 percentage points at reference ranks 8, 32, and 128, respectively, indicating an overall narrowing trend.

Its advantages over GoRA are 0.50, 0.91, and 0.74 percentage points, respectively. The advantage at the highest budget is smaller than at reference rank 32.

Furthermore, increasing the reference rank from 8 to 32 improves CovRA's accuracy by 2.73 percentage points, whereas increasing it from 32 to 128 yields only 0.16 percentage points.

These results suggest diminishing absolute performance gains from additional capacity within the tested model, task, and budget range. However, they do not establish a precise threshold at which the benefits of intelligent rank allocation disappear.

5. Compatibility with Dynamic Expert Routing in MoE Models (Q4)

Adapter rank determines the trainable capacity of an expert, whereas dynamic routing determines which experts are activated for each input. Consequently, input-dependent routing during inference does not inherently require the allocated adapter ranks to change for every input.

A possible extension of CovRA to mixture-of-experts (MoE) architectures would construct expert-specific responses using the actual routing of calibration tokens and account for expert coverage during calibration sampling.

We emphasize that the present empirical findings are limited to the evaluated dense models; MoE applicability remains a direction for future investigation.

6. Improving Presentation and Clarity

We will reorganize the methodological explanation around three essential steps:

Construct per-sample responses → Measure residual response energy → Allocate ranks under a parameter budget.

SelInit will be introduced separately as an optional initialization extension. Numerical implementation details and exact-budget repair will be moved to the appendix to keep the main methodological contribution clear and focused.

Reviewer_ahmPOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

(All my reviews are rewritten)

This paper proposes CovRA. It sets LoRA ranks before fine-tuning under a fixed budget. It builds rank-one candidates from calibration gradients and picks them by conditional marginal gain. The gains are turned into integer ranks that fit the budget. Experiments use two Llama models and three tasks, with three runs each.

Critical strengths

S1. The controlled comparison is well designed and honestly reported. Only the valuation rule changes. CovRA-RI beats each control on GSM8K in all three seeds. LoRA and CovRA use exactly the same budget.

Critical weaknesses

W1. Lack of novelty. Also, the gain over the closest methods is not shown.

My main concerns include: 1) The selection rule is classical. Appendix B says it has the same form as pivoted QR, and SubLoRA also values candidates this way; 2) Against GoRA, the main baseline, CovRA gains only 0.18 to 0.41 points. By my Welch tests, none of these gaps is clear; 3) Independent valuation is the authors' own variant. GoRA uses a different setup. No experiment compares rank choices in the same setup; 4) EVA and GeLoRA are named but not tested.

W2. The evidence that conditional valuation causes the gains is weak and mixed with other factors.

My main concerns include: 1) The 1.31-point GSM8K gain over CovRA-I-RI has a paired p of about 0.08; 2) For HumanEval, there are no per-seed values, and the gap gives p ≈ 0.39. All controlled results are on Llama-3.1 only; 3) CovRA uses scaling α/rm\alpha/r_m, so changing the ranks also changes the update scale. This may affect results; 4) The rank allocation rule is not fully described. Ranks reach 20, above K=16K = 16, and it is not clear how..

Questions for authors

Questions

Q1. How do ranks above KK happen? Please give full pseudocode and say how many rank units come from each step.

Q2. What happens if you put GoRA's ranks into the rest of CovRA? Please also add EVA and GeLoRA with the same budget.

Q3. Do the controlled gains hold with more seeds, per-seed HumanEval results and Llama-2?

Q4. Does the gain stay with scaling that does not depend on rank?


Suggestions:

Suggestion 1 How did you choose N=256N = 256? No validation split is described.

Suggestion 2 Please add LoRA and GoRA at rref=32r_{ref} = 32, and give the constants in Eqs. 8-11.


References to correct:

Correction 1. "Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations, 2023b." The ICLR 2023 version has seven authors. Nikos Karampatziakis is not one of them.

This is an example, not a full list. Please check the whole bibliography.

8 comments
AuthorsOpenReview ↗

Response to Reviewer ahmP (1/4): Novelty and Controlled Evidence

We thank the reviewer for recognizing the controlled design and transparent budgets. In four comments, we report fixed-scaling ablations, GoRA rank transfer, matched-budget baselines, additional HumanEval and Llama-2 experiments, and the complete allocation procedure.

Extended mechanism experiments use paired seeds 42–46; budget-matched full-method experiments use 42–44 unless specified. Results are mean ± sample SD. Paired CIs and two-sided paired tt tests use unrounded per-seed scores and are not adjusted for multiple comparisons. pp denotes percentage points.

1. Contribution: Capacity Valuation Based on Per-Sample Response Dependencies (W1)

Before adapter fine-tuning, CovRA conditionally values candidate updates using their per-sample first-order loss responses. It aggregates the resulting gains into module demands and allocates ranks subject to a parameter budget. Even when two candidates have similar individual response energies and orthogonal directions in parameter space, their per-sample loss responses can overlap substantially. Conditional valuation measures newly covered responses after subtracting the component explained by previously selected responses.

Distinction from SubLoRA. The two methods differ in both what they value and their objectives. SubLoRA evaluates singular-value components of an already trained LoRA using a second-order loss surrogate that accounts for component interactions. CovRA instead constructs candidate updates on a frozen model and uses them as capacity probes: it measures additional within-module coverage through per-sample first-order loss responses, and then aggregates the results into module demands.

This objective motivates three design choices: sign-aware encoding distinguishes positive from negative responses; subspace projection removes the part explained by selected responses; and residual energy quantifies additional coverage. The connections to QR decomposition, Schur complements, and, under appropriate conditions, log-determinants characterize the underlying mathematics. The coverage information is then transformed into integer ranks through standardization, demand aggregation, and parameter-cost constraints.

Empirical evidence. Table 4 uses a matched-permutation null that preserves each candidate's response distribution and energy while disrupting sample-wise correspondence across candidates. Actual response redundancy exceeds this null in 125 of 128 modules. The controlled experiments below further test the downstream value of this signal.

2. Gains from Conditional Valuation Persist under Constant Effective Scaling (W2, Q4)

On Llama-3.1-8B-Base/GSM8K, we use

αm=2rm,αmrm=2,\alpha_m = 2r_m, \qquad \frac{\alpha_m}{r_m} = 2,

while holding the actual parameter budget, random-initialization protocol, and other optimization settings fixed. Results over five paired seeds (42–46) are as follows.

MethodGSM8K (%, mean ± SD)
CovRA-I-RI71.07 ± 0.58
CovRA-SD-RI71.65 ± 0.44
GoRA-rank-RI72.02 ± 0.23
CovRA-RI72.30 ± 0.34
ComparisonMean gain (pp)95% paired CI (pp)Two-sided pp
CovRA-RI - CovRA-I-RI+1.23[0.89, 1.56]0.00053
CovRA-RI - CovRA-SD-RI+0.65[0.48, 0.82]0.00045
CovRA-RI - GoRA-rank-RI+0.27[−0.24, 0.78]0.2112

Compared with independent valuation and shared rank-position decay, the paired differences are positive for all five seeds, and both 95% paired CIs lie above zero. This supports the benefit of conditional valuation when effective scaling is held constant.

For the Uniform control, we use the three shared seeds (42–44). Uniform and the corresponding CovRA-RI runs achieve 67.78 ± 1.19% and 72.33 ± 0.42%, respectively, yielding a mean paired gain of 4.55 pp (95% CI [1.61, 7.48]). Thus, the improvement over uniform allocation cannot be explained solely by rank-dependent scaling differences.

Supplementary record: Per-Seed Results under Constant Effective Scaling

Each GSM8K evaluation contains 1,319 test questions; entries are the numbers answered correctly.

SeedCovRA-I-RICovRA-SD-RIGoRA-rank-RICovRA-RI
42929937952948
43938949954955
44949951949959
45932941946950
46939947949956

In the original GSM8K experiment with fixed α\alpha, the paired comparison against independent valuation yielded p≈0.080p \approx 0.080. The table above reports the new experiments with constant effective scaling.

AuthorsOpenReview ↗

Response to Reviewer ahmP (2/4): Rank Transfer and Matched-Budget Comparisons

3. Transferring GoRA Ranks and Comparing Full Methods at Exactly Matched Budgets (W1, Q2)

GoRA rank transfer. GoRA-rank-RI uses the rank configuration produced by GoRA, retaining its allocation bounds rmin⁡=4r_{\min}=4 and rmax⁡=32r_{\max}=32 during rank generation. After budget alignment, it shares CovRA-RI's random initialization, constant effective scaling, and optimization settings. As reported in Reply 1, CovRA-RI is higher by 0.27 pp on average (95% CI [−0.24, 0.78], p=0.2112p=0.2112). This difference is not statistically significant and does not establish a reliable advantage over GoRA's rank configuration.

Full-method comparison. In the original paper, GoRA uses 102.70% of the reference parameter budget. We now strictly match all methods to 6,815,744 actual trainable parameters and add EVA and GeLoRA. The following results are from three paired seeds on Llama-3.1-8B-Base/GSM8K.

MethodGSM8K (%, mean ± SD)95% paired CI for CovRA - baseline (pp)Trainable parameters
CovRA (full)72.93 ± 0.50-6,815,744
GoRA (matched budget)72.43 ± 0.61[0.22, 0.79]6,815,744
EVA (matched budget)72.25 ± 0.65[0.31, 1.06]6,815,744
GeLoRA (matched budget)72.20 ± 0.68[0.26, 1.21]6,815,744

CovRA improves over GoRA, EVA, and GeLoRA by approximately 0.50, 0.68, and 0.73 pp, respectively. All three paired CIs lie above zero, supporting the performance advantage of the full method under this strictly matched-budget setting for the tested model and task.

The full-method comparison preserves method-specific initialization, optimization, and scaling configurations, including GoRA's 16:1 learning-rate ratio for B/AB/A. In contrast, the rank-only transfer control in Reply 1 standardizes these training components. The two comparisons therefore address distinct questions: the performance of full methods and the effect of their rank configurations. We will distinguish internal ablation findings from comparisons with published methods in the abstract and conclusion.

Supplementary detail: Exact-Budget Alignment

Starting from each method's original rank configuration r^\hat{\mathbf r}, we apply a deterministic, minimum-change integer adjustment within that method's permitted rank bounds:

\min_{\mathbf r\in\mathbb Z^M}\quad &\sum_{m=1}^{M}|r_m-\hat r_m|\\\\ \text{s.t.}\quad &\sum_{m=1}^{M}r_m(d_{m,\mathrm{in}}+d_{m,\mathrm{out}}) =6{,}815{,}744,\\\\ &r_{m,\min}\le r_m\le r_{m,\max}. \end{aligned}$$ Ties between optimal solutions are resolved using a fixed module order; test-set scores are not used in the adjustment. **Full-method validation on Qwen.** We evaluate Qwen2.5-7B-Base using three paired seeds. Results are mean ± sample SD. | Task | LoRA (%) | GoRA (%) | CovRA (full, %) | |---|---:|---:|---:| | GSM8K | 72.10 ± 0.62 | 73.16 ± 0.53 | **75.89 ± 0.79** | | HumanEval | 59.96 ± 0.70 | 63.01 ± 0.70 | **64.43 ± 0.70** | CovRA outperforms GoRA by **2.73 pp on GSM8K and 1.42 pp on HumanEval**, indicating that the benefit of the full method is not confined to the Llama family. All methods adapt `q_proj`, `k_proj`, `v_proj`, and `o_proj` in all 28 layers, using uniform rank 8 as the reference. Counting only LoRA A/B matrices, all methods have **5,046,272 trainable parameters**. **Higher budgets.** At $r_{\mathrm{ref}}=32$ with an actual budget of **27,262,976** trainable parameters, the accuracies of LoRA, GoRA, and CovRA on Llama-3.1-8B-Base/GSM8K are **73.16 ± 0.47%, 74.75 ± 0.50%, and 75.66 ± 0.38%**, respectively. At four times the default reference budget, CovRA still exceeds GoRA by **0.91 pp**.
AuthorsOpenReview ↗

Response to Reviewer ahmP (3/4): Additional Seeds and Robustness

4. Additional Seeds, HumanEval, and Llama-2 Validation (W2, Q3)

HumanEval. We extend the original valuation-ablation protocol to five seeds (42–46). For each seed, one completion is generated for each of the 164 problems.

MethodHumanEval pass@1 (%, mean ± SD)
CovRA-I-RI45.61 ± 0.80
CovRA-SD-RI46.34 ± 1.14
CovRA-RI47.32 ± 1.40
ComparisonMean gain (pp)95% paired CI (pp)Two-sided p-value
CovRA-RI - CovRA-I-RI+1.71[0.46, 2.95]0.0189
CovRA-RI - CovRA-SD-RI+0.98[−1.68, 3.63]0.3653

These expanded results support a gain over independent valuation on HumanEval, whereas the CI for the comparison with shared rank-position decay includes zero.

Llama-2. On Llama-2-7B-Base/GSM8K, we use five paired seeds (42–46), random initialization, and constant effective scaling αm/rm=2\alpha_m/r_m=2, keeping all other training settings fixed. We adapt q/k/v/o/up/gate/down across all 32 layers (224 modules). The rank-8 reference budget calculated from this configuration is 19,988,480 trainable parameters.

MethodGSM8K (%, mean ± SD)
CovRA-I-RI52.66 ± 0.64
CovRA-SD-RI53.15 ± 0.51
CovRA-RI53.68 ± 0.38
ComparisonMean gain (pp)95% paired CI (pp)Two-sided p-value
CovRA-RI - CovRA-I-RI+1.02[0.12, 1.92]0.0350
CovRA-RI - CovRA-SD-RI+0.53[−0.14, 1.20]0.0924

The Llama-2 results support the benefit of conditional valuation over independent valuation under constant effective scaling. The mean gain over shared rank-position decay is positive, but its CI includes zero.

Supplementary Record: Per-Seed HumanEval and Llama-2 Results

HumanEval: numbers of passing problems out of 164.

SeedCovRA-I-RICovRA-SD-RICovRA-RI
42737674
43767380
44767877
45757779
46747678

Llama-2/GSM8K: numbers of correct answers out of 1,319.

SeedCovRA-I-RICovRA-SD-RICovRA-RI
42690697708
43704705711
44695707714
45683691706
46701705701

Both additional experiments support the gain of conditional over independent valuation, indicating that this mechanism is not limited to a single task or model in the tested settings.

Supplementary Validation: Calibration Robustness and Initialization Details

Independent calibration subsets. Eight resampled calibration subsets, each containing 256 samples, support robustness to calibration sampling changes within the training distribution. Relative to independent valuation, conditional valuation gives a mean improvement of 0.9 pp (95% CI [0.2, 1.6]).

Calibration size. The default N=256N=256 was set in advance based on the preprocessing-compute budget. Calibration samples come from the training set, and evaluation scores were not used for this choice. Appendix Table H2 is a subsequent sensitivity analysis: N=256N=256 and N=512N=512 achieve 72.93 ± 0.50% and 72.78 ± 0.40%, respectively, with allocation times of 985 ± 1 s and 1,656 ± 4 s. Thus, N=256N=256 provides a favorable performance-cost trade-off in the tested settings.

Numerical constants and initialization. The constants and initialization targets are:

Symbol / parameterValueUsed for
εδ\varepsilon_\delta10−1210^{-12}Log transform of marginal gains
εMAD\varepsilon_{\mathrm{MAD}}10−1210^{-12}Within-type MAD standardization
εD\varepsilon_D10−1210^{-12}Square-root compression of module demand
ϵ\epsilon10−1210^{-12}Denominator protection for quota progress and shortfall
β\beta0.5Cost exponent in dynamic priority
Direction validity / orthogonalization threshold10−1210^{-12}Valid-direction checks and residual-direction normalization

Initialization matches the expected squared Frobenius norm of random initialization at the same rank. Each entry of the random initialization matrix is sampled independently from a uniform distribution over [−1/√d, +1/√d], where d is the input dimension of the corresponding module. The target squared Frobenius norm is therefore

Tm2=E[∥Amrand∥F2]=rm3.T_m^2 = \mathbb{E}\left[\|A_m^{\mathrm{rand}}\|_F^2\right] = \frac{r_m}{3}.

For a row-orthonormal input basis QmQ_m, we use the per-row scale

σminit=Tmrm=13.\sigma_m^{\mathrm{init}} = \frac{T_m}{\sqrt{r_m}} = \frac{1}{\sqrt{3}}.

The initialized matrices are

Am,0=σminitQm,Bm,0=0.A_{m,0} = \sigma_m^{\mathrm{init}} Q_m, \qquad B_{m,0} = 0.

The target total Frobenius norm scales with rm\sqrt{r_m}, and both selected directions and randomly completed directions use the same per-row scale.

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their rebuttals. As the author-reviewer discussion period is coming to a close soon, please read their responses and engage in the discussion. Thank you for your time and effort.

Best,

AC

AuthorsOpenReview ↗

Summary of Revisions and Thanks to the Reviewers and Area Chair

Dear Area Chair and Reviewers,

We thank the Area Chair for coordinating the review and discussion, and all three reviewers for carefully considering the paper, our additional experiments, and our responses. We have incorporated the main clarifications and results into the revised manuscript, with the principal changes marked in blue. The discussion centered on two related questions.

How does CovRA decide which modules receive more rank under a fixed parameter budget? Section 3.4 now explains how CovRA estimates each module’s capacity demand from the response energy that its candidates contribute beyond the span of the responses already selected in that module. Appendix C provides pseudocode and allocation records from individual runs. These records show how many rank units come from scored candidates and how quota-based increments can raise a module’s final rank above the number of candidates directly scored.

How much of the observed gain can be attributed to the valuation rule itself? In Section 4.2, we match the actual number of trainable parameters across CovRA, GoRA, EVA, and GeLoRA while retaining each method’s own training configuration. On Llama-3.1-8B-Base/GSM8K, CovRA’s mean accuracy exceeds that of the three baselines by 0.50–0.73 percentage points. Section 4.3 examines the valuation rule under a common random initialization protocol and fixed effective scaling. Across five paired seeds, conditional valuation improves mean accuracy by 1.23 percentage points over independent valuation and 0.65 points over the shared rank-position decay control. We also report controls that transfer GoRA’s rank configuration, shuffle ranks among equal-cost modules, and compare SelInit with norm-matched random orthogonal initialization. The appendix provides per-seed results and paired confidence intervals.

The reviewers’ follow-up comments helped us state these findings more precisely. Reviewer X4ct recognized the rationale for evaluating each candidate in light of the updates already selected and indicated that our responses had addressed their concerns and that they would raise their rating to 3. Reviewer deyV noted that the controlled ablations provide direct evidence for conditional valuation and indicated that most of their concerns had been resolved, maintaining their rating of 3. Reviewer ahmP recognized the design and reporting of the controlled comparisons and, after reconsidering the paper, indicated that our responses had addressed some of their concerns and raised their rating from 2 to 3.

We thank the Area Chair and all three reviewers again for their time and care. Their questions about the source of the gains helped us distinguish more clearly between the performance of CovRA as a complete method and the contribution supported by each controlled comparison.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.