- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CovRA determines module-specific LoRA ranks before fine-tuning. It constructs candidate rank-one updates from calibration gradients on a frozen model and computes their per-sample first-order loss responses. Within each module, candidates are selected sequentially using residual response energy after projection onto the span of previously selected responses. The resulting gains are standardized and aggregated into module demands, which guide integer rank allocation under a parameter budget. SelInit additionally reuses retained input directions for initialization. Experiments on two Llama base models and three tasks demonstrate improvements over standard LoRA at matched parameter budgets. Internal ablations and permutation diagnostics further support the usefulness of conditional valuation and the existence of response redundancy.
Critical strengths
The ablations directly target the central valuation mechanism. Table 3 compares valuation strategies using a shared candidate pool, budget, downstream allocation procedure, and random initialization. CovRA-RI improves GSM8K accuracy by 1.31 percentage points over independent valuation and by 0.99 points over shared rank-position decay, with positive differences across all three paired seeds in Table F3. This provides direct evidence for conditional valuation without attributing the full improvement over LoRA to this component.
Critical weaknesses
-
The connection between response de-redundancy and final capacity allocation is insufficiently validated. Conditional gains describe uncovered response energy over a finite candidate pool on calibration data. Final ranks additionally depend on within-type standardization, score aggregation, square-root compression, dynamic priorities, and soft quotas, and may exceed the number of scored candidates. Table 3 supports conditional valuation as a useful input to this pipeline, but does not establish whether aggregated demand identifies modules that actually benefit more from additional rank. The discussion acknowledges this missing validation. Furthermore, fixed-α, α/r scaling changes module scaling alongside capacity when allocations change. This does not invalidate the practical gains, but limits their interpretation as evidence of improved capacity allocation.
-
The incremental value over competing methods and the specific initialization contribution are not sufficiently established. Compared with GoRA, the GSM8K and HumanEval improvements are approximately 0.22 and 0.20 percentage points in Table 1, and 0.18 and 0.41 points in Table 2, without a consistent advantage on MT-Bench. Given variability across three runs, these differences do not establish a reliable advantage, and GoRA and CovRA have unmatched budgets on Llama-3.1. The directly relevant EVA baseline is also absent. Meanwhile, SelInit changes input directions, orthogonalization, and norm handling simultaneously relative to random initialization, without a random-subspace control matching the latter two factors. Consequently, the evidence does not sufficiently isolate the value of the specific design from existing data-driven allocation or more general initialization improvements.
Questions for authors
-
Can the authors validate whether module demand predicts the actual benefit of additional rank, or compare against budget-preserving random allocations? This would help establish whether the proxy identifies modules that benefit from additional capacity.
-
Do the advantages of CovRA-RI over independent valuation and shared decay persist when module scaling coefficients are independent of rank?
-
How does CovRA compare with GoRA and EVA under exactly matched parameter budgets? Could additional runs provide more precise estimates and uncertainty intervals for its differences against GoRA?
4 comments
Response to Reviewer deyV (2/2): Matched-Budget Baselines and SelInit Attribution
We further address the reviewer's concerns regarding comparisons with competing methods and the independent contribution of SelInit.
Unless otherwise specified, the following experiments use three paired seeds (42–44). Results are reported as mean ± sample SD, with 95% paired CIs computed from unrounded per-seed differences using the two-sided paired procedure, without correction for multiple comparisons.
3. Full-Method Comparisons at Strictly Matched Parameter Budgets (W2)
The original experiments preserved method-specific initialization and optimization configurations to evaluate complete methods, while reporting their actual trainable-parameter budgets. We now additionally conduct strictly budget-matched comparisons and include EVA and GeLoRA.
On Llama-3.1-8B-Base/GSM8K, all methods use exactly 6,815,744 trainable parameters:
| Method | GSM8K (%, mean ± SD) | 95% paired CI for CovRA - baseline (pp) | Trainable parameters |
|---|---|---|---|
| CovRA (full) | 72.93 ± 0.50 | — | 6,815,744 |
| GoRA (matched budget) | 72.43 ± 0.61 | [0.22, 0.79] | 6,815,744 |
| EVA (matched budget) | 72.25 ± 0.65 | [0.31, 1.06] | 6,815,744 |
| GeLoRA (matched budget) | 72.20 ± 0.68 | [0.26, 1.21] | 6,815,744 |
CovRA improves over GoRA, EVA, and GeLoRA by approximately 0.50, 0.68, and 0.73 pp, respectively. All three paired CIs lie above zero, supporting the performance advantage of the full method under this strictly matched-budget setting for the tested model and task.
Budget alignment. Starting from each method's original rank configuration, we apply a deterministic integer adjustment within its permitted rank bounds. The adjustment exactly matches the target trainable-parameter budget while minimizing the total absolute rank deviation from the original configuration.
Ties between optimal adjustments are resolved using a fixed module order. Test-set performance is never used to guide budget alignment.
Experimental fairness. Full-method comparisons preserve the essential method-specific settings, including initialization, optimization, scaling, and GoRA's 16:1 learning-rate ratio for . In contrast, the rank-only comparison in Reply 1 standardizes these training components.
These complementary experiments separately evaluate full-method performance and the contribution of the rank configurations.
4. Norm-Matched Random Orthogonal Control for SelInit (W2)
To isolate the contribution of calibration-selected input directions, we fix the CovRA rank configuration and training conditions, and compare SelInit against a random orthogonal initialization control.
Both use identical effective scaling, orthogonalization procedures, direction-completion rules, and target norms. The only difference is whether the input directions are selected using calibration data.
Both initializations match the expected squared Frobenius norm of standard random initialization at the same rank:
=\frac{r_m}{3}.$$ For a row-orthonormal input basis $Q_m$, both use $$A_{m,0}=\frac{Q_m}{\sqrt{3}}, \qquad B_{m,0}=0.$$ The results over three paired seeds are: | Initialization | GSM8K (%) | HumanEval (%) | |---|---:|---:| | Random orthogonal directions + norm matching | 72.73 ± 0.31 | 47.76 ± 1.86 | | **SelInit** | **72.93 ± 0.50** | **48.98 ± 1.96** | | Mean paired improvement | +0.20 pp | +1.22 pp | | 95% paired CI | [−0.27, 0.68] | [−1.40, 3.84] | SelInit achieves higher mean performance on both tasks. However, both paired CIs include zero. **With the current sample size, these results do not establish a statistically reliable advantage for calibration-selected directions over a matched random orthogonal subspace.** We will therefore clarify the scope of the direction-selection contribution and present SelInit as an optional initialization extension. Importantly, **the rank-shuffling experiment and the constant-scaling comparisons against independent valuation and shared rank-position decay use random initialization throughout**. Thus, the evidence for the core capacity-allocation contribution does not depend on SelInit. We also acknowledge that the paired CI for the GoRA rank-transfer comparison includes zero, and we will interpret that result within its supported statistical scope.Thanks for your detailed response, which resolve most of my concerns. I would like to maintain my score.
Dear Reviewer deyV,
Thank you for taking the time to read our response and for your constructive feedback.
Best regards,
The Authors
Response to Reviewer deyV (1/2): Capacity Allocation and Fixed-Scaling Evidence
We thank the reviewer for recognizing that our controlled ablations directly support conditional valuation. We address the concerns about capacity allocation, scaling confounds, baseline fairness, and initialization attribution through four additional analyses, presented in two replies.
The constant-scaling mechanism experiments use five paired seeds (42–46), while the other experiments use three paired seeds (42–44). Results are reported as mean ± sample standard deviation (SD). Confidence intervals (CIs) and tests are computed from unrounded per-seed differences using two-sided paired t procedures, without correction for multiple comparisons. Here, pp denotes percentage points.
1. Task-Informed Capacity Allocation Outperforms the Tested Random Layouts (W1)
To address the concern about whether the final rank allocation is meaningful, we randomly permute the allocated ranks among modules with identical per-rank parameter costs. We preserve the total parameter budget, rank distribution, random-initialization protocol, and effective scaling, changing only the assignment of capacity to modules.
Experiments are conducted on Llama-3.1-8B-Base/GSM8K with constant effective scaling of 2.
Each seed is paired with one budget-preserving random rank configuration. CovRA outperforms all three tested random layouts, providing direct evidence that its final capacity allocation contains useful task-specific information.
This experiment validates the effectiveness of the overall allocation layout. It does not establish that the aggregated module demand Dm precisely predicts the actual marginal benefit of adding rank to each module.
The allocation procedure includes base ranks, scored-candidate increments, soft-quota capacity supplementation, and exact-budget repair. In the default setting, K=16 limits the number of directly evaluated candidates, while the maximum allowed final rank is 32. Capacity beyond the directly scored candidates is supplied through quota increments and budget repair, without extrapolating additional conditional marginal gains.
2. Conditional-Valuation Gains Persist under Constant Effective Scaling (W1)
To isolate the effect of rank allocation from rank-dependent scaling, we set
\frac{\alpha_m}{r_m}=2,$$ while keeping the actual parameter budget, random initialization, and other training settings fixed. On Llama-3.1-8B-Base/GSM8K, the results over five paired seeds (42–46) are: | Method | GSM8K (%, mean ± SD) | CovRA-RI gain and 95% paired CI (pp) | Two-sided $p$ | |---|---:|---:|---:| | CovRA-I-RI | 71.07 ± 0.58 | **+1.23 [0.89, 1.56]** | **0.00053** | | CovRA-SD-RI | 71.65 ± 0.44 | **+0.65 [0.48, 0.82]** | **0.00045** | | GoRA-rank-RI | 72.02 ± 0.23 | +0.27 [−0.24, 0.78] | 0.2112 | | **CovRA-RI** | **72.30 ± 0.34** | — | — | **Relative to independent valuation and shared rank-position decay, the paired differences are positive for all five seeds, and both 95% paired CIs lie above zero.** These results support the benefit of conditional valuation when effective scaling is held constant. **GoRA rank transfer.** GoRA-rank-RI retains GoRA's rank-generation bounds, $r_{\min}=4$ and $r_{\max}=32$. After budget alignment, it uses the same random initialization, constant effective scaling, and optimization settings as CovRA-RI. CovRA-RI achieves a mean improvement of **0.27 pp**, but the 95% paired CI **[−0.24, 0.78]** includes zero ($p=0.2112$). Therefore, the current evidence does not establish a statistically reliable advantage over GoRA's rank configuration. Together, the rank-shuffling and constant-scaling experiments support the usefulness of CovRA's task-informed capacity allocation, while distinguishing this evidence from a direct quantitative validation of module-level demand estimates.