CovRA: LoRA Rank Allocation via Conditional Marginal Gain

Low-rank adaptation (LoRA) is a widely used method for parameter-efficient fine-tuning, and its effectiveness is shaped by rank allocation and weight initialization. Existing pre-fine-tuning rank-allocation methods typically determine module ranks from importance scores of modules or candidate rank-one updates, but these scores are computed once and not updated as candidates are selected. When different candidates within a module exhibit similar first-order loss-sensitivity patterns across calibration samples, independent scores may count this overlap repeatedly, overstating the additional value of candidates similar to those already selected. We propose Coverage-based Rank Allocation (CovRA). Using a small calibration set on a frozen model, CovRA constructs candidate rank-one updates for each module and takes their per-sample first-order loss sensitivities as responses. It then selects candidates sequentially within each module: at each step, it reevaluates all unselected candidates, treats the part of their responses explained by selected candidates as already covered, and defines the residual response energy as their additional value, yielding a conditional marginal gain. These gains are standardized across modules and used to assign integer ranks under a fixed parameter budget. CovRA additionally reuses the input directions retained during allocation to initialize LoRA. Across two base models and three tasks, CovRA outperforms standard LoRA at the same parameter budget in every setting. On Llama-3.1-8B-Base, it improves GSM8K accuracy over standard LoRA by 5.16 percentage points. In a controlled comparison in the same setting with random initialization and all else fixed, replacing independent valuation with conditional marginal valuation alone improves accuracy by 1.31 points, indicating that updating valuation with the selected set can yield more effective cross-module rank configurations.