Learning What to Preserve and Where to Adapt in LoRA

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning, yet learning a new task can still compromise a model's existing capabilities. To balance adaptation with retention, existing approaches guide capacity allocation by protecting dominant singular subspaces or amplifying task-specific directions. Yet identifying a task-dependent component leaves open whether to preserve its existing mapping or allocate new capacity along its input direction. We find that using components with high task dependence but low pre-trained spectral strength as adaptation support effectively balances downstream learning with performance on held-out tasks outside the fine-tuning domain. Our method, Capacity Allocation through Role Discovery for Low-Rank Adaptation (CARD-LoRA), probes task dependence with suppression gates and combines it with spectral strength to select the input directions a LoRA update can act on, while leaving its output correction freely learnable. On LLaMA-2-7B, CARD-LoRA achieves the highest combined downstream and held-out evaluation average among the compared methods in both commonsense and mathematical fine-tuning. The gains over vanilla LoRA are 1.23 and 0.62 percentage points, respectively. Restriction-side ablations further show the benefit of learning the output correction independently of the selected components' pre-trained output directions. More broadly, this work frames low-rank adaptation as deciding where to intervene in a pre-trained model, not only how to parameterize its update.