acceptodds
Withdrawn from ICLR 2027

Learning What to Preserve and Where to Adapt in LoRA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning, yet learning a new task can still compromise a model's existing capabilities. To balance adaptation with retention, existing approaches guide capacity allocation by protecting dominant singular subspaces or amplifying task-specific directions. Yet identifying a task-dependent component leaves open whether to preserve its existing mapping or allocate new capacity along its input direction. We find that using components with high task dependence but low pre-trained spectral strength as adaptation support effectively balances downstream learning with performance on held-out tasks outside the fine-tuning domain. Our method, Capacity Allocation through Role Discovery for Low-Rank Adaptation (CARD-LoRA), probes task dependence with suppression gates and combines it with spectral strength to select the input directions a LoRA update can act on, while leaving its output correction freely learnable. On LLaMA-2-7B, CARD-LoRA achieves the highest combined downstream and held-out evaluation average among the compared methods in both commonsense and mathematical fine-tuning. The gains over vanilla LoRA are 1.23 and 0.62 percentage points, respectively. Restriction-side ablations further show the benefit of learning the output correction independently of the selected components' pre-trained output directions. More broadly, this work frames low-rank adaptation as deciding where to intervene in a pre-trained model, not only how to parameterize its update.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, spectral analysis, capability retention, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_k39xOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CARD-LoRA, which combines task dependence estimated through suppression gates with pre-trained spectral strength to select input directions for LoRA adaptation. The method restricts the input update space while allowing the output correction to be learned freely. Experiments on LLaMA-2-7B and Qwen2.5-7B evaluate downstream performance and capability retention, with comparisons against existing LoRA methods and several ablation studies.

Critical strengths
  1. The method has a clear and reasonable design. CARD-LoRA separates direction discovery from adapter training. It uses task dependence and spectral strength to select input directions, while keeping the output correction learnable. This is a useful way to think about the trade-off between learning new tasks and preserving existing capabilities, even though the motivation for the specific selection rule is not fully justified.

  2. The paper provides useful ablations and some promising results. Table 3 shows that combining task dependence with spectral strength performs better than using either signal alone in terms of the overall average. Table 4 also supports the choice of input-only projection over two-sided constraints. The additional results on Qwen2.5-7B provide some evidence that the method is not limited to a single backbone. These experiments give the proposed design some empirical support.

Critical weaknesses
  1. The main motivation is not sufficiently justified, and the experimental evidence is less convincing than the overall averages suggest. The suppression gates measure how much the task relies on existing pretrained mappings, but this does not mean that updating along those directions will improve performance. In particular, the paper does not explain why high task dependence combined with low spectral strength should identify the most useful directions for adaptation. This is important because the main novelty largely comes from this selection rule. The empirical evidence also raises concerns. In Table 1, about 84% of CARD-LoRA-16's average improvement over vanilla LoRA comes from MBPP. Without MBPP, the gain drops to only 0.22 points, while several commonsense benchmarks get worse. Moreover, the task-specific direction analysis in Figure 4 uses only one archived discovery result per task, without checking stability across discovery seeds. These issues weaken the evidence for the paper's central claims.

  2. The released code does not appear to implement the proposed method, making the main results difficult to reproduce. The anonymous repository mainly contains PC-LoRA, which uses complementary-space projections and additional regularization losses rather than CARD-LoRA's input-support projection. I could not find the suppression-gate discovery, the Ti(1−Si)T_i(1-S_i) direction selection, or the complete CARD-LoRA training pipeline. The available configurations and results also do not match the LLaMA-2-7B and Qwen2.5-7B experiments reported in the paper. As a result, the provided code cannot be used to independently verify the proposed method or its main results. This is a serious concern about reproducibility.

Questions for authors
  1. What is the evidence that suppression dependence predicts adaptation benefit? A direction that is important to preserve is not necessarily a good direction to update. Can the authors provide direct evidence that directions with higher Ti(1−Si)T_i(1-S_i) actually lead to larger improvements when used for adaptation, rather than only showing that the proposed ranking performs better on average?

  2. Can the authors provide the actual CARD-LoRA implementation and matching experiment configurations? The current repository appears to implement PC-LoRA instead. Please clarify whether the wrong code version was released and provide the suppression-gate discovery, direction selection, adapter training, and evaluation code, along with the configurations used for the main tables. It would also be important to show how the discovered supports correspond to the checkpoints used in the reported experiments.

  3. Can the authors explain and verify the unusually large MBPP improvement in Table 1? CARD-LoRA-16 improves MBPP from 2.76 to 14.05 after commonsense fine-tuning, and this accounts for most of its overall gain. Please provide per-example results, MBPP scores across random seeds, exact evaluation settings, and checks for possible data overlap. This would help determine whether the improvement reflects a reliable retention benefit.

Reviewer_EwJSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the fundamental tension in LoRA between adapting a pre-trained language model to a new task and preserving its original pre-trained performance. The authors argue that standard LoRA fails to distinguish between pre-trained directions that should be protected and those that require modification. To solve this issue, the authors proposed CARD-LoRA, which involves two principal phrases:

  1. To identify task dependence, the proposed method trains a gate function over the singular value of the decomposed pre-trained weight over the target task, so as to determine which parts of the weight are critical for this task. Meanwhile, the authors treat the singular-value strengths as the pre-trained strengths.
  2. The LoRA training is conducted on the restricted directions identified as high task dependence and low pre-trained strength.

By strategically deciding where to intervene in the model’s weight space, the method aims to balance downstream task performance and knowledge retention.

Critical strengths
  1. This paper designed a trainable gate function along with a loss function to identify the task dependence over the pre-trained model weights.
  2. Combined with pre-trained model strength and the gate values from pre-staged training, this method updates LoRA in specified directions.
  3. The proposed methods showed improved task performance while maintain less degradation over OOD tasks.
Critical weaknesses
  1. The introduction of a probing phrase involving SVD on pre-trained weights and gate function optimization would introduce non-negligible computational overhead before actual training happens.
  2. Compared to LoRA, CARD-LoRA introduced new hyperparameters related to the gate function training and k selection, making this method less applicable and requiring careful tuning for different models or tasks.
  3. This analysis is primarily on two legacy models (Llama-2-7B and Qwen-2.5-7B), and does not explore how this role-discovery logic might translate to other modern LLMs or newer architectures (such as MoE models or linear attention mixed models, e.g., Qwen3.5).
Questions for authors
  1. How sensitive is the role discovery phase to the size of the probing dataset?
  2. How does this structural intervention compare to traditional "replay" or "distillation" methods for preventing catastrophic forgetting in terms of both memory efficiency and final performance?
  3. As the base model size increases, does the gap between CARD-LoRA and vanilla LoRA widen, or do the benefits of specific direction selection diminish?
Reviewer_9ay8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CARD-LoRA is a LoRA variant that selects a subset of the right-singular vectors of each pre-trained weight in advance, restricts the input of LoRA to that subspace, and trains the output factor freely. It decomposes the weight with SVD, attaches a sigmoid gate to every singular component, and with the model frozen, trains only the gates with the task loss and a sparsity penalty to obtain a task-dependence score. It combines this with the normalized singular value strength to build an adaptation score that gives high values to components that are weak but relied on by the task, and selects the top-k directions for each weight. The gates are then removed and a new adapter is trained.

Critical strengths

(S1) The method is simple — a fixed projector before the LoRA input — and merges into the base weight at inference with no overhead. (S2) The paper is clearly written and easy to follow.

Critical weaknesses

(W1) LoRA-DASH and its follow-up work propose a similar concept under the name “task-specific direction”. A qualitative and quantitative comparison with those papers is needed. Can the proposed model be evaluated in the experimental setting of those papers?

(W2) The results in Table 3 do not match the intended role of each score. Conflict-only, which the task is supposed to depend on, is the lowest on downstream, and weak-only, which the task is supposed not to use, is the highest on downstream. In other words, it is hard to see T as reflecting “which components learn when given capacity” as designed.

Questions for authors

(Q1) For rectangular matrices, is the null space excluded? Papers such as LoRA-Null make use of this null space. Please compare.

(Q2) A memory and runtime analysis is needed. In the probe stage, the gates are trained while the full SVD of the weights is kept in memory, so the memory cost is expected to be large for large models, and since reconstruction is required at every step, the runtime is expected to increase substantially compared to standard LoRA. Please provide both an analytical and an empirical analysis.

(Q3) Table 1 and Table 2 appear to follow the experimental setting of OPLoRA, but the results differ considerably (LoRA average of 57.03 in OPLoRA vs. 50.59 in this paper). What is different? Can the comparison be made under the same setting?

(Q4) Can you show which axes are actually selected? The expectation is that many of the middle axes would be selected. Since S is a heavily skewed quantity, it is not clear whether the selection actually matches the intended “high dependence, low strength” rule.

1 comment

Thank you for the detailed response. Some of my concerns have been addressed, but the following remain unresolved.

(W2) My main concern is whether the directions identified by the proposed scores actually play the roles the authors assign to them. It is unclear whether a direction labeled as task-dependent is truly one the task relies on, or whether the adapt/conflict/preserve/weak roles reflect real functional differences. Table 3 does not confirm this: among weak directions, downstream performance is nearly the same whether high- or low-dependence directions are selected, and the conflict-only selection, which is supposed to contain task-dependent directions, is the lowest. Direct validation is needed, such as the controlled experiment mentioned in the response (matched spectral strength, high vs. low dependence) and evidence that the same directions are identified as task-dependent across discovery seeds.

(Q2) Summing the reported numbers, CARD-LoRA-16 takes 279.6 minutes in total (16.72 SVD + 5.73 discovery + 257.15 adapter training), about 1.46×1.46\times LoRA's 191.80 minutes, and the discovery-stage GPU peak is about 3×3\times LoRA's. This cost seems high relative to the average gains.

(Q3) First, OPLoRA's setting is not r=3r=3, α=3\alpha=3; it is r=32r=32, α=32\alpha=32 for commonsense and r=64r=64, α=64\alpha=64 for math (Table 8 of OPLoRA). The Commonsense170K benchmark is standard in LLM-Adapters, DoRA, and follow-up work (e.g., ReFT, MiLoRA, LoRA-Dash), where LoRA variants are commonly evaluated with r=32r=32. Compared with this, r=8r=8 in this paper seems too small. I agree that CARD-LoRA is ahead under the current setting (r=8r=8, α=16\alpha=16), but for a fair comparison I recommend adding results at r=32r=32.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.