- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CARD-LoRA, which combines task dependence estimated through suppression gates with pre-trained spectral strength to select input directions for LoRA adaptation. The method restricts the input update space while allowing the output correction to be learned freely. Experiments on LLaMA-2-7B and Qwen2.5-7B evaluate downstream performance and capability retention, with comparisons against existing LoRA methods and several ablation studies.
Critical strengths
-
The method has a clear and reasonable design. CARD-LoRA separates direction discovery from adapter training. It uses task dependence and spectral strength to select input directions, while keeping the output correction learnable. This is a useful way to think about the trade-off between learning new tasks and preserving existing capabilities, even though the motivation for the specific selection rule is not fully justified.
-
The paper provides useful ablations and some promising results. Table 3 shows that combining task dependence with spectral strength performs better than using either signal alone in terms of the overall average. Table 4 also supports the choice of input-only projection over two-sided constraints. The additional results on Qwen2.5-7B provide some evidence that the method is not limited to a single backbone. These experiments give the proposed design some empirical support.
Critical weaknesses
-
The main motivation is not sufficiently justified, and the experimental evidence is less convincing than the overall averages suggest. The suppression gates measure how much the task relies on existing pretrained mappings, but this does not mean that updating along those directions will improve performance. In particular, the paper does not explain why high task dependence combined with low spectral strength should identify the most useful directions for adaptation. This is important because the main novelty largely comes from this selection rule. The empirical evidence also raises concerns. In Table 1, about 84% of CARD-LoRA-16's average improvement over vanilla LoRA comes from MBPP. Without MBPP, the gain drops to only 0.22 points, while several commonsense benchmarks get worse. Moreover, the task-specific direction analysis in Figure 4 uses only one archived discovery result per task, without checking stability across discovery seeds. These issues weaken the evidence for the paper's central claims.
-
The released code does not appear to implement the proposed method, making the main results difficult to reproduce. The anonymous repository mainly contains PC-LoRA, which uses complementary-space projections and additional regularization losses rather than CARD-LoRA's input-support projection. I could not find the suppression-gate discovery, the direction selection, or the complete CARD-LoRA training pipeline. The available configurations and results also do not match the LLaMA-2-7B and Qwen2.5-7B experiments reported in the paper. As a result, the provided code cannot be used to independently verify the proposed method or its main results. This is a serious concern about reproducibility.
Questions for authors
-
What is the evidence that suppression dependence predicts adaptation benefit? A direction that is important to preserve is not necessarily a good direction to update. Can the authors provide direct evidence that directions with higher actually lead to larger improvements when used for adaptation, rather than only showing that the proposed ranking performs better on average?
-
Can the authors provide the actual CARD-LoRA implementation and matching experiment configurations? The current repository appears to implement PC-LoRA instead. Please clarify whether the wrong code version was released and provide the suppression-gate discovery, direction selection, adapter training, and evaluation code, along with the configurations used for the main tables. It would also be important to show how the discovered supports correspond to the checkpoints used in the reported experiments.
-
Can the authors explain and verify the unusually large MBPP improvement in Table 1? CARD-LoRA-16 improves MBPP from 2.76 to 14.05 after commonsense fine-tuning, and this accounts for most of its overall gain. Please provide per-example results, MBPP scores across random seeds, exact evaluation settings, and checks for possible data overlap. This would help determine whether the improvement reflects a reliable retention benefit.
Thank you for the detailed response. Some of my concerns have been addressed, but the following remain unresolved.
(W2) My main concern is whether the directions identified by the proposed scores actually play the roles the authors assign to them. It is unclear whether a direction labeled as task-dependent is truly one the task relies on, or whether the adapt/conflict/preserve/weak roles reflect real functional differences. Table 3 does not confirm this: among weak directions, downstream performance is nearly the same whether high- or low-dependence directions are selected, and the conflict-only selection, which is supposed to contain task-dependent directions, is the lowest. Direct validation is needed, such as the controlled experiment mentioned in the response (matched spectral strength, high vs. low dependence) and evidence that the same directions are identified as task-dependent across discovery seeds.
(Q2) Summing the reported numbers, CARD-LoRA-16 takes 279.6 minutes in total (16.72 SVD + 5.73 discovery + 257.15 adapter training), about 1.46× LoRA's 191.80 minutes, and the discovery-stage GPU peak is about 3× LoRA's. This cost seems high relative to the average gains.
(Q3) First, OPLoRA's setting is not r=3, α=3; it is r=32, α=32 for commonsense and r=64, α=64 for math (Table 8 of OPLoRA). The Commonsense170K benchmark is standard in LLM-Adapters, DoRA, and follow-up work (e.g., ReFT, MiLoRA, LoRA-Dash), where LoRA variants are commonly evaluated with r=32. Compared with this, r=8 in this paper seems too small. I agree that CARD-LoRA is ahead under the current setting (r=8, α=16), but for a fair comparison I recommend adding results at r=32.