acceptodds
Under review as a conference paper at ICLR 2027

From Uniform Capacity to Functional Roles: Efficient Visual LoRA for Source-Free Cross-Domain Few-Shot Learning

Abstract

Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) aims to rapidly adapt pre-trained vision-language models to a target domain using only a few samples, without access to source domain data. However, existing vision LoRA methods typically employ a uniform Q/K/V adaptation structure across different Transformer layers, overlooking the distinct functional roles played by layers at different depths. This paper reveals an intriguing phenomenon regarding vision LoRA in SF-CDFSL: despite being assigned identical adaptation capacities, the layers contribute highly unevenly to the final decision. Specifically, shallow layers have a minimal impact, middle-to-deep layers rely primarily on the V-path, and final layers exhibit significant QK sensitivity. We investigate and explain this phenomenon; through counterfactual interventions on identical hidden states, we attribute this behavior to a stable functional division of labor emerging among Transformer depths and QK/V projection paths during target domain adaptation. Based on this insight, we propose Role-Aligned LoRA. This method aligns adaptation capacity with functional roles by freezing low-impact layers, retaining V-LoRA in V-sensitive regions, QKV-LoRA in transitional layers, and QK-LoRA in final layers. Experiments across four SF-CDFSL datasets validate our explanation and the proposed method. Results demonstrate that Role-Aligned LoRA maintains overall accuracy on CLIP while utilizing only 27.8% of the trainable parameters found in Uniform-QKV LoRA, and reduces support-set adaptation time by 34.3%.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
source-free cross-domain few-shot learning, parameter-efficient fine-tuning, visual LoRA, vision-language models, transfer learning, few-shot learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L9AaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on depth-dependent heterogeneity in visual LoRA adaptation for source-free cross-domain few-shot learning. It proposes counterfactual QK/V interventions to characterize layer contributions and projection sensitivities, and uses the resulting profiles to construct a sparse LoRA fine-tuning layout. The proposed method reduces trainable parameters and adaptation time while largely preserving classification accuracy.

Critical strengths
  1. The study of improving efficiency of fine-tuning visual models via LoRA adaptation is meaningful.

  2. The counterfactual analysis offers a useful perspective on projection-specific adaptation effects.

Critical weaknesses
  1. The connection between intervention sensitivity and optimal adaptation placement is insufficiently established. The proposed diagnostics measure the effect of reverting individual LoRA components in a fully adapted model, but these effects do not necessarily reflect the importance of these components during retraining. The current experiments validate a particular sparse configuration without sufficiently establishing that the proposed functional measurements reliably identify optimal placements.

  2. The claimed functional specialization is stronger than the supporting evidence. The QK/V sensitivity scores are intervention-dependent operational measurements. They do not directly establish stable or intrinsic computational roles. Moreover, a larger QK contribution does not imply that the V adaptation is dispensable.

  3. The methodological novelty is limited relative to existing selective adaptation and pruning approaches. The proposed method ultimately instantiates a fixed, manually specified LoRA layout. Its advantage over simpler depth-based pruning, projection-specific adaptation, and importance-based selection methods is not fully supported.

Questions for authors

Please refer to Weakness.

Reviewer_yw81OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Role-Aligned LoRA, a parameter-efficient adaptation method for source-free cross-domain few-shot learning. Instead of uniformly applying LoRA to the query, key, and value projections of every Transformer layer, the method first performs counterfactual interventions to estimate the functional contributions and relative QK/V sensitivities of different layers. Based on these diagnostic signals, it assigns different adaptation roles to different layers, including freezing early layers and selectively adapting QK, V, or all three projections in later layers. Experiments on four cross-domain datasets using CLIP and PE-Core show that the proposed method substantially reduces the number of trainable parameters and adaptation time while maintaining comparable classification accuracy. The paper addresses a practical efficiency problem, although the novelty and generality of the proposed role diagnosis mechanism require further clarification.

Critical strengths
  1. The paper addresses a practical and relevant problem in parameter-efficient transfer learning. Uniformly applying LoRA to all layers and projections may introduce unnecessary trainable parameters and computational overhead, particularly in few-shot adaptation settings.

  2. The proposed method goes beyond uniform LoRA allocation by investigating layer-wise functional differences and selectively assigning adaptation capacity to QK and V projections. This provides an intuitive motivation for reducing redundant adaptation parameters.

Critical weaknesses
  1. The novelty and validity of the proposed method are insufficiently established. The authors should clarify how their QK/V-based functional diagnosis fundamentally differs from existing layer-importance estimation and LoRA allocation strategies. Moreover, the robustness of the diagnosed roles remains unclear, as component interactions and variations across random seeds, samples, and datasets may affect the results.

  2. The generalizability and practical effectiveness of the proposed method require further validation. The experiments cover only two 12-layer backbones, and the diagnosed roles vary across domains. Does the method require re-running QK/V interventions for each new target domain? The authors should clarify the reuse scope of the role profile and provide cross-domain transfer experiments, while further validating the robustness of performance gains and accounting for the diagnostic overhead.

Questions for authors
  1. How does the proposed counterfactual role diagnosis fundamentally differ from existing layer-importance estimation, ablation-based analysis, and adaptive LoRA allocation methods? Can the authors provide direct comparisons with representative alternatives under comparable parameter and computational budgets?

  2. How robust are the identified QK/V functional roles to random seeds, sampled diagnostic examples, and changes in the target dataset? Can the authors provide quantitative evidence that the resulting role assignments remain consistent under these variations?

  3. Can the proposed role assignment strategy generalize to backbones with different depths or architectures? If the role structure is architecture- or dataset-dependent, how much of the proposed configuration must be re-diagnosed when adapting to a new domain?

  4. How is the reported diagnostic overhead amortized in practice? Please clarify whether the diagnostic stage is performed once per backbone, once per target dataset, or once per adaptation episode, and report the overall cost under these different settings.

  5. In the parameter-matched comparison, how were the random LoRA layouts selected? Were the reported configurations chosen independently of the final evaluation data? Can the authors report results across multiple randomly sampled layouts to establish the robustness of the comparison?

  6. Given that the main accuracy difference over uniform QKV LoRA is very small and its 95% confidence interval includes zero, can the authors clarify whether the primary contribution is intended to be accuracy improvement or efficiency improvement? Please provide sufficient statistical details to support the reported comparisons.

Reviewer_h9WpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is targeting solving the issue of Source-Free Cross-Domain Few-Shot Learning, the core idea is that they found that each layer contribute highly unevenly to the final decision, therefore, they proposed Role-Aligned LoRA, which could reduce the required training parameter and improve the adaptation time.

Critical strengths

None

Critical weaknesses

The paper is written poorly and hard to follow what's the difference between this compared to previous works, and the experiments are lacking, I do not see anything significant to support this paper to be accepted to ICLR. Here are some examples for my justification:

  1. At related work sections, authors just cite many papers without properly position the work against previous works. It look messy and hard to read without giving any useful information. For example, the core idea of the paper: different layers contribute differently is well known from the MLP, CNN era, however, authors do not mention any of the related works on this topic.
  2. Authors should compare it to previous works, there are so much LoRA variants, the authors should properly compared with those.
  3. The figures are hard to understand by its own. For example:
    1. What is LL, BL, LB, BB at figure 4?
    2. For figure 3, what's the main information the authors tried to conduct ? And the scale seems really small, make me wonder, are they just noises? The curve of ChestX looks flat, what's the point of presenting it?

Overall, whole paper feel disconnected, the core idea comes from a well-known phenomenon that plenty of works have discussed it, the methods itself is also pretty standard, the experiments coverage is also narrow. However, the authors did nothing to position their works properly against those works, which is the core reason for rejecting the paper.

Questions for authors

The authors should properly position and refining their works before submitting. The core idea is well-known, authors should properly distinguish what's the contribution it brings compared to previous works. The authors should have substantial revised the paper properly to meet the ICLR standard.

1 comment

Thank you for the response and revisions. I acknowledge that the authors have clarified some of my original concerns, especially the figures and experimental details. However, my main concern about the novelty and significance of this work remains.

  1. The core idea is still very similar to previous importance-based LoRA methods. For example, AdaLoRA and LoRA-drop already study how to allocate or retain LoRA components based on their importance. I have also checked the revised Related Work, but I still do not think the authors properly position their contribution against these methods. The authors mainly explain how their importance measurement differs technically, without explaining what limitations of previous methods they address or what significant new insights their approach provides.

  2. More importantly, why are these methods not included in the experiments? The authors compare with different ranks and random placements, but not with existing importance-based LoRA methods. Without direct comparisons under similar parameter budgets, it is difficult to judge whether the proposed method actually brings meaningful improvements.

  3. The presentation is still difficult to follow. The paper contains too many statistical details and confidence intervals in the main text without clearly explaining what readers should learn from them. Even Table 3 reports unnecessarily precise parameter counts, making the results harder to read. Overall, the writing feels AI generated, as if the manuscript has not been carefully reviewed and polished by the authors.

Overall, while the revisions improve some details, I am still not convinced that the paper provides sufficiently new insights or significant improvements over previous works. Therefore, I maintain my rejection recommendation.

Comments on the paper

1 comment
Area_Chair_dg4eOpenReview ↗

ICLR 2027 Significance Reminder

Dear Reviewers,

Please review the authors' rebuttal to your comments and the other reviews and to respond.

Please remember to update your ratings and final justification if your concerns have been addressed. If ratings are not updated, clearly explain any remaining concerns in your final justification.

As a reminder, the discussion period ends on November 18th, 11:59 PM AoE.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.