acceptodds
Under review as a conference paper at ICLR 2027

SAHR: Semantics-Aware Head Reconfiguration for Efficient Task Adaptation

Abstract

Vision-language models (VLMs) such as CLIP have shown strong transferability across downstream tasks. However, efficiently adapting CLIP under limited supervision remains challenging because existing parameter-efficient fine-tuning (PEFT) methods typically modify prompts, lightweight modules, or support features without explicitly identifying which internal visual Transformer units are most relevant to a target task. In this work, we observe that attention heads in the CLIP visual Transformer exhibit semantic specialization, with different heads consistently capturing different visual or semantic attributes. Accordingly, we propose Semantics-Aware Head Reconfiguration (SAHR), an efficient framework that directly updates task-relevant attention heads during CLIP adaptation. SAHR constructs text-aligned semantic profiles for attention heads and quantifies their semantic correspondence to downstream tasks through task-aware descriptions. Based on this head-task correspondence, SAHR adapts only a sparse set of top-ranked heads. Extensive experiments on 11 few-shot classification benchmarks, Flickr30K and MSCOCO image-text retrieval, and OOD robustness show that SAHR outperforms recent PEFT methods with low parameter scale. In the 16-shot setting, SAHR achieves 83.1% average accuracy on 11 benchmarks with only 1.18M parameters, outperforming the strongest PEFT method while reducing parameters by 94.6%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, parameter-efficient fine-tuning, multimodal representation learning, interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_umaTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAHR, a parameter-efficient approach to adapting CLIP by fine-tuning task-relevant attention heads. The key idea is to use the semantic specialization of attention heads as a signal for deciding which parameters to update. The authors construct text-aligned semantic profiles for late-layer attention heads using unlabeled images and general visual descriptions. They then generate task-specific descriptions with an LLM and compute a specialty score to rank heads according to their relevance to the target task. Only the Q/K/V/O projection slices associated with the top-ranked heads are updated. The method is evaluated on 11 few-shot classification benchmarks, Flickr30K and MSCOCO image-text retrieval, and OOD generalization benchmarks.

Critical strengths
  1. The motivation behind SAHR reasonable. This paper provides a concrete way to turn head-level semantic profiles into a selection criterion for fine-tuning.
  2. The experiments cover a diverse set of downstream tasks. The ablations in Table 6 are also helpful to understand the method.
Critical weaknesses
  1. Limited evidence of novelty and practical advantages over existing adaptation strategies. The main components, including text-based head interpretation and selective head adaptation, are closely related to prior work. The proposed Specialty Score appears to be the main methodological addition.
  2. The Specialty Score relies on a predefined collection of benchmark tasks. It is unclear whether the selected heads would remain consistent when this collection changes or when adapting to an unseen task.
Questions for authors

None

3 comments

We sincerely thank Reviewer umaT for the positive and insightful feedback. We appreciate the reviewer's recognition of the methodological rationale of our proposed Semantics-Aware Head Reconfiguration (SAHR) framework, particularly the use of semantic profiles to guide task-specific attention head selection. We are encouraged that the reviewer finds our experimental evaluation comprehensive and the overall method design reasonable.

We also appreciate the reviewer's thoughtful concerns regarding the technical novelty of SAHR and the generalizability of the Specialty Score under different task collections. These are important aspects of our method, and we have carefully considered them in our response. We provide further clarification of the technical contributions of SAHR and additional analysis to address the robustness of task-aware head selection.


Q1: Limited evidence of novelty and practical advantages over existing adaptation strategies. The main components, including text-based head interpretation and selective head adaptation, are closely related to prior work. The proposed Specialty Score appears to be the main methodological addition.

A1: Thank you for this insightful comment. We agree that text-based head interpretation and selective parameter adaptation have been explored in prior studies. However, SAHR's contribution lies in explicitly quantifying the semantic correspondence between downstream task characteristics and the functional specialization of individual attention heads, and leveraging this correspondence to guide task-specific parameter adaptation through three closely integrated stages.

(1) Contribution-aware semantic profiling. Rather than simply assigning textual descriptions to attention heads, SAHR constructs weighted semantic profiles by decomposing head contributions along text-aligned directions. This provides a structured representation of the complementary semantic factors captured by individual heads.

(2) Task-relative semantic specialization scoring. This is the central methodological contribution of SAHR. Raw semantic similarity alone cannot distinguish heads that are specifically relevant to a target task from those that are broadly relevant across multiple tasks. SAHR addresses this through the Specialty Score, which normalizes each head's task relevance against its average relevance across the task collection. This explicitly quantifies task-specific specialization and provides a criterion for ranking adaptation-relevant heads.

(3) Structured sparse head adaptation. SAHR translates the resulting task-specific ranking into structured parameter updates, selectively adapting the Q/K/V/O parameter slices of the selected heads while freezing the remaining parameters. Thus, semantic interpretation directly determines where adaptation capacity is allocated.

The practical advantages are also supported by our ablation results. Removing Specialty Score normalization reduces the average 16-shot accuracy from 83.1% to 82.1%, demonstrating the importance of distinguishing task-specific relevance from general relevance.

R3-Table 1: Ablation on adaptation design.

MethodParams1-shot2-shot4-shot8-shot16-shot
w/o Spec1.18M71.073.976.579.582.1
SAHR (Ours)1.18M72.274.877.880.483.1

Furthermore, compared with gradient-based and Taylor-based head selection, SAHR achieves the highest accuracy across all five shot settings while requiring substantially less selection time.

R3-Table 2: Ablation on head selection strategies.

MethodTime (s)1-shot2-shot4-shot8-shot16-shot
Random--69.472.875.978.981.9
Grad.-Norm7.8470.573.776.579.382.1
Grad.-Gate8.7670.873.876.479.181.6
Taylor9.1869.973.476.479.581.2
SAHR (Ours)0.5372.274.877.880.483.1

These results further demonstrate that both task-relative semantic scoring and structured Q/K/V/O adaptation contribute to the effectiveness of SAHR. Therefore, the novelty of SAHR is not the isolated use of text-based interpretation or sparse adaptation, but the integration of contribution-aware semantic profiling and task-relative specialization scoring into an explicit mechanism for structured, task-specific head adaptation. This connection transforms head interpretability from a descriptive analysis into an actionable signal, enabling semantics-guided, task-specific and parameter-efficient reconfiguration of attention heads.

Revisions in the manuscript: Sections 2 and 3.1–3.3, as well as Section 4.4 (Tables 8 and 9). We have further clarified the technical contributions of SAHR and the respective roles of semantic profiling, Specialty Score, and structured sparse adaptation.


Q2: The Specialty Score relies on a predefined collection of benchmark tasks. It is unclear whether the selected heads would remain consistent when this collection changes or when adapting to an unseen task.

A2: Thank you for raising this important concern. We adopt 11 widely used benchmark datasets in few-shot CLIP adaptation to cover diverse visual domains and provide a comprehensive evaluation across different downstream tasks. Accordingly, these datasets constitute the reference task collection D\mathcal{D} for computing task-relative normalization in the Specialty Score. However, this collection serves only as a reference for measuring relative task specificity, rather than restricting the downstream tasks to which SAHR can be applied.

For a new downstream task, SAHR constructs task-aware semantic descriptions and computes the corresponding head–task matching scores. The Specialty Score then compares the relevance to the target task against the average relevance over the reference task collection. Thus, a new target task does not need to be included in the predefined reference collection, as its semantic descriptions can be used to calculate its relevance to existing head profiles.

Generalization to unseen classes. We have also evaluated SAHR under the base-to-novel protocol across 11 benchmarks. During adaptation, only base-class descriptions are used for head selection, while novel classes are excluded from the adaptation process. The results are summarized below.

R3-Table 3: Base-to-novel generalization performance across 11 benchmarks.

MethodBase (%)Novel (%)HM (%)
CLIP69.3474.2271.70
CoOp82.6963.2271.66
CoCoOp80.4771.6975.83
TextRefiner79.7474.3276.94
SAHR (Ours)84.3573.5378.57

SAHR achieves the highest Base accuracy of 84.35% and HM of 78.57%, while maintaining a Novel accuracy of 73.53%, close to the zero-shot CLIP result of 74.22%. These results provide complementary evidence that SAHR can adapt to observed classes while largely retaining its ability to recognize unseen classes.

Revisions in the manuscript: Section 3.2 and Section 4.3 (Table 2, Page 7), with detailed per-dataset base-to-novel results in Appendix C (Table 12, Page 18).


We are pleased that the reviewer has an overall positive view of our work and sincerely appreciate the thoughtful feedback. We hope that our responses and additional clarifications have adequately addressed the reviewer's concerns. Thank you once again for your time and effort.

Thank for your reply. I will keep my positive rating.

Reviewer_q879OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents SAHR, a PEFT method for CLIP built upon the semantic specialization property of vision transformer attention heads. It first offline builds semantic profiles for late‑layer heads, then uses LLM‑generated task descriptions to score and select top‑K task‑relevant heads, only fine‑tuning their Q/K/V/O projections while freezing rest weights. SAHR adds no inference overhead. Evaluations cover 11 few‑shot classification benchmarks, image‑text retrieval and OOD robustness. Under 16‑shot ViT‑B/16, it reaches 83.1% accuracy with merely 1.18M trainable parameters.

Critical strengths
  1. Turns transformer interpretability insight into practical PEFT algorithm, conceptually distinct from prompt/adapter methods.
  2. Great parameter‑accuracy tradeoff; zero extra inference compute.
  3. Comprehensive evaluation: few‑shot classification, retrieval, OOD robustness, multiple CLIP backbones; solid ablations and t‑SNE qualitative analysis.
  4. Well‑documented reproducibility details in appendix.
Critical weaknesses
  1. The core pipeline is a straightforward combination of existing building blocks: prior work has demonstrated attention‑head semantic specialization, LLM‑driven prompt augmentation, and sparse head fine‑tuning. The main contribution is their heuristic combination, lacking novel model designs or new theoretical formulation for head‑task semantic matching.
  2. No ablation for key hyper‑parameter M (fixed to 20). It controls profile richness, yet its impact on performance/computation is unknown.
  3. Heavy dependency on closed‑source LLM (GPT‑4.5). No experiments with GPT‑3.5‑turbo or open‑source LLMs; offline deployment feasibility unvalidated.
  4. Missing comparison against gradient‑based head‑selection baselines; only random selection is ablated.
  5. Offline profiling cost and sensitivity to profiling dataset size are not studied.
  6. Limited gains on MSCOCO retrieval without further diagnosis; no failure‑case analysis.
Questions for authors
  1. Please add ablation over hyper‑parameter M (e.g. {5,10,20,30,40}), how does M affect accuracy and computational cost?
  2. What performance drop occurs if you replace GPT‑4.5 with GPT‑3.5‑turbo / open‑source LLMs like Llama‑3? Can prompt engineering mitigate degradation?
  3. Could you add ablation on profiling‑set size (10k/50k/100k images)? Report the compute overhead of offline profiling.
  4. How does your semantic‑based head ranking compare against gradient‑magnitude based head importance on downstream support set?
  5. Why are SAHR’s gains much weaker for MSCOCO retrieval than few‑shot classification?
  6. Show several representative failure cases: what tasks degrade SAHR’s semantic matching?
  7. Given that each key component has been studied in prior literature, what distinct technical innovation does this work bring beyond heuristic combination of existing techniques?
6 comments

We sincerely thank the reviewer for the detailed and constructive feedback. We appreciate the positive assessment of our proposed Semantics-Aware Head Reconfiguration (SAHR) framework, particularly its semantic-aware head selection mechanism and comprehensive evaluation across few-shot classification, image-text retrieval, and out-of-distribution generalization.

In response to the reviewer's concerns, we have conducted additional experiments and analyses to further examine the sensitivity, robustness, and computational efficiency of SAHR. Specifically, we have expanded the ablation studies on the semantic profile size and profiling dataset size, investigated the robustness to different LLMs and prompt designs, and added comparisons with gradient-based and Taylor-based head selection strategies. We have also further analyzed the offline profiling cost. We have carefully revised the manuscript to address the reviewer's questions and provide additional empirical evidence supporting the design choices and effectiveness of our method. We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable to re-consider the rating in light of our efforts and dedication.


Q1: Please add ablation over hyper-parameter MM (e.g., 5, 10, 20, 30, 40). How does MM affect accuracy and computational cost?

A1: Thank you for this helpful suggestion. In SAHR, MM denotes the number of text-aligned semantic spans retained in each attention head's semantic profile. A larger MM allows more semantic directions to be selected, providing a richer representation of each head's contribution.

We have conducted additional experiments with M∈{5,10,20,30,40}M \in \{5, 10, 20, 30, 40\}, evaluating both downstream classification accuracy and computational cost. Specifically, ImageNet Acc. denotes the 16-shot classification accuracy on ImageNet; Semantic Decomp. denotes the time required to construct semantic profiles for 48 candidate attention heads; and Scoring denotes the time required to compute head--task matching scores across 11 reference tasks. The results are summarized below.

R2-Table 1: Sensitivity to the semantic profile size MM.

MetricM=5M=5M=10M=10M=20M=20M=30M=30M=40M=40
ImageNet Acc. (%)71.872.572.772.571.8
Semantic Decomp. (s)105.8123.5159.1194.7230.3
Scoring (s)5.715.745.845.936.01

The results show that increasing MM from 5 to 20 improves ImageNet accuracy from 71.8% to 72.7%. However, further increasing MM does not yield additional accuracy gains. In contrast, semantic decomposition time increases substantially, from 159.1s at M=20M=20 to 230.3s at M=40M=40, while task-specific scoring time increases only slightly. Therefore, we adopt M=20M=20 as the default setting, as it achieves the highest classification accuracy while maintaining a favorable balance between semantic representation richness and computational cost.

Revisions in the manuscript: We have added the MM sensitivity analysis in Section 4.4 (Table 10, Page 10) and provided the detailed results and discussion in Appendix E.2 (Table 15, Page 22).


Q2: What performance drop occurs if you replace GPT-4.5 with GPT-3.5-turbo/open-source LMs like Llama-3? Can prompt engineering mitigate degradation?

A2: Thank you for raising this important question. We agree that the dependence on a specific LLM is an important consideration for the practical applicability of SAHR. To investigate this, we have evaluated SAHR using three alternative LLMs, including the open-source Llama 3.1 and GPT-5.6 Sol and GPT-6.1 Sol, while retaining GPT-4.5 as the default reference. Regarding GPT-3.5-turbo, it is a proprietary OpenAI model rather than an open-source LLM, and its API access was officially discontinued on October 23, 2026.

R2-Table 2: Robustness to different LLMs across five few-shot settings.

LLM1-shot2-shot4-shot8-shot16-shot
Llama 3.172.074.477.279.982.8
GPT-5.6 Sol71.873.677.880.683.2
GPT-6.1 Sol72.574.977.880.583.1
SAHR (GPT-4.5)72.274.877.880.483.1

The results demonstrate that SAHR is robust to the choice of LLM. GPT-6.1 Sol also achieves comparable or slightly better results than GPT-4.5 across the evaluated settings.

Importantly, SAHR does not rely on a specific proprietary LLM. Its semantic profiling and head selection framework can accommodate different language models, including open-source alternatives such as Llama 3.1. This provides a practical option for local LLM deployment, reducing reliance on external API services and facilitating offline use in resource- or privacy-constrained environments.

Revisions in the manuscript: Appendix E.2, Robustness to Different LLM Generators (Table 17, Page 22).


Q3: Could you add ablation on profiling-set size (10k/50k/100k images)? Report the compute overhead of offline profiling.

A3: Thank you for this helpful suggestion. We have conducted additional experiments to investigate the effect of profiling dataset size using CLIP ViT-B/16, with the semantic profile size fixed at M=20M=20. Specifically, we construct semantic profiles using 10K, 50K, and 100K images sampled from the Open Images V7 training split. We evaluate both the downstream 16-shot ImageNet accuracy and the stability of the selected top-6 attention heads.

R2-Table 3: Sensitivity to profiling dataset size and corresponding offline profiling costs.

Profiling ImagesImageNet Acc. (%)Top-6 OverlapExtraction TimeDecomposition TimeTotal Time
10K72.55/621m 28s27.24s21m 55s
50K72.76/61h 38m 44s159.05s1h 41m 23s
100K72.76/63h 11m 12s338.39s3h 16m 51s

The results show that the 10K configuration already identifies five of the six heads selected by the 50K and 100K configurations, while the 50K and 100K configurations produce identical top-6 head selections. Moreover, increasing the profiling set from 50K to 100K does not improve ImageNet accuracy, which remains at 72.7%, but nearly doubles the offline profiling time. These results indicate that 50K images provide sufficient semantic coverage for stable head selection under our experimental setting, achieving a favorable balance between downstream accuracy, selection stability, and computational cost.

Importantly, offline semantic profiling is performed only once for each CLIP backbone. The resulting semantic profiles are stored and reused across downstream datasets, few-shot settings, and random seeds. Therefore, the reported profiling cost is a one-time overhead per backbone rather than a recurring cost for each downstream task. For new tasks, only task-specific head scoring needs to be performed using the precomputed profiles, which incurs substantially lower computational cost.

Revisions in the manuscript: Appendix E.2, Sensitivity to Profiling Dataset Size (Table 16), and Appendix E.3, Offline Profiling Cost Analysis. We have also clarified the profile reuse mechanism in Appendix A.1.


Q4: How does your semantic-based ranking compare against gradient-magnitude-based head importance on the downstream support set? A4: Thank you for this important question. To directly evaluate the effectiveness of semantic-based head selection, we compare SAHR with three importance-based alternatives using the same downstream support set and head selection budget: parameter-gradient norm (Grad.-Norm), head-gate gradient (Grad.-Gate)[1], and Taylor-based parameter importance (Taylor)[2]. Specifically, Grad.-Norm ranks heads by the L2 norms of their Q/K/V/O parameter gradients, Grad.-Gate uses the mean absolute gradients of head-wise output gates, and Taylor ranks heads using first-order parameter importance scores. We also include random selection as a reference baseline.

R2-Table 4: Comparison of different head selection strategies across five few-shot settings.

MethodSelection Time (s)1-shot2-shot4-shot8-shot16-shot
Random--69.472.875.978.981.9
Grad.-Norm7.8470.573.776.579.382.1
Grad.-Gate8.7670.873.876.479.181.6
Taylor9.1869.973.476.479.581.2
SAHR (Ours)0.5372.274.877.880.483.1

SAHR achieves the highest accuracy across all five shot settings, consistently outperforming random, gradient-based, and Taylor-based selection under the same head selection budget. Moreover, its average head selection time across 11 benchmarks is only 0.53s, substantially lower than Grad.-Norm (7.84s), Grad.-Gate (8.76s), and Taylor (9.18s).

These results demonstrate that task-aware semantic matching provides a more effective and computationally efficient criterion for identifying adaptation-relevant attention heads than gradient- or parameter-based importance estimation on downstream support sets.

Revisions in the manuscript: Section 4.4, Ablation on Selection and Adaptation Design (Table 8, Page 9). We have added the gradient-based and Taylor-based selection comparisons, together with their computational costs.


Q5: Why are SAHR's gains much weaker for MSCOCO retrieval than for few-shot classification?

A5: Thank you for raising this important question. We believe that this difference is related to the distinct semantic requirements of classification and image-text retrieval.

For few-shot classification, task semantics are primarily organized around class-level visual concepts, such as object categories, attributes, parts, and textures. SAHR matches these task-aware semantic descriptions with attention-head profiles to identify heads that capture task-relevant visual characteristics. In contrast, MSCOCO retrieval requires instance-level cross-modal alignment between images and free-form captions, which often involve multiple objects, fine-grained attributes, and compositional relationships.

Nevertheless, SAHR achieves competitive retrieval performance on both Flickr30K and MSCOCO, as summarized below.

R2-Table 5: Representative image-text retrieval results on Flickr30K and MSCOCO.

DatasetDirectionMetricSAHRBest PEFT Baseline (Score; Params)
Flickr30KTRR@1099.4MaPLe (99.1; 4.8M)
Flickr30KIRR@592.1MaPLe (91.9; 4.8M)
MSCOCOIRR@143.5MaPLe (43.4; 4.8M)
MSCOCOIRR@1080.7Partial (81.5; 10.2M)
MSCOCOTRR@157.6MaPLe (59.8; 4.8M)
MSCOCOTRR@582.4Partial (83.3; 10.2M)
MSCOCOTRR@1090.0Partial (90.5; 10.2M)

Importantly, SAHR achieves these results with only 1.18M trainable parameters, compared with 4.8M for MaPLe and 10.2M for Partial, reducing the trainable parameter count by 75.4% and 88.4%, respectively. These results demonstrate that SAHR achieves a favorable balance between retrieval performance and parameter efficiency.


Q6: Show several representative failure cases: what tasks degrade SAHR's semantic matching?

A6: Our analysis suggests that semantic matching is less reliable for fine-grained recognition tasks whose discriminative evidence is concentrated in subtle local visual attributes. For example, distinguishing closely related aircraft variants often requires identifying small structural differences in wing shapes, engine configurations, tail designs, and fuselage details. This is reflected in confusions between visually similar aircraft variants, such as **Boeing 747-400 and 747-300, as well as CRJ-700 and CRJ-900. Such distinctions are difficult to fully characterize through general text-aligned semantic profiles, which may instead emphasize broader object-level or scene-level concepts.


Q7: Given that each key component has been studied in prior literature, what distinct technical innovation does this work bring beyond a heuristic combination of existing techniques?

A7: Thank you for raising this important question. We agree that several individual components used in SAHR, such as LLM-generated descriptions, text embeddings, semantic similarity, and parameter-efficient fine-tuning, have been explored in prior studies. However, the contribution of SAHR lies not in introducing these components independently, but in formulating a unified framework that quantifies the semantic correspondence between attention heads and downstream tasks, and uses this correspondence to guide selective head adaptation.

(1) Contribution-aware semantic profiling of attention heads: Constructing weighted semantic profiles that characterize the semantic specialization of individual attention heads.

(2) Task-relative semantic specialization scoring: Identifying task-specific attention heads by measuring their semantic correspondence to the target task relative to their relevance across other tasks.

(3) Structured sparse head adaptation: Translating task-aware head rankings into selective Q/K/V/O parameter updates while keeping the remaining parameters frozen.

Taken together, these components form a coherent adaptation mechanism rather than an independent combination of existing techniques: head contributions are decomposed into weighted semantic profiles, these profiles are evaluated against task-specific semantic descriptions, and the resulting task-relative rankings directly control structured parameter updates. The central innovation is therefore the use of head-level semantic specialization as an explicit and reusable control signal for parameter-efficient adaptation.


References

[1] Are Sixteen Heads Really Better than One? Michel et al., NeurIPS 2019.

[2] Importance Estimation for Neural Network Pruning. Molchanov et al., CVPR 2019.


We hope that our responses have adequately addressed your concerns through the additional experiments, analyses, and clarifications provided. We would be grateful if you would consider revising your rating in light of these additional results and explanations. Thank you once again for your time and effort.


Reviewer_xKtzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Semantics-Aware Head Reconfiguration, a method that enables the direct updating of task-specific attention heads during the adaptation of CLIP. The authors validate the effectiveness of this approach across three benchmarks.

Critical strengths
  1. The paper presents a clear line of reasoning; the motivation is sound, and the writing is of high quality.

  2. The approach of fine-tuning attention heads appears logical and demonstrates strong performance across multiple benchmarks.

Critical weaknesses
  1. How does this method compare to full fine-tuning? Furthermore, what do the attention heads look like after fine-tuning—are there any visualizations available?

  2. The performance gains reported are relatively modest. Notably, the improvement on ImageNet is quite limited—in fact, it underperforms compared to other methods. Does this suggest that the method’s effectiveness diminishes as the dataset size increases? Given that larger datasets often involve overlapping concepts, does this approach struggle with fine-grained recognition tasks?

Questions for authors

My primary concern relates to the method's generalizability. While it improves performance in few-shot scenarios, does it manage to preserve the original model's zero-shot capabilities? How does the method perform in "base-to-novel" generalization experiments? If the generalization performance can be verified, I would be willing to increase my score.

5 comments

Comment:

We sincerely thank the reviewer for the detailed and constructive feedback. We are encouraged that the reviewer recognizes our proposed Semantics-Aware Head Reconfiguration (SAHR) approach as a meaningful and practically impactful direction.

In response to the reviewer's concerns, we have conducted additional experiments and analyses, including comparisons with full fine-tuning methods, visualizations of attention heads before and after adaptation, and comprehensive base-to-novel generalization evaluations. We have carefully revised the manuscript to address the raised concerns and further strengthen the empirical evidence supporting our method. We sincerely appreciate the reviewer's willingness to increase the score if the generalization performance can be verified, and we hope that our additional experiments adequately address this concern.


Q1: How does this method compare to full fine-tuning?

A1: Thank you for raising this important question. To provide a direct comparison, we have added two full fine-tuning baselines: full visual-encoder fine-tuning (Visual FT) and joint full CLIP fine-tuning (Joint CLIP FT). All methods are evaluated using CLIP ViT-B/16 under five few-shot settings.

R1-Table 1: Comparison with full fine-tuning under five few-shot settings.

MethodParams1-shot2-shot4-shot8-shot16-shot
Visual FT86.19M66.471.474.677.980.7
Joint CLIP FT149.62M69.973.275.078.782.0
SAHR (Ours)1.18M72.274.877.880.483.1

SAHR achieves the highest accuracy across all five shot settings, while updating only 1.18M parameters, compared with 86.19M and 149.62M for the two full fine-tuning baselines, respectively. This corresponds to reductions of 98.6% and 99.2% in trainable parameters. These results demonstrate that SAHR achieves superior few-shot classification performance while maintaining high parameter efficiency through selective head adaptation.

Revisions in the manuscript:Section 4.3, Table 5 (Page 9).


Q2: Furthermore, what do the attention heads look like after fine-tuning—are there any visualizations available?

A2: Thank you for this helpful suggestion. We have added visualizations of representative attention heads before and after SAHR fine-tuning to provide a qualitative view of how the selected heads change during adaptation.

R1-Figure 1: Visualization of attention maps before and after SAHR fine-tuning.

The visualization covers representative heads from different layers using flower and cat images, with their corresponding layer and head indices explicitly indicated. Compared with pretrained CLIP, the fine-tuned heads exhibit changes in their spatial attention patterns, with responses becoming more focused on task-relevant visual regions. These visualizations provide qualitative evidence of how SAHR reconfigures selected attention heads to better capture task-relevant visual cues.

Revisions in the manuscript: Section 4.5, Figure 7 (Page 10). We have also added a corresponding discussion of the attention map changes.


Q3: The performance gains reported are relatively modest. Notably, the improvement on ImageNet is quite limited—in fact, it underperforms compared to other methods. Does this suggest that the method's effectiveness diminishes as the dataset size increases? Given that larger datasets often involve overlapping concepts, does this approach struggle with fine-grained recognition tasks?

A3:Thank you for raising these important concerns. We agree that SAHR does not achieve the best performance on every individual benchmark. However, its effectiveness should be assessed across different downstream tasks rather than based on ImageNet alone. More importantly, across all 11 benchmarks, SAHR achieves the highest average accuracy of 83.1%, outperforming TOGA (82.3%) by 0.8 percentage points. These results suggest that the effectiveness of SAHR depends on the semantic characteristics of downstream tasks. This advantage stems from SAHR's semantics-aware adaptation mechanism, which identifies and selectively updates attention heads whose semantic profiles are most relevant to the target task.


Comment:

Q4: My primary concern relates to the method's generalizability. While it improves performance in few-shot scenarios, does it manage to preserve the original model's zero-shot capabilities? How does the method perform in "base-to-novel" generalization experiments? If the generalization performance can be verified, I would be willing to increase my score?

A4: We sincerely thank the reviewer for highlighting this important concern. We agree that evaluating few-shot performance alone is insufficient, and that it is essential to examine whether adaptation preserves the pretrained model's ability to recognize unseen classes. To address this concern, we have conducted additional base-to-novel generalization experiments on all 11 benchmark datasets under the 16-shot setting.

Following the standard base-to-novel protocol, each dataset is divided into disjoint base and novel classes. Only 16 labeled training images per base class are used for adaptation, while novel classes are excluded from training. Importantly, task-aware semantic descriptions for head selection are constructed using base classes only.

R1-Table 2: Comparison of base-to-novel generalization performance across 11 benchmark datasets.

MethodBase (%)Novel (%)HM (%)
CLIP69.3474.2271.70
CoOp82.6963.2271.66
CoCoOp80.4771.6975.83
TextRefiner79.7474.3276.94
SAHR (Ours)84.3573.5378.57

The results show that SAHR achieves the highest Base accuracy (84.35%) and HM (78.57%) among the compared methods, while maintaining a Novel accuracy of 73.53%, close to the zero-shot CLIP result (74.22%).

These results demonstrate that SAHR achieves a favorable balance between adaptation to seen classes and generalization to unseen classes. In particular, SAHR substantially improves few-shot adaptation while largely retaining the pretrained CLIP model's zero-shot recognition capability on unseen classes.

We have also provided complete per-dataset Base, Novel, and HM results across all 11 benchmarks in Appendix C to further validate the generalization behavior of SAHR.

Revisions in the manuscript: Section 4.3, Table 2 (Page 7), and Appendix C, Table 12 (Page 18). We have added the base-to-novel evaluation protocol, quantitative results, and corresponding analysis.


We hope that our additional experiments and clarifications have adequately addressed the reviewer's concerns and strengthened the evidence supporting SAHR. We are particularly grateful for the reviewer's willingness to reconsider the score in light of our additional results. Thank you once again for your time and effort. We look forward to incorporating these helpful suggestions and discussions into the final camera-ready version.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the Area Chair and all three reviewers for their thoughtful and constructive feedback and continued consideration of our work. We are encouraged by the reviewers' positive follow-up feedback and their willingness to update their evaluations after considering our additional experiments and clarifications.

During the rebuttal, we have carefully addressed the concerns regarding methodological novelty, generalization, robustness, and computational efficiency. We have also revised the manuscript with additional experiments and analyses to strengthen the evidence supporting SAHR. Below, we summarize our core technical contributions and the major updates made in response to the reviews.


Core Contributions of Our Work

  1. Contribution-Aware Semantic Profiling: SAHR constructs weighted, text-aligned semantic profiles by decomposing attention-head contributions into complementary semantic directions.
  2. Task-Relative Semantic Specialization: SAHR introduces the Specialty Score to quantify the semantic correspondence between downstream tasks and attention heads, distinguishing task-specific relevance from general cross-task relevance.
  3. Semantics-Guided Sparse Adaptation: SAHR translates task-relative semantic rankings into selective Q/K/V/O parameter updates, enabling task-specific head reconfiguration while freezing the remaining parameters.

Updates of Experimental Results during Rebuttal

  • Main Text, Section 4.3: Added comparisons with full visual-encoder fine-tuning and joint full CLIP fine-tuning under five few-shot settings (Table 5, R1).
  • Main Text, Section 4.3: Added base-to-novel generalization experiments across 11 benchmarks, evaluating adaptation to seen classes and generalization to unseen classes (Table 2, R1/R3).
  • Main Text, Section 4.5: Added visualizations of representative attention heads before and after SAHR adaptation (Figure 7, R1).
  • Main Text, Section 4.4: Added sensitivity analysis of the semantic profile size MM, examining classification accuracy and computational cost (Table 10, R2).
  • Main Text, Section 4.4: Added gradient-based and Taylor-based head selection comparisons, including selection costs (Table 8, R2/R3).
  • Appendix E.2: Added profiling dataset size ablations using 10K, 50K, and 100K images, including head selection consistency and offline profiling costs (Table 16, R2).
  • Appendix E.2: Added robustness evaluations with GPT-5.6 Sol, GPT-6.1 Sol, and Llama 3.1 (Table 17, R2).
  • Appendix E.3: Clarified the one-time offline profiling overhead and reuse of semantic profiles across downstream tasks (R2).

We believe these additions and clarifications comprehensively address the reviewers' concerns and enhance the overall quality of our manuscript. All revisions are highlighted in blue-colored text for ease of reference.

We are grateful to the Area Chair and reviewers for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.