- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SAHR, a parameter-efficient approach to adapting CLIP by fine-tuning task-relevant attention heads. The key idea is to use the semantic specialization of attention heads as a signal for deciding which parameters to update. The authors construct text-aligned semantic profiles for late-layer attention heads using unlabeled images and general visual descriptions. They then generate task-specific descriptions with an LLM and compute a specialty score to rank heads according to their relevance to the target task. Only the Q/K/V/O projection slices associated with the top-ranked heads are updated. The method is evaluated on 11 few-shot classification benchmarks, Flickr30K and MSCOCO image-text retrieval, and OOD generalization benchmarks.
Critical strengths
- The motivation behind SAHR reasonable. This paper provides a concrete way to turn head-level semantic profiles into a selection criterion for fine-tuning.
- The experiments cover a diverse set of downstream tasks. The ablations in Table 6 are also helpful to understand the method.
Critical weaknesses
- Limited evidence of novelty and practical advantages over existing adaptation strategies. The main components, including text-based head interpretation and selective head adaptation, are closely related to prior work. The proposed Specialty Score appears to be the main methodological addition.
- The Specialty Score relies on a predefined collection of benchmark tasks. It is unclear whether the selected heads would remain consistent when this collection changes or when adapting to an unseen task.
Questions for authors
None
3 comments
Q2: The Specialty Score relies on a predefined collection of benchmark tasks. It is unclear whether the selected heads would remain consistent when this collection changes or when adapting to an unseen task.
A2: Thank you for raising this important concern. We adopt 11 widely used benchmark datasets in few-shot CLIP adaptation to cover diverse visual domains and provide a comprehensive evaluation across different downstream tasks. Accordingly, these datasets constitute the reference task collection for computing task-relative normalization in the Specialty Score. However, this collection serves only as a reference for measuring relative task specificity, rather than restricting the downstream tasks to which SAHR can be applied.
For a new downstream task, SAHR constructs task-aware semantic descriptions and computes the corresponding head–task matching scores. The Specialty Score then compares the relevance to the target task against the average relevance over the reference task collection. Thus, a new target task does not need to be included in the predefined reference collection, as its semantic descriptions can be used to calculate its relevance to existing head profiles.
Generalization to unseen classes. We have also evaluated SAHR under the base-to-novel protocol across 11 benchmarks. During adaptation, only base-class descriptions are used for head selection, while novel classes are excluded from the adaptation process. The results are summarized below.
R3-Table 3: Base-to-novel generalization performance across 11 benchmarks.
| Method | Base (%) | Novel (%) | HM (%) |
|---|---|---|---|
| CLIP | 69.34 | 74.22 | 71.70 |
| CoOp | 82.69 | 63.22 | 71.66 |
| CoCoOp | 80.47 | 71.69 | 75.83 |
| TextRefiner | 79.74 | 74.32 | 76.94 |
| SAHR (Ours) | 84.35 | 73.53 | 78.57 |
SAHR achieves the highest Base accuracy of 84.35% and HM of 78.57%, while maintaining a Novel accuracy of 73.53%, close to the zero-shot CLIP result of 74.22%. These results provide complementary evidence that SAHR can adapt to observed classes while largely retaining its ability to recognize unseen classes.
Revisions in the manuscript: Section 3.2 and Section 4.3 (Table 2, Page 7), with detailed per-dataset base-to-novel results in Appendix C (Table 12, Page 18).
We are pleased that the reviewer has an overall positive view of our work and sincerely appreciate the thoughtful feedback. We hope that our responses and additional clarifications have adequately addressed the reviewer's concerns. Thank you once again for your time and effort.
Thank for your reply. I will keep my positive rating.
We sincerely thank Reviewer umaT for the positive and insightful feedback. We appreciate the reviewer's recognition of the methodological rationale of our proposed Semantics-Aware Head Reconfiguration (SAHR) framework, particularly the use of semantic profiles to guide task-specific attention head selection. We are encouraged that the reviewer finds our experimental evaluation comprehensive and the overall method design reasonable.
We also appreciate the reviewer's thoughtful concerns regarding the technical novelty of SAHR and the generalizability of the Specialty Score under different task collections. These are important aspects of our method, and we have carefully considered them in our response. We provide further clarification of the technical contributions of SAHR and additional analysis to address the robustness of task-aware head selection.
Q1: Limited evidence of novelty and practical advantages over existing adaptation strategies. The main components, including text-based head interpretation and selective head adaptation, are closely related to prior work. The proposed Specialty Score appears to be the main methodological addition.
A1: Thank you for this insightful comment. We agree that text-based head interpretation and selective parameter adaptation have been explored in prior studies. However, SAHR's contribution lies in explicitly quantifying the semantic correspondence between downstream task characteristics and the functional specialization of individual attention heads, and leveraging this correspondence to guide task-specific parameter adaptation through three closely integrated stages.
(1) Contribution-aware semantic profiling. Rather than simply assigning textual descriptions to attention heads, SAHR constructs weighted semantic profiles by decomposing head contributions along text-aligned directions. This provides a structured representation of the complementary semantic factors captured by individual heads.
(2) Task-relative semantic specialization scoring. This is the central methodological contribution of SAHR. Raw semantic similarity alone cannot distinguish heads that are specifically relevant to a target task from those that are broadly relevant across multiple tasks. SAHR addresses this through the Specialty Score, which normalizes each head's task relevance against its average relevance across the task collection. This explicitly quantifies task-specific specialization and provides a criterion for ranking adaptation-relevant heads.
(3) Structured sparse head adaptation. SAHR translates the resulting task-specific ranking into structured parameter updates, selectively adapting the Q/K/V/O parameter slices of the selected heads while freezing the remaining parameters. Thus, semantic interpretation directly determines where adaptation capacity is allocated.
The practical advantages are also supported by our ablation results. Removing Specialty Score normalization reduces the average 16-shot accuracy from 83.1% to 82.1%, demonstrating the importance of distinguishing task-specific relevance from general relevance.
Furthermore, compared with gradient-based and Taylor-based head selection, SAHR achieves the highest accuracy across all five shot settings while requiring substantially less selection time.
These results further demonstrate that both task-relative semantic scoring and structured Q/K/V/O adaptation contribute to the effectiveness of SAHR. Therefore, the novelty of SAHR is not the isolated use of text-based interpretation or sparse adaptation, but the integration of contribution-aware semantic profiling and task-relative specialization scoring into an explicit mechanism for structured, task-specific head adaptation. This connection transforms head interpretability from a descriptive analysis into an actionable signal, enabling semantics-guided, task-specific and parameter-efficient reconfiguration of attention heads.
Revisions in the manuscript: Sections 2 and 3.1–3.3, as well as Section 4.4 (Tables 8 and 9). We have further clarified the technical contributions of SAHR and the respective roles of semantic profiling, Specialty Score, and structured sparse adaptation.