- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors try to solve the problem of how to quickly transfer knowledge between different tasks through finetuning with limited support images and labels.
Critical strengths
The authors try to solve a fundamental problem in VLM for few-shot learning. They aim to reduce computational cost and the number of learnable parameters for fine-tuning with limited labeled samples. Compared to basic LoRA structures, the proposed method reduces the large number of learnable parameters and maintains comparable performance for multiple downstream tasks.
Critical weaknesses
The writing needs significant improvement, as both the motivation and the proposed method are difficult to follow. Many equations are presented without clear definitions of the corresponding notations, making it challenging for readers to understand the underlying methodology. Furthermore, the connection between the motivation and the proposed method is not clearly established, making it difficult to understand how the proposed approach addresses the identified problems.
Questions for authors
- I find the examples provided in the first paragraph of the Introduction somewhat confusing. The examples of the pathology system and robot seem more closely related to cross-domain generalization problems rather than few-shot learning. In general, few-shot learning aims to enable a model to make predictions on novel tasks or classes using a limited number of labeled support samples, which provide guidance for the model's predictions. However, the proposed method appears to focus on addressing the challenges arising from training with limited labeled data, rather than leveraging support samples to guide predictions. Therefore, the motivation and problem formulation seem somewhat inconsistent with the typical few-shot learning setting. The authors should clarify the specific problem they aim to address and explain how it relates to few-shot learning.
- Based on the second paragraph of the Introduction, it seems that the authors aim to adapt existing vision-language models (VLMs) using parameter-efficient fine-tuning (PEFT) with a limited number of labeled samples.
- In the fourth paragraph of the Introduction, what exactly do the authors mean by “visual K/V states”? If they refer to the key/value (K/V) states in the attention mechanism, it should be noted that VLMs typically contain multiple attention layers, each with its own K/V states. Therefore, it is unclear which specific K/V states the authors are referring to. The authors should provide a clearer definition and specify which components or layers of the VLM are involved.
- For equations and notations in the Method, they are quite difficult to follow. For example, what are xi, yi, and N? Does xi denote images or something else? Does yi denote a mask or an image-level label? Does N denote the number of support samples? Moreover, what is Mi and how to derive Mi. Moreover, you state that each input contains a question. However, there are no questions in Fig.2, and the question is no longer mentioned in the following method descriptions. What is the role of the question?
- The same problems also appear in sections 3.2 and 3.3. A lot of notations lack definitions. Thus, it is difficult for me to follow the method process. As the unclear description of how to derive the K/V states, I cannot link the equations with the K/V states. How can those equations identify the visual K/V direction that affects the correct-answer loss? How can the visual K/V directions be linked to the support samples?
- It would be helpful to provide an overview of the proposed method at the beginning of Section 3, before introducing the individual components. This would allow readers to better understand the overall workflow and how the different components are connected.
5 comments
Thank you for the detailed response and substantial revisions. The authors have effectively addressed my concerns regarding the manuscript's clarity and presentation, particularly in explaining the overall framework and methodological details.
However, I remain concerned about the connection between the proposed K/V-state intervention and its effectiveness across different visual domains. While the experimental results demonstrate substantial parameter efficiency and competitive performance, the underlying motivation remains insufficiently justified. In particular, it remains unclear why modifying the decoder K/V representations would effectively address visual domain shifts, given that there are many components that can be modified within the transformer architecture. Why do the authors try to just modify the K/V-state? Thus, I keep my original rating.
Why intervene in decoder K/V states?
Thank you for the follow-up and for recognizing the revisions. To be hoest, our motivation comes from looking at the problem a little differently. From our perspective, pretrained VLMs already have strong general visual recognition capabilities. However, when adapting to a new hospital or disaster dataset, unfamiliar visual patterns may not be well aligned with the new dataset's labels, introducing irrelevant or misleading information into the visual representations used by the decoder. We believe that directly adjusting the K/V states is a more targeted and efficient way to reduce such noise, rather than updating model weights to learn how to handle these new patterns, which can be expensive with only a few labeled examples. This is why we focus on visual K/V states rather than MLP weight updates: keys influence which visual evidence later answer tokens attend to, while values determine what information that evidence contributes. APERTURE uses a few labeled examples to adjust both, helping the decoder rely less on irrelevant visual evidence while keeping the pretrained weights fixed.
The two interventions are complementary. K-only can redirect attention but leaves the attended values unchanged; V-only can change the contributed information but does not directly redirect attention. In Table 4, K-only outperforms V-only on Hawaii, Libya, and UMCU (27.71/31.05/65.78 versus 18.43/27.71/49.05 macro-F1). The best joint K/V configurations reach 28.84/32.56/68.01 on these tasks (Uniform for Hawaii and Libya; Gated for UMCU). Modifying visual-token Q would primarily affect how those tokens query their context, rather than directly changing how later answer tokens access visual evidence.
We therefore see K/V as a direct and targeted interface for this goal. We hope this perspective offers a fresh way of thinking about visual domain adaptation!
Thank you for identifying the gaps between the original motivation, notation, and implementation. We have substantially revised the Introduction (Section 1), reorganized the method presentation (Sections 3.1–3.4), simplified Figure 2, and expanded the derivations and implementation details in Appendices A and B. The paper now explains the complete path from a few labeled target-domain examples to a reusable visual-state correction before presenting the detailed equations.
The problem is supervised few-shot target-domain adaptation, not zero-shot domain generalization (Introduction; Section 3.1; Appendix B.1). Each target hospital, disaster event, or robot decision task provides a small labeled support set. We use the fitting labels to learn one domain-specific controller and evaluate it on separate target-domain queries, without optimizing on those queries. For example, each CAMELYON17 hospital uses 12 fitting examples, 4 calibration examples, and 500 held-out queries (Table 1). The support data are therefore the supervision that determines the adaptation, not simply examples illustrating a distribution shift.
The K/V intervention and its layer locations are now described explicitly (Sections 3.1 and 3.4; Appendix B.3). The intervened states are the outputs of the key and value projection modules in selected attention blocks of the language decoder—not a single monolithic KV cache and not the pretrained projection weights. We index each site by a decoder layer and projection kind, j = (layer, K/V), and intervene only at eligible visual-token positions. The general implementation supports a depth-relative middle/final-layer convention: for a decoder with L layers (zero-based indexing), the two layers are floor(L/2) and L−1, with K and V at both sites. Thus, for the 28-layer Qwen2.5-VL-7B, the locations are 14 and 27. The exact layer indices for an experiment must follow its backbone and saved configuration, rather than treating 14 and 27 as universal identifiers. Choosing intermediate and late decoder sites covers stages of visual–language integration and final-answer formation identified in prior VLM analyses (Zhang et al., CVPR 2025; Kaduri et al., CVPR 2025).
All inputs, masks, and learning stages are connected to the equations (Sections 3.1–3.4; Figure 2; Appendix A). We define S_e = {(x_i, y_i)} with x_i = (v_i, q_i): v_i is the image input, q_i is its question/instruction, and y_i is the correct answer. The question conditions the answer-token likelihood and therefore its loss and gradients. The visual-token mask is constructed from positions identified by the multimodal processor, selecting the post-event image tokens for paired disaster images and the image tokens for single-image tasks. Stage I differentiates the correct-answer loss with respect to those selected K/V outputs, forms the uncentered second moment of the resulting gradients, and extracts its leading rank-r directions. Stage II fixes this basis and learns a bounded signed residual; Uniform applies a shared subspace operator, whereas Gated learns token-conditioned modulation. Figure 2 presents these steps in order, and Appendix A provides the full derivation and algorithm.
Expanded experiments connect the clarified mechanism to measurable results (Section 4; Tables 1–5; Appendix C). The revised Table 1 covers 12 hospital–backbone settings: an APERTURE variant leads in 11/12, and Uniform achieves 54.26% mean macro-F1 versus 42.53% for LoRA. Lower variability across support draws is an additional empirical finding: Uniform/Gated have lower SD than LoRA in 8/12 and 9/12 comparisons. Table 2 examines 16, 32, and 64 labels against calibration-selected LoRA ranks; Table 3 extends the training-budget comparison to 128 epochs. Most directly, Table 4 tests the gradient-selected directions: on UMCU, Gated reaches 68.01%, compared with 62.60% using a random basis and 44.55% using activation PCA. Appendix C adds further visual tasks and controller-hyperparameter sensitivity.
In summary, the revised manuscript states exactly what is learned from the labeled support set, where the visual K/V intervention is applied, why the question and visual mask enter the gradient computation, and how the two learning stages produce a reusable controller. The expanded evaluation then links that mechanism to cross-backbone accuracy, few-shot support consistency, and deployment efficiency. The General Response contains the complete revised tables and fine-tuning protocols.