acceptodds
Under review as a conference paper at ICLR 2027

APERTURE: Few-Shot Refocusing of Visual Evidence in Vision–Language Models

Abstract

In a new visual domain, a vision–language model may need to make many predictions using only a few labeled images. This situation arises, for example, when identifying tumor tissue at a new hospital or assessing building damage after a new disaster. We introduce APERTURE, a small, reusable module that adjusts image-token key/value states while keeping the backbone frozen. Gradients of the correct-answer loss on labeled support images identify sensitive directions in these states. The module learns a bounded transformation within those directions and applies it to each new image’s activations, without demonstrations or further optimization. Across four BRIGHT events and three support sets per event, APERTURE increases Qwen2.5-VL’s mean macro-F1 from 22.55% to 31.33% using only 1,024 learned coefficients, compared with 31.46% for LoRA-1E. On CAMELYON17–H2, it achieves lower mean raw candidate negative log-likelihood than LoRA-1E and LoRA-4E across the three evaluated Qwen model variants. In the measured Qwen2.5-VL configuration, its saved adaptation state, including the fixed bases, occupies approximately 1/291 of the storage required by rank-16 LoRA. These results show that APERTURE can achieve performance close to LoRA-1E on BRIGHT with substantially less adaptation storage, while its benefits vary across tasks and models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision–language models, few-shot adaptation, visual-state adaptation, parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SXxoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors try to solve the problem of how to quickly transfer knowledge between different tasks through finetuning with limited support images and labels.

Critical strengths

The authors try to solve a fundamental problem in VLM for few-shot learning. They aim to reduce computational cost and the number of learnable parameters for fine-tuning with limited labeled samples. Compared to basic LoRA structures, the proposed method reduces the large number of learnable parameters and maintains comparable performance for multiple downstream tasks.

Critical weaknesses

The writing needs significant improvement, as both the motivation and the proposed method are difficult to follow. Many equations are presented without clear definitions of the corresponding notations, making it challenging for readers to understand the underlying methodology. Furthermore, the connection between the motivation and the proposed method is not clearly established, making it difficult to understand how the proposed approach addresses the identified problems.

Questions for authors
  1. I find the examples provided in the first paragraph of the Introduction somewhat confusing. The examples of the pathology system and robot seem more closely related to cross-domain generalization problems rather than few-shot learning. In general, few-shot learning aims to enable a model to make predictions on novel tasks or classes using a limited number of labeled support samples, which provide guidance for the model's predictions. However, the proposed method appears to focus on addressing the challenges arising from training with limited labeled data, rather than leveraging support samples to guide predictions. Therefore, the motivation and problem formulation seem somewhat inconsistent with the typical few-shot learning setting. The authors should clarify the specific problem they aim to address and explain how it relates to few-shot learning.
  2. Based on the second paragraph of the Introduction, it seems that the authors aim to adapt existing vision-language models (VLMs) using parameter-efficient fine-tuning (PEFT) with a limited number of labeled samples.
  3. In the fourth paragraph of the Introduction, what exactly do the authors mean by “visual K/V states”? If they refer to the key/value (K/V) states in the attention mechanism, it should be noted that VLMs typically contain multiple attention layers, each with its own K/V states. Therefore, it is unclear which specific K/V states the authors are referring to. The authors should provide a clearer definition and specify which components or layers of the VLM are involved.
  4. For equations and notations in the Method, they are quite difficult to follow. For example, what are xi, yi, and N? Does xi denote images or something else? Does yi denote a mask or an image-level label? Does N denote the number of support samples? Moreover, what is Mi and how to derive Mi. Moreover, you state that each input contains a question. However, there are no questions in Fig.2, and the question is no longer mentioned in the following method descriptions. What is the role of the question?
  5. The same problems also appear in sections 3.2 and 3.3. A lot of notations lack definitions. Thus, it is difficult for me to follow the method process. As the unclear description of how to derive the K/V states, I cannot link the equations with the K/V states. How can those equations identify the visual K/V direction that affects the correct-answer loss? How can the visual K/V directions be linked to the support samples?
  6. It would be helpful to provide an overview of the proposed method at the beginning of Section 3, before introducing the individual components. This would allow readers to better understand the overall workflow and how the different components are connected.
5 comments

Thank you for identifying the gaps between the original motivation, notation, and implementation. We have substantially revised the Introduction (Section 1), reorganized the method presentation (Sections 3.1–3.4), simplified Figure 2, and expanded the derivations and implementation details in Appendices A and B. The paper now explains the complete path from a few labeled target-domain examples to a reusable visual-state correction before presenting the detailed equations.

  1. The problem is supervised few-shot target-domain adaptation, not zero-shot domain generalization (Introduction; Section 3.1; Appendix B.1). Each target hospital, disaster event, or robot decision task provides a small labeled support set. We use the fitting labels to learn one domain-specific controller and evaluate it on separate target-domain queries, without optimizing on those queries. For example, each CAMELYON17 hospital uses 12 fitting examples, 4 calibration examples, and 500 held-out queries (Table 1). The support data are therefore the supervision that determines the adaptation, not simply examples illustrating a distribution shift.

  2. The K/V intervention and its layer locations are now described explicitly (Sections 3.1 and 3.4; Appendix B.3). The intervened states are the outputs of the key and value projection modules in selected attention blocks of the language decoder—not a single monolithic KV cache and not the pretrained projection weights. We index each site by a decoder layer and projection kind, j = (layer, K/V), and intervene only at eligible visual-token positions. The general implementation supports a depth-relative middle/final-layer convention: for a decoder with L layers (zero-based indexing), the two layers are floor(L/2) and L−1, with K and V at both sites. Thus, for the 28-layer Qwen2.5-VL-7B, the locations are 14 and 27. The exact layer indices for an experiment must follow its backbone and saved configuration, rather than treating 14 and 27 as universal identifiers. Choosing intermediate and late decoder sites covers stages of visual–language integration and final-answer formation identified in prior VLM analyses (Zhang et al., CVPR 2025; Kaduri et al., CVPR 2025).

  3. All inputs, masks, and learning stages are connected to the equations (Sections 3.1–3.4; Figure 2; Appendix A). We define S_e = {(x_i, y_i)} with x_i = (v_i, q_i): v_i is the image input, q_i is its question/instruction, and y_i is the correct answer. The question conditions the answer-token likelihood and therefore its loss and gradients. The visual-token mask is constructed from positions identified by the multimodal processor, selecting the post-event image tokens for paired disaster images and the image tokens for single-image tasks. Stage I differentiates the correct-answer loss with respect to those selected K/V outputs, forms the uncentered second moment of the resulting gradients, and extracts its leading rank-r directions. Stage II fixes this basis and learns a bounded signed residual; Uniform applies a shared subspace operator, whereas Gated learns token-conditioned modulation. Figure 2 presents these steps in order, and Appendix A provides the full derivation and algorithm.

  4. Expanded experiments connect the clarified mechanism to measurable results (Section 4; Tables 1–5; Appendix C). The revised Table 1 covers 12 hospital–backbone settings: an APERTURE variant leads in 11/12, and Uniform achieves 54.26% mean macro-F1 versus 42.53% for LoRA. Lower variability across support draws is an additional empirical finding: Uniform/Gated have lower SD than LoRA in 8/12 and 9/12 comparisons. Table 2 examines 16, 32, and 64 labels against calibration-selected LoRA ranks; Table 3 extends the training-budget comparison to 128 epochs. Most directly, Table 4 tests the gradient-selected directions: on UMCU, Gated reaches 68.01%, compared with 62.60% using a random basis and 44.55% using activation PCA. Appendix C adds further visual tasks and controller-hyperparameter sensitivity.

In summary, the revised manuscript states exactly what is learned from the labeled support set, where the visual K/V intervention is applied, why the question and visual mask enter the gradient computation, and how the two learning stages produce a reusable controller. The expanded evaluation then links that mechanism to cross-backbone accuracy, few-shot support consistency, and deployment efficiency. The General Response contains the complete revised tables and fine-tuning protocols.

Thank you for the detailed response and substantial revisions. The authors have effectively addressed my concerns regarding the manuscript's clarity and presentation, particularly in explaining the overall framework and methodological details.

However, I remain concerned about the connection between the proposed K/V-state intervention and its effectiveness across different visual domains. While the experimental results demonstrate substantial parameter efficiency and competitive performance, the underlying motivation remains insufficiently justified. In particular, it remains unclear why modifying the decoder K/V representations would effectively address visual domain shifts, given that there are many components that can be modified within the transformer architecture. Why do the authors try to just modify the K/V-state? Thus, I keep my original rating.

AuthorsOpenReview ↗

Why intervene in decoder K/V states?

Thank you for the follow-up and for recognizing the revisions. To be hoest, our motivation comes from looking at the problem a little differently. From our perspective, pretrained VLMs already have strong general visual recognition capabilities. However, when adapting to a new hospital or disaster dataset, unfamiliar visual patterns may not be well aligned with the new dataset's labels, introducing irrelevant or misleading information into the visual representations used by the decoder. We believe that directly adjusting the K/V states is a more targeted and efficient way to reduce such noise, rather than updating model weights to learn how to handle these new patterns, which can be expensive with only a few labeled examples. This is why we focus on visual K/V states rather than MLP weight updates: keys influence which visual evidence later answer tokens attend to, while values determine what information that evidence contributes. APERTURE uses a few labeled examples to adjust both, helping the decoder rely less on irrelevant visual evidence while keeping the pretrained weights fixed.

The two interventions are complementary. K-only can redirect attention but leaves the attended values unchanged; V-only can change the contributed information but does not directly redirect attention. In Table 4, K-only outperforms V-only on Hawaii, Libya, and UMCU (27.71/31.05/65.78 versus 18.43/27.71/49.05 macro-F1). The best joint K/V configurations reach 28.84/32.56/68.01 on these tasks (Uniform for Hawaii and Libya; Gated for UMCU). Modifying visual-token Q would primarily affect how those tokens query their context, rather than directly changing how later answer tokens access visual evidence.

We therefore see K/V as a direct and targeted interface for this goal. We hope this perspective offers a fresh way of thinking about visual domain adaptation!

Reviewer_cJAmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

APERTURE proposes a reusable visual Key/Value state controller that identifies task-sensitive directions using a small number of labeled samples and learns bounded residual corrections along these directions. The paper evaluates the method across multiple vision-language model backbones. Beyond classification accuracy, it also reports negative log-likelihood of candidate answers, Brier scores, and storage/computational costs—demonstrating a highly detailed and rigorous approach.

Critical strengths
  1. APERTURE proposes a reusable visual Key/Value state controller that identifies task-sensitive directions using a small number of labeled samples and learns bounded residual corrections along these directions.

  2. The paper addresses the high cost of model parameter updates in few-shot visual domain adaptation; the research objective is clear and holds practical application value.

  3. The paper evaluates the method across multiple vision-language model backbones. Beyond classification accuracy, it also reports negative log-likelihood of candidate answers, Brier scores, and storage/computational costs—demonstrating a highly detailed and rigorous approach.

Critical weaknesses
  1. The paper primarily compares its method against LoRA, random-direction approaches, and simple output calibration; it should incorporate additional, more robust baselines for comparison.

  2. There is a lack of systematic evaluation using larger support sets. Since the method is primarily designed for extremely low-shot adaptation, it remains unclear whether APERTURE would be outperformed by more extensively trained LoRA or other parameter-efficient fine-tuning methods as the number of labeled samples increases.

  3. The scope is limited to classification tasks; current evaluations rely on a fixed set of candidate answers and classification probabilities. The method’s efficacy for open-ended VQA, complex reasoning, and multi-turn dialogue has yet to be verified.

  4. As shown in Table 3, the fitting time for APERTURE is 22.14 seconds, which is significantly higher than the 7.92 seconds required for LoRA-1E. Consequently, it offers no advantage in terms of computational overhead.

Questions for authors

For further details, please refer to the "Critical Weaknesses" section.

4 comments

Thank you for your comments on baseline strength, label budgets, computational cost, and evaluation scope. Our expanded results establish where APERTURE is most effective: accurate and consistent adaptation from scarce labels, strong performance at limited training budgets, and a much smaller reusable state. We address each concern directly below; the complete results appear in our General Response and revised Tables 1–5.

  1. Stronger LoRA baselines: APERTURE's cross-hospital advantage is not limited to a one-epoch LoRA comparison.

We expanded LoRA evaluation along three complementary axes. Table 1 covers ranks {1, 4, 16} and training epochs {2, 4, 8} across three support seeds. Table 2 independently tunes ranks 4 and 16 over three learning rates and {4, 16} epochs, selecting each rank's configuration by calibration NLL. Table 3 extends rank-16 training from 1 to 128 epochs across three datasets. Against the Table 1 references, APERTURE achieves the highest macro-F1 in 11/12 backbone–hospital settings, with Uniform averaging 54.26% versus 42.53% for LoRA. These distinct experiments test both stronger calibrated LoRA and the effect of substantially longer optimization.

  1. More labels: APERTURE retains a clear advantage in low-label accuracy and support-seed consistency.

Table 2 explicitly evaluates 16, 32, and 64 labels, with APERTURE leading 6/12 calibrated-LoRA comparisons. With only 16 labels on CWZ/Gemma-3, Gated achieves 56.47 ± 0.33, compared with 50.20 ± 14.15 for LoRA-r16 and 48.01 ± 14.77 for LoRA-r4. At 32 labels on the same model and hospital, Gated achieves 56.96 ± 0.97 versus 49.16 ± 6.32 for the stronger LoRA rank. At 64 labels on UMCU/Qwen2.5-VL, Uniform obtains 64.80 ± 0.47 versus 64.48 ± 19.58 (LoRA-r4) and 64.37 ± 19.28 (LoRA-r16). The broader cross-hospital results in Table 1 show the same consistency pattern: average row-wise SD is 6.40 for Uniform and 5.20 for Gated, compared with 12.94 for LoRA. Calibration-tuned LoRA also has a clear advantage in some settings—for example, 72.93% versus APERTURE's best 64.90% at 32 labels on UMCU/Qwen2.5-VL—identifying where weight-space adaptation is more effective.

  1. Training cost: APERTURE provides stronger early adaptation in some tasks and substantially lower deployment storage, rather than the fastest one-epoch fit.

Table 3 shows that on UMCU, Gated reaches 60.06% after one controller-training epoch versus 29.91% for one-epoch LoRA. On Libya, Gated reaches 37.82% versus LoRA's 33.84% at 16 epochs and 36.18% versus 32.51% at 128 epochs. For wall-clock fitting, Table 5 confirms that one-epoch LoRA is faster (Hawaii: 12.62 s versus Uniform's 38.22 s). Against four-epoch LoRA, however, Uniform takes 38.22 versus 50.03 s on Hawaii and 15.82 versus 24.90 s on UMCU, including basis extraction. The central deployment gain is storage: APERTURE needs only 0.140–0.142 MB of complete task-specific state, including fixed bases, versus 40.41 MB for rank-16 LoRA, a 285–290× reduction.

  1. Additional controls and open-ended evaluation: we expanded the mechanism comparisons and state the task scope explicitly.

Table 4 adds random and activation-PCA subspaces, fixed-half and learned-scalar gates, and K-only/V-only interventions. On UMCU, the full Gated method reaches 68.01%, compared with 62.60% for random directions and 44.55% for activation-PCA, connecting the gains to the gradient-selected visual K/V subspaces. These are mechanism controls; the expanded calibrated-LoRA studies provide the stronger weight-space comparison. Our quantitative evidence covers supervised visual decisions in pathology, disaster assessment, and robotics. Appendix B specifies a TextVQA protocol, but we do not present an open-ended VQA or multi-turn dialogue performance result as completed evidence.

In summary, the revision directly addresses the baseline and label-budget concerns with calibrated LoRA ranks, up to 64 labels, and training durations up to 128 epochs. The combined evidence identifies APERTURE's strongest use case: high-quality, lower-variance few-shot adaptation with a tiny reusable state, while explicitly showing the settings where longer or better-tuned LoRA achieves higher accuracy. The revised Experimental Setup, Appendix B, and General Response provide the full protocols and tables.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our response and let us know whether any further clarification is needed.

Kind regards, The authors of APERTURE

Thank you for providing the additional experiments; they address my primary concerns regarding the comparative experiments. However, some issues remain unresolved, and the benchmarking against other PEFT methods is still insufficient. Furthermore, the scope of the tasks is limited to classification and decision-making among a finite set of options. I will maintain my score of 3 (marginally acceptable).

Thank you for acknowledging that our additional experiments have addressed your primary concerns. We agree that broader comparisons with other PEFT methods and open-ended evaluations would be valuable. As LoRA is a widely adopted PEFT baseline for VLM adaptation, to strengthen the baselines, we have substantially strengthened our comparisons across different ranks, learning rates, training budgets, and visual domains. Besdies, our work focuses specifically on few-shot visual adaptation for classification and decision-making, where APERTURE demonstrates advantages in accuracy, consistency, and storage efficiency. We sincerely appreciate your constructive feedback and recognition of our substantial revisions, which have greatly strengthened the paper.

Reviewer_qmC7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose APERTURE, a way to learn task-specific corrections/improvements based on a small labels set of data. A novel correct-answer loss identifies which gradients are “sensitive”, which are then fixed and used to train a small adapter to strength, improve those portions know as support labels. This leads to the key contributions: a reusable K/V module honed on sensitive directions, and a thorough evaluation of the approach across various datasets, backbones, inference costs, training times, with thorough and well thought out ablations and mathematical formulation.

Critical strengths

The paper is well written, and the use-case and the domains, and formulations are meaningful and clear. The proposed approach excels at storage efficiency with a 1/291 required storage relative to LORA and improves upon NLL and Brier score for answer probabilities - performance maintains a similar performance on BRIGHT evaluation. The authors have put quite some thought into their formulation, and have laid out clear questions and tradeoffs of approaches throughout the narrative.

Critical weaknesses

The overall framing of the paper is a bit disconnected from the key measurable gains reported. The support-gradient leads the reader to believe that the module identifies the key portions, though this does not necessarily lead to accuracy improvements over LORA - instead, this quite literally focuses the architecture only needed a significantly smaller adaption storage footprint. On performance, results are mixed with 4-event mean being outperformed by LORA-1E with some cases regressing (Hawaii). On overall novelty, the work is narrow for a more general conference, and (as noted by the author) varies depending on task/how it is applied (training budget constraints, etc).

Questions for authors

Additional qualitative discussion and analysis of the performance tradeoff across datasets would be helpful to help reinforce to the reader why in some cases performance improves and/or regresses. For example, the Hawaii wildfire dataset seems to be a solid dataset where APERTURE helps, and Libya, being a case where the model struggles - discussion and analysis on this (why it improves here but say not as much on the other planet earth scans such as Libya, Noto, etc could help the reader better understand the empirical findings).

4 comments

Thank you for asking us to connect APERTURE's mechanism to measurable improvements and to explain the differences between disaster events. The expanded results identify where visual-state correction is most effective: adaptation with scarce labels, reduced variation across support draws, and compact reusable state.

  1. Accuracy and consistency beyond storage savings. APERTURE is not merely a smaller alternative to LoRA. In the revised Table 1, an APERTURE variant has the highest macro-F1 in 11 of 12 backbone–hospital settings. APERTURE-Uniform averages 54.26%, compared with 42.53% for LoRA and 34.76% for Frozen. Uniform and Gated also have lower support-seed SD than LoRA in 8/12 and 9/12 settings, respectively; the mean row-wise SDs are 6.40 and 5.20 points, versus 12.94 for LoRA. The mechanism ablation in Table 4 links these gains to the selected representation directions: on UMCU, Gated achieves 68.01%, compared with 62.60% using a random basis and 44.55% using an activation-PCA basis. The contribution is thus a support-gradient-selected visual K/V correction, with measurable gains in both predictive quality and support-draw consistency.

  2. Why Hawaii and Libya behave differently. The revised Table 3 tracks both methods from 1 to 128 training epochs, showing that their relative performance changes with the adaptation budget. On Libya, LoRA initially leads at 1 epoch (34.67% versus 27.65% for Gated), but APERTURE-Uniform overtakes LoRA at 8 epochs (36.05% versus 34.24%). Gated then reaches 37.82% versus LoRA's 33.84% at 16 epochs and maintains 36.18% versus 32.51% at 128 epochs. Hawaii follows a different trajectory: Gated reaches 41.74% versus 41.27% for LoRA at 32 epochs, while LoRA achieves 42.42% versus 39.22% for Gated at 128 epochs. Thus, visual-state correction is stronger at later budgets on Libya, whereas extended weight-space adaptation ultimately leads on Hawaii. Appendix C.1 adds Noto and Türkiye Earthquake, where Uniform achieves 30.62% versus 27.20% and 32.44% versus 26.18% for LoRA, respectively. These event-specific comparisons clarify the trade-off rather than relying on a single four-event average.

  3. The few-shot and adaptable correction strength. Table 2 evaluates calibration-selected LoRA ranks 4 and 16 under larger training budgets and 16/32/64 labeled examples. With only 16 labels on CWZ/Gemma-3, APERTURE-Gated achieves 56.47 ± 0.33, versus 50.20 ± 14.15 for LoRA-r16 and 48.01 ± 14.77 for LoRA-r4. Appendix C.2 shows that the bounded residual scale is an important adaptation parameter: on UMCU, Uniform reaches 68.17% with α=2, η=0.05, versus 64.86% at α=3, η=0.05. Table 5 further shows that the complete APERTURE state, including fixed bases, occupies 0.140–0.142 MB versus 40.41 MB for rank-16 LoRA—approximately 285–290× less storage.

We have substantially revised the Introduction, method overview, Figure 2, and Results/Discussion to make this relationship explicit: APERTURE is especially effective when adaptation must be accurate and consistent from very small support sets while retaining only a tiny domain-specific state. The General Response contains the complete revised tables and the per-experiment fine-tuning settings.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could review our response and let us know whether any further clarification is needed.

Kind regards, The authors of APERTURE

Thank you for adding the additional analysis and ablations - the updated uniform results paint a significantly stronger picture beyond simply an improvement in storage and minimized weight updates - the reframing and updates in the writeup are also clearer/easier to read.I have updated my raiting to 3.

Thank you very much for taking the time to review our revisions and additional experiments. We truly appreciate your recognition of the improved results, clearer presentation, and stronger contributions beyond storage efficiency. Your constructive feedback has been very helpful in strengthening our paper, and we sincerely appreciate your reconsideration and updated rating. Thank you again for your time and thoughtful feedback!

Comments on the paper

4 comments
AuthorsOpenReview ↗

Manuscript Revision and Expanded Experiments

We thank the reviewers for their constructive feedback. We have substantially revised the Abstract, Introduction, Method, Experimental Setup, Results, and Discussion to make the motivation and technical workflow clearer and to connect the method directly to its measurable advantages. The revised manuscript explicitly formulates supervised few-shot domain adaptation: a small labeled support set is used to learn a reusable visual-state controller for a frozen VLM, which is then applied to held-out queries without additional training.

The technical presentation now explains the two-stage procedure before its detailed derivation: support-answer gradients identify task-sensitive decoder visual K/V subspaces, and a bounded controller learns corrections within those subspaces. We have clarified the visual observations, text questions, answer labels, visual-token masks, projection sites, and the distinction between the shared Uniform operator and the token-conditioned Gated operator. Revised Figure 2 summarizes the full process; Appendix A contains the mathematical derivations.

We have also substantially expanded the experimental evidence. The revised paper presents five main tables and two additional appendix tables, designed to systematically evaluate APERTURE against well-trained LoRA baselines across model architectures, labeled-data budgets, optimization durations, adaptation mechanisms, and deployment costs. Importantly, we extend LoRA training across multiple ranks, learning rates, and epochs to establish competitive weight-space adaptation baselines rather than relying solely on short-budget fine-tuning.

Specifically, Table 1 examines cross-hospital generalization across VLM backbones, comparing APERTURE with LoRA configurations spanning ranks {1, 4, 16} and training epochs {2, 4, 8}, evaluated across three support seeds. Table 2 investigates the effect of increasing labeled-data budgets (16, 32, and 64 examples), with LoRA ranks 4 and 16 independently tuned over three learning rates and 4/16 epochs using calibration NLL. Table 3 directly examines whether additional fine-tuning improves LoRA, extending the training comparison from 1 to 128 epochs across three datasets. This provides a systematic assessment of performance under substantially larger optimization budgets, including settings where extensively trained LoRA achieves its strongest results.

Beyond these baseline comparisons, Table 4 isolates the contributions of gradient-guided subspace selection, token-conditioned gating, and K/V intervention placement. Table 5 compares trainable parameters, serialized adaptation storage, and measured fitting costs against LoRA trained for 1, 4, and 16 epochs. Appendix C.1 extends the comparison to additional disaster, pathology, and robotic decision tasks, while Appendix C.2 studies the sensitivity of APERTURE to its residual scale and controller learning rate.

Together, these experiments evaluate not only whether APERTURE improves predictive performance, but also how its accuracy, support-seed consistency, and storage efficiency compare with increasingly capable LoRA configurations. The expanded evaluation establishes the advantages and operating characteristics of visual-state correction across different adaptation budgets and application domains. The full numerical comparisons are reproduced below. Except for Table 5, scores are macro-F1 (%) and higher is better. Values include sample SD over three support seeds where shown; Frozen is displayed without SD, and Table 3 reports means only. Underlined entries indicate the second-highest mean in each comparison, following the manuscript's table convention; numerical results are otherwise unformatted. Fine-tuning and selection protocols are detailed in revised Appendix B.3–B.4.

Table 1 — Adaptation Across VLM Backbones and Differet Scenarios

Settings: CAMELYON17; 16 labels (12 fit / 4 calibration), 500 queries, seeds 0/1/2. LoRA: ranks {1, 4, 16}, epochs {2, 4, 8}, reference LR 2×10⁻⁴. APERTURE: rank 16, 4 controller epochs, α=3, η=0.05, plus basis extraction. Result: APERTURE leads in 11/12 rows; Uniform averages 54.26 versus 42.53 for LoRA. Mean row-wise SD is 6.40 (Uniform), 5.20 (Gated), and 12.94 (LoRA).

HospitalBackboneFrozenLoRA (r=1/4/16)APERTURE-UniformAPERTURE-Gated
UMCUGemma-3-4B44.4457.40‾\underline{57.40} ± 31.0465.59 ± 1.5853.98 ± 10.90
UMCUInternVL3-8B22.7822.64 ± 2.1749.76 ± 7.0246.25‾\underline{46.25} ± 5.42
UMCUPhi-3.5-Vision24.2733.69 ± 14.8463.11 ± 10.1244.45‾\underline{44.45} ± 4.70
UMCUQwen2.5-VL-7B63.2064.10 ± 5.4264.86‾\underline{64.86} ± 1.6268.01 ± 0.92
RUMCGemma-3-4B11.9420.33 ± 30.8236.22‾\underline{36.22} ± 15.2446.91 ± 6.24
RUMCInternVL3-8B3.5824.82 ± 19.3539.51 ± 9.8427.67‾\underline{27.67} ± 5.90
RUMCPhi-3.5-Vision11.5942.06 ± 15.6551.02 ± 2.6647.75‾\underline{47.75} ± 8.26
RUMCQwen2.5-VL-7B37.1440.84‾\underline{40.84} ± 6.8046.35 ± 5.3440.55 ± 3.63
LPONGemma-3-4B54.4057.07‾\underline{57.07} ± 17.1566.03 ± 1.6742.86 ± 1.09
LPONInternVL3-8B26.5141.81 ± 0.8754.51‾\underline{54.51} ± 4.0456.01 ± 4.29
LPONPhi-3.5-Vision60.1354.95 ± 4.5756.48‾\underline{56.48} ± 10.4849.87 ± 5.92
LPONQwen2.5-VL-7B57.09‾\underline{57.09}50.59 ± 6.6457.68 ± 7.1753.90 ± 5.18

Table 2 — Few-Shot Label Scaling on CAMELYON17

Settings: CWZ/UMCU; 16, 32, 64 labels; seeds 0/1/2. LoRA: ranks {4,16}, LRs {5×10⁻⁵, 2×10⁻⁴, 8×10⁻⁴}, epochs {4,16}, selection by calibration NLL within each rank. APERTURE: rank 16, 4 controller epochs.

HospitalBackboneLabelsFrozenLoRA-r4LoRA-r16APERTURE-UniformAPERTURE-Gated
CWZQwen2.5-VL-7B1646.6754.32‾\underline{54.32} ± 6.7943.06 ± 14.2255.14 ± 2.0752.95 ± 2.29
CWZQwen2.5-VL-7B3246.6753.59 ± 20.1249.05 ± 16.0652.30‾\underline{52.30} ± 2.1649.02 ± 2.38
CWZQwen2.5-VL-7B6446.6754.02‾\underline{54.02} ± 17.4033.53 ± 0.7256.60 ± 0.7151.97 ± 0.37
CWZGemma-3-4B1650.3548.01 ± 14.7750.20 ± 14.1556.17‾\underline{56.17} ± 2.0456.47 ± 0.33
CWZGemma-3-4B3250.3549.16 ± 6.3241.72 ± 14.5555.50‾\underline{55.50} ± 1.8556.96 ± 0.97
CWZGemma-3-4B6450.3559.05 ± 8.1233.53 ± 0.7254.08 ± 3.4956.15‾\underline{56.15} ± 0.19
UMCUQwen2.5-VL-7B1663.2068.51 ± 2.7539.49 ± 31.3666.00‾\underline{66.00} ± 2.9066.00‾\underline{66.00} ± 2.58
UMCUQwen2.5-VL-7B3263.2072.93 ± 2.2462.49 ± 17.7964.90‾\underline{64.90} ± 1.7664.27 ± 1.83
UMCUQwen2.5-VL-7B6463.2064.48‾\underline{64.48} ± 19.5864.37 ± 19.2864.80 ± 0.4764.47 ± 4.78
UMCUGemma-3-4B1644.4465.55 ± 10.3557.26 ± 31.0465.48‾\underline{65.48} ± 0.7465.12 ± 1.76
UMCUGemma-3-4B3244.4466.21 ± 12.5260.59 ± 16.1063.24‾\underline{63.24} ± 0.8559.29 ± 6.41
UMCUGemma-3-4B6444.4463.10‾\underline{63.10} ± 31.2556.86 ± 30.7563.87 ± 3.7263.09 ± 3.67

Table 3 — Effect of Training Epochs

Settings: Qwen2.5-VL-7B; Hawaii/Libya 24 labels, UMCU 16. LoRA/APERTURE: rank 16, epochs {1,2,4,8,16,32,64,128}; APERTURE also performs basis extraction. Mean only. Result: On Libya, Gated peaks at 37.82 versus LoRA’s 35.75; at one epoch on UMCU, Gated achieves 60.06 versus 29.91.

DatasetMethod1248163264128
Hawaii WildfireLoRA-r1629.7618.3534.6637.1240.1541.27‾\underline{41.27}41.6542.42
Hawaii WildfireAPERTURE-Uniform20.23‾\underline{20.23}23.2428.84‾\underline{28.84}36.92‾\underline{36.92}39.0939.0238.4737.96
Hawaii WildfireAPERTURE-Gated16.4320.25‾\underline{20.25}27.6134.3239.86‾\underline{39.86}41.7439.30‾\underline{39.30}39.22‾\underline{39.22}
Libya FloodLoRA-r1634.6730.3235.7534.2433.8431.7731.4332.51
Libya FloodAPERTURE-Uniform28.79‾\underline{28.79}29.45‾\underline{29.45}32.56‾\underline{32.56}36.0537.02‾\underline{37.02}37.20‾\underline{37.20}36.26‾\underline{36.26}35.24‾\underline{35.24}
Libya FloodAPERTURE-Gated27.6528.4729.7334.85‾\underline{34.85}37.8237.6936.3136.18
UMCULoRA-r1629.9155.1071.7277.8978.0378.4178.3878.28
UMCUAPERTURE-Uniform58.59‾\underline{58.59}59.38‾\underline{59.38}64.8670.5576.67‾\underline{76.67}76.48‾\underline{76.48}75.33‾\underline{75.33}73.66
UMCUAPERTURE-Gated60.0659.7368.01‾\underline{68.01}70.90‾\underline{70.90}75.5475.3374.8173.89‾\underline{73.89}

Table 4 — Mechanism Ablation of APERTURE

Settings: Qwen2.5-VL-7B; Hawaii/Libya/UMCU; three support seeds. APERTURE: rank 16, 4 epochs, α=3, η=0.05; vary the basis, gate, or K/V intervention. No LoRA arm.Result: On UMCU, Gated reaches 68.01 ± 0.92 compared with 62.60 ± 0.50 for Random Gated and 44.55 ± 1.56 for PCA Gated.

Method / ControlHawaii WildfireLibya FloodUMCU
Frozen18.4326.2263.20
Random Gated19.67 ± 1.2828.55 ± 0.5862.60 ± 0.50
PCA Gated27.57 ± 8.1230.98 ± 1.0944.55 ± 1.56
Fixed Half25.18 ± 4.3129.56 ± 0.4765.21 ± 2.53
Learned Scalar25.79 ± 3.8928.86 ± 0.4665.13 ± 2.54
K-only Gated27.71‾\underline{27.71} ± 8.2731.05‾\underline{31.05} ± 0.7165.78‾\underline{65.78} ± 0.52
V-only Gated18.43 ± 0.0027.71 ± 0.0049.05 ± 2.94
APERTURE-Uniform28.84 ± 7.4532.56 ± 3.3864.86 ± 1.62
APERTURE-Gated27.61 ± 2.4129.73 ± 0.5568.01 ± 0.92

Table 5 — Adaptation Efficiency and Storage Cost

Settings: Qwen2.5-VL-7B; Hawaii/UMCU; seed 0. LoRA: rank 16, epochs {1,4,16}. APERTURE: rank 16, 4 controller epochs + basis extraction. Saved-state size includes the bases. Result: APERTURE stores 0.140/0.142 MB versus 40.41 MB for LoRA, an approximately 285–290× reduction.

MethodTrainable parametersSaved state (MB)CompressionHawaii fit (s)UMCU fit (s)
Frozen0--------
LoRA-r16 (1 epoch)10,092,54440.411×12.626.92
LoRA-r16 (4 epochs)10,092,54440.411×50.0324.90
LoRA-r16 (16 epochs)10,092,54440.411×189.9695.48
APERTURE-Uniform1,0240.140~290×38.22‾\underline{38.22}15.82‾\underline{15.82}
APERTURE-Gated1,092‾\underline{1,092}0.142‾\underline{0.142}~285‾\underline{285}×39.0615.98

Appendix C.1 — Additional Cross-Scenario Few-Shot Adaptation

Settings: Three VLM backbones; disaster, PathMNIST, and robotic tasks with task-specific label budgets. LoRA: rank 16, 1 or 4 epochs. APERTURE: rank 16, four-epoch controller protocol; three support seeds. Result: An APERTURE variant leads in 8/12 rows, including Noto Earthquake, Türkiye Earthquake, and InternVL3 DROID Articulation.

BackboneScenario / DatasetFrozenLoRA-r16APERTURE-UniformAPERTURE-Gated
Qwen2.5-VL-7BNoto Earthquake19.9027.20 ± 0.7430.62 ± 5.5429.41‾\underline{29.41} ± 0.07
Qwen2.5-VL-7BTürkiye Earthquake24.4026.18 ± 2.0732.44 ± 3.1529.95‾\underline{29.95} ± 1.46
Qwen2.5-VL-7BPathMNIST29.0545.09 ± 5.0244.41‾\underline{44.41} ± 1.8441.43 ± 0.86
Qwen2.5-VL-7BDROID P&P -- Ending Tile43.9951.14 ± 3.4449.12 ± 1.4749.94‾\underline{49.94} ± 2.60
Qwen2.5-VL-7BGuardian UR541.4042.08 ± 13.6447.76‾\underline{47.76} ± 6.0948.43 ± 4.72
InternVL3-8BBridge P&P -- Pick Point24.2627.84 ± 2.5428.39‾\underline{28.39} ± 1.0528.97 ± 1.78
InternVL3-8BDROID Articulation -- Ending Tile41.1843.52 ± 11.4049.65 ± 4.9947.82‾\underline{47.82} ± 2.98
InternVL3-8BDROID P&P -- Pick Point24.9625.87 ± 0.9927.39 ± 0.8527.30‾\underline{27.30} ± 1.19
Phi-3.5-VisionNoto Earthquake17.3724.60‾\underline{24.60} ± 7.3323.79 ± 1.9724.78 ± 1.04
Phi-3.5-VisionTürkiye Earthquake17.3722.38 ± 6.9321.60‾\underline{21.60} ± 3.9520.37 ± 1.03
Phi-3.5-VisionDROID Articulation -- Ending Tile43.9255.63 ± 7.5154.32‾\underline{54.32} ± 6.8851.23 ± 5.71
Phi-3.5-VisionGuardian UR532.9746.15 ± 4.7947.98 ± 2.3447.08‾\underline{47.08} ± 1.42

Appendix C.2 — Hyperparameter Sensitivity of APERTURE

Settings: Qwen2.5-VL-7B; Hawaii/Libya/UMCU; three support seeds. APERTURE: rank 16, 4 epochs, α∈{1,2,3}, η∈{0.01,0.05}. No LoRA arm. Result: The default α=3, η=0.05 is best for Uniform on Hawaii/Libya and Gated on Hawaii/UMCU; Uniform reaches 68.17 on UMCU at α=2.

αηUniform — HawaiiUniform — LibyaUniform — UMCUGated — HawaiiGated — LibyaGated — UMCU
10.0117.28 ± 0.3926.77 ± 0.9662.61 ± 0.3318.39 ± 0.0326.79 ± 0.1261.76 ± 0.53
10.0520.00 ± 4.0229.83‾\underline{29.83} ± 1.4264.83 ± 0.9416.67 ± 0.4427.60 ± 0.6964.32 ± 1.12
20.0116.93 ± 0.2827.82 ± 0.2963.80 ± 0.1716.84 ± 0.7727.20 ± 1.3063.17 ± 0.20
20.0528.06‾\underline{28.06} ± 4.2928.59 ± 0.6368.17 ± 1.5424.03‾\underline{24.03} ± 4.3429.74 ± 1.4465.10‾\underline{65.10} ± 1.07
30.0120.34 ± 5.7428.98 ± 0.4765.30‾\underline{65.30} ± 0.8716.51 ± 0.1126.93 ± 0.5864.05 ± 0.56
30.0528.84 ± 7.4532.56 ± 3.3864.86 ± 1.6227.61 ± 2.4129.73‾\underline{29.73} ± 0.5568.01 ± 0.92

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.