acceptodds
Under review as a conference paper at ICLR 2027

Which Attention Heads in VLMs Should We Adapt at Test Time?

Abstract

Training vision-language models (VLMs) at test time has emerged as a promising approach to improving visual understanding and mitigating prediction failures during inference. However, existing methods largely predefine the model components to be adapted, risking inefficient parameter updates and unnecessary model drift, which are particularly problematic in the data-scarce test-time setting. We introduce TTT-Head (Test-Time Training of Attention Heads), a fine-grained TTT framework that automatically identifies tunable attention heads and performs instance-specific adaptation in VLMs. Experiments on seven benchmarks covering diverse VQA scenarios—including visual reasoning and compositional question answering, hallucination recognition, multimodal perception and reasoning, and optical character recognition—show that TTT-Head consistently improves the corresponding frozen baselines across all evaluated VLMs, with an average gain of 2.65 points on the LLaVA-1.5-7B backbone. We further demonstrate its generalizability across three widely used VLM families spanning 7B and 13B model scales. In-depth analysis reveals that suppressive interventions on identified key heads are more frequently associated with successful corrections than amplifying interventions, while tunable heads exhibit structured layer-wise preferences across different answer formats.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
test-time training, vision-language models, interpretability, vision-language understanding, multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_EWRpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies which internal components of vision–language models (VLMs) should be adapted at test time. The proposed framework, TTT-Head, distinguishes visual relevance from adaptation eligibility and dynamically selects attention heads for instance-specific adaptation. Experiments cover seven benchmarks and five model configurations across three VLM families. TTT-Head improves the frozen LLaVA-1.5-7B baseline by approximately 2.65 percentage points on average, with consistent improvements across the other evaluated configurations.

Critical strengths

Clear and meaningful problem formulation.

The distinction between heads associated with visual evidence and heads suitable for adaptation provides a useful perspective on test-time training. The preliminary intervention analysis and the visual-only selection ablation support the motivation for looking beyond functional relevance when selecting adaptation targets.

Coherent instance-specific adaptation design.

The framework jointly determines which heads to adjust, their adjustment directions, and the supervision used for optimization. Candidate-relative responses account for score changes shared by the alternative and baseline answers, while baseline anchoring and directional constraints provide explicit mechanisms for moderating unreliable updates.

Broad empirical coverage and informative ablations.

Evaluation spans different visual–language interfaces, model scales, and answer formats. Table 2 includes useful controls covering head discovery, supervision construction, and adaptation realization. In particular, comparisons against entropy minimization on the same heads, direct target-based prediction, and joint probe gates without optimization help disentangle the contributions of the proposed components.

Useful supplementary diagnostics and clearly described data separation.

The paper distinguishes unlabeled calibration, held-out evaluation, and labeled development data, with image-disjoint calibration and evaluation splits. The oracle-head comparison, oracle-target diagnostic, and correction–disruption analysis provide additional evidence. Notably, Table 10 shows that baseline mixing reduces disruptions from 38 to 18, directly supporting its role in limiting overcorrection.

Critical weaknesses

Intervention support primarily measures controllability; its relationship to answer correctness remains insufficiently validated.

An alternative answer becoming more competitive under internal interventions does not necessarily make it more likely to be correct. This concern is particularly relevant because the method probes both suppression and amplification and retains the strongest positive response. For a candidate-relative score with a nonzero local slope around the baseline gate, opposite perturbation directions can naturally yield a positive response in one direction. Consequently, positive responses from multiple heads do not necessarily constitute independent evidence of correctness.

The complete computational cost and accuracy–cost trade-off remain unclear.

Appendix H reports online latency, with TTT-Head requiring approximately 10.9 times the runtime of frozen inference. However, this excludes task-level discovery and calibration, which involve interventions across all attention heads and comparisons involving original and perturbed inputs. Although an amortization formula is provided, the actual calibration cost is not reported.

Moreover, TTAug-8 and TTAug-16 are included as runtime references without corresponding accuracy results. This prevents a direct assessment of whether TTT-Head offers a better trade-off than additional inference-time sampling under comparable budgets. Calibration runtime, stage-level latency, peak memory, and compute-matched accuracy comparisons would make the practical benefits more convincing.

Open-ended candidate construction is restrictive and insufficiently analyzed.

Appendix B.3 constructs alternatives by replacing the first answer token with four high-probability alternatives and completing each prefix. This can miss correct answers that share the baseline’s first token but differ later, while potentially allocating candidates to semantically redundant responses.

Although final decoding is unrestricted, candidate construction still determines the selected alternative, supporting heads, and training target. Candidate-set answer coverage, candidate-oracle performance, and sensitivity to candidate budget or generation strategy are therefore important missing analyses.

The head-level analyses are descriptive and do not fully establish the proposed functional interpretations.

The predominance of suppressive actions among successful corrections does not establish that suppression is more effective than amplification without accounting for their frequencies and outcomes across all attempted adaptations. Likewise, task-dependent depth distributions may reflect answer format, output length, or confidence differences alongside task requirements. Suppression-only and amplification-only comparisons, along with depth analyses controlling for answer format, would strengthen these interpretations.

Questions for authors

Please refer to the weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer EWRp (1/6)

We sincerely appreciate your careful review and positive assessment of our work, particularly your recognition of the problem formulation, instance-level adaptation design, experimental coverage, and supplementary diagnostic analyses. In response to your specific concerns regarding the reliability of Intervention Support, complete computational cost, limitations of candidate construction, and Head-level functional interpretations, we have conducted additional controlled experiments and analyses to further clarify the underlying design rationale and the scope of our conclusions.

Data Split Clarification. The candidate correctness analysis in W1, candidate construction sensitivity analysis in W3, and intervention direction analysis in W4.1 use the previously established independent set of 500 GQA Open Analysis Samples. The computational cost and accuracy comparison in W2 use a separate set of 1,000 GQA Open Main Test Samples. These two evaluation sets are independent of each other and maintain Image-level Disjointness from their corresponding unlabeled Calibration Samples.

W1. Intervention Support and Answer Correctness

We appreciate your pointing out that an increase in the relative preference for an alternative candidate induced by internal interventions does not necessarily mean that the candidate is more accurate. We agree with this observation. In particular, TTT-Head tests both Suppression and Amplification and retains the strongest positive response. Therefore, even an incorrect candidate may naturally produce a positive response in one intervention direction due to the non-zero local slope around the baseline gate. Furthermore, positive responses from multiple Heads should not be directly regarded as independent evidence of correctness.

Regarding the method design and the role of Intervention Support, TTT-Head uses Candidate-Relative Response to measure changes in the preference for an alternative candidate relative to the original answer. Specifically, we compute the change in the score margin between the alternative candidate and the Frozen Answer before and after intervention, rather than examining only the score change of the alternative candidate itself. This avoids interpreting a score increase shared by both answers as stronger support for the alternative candidate. However, such relative score changes cannot completely eliminate local slope effects around the baseline gate, nor can they independently establish candidate correctness. We subsequently aggregate responses from different Heads to obtain Aggregated Support, which is used for candidate selection and the construction of Response-derived Supervision. Aggregated Support primarily reflects the responsiveness of candidate preferences to internal interventions and the resulting relative preference changes, and should not be interpreted as the probability of candidate correctness. We further employ Baseline Anchoring to limit deviations from the original prediction and adjust the contribution of response-based supervision according to Support. However, these designs alone do not establish a relationship between Support and answer correctness.

Therefore, we further examine whether Aggregated Support contains information that distinguishes correct from incorrect candidates, and whether this discriminative ability goes beyond individual-head responses and positive responses that may naturally arise from bidirectional interventions.

AuthorsOpenReview ↗

Response to Reviewer EWRp (2/6)

W1 (continued).

To investigate this question, we conduct additional controlled experiments on 500 independent Analysis Samples from GQA Open using LLaVA-1.5-7B. Following the candidate construction method in Appendix B.3 of the original paper, answer normalization and deduplication yield 1,824 candidate records, including 121 correct and 1,703 incorrect candidates. We compare Aggregated Support, Frozen Margin, and the strongest single-head response, and introduce two controls: Random-head Control randomly selects 16 Heads from the initial Visual Head Shortlist that are not included in the final Eligible Head Pool, while retaining the same candidate set, bidirectional interventions, and maximum-positive-response calculation; Shuffled Support randomly permutes Support values among candidates within the same instance, disrupting the correspondence between Support and individual candidates. Ground Truth is used only for post-hoc evaluation. Pairwise Win Rate is calculated over 278 valid within-instance pairs. Results for Random-head and Shuffled Support are reported as the mean and standard deviation over five random seeds.

Table 1. Correctness Discrimination under Matched Intervention Controls

Score / ControlAUROC ↑Pairwise Win Rate ↑
Frozen Margin0.62059.5%
Aggregated Support0.73069.8%
Strongest-single-head0.68065.4%
Random-head Support0.540 ± 0.02052.4 ± 2.3%
Shuffled Support0.510 ± 0.02050.5 ± 1.8%

As shown in Table 1, Aggregated Support achieves an AUROC of 0.73, higher than Frozen Margin (0.62) and Strongest-single-head (0.68). AUROC measures the ability of a scoring signal to distinguish correct from incorrect candidates: an AUROC of 0.73 means that, when randomly selecting one correct candidate and one incorrect candidate, the correct candidate receives a higher score with a probability of 73%, with ties counted as 0.5. Its within-instance Pairwise Win Rate also reaches 69.8%, exceeding Frozen Margin (59.5%). Notably, Random-head Control retains the same bidirectional intervention and maximum-positive-response mechanism but achieves an AUROC of only 0.54, while Shuffled Support achieves an AUROC close to random chance. These results indicate that the candidate discrimination ability of our method does not arise solely from taking the maximum positive response across bidirectional interventions, nor can it be simply attributed to positive responses from arbitrary Heads. Multi-head aggregation and its correspondence with individual candidates provide additional information for distinguishing candidate correctness.

These results support using Aggregated Support to construct response-based supervision, but we do not directly treat it as a correctness label or probability for candidate answers. Specifically, Support is used to construct the Response-derived Distribution, while the final Soft Supervision Target combines this response distribution with the Frozen Prediction and uses a dynamic weight to control their respective contributions. Therefore, the final supervision target does not rely on Support alone, but retains the original prediction as an anchor while incorporating intervention-derived information.

W2. Complete Computational Cost and Accuracy–Runtime Trade-off

We appreciate your pointing out that the original paper reports only online runtime without fully accounting for Task-level Discovery and Calibration costs, and that the corresponding accuracy results for TTAug-8 and TTAug-16 are missing. We agree that reporting online latency alone is insufficient to fully assess the actual computational cost of TTT-Head.

AuthorsOpenReview ↗

Response to Reviewer EWRp (3/6)

W2 (continued).

Appendix H, Table 17 of the original paper reports the Instance-level Online Runtime, where TTT-Head takes an average of 5.68 seconds per sample on LLaVA-1.5-7B / GQA Open, approximately 10.9 times that of Frozen Inference. This runtime includes Candidate Construction, Head–Action Probing, Sparse Test-time Training, and final prediction, but excludes the one-time Task-level Head Discovery and Calibration costs.

Regarding Task-level computational cost, we use the same batch of 250 unlabeled Calibration Samples on LLaVA-1.5-7B / GQA Open and repeat the complete Task-level Discovery and Calibration pipeline three times on a single NVIDIA Tesla V100 32GB GPU. All stages are re-executed in each run, and their runtimes and peak GPU memory usage are recorded. Runtime is reported as the mean and standard deviation over the three complete runs to assess the actual computational cost and runtime variation.

Table 2. Task-level Discovery and Calibration Runtime

StageRuntime (min)Peak GPU Memory (GiB)
Frozen Inference & Entropy Collection4.26 ± 0.1515.25
Visual Head Discovery8.52 ± 0.2416.40
Responsiveness & Preservation Screening31.56 ± 0.6517.80
Support Calibration3.72 ± 0.1918.65
Total Task-level Discovery & Calibration48.06 ± 0.7618.65

As shown in Table 2, the complete Task-level Discovery and Calibration pipeline takes 48.06 ± 0.76 minutes, with a peak GPU memory usage of 18.65 GiB. Responsiveness & Preservation Screening accounts for the largest portion of the computational cost. Since these operations are performed only once for each Model–Benchmark Configuration, the resulting Head Pool and calibration statistics can be reused for subsequent test samples. Therefore, following Eq. (17) in the original paper, we amortize the one-time calibration cost over different test set sizes:

Tamortized=Tonline+60TcalNT_{\text{amortized}}=T_{\text{online}}+\frac{60T_{\text{cal}}}{N}

Based on the online runtime of 5.68 seconds per sample, the amortized average runtimes are 11.45, 8.56, and 5.97 seconds per sample for test set sizes of 500, 1,000, and 10,000, respectively.

Regarding the Accuracy–Runtime Trade-off, we further evaluate TTAug-8 and TTAug-16 on the same 1,000 GQA Open Main Test Samples using a single V100 GPU, while keeping the Prompt and Decoding Settings identical. TTAug-8 and TTAug-16 serve as Augmentation-only Inference References and do not require additional Task-level Discovery or Calibration; therefore, their amortized runtimes equal their online runtimes. This comparison evaluates methods under comparable runtime budgets using the same hardware and evaluation set, rather than strictly matched computational operations.

Table 3. Accuracy–Runtime Comparison on GQA Open

MethodAccuracy (%) ↑Online TimeAmortized Time (N=1,000)
Frozen VLM58.400.52 s0.52 s
TTAug-859.605.12 s5.12 s
TTT-Head61.205.68 s8.56 s
TTAug-1660.009.46 s9.46 s

As shown in Table 3, TTT-Head achieves an accuracy of 61.20%, outperforming TTAug-8 (59.60%) and TTAug-16 (60.00%). After accounting for the one-time calibration cost, its amortized runtime is 8.56 seconds per sample. Compared with TTAug-8, TTT-Head requires approximately 3.44 additional seconds per sample but achieves a 1.6 pp accuracy improvement. Compared with TTAug-16, it requires approximately 0.90 fewer seconds per sample while still achieving a 1.2 pp higher accuracy. Thus, TTT-Head trades additional runtime for higher accuracy compared with TTAug-8, while achieving both higher accuracy and lower amortized runtime than TTAug-16. As the test set size increases, the one-time calibration cost is further amortized. For example, with 10,000 test samples, the estimated average runtime decreases to 5.97 seconds per sample, approaching the online runtime. These results supplement the one-time calibration costs omitted from the original paper and further demonstrate the accuracy–runtime trade-off between TTT-Head and inference-time augmentation methods.

Reviewer_ie5iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses which attention heads should be selectively adapted in Test-Time Training (TTT) of Vision-Language Models and proposes the TTT-Head framework for this purpose. The authors point out the limitation that existing TTT methods predefine the parameters or modules to be trained, and present the observation that attention heads associated with visual information are not necessarily effective in improving predictions. To address this, they first use unlabeled calibration data to select task-level attention head candidates based on their relevance to visual evidence, intervention responsiveness, and preservation of stable predictions. Then, for each test input, they perform Head–Action probing that suppresses or amplifies the selected heads to identify alternative predictions and the heads that support them. They combine the resulting intervention responses with the original frozen prediction to construct soft supervision and perform instance-specific adaptation by optimizing only the scalar gates of the selected heads. The authors evaluate the proposed method on five model configurations and seven VQA benchmarks. Experimental results show that TTT-Head achieves consistent performance improvements over the original frozen models and several test-time adaptation baselines, and the authors validate the effects of head selection, corrective supervision, and sparse adaptation through various ablations and additional analyses.

Critical strengths
  1. Rather than simply proposing a new method to improve TTT performance, this paper focuses on the issue that existing methods predetermine the parameters to be adapted. In particular, I think the observation that attention heads associated with visual evidence do not necessarily induce positive changes in predictions through intervention is a convincing starting point. Based on this, the authors propose a head selection strategy that considers both intervention responsiveness and prediction preservation beyond simple visual relevance. This design naturally connects to task-level head discovery and instance-specific adaptation, and I think it is a technically well-structured approach in that each stage of the proposed method has a relatively clear purpose.

  2. The authors validate the proposed method on three VLM families with different architectures and seven benchmarks covering diverse visual understanding tasks. In particular, they report an average performance improvement of 2.65 percentage points over the frozen model on LLaVA-1.5-7B, and consistent improvements are also observed across other model configurations. In addition, the ablations in Table 2 provide evidence that head selection, corrective supervision, and gate optimization each contribute to performance improvements.

Critical weaknesses
  1. This paper aggregates intervention responses from attention heads to select an alternative candidate and adjusts the adaptation strength according to the magnitude of the corresponding support. However, the fact that adjusting a particular head increases the relative preference for an alternative candidate does not mean that the candidate is actually a more accurate answer. In particular, Eq. (11) is designed to allow stronger corrections as intervention support increases, but whether this relationship actually leads to correct prediction revisions has not been sufficiently validated. The authors analyze the risk of excessive correction by comparing against the case without baseline mixing in Appendix D.4, but this experiment does not directly validate the necessity of support-dependent adaptation strength itself. Therefore, it remains unclear whether the use of intervention-derived support as a key signal for determining corrective supervision and adaptation strength is sufficiently justified.

  2. This paper considers samples with low prediction entropy in the calibration data to be in relatively stable states and excludes heads that respond strongly to intervention in these states from the adaptation candidates. However, low prediction entropy does not mean that the corresponding prediction is actually correct. Therefore, when the model generates an incorrect answer with high confidence, even heads useful for correcting that error may be excluded during preservation screening. Although the authors show through the ablation in Table 2 that preservation screening contributes to overall performance improvement, they do not sufficiently analyze how this criterion affects confidently correct and confidently incorrect predictions, respectively. Therefore, further validation is needed to determine whether it is appropriate to use intervention sensitivity on low-uncertainty predictions as a criterion for head exclusion.

Questions for authors

This paper considers samples with low prediction entropy to be in relatively stable states and excludes attention heads that respond strongly to intervention in these states from the adaptation candidates. However, because high confidence does not necessarily indicate a correct prediction, heads useful for correcting confidently incorrect predictions may also be excluded. Table 2 shows the overall effect of preservation screening, but could the authors analyze how this screening affects actual correction performance by distinguishing confidently correct and confidently incorrect cases? In addition, the oracle-head recovery analysis in Appendix C.3 is conducted only within the top 10% of attention heads selected based on the initial visual relevance criterion. Therefore, it is unclear how many heads useful for actual correction exist among the 90% of heads excluded initially. If the authors varied the scope of initial head selection or compared against an oracle over all heads, it would be possible to assess more clearly whether the proposed head discovery procedure sufficiently identifies important adaptation targets.

6 comments
AuthorsOpenReview ↗

Response to Reviewer ie5i (1/5)

We appreciate your thoughtful review and specific, constructive comments. We appreciate your recognition of our problem formulation, method design, and experimental analysis. In response to your concerns regarding Intervention Support, Preservation Screening, and the initial Visual Head Selection scope, we provide further clarifications and experimental analyses below.


W1. Role of Intervention-derived Support in Guiding Corrective Supervision as well as Adaptation Strength

We appreciate your pointing out that intervention-induced preference changes do not necessarily indicate candidate correctness, and that the design of adjusting adaptation strength based on Support in Eq. (11) requires further validation. We clarify the role of Intervention-derived Support in TTT-Head. For each test instance, we construct a Candidate Set containing the Frozen Answer and alternative candidates, and use single-head interventions to compare changes in the score margins of alternative candidates relative to the original prediction. We then aggregate the positive intervention responses from different Heads for each candidate to obtain its Aggregated Support S(c)S(c). Based on this Support, we select a sufficiently supported alternative candidate c∗c^*, together with its corresponding Supporting Heads and intervention directions. Thus, Support identifies promising alternatives and determines which Heads participate in subsequent adaptation.

Regarding Response-derived Soft Supervision, Aggregated Support is used to identify a promising alternative candidate and its corresponding Supporting Heads, while the supervision target is constructed from the intervention responses of these Heads across the entire Candidate Set. Specifically, we aggregate the intervention-induced score changes over the entire Candidate Set, weighted according to the degree of support each Supporting Head provides to the selected candidate, and convert the aggregated scores into a probability distribution qresq_{\text{res}}. This distribution captures intervention-induced shifts in candidate preferences and provides soft supervision without assigning a definitive correctness label to any single candidate. We further combine it with the original prediction distribution q0q_0 of the Frozen Model to obtain the final Soft Supervision Target:

qT=(1−β(x))q0+β(x)qresq_T = (1 - \beta(x))q_0 + \beta(x)q_{\text{res}}

Here, q0q_0 denotes the original prediction distribution of the Frozen Model, qresq_{\text{res}} denotes the candidate distribution constructed from intervention responses, and β(x)\beta(x) controls the contribution of response-based supervision to the final target. We then optimize the selected Head Gates using Eq. (7), encouraging the adapted prediction distribution to approach this target.

Regarding the rationale for Support-dependent Adaptation Strength, different test instances receive different levels of intervention support. This motivates the use of instance-specific adaptation weights, allowing the contribution of response-based supervision to vary according to the strength of intervention support. For instances with weaker intervention support, a smaller response-supervision weight helps limit potentially unreliable prediction modifications. Conversely, stronger intervention support allows response-based supervision to contribute more substantially to the adaptation process. Accordingly, we dynamically adjust β(x)\beta(x) based on the Support of each instance, so that the contribution of response-based supervision reflects the strength of intervention support. Meanwhile, retaining the original prediction distribution q0q_0 as an anchor reduces the risk of over-relying on intervention-induced candidate preferences.

To further assess the contribution of dynamic weighting, we compare three strategies while retaining Baseline Anchoring. The experiment uses independent Analysis Sets, consisting of 500 samples each from GQA Open and AMBER, for a total of 1,000 sample instances. Dynamic β\beta uses the sample-specific dynamic weight; Fixed β\beta uses the average dynamic weight among adaptation-eligible samples from the corresponding independent Development Set, which is 0.28 for GQA and 0.30 for AMBER; Shuffled β\beta preserves the distribution of dynamic weights but randomly permutes their assignments across test samples. All three variants maintain the same Candidate Selection, Supporting Heads, intervention directions, Adaptation Coverage, and Gate Optimization settings, differing only in β(x)\beta(x). Shuffled β\beta uses five independent random permutations with different seeds. Its C/D values are reported as the average counts over five runs, while Accuracy is reported as the mean and standard deviation. The standard deviation of Mean β\beta reflects the variation in dynamic weights across different samples. C/D denotes Corrections / Disruptions.

AuthorsOpenReview ↗

Response to Reviewer ie5i (2/5)

W1 (continued).

Table 1. Ablation of Support-dependent Adaptation Strength

BenchmarkVariantMean β\betaCoverageC / DAccuracy
GQADynamic β\beta0.28 ± 0.1047.0%28 / 1560.40%
GQAFixed β\beta0.2847.0%26 / 2059.00%
GQAShuffled β\beta0.28 ± 0.1047.0%25.0 / 20.458.72 ± 0.33%
AMBERDynamic β\beta0.30 ± 0.0951.0%24 / 975.60%
AMBERFixed β\beta0.3051.0%22 / 1174.80%
AMBERShuffled β\beta0.30 ± 0.0951.0%23.0 / 13.274.56 ± 0.17%

As shown in Table 1, Dynamic β\beta achieves the highest accuracy on both benchmarks. Compared with Fixed β\beta, dynamic weighting yields more corrections and fewer disruptions. Compared with Shuffled β\beta, preserving the correspondence between Support and individual samples also leads to better results. These findings indicate that dynamically adjusting the weight of response-based supervision according to the Intervention Support of each instance is more effective than using a uniform fixed weight or randomly assigning weights, providing direct experimental support for the Support-dependent Adaptation Strength in Eq. (11).


W2 & Q1. Appropriateness of Preservation Screening for Selecting Adaptable Heads and Its Implications for Confidently Correct/Incorrect Predictions

We appreciate your pointing out that low prediction entropy does not necessarily imply a correct prediction, and that Preservation Screening may exclude some Heads with corrective potential. This is an important distinction, as prediction entropy reflects the concentration of the model's prediction distribution rather than the actual correctness of its answer. Therefore, low-entropy samples may contain both initially correct predictions and predictions that are made with high confidence but are actually incorrect.

Regarding the design and necessity of Preservation Screening, TTT-Head's Task-level Head Selection consists of three sequential and complementary screening stages. First, Visual Relevance identifies Attention Heads associated with processing task-relevant visual evidence and narrows the initial search space. Second, Responsiveness Screening uses high-entropy calibration samples to identify Heads with sufficient intervention responses when the model's predictions are uncertain. Finally, Preservation Screening uses low-entropy calibration samples to further assess whether these Heads still induce strong intervention effects when the model's predictions are relatively stable, excluding Heads that are more likely to disturb stable predictions. These three stages progressively identify Heads suitable for subsequent adaptation based on visual relevance, potential corrective responsiveness, and prediction preservation, respectively. Visual Relevance and Responsiveness provide complementary evidence for Head Selection, but neither alone captures the potential risk of disrupting originally stable predictions. Preservation Screening therefore helps retain corrective potential while reducing the risk of disrupting the original predictions during subsequent adaptation.

Regarding the trade-off between prediction confidence and correctness, low prediction entropy is not a direct indicator of answer correctness. Some Heads with corrective potential for confidently incorrect predictions may nevertheless be filtered out by Preservation Screening if they also produce strong intervention responses on low-entropy calibration samples. This introduces a trade-off: the screening may leave some potentially corrective Heads unselected, while reducing the risk of disrupting originally correct predictions. Accordingly, prediction entropy serves to distinguish different prediction states rather than determine answer correctness, helping balance potential corrective utility against prediction preservation. To examine this design and its potential impact, we use the entropy thresholds determined from the original 250 unlabeled Calibration Samples on LLaVA-1.5-7B and apply them to the respective independent sets of 500 Analysis Samples from GQA Open and AMBER. This yields 154 low-entropy samples on GQA Open and 152 on AMBER. Among them, GQA Open contains 108 initially correct and 46 initially incorrect predictions, while AMBER contains 126 initially correct and 26 initially incorrect predictions. Ground Truth is used only for post-hoc correctness evaluation and is not involved in entropy-threshold determination or Head Selection.

AuthorsOpenReview ↗

Response to Reviewer ie5i (3/5)

W2 & Q1 (continued).

First, we conduct a Head-level diagnosis to examine whether Heads excluded by Preservation Screening still exhibit corrective potential. Under the default Top-10% Visual Head Shortlist, each benchmark initially contains 102 Heads. After Responsiveness Screening, approximately 41 Heads pass the screening. Preservation Screening subsequently retains 16 Heads and excludes the remaining 25. We therefore compare the 16 Retained Heads with the 25 Excluded Heads that pass Responsiveness Screening but are removed by Preservation Screening. For each Head–Sample Pair, we take the maximum response over the three Scaling Factors separately within the suppression and amplification directions, and then compute the equally weighted average across the two directions. Thus, each pair produces only one diagnostic response. For initially incorrect predictions, an intervention that increases the Ground Truth margin over the original prediction is counted as a Positive GT-margin Response. For initially correct predictions, an intervention that decreases the margin of the original correct answer over other candidates is counted as a Negative Correct-margin Response. If the Ground Truth answer is absent from the original candidate set, we additionally compute its sequence score under the Frozen Model solely for post-hoc diagnosis. All reported proportions use Head–Sample Pairs as the statistical unit.

Table 2. Post-hoc Responses of Preservation-retained and Excluded Heads

BenchmarkHead GroupHeadsPositive GT-margin ResponsesNegative Correct-margin Responses
GQARetained16231/736 (31.4%)188/1,728 (10.9%)
GQAExcluded25248/1,150 (21.6%)564/2,700 (20.9%)
AMBERRetained16139/416 (33.4%)178/2,016 (8.8%)
AMBERExcluded25134/650 (20.6%)682/3,150 (21.7%)

As shown in Table 2, Preservation Screening does exclude some Heads that still exhibit positive corrective responses, but these Excluded Heads also have substantially higher rates of negative responses on originally correct predictions. For example, on GQA Open, the proportion of Negative Correct-margin Responses increases from 10.9% for Retained Heads to 20.9% for Excluded Heads. The same trend is observed on AMBER, where it increases from 8.8% to 21.7%. This indicates that although the excluded Heads can provide some corrective responses, they are also more likely to interfere with originally correct predictions. To further examine how this screening trade-off affects actual prediction accuracy, we conduct the following adaptation experiments.

At the Prediction level, we compare Full TTT-Head with w/o Preservation. The latter removes only Preservation Screening while retaining Visual Head Discovery and Responsiveness Screening, and therefore uses all approximately 41 Heads that pass Responsiveness Screening. Full TTT-Head uses the final 16 retained Heads. We re-execute Support Calibration separately for each configuration based on its corresponding Head Pool, while keeping all other Instance-level settings unchanged. Corrections and Disruptions are counted at the unique-sample level, representing initially incorrect predictions that become correct and initially correct predictions that become incorrect, respectively.

Table 3. Correction–Disruption Analysis on Low-entropy Samples

BenchmarkVariantCorrections ↑Disruptions ↓Accuracy ↑
GQA (n=154)Frozen VLM——70.1%
GQAw/o Preservation9771.4%
GQAFull TTT-Head5272.1%
AMBER (n=152)Frozen VLM——82.9%
AMBERw/o Preservation7683.6%
AMBERFull TTT-Head3184.2%

As shown in Table 3, removing Preservation Screening increases the number of corrections on both benchmarks, but also leads to more disruptions of originally correct predictions. In contrast, the full method achieves higher final accuracy by reducing disruptions, despite making fewer corrections. Specifically, on GQA Open, Full TTT-Head achieves a net gain of 3 correct predictions, compared with 2 for w/o Preservation; on AMBER, the corresponding gains are 2 and 1. Overall, these results support Preservation Screening as a conservative Head Selection criterion that balances potential corrective utility against the risk of disrupting originally correct predictions, rather than treating low-entropy predictions as necessarily correct.

Reviewer_gAdhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TTT-Head, a fine-grained test-time training framework for VLMs. It identifies adaptation-eligible attention heads using visual relevance and intervention responsiveness, then performs instance-specific head selection and adaptation using response-derived soft supervision. Experiments on seven benchmarks and five VLM configurations demonstrate consistent improvements over frozen models and competitive baselines.

Critical strengths
  1. The paper is well motivated and highlights that visual relevance does not guarantee adaptation effectiveness. The proposed task-to-instance pipline provides a flexible approach to selecting and adapting attention heads.

  2. Experiments across different benchmarks and VLM architectures demonstrate consistent improvements. The ablation studies provide useful evidence supporting the effectiveness of the proposed components.

Critical weaknesses
  1. Although baseline-anchored supervision effectively mitigates overcorrection, TTT-Head relies on aggregated head responses to select alternative predictions and determine adaptation strength. However, the paper does not directly establish whether stronger head support indicates a higher likelihood of selecting a correct alternative.

  2. TTT-Head introduces a considerable number of hyperparameters across different stages. Although the paper state that these hyperparameters are selected on a development set, the justification for their selected values are not sufficiently explained. Moreover, the sensitivity analysis covers only a subset of these parameters, leaving many other hyperparameters unexplored and making it unclear how robust the method is to different hyperparameter choices. This raises concerns about TTT-Head's dependence on extensive hyperparameter tuning.

Questions for authors
  1. Could the authors analyze whether stronger aggregated head support is associated with a higher probability of selecting the correct alternative?

  2. Could the authors clarify how the hyperparameters are selected and provide more comprehensive sensitivity analyses of the unexplored hyperparameters?

  3. Could the authors report the total computational overhead including task-level head discovery and calibration, this result can provide a more complete assessment of the TTT-Head's efficiency?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gAdh (1/3)

We sincerely thank the reviewer for the positive assessment of our motivation, methodological design, and experimental results, as well as for the constructive suggestions. To address the concerns regarding candidate correctness, hyperparameter sensitivity, and computational overhead, we provide additional analyses and experiments below.


W1 & Q1. Aggregated Head Support and Answer Correctness.

Thanks for pointing out the need to further validate the relationship between Aggregated Support and candidate correctness. To examine whether stronger head support tends to favor correct candidates, we conduct a post-hoc correctness analysis on 500 independent GQA Open analysis samples using LLaVA-1.5-7B.

We use the fixed eligible head pool obtained from the default 250 unlabeled calibration samples in the original paper, and perform Candidate Construction and instance-level Head–Action probing on each of the 500 analysis samples, without repeating task-level head discovery or calibration. Following the candidate construction protocol in Appendix B.3, we generate up to four non-baseline alternatives per sample. After answer normalization and deduplication, we obtain 1,824 candidate records, including 121 correct and 1,703 incorrect alternatives. These 121 correct alternatives come from 121 distinct samples. We compute the Aggregated Support (S(c)) for each candidate and compare it with the Frozen Alternative Margin, defined as the score difference between the candidate and the original prediction: �0��0� Ground-truth labels are used solely for post-hoc evaluation and are not involved in head selection, candidate construction, or test-time adaptation.

Table 1. Association Between Aggregated Support and Candidate Correctness (GQA Open)

MetricResult
Support AUROC0.73
Frozen Margin AUROC0.62
AUROC Improvement+0.11
Within-instance Pairwise Win Rate69.8% (278 pairs)
Support Top-1 Selection Accuracy66.1% (80/121)
Frozen Margin Top-1 Selection Accuracy50.4% (61/121)
Support AUROC (Top-25% Frozen Margin)0.66
Frozen Margin AUROC (Same Subset)0.56

As shown in Table 1, Aggregated Support achieves an AUROC of 0.73 across all candidates, outperforming the Frozen Margin (0.62) by 0.11. AUROC measures the ability of a scoring signal to distinguish correct from incorrect candidates, i.e., the probability that a randomly selected correct candidate receives a higher score than a randomly selected incorrect candidate, with ties counted as 0.5. Since actual adaptation involves comparing candidates within the same question, we further compute the Within-instance Pairwise Win Rate. Among 278 valid within-instance pairs, correct candidates receive higher Support in 69.8% of comparisons.

To further evaluate the two scoring signals in candidate ranking, we conduct a post-hoc Top-1 ranking analysis on the 121 distinct samples containing at least one correct alternative. Specifically, for each sample, we select the alternative with the highest Aggregated Support or Frozen Margin, respectively, and evaluate its correctness. This is a conditional metric computed only on samples with at least one correct alternative; it does not represent the end-to-end accuracy over all 500 samples or alter TTT-Head's original soft-supervision mechanism. The results show that Support achieves a Top-1 Selection Accuracy of 66.1% (80/121), substantially higher than Frozen Margin (50.4%, 61/121). These results indicate that Aggregated Support not only provides better candidate correctness discrimination but also leads to a higher proportion of correct selections in post-hoc candidate ranking.

Furthermore, to examine whether Aggregated Support can distinguish correct candidates from alternatives that are relatively preferred by the original model but are actually incorrect, we analyze the top 25% of candidates ranked by Frozen Alternative Margin. Among these 456 candidates, 400 are incorrect and 56 are correct. Within this subset, Support achieves an AUROC of 0.66, compared with 0.56 for Frozen Margin. This indicates that even among candidates receiving relatively high preference from the original model, Aggregated Support more effectively distinguishes correct from incorrect alternatives and ranks correct candidates higher.

AuthorsOpenReview ↗

Response to Reviewer gAdh (2/3)

W1 & Q1 (continued).

In summary, stronger Aggregated Support is positively associated with candidate correctness, but it does not directly represent the probability of candidate correctness or guarantee that every selection is correct. Therefore, we do not treat Aggregated Support as a hard correctness label. Instead, we use it to construct Response-derived Soft Supervision and combine it with the dynamic weight (β(x)\beta(x)) and Baseline Anchoring to mitigate the risk of unreliable interventions disrupting the original predictions.


W2 & Q2. Hyperparameter Selection and Sensitivity.

Hyperparameter Selection Protocol. We appreciate your highlighting the need to clarify hyperparameter selection and sensitivity. We understand that the number of hyperparameters may raise concerns about extensive tuning. We emphasize that TTT-Head does not require repeated label-based searches for different models and tasks, but follows a unified global selection protocol. Specifically, following Appendix A.2, we use labeled development sets from seven benchmarks to select global hyperparameters once on LLaVA-1.5-7B. GQA Open, COCO-QA, POPE, AMBER, and MME each contain 500 development samples, while MMStar and TextVQA each contain 300. These samples are independent of the unlabeled calibration sets and held-out evaluation sets used in the main experiments.

Each development set is further divided into Development Calibration and Development Evaluation splits, following the same task-level head discovery, calibration, and instance-level adaptation pipeline as the main experiments. We use the equally weighted average accuracy (Macro-average Accuracy) across the seven benchmarks on Development Evaluation as the selection criterion. Global hyperparameters are selected once on LLaVA-1.5-7B and then fixed across all subsequent model families, scales, and tasks, without further label-based tuning. They include Responsiveness and Preservation Screening Percentiles, Support Qualification Quantile (qq), Maximum Response Weight (βmax⁡\beta_{\max}), and the Probe Budget per direction. Model–task-specific calibration thresholds are independently estimated from the corresponding unlabeled calibration sets without ground-truth-based tuning. Other implementation details are provided in Appendix B.4.

Additional Hyperparameter Robustness Analysis. Appendix C.2 (Figures 6–8) analyzes the effect of the Preservation Screening Percentile on the adaptation space and performance, while Appendix D.2 (Table 8) examines different Probe Budgets. To further evaluate hyperparameter sensitivity, we analyze two additional parameters related to candidate support. Support Qualification Quantile (qγq_{\gamma}) determines the candidate qualification threshold based on the Stable-response Reference, controlling the strictness of candidate selection. Maximum Response Weight (βmax⁡\beta_{\max}) limits the maximum dynamic weight (β(x)\beta(x)), controlling the maximum contribution of Response-derived Supervision. Thus, the former determines which candidates qualify for adaptation, while the latter controls the strength of response-based supervision. We conduct sensitivity experiments on 500 independent analysis samples each from GQA Open and AMBER using LLaVA-1.5-7B. We vary one parameter at a time while keeping all others fixed. Coverage denotes the proportion of samples qualifying for adaptation, and C/D denotes Corrections / Disruptions. We fix (βmax⁡=0.50\beta_{\max}=0.50) when varying (qγq_{\gamma}), and (qγ=0.70q_{\gamma}=0.70) when varying (βmax⁡\beta_{\max}), under which candidate qualification and Adaptation Coverage remain unchanged.

Table 2. Sensitivity to Support Qualification Quantile (qγq_{\gamma})

qγq_{\gamma}GQA CoverageGQA C/DGQA Acc.AMBER CoverageAMBER C/DAMBER Acc.
0.5062.0%35 / 2460.0%66.0%29 / 1575.4%
0.6054.0%32 / 1960.4%58.0%27 / 1275.6%
0.70 (Default)47.0%28 / 1560.4%51.0%24 / 975.6%
0.8039.0%22 / 1160.0%42.0%19 / 775.0%
0.9030.0%14 / 659.4%33.0%14 / 474.6%
AuthorsOpenReview ↗

Response to Reviewer gAdh (3/3)

W2 & Q2 (continued).

Table 3. Sensitivity to Maximum Response Weight (βmax⁡\beta_{\max})

βmax⁡\beta_{\max}GQA C/DGQA Acc.AMBER C/DAMBER Acc.
0.2018 / 760.0%15 / 474.8%
0.3524 / 1160.4%21 / 775.4%
0.50 (Default)28 / 1560.4%24 / 975.6%
0.6534 / 2360.0%29 / 1675.2%
0.8039 / 3159.4%34 / 2574.4%

Tables 2–3 show that lowering qγq_{\gamma} increases Adaptation Coverage and Corrections, but also Disruptions; raising it has the opposite effect. The default qγ=0.70q_{\gamma}=0.70 matches the accuracy of 0.60 on both benchmarks, with lower coverage and fewer disruptions. Smaller βmax⁡\beta_{\max} values limit response supervision, reducing both corrections and disruptions, while larger values strengthen adjustments but introduce more disruptions. The default βmax⁡=0.50\beta_{\max}=0.50 achieves tied-best accuracy on GQA and the highest accuracy on AMBER. Together with the original paper's two sensitivity analyses, these results support the robustness of our default configuration. We will clarify the settings and selection protocol and include these analyses in the revision.


Q3. Complete Computational Cost.

We appreciate the suggestion to report the full computational cost, including task-level discovery and calibration. We evaluate LLaVA-1.5-7B on GQA Open using 250 unlabeled calibration samples and a single NVIDIA Tesla V100 32GB GPU. We repeat the complete discovery and calibration pipeline three times, rerunning each stage and recording runtime and peak GPU memory to capture runtime variability.

Table 4. Task-level Discovery and Calibration Cost

StageRuntime (min)Peak GPU Memory (GiB)
Frozen Inference & Entropy Collection4.26 ± 0.1515.25
Visual Head Discovery8.52 ± 0.2416.40
Responsiveness & Preservation Screening31.56 ± 0.6517.80
Support Calibration3.72 ± 0.1918.65
Total Task-level Discovery & Calibration48.06 ± 0.7618.65

Table 4 shows that task-level discovery and calibration take 48.06 ± 0.76 minutes, with peak GPU memory of 18.65 GiB; Responsiveness & Preservation Screening is the most costly stage. Since this process runs only once per model–task configuration, its head pool and calibration statistics can be reused for subsequent test samples. Following Eq. (17) in the original paper, we amortize this cost over the evaluation set:

Ttotal=Tonline+TcalNT_{\text{total}}=T_{\text{online}}+\frac{T_{\text{cal}}}{N}

With an online runtime of 5.68 seconds per sample, the amortized runtimes for 500, 1,000, and 10,000 samples are 11.45, 8.56, and 5.97 seconds per sample, respectively. We also compare TTT-Head with TTAug-8 and TTAug-16 on the same 1,000 held-out GQA Open samples using one V100 GPU. TTAug-8 and TTAug-16 are augmentation-only inference references requiring no task-level discovery or calibration; thus, their amortized and online runtimes are identical.

Table 5. Accuracy–Cost Comparison on GQA Open

MethodAccuracy (%) ↑Online TimeAmortized Time (N=1,000)
Frozen VLM58.400.52 s0.52 s
TTAug-859.605.12 s5.12 s
TTT-Head61.205.68 s8.56 s
TTAug-1660.009.46 s9.46 s

TTT-Head achieves 61.20% accuracy, exceeding TTAug-8 (59.60%) and TTAug-16 (60.00%). After calibration-cost amortization, its runtime is 8.56 seconds per sample—higher than TTAug-8 (5.12 seconds) but lower than TTAug-16 (9.46 seconds). Thus, TTT-Head trades additional computation for higher accuracy than TTAug-8, while outperforming TTAug-16 in both accuracy and amortized runtime. At 10,000 samples, its estimated runtime falls to 5.97 seconds per sample, approaching the online runtime. We will include the full cost, measurement protocol, and comparisons in the revision.


We sincerely appreciate the reviewer’s insightful comments and constructive suggestions, which have helped strengthen the empirical evaluation of TTT-Head. We hope these responses and additional analyses address the concerns.

We sincerely thank you for your time and constructive feedback on our work. We understand that this is a particularly busy period, and as the discussion deadline approaches, we hope that our detailed responses and additional experimental analyses have adequately addressed your concerns. If you have any remaining questions or suggestions, we would be very happy to provide further clarification and discussion.

Thank you again for your time and consideration.

Reviewer_GbjuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an important problem in test-time training for VLMs. The authors conduct mechanistic interpretability experiments to study attention heads associated with visual evidence. Based on the observation that certain heads respond positively to adaptation, they identify adaptation-eligible heads and propose TTT-Head (Test-Time Training of Attention Heads). Experiments demonstrate that TTT-Head outperforms existing baselines.

Critical strengths

Strengths

  1. This paper studies an important gap between context relevance and adaptation effectiveness. Existing mechanistic interpretability method often fail to translate explainability to practical intervention on models.
  2. Two level identifications (task and instance levels) are efficient to locate specific heads contributing in different settings.
  3. Ablation studies are helpful to distinguish the contributions of visual relevance, head selection and probing. The paper is well written and organized (except some formatting errors).
Critical weaknesses

Weakness

  1. Motivation and Training. The motivation from visual evidence to adaptation eligibility should be strengthened. Why heads that respond strongly to individual interventions are suitable targets for adaptation, some heads may be effective under different scaling factors. Although Eq. (7) jointly optimizes the selected head gates, (a) head selection and adjustment directions are determined through single-head interventions. This may overlook interactions among heads, particularly combinations that are effective only when modified jointly. (b) the training details are not provided (hyper parameter, loss curves), and the update stability of the gate are not verified.
  2. Sensitivity of Head Selection. In task level discovery, this paper use frozen-response entropy to form high- and low-uncertainty calibration subsets D_u and D_s. And the author defend that these subsets reflect predictive states rather than correctness labels. Uncertainty does not guarantee correctness. However, the number of the calibration set is 250. After being divided to subsets, the number of the subsets is too small to support the generalization of the proposed method. Also, it remains unclear whether greater intervention-induced responsiveness on the high-entropy subset D_u translates into improved answer correctness.
  3. Experiments. Current experimental results are not adequate to prove the effectiveness of proposed TTT-Head. (a) The main experiments are concentrated on LLaVA-1.5-7B, more results on other VLM models are appreciated for comprehensive evaluation. (b) Additionally, hyper parameters are selected once on a separate labeled development set and fixed across all model–task configurations. The details of hyper parameter selection on other baselines are missing to make fair comparison.
  4. Minors. Some figures such as Figure 2 are dense and difficult to interpret.
Questions for authors

Questions

Head–Action probes measure intervention-induced preference shifts, which do not necessarily indicate correctness. Can the authors show whether the aggregated support score S(c) distinguishes correct alternatives from incorrect ones? Could the authors test whether correct alternatives receive systematically stronger support, rather than confident but incorrect alternatives?

9 comments
AuthorsOpenReview ↗

Response to Reviewer Gbju (1/6)

We sincerely thank you for your careful review and constructive suggestions, as well as for recognizing the motivation, methodological design, and experimental results of our work. In response to your comments, we would like to further clarify the rationale behind our methodological design and implementation details, and provide additional experimental analyses.


W1.1 Rationale for determining head adaptation eligibility based on single-head intervention responses despite scaling sensitivity and head-head interactions.

Thanks for raising this important point. We hope our following response could address your concern and would be happy to clarify further if needed.

Motivation. We would like to clarify that our design is motivated by two complementary considerations: (1) prior interpretability studies [1, 2] have shown that attention heads in VLMs can identify question-relevant visual evidence and route the corresponding information to subsequent computations for prediction. Identifying such heads therefore provides a principled way to narrow the search space for test-time adaptation; and (2) our preliminary intervention analysis reveals that visual relevance establishes a head's functional connection to task-relevant evidence, but does not by itself establish its benefits for adaptation. This motivates our two-stage rationale: using visual relevance as a grounding criterion to identify potentially relevant heads, followed by intervention responsiveness as an additional criterion to determine their adaptation eligibility.

[1] Yanbei Jiang, Xueqi Ma, Shu Liu, Sarah Monazam Erfani, Tongliang Liu, James Bailey, Jey Han Lau, and Krista A. Ehinger. 2025. Investigating the Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules. arXiv preprint arXiv:2512.10300.

[2] Jing Bi, Junjia Guo, Yunlong Tang, Lianggong Bruce Wen, Zhang Liu, Bingjie Wang, and Chenliang Xu. 2025. Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4135–4144.

Scaling Sensitivity. We do share the reviewer’s consideration of head effectiveness to scaling sensitivity. We clarify that Task-level Head Screening does not rely on a fixed scaling factor. Instead, for each calibration sample, we randomly sample one suppression factor and one amplification factor, respectively, and share these factors across all candidate heads for that sample. We then aggregate intervention responses across calibration samples, reducing dependence on any single scaling factor while keeping the computational cost manageable.

To further validate our design, we compare screening with 1, 3, and 5 sampled scaling factors per direction (either suppression or amplification) on GQA and AMBER. Here, Oracle Head Recovery measures the proportion of ground-truth-guided oracle heads also identified by the label-free selection method, with both head sets selected from the same initial visual shortlist (see details in Appendix C.3, Eq. (12)). As shown in Table 1, increasing the number of sampled factors improves Oracle Head Recovery from 75.0% to 81.3% on GQA and from 81.3% to 87.5% on AMBER. However, the corresponding final accuracy improves by only 0.1 percentage points on GQA and remains unchanged on AMBER. These results suggest that denser scaling-factor sampling can identify a small number of additional effective heads, but yields limited gains in final adaptation performance. Therefore, our current setting (K=1) provides a reasonable trade-off between head-screening effectiveness and computational efficiency.

Table 1. Task-level Scaling Factor Sensitivity

BenchmarkKJaccardOracle RecoveryAccuracy (%)
GQA1100%75.0%61.20
GQA388.2%81.3%61.30
GQA577.8%81.3%61.30
AMBER1100%81.3%76.70
AMBER388.2%87.5%76.70
AMBER577.8%87.5%76.70
AuthorsOpenReview ↗

Response to Reviewer Gbju (2/6)

W1.1 (continued).

Head–Head Interactions. We thank the reviewer for highlighting the possibility that single-head interventions may overlook synergistic interactions among heads. At the instance level, we use single-head Head–Action probing to identify supporting heads and their adjustment directions, followed by joint optimization of the selected head gates using Eq. (7). To assess whether this strategy may overlook potentially beneficial head combinations and its practical impact, we conduct an additional pairwise head-head interaction analysis.

Specifically, we select 100 samples from GQA Open and conduct experiments on LLaVA-1.5-7B. Following our instance-level probing design, we test three scaling factors for each head in both the suppression and amplification directions. We then categorize each head based on its best Candidate-Relative Head Response across all single-head Head–Actions (Eq. (4)): a head is labeled as a Supporting Head if its response is positive, and as a Weak Head otherwise. These labels are determined by intervention responses on each individual sample and are used only to characterize changes in candidate preference, not answer correctness. Based on this categorization, we test 10 head pairs per sample instance: 4 Weak-Weak pairs, 4 Supporting-Weak pairs, and 2 randomly selected pairs, resulting in 1,000 (10 head pairs/sample × 100 samples) pairwise head-head intervention tests in total. This allocation prioritizes two complementary types of potentially overlooked interactions (Weak-Weak and Supporting-Weak), while retaining random pairs as a reference and keeping the computational cost manageable. We compare the Candidate-Relative Head Response under joint two-head intervention with the sum of the responses obtained by intervening on each head separately. If the joint response exceeds the sum of the two individual responses, we consider this evidence of an interaction between the two heads.

As shown in Table 2, 140 of the 1,000 pairwise tests (14.0%) yield a joint response greater than the sum of the individual responses. Among the 400 Weak-Weak pairs, 48 (12.0%) show this pattern, but only 24 (6.0%) produce a positive candidate-relative response under joint intervention. Similar patterns are observed in the Supporting-Weak and Random Pair groups, with only 16 and 4 positive responses, respectively. Overall, only 44 of the 1,000 pairs produce positive joint responses, indicating that positive joint candidate-relative responses are relatively uncommon among the sampled head pairs.

However, positive candidate-relative responses do not necessarily lead to correct predictions. We further use ground-truth labels solely for post-hoc correctness evaluation. Of the 100 samples, 42 were initially incorrect and 58 were initially correct, corresponding to 420 and 580 pairwise head tests, respectively. Among the 420 tests on initially incorrect samples, 14 (3.33%) result in corrected predictions, while 8 of the 580 tests on initially correct samples (1.38%) disrupt correct predictions, with the detailed results reported in Tables 2 and 3.

These results suggest that although some synergistic head combinations do exist, their corrective benefits are limited, and joint interventions can also disrupt initially correct predictions. Moreover, searching for all possible synergistic combinations may require substantial computational cost. Given that, we believe that our design, i.e., single-head screening reduces the search space while identifying heads with positive individual intervention responses, and joint optimization of the selected head gates accounts for interactions among these heads during adaptation, offers a practical balance between computational efficiency and adaptation effectiveness.

Table 2. Pairwise Head Interaction Analysis

Pair Type#PairsJoint Response > Sum of Individual ResponsesPositive Joint Response
Weak + Weak4004824
Supporting + Weak4007616
Random Pair200164
Total1,00014044

Table 3. Correction and Disruption Analysis

Pair TypeTests on Initially Incorrect SamplesCorrectionsTests on Initially Correct SamplesDisruptions
Weak + Weak16842322
Supporting + Weak16882324
Random Pair8421162
Total420145808
AuthorsOpenReview ↗

Response to Reviewer Gbju (3/6)

W1.2 Missing training details: hyperparameters, learning curves, gate update stability.

We thank the reviewer for pointing out the missing training details and the need to verify gate update stability. We clarify that TTT-Head optimizes only the gates of the heads selected for the current instance using AdamW, while keeping all other model parameters frozen. The optimization objective is the KL divergence in Eq. (7). We use a learning rate of 5e-3, weight decay of 0, Adam betas of (0.9, 0.999), and three optimization steps per instance. All gates are initialized to 1 and projected after each update to the corresponding suppression interval [0, 1] or amplification interval [1, 2]. They are reset to 1 after each instance.

To further examine optimization stability, we analyze 500 samples from GQA Open using LLaVA-1.5-7B. As shown in Table 4, the mean KL loss consistently decreases from 0.0817 to 0.0419 over three optimization steps, corresponding to a 48.7% reduction. Meanwhile, the mean gate update magnitudes in both suppression and amplification directions gradually decrease. These results suggest that the optimization exhibits stable behavior over the three update steps. We will include these training configurations and the corresponding stability analysis in the revised manuscript.

Table 4. Gate Optimization Stability on GQA Open (500 Samples)

StepMean KL Loss ↓Mean Absolute Gate Update (Suppression) ↓Mean Absolute Gate Update (Amplification) ↓
Initialization0.0817 ± 0.0305——
Step 10.0520 ± 0.02280.0051 ± 0.00240.0053 ± 0.0026
Step 20.0460 ± 0.02070.0046 ± 0.00260.0047 ± 0.0023
Step 3 (Final)0.0419 ± 0.01890.0040 ± 0.00200.0043 ± 0.0025

W2.1 Calibration Size Sensitivity.

We thank the reviewer for raising the concern that 250 calibration samples may be insufficient. We further compare four calibration sizes: 150, 200, 250, and 500 unlabeled samples. Specifically, we collect 2,500 additional unlabeled GQA Open samples and divide them into five non-overlapping groups of 500 samples each. Within each group, we construct nested calibration subsets of sizes 150, 200, 250, and 500. These additional calibration samples are image-level disjoint from the fixed 500 analysis samples and 1,000 main test samples.

For each calibration size and run, we independently repeat the complete task-level head discovery and calibration pipeline from the same frozen model state, including Visual Head Discovery, Responsiveness and Preservation Screening, and Support Calibration. No head responses or calibration statistics are reused across runs. All configurations are evaluated on the same 500 analysis samples, and we report the mean accuracy and standard deviation over five independent runs.

Table 5. Sensitivity to Calibration Set Size

Calibration SizeGQA Open Accuracy (%)
15059.84 ± 0.38
20060.08 ± 0.23
250 (Default)60.36 ± 0.17
50060.40 ± 0.14

As shown in Table 5, increasing the calibration size from 250 to 500 yields only a 0.04 percentage-point improvement in mean accuracy. Moreover, the mean pairwise Jaccard similarity between the head pools obtained from the five runs at the default size of 250 is 0.76, computed over all 10 pairwise comparisons. These results indicate that, on LLaVA-1.5-7B / GQA Open, 250 calibration samples provide relatively consistent head selection and adaptation performance close to that obtained with a larger calibration set.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.