- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies which internal components of vision–language models (VLMs) should be adapted at test time. The proposed framework, TTT-Head, distinguishes visual relevance from adaptation eligibility and dynamically selects attention heads for instance-specific adaptation. Experiments cover seven benchmarks and five model configurations across three VLM families. TTT-Head improves the frozen LLaVA-1.5-7B baseline by approximately 2.65 percentage points on average, with consistent improvements across the other evaluated configurations.
Critical strengths
Clear and meaningful problem formulation.
The distinction between heads associated with visual evidence and heads suitable for adaptation provides a useful perspective on test-time training. The preliminary intervention analysis and the visual-only selection ablation support the motivation for looking beyond functional relevance when selecting adaptation targets.
Coherent instance-specific adaptation design.
The framework jointly determines which heads to adjust, their adjustment directions, and the supervision used for optimization. Candidate-relative responses account for score changes shared by the alternative and baseline answers, while baseline anchoring and directional constraints provide explicit mechanisms for moderating unreliable updates.
Broad empirical coverage and informative ablations.
Evaluation spans different visual–language interfaces, model scales, and answer formats. Table 2 includes useful controls covering head discovery, supervision construction, and adaptation realization. In particular, comparisons against entropy minimization on the same heads, direct target-based prediction, and joint probe gates without optimization help disentangle the contributions of the proposed components.
Useful supplementary diagnostics and clearly described data separation.
The paper distinguishes unlabeled calibration, held-out evaluation, and labeled development data, with image-disjoint calibration and evaluation splits. The oracle-head comparison, oracle-target diagnostic, and correction–disruption analysis provide additional evidence. Notably, Table 10 shows that baseline mixing reduces disruptions from 38 to 18, directly supporting its role in limiting overcorrection.
Critical weaknesses
Intervention support primarily measures controllability; its relationship to answer correctness remains insufficiently validated.
An alternative answer becoming more competitive under internal interventions does not necessarily make it more likely to be correct. This concern is particularly relevant because the method probes both suppression and amplification and retains the strongest positive response. For a candidate-relative score with a nonzero local slope around the baseline gate, opposite perturbation directions can naturally yield a positive response in one direction. Consequently, positive responses from multiple heads do not necessarily constitute independent evidence of correctness.
The complete computational cost and accuracy–cost trade-off remain unclear.
Appendix H reports online latency, with TTT-Head requiring approximately 10.9 times the runtime of frozen inference. However, this excludes task-level discovery and calibration, which involve interventions across all attention heads and comparisons involving original and perturbed inputs. Although an amortization formula is provided, the actual calibration cost is not reported.
Moreover, TTAug-8 and TTAug-16 are included as runtime references without corresponding accuracy results. This prevents a direct assessment of whether TTT-Head offers a better trade-off than additional inference-time sampling under comparable budgets. Calibration runtime, stage-level latency, peak memory, and compute-matched accuracy comparisons would make the practical benefits more convincing.
Open-ended candidate construction is restrictive and insufficiently analyzed.
Appendix B.3 constructs alternatives by replacing the first answer token with four high-probability alternatives and completing each prefix. This can miss correct answers that share the baseline’s first token but differ later, while potentially allocating candidates to semantically redundant responses.
Although final decoding is unrestricted, candidate construction still determines the selected alternative, supporting heads, and training target. Candidate-set answer coverage, candidate-oracle performance, and sensitivity to candidate budget or generation strategy are therefore important missing analyses.
The head-level analyses are descriptive and do not fully establish the proposed functional interpretations.
The predominance of suppressive actions among successful corrections does not establish that suppression is more effective than amplification without accounting for their frequencies and outcomes across all attempted adaptations. Likewise, task-dependent depth distributions may reflect answer format, output length, or confidence differences alongside task requirements. Suppression-only and amplification-only comparisons, along with depth analyses controlling for answer format, would strengthen these interpretations.
Questions for authors
Please refer to the weaknesses.
8 comments
Response to Reviewer EWRp (2/6)
W1 (continued).
To investigate this question, we conduct additional controlled experiments on 500 independent Analysis Samples from GQA Open using LLaVA-1.5-7B. Following the candidate construction method in Appendix B.3 of the original paper, answer normalization and deduplication yield 1,824 candidate records, including 121 correct and 1,703 incorrect candidates. We compare Aggregated Support, Frozen Margin, and the strongest single-head response, and introduce two controls: Random-head Control randomly selects 16 Heads from the initial Visual Head Shortlist that are not included in the final Eligible Head Pool, while retaining the same candidate set, bidirectional interventions, and maximum-positive-response calculation; Shuffled Support randomly permutes Support values among candidates within the same instance, disrupting the correspondence between Support and individual candidates. Ground Truth is used only for post-hoc evaluation. Pairwise Win Rate is calculated over 278 valid within-instance pairs. Results for Random-head and Shuffled Support are reported as the mean and standard deviation over five random seeds.
Table 1. Correctness Discrimination under Matched Intervention Controls
| Score / Control | AUROC ↑ | Pairwise Win Rate ↑ |
|---|---|---|
| Frozen Margin | 0.620 | 59.5% |
| Aggregated Support | 0.730 | 69.8% |
| Strongest-single-head | 0.680 | 65.4% |
| Random-head Support | 0.540 ± 0.020 | 52.4 ± 2.3% |
| Shuffled Support | 0.510 ± 0.020 | 50.5 ± 1.8% |
As shown in Table 1, Aggregated Support achieves an AUROC of 0.73, higher than Frozen Margin (0.62) and Strongest-single-head (0.68). AUROC measures the ability of a scoring signal to distinguish correct from incorrect candidates: an AUROC of 0.73 means that, when randomly selecting one correct candidate and one incorrect candidate, the correct candidate receives a higher score with a probability of 73%, with ties counted as 0.5. Its within-instance Pairwise Win Rate also reaches 69.8%, exceeding Frozen Margin (59.5%). Notably, Random-head Control retains the same bidirectional intervention and maximum-positive-response mechanism but achieves an AUROC of only 0.54, while Shuffled Support achieves an AUROC close to random chance. These results indicate that the candidate discrimination ability of our method does not arise solely from taking the maximum positive response across bidirectional interventions, nor can it be simply attributed to positive responses from arbitrary Heads. Multi-head aggregation and its correspondence with individual candidates provide additional information for distinguishing candidate correctness.
These results support using Aggregated Support to construct response-based supervision, but we do not directly treat it as a correctness label or probability for candidate answers. Specifically, Support is used to construct the Response-derived Distribution, while the final Soft Supervision Target combines this response distribution with the Frozen Prediction and uses a dynamic weight to control their respective contributions. Therefore, the final supervision target does not rely on Support alone, but retains the original prediction as an anchor while incorporating intervention-derived information.
W2. Complete Computational Cost and Accuracy–Runtime Trade-off
We appreciate your pointing out that the original paper reports only online runtime without fully accounting for Task-level Discovery and Calibration costs, and that the corresponding accuracy results for TTAug-8 and TTAug-16 are missing. We agree that reporting online latency alone is insufficient to fully assess the actual computational cost of TTT-Head.
Response to Reviewer EWRp (3/6)
W2 (continued).
Appendix H, Table 17 of the original paper reports the Instance-level Online Runtime, where TTT-Head takes an average of 5.68 seconds per sample on LLaVA-1.5-7B / GQA Open, approximately 10.9 times that of Frozen Inference. This runtime includes Candidate Construction, Head–Action Probing, Sparse Test-time Training, and final prediction, but excludes the one-time Task-level Head Discovery and Calibration costs.
Regarding Task-level computational cost, we use the same batch of 250 unlabeled Calibration Samples on LLaVA-1.5-7B / GQA Open and repeat the complete Task-level Discovery and Calibration pipeline three times on a single NVIDIA Tesla V100 32GB GPU. All stages are re-executed in each run, and their runtimes and peak GPU memory usage are recorded. Runtime is reported as the mean and standard deviation over the three complete runs to assess the actual computational cost and runtime variation.
Table 2. Task-level Discovery and Calibration Runtime
| Stage | Runtime (min) | Peak GPU Memory (GiB) |
|---|---|---|
| Frozen Inference & Entropy Collection | 4.26 ± 0.15 | 15.25 |
| Visual Head Discovery | 8.52 ± 0.24 | 16.40 |
| Responsiveness & Preservation Screening | 31.56 ± 0.65 | 17.80 |
| Support Calibration | 3.72 ± 0.19 | 18.65 |
| Total Task-level Discovery & Calibration | 48.06 ± 0.76 | 18.65 |
As shown in Table 2, the complete Task-level Discovery and Calibration pipeline takes 48.06 ± 0.76 minutes, with a peak GPU memory usage of 18.65 GiB. Responsiveness & Preservation Screening accounts for the largest portion of the computational cost. Since these operations are performed only once for each Model–Benchmark Configuration, the resulting Head Pool and calibration statistics can be reused for subsequent test samples. Therefore, following Eq. (17) in the original paper, we amortize the one-time calibration cost over different test set sizes:
Based on the online runtime of 5.68 seconds per sample, the amortized average runtimes are 11.45, 8.56, and 5.97 seconds per sample for test set sizes of 500, 1,000, and 10,000, respectively.
Regarding the Accuracy–Runtime Trade-off, we further evaluate TTAug-8 and TTAug-16 on the same 1,000 GQA Open Main Test Samples using a single V100 GPU, while keeping the Prompt and Decoding Settings identical. TTAug-8 and TTAug-16 serve as Augmentation-only Inference References and do not require additional Task-level Discovery or Calibration; therefore, their amortized runtimes equal their online runtimes. This comparison evaluates methods under comparable runtime budgets using the same hardware and evaluation set, rather than strictly matched computational operations.
Table 3. Accuracy–Runtime Comparison on GQA Open
| Method | Accuracy (%) ↑ | Online Time | Amortized Time (N=1,000) |
|---|---|---|---|
| Frozen VLM | 58.40 | 0.52 s | 0.52 s |
| TTAug-8 | 59.60 | 5.12 s | 5.12 s |
| TTT-Head | 61.20 | 5.68 s | 8.56 s |
| TTAug-16 | 60.00 | 9.46 s | 9.46 s |
As shown in Table 3, TTT-Head achieves an accuracy of 61.20%, outperforming TTAug-8 (59.60%) and TTAug-16 (60.00%). After accounting for the one-time calibration cost, its amortized runtime is 8.56 seconds per sample. Compared with TTAug-8, TTT-Head requires approximately 3.44 additional seconds per sample but achieves a 1.6 pp accuracy improvement. Compared with TTAug-16, it requires approximately 0.90 fewer seconds per sample while still achieving a 1.2 pp higher accuracy. Thus, TTT-Head trades additional runtime for higher accuracy compared with TTAug-8, while achieving both higher accuracy and lower amortized runtime than TTAug-16. As the test set size increases, the one-time calibration cost is further amortized. For example, with 10,000 test samples, the estimated average runtime decreases to 5.97 seconds per sample, approaching the online runtime. These results supplement the one-time calibration costs omitted from the original paper and further demonstrate the accuracy–runtime trade-off between TTT-Head and inference-time augmentation methods.
Response to Reviewer EWRp (1/6)
We sincerely appreciate your careful review and positive assessment of our work, particularly your recognition of the problem formulation, instance-level adaptation design, experimental coverage, and supplementary diagnostic analyses. In response to your specific concerns regarding the reliability of Intervention Support, complete computational cost, limitations of candidate construction, and Head-level functional interpretations, we have conducted additional controlled experiments and analyses to further clarify the underlying design rationale and the scope of our conclusions.
Data Split Clarification. The candidate correctness analysis in W1, candidate construction sensitivity analysis in W3, and intervention direction analysis in W4.1 use the previously established independent set of 500 GQA Open Analysis Samples. The computational cost and accuracy comparison in W2 use a separate set of 1,000 GQA Open Main Test Samples. These two evaluation sets are independent of each other and maintain Image-level Disjointness from their corresponding unlabeled Calibration Samples.
W1. Intervention Support and Answer Correctness
We appreciate your pointing out that an increase in the relative preference for an alternative candidate induced by internal interventions does not necessarily mean that the candidate is more accurate. We agree with this observation. In particular, TTT-Head tests both Suppression and Amplification and retains the strongest positive response. Therefore, even an incorrect candidate may naturally produce a positive response in one intervention direction due to the non-zero local slope around the baseline gate. Furthermore, positive responses from multiple Heads should not be directly regarded as independent evidence of correctness.
Regarding the method design and the role of Intervention Support, TTT-Head uses Candidate-Relative Response to measure changes in the preference for an alternative candidate relative to the original answer. Specifically, we compute the change in the score margin between the alternative candidate and the Frozen Answer before and after intervention, rather than examining only the score change of the alternative candidate itself. This avoids interpreting a score increase shared by both answers as stronger support for the alternative candidate. However, such relative score changes cannot completely eliminate local slope effects around the baseline gate, nor can they independently establish candidate correctness. We subsequently aggregate responses from different Heads to obtain Aggregated Support, which is used for candidate selection and the construction of Response-derived Supervision. Aggregated Support primarily reflects the responsiveness of candidate preferences to internal interventions and the resulting relative preference changes, and should not be interpreted as the probability of candidate correctness. We further employ Baseline Anchoring to limit deviations from the original prediction and adjust the contribution of response-based supervision according to Support. However, these designs alone do not establish a relationship between Support and answer correctness.
Therefore, we further examine whether Aggregated Support contains information that distinguishes correct from incorrect candidates, and whether this discriminative ability goes beyond individual-head responses and positive responses that may naturally arise from bidirectional interventions.