acceptodds
Withdrawn from ICLR 2027

VEC-TTA: Visual-Evidence-Calibrated Test-Time Adaptation for Compact Multimodal Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time computation provides an opportunity to improve multimodal large language models (MLLMs) beyond their pretrained capabilities.However, reinforcing model predictions does not ensure that the resulting answers are driven by query-relevant visual evidence.Models can favor answers elicited without visual evidence even when the original image is present, and retain original answers after critical visual evidence is altered.To address these limitations, we propose Visual Evidence Calibrated Test-Time Adaptation (VEC-TTA), an instance-specific adaptation framework for compact MLLMs that performs temporary low-rank updates while keeping the backbone frozen.Specifically, VEC-TTA comprises three complementary modules:(1) Query-Conditioned Evidence Elicitation (QEE), which constructs pseudo-supervision from the current image–query instance;(2) Invariant Evidence Consolidation (IEC), which supplements answer fitting with visual consistency, jointly fitting pseudo-targets and aligning predictions across online content-preserving views; and(3) Counterfactual Evidence Calibration (CEC),which constrains persistent answer support by encouraging stronger support for the same pseudo-target when visual content is present than when it is removed.Experiments on two compact MLLMs across multiple benchmarks show consistent improvements over representative test-time methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Model, Test Time Adaption

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HQfVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VEC-TTA, a per-sample test-time adaptation method for small multimodal language models. The method first constructs five query-related pseudo-supervision samples, then updates a temporary LoRA adapter through invariant evidence consolidation, which enforces consistency between answer fitting and weak image augmentation, and counterfactual evidence calibration, which encourages pseudo-answers to receive stronger support when the image is present than when a fully black image is used. Experiments on two models of approximately 2 billion parameters and five benchmarks show that the method improves image captioning and question answering performance. The paper further validates the approach through module ablation, occlusion of matched evidence regions and control regions, analysis of the adaptation process, and measurement of computational cost.

Critical strengths
  1. The paper provides a more direct test of visual reliance than overall task accuracy. Under a fixed post-adaptation model, Table 3 compares occlusion of evidence regions with occlusion of non-overlapping control regions of equal area. VEC-TTA improves original-image accuracy and the evidence selectivity gap on both models, while accuracy after control occlusion remains close to original-image performance. This offers substantive evidence for the intended use of visual grounding, going beyond merely suppressing answers on fully black images.

  2. The method achieves generally favorable results across two models and multiple task types, with particularly clear gains for SmolVLM on image captioning and VizWiz. Table 2 supports the complementary roles of IEC and CEC, and the paper explicitly analyzes low-quality pseudo-supervision and failure cases. In addition, Table 6 reports that single-sample end-to-end time decreases from 207.50 seconds to 103.69 seconds relative to TTAdapt, giving the comparison clearer practical reference value.

Critical weaknesses
  1. The core contribution has not yet been sufficiently distinguished from ordinary query-conditioned self-training. Table 2 removes IEC or CEC at the full-stage level, but does not include a baseline trained solely with cross-entropy under the same QEE sample set, LoRA placement, and update budget. Consequently, the large improvement attributed to IEC alone cannot be specifically tied to the cross-view consistency constraint. Without this control, the contribution of the evidence-constrained objective is difficult to separate from that of pseudo-supervision and the adaptation configuration itself.

  2. Existing evaluations do not yet adequately distinguish improvements in visual understanding from effects arising from output style and differences in computational budget. Appendix C.1 acknowledges that caption length and style affect metrics, and several baselines in Table 5 show CIDEr-D values near zero. These anomalously low values require verification under unified prompting, decoding, and scoring settings. Table 6 valuably reports runtime comparisons with TTAdapt, but has not yet established accuracy-computation comparisons against all test-time computation baselines. These issues limit the interpretation of the large gains in image captioning and the assessment of the method's practical advantages over other test-time approaches.

Questions for authors
  1. With QEE, LoRA placement, and update budget held fixed, how do cross-entropy fitting alone, the addition of weak-view augmentation, the further addition of symmetric KL, and the full method perform respectively?

  2. If the no-visual-content reference is replaced with a mean-color image or a randomly paired unrelated image, do the benefits of CEC still persist? Table 3 already provides valuable local occlusion evidence; this additional control would help determine how much the effect depends on the fully black image as a specific reference.

Reviewer_5ck8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Motivated by the phenomenon that existing MLLMs may be dominated by language priors and neglect key visual evidence, this paper proposes VEC-TTA, a test-time adaptation framework. VEC-TTA constructs pseudo-supervision from input image-text pairs, which are further utilized to update instance-specific LoRA to encourage consistent predictions to rely on visual inputs. Experiments on two 2B-size models validate the effectiveness of VEC-TTA.

Critical strengths
  1. This paper is strongly motivated by an important research problem in MLLMs, which is supported by a preliminary experiment.
  2. The proposed method is intuitive and carefully designed; at the same time, the training-free nature of VEC-TTA could enlarge the scope of the availability of this method.
  3. Several interesting experiments are conducted in this paper, revealing the effects of different components of VEC-TTA.
Critical weaknesses

Suggestions on Presentation Improvement

  1. Citation formatting should be improved. In several places, in-text references should use \citep{} rather than the current citation format.

  2. Figure 1 has visual presentation issues. The axis titles are placed too close to the axes, which reduces readability.

  3. The experimental setup for Figures 1(a) and 1(b) is underspecified. For example, it is unclear what the horizontal axis in Figure 1(b) represents—does each column correspond to a different sample? Since the experiments in Figure 1 provide the main motivation for the method, their setup and interpretation should be described explicitly in Section 3 rather than being left mainly to the appendix.

  4. Computational efficiency is an important part of the evaluation but is relegated to the appendix. The current analysis is also not sufficiently detailed. Since test-time adaptation inevitably introduces a performance–efficiency trade-off, efficiency should be integrated into the main experiments. The paper should evaluate whether VEC-TTA achieves a better quality–efficiency trade-off, rather than focusing primarily on performance improvements.

Concerns about the Experimental Setup

  1. The metrics used in Figure 1 are not fully justified. (i) It is unclear whether the null-visual condition should be implemented using an all-black image, removing the image entirely, or using another no-visual-input condition. (ii) It is also unclear why the analysis only considers the first answer token, especially for multi-token answers. (iii) The interpretation of Figure 1(c) is also questionable. Performance on OCRVQA and GQA already drops substantially under the counterfactual images, so the statement that the model “retains substantial performance” appears overstated. The paper should report appropriate baselines, such as random-guessing performance, black-image performance, and no-image or language-only performance.

  2. The validation platform is limited.

2.1 The method is evaluated only on two approximately 2B-scale models released in 2025. This does not sufficiently establish its generality across newer models, different model scales, or alternative architectures such as MoE models.

2.2 The benchmark coverage is also limited. COCO with ROUGE-L is not an especially convincing benchmark for evaluating modern MLLM image-captioning ability. Beyond hallucination-oriented benchmarks, the authors should evaluate VEC-TTA on more widely used multimodal benchmarks, such as MMStar, MMMU, MathVision, RealWorldQA, OCR-related tasks, and document-understanding benchmarks.

Concerns about the Motivation

The motivation for using TTA is not sufficiently justified.

  • The paper addresses a relatively general problem of improving visual grounding, but it does not explain why test-time adaptation is preferable to additional training or post-training.

  • Why not directly correct the model’s behavior through supervised or preference-based training? The paper should clarify the scenarios in which test-time adaptation provides a meaningful advantage over training-based approaches, such as the absence of additional training data, distribution-specific adaptation, or the need to preserve the frozen backbone.

Questions for authors

Please answer the questions raised in the weakness part.

Reviewer_HtPeOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes VEC-TTA, an episodic, per-instance test-time adaptation method for compact MLLMs.

The motivation is a logit-lens diagnostic (Fig. 1) showing that a language prior dominates late layers for many instances. Experiments use SmolVLM2-2.2B and Qwen3-VL-2B on 1,000-item strided subsets of COCO, AMBER, GQA, VizWiz and 951 HallusionBench items. The headline result is COCO ROUGE-L 8.8 → 45.3 on SmolVLM (+29.4 over the best baseline), with smaller gains elsewhere. Supporting analyses cover LoRA update geometry

Critical strengths
  1. Well-posed problem. Whether test-time gains are visually grounded, rather than only more accurate, is the right question for TTA on MLLMs. The two failure modes in Sec. 1 (prior-elicited answers winning with the image present; answers surviving evidence removal) are stated clearly.

Critical weaknesses

The largest gains are most likely output-format/length adaptation, not grounding. Vanilla SmolVLM scores 8.8 ROUGE-L on COCO and 8.9% on VizWiz. These are format-failure numbers, not perception-failure numbers. Fitting five short self-generated answers for 5–10 steps plausibly teaches the model to answer tersely, which ROUGE-L and VizWiz exact-match reward heavily.

VEC-TTA takes ≈ 94 s of adaptation per instance on an A6000 for a 2B model (Table 6), orders of magnitude above Vanilla. Self-Consistency and SCI are not run at matched compute, and TTAug (training-free, same paper as TTAdapt, reported stronger than voting) is absent from Table 1.

Questions for authors
  1. Length and format. What are the mean output lengths (tokens) on COCO and VizWiz for Vanilla, IEC-only and VEC-TTA? What does Vanilla score with a concise-answer prompt, and with VLMEvalKit's default prompts?
  2. Prompt and decoding parity. Do the Vanilla row and QEE's Decode use the same prompt, chat template, and greedy decoding with the same max length? On what fraction of instances does the final prediction differ from the pseudo-target y, and what is the accuracy of y, y^s and y^e on GQA, AMBER and VizWiz?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.