acceptodds
Withdrawn from ICLR 2027

On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shifts in CLIP-like contrastive models, its application to generative multimodal large language models (MLLMs) remains underexplored and uniquely challenging due to their sequential token generation. In this work, we demonstrate that representation-based TTA fails to recover performance in MLLMs. Our experiments reveal that even when violating standard TTA assumptions by leveraging target data offline, sample-blind feature alignment recovers only marginal accuracy. We attribute this failure to the nature of corruption shifts: unlike CLIP-like architectures, which benefit from a shared domain-shift direction within a single pooled classification embedding, MLLM visual token sequence shifts are nearly orthogonal and span the full embedding space. These findings bound what any sample-blind, position-shared correction at a single encoder site can achieve, establishing that representation-space interventions alone might be insufficient to ensure corruption robustness in MLLMs.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
MLLMs, VLMs, test-time adaptation, TTA, common corruptions, robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AMJNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether representation-based test-time adaptation (TTA) can restore MLLM performance under common image corruptions. Testing "Gaussian-AcT" across three MLLMs and 15 corruption types from the RealWorldQA dataset, the authors find that it recovers only a small fraction of the performance loss. Through subsequent experiments—including offline fitting, full-covariance alignment, paired ridge/MLP regression, and activation patching—the authors conclude that this failure stems not primarily from estimation errors, mapping expressivity, or the choice of intervention location, but rather from the fact that corruption-induced shifts in MLLMs are highly token-specific; these shifts are approximately orthogonal across samples and exhibit high effective rank. In contrast, pooled representations exhibit more pronounced shared shifts, explaining why traditional feature alignment is more effective for classification tasks. Consequently, the paper’s primary contribution is an empirical, diagnostic negative result rather than the proposal of a new TTA method.

Critical strengths

I believe the paper's primary strength lies in the fact that it moves beyond the mere observation that ''a certain simple TTA method performs poorly.'' Instead, through a series of oracle and ablation experiments, the authors attempt to rule out alternative explanations—such as the estimator, mapping expressivity, supervision, and the intervention site—and further analyze the structure of the corruption shift. This lends the paper diagnostic value within the scope of sample-blind correction.

Additionally, the paper presents a clear core narrative: the authors contrast token-level MLLM outputs with pooled classification representations, employing shift decomposition, cosine similarity, and effective rank to explain the divergent results. The overall writing and experimental organization are easy to follow.

Critical weaknesses

The paper lacks a solution—or at least a constructive validation—that naturally derives from the core analysis. The paper primarily demonstrates the limited effectiveness of existing ''sample-blind representation correction'' methods in the face of MLLM corruption. It further suggests that effective remediation might require sample-specific or token-specific approaches, or even the utilization of contextual information from other visual tokens within the same image. While this diagnostic insight is valuable, the paper essentially stops there without verifying the feasibility of such directions. In particular, the authors could have easily incorporated a relatively simple contextual (or input-dependent) "oracle corrector" capable of accessing the full visual-token sequence to predict token-level corrections. Even if such a method were not practically deployable for TTA, it would have answered a crucial question: does the paper reveal a fundamental limitation of representation-based adaptation itself, or merely a limitation of shared/pointwise correction strategies? In my view, the absence of this experiment makes the work feel more like an analysis of failure cases rather than a fully scientific story.

Questions for authors

There are minor cross-reference formatting inconsistencies: ''Figure 1'' and ''Figure 4'' in the Introduction do not appear as hyperlinks, unlike the surrounding figure/section references.

Figure 1 is not fully self-contained. The abbreviation “MCQ” is used before it is defined, and the figure/caption does not clearly explain what the “classification” task refers to. Since Figure 1 presents the paper’s central contrast, these quantities should be defined directly in the caption or surrounding text.

Reviewer_ykKaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates representation-based test-time adaptation for multimodal LLMs under common image corruptions. Experiments on RealWorldQA show that several feature-correction methods recover only a limited fraction of the average accuracy degradation. The authors decompose corruption-induced activation shifts into global, position-specific, sample-specific, and residual components and compare the cross-image alignment and dimensionality of token-level and pooled shifts. They also replace selected visual tokens or feature dimensions with their clean counterparts to test whether repairing a small subset of the representation can recover performance. They argue that the token-specific structure of these shifts limits the effectiveness of shared representation corrections.

Critical strengths

The paper provides a detailed analysis of why the studied feature-correction approaches achieve limited effectiveness in MLLMs. Moreover, the decomposition of corruption-induced shifts and the comparison between pooled and token-level representations help characterize the limitations of these corrections.

Critical weaknesses
  1. The paper primarily identifies a limitation without establishing a sufficiently substantial practical or methodological contribution. Its main outcome is that several existing feature-correction approaches achieve limited recovery in the tested setting. It does not develop an effective adaptation strategy or demonstrate that the analysis leads to a useful intervention. The suggestion that successful adaptation requires sample- and token-specific restoration remains a broad direction rather than an operational principle supported by a working method. Consequently, the paper offers limited progress toward resolving the robustness problem it identifies.

  2. The diagnostic findings are not sufficiently general or conclusive to compensate for the limited practical contribution. A negative-results paper can be valuable when it establishes a broadly applicable and nontrivial limitation. However, the evaluation is limited to one benchmark, one corruption severity, and three models with similar architectures. It remains unclear whether the findings generalize beyond the tested settings.

Questions for authors
  1. What concrete adaptation strategy or experimentally validated design principle follows from the diagnostic analysis?

  2. What general limitation can be established beyond the observed failure of the particular correction methods evaluated here?

Reviewer_gLAQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the limitations of representation-based TTA for MLLMs under common image corruptions. Through extensive experiments on three MLLMs and 15 corruption types, the authors show that feature alignment methods provide only marginal improvements in visual question answering accuracy, even with stronger correction mappings and paired supervision. The authors further analyze corruption-induced representation shifts and attribute the limited effectiveness of these methods to the highly sample-specific and token-specific nature of such shifts.

Critical strengths
  • The paper is clearly structured, well-written, and easy to follow. The overall presentation is logical and the arguments are communicated effectively.
  • The paper conducts a relatively extensive set of experiments and provides theoretical analyses to support its findings.
Critical weaknesses
  • The practical significance of the studied task is limited. Modern MLLMs have been exposed to visual data at a substantially different scale from earlier CLIP-based models during pretraining, and they already exhibit relatively strong robustness to mild image corruptions. However, this paper focuses exclusively on severity level 5, which represents extremely severe image degradation. Such heavily degraded images are rarely encountered in typical real-world MLLM applications. Therefore, I question the practical relevance of the experimental setting and whether the conclusions are representative of realistic scenarios.

  • For images suffering from severe degradation, rather than expecting an MLLM to directly answer visual questions correctly, a straightforward solution would be to first restore the degraded images using a dedicated image restoration model and then feed the restored images into the MLLM. This is a natural and practical approaches to handling severe image corruptions. I'm curious about the effectiveness of such baseline.

  • The methodological novelty is limited. Gaussian-AcT follows essentially the same underlying principle as previous feature alignment-based TTA methods developed for CLIP-like models. The proposed approach does not introduce a fundamentally new mechanism or methodological insight beyond these existing methods.

  • The experiments in Section 3.2 are not sufficiently convincing. The authors train a two-layer MLP using paired clean and corrupted visual tokens and observe that it still achieves only marginal accuracy recovery. However, this experiment is insufficient to support a general conclusion about the limitations of nonlinear feature correction: 1) a two-layer MLP has limited representational capacity and may not be expressive enough to learn the complex mapping; 2) the training set is relatively small, consisting of only 500 images, which may further limit the model's ability to learn an effective restoration mapping. Therefore, the current experimental setting is not sufficiently convincing to establish that more expressive nonlinear correction methods would also fail.

Questions for authors

See the weaknesses session~

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.