acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Training for Modality Order Consistency in Vision-Language Models

Abstract

We find that vision-language models are sensitive to a specific semantically irrel- evant change: the order in which the image and question are presented. Across three models on three core benchmarks, with additional evaluation on three further benchmarks, image-first prompting consistently outperforms question-first prompt- ing, revealing a repeatable modality order failure. We use this gap to design an order-consistent test-time training method. Our method substantially closes the modality-order gap across all evaluated settings. The stronger image-first branch is largely preserved, with observed improvements in several settings, hence boot- strapping both orderings toward mutual consistency. Activation patching localizes the ordering failure to a narrow mid-network region where representations diverge sharply between prompt orders. We find that the test-time training method repairs this misalignment across layers. Together, our results identify modality-order sen- sitivity as a circuit-level failure in VLMs and demonstrate that simple, asymmetric test-time adaptation can effectively mitigate it and even improve performance over the baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time training, vision language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9iSQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies modality-order sensitivity in vision-language models (VLMs), observing that presenting the image before the question ("image-first") consistently yields higher accuracy than the reverse ("question-first") across three models and multiple benchmarks. The authors frame this as a semantically irrelevant perturbation that nonetheless induces a structured, repeatable failure. To address it, they propose an asymmetric test-time training (TTT) procedure: for each test instance, both prompt orders are constructed, and the stronger image-first prediction serves as a detached (moving) teacher while the question-first prediction is adapted toward it via a KL divergence objective, with parameters reset between instances. Adaptation is restricted to a mid-network layer window, motivated by activation-patching analysis that localizes the ordering failure to a narrow mid-layer band. The paper further reports ablations on the loss function (asymmetric vs. symmetric), layer window, and moving vs. frozen teacher, and presents mechanistic evidence that TTT increases late-layer MLP overlap between prompt orders.

Critical strengths
  1. The paper identifies a clear, reproducible robustness failur—modality-order sensitivity in VLMs—and supports it with consistent evidence across three models and multiple benchmarks, making the phenomenon convincing and practically relevant.

2.The proposed method use activation patching localizes the failure to mid-network layers, and the asymmetric TTT objective follows this analysis. Improvement on QF style input is substantial.

Critical weaknesses
  1. Weakness on why this method matter

It is well known that LVLMs place the image first during both training and testing. Changing the positions means the information layout differs from training, so it is expected to find a drop in performanc. The method perform distillation on QF. QF itself is already poor, so learning from IF, the improvement could be expecte——which means this does not count as a contribution. For the reason in the improvement in IF results, I think this is the most critical part, but the authors' explanation for this improvement is only a conjecture; the claim about a "sharper" distribution is also not supported by any experiment. I suggest the authors add experiments to increase the persuasiveness of the method.

  1. Limited novelty

The overall method looks like just a distillation, the dynamic teacher already exists in prior work. The local-layer choice looks like a cherry-picked result from the ablation, with no novelty.

  1. The ablation does not investigate the mechanism thoroughly enough.

In Section 5, Figure 3a shows that when IF hidden-state token replace the tokens in QF at later layers has a relatively large effect on the result——looks like 20 points. But in the ablation results in Table 3, when different layers are used, the gap between the best QF result (61.7) and the worst (60.7) is no more than 1 point. How this happen? If Figure 3 suggests such a large effect, why is the final effect so small? Is this really effective? I suggust author could add more experiments to improve improve the interpretability.

  1. The models are outdated.

InternVL2 is still being used.

Questions for authors
  1. So the first question raise in my mind when reading this paper is: if LLaVA is trained by their open-source data with the text always placed first and the image after it. Then tested in QF form, would it show a large improvement in QF results. I think the asnwer is yes. Then, the question comes: whether image-first or text-first is easier to learn? A study on the underlying mechanism might be a good research. I would like to know what the authors think about this question, and what exactly their work contributes to the field.
3 comments

I used an LLM to help look up relevant related work and verify the accuracy of the references in the paper. I also used it to retrieve information to help me understand the method, and refresh my review writing.

We thank the reviewer for their assessment, we discuss their concerns below.

Regarding W1 and W2. Our central contribution is not simply distilling question-first (QF) predictions from image-first (IF). We demonstrate that cross-order disagreement provides a label-free adaptation signal whose benefits can extend to the initially stronger IF ordering. Because the teacher and student share parameters, adapting QF also modifies IF predictions. Recomputing the teacher after each update allows both orderings to evolve, rather than distilling toward a fixed prediction. We distinguish our empirical observations from their proposed mechanistic explanation:

  • Tables 1–2: IF accuracy improves in 7/9 primary settings (up to +1.2 percentage points) and across all three additional benchmarks.
  • Table 3: Asymmetric KL achieves 61.5% QF accuracy, compared with 45.3% for symmetric CE, establishing the importance of directional consistency.
  • Table 5: The moving teacher achieves 62.3% IF and 61.5% QF accuracy, compared with 61.5% IF and 43.9% QF for the frozen teacher under the same adaptation budget.

These results establish that the adaptation objective and teacher-update strategy affect predictive performance, with observed benefits extending to IF. We interpret the IF improvements as consistent with iterative bootstrapping, rather than claiming that progressively sharper teacher distributions have been established as the causal mechanism. The precise contribution of confidence changes versus shared-representation updates remains to be characterized and is an interesting future direction. Thus, while moving teachers and distillation are established techniques, our contribution lies in exploiting the directional reliability asymmetry between semantically equivalent modality orderings for test-time adaptation, with observed improvements beyond merely imitating the initial teacher.

Regarding W3. Figure 3 and Table 4 investigate fundamentally different interventions, and their effect sizes need not coincide.

Figure 3 directly replaces the QF final-token representation with its IF counterpart at a selected layer, measuring where IF information becomes sufficient to repair QF predictions. Recovery increases sharply around layers 18–20.

Table 4, in contrast, restricts gradient-based adaptation to different layer windows. All evaluated windows recover over 25 percentage points of QF accuracy, demonstrating that different parameter subsets can achieve similar output-level improvements.

Appendix D further shows that these windows achieve high output agreement through distinct internal representation changes. Thus, the small variation in final QF accuracy does not contradict the sharp depth-dependent effect observed under activation patching. The mid-layer window is motivated by its correspondence with the patching results and its favorable IF preservation, rather than being selected solely for the highest QF accuracy. We will make this more explicit in a revised manuscript.

Regarding W4. Our evaluation includes Qwen2.5-VL-7B and Qwen3-VL-8B, in addition to InternVL2-8B, with consistent QF improvements across all nine primary model–dataset settings. We emphasize InternVL2 in the mechanistic analysis because its larger initial ordering gap makes the intervention effects easier to study. Appendix C also reports activation-patching results on Qwen3-VL-8B, showing a similar qualitative depth-dependent recovery pattern across model families.

Regarding Q1. We agree that training-time modality-order conventions may influence which ordering performs better, and a model trained predominantly on question-first inputs could plausibly exhibit a different preference. However, our approach does not assume that image-first is intrinsically easier to learn or universally preferable. Instead, we reiterate that our proposed method exploits the empirically stronger ordering as a directional supervisory signal. The key contribution is therefore not identifying which ordering should be preferred during training, but demonstrating that an existing reliability asymmetry can be leveraged for label-free test-time model adaptation, with improvements potentially extending to the stronger ordering itself as a bootstrapped iterative method.

Investigating how training-time ordering determines this asymmetry is a complementary research direction, rather than a prerequisite for the proposed adaptation method and remains and interesting future direction.

Thank you for the authors’ response. However, with regard to the improvement observed in the IF results, I consider that substantial issues remain in this study. The underlying mechanism has not been clearly elucidated, and the rationale behind the reported improvement remains ambiguous. Given these concerns, I will retain my original score.

Reviewer_GVh9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors proposed an asymmetric test-time training method to address modality-order sensitivity in vision-language models. The method aligns question-first predictions with a moving image-first teacher through a KL-divergence objective. The authors also use activation patching and layer-window ablations to investigate order-dependent internal computations and guide adaptation.

Critical strengths
  • The method is intuitive. Using the empirically stronger image-first prediction to guide the weaker question-first prediction provides a simple rationale for asymmetric consistency training.

  • The evaluation includes multiple models and tasks.

Critical weaknesses
  • The authors provide a useful investigation of modality-order sensitivity. However, the practical advantage over simply using image-first prompting remains unclear, raising questions about the value of closing the gap between image-first and question-first performance. The method already requires an image-first forward pass, yet the adapted question-first accuracy generally only approaches the original image-first baseline.

  • In section 5.1, replacing the final-token representation identifies depths where donor information becomes sufficient to recover an answer. This does not establish that the underlying error originates at those depths, particularly when other token representations remain unpatched

  • The benefit of the moving teacher is not sufficiently explained. Table 5 shows that updating the teacher improves performance, but it remains unclear whether the teacher becomes more accurate or simply more confident during adaptation.

  • The reported experimental results are inconsistent. For example, Tables 1 and 6 report different image-first and question-first baseline accuracies for Qwen2.5-VL on RealWorldQA.

Questions for authors
  • Since the method already requires an image-first forward pass, and adapted question-first performance generally only approaches the original image-first baseline, why not directly use the image-first answer? What practical benefit justifies the additional computation?

  • Since the moving teacher and student share parameters, how do the authors establish that the teacher becomes more accurate rather than merely more confident? In particular, what prevents adaptation from reinforcing an incorrect initial image-first prediction?

1 comment

We thank the reviewer for the thoughtful assessment and for recognizing the consistent improvements and informative ablations.

Regarding W1 and Q1. The distinction from simply using image-first (IF) prompting is central to our contribution. Our proposed method exploits cross-order disagreement as a label-free adaptation signal that can improve the initially stronger IF branch itself. It does not merely transfer IF predictions to question-first (QF). Because the teacher and student share parameters, adapting QF also changes IF predictions. Recomputing the teacher after each update enables an iterative bootstrapping process rather than distillation toward a fixed prediction.

  • Tables 1–2: IF accuracy improves in 7/9 primary settings (up to +1.2 percentage points) and across all three additional benchmarks.
  • Table 5: The moving teacher improves both IF (61.6→62.3%) and QF (35.5→61.5%), whereas the frozen teacher yields 61.5% IF and only 43.9% QF under the same adaptation budget.

These findings support our central insight which is that canonicalization selects an existing prediction, whereas consistency-based TTT adapts the model and can improve both orderings. We do not claim a computational advantage over canonicalization, the contribution is a self-supervised adaptation mechanism with observed benefits extending beyond the weaker ordering. We will clarify this in the manuscript.

Regarding W2. Our activation-patching experiments are designed to identify where IF information can causally repair QF computation, rather than to isolate the original source of the ordering discrepancy. Figure 3 demonstrates a sharp transition in repair effectiveness around layers 18–20, while Table 4 evaluates the effectiveness of adaptation across layer windows. Together, these provide complementary evidence for identifying effective intervention depths. The results do not require the initial ordering discrepancy to originate at those layers.

Regarding W3 and Q2. The improvements reported in Table 5 concern actual predictive accuracy and not confidence. With a moving teacher, IF accuracy increases from 61.6% to 62.3%, whereas the frozen-teacher variant decreases it to 61.5%. The corresponding QF accuracies are 61.5% and 43.9%, respectively. Thus, recomputing the teacher produces differences in both branches' accuracy under identical adaptation budgets. We will make this more explicit in the manuscript.

The concern about reinforcing incorrect predictions is explicitly investigated in Section 4.2 and Appendix I. On MMMU, TTT repairs 86/130 initially IF-correct/QF-wrong examples, while corrupting 32/69 initially IF-wrong/QF-correct examples. Importantly, it also recovers 31 examples where both initial predictions were incorrect. Overall, QF accuracy increases from 329/800 to 384/800.

These results demonstrate that adaptation is not simply copying the initial teacher's predictions and it can recover examples where the teacher is initially incorrect, although individual regressions are also possible. The asymmetric objective exploits the stronger ordering's aggregate reliability rather than assuming that every teacher prediction is correct. While the precise contribution of confidence changes versus improved representations remains to be characterized, the reported results establish improvements in predictive correctness.

Regarding W4. Thank you for highlighting this difference. The baseline values in Tables 1 and 6 originate from separate evaluation runs, explaining the numerical differences (67.8%/60.0% versus 70.2%/63.8% for IF/QF on RealWorldQA). Both runs support the same qualitative finding that IF prompting outperforms QF prompting. We will clarify the distinction between these runs and harmonize the baseline reporting in the revised manuscript.

Reviewer_QGjaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the sensitivity of vision-language models to the relative order of images and questions. It proposes an instance-wise test-time adaptation method that uses the image-first prediction as a stop-gradient teacher and updates selected language-model layers to align the question-first distribution through KL divergence. The teacher is recomputed after each update, and parameters are reset between examples. Experiments cover three models from two families on three core benchmarks, with three additional benchmarks evaluated using InternVL2-8B. The results show substantial recovery of question-first accuracy while largely preserving image-first performance, with reported image-first gains of up to 1.2 percentage points. Activation patching and layer-window ablations are used to motivate a mechanistic interpretation. Per-instance adaptation requires no answer labels, although model-level hyperparameters are selected using a labeled calibration subset.

Critical strengths

S1. Consistent recovery across models and tasks. Tables 1–2 report substantial question-first improvements across all evaluated settings while preserving aggregate image-first accuracy. Reusing model-specific adaptation configurations across benchmarks provides useful evidence that the repair is not restricted to one task. This supports the method’s value as a targeted robustness intervention.

S2. Informative ablations of the adaptation dynamics. Tables 3 and 5 examine asymmetric versus symmetric objectives, soft versus hard targets, and moving versus frozen teachers. In particular, the moving-teacher variant achieves 61.5% question-first accuracy compared with 43.9% for the frozen-teacher variant on InternVL2-8B/MMStar. These results identify consequential design choices and provide a useful empirical observation about consistency-based test-time adaptation.

Critical weaknesses

W1. The necessity and incremental benefit of test-time adaptation over input canonicalization are not sufficiently established. Since the method already constructs the image-first input to obtain its teacher distribution, directly returning the image-first prediction is an immediately available alternative. In Table 1, adapted question-first accuracy remains below the original image-first baseline in eight of nine settings, while improvements to image-first accuracy are limited to 0–1.2 percentage points and are reported without uncertainty estimates. These results demonstrate recovery of order robustness more clearly than an accuracy advantage over simply standardizing the input order. Given the additional forward and backward passes required by TTT, the practical motivation would be stronger with either a concrete setting where image-first canonicalization is insufficient, or reliable improvements over this strategy and other prompt-only alternatives under a comparable computational budget.

W2. The mechanistic interpretation does not sufficiently isolate contextual-position effects and differences in visual-token budgets. Related work, such as [1][2], has linked question-first degradation to reduced access to the question at answer generation, with the intervening image-token sequence increasing the question–answer distance. This provides a plausible alternative explanation that should be distinguished from the proposed circuit-level account. In particular, InternVL and Qwen-VL use different image-processing pipelines, which can produce different visual-sequence lengths even for the same image. The supplied main text does not report the resulting token-count distributions or control for question–answer distance. Consequently, the larger absolute recovery on InternVL could reflect differences in visual-sequence length and a larger initial ordering gap, rather than independently supporting the proposed mechanism. Reporting actual visual-token counts and conducting within-model token-budget sweeps and distance-controlled or question-echoing comparisons would help clarify which factors drive the observed effect.

[1] Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models [2] CoMemo: LVLMs Need Image Context with Image Memory

Questions for authors

See weakness.

1 comment

We thank the reviewer for the thoughtful assessment and for recognizing the consistent improvements and informative ablations.

Regarding W1. We agree that directly using image-first (IF) prompting is a simpler and computationally cheaper alternative. However, our contribution goes beyond transferring IF predictions to question-first (QF): we show that cross-order disagreement can provide a label-free adaptation signal that also improves the initially stronger IF branch. Crucially, the IF teacher and QF student share parameters, and the teacher is recomputed after each update. Thus, adapting QF also modifies IF, enabling an iterative bootstrapping method rather than distillation toward a fixed prediction.

Our results support this interpretation:

  • Tables 1–2: IF accuracy improves in 7/9 primary settings (up to +1.2 points) and across all three additional benchmarks.
  • Table 5: The moving teacher improves both IF (61.6→62.3%) and QF (35.5→61.5%), whereas the frozen teacher yields 61.5% IF and only 43.9% QF, under the same adaptation budget.

While the IF gains are modest, these results support our central insight that canonicalization selects the stronger prediction, whereas consistency-based TTT can improve the model's predictions under both orderings.

Regarding W2. We agree that question–answer distance and visual-token budgets may contribute to modality-order sensitivity, and our results do not rule out these explanations. We wish to clarify that our activation-patching analysis identifies where IF representations can causally repair QF predictions, not necessarily where the original failure originates. Specifically, Figure 3 shows that patching becomes effective around layers 18–20, while Table 4 evaluates the effectiveness of adapting different layer windows.

These findings are compatible with positional explanations. Our central contribution is complementary in a way that regardless of the source of the discrepancy, modality-order disagreement can be exploited as a self-supervised signal for test-time adaptation. We will make this distinction more explicit and acknowledge the potential effects of visual-token budgets and contextual position.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.