- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies modality-order sensitivity in vision-language models (VLMs), observing that presenting the image before the question ("image-first") consistently yields higher accuracy than the reverse ("question-first") across three models and multiple benchmarks. The authors frame this as a semantically irrelevant perturbation that nonetheless induces a structured, repeatable failure. To address it, they propose an asymmetric test-time training (TTT) procedure: for each test instance, both prompt orders are constructed, and the stronger image-first prediction serves as a detached (moving) teacher while the question-first prediction is adapted toward it via a KL divergence objective, with parameters reset between instances. Adaptation is restricted to a mid-network layer window, motivated by activation-patching analysis that localizes the ordering failure to a narrow mid-layer band. The paper further reports ablations on the loss function (asymmetric vs. symmetric), layer window, and moving vs. frozen teacher, and presents mechanistic evidence that TTT increases late-layer MLP overlap between prompt orders.
Critical strengths
- The paper identifies a clear, reproducible robustness failur—modality-order sensitivity in VLMs—and supports it with consistent evidence across three models and multiple benchmarks, making the phenomenon convincing and practically relevant.
2.The proposed method use activation patching localizes the failure to mid-network layers, and the asymmetric TTT objective follows this analysis. Improvement on QF style input is substantial.
Critical weaknesses
- Weakness on why this method matter
It is well known that LVLMs place the image first during both training and testing. Changing the positions means the information layout differs from training, so it is expected to find a drop in performanc. The method perform distillation on QF. QF itself is already poor, so learning from IF, the improvement could be expecte——which means this does not count as a contribution. For the reason in the improvement in IF results, I think this is the most critical part, but the authors' explanation for this improvement is only a conjecture; the claim about a "sharper" distribution is also not supported by any experiment. I suggest the authors add experiments to increase the persuasiveness of the method.
- Limited novelty
The overall method looks like just a distillation, the dynamic teacher already exists in prior work. The local-layer choice looks like a cherry-picked result from the ablation, with no novelty.
- The ablation does not investigate the mechanism thoroughly enough.
In Section 5, Figure 3a shows that when IF hidden-state token replace the tokens in QF at later layers has a relatively large effect on the result——looks like 20 points. But in the ablation results in Table 3, when different layers are used, the gap between the best QF result (61.7) and the worst (60.7) is no more than 1 point. How this happen? If Figure 3 suggests such a large effect, why is the final effect so small? Is this really effective? I suggust author could add more experiments to improve improve the interpretability.
- The models are outdated.
InternVL2 is still being used.
Questions for authors
- So the first question raise in my mind when reading this paper is: if LLaVA is trained by their open-source data with the text always placed first and the image after it. Then tested in QF form, would it show a large improvement in QF results. I think the asnwer is yes. Then, the question comes: whether image-first or text-first is easier to learn? A study on the underlying mechanism might be a good research. I would like to know what the authors think about this question, and what exactly their work contributes to the field.
3 comments
We thank the reviewer for their assessment, we discuss their concerns below.
Regarding W1 and W2. Our central contribution is not simply distilling question-first (QF) predictions from image-first (IF). We demonstrate that cross-order disagreement provides a label-free adaptation signal whose benefits can extend to the initially stronger IF ordering. Because the teacher and student share parameters, adapting QF also modifies IF predictions. Recomputing the teacher after each update allows both orderings to evolve, rather than distilling toward a fixed prediction. We distinguish our empirical observations from their proposed mechanistic explanation:
- Tables 1–2: IF accuracy improves in 7/9 primary settings (up to +1.2 percentage points) and across all three additional benchmarks.
- Table 3: Asymmetric KL achieves 61.5% QF accuracy, compared with 45.3% for symmetric CE, establishing the importance of directional consistency.
- Table 5: The moving teacher achieves 62.3% IF and 61.5% QF accuracy, compared with 61.5% IF and 43.9% QF for the frozen teacher under the same adaptation budget.
These results establish that the adaptation objective and teacher-update strategy affect predictive performance, with observed benefits extending to IF. We interpret the IF improvements as consistent with iterative bootstrapping, rather than claiming that progressively sharper teacher distributions have been established as the causal mechanism. The precise contribution of confidence changes versus shared-representation updates remains to be characterized and is an interesting future direction. Thus, while moving teachers and distillation are established techniques, our contribution lies in exploiting the directional reliability asymmetry between semantically equivalent modality orderings for test-time adaptation, with observed improvements beyond merely imitating the initial teacher.
Regarding W3. Figure 3 and Table 4 investigate fundamentally different interventions, and their effect sizes need not coincide.
Figure 3 directly replaces the QF final-token representation with its IF counterpart at a selected layer, measuring where IF information becomes sufficient to repair QF predictions. Recovery increases sharply around layers 18–20.
Table 4, in contrast, restricts gradient-based adaptation to different layer windows. All evaluated windows recover over 25 percentage points of QF accuracy, demonstrating that different parameter subsets can achieve similar output-level improvements.
Appendix D further shows that these windows achieve high output agreement through distinct internal representation changes. Thus, the small variation in final QF accuracy does not contradict the sharp depth-dependent effect observed under activation patching. The mid-layer window is motivated by its correspondence with the patching results and its favorable IF preservation, rather than being selected solely for the highest QF accuracy. We will make this more explicit in a revised manuscript.
Regarding W4. Our evaluation includes Qwen2.5-VL-7B and Qwen3-VL-8B, in addition to InternVL2-8B, with consistent QF improvements across all nine primary model–dataset settings. We emphasize InternVL2 in the mechanistic analysis because its larger initial ordering gap makes the intervention effects easier to study. Appendix C also reports activation-patching results on Qwen3-VL-8B, showing a similar qualitative depth-dependent recovery pattern across model families.
Regarding Q1. We agree that training-time modality-order conventions may influence which ordering performs better, and a model trained predominantly on question-first inputs could plausibly exhibit a different preference. However, our approach does not assume that image-first is intrinsically easier to learn or universally preferable. Instead, we reiterate that our proposed method exploits the empirically stronger ordering as a directional supervisory signal. The key contribution is therefore not identifying which ordering should be preferred during training, but demonstrating that an existing reliability asymmetry can be leveraged for label-free test-time model adaptation, with improvements potentially extending to the stronger ordering itself as a bootstrapped iterative method.
Investigating how training-time ordering determines this asymmetry is a complementary research direction, rather than a prerequisite for the proposed adaptation method and remains and interesting future direction.
Thank you for the authors’ response. However, with regard to the improvement observed in the IF results, I consider that substantial issues remain in this study. The underlying mechanism has not been clearly elucidated, and the rationale behind the reported improvement remains ambiguous. Given these concerns, I will retain my original score.
I used an LLM to help look up relevant related work and verify the accuracy of the references in the paper. I also used it to retrieve information to help me understand the method, and refresh my review writing.