- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether continual test-time adaptation (CTTA) of multimodal LLMs produces improvements that persist beyond the adaptation stream. It introduces a two-stage multiple-choice VQA benchmark: models first adapt on an unlabeled stream under paired interface and image perturbations, then undergo frozen evaluation on shared held-out questions against inference-matched frozen baselines. Experiments with ten existing methods across two Qwen2.5-VL sizes show limited retained improvements and persistent, order-dependent answer biases. The paper also proposes Deliberate-then-Consolidate (DtC), which distills the current model’s CoT answer distribution into direct predictions. Validation includes controlled training targets, ablations, source-order changes, multiple question splits, and additional model/data checks. DtC shows positive retained gains on the main split, but these gains are not consistently reproduced.
Critical strengths
The evaluation separates inference improvements from retained adaptation effects. Comparing adapted and frozen models on the same held-out questions with the same inference procedure is a useful control. The paired presentation histories additionally help isolate the effects of earlier interface changes. I also appreciate the reporting of unsuccessful replications, computational costs, and statistical qualifications. These make the empirical findings more interpretable than a comparison based solely on online accuracy.
Critical weaknesses
-
Limited scope and practical relevance. The evaluation focuses on four-way option selection, whose position biases and exploitable option information are already well documented [1–3](please cite these papers). Although the paper studies their persistence after adaptation, it does not establish relevance to open-ended visual answering. This limits the broader significance of the findings and the practical value of improving direct-option accuracy.
-
Limited methodological novelty and inconsistent gains. DtC combines established CoT self-distillation with option canonicalization and averaging, without demonstrating a clear advantage over fixed-teacher distillation. In Table 15, its mean retained gain is only +0.6 points, versus +0.9 for the fixed CoT target. Gains vary across splits, and the main-split improvement does not survive the reported multiple-comparison correction. These results do not establish a reliable benefit from the proposed online updates.
[1] Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? ACL, 2024.
[2] Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large Language Models Are Not Robust Multiple Choice Selectors. ICLR, 2024; arXiv:2309.03882, 2023.
[3] Yesheng Liu et al. Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT. CVPR, 2026, pp. 18859–18869.
Questions for authors
Clarification of the methodological contribution: What benefit is specifically attributable to the evolving teacher in DtC beyond distilling a fixed frozen CoT policy? How should the proposed design be interpreted given that the fixed-target control has a larger mean retained gain in Table 15?
Clarification of practical scope: What concrete deployment setting is represented by the combination of an unlabeled, continuously updated stream and subsequent frozen direct-option evaluation? In that setting, what evidence supports preferring DtC over frozen PA+CoT or fixed-target distillation, considering both adaptation cost and the inconsistent retained gains?
Response to Reviewer 8Z9C
We thank the reviewer for the suggestions on broader evaluation and fixed-teacher comparisons.
We tested the adapted models without options, as your first weakness asks: the collapse carries over to open-ended questions, the label preference mostly does not. We find no benefit of the evolving teacher and withdraw the method claim. Thank you for the review and the three references, which we will cite.
W1. Scope: four-way option selection. Agreed; the abstract and the introduction will state this limit. [1, 2] document content shortcuts and an option-ID bias in fixed models, and [3] replaces multiple choice with verifiable open-form questions. What our study adds is narrower than the submission suggested: the biases in [1, 2] belong to a fixed model, and we measure a bias that test-time updates create.
That self-training can collapse is known. What is new is where the damage sits (the label interface), that online accuracy shows it late, and that PA can hide it.
Beyond option selection (new). Following your point and [3], we tested the saved states without the scoring readout and without options.
W2 and Q1. DtC versus a fixed teacher; what is attributable to the evolving teacher? No benefit that we can measure. Over the four splits (7B), the fixed CoT target retains +0.9 [-0.1, +2.1] and DtC +0.6 [-0.7, +1.9]; DtC minus the fixed target is -0.3 [-1.1, +0.4] under canonical PA and -1.1 [-1.9, -0.4] in a single pass (clean images), and the main-split gain does not survive the multiple-comparison correction. An evolving target can also fail badly: trained toward its own PA instead of the frozen model's, the model collapses to tied scores on about 70% of questions (-41 points). We withdraw the claim of a reliable benefit.
Q2. What deployment setting is this, and why prefer DtC?
If the narrowed claims and the new evidence address your two weaknesses, we would be grateful if you would reconsider your assessment.
[1] Balepur et al., ACL 2024. [2] Zheng et al., ICLR 2024. [3] Liu et al., CVPR 2026.