acceptodds
Withdrawn from ICLR 2027

Beyond Online Accuracy: What Persists After Continual Test-Time Adaptation of Multimodal LLMs on Multiple-Choice VQA?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual test-time adaptation (CTTA) lets multimodal large language models (MLLMs) update from unlabeled test streams, but it is usually evaluated only by online accuracy. This can confound the effect of adaptation with the inference procedure and does not reveal what the model retains after the stream. We introduce a two-stage benchmark for multiple-choice VQA that first measures online performance under paired presentation histories, then freezes each adapted model and evaluates it on shared held-out questions against a frozen model using the same inference procedure. Across ten published TTA and CTTA methods on Qwen2.5-VL, adapted models rarely outperform their matched frozen baselines, while temporary interface changes can leave persistent, order-dependent answer biases. Controlled updates further show that retained gains depend on the information available in the update target. Based on this finding, we propose Deliberate-then-Consolidate (DtC), which uses chain-of-thought (CoT) inference online and distills its CoT answer distribution into direct predictions. On Qwen2.5-VL-7B, DtC reaches the strongest frozen online inference and, on the benchmark split, improves clean held-out direct-answer accuracy by 2.1 points over frozen PA. Across four question splits, retained gains vary with the advantage of CoT over direct inference, a distinction that online accuracy alone does not reveal.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual test-time adaptation, multimodal large language models, multiple-choice visual question answering, distribution shift, evaluation protocols

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8Z9COpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether continual test-time adaptation (CTTA) of multimodal LLMs produces improvements that persist beyond the adaptation stream. It introduces a two-stage multiple-choice VQA benchmark: models first adapt on an unlabeled stream under paired interface and image perturbations, then undergo frozen evaluation on shared held-out questions against inference-matched frozen baselines. Experiments with ten existing methods across two Qwen2.5-VL sizes show limited retained improvements and persistent, order-dependent answer biases. The paper also proposes Deliberate-then-Consolidate (DtC), which distills the current model’s CoT answer distribution into direct predictions. Validation includes controlled training targets, ablations, source-order changes, multiple question splits, and additional model/data checks. DtC shows positive retained gains on the main split, but these gains are not consistently reproduced.

Critical strengths

The evaluation separates inference improvements from retained adaptation effects. Comparing adapted and frozen models on the same held-out questions with the same inference procedure is a useful control. The paired presentation histories additionally help isolate the effects of earlier interface changes. I also appreciate the reporting of unsuccessful replications, computational costs, and statistical qualifications. These make the empirical findings more interpretable than a comparison based solely on online accuracy.

Critical weaknesses
  1. Limited scope and practical relevance. The evaluation focuses on four-way option selection, whose position biases and exploitable option information are already well documented [1–3](please cite these papers). Although the paper studies their persistence after adaptation, it does not establish relevance to open-ended visual answering. This limits the broader significance of the findings and the practical value of improving direct-option accuracy.

  2. Limited methodological novelty and inconsistent gains. DtC combines established CoT self-distillation with option canonicalization and averaging, without demonstrating a clear advantage over fixed-teacher distillation. In Table 15, its mean retained gain is only +0.6 points, versus +0.9 for the fixed CoT target. Gains vary across splits, and the main-split improvement does not survive the reported multiple-comparison correction. These results do not establish a reliable benefit from the proposed online updates.

[1] Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? ACL, 2024.

[2] Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large Language Models Are Not Robust Multiple Choice Selectors. ICLR, 2024; arXiv:2309.03882, 2023.

[3] Yesheng Liu et al. Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT. CVPR, 2026, pp. 18859–18869.

Questions for authors

Clarification of the methodological contribution: What benefit is specifically attributable to the evolving teacher in DtC beyond distilling a fixed frozen CoT policy? How should the proposed design be interpreted given that the fixed-target control has a larger mean retained gain in Table 15?

Clarification of practical scope: What concrete deployment setting is represented by the combination of an unlabeled, continuously updated stream and subsequent frozen direct-option evaluation? In that setting, what evidence supports preferring DtC over frozen PA+CoT or fixed-target distillation, considering both adaptation cost and the inconsistent retained gains?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8Z9C

We thank the reviewer for the suggestions on broader evaluation and fixed-teacher comparisons.

We tested the adapted models without options, as your first weakness asks: the collapse carries over to open-ended questions, the label preference mostly does not. We find no benefit of the evolving teacher and withdraw the method claim. Thank you for the review and the three references, which we will cite.

W1. Scope: four-way option selection. Agreed; the abstract and the introduction will state this limit. [1, 2] document content shortcuts and an option-ID bias in fixed models, and [3] replaces multiple choice with verifiable open-form questions. What our study adds is narrower than the submission suggested: the biases in [1, 2] belong to a fixed model, and we measure a bias that test-time updates create.

  • It is absent before adaptation and present after it (on one history a 7B model answers "A" on 91% of held-out questions), and it persists after the format returns to the standard one.
  • An ensembled readout can hide it: on 4,080 new questions, COME on InternVL3.5-2B loses 28.0 points in a single pass and 6.1 under permutation averaging (PA).
  • It is attached to the label, as [2] finds for fixed models: with shuffled letters, the median state gives 76% of its answers to the option that now carries its letter and 1% to the letter's old position (the 19 states with a strong preference; the label wins in all 19).

That self-training can collapse is known. What is new is where the damage sits (the label interface), that online accuracy shows it late, and that PA can hide it.

Beyond option selection (new). Following your point and [3], we tested the saved states without the scoring readout and without options.

  • Written answers (256 panel questions, options shown): the frozen models are exactly as accurate written as scored, and the damaged states stay damaged (InternVL3.5-8B, Tent: -50.3 points written, -30.4 scored).
  • Open-ended questions (768 questions from TextVQA, ChartQA, GQA and OK-VQA, no options; both metrics below were registered before the run; the states were adapted on multiple-choice streams). The collapse carries over: Tent, DeYO and COME on InternVL3.5-8B lose 43.6, 51.6 and 16.9 points. The label preference mostly does not: the three InternVL3.5-2B methods that lose 11.2 to 28.0 points on multiple-choice questions lose 0.5 to 3.3 here.
  • Two gains, and what they are. Under each benchmark's standard metric (primary), two of 24 method-model pairs gain: COME +3.8 [+2.2, +5.4] and Tent +2.4 [+0.9, +3.9] at 7B. Under the secondary metric, which accepts a correct answer inside a longer one, none gains. A post hoc description explains the difference: both gains come from the 106 questions that the frozen 7B model answers with a sentence, which exact match counts as wrong; on the other 662 questions the two methods change by +0.4 [-0.8, +1.7] and -0.4 [-1.7, +0.8]. The adapted states answer more tersely, and we find no evidence that they answer more correctly. The revised paper reports both metrics and limits the no-gain statement to multiple-choice questions.

W2 and Q1. DtC versus a fixed teacher; what is attributable to the evolving teacher? No benefit that we can measure. Over the four splits (7B), the fixed CoT target retains +0.9 [-0.1, +2.1] and DtC +0.6 [-0.7, +1.9]; DtC minus the fixed target is -0.3 [-1.1, +0.4] under canonical PA and -1.1 [-1.9, -0.4] in a single pass (clean images), and the main-split gain does not survive the multiple-comparison correction. An evolving target can also fail badly: trained toward its own PA instead of the frozen model's, the model collapses to tied scores on about 70% of questions (-41 points). We withdraw the claim of a reliable benefit.

Q2. What deployment setting is this, and why prefer DtC?

  • The setting. In continual test-time adaptation, the model that answers the next request is the model left by the updates on earlier requests, as test-time learning methods such as TLM and TTRV propose. Stage 2 measures that model. We do not claim that such deployment is advisable; our results bear on whether it is.
  • Why prefer DtC? The evidence gives no reason to. Online, DtC and the frozen model with the same PA+CoT inference are indistinguishable (69.8 versus 69.5; difference +0.2 [-1.1, +1.5] before rounding), and the frozen model needs no updates.
  • A reference that needs no update. On the new panel, no update is detectably better than the frozen model averaging four option orders: for each of the 19 pairs of a published method and a model, a one-point advantage is excluded. We propose this frozen ensemble as a required reference, and the revised paper makes no method recommendation.

If the narrowed claims and the new evidence address your two weaknesses, we would be grateful if you would reconsider your assessment.

[1] Balepur et al., ACL 2024. [2] Zheng et al., ICLR 2024. [3] Liu et al., CVPR 2026.

Reviewer_U9e2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether continual test-time adaptation (CTTA) of multimodal large language models leads to persistent improvements beyond online accuracy by introducing a two-stage evaluation benchmark that separates online adaptation performance from retained model improvements on held-out questions. Through extensive experiments with existing TTA/CTTA methods, they find that online accuracy gains often fail to translate into meaningful retained improvements and that adaptation can introduce persistent answer biases. Motivated by these observations, the paper proposes Deliberate-then-Consolidate (DtC), which distills chain-of-thought (CoT) predictions into direct-answer predictions through continual adaptation. Experiments demonstrate promising improvements in certain settings, although the retained gains vary across evaluation splits.

Critical strengths

The paper highlights an important limitation of existing CTTA evaluations: improvements in online accuracy do not necessarily indicate that the adapted model has acquired transferable capabilities. The proposed two-stage benchmark, which evaluates adapted models on held-out questions against frozen baselines under matched inference procedures, provides a meaningful framework for assessing whether adaptation produces persistent benefits.

Critical weaknesses
  • Limited methodological novelty. The proposed Deliberate-then-Consolidate (DtC) approach primarily combines several established techniques, including chain-of-thought (CoT) distillation, self-training, permutation averaging, and LoRA-based adaptation. While applying these components to continual test-time adaptation is potentially useful, the underlying optimization objective is essentially a form of knowledge distillation from CoT-based predictions to direct-answer predictions. The paper does not sufficiently clarify what fundamentally new methodological insight DtC offers beyond integrating existing techniques.
  • Lack of reliability control for potentially incorrect CoT targets. DtC relies on the model's own CoT-generated answer distributions as pseudo-supervision for continual parameter updates. However, when CoT predictions are incorrect, directly distilling them may reinforce erroneous beliefs. This concern is particularly relevant given the results in Table 3, where CoT-based distillation fails to improve retained accuracy on the 3B model. Although the paper explores several training variants, it lacks a systematic analysis of how incorrect or low-confidence CoT targets influence adaptation. An analysis distinguishing cases where CoT corrects versus degrades direct predictions would help clarify the reliability and limitations of the proposed approach.
  • Limited robustness and statistical significance of the reported retained improvements. The main empirical claim regarding DtC is not consistently supported across different evaluation splits. Although Table 2 reports a +2.1 percentage-point improvement in clean held-out accuracy over the frozen PA baseline, the gains vary substantially across the four splits reported in Table 15 (+2.3, −1.0, +1.7, and −0.5 percentage points) in Appendix D. These results raise concerns about whether DtC consistently produces meaningful improvements in the adapted model rather than benefits specific to particular data partitions.
Questions for authors

The paper itself identifies persistent answer biases as a major limitation of existing CTTA methods. However, it seems that the proposed DtC still suffers from such problems. As discussed in Appendix D, the clean retained gain is particularly large on PhyX (+9.6 percentage points), while excluding PhyX reduces the overall gain to +1.0 point, with a confidence interval that includes zero. Furthermore, the authors note that 51 of the 96 held-out PhyX questions have their correct answers in the third position under canonical ordering, and that DtC develops a preference for later answer positions.

Could the authors clarify to what extent the reported retained gains of DtC reflect real improvements in transferable reasoning capabilities rather than the exploitation of dataset-specific answer-position biases? In particular, would the improvements persist if the correct answer positions were explicitly balanced across the held-out evaluation set?

Given that mitigating persistent answer biases is one of the paper's central motivations, how do the authors reconcile this observation with the claimed effectiveness of DtC?

1 comment
AuthorsOpenReview ↗

Response to Reviewer U9e2

We thank the reviewer for the comments on CoT reliability and answer-position bias.

Your three weaknesses concern DtC and we agree with each, so we withdraw the method claim. The analysis you ask for is below: the CoT target corrects and degrades direct answers at comparable rates. Thank you for the review.

What remains, and why it is new. The strength you identify, that a gain in online accuracy need not mean the adapted model acquired anything transferable, is now the claim of the paper, and it rests on three findings that go beyond "published methods do not help" (see our response to all reviewers):

  • Online accuracy shows the damage late. On InternVL3.5-8B, five of six trajectories fall from at most 5.6 points below the frozen model to 13.6-42.9 points below within 24 to 96 questions, while accuracy on the stream so far is only 0.7-6.9 points below it.
  • Adaptation creates a preference for an answer label, and an ensembled readout hides it. COME on InternVL3.5-2B loses 28.0 points in a single pass and 6.1 under permutation averaging (PA); when the labels are shuffled, such states follow the letter (a median 76% of their answers) and not its old position (1%).
  • The test can find a gain. On 4,080 new questions we detect none for any published method, while a supervised positive control (the same update trained with ground-truth labels) gains +3.9 [+2.1, +5.7] points at 7B.

Limited novelty of DtC. Agreed. DtC distills CoT answers into direct answers with known components; it remains only as one row in a comparison of update targets.

No reliability control for wrong CoT targets. Agreed, and we ran the analysis you describe from the saved logs: for every stream question, the model's direct answer just before the update (PA over four option orders) against the CoT target it is then trained on.

ModelPA wrong, CoT rightPA right, CoT wrongCoT minus PA accuracy
Qwen2.5-VL-7B266199+2.2 points
Qwen2.5-VL-3B278290-0.4 points
InternVL3.5-2B147125+1.4 points

(Counts over several adaptation runs on the same 768 questions; descriptive.) At 3B the target degrades slightly more often than it corrects, and 3B shows no retained gain. A more accurate target is also not sufficient: on InternVL3.5-2B the CoT target is 1.4 points better on the stream, and the retained model is 7.0 points worse on new questions.

Gains vary across splits. Agreed. The four values are +2.3, -1.0, +1.7 and -0.5; their mean is +0.6 [-0.7, +1.9].

Your question: transferable reasoning or answer-position bias, and does the gain survive balanced positions? (7B, main split.)

  1. Are the positions balanced? As displayed, exactly: each letter is correct on 192 of the 768 held-out questions. In the sorted order that DtC trains and is read out on, no: 169 / 188 / 238 / 173.
  2. Does the gain survive balancing? Yes, under the readout DtC trains on (canonical PA): equal weight on the four sorted positions changes it from +2.1 to +2.2 [+0.4, +4.1].
  3. Is it transferable reasoning? The evidence does not establish that. The gain is concentrated in one source: +9.6 points on PhyX, +1.0 [-0.6, +2.6] on the other seven, and -5.7 on PhyX in the role-swap split. Under a single pass it sits on one displayed position: +8.9 [+5.2, +12.7] on questions whose correct option is shown first and about zero on the others; the frozen model chooses the first option on 20% of questions, the adapted model on 24%. So position imbalance does not explain the canonical-PA gain, and the single-pass gain comes with a shift toward the first option. On the 4,080 new questions the gain is +0.7 [-0.2, +1.7] at 7B, +0.1 [-0.7, +0.9] at 3B and -7.0 [-9.0, -5.0] on InternVL3.5-2B.

How do we reconcile this with the claimed effectiveness? We do not. The effect depends on the split, the source and the readout; we withdraw the effectiveness claim and report this analysis in the main text.

Your question about answer positions is the one the revised paper now asks of every published method, and it led to the label-preference finding above. Since your concerns were about the method we have withdrawn, we hope that the paper, read as the evaluation study it now is, merits a second look. We are glad to answer further questions before the discussion closes.

Reviewer_Zt5mOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the evaluation of continual test-time adaptation (CTTA) for multimodal large language models (MLLMs). The authors argue that reporting online accuracy alone conflates the effects of parameter updates with those of the inference procedure itself, and does not reveal what the model retains after adaptation. To address this issue, the paper proposes a two-stage evaluation protocol. In Stage 1, the model performs unlabeled online adaptation under four paired interface/image histories. In Stage 2, the adapted model is frozen and evaluated on a shared held-out question set that was not used for updates, using the same inference procedure as the frozen reference model, thereby measuring the retained gain.

The authors reimplement and compare ten existing TTA/CTTA methods on Qwen2.5-VL-3B/7B and propose Deliberate-then-Consolidate (DtC), which uses the model's CoT answer distribution as a pseudo-target to update direct answer predictions through LoRA. The paper's robust conclusions are that online performance and post-adaptation retained gains are different quantities, that interface changes can induce substantial and order-dependent answer-letter biases, and that matched frozen baselines are necessary. The evidence for a stable positive retained gain from DtC is weaker: the direct-answer improvement is approximately 2.1 percentage points on the main split, but is not significant after Holm correction; the improvement is not reproduced on the role-swap split or the new-question panel, and the mean gain across four splits is only about +0.6 points with an unstable sign.

The evaluation perspective and empirical auditing value are strong, but the generality of the method contribution and the statistical evidence for the main positive claim are insufficient for acceptance under a CCF A-level standard. The work would be more convincing if positioned as an evaluation benchmark/audit study and supported by broader independent testing. In its current form as a paper proposing an effective CTTA method, the evidence remains insufficient.

Critical strengths

Strengths 1: Clearly separating online gains from post-adaptation retained gains.

The Stage 1/Stage 2 design addresses a question often overlooked in TTA work: when a model improves on the stream it has seen, does its state actually change in a way that improves later unseen questions? Comparing adapted and frozen models on the same held-out questions with the same inference procedure prevents CoT, PA (permutation averaging), and similar inference techniques from being misattributed to parameter-update gains. This is the most valuable aspect of the paper and the part most likely to influence subsequent work.

Strengths 2: Paired histories and order controls reveal interface bias.

The four Standard/Interface/Perceptual/Compound histories change only the presentation while keeping the questions and their order fixed; the paper also evaluates both O1 and O2 source-block orders. The experiments show that naive self-training can push the model toward a single answer letter after interface changes, with the 7B model selecting A for 91% of held-out questions in one setting. This demonstrates that the answer interface in multiple-choice VQA can become an absorbed state variable, and the phenomenon is concrete and diagnostically useful.

Strengths 3: Good disclosure of baseline implementations and computational costs.

The appendices list parameter scopes, learning rates, filtering rates, protocol modifications, seeds, and runtime costs. The authors also clearly acknowledge deviations from the original settings of TTAdapt, SAR, ZeroSiam, TTRV, and other methods. Learning-rate grids, a second random seed, a role-swap split, and a new-question panel are reported, making the evaluation substantially more informative than a single aggregate accuracy number.

Strengths 4: DtC has a clear mechanism and targeted ablations.

DtC converts expensive CoT inference into direct-answer predictions through online distillation. The method is simple and interpretable. The ablations show that using the PA target can make the direct predictor nearly degenerate, whereas the CoT target preserves more useful information; training over all four option orders also outperforms training on a single order. These results support a testable hypothesis that the information content of the update target determines the amount of gain that can be retained.

Critical weaknesses

Weakness 1: The central retained-gain claim for DtC does not pass the paper's own strict statistical test.

The main table reports a +2.1-point clean held-out gain of DtC over frozen PA, but the prespecified comparison has a Holm-adjusted p-value of 0.0937. The +1.6-point gain over frozen PA+CoT has an adjusted p-value of 0.1025. The direct-answer gain on the role-swap split is −1.0 points, with a 95% confidence interval containing zero; the gain on the new-question panel is −0.4 points; and the direct-answer gains across four splits are only +0.6 points on average, with values of +2.3, −1.0, +1.7, and −0.5. These results support the claim that the gain depends on the split and on the frozen model's CoT advantage over PA, but they do not support the stronger claim that DtC reliably improves the adapted model. The positive DtC results should be explicitly labeled exploratory, with corresponding changes to the title, abstract, and conclusion.

Weakness 2: The main question set was exposed during development, creating selection and tuning concerns.

The paper explicitly states that the 1,536 core questions were used during development, while the largest DtC gain occurs on the main split. The preregistered new-question panel does not reproduce this gain. Although the authors provide a role-swap split and random re-splits, these still come from the same core question pool and cannot replace a fully independent benchmark. The main claims should be reevaluated on data that were not used for method selection, prompt design, learning-rate selection, or result inspection, with preregistered endpoints reported separately for each source and model.

Weakness 3: Substantial protocol asymmetries weaken cross-method comparisons.

To impose a unified single-question continual protocol, the paper modifies several methods: TTAdapt's original per-question restoration is removed, SAR is evaluated without its reset in the main comparison, ZeroSiam uses its vision implementation rather than its language-model setting, CoTTA and TTAdapt are run only at 3B, and TTRV uses a different setup and inference path from its original formulation. These changes are disclosed, but the claim that “ten published methods generally fail to retain gains” should be restricted to these transplanted continual implementations. It would be preferable to report native-protocol results, unified-protocol results, and their differences, and to include recent methods that support single-question streams or clearly explain why they cannot be included.

Weakness 4: The scope of the contribution is narrow, and synthetic perturbations may not represent real test-time drift.

The full method comparison is conducted primarily on Qwen2.5-VL-7B; Qwen2.5-VL-3B, InternVL3.5, and SmolVLM2 are covered only in partial experiments. The task is also limited to single-image, four-option VQA. The four interface changes and four image corruptions are synthetic and were selected by the authors to be clearly visible. They are sufficient to demonstrate that interfaces can be absorbed, but do not establish that natural format changes, topical drift, or cross-task streams in deployment have the same magnitude or direction of effect. The paper should narrow its claims about general CTTA or add experiments on natural continuous streams and more model families.

Weakness 5: The methodological novelty is mainly a combination of existing ideas, and the theoretical explanation remains correlational.

DtC combines CoT/self-consistency, permutation averaging, and online self-distillation. Related ideas already exist in offline distillation, System-2-to-System-1 transfer, and test-time self-training. The paper shows empirically that the CoT target contains more useful information than the PA target, but does not provide a theory or stronger mechanistic evidence concerning online-update stability, error reinforcement, distribution shift, or when the target will generalize to held-out data. If presented as a method paper, the authors should more clearly define the new algorithmic contribution relative to existing CoT distillation and test-time learning work.

Weakness 6: The evidence for “retained gains follow target information” may be confounded by source difficulty and answer-position structure.

The clean direct-answer gain of DtC is strongly influenced by PhyX: removing PhyX reduces the gain from +2.3 to +1.0, while the role-swap result for PhyX is −5.7. The authors also observe that correct answers are concentrated in later sorted positions in some splits. The paper should report effects stratified by source, answer position, image condition, and question type, and test whether the gain comes from particular sources or positional biases rather than general CoT information distillation.

Questions for authors

The following questions may materially affect the assessment of the paper's novelty and experimental reliability.

Q1: When the DtC direct-answer gain and CoT-deployment gain are both non-significant after Holm correction, and neither the role-swap split nor the new-question panel reproduces the result, what is the strongest conclusion the authors believe can be retained without relying on exploratory analyses? Would the authors be willing to describe the retained DtC gain as a conditional, split-dependent phenomenon?

Q2: The core question set was used during development. Can the authors provide a fully held-out, multi-model, multi-source test set and rerun the experiments after fixing DtC's prompt, learning rate, target, and primary endpoints in advance? If not, how can the potential effect of development exposure on the +2.1-point result be quantified?

Q3: Can the authors report, for each published baseline, native-protocol results, results under the unified continual protocol, and the corresponding matched-frozen references, while explaining how removing TTAdapt's reset, disabling SAR's reset, and using the vision implementation of ZeroSiam affect the rankings? This would distinguish “the method is effective under its original protocol” from “the method does not retain gains under the protocol studied here.”

1 comment
AuthorsOpenReview ↗

Response to Reviewer Zt5m

We thank the reviewer for the detailed assessment and suggestions on evaluation rigor.

The independent test you ask for in Q2 now exists (4,080 new questions, endpoints fixed in advance): we detect no retained gain for any published method, a supervised control gains, and DtC's gain is not reproduced. We withdraw the method claim and reposition the paper as the benchmark and audit you describe. Thank you for the review.

W1, Q1, W5 (DtC). You are right. What we can keep is: on the 7B model, DtC's gain is conditional and split-dependent, and we do not claim it as a reliable improvement. The four-split mean is +0.6 [-0.7, +1.9]; the main split exceeds the other three by 2.2 points [+0.4, +4.1]. DtC combines known components and leaves the abstract, the contributions and the conclusion.

W2, Q2 (an independent test). The panel has 4,080 questions excluded from all earlier experiments (1,180 unused questions from the benchmark sources, 2,900 from six outside datasets). The selection rule, the endpoints, equal weights for the sources and a 1-point equivalence margin were fixed before any evaluation; nothing is re-tuned.

  • DtC (canonical PA): 7B +0.7 [-0.2, +1.7], 3B +0.1 [-0.7, +0.9], InternVL3.5-2B -7.0 [-9.0, -5.0]. The +2.1 of the main split is not reproduced.
  • Published methods: none of 19 method-model pairs (three models) gains under either readout.
  • 1,024 of the 1,180 in-source questions come from PathMMU. With equal source weights it is one of eight strata of every estimate, and we added a cross-fitted replication on four models in which every state is tested on the sources it adapted on: none of 24 pairs gains (largest single-pass entry +0.4), while a supervised positive control, the same update trained with ground-truth labels, gains +3.9 [+2.1, +5.7] at 7B and +3.3 [+1.5, +5.2] at 3B.

W3, Q3 (native versus continual protocols). Agreed: the abstract and the table caption restrict the claim to continual implementations.

  • TTRV is the method for which we have both columns you ask for. Native (InternVL3-2B, MathVista, ten registered repetitions): +2.8 [+1.1, +4.6], all ten positive. Continual multi-source stream on 7B: +0.2 clean, -0.6 corrupted. This supports your distinction between "effective under its original protocol" and "does not retain a gain under the protocol studied here"; the two settings also differ in model and data, so we do not attribute the difference to the protocol alone.
  • TTAdapt: removing the per-question restoration is what makes it continual, so our result does not assess native TTAdapt. SAR with its reset is also equivalent to the frozen model (-0.1 [-0.4, +0.1]). ZeroSiam follows its vision implementation and is labeled the least faithful port. CoTTA and TTAdapt run at 3B only and are excluded from 7B claims.
  • Two checks on the ports: Tent with the entropy over the whole next-token distribution loses as much or more (InternVL3.5-8B: -44.5); with learning rates chosen by cross-validation on labels, one of 24 pairs gains 0.7 points in a single pass and none under PA.

W4 (scope). We narrow the claims to single-image four-option VQA and models of 2B to 8B; we have no data on natural format changes, topical drift or cross-task streams. Three additions:

  • Models. InternVL3.5-8B is new, and the damage is larger there. SmolVLM2, whose language model belongs to another family, was run with the label tokens it writes (our appendix scored its labels without their leading space; now corrected) and the learning rates of InternVL3.5-2B (not tuned): no method gains; Tent, DeYO and COME answer at the level of guessing in all 24 states (-28.2, -28.2 and -28.1 points); ETA is equivalent to the frozen model.
  • Learning rate and stream length. At a tenth of the rate no state of the 8B model collapses on streams of up to 2,040 questions; repeated eight times (16,320 questions), one of ten chains still breaks down (DeYO, after about 14,700 questions), and no method gains after any pass.
  • The no-gain result does not depend on the synthetic changes: it holds in the standard history, where the presentation never changes.

W6 (source and answer position; 7B, main split). PhyX carries the gain: +9.6 points on PhyX, +1.0 [-0.6, +2.6] without it, and -5.7 on PhyX in the role-swap split. Displayed answer positions are exactly balanced (192 per letter); in the sorted order that DtC trains on they are not (169 / 188 / 238 / 173), and weighting the four positions equally changes the canonical-PA gain from +2.1 to +2.2 [+0.4, +4.1] (not the +2.2 of W1; the rounded intervals coincide), so the imbalance alone does not create it. Under a single pass the gain sits on one position: +8.9 [+5.2, +12.7] where the correct option is displayed first, about zero elsewhere. The effect depends on the split, the source and the readout, and does not establish that reasoning was distilled.

We are glad to provide further analysis before the discussion closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers on What Changes and What We Tested

We thank the reviewers for careful and specific reviews. The three reviews agree on the strength of the paper (the two-stage benchmark that separates online gains from retained gains) and on its weakness (the retained gain of our method, DtC, is not reliable). We agree with both.

What changes. We withdraw the method claim and follow the positioning that Reviewer Zt5m proposes: an evaluation benchmark and audit. The title, the two-stage protocol and the audit of ten published methods are those of the submission; the revision narrows the claims to what the evidence supports. DtC leaves the abstract, the contributions and the conclusion and remains as one row of a controlled comparison; the negative evidence moves from the appendix into the main text; claims about published methods are restricted to their continual implementations.

What the paper claims now. Three findings on how continual test-time adaptation should be evaluated: accuracy on the stream lags behind damage to the adapted model (point 2 below); adaptation creates a preference for an answer label that an ensembled readout hides (point 3); and a frozen model with the same inference is a reference that no published method is found to exceed (point 1). The first two apply beyond these ten methods: any result reported as accuracy during adaptation, or with ensembled inference, is open to them.

The evidence, new since submission. Unless stated otherwise, the results use questions never used in development and the submitted code and settings, and were specified in writing before their outputs existed.

  1. We detect no retained gain for any published method, and the test can detect one. On a registered panel of 4,080 new questions (1,180 unused questions from the benchmark sources, 2,900 from six other datasets) we test the seven methods that share the standard readout: 19 method-model pairs on three models (CoTTA, TTAdapt and TTRV bring inference procedures of their own and are compared on the core questions). Against the frozen model with the same readout, no pair gains: in a single pass 9 are equivalent within one point, 5 are worse and 5 have intervals too wide to decide; under permutation averaging (PA) the counts are 11, 7 and 1. Against the frozen model that averages four option orders, a one-point advantage is excluded for every pair (pairwise 90% bounds). A cross-fitted replication on four models (adapt on one half of the panel, test on the other) agrees: none of 24 pairs gains and the largest single-pass entry is +0.4, while a supervised positive control, the same update trained with ground-truth labels, gains +3.9 [+2.1, +5.7] points at 7B. With the learning rate chosen by cross-validation on labels, one of 24 pairs gains (+0.7, single pass) and none does under PA.
  2. Accuracy on the stream lags behind the damage. On InternVL3.5-8B, five of six trajectories fall from at most 5.6 points below the frozen model to 13.6-42.9 points below within 24 to 96 questions, while accuracy on the stream so far is only 0.7-6.9 points below it.
  3. Adaptation creates an answer-label preference that an ensembled readout can hide. COME on InternVL3.5-2B loses 28.0 points in a single pass and 6.1 under PA. When the labels are shuffled, the 19 adapted states that developed a strong option preference give a median 76% of their answers to the option that now carries their letter and 1% to its old position.
  4. The damage is not an artifact of the scoring readout. When the models write their answers, the frozen models are exactly as accurate as when scored, and the damaged states stay damaged (InternVL3.5-8B, Tent: -50.3 points written, -30.4 scored).
  5. DtC does not hold up (7B): +0.6 [-0.7, +1.9] over the four core splits (a re-analysis of the submitted records), +0.7 [-0.2, +1.7] on the new panel, and no better than a fixed teacher.

Three further registered experiments (a model with another language model, more adaptation at a lower learning rate, and questions without options) are reported in our replies to Reviewers Zt5m and 8Z9C. Each reply is self-contained.

The revision reports every estimate with its interval, and we will release the code, the dated analysis plan, the panel manifests and the per-question records.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.