- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper addresses missing-modality robustness in autoregressive video-language models. The proposed Test-Time Self-Distillation (TTSD) framework uses complete-modality examples to distill a frozen teacher's hidden representations into a student LoRA operating on synthetically masked inputs. Fisher-Anchored Restoration (FAR) controls continual adaptation through an ACTIVE/ANCHORED state machine, using an exponential moving average of squared gradients to determine when to suspend and resume updates.
Experiments on MELD, DFEW, and BAH report performance improvements over no adaptation, entropy-based adaptation, retrieval-augmented generation, and perplexity-based methods.
Critical strengths
(1) For autoregressive multimodal models, maintaining robust performance under missing modalities is a non-trivial challenge in real-world deployment. Performing self-distillation at the feature level could serve as a fresh approach in place of entropy-centric optimization. (2) The authors conduct experiments across several datasets, covering varying modality-missing ratios; they further examine aspects such as FAR, LoRA rank, smaller backbone architectures, and loss-based stopping criteria.
Critical weaknesses
(1) The method updates the student only when a complete-modality example is available, allowing the teacher to process the complete observation while the student receives a synthetically masked view. Genuinely incomplete examples are used only for inference, without adaptation updates. Consequently, TTSD-FAR cannot adapt during a deployment stream consisting entirely of incomplete observations, and FAR cannot reactivate in the absence of complete-modality samples. Although this restriction is explicitly stated in Section 4.1, it substantially narrows the deployment setting: TTSD-FAR learns to handle missing-modality predictions through adaptation on interleaved complete-modality examples, rather than adapting directly from genuinely incomplete target observations.
(2) FAR's consolidation criterion does not, by itself, establish convergence or stable coordinate-wise parameter importance.
The theoretical justification of FAR contains a significant gap. From Eqs. (12)–(13), the Consolidation Index satisfies
Let and . By nonnegativity and the triangle inequality,
With the reported and , the condition is sufficient for , regardless of which coordinates contain the current gradient. In particular, the gradient can move to coordinates disjoint from the previous importance estimate while still satisfying the consolidation condition, provided its total squared norm is smaller.
A simple counterexample considers a constant nonzero gradient with . Ignoring the negligible numerical stabilizer,
Under the reported hyperparameters, CI first falls below the threshold at step 41, and FAR enters ANCHORED at step 45, despite the gradient remaining nonzero.
The authors acknowledge in Appendix A.1 that stable gradient second moments need not imply a stationary point. My concern is therefore not that the paper explicitly proves zero-gradient convergence, but that the main text interprets CI as a convergence-aware stopping signal and that Proposition 1 asserts coordinate-wise expected second-moment stability in Eq. (25) without a sufficient justification from the stated assumptions and finite-step CI threshold.
Similarly, Proposition 2 establishes the programmed state transition conditional on the FMI threshold being crossed, rather than validating the identification of genuine distributional shifts.
Questions for authors
The empirical evidence is insufficient to establish robust gains and the claimed continual-adaptation advantages. Although the proposed method generally outperforms the reported baselines, several improvements are very small. For example, the F1 gains are only 0.0008 on MELD with 10% vision missing and 0.0010/0.0020 on DFEW with 10%/20% text missing. Furthermore, TENT and EATA perform substantially worse than the unadapted LVLM, making them relatively weak comparisons in this setting. The paper already includes comparisons against plain TTSD without FAR (Figure 3 and Appendix Table 8) and loss-plateau stopping (Appendix Table 8). However, these experiments do not establish whether FAR outperforms simpler stopping and reactivation strategies. More informative controls would include matched-data modality-dropout LoRA training, compute-matched periodic-reset or gradient-norm-based gating mechanisms, and evaluation on controlled streams with known distribution-shift points to measure false reactivations and detection delays.
7 comments
continued
W2: FAR as Convergence and Distribution Shift Detector
A2: We thank the reviewer for identifying this limitation. We agree that a small CI does not establish convergence or coordinate-wise stability. In particular, a nonzero but constant gradient can produce a stabilizing EMA and cause CI to fall below the consolidation threshold. We have therefore revised the interpretation of CI throughout the paper: it is a heuristic measure of relative change in the aggregate gradient second-moment estimate, not a convergence criterion. Entering the ANCHORED state consequently does not imply vanishing gradients or a stationary point. We have also clarified that FMI measures mismatch with the stored gradient-importance estimate, rather than proving a genuine distribution shift. Such mismatch may also result from sample difficulty or stochastic gradient variation. Accordingly, FAR is presented as a heuristic pause-and-reactivation mechanism, not a distribution-shift detector. Further, we would like to state that detecting the change in the stream is not the aim of FAR. Its aim is a reliable signal for when the student should resume adaptation. FMI measures the mismatch between incoming gradients and the statistics of the consolidated student, so it depends on the student and not only on the input. An input change that the consolidated student already handles produces little gradient mismatch and requires no new adaptation, whereas a change that moves the gradients onto parameters that were unimportant at consolidation. For scheduling adaptation, detecting every input change is therefore neither necessary nor sufficient.
Q1:Magnitude of the Gains
A2: We would like to point out that in most of the settings that stress the model, the gains are substantial. On MELD with 50% text missing, F1 improves from 0.4785 to 0.5124 (+3.39%); on DFEW with 50% vision missing, from 0.4430 to 0.4784 (+3.54%); and on BAH with 50% text missing, from 0.6115 to 0.6294 (+1.79%). These are the conditions in which the missing modality is the dominant one and the unadapted model degrades most, and they are also the conditions in which every competing adaptation method either collapses or fails to improve over the unadapted model. The small gains cited by the reviewer, +0.0008 for MELD with 10% vision missing and +0.0010 and +0.0020 for DFEW with 10% and 20% text missing, occur in cases where the modality drop itself barely affects performance. Vision is the weaker modality on MELD and text is the weaker modality on DFEW (Appendix B notes that 4,960 of the DFEW clips contain no utterance at all), so removing 10%–20% of that modality lowers the unadapted F1 by only 0.0027 on MELD and by 0.0019 and 0.0059 on DFEW. There is very little room for improvement in these cells, and the absolute gain is bounded by the size of the drop. We also acknowledge that the main 7B results are reported for a single configuration, while multi-seed statistics are reported only for the 0.5B model (Table 7). This was done because each experiment requires repeated runs with different missing levels and baselines. However, in the revision, we have added repeated runs with 3 different seeds for some Video-LLaVA-7B experiments (Table 2 - DFEW Dataset and Table 3 BAH Dataset) as well, so that the claims for those cells rest on measured variability. For the camera-ready version, we will include this analysis for the remaining tables as well.
continued
Q2: Choice for SOTA Comparisons
A2: We chose the baselines to answer a specific question, namely whether the test-time adaptation signals that work for discriminative models and for text-only LLMs transfer to autoregressive LVLMs under missing modalities. TENT and EATA are the canonical entropy-based test-time adaptation methods and used by multiple recent studies Ramazanova et al. (2025); Hu et al. (ICML 2025), and EATA is of particular relevance because it also uses a Fisher-based regularizer, which makes it the most direct point of comparison for the way FAR uses Fisher information. Their poor performance is therefore an informative result and not the sign of a weak comparison. It is precisely the behavior shown by Hu et al. ( ICML 2025), who show that entropy minimization is ill-suited to test-time adaptation of large language models, because confident predictions from an autoregressive model under distribution shift are frequently incorrect, so minimizing entropy reinforces errors. For the same reason, we included a perplexity-based generation baseline that follows the principle of Hu et al. (ICML 2025), which is the adaptation signal designed for LLMs and the strongest published alternative to entropy for generative models. In addition, we included the retrieval-augmented baseline (RAG) because it is the main inference-time approach that requires no retraining and that is used for LVLMs, and because it fails in a way that our analysis predicts, namely when the observed modality is too weak to retrieve informative neighbors. MiDl (Ramazanova et al., 2025), is the closest to our work in the evaluation protocol, but its published form is not directly applicable to an autoregressive LVLM, and any LVLM version requires design choices that we would have had to make ourselves. First, both of its objectives are defined on a categorical distribution set produced by a classifier head, namely the mutual information between that distribution and the modality combination, and a KL term to the original model’s distribution. An autoregressive LVLM has no such head, so a surrogate must be chosen. Second, MiDl adapts the normalization layers of the backbone, and the corresponding choice for an LVLM (the RMSNorm weights, or LoRA as in our student) changes the method. The point that matters for our claims is that the unadapted model is the relevant reference for whether adaptation helps, and in this setting the existing test-time methods do not improve over it, whereas TTSD-FAR does.
Q3: Comparison with simpler start-stop strategy
A3: We agree that comparing FAR only with plain TTSD and with a loss-plateau rule that never resumes did not test a simple stop-and-reactivate strategy. To that end, we have added an experiment to Appendix Table 8:
| Method | Full | 10% | 20% | 30% | 40% | 50% |
|---|---|---|---|---|---|---|
| No-Adaptation | 0.6341 | 0.5905 | 0.5798 | 0.5710 | 0.5662 | 0.5614 |
| TTSD | -- | 0.6065 | 0.6039 | 0.5887 | 0.5816 | 0.5728 |
| TTSD-LS | -- | 0.5910 | 0.5820 | 0.5730 | 0.5645 | 0.5560 |
| TTSD-LS w/ restart | -- | 0.5916 | 0.5835 | 0.5730 | 0.5645 | 0.5560 |
| TTSD-FAR (Ours) | -- | 0.6185 | 0.6120 | 0.5955 | 0.5910 | 0.5870 |
Caption: Ablation comparing TTSD-FAR against plain TTSD (no stopping criterion), TTSD-LS (loss-plateau stopping criterion), and TTSD-LS with restart under progressively missing text conditions. F1 scores are reported on the BAH dataset.
TTSD-LS w/ restart, which freezes the student when the distillation loss has plateaued for 5 samples and resumes adaptation when the loss varies for 5 consecutive samples. It uses the loss for both stopping and reactivation, as FAR uses its gradient statistics for both. On BAH with progressively missing text, TTSD-LS w/ restart behaves almost identically to TTSD-LS (a difference of at most 0.0015 F1, and identical scores at 30–50% missing text, where the restart rule never triggered), so loss-based reactivation recovers little of the gap to plain TTSD. TTSD-FAR outperforms it by 0.023–0.031 F1 and plain TTSD by 0.007–0.014 F1 at every missing ratio. The per-sample loss mostly reflects sample difficulty, which makes it a weak signal for both stopping and resuming. We note the limits of this evidence: it is a single run on one dataset, and we do not claim that FAR is optimal among all stop-and-resume rules, only that it is better than the loss-based rule tested here and than adapting on every sample.
Author's Response
We thank reviewer 6gyM for their constructive feedback and for the careful review and for recognizing the relevance of our research problem and the methodological contributions of TTSD-FAR. We also appreciate the reviewer acknowledging the strong experimental results across several datasets, covering varying modality- missing ratios, LoRA rank, smaller backbone architectures, and loss-based stopping criteria. We address each comment below, where “W” denotes a weakness, “Q” a question, and “A” our response.
Summary of the changes We addressed the reviewer's concerns regarding the reliance on complete-modality samples, the magnitude of performance improvements, baseline selection, and the effectiveness of FAR. We clarified that TTSD-FAR targets intermittent modality loss, where complete observations remain available within the test stream, and distinguished this setting from persistent modality absence, which belongs to the privileged-modality research setting. We further explained why complete observations are necessary to obtain the teacher representations used as distillation targets. Furthermore, this setting is consistent with the relevant test-time adaptation for missing modalities. Regarding performance, we highlighted the substantial gains under more challenging missing-modality conditions, particularly on MELD, DFEW, and BAH, while explaining the smaller improvements in cases where removing the weaker modality has minimal impact on the original performance. We strengthened the statistical evaluation by adding three-seed experiments for selected Video-LLaVA-7B results. We also expanded the justification for baseline selection, explaining the limitations of transferring entropy-based, perplexity-based, retrieval-based, and MiDl approaches to autoregressive LVLMs. To isolate the contribution of FAR, we introduced a loss-based start-stop and reactivation baseline, against which FAR demonstrates consistent improvements. Finally, we clarified the teacher's modality-dropout training procedure, explained why periodic resets are unsuitable for the warm-started student, and supplemented the FAR analysis with MMD visualizations to examine its relationship with changes in the incoming data distribution.
W1: Reliance on Complete Modality Samples
A1: We agree that TTSD-FAR adapts only on complete-modality samples, and that a deployment stream containing no complete observation falls outside its scope. In the literature, this is termed as privileged modality, and has a separate line of research. Privileged modality methods typically present train-time methods, whereas TTSD is a test-time protocol for multi-billion-parameter models, where retraining or source data availability limits the train-time approaches. This is already stated in Section 4.1. The evaluation protocol is standard in missing modality literature. We follow the evaluation protocol of Ramazanova er al. (ICLR 2025) [1], which also relies on complete modality samples for adaptation. Similar to that study, in the 100% missing case, we would always predict using the student LoRA and maintain the performance. We nevertheless think the restriction is a property of the problem rather than a defect of the method, for two reasons. First, the target of adaptation is a mapping from a masked view of an input to the complete-modality representation of the same input. A genuinely incomplete observation has no complete-modality counterpart, so no method can obtain such a target from it without an external generative model of the missing modality, which is precisely the assumption that breaks down for billion-parameter autoregressive decoders (Section 2). The student therefore does learn to handle incomplete observations, but it learns the structure of the missing-modality shift from the complete samples that make the shift observable. The fully incomplete regime (100% missing) is a privileged-modality problem and is explicitly outside our scope (Section 4.1). Second, the regime we address, in which a fraction of the stream is complete and a fraction is not, is the typical deployment condition for sensor dropout, transcription failure, and audio corruption, which are intermittent rather than permanent. Evaluation across 0%–50% missing therefore covers the range in which complete samples remain available.