acceptodds
Under review as a conference paper at ICLR 2027

TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs

Abstract

LVLMs have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in-the-wild. ER is inherently multimodal, requiring a joint understanding of facial expressions, vocalizations, language, biosignals, and gestures. However, real-world deployment remains challenging: modalities may be missing or noisy at test time. Partial observations can be viewed as a distribution shift relative to the complete-modality distribution. SOTA TTA methods based on entropy or perplexity do not transfer to autoregressive LVLMs, while RAG degrades when the observed modality is weak. Because no ground-truth supervision exists to verify individual updates, adaptation across this stream risks accumulating drift and degrading once the model departs from a reliable solution. An effective solution must therefore adapt to missing modalities and remain effective during continual adaptation. We address both jointly with Test-Time Self-Distillation (TTSD), a parameter-efficient framework in which a frozen teacher, trained on complete modalities, guides an adaptive low-rank student via self-distillation, updating only a negligible number of parameters. Stability is built into this same loop through Fisher-Anchored Restoration (FAR), which monitors Fisher information stability to detect convergence versus drift and restores the student to the ACTIVE state when distributional shifts are identified. Our experiments on MELD, DFEW, and BAH under 0%–50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Test time adaptation, missing modality, fisher anchored restoration, large video language models, LoRA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6gyMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper addresses missing-modality robustness in autoregressive video-language models. The proposed Test-Time Self-Distillation (TTSD) framework uses complete-modality examples to distill a frozen teacher's hidden representations into a student LoRA operating on synthetically masked inputs. Fisher-Anchored Restoration (FAR) controls continual adaptation through an ACTIVE/ANCHORED state machine, using an exponential moving average of squared gradients to determine when to suspend and resume updates.

Experiments on MELD, DFEW, and BAH report performance improvements over no adaptation, entropy-based adaptation, retrieval-augmented generation, and perplexity-based methods.

Critical strengths

(1) For autoregressive multimodal models, maintaining robust performance under missing modalities is a non-trivial challenge in real-world deployment. Performing self-distillation at the feature level could serve as a fresh approach in place of entropy-centric optimization. (2) The authors conduct experiments across several datasets, covering varying modality-missing ratios; they further examine aspects such as FAR, LoRA rank, smaller backbone architectures, and loss-based stopping criteria.

Critical weaknesses

(1) The method updates the student only when a complete-modality example is available, allowing the teacher to process the complete observation while the student receives a synthetically masked view. Genuinely incomplete examples are used only for inference, without adaptation updates. Consequently, TTSD-FAR cannot adapt during a deployment stream consisting entirely of incomplete observations, and FAR cannot reactivate in the absence of complete-modality samples. Although this restriction is explicitly stated in Section 4.1, it substantially narrows the deployment setting: TTSD-FAR learns to handle missing-modality predictions through adaptation on interleaved complete-modality examples, rather than adapting directly from genuinely incomplete target observations.

(2) FAR's consolidation criterion does not, by itself, establish convergence or stable coordinate-wise parameter importance.

The theoretical justification of FAR contains a significant gap. From Eqs. (12)–(13), the Consolidation Index satisfies

CIt=(1−βF)∥gt⊙2−Ft−1∥1∥Ft∥1+ϵ.CI_t = \frac{(1-\beta_F)\|g_t^{\odot 2}-F_{t-1}\|_1}{\|F_t\|_1+\epsilon}.

Let St−1=∥Ft−1∥1S_{t-1}=\|F_{t-1}\|_1 and qt=∥gt∥22q_t=\|g_t\|_2^2. By nonnegativity and the triangle inequality,

CIt≤(1−βF)(St−1+qt)βFSt−1+(1−βF)qt+ϵ.CI_t \leq \frac{(1-\beta_F)(S_{t-1}+q_t)}{\beta_F S_{t-1}+(1-\beta_F)q_t+\epsilon}.

With the reported βF=0.99\beta_F=0.99 and τfc=0.02\tau_{fc}=0.02, the condition qt<St−1q_t<S_{t-1} is sufficient for CIt<τfcCI_t<\tau_{fc}, regardless of which coordinates contain the current gradient. In particular, the gradient can move to coordinates disjoint from the previous importance estimate while still satisfying the consolidation condition, provided its total squared norm is smaller.

A simple counterexample considers a constant nonzero gradient gt=gg_t=g with F0=0F_0=0. Ignoring the negligible numerical stabilizer,

CIt=(1−βF)βFt−11−βFt.CI_t=\frac{(1-\beta_F)\beta_F^{t-1}}{1-\beta_F^t}.

Under the reported hyperparameters, CI first falls below the threshold at step 41, and FAR enters ANCHORED at step 45, despite the gradient remaining nonzero.

The authors acknowledge in Appendix A.1 that stable gradient second moments need not imply a stationary point. My concern is therefore not that the paper explicitly proves zero-gradient convergence, but that the main text interprets CI as a convergence-aware stopping signal and that Proposition 1 asserts coordinate-wise expected second-moment stability in Eq. (25) without a sufficient justification from the stated assumptions and finite-step CI threshold.

Similarly, Proposition 2 establishes the programmed state transition conditional on the FMI threshold being crossed, rather than validating the identification of genuine distributional shifts.

Questions for authors

The empirical evidence is insufficient to establish robust gains and the claimed continual-adaptation advantages. Although the proposed method generally outperforms the reported baselines, several improvements are very small. For example, the F1 gains are only 0.0008 on MELD with 10% vision missing and 0.0010/0.0020 on DFEW with 10%/20% text missing. Furthermore, TENT and EATA perform substantially worse than the unadapted LVLM, making them relatively weak comparisons in this setting. The paper already includes comparisons against plain TTSD without FAR (Figure 3 and Appendix Table 8) and loss-plateau stopping (Appendix Table 8). However, these experiments do not establish whether FAR outperforms simpler stopping and reactivation strategies. More informative controls would include matched-data modality-dropout LoRA training, compute-matched periodic-reset or gradient-norm-based gating mechanisms, and evaluation on controlled streams with known distribution-shift points to measure false reactivations and detection delays.

7 comments
AuthorsOpenReview ↗

Author's Response

We thank reviewer 6gyM for their constructive feedback and for the careful review and for recognizing the relevance of our research problem and the methodological contributions of TTSD-FAR. We also appreciate the reviewer acknowledging the strong experimental results across several datasets, covering varying modality- missing ratios, LoRA rank, smaller backbone architectures, and loss-based stopping criteria. We address each comment below, where “W” denotes a weakness, “Q” a question, and “A” our response.

Summary of the changes We addressed the reviewer's concerns regarding the reliance on complete-modality samples, the magnitude of performance improvements, baseline selection, and the effectiveness of FAR. We clarified that TTSD-FAR targets intermittent modality loss, where complete observations remain available within the test stream, and distinguished this setting from persistent modality absence, which belongs to the privileged-modality research setting. We further explained why complete observations are necessary to obtain the teacher representations used as distillation targets. Furthermore, this setting is consistent with the relevant test-time adaptation for missing modalities. Regarding performance, we highlighted the substantial gains under more challenging missing-modality conditions, particularly on MELD, DFEW, and BAH, while explaining the smaller improvements in cases where removing the weaker modality has minimal impact on the original performance. We strengthened the statistical evaluation by adding three-seed experiments for selected Video-LLaVA-7B results. We also expanded the justification for baseline selection, explaining the limitations of transferring entropy-based, perplexity-based, retrieval-based, and MiDl approaches to autoregressive LVLMs. To isolate the contribution of FAR, we introduced a loss-based start-stop and reactivation baseline, against which FAR demonstrates consistent improvements. Finally, we clarified the teacher's modality-dropout training procedure, explained why periodic resets are unsuitable for the warm-started student, and supplemented the FAR analysis with MMD visualizations to examine its relationship with changes in the incoming data distribution.

W1: Reliance on Complete Modality Samples

A1: We agree that TTSD-FAR adapts only on complete-modality samples, and that a deployment stream containing no complete observation falls outside its scope. In the literature, this is termed as privileged modality, and has a separate line of research. Privileged modality methods typically present train-time methods, whereas TTSD is a test-time protocol for multi-billion-parameter models, where retraining or source data availability limits the train-time approaches. This is already stated in Section 4.1. The evaluation protocol is standard in missing modality literature. We follow the evaluation protocol of Ramazanova er al. (ICLR 2025) [1], which also relies on complete modality samples for adaptation. Similar to that study, in the 100% missing case, we would always predict using the student LoRA and maintain the performance. We nevertheless think the restriction is a property of the problem rather than a defect of the method, for two reasons. First, the target of adaptation is a mapping from a masked view of an input to the complete-modality representation of the same input. A genuinely incomplete observation has no complete-modality counterpart, so no method can obtain such a target from it without an external generative model of the missing modality, which is precisely the assumption that breaks down for billion-parameter autoregressive decoders (Section 2). The student therefore does learn to handle incomplete observations, but it learns the structure of the missing-modality shift from the complete samples that make the shift observable. The fully incomplete regime (100% missing) is a privileged-modality problem and is explicitly outside our scope (Section 4.1). Second, the regime we address, in which a fraction of the stream is complete and a fraction is not, is the typical deployment condition for sensor dropout, transcription failure, and audio corruption, which are intermittent rather than permanent. Evaluation across 0%–50% missing therefore covers the range in which complete samples remain available.

W2: FAR as Convergence and Distribution Shift Detector

A2: We thank the reviewer for identifying this limitation. We agree that a small CI does not establish convergence or coordinate-wise stability. In particular, a nonzero but constant gradient can produce a stabilizing EMA and cause CI to fall below the consolidation threshold. We have therefore revised the interpretation of CI throughout the paper: it is a heuristic measure of relative change in the aggregate gradient second-moment estimate, not a convergence criterion. Entering the ANCHORED state consequently does not imply vanishing gradients or a stationary point. We have also clarified that FMI measures mismatch with the stored gradient-importance estimate, rather than proving a genuine distribution shift. Such mismatch may also result from sample difficulty or stochastic gradient variation. Accordingly, FAR is presented as a heuristic pause-and-reactivation mechanism, not a distribution-shift detector. Further, we would like to state that detecting the change in the stream is not the aim of FAR. Its aim is a reliable signal for when the student should resume adaptation. FMI measures the mismatch between incoming gradients and the statistics of the consolidated student, so it depends on the student and not only on the input. An input change that the consolidated student already handles produces little gradient mismatch and requires no new adaptation, whereas a change that moves the gradients onto parameters that were unimportant at consolidation. For scheduling adaptation, detecting every input change is therefore neither necessary nor sufficient.

Q1:Magnitude of the Gains

A2: We would like to point out that in most of the settings that stress the model, the gains are substantial. On MELD with 50% text missing, F1 improves from 0.4785 to 0.5124 (+3.39%); on DFEW with 50% vision missing, from 0.4430 to 0.4784 (+3.54%); and on BAH with 50% text missing, from 0.6115 to 0.6294 (+1.79%). These are the conditions in which the missing modality is the dominant one and the unadapted model degrades most, and they are also the conditions in which every competing adaptation method either collapses or fails to improve over the unadapted model. The small gains cited by the reviewer, +0.0008 for MELD with 10% vision missing and +0.0010 and +0.0020 for DFEW with 10% and 20% text missing, occur in cases where the modality drop itself barely affects performance. Vision is the weaker modality on MELD and text is the weaker modality on DFEW (Appendix B notes that 4,960 of the DFEW clips contain no utterance at all), so removing 10%–20% of that modality lowers the unadapted F1 by only 0.0027 on MELD and by 0.0019 and 0.0059 on DFEW. There is very little room for improvement in these cells, and the absolute gain is bounded by the size of the drop. We also acknowledge that the main 7B results are reported for a single configuration, while multi-seed statistics are reported only for the 0.5B model (Table 7). This was done because each experiment requires repeated runs with different missing levels and baselines. However, in the revision, we have added repeated runs with 3 different seeds for some Video-LLaVA-7B experiments (Table 2 - DFEW Dataset and Table 3 BAH Dataset) as well, so that the claims for those cells rest on measured variability. For the camera-ready version, we will include this analysis for the remaining tables as well.

Q2: Choice for SOTA Comparisons

A2: We chose the baselines to answer a specific question, namely whether the test-time adaptation signals that work for discriminative models and for text-only LLMs transfer to autoregressive LVLMs under missing modalities. TENT and EATA are the canonical entropy-based test-time adaptation methods and used by multiple recent studies Ramazanova et al. (2025); Hu et al. (ICML 2025), and EATA is of particular relevance because it also uses a Fisher-based regularizer, which makes it the most direct point of comparison for the way FAR uses Fisher information. Their poor performance is therefore an informative result and not the sign of a weak comparison. It is precisely the behavior shown by Hu et al. ( ICML 2025), who show that entropy minimization is ill-suited to test-time adaptation of large language models, because confident predictions from an autoregressive model under distribution shift are frequently incorrect, so minimizing entropy reinforces errors. For the same reason, we included a perplexity-based generation baseline that follows the principle of Hu et al. (ICML 2025), which is the adaptation signal designed for LLMs and the strongest published alternative to entropy for generative models. In addition, we included the retrieval-augmented baseline (RAG) because it is the main inference-time approach that requires no retraining and that is used for LVLMs, and because it fails in a way that our analysis predicts, namely when the observed modality is too weak to retrieve informative neighbors. MiDl (Ramazanova et al., 2025), is the closest to our work in the evaluation protocol, but its published form is not directly applicable to an autoregressive LVLM, and any LVLM version requires design choices that we would have had to make ourselves. First, both of its objectives are defined on a categorical distribution set produced by a classifier head, namely the mutual information between that distribution and the modality combination, and a KL term to the original model’s distribution. An autoregressive LVLM has no such head, so a surrogate must be chosen. Second, MiDl adapts the normalization layers of the backbone, and the corresponding choice for an LVLM (the RMSNorm weights, or LoRA as in our student) changes the method. The point that matters for our claims is that the unadapted model is the relevant reference for whether adaptation helps, and in this setting the existing test-time methods do not improve over it, whereas TTSD-FAR does.

Q3: Comparison with simpler start-stop strategy

A3: We agree that comparing FAR only with plain TTSD and with a loss-plateau rule that never resumes did not test a simple stop-and-reactivate strategy. To that end, we have added an experiment to Appendix Table 8:

MethodFull10%20%30%40%50%
No-Adaptation0.63410.59050.57980.57100.56620.5614
TTSD--0.60650.60390.58870.58160.5728
TTSD-LS--0.59100.58200.57300.56450.5560
TTSD-LS w/ restart--0.59160.58350.57300.56450.5560
TTSD-FAR (Ours)--0.61850.61200.59550.59100.5870

Caption: Ablation comparing TTSD-FAR against plain TTSD (no stopping criterion), TTSD-LS (loss-plateau stopping criterion), and TTSD-LS with restart under progressively missing text conditions. F1 scores are reported on the BAH dataset.

TTSD-LS w/ restart, which freezes the student when the distillation loss has plateaued for 5 samples and resumes adaptation when the loss varies for 5 consecutive samples. It uses the loss for both stopping and reactivation, as FAR uses its gradient statistics for both. On BAH with progressively missing text, TTSD-LS w/ restart behaves almost identically to TTSD-LS (a difference of at most 0.0015 F1, and identical scores at 30–50% missing text, where the restart rule never triggered), so loss-based reactivation recovers little of the gap to plain TTSD. TTSD-FAR outperforms it by 0.023–0.031 F1 and plain TTSD by 0.007–0.014 F1 at every missing ratio. The per-sample loss mostly reflects sample difficulty, which makes it a weak signal for both stopping and resuming. We note the limits of this evidence: it is a single run on one dataset, and we do not claim that FAR is optimal among all stop-and-resume rules, only that it is better than the loss-based rule tested here and than adapting on every sample.

Reviewer_8pnaOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes TTSD-FAR for missing-modality emotion recognition using large video-language models. TTSD aligns the hidden representations of a LoRA student with those of a frozen teacher using complete and synthetically masked inputs. FAR monitors squared-gradient statistics to suspend and resume test-time adaptation.

Experiments on MELD, DFEW, and BAH report improvements over entropy-based adaptation, RAG, and perplexity-based methods. Additional experiments examine FAR, LoRA rank, computational cost, and model scalability.

Critical strengths
  1. The paper addresses missing-modality robustness in large video-language models. Parameter-efficient adaptation is practically relevant when full model retraining is expensive.
  2. The experiments include three affective computing corpora and several missing-modality ratios. The authors also provide FAR ablations, a smaller-backbone experiment, and computational measurements.
Critical weaknesses
  1. The main components have substantial precedents. MiDl (ICLR'25) combines test-time adaptation and self-distillation for missing modalities. SuMi (ICLR'25) addresses stable multimodal adaptation. UMDF (AAAI'24), CorrKD (CVPR'24), and CMAD (ICCV'25) investigate knowledge distillation under incomplete modalities. LDSD (Pattern Recognition 26) and HCKD (Information Fusion 26) further develop representation distillation. EATA (ICML 22) already uses Fisher-based regularization during adaptation. The paper does not establish sufficient methodological novelty beyond combining established techniques with LoRA and a stopping mechanism.
  2. The proposed Fisher statistic is an exponential moving average of squared distillation gradients rather than classical Fisher information. A stable Consolidation Index does not establish optimization convergence. With the reported smoothing coefficient of 0.99, even a twofold increase in the current squared gradient relative to its previous estimate can yield a CI below the stopping threshold. Similarly, the Fisher Mismatch Index does not reliably distinguish distribution shifts from ordinary gradient fluctuations. The theoretical propositions do not establish the claimed convergence or drift-detection guarantees.
  3. The experiments do not evaluate drift detection against known distribution changes. Detection accuracy, false positives, and detection delays are missing. The loss-based stopping baseline remains permanently frozen, whereas FAR can reactivate adaptation. This comparison does not isolate the benefit of Fisher-based monitoring. Stronger controls using loss-based reactivation, gradient-based criteria, and periodic updates are necessary.
  4. The student is updated only when complete-modality samples are available. Missing-modality samples are processed without adaptation. The approach therefore depends on complete observations during deployment. Its effectiveness under persistent modality absence remains untested. Algorithm 1 also appears inconsistent with this protocol. The prediction and update order is insufficiently specified.
  5. The evaluation primarily relies on TENT, EATA, RAG, and perplexity-based generation. Several of these methods were not designed for autoregressive LVLMs or the proposed adaptation setting. Their poor performance does not establish the superiority of TTSD-FAR. Direct comparisons with closely related missing-modality adaptation and distillation methods are absent. The independent contributions of LoRA adaptation, teacher initialization, self-distillation, and FAR are not sufficiently isolated under matched computational budgets.
  6. The main 7B experiments lack repeated runs, standard deviations, and confidence intervals. The three-seed analysis is restricted to a smaller backbone. Several improvements over the non-adapted model are relatively small, reaching only 1.40 percentage points in the MELD vision-missing setting at 50% missingness. Their statistical significance is unknown. Despite updating approximately 0.63% of the model parameters, the reported inference and adaptation time increases by approximately 40%. The practical benefit relative to this overhead is not established.
  7. The paper describes an older BAH version with 1,118 videos and 224 participants, whereas the published ICLR'26 dataset contains 1,427 videos and 300 participants. The exact evaluation version and F1 definition are unclear. Relevant CVPR'26 and ECCV'26 ABAW challenge results are not discussed. The reported full-modality score of 0.7142 also matches the published result (CVPR 2026 ABAW) and requires clarification. Furthermore, Table 9 presents prompt selection based on F1 without establishing whether an independent validation set was used.
  8. The teacher LoRA training procedure and its data requirements are insufficiently documented. The substantial difference between results with and without teacher LoRA makes this omission important. The source of the textual modality in DFEW is also unclear. Hidden-state alignment under different token sequence lengths is not explained. The experiments primarily simulate random modality removal and do not establish robustness under realistic temporal failures, persistent missingness, or changing deployment conditions.
Questions for authors
  1. What is the specific methodological contribution beyond existing missing-modality distillation and test-time adaptation methods? Can the authors provide controlled comparisons with closely related methods such as MiDl and SuMi under comparable experimental conditions?
  2. How do the authors justify using CI as a convergence indicator and FMI as a distribution-shift detector? Can they demonstrate reliable detection under known distribution shifts and report false-positive rates and detection delays?
  3. Does FAR outperform simpler stopping and reactivation strategies under matched computational budgets? Can the authors compare it with loss-based reactivation, gradient-based monitoring, and periodic updates rather than permanent stopping alone?
  4. Why is adaptation performed only on complete-modality samples? How does the method behave when a modality is persistently unavailable? What is the exact prediction and update order, and how does it correspond to Algorithm 1?
  5. How is the teacher LoRA trained, and which corpora and labels are used? How are teacher and student hidden representations aligned when masking changes sequence lengths? Can the authors clarify the gradient accumulation procedure and its effect on online updates?
  6. Are the reported improvements statistically significant across repeated runs of the 7B model? Can the authors provide confidence intervals, compute-matched comparisons, and full-modality performance after adaptation? Do the improvements justify the reported 40% runtime increase?
  7. Which BAH version, evaluation split, and F1 definition are used? How does the reported baseline of 0.7142 relate to the published CVPR'26 and ECCV'26 challenge results? Was prompt selection performed on an independent validation set? What is the source of the textual modality in DFEW?
4 comments
AuthorsOpenReview ↗

Author's Response

We thank reviewer 8pna for their constructive feedback and for the careful review and for recognizing the relevance of our research problem and the methodological contributions of TTSD-FAR. We also appreciate the reviewer's acknowledgment of the strong experimental results across several datasets. We address each comment below, where “W” denotes a weakness, “Q” a question, and “A” our response.

Summary of the changes: i) We clarified the novelty of TTSD-FAR relative to existing test-time adaptation, missing-modality distillation, and Fisher-based regularization methods. In particular, we emphasized that TTSD uses complete-modality teacher representations as supervision for masked student inputs, adapts only a low-rank student on an unlabeled test stream, and introduces FAR as an adaptation control mechanism rather than a conventional Fisher-based regularizer. ii) We addressed concerns regarding the dependence on complete-modality samples by explaining the adaptation protocol and its distinction from persistent modality absence (privileged modality settings). iii) We also corrected Algorithm 1 to explicitly enforce the completeness condition and clarify the prediction-before-update order. iv) To strengthen the experimental comparisons, we introduced a loss-based stopping and reactivation baseline, demonstrating that FAR provides consistent improvements over simpler alternatives. v) Additional clarifications and experiments addressed statistical significance, teacher LoRA training and modality dropout (the teacher was already trained with modality dropout), hidden-state alignment under different sequence lengths, gradient accumulation, BAH dataset details and evaluation metrics, prompt selection, and the source of DFEW transcripts. vi) We also clarified the different full-modality reference scores arising from the use of different model configurations.

W1 and Q1: Novelty relative to existing techniques and Comparison with MiDl and SuMi

A1: Knowledge distillation under incomplete modalities has been studied extensively, and, as we understand these methods, they train task-specific, typically discriminative, models on modality-incomplete training data and so require a training phase and training data, whereas TTSD adapts a frozen LVLM on an unlabeled test stream with a low-rank student. The use of Fisher statistics also differs from EATA: FAR does not regularize individual updates but uses the statistic as a control signal that suspends and resumes adaptation. The claim of the paper is therefore that the existing Test-Time canonical methods do not transfer to autoregressive LVLMs, where entropy- based and retrieval-based signals fail, the proposed method is able to improve the test-time performance of multi-billion-parameter models. Beyond existing missing-modality distillation and test-time adaptation methods, TTSD-FAR makes three specific contributions for a frozen autoregressive LVLM. (i) It supervises the masked input with the hidden representation that a teacher computes from the complete observation of the same instance, so the student receives information about the missing modality, whereas MiDl and SuMi act on the output distribution of a classifier (mutual-information and KL terms in MiDl; entropy minimization with sample selection and smoothing in SuMi) and give the incomplete view no target derived from the complete one. (ii) It adapts only a low-rank student, warm-started from the teacher LoRA, on an unlabeled test stream. (iii) FAR schedules when adaptation starts and stops. We do not report controlled comparisons with MiDl and SuMi because neither can be run as published on an LVLM. Both are defined on a class posterior produced by a classifier head, MiDl adapts normalization layers and needs one forward pass per modality combination, and SuMi is built on entropy optimization over separate unimodal encoders with fusion layers. Applying them would require choices of our own (a surrogate class distribution from label tokens, which parameters to adapt, how to form unimodal branches), so the result would be a vastly new method and not a comparison under matched conditions. Entropy-based adaptation, which SuMi builds on, is loosely represented by TENT and EATA, and it fails on autoregressive models (Hu et al., ICML 2025).

References:

[1] Jinwu Hu, Zhitian Zhang, Guohao Chen, Xutao Wen, Chao Shuai, Wei Luo, Bin Xiao, Yuanqing Li, and Mingkui Tan. Test-time learning for large language models, ICML 2025.

[2] Merey Ramazanova, Alejandro Pardo, Bernard Ghanem, and Motasem Alfarra. Test-time adaptation for combating missing modalities in egocentric videos, ICLR 2025.

W2, W3 and Q2: FAR Convergence and Drift Detection under known distribution shifts

A2: We agree that a small CI does not establish convergence or coordinate-wise stability. In particular, a nonzero but constant gradient can produce a stabilizing EMA and cause CI to fall below the consolidation threshold. We have therefore revised the interpretation of CI throughout the paper: it is a heuristic measure of relative change in the aggregate gradient second-moment estimate, not a convergence criterion. Entering the ANCHORED state consequently does not imply vanishing gradients or a stationary point.

We have also clarified that FMI measures mismatch with the stored gradient-importance estimate, rather than proving a genuine distribution shift. Such mismatch may also result from sample difficulty or stochastic gradient variation. Accordingly, FAR is presented as a heuristic pause-and-reactivation mechanism, not a distribution-shift detector.

An analysis against known shift points is not applicable, because the benchmark streams carry no ground truth for when or whether a shift occurs, and a deployed stream would not either; shifts that we inserted ourselves would only test the trigger on a change that we chose. We can, however, monitor the shifts/change points by visualizing the MMD (independent of FAR statistics) to estimate where the input stream changes. This analysis shows that the FAR reactivation corresponds to the change in MMD shift. This analysis has been added to Appendix C in the revised paper.

Further, we would like to state that detecting the change in the stream is not the aim of FAR. Its aim is a reliable signal for when the student should resume adaptation. FMI measures the mismatch between incoming gradients and the statistics of the consolidated student, so it depends on the student and not only on the changes in the input stream. For scheduling adaptation, detecting every input change is therefore neither necessary nor sufficient.

W3 and Q3: Comparison against simpler Start-stop strategy.

A3: We agree that comparing FAR only with plain TTSD and with a loss-plateau rule that never resumes did not test a simpler stop-and-reactivate strategy. To that end, we have added an experiment to Appendix Table 8:

MethodFull10%20%30%40%50%
No-Adaptation0.63410.59050.57980.57100.56620.5614
TTSD--0.60650.60390.58870.58160.5728
TTSD-LS--0.59100.58200.57300.56450.5560
TTSD-LS w/ restart--0.59160.58350.57300.56450.5560
TTSD-FAR (Ours)--0.61850.61200.59550.59100.5870

TTSD-LS w/ restart, which freezes the student when the distillation loss has plateaued for 5 samples and resumes adaptation when the loss varies for 5 consecutive samples. It uses the loss for both stopping and reactivation, as FAR uses its gradient statistics for both. On BAH with progressively missing text, TTSD-LS w/ restart behaves almost identically to TTSD-LS (a difference of at most 0.0015 F1, and identical scores at 30–50% missing text, where the restart rule never triggered), so loss-based reactivation recovers little of the gap to plain TTSD. TTSD-FAR outperforms it by 0.023–0.031 F1 and plain TTSD by 0.007–0.014 F1 at every missing ratio. The per-sample loss mostly reflects sample difficulty, which makes it a weak signal for both stopping and resuming. We note the limits of this evidence: it is a single run on one dataset, and we do not claim that FAR is optimal among all stop-and-resume rules, only that it is better than the loss-based rule tested here and than adapting on every sample.

W4 and Q4: Reliance on complete samples, persistent absence and update order

A4: Adaptation uses complete samples because they provide the teacher target (Section 1.4). Persistent absence is outside the scope of the missing-modality literature, as it drifts toward the concept of privileged modality, where one modality is completely absent. This evaluation protocol is similar to Ramazanova et al. (ICLR 2025), and is also followed by missing modality literature in general. As for the privileged modality papers, they normally present train-time methods. The exact update order is as follows: the incoming sample xtx_t is predicted with the current parameters; if it is complete (prediction is made with teacher LoRA), a synthetic mask is applied, the student gradient is computed, and the update or the FMI computation follows according to the state; if it is incomplete, prediction is made with the current adapted LoRA. The corrected Algorithm 1 states this order, and the text and the algorithm now describe the same procedure. Under persistent absence, the unadapted baseline results are maintained; this behavior is consistent with the recent test-time missing modality paper Ramazanova et al. (ICLR 2025)

W5 and Q5: Teacher LoRA details and Hidden State Alignment

A5: The teacher LoRA is trained on complete-modality inputs. This paper mainly focuses on the student adaptation under missing conditions, which is why the details on teacher training were originally omitted from the paper. However, we agree that this information becomes pertinent where we claim that the proposed method works both with and without teacher LoRA. The teacher LoRA, in this case, is trained on the training set of the dataset and tuned using the validation set. The aforementioned does not cause a bias in our paper because TTSD-FAR is purely test-time and only adapts on the test set under unsupervised conditions. We have documented the teacher LoRA training along with its parameters in Appendix D. Table 4 shows that the method also works with no teacher LoRA, distilling from the frozen base model, which indicates that the gain does not rest on task-specific teacher supervision.

Alignment Eq. (10) assumes equal sequence lengths. With zero-imputed frames, the number of visual tokens is unchanged. With an empty transcript, the student sequence is shorter. We tested two strategies: i) using the last token representation in each layer, and ii) adding a placeholder token for the student ( this information is available because, in the adaptation step, the teacher has access to the transcript). The latter strategy could cause some information leakage about the length of the missing transcript. We retained the former strategy of last-token embedding-based distillation. This is now explicitly stated in the proposed methodology section. As mentioned earlier, the choices of comparison answer a particular question for LVLM test-time adaptation. TENT and EATA are standard test-time methods and are also used as comparison points in Ramanazova et al. (ICLR 2025) and Hu et al. (ICML 2025). RAG- and perplexity-based methods are designed for LLMs, but they fail for a different reason (poor retrieval under a weak observed modality).

W6 and Q6: Statistical Significance, Full Modality results after adaptation

A6: We acknowledge that the main 7B results are reported for a single configuration, while multi-seed statistics are reported only for the 0.5B model (Table 7). This was done because each experiment requires repeated runs with different missing levels and baselines. However, in the revision, we have added repeated runs with different seeds for some Video-LLaVA-7B experiments (DFEW-Table 2, and BAH Table 3) as well. We will add the remaining multi-seed results in the camera-ready version.

Table2: Comparison of methods under progressively missing visual modality. We report F1 on the DFEW dataset as the proportion of unavailable visual input increases. Seeds: 42, 1000,1999

MethodFull10%20%30%40%50%
No Adap.0.5565 ± 0.0010.5310 ± 0.0040.5095 ± 0.0050.4995 ± 0.0050.4845 ± 0.0150.4430 ± 0.014
TENT--0.1556 ± 0.0070.1557 ± 0.0050.1643 ± 0.0150.1695 ± 0.0200.1655 ± 0.025
EATA--0.2555 ± 0.0080.2778 ± 0.0120.2795 ± 0.0250.2894 ± 0.0250.2910 ± 0.024
RAG--0.5395 ± 0.0140.5055 ± 0.0100.5038 ± 0.0220.4835 ± 0.0210.4463 ± 0.028
(Ours)--0.5518 ± 0.0120.5310 ± 0.0140.5215 ± 0.0200.5005 ± 0.0250.4784 ± 0.023

Table 3: Comparison of methods under progressively missing textual modality conditions for BAH dataset. Seeds: 42, 1000, 1999.

MethodFull10%20%30%40%50%
No Adap.0.7142 ± 0.0020.6844 ± 0.0100.6744 ± 0.0050.6534 ± 0.0150.6204 ± 0.0230.6115 ± 0.025
TENT--0.4934 ± 0.0230.4855 ± 0.0250.4751 ± 0.0220.4554 ± 0.0180.4565 ± 0.015
EATA--0.5625 ± 0.0150.5460 ± 0.0110.5365 ± 0.0210.5334 ± 0.0230.5248 ± 0.014
RAG--0.6830 ± 0.0200.6734 ± 0.0130.6532 ± 0.0190.6214 ± 0.0110.5955 ± 0.019
Perp. Gen.--0.6873 ± 0.0220.6752 ± 0.0160.6530 ± 0.0170.6294 ± 0.0250.5994 ± 0.010
TTSD-FAR (Ours)--0.7055 ± 0.0100.6945 ± 0.0140.6698 ± 0.0220.6386 ± 0.0230.6294 ± 0.026

Full-modality performance after adaptation: After adaptation, the model maintains its upper-bound performance under the 100% available case, because for complete modality samples, the model always predicts using the frozen Teacher LoRA. Furthermore, TTSD-FAR is not an offline adaptation mechanism, where the full-modality results after adaptation are required. Instead, it is an online adaptation mechanism that is applied continually; the full-modality results are presented as an upper bound. However, if it encounters 100% available modalities, the mechanism is designed to maintain the upper-bound results with complete modalities.

W7-W8 and Q7: BAH version, validation split, F1 metric, the 0.7142 reference, prompt selection, and DFEW text transcript source

A7:

BAH version and F1: The evaluation for BAH dataset was performed on the published version of the dataset with 300 subjects. All the comparisons with baselines are performed under the same setting, which makes the comparison valid. The evaluation metric used was Macro-F1, as used by the dataset authors. The dataset description in the appendix was inaccurate and was carried forward from an earlier version of the dataset.

The 0.7142 reference for BAH upper bound: The full-modality reference on BAH is 0.7142 in Table 3 and 0.6341 in Table 4 because the two tables use different base models for the unadapted reference. This is done intentionally to show the applicability of the proposed method on a variety of configurations, in this case, particularly when no Teacher LoRA is available. Table 3 (0.7142-matches the published version of the dataset as well as CVPR 26 challenge results) shows the regular setting where the teacher representations are obtained from a tuned teacher for the dataset. The value 0.6341 is the zero-shot score of the frozen base model with the best prompt (the transcript and definition prompt in Table 9), and Table 4 is the setting without a teacher LoRA. In each table, the reference and the adapted methods are compared within the same configuration, so the gains are measured against a matching baseline. We will state the configuration explicitly in the caption of every table.

Prompt selection: The prompts in Table 9 follow the original dataset paper. The prompt selection mechanism is outside the scope of this paper, as it deals with teacher training, but was included explicitly because the BAH dataset is a fairly new dataset and requires specialized prompts for better performance. However, as mentioned by the other reviewers, teacher training details are also pertinent in our case, so we supplemented this information with the complete teacher LoRA training pipeline. Further, the prompt selection was verified by us on the validation set, but this causes no bias, because TTSD-FAR is purely an unsupervised test-time method.

DFEW text: The reviewer has correctly identified that the DFEW dataset does not come with textual transcripts. These transcripts were manually extracted using OpenAI’s Whisper model, and non-English text was translated to English using DeepTranslate from the Google Translate API. We have described it in detail in Appendix D, and will also provide the final transcripts for reproducible research.

Reviewer_268POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates missing-modality inference in autoregressive video-language models using online feature distillation. A frozen complete-modality teacher supervises a masked-input LoRA student on complete test samples, while Fisher-Anchored Restoration (FAR) freezes and reactivates student adaptation. Experiments cover MELD, DFEW, and BAH with simulated 10-50% missingness.

Critical strengths
  • The missing-modality setting is practically relevant, and the proposed LoRA-only student adaptation avoids full-model updates. The use of available complete samples to supervise synthetically masked inputs is a clear, implementable strategy.
  • Tables 1-3 report consistently positive F1 gains over no adaptation across the three datasets and multiple missing-modality rates. The FAR ablation (Fig. 3; Appendix Table 8), smaller-model multi-seed study (Appx Table 7), and cost analysis provide useful supporting evidence.
Critical weaknesses
  • Teacher-to-masked-student feature distillation and missing-modality test-time adaptation already have close precedents, including MiDl (ICLR 2025), but Tables 1-3 omit a directly relevant comparison. The paper does not yet isolate what is gained specifically by TTSD beyond established distillation and adaptation approaches.
  • The convergence and drift-detection interpretation of FAR is not adequately established: a slowly varying gradient-square EMA need not imply optimization convergence or stable underlying moments (Sec. 3.4; Appx Proposition 1). Proposition 2 primarily restates the state-transition rule, while Figs. 7-8 do not verify detections against known distribution shifts. The main random-missingness evaluations do not, by themselves, establish performance under non-stationary test streams.
Questions for authors

Q1. How was the teacher LoRA trained, on which data and with what supervision, and were the baseline methods given comparable task-specific information? Please explain the different full-modality reference scores between main Table 3 and base-teacher Appendix Table 4. Q2. Can Tables 1-3 separate F1 on complete and actually missing-modality test samples, rather than reporting only pooled scores? Please also reconcile the ACTIVE-state behavior in Algorithm 1 with the complete-sample-only adaptation rule. Q3. Can TTSD-FAR be compared with a carefully adapted MiDl baseline and with compute-matched fixed-budget or gradient-based stop/reactivate strategies? Q4. Can FAR be tested on controlled non-stationary streams with known change points and evaluated using false-alarm rate, detection delay, and post-shift F1 recovery? The currently reported transitions in Figs. 7-8 do not verify that identified events correspond to genuine shifts. Q5. How are teacher and student hidden-state sequences aligned for the layerwise MSE loss when the student transcript is replaced with an empty string (Eq. 10; Appendix D)?

4 comments
AuthorsOpenReview ↗

Author's Response

We thank Reviewer 268P for their careful reading of the paper and for recognizing the clarity of our paper and the methodological contributions. We also appreciate the reviewer's acknowledgment of the strong experimental performance. We address each comment below, where “W” denotes a weakness, “Q” a question, and “A” our response.

Summary of the changes We addressed the reviewer’s concerns regarding i) the applicability of MiDl to autoregressive LVLMs, the choice of baselines, the adaptation protocol, and the methodological details of TTSD-FAR. We clarified that MiDl relies on categorical distributions and normalization-layer adaptation, making its direct application to LVLMs non-trivial and requiring substantial design modifications. We further explained the motivation for selecting TENT, EATA, perplexity-based adaptation, and RAG as comparison methods. ii) Regarding FAR, we acknowledged the limitations of evaluating shift detection without ground-truth change points and supplemented the analysis with MMD visualizations to examine the relationship between distribution changes and FAR reactivation. iii) We also clarified the teacher LoRA training procedure, demonstrated that TTSD-FAR can improve performance even without a task-specific teacher LoRA, explained the differences in full-modality reference scores, and iv) corrected Algorithm 1 to ensure that only complete-modality samples contribute to adaptation. v) Finally, we clarified the hidden-state alignment strategy, using the last-token representations of teacher and student at each supervised layer to accommodate different sequence lengths. vi) We also provide a comparison with a simpler start/stop strategy and show that FAR outperforms a Loss-based start-resume strategy.

W1: Comparison with MiDl

A1: MiDl is a test-time method for missing modality robustness, and we follow its protocol (adaptation only on complete samples, prediction for all) as mentioned in the introduction section. It was not included because its published form is not directly applicable to an autoregressive LVLM, and any LVLM version requires design choices that we would have had to make ourselves. First, both of its objectives are defined on a categorical distribution set produced by a classifier head, namely the mutual information between that distribution and the modality combination, and a KL term to the original model’s distribution. An autoregressive LVLM has no such head, so a surrogate must be chosen. Second, MiDl adapts the normalization layers of the backbone, and the corresponding choice for an LVLM (the RMSNorm weights, or LoRA as in our student) changes the method. Rationale for choosing the baselines: We compare with TENT and EATA methods because they were presented by Hu et al. (ICML 2025) and Ramazanova et al. (ICLR 2025). We additionally used the perplexity-based objective for the text-missing case, because it is not applicable to the vision-missing scenario. Finally, we chose to compare with a RAG-based baseline, because it is a direct competitor in the LLM and LVLM literature to improve the performance of autoregressive models at test-time, Contributions The contribution is a test-time procedure for a frozen autoregressive LVLM under missing modalities, and it addresses three gaps that existing methods leave open for this model class. (i) Supervision signal. Retraining a billion-parameter LVLM for each modality combination is impractical, and removing the text modality induces a measurable shift in the model’s representations (MMD 0.8350, cosine similarity 0.6342, F1 0.6149 → 0.4785 on MELD). Existing test-time signals do not supply a usable target: entropy minimization is ill-suited to autoregressive models (Hu et al., 2025), retrieval fails when the observed modality is too weak to retrieve informative neighbors, and MiDl enforces agreement across modality views on a class distribution and anchors the full view to the original model, so the incomplete view never receives a target derived from the complete observation. TTSD instead supervises the masked view with the teacher’s hidden state computed from the complete observation of the same instance (Eqs. 3 to 5). (ii) Mechanism. Train-time distillation methods for incomplete modalities need a training phase on task data. TTSD updates only a low-rank student (≈ 0.63% of the parameters) on an unlabeled test stream, with two forward passes and one backward pass per adapted sample regardless of the number of modalities. (iii) Control. EATA uses a Fisher term to regularize every update. FAR uses a gradient second-moment statistic to schedule when adaptation is paused and resumed, which we show improves on adaptation without a schedule (Figure 3, Table 8). The ablations (Table 4, Table 8, and the additional ones described above) attribute the gain to the latent target, the low-rank student, and the schedule separately.

W2 and Q4: The convergence and drift-detection interpretation of FAR is not adequately established

A2: We agree that evidence from streams with known change points would be the direct way to measure false-alarm rate, detection delay and F1 recovery. The benchmark streams provide no labeled shifts, and any shift we inserted by construction would test the trigger on a change that we chose, so it could not establish that the triggers fire on real shifts in deployment. We can, however, monitor the shifts/change points by visualizing the MMD (independent of FAR statistics) to estimate where the input stream changes. To this end, we have performed further analysis of input distribution shift (MMD) and plotted it against FMI. We have added this in Appendix C of the revised paper. The analysis reveals that the FMI reactivation corresponds to change points identified by the MMD plot. Furthermore, we soften the claim that FAR detects distribution shift by saying that these state transitions suggest that a distribution shift may have occurred, and maintaining optimum threshold values avoids frequent state transitions, and TTSD-FAR is only triggered when there is a mismatch between the frozen FAR statistics and the incoming samples. It is also important to mention that the primary goal for FAR is not to accurately detect shifts in the incoming stream; rather, it proposes a pause/restart mechanism to reduce unnecessary updates to the student, so the primary aim is to monitor and detect mismatch with the frozen/stored Fisher statistics.

Q1:Teacher LoRA, BAH Results, and Dependence on Teacher supervision and Algorithm 1

Teacher LoRA: The teacher LoRA is trained on complete-modality (with random modality dropout applied) inputs. The Teacher LoRA was trained using the train data, and tuned using the separate validation set provided in the datasets. No test set was used in the tuning of the teacher model. We have added these details to Appendix D. We also note that the sentence in Appendix B describing the teacher as trained “with the objective in Eq. (11)” is incorrect, because Eq. (11) is the student objective, and we have corrected it.

Dependence on teacher supervision: The gains do not depend on the task-specific supervision of the teacher. Table 4 distills from the frozen base model with no teacher LoRA and no task-specific training. In that case, with BAH and text missing, TTSD-FAR reaches 0.6185 at 10% and 0.5870 at 50% against 0.5905 and 0.5614 for the unadapted model, while TENT, EATA, RAG and perplexity-based generation all fall below the unadapted model at every missing rate. Thus, when no method has task-specific training, TTSD-FAR improves on the zero-shot model, and the others do not. Information available to the baselines. All baselines use the same backbone, the same prompts, and the same test stream. TENT, EATA and the perplexity-based baseline are unsupervised and use no labels, as does TTSD-FAR. The RAG baselines, however, do have some task-specific information, as the memory is built using labels on the training data.

Full-modality scores on BAH in Table 3 and App.Tab. 4: The full-modality reference on BAH is 0.7142 in Table 3 and 0.6341 in Tab. 4 because the two tables use different base models for the unadapted reference. This is done intentionally to show the applicability of the proposed method on a variety of configurations, in this case, particularly when no Teacher LoRA is used. Tab. 3 (0.7142) shows the regular setting where the teacher representations are obtained from a tuned teacher for the dataset. The value 0.6341 is the zero-shot score of the frozen base model with the best prompt (the transcript and definition prompt in Table 9), and Table 4 is the setting without a teacher LoRA. In each table, the reference and the adapted methods are compared within the same configuration, so the gains are measured against a matching baseline. We have stated this configuration explicitly in the caption of every table.

Algorithm 1 Inconsistency: The reviewer has identified a real inconsistency in the presentation of the algorithm. The anchored branch of Algorithm 1 contains this check (line 14), but the active branch (lines 3 to 11) does not, and it uses the symbol ˜x both for a genuinely incomplete observation and for a synthetically masked view of a complete sample. We have corrected the algorithm by adding at the start of the loop a branch that, for an incomplete sample, performs inference with the current student parameters and skips all updates, including the update of Ft, the Consolidation Index, and the counter, and by using a separate symbol for the synthetic mask applied to a complete sample x. This matches the description in Section 3.3, where Ft is updated only during the active state and the consolidation counter counts adaptation steps.

Q3: Comparison with Adapted MiDL, and compute-matched start/stop strategy

A3: We have now performed an experiment with a simpler loss-based start-stop strategy and compared it with FAR (Appendix Table 8).

MethodFull10%20%30%40%50%
No-Adaptation0.63410.59050.57980.57100.56620.5614
TTSD--0.60650.60390.58870.58160.5728
TTSD-LS--0.59100.58200.57300.56450.5560
TTSD-LS w/ restart--0.59160.58350.57300.56450.5560
TTSD-FAR (Ours)--0.61850.61200.59550.59100.5870

TTSD-LS freezes the student when the distillation loss plateaus for 5 samples, and TTSD-LS w/ restart also resumes adaptation when the loss varies for 5 consecutive samples, so the loss plays the role that gradient statistics play in FAR. TTSD-LS w/ restart behaves almost identically to TTSD-LS (a difference of at most 0.0015 F1, and its restart rule never triggered at 30–50% missing text), and TTSD-FAR outperforms it by 0.023–0.031 F1 and plain TTSD by 0.007–0.014 F1 at every missing ratio.

The question about the MiDl comparison is answered in response to W1.

Q5: Alignment of teacher and student hidden states In our implementation, the distillation target is the hidden state of the last token, which is the position from which the first answer token is predicted, taken at each supervised layer l∈Ll\in\mathcal{L}. For the teacher, this is position steas_{\text{tea}} and for the student it is position sstus_{\text{stu}}. Both are dd-dimensional vectors, so no token correspondence, padding or projector is needed, and the loss does not depend on the length difference.

References:

[1] Jinwu Hu, Zhitian Zhang, Guohao Chen, Xutao Wen, Chao Shuai, Wei Luo, Bin Xiao, Yuanqing Li, and Mingkui Tan. Test-time learning for large language models, ICML 2025.

[2] Merey Ramazanova, Alejandro Pardo, Bernard Ghanem, and Motasem Alfarra. Test-time adaptation for combating missing modalities in egocentric videos, ICLR 2025.

Dear Reviewer,

As the discussion period is coming to an end soon, we were wondering if our rebuttal addresses your current set of concerns. We are happy to provide further clarifications or answer any other questions if there are any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.