acceptodds
Withdrawn from ICLR 2027

MUTE: Multimodal Uncertainty-Tempered Mutual Teaching for Lifelong Test-Time Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time adaptation (TTA) is crucial for deploying multimodal models in real-world scenarios where test distributions differ from training data. Multimodal test-time adaptation (MMTTA) further leverages complementary information across multiple modalities to improve robustness. In this work, we focus on audio-video MMTTA, where models process temporally aligned audio and video inputs to produce audio, video, and fused predictions. However, we observe that existing TTA methods suffer significant performance degradation in lifelong MMTTA settings, where models must adapt sequentially to a stream of diverse corruptions without forgetting previous knowledge. To address this challenge, we propose MUTE (Multimodal Uncertainty-Tempered Mutual Teaching), a plug-and-play framework that uses confidence-based routing with a modality-advantage criterion to select the most reliable teacher among the audio, video, and fused heads and transfer its supervision to the others. Confidence-conditioned temperature scheduling and adaptive weight modulation stabilize mutual teaching, while our analysis provides a margin-dependent per-step risk bound and a temperature-dependent teacher-target sensitivity bound. MUTE is buffer-free and composes seamlessly with existing TTA methods. Extensive experiments with twelve base TTA methods across two datasets and both audio and video corruption streams demonstrate that MUTE generally improves average accuracy, with gains of up to 42.9 percentage points for severely degraded base methods and up to 2.1 points for already stable ones. Code is available at https://anonymous.4open.science/r/MUTE-ICLR27-C2B4/.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, multimodal learning, lifelong test-time adaptation, continual test-time adaptation, audio-visual learning, mutual teaching, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UY32OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MUTE, a plug-and-play framework for lifelong audio-video test-time adaptation. It selects a teacher among audio, video, and fused prediction heads using entropy-based confidence and a modality-advantage criterion, and employs temperature-scaled KL mutual teaching with scheduled loss weighting. Experiments augmenting twelve base TTA methods on two datasets under audio and video corruption streams show varying degrees of improvement.

Critical strengths

The paper addresses an important challenge in lifelong multimodal test-time adaptation: modality reliability can vary under sequential distribution shifts, making adaptation vulnerable to error accumulation. Leveraging complementary predictions across modalities to improve adaptation reliability is a meaningful research direction.

Critical weaknesses
  1. The theoretical risk bound relies on a key assumption that a sufficiently large confidence advantage implies a lower-risk teacher. Whether this assumption holds in the considered setting remains insufficiently justified.

  2. The performance gains from MUTE vary substantially across experimental settings for the same baseline, particularly SURGEON. The authors' explanation is insufficient, and the claim that MUTE yields smaller marginal gains in already stable regimes is not consistently supported by the reported results.

Questions for authors
  1. Could the substantial variation in SURGEON's performance gains across experimental settings be explained? In particular, why does it improve by 42.9 percentage points on KS50 video corruptions but only 0.1 points on each VGGSound corruption stream?

  2. Could empirical evidence be provided to assess whether the confidence advantage reliably identifies the lower-risk teacher across audio, video, and fused heads under different corruption types?

  3. Could MUTE's methodological novelty beyond existing confidence-based routing, mutual teaching, temperature scaling, and loss scheduling be clarified? Given that the current component ablation is limited to READ on KS50 video corruptions, how can the independent contributions of the proposed components be established?

  4. All experiments use a single CAV-MAE backbone. Could the generalizability of MUTE to other multimodal backbones be further clarified?

  5. In Appendix Tables 10 and 11, the per-corruption results for MEMO+MUTE appear nearly identical to those of the original CoTTA baseline. Could these entries be checked, and could any potential reporting errors affecting the reported average results be clarified?

Reviewer_nu59OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MUTE, a plug-and-play framework for lifelong multimodal test-time adaptation on audio-visual classification. MUTE addresses error accumulation when a model adapts online to a sequence of distribution shifts without reset, letting the audio, video, and fused prediction heads of the same network selectively teach one another. For each unlabeled sample, the most confident head teaches another head only if it exceeds a confidence threshold and leads the student by a margin. MUTE tempers the teacher distributions with a confidence-dependent temperature and adds a scheduled KL-divergence term to the base adaptation objective. The authors provide a per-step risk bound and a target-sensitivity bound, and evaluate MUTE on top of twelve existing TTA methods on Kinetics50-C and VGGSound-C under sequential video- and audio-corruption streams. The results show improvements across most tested configurations, with particularly large gains for base methods that degrade severely during sequential adaptation and smaller gains for methods that are already stable.

Critical strengths
  1. The paper is well motivated by a clear research problem that lifelong multimodal test-time adaptation in audio–video scenarios collapses as the amount of corruption sequentially grows. The paper is clearly written and easy to follow.
  2. The plug-and-play nature of MUTE is attractive and lightweight. It is flexible and can be applied on top of existing TTA methods with different adaptation objectives, adds no trainable parameters, and uses one fixed hyperparameter setting across all twelve base methods, both datasets, and both corruption modalities.
Critical weaknesses
  1. Positioning against continual multimodal test-time adaptation (MMTTA) is incomplete. The paper identifies degradation under lifelong streams as a research gap, but does not cite or properly discuss closely related works on single-step or lifelong MMTTA. For example, MDAA [1] studies MMTTA using a mechanistically similar approach: confidence-based dynamic selection among audio, video, and multimodal classifiers. But this paper does not discuss it. Another uncited work, AVReCAP [2], studies source-free audio-visual lifelong TTA under unimodal and bimodal corruptions without task boundaries. The paper cites DASP [3] and BriMPR [4], but it does not acknowledge that both already define continual protocols, including interleaved-modality streams that this paper does not evaluate. Overall, the paper does not clarify the distinction between MUTE and existing lifelong MMTTA works or MDAA.
  2. Although motivated by issues in lifelong settings, the paper does not demonstrate a specific design that brings benefits specific to lifelong adaptation. Nothing in the method depends on stream history beyond the three-phase schedule over iterations. Mute compares with single-step methods but never compares against lifelong works as baselines. And MUTE is not evaluated under the single-step protocol, while the listed baselines are all single-step.

References:

[1] Zhang, Yufei, et al. "Analytic continual test-time adaptation for multi-modality corruption." Proceedings of the 33rd ACM International Conference on Multimedia. 2025.

[2] Maharana, Sarthak Kumar, et al. "Audio-visual continual test-time adaptation without forgetting." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[3] He, Yongbo, Zirun Guo, and Tao Jin. "Decoupling stability and plasticity for multi-modal test-time adaptation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.

[4] Li, Jiacheng, and Songhe Feng. "Bridging modalities via progressive re-alignment for multimodal test-time adaptation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 27. 2026.

Questions for authors
  1. Can the authors discuss in the related work section MUTE's positioning against MDAA [1] and AVReCAP [2], acknowledge the continual protocols of DASP [3]/BriMPR [4], and report at least one dedicated lifelong-MMTTA method under the same protocol?
  2. Can the authors evaluate on a joint audio-video corruption lifelong stream (e.g., existing benchmarks like AVROBUSTBENCH [5]) and on a second backbone or modality pair? This will demonstrate MUTE's generalizability and make the experiments more comprehensive.

I would be open to raising my score if the authors address my concerns listed in Weakness and Questions, especially by clarifying MUTE's positioning against related work and demonstrating promising additional empirical evidence.

References:

[5] Maharana, Sarthak Kumar, et al. "$\texttt {AVROBUSTBENCH} $: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time." Advances in Neural Information Processing Systems 38 (2026).

Reviewer_4LGhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MUTE, a plug-and-play framework for lifelong audio-video test-time adaptation. It uses confidence-based teacher selection, temperature scaling, and adaptive loss weighting to guide knowledge transfer among audio, video, and fused prediction heads. Experiments with twelve TTA methods on two datasets report improved performance under sequential corruptions.

Critical strengths
  1. The method is compatible with a range of TTA approaches and can be integrated as an add-on module without introducing additional model copies or trainable parameters.

  2. The paper addresses a practically relevant problem: adapting multimodal models to sequential distribution shifts. This setting better reflects continuous deployment than evaluating adaptation to individual corruptions in isolation.

Critical weaknesses
  1. The method introduces a large number of hyperparameters. Although some ablation studies and parameter analyses are provided, they do not sufficiently establish whether the added complexity is justified.

  2. Assumption 1 stipulates that higher entropy-based confidence implies lower true risk, with an improvement of at least κδ\kappa\delta. This assumption is not self-evident in lifelong adaptation, where distribution shifts may lead to confidently incorrect predictions. In effect, the assumption presupposes that the selected teacher is better. Using it to justify the effectiveness of the teacher selection mechanism therefore risks circular reasoning.

  3. The abstract and introduction do not clearly define the lifelong adaptation setting or explain the “single-step” setting used for comparison. The term “head” is also used without an explicit definition or architectural description, although the subsequent discussion of logits suggests that it refers to a classification head.;

Questions for authors
  1. The experiments include many methods originally developed for other backbones or tasks, yet omit several more directly relevant multimodal methods discussed in the related work. Why are these methods not included? Could they be evaluated using the same backbone, datasets, and lifelong adaptation protocol?

  2. Why is it necessary to apply temperature scaling to the teacher predictions rather than directly minimize the KL divergence between the student predictions and the original teacher predictions? Please include an ablation with T=1T=1. Moreover, all temperatures used in practice are below 1, which sharpens the teacher distribution rather than softening it. When the teacher prediction is incorrect, how does the method prevent this sharpening from further reinforcing erroneous supervision?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.