acceptodds
Under review as a conference paper at ICLR 2027

Decorate Your Incomplete Multimodal Inputs: Towards Test-Time Adaptation for Missing Modalities

Abstract

The real-world deployed systems often encounter the incomplete multimodal instances, which incentivize the study of Modality-Missing Learning (MML). Nevertheless, existing methods rely on customized model architectures and training strategies to handle incomplete multimodal inputs. As a result, they are less compatible with the real-world applications, where downstream models are often already designed and trained, and the training process is also often inaccessible due to the privacy issues. To address these limitations, we propose a more practical research problem, MML-TTA, which aims to improve the robustness of multimodal models, previously designed and pre-trained on modality-complete samples, to incomplete instances at test time. Accordingly, we introduce a new framework IM-DECOR. It treats latent incomplete multimodal inputs as a model-agnostic adaptation anchor, and decorates these inputs to mimic complete ones observed during training, bridging the distribution shift in MML-TTA caused by input-level structural mismatches. Specifically, IM-DECOR introduces a modality-wise Global Decorator (G-DECOR) and an instance-aware Local Decorator (L-DECOR). They progressively transform incomplete inputs from mimicking modality-level distributions of complete data to resembling instance-specific complete inputs, enabling robust, on-the-fly adaptation of diverse downstream models to test-time missing inputs. Experiments on six benchmarks under MML-TTA show the efficacy of our method, achieving an average relative improvement of 9.86% over the strongest baseline.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Incomplete multimodal learning, test-time modality-missing adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Bo86OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies TTA method to adapt pretrained multimodal models to handle conditions with missing modalities. The name of the method is IM-DECOR, which reconstructs missing modality representations using online prototype containers and instance-conditioned retrieval. The method is evaluated extensively across multiple multimodal benchmarks and missing-modality settings.

Critical strengths
  1. Clear motivation and paper is easy to follow.
  2. The paper evaluates the method on six benchmarks with different settings.
  3. IM-DECOR generally outperforms all selected TTA baselines.
Critical weaknesses
  1. The core method is largely a combination of standard components. It consists of K-means prototypes, an online feature memory, MLP-based query projection, cosine-similarity retrieval, weighted prototype aggregation, and entropy minimization. While the overall design is reasonable and empirically effective, I find the technical contribution fairly incremental. Most of these components have already appeared in prior work, and the proposed framework feels more like a combination of existing techniques in an “A + B + C” manner than a genuinely new methodological contribution. Overall, I view this as a successful engineering pipeline rather than a method with substantial technical novelty.

  2. MML-TTA itself does not appear to be a new research problem. Missing-modality adaptation at test time has already been studied in some prior works (e.g., MiDl ICLR 2025). Therefore, I do not think formulating MML-TTA should be presented as a major contribution.

  3. The terminology around “decoration” is somewhat unconventional. “Decoration” is not a standard term in the missing-modality literature, while the proposed operation is conceptually much closer to feature imputation or missing-modality-representation completion. Introducing new terminology does not by itself constitute novelty, and in this case it arguably makes the method less transparent.

  4. No systematic ablation study was conducted regarding batch composition or "small-batch stability." In the main experiments, the batch size was set to 128 for MM-IMDb, Food101, IEMOCAP, and MOSI, and to 32 for Kinetics50 and VGGSound. These are relatively large batch sizes, particularly for a testing environment. Therefore, I suggest adding an analysis of prototype stability under varying batch sizes, different modality sample proportions, and class compositions.

  5. The method relies on a fairly strong stream-level modality coverage assumption. Each modality must appear at least occasionally in the test stream so that its prototype container can be initialized and updated. This raises concerns in persistent cold-start scenarios where a modality is unavailable for an extended period or never appears at all. The authors appear to have partially validated this in the context of "Robustness to Extreme Missing Scenarios". However, this seems insufficient, as they did not verify the specific case where the text is completely absent from the very beginning of deployment. Furthermore, batch sizes of 128 or 32 are relatively large. In real-world deployment, inference often involves single samples or sparse requests, such as a medical diagnosis, a multimodal query, or sporadic event detection, where there are insufficient historical test samples to construct stable prototypes.

  6. Moreover, the "stream" used in the experiments likely consists merely of the test set fed in sequentially by batch. This differs significantly from real-world continuous temporal streams, user streams, or sensor streams, and also differs from the video-based stream definition used in the MiDl baseline paper. In short, the authors' assumption that deployment requires a dense stream is not entirely reasonable, which undermines the claim that the model is "model-agnostic." (what about the sparse query medical multimodal model) Additionally, the authors assume the existence of a stream with a continuous influx of diverse samples for online updates. However, a key challenge with online updates is addressing long-term distribution drift and error accumulation. How do the authors handle this?

Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Bo86 (1/4)

We sincerely thank you for the thoughtful comments and constructive suggestions. We are particularly encouraged by your recognition of (1) the clear motivation and presentation of the paper, (2) the evaluation on six benchmarks under different settings, and (3) the improvements of IM-DECOR over all selected TTA baselines. Below, we carefully address your remaining concerns.

[W1] The core method is largely a combination of standard components (K-means prototypes, an online feature memory, MLP-based query projection, cosine-similarity retrieval, weighted prototype aggregation, and entropy minimization). The design is reasonable and empirically effective, but the framework feels like an "A + B + C" combination of existing techniques, i.e., a successful engineering pipeline rather than a method with substantial technical novelty.

We sincerely thank you for this candid assessment, and for acknowledging that the design is reasonable and empirically effective. We agree that the building blocks of IM-DECOR are deliberately simple. Our technical contribution lies in where and how the adaptation is performed under MML-TTA, and we elaborate on our considerations below.

  1. Adapting the incomplete inputs, rather than the model, is the new mechanism. We examine how the shift in MML-TTA differs from those in conventional TTA (Appendix H.2). Existing unimodal and multimodal TTA methods adapt the model to a shifted representation that still exists, i.e., z(m)→z~(m)z^{(m)} \rightarrow \tilde{z}^{(m)}. In contrast, modality missing removes the representation entirely, i.e., z(m)→∅z^{(m)} \rightarrow \varnothing, so there is no representation left to adapt. Therefore, IM-DECOR treats the latent incomplete inputs as a general adaptation anchor and decorates them to mimic complete ones, while the source model stays frozen. This also explains why the seven TTA baselines bring limited or even negative gains over the Source model (Tables 1 and 2 in the main paper).

  2. The gain comes from decoration, not from the standard components. We design two variants that separate decoration from generic entropy minimization (Table 9 in the appendix). On MM-IMDb with text missing, EM w/o Decoration falls below the Source model (36.35 vs. 39.28), whereas Decoration w/o EM already reaches 46.92 and the full IM-DECOR reaches 48.22. Moreover, we adapt imputation, reconstruction, and alignment to MML-TTA (AVG, RECON, and ALIGN, Figure 10 in the appendix), and all three remain below IM-DECOR and even underperform the Source model in some cases.

  3. Each component answers a constraint of MML-TTA. Under MML-TTA, the test stream is unlabeled, the source data are inaccessible, and each batch offers only a few samples. Accordingly, (i) G-DECOR keeps only K=4K = 4 prototypes per modality, which need no training and are accumulated online, so that decoration remains available even when a modality is entirely missing over consecutive batches (Figure 4 in the main paper); (ii) L-DECOR turns the remaining modalities into learnable queries, so that the decoration is instance-specific rather than a shared average, and removing it lowers the result from 48.22 to 45.10 (Table 3 in the main paper); and (iii) LIMDRL_\text{IMDR} draws self-supervision from relatively complete samples without any label.

  4. The minimalist design keeps the method practical. Since backpropagation bypasses the modality-specific encoders and only the decoration parameters are updated, IM-DECOR has a modest runtime and one of the smallest peak GPU memory footprints among all methods (Figure 6 in the appendix). Moreover, it applies to six benchmarks and four source-model architectures without modifying the source models.

Overall, IM-DECOR achieves an average relative improvement of 9.86% over the strongest baseline.

AuthorsOpenReview ↗

Response to Reviewer Bo86 (2/4)

(Part 2/4)

[W2] MML-TTA does not appear to be a new research problem. Missing-modality adaptation at test time has already been studied in prior works (e.g., MiDl, ICLR 2025). Therefore, formulating MML-TTA should not be presented as a major contribution.

We thank you for this important point, and we agree that MiDl is a pioneering work on addressing missing modalities at test time. We discuss it in Section 2 and compare with it throughout our main experiments, and we apologize that our wording did not give it sufficient credit. What MML-TTA contributes is a more general formulation and a systematic study of this problem, and we clarify the differences below.

  1. MML-TTA is not tied to a task or an architecture. MiDl studies the downstream task of egocentric video understanding. In contrast, we formulate MML-TTA for a generic source model Fθa={fθ(1),fθ(2),Hθ,Pθ}F_{\theta}^{a} = \{f_{\theta}^{(1)}, f_{\theta}^{(2)}, H_{\theta}, P_{\theta}\} (Section 3), and instantiate it on six benchmarks that cover text, image, audio, and video, dual- and tri-modality compositions, and four source-model architectures.

  2. MML-TTA does not require modality-complete samples. The adaptation of MiDl relies on modality-complete samples in the test stream. In contrast, MML-TTA only assumes stream-level modality coverage, which also admits streams without any complete sample. We evaluate this complete-sample-free scenario in Figure 4(c)–(d) in the main paper, where IM-DECOR remains effective across all image-to-text missing ratios.

  3. MML-TTA comes with a systematic evaluation protocol. We examine three missing rates (50%, 70%, and 90%), single- and multi-modality missing types, missing patterns that change in every batch (Figure 3 in the main paper), modalities that are entirely missing over contiguous segments (Figure 4(a)–(b) in the main paper), and the cold-start boundary (Appendix E).

  4. The general problem remains open for existing methods. Under this protocol, MiDl stays close to the Source model, e.g., 36.33 vs. 36.06 on MM-IMDb on average (Table 1 in the main paper), whereas IM-DECOR reaches 48.70.

Following your comment, we will credit MiDl explicitly in the Introduction, and rephrase our first contribution as generalizing and systematically studying test-time missing-modality adaptation rather than introducing a brand-new problem.

[W3] "Decoration" is not a standard term in the missing-modality literature, while the proposed operation is conceptually much closer to feature imputation or missing-modality-representation completion. Introducing new terminology does not by itself constitute novelty, and in this case it arguably makes the method less transparent.

We thank you for this helpful comment. We fully agree that a new term does not constitute novelty by itself, and we do not claim novelty from the name. We explain our considerations behind it below.

  1. Decoration is driven by the frozen model rather than by fidelity to the missing modality. In the MML literature, imputation and reconstruction usually denote recovering the missing modality itself with networks trained on the source data. In contrast, our operation is optimized at test time so that the incomplete input mimics a complete one for the frozen source model. Specifically, it is updated by LEML_\text{EM} on the downstream prediction, and it contains learnable residual biases Δ(m)\Delta^{(m)} and ξi(m)\xi_i^{(m)} that go beyond any retrieved modality content (Section 3.2).

  2. Decoration is built from the test stream. The decoration of a sample aggregates prototypes accumulated from other test samples, conditioned on its remaining modalities, rather than being produced by a trained cross-modal generator.

  3. The term conveys input-side adaptation. We use "decorate" to stress that the source model is wrapped rather than modified, which is the key property under MML-TTA.

Nevertheless, we agree that the link to the existing literature should be transparent.

AuthorsOpenReview ↗

Response to Reviewer Bo86 (3/4)

(Part 3/4)

[W4] No systematic ablation study was conducted regarding batch composition or "small-batch stability". The main experiments use a batch size of 128 (MM-IMDb, Food101, IEMOCAP, MOSI) or 32 (Kinetics50, VGGSound), which are relatively large for a testing environment. I suggest adding an analysis of prototype stability under (1) varying batch sizes, (2) different modality sample proportions, and (3) class compositions.

We sincerely thank you for this constructive suggestion. We agree that the stability of the decoration containers should be examined beyond the default batch sizes, and we address the three factors below.

  1. Batch size. We first clarify how the containers depend on the batch. The online update of G-DECOR (Eq. (4)) is a per-sample running mean with a counter, so the prototypes are driven by all samples accumulated over the stream rather than by any single batch. Moreover, when a batch holds fewer than KK samples of a modality, e.g., under an extremely small batch size, IM-DECOR temporarily sets KK to this number and reinitializes the container once more samples arrive (Appendix F.4). To verify this, we vary the online batch size from 128 down to 1 on MM-IMDb with text missing at the 70% rate.

    Method1832128 (paper)
    Source39.2839.2839.2839.28
    DASP26.7837.0942.6241.09
    IM-DECOR45.9147.5747.8748.22

    As shown, IM-DECOR remains stable across batch sizes. Even with batch size 1, it reaches 45.91, well above the Source model (39.28) and DASP (26.78). Therefore, the decoration containers do not depend on large batches.

  2. Modality sample proportions. We already vary the proportions along three axes in the paper. Specifically, we evaluate missing rates of 50%, 70%, and 90% on all six benchmarks (Tables 1 and 2 in the main paper, Tables 6 and 7 in the appendix), image-to-text missing ratios from 10%:90%10\%:90\% to 90%:10%90\%:10\% without any modality-complete sample (Figure 4(c)–(d) in the main paper), and missing types that change in every incoming batch (Figure 3 in the main paper). IM-DECOR outperforms all baselines in each case. For instance, at the 90% missing rate, where only about one tenth of the samples in a batch still carry the affected modality, IM-DECOR improves the Source model from 26.67 to 44.92 on MM-IMDb on average.

  3. Class composition. We further evaluate a class-sorted stream on Food101 with text missing at the 70% rate, where the samples arrive class by class and each batch therefore covers only one or a few classes.

    MethodPaper streamClass-sorted stream
    Source76.6976.69
    DASP77.6277.38
    IM-DECOR79.3278.67

    As shown, IM-DECOR still outperforms the Source model and DASP on the class-sorted stream, and stays within 0.65 points of its result on the paper stream. Therefore, the prototypes remain stable when each batch covers only a few classes.

    Moreover, as discussed in Appendix I.2, G-DECOR can incorporate techniques such as pseudo-labeling-based filtering to further stabilize the prototypes under highly imbalanced or temporally correlated streams.

We will add both studies to Appendix G of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Bo86 (4/4)

(Part 4/4)

[W5] The method relies on a fairly strong stream-level modality coverage assumption. (1) This raises concerns in persistent cold-start scenarios where a modality is unavailable for an extended period or never appears, and the case where the text is completely absent from the very beginning of deployment is not verified. (2) Real-world inference often involves single samples or sparse requests rather than batches of 128 or 32, with insufficient historical test samples to construct stable prototypes.

We thank you for this thoughtful concern. We first clarify that stream-level modality coverage requires neither modality-complete samples nor the presence of every modality in every batch (Appendix H.4). Once a modality has been observed, the container retains it over later segments where it is missing (Figure 4(a)–(b) in the main paper). We address the remaining cases below.

  1. Cold start. We apologize for the lack of clarity, since this case is studied in Appendix E rather than in the main paper.

    • Absent throughout the stream. We evaluate the most extreme form on MM-IMDb and Food101, where the missing rate of a modality stays at η=100%\eta = 100\% throughout the entire stream (Figure 7 in the appendix). Following prior TTA works that allow minimal source-domain statistics, IM-DECOR uses lightweight modality-wise source prototypes, without accessing any training sample. With text absent throughout, it improves the Source model from about 27 to about 42 on MM-IMDb, whereas DASP stays close to the Source model.
    • Absent for an extended period. Without source statistics, IM-DECOR behaves like a standard TTA method until the modality first appears, and launches decoration from then on (Appendix E). To verify this, we keep text absent during the first 40% of the stream on MM-IMDb. Once text appears, IM-DECOR quickly reaches 48 on the remaining stream, compared with 39 for the Source model.
  2. Single samples and sparse requests. The decoration container is a persistent state carried across requests. A request is therefore decorated from the prototypes accumulated over earlier requests, rather than from the samples that share its batch, and only K=4K = 4 prototypes per modality need to be stored. The single-sample case is covered by batch size 1 in our response to [W4], where IM-DECOR still improves the Source model from 39 to 46.

[W6] (1) The "stream" is likely just the test set fed in sequentially by batch, which differs from real-world temporal, user, or sensor streams and from the video-based stream in the MiDl paper. (2) Assuming a dense stream is not entirely reasonable, which undermines the "model-agnostic" claim (what about a sparse-query medical multimodal model?). (3) How do the authors handle long-term distribution drift and error accumulation under online updates?

We address these questions below.

  1. Stream protocol. We follow the practical online TTA setting (Wang et al., 2021), where unlabeled test data arrive sequentially in batches, and all baselines, including MiDl, are evaluated on the same streams. Beyond this standard protocol, we stress the stream with missing types that change in every batch (Figure 3 in the main paper) and with contiguous segments where a modality is entirely missing (Figure 4(a)–(b) in the main paper). In addition, the class-sorted stream in our response to [W4] examines temporally correlated arrivals.

  2. Dense streams and the model-agnostic claim. Model-agnostic refers to the source model, i.e., independence from model-specific redesign or retraining (Appendix H.1), whereas the stream is a property of the test data. As explained in [W5], IM-DECOR does not require a dense stream. For a sparse-query medical model, the container is accumulated from the modalities observed in earlier queries, or initialized from lightweight source prototypes when available (Appendix E).

  3. Long-term drift and error accumulation. IM-DECOR limits error accumulation by design.

    • Frozen source model. The source model is never updated, so no error accumulates in its parameters, and modality-complete samples are always predicted from their original modalities without decoration (Section 3.3).
    • Prediction-free containers. The prototypes are running means of the frozen encoders' outputs on observed modalities and involve no pseudo-labels, so prediction errors are never written back into them.
    • Bias reset. The sample-specific biases are reset after each adaptation batch to mitigate parameter drift (Section 3.3).

    Empirically, IM-DECOR stays above the Source model and MiDl along the whole stream, including the 439 online batches of VGGSound (Figure 3 in the main paper). Semantic drift on top of modality missing forms a compound shift, which we discuss as future work in Appendix I.1.

We would be most grateful if you might consider reflecting this in your rating based on clarifications and more empirical results.

Reviewer_Ur9QOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies test-time adaptation of pretrained multimodal models to missing-modality inputs. Its proposed framework, IM-DECOR, maintains modality-specific prototypes from the test stream and uses the available modalities to retrieve representations for missing inputs. The decoration parameters are optimized online while the downstream model remains frozen. Experiments cover six benchmarks, different missing rates and patterns, and multiple model architectures.

Critical strengths
  1. Continually adapting model to missing-modality inputs at test time is an insteresting topic.
  2. Evaluations are well-conducted and the performance of the proposed approach is good.
Critical weaknesses

1.The novelty of the proposed problem setting is overstated. The claimed new problem setting is already substantially covered by MiDl. 2. The comparison with existing missing-modality imputation methods is insufficient. One concrete example to discuss is Knowledge Bridger (CVPR 2025), a training free approach, which the submission already cites but not compared. Appendix G.6 includes mean imputation and simplified reconstruction and alignment baselines, which provide limited evidence of the method’s advantage over established imputation approaches.

I would consider raise my rating if the above two issues could be addressed by the authors.

Questions for authors

See above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Ur9Q (1/2)

We sincerely thank you for the thoughtful comments and constructive suggestions. We are particularly encouraged by your recognition that (1) continually adapting models to missing-modality inputs at test time is an interesting topic, and (2) our evaluations are well-conducted and the performance of IM-DECOR is good. Below, we carefully address your two concerns.

[W1] The novelty of the proposed problem setting is overstated. The claimed new problem setting is already substantially covered by MiDl.

We thank you for this important point, and we agree that MiDl is a pioneering work on addressing missing modalities at test time. We discuss it in Section 2 and compare with it throughout our main experiments, and we apologize that our wording overstated the novelty of the setting. What MML-TTA contributes is a more general formulation and a systematic study of this problem, and we clarify what it adds beyond the setting of MiDl below.

  1. MML-TTA is not tied to a task or an architecture. MiDl studies the downstream task of egocentric video understanding. In contrast, we formulate MML-TTA for a generic source model Fθa={fθ(1),fθ(2),Hθ,Pθ}F_{\theta}^{a} = \{f_{\theta}^{(1)}, f_{\theta}^{(2)}, H_{\theta}, P_{\theta}\} (Section 3), and instantiate it on six benchmarks that cover text, image, audio, and video, dual- and tri-modality compositions, and four source-model architectures.

  2. MML-TTA does not require modality-complete samples. The adaptation of MiDl relies on modality-complete samples in the test stream. In contrast, MML-TTA only assumes stream-level modality coverage, i.e., any modality may be missing in individual samples as long as the stream collectively retains all modalities (Section 3 and Appendix H.4). This also admits streams without any complete sample. We evaluate this complete-sample-free scenario in Figure 4(c)–(d) in the main paper, where IM-DECOR remains effective across all image-to-text missing ratios.

  3. MML-TTA comes with a systematic evaluation protocol. We examine three missing rates (50%, 70%, and 90%), single- and multi-modality missing types, missing patterns that change in every batch (Figure 3 in the main paper), modalities that are entirely missing over contiguous segments (Figure 4(a)–(b) in the main paper), and the cold-start boundary (Appendix E).

  4. The general problem remains open for existing methods. We compare MiDl with the Source model under this protocol. MiDl stays close to the Source model, e.g., 36.33 vs. 36.06 on MM-IMDb and 44.14 vs. 44.34 on VGGSound on average (Table 1 in the main paper), whereas IM-DECOR reaches 48.70 and 46.47.

Following your comment, we will credit MiDl explicitly in the Introduction, and rephrase our first contribution, together with the corresponding statements in the Abstract and Conclusion, as generalizing and systematically studying test-time missing-modality adaptation rather than introducing a brand-new problem. We will also state these differences explicitly in Section 2.

AuthorsOpenReview ↗

Response to Reviewer Ur9Q (2/2)

(Part 2/2)

[W2] The comparison with existing missing-modality imputation methods is insufficient. One concrete example to discuss is Knowledge Bridger (CVPR 2025), a training-free approach, which the submission already cites but does not compare with. Appendix G.6 includes mean imputation and simplified reconstruction and alignment baselines, which provide limited evidence of the method's advantage over established imputation approaches.

We sincerely thank you for this valuable suggestion. We agree that a comparison with established imputation approaches makes the advantage of IM-DECOR more convincing, and we address it below.

  1. The baselines in Appendix G.6 are simplified to comply with MML-TTA. Established MML methods rely on labeled source data, dedicated training, and customized architectures. None of these is available under MML-TTA, where the source model is frozen and only the unlabeled test stream is accessible. We therefore adapt one representative of each family, namely imputation (AVG), reconstruction (RECON), and alignment (ALIGN). As shown in Figure 10 in the appendix, all three remain below IM-DECOR and even underperform the Source model in some cases, mainly because reconstruction and alignment have to be learned online from the few unlabeled modality-complete samples in the stream.

  2. Knowledge Bridger is the closest training-free approach. Since it requires no training, it is compatible with a frozen source model. It completes the missing modality at the raw-data level, where a large multimodal model extracts knowledge from the available modality, generates candidates for the missing one, and ranks them. In contrast, IM-DECOR decorates in the latent space of the source model with lightweight prototypes collected from the test stream. Following your suggestion, we apply Knowledge Bridger to each incomplete sample and feed the completed input into the same frozen source model. We compare on the two image–text benchmarks under the 70% missing rate.

    MethodMM-IMDb TMM-IMDb IFood101 TFood101 I
    Source39.2831.7976.6983.89
    Knowledge Bridger43.7540.3478.0185.82
    IM-DECOR48.2248.8879.3287.76

    As shown, IM-DECOR outperforms Knowledge Bridger in all four settings, e.g., 48.22 vs. 43.75 on MM-IMDb with text missing. This suggests that completing the raw modality does not by itself match what the frozen source model expects, whereas IM-DECOR decorates directly in its latent space.

    Moreover, the two approaches differ in cost and coverage. Knowledge Bridger queries a large multimodal model to generate and rank candidates for every incomplete sample (about 12.94 s per sample), whereas IM-DECOR takes less than one second for an online batch of 128 samples (Figure 6 in the appendix). In addition, decoration in the latent space applies directly to audio, video, and pre-extracted features, as in Kinetics50, VGGSound, IEMOCAP, and MOSI, where raw-level generation is not directly applicable.

  3. IM-DECOR also outperforms a supervised alternative. We further compare with unimodal models that are trained with labels for each available modality (Table 11 in the appendix). IM-DECOR still outperforms them, e.g., 48.22 vs. 45.48 on MM-IMDb with text missing, although it needs no additional supervised training.

We will add the comparison with Knowledge Bridger and its discussion to Appendix G.6 of the revised manuscript.

We sincerely appreciate your willingness to reconsider the rating. We hope that our responses address your two concerns, and we would be glad to continue the discussion. We would sincerely appreciate it if these clarifications might provide a basis for a favorable reconsideration of the rating.

Reviewer_b2L3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IM-DECOR, which enables test-time adaptation to missing modalities through global and local modality decoration without modifying the parameters of pre-trained models, achieving competitive performance across multiple multimodal benchmarks.

Critical strengths
  1. The paper addresses the problem of missing modalities at test time, which has practical relevance to real-world applications.

  2. The proposed framework combines global modality prototypes with instance-specific information for missing modality completion, presenting a relatively clear overall design.

  3. Experiments are conducted on six multimodal datasets, including ablation studies and evaluations under different missing modality conditions.

Critical weaknesses
  1. The methodological novelty is limited, as the core components mainly rely on existing techniques such as K-means clustering, prototype retrieval, and entropy minimization, without introducing substantially new mechanisms.

  2. The proposed MML-TTA setting substantially overlaps with existing studies on test-time adaptation for missing modalities, and its novelty as an independent research problem is not sufficiently established.

  3. The method relies on intermediate modality representations and feature injection interfaces of pre-trained models, which limits the claimed model-agnostic and plug-and-play capabilities.

  4. The experimental comparisons mainly focus on general TTA methods, while comparisons with missing modality completion approaches are limited, making it difficult to fully establish the advantages of the proposed mechanisms.

  5. The theoretical analysis largely builds upon existing domain adaptation error bounds and does not sufficiently establish the connection between the proposed global and local decoration mechanisms and the observed performance improvements.

  6. The method assumes that each modality remains available at least occasionally in the test data stream and does not sufficiently consider scenarios where modality missingness and semantic distribution shifts occur simultaneously, limiting its practical applicability.

Questions for authors

Please see above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer b2L3 (1/3)

We sincerely thank you for the thoughtful comments and constructive suggestions. We are particularly encouraged by your recognition of (1) the practical relevance of handling missing modalities at test time, (2) the clear overall design that combines global modality prototypes with instance-specific information, and (3) the experiments on six multimodal datasets with ablation studies and different missing conditions. Below, we carefully address your concerns.

[W1] The methodological novelty is limited, as the core components mainly rely on existing techniques such as K-means clustering, prototype retrieval, and entropy minimization, without introducing substantially new mechanisms.

We thank you for this comment. We agree that the building blocks of IM-DECOR are deliberately simple. The new mechanism lies in where the adaptation is performed under MML-TTA, and we elaborate on our considerations below.

  1. Adapting the incomplete inputs, rather than the model, is the new mechanism. We examine how the shift in MML-TTA differs from those in conventional TTA (Appendix H.2). Existing TTA methods adapt the model to a shifted representation that still exists, i.e., z(m)→z~(m)z^{(m)} \rightarrow \tilde{z}^{(m)}. In contrast, modality missing removes the representation entirely, i.e., z(m)→∅z^{(m)} \rightarrow \varnothing, so there is no representation left to adapt. Therefore, IM-DECOR treats the latent incomplete inputs as a general adaptation anchor and decorates them to mimic complete ones, while the source model stays frozen.

  2. The gain comes from decoration, not from the existing techniques. We design two variants that separate decoration from generic entropy minimization (Table 9 in the appendix). On MM-IMDb with text missing, EM w/o Decoration falls below the Source model (36.35 vs. 39.28), whereas Decoration w/o EM already reaches 46.92 and the full IM-DECOR reaches 48.22. Moreover, the global-to-local design matters. Specifically, removing G-DECOR lowers the result to 40.60, and replacing L-DECOR with a simple average of the prototypes lowers it to 45.10 (Table 3 in the main paper).

  3. The simple components are chosen for the constraints of MML-TTA. Under MML-TTA, the test stream is unlabeled, the source data are inaccessible, and each batch offers only a few samples. Therefore, we keep only K=4K = 4 prototypes per modality that need no training and are accumulated online, and we update only the lightweight decoration parameters. This minimalist design (Appendix I.2) gives IM-DECOR a modest runtime and one of the smallest peak GPU memory footprints among all methods (Figure 6 in the appendix).

[W2] The proposed MML-TTA setting substantially overlaps with existing studies on test-time adaptation for missing modalities, and its novelty as an independent research problem is not sufficiently established.

We thank you for this important point, and we agree that MiDl is a pioneering work on addressing missing modalities at test time. We discuss it in Section 2 and compare with it throughout our main experiments, and we apologize that our wording did not give it sufficient credit. What MML-TTA contributes is a more general formulation and a systematic study of this problem, as clarified below.

  1. MML-TTA is not tied to a task or an architecture. MiDl studies the downstream task of egocentric video understanding. In contrast, we formulate MML-TTA for a generic source model (Section 3), and instantiate it on six benchmarks that cover text, image, audio, and video, dual- and tri-modality compositions, and four source-model architectures.

  2. MML-TTA does not require modality-complete samples. The adaptation of MiDl relies on modality-complete samples in the test stream. In contrast, MML-TTA only assumes stream-level modality coverage, which also admits streams without any complete sample (Figure 4(c)–(d) in the main paper).

  3. MML-TTA comes with a systematic evaluation protocol. We examine three missing rates, single- and multi-modality missing types, missing patterns that change in every batch (Figure 3 in the main paper), modalities that are entirely missing over contiguous segments (Figure 4(a)–(b) in the main paper), and the cold-start boundary (Appendix E). Under this protocol, MiDl stays close to the Source model, e.g., 36.33 vs. 36.06 on MM-IMDb on average (Table 1 in the main paper), whereas IM-DECOR reaches 48.70.

Following your comment, we will credit MiDl explicitly in the Introduction, and rephrase our first contribution as generalizing and systematically studying test-time missing-modality adaptation rather than introducing a brand-new problem.

AuthorsOpenReview ↗

Response to Reviewer b2L3 (2/3)

(Part 2/3)

[W3] The method relies on intermediate modality representations and feature injection interfaces of pre-trained models, which limits the claimed model-agnostic and plug-and-play capabilities.

We apologize for the lack of clarity, and we clarify the scope of this claim below.

  1. Model-agnostic refers to redesign and retraining, not to black-box access. We use the term to denote independence from model-specific architectural redesign or retraining (footnote 1 in the main paper). As stated in Appendix H.1, IM-DECOR does require a generic encode-then-fuse interface that exposes the modality-specific latent representations and allows decorated ones to be injected before fusion. Within this interface, it modifies neither the architecture nor the parameters of the source model.

  2. This access is lighter than what TTA methods commonly use. We examine the access used by the seven TTA baselines (Appendix F.3). All of them update or insert parameters inside the source model, such as normalization layers, attention layers, or adapters. In contrast, IM-DECOR only reads and writes latent representations, keeps the source model frozen, and bypasses the modality-specific encoders during backpropagation.

  3. The interface is shared by diverse architectures. We evaluate IM-DECOR on three source-model families, namely CLIP encoders with concatenation fusion, CAV-MAE branches with unified transformer blocks, and MLP encoders with a Transformer fusion network over three modalities (Appendix F.2). Moreover, we evaluate it on the early-fusion ViLT, which has almost no modality-specific encoders. There, we simply treat the first half of the fusion layers as the encoders, and IM-DECOR still achieves the best performance (Figure 9 in the appendix).

[W4] The experimental comparisons mainly focus on general TTA methods, while comparisons with missing modality completion approaches are limited, making it difficult to fully establish the advantages of the proposed mechanisms.

We sincerely thank you for this valuable suggestion, and we address it below.

  1. Completion approaches need to be adapted to MML-TTA. Established MML methods rely on labeled source data, dedicated training, and customized architectures, none of which is available when the source model is frozen and only the unlabeled test stream is accessible. This is why TTA methods form our main baselines. We therefore adapt one representative of each MML family, namely imputation (AVG), reconstruction (RECON), and alignment (ALIGN). As shown in Figure 10 in the appendix, all three remain below IM-DECOR and even underperform the Source model in some cases, mainly because reconstruction and alignment have to be learned online from the few unlabeled modality-complete samples in the stream.

  2. Knowledge Bridger is the closest training-free completion approach. To further address your concern, we add Knowledge Bridger (CVPR 2025), which completes the missing modality at the raw-data level with a large multimodal model and requires no training. We apply it to each incomplete sample and feed the completed input into the same frozen source model. We compare on the two image–text benchmarks under the 70% missing rate.

    MethodMM-IMDb TMM-IMDb IFood101 TFood101 I
    Source39.2831.7976.6983.89
    Knowledge Bridger43.7540.3478.0185.82
    IM-DECOR48.2248.8879.3287.76

    As shown, IM-DECOR outperforms Knowledge Bridger in all four settings, e.g., 48.22 vs. 43.75 on MM-IMDb with text missing. This suggests that completing the raw modality does not by itself match what the frozen source model expects, whereas IM-DECOR decorates directly in its latent space.

    Moreover, decoration in the latent space applies directly to audio, video, and pre-extracted features, as in Kinetics50, VGGSound, IEMOCAP, and MOSI, where raw-level generation is not directly applicable.

  3. IM-DECOR also outperforms a supervised alternative. We further compare with unimodal models that are trained with labels for each available modality (Table 11 in the appendix). IM-DECOR still outperforms them, e.g., 48.22 vs. 45.48 on MM-IMDb with text missing, although it needs no additional supervised training.

We will add the comparison with Knowledge Bridger to Appendix G.6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer b2L3 (3/3)

(Part 3/3)

[W5] The theoretical analysis largely builds upon existing domain adaptation error bounds and does not sufficiently establish the connection between the proposed global and local decoration mechanisms and the observed performance improvements.

We thank you for this insightful comment. We agree that Lemma 1 is an existing bound, which we use as the starting point. The purpose of Appendix D is to explain why decorating the inputs reduces the target risk under MML-TTA, and we clarify the connection to the two decorators below.

  1. The coupling in Lemma 2 is specific to MML-TTA. We observe that a modality-missing sample and its complete counterpart are two views of the same underlying instance, which offers a natural coupling between the source and target representations. With this coupling, the Wasserstein term reduces to the expected per-sample distance between the fused representation of the complete input and that of the decorated input (Proposition 1). Therefore, the target risk decreases as the decorated input mimics the complete one, which is the objective of IM-DECOR.

  2. The two decorators control two parts of this distance. To make the role of each decorator explicit, we further decompose the per-sample distance. Assume that HθH_\theta is KH\mathcal{K}_ H-Lipschitz in the decorated input, and let gk∗(1)\mathbf{g}^{(1)}_ {k^{\ast}} be the prototype nearest to the true representation z(1)\mathbf{z}^{(1)}. The triangle inequality then gives

    ∥Hθ(z(1),z(2))−Hθ(l^(1),z(2))∥2≤KH(∥z(1)−gk∗(1)∥2+∥gk∗(1)−l^(1)∥2).\big\Vert H_\theta(\mathbf{z}^{(1)}, \mathbf{z}^{(2)}) - H_\theta(\hat{\mathbf{l}}^{(1)}, \mathbf{z}^{(2)})\big\Vert_2 \le \mathcal{K}_ H \Big( \big\Vert\mathbf{z}^{(1)} - \mathbf{g}^{(1)}_ {k^{\ast}}\big\Vert_2 + \big\Vert\mathbf{g}^{(1)}_ {k^{\ast}} - \hat{\mathbf{l}}^{(1)}\big\Vert_2 \Big).

    The first term is the quantization error of the decoration container, which the K-means objective of G-DECOR minimizes (Eq. (3)) and the online update keeps refining. The second term measures whether the decoration retrieved from the remaining modality reaches the right region of the container, which L-DECOR learns and LIMDRL_\text{IMDR} supervises on relatively complete samples.

  3. The empirical results agree with this decomposition. We examine the two terms through the ablation study (Table 3 in the main paper). On MM-IMDb with text missing, removing G-DECOR lowers the result from 48.22 to 40.60, and removing L-DECOR lowers it to 45.10. Moreover, IM-DECOR reduces the MMD between the source and target representations far more than MiDl and DASP (Figure 8 in the appendix).

We will add this decomposition as a remark after Proposition 1 in Appendix D of the revised manuscript.

[W6] (1) The method assumes that each modality remains available at least occasionally in the test data stream. (2) It does not sufficiently consider scenarios where modality missingness and semantic distribution shifts occur simultaneously, limiting its practical applicability.

We thank you for these two points and address them below.

  1. Stream-level modality coverage is a mild assumption. It requires neither modality-complete samples nor the presence of every modality in every batch (Appendix H.4). Once a modality has been observed, the decoration container retains it, so IM-DECOR stays robust when the modality is entirely missing over contiguous segments or when no complete sample exists (Figure 4 in the main paper). When a modality never appears, no purely test-time method can characterize it without additional information. For this boundary case, IM-DECOR can use lightweight modality-wise source prototypes, following prior TTA works that allow minimal source-domain statistics. With a modality missing at η=100\eta = 100% throughout the entire stream, it still clearly outperforms the Source model and DASP (Figure 7 in the appendix).

  2. Simultaneous semantic shifts are a natural extension. In this work, we focus on the structural shift induced by modality missing, which aligns with the standard setting in multimodal learning, and we state this scope in the assumption of Lemma 2 and in Appendix I.1. We verify that this shift dominates in our benchmarks. Specifically, the MMD between the training set and the modality-missing test sets is 0.19–0.46 on MM-IMDb, several orders of magnitude larger than the 0.00016 between the original training and test sets (Table 10 in the appendix). Moreover, IM-DECOR adapts the incomplete inputs and keeps the source model frozen, whereas conventional TTA methods adapt the model to shifted inputs (Appendix H.2). The two directions are therefore complementary, and we leave their combination under compound shifts as future work.

We will expand this discussion in Appendix I.1 of the revised manuscript.

We sincerely appreciate your suggestions, which have helped us strengthen this work. We hope that our responses address your concerns, and we would be glad to continue the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.