- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies TTA method to adapt pretrained multimodal models to handle conditions with missing modalities. The name of the method is IM-DECOR, which reconstructs missing modality representations using online prototype containers and instance-conditioned retrieval. The method is evaluated extensively across multiple multimodal benchmarks and missing-modality settings.
Critical strengths
- Clear motivation and paper is easy to follow.
- The paper evaluates the method on six benchmarks with different settings.
- IM-DECOR generally outperforms all selected TTA baselines.
Critical weaknesses
-
The core method is largely a combination of standard components. It consists of K-means prototypes, an online feature memory, MLP-based query projection, cosine-similarity retrieval, weighted prototype aggregation, and entropy minimization. While the overall design is reasonable and empirically effective, I find the technical contribution fairly incremental. Most of these components have already appeared in prior work, and the proposed framework feels more like a combination of existing techniques in an “A + B + C” manner than a genuinely new methodological contribution. Overall, I view this as a successful engineering pipeline rather than a method with substantial technical novelty.
-
MML-TTA itself does not appear to be a new research problem. Missing-modality adaptation at test time has already been studied in some prior works (e.g., MiDl ICLR 2025). Therefore, I do not think formulating MML-TTA should be presented as a major contribution.
-
The terminology around “decoration” is somewhat unconventional. “Decoration” is not a standard term in the missing-modality literature, while the proposed operation is conceptually much closer to feature imputation or missing-modality-representation completion. Introducing new terminology does not by itself constitute novelty, and in this case it arguably makes the method less transparent.
-
No systematic ablation study was conducted regarding batch composition or "small-batch stability." In the main experiments, the batch size was set to 128 for MM-IMDb, Food101, IEMOCAP, and MOSI, and to 32 for Kinetics50 and VGGSound. These are relatively large batch sizes, particularly for a testing environment. Therefore, I suggest adding an analysis of prototype stability under varying batch sizes, different modality sample proportions, and class compositions.
-
The method relies on a fairly strong stream-level modality coverage assumption. Each modality must appear at least occasionally in the test stream so that its prototype container can be initialized and updated. This raises concerns in persistent cold-start scenarios where a modality is unavailable for an extended period or never appears at all. The authors appear to have partially validated this in the context of "Robustness to Extreme Missing Scenarios". However, this seems insufficient, as they did not verify the specific case where the text is completely absent from the very beginning of deployment. Furthermore, batch sizes of 128 or 32 are relatively large. In real-world deployment, inference often involves single samples or sparse requests, such as a medical diagnosis, a multimodal query, or sporadic event detection, where there are insufficient historical test samples to construct stable prototypes.
-
Moreover, the "stream" used in the experiments likely consists merely of the test set fed in sequentially by batch. This differs significantly from real-world continuous temporal streams, user streams, or sensor streams, and also differs from the video-based stream definition used in the MiDl baseline paper. In short, the authors' assumption that deployment requires a dense stream is not entirely reasonable, which undermines the claim that the model is "model-agnostic." (what about the sparse query medical multimodal model) Additionally, the authors assume the existence of a stream with a continuous influx of diverse samples for online updates. However, a key challenge with online updates is addressing long-term distribution drift and error accumulation. How do the authors handle this?
Questions for authors
See above.
4 comments
Response to Reviewer Bo86 (2/4)
(Part 2/4)
[W2] MML-TTA does not appear to be a new research problem. Missing-modality adaptation at test time has already been studied in prior works (e.g., MiDl, ICLR 2025). Therefore, formulating MML-TTA should not be presented as a major contribution.
We thank you for this important point, and we agree that MiDl is a pioneering work on addressing missing modalities at test time. We discuss it in Section 2 and compare with it throughout our main experiments, and we apologize that our wording did not give it sufficient credit. What MML-TTA contributes is a more general formulation and a systematic study of this problem, and we clarify the differences below.
-
MML-TTA is not tied to a task or an architecture. MiDl studies the downstream task of egocentric video understanding. In contrast, we formulate MML-TTA for a generic source model (Section 3), and instantiate it on six benchmarks that cover text, image, audio, and video, dual- and tri-modality compositions, and four source-model architectures.
-
MML-TTA does not require modality-complete samples. The adaptation of MiDl relies on modality-complete samples in the test stream. In contrast, MML-TTA only assumes stream-level modality coverage, which also admits streams without any complete sample. We evaluate this complete-sample-free scenario in Figure 4(c)–(d) in the main paper, where IM-DECOR remains effective across all image-to-text missing ratios.
-
MML-TTA comes with a systematic evaluation protocol. We examine three missing rates (50%, 70%, and 90%), single- and multi-modality missing types, missing patterns that change in every batch (Figure 3 in the main paper), modalities that are entirely missing over contiguous segments (Figure 4(a)–(b) in the main paper), and the cold-start boundary (Appendix E).
-
The general problem remains open for existing methods. Under this protocol, MiDl stays close to the Source model, e.g., 36.33 vs. 36.06 on MM-IMDb on average (Table 1 in the main paper), whereas IM-DECOR reaches 48.70.
Following your comment, we will credit MiDl explicitly in the Introduction, and rephrase our first contribution as generalizing and systematically studying test-time missing-modality adaptation rather than introducing a brand-new problem.
[W3] "Decoration" is not a standard term in the missing-modality literature, while the proposed operation is conceptually much closer to feature imputation or missing-modality-representation completion. Introducing new terminology does not by itself constitute novelty, and in this case it arguably makes the method less transparent.
We thank you for this helpful comment. We fully agree that a new term does not constitute novelty by itself, and we do not claim novelty from the name. We explain our considerations behind it below.
-
Decoration is driven by the frozen model rather than by fidelity to the missing modality. In the MML literature, imputation and reconstruction usually denote recovering the missing modality itself with networks trained on the source data. In contrast, our operation is optimized at test time so that the incomplete input mimics a complete one for the frozen source model. Specifically, it is updated by on the downstream prediction, and it contains learnable residual biases and that go beyond any retrieved modality content (Section 3.2).
-
Decoration is built from the test stream. The decoration of a sample aggregates prototypes accumulated from other test samples, conditioned on its remaining modalities, rather than being produced by a trained cross-modal generator.
-
The term conveys input-side adaptation. We use "decorate" to stress that the source model is wrapped rather than modified, which is the key property under MML-TTA.
Nevertheless, we agree that the link to the existing literature should be transparent.
Response to Reviewer Bo86 (3/4)
(Part 3/4)
[W4] No systematic ablation study was conducted regarding batch composition or "small-batch stability". The main experiments use a batch size of 128 (MM-IMDb, Food101, IEMOCAP, MOSI) or 32 (Kinetics50, VGGSound), which are relatively large for a testing environment. I suggest adding an analysis of prototype stability under (1) varying batch sizes, (2) different modality sample proportions, and (3) class compositions.
We sincerely thank you for this constructive suggestion. We agree that the stability of the decoration containers should be examined beyond the default batch sizes, and we address the three factors below.
-
Batch size. We first clarify how the containers depend on the batch. The online update of G-DECOR (Eq. (4)) is a per-sample running mean with a counter, so the prototypes are driven by all samples accumulated over the stream rather than by any single batch. Moreover, when a batch holds fewer than samples of a modality, e.g., under an extremely small batch size, IM-DECOR temporarily sets to this number and reinitializes the container once more samples arrive (Appendix F.4). To verify this, we vary the online batch size from 128 down to 1 on MM-IMDb with text missing at the 70% rate.
Method 1 8 32 128 (paper) Source 39.28 39.28 39.28 39.28 DASP 26.78 37.09 42.62 41.09 IM-DECOR 45.91 47.57 47.87 48.22 As shown, IM-DECOR remains stable across batch sizes. Even with batch size 1, it reaches 45.91, well above the Source model (39.28) and DASP (26.78). Therefore, the decoration containers do not depend on large batches.
-
Modality sample proportions. We already vary the proportions along three axes in the paper. Specifically, we evaluate missing rates of 50%, 70%, and 90% on all six benchmarks (Tables 1 and 2 in the main paper, Tables 6 and 7 in the appendix), image-to-text missing ratios from to without any modality-complete sample (Figure 4(c)–(d) in the main paper), and missing types that change in every incoming batch (Figure 3 in the main paper). IM-DECOR outperforms all baselines in each case. For instance, at the 90% missing rate, where only about one tenth of the samples in a batch still carry the affected modality, IM-DECOR improves the Source model from 26.67 to 44.92 on MM-IMDb on average.
-
Class composition. We further evaluate a class-sorted stream on Food101 with text missing at the 70% rate, where the samples arrive class by class and each batch therefore covers only one or a few classes.
Method Paper stream Class-sorted stream Source 76.69 76.69 DASP 77.62 77.38 IM-DECOR 79.32 78.67 As shown, IM-DECOR still outperforms the Source model and DASP on the class-sorted stream, and stays within 0.65 points of its result on the paper stream. Therefore, the prototypes remain stable when each batch covers only a few classes.
Moreover, as discussed in Appendix I.2, G-DECOR can incorporate techniques such as pseudo-labeling-based filtering to further stabilize the prototypes under highly imbalanced or temporally correlated streams.
We will add both studies to Appendix G of the revised manuscript.
Response to Reviewer Bo86 (4/4)
(Part 4/4)
[W5] The method relies on a fairly strong stream-level modality coverage assumption. (1) This raises concerns in persistent cold-start scenarios where a modality is unavailable for an extended period or never appears, and the case where the text is completely absent from the very beginning of deployment is not verified. (2) Real-world inference often involves single samples or sparse requests rather than batches of 128 or 32, with insufficient historical test samples to construct stable prototypes.
We thank you for this thoughtful concern. We first clarify that stream-level modality coverage requires neither modality-complete samples nor the presence of every modality in every batch (Appendix H.4). Once a modality has been observed, the container retains it over later segments where it is missing (Figure 4(a)–(b) in the main paper). We address the remaining cases below.
-
Cold start. We apologize for the lack of clarity, since this case is studied in Appendix E rather than in the main paper.
- Absent throughout the stream. We evaluate the most extreme form on MM-IMDb and Food101, where the missing rate of a modality stays at throughout the entire stream (Figure 7 in the appendix). Following prior TTA works that allow minimal source-domain statistics, IM-DECOR uses lightweight modality-wise source prototypes, without accessing any training sample. With text absent throughout, it improves the Source model from about 27 to about 42 on MM-IMDb, whereas DASP stays close to the Source model.
- Absent for an extended period. Without source statistics, IM-DECOR behaves like a standard TTA method until the modality first appears, and launches decoration from then on (Appendix E). To verify this, we keep text absent during the first 40% of the stream on MM-IMDb. Once text appears, IM-DECOR quickly reaches 48 on the remaining stream, compared with 39 for the Source model.
-
Single samples and sparse requests. The decoration container is a persistent state carried across requests. A request is therefore decorated from the prototypes accumulated over earlier requests, rather than from the samples that share its batch, and only prototypes per modality need to be stored. The single-sample case is covered by batch size 1 in our response to [W4], where IM-DECOR still improves the Source model from 39 to 46.
[W6] (1) The "stream" is likely just the test set fed in sequentially by batch, which differs from real-world temporal, user, or sensor streams and from the video-based stream in the MiDl paper. (2) Assuming a dense stream is not entirely reasonable, which undermines the "model-agnostic" claim (what about a sparse-query medical multimodal model?). (3) How do the authors handle long-term distribution drift and error accumulation under online updates?
We address these questions below.
-
Stream protocol. We follow the practical online TTA setting (Wang et al., 2021), where unlabeled test data arrive sequentially in batches, and all baselines, including MiDl, are evaluated on the same streams. Beyond this standard protocol, we stress the stream with missing types that change in every batch (Figure 3 in the main paper) and with contiguous segments where a modality is entirely missing (Figure 4(a)–(b) in the main paper). In addition, the class-sorted stream in our response to [W4] examines temporally correlated arrivals.
-
Dense streams and the model-agnostic claim. Model-agnostic refers to the source model, i.e., independence from model-specific redesign or retraining (Appendix H.1), whereas the stream is a property of the test data. As explained in [W5], IM-DECOR does not require a dense stream. For a sparse-query medical model, the container is accumulated from the modalities observed in earlier queries, or initialized from lightweight source prototypes when available (Appendix E).
-
Long-term drift and error accumulation. IM-DECOR limits error accumulation by design.
- Frozen source model. The source model is never updated, so no error accumulates in its parameters, and modality-complete samples are always predicted from their original modalities without decoration (Section 3.3).
- Prediction-free containers. The prototypes are running means of the frozen encoders' outputs on observed modalities and involve no pseudo-labels, so prediction errors are never written back into them.
- Bias reset. The sample-specific biases are reset after each adaptation batch to mitigate parameter drift (Section 3.3).
Empirically, IM-DECOR stays above the Source model and MiDl along the whole stream, including the 439 online batches of VGGSound (Figure 3 in the main paper). Semantic drift on top of modality missing forms a compound shift, which we discuss as future work in Appendix I.1.
We would be most grateful if you might consider reflecting this in your rating based on clarifications and more empirical results.
Response to Reviewer Bo86 (1/4)
We sincerely thank you for the thoughtful comments and constructive suggestions. We are particularly encouraged by your recognition of (1) the clear motivation and presentation of the paper, (2) the evaluation on six benchmarks under different settings, and (3) the improvements of IM-DECOR over all selected TTA baselines. Below, we carefully address your remaining concerns.
We sincerely thank you for this candid assessment, and for acknowledging that the design is reasonable and empirically effective. We agree that the building blocks of IM-DECOR are deliberately simple. Our technical contribution lies in where and how the adaptation is performed under MML-TTA, and we elaborate on our considerations below.
Adapting the incomplete inputs, rather than the model, is the new mechanism. We examine how the shift in MML-TTA differs from those in conventional TTA (Appendix H.2). Existing unimodal and multimodal TTA methods adapt the model to a shifted representation that still exists, i.e., z(m)→z~(m). In contrast, modality missing removes the representation entirely, i.e., z(m)→∅, so there is no representation left to adapt. Therefore, IM-DECOR treats the latent incomplete inputs as a general adaptation anchor and decorates them to mimic complete ones, while the source model stays frozen. This also explains why the seven TTA baselines bring limited or even negative gains over the Source model (Tables 1 and 2 in the main paper).
The gain comes from decoration, not from the standard components. We design two variants that separate decoration from generic entropy minimization (Table 9 in the appendix). On MM-IMDb with text missing, EM w/o Decoration falls below the Source model (36.35 vs. 39.28), whereas Decoration w/o EM already reaches 46.92 and the full IM-DECOR reaches 48.22. Moreover, we adapt imputation, reconstruction, and alignment to MML-TTA (AVG, RECON, and ALIGN, Figure 10 in the appendix), and all three remain below IM-DECOR and even underperform the Source model in some cases.
Each component answers a constraint of MML-TTA. Under MML-TTA, the test stream is unlabeled, the source data are inaccessible, and each batch offers only a few samples. Accordingly, (i) G-DECOR keeps only K=4 prototypes per modality, which need no training and are accumulated online, so that decoration remains available even when a modality is entirely missing over consecutive batches (Figure 4 in the main paper); (ii) L-DECOR turns the remaining modalities into learnable queries, so that the decoration is instance-specific rather than a shared average, and removing it lowers the result from 48.22 to 45.10 (Table 3 in the main paper); and (iii) LIMDR draws self-supervision from relatively complete samples without any label.
The minimalist design keeps the method practical. Since backpropagation bypasses the modality-specific encoders and only the decoration parameters are updated, IM-DECOR has a modest runtime and one of the smallest peak GPU memory footprints among all methods (Figure 6 in the appendix). Moreover, it applies to six benchmarks and four source-model architectures without modifying the source models.
Overall, IM-DECOR achieves an average relative improvement of 9.86% over the strongest baseline.