Decorate Your Incomplete Multimodal Inputs: Towards Test-Time Adaptation for Missing Modalities

The real-world deployed systems often encounter the incomplete multimodal instances, which incentivize the study of Modality-Missing Learning (MML). Nevertheless, existing methods rely on customized model architectures and training strategies to handle incomplete multimodal inputs. As a result, they are less compatible with the real-world applications, where downstream models are often already designed and trained, and the training process is also often inaccessible due to the privacy issues. To address these limitations, we propose a more practical research problem, MML-TTA, which aims to improve the robustness of multimodal models, previously designed and pre-trained on modality-complete samples, to incomplete instances at test time. Accordingly, we introduce a new framework IM-DECOR. It treats latent incomplete multimodal inputs as a model-agnostic adaptation anchor, and decorates these inputs to mimic complete ones observed during training, bridging the distribution shift in MML-TTA caused by input-level structural mismatches. Specifically, IM-DECOR introduces a modality-wise Global Decorator (G-DECOR) and an instance-aware Local Decorator (L-DECOR). They progressively transform incomplete inputs from mimicking modality-level distributions of complete data to resembling instance-specific complete inputs, enabling robust, on-the-fly adaptation of diverse downstream models to test-time missing inputs. Experiments on six benchmarks under MML-TTA show the efficacy of our method, achieving an average relative improvement of 9.86% over the strongest baseline.