acceptodds
Under review as a conference paper at ICLR 2027

Understanding Missing-Modality Robustness through Target-Relevant Redundancy

Abstract

Multimodal models often suffer substantial performance degradation when partial modalities become unavailable. Existing approaches primarily focus on improving performance under incomplete inputs, while leaving unclear what information property makes multimodal model inherently robust to modality incompleteness. In this work, we study missing-modality robustness from the perspective of cross-modal redundancy. We formulate cross-modal redundancy as the target-relevant information shared between modalities, and establish its theoretical connection to robustness under modality incompleteness. Our analysis shows that larger cross-modal redundancy leads to a more favorable bound on the average prediction error when either modality is unavailable. Guided by this analysis, we develop a Redundancy-Oriented Multimodal learning (ROM) framework. Specifically, we introduce two complementary objectives that strengthen target-mediated cross-modal dependence while suppressing residual class-conditional dependence, thereby encouraging the model to obtain more target-relevant redundancy. Extensive experiments across different modality availability settings demonstrate improved robustness to missing modalities. Further analyses reveal a consistent correspondence between increased target-relevant redundancy and improved missing-modality performance, supporting our theoretical predictions. Together, our results establish target-relevant cross-modal redundancy as a principled information perspective for understanding and improving multimodal robustness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Redundancy, Missing Modality Robustness, Multimodal Information Theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uaSpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that target-relevant cross-modal redundancy explains why multimodal models stay robust when a modality is missing. It defines redundancy as RX=I(X1;X2)−I(X1;X2∣Y)R_X = I(X1;X2) − I(X1;X2|Y), derives upper and lower bounds on the average error when one modality is missing, and proposes ROM.

ROM has two losses: L_LMR (cross-modal contrastive loss) and L_CCD (within-class cross-covariance penalty)

Experiments were ran on Food-101, CUB-200 and HatefulMemes, with ViLT and CLIP, and shows gains

Critical strengths
  • The question of what information makes representations robust to missing modalities is important.
  • The method is simple, cheap, and works with both single-stream and dual-stream models.
  • Gains are large in several mismatched train/test settings.
  • The ablation is informative and honestly reports where L_CCD does not help.
Critical weaknesses
  • W1. The central question is not fully answered. The introduction asks what property makes representations robust. The paper proposes redundancy, but the evidence is mainly a correlation, and the theory (see W2) points to a simpler property: how well each modality predicts the label on its own. No experiment separates redundancy from unimodal predictive strength.

  • W2. The link between the bound and redundancy is unclear. Φ_X is defined as the average of H(Y|X1) and H(Y|X2), so the bound depends only on how informative each modality is alone. Redundancy appears only through an identity, and only when the joint information I(X1,X2;Y) is held fixed. For example, two datasets with the same unimodal informativeness but different redundancy give the same bound. Also, R_X is the co-information, which equals redundancy minus synergy. R_X can therefore increase because synergy decreases, and it can be negative on synergy-heavy tasks such as HatefulMemes. Synergy is never discussed. Meanwhile the experiments measure redundancy with a different, PID-based measure (Redundancy Bottleneck), so the paper defines one quantity and validates with another.

  • W3. The bound has limited practical value. The lower bound is meaningful only when a modality is weak; for a reasonably good modality it becomes negative and says nothing. The upper bound is very loose. The bound is never computed on any dataset. It describes an ideal single-modality classifier, not the trained multimodal model with a missing input. It depends on R_X, which training cannot change; no result connects the learned R_θ to the model's error.

  • W4. Novelty is overstated. Robult (Nguyen et al., IJCAI 2025) already targets redundancy for missing-modality robustness, using a label-aware contrastive loss. It is cited but not compared or discussed in depth. UME-MMA (arXiv 2310.06383) already gives an information-theoretic analysis of missing modalities. FactorCL (Liang et al., NeurIPS 2023) uses the same decomposition I(X1;X2) − I(X1;X2|Y) with a similar two-term strategy. It is not a missing-modality method, but it should be cited and the difference stated. L_LMR has the standard form of a cross-modal supervised contrastive loss. The bounds follow standard Fano and Feder–Merhav arguments. The genuinely new part seems to be L_CCD, which has mixed results (it hurts on CUB)

  • W5. Baselines are incomplete. Robult should be compared directly. It reports on overlapping benchmarks. A unimodal-ensemble baseline (e.g. UME-MMA in the CLIP setting) and simple controls (modality dropout, per-modality auxiliary cross-entropy) are needed. If these match ROM, the gains come from unimodal strength, not redundancy. SimMLM and gradient-guided modality decoupling are cited as key prior approaches but not evaluated. MM-IMDb, standard in the MAP/MSP line, is missing.

  • W6. The redundancy measurement needs clarification. How is the Redundancy Bottleneck estimated on high-dimensional embeddings? Since redundancy is bounded by unimodal informativeness, the r = 0.97 correlation may largely reflect unimodal accuracy. Please show the correlation after controlling for it. The y-axis of Figure 4 (right) is undefined, and the redundancy values in Figure 1 (0.27 vs 0.50) look inconsistent with Figure 4 (0 to 5 bits).

Questions for authors

I am open to update my score provided authors could answer the following questions

  • Does ROM outperform per-modality auxiliary cross-entropy, modality dropout, and a unimodal ensemble under the same backbone and budget?
  • How does ROM compare with Robult, and what is the technical difference between L_LMR and Robult's contrastive loss?
  • How do you reconcile R_X (redundancy minus synergy) with the PID redundancy measure used in the experiments? Can ROM reduce useful synergy?
  • What are the numerical bound values on Food-101 and CUB-200?
  • Which parameters are trained for each method in the ViLT setting?
  • How is L_CCD computed for classes with only one or two samples in a batch (about 2.5 samples per class on CUB with batch size 512)?
Reviewer_b2Z5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies missing-modality robustness through target-relevant cross-modal redundancy. It derives Fano-style and MAP error bounds connecting this quantity to average prediction error when either modality is missing. Based on this analysis, ROM combines same-label cross-modal contrastive learning (LMR) with class-conditional decorrelation (CCD). Experiments on FOOD-101, CUB-200, and HatefulMemes use ViLT and CLIP under several train/test missingness patterns. ROM generally improves missing-modality accuracy, and the reported redundancy proxy is strongly associated with robustness.

Critical strengths
  • The paper addresses an important practical problem and attempts to explain the source of robustness rather than merely adding an empirical module.
  • It clearly distinguishes the data-level quantity R_X from the representation-level quantity R_theta and provides a complete derivation of the error bounds.
  • The evaluation covers ViLT and CLIP, complete and incomplete training, and matched and mismatched missingness distributions.
  • The main improvements are large. Table 2 reports both a redundancy proxy and robustness, and transparently shows that CCD alone reduces average performance on CUB-200.
  • The HatefulMemes appendix reports mixed or negative outcomes rather than hiding them, and the overall presentation is relatively restrained.
Critical weaknesses
  1. The definition of "redundancy" is not generally valid. R_X is co-information, not PID redundancy in general. The paper does not state assumptions that rule out synergy, guarantee nonnegativity, or make these quantities equal.
  2. The theory, method, and empirical metric do not form a closed loop. The theory uses R_X/R_theta, while Figure 4 and Table 2 use a different Redundancy Bottleneck metric. LMR and CCD are also not shown to increase R_theta.
  3. The theoretical result has limited explanatory power for the trained model. The upper bound assumes MAP/Bayes decoding, whereas the finite-capacity ROM classifier is not a MAP decoder. Holding joint predictive information fixed and increasing R is algebraically equivalent to increasing the sum of the two unimodal mutual informations, rather than providing a nontrivial guarantee for ROM.

Besides, I have these small concerns regrading this paper.

  • The paper does not explain how Z1 and Z2 are defined in ViLT, or how LMR/CCD are computed when one modality is missing during training.
  • The CCD approximation assumes class-conditional Gaussianity, weak correlation, and approximate whitening, but the implementation only applies L2 normalization. With 101/200 classes and batch size 512, some classes may have very few samples; handling of Nc=1 and covariance stability is not specified.
  • The r=0.97 result in Figure 4 pools architectures, methods, and training conditions. The points are not independent, no confidence interval is provided, and the redundancy proxy itself uses target information.
  • The paper reports no multi-seed means, standard deviations, or significance analyses. The joint missing-modality sampling protocol is also underspecified.
Questions for authors

Can you prove that R_X equals a well-defined PID redundancy under explicit assumptions, rather than redundancy minus synergy? What is the quantitative relationship between the Redundancy Bottleneck metric and R_theta in Equation (3)? How are modality-specific representations obtained from ViLT, and how is CCD handled when a class appears only once in a batch? Can you report means and variances over at least 3-5 seeds and compute the redundancy-robustness correlation separately by dataset and architecture?

Reviewer_CkMAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies missing-modality robustness through target-relevant cross-modal redundancy, defined as RX=I(X1;X2)−I(X1;X2∣Y)R_X=I(X_1;X_2)-I(X_1;X_2\mid Y). It derives information-theoretic bounds on the average prediction error when either modality is missing and proposes Redundancy-Oriented Multimodal learning (ROM), combining label-consistent cross-modal contrastive learning with class-conditional decorrelation. Experiments on Food-101, CUB-200, and HatefulMemes evaluate ViLT and CLIP-based models under different training and inference modality-availability patterns. ROM improves robustness in several settings, and the authors report a correlation between a redundancy-bottleneck proxy and missing-modality accuracy.

Critical strengths
  1. Clear motivation and formulation. The paper distinguishes redundancy intrinsic to the data distribution from redundancy captured in learned representations, and uses this distinction to motivate ROM.
  2. Evaluation of varied modality-availability patterns. The experiments consider both complete and incomplete training inputs, including mismatched training and inference conditions, and evaluate single-stream and dual-stream models. ROM achieves substantial improvements in several settings.
  3. Informative ablations and discussion of limitations. Table 2 separates the two training objectives, showing that their contributions vary by dataset and that decorrelation alone can hurt performance.
Critical weaknesses
  1. Overstated novelty and insufficient differentiation from prior work. The abstract and §2.2 present the theoretical connection between target-relevant redundancy and missing-modality robustness as a major new contribution. However, prior work has studied task-relevant multimodal redundancy theoretically [1–3] and explored related information-theoretic or redundancy-based approaches to missing-modality robustness [4, 5]. Moreover, the quantity in Eq. (2) satisfies ΦX=[H(Y∣X1)+H(Y∣X2)]/2\Phi_X=[H(Y\mid X_1)+H(Y\mid X_2)]/2, so the redundancy-based error bound is a reformulation of a conditional-entropy bound. Although this interpretation may be useful, its additional theoretical insight is unclear. ROM employs different training objectives, but its advantages over relevant prior approaches are not established through direct comparisons.
  2. Limited empirical validation of the theoretical formulation. The paper defines target-relevant redundancy using signed co-information in Eqs. (1) and (3), but evaluates redundancy using a different PID-based measure. While using a proxy is reasonable given the difficulty of estimating mutual information, the relationship between these quantities is not sufficiently discussed. Consequently, the observed correlation between redundancy and missing-modality robustness supports the general motivation, but does not directly validate the specific theoretical formulation (§3, §5.3).
  3. Incomplete specification of training with missing modalities. The experiments include partially observed training examples, but Eqs. (6)–(7) and (10)–(11) require cross-modal representations, and the paper does not clearly explain how these objectives are computed when a modality is absent. The class-conditional covariance estimate also contains 1/(Nc−1)1/(N_c-1) without specifying how classes with fewer than two examples in a minibatch are handled. These implementation details matter for reproducibility, particularly on CUB-200 (§4–5; Appendix A.1).

[1] M. Federici, A. Dutta, P. Forré, N. Kushman, and Z. Akata. Learning Robust Representations via Multi-View Information Bottleneck. ICLR, 2020.

[2] C. Tosh, A. Krishnamurthy, and D. Hsu. Contrastive Learning, Multi-View Redundancy, and Linear Models. ALT, 2021.

[3] P. P. Liang et al. Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework. NeurIPS, 2023.

[4] T. Wörtwein, N. B. Allen, J. F. Cohn, and L.-P. Morency. SMURF: Statistical Modality Uniqueness and Redundancy Factorization. ICMI, 2024.

[5] S. Li, C. Du, Y. Zhao, Y. Huang, and H. Zhao. What Makes for Robust Multi-Modal Models in the Face of Missing Modalities? arXiv:2310.06383, 2023.

Questions for authors
  1. What additional insight or guarantee does Eq. (2) establish beyond existing theoretical analyses of multi-view redundancy and missing-modality prediction [1–3, 5], given its equivalent conditional-entropy form?
  2. What are ROM's key advances over related missing-modality approaches [4, 5]? Could the authors provide matched comparisons where feasible, or explain why these would not be appropriate?
  3. Could the authors clarify the relationship between the redundancy-bottleneck proxy used in the experiments and the theoretical quantities RXR_X and RθR_\theta? In particular, to what extent does the empirical proxy support the theoretical claims, given that these quantities capture different notions of redundancy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.