acceptodds
Withdrawn from ICLR 2027

Robust Multimodal Learning via Dual-Constrained Informative Latent Space

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal systems deployed in real-world scenarios often encounter both missing modalities and corrupted observations. Such inputs make it challenging for existing methods to identify reliable information for effective multimodal interaction. Moreover, noisy and missing modalities are typically handled separately, hindering unified robustness under complex multimodal conditions. To address these challenges, we propose Reliability-aware Dual-constraint Information Bottleneck (RDIB). RDIB introduces a dual-constraint information bottleneck that filters noisy information while preserving both target-modality and task-relevant information, providing an information-theoretic basis for representation reliability. Building on the informative representations, a unified reliability estimation mechanism combines bottleneck-derived reliability for observed representations with estimated reliability for reconstructed representations. We further construct reliability-aware complete and incomplete fusion streams and impose a fusion-level dual-constraint information bottleneck to learn robust multimodal representations under incomplete inputs. Extensive experiments on MOSEI, D-Vlog, LMVD and Food-101 under diverse modality conditions demonstrate that RDIB consistently achieves robust and competitive performance across multimodal learning scenarios.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Robust Multimodal Learning, Information Bottleneck, Missing Modalities

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1EmpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reliability-aware Dual-constraint Information Bottleneck (RDIB) to address missing and noisy modalities in multimodal learning. The framework combines a dual-constraint information bottleneck, unified reliability estimation, and reliability-aware dual-stream fusion to learn robust multimodal representations. Experiments on four datasets demonstrate competitive performance under different modality-missing conditions, supported by ablation studies, noise robustness evaluations, and complexity analysis.

Critical strengths

1. Well-Motivated Problem Setting. The paper addresses the practical challenges of missing and noisy modalities in multimodal learning, which are highly relevant to real-world applications. The motivation for developing a unified framework to handle both issues is well justified.

2. Clear Theoretical Motivation. The proposed DIB aims to preserve both target-modality information and task-relevant information, supported by corresponding information-theoretic derivations. This provides a clear theoretical motivation for the proposed method.

3. Comprehensive Experimental Evaluation. The authors conduct experiments on datasets covering four different tasks, along with ablation studies, noise robustness evaluations, visualization analyses, and computational complexity comparisons, providing relatively comprehensive evidence of the method's effectiveness.

Critical weaknesses

1. Limited Novelty. The core approach essentially adds target-modality reconstruction supervision to the conventional Variational Information Bottleneck (VIB) framework. Although the paper provides an information-theoretic analysis, the actual optimization objective remains a combination of KL divergence, reconstruction loss, and task loss.

2. Insufficient Evaluation of Noise Robustness. The paper emphasizes its ability to handle both missing and noisy modalities, but the main experiments primarily focus on missing-modality scenarios. Therefore, its robustness against complex real-world noise remains insufficiently validated.

3. Marginal Performance Improvements on Some Datasets. The performance gains on certain datasets are relatively limited. For example, on MOSEI, RDIB improves the average accuracy over HME by only 0.2 percentage points (78.1 vs. 77.9), while achieving exactly the same average F1 score (76.9).

4. Limited and Outdated Baseline Comparisons. The paper compares against a relatively small set of baselines, many of which are outdated. Several recent competitive methods are not included, making the claim of achieving state-of-the-art performance less convincing.

Questions for authors

1. Novelty: What is the key methodological difference between the proposed DIB and a conventional VIB framework augmented with target-modality reconstruction? Could the authors clarify which aspects constitute the main methodological innovations of this work?

2. Noise Robustness: Although the paper claims robustness to both missing and noisy modalities, the experiments primarily focus on missing modalities. What evidence supports the effectiveness of RDIB under more complex and realistic noise conditions?

3. Marginal Improvements: On the MOSEI dataset, RDIB improves the average accuracy over HME by only 0.2 percentage points, while achieving an identical average F1 score. Are these improvements statistically significant across multiple runs?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1Emp - Part 1

Q1 and W1. Novelty Beyond Conventional VIB

A: RDIB’s methodological novelty lies in how it connects representation learning, reliability estimation, and multimodal fusion to handle both noisy and missing modalities.

  • Representation Learning. A supervised VIB that reconstructs the target modality from the same latent has the same loss form as Eq. 4. DIB gives this objective an explicit information-theoretic interpretation. Under one compression penalty, the same latent is encouraged to retain information about both the target features and the label (Eq. 1). Conventional supervised VIB imposes a single retention constraint [1][2][3], while CyIN [4] applies two bottlenecks at different stages, respectively.
  • Unified Reliability. Building on HME’s use of bottleneck uncertainty, RDIB estimates token reliability with respect to the same target representation. RDIB uses the encoder’s posterior variance for observed modalities and uses the translator’s predicted variance for missing modalities. Both yield centered token-wise reliability (Eqs. 7–11). Appendix B.2 relates the translation variance to expected reconstruction error, and ablation study reports its effectiveness.
  • Fusion Mechanism. RDIB adds each token’s reliability score to the cross-modal attention logits. The case study illustrates how these scores redistribute attention (Figs. 5–6). During training, a fusion-level DIB uses the complete stream and the label to supervise the incomplete stream. This formulation can also be applied to other missing-modality frameworks that provide paired complete and incomplete representations during training.

[1] Zhuang Y, Liu M, Bai W, et al. Hyper-modality enhancement for multimodal sentiment analysis with missing modalities[J]. Advances in Neural Information Processing Systems, 2026.

[2] Lin R, He Q, Zeng Y, et al. Affection-guided bottleneck diffusion for missing modality issue in multimodal affective computing[J]. IEEE Transactions on Affective Computing, 2025.

[3] Xiao X, Liu G, Gupta G, et al. Neuro-inspired information-theoretic hierarchical perception for multimodal learning[C]//International Conference on Learning Representations. 2024.

[4] Lin R, He Q, Mai S, et al. Cyin: Cyclic informative latent space for bridging complete and incomplete multimodal learning[J]. Advances in Neural Information Processing Systems, 2026.

Q2 and W2. Robustness Under Realistic Noise

A: In real-world scenarios, noise is often complex and may involve multiple corruption types, making it difficult to comprehensively simulate all possible noise distributions. Since the primary objective of noise-robust multimodal learning is to suppress irrelevant information while preserving task-relevant representations, we initially adopted Gaussian noise as a representative corruption to evaluate model robustness.

To address the concern, we extend our experiments to non-Gaussian corruption (Salt-and-Pepper noise) under both complete and missing-modality settings. We further introduce TMDC, a recent method without an information bottleneck (IB) mechanism, as an additional baseline beyond HME. For fair comparison, all methods are evaluated under identical modality availability patterns and noise levels (0–7).

Noise TypeAvailabilityModel0.01.02.03.04.05.06.07.0
Salt-PepperBothRDIB74.8171.9168.9963.1661.5060.2159.8159.51
Salt-PepperBothHME73.2871.6262.5959.2557.6356.4456.0254.88
Salt-PepperBothTMDC72.0669.4867.7961.3557.6657.0956.4154.10
Salt-PepperAudioRDIB74.5571.8867.8964.3662.1160.0559.3058.60
Salt-PepperAudioHME72.8770.3163.6555.8054.0753.0752.6152.28
Salt-PepperAudioTMDC70.4269.4361.9757.4555.3954.3353.9352.97
Salt-PepperVisionRDIB74.3071.0968.8159.9655.0454.6254.4354.43
Salt-PepperVisionHME70.0765.2056.2553.6853.0652.9252.9852.98
Salt-PepperVisionTMDC71.0067.4764.7855.3254.2152.8252.1152.75

Due to the stronger feature perturbations introduced by Salt-and-Pepper noise, all methods exhibit noticeable performance degradation, particularly under severe corruption. Nevertheless, RDIB consistently outperforms both HME and TMDC across all noise levels and modality availability patterns. These results further demonstrate RDIB's robustness to non-Gaussian corruption, even under severe noise and missing modalities. The corresponding line plots and further analysis have been included in the Appendix of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 1Emp - Part 2

W4. Comparison with Latest Baselines

A: Thank you for the suggestion. In the main paper, we selected representative baselines covering widely adopted methods, including recent methods published in 2026. To further strengthen the comparison, we have added recent competitive methods, including CyIN (NeurIPS 2025), EBMC (CVPR 2026), TMDC (AAAI 2026), and MIDAS (TPAMI 2026) on MOSEI, as well as TMDC and MIDAS on D-Vlog and LMVD. As shown below, our method remains highly competitive in average performance. The complete results are provided in the appendix of the revised manuscript.

DatasetsModelsVenueAcc2F1
MOSEICyInNeurIPS 202575.374.8
----EBMCCVPR 202674.674.1
----TMDCAAAI 202677.477.1
----MIDASTPAMI 202676.274.4
----Ours—78.176.9

Q3 and W3. Statistical Reliability of Improvements

A: First, we would like to clarify that all RDIB (Ours) results reported in the manuscript are averaged over five independent runs with different seeds. To further assess the reliability and stability of our method, we additionally report the standard deviations in some settings across these five runs, together with those of the reproduced baselines. Some baseline results on MOSEI and Food-101 are directly adopted from their original publications (as marked in the manuscript).

ModelMOSEI Acc2 (MR = 0.7)MOSEI F1 (MR = 0.7)D-Vlog F1 (Both)LMVD F1 (Both)
HME72.469.273.28 ± 1.3872.39 ± 2.86
TMDC71.1 ± 1.2270.4 ± 0.9972.06 ± 0.9570.09 ± 1.13
MIDAS70.1 ± 0.5666.1 ± 0.8773.93 ± 1.4373.34 ± 1.24
RDIB72.6 ± 0.7369.1 ± 0.6474.81 ± 0.3975.27 ± 1.56

We acknowledge that the improvements on MOSEI and Food-101 are relatively marginal. Nevertheless, the additional results demonstrate the stability of RDIB across different seeds. RDIB achieves more substantial improvements on D-Vlog and LMVD. Together with its competitive performance on MOSEI and Food-101, these results demonstrate the effectiveness and cross-dataset applicability of RDIB across diverse multimodal learning tasks and modality-missing scenarios.

Thank you for the response and the additional Salt-and-Pepper noise and baseline experiments, which partially address my concerns about noise robustness and baseline coverage. However, the response confirms that the DIB objective has the same loss form as a supervised VIB with target-modality reconstruction, the added noise evaluation is still synthetic, and the improvements on MOSEI remain marginal, with TMDC achieving a higher F1 in the new comparison, so I maintain my rating.

AuthorsOpenReview ↗

Response to Reviewer 1Emp

Thank you for your follow-up and for acknowledging our additional experiments. We would like to further clarify the following points.

  • DIB shares the same loss form as supervised VIB with target-modality reconstruction only when the bottleneck, parameterization, and loss weights are identical. However, they differ in their underlying motivations and learning objectives. DIB explicitly aims to jointly preserve cross-modal and task-relevant information under a shared compression budget. Our theoretical analysis (Section 3.2 and Appendix B.1) characterizes the interplay between these two retention requirements and their optimization trade-offs.

  • We agree that synthetic corruption cannot fully capture complex real-world noise. Nevertheless, our experiments with Gaussian and Salt-and-Pepper noise across different corruption intensities and modality availability patterns provide consistent evidence of RDIB's robustness under the evaluated conditions. We leave broader validation under more realistic and complex noise environments to future work.

  • Our extensive experiments across four datasets, including ablation studies, visual analyses, case studies, and parameter sensitivity analyses, support both our motivation and methodological design beyond a single benchmark gain. As the ICLR reviewer guidelines state, “a lack of state-of-the-art results does not by itself constitute grounds for rejection.” We sincerely hope that our broader empirical evidence will be considered collectively when assessing the contribution of this work.

Thank you again for your careful evaluation and constructive feedback. We will continue refining the method and strengthening its evaluation.

Reviewer_QPPZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RDIB for multimodal learning with missing and noisy inputs. It combines a dual-constraint information bottleneck, reconstruction of missing representations, variance-based reliability scores, and reliability-aware fusion. The paper reports results on four datasets under missing-modality settings and tests Gaussian feature noise in the appendix.

Critical strengths
  1. Studying missing and noisy modalities together is useful. The paper gives a clear reason to consider both representation quality and fusion reliability.

  2. The method describes separate roles for its bottleneck, reconstruction, and fusion components. The ablations test the contribution of these parts.

  3. The experiments cover four datasets with different modality settings. The LMVD results show gains across all three reported availability patterns, and the supplement includes a cost analysis.

Critical weaknesses
  1. The method combines several existing ideas, but the benefit of its variance-based reliability score is not isolated. Removing reliability-aware fusion shows that the module matters, but does not show whether this score is better than a simpler quality score.

  2. The reliability mechanism may not detect a uniformly bad modality. Eq 7 centers reliability within each modality, and Eq 13 normalizes attention over tokens from that modality. If all token uncertainties rise together, the relative attention weights may stay the same.

  3. Several reported gains are small without uncertainty estimates. In Table 3, average MOSEI accuracy is 78.1 for RDIB versus 77.9 for HME, while their average F1 scores both equal 76.9. Repeated runs are needed to judge these differences.

Questions for authors

Please address the above weaknesses and answer the following questions.

  1. Do the reliability scores track measured corruption or reconstruction error? What happens to fusion when every token in one modality becomes less reliable by the same amount?

  2. Can you replace the variance-based score with a simple confidence or reconstruction-error score while keeping the fusion architecture fixed? Does RDIB still improve performance?

  3. Can you report variation over training seeds for MOSEI and Food-101? Are the small average gains stable?

  4. How does RDIB perform under non-Gaussian corruption and against more than HME in the joint missing-plus-noise setting? Please report the same missingness and noise levels for each method.

3 comments
AuthorsOpenReview ↗

Response to Reviewer QPPZ - Part 1

Thanks for your insightful comments. We believe we have resolved each of your constructive concerns with additional experiments and further explanations.

Q1, Q2 and W1. Effectiveness of Variance-Based Reliability

A: We use the posterior variance produced by the information bottleneck as an intrinsic measure of token-level reliability, inspired by HME. Unlike prediction confidence method, this score directly captures latent uncertainty without requiring additional quality estimators. To isolate its contribution, we keep the fusion architecture fixed and compare three alternative scores: uniform weights, token-level maximum softmax confidence, and reconstruction residuals. Our variance-based score consistently achieves the best performance, outperforming the strongest alternative by 0.77 points on D-VLog and 1.32 points on LMVD. These results demonstrate that the gains arise not only from reliability-aware fusion but also from the effectiveness of the proposed variance-based reliability estimation

quality scoreD-VlogLMVD
w/o reliability71.8872.08
ours74.8175.27
uniform71.2172.20
reconstruction74.0473.95
max softmax72.0272.78

Q3 and W3. Statistical Significance

A: First, we would like to clarify that all RDIB (Ours) results reported in the manuscript are averaged over five independent runs with different seeds. To further assess the reliability and stability of our method, we additionally report the standard deviations in some settings across these five runs, together with those of the reproduced baselines. Some baseline results on MOSEI and Food-101 are directly adopted from their original publications (as marked in the manuscript).

ModelMOSEI Acc2 (MR = 0.7)MOSEI F1 (MR = 0.7)D-Vlog F1 (Both)LMVD F1 (Both)
HME72.469.273.28 ± 1.3872.39 ± 2.86
TMDC71.1 ± 1.2270.4 ± 0.9972.06 ± 0.9570.09 ± 1.13
MIDAS70.1 ± 0.5666.1 ± 0.8773.93 ± 1.4373.34 ± 1.24
RDIB72.6 ± 0.7369.1 ± 0.6474.81 ± 0.3975.27 ± 1.56

Since the run-level statistics of some externally reported baselines are unavailable, we cannot establish whether these improvements on MOSEI are statistically significant across multiple runs. Nevertheless, the additional results demonstrate the stability of RDIB across different seeds. RDIB achieves more substantial improvements on D-Vlog and LMVD. Together with its competitive performance on MOSEI and Food-101, these results demonstrate the effectiveness and cross-dataset applicability of RDIB across diverse multimodal learning tasks and modality-missing scenarios.

AuthorsOpenReview ↗

Response to Reviewer QPPZ - Part 2

Q4. Non-Gaussian and Joint Corruption Robustness

A: Thank you for the suggestion. In real-world scenarios, noise can be complex and heterogeneous, making it difficult to simulate all possible corruption types. We initially adopted Gaussian noise as a representative corruption to evaluate model robustness.

To address the concern, we extend our experiments to non-Gaussian corruption (Salt-and-Pepper noise) under both complete and missing-modality settings. We further introduce TMDC, a recent method without an information bottleneck (IB) mechanism, as an additional baseline beyond HME. For fair comparison, all methods are evaluated under identical modality availability patterns and noise levels (0–7).

Noise TypeAvailabilityModel0.01.02.03.04.05.06.07.0
Salt-PepperBothRDIB74.8171.9168.9963.1661.5060.2159.8159.51
Salt-PepperBothHME73.2871.6262.5959.2557.6356.4456.0254.88
Salt-PepperBothTMDC72.0669.4867.7961.3557.6657.0956.4154.10
Salt-PepperAudioRDIB74.5571.8867.8964.3662.1160.0559.3058.60
Salt-PepperAudioHME72.8770.3163.6555.8054.0753.0752.6152.28
Salt-PepperAudioTMDC70.4269.4361.9757.4555.3954.3353.9352.97
Salt-PepperVisionRDIB74.3071.0968.8159.9655.0454.6254.4354.43
Salt-PepperVisionHME70.0765.2056.2553.6853.0652.9252.9852.98
Salt-PepperVisionTMDC71.0067.4764.7855.3254.2152.8252.1152.75

Due to the stronger feature perturbations introduced by Salt-and-Pepper noise, all methods exhibit noticeable performance degradation, particularly under severe corruption. Nevertheless, RDIB consistently outperforms both HME and TMDC across all noise levels and modality availability patterns. These results further demonstrate RDIB's robustness to non-Gaussian corruption, even under severe noise and missing modalities. The corresponding line plots and further analysis have been included in the Appendix of the revised manuscript.

Q1 and W2. Robustness to Uniform Modality Degradation

A: Uniform degradation of all tokens within a modality is an relatively uncommon scenario, as real-world corruption typically affects tokens unevenly. We examine a related setting in which only one modality is available and is subject to severe salt-and-pepper noise. Even if all tokens in the available modality receive the same reliability score, RDIB can still assign them different attention weights based on their content.

Noise TypeAvailabilityModel0.01.02.03.04.05.06.07.0
Salt-PepperAudioRDIB74.5571.8867.8964.3662.1160.0559.3058.60
Salt-PepperAudioHME72.8770.3163.6555.8054.0753.0752.6152.28
Salt-PepperAudioTMDC70.4269.4361.9757.4555.3954.3353.9352.97
Salt-PepperVisionRDIB74.3071.0968.8159.9655.0454.6254.4354.43
Salt-PepperVisionHME70.0765.2056.2553.6853.0652.9252.9852.98
Salt-PepperVisionTMDC71.0067.4764.7855.3254.2152.8252.1152.75

At the highest noise level (7.0), RDIB achieves 58.60% and 54.43% under Audio-only and Vision-only settings, outperforming the strongest baselines by 5.63 and 1.45 points, respectively. Although these experiments do not impose strictly uniform token degradation, they demonstrate RDIB's superior robustness when only one heavily corrupted modality remains available. These results are consistent with the design of RDIB’s dual-constraint information bottleneck, which is trained to suppress modality-specific noise while retaining cues that support both target-modality reconstruction and label prediction.

Thank you for comparing alternative reliability scores and adding the salt-and-pepper experiments. The new results support the variance-based score, but they leave open whether RDIB can downweight a uniformly degraded modality when a clean modality is also available. I will keep my rating at 2.

Reviewer_42AGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reliability-aware Dual-constraint Information Bottleneck (RDIB) to simultaneously handle missing modalities and noisy observations in multimodal learning. The method consists of three parts. First, a Dual-constraint Information Bottleneck (DIB) imposes both a target-modality constraint I(B_S; F_T) and a task constraint I(B_S; Y) on the same latent variable, and is trained with variational IB. Second, a unified reliability estimator uses the DIB posterior variance to characterize token-level reliability of observed modalities, and uses a heteroscedastic Gaussian translator to predict the variance of reconstructed representations for missing modalities, which is then converted into log-precision reliability. Third, reliability-aware dual-stream fusion incorporates reliability as an additive prior in cross-modal attention and uses a fusion-level DIB to regularize the incomplete stream toward the informative space of the complete stream. Experiments on MOSEI, D-Vlog, LMVD, and Food-101 cover missing and noisy conditions and report robustness and competitiveness relative to existing methods.

Critical strengths

The idea of unified reliability modeling is valuable. The paper maps both the DIB posterior variance of observed modalities and the predictive variance of missing-modality reconstruction onto the same target representation B_T, and incorporates them into attention in log-precision form, so that "observed reliability" and "reconstruction reliability" have a consistent interpretation during fusion. The conditional variance decomposition in Appendix B.2 also shows that reconstruction variance contains both target posterior uncertainty and source-conditioned recovery error, which is better grounded than simple confidence weighting.

The experiments cover multiple missing and noisy conditions, and the ablation is relatively complete. The paper evaluates random missingness on MOSEI, fixed missingness on LMVD/D-Vlog, proportional missingness on Food-101, and Gaussian noise. The ablations in Tables 4/6 show that DIB, Est, DF, and RF each contribute; in particular, Est causes the largest drop under severe missingness, indicating that reliability estimation and the reconstruction module do play a role. The computational complexity table also shows that parameter count and inference time are within an acceptable range.

Critical weaknesses

The main experimental results show only marginal improvement over strong baselines, and in some settings do not outperform HME. In Table 3, the average ACC/F1 on MOSEI is 78.1/76.9 for RDIB and 77.9/76.9 for HME, with F1 tied; at MR = 0.2 and 0.3, RDIB's F1 is 81.1 and 79.2, lower than HME's 81.9 and 79.6; at MR = 0.1, F1 is 83.7, also slightly lower than HME's 83.8. The main comparisons on D-Vlog and LMVD are not given as complete tables in the main text, and Food-101 is shown only as a bar chart without numerical values or confidence intervals. If the core claim is "consistently achieves robust and competitive performance," more complete main results and statistical significance statements are needed; otherwise, it is difficult to support a substantial advantage over SOTA.

The rigor of the theoretical analysis and implementation details is insufficient, which affects the credibility of the method. The weight notation in Equations (1) and (4) is inconsistent; E(B_T, B̂_T←S) in Equation (10) is not defined in the main text; the reliability selection in Equation (12) and the centering notation in Equation (7) need clarification; the "theoretical analysis" in Appendix B.1 does not state sufficient conditions for information retention in theorem form and is more a discussion of lower bounds and approximation gaps. In addition, using the DIB posterior variance as reliability relies on the diagonal Gaussian assumption, and the heteroscedastic variance of the translator depends on the missing pattern during training; the paper does not experimentally answer whether reliability scores are calibrated when the test missing pattern differs from training, or whether temperature scaling or a calibration set is needed. These are not fatal flaws, but they weaken the claim of an "information-theoretic basis for representation reliability."

Questions for authors

(1) In Table 3, MOSEI F1 is lower than HME at MR = 0.1/0.2/0.3, and average F1 is tied. Can the authors provide multi-seed means, standard deviations, and significance tests? Can the complete main result tables for D-Vlog and LMVD be placed in the main text or appendix with all baselines explicitly listed? Can the Food-101 bar chart be supplemented with numerical values and confidence intervals? (2) How exactly does MR = 0.7 in the missing protocol correspond to the definition of "at least one modality remains available"? The paper says "exactly one modality retained per sample," but under three modalities, if each sample retains one modality, the average MR should be 2/3. Does the actual sampling retain one modality for some samples and two for others? Please provide the distribution of the number of available modalities under each MR. (3) The weight notation in Equations (1) and (4) is inconsistent: Equation (1) uses β_I, while Equation (4) uses β_t. Please unify the notation and explain the optimal ranges of β_t and β_y. Can the "theoretical analysis" in Appendix B.1 be stated in theorem form? For example, under what conditions is the retention lower bound of DIB strictly better than that of single-constraint IB? (4) Are reliability scores calibrated when the test missing pattern differs from training? Can the authors provide correlation curves between reliability scores and true reconstruction error/task loss (e.g., error after reliability binning), and state whether temperature scaling or a calibration set is used? When the diagonal Gaussian posterior is mismatched, is the log-precision reliability still monotonic? (5) Dual-stream fusion uses the complete stream F_f as the target during training, but the complete stream is unavailable at test time. Is the input to B_f at test time only from F_est_f? Does the fusion-level DIB introduce a dependence on the complete stream at test time? Can the authors provide an ablation that removes the fusion-level DIB and keeps only dual-stream attention? (6) What is the core difference between RDIB and IB/reliability methods such as HME, CyIN, and ABDiff? HME also uses bottleneck uncertainty to characterize modality confidence; under what conditions is RDIB's "unified reliability" strictly better than HME's reliability? Can the authors provide a direct ablation against HME under the same backbone and the same preprocessing?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 42AG - Part 1

Thanks for your comments. Our responses to your concerns and mistakes are presented in lists as follows.

Q1 and W1. Performance and Statistical Reliability

A: First, we would like to clarify that all RDIB (Ours) results reported in the manuscript are averaged over five independent runs with different seeds. To further assess the reliability and stability of our method, we additionally report the standard deviations in some settings across these five runs, together with those of the reproduced baselines. Some baseline results on MOSEI and Food-101 are directly adopted from their original publications (as marked in the manuscript).

ModelMOSEI Acc2 (MR = 0.7)MOSEI F1 (MR = 0.7)D-Vlog F1 (Both)LMVD F1 (Both)
HME72.469.273.28 ± 1.3872.39 ± 2.86
TMDC71.1 ± 1.2270.4 ± 0.9972.06 ± 0.9570.09 ± 1.13
MIDAS70.1 ± 0.5666.1 ± 0.8773.93 ± 1.4373.34 ± 1.24
RDIB72.6 ± 0.7369.1 ± 0.6474.81 ± 0.3975.27 ± 1.56

Since the run-level statistics of some externally reported baselines are unavailable, we cannot establish whether these improvements on MOSEI are statistically significant across multiple runs. Nevertheless, the additional results demonstrate the stability of RDIB across different seeds. RDIB achieves more substantial improvements on D-Vlog and LMVD. Together with its competitive performance on MOSEI and Food-101, these results demonstrate the effectiveness and cross-dataset applicability of RDIB across diverse multimodal learning tasks and modality-missing scenarios.

We have supplement the complete main result tables for D-Vlog, LMVD, MOSEI and Food-101 in the appendix of revised manuscript.

Q2. Missingness Protocol Explanation

A: Following previous works [1][2], MR = 0.7 is a commonly adopted nominal setting approximating the theoretical maximum of 2/3 for three modalities. Thus, at MR = 0.7, exactly one modality is retained for every sample (100%), without mixing samples with two available modalities. For MR = 0.1–0.6, the number of available modalities varies according to random masking.

[1] Lian Z, Chen L, Sun L, et al. Gcnet: Graph completion network for incomplete multimodal learning in conversation[J]. IEEE Transactions on pattern analysis and machine intelligence, 2023, 45(7): 8419-8432. [2] Zhuang Y, Liu M, Bai W, et al. Hyper-modality enhancement for multimodal sentiment analysis with missing modalities[J]. Advances in Neural Information Processing Systems, 2026, 38: 142100-142129.

Q3. Theoretical Justification and Notation

A: [Mistake]: Equations (1) did not use β_I as weight notation in initial manuscript pdf. We have evaluated both weights over {0.01,0.1,0.5,1.0}\{0.01, 0.1, 0.5, 1.0\} (Appendix D.3, Fig. 8). Among the tested pairs, (βt,βy)=(0.5,0.5)(\beta_t,\beta_y)=(0.5,0.5) gave the highest F1 on D‑Vlog, while (0.1,1.0)(0.1,1.0) gave the highest F1 on LMVD. D‑Vlog favored equal weights, while LMVD favored a larger weight on task supervision.

[Theorem form]. Building on Appendix B.1, we compare DIB with target-only IB under the same target-modality retention requirement ata_t, and with label-only IB under the same label retention requirement aya_y. As in Proposition 1 of SIB [2], we give an explicit condition for strict improvement.

Let BS∼qθ(⋅∣FS)B_S\sim q_\theta(\cdot\mid F_S) satisfy I(BS;FT)≥atI(B_S;F_T)\ge a_t and I(BS;Y)≥ayI(B_S;Y)\ge a_y, and let c=I(FT;Y)c=I(F_T;Y). The lower bound in Eq. (20) strictly exceeds the target-only bound ata_t if and only if ay>min⁡at,ca_y>\min\\{a_t,c\\}. It strictly exceeds both single-constraint bounds if and only if min⁡at,ay>c\min\\{a_t,a_y\\}>c.

Proof. By the chain rule in Eq. (19) and nonnegativity of conditional mutual information,

max⁡at,ay,at+ay−c≤I(BS;FT,Y)≤R=I(FS;BS).\max\\{a_t,a_y,a_t+a_y-c\\}\le I(B_S;F_T,Y)\le R=I(F_S;B_S).

Comparing the left-hand side with ata_t gives the first condition. Its gain over max⁡at,ay\max\\{a_t,a_y\\} is exactly max⁡0,min⁡at,ay−c\max\\{0,\min\\{a_t,a_y\\}-c\\}, which gives the second. Using variational bounds as in CyIN [1], Eq. (24) gives the same comparison with at,aya_t,a_y replaced by ℓt,ℓy\ell_t,\ell_y.

[1] Lin R, He Q, Mai S, et al. Cyin: Cyclic informative latent space for bridging complete and incomplete multimodal learning[J]. Advances in Neural Information Processing Systems, 2026.

[2] Liu T, Yang X, Wang W, et al. Sequential Information Bottleneck Fusion: Towards Robust and Generalizable Multi-Modal Brain Tumor Segmentation[C]//The Fourteenth International Conference on Learning Representations. 2026.

AuthorsOpenReview ↗

Response to Reviewer 42AG - Part 2

Q4. Reliability Estimation and Calibration

A: We conduct additional experiments on MOSEI under matched and mismatched missing patterns. Specifically, we train models at MR = 0.3 and evaluate them under different test conditions. For reliability analysis, we use the model trained at MR = 0.3 and evaluate it at MR = 0.3, Diff. 0.3 (sample-shuffled masks), and MR = 0.6. The same five reliability bins defined under standard MR = 0.3 are used to measure reconstruction MSE against targets, with 95% bootstrap confidence intervals.

DatasetsTraining MRTest MRAcc2F1
MSOEI0.30.380.179.2
0.3Diff 0.379.778.8
0.60.673.973.3
0.30.674.472.7

The results demonstrate robustness to missing-pattern shifts. Under Diff. 0.3, both Acc2 and F1 decrease by only 0.4 points. When the test MR increases to 0.6, the model trained at MR = 0.3 achieves performance comparable to matched training at MR = 0.6.

Training MRTest MRBin 1 MSEBin 2 MSEBin 3 MSEBin 4 MSEBin 5 MSESpearman ρ
0.30.30.08610.05820.06360.06040.0609-0.30
0.3Diff. 0.30.08120.06380.06080.06590.0657-0.20
0.30.60.08730.06580.06870.06420.0637-0.90

The lowest-reliability bin consistently exhibits the highest reconstruction MSE across all three conditions. The negative Spearman correlations indicate that reliability scores retain error-discriminative ability under missing-pattern shifts, although the relationship is not strictly monotonic. We use the raw reliability scores without additional calibration. The corresponding correlation curves are provided in the Appendix of the revised manuscript.

For the concern about reliability monotonic under posterior mismatched, the log-precision score in Eq. (7), rt=−log⁡(vt+ϵ)r_t=-\log(v_t+\epsilon), is strictly decreasing in the token’s estimated mean posterior variance. Centering preserves this ordering within each modality. Even under mismatch, the score ranks tokens by their true uncertainty as long as the estimated variances preserve their relative order. Under severe mismatch, the Gaussian approximation may underestimate long-tailed uncertainty more for some tokens than others, which can change that ranking (Appendix D.5).

Q5. Training–Inference Consistency

A: At test time, the fusion encoder takes only FestfF_{est}^{f} as input to produce BfB_f (See in main text Sec. 3.4). The complete stream FfF^{f} serves only as a reconstruction target during training, so inference requires no access to it.

The requested ablation of fusion-level DIB is 'w/o DF' in main text Table 4. As described in Sec. 3.4, the fusion-level DIB uses the complete stream FfF^{f} as a training target for the incomplete stream. w/o DF removes this bottleneck while retaining the reliability-aware attention that computes FestfF_{est}^{f}. LMVD F1 drops from 71.13 to 69.32, and MOSEI ACC/F1 drops from 78.1/76.5 to 76.9/74.8. The separate w/o RF ablation tests the contribution of the token-wise reliability prior in attention.

AuthorsOpenReview ↗

Response to Reviewer 42AG - Part 3

A: RDIB’s main distinction from other methods based information bottleneck is its introduction of a dual constraint that jointly preserves information relevant to the target modality and the downstream task (Eq. 1).

  • HME motivated our use of bottleneck uncertainty. It uses posterior uncertainty to weight representations enriched with cues from other samples when fusing them with the original inputs.
  • CyIN uses token- and label-level bottlenecks with cyclic cross-modal translation to recover missing representations.
  • ABDiff uses an affect-guided bottleneck to filter noise during diffusion-based reconstruction.

In RDIB, “unified” refers to how reliability guides fusion for both observed and reconstructed modalities. RDIB computes token-wise scores for cross-modal attention from the posterior variance of the observed target latent or the predicted variance of its reconstruction (Eqs. 7, 11–13). HME uses posterior standard deviation to balance original representations with cross-sample enhancements in sentiment analysis with missing modalities. The two reliability measures operate at different stages and apply to different representations, so a strict comparison of their effectiveness is difficult.

HME could also adopt the dual constraint in its enhancement branch to preserve cross-modal cues alongside sentiment information, potentially improving fusion. To ensure a fair comparison, we conduct a direct ablation on HME using the same backbone and preprocessing. Specifically, we replace its original information bottleneck (IB) with our dual information bottleneck (DIB) or incorporate our dual-stream fusion (DF) mechanism.

Dataset (Setting)HMEw DIBw DF
LMVD (audio)62.9963.1462.02
LMVD (vision)71.2972.9972.54

Replacing IB with DIB consistently improves HME, yielding gains of 0.15 and 1.70 points under Audio-only and Vision-only settings, respectively. Incorporating DF also improves Vision-only performance by 1.25 points, although its effectiveness varies across settings. These results suggest that the proposed DIB and dual-stream fusion can also benefit existing frameworks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.