acceptodds
Under review as a conference paper at ICLR 2027

Uncertainty-Driven Trustworthy Multi-modal Information Bottleneck Clustering

17 views

Abstract

The Information Bottleneck (IB) theory seeks to extract minimal sufficient representations and filter redundant noise, providing a solid foundation for multi-modal clustering. Although traditional IB-based methods demonstrate significant advantages, they still face several challenges. These include the inability to effectively model modality uncertainty, the tendency to yield overconfident incorrect predictions in the presence of modality conflicts, and the disconnection between representation learning and the final decision-making process. To address these issues, we propose a novel clustering method, the UNcertainty-drIven Trustworthy Multi-modal Information Bottleneck (UNIT-MIB). This approach adaptively regulates compression intensity based on uncertainty estimations and employs a dual-path fusion strategy. Specifically, it integrates semantic features via uncertainty-weighted aggregation and uncertainty-aware opinions via the Dempster-Shafer theory. Furthermore, based on Deng entropy, we formulate a Deng mutual information measure to characterize the discrepancy between semantic and evidential opinions, and introduce a stop-gradient-based directed alignment mechanism to impose asymmetric optimization. This mechanism steers semantic representation learning toward trustworthy evidential decision-making, thereby bridging the semantic-logical gap. Experimental results demonstrate that the proposed UNIT-MIB significantly outperforms state-of-the-art methods in terms of clustering accuracy across multiple real-world datasets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Deep Multi-modal Clustering, Trustworthy Learning, Information Bottleneck

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_wCKLOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a novel framework for trustworthy multi-modal clustering. It integrates information bottleneck theory with evidential deep learning, which represents a fresh and reasonable direction that has not been fully explored before. The experiments are sufficient and cover multiple public multi-modal datasets. The evaluation is conducted on several benchmark datasets and compared against a broad set of recent baselines, which lends credibility to the reported results.

Critical strengths
  1. The research problem addressed in this paper has strong practical significance. Clustering multi-modal data in the presence of modality conflict is a challenging and widely encountered task, and the paper is well motivated in this respect.
  2. The modality weights in the proposed method are interpretable. Since they are derived from epistemic uncertainty, they directly reflect the reliability of each modality.
Critical weaknesses
  1. The necessity of Dempster-Shafer fusion is not justified. Why a simpler probability-level fusion would not be sufficient, and the authors should explain the advantage of this.
  2. Since pseudo-labels are generated from the model's own predictions, there is a risk that errors may propagate and accumulate during training. How is this error propagation handled?.
  3. The necessity of maintaining two fusion paths is not established. It is not clear what each path contributes to the final result and why both are required.
Questions for authors
  1. The key differences between the proposed method and existing multi-modal clustering methods are not presented clearly enough.
  2. Whether the proposed method can be transferred to other unsupervised learning tasks is not demonstrated. The evaluation is limited to multi-modal clustering, and its generalizability remains unclear.
  3. The reason for using Deng mutual information instead of conventional mutual information is not explained. The paper should clarify the effect that Deng mutual information brings in this task.
2 comments
AuthorsOpenReview ↗

Response to Reviewer wCKL

We appreciate the reviewer’s constructive comments and suggestions, which have helped us improve the clarity and presentation of UNIT-MIB.


Weakness 1

D-S fusion is adopted to preserve the complete evidential information from different modalities. Unlike probability-level fusion, which primarily combines normalized class probabilities, D-S fusion operates on subjective opinions containing both class-wise belief masses and epistemic uncertainty. Thus, it preserves uncertainty information that may be lost through conventional probability averaging. Moreover, its combination rule incorporates the conflict coefficient to account for disagreements among modalities. These properties make D-S fusion suitable for constructing a global evidential consensus from modalities with different uncertainty levels and potentially conflicting evidence.


Revision:

  1. In Section 2.5.3 (Decision-level Fusion via D-S Theory), we have clarified the motivation and advantages of D-S fusion.

Weakness 2

UNIT-MIB mitigates potential pseudo-label error propagation through confidence-based pseudo-label selection. Specifically, soft pseudo-labels are first generated from the global D-S consensus, which integrates evidential information from multiple modalities. Samples are then ranked according to their confidence scores derived from the D-S consensus, and only the pseudo-labels associated with the top-kk% most confident samples are retained for EDL supervision, while ambiguous ones are excluded. Furthermore, pseudo-labels are refreshed at the end of each epoch and remain fixed within each epoch to provide stable supervision. These mechanisms help reduce the influence of erroneous pseudo-labels and limit error accumulation during training.


Revision:

  1. In Section 2.5.2 (Cross-branch Evidential Supervision), we have clarified the pseudo-label updating strategy and the mechanisms for mitigating error propagation.

Weakness 3

The two fusion paths serve complementary roles. The feature-level path performs uncertainty-weighted semantic aggregation, while the decision-level path establishes an evidential consensus through D-S fusion and provides guidance via Deng-MI-based alignment. To validate their respective contributions, we conducted additional path-level ablation experiments. The results show that removing either path leads to decreases in both ACC and NMI across all five datasets, demonstrating their complementary contributions to the final clustering performance.

Revision:

  1. In Section 3.3 (Ablation Study), we have added two path-level ablation variants and clarified the complementary contributions of the two fusion paths.

Question 1

Compared with existing multi-modal clustering methods, UNIT-MIB explicitly incorporates EDL-derived modality-level epistemic uncertainty into IB optimization and feature-level fusion. Furthermore, DS-based evidential fusion and Deng-MI-based directed alignment connect uncertainty-aware representation learning with evidential decision-making, providing a unified framework for trustworthy multi-modal clustering.


Revision:

  1. In Section 2.8 (Discussion), we have clarified the key methodological differences between UNIT-MIB and existing multi-modal clustering methods.

Question 2

The primary focus of UNIT-MIB is multi-modal clustering, and all experiments are conducted within this task. Therefore, the current experimental results do not establish its generalizability to other unsupervised learning tasks. Although the uncertainty regulation and IB-based representation learning have potential for broader applications, extending UNIT-MIB to other tasks would require adapting its task-specific components and learning objectives.


Question 3

In UNIT-MIB, conventional mutual information is retained in the information bottleneck objective for information compression and preservation, whereas Deng MI is specifically adopted for opinion-level alignment.

We adopt Deng MI because it accounts for both class-wise belief masses and uncertainty through Deng entropy, making it suitable for measuring discrepancies between complete subjective opinions. In contrast, alignment based solely on normalized class probabilities does not explicitly preserve the uncertainty information contained in subjective opinions.

In this task, Deng MI enables the feature-fused opinion MS\mathcal{M}^{S} to align with the D-S evidential consensus MDS\mathcal{M}^{DS} in terms of both class beliefs and uncertainty. This provides an uncertainty-aware alignment constraint that helps reduce the semantic-logical gap..


Revision:

  1. In Section 2.6 (Directed Alignment via Deng Mutual Information), we have clarified the motivation and role of Deng MI in opinion-level alignment.

Thanks for the response. This addresses my concerns, and I have no further questions. I will consider raising my score accordingly.

Reviewer_C3P9OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

A new clustering method, named UNIT-MIB, is proposed for trusted multi-modal clustering. An alignment mechanism between the semantic space and the evidential space is designed to directly target the semantic-logical gap, which is a meaningful issue in multi-modal clustering. The ablation studies are relatively complete, verifying the contribution of each proposed component and provide supporting evidence for the design choices made in the paper. The experimental section is fairly comprehensive, but additional experiments are still needed. Evidential deep learning is utilized to improve the trustworthiness of the model. By explicitly modeling uncertainty instead of relying on overconfident softmax predictions, the model can better express its confidence in the clustering results.

Critical strengths
  1. The paper has a complete structure and a clear methodological logic. The presentation is well organized and easy to follow, which makes the contribution accessible to the reader.
  2. The dual-path multi-modal fusion mechanism is a sensible design. It combines information at both the feature level and the decision level, allowing the model to leverage complementary information from two different perspectives. 3.The stop-gradient mechanism is designed to improve training stability. It is a simple yet effective technique that prevents noisy semantic features from contaminating the evidential reasoning, making the training process more robust.
Critical weaknesses
  1. It is unclear whether the uncertainty is truly effective or merely functions as an attention weight over modalities. The paper does not demonstrate what the uncertainty estimate provides beyond a standard attention mechanism.
  2. The design of beta(u) appears to be an empirical choice. The paper does not justify why this particular function form is selected. 3.While the experimental section is more complete than average, it still lacks (i) noisy-/missing-modality robustness experiments that the "trustworthy" title implies. 4.Reproducibility details. Encoder architecture , batch size B, learning rate, and the final per-dataset values are not tabulated.
Questions for authors
  1. The choice of evidential deep learning over other uncertainty estimation methods is not justified. The paper should explain why this particular approach is selected.
  2. What are the advantages of combining both the information bottleneck and evidential deep learning?
1 comment
AuthorsOpenReview ↗

Response to Reviewer C3P9

We sincerely appreciate the reviewer's constructive comments and suggestions.


Weakness 1

The uncertainty in UNIT-MIB is not merely an attention-like modality weight. Specifically, um=C/Smu^m=C/S^m represents the uncertainty mass in a subjective opinion. It not only determines feature-fusion weights but also regulates evidential guidance in the IB objective through β(um)\beta(u^m). Moreover, modality-level uncertainty contributes to D-S fusion and subsequent opinion alignment. Thus, uncertainty influences feature fusion, and evidential decision-making rather than serving solely as a modality aggregation weight.


Revision:

  1. In Section 2.4 (Uncertainty-weighted Feature Fusion), we have clarified the roles of modality-level uncertainty beyond feature aggregation, including its contributions to IB optimization and evidential fusion.

Weakness 2

The choice of β(um)=(1−um)2\beta(u^m)=(1-u^m)^2 is motivated by the need to adaptively regulate the evidential guidance according to modality-level uncertainty. Specifically, we first convert uncertainty umu^m into a reliability measure 1−um1-u^m. We then adopt a quadratic mapping to introduce stronger nonlinear attenuation for uncertain modalities compared with a linear mapping. This design allows reliable modalities to retain stronger evidential guidance while suppressing the influence of highly uncertain modalities.


Revision:

  1. In Section 2.3 (Uncertainty-Driven Information Bottleneck), we have clarified the motivation for adopting the quadratic form of β(u)\beta(u) and its selective attenuation effect.

Weakness 3

We thank the reviewer for raising the concern regarding robustness to noisy and missing modalities.

In UNIT-MIB, the term "trustworthy" primarily refers to uncertainty-driven representation learning and evidential fusion, rather than a comprehensive guarantee of robustness to noisy or missing modalities. Although the proposed mechanisms incorporate modality-level uncertainty into representation learning and fusion, their effectiveness under different modality corruption conditions requires further investigation. Moreover, the current framework assumes complete and sample-aligned multi-modal data and does not explicitly handle missing modalities. Systematic robustness evaluation and extensions to incomplete multi-modal settings will be considered in future work.


Revisions:

  1. In Section 4 (Conclusion), we have clarified the limitations of UNIT-MIB regarding noisy and missing modalities, and identified systematic robustness evaluation and extensions to incomplete multi-modal settings as future research directions.

Weakness 4

To improve reproducibility, we have supplemented the implementation details and dataset-specific configurations. The encoder architectures, latent dimensions, batch sizes, learning rates, and EDL and Deng MI coefficients are summarized in Appendix A.6.


Revisions:

  1. In Appendix A.6 (Implementation Details), we have added a table summarizing the encoder architectures and dataset-specific hyperparameters.

Question 1

We have clarified the rationale for adopting EDL over alternative uncertainty estimation methods. Specifically, EDL provides explicit Dirichlet-based subjective opinions containing both class-wise belief masses and epistemic uncertainty, which are directly compatible with the D-S evidential fusion and Deng-MI-based opinion alignment in UNIT-MIB. Moreover, the estimated modality-level uncertainty can be directly incorporated into IB optimization and feature-level fusion. EDL also offers computational efficiency by estimating uncertainty through a single deterministic forward pass, without requiring multiple models or repeated stochastic inference.


Revision:

  1. In Section 2.5.1 (Subjective Opinion Modeling), we have clarified the rationale for adopting EDL.

Question 2

IB and EDL provide complementary benefits to each other. IB learns compact and discriminative representations for evidential estimation, while EDL provides modality-level uncertainty estimates to guide information compression. Their integration connects representation learning with uncertainty estimation, and its advantages have been further discussed in the revised manuscript.

Revision:

  1. In Section 2.8 (Discussion), we have clarified the complementary roles and benefits of combining IB and EDL.
Reviewer_XVh5OpenReview ↗
Rating
3(−1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

To address the limitations of traditional IB methods, such as poor modeling of modality uncertainty, overconfident errors under conflicts, and the disconnect between representation learning and decision-making, the authors propose UNIT-MIB. It adaptively adjusts compression via uncertainty estimation, fuses semantic features and evidential opinions through a dual-path strategy using DS theory, and introduces a Deng-entropy-based mutual information measure with stop-gradient directed alignment to bridge the semantic-logical gap. Experiments demonstrate that UNIT-MIB significantly outperforms state-of-the-art methods across many real-world datasets.

Critical strengths
  1. The paper is well-written and is easy to understand the motivations and the innovations of the method.
  2. The paper is logically rigorous and presents a novel uncertainty driven IB-related approach, which has the potential to alleviate the overconfidence issues in existing methods.
  3. The model achieved notable experimental results on different multimodal datasets, illustrating its superiority.
Critical weaknesses
  1. After carefully reading the work, I am not sure whether the work is the first uncertainty-driven IB method for multi-modal clustering. If not, the author should give the relationship with them in a clear way.
  2. The details of adopting Deng MI not general one is not clear in the main part of the work.
  3. It seems that the proposed method is used for complete and aligned multi-modal data. In real-world scenarios, incomplete/unaligned datasets also show wide attention, could the proposed method be modified into for the mentioned datasets?
Questions for authors
  1. What are the rationale for adopting the Deng MI?
  2. What are the relationships among current IB-based methods and proposed uncertainty-driven IB methods?
  3. Could the proposed UNIT-IB method be modified into addressing the other more challenging incomplete multi-modal datasets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer XVh5

Thanks for your valuable comments.


Weakness 1 & Question 2

To the best of our knowledge, UNIT-MIB is the first uncertainty-driven IB framework for multi-modal clustering that explicitly incorporates EDL-derived modality-level epistemic uncertainty into IB optimization.

Existing IB-based methods share the objective of learning compact and informative representations, but differ in the signals used to guide IB optimization. In contrast, UNIT-MIB uses EDL-derived modality-level uncertainty as a unified reliability signal to regulate both evidential guidance in IB optimization through β(um)\beta(u^m) and feature-level fusion weights.


Revision:

  1. In Section 2.8 (Discussion), we have clarified the relationship between UNIT-MIB and existing IB-based methods.

Weaknesses 2 & Questions 1

In UNIT-MIB, Deng MI is constructed based on Deng entropy by measuring the entropy difference between the average opinion and the two individual opinions. The average opinion is obtained through component-wise averaging of their belief and uncertainty masses.

We adopt Deng MI because it accounts for both class-wise belief masses and uncertainty through Deng entropy, making it suitable for measuring discrepancies between complete subjective opinions. In contrast, alignment based solely on normalized class probabilities does not explicitly preserve the uncertainty information contained in subjective opinions.


Revision:

  1. In Section 2.6 (Directed Alignment via Deng Mutual Information), we have clarified the details of adopting Deng MI and the rationale for its use. The detailed mathematical derivation is provided in Appendix A.1.

Weaknesses 3 & Questions 3

The current UNIT-MIB framework is designed and evaluated under the assumption of complete and sample-aligned multi-modal data. Therefore, it does not explicitly handle missing modalities or unknown cross-modal correspondences, which remain limitations of the current framework.

In principle, UNIT-MIB could be extended to incomplete and unaligned multi-modal clustering by incorporating missing-modality handling and cross-modal correspondence learning mechanisms. However, these extensions require additional methodological designs and are beyond the scope of the current work. We will investigate these directions in future research.


Revisions:

  1. In Section 4 (Conclusion), we have clarified the limitations of the current framework and identified extending UNIT-MIB to incomplete and unaligned multi-modal clustering as a future research direction.
Reviewer_XVh5OpenReview ↗

Response to authors' rebuttal

The authors' response has resolved most of my concerns, and I would rate this version as weak acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.