acceptodds
Under review as a conference paper at ICLR 2027

What to Preserve and Trust in Multi-modal Clustering: Cross-Guided Information Bottleneck with Reliable Contrastive Learning

21 views

Abstract

Multi-modal clustering (MMC) aims to discover common cluster structures from multiple modalities. Effective MMC requires determining what information to preserve and how strongly negative relations are trusted. Existing information bottleneck (IB)-based MMC methods can suppress unnecessary information while preserving shared content, but fail to explicitly consider that the contribution of each modality may vary across samples. Moreover, existing contrastive MMC methods can promote cross-modal consistency, but often overlook that negative pairs may contribute differently, especially when their relations are ambiguous or the samples involved are uncertain or inconsistent across modalities. To address these issues, we propose a novel reliable multi-modal clustering framework named CGIB-RCL, consisting of Cross-Guided Information Bottleneck (CGIB) and Reliable Contrastive Learning (RCL). Specifically, CGIB constructs guidance for each target modality by adaptively combining the remaining modalities for each sample, preserving information supported across modalities while reducing modality-specific content. RCL separately models relation ambiguity and sample reliability to determine the contribution of each negative pair, reducing the influence of ambiguous relations and pairs involving unreliable samples during contrastive learning. Together, CGIB and RCL regulate what information is preserved and how strongly negative relations are trusted during learning. Extensive experiments on five datasets demonstrate the effectiveness of the proposed framework. The source code will be available on the author's GitHub homepage upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-modal Clustering, Trust Learning, Information Bottleneck, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_zzbtOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

A novel framework named CGIB-RCL is proposed in this paper, which mainly contains Cross-Guided Information Bottleneck and Reliable Contrastive Learning module. In detail, CGIB adaptively integrates remaining modalities to preserve shared information while reducing specific content, whereas RCL models reliability to weight negative pairs. Experiments on five datasets verify its effectiveness in balancing information preservation and relation trust.

Critical strengths

1 The overall framework is well motivated. CGIB and RCL address different but complementary aspects of representation learning within a unified framework.

2 The method combines information bottleneck learning with a probabilistic guidance construction, providing a clear motivation for preserving cross-modal information while suppressing unnecessary content.

3 The mechanism analysis provide useful evidence that corrupted supporting views receive lower guidance weights, false negatives receive lower relation scores, and unreliable samples receive lower reliability scores.

4 The ablation studies examine the contributions of CGIB and RCL, as well as the main factors used in negative-pair weighting.

Critical weaknesses
  1. CGIB tries to preserve cross-modal shared information while compressing modality-specific information. However, modality-specific information may also be useful for clustering. How does CGIB avoid losing important discriminative information during compression?

  2. When evaluating a supporting modality, the reference representation already contains that modality. Could this affect the reliability of the guidance weights?

Questions for authors
  1. Does uncertainty-weighted feature fusion perform better than simple average fusion?

  2. Although RCL assigns lower weights to unreliable negative pairs, it remains unclear whether this weighting strategy actually reduces their negative impact on clustering performance.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zzbt

Thank you for your constructive comments. We have clarified the relevant mechanisms and added supporting experiments.


Weakness 1

We agree that modality-specific information may contain useful discriminative cues. CGIB does not explicitly remove such information but balances compression and preservation through regularization. The KL term regulates information retention relative to cross-modal guidance, while RmR_m encourages feature-level cross-modal associations. DDC and RCL provide additional learning signals for clustering.

Table 2 shows that adding CGIB to DDC improves both ACC and NMI on four datasets, while the full model outperforms DDC on all five. These results support the practical effectiveness of this design, although retaining every useful modality-specific feature is not guaranteed.

Revision: We have clarified these complementary roles in Section 3.5.


Weakness 2

We acknowledge that the reference representation includes the supporting modality being evaluated, potentially introducing self-inclusion bias. However, guidance weights depend on the gap between matched and mismatched similarities computed against the same reference, rather than on matched similarity alone.

Figure 5 shows that corrupted supporting modalities receive lower guidance weights and that adaptive guidance is less affected by corruption than uniform aggregation. These results support the practical reliability of the weighting mechanism, although self-inclusion bias cannot be completely ruled out.

Revision: We have clarified the guidance-weight construction and potential self-inclusion effect in Section 3.3.


Question 1

We compared uncertainty-weighted representation fusion with Average Fusion, which assigns equal weights to all modalities. The full model achieves higher ACC and NMI across all five datasets, supporting the benefit of uncertainty-based fusion.

Revision: We have added the average-fusion ablation in Appendix A.7.


Question 2

RCL reduces the influence of potentially unreliable negative pairs through adaptive weighting. Figure 6 supports this mechanism, while the w/o Negative Weighting ablation in Table 3 confirms its contribution to clustering performance across all five datasets.

Additional Constant Weight and Shuffled Weight controls show that the improvement depends on assigning appropriate weights to individual negative relations rather than simply reducing negative-pair contributions.

Revision: We have added these control experiments in Section 4.4.

The authors have satisfactorily addressed my concerns. I have therefore raised my score to 4.

Reviewer_M7e3OpenReview ↗
Rating
3(−1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a new CGIB-RCL method for reliable multi-modal clustering, addressing two typical limitations in existing IB and contrastive learning approaches. The proposed method constructs sample-specific guidance by fusing other modalities, filtering out modality-specific noise while retaining shared semantics. Meanwhile, RCL quantifies relation ambiguity and sample uncertainty to down-weight unreliable negative pairs during contrastive learning process. This joint design improves clustering robustness, as demonstrated by different experiments and their corresponding results.

Critical strengths

1). The paper addresses two meaningful challenges in multi-modal clustering: what information should be preserved and how negative sample relations should be handled. 2). CGIB-RCL achieves the best reported ACC and NMI results across all five evaluated datasets, showing consistent improvements over the compared methods. 3). The paper includes additional analyses of parameter sensitivity, training convergence, and learned representations, helping readers better understand the behavior of the proposed framework.

Critical weaknesses

1). Uncertainty and modality agreement are estimated from the model's own predictions. Could inaccurate estimates during early training reinforce incorrect representations? 2). Though the similarity gap provides a measure of relation ambiguity, it is not clear whether this measure can reliably distinguish ambiguous negative relations across different samples and datasets.

Questions for authors

1). Is adaptive guidance actually better than uniform guidance in terms of final clustering performance? 2). Are both uncertainty and cross-modality disagreement necessary for estimating sample reliability?

2 comments
AuthorsOpenReview ↗

Response to Reviewer M7e3

Thank you for your constructive comments. We provide the following clarifications and additional experimental evidence.


Weakness 1

We agree that reliability estimates may be inaccurate during early training. RCL mitigates this issue by continuously updating negative-pair weights using current representations and predictions, rather than relying on fixed early estimates. DDC and CGIB also provide complementary learning signals.

Figure 3 shows stable convergence across all five datasets, while Figure 6(b) shows that corrupted samples receive lower reliability scores. These results support the practical effectiveness of dynamic weighting, although they do not guarantee the accuracy of every early-stage estimate.

Revision: We have clarified the dynamic computation of negative-pair weights in Section 4.1.


Weakness 2

The relation score in Eq. (10) compares each negative-pair similarity with the corresponding matched positive similarity. A smaller gap indicates a more ambiguous relation and leads to a weaker negative-pair constraint.

Figure 6(a) shows that false negatives receive lower average relation scores than true negatives. To further evaluate discrimination at the individual-pair level, we report two additional metrics:

  • AUROC: Measures how often a false negative receives a lower score than a true negative.
  • AUPRC: Evaluates how effectively false negatives can be identified from these scores, accounting for precision and recall.

Ground-truth labels are used only for evaluation, not for training. The false-negative rate provides the random baseline for AUPRC.

Table 1. False-negative discrimination using relation scores.

DatasetFalse-negative rate (%)AUROC ↑AUPRC ↑
Caltech-2V14.260.8840.654
Caltech-3V14.180.9090.750
WVU9.730.8320.520
MIRFlickr16.670.7500.439
Prokaryotic39.420.6620.577

For example, on Caltech-2V, an AUROC of 0.884 means that a randomly selected false negative receives a lower score than a true negative in about 88.4% of comparisons. Its AUPRC of 0.654 also exceeds the random baseline of 0.1426. Across all five datasets, AUROC exceeds 0.5 and AUPRC exceeds the corresponding false-negative rate.

We also examined reversed relations, where negative similarity exceeds positive similarity. As reported in Appendix A.6, these account for 2.01%–9.14% of negative pairs and at most 1.13% of the total relation-score mass, suggesting a limited influence on the relation-weighting signal in the evaluated datasets.

Together, these results support the similarity gap as a useful continuous measure of negative-relation ambiguity rather than a guaranteed false-negative classifier.


Question 1

We compared sample-specific guidance with Uniform Guidance. The full model achieves higher ACC and NMI on all three datasets with multiple supporting modalities. For two-view datasets, uniform guidance is equivalent to the original formulation.

Figure 5 further shows that corrupted supporting modalities receive lower guidance weights and that adaptive guidance is less affected by corruption than uniform aggregation.

Revision: We have added the uniform-guidance ablation in Appendix A.7.


Question 2

Fused uncertainty reflects predictive uncertainty, while cross-modal disagreement captures inconsistency across modalities. Both are needed because predictions may be confident but inconsistent, or consistent but uncertain.

We compared the full model with Uncertainty-only and Disagreement-only variants. The full model achieves higher ACC and NMI than both across all five datasets, supporting their complementary roles in sample reliability estimation.

Revision: We have added the corresponding reliability-component ablations in Appendix A.7.

The added rebuttal and explanations address my concerns. I therefore keep my score.

Reviewer_xFkMOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CGIB-RCL, a multi-modal clustering framework that jointly addresses “what information to preserve” and “how to trust negative relations.” CGIB adaptively combines the Gaussian latent distributions of the remaining modalities for each sample to construct information bottleneck guidance for the target modality. RCL combines relation ambiguity, predictive uncertainty, and cross-modal consistency to weight negative pairs. The model is jointly optimized with the DDC clustering objective and reports higher ACC and NMI than the compared methods across five benchmark settings, accompanied by ablation studies and mechanism analyses.

Critical strengths

1.The reported performance improvements are substantial and cover different data settings.

2.The proposed CGIB addresses cross-modal information preservation, while RCL distinguishes relation ambiguity from sample reliability.

Critical weaknesses
  1. PTIB [1] already uses the remaining modalities to evaluate the target modality, and STCMC-UR [2] already combines evidential uncertainty with false-negative handling. However, Table 1 lacks direct comparisons with these two methods.

  2. The upper bound in Eq. (12) itself is valid, but the dependency regularizer RmR_m in Appendix A.5 has no established explicit relationship with the target mutual information. Even when representations are constant across samples, different constant representations across modalities may still produce a positive RmR_m. Therefore, whether increasing this regularizer actually means preserving more cross-modal information remains insufficiently justified.

  3. The negative-pair weights in Eq. (11) are generated by the model itself, but the paper does not specify whether gradients through these weights are stopped. Decreasing the relation scores or reliability can directly reduce the contrastive loss. It is therefore necessary to rule out the possibility that the gains arise from decreasing weights rather than more effective representation learning.

  4. Table 3 lacks controls using uniform guidance, as well as constant or shuffled negative weights with the same mean weight. Removing the weighting factors also changes the overall strength of the negative-pair contribution. Consequently, it remains unclear whether some of the gains come from generally weakening the constraints imposed by negative pairs.

Questions for authors
  1. Can the authors provide comparisons with PTIB and STCMC-UR using the same features and corresponding protocols, and clarify the substantive added value of the proposed sample-level guidance and continuous negative-pair weighting?
  2. How do the authors explain the possibility of obtaining a positive RmR_m when sample representations are constant in Appendix A.5? What theoretical properties or empirical evidence support its use as a surrogate measure of cross-modal dependence?
  3. Are gradients propagated through the relation scores and reliability weights in Eq. (11)? If so, how does the model avoid optimizing the loss by reducing the weights rather than improving the representations?
  4. Does the proposed adaptive mechanism retain a clear advantage over uniform guidance, constant negative weights with a matched mean, or shuffled negative weights?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xFkM

Thank you for your constructive comments. We have conducted additional experiments and clarified the relevant details in the revised manuscript.


Weakness 1 & Question 1

Using the same input features and corresponding evaluation protocols, we compared CGIB-RCL with PTIB and STCMC-UR (ACC/NMI, %).

DatasetPTIBSTCMC-UROurs
Caltech-2V59.4/45.365.3/56.273.3/63.9
Caltech-3V63.9/51.071.6/63.379.1/71.6
WVU54.8/60.353.2/52.064.5/67.2
MIRFlickr33.7/29.248.7/32.359.4/37.0
Prokaryotic52.5/26.054.5/31.267.3/41.6

CGIB-RCL achieves higher ACC and NMI than both methods across all five datasets. Unlike PTIB, which uses modality-level weights, CGIB constructs sample-specific guidance. Unlike STCMC-UR, which excludes potential false negatives through binary filtering, RCL continuously adjusts their contributions based on relation ambiguity and sample reliability.

Revision: We have clarified these differences in Sections 2.1 and 2.2.


Weakness 2 & Question 2

We agree that symmetrization can produce a positive RmR_m even when representations are constant across samples. Thus, RmR_m is neither a guaranteed estimator nor a universal lower bound of I(Zm;X−m)I(Z^m;X^{-m}), and maximizing it need not increase true mutual information. Accordingly, Eq. (14) provides a practical dependency-regularized objective rather than a guaranteed variational bound on Eq. (3). The KL upper bound in Eq. (12) remains valid.

Existing challenges in directly estimating mutual information motivate the use of a tractable dependency regularizer as a practical surrogate. Our formulation follows an IIC-inspired feature-association construction, and a similar construction is used in MSDIB (AAAI 2025, Eqs. 14–15). This provides methodological motivation without establishing a theoretical mutual information guarantee.

To examine whether RmR_m captures meaningful cross-modal associations, we added a correspondence-shuffling analysis in Appendix A.5. With representations fixed, complete shuffling reduces RmR_m by 94.1%–98.3% across the five datasets, demonstrating its sensitivity to cross-modal sample correspondence.

We further evaluated its practical contribution by removing RmR_m, removing symmetrization, and shuffling the sample pairings used by RmR_m during training.

Clustering performance (ACC/NMI, %).

DatasetFullw/o RmR_mNonSym-RmR_mShuffled-RmR_m
Caltech-2V73.3/63.965.9/57.764.8/58.968.8/58.7
Caltech-3V79.1/71.667.9/58.970.7/62.476.1/68.3
WVU64.5/67.263.4/66.364.2/67.062.3/66.5
MIRFlickr59.4/37.058.2/36.857.6/36.357.2/35.8
Prokaryotic67.3/41.664.6/39.167.0/41.065.7/40.0

The full model outperforms all three variants across all five datasets. These results support the contribution of RmR_m, the empirical benefit of symmetrization, and the importance of correct cross-modal correspondence. Together, they support the practical value of RmR_m as a cross-modal dependency regularizer.

Revision: We have clarified the role and limitations of RmR_m in Section 3.5 and Appendix A.5, and added the correspondence-shuffling analysis in Appendix A.5.


Weakness 3 & Question 3

Gradients flow through relation scores and fused uncertainty, while cross-modal disagreement is detached to limit direct manipulation of negative-pair weights. These weights are computed from current representations and predictions rather than learned independently. DDC and CGIB also provide complementary learning constraints.

The gradient-detachment ablations show that retaining these gradient pathways improves ACC and NMI across all five datasets. Although these results do not completely rule out optimization shortcuts, they support the effectiveness of our gradient design.

Revision: We have clarified gradient propagation in Section 4.1 and added the corresponding ablations in Appendix A.7.


Weakness 4 & Question 4

We added three controls: Uniform Guidance, Constant Weight, and Shuffled Weight. The full model outperforms uniform guidance on datasets with multiple supporting modalities and both negative-weight controls across all five datasets. These results support the benefits of sample-specific guidance and adaptive weighting of individual negative pairs.

Revision: We have added comparisons with constant and shuffled negative weights in Section 4.4, and the uniform-guidance ablation in Appendix A.7.

I thank the authors for the additional experiments and for the comments addressing my concerns. I raised my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.