acceptodds
Under review as a conference paper at ICLR 2027

Justified Cross-View Interaction: Beyond Trivial Alignment in Multi-View Clustering

Abstract

Deep multi view clustering seeks to discover coherent semantic structures from heterogeneous views by exploiting cross view information. However, existing methods often treat cross view interaction as inherently beneficial, overlooking that information exchanged across views can be unreliable, structurally incompatible, or redundant. To address these issues, we formulate multi view clustering from the perspective of Justified Cross View Interaction (JCVI), where an effective interaction should be both useful and non trivial. Specifically, we develop a utility aware interaction mechanism that jointly characterizes semantic reliability, structural compatibility, and feature complementarity, and adaptively regulates cross view information exchange to selectively incorporate complementary cues while preserving view specific discriminative semantics. To promote interaction non triviality, we further construct a target exclusive supervisory context for each view using only the remaining views, thereby removing the direct self information path and encouraging the target representation to capture semantic structures predictable from genuinely external views. Multi granularity structural constraints are further imposed at pairwise and neighborhood levels to preserve coherent cross view organization. Extensive experiments on five benchmark datasets demonstrate the effectiveness of the proposed framework and its strong clustering performance against representative state of the art methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-view Clustering, Cross-View Interaction, Contrastive Learning, No Trivial Alignment.

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_7UNeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses unreliable information exchange and self-containing consensus supervision in multi-view clustering by proposing a framework with utility-aware interaction and target-excluding supervision. Guided by interaction utility and non-triviality, the framework enables utility-aware adaptive information exchange by jointly modeling semantic reliability, structural compatibility, and feature complementarity.

Critical strengths
  1. The paper examines whether cross-view interaction is beneficial and whether consensus supervision introduces self-information shortcuts, offering a meaningful perspective on multi-view fusion.

  2. Utility assessment, the interaction matrix, feature retention gates, and multi-granularity structural constraints are connected through a clear logical progression.

Critical weaknesses
  1. The paper should explicitly discuss the applicability of the framework to two-view settings.

  2. Although the related work covers a broad range of methods, the authors should clarify the specific differences between their approach and conventional attention-based and gated fusion methods.

  3. Since interaction utility and view importance depend on batch statistics, the authors should clarify how weights are computed during training and inference, and how weight consistency is maintained when extracting representations for the entire dataset.

Questions for authors
  1. Is reconstruction performed by decoding the original latent representations or the enhanced representations after cross-view interaction? How does this choice affect the preservation of view-specific information?
  2. What initialization and repeated-run strategies are used for the final k-means clustering? Do the reported results account for randomness in both network training and k-means initialization?
  3. How are implementation details, such as the softmax dimension, feature normalization, and the number of Sinkhorn iterations, specified?
1 comment

Thanks for your careful review. We are glad to address your questions one by one.

W1: JCVI is applicable to two-view settings, as its core mechanisms, including utility-aware cross-view interaction, target-exclusive supervision, and multi-granularity structural regularization, can still be implemented with two views. In this case, each target view receives information exclusively from the other view, and the target-exclusive context is directly constructed from that remaining view. Although the adaptive selection among multiple source views becomes unnecessary, the pairwise interaction mechanism and feature-wise gating still regulate cross-view information integration, while the structural constraints continue to promote semantic consistency.

W2: Unlike conventional attention-based fusion, which primarily assigns importance weights to different views, JCVI explicitly evaluates the utility of each cross-view interaction by jointly considering semantic reliability, structural compatibility, and feature complementarity. Compared with conventional gated fusion, which mainly regulates feature contributions during aggregation, JCVI further introduces an interaction non-triviality mechanism through target-exclusive supervision, preventing the direct inclusion of target-view information in its own supervisory context. Moreover, pairwise- and neighborhood-level structural regularization further preserves semantic consistency at different granularities. Therefore, JCVI goes beyond conventional importance-based weighting or feature-level gating by explicitly addressing both the usefulness and non-triviality of cross-view interactions.

W3: In JCVI, the interaction utilities and corresponding weights are computed dynamically within each mini-batch based on the current batch's view-specific representations. Specifically, semantic reliability, structural compatibility, and feature complementarity are evaluated to construct the interaction matrix through Sinkhorn normalization. During inference, the trained network parameters are kept fixed, while the interaction weights are dynamically computed for each mini-batch using the same procedure. To obtain representations for the entire dataset, we process the samples in mini-batches and concatenate their corresponding outputs in the original sample order. Although the interaction weights may vary across batches due to differences in batch statistics, the consistent computation procedure and fixed inference configuration ensure a well-defined representation extraction process.

Q1: In JCVI, reconstruction is performed using the original view-specific latent representations Z(v)Z^{(v)}, rather than the enhanced representations Z^(v)\hat{Z}^{(v)} after cross-view interaction. This design is intentional, as it allows each view-specific autoencoder to preserve the fundamental characteristics and information distribution of its original input without relying excessively on information from other views. Meanwhile, the enhanced representations are optimized through the cross-view interaction and contrastive objectives to promote complementary information integration and semantic consistency. Therefore, the reconstruction objective and cross-view learning objectives play complementary roles: the former preserves view-specific information, while the latter facilitates cross-view semantic enhancement. This separation also helps prevent excessive cross-view fusion from compromising the distinctive information of individual views.

Q2: In our experiments, JCVI is independently trained five times to account for the randomness introduced by network initialization and stochastic optimization. After each training run, standard K-means is applied to the learned comprehensive representation HH to obtain the final clustering assignments. Therefore, the reported results are averaged over five independent runs, accounting for the variability in network training.

Q3: For the cluster-assignment matrix Q(v)∈RB×KQ^{(v)} \in \mathbb{R}^{B \times K}, softmax is applied over the cluster dimension (dim=1), while the sample-similarity matrix S(v)∈RB×BS^{(v)} \in \mathbb{R}^{B \times B} is normalized row-wise (dim=1). View-level weights are normalized across views (dim=0). Sample representations and cluster prototypes are L2L_2-normalized along the feature dimension (p=2, dim=1) for similarity computation and contrastive learning, while reconstruction uses unnormalized encoder features. LayerNorm follows gated fusion, and the final global and contrastive representations are L2L_2-normalized. For Sinkhorn-Knopp, we mask diagonal entries, construct the kernel as exp⁡(U/τs)\exp(U/\tau_s) with τs=0.2\tau_s=0.2, and perform five rounds of alternating row-column normalization using ϵ=10−8\epsilon=10^{-8}. All settings are consistent across datasets.

Reviewer_udmgOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents JCVI for deep multi-view clustering. It questions if sharing information across all views is always good. The method controls information flow based on utility and uses a target-exclusive context to avoid shortcuts during alignment.

Critical strengths
  1. The idea of looking at cross-view information exchange is interesting.

  2. Formulating the weight calculation as an optimization problem makes sense mathematically.

  3. The experiments cover many different types of baselines.

Critical weaknesses
  1. Equation (4) uses γ\gamma to balance two factors. It would be better to show a sensitivity test for γ\gamma to see if the performance is stable.

  2. The semantic reliability uses an auxiliary clustering head, which might suffer from confirmation bias if the data is noisy.

  3. Some figure and table captions are too long. Please make them shorter.

Questions for authors
  1. Can you show a quick sensitivity test for γ\gamma (e.g., in [0.1,0.9][0.1, 0.9] on Synthetic3D or Caltech-5V)?

  2. For Equation (13), did you try concatenating the remaining views directly instead of a weighted sum? If not, why?

  3. Does the gating mechanism in Equation (15) change if we swap views uu and vv? Is it sensitive to the order?

2 comments

Thanks for your careful review. We are glad to address your questions one by one.

W1: We conducted additional hyperparameter sensitivity experiments on the Synthetic3D dataset by varying γ\gamma, which controls the balance between structural compatibility and feature complementarity in the interaction utility, while keeping all other hyperparameters fixed. Specifically, we evaluated the clustering performance under different values of γ\gamma to further investigate the effects of structural compatibility and feature complementarity on the interaction utility and assess the robustness of JCVI to different weight assignments.

Metric0.10.20.30.40.50.60.70.80.9
ACC97.6797.0097.1796.1797.0097.3396.8396.6796.83
NMI90.7988.1188.7185.4387.9788.5088.1988.1587.61
PUR97.6797.0097.1796.1797.0097.3396.8396.6796.83

W2: The auxiliary clustering head may produce overconfident but incorrect predictions, particularly during early training or under noisy conditions. However, in JCVI, semantic reliability is not used as the sole criterion for determining cross-view interactions. Instead, it is jointly considered with structural compatibility and feature complementarity to calculate the interaction utility. This multi-criteria design helps prevent potentially unreliable semantic predictions from dominating the interaction process. Moreover, semantic reliability acts as a continuous weighting factor rather than a hard selection criterion, allowing the model to regulate information exchange without completely excluding individual views.

W3: Following this suggestion, we have carefully revised the figure and table captions throughout the manuscript to improve their conciseness and clarity. For example, the caption of Figure 4 has been shortened to: “Illustration of JCVI's utility-aware cross-view interaction based on semantic reliability, structural compatibility, and feature complementarity.” Similarly, the caption of Figure 5 has been refined to: “t-SNE visualizations of raw data and learned representations at different training stages across five benchmark datasets.” These revisions remove repetitive descriptions while retaining the essential information, thereby improving the clarity and readability of the manuscript.

Q1: To address this concern, we conducted an additional sensitivity analysis on the Synthetic3D dataset by varying the trade-off parameter γ\gamma within the range of [0.1,0.9][0.1, 0.9], while keeping all other hyperparameters fixed. Specifically, we evaluated the clustering performance under different values of γ\gamma to examine the respective contributions of structural compatibility and feature complementarity to the overall framework and assess the robustness of JCVI to different parameter settings. The detailed experimental results are provided in our response to W1.

Q2: We considered the design choice between weighted aggregation and direct concatenation when constructing the target-exclusive context. We adopt weighted summation for three main reasons. First, the normalized view importance weights allow the model to adaptively emphasize more informative views while suppressing less contributive ones. Second, weighted aggregation produces a fixed-dimensional context representation regardless of the number of remaining views, avoiding the dimensional growth and additional MLP parameters introduced by concatenation. Third, it provides a compact consensus representation that facilitates cross-view semantic alignment. Although direct concatenation can preserve more view-specific information, it may increase model complexity and make the context construction more dependent on the number and ordering of views. Therefore, weighted aggregation provides a more compact and adaptive solution that is consistent with the objective of target-exclusive supervision in JCVI.

Q3: We thank the reviewer for raising this question. In Eq. (15), the concatenated features of two views, together with their absolute difference and element-wise product, are first fed into an MLP to learn an adaptive gating weight. This weight and its complementary weight are then used to control the respective contributions of the two view representations during fusion. Therefore, the mechanism does not assign a predefined priority to either view based on its position in the input. Instead, the fusion weights are dynamically learned from the joint feature information of the two views. With a consistent view ordering throughout training and evaluation, the mechanism does not introduce an explicit view-order preference.

Thanks for the response. After reading the response and the comments from other reviewers, I think my score is appropriate, and I am currently inclined to keep it as it is.

Reviewer_pT9ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JCVI, a deep multi-view clustering framework that regulates cross-view information exchange using utility-based weighting and constructs target-excluding contexts for contrastive supervision. Combined with pairwise and neighborhood structural regularization, the framework achieves improved clustering performance over comparison methods on five benchmark datasets.

Critical strengths
  1. The paper offers a novel perspective by taking “when is interaction justified?” as its central question, with a clear and coherent motivation.
  2. The target-excluding supervision mechanism is simple, easy to implement, and intuitive.
  3. The paper is generally well written, and the framework diagram is easy to follow.
Critical weaknesses
  1. Semantic reliability relies on the clustering head’s confidence, which may be unreliable early in training and risks reinforcing confident but incorrect predictions.
  2. Controlled robustness experiments, such as adding noise to a specific view, are missing and would help assess whether the method can suppress unreliable views.
  3. The datasets are relatively simple and small, with performance approaching saturation. Evaluation on larger datasets with more views is recommended.
Questions for authors
  1. How is the auxiliary clustering head trained, and are gradients backpropagated through the reliability scores?
  2. What is the distribution of the learned interaction matrix? Are the weights nearly uniform, and does the method reduce the weights assigned to noisy views?
  3. Given the substantial differences in scale between the reconstruction and contrastive losses, do the three weighting coefficients require dataset-specific tuning, or is there a common default configuration?
2 comments

Thanks for your careful review. We are glad to address your questions one by one.

W1: The clustering head may produce unreliable confidence estimates during early training. However, in JCVI, semantic reliability is not used as an independent criterion or hard pseudo-label supervision; instead, it is incorporated as a continuous soft weighting factor jointly with structural compatibility and feature complementarity to determine cross-view interaction utility. Therefore, even if the confidence estimates are imperfect, the interaction mechanism is not solely driven by potentially incorrect predictions. Furthermore, the reconstruction objective and multi-granularity structural constraints provide additional learning signals to facilitate representation learning and stabilize the clustering process.

W2: We conducted additional robustness experiments on Fashion and Synthetic3D under noisy-view and incomplete-view settings (corruption rate: 0.1). JCVI was compared with four representative baselines (CSOT, SSLNMVC, MISCC, and GFSAF) using ACC, NMI, and PUR. The results show that JCVI consistently outperforms all baselines across both datasets and settings, demonstrating its robustness to view corruption and missing information. These findings further validate the effectiveness of utility-aware interaction in mitigating the influence of unreliable views.

Noise (0.1)Incomplete (0.1)
FashionSynthetic3DFashionSynthetic3D
MethodACCNMIPURACCNMIPURACCNMIPURACCNMIPUR
CSOT (2024)98.7597.0498.7594.7585.5394.7585.3870.0285.3882.5671.3282.56
SSLNMVC (2025)97.4996.5397.4993.3782.2093.3792.7589.6492.7589.6478.8789.64
MISCC (2026)98.5696.4798.5695.1184.5595.1196.7592.3696.7583.6764.0383.67
GFSAF (2026)98.5496.8798.5495.6285.2195.6296.3692.3796.3692.4881.6292.48
JCVI99.2998.1899.2997.5089.6097.5099.0797.7099.0796.1784.4996.17

W3: To further validate the effectiveness and scalability of JCVI on larger-scale datasets with more views, we have introduced the YouTubeFace dataset as an additional benchmark, providing a more challenging evaluation scenario. Specifically, we conducted comparative experiments with four representative methods, including CSOT, SSLNMVC, MISCC, and GFSAF, using ACC, NMI, and PUR as evaluation metrics. The experimental results demonstrate that JCVI consistently achieves the best clustering performance across all three evaluation metrics on this larger-scale multi-view dataset, further supporting its effectiveness and generalization capability beyond the original benchmarks.

MethodYouTubeFace
ACCNMIPUR
CSOT (2024)30.9130.9040.04
SSLNMVC (2025)24.5625.7434.83
MISCC (2026)23.6824.5733.62
GFSAF (2026)18.9919.9732.76
JCVI34.9432.2744.85

Q1: In JCVI, the auxiliary clustering head uses a learnable prototype matrix CC to generate soft cluster assignments via softmax, with semantic reliability estimated from normalized prediction entropy. The head is trained end-to-end without a separate clustering loss. Gradients propagate through the reliability scores to update the prototypes and view-specific encoders via the interaction utility and subsequent objectives, enabling reliability estimation to evolve with representation learning.

Q2: In JCVI, interaction weights are adaptively determined by pairwise utilities based on semantic reliability, structural compatibility, and feature complementarity, rather than being uniformly assigned. Entropy regularization prevents excessive weight concentration while preserving utility-based differences. When a view becomes noisy, its reduced reliability and structural compatibility can lower its interaction utility, thereby suppressing unreliable information exchange. Thus, JCVI adaptively regulates cross-view information flow according to view quality.

Q3: We adopt a unified configuration across all datasets without extensive dataset-specific tuning. The three coefficients balance the non-trivial cross-view interaction and two structural regularization losses against reconstruction. As shown in Appendix A.2.3, JCVI maintains stable clustering performance over a broad coefficient range [0.1,10.0][0.1, 10.0], demonstrating its robustness to hyperparameter selection.

Thank you for your response. You have addressed my concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.