acceptodds
Withdrawn from ICLR 2027

Beyond Feature Geometry: Semantic Evidence for Contrastive Multi-View Clustering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-view clustering (MVC) aims to discover semantically meaningful groups by exploiting consistency and complementarity across views. Many contrastive MVC methods construct supervision from learned representations or model predictions. However, feature geometry may capture attributes that do not align with the task-specific semantic criterion. Consequently, even high-confidence supervision can reinforce semantically incorrect relationships. We refer to this mismatch as Semantic-Blind Correspondence (SBC). To address SBC, we propose Semantic Evidence for Contrastive Multi-View Clustering (SECMVC), which introduces semantic evidence through a small budget of multimodal large language model (MLLM) queries. SECMVC first constructs verified semantic anchors under a fixed task criterion and partitions the remaining samples into reliable and unreliable sets. Reliable samples learn directly from semantic prototypes without additional queries. For unreliable samples, SECMVC queries a representative subset and uses confirmed assignments to guide reference-based contrastive learning, while uncertain responses provide no semantic supervision. In this way, SECMVC propagates semantic evidence when feature-based assignments are reliable and directs MLLM queries toward uncertain cases. Experiments on four multimodal clustering benchmarks demonstrate the effectiveness of SECMVC.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-view Clustering, Multimodal Large Language Model, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QtcHOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SECMVC, a contrastive multi-view clustering method that utilizes multimodal large language models (MLLM) to provide semantic supervision. This method constructs semantic anchors through a small number of MLLM queries, and adopts prototype learning and query-guided contrastive learning respectively based on the reliability of the samples. The authors conducted experiments on four multimodal datasets and reported superior results compared to traditional multi-view clustering methods.

Critical strengths

The overall structure of the paper is clear, and the description of the methods and experiments is relatively complete.

Critical weaknesses
  1. The practical significance of the research question and the necessity of the methods are insufficient. With the development of multimodal large models, many traditional multimodal clustering tasks can now be directly solved by the semantic understanding capabilities of pre-trained models. Although this paper introduces MLLM, it still builds a complex framework around traditional feature alignment, prototype learning, and contrastive clustering. The authors do not adequately explain what this framework actually solves, compared to directly using large models for semantic classification or clustering. This is an important issue that cannot be simply replaced by simple methods, and it is still crucial. Therefore, I have significant doubts about the practical value and research significance of this work.
  2. There is a lack of key large model baselines, and the experimental persuasiveness is insufficient. The 18 baselines compared in the paper did not use MLLM, while the proposed method can utilize MLLM to obtain additional semantic information. Such a comparison cannot distinguish whether the performance improvement comes from the large model itself or from the proposed clustering algorithm. In particular, there is a lack of comparisons that directly use MLLM, MLLLM to generate pseudo-labels, and modern multimodal semantic embeddings combined with simple clustering algorithms. This makes the significant performance improvement reported in the paper difficult to fully prove the independent value of the method.
Questions for authors
  1. Given that multimodal large models already possess strong semantic understanding capabilities, the author asks: Compared to directly using MLLM for semantic classification or clustering, what are the irreplaceable advantages of SECMVC? Are there any practical application scenarios that can only be addressed by relying on the proposed framework?
  2. If under the same MLLM query budget, using MLLM to generate pseudo-labels and combining simple label propagation or K-means, how would the performance be? How do the current experiments prove that the performance improvement comes from the proposed clustering mechanism rather than the semantic understanding ability of MLLM itself?
Reviewer_ufUCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SECMVC, a semantic-guided contrastive multi-view clustering framework designed to address Semantic-Blind Correspondence (SBC), where feature-space relationships may conflict with task-specific semantics. The method leverages MLLM-generated semantic anchors and a reliability-aware two-path learning strategy to guide representation learning and clustering. Experiments on four benchmarks report improvements over conventional multi-view clustering baselines.

Critical strengths
  1. The paper identifies the potential mismatch between feature-space relationships and task-specific semantics, providing an intuitive motivation for incorporating external semantic guidance.
  2. The proposed framework integrates semantic anchor construction, reliability estimation, and two-path contrastive learning into a coherent pipeline.
  3. The evaluation covers four datasets involving images, text, audio, and video. The paper also includes ablation studies and analyses of query budgets and hyperparameter sensitivity.
Critical weaknesses
  1. A key contribution of this work is to leverage limited MLLM queries for effective semantic-guided clustering. However, all baselines lack external semantic supervision. Without comparisons to existing semantic-guided clustering methods in terms of both performance and query cost, the effectiveness and efficiency of the proposed approach remain insufficiently validated.
  2. The proposed method relies on manually specified Criterion Cards that closely align with the ground-truth categories of the evaluated datasets. However, the construction of these criteria and their influence on clustering performance are not sufficiently discussed.
  3. The paper highlights that high-confidence relationships in feature space may conflict with task-specific semantics. Nevertheless, semantic anchors are selected based on proximity to K-means cluster centers, while unreliable samples are audited according to local neighborhood similarity. Both strategies still rely heavily on feature geometry, potentially overlooking semantically informative samples that are not geometrically representative. The effectiveness of these selection strategies is not adequately validated through ablation studies.
  4. In Figure 3(b), the Anchors-only variant already outperforms all conventional baselines on Caltech101-20. However, this experiment reuses semantic evidence generated by the full method, making the experimental setup and interpretation of the results somewhat unclear.
Questions for authors
  1. Could the authors compare SECMVC with existing externally guided clustering methods, under comparable semantic supervision or query budgets? Additionally, could the authors report the actual inference time and computational cost of MLLM queries to better support the claimed efficiency?
  2. How are the Criterion Cards constructed, and do they require prior knowledge of the dataset's category definitions? How sensitive is the method to different task descriptions, particularly ambiguous or less precisely specified criteria?
  3. Could the authors compare the proposed selection strategies against random sampling, uncertainty-based sampling, and diversity-based sampling under the same MLLM query budget?
  4. Could the authors clarify the complete training and evaluation procedure of the Anchors-only variant in Figure 3(b)?
Reviewer_BpiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the semantic-blind correspondence problem in contrastive multi-view clustering. The proposed SECMVC builds semantic anchors from representative samples using a frozen multimodal large language model; the remaining samples are partitioned by cross-view agreement and the margin between their nearest and second-nearest semantic prototypes. Reliable samples learn directly from semantic prototypes, while selected unreliable samples are audited and supervised via confirmed instance references. The two paths are combined with cross-view alignment, reconstruction, and clustering with semantic priors. Experiments on four benchmarks show improvements over eighteen baselines, with ablations validating the contribution of each path

Critical strengths
  1. The paper distinguishes between confidence in feature-space relationships and correctness under the intended clustering criterion. The examples make this distinction easy to follow and explain why filtering low-confidence pairs may leave some errors unresolved.
  2. The method uses semantic prototypes to supervise samples with consistent assignments across views, while reserving additional MLLM queries for ambiguous samples. It also leaves uncertain responses unlabeled and uses negative relations only when they are explicitly confirmed.
  3. SECMVC improves clustering performance on all four benchmarks. On Caltech101-20, the comparison using the same frozen semantic evidence for initialization, supervised contrastive learning, and the full method helps demonstrate the benefit of the two-path design.
Critical weaknesses
  1. All eighteen baselines operate without the MLLM queries and raw-input semantic evidence used by SECMVC. Although the frozen-evidence comparison on Caltech101-20 helps isolate the contribution of the learning design, comparisons with existing semantic-guided clustering methods and simpler MLLM-assisted approaches under comparable semantic access and query budgets are still needed.
  2. The paper does not directly report how often the semantic anchors and audited assignments are correct, or how performance changes when the MLLM makes mistakes. Since an incorrect anchor can affect many unqueried samples, the paper should assess the frequency of such errors and their impact on the final clustering.
  3. Appendix A.6 reports 2,481 to 12,370 API calls per full run and distinguishes the worst-case quadratic bound from the reduced call count achieved through batching. However, end-to-end runtime and monetary cost are not reported. These measurements, together with comparisons against simpler semantic-guided approaches under matched query budgets, would help assess the practical efficiency of the method.
Questions for authors

please refer to critical weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.