acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Multi-View Clustering through Semantic Recomposition

Abstract

Multi-view clustering (MVC) seeks to discover meaningful groups by integrating complementary descriptions of the same samples. However, geometric consistency alone does not ensure semantic correctness: nuisance variation can fragment one class across clusters, while shared contextual cues can mix distinct classes. Correcting these errors calls for semantic evidence that reveals both shared identities across clusters and distinct identities within them. We propose Semantic Recomposition (SemReC), which starts from a geometric partition, extracts semantic descriptions from sparse, coverage-oriented representatives using a multimodal large language model, and organizes them into frozen semantic seeds. These seeds guide split proposals, while a joint geometric–semantic objective evaluates coupled merge–split moves that preserve the cluster count. Across four benchmarks spanning visual, image–text, and audio–text settings, SemReC achieves an average ACC of 84.97% and the highest ACC on all four datasets among the compared baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-view Clustering, Semantic Recomposition, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4vybOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SemReC addresses semantic mixing and fragmentation in multi-view clustering by correcting an existing geometric partition. It selects coverage-oriented representatives, forms frozen local units, and uses two MLLM passes to produce normalized descriptions that are grouped into semantic seeds. A quotient graph retains sample-level connectivity, and a joint objective combines geometric crossing cost with two conditional-entropy terms. The optimizer accepts coupled merge–split moves only when they lower this objective while preserving exactly K nonempty clusters. The paper reports the highest ACC among eight baselines on four visual, image–text, and audio–text benchmarks, averaging 84.97%. Controls examine direct semantic assignment, semantic-seed permutation, auditing, and the application of recomposition to frozen baseline partitions.

Critical strengths

1.The paper introduces semantic evidence to MVC with a concrete motivation, which is correcting fragmentation across clusters and mixing within them. The objective and structural edits follow directly from these two failure modes. 2.SemReC achieves the highest ACC on all four benchmarks, with gains over the strongest baseline ranging from 1.79 points on AwA2 to 41.35 points on Caltech. The improvement extends across visual, image–text, and audio–text settings rather than being confined to one modality. Also, Applying Stages II–III to frozen CorreGen, CANDY, and DIVIDE partitions improves all three metrics in all six reported cases. This supports recomposition as a reusable correction plug-in, which can be useful to future research.

Critical weaknesses

1.The evidence for the two entropy terms is uneven: removing H(S | C) costs 39.17 ACC points on Caltech, whereas removing H(C | S) costs 1.90. Since this ablation is reported only on Caltech, the empirical importance of the reverse term across the other settings is less established. 2.The 30-operation cap may be active in practice. Mean accepted counts reach 28.3 on AwA2 and 27.0 on Fluent. The results do not distinguish runs that exhaust this cap from runs with no remaining improving candidate.

Questions for authors

1.On other datasets, does adding H(C | S) reduce label-based fragmentation relative to the objective without this term, and does that reduction translate into higher ACC?
2.How many runs terminate at the 30-operation cap, and does increasing this cap change the final objective or evaluation scores? 3.What fraction of representatives, and separately of sample coverage, is marked uncertain on each dataset, and how accurately are those unsupported units clustered after recomposition?

1 comment

Thanks to the authors for the responses, which address the questions and concerns I raised earlier. SemReC puts mixing and fragmentation into a single objective instead of handling them separately, and it corrects an existing partition by merge–split moves that keep the number of clusters KK fixed. More importantly, it can serve as a plug-and-play correction module on top of existing methods. Overall, this paper gives the community something worth thinking about, and I recommend acceptance.

Reviewer_AtTuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Semantic Recomposition (SemReC), a framework for correcting semantic mixing and fragmentation in multi-view clustering. It extracts semantic descriptions from representative samples using a multimodal large language model and organizes them into frozen semantic seeds. A joint objective combines semantic discrepancies with a geometric graph constraint. The method then performs merge–split operations while preserving the number of clusters. Experiments on four benchmarks show improvements over the compared baselines

Critical strengths
  1. The paper distinguishes semantic mixing from fragmentation and explains their implications for clustering. This formulation provides a coherent motivation for jointly considering merging and splitting.
  2. The method gives sparse semantic evidence a concrete role in correcting existing partitions. Its combination of frozen evidence and constrained structural updates provides a clear methodological distinction from the approaches discussed in the related work.
  3. Multiple benchmarks, component ablations, and frozen-backbone experiments support the method’s effectiveness.
Critical weaknesses
  1. The paper combines MLLM-based semantic extraction, semantic clustering, graph regularization, and merge–split optimization. However, it does not sufficiently isolate which component constitutes the main algorithmic innovation beyond their integration.
  2. Each representative defines a group whose membership remains unchanged during optimization. This design preserves local structure but prevents the method from correcting errors that require splitting a local unit or reassigning individual samples.
  3. Equations 1–2 use the unavailable ground-truth variable YY, whereas the optimization objective uses the semantic surrogate SS. Although the paper explains their intended correspondence, it does not establish when improvements in the surrogate-based objective reliably imply improvements in the true semantic structure.
Questions for authors
  1. What is the main algorithmic innovation beyond component integration?
  2. How do fixed local units limit error correction?
  3. When does improving the surrogate objective reliably improve true semantic structure?
Reviewer_6jHBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SemRec, a semantic recomposition framework for multi-view clustering (MVC) that leverages sparse semantic evidence to correct geometrically derived multi-view partitions. Several experiments are conducted to verify the effectiveness of the proposed method.

Critical strengths

S1. Addressing semantic fragmentation and semantic mixing is an important and timely problem in multi-view clustering.

S2. Several experiments have been conducted on four heterogeneous benchmark datasets, and the results show the effectiveness of the proposed method.

Critical weaknesses

W1. The logical consistency of the proposed SemRec framework is unclear. For instance, the Introduction states that the method comprises three stages, whereas the Method section describes only two phases.

W2. Notational inconsistencies hinder readability. Specifically, symbols such as H(Y∣C)H(Y|C) and H(C∣Y)H(C|Y) are used in the Introduction but are not formally defined until the Method section.

W3. The overall organization requires improvement. For example, the opening paragraph of the Method section redundantly overlaps with the Problem Formulation subsection; evaluation metrics (ACC, NMI, ARI) are referenced before being defined; and Figure 2 is placed far from its corresponding textual description.

Questions for authors

Q1. Could you clarify the discrepancy between the three-stage description in the Introduction and the two-phase formulation in the Method section?

Q2. Can you ensure all notations, particularly H(Y∣C)H(Y|C) and H(C∣Y)H(C|Y), are formally defined at their first occurrence to improve self-containment and readability?

Q3. Will you restructure the Method section to eliminate redundancy with Problem Formulation, define all metrics before use, and relocate Figure 2 adjacent to its relevant discussion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.