acceptodds
Under review as a conference paper at ICLR 2027

Semantic Calibration Beyond Anchor Correspondence for Multi-View Clustering

Abstract

Multi-view clustering aims to uncover a shared partition from heterogeneous representations of the same underlying data. Most existing methods emphasize cross-view consistency, implicitly assuming that a global category exhibits compatible local semantics across views. In practice, however, differences in modalities, feature representations, anchor construction, missing or degraded observations, and other view-specific factors can induce class-level semantic shifts, making this assumption overly restrictive. We propose Calibrated Anchor Semantic Transitions (CAST), a probabilistic framework that explicitly models how global cluster semantics are expressed through view-local semantic states. CAST operates directly on heterogeneous sample-to-anchor assignments, allows different anchor cardinalities across views, and avoids requiring cross-view anchor correspondence. Learned class-level transitions use cross-view evidence to infer global semantics while identifying and calibrating systematic sample–view shifts. We further establish theoretical properties concerning correspondence-free semantic evidence, semantic discriminability, multi-view complementarity, and the convergence behavior of the proposed MAP-EM optimization. Extensive experiments on multiple public multi-view clustering benchmarks, together with controlled semantic-shift evaluations, demonstrate strong clustering performance across both naturally heterogeneous and deliberately shifted settings. Ablation studies on the semantic transition structure further verify that explicitly modeling class-level semantic transitions is important for capturing cross-view semantic heterogeneity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-view Clustering, Anchor Graph Learning, Semantic Calibration, Probabilistic Modeling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ft6aOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The submission studies whether systematic differences in view-local semantics can be exploited for clustering rather than simply discouraged. Its method, CAST, learns class-conditional transitions between a shared partition and local semantic states represented by independent anchor graphs. The evaluation combines eight public benchmarks, eleven comparison methods, two controlled shift mechanisms, transition-channel ablations, unequal-anchor experiments, and view-wise semantic diagnostics. The reported results support the usefulness of semantic calibration on both ordinary and deliberately shifted data, with additional experiments examining which aspects of the transition model matter.

Critical strengths

First, Table 1 makes a persuasive empirical case for the semantic-shift hypothesis by combining results on the original benchmarks with controlled Euclidean and spherical shifts. The central idea is that views can share a global partition while their local semantics differ systematically. CAST is already among the strongest methods on the original data, making it relevant to routine clustering. The YTF results after the controlled shifts are particularly informative: on YTF10, ACC rises from 88.41% to 97.59% and 96.87%, and YTF20 and YTF50 also improve. I find this evidence convincing because it fits the explanation that systematic semantic changes can preserve useful class information and sometimes produce structure that a calibrated model can use more effectively.

Second, the mechanism is examined through complementary structural ablations and semantic diagnostics. On unmodified MSRC and YTF10, Table 3 shows substantial gains over Identity, supporting the relevance of naturally occurring semantic discrepancies in heterogeneous handcrafted representations. Under controlled shifts, full CAST generally does better than the restricted transition variants; some restrictions even perform worse than Identity. These comparisons provide good evidence that class-dependent shift rates and destination distributions serve complementary purposes, and that the chosen parameterization of T(v)T^{(v)} has an appropriate level of expressiveness for systematic shifts. The semantic diagnostics in Figures 2 and 3 complement these ablations: view-wise ACC, mean semantic preservation, and normalized semantic information follow broadly similar trends. Their departures from exact agreement also make sense given the theoretical distinction between preserving semantics and retaining class information.

Critical weaknesses

My main reservation concerns how the experiments are reported. I would like the numerical shift settings, the order of preprocessing and graph construction, and the tuning and randomization behind the five-run means to be described together more clearly for reproducibility. A measure of dispersion would also help me judge the close comparisons.

For Table 4, I would like to see the selected anchor-count configurations for each view and a short discussion of why the gains vary across datasets. The ability to accommodate heterogeneous anchor graphs remains useful even when the performance change is small.

Questions for authors
  1. For Table 1 and Figures 2 to 5, could the authors give a small configuration table listing δ,η\delta,\eta, the affected views, and any view left unchanged? Are these settings shared across datasets?

  2. Could the authors spell out the sequence of feature preprocessing, shift generation, any preprocessing after the shift, and anchor-graph construction? I would especially like to know when sample-wise normalization is applied in the spherical and Euclidean settings, and whether anchors and assignments are rebuilt from the transformed features or partly reused. If Euclidean relocation is followed by sample-wise normalization, which stage does the within-subgroup distance-preservation statement refer to?

  3. I would like to understand what "best mean performance over five runs" means here. Are hyperparameters and anchor counts searched separately for each shifted condition, or carried over from the original data? How is the final configuration chosen for each method, and how is the selection rule kept consistent across methods?

  4. Which random choices are made when generating shifts, constructing anchor graphs, and initializing CAST? Please give the seed lists or a deterministic generation rule, and say whether the predefined seeds are shared across datasets or specified separately. Across the five runs, are the perturbed data and graphs fixed, or is either regenerated? This would make clear which sources of randomness the reported means cover.

  5. A standard deviation, range, or another compact dispersion statistic from the existing runs would make the close comparisons easier to judge. It would also help assess whether the main full-versus-restricted transition conclusions recur across runs.

  6. For Table 4, what are the selected (m1,…,mV)(m_1,\ldots,m_V) configurations, and how were they chosen? Where there is little or no gain, is the common count already sufficient, is performance near saturation, or do the views gain little from different representation sizes? The flexibility to accommodate heterogeneous graphs remains useful in its own right, even where individual metrics do not improve.

2 comments

Response to Question 1:

For both controlled transformations, δ\delta is the fraction selected from each source class and η\eta controls the transformation strength. We use δ=0.5\delta=0.5 and η=1\eta=1 across all datasets. In each affected view, round⁡(0.5nc)\operatorname{round}(0.5n_c) observations are selected from source class cc. The selected observations receive the full source-to-target centroid displacement in the Euclidean setting or the full source-to-target angular displacement in the spherical setting.

Datasetδ\deltaη\etaAffected viewsUnchanged view
EuroSAT, RSICB128, YTF10, YTF20, YTF500.511, 2, 34
MSRC0.511, 2, 3, 4, 56
CIFAR10, CIFAR1000.511, 23

The indices follow the original view order.

Response to Question 2:

Both controlled conditions apply sample-wise normalization before and after the shift:

\longrightarrow \ell_2\text{ normalization} \longrightarrow \text{Euclidean relocation or spherical rotation} \longrightarrow \ell_2\text{ normalization} \longrightarrow \text{new anchors and assignments} \longrightarrow \text{CAST}.$$ Both the anchors and the sample-to-anchor assignments are constructed from the transformed, normalized features. Selected observations from the same source class and view receive the same displacement, so $\widetilde x_i^{(v)}-\widetilde x_j^{(v)}=x_i^{(v)}-x_j^{(v)}$. Their pairwise distances and centered covariance are preserved at that step. Subsequent sample-wise normalization can change those distances. Spherical rotation preserves norms and inner products within the subgroup receiving the same orthogonal transformation. Its inputs already have unit norm, so the subsequent normalization preserves this geometry up to numerical precision. **Response to Question 3:** The original, Euclidean-shift, and spherical-shift conditions are tuned separately. For standard CAST, the common anchor count is searched over $m\in\\{C,2C,3C\\}$, with $m_v=m$ for every view. At each candidate configuration, ACC is averaged over five fits. The configuration with the highest mean ACC is selected, and the reported NMI and ARI are computed from the same predicted labels used to obtain the best ACC. Anchor-based comparison methods use the same anchor-count candidate set; their other method-specific hyperparameter ranges follow their original papers. The mean-ACC selection rule is shared across methods. Table 3 reuses the selected CAST configuration for each dataset and condition, fixing the graphs and non-transition settings across the transition variants. Each variant is fitted five times.

Response to Question 4:

To generate shifts, we draw a separate within-class sample ordering for each affected view. For each source class in that view, one target class is sampled uniformly from the other classes, and all selected observations from that source class use this target. Different source classes may share a target.

Randomness also enters through the initial anchors and sample-to-anchor assignments in graph construction and the spherical K-means starts used to initialize CAST. For reproducibility and fair comparisons, we fix the random seed at 1 across all datasets for shift generation, graph construction, and CAST fitting. The transition variants follow the same initialization rule for paired comparisons.

Within a CAST call, spherical K-means is run with ten starts for each view. These starts use successive draws from that call's random stream, and the solution with the largest within-cluster cosine objective initializes the view's local components.

Shifted observations are generated once per dataset and condition. Graphs are constructed once per candidate configuration and reused by the five fits and the paired transition variants. The reported means therefore measure repeatability under fixed inputs and initialization. Figures 2–5 use a single fit at the corresponding selected configuration.

Response to Question 5 and Weakness 1:

For the reported five-run experiments, the standard deviations of ACC, NMI, and ARI are all 0. This includes both full-transition settings, CAST with γS=0\gamma_S=0 and γS=2\gamma_S=2, and the restricted transition variants. Under the protocol described above, the reported metric differences and full-versus-restricted rankings are identical across all five repetitions, including the close comparisons.

Response to Question 6 and Weakness 2:

For CAST-H, each view has the candidate set mv∈C,2C,3Cm_v\in\\{C,2C,3C\\}. We search the full Cartesian product C,2C,3CV\\{C,2C,3C\\}^{V}, considering all 3V3^V combinations. The selected configuration maximizes five-run mean ACC; NMI and ARI are computed from the same predicted labels used to obtain the best ACC. The counts, in original view order, are:

DatasetSelected (m1,…,mV)(m_1,\ldots,m_V)
EuroSAT(20,20,30,30)(20,20,30,30)
MSRC(21,14,21,7,21,14)(21,14,21,7,21,14)
RSICB128(135,45,135,45)(135,45,135,45)
YTF10(10,30,20,20)(10,30,20,20)
YTF20(60,40,20,20)(60,40,20,20)
YTF50(100,50,150,150)(100,50,150,150)
CIFAR10(10,10,10)(10,10,10)
CIFAR100(100,300,200)(100,300,200)

Anchor counts control the granularity of each view's representation. Different feature geometries can benefit from different granularities, so a common count can be suitable for some views while being less suitable for others.

The selected unequal counts improve YTF20 from 76.49% to 79.76% ACC and from 65.88% to 72.71% ARI; YTF10 improves from 76.74% to 84.86% ARI.

The revised MSRC-v6 comparison will use the same default environment as the other experiments. The earlier 95.71% ACC was obtained on an Intel Core i7-14700 with MATLAB R2023b. After correction, CAST still ranks first on MSRC-v6 in Table 1. The Table 4 comparison uses the following two configurations.

MethodAnchor countsACC / NMI / ARI (%)
CAST(21,21,21,21,21,21)(21,21,21,21,21,21)93.81 / 87.43 / 86.13
CAST-H(21,14,21,7,21,14)(21,14,21,7,21,14)95.24 / 90.69 / 88.73

The gains are 1.43, 3.26, and 2.60 percentage points, respectively.

Equal-count configurations are also included in the search. CIFAR10 selects (10,10,10)(10,10,10), the standard CAST configuration, and retains its 99.07% ACC. This is consistent with a common count already being sufficient within the tested grid and limited remaining performance headroom. CAST-H accommodates independently constructed heterogeneous graphs even when the selected configuration or a particular metric does not improve.

Reviewer_yBNBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

CAST presents a probabilistic approach to multi-view clustering in which a shared global class generates view-specific semantic states, which in turn account for directional observations obtained from sample-to-anchor probabilities. By learning class-conditional transition matrices, the method allows local class semantics to differ systematically across views and calibrates their evidence without imposing cross-view anchor correspondence. The latent variables admit exact posterior inference, leading to a computationally compact MAP-EM procedure. The paper also examines the discriminative capacity of non-identity transitions, the semantic information they preserve, the potential complementarity of calibrated views, and convergence of the bounded MAP objective values. Experiments on standard benchmarks, controlled semantic shifts, and transition-structure ablations evaluate the practical value of this modeling choice.

Critical strengths
  1. A key merit of CAST is that it develops the semantic-shift motivation into a flexible probabilistic model without making inference prohibitively expensive. Rather than assuming that every view expresses a global class through the same local semantic state, it models global and local states separately while still recovering a common clustering. The discussion in the introduction and Appendix A makes this distinction plausible, and the improvements over the Identity variant on some unmodified datasets provide empirical support. I particularly appreciate the decomposition of the transition probabilities: ρc(v)\rho_c^{(v)} represents the chance of leaving the corresponding local state, whereas Sc:(v)S_{c:}^{(v)} describes where such a shift goes. This preserves the expressiveness of a general row-stochastic transition matrix while making its parameters interpretable and amenable to distinct priors, as further explored by the ablations. The added latent semantic layer also has a manageable computational cost. The E-step computes exact global and global/local responsibilities without explicitly considering all CVC^V local-state combinations; the probability and prototype-direction updates have closed forms, and only a finite scalar concentration equation remains to be solved for each view.

  2. The theoretical results offer a useful explanation of when semantic disagreement across views need not prevent accurate clustering. Proposition 4.3 is particularly informative: even a transition that permutes local semantic states can retain all global class information at the latent-state level, despite having no diagonal preservation under the chosen labeling. Proposition 4.2 complements this observation by expressing class separation through δGδ⊤\delta G\delta^\top, thereby linking differences between transition rows to the overlap and geometry of local emission distributions. Together, these results clarify the conditions under which non-identity transitions still provide discriminative evidence. The Bhattacharyya bound additionally characterizes how calibrated evidence may accumulate across conditionally independent views. The analysis is rounded out by correspondence-free posterior inference, monotonic improvement and convergence of the bounded MAP objective values, and explicit computational complexity estimates.

Critical weaknesses
  1. Some statistical modeling decisions would benefit from more explanation, notably the use of a single concentration κ(v)\kappa^{(v)} across all local components of a view and the way the destination-prior profile is constructed. Although Table 3 contrasts γS=0\gamma_S=0 and γS=2\gamma_S=2, this comparison does not isolate the influence of the regularization strength from that of the pooled, data-derived destination profile. I regard these as relatively minor gaps in the justification of the model design.

  2. The positioning relative to recent work on robust multi-view clustering remains somewhat limited. In particular, [A] studies the recovery of underlying cross-view correspondences under noisy matching, while [B] considers incomplete views and unreliable correspondences together. These are not the same problem as CAST's calibration of class-level view-local semantic shifts, but discussing how the assumptions and mechanisms differ would make the scope and distinctiveness of CAST clearer.

  3. While the paper gives a helpful asymptotic cost analysis, it provides little direct evidence of the actual computational expense. Representative wall-clock times and iteration counts, especially on a larger benchmark, would make it easier to assess the practical efficiency of the proposed inference procedure.

Questions for authors
  1. What motivated the decision to use one concentration parameter κ(v)\kappa^{(v)} for all local semantic components in Equation (2)? Is the primary consideration statistical stability, fewer parameters, or computational simplicity? Since separate parameters κk(v)\kappa_k^{(v)} could also be updated through scalar equations, I would appreciate clarification of how shared concentration affects the distinction between within-component dispersion and cross-class semantic transitions, particularly when component dispersions differ substantially. 2 .Could the authors elaborate on the default choice γS=2\gamma_S=2? The ablations in Table 3 suggest that destination shrinkage is valuable in some cases but has limited or occasionally negative effects in others. How should these variations be understood? Because γS=0\gamma_S=0 disables the Dirichlet destination shrinkage while leaving the Beta prior on shift rates intact, it specifically tests one form of regularization. I am also interested in why the destination-prior profile is estimated by pooling information across source classes. If results with a uniform profile at the same γS\gamma_S are available, they could help distinguish the impact of the prior profile from that of its strength.
  2. The multi-view Bhattacharyya bound is informative under the assumed conditional independence of the calibrated views. When views are produced by similar feature extractors, how does their semantic agreement relate to the amount of complementary class evidence they provide? In particular, what might residual dependence conditional on the global class mean for fused posterior confidence, the recovered partition, or the estimated transition matrices? A short qualitative explanation would be sufficient.
  3. How does CAST compare conceptually with [A] and [B], which address uncertain cross-view correspondence and imperfect observations rather than class-level semantic transitions? It would be useful to clarify which assumptions these methods share with CAST, where their robustness mechanisms differ, and whether they address complementary sources of heterogeneity. If a compatible experimental protocol can be established, could the authors also consider an empirical comparison with [A] and [B]? Otherwise, an explanation of why direct numerical comparison would not be meaningful would help readers understand CAST's position relative to these approaches.

[A] Zhou et al. Uncover Underlying Correspondence for Robust Multi-view Clustering.

[B] Huang et al. Robust Multi-view Clustering against Imperfect Information.

9 comments

Response to Weakness 3:

The table below reports mean wall-clock times over repeated runs on all eight original datasets, including anchor-graph construction, CAST execution, and their combined total.

DatasetAnchor construction (s)CAST execution (s)Total (s)Anchor share
EuroSAT21.552.9424.4988.00%
MSRC-v61.190.091.2893.00%
RSICB128118.7632.02150.7778.77%
YTF1031.123.1434.2690.84%
YTF20132.0213.81145.8390.53%
YTF50605.38161.12766.5078.98%
CIFAR1045.711.8447.5596.13%
CIFAR100371.5295.26466.7879.59%

Across the eight datasets, CAST execution takes only 0.09–161.12 s, while anchor construction accounts for 78.77–96.13% of the total time. These measurements show that the semantic calibration and clustering stage itself is efficient, with the current anchor-construction strategy being the main runtime bottleneck.

The following table compares runtimes on four larger datasets.

MethodYTF20YTF50CIFAR10CIFAR100
LMVSC104.30329.5288.26210.71
FPMVS524.34—223.08—
OMSC266.80—194.45—
EMVGCLG293.05757.74228.47411.83
FDAGF736.53751.96959.881357.47
AEVC391.501363.57200.96601.94
ESTMC264.33477.28272.15578.25
MVS4C29.20139.8520.3970.83
RTGD178.84612.69204.49504.97
TMC399.141029.80192.38371.55
SLWSE239.04—286.82494.66
CAST (anchor construction + execution)145.83766.5047.55466.78

Together with Table 1, these results show that CAST maintains high computational efficiency while generally achieving better clustering performance. On YTF50, its total runtime is comparable to EMVGCLG and FDAGF, while its ACC, NMI, and ARI are higher. MVS4C prioritizes efficiency and avoids alternating optimization over multiple variables, making it faster, but its ACC, NMI, and ARI are lower than CAST's throughout our Table 1 evaluation.

On YTF50, CAST converged after 14 MAP-EM updates. MAP-EM permits at most 100 updates, with early stopping when

∣Jt−Jt−1∣max⁡1,∣Jt−1∣<10−6.\frac{|J_t-J_{t-1}|}{\max\\{1,|J_{t-1}|\\}}<10^{-6}.

Response to Question 1 and Weakness 1 (1/2):

Shared concentration.

Our main motivation for sharing κ(v)\kappa^{(v)} is to pool statistical evidence across local components and reduce the number of concentration parameters from VCVC to VV. Components with few effective members can therefore use information from the whole view. For local responsibilities rik(v)r_{ik}^{(v)}, define Nk(v)=∑irik(v)N_k^{(v)}=\sum_i r_{ik}^{(v)} and, for occupied components, Rk(v)=∣∑irik(v)ui(v)∣/Nk(v)R_k^{(v)}=\\|\sum_i r_{ik}^{(v)}\boldsymbol u_i^{(v)}\\|/N_k^{(v)}. The shared update uses the membership-weighted mean resultant length

=\sum_{k:N_k^{(v)}>0}\frac{N_k^{(v)}}{N}R_k^{(v)}.$$ Concentration controls angular dispersion within a local component, while $T_{ck}^{(v)}$ describes how global class $c$ distributes across local states. Separate concentrations also admit scalar updates and can model unequal dispersions. Sharing additionally reduces the number of concentration solves from one per component to one per view. If component dispersions differ substantially, the shared estimate can fit some components too sharply or too diffusely. This changes their posterior responsibilities, which in turn enter the transition updates. **Why we use Pooled.** We adopt Pooled because it gives sparse transition rows a shared destination preference inferred from the view. It aggregates initial off-diagonal evidence across source classes: $$\zeta_k^{(v)}=\sum_{a\ne k}N_{ak}^{(v,0)},\qquad \omega_{ck}^{(v)}=\frac{\zeta_k^{(v)}}{\sum_{j\ne c}\zeta_j^{(v)}}\quad(k\ne c),\qquad \omega_{cc}^{(v)}=0,$$ where $N_{ak}^{(v,0)}$ comes from initial leave-one-view-out global-to-local overlaps. Larger prior weights thus identify destinations supported by more initial transition evidence, giving the regularization a direct interpretation in terms of transition structure. This lets a sparse row borrow destination information from other source classes, while each learned row remains source-class dependent through its own posterior counts. The profile is computed once and fixed during MAP-EM, with a uniform fallback if the denominator is zero.

Response to Question 1 and Weakness 1 (2/2):

Regularization strength.

Pooled and Uniform differ only in how destination pseudo-counts are allocated; Uniform uses ωck=1/(C−1)\omega_{ck}=1/(C-1) for k≠ck\ne c. We set γS=2\gamma_S=2 across datasets and conditions, adding two total pseudo-counts per off-diagonal row as a weak prior. With posterior counts nck(v)n_{ck}^{(v)} and n−c(v)=∑k≠cnck(v)n_{-c}^{(v)}=\sum_{k\ne c}n_{ck}^{(v)}, the update for a positive denominator is

\frac{n_{ck}^{(v)}+\gamma_S\omega_{ck}^{(v)}}{n_{-c}^{(v)}+\gamma_S},\qquad k\ne c.$$ The prior weight is $\gamma_S/(n_{-c}^{(v)}+\gamma_S)$. When $\gamma_S\ll n_{-c}^{(v)}$, the data dominate this update and the choice of profile has limited influence. Here $n_{-c}^{(v)}$ is the effective off-diagonal count for the row. The effect can be stronger in small datasets or sparse rows and depends on how well the profile matches their destination tendencies. Posterior counts are recomputed during EM, allowing the prior to influence later responsibilities and the optimization trajectory. This feedback helps explain why a weak prior can sometimes produce a sizeable change in clustering performance. Both $\gamma_S=0$ and $\gamma_S=2$ retain the full transition parameterization and the Beta prior on shift rates. At $\gamma_S=0$, the destination regularization term vanishes, giving Pooled and Uniform the same objective. Table 3 reflects this variation: increasing $\gamma_S$ from 0 to 2 raises Euclidean-shift ACC from 82.1% to 93.8% on EuroSAT and from 87.4% to 97.6% on YTF10, while original CIFAR100 changes from 98.0% to 97.0%. Under controlled shifts, one or both full-model settings perform best among the transition variants in nearly all cases, supporting the effectiveness of the full parameterization. On original datasets where Identity leads, CAST remains a close second and competitive with the Table 1 baselines. This is consistent with views already expressing similar class structure, as can occur with different neural feature extractors. **Pooled versus Uniform at fixed strength.** We compared the profiles at $\gamma_S=2$ on six original datasets, using identical saved graphs, initial model parameters, and fitting settings. Both branches retained the original initialization, and only the fixed profile used in the subsequent MAP-EM objective and updates changed. Each profile was fitted once with seed 1, ten spherical K-means initialization starts per view, a 100-iteration MAP-EM limit, and tolerance $10^{-6}$ on Intel Core Ultra 9 386H / MATLAB R2025b. All fits converged. | Dataset | Pooled: ACC / NMI / ARI (%) | Uniform: ACC / NMI / ARI (%) | | -------- | --------------------------- | ---------------------------- | | MSRC-v6 | 93.81 / 87.43 / 86.13 | 94.29 / 88.49 / 87.15 | | EuroSAT | 89.77 / 81.01 / 79.26 | 89.76 / 81.01 / 79.25 | | RSICB128 | 79.16 / 88.40 / 78.64 | 79.14 / 88.38 / 78.60 | | CIFAR10 | 99.07 / 97.48 / 97.97 | 99.07 / 97.48 / 97.97 | | YTF10 | 88.41 / 84.18 / 76.74 | 88.41 / 84.17 / 76.74 | | YTF20 | 76.49 / 80.66 / 65.88 | 76.49 / 80.67 / 65.88 | Pooled is slightly higher on EuroSAT, RSICB128, and YTF10, identical on CIFAR10, and practically equivalent on YTF20, where each decrease is below 0.01 percentage points. All metric differences on these five datasets are below 0.04 percentage points. On MSRC-v6, Pooled is lower by 0.48, 1.06, and 1.02 percentage points in ACC, NMI, and ARI, respectively.
Reviewer_4vehOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper revisits what should be consistent across views in multi-view clustering. It retains a shared global partition but questions the stronger assumption that every global class must have the same local semantic realization in every view. CAST replaces this identity coupling with class-conditional semantic transition channels over independently constructed anchor representations. The introduction and Appendix A motivate how such discrepancies can arise, while experiments on unmodified benchmarks and controlled shifts investigate their practical relevance. The resulting framework accommodates unrelated anchor coordinates and different anchor counts, with a tractable probabilistic algorithm for jointly learning global assignments and view-local semantic relations.

Critical strengths
  1. The paper distinguishes a shared global class from its view-local semantic realization and questions whether these must be coupled through an identity relation. This is a meaningful conceptual contribution: systematic disagreement can still provide useful evidence for a common partition. The introduction and Appendix A explain plausible sources of such discrepancies, while Table 3 establishes their practical relevance through targeted original-data comparisons. For example, full CAST raises MSRC ACC from 72.4% for Identity to 95.7%, with the local emission model and inference procedure held fixed. The substantial gain on YTF10 provides further support. Together with the favorable behavior on shifted YTF datasets, these findings make a persuasive case for explicitly calibrating semantic relations. I regard this combination of a clearly articulated hypothesis and relevant empirical evidence as the main reason to support acceptance.

  2. CAST allows each view to retain its own anchor coordinates, cardinality, and upstream construction procedure. Views communicate through class-level semantic evidence, so the method does not require anchor matching or joint construction of a common anchor system. This is an appealing practical consequence of the formulation: a representation suitable for one view need not impose the same anchor identities or count on the others. Table 4 supports unequal cardinalities as a useful capability, showing that this flexibility can also improve clustering performance. The combination of semantic calibration and a modular anchor interface gives the framework value beyond a single graph-construction pipeline.

Critical weaknesses

None at the level of a rejection reason. The consequences of the ambient working likelihood and the interpretation of the learned semantic quantities could be explained more fully. I would also welcome a clearer account of the concentration-boundary choice and of the model's behavior when additional calibration offers little benefit. These are limited clarification points concerning a contribution I otherwise find convincing.

Questions for authors
  1. The square-root map has a clear Hellinger-geometric motivation, and the ambient vMF working-likelihood convention is explicitly acknowledged. Could the authors clarify the interpretation of relative local semantic evidence, concentration estimates, and T(v)T^{(v)} under this working model, given that observations occupy the positive orthant? A brief explanation of the statistical and computational implications of an orthant-normalized directional model or a distribution defined directly on the simplex would also be helpful. A conceptual discussion would suffice.

  2. Appendix B.7 relates the capped mean-resultant target to the direct bound Kv=Amv−1(1−ϵκ)K_v=A_{m_v}^{-1}(1-\epsilon_\kappa). Could the authors explain more intuitively why they parameterize the boundary through ϵκ\epsilon_\kappa rather than specifying a raw κmax⁡\kappa_{\max}, particularly when anchor dimensions differ across views? I recognize that imposing the same induced KvK_v directly gives the same constrained update; the question concerns the practical choice between a common resultant threshold and a common raw concentration cap. How often is the bound active in the experiments? Any existing observations about sensitivity to modest changes in ϵκ\epsilon_\kappa would also be useful.

  3. Could a compact example illustrate the complementary roles of Propositions 4.2 and 4.3, for instance a transition channel retaining considerable class information while overlapping local emissions still limit observation-level class distinguishability? How does this distinction relate to the single-view ACC diagnostic? For Figures 2–3, please specify how the single-view labels are obtained, whether mean preservation is averaged from sample posteriors or from transition diagonals with particular class weights, and how semantic information is normalized. Clarifying what these quantities measure and connecting their observed agreement or divergence to the two propositions would make the diagnostic interpretation more accessible.

  4. On original EuroSAT, RSICB128, and CIFAR10, where Identity is competitive, does full CAST learn predominantly preserving channels, or can nontrivial transitions remain while the global partition stays similar? A brief interpretation using existing preservation statistics would be helpful; exact recovery of T=IT=I is not expected. Relatedly, why is YTF10/20's fourth view more diagonal-dominant in Figures 4–5 than any MSRC view? How much does this reflect which views were perturbed, their original representations, or both? Please relate this pattern to the available preservation and information diagnostics, without equating diagonal dominance alone with discriminative quality.

6 comments

Response to Weaknesses:

We address the implications of the ambient working likelihood and the interpretation of its fitted quantities in Q1. Q2 covers the concentration boundary and its observed activation, while Q3 relates the learned semantic quantities to the diagnostics. Q4 explains CAST's behavior when additional calibration offers little benefit.

Response to Question 1:

The square-root map gives the anchor probabilities an exact Hellinger-geometric representation: H2(z,z′)=1−(z)⊤z′H^2(\boldsymbol z,\boldsymbol z')=1-(\sqrt{\boldsymbol z})^\top\sqrt{\boldsymbol z'}. It is invertible through z=u⊙u\boldsymbol z=\boldsymbol u\odot\boldsymbol u, so it preserves the information in the anchor probabilities. CAST then scores these positive-orthant embeddings with an ambient vMF working likelihood, normalized on the entire sphere as specified in Equation (2). Within each view, relative local evidence is determined by angular similarity. Because concentration is shared, the vMF normalizer cancels between local components:

=\kappa^{(v)}(\boldsymbol\mu_k^{(v)}-\boldsymbol\mu_{k'}^{(v)})^\top\boldsymbol u.$$ Here $\kappa^{(v)}$ controls the sharpness of the angular evidence. When estimating concentration, the vMF normalizer remains part of the likelihood and yields the mean-resultant equation used in the scalar update. The row $T_{c:}^{(v)}$ gives the distribution of local semantic states for global class $c$, and $\psi_c^{(v)}(\boldsymbol u)=\sum_k T_{ck}^{(v)}f_k^{(v)}(\boldsymbol u)$ combines their evidence for that global class. Concentrations and transition probabilities are therefore fitted quantities of the ambient working model. The exact geometry of the embedding and the chosen statistical likelihood play separate roles in this interpretation. Conditioning the likelihood on that orthant would give $f_{k,+}^{(v)}=f_k^{(v)}/Z_k^{(v)}$, where $Z_k^{(v)}$ is its component-specific orthant mass. The evidence ratio would acquire $-\log Z_k^{(v)}+\log Z_{k'}^{(v)}$. These masses depend on direction and concentration, so the correction can change relative component evidence, posterior responsibilities, and consequently the fitted concentrations and transitions. Computationally, such a model requires evaluating the orthant masses and their parameter dependence in the emission updates, replacing the current closed-form direction updates and scalar concentration equation with updates derived for that likelihood. A distribution defined directly on the simplex would also change the observation model. For example, a Dirichlet mixture scores log probability coordinates and has its own parameter updates and boundary behavior, including the treatment of zero anchor probabilities. The ambient vMF choice gives a tractable directional model tied to the Hellinger embedding; an orthant-conditioned or simplex likelihood would attach a different statistical interpretation to the fitted semantic quantities.

Response to Question 2:

A common ϵκ\epsilon_\kappa specifies the same distance from perfect expected directional alignment in every view. The mean-resultant function Am(κ)A_m(\kappa) has range [0,1)[0,1), so the threshold 1−ϵκ1-\epsilon_\kappa has a common, dimensionless interpretation even when anchor dimensions differ. Its induced concentration bound Kv=Amv−1(1−ϵκ)K_v=A_{m_v}^{-1}(1-\epsilon_\kappa) adapts to the anchor dimension. A common raw κmax⁡\kappa_{\max} would give different alignment limits Amv(κmax⁡)A_{m_v}(\kappa_{\max}). As Appendix B.7 shows, directly imposing the corresponding KvK_v gives the same constrained update. Parameterizing it through ϵκ\epsilon_\kappa makes the alignment threshold comparable across views. The update solves Amv(κ^(v))=min⁡Rˉ(v),1−ϵκA_{m_v}(\widehat\kappa^{(v)})=\min\\{\bar R^{(v)},1-\epsilon_\kappa\\}, keeping concentration finite as Rˉ(v)\bar R^{(v)} approaches one.

The saved diagnostics summarize the concentration solves from one complete seed-1 fit on each of the eight original datasets at ϵκ=10−3\epsilon_\kappa=10^{-3}. The table gives the recovered maximum pre-cap resultant and the number of solves reaching the ceiling of 0.999, separating initialization from MAP-EM.

DatasetMaximum pre-cap Rˉ\bar RInit.: at-cap / solvesMAP-EM: at-cap / solves
EuroSAT0.8364550/80/80
MSRC-v60.9620630/120/324
RSICB1280.8406450/80/72
YTF100.9986400/80/72
YTF200.9009560/80/52
YTF500.8879020/80/56
CIFAR100.8946690/60/24
CIFAR1000.7733300/60/39
All records0.9986400/640/719

All 783 solves were below the ceiling, and all eight fits converged. With 60 bisection refinements, the largest absolute root residual was approximately 4.01×10−144.01\times10^{-14}. The bound therefore provided numerical protection without clipping any concentration target in these runs. Reducing ϵκ\epsilon_\kappa would relax this inactive upper bound and leave the constrained concentration optima unchanged. Increasing it tightens the allowable range and can affect estimation once the bound becomes active.

Response to Question 3:

The propositions describe successive stages of Y→L(v)→U(v)Y\to L^{(v)}\to U^{(v)}. Proposition 4.3 quantifies the class information retained by the transition channel. Proposition 4.2 combines differences between transition rows with the Gram matrix of the local emissions, thereby accounting for whether those local states can actually be distinguished from observations.

For example, let π=(1/2,1/2)\pi=(1/2,1/2) and T=(01\10)T=\begin{pmatrix}0&1\\\1&0\end{pmatrix}. This channel has zero diagonal preservation but retains all class information: I(Y;L)=H(Y)=log⁡2I(Y;L)=H(Y)=\log2. The induced observation densities are ψ1=f2\psi_1=f_2 and ψ2=f1\psi_2=f_1. If the emissions satisfy f1=f2=ff_1=f_2=f, these densities are identical, so I(Y;U)=0I(Y;U)=0 and population Bayes accuracy is 1/21/2. In Proposition 4.2, G=∥f∥L2211⊤G=\|f\|_{L^2}^2\boldsymbol{1}\boldsymbol{1}^\top and δ=(−1,1)\boldsymbol{\delta}=(-1,1) give δGδ⊤=0\boldsymbol{\delta}G\boldsymbol{\delta}^\top=0. Thus, information retained by the latent channel can coexist with indistinguishable observations. With well-separated emissions instead, the same swapped channel can support accurate observation-level classification after relabeling, while its diagonal preservation remains zero.

Figures 2–3 use the following diagnostics:

  • View-wise ACC: hard spherical K-means labels obtained from that view's square-root anchor embedding during CAST initialization, optimally matched to ground-truth labels for evaluation. The plotting routine reuses these exact initialization labels. This measures recoverability from one view under the specified clustering procedure.

  • Mean preservation: the sample average

    \xi_{ick}^{(v)}=P(Y_i=c,L_i^{(v)}=k\mid\boldsymbol u_i^{(1)},\ldots,\boldsymbol u_i^{(V)}).$$ Samples have equal weight; class contributions follow their posterior responsibilities. These posteriors condition on all views. The class-average diagonal $C^{-1}\sum_c T_{cc}^{(v)}$ and model-prior-weighted diagonal $\sum_c\widehat\pi_cT_{cc}^{(v)}$ are separate summaries; the plotted preservation is computed from the sample posteriors above.
  • Normalized semantic information: Iπ^,T^(Y;L(v))/H(π^)I_{\widehat\pi,\widehat T}(Y;L^{(v)})/H(\widehat\pi), calculated from the fitted prior and channel. Writing p^k(v)=∑cπ^cT^ck(v)\widehat p_k^{(v)}=\sum_c\widehat\pi_c\widehat T_{ck}^{(v)}, the numerator and denominator are

    =\sum_{c,k}\widehat\pi_c\widehat T_{ck}^{(v)} \log\frac{\widehat T_{ck}^{(v)}}{\widehat p_k^{(v)}},\qquad H(\widehat\pi)=-\sum_c\widehat\pi_c\log\widehat\pi_c.$$ Zero-mass terms contribute zero, and the normalized diagnostic is set to zero when $H(\widehat\pi)=0$. The information calculation uses the fitted latent-channel model; ground-truth labels enter the ACC evaluation.

Figures 2–3 show broadly similar trends across these diagnostics, consistent with informative channels also supplying useful local evidence. Their differences have specific interpretations. Emission overlap can reduce single-view recoverability even when the channel is informative. Systematic semantic reassignment can lower preservation while retaining information, as in the swapped-channel example. ACC uses optimal label matching, whereas preservation uses the fitted global/local index convention. Class weighting and the use of all views in posterior preservation can also produce differences. Together, the diagnostics distinguish semantic-index agreement, channel informativeness, and empirical recoverability from the observations.

Comments on the paper

3 comments

Dear Reviewers and Authors,

As a quick reminder, the author-reviewer discussion period will close on November 18.

@Reviewers: Please make sure to read the authors' rebuttals, acknowledge their responses, and engage in further discussion if any points remain unclear. While final decisions are not made at this stage, we ask that you carefully re-evaluate your ratings in light of the new clarifications and experimental results:

Increase your score if the authors have satisfactorily resolved your concerns and strengthened the paper.

Decrease your score if new, unaddressed issues have emerged during the discussion.

Maintain your score if critical concerns remain unresolved.

Please note that you are under no obligation to change your rating, provided your final score accurately reflects the current state of the manuscript. To ensure a fair evaluation, please ground your assessment in concrete evidence rather than personal preference, and consider whether the work offers meaningful scientific value to the ICLR community.

Thank you very much for your time, thoughtful engagement, and contributions to the review process.

Best regards,

Area Chair

AuthorsOpenReview ↗

Summary for the Area Chair and Reviewers

Thank you to the Area Chair and all three reviewers for the discussion. CAST asks whether views can still support a common partition when some samples from a global class systematically appear in other local states in a particular view. Both feature extraction and independent construction of each view’s anchor graph can introduce such differences. Our probabilistic model learns a transition matrix for each view, with entries Tck(v)=Pr⁡(L(v)=k∣Y=c)T^{(v)}_{ck}=\Pr(L^{(v)}=k\mid Y=c). Each row gives the probability of retaining the corresponding local state and shows which destinations samples of that class favor when they shift. For example, in one view some samples from class A retain local state A while others tend toward B. Some samples from class C may also shift toward B. Such systematic disagreement can itself inform clustering. For an existing feature extractor, users can also identify the global classes its representation tends to preserve and those whose samples tend to appear in other local states.

Theory and positioning. Two classes can share a destination and still have different preferences over local states. If all global classes have the same transition distribution in a view, its local state carries no information about the global class (Proposition 4.3). How well observations distinguish those two classes also depends on the overlap between the local emission distributions (Proposition 4.2). When the views are conditionally independent given the global class, views that distinguish different class pairs can provide complementary evidence (Theorem 4.4). The MAP-EM objective values are nondecreasing and converge to a finite limit (Theorem 4.5). Mainstream multi-view clustering methods start from cross-view agreement and seek shared class structure by fusing complementary features, aligning representations, or learning a consensus graph. These approaches often implicitly assume that a global class has consistent local semantics across views. CAST relaxes this assumption and models class-level semantic shift directly. It learns the transition probabilities from each global class to the local states of each view, using systematic disagreement to infer a shared partition.

Performance on original and shifted data. In Table 1, CAST ranks first in ACC on seven of the eight original datasets when compared with 11 baselines. Many baselines show large drops in ACC under controlled shifts. When CAST’s ACC does fall, the drop is generally smaller. On five of the eight datasets, both Euclidean relocation and spherical rotation increase CAST’s ACC relative to its original-data result. It ranks first in ACC on all eight datasets under both shifts. Table 3 compares CAST with Identity (T(v)=IT^{(v)}=I) using the same anchor graphs and observation model. On both MSRC-v6 and YTF10, each view comes from a different handcrafted feature descriptor. On the original data, learning the transitions raises ACC from 81.9% to 93.8% on MSRC-v6 and from 79.7% to 88.4% on YTF10. These gains suggest that systematic semantic shifts can arise naturally across descriptor-based views and that modeling them can improve clustering. Under Euclidean relocation on YTF10, CAST’s ACC rises further to 97.6% while Identity’s falls to 74.6%. Where Identity performs well, one possible explanation is weak or absent systematic semantic shift. In such cases, the full CAST model can tie Identity for the best result on all three metrics. CAST can therefore be used without first determining whether semantic shift is present.

Practical use. CAST fuses information at the class level, so each view can use an independently constructed anchor graph with its own anchor coordinates and count (Proposition 4.1). This flexibility can also improve clustering. On YTF20, selecting the anchor count separately for each view raises ACC from 76.49% to 79.76% and ARI from 65.88% to 72.71% (Table 4). CAST computes exact posterior probabilities by summing over each view’s local states separately. Both the time per iteration and the working memory needed for fitting scale linearly with sample count. On YTF50 (126,054 samples), fitting CAST took 161.12 seconds on average.

AuthorsOpenReview ↗

Reviewer comments and our responses

  • 4veh asked about the directional likelihood, the concentration cap, and the semantic diagnostics. We explained how to interpret the fitted parameters and checked all 783 concentration solves across eight fits, one on each original dataset. None reached the cap. We defined the diagnostics and reported per-view semantic preservation, normalized semantic information, and summary statistics for the transition-matrix diagonals on the original EuroSAT, RSICB128, and CIFAR10 data. We explained the differences in the YTF and MSRC plots in terms of the original representations and the perturbations applied to each view. The reviewer later confirmed that the key questions had been answered.
  • ft6a asked about experimental reproducibility and when using different anchor counts across views helps. We specified the shift settings and affected views, the order of preprocessing and anchor-graph reconstruction, the rule for selecting configurations, and the randomization procedure. We also listed the selected anchor counts for each view on all eight datasets and explained why a common count was sufficient on CIFAR10.
  • yBNB asked about the shared concentration parameter, the prior over shift destinations, dependence between views, related work, and runtime. We explained why pooling information across local components within a view helps estimate their shared concentration, compared Pooled and Uniform at the same prior strength, and explained how counting some evidence more than once can affect posterior confidence and transition estimates. We distinguished CAST’s class-level semantic calibration from the uncertain sample correspondences studied by CorreGen and PLCI. We also provided timings for eight datasets and the iteration count on YTF50. The reviewer later confirmed that the replies had addressed their main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.