acceptodds
Under review as a conference paper at ICLR 2027

A Nonlinear Kernel Spectral Perspective on Multi-View Contrastive Learning

Abstract

Multi-view contrastive learning has become a powerful paradigm for exploiting cross-view consistency, yet most existing methods treat contrastive objectives primarily as representation-level regularizers within larger learning pipelines. The spectral structure induced by these objectives remains less understood. We present a nonlinear kernel spectral perspective on multi-view contrastive learning and derive MVCL-KS, a direct spectral realization of this perspective. Starting from bidirectional InfoNCE, we show that its exact dot-product gradient induces a signed cross-view relation operator; aggregating all view pairs yields a unified multi-view block spectral system, which is further lifted to view-specific reproducing kernel Hilbert spaces. Because the contrastive probabilities depend on the current representation, the resulting formulation is a nonlinear self-consistent spectral system. This formulation leads to dominant-eigenspace spectral refinement with basis-invariant cycle-aware stabilization, together with low-rank and anchor-based implementations for scalability. Under the same evaluation setting, MVCL-KS is to faster than the fastest competing multi-view baseline across all benchmarks, while maintaining competitive or superior clustering quality. Experiments with multiple kernel families further show that the same spectral formulation works with different nonlinear view geometries.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
contrastive learning, spectral learning, kernel methods, multi-view clustering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fcaVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MVCL-KS, a multi-view clustering framework that interprets contrastive learning as signed cross-view relationships and combines them with view-specific kernel geometry. Its central idea is to refine the clustering representation through a self-consistent spectral system. Experiments on conventional benchmarks show strong ACC and substantially faster training, though NMI and ARI results are less consistent and the validation offers limited insight into how the proposed mechanism works during learning.

Overall: an interesting, efficient framework with strong ACC results, but inconsistent NMI/ARI and limited empirical evidence for its core refinement mechanism. Borderline, leaning reject, but open to revision after rebuttal.

Critical strengths
  1. The signed-operator formulation explicitly models cross-view attraction and repulsion while incorporating view-specific kernel geometry. This provides a useful perspective on multi-view contrastive learning and its connection to spectral methods.

  2. MVCL-KS demonstrates an advantage in terms of ACC across multi-view benchmarks while substantially reducing runtime compared with existing methods.

Critical weaknesses
  1. Across datasets, MVCL-KS demonstrates advantages in terms of ACC, but its performance on ARI and NMI remains inconsistent. For instance, a 10.11% performance gap in ARI is observed on NUS-WIDE. These results raise questions about the generalizability in clustering quality. Further analysis of the reasons behind these discrepancies would help clarify the strengths and limitations of MVCL-KS.

  2. The experimental validation is limited mainly to standard sensitivity analyses and component ablations. It does not show how the cross-view probabilities QtQ_t, signed weights StS_t, or leading spectral subspace evolve during refinement. Tracking positive-pair and hard-negative probabilities, one- and two-step subspace distances, and ACC/NMI/ARI across iterations would clarify whether the proposed dynamics approach self-consistency and improve clustering beyond the initial kernel embedding.

  3. Multi-view data have become increasingly prevalent in real-world applications, such as autonomous driving perception and multimodal medical analysis. The experiments here focus on established clustering benchmarks such as 3Sources, BBCSport, and Caltech. These benchmarks support comparison with prior work, but leave the method’s applicability to more complex multimodal settings unclear. A discussion of view alignment and kernel construction costs in such settings would strengthen its practical motivation. I recognize that a full application study may be challenging, so I offer this as a suggestion rather than a requirement for this paper.

  4. The framework diagram could be substantially improved. It is visually cluttered and largely repeats equations from the text, offering little intuition about the role of each component or how information flows through the method.

Questions for authors

Please refer to Major Weakness.

1 comment

We thank the reviewer for identifying the metric inconsistency and requesting direct evidence of the refinement mechanism. We have corrected the affected comparisons and added iteration diagnostics to the main text.

W1 — Clustering consistency.

We identified a column-assignment error in the original submission, where the ARI and F1 values for RMCNC and ROLL were interchanged. We have corrected the affected entries in Tables 1, 2, and 4 and verified them against the original evaluation records. All MVCL-KS results remain unchanged. We apologize for this reporting error.

With this correction, the previously reported 10.11-point ARI deficit on NUS-WIDE no longer holds. The corrected ARIs are 8.77% for RMCNC and 6.26% for ROLL, compared with 17.54% for MVCL-KS. Our method therefore ranks second in ARI, only 1.11 percentage points below OMVCDR (18.65%), while retaining the highest ACC (39.88%). These corrected results provide a more accurate assessment of MVCL-KS across clustering metrics.

W2 — Refinement dynamics and gains beyond initialization.

New Figure 3 (Section 5.3) directly tracks positive-pair and hard-negative probabilities, one- and two-step subspace distances, and ACC across refinement iterations. BBCSport meets the one-step tolerance at t=2t=2, while 3Sources and Caltech meet the two-cycle criterion at t=7t=7 and t=4t=4. The remaining datasets reach the iteration limit. These diagnostic trajectories continue for all 15 raw updates, with markers indicating the first stopping decision. Table 8 separately evaluates the standard stopping and stabilization procedure, which improves ACC over initialization on all six datasets by 0.19–1.74 percentage points.

W3 — Multimodal applicability.

Section 4.2 now states the sample-level correspondence requirement and explains how view-specific kernels accommodate different feature dimensions. It also gives the dense kernel construction cost, O(N2∑vdv)O(N^2\sum_v d_v), the current kernel eigendecomposition cost, O(VN3)O(VN^3), and the anchor-based alternative. For sensor or medical inputs, correspondence is supplied by acquisition or preprocessing.

W4 — Motivation and framework figures.

New Figure 1 explains how target–prediction mismatch produces signed attraction–repulsion corrections. Revised Figure 2 shows how these corrections combine with view-specific kernels in a joint spectral operator and how the resulting embedding updates the probabilities. The figures now emphasize component roles and information flow.

Reviewer_wAnFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MVCL-KS, which derives a signed cross-view operator from the gradient of bidirectional InfoNCE and unifies all view pairs within a single kernelized multi-view spectral system. Because the contrastive probabilities depend on the current representation, the method alternates between updating the operator and estimating its dominant eigenspace. The framework also includes two-cycle detection, projector averaging, low-rank factorization, and an anchor-based Nyström extension.

Critical strengths
  1. The paper derives a signed operator from the exact InfoNCE gradient, making a clear connection between positive-pair attraction, negative-sample competition, and multi-view spectral structure. The motivation is easy to follow, and the different components fit together coherently.
  2. View-specific kernels allow different views to retain their own geometry, while the low-rank formulation and anchor-based Nyström extension avoid the repeated training required by deep networks. The runtime gains reported in the experiments are substantial and suggest good practical value.
  3. The experimental evaluation is fairly comprehensive. In addition to standard clustering comparisons, the paper includes experiments with multiple kernel families, ablation studies, parameter sensitivity analyses, and runtime results.
Critical weaknesses
  1. The paper should more clearly articulate its substantive novelty relative to the most closely related work on spectral contrastive learning and kernel-based gradient equivalence. The InfoNCE gradient, Rayleigh–Ritz eigendecomposition, RKHS coordinate transformation, Nyström extension, and spectral perturbation analysis are all individually well-established. The main contribution appears to be the integration of these components into a signed kernel spectral system for multi-view learning, but the paper does not yet make sufficiently clear what distinctive property or capability this system provides beyond a meaningful combination of existing tools.
  2. The main results and ablation studies report only single-run values, without means or standard deviations. Performance is also relatively weak on some datasets. For example, ACC above 72% has been reported by a number of methods on 3Sources, whereas MVCL-KS obtains 65.09%. This makes the claimed empirical advantage less convincing.
Questions for authors

The final representation is obtained by simply averaging the view-specific blocks. Have the authors evaluated settings with low-quality or noisy views? How much would the results change if reliability-based weighting were used, or if some views were removed?

1 comment

We thank the reviewer for recognizing our coherent formulation and runtime advantage. We have clarified the distinctive properties of our method, the evaluation setup, and its behavior under degraded views.

W1 — Distinctive property of the combined system.

MVCL-KS couples all view-pair contrastive corrections through one signed kernel operator whose probabilities depend on the shared spectral representation. The resulting joint eigensystem imposes simultaneous constraints across views (Section 2.3 and Appendix E.3). We also added Proposition 5, Appendix E.6, which proves that the dominant-eigenspace step exactly minimizes a frozen surrogate that matches the original InfoNCE objective in value and first-order gradient at the current iterate. This identifies both the joint coupling and the local contrastive information preserved by the spectral update.

W2 — Statistical stability and clustering performance

We have added Appendix C.6 (Stability of the Final Clustering Results) and Table 9 to evaluate sensitivity to K-means initialization. Specifically, we repeat K-means ten times with different random seeds while keeping the learned representations fixed for both MVCL-KS and BMvC. The mean and standard deviation of ACC and NMI (%) are reported below.

DatasetMVCL-KS ACCBMvC ACCMVCL-KS NMIBMvC NMI
3Sources64.14±0.9256.98±5.4561.77±1.3356.15±2.56
BBCSport88.24±0.0080.70±8.3378.45±0.0064.30±7.83
Caltech82.76±0.5280.25±3.5371.03±0.3978.21±1.41
100leaves89.39±0.8986.29±1.7595.44±0.4294.60±0.35
NUS-WIDE37.96±1.1435.26±0.7520.62±0.3420.94±0.40
Notting-Hill87.34±5.7780.57±1.1181.94±5.2272.26±2.41

MVCL-KS achieves higher mean ACC on all six datasets, with lower ACC standard deviations on four of them. Notably, on 3Sources, MVCL-KS obtains 64.14±0.92% ACC, compared with 56.98±5.45% for BMvC. These results demonstrate that its clustering performance is not dependent on a single favorable K-means initialization.

Regarding the reported ACC on 3Sources, we emphasize that our main experiments use a fixed RBF kernel family across all six datasets and consistent label-free stopping criteria with a maximum of 15 refinement iterations. We do not select checkpoints based on ground-truth ACC, as such selection would introduce label information into an otherwise unsupervised evaluation.

Importantly, the reported results do not reflect the best achievable performance of MVCL-KS. Table 7 demonstrates the benefits of alternative kernel configurations.

Kernel setting3SourcesBBCSportCaltech100leavesNotting-HillNUS-WIDE
Default RBF ( MVCL-KS)65.0988.2483.4391.3889.2539.88
Best-tested kernel68.0588.2483.4395.3893.4342.69

For example, kernel adaptation improves ACC from 65.09% to 68.05% on 3Sources and from 91.38% to 95.38% on 100leaves. We deliberately keep these selected configurations separate from the main results to preserve a transparent evaluation setup.

Therefore, our primary comparison emphasizes a practical balance between clustering quality, computational efficiency, and label-free model selection, while achieving the reported 1.7–22.1× runtime advantage.

Q — Equal averaging with a noisy view.

We added a Gaussian-noise experiment (see Appendix C.4, Figure 4) that perturbs only the first view at levels 0.0–0.9, leaving the others unchanged. MVCL-KS achieves higher ACC in 35 of the 36 dataset–noise settings. Its six-dataset mean ACC declines by 2.77 percentage points between levels 0.0 and 0.9, versus 10.75 points for BMvC. Equal averaging isolates the contribution of the signed kernel-spectral mechanism under this single-view perturbation.

The raw ACC (%) values for the added figure on Gaussian-noise robustness are provided below.

Datasetsmethods0.00.10.30.50.70.9
3SourcesBMvC59.1456.8049.7055.6250.8949.11
3SourcesMVCL-KS65.0962.7263.3168.6465.0969.23
BBCSportBMvC82.6450.7459.5658.6461.2152.57
BBCSportMVCL-KS88.2488.4287.1386.0385.1184.38
Notting_HillBMvC85.3180.3483.2684.8979.7984.29
Notting_HillMVCL-KS89.2589.0390.4190.9090.7190.60
Reviewer_DkhdOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MVCL-KS, a kernel-based spectral method for multi-view contrastive learning.

The author reformulates InfoNCE gradients into a unified spectral system and introduces an iterative refinement algorithm to improve efficiency.

Experiments on eight datasets show competitive performance with reduced runtime.

Critical strengths

The paper provides a clear mathematical formulation that connects multi-view contrastive learning with spectral methods. The theoretical derivation is relatively complete and well-organized.

The method achieves strong clustering performance while significantly reducing runtime, with reported speedups of 1.7× to 22.1× over competing methods.

Critical weaknesses
  1. Previous studies have already connected contrastive learning with spectral clustering and kernel-based optimization, such as Tan et al. (ICLR 2024) and Sui et al. (ICLR 2026). Although this paper extends these ideas to a unified multi-view spectral system, it could better explain what fundamentally new insights this formulation provides.

  2. The connection between the theoretical formulation and the proposed algorithm needs further justification. While the authors derive an exact gradient representation of InfoNCE, the subsequent spectral refinement solves a conditional eigenvalue problem rather than directly optimizing the original objective. The paper could better explain what properties of contrastive learning are preserved through this reformulation and why this particular spectral procedure is appropriate.

Questions for authors
  1. Compared with Tan et al. (ICLR 2024) and Sui et al. (ICLR 2026), what is the most important new theoretical insight provided by this work? I mean "theoretical". Could the authors identify which results go beyond extending existing spectral or kernel-based formulations to the multi-view setting?

  2. The proposed refinement repeatedly solves a spectral problem with a fixed operator, while the operator itself depends on the current representations. Beyond the established connection between fixed points and stationary points, what justifies using dominant-eigenspace replacement to optimize the original InfoNCE objective?

1 comment

We thank the reviewer for the focused questions about theoretical novelty and the justification of the spectral update. The revision makes the joint coupling explicit and adds a first-order-consistent surrogate result.

W1 / Q1 — Joint spectral compatibility beyond pairwise contrastive alignment.

We appreciate the reviewer's request for a more precise theoretical distinction. Tan et al. (ICLR 2024) establish a connection between InfoNCE and spectral clustering, while Sui et al. (ICLR 2026) develop gradient-equivalent kernelized contrastive alignment with spectral updates. Our contribution is not the gradient identity, kernelization, or eigendecomposition individually, but the joint spectral compatibility structure induced by multiple interacting contrastive objectives.

The central theoretical insight is that pairwise contrastive alignment does not generally yield a globally consistent multi-view embedding. In MVCL-KS, all bidirectional InfoNCE gradients induce a single symmetric, signed VV-partite operator. Its kernelized conditional eigenspace must satisfy, simultaneously for every view uu,

=Z_{t+1}^{(u)}\Lambda_t, \qquad \sum_{u=1}^{V}Z_{t+1}^{(u)\top}Z_{t+1}^{(u)}=I_C.$$ **These equations reveal a coupling absent from independently solved view-pair spectral problems.** Every view participates in multiple contrastive interactions, yet all views must satisfy one global orthogonality constraint and share the same symmetric spectral multiplier $\Lambda_t$. Consequently, individually optimal pairwise embeddings need not satisfy the joint system. While the two-view case reduces to a bipartite spectral/SVD formulation, the general multi-view case requires a globally coupled eigenspace. Moreover, the operator is constructed from the current bidirectional InfoNCE predictions, so **the shared embedding and all pairwise contrastive relations jointly determine one another**. This characterizes a representation-dependent multi-view spectral system rather than a collection of independent alignment problems. We have made these structural properties explicit in **Section 2.3 and Appendix E (Joint View-Coupling Conditions)**. The additional first-order-consistency result in **Proposition 5** further characterizes the conditional spectral update, as discussed in our response to W2/Q2. **W2 / Q2 — What is preserved by the spectral reformulation, and why is spectral refinement justified?** **The key distinction is that our spectral refinement preserves the exact first-order structure of InfoNCE, rather than directly minimizing the original nonlinear objective.** Specifically, the signed operator retains the positive-pair attraction and softmax-weighted negative competition induced by the InfoNCE gradient, while the kernelized formulation expresses these interactions jointly across views. To further justify the update, we added a proposition establishing **first-order consistency between InfoNCE and the conditional spectral surrogate**. Let $\Phi(Z)=\mathcal L_{\mathrm{MVC}}(\mathcal K^{1/2}Z)$ and freeze $\mathcal M_t=\mathcal M(Z_t)$. We define $$\mathcal Q_t(Z)=\Phi(Z_t)-\frac{\operatorname{Tr}(Z^\top\mathcal M_t Z)-\operatorname{Tr}(Z_t^\top\mathcal M_t Z_t)}{4N\tau}.$$ The proposition establishes $$\mathcal Q_t(Z_t)=\Phi(Z_t),\qquad \nabla\mathcal Q_t(Z_t)=\nabla\Phi(Z_t) =-\frac{1}{2N\tau}\mathcal M_t Z_t.$$ Thus, **the surrogate matches both the objective value and the exact gradient of InfoNCE at the current representation**. By the Rayleigh–Ritz theorem, its global minimizer under the orthogonality constraint is the dominant algebraic eigenspace of $\mathcal M_t$, restricted to the effective RKHS range. Dominant-eigenspace replacement therefore provides a principled solution to a **first-order-consistent conditional spectral problem**. Importantly, MVCL-KS seeks spectral self-consistency as the representation-dependent operator evolves. **Figure 3 (Section 5.3)** illustrates both convergence and oscillatory behavior, while **Table 8** demonstrates improvements over the initial kernel embedding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.