acceptodds
Under review as a conference paper at ICLR 2027

Identifiability of Nonlinear Multi-view CCA under Partial Latent Observability: Limits and Guarantees of Recovery

Abstract

We investigate the identifiability of nonlinear multi-view canonical correlation analysis (CCA) under partial latent observability. We formally show that existing pairwise identifiability does not guarantee simultaneous source recovery using a single encoder per view. When the representation dimension matches the source dimension, sum-of-correlations (SUMCOR), a generalized CCA objective variant, can favor higher-order modes over first-order source directions, causing representation drift. Under isotropic Gaussian prior and full cross-view latent coverage, we establish three recovery regimes at population global optima. First, global first-order dominance guarantees recovery of each view-specific source up to an affine transformation. Second, known overlap ranks enable Top- and masked SUMCOR to achieve the same recovery under pairwise first-order dominance. Third, even without either dominance condition, SUMCOR admits exact affine source readouts at finite, spectrum-dependent representation dimensions under a joint spectral compatibility condition. The enlarged representations retain source directions alongside stronger nonlinear modes. Our results demonstrate how spectral structure, rank information, and representation capacity govern source recovery. Synthetic experiments confirm the predicted recovery regimes and representation drift.

Primary area:
learning theory
Keywords:
Canonical Correlation Analysis, Identifiability Analysis, Self-supervised Learning, Partial Latent Observability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7UV8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies source recovery in nonlinear multi-view CCA, focusing on the conditions under which optimizing SUMCOR leads to recovery. It considers three cases: 1) recovery under global first-order dominance, 2) recovery using rank-constrained objectives under pairwise first-order dominance and known overlap ranks, and 3) recovery using finite overcomplete representations under joint spectral compatibility. These results highlight the role of first-order dominance in source recovery.

Critical strengths
  1. The paper analyzes the global optimality properties of SUMCOR for source recovery in nonlinear multi-view CCA.

  2. It discusses three recovery regimes under different conditions and provides experiments supporting the theoretical results.

Critical weaknesses
  1. The recovery guarantees seem to rely heavily on the first-order dominance assumptions. However, it is not clearly explained why source recovery requires the stated assumptions (Assumptions 2, 4, and 5) or what role each assumption plays in establishing the corresponding theorems. The authors should provide more insights or high-level interpretations.

  2. The criterion for successful recovery needs clearer explanation. How should readers determine whether optimizing Equation (2) successfully recovers si\mathbf{s}_i? In particular, what is meant by "up to an affine transformation"? Could the authors state an explicit mathematical definition or criterion for correct recovery of si\mathbf{s}_i in this sense? In the experiments, how is si\mathbf{s}_i recovered from the learned representation zi\mathbf{z}_i? Please explain how the affine readout is fitted and how the recovery is evaluated.

  3. Many equations are presented with insufficient intuition or explanation. In particular, Section 3.2 is difficult to follow, and the motivation for its mathematical constructions needs to be clearer.

  4. Is Proposition 2 intended to state that, under Assumptions 1–3, if there exist matrices Qi\mathbf{Q}_i and functions ψi\boldsymbol{\psi}_i satisfying the stated requirements and Equation (3), then any encoder collection that fully recovers the sources in all views is strictly suboptimal for SUMCOR. If so, do such Qi\mathbf{Q}_i and ψi\boldsymbol{\psi}_i necessarily exist under Assumptions 1–3, and how can their existence be proved? If no such Qi\mathbf{Q}_i and ψi\boldsymbol{\psi}_i exist, would the conclusion of Proposition 2 change?

  5. The representation dimension dZid_{\mathcal{Z}_i} required by Theorem 4.3 depends on Di(γ)D_i(\gamma), but how to calculate this quantity in practice needs clearer explanation. In addition, please clarify the distribution with respect to which the expectation in Tijf:=E[f(sj)∣si]\mathcal{T}_{ij}f := \mathbb{E}[f(\mathbf{s}_j) \mid \mathbf{s}_i] is taken. Is there any assumptions on the distributions of si\mathbf{s}_i and sj\mathbf{s}_j? In particular, it would be helpful to show how to calculate Di(γ)D_i(\gamma) for the Gaussian model in Equation (1) under Assumption 2.

  6. The function class and practical implementation of fi\mathbf{f}_i in Equation (2) need clearer explanation. Is fi\mathbf{f}_i restricted to linear or affine maps, or can it be nonlinear? Is the optimization in Equation (2) performed over the function space L2L^2? If so, how can a solution be obtained in practice? Although Section 5.1 mentions residual encoders and Adam, please explain more explicitly how fi\mathbf{f}_i is parameterized in the experiments and how the optimization problem is solved.

Questions for authors

see the weaknesses.

5 comments

We thank the reviewer for the careful reading and constructive questions. We agree that the recovery criterion and the roles of the assumptions needed clearer explanation. We have revised these passages and address each concern below. All locations refer to the revised PDF identified in our general comment. A1–A5 denote Assumptions 1–5.

W1. Why are A2, A4, and A5 needed?

These assumptions are the key conditions of three alternative sufficient routes to full affine source recovery. They are not simultaneously required, e.g., Theorem 4.3 requires neither A2 nor A4.

One major difficulty of source recovery in multi-view setting is that one view-specific encoder must serve several view pairs simultaneously. A higher-order mode of a widely shared coordinate can earn correlation on several pairs, while a pair-specific first-order source direction earns it once. Pair-specific first-order directions are even harder to recover when they have low intra-pair correlation. At minimum-sufficient capacity, namely dZi=mid_{\mathcal Z_i}=m_i, SUMCOR can therefore prefer dominant higher-order modes over the desired first-order directions, causing representation drift. The three routes prevent this in different ways:

ResultAssumptions and widthWhy recovery follows
Theorem 4.1: SUMCORA1, A3, A4; dZi=mid_{\mathcal Z_i}=m_iGlobal first-order dominance (A4) makes the aggregate correlation loss from reducing first-order energy exceed any gain from higher-order components.
Theorem 4.2: Top-rr SUMCOR or masked SUMCORA1, A2, A3; dZi=mid_{\mathcal Z_i}=m_i and known overlap ranks rijr_{ij}Pairwise first-order dominance (A2) places all correlated first-order modes ahead of higher-order modes on each edge. Every population maximizer attains all pairwise bounds and preserves the pairwise first-order targets.
Theorem 4.3: Finite-overcomplete affine source recoveryA1, A3, A5; dZi=Di(γ)d_{\mathcal Z_i}=D_i(\gamma)For 0<γ<min⁡iαi0<\gamma<\min_i\alpha_i, Di(γ)D_i(\gamma) counts the eigenvalues of Γi\Gamma_i at least γ\gamma, with multiplicity. The cutoff spaces contain the full first-order source spaces and any retained higher-order modes. Joint spectral compatibility of finite cutoffs (A5) makes them attain the SUMCOR trace bound, and the spectral gap below each cutoff forces every maximizer to span them.

Role of A1. Under the isotropic Gaussian model (A1), Hermite polynomials separate first-order source functions from higher-order ones, and every higher-order canonical correlation on edge {i,j}\{i,j\} is at most tij,12t_{ij,1}^2. A2 and A4 compare against this bound. A1 also makes Di(γ)D_i(\gamma) finite.

Role of A3. Full cross-view latent coverage (A3) requires every latent coordinate visible in a view to be shared with a neighbor. Under A1, the pairwise first-order targets then span the full first-order source space, so every nonzero linear source direction carries cross-view signal.

Failure outside guarantees. In Setup B, A2 and A3 hold but A4 fails, and full affine source recovery is strictly suboptimal for SUMCOR at dZi=mi=3d_{\mathcal Z_i}=m_i=3 by Proposition 2 applied to the construction in Appendix B.2. In Setup C, both first-order dominance conditions fail. Even with the true overlap ranks rij=2r_{ij}=2, the rank-constrained methods show incomplete source recovery at the tested widths (Tables 1 and 3), whereas SUMCOR at dimension six achieves near-perfect affine source recovery (Table 1), consistent with Theorem 4.3 at Di(γ)=6D_i(\gamma)=6, as calculated in Appendix D.4.

W2. What counts as recovery, and how is it evaluated?

We now state an explicit recovery criterion in Definition 2 and explain how the affine readout is fitted and scored. A high SUMCOR value alone does not establish recovery.

Recovery criterion. An encoder recovers the noisy source si\mathbf s_i if

\quad\text{almost surely}$$ for fixed $\mathbf L_i\in\mathbb R^{m_i\times d_{\mathcal Z_i}}$ and $\mathbf b_i\in\mathbb R^{m_i}$. Equivalently, every source coordinate, and hence every nonzero linear source function, has population $R^2=1$ under its best affine readout. Our theorems ensure this at **every population global optimum** under their stated assumptions and widths. **Affine ambiguity.** At matched width $d_{\mathcal Z_i}=m_i$, $\mathbf L_i$ is invertible because the source covariance is nonsingular under A1. Thus $\mathbf z_i$ is an invertible affine transformation of $\mathbf s_i$. SUMCOR cannot remove this ambiguity because Equation (2) is invariant to such transformations. At larger width, recovery requires only an affine source readout, and the remaining coordinates may be nonlinear. **Readout evaluation.** Sources do not enter the encoder training objective. Evaluation uses ten batches of 4,096 samples from a random stream independent of the training samples. On each batch and view, we fit an affine least-squares readout, including an intercept, from the complete representation to the noisy source and score it on the same batch. We average coordinate-wise $R^2$ over source coordinates, views, and batches. Appendix F.3 estimates the optimism from same-batch fitting under a homoscedastic approximation. This is not a finite-sample bound and does not account for initialization selection on these observations. **Detecting missing directions.** Worst-direction $R^2$ is the smallest same-batch score over all nonzero linear source functions $\mathbf a^\top\mathbf s_i$. It equals the squared smallest source–representation canonical correlation on each batch. Its population value is one if and only if Definition 2 holds. For Setups B/C, Table 3 reports the batch mean of this correlation, **unsquared**, in the worst view as $\gamma_{\min}$. The Setup B results at width three show why the worst-direction diagnostic matters (Tables 1 and 3): | Method | Average $R^2$ (%) | $\gamma_{\min}$ (%) | |---|---|---| | SUMCOR | 68.47 ± 0.21 | 1.06 ± 0.70 | | tSUMCOR | 99.93 ± 0.00 | 99.95 ± 0.01 | | mSUMCOR | 99.94 ± 0.00 | 99.94 ± 0.01 | Here tSUMCOR and mSUMCOR denote Top-$2$ and masked SUMCOR. The standard deviation is over ten evaluation batches of one trained model. SUMCOR's average $R^2$ indicates incomplete recovery, while $\gamma_{\min}$ reveals an almost entirely lost source direction. The corresponding batch-averaged worst-direction $R^2$ is below 0.1% for SUMCOR and above 99.8% for both rank-constrained methods. The five-seed results show the same separation (Tables 4 and 12). Their worst-direction $R^2$ is averaged over batches, then minimized over views. Initialization selection uses the objective on evaluation observations without sources (Appendix F.6). Setup A reports average $R^2$ only.

W3. What is the intuition behind Section 3.2?

Section 3.2 introduces the spectral tools and pairwise first-order dominance condition used in our later theorems, connecting our analysis to classic spectral theory and prior nonlinear CCA results. The constructions serve three purposes:

  1. Whitening and canonical coordinates. Whitening the representations removes arbitrary scale. Under A1, whitening the sources and applying the SVD of the normalized source cross-covariance Rij\mathbf R_{ij} decomposes their joint law into mutually independent Gaussian pairs and uncoupled coordinates, while Rij\mathbf R_{ij} is not required to be full rank. Within each pair, the two variables retain correlation tij,kt_{ij,k}. This factorization makes the cross-view dependence explicit.

  2. Hermite expansion. Under the isotropic Gaussian model (A1), the orthonormal Hermite basis separates polynomial degrees, isolating first-order source functions from higher-order components. Hermite expansion also gives a closed-form spectral description: for a canonical Gaussian pair with correlation tt, matching degree-nn features have correlation tnt^n for n≥1n\ge1, and different degrees are uncorrelated. For multiple canonical pairs, a nonconstant mode with degrees nkn_k has strength ∏ktij,knk\prod_k t_{ij,k}^{n_k}. These are the nonzero singular values of the cross-view conditional-expectation operator on centered functions. Whitening the representations makes their Hermite coefficient rows orthonormal, allowing CCA to compare subspaces of these modes. The scalar Gaussian case is classical (Lancaster, 1958), and Han et al. (2026a; 2026b) used it with whitened encoders for nonlinear CCA. We do not claim it as our contribution. Appendix B.3 states the bound we use (Equation (18)).

  3. Connection to prior theory. The weakest nonzero first-order canonical correlation is tij,rijt_{ij,r_{ij}}, while the strongest higher-order one is tij,12t_{ij,1}^2. Pairwise first-order dominance (A2) places every correlated first-order mode ahead of all higher-order modes. Thus, under A1 and A2, every population pairwise CCA maximizer with representation dimension rijr_{ij} recovers the pairwise first-order targets, as recalled from Han et al. (2026b) in Proposition 1. Appendix B.3 reuses this Hermite–Mehler spectral bound to control higher-order correlations, then compares their aggregate gains with first-order losses when one encoder serves several view pairs.

W4. Is the existence condition in Proposition 2 guaranteed?

Proposition 2 is conditional. If there exist matrices Qi\mathbf Q_i and functions ψi\boldsymbol\psi_i satisfying the stated requirements and Equation (3), then every encoder collection permitting full affine source recovery in all views at minimum-sufficient capacity is strictly suboptimal for SUMCOR. A1–A3 alone do not guarantee that such matrices and functions exist.

Explicit construction. Appendix B.2 gives a three-view example with three-dimensional sources and full cross-view latent coverage. In its aligned, whitened source coordinates, let ζi\zeta_i denote the standardized globally shared coordinate and choose

\boldsymbol{\psi}_i(\mathbf{s}_i) =\bigl(H_2(\zeta_i),H_3(\zeta_i)\bigr)^{\top},$$ where $H_2$ and $H_3$ are normalized Hermite polynomials. These functions are centered, whitened, and orthogonal to all first-order source coordinates, so concatenating them with $\zeta_i$ gives a feasible three-dimensional representation. For the Setup B correlations $\rho=0.6$ (globally shared) and $\tau=0.45$ (pair-specific), A2 holds because $0.45 > 0.6^2=0.36$, yet this construction gives $$J_{\mathrm{drift}}=3(\rho+\rho^2+\rho^3)=3.528 \gt 3.15=3(\rho+\tau)=J_{\mathrm{aff}}.$$ Here $J_{\mathrm{aff}}$ is the value of every collection permitting full affine source recovery at minimum-sufficient capacity. A feasible collection with a larger value proves strict suboptimality. It need not itself be a global maximizer. Appendix B.2 verifies Equation (3) explicitly. **If the existence condition fails.** Proposition 2 implies neither recovery nor failure. For example, when A4 also holds, Theorem 4.1 makes full affine source recovery optimal, ruling out the strict inequality in Equation (3). Among A1–A3, the proof uses only A1. A2 and A3 show that failure can occur despite pairwise subspace recovery guarantees and full cross-view latent coverage.
Reviewer_UCf7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the source recovery problem in multi-view CCA. In particular, the authors investigate representation drift that may arise in multi-view settings and its underlying mechanism. They theoretically study sufficient conditions under which representations learned with SUMCOR can achieve full source recovery, and develop corresponding CCA approaches with recovery guarantees.

Critical strengths
  1. Building on existing pairwise identifiability theory for CCA, the authors further investigate simultaneous source recovery in multi-view settings, extending the scope of previous studies.

  2. The authors analyze the specific mechanism underlying representation drift, investigate theoretical conditions that may lead to this phenomenon, and provide experimental validation.

  3. Based on the theoretical analysis, the authors provide recovery mechanisms that adjust the objective function according to overlap ranks and use overcomplete representations to retain higher-order information, together with corresponding theoretical recovery guarantees.

Critical weaknesses
  1. Although the authors have discussed the limitations of the assumed structured model A_i=B_iP_i, more general multi-view models may involve arbitrary linear mixtures of latent subspaces. To what extent do the different recovery theorems depend on this structural assumption? Which results could potentially be extended to more general loading matrices A_i?

  2. The recovery target of this paper is the full view-specific source, which also includes independent random perturbations in the model. However, in practical multi-view representation learning, these perturbations may not be information that we wish to preserve. Could the authors further explain the practical significance of fully recovering such noisy sources? How does this recovery setting differ from the common goal of extracting shared information in CCA?

  3. In Theorem 4.1, the authors provide a sufficient theoretical condition that guarantees full source recovery under SUMCOR. It would be useful to discuss how applicable this condition is to real-world data. When the true latent sources are unknown, is there any feasible strategy to assess whether this condition holds? This would help determine whether the corresponding theoretical recovery guarantee is applicable.

Minor: 4. The proposed overlap rank based methods require the corresponding r_{ij} to be known in advance, but this information may not be available in practical applications. Although the authors have mentioned estimating overlap ranks as a future direction, could they consider a preliminary evaluation of the methods' robustness to inaccurate rank estimates?

  1. The experiments mainly validate source recovery performance under different conditions, but each reported result corresponds to only one trained residual encoder model. If computational resources permit, could the authors consider repeated training with different random initializations, as well as independent training and evaluation on differently generated datasets or observation models?
Questions for authors

My main question concerns the practical relevance of the structured modeling assumption A_i=B_iP_i. Is this assumption widely applicable to real world multi-view CCA settings? Could the authors provide some practical examples where this modeling assumption holds?

5 comments

We thank the reviewer for their time, careful reading, and thoughtful engagement with the technical details of our work. Their constructive questions have helped us clarify the scope of our guarantees and the evidence needed to assess them. We address each concern below. All locations refer to the revised PDF identified in our general comment. A1–A5 denote Assumptions 1–5.

W1. How restrictive is the loading structure, and which results extend beyond it?

For the main Gaussian results, only the proof of masked SUMCOR uses the coordinate structure of Ai=BiPi\mathbf A_i=\mathbf B_i\mathbf P_i. The other proofs use the loadings only through the normalized source cross-covariances Rij\mathbf R_{ij} and linear coverage, see Remark 1.

What the structure adds. The factorization restricts loading row spaces to coordinate subspaces of a shared latent basis, while allowing invertible within-view mixing. It excludes arbitrary loadings. At each view, we distinguish the two properties defined by Equations (33)–(34) in Appendix E.1:

  • Linear coverage: the pairwise first-order targets span the full first-order source space.
  • Simultaneous coordinate realizability: each target is spanned by a coordinate block of the same invertible linear source representation.

Established and potential extensions to general loadings. By Lemma 5 in Appendix E.1, the structured model satisfies simultaneous coordinate realizability, and under A3 also linear coverage. With general loadings, linear coverage must be assumed and realizability can fail. The table lists the resulting conditions for Gaussian sources, with rij=rank⁡(Rij)r_{ij}=\operatorname{rank}(\mathbf R_{ij}). Each result retains its observation and encoder requirements. Minimum-sufficient capacity means dZi=mid_{\mathcal Z_i}=m_i.

ResultConditions for Gaussian sources with general loadingsExtension Status
Theorem 4.1: SUMCORLinear coverage and global first-order dominance at minimum-sufficient capacity.Proved by Theorem E.1 in Appendix E.2, with the Gaussian specialization in Appendix E.4.
Proposition 2: Representation driftA feasible spectral substitution whose higher-order spectral gain exceeds its first-order spectral loss, at minimum-sufficient capacity.The Setup B counterexample is itself a general-loading instance. The general condition is not formally restated.
Theorem 4.2: Top-rr SUMCORKnown ranks rijr_{ij}, pairwise first-order dominance, linear coverage, and minimum-sufficient capacity.Potential extension (not formally restated). Its proof does not use realizability.
Theorem 4.2: masked SUMCORThe Top-rr SUMCOR conditions above, plus simultaneous coordinate realizability.Proved by Theorem E.2 in Appendix E.2, with the Gaussian specialization in Appendix E.4.
Theorem 4.3: Finite-overcomplete affine source recoveryLinear coverage, joint spectral compatibility of finite cutoffs, and prescribed dimensions dZi=Di(γ)d_{\mathcal Z_i}=D_i(\gamma) with 0<γ<min⁡iαi0<\gamma<\min_i\alpha_i.Potential extension (not formally restated).

For example, realizability fails when three pairwise targets at one view are distinct lines in a two-dimensional source space, since they cannot all be coordinate axes of one basis. The proof of masked SUMCOR then no longer establishes simultaneous attainment of the pairwise bounds. This is a proof limitation, not a claim that recovery must fail.

Practical interpretation. Appendix A.2 gives an idealized example: sensor groups measure fixed, overlapping subsets of physical quantities, with invertible mixing, additive noise, and invertible nonlinear responses in each view. Appendix A.4 explains how our noiseless structural equation matches that of Yao et al. (2024), and how our model specializes the additive-loading model of Han et al. (2026b). These are modeling correspondences. We do not claim that this structure, A1, or A3 holds widely in real data. Our main theorems use all three, and the table above states what still holds for Gaussian sources with general loadings.

W2. Why recover the noisy source instead of only shared information?

We agree that noise itself is not worth preserving. It enters our target only because it cannot be separated from the signal. Since ϵi\boldsymbol\epsilon_i has the same dimension as Pic\mathbf P_i\mathbf c and Bi\mathbf B_i is invertible, every direction of si\mathbf s_i mixes both.

Single-view estimation. Under A1, the best single-view estimate of the visible latents under squared error, E[Pic∣xi]\mathbb E[\mathbf P_i\mathbf c\mid\mathbf x_i], is an invertible linear function of si\mathbf s_i as shown in Appendix A.3. Recovering si\mathbf s_i is thus the same as recovering this estimate, up to the affine ambiguity of Definition 2.

Relation to shared information in CCA. Correlation maximization does not remove the perturbation either. Already for one view pair, the population optimum in Proposition 1 is a linear function of si\mathbf s_i. Under partial observability, a view shares different latent coordinates with different neighbors, and its single encoder must retain the corresponding pairwise first-order source targets. Under A1 and A3, every direction of si\mathbf s_i is correlated with the source of some neighboring view, so retaining all pairwise shared directions is the same as recovering si\mathbf s_i. View-private latent content would violate A3 and lies outside our guarantees.

W3. How applicable is Theorem 4.1, and can its conditions be assessed?

A4 is a sufficient condition for Theorem 4.1. It depends on source-level cross-covariances, which are not directly accessible under unknown nonlinear observation maps. We provide no finite-sample test or estimation guarantee for A4. Theorems 4.2 and 4.3 offer alternative recovery routes under their own stated assumptions, which we likewise do not certify from observations.

When A4 is plausible. A4 compares, at each view, the weakest aggregate cross-view support of a source direction with an upper bound on what higher-order features can collect. It is doubtful when a strong factor shared by many views coexists with weaker factors shared by only a few view pairs. Setup B is such a case: every view pair satisfies pairwise dominance, yet SUMCOR drifts. A4 is more plausible when every source direction is supported on several edges with correlations comparable to the strongest one. We do not know how often this holds in real data. A4 is sufficient, not necessary, for recovery.

Minor 4. What happens when overlap ranks are inaccurate?

We added a five-seed comparison in Setup B. There, both rank-dependent methods are sensitive to a wrong rank. With the true rank, they achieve near-perfect affine source recovery in every seed. With an underestimated rank, they keep the globally shared direction but lose pair-specific signal in every seed. Below, r^\hat r is the supplied rank, and tSUMCOR and mSUMCOR denote Top-r^\hat r and masked SUMCOR.

Design. We use Setup B at width dZ=3d_{\mathcal Z}=3 with five seeds. Every edge has true overlap rank rij=2r_{ij}=2. Pairwise first-order dominance holds but global dominance fails, so with the true rank Theorem 4.2 guarantees full recovery at population maximizers, while SUMCOR drifts. The comparison therefore isolates the effect of the supplied rank. Width three is the minimum-sufficient capacity required by Theorem 4.2, and at this width r^=1\hat r=1 and r^=3\hat r=3 are the only wrong ranks. We train only r^=1\hat r=1. At r^=3\hat r=3, both rank-dependent objectives coincide with SUMCOR.

The table gives mean ± SD over five seeds, using same-batch affine readouts and the initialization selection in Minor 5. Worst-direction R2R^2 near 100% indicates accurate affine readout of every source direction, whereas a value near 0% indicates that at least one direction is barely recovered. Shared-direction R2R^2 measures the readout of the source direction driven by the globally shared latent. Both scores are averaged over batches, then minimized over views, as detailed in Appendices D.4, F.3, and F.6.

Effect of rank errors. The two errors have different population implications:

  • Overestimating to r^=3\hat r=3 reduces both objectives to SUMCOR, so the drift example applies. If A4 also holds, overestimated ranks are safe: with any rij≤r^ij≤min⁡(mi,mj)r_{ij}\le\hat r_{ij}\le\min(m_i,m_j), every maximizer of either objective recovers the sources (Appendix C.2). Setup B violates A4, so this guarantee does not apply there.
  • Underestimating to r^=1\hat r=1 rewards only each edge's leading correlation, attained by the globally shared direction. Fully recovering and non-recovering population maximizers both exist. Full recovery is no longer guaranteed, and the theory does not predict which outcome training selects.

Results. Underestimating the rank fails in every seed. Both methods keep the globally shared direction, but at least one source direction is lost (worst-direction R2≤0.04%R^2\le0.04\%), and average R2R^2 drops to about 47%, below the 69% of SUMCOR. Training nearly attains the maximum objective value, which non-recovering collections also attain, so a near-optimal objective value does not guarantee recovery at the wrong rank. Table 12 and Appendix F.6 report the per-seed values.

MethodSupplied rank r^\hat rR2R^2 (%)Worst-direction R2R^2 (%)Shared-direction R2R^2 (%)
tSUMCOR2 (true)99.94 ± 0.0099.89 ± 0.0399.96 ± 0.01
mSUMCOR2 (true)99.94 ± 0.0099.90 ± 0.0199.95 ± 0.01
tSUMCOR146.80 ± 2.230.03 ± 0.0199.97 ± 0.00
mSUMCOR146.89 ± 1.960.02 ± 0.0199.97 ± 0.00
SUMCOR (= both at r^=3\hat r=3)369.17 ± 0.780.02 ± 0.0199.94 ± 0.01
Reviewer_EYaoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies nonlinear multi-view CCA when different views observe overlapping subsets of a latent vector. Its objective sums the nuclear norms of pairwise whitened cross-covariances, using one encoder per view. The recovery target is the noisy view-specific source, up to an invertible affine transformation at matched dimension, or an affine readout in an enlarged representation.

The central observation is that pairwise first-order identifiability does not ensure simultaneous source recovery: nonlinear functions of a widely shared coordinate can be rewarded on more edges than first-order directions shared by only one pair. With a fixed representation budget, this can make source-preserving representations strictly suboptimal.

Critical strengths

One strength that I liked is that the failure mechanism is presented clearly and in a meaningful way. Representation drift is not ordinary covariance collapse: the representations remain whitened and full rank, but their function spaces omit source directions. The three-view construction explains why a mode that is weaker on every individual edge can nevertheless be attractive after aggregation.

Another critical strength is that the main recovery arguments address every population maximizer. Theorem 4.1 provides an objective-deficit bound that forces the higher-order residual to vanish. Theorem 4.2 uses simultaneous attainment of all pairwise bounds and coverage to force the representation span to contain the entire first-order source space. These arguments establish more than the existence of a desirable solution.

Critical weaknesses

Maybe the main weakness is that the joint spectral compatibility substantially limits the overcomplete guarantee. The explicit example verifies compatibility through aligned Hermite modes; the dense orthogonal source rotations preserve that underlying structure. Thus, the paper establishes that suitable additional capacity enables recovery in compatible models. It does not establish that increasing width generally cures representation drift, or that every sufficiently large allocation works. The manuscript acknowledges this, but the practical significance of the third recovery regime depends on understanding how broad and stable the compatible class is. A structural characterization or an approximate-compatibility result would materially strengthen it.

Another weakness is that the sufficient conditions leave substantial questions about applicability. Both rank-constrained objectives require known overlap ranks, and the overcomplete construction requires spectrum-dependent widths. These are population quantities whose estimation is not addressed. The coordinate-subset loading model also imposes simultaneous geometric structure that general rectangular loadings need not possess. These restrictions do not invalidate the results, but they make the contribution primarily an explanation of population spectral behavior. The paper would benefit from a clearer account of which conclusions survive approximate ranks, imperfect dominance, or modest departures from exact compatibility.

Questions for authors

None

3 comments

We thank the reviewer for the careful reading. We also thank the reviewer for recognizing the clarity of our explanation of the failure mechanism. We address the two main concerns from the reviewer in the following. All locations refer to the revised PDF identified in our general comment. A1–A5 denote Assumptions 1–5.

W1. How broad and stable is the compatible class?

We agree that the key issue is the breadth and stability of joint spectral compatibility of finite cutoffs (A5), and we can only partially address it. We have added an approximate-compatibility bound that controls affine readout error by the compatibility defect relative to the cutoff spectral gap. We then evaluate this bound under non-orthogonal loading perturbations that leave the aligned structure. These computations give limited evidence, not a stability theorem.

Scope of exact compatibility. In the aligned three-view family underlying Setups B and C, we now establish exact compatibility at every cutoff 0<γ<τ0<\gamma<\tau, with full affine source recovery at the prescribed representation dimensions Di(γ)D_i(\gamma). We have extended Proposition 3 in Appendix D.4 accordingly, but this extension remains within the original aligned family.

Approximate compatibility. A compatibility defect small relative to the cutoff spectral gap guarantees accurate affine source readout. Under A1 and A3, at dZi=Di(γ)d_{\mathcal Z_i}=D_i(\gamma) with 0<γ<min⁡iαi0<\gamma<\min_i\alpha_i, every population maximizer, if one exists, satisfies ei≤2εγδi.e_i\le\frac{2\varepsilon_\gamma}{\delta_i}. Here eie_i is the minimum expected squared error of an affine readout of the whitened source, summed over source coordinates. The compatibility defect εγ≥0\varepsilon_\gamma\ge0 measures how far the cutoff spaces fall below the SUMCOR trace bound, and δi>0\delta_i>0 is the cutoff spectral gap. Neither dominance condition is required. We have added this bound as Corollary 1 in Appendix D.5.

Non-orthogonal loading perturbations. To leave the aligned structure, we replace the orthogonal factor of each mixing matrix in Setups B and C by a non-orthogonal one, using 20 random perturbation directions at each of three strengths. The perturbed models still satisfy A1 and A3. In these population computations, exact compatibility fails at every evaluated cutoff. Nevertheless, in Setup B at representation dimension five, every draw at each strength has a common cutoff that gives dimension five at all three views, and the bound certifies population R2≥0.96R^2\ge0.96 for every source direction at every population maximizer, if one exists. At dimension four, 12 of 20 draws have no common cutoff giving dimension four at all three views, even at the weakest strength. Small gaps can also make the bound vacuous when such a cutoff exists. We have added these computations and their method in Appendix D.5, with a summary in Section 4.3.

What still remains open. We do not provide a structural characterization of compatibility beyond the aligned family of Appendix D.4. We also do not prove that small loading perturbations keep the compatibility defect small relative to the cutoff spectral gaps. The results above do not establish exact recovery at arbitrary larger representation dimensions. Finally, Di(γ)D_i(\gamma), εγ\varepsilon_\gamma, and δi\delta_i remain population quantities that we do not estimate from observations.

W2. What survives approximate ranks, imperfect dominance, or approximate compatibility?

We agree that the contribution is primarily a population analysis, and we do not estimate overlap ranks or the minimum representation dimensions for full affine source recovery, or provide procedures for addressing these practical issues. At the population level, overestimated overlap ranks can be tolerated under explicit conditions. Positive dominance margins or small compatibility defects give explicit affine readout-error bounds.

Approximate overlap ranks. Theorem 4.2 uses the true overlap ranks rijr_{ij}. We analyze two types of overlap rank estimation error:

  • Overestimated overlap ranks, r^ij>rij\hat r_{ij}>r_{ij}, inherit the SUMCOR guarantee. Under the assumptions of Theorem 4.1 at dZi=mid_{\mathcal Z_i}=m_i, with rij≤r^ij≤min⁡(mi,mj)r_{ij}\le\hat r_{ij}\le\min(m_i,m_j) on every active edge, every population maximizer of Top-r^\hat r SUMCOR or masked SUMCOR achieves full affine source recovery. This requires A4. Setup B violates A4: there, r^=3\hat r=3 reduces both rank-constrained objectives to SUMCOR, and full recovery is strictly suboptimal. The overestimation guarantee is new in Appendix C.2.
  • Underestimated overlap ranks, r^ij<rij\hat r_{ij}<r_{ij}, need not preserve full affine source recovery. In Setup B, at representation dimension three and r^=1\hat r=1, every population maximizer of either objective retains the globally shared source direction, which is the unique leading mode on every edge, but may omit other source directions. The experiments summarized in Table 4 of Section 5.2 exhibit this behavior. Across all five seeds, both methods retain the shared direction (R2R^2: 99.96%–99.98%) while losing at least one other source direction (worst-direction R2R^2 at most 0.04%).

Small dominance margins. Positive margins αi−βi\alpha_i-\beta_i at every view (A4) bound the affine readout error of near-optimal encoders. Under A1 at dZi=mid_{\mathcal Z_i}=m_i, Equation (19) in Appendix B.3 implies that every feasible collection with JSUMCORE≥Jaff−ωJ_{\mathrm{SUMCOR}}^{\mathcal E}\ge J_{\mathrm{aff}}-\omega, where ω≥0\omega\ge0, satisfies ∑i(αi−βi)ei≤2ω.\sum_i(\alpha_i-\beta_i)e_i\le2\omega. Here eie_i is the affine readout error defined in W1, which equals ℓi\ell_i in Equation (19) at this representation dimension. Under A4, each ei≤2ω/(αi−βi)e_i\le2\omega/(\alpha_i-\beta_i), and the bound weakens as the margins approach zero. At nonpositive margins, Theorem 4.1 gives no recovery guarantee, and representation drift can occur.

Approximate compatibility. See W1 for the affine readout-error bound at the prescribed cutoff dimensions and its limitations.

General loadings. Under our coordinate-subset loading model, only the masked SUMCOR proof uses simultaneous coordinate realizability of the pairwise first-order targets. The other proofs use the loading factorization only through Rij\mathbf R_{ij} and the linear coverage implied by A3. We have updated Remark 1 to highlight this point. Theorem E.1 in Appendix E.2, with the Gaussian specialization in Appendix E.4, extends Theorem 4.1 to general Gaussian loadings under linear coverage and global first-order dominance. Theorem E.2 extends the masked SUMCOR guarantee under its pairwise spectral gaps, linear coverage, and simultaneous coordinate realizability, using known ranks rij=rank⁡(Rij)r_{ij}=\operatorname{rank}(\mathbf R_{ij}). Both results retain dZi=mid_{\mathcal Z_i}=m_i and their stated encoder constraints.

Please let us know if any concern remains unresolved.

Dear Reviewer,

Thank you again for your thoughtful feedback. In response to your main concern, we added an approximate-compatibility bound and evaluated it under non-orthogonal loading perturbations (Appendix D.5), while stating the remaining limitations explicitly. We would appreciate it if you could revisit our responses and update your review to reflect your current assessment. Please let us know if any concerns remain.

Comments on the paper

3 comments

We thank the reviewers for their careful reading and high-quality feedback, which helped us improve the manuscript. We have clarified the recovery criterion, the roles of the assumptions, and the limits of the guarantees. We have also added population results on approximate overlap ranks and approximate compatibility, and repeated Setup B experiments. All locations in our responses refer to the revised PDF accompanying these responses.

The key message. Pairwise recovery does not ensure multi-view recovery: at fixed representation dimension, a nonlinear feature rewarded by several view pairs can displace a source direction shared by only one pair. We give a sufficient failure criterion and three sufficient remedies: global first-order dominance (Theorem 4.1), rank-constrained objectives with known overlap ranks under pairwise first-order dominance (Theorem 4.2), or prescribed extra capacity with joint spectral compatibility (Theorem 4.3). These results concern population global optima in the Gaussian coordinate-subset model with full cross-view latent coverage. They do not establish exact recovery at arbitrary larger representation dimensions.

What we clarified and added.

  • Recovery and intuition. Definition 2 requires affine reconstruction of each noisy source, an invertible affine relation at matched representation dimension. Sections 3.2 and 4 explain feature competition and the assumptions. Proposition 2 clarifies that its sufficient failure condition need not hold under Assumptions 1–3.
  • Applicability. We state which results extend to general Gaussian loadings. Only the masked SUMCOR proof uses the coordinate structure. Under full cross-view latent coverage, retaining all pairwise shared directions is equivalent to recovering the noisy source, the recoverable single-view target. A worked Setup B example gives the smallest prescribed representation dimension as four.
  • Approximate overlap ranks. New Appendix C.2 proves full affine source recovery with overestimated overlap ranks at matched representation dimension under global first-order dominance. Underestimated ranks can lose source directions, as the repeated runs below illustrate.
  • Compatibility. Proposition 3 gives exact compatibility and recovery in the aligned family of Setups B and C for every cutoff 0<γ<τ0<\gamma<\tau, where τ\tau is the pair-specific canonical correlation. New Corollary 1 bounds the affine readout error by the compatibility defect relative to the cutoff spectral gap at the prescribed representation dimensions. Under non-orthogonal loading perturbations, exact compatibility fails at every evaluated cutoff. For all sampled Setup B draws at representation dimension five, it certifies population R2≥0.96R^2\ge0.96 for every source direction at every population maximizer, if one exists. New Corollary 2 bounds readout error without compatibility at large representation dimensions, but is informative only at hundreds of dimensions in Setups B and C.
  • Repeated training and rank sensitivity. Across five seeds fixed before training, Setup B compares SUMCOR, Top-22 SUMCOR, and masked SUMCOR at representation dimension three, plus rank-1 variants of the latter two, giving 25 runs paired by seed. Each run selects the best of four initializations after 10,000 updates per initialization by its training objective on evaluation observations, without sources, then retrains from scratch for 50,000 updates. The evaluation stream is independent of training samples, but its observations are reused for selection. With frozen encoders, affine readouts are fitted and scored on each evaluation batch. In every seed, both rank-constrained objectives at the true rank retain all source directions (worst-direction R2R^2 above 99.83%), whereas SUMCOR loses one (below 0.04%). At supplied rank 1, the shared direction is retained but pair-specific signal is lost (worst-direction R2R^2 at most 0.04%).

Where to find the revisions. This table maps the concerns addressed in the individual responses to the revised PDF.

Change and concernRevised PDF location
Recovery criterion and noisy-source target (7UV8-W2, UCf7-W2)Definition 2 and following paragraph in Section 3.1; Appendix A.3
Loading structure and idealized sensing example (UCf7-W1 and main question, EYao-W2)Remark 1 in Section 3.1; Appendices A.2 and E.4
Pairwise intuition and roles of the assumptions (7UV8-W1/W3)Paragraph before Assumption 2 in Section 3.2; “Roadmap and role of the assumptions” in Section 4
Conditional failure criterion (7UV8-W4)Proposition 2 and paragraph after its proof sketch in Section 4.1
Distribution and capacity calculation, including Setup B under Assumption 2 (7UV8-W5)Section 4.3; Appendix D.1; Equation (30) and Proposition 3 in Appendix D.4
Overestimated overlap ranks under global first-order dominance (EYao-W2, UCf7-Minor 4)Appendix C.2
Exact compatibility at every cutoff of the aligned family (EYao-W1)Proposition 3 in Appendix D.4
Approximate compatibility and loading perturbations (EYao-W1/W2)Paragraph after Remark 3 in Section 4.3; Corollary 1 and “Perturbed loadings” in Appendix D.5
Readout error at large representation dimensions without compatibility (EYao-W1)Corollary 2 in Appendix D.6
Encoder implementation and readout protocols (7UV8-W2/W6)Section 5.1; Appendices F.2–F.3 and F.6
Repeated runs and rank sensitivity (UCf7-Minor 4/5, 7UV8-W2, EYao-W2)Table 4 in Section 5.2; Table 12 in Appendix F.6
Empirical scope and limits of checking the conditions (UCf7-W3/Minor 5, EYao-W2)Section 5.2; Section 6; “consistent with” in the abstract and Section 1

What remains unresolved. The theory does not address estimating ranks or representation dimensions, testing spectral assumptions from observations, or neural optimization. We give no structural characterization of compatibility beyond the aligned family of Appendix D.4, and do not prove that small loading perturbations keep its defect small relative to the cutoff spectral gaps. Finite-overcomplete recovery in Setup C still has only single-run evidence, without a claim of stability across training runs. For the final version, we will extend the same five-seed protocol to Setup C at representation dimension six and report all runs.

Dear SAC, AC, and Reviewers,

Thank you for your time and effort in evaluating our submission. Unfortunately, as the author-reviewer discussion period draws to a close, we have not yet received any replies from reviewers. We had hoped to engage in a constructive discussion and clarify any remaining outstanding concerns, but have not had the opportunity to do so.

We take the reviewers' feedback very seriously and have responded to all of their concerns. Our responses clarified the modeling assumptions and recovery target and added five-seed rank-sensitivity experiments for UCf7, clarified the recovery criterion, roles of the assumptions, and implementation and evaluation details for 7UV8, and added an affine recovery error bound under approximate spectral compatibility for EYao.

We would be grateful for your help in coordinating reviewer feedback on our responses during the remaining review process. We have provided an overview of the paper and details of the revisions in our first official comment. We hope this message provides a useful summary of our responses and supports your assessment during the remaining review process.

Thank you for your time and assistance.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.