acceptodds
Under review as a conference paper at ICLR 2027

Procrustes Similarity with the Reconstructed Space: A Framework for Representational Similarity and Relative Information Content

Abstract

With the growing reliance on high-dimensional embeddings in many areas including machine learning, neuroscience, and computational biology, the need for rigorous methods to compare and assess these representations and their information content is critical. Quantifying the similarity between two variables or distributions is a longstanding challenge in statistics and information theory, and this challenge becomes even more pronounced in high-dimensional spaces. Established methods, such as Mutual Information (MI), Canonical Correlation Analysis (CCA), and Centered Kernel Alignment (CKA), struggle with accurately capturing complex dependencies, handling noise, and modeling nonlinear relationships in high-dimensional settings, limiting their practical usability. To address this, we introduce Procrustes Similarity with the Reconstructed Space (PSRS), a novel approach that quantifies the Relative Information Content (RIC) between two embedding spaces by reconstructing one from another using deep learning techniques and measuring the reconstruction fidelity via Procrustes similarity. Through a combination of synthetic tests and real-world case studies, we demonstrate that PSRS significantly outperforms state-of-the-art methodologies in capturing both linear and nonlinear relationships while exhibiting superior robustness to noise and extraneous information.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representational Similarity, Relative Information Content, Embedding similarity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8oowOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Procrustes Similarity with the Reconstructed Space (PSRS), a framework for quantifying the relative information content between two embedding spaces. The approach consists of two main steps: first, a neural network is trained to reconstruct one embedding space from the other; second, Procrustes similarity is used to evaluate the reconstruction fidelity. Interestingly, the resulting score is directional, allowing to investigate asymmetric relationships. A symmetric variant, sPSRS, is also introduced by taking the minimum of the two directional scores. The proposed method is evaluated via synthetic experiments involving independent representations, PCA-based dimensionality reduction, nonlinear coordinate transformations, irrelevant dimensions, and additive Gaussian noise. Its performance is compared with Deep Canonical Correlation Analysis (DCCA) and nonlinear Centered Kernel Alignment (CKA). The authors also present a real-world case study on Matryoshka embeddings.

Critical strengths

A main strength of PSRS is its ability to evaluate representational similarity directionally, providing a potentially useful operational approach going beyond symmetric similarity measures.

Critical weaknesses

My primary concern is the theoretical interpretation of the proposed measure. The authors adopt variance as an information proxy and argue that Procrustes similarity between the reconstructed and original representations quantifies the proportion of information preserved between embedding spaces. However, this interpretation is not supported by a sufficiently rigorous theoretical justification. A related concern arises from the interpretation of Procrustes similarity as analogous to the coefficient of determination. The proposed framework combines two established and independent components, namely neural networks for reconstructing one representation from another, and Procrustes analysis for evaluating similarity. While this combination may be practically useful, the manuscript does not clearly establish what fundamentally new capability it provides. In particular, the experimental evaluation does not include comparisons with sufficiently many alternative baselines, making it difficult to determine whether the proposed framework provides a meaningful methodological improvement. The empirical validation remains relatively narrow for a method presented as a general framework for comparing high-dimensional representations.

Questions for authors

Question 1. Could the authors provide a rigorous mathematical justification for interpreting PSRS as a relative information content? In particular, how does the proposed score relate to the conventional coefficient of determination after centering, normalization, and orthogonal Procrustes alignment? Question 2. Since PSRS reconstructs one embedding space from another using a neural network trained with MSE, why is Procrustes similarity needed to evaluate reconstruction quality instead of a standard out-of-sample multivariate R2? Using the same trained neural network, a comparison between these two evaluation measures would help clarify the specific contribution of the Procrustes step. Question 3. To what extent do the reported results for DCCA and nonlinear CKA depend on their hyperparameter configurations, e.g., the kernel bandwidth used for CKA? Could the authors clarify whether the claimed advantages of PSRS remain consistent under alternative reasonable configurations of hyperparameters? Question 4. Can the authors investigate whether PSRS reliably compares independently learned representations, such as embeddings generated by different neural architectures? This would be needed, as the current experiments involve synthetic transformations or prefixes of the same representation. Question 5. Could the authors provide an analysis of the computational cost and scalability of PSRS? Since the proposed approach requires training a neural network for each directional comparison, reporting training times and computational requirements, alongside those of the competing methods, would help assess its practical applicability.

Reviewer_dhBJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new similarity measure, PSRS, for high-dimensional representations of the same samples. It proposes to first reconstruct one representation from the other with a custom deep learning architecture and then compute the Procrustes similarity between them. On synthetic data, the authors compare PSRS' behaviour with two competitors. Finally, they include a case study that illustrates how PSRS works on real data.

Critical strengths
  • S1. The paper is, for the most part, easy to follow, even for a non-domain-expert. It is also clearly structured.
  • S2. Given the popularity of learning high-dimensional representations and the ensuing need to compare them, the paper addresses a relevant topic.
Critical weaknesses
  • W1. DeepShallowNet under-analyzed: Despite the discussion in the paragraph l140-l145, I still find the large number of moving parts in the DeepShallowNet problematic. The authors also never analyze whether the particular architecture they propose is relevant. For instance, the network could always choose to learn a zero linear branch and thus behave like a normal deep neural network. Given the numerous synthetic tasks on which a linear map is actually optimal, it would be great to check if DeepShallowNet actually learns the optial linear map and no non-linear part.

  • W2. Experimental validation very thin: Almost all of the experiments are synthetic. While I appreciate the usefulness of GT, this still renders the validation fairly thin. In a similar vein, only two competitors are shown, none of which covers the information-theoretic flavor of intrinsic measures. Extrinsic measures were not compared against at all. On the single real world case study no competitors are reported at all. Also, no contextualization of the PSRS results was provided for the use case. Could one, e.g., compute something akin to explained variance, since the Matryoshka embeddings are also nested.

Questions for authors

Q1: Why is the Procrustes analysis even necessary? SSR is essentially the MSE training objective of the network. So the network should be able to learn the optimal translation, rotation, and scaling. (Normalization might still be required for comparability between datasets). How much do the results change when omitting the optimal Procrustes rotation?

Q2: What is the lower bound of PSRS and under which null model does it yield zero scores?

Q3: What are the ranges of CKA and DCCA? This would be helpful to contextualize their results.

Q4: Line 368-370: Both DCCA and CKA have the same failure mode, no? They have the same score independent of the addition of the noise. The current phrasing does not make this immediately clear.

Reviewer_c38FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Procrustes Similarity with the Reconstructed Space (PSRS), an intrinsic measure of representational similarity between two embedding spaces of possibly different dimensionality, defined as the Procrustes similarity between a target space and a deep-learning reconstruction of it from the other space. The authors claim that PSRS yields a directional, R2R^2-like score of how much of one representation's variance is predictable from the other. The metric is validated across synthetic benchmarks and OpenAI's Matryoshka embeddings. The results suggest that PSRS has advantages over the baselines (DCCA, RBF-CKA): it is better at capturing non-linear dependencies (vs. CKA) and the null case (independence, vs. DCCA).

Critical strengths
  1. The proposed quantity is simple to understand and define, yet it is meaningful since it captures direct non-linear dependence between embeddings. The normalization to [0,1][0,1] and the explained variance-based approach are also useful and interpretable.
  2. The work addresses well some methodological problems when comparing distinct similarity measures: the trends are compared instead of the quantities themselves.
Critical weaknesses
  1. The proposed measure is fundamentally limited regardless of the network capacity: MSE minimization yields the conditional expectation (given a wide enough model class). Therefore, PSRS is structurally blind to any dependence that does not move the conditional mean. If B=S⋅A+ZB = S \cdot A + Z, where A∼N(0,1)A \sim \mathcal{N}(0,1), Z∼N(0,σ2)Z \sim \mathcal{N}(0,\sigma^2), S∼2⋅Be(1/2)−1S \sim 2 \cdot Be(1/2) - 1 and A,S,ZA,S,Z are independent, PSRS (in one direction) yields 00, whereas the mutual information is non-zero and can be made arbitrarily high.
  2. Even though the authors contrast PSRS with Mutual Information, their method is tied to a crude MSE-based lower bound on MI, see [1]: I(A;B)≥C1−C2⋅log⁡MSEI(A;B) \geq C_1 - C_2 \cdot \log MSE (equality in the Gaussian case).
  3. "Significantly outperforms state-of-the-art" from the Abstract is to some extent overclaiming. Table 1: PSRS is second in 2 of 3 scenarios (independent: CKA 0.008 is closer to the ground truth 0 than PSRS 0.016; perfect nonlinear: DCCA 0.999 > PSRS 0.986) and tied-best in 1 (perfect linear: all 1.00).

[1] Benoit Frenay et al. "Is mutual information adequate for feature selection in regression?", Neural Networks.

Questions for authors
  1. Since the correct methodology is comparing the trends, why is MI not included as an additional baseline? Some synthetic experiments (like the Gaussian) also admit closed-form expressions for MI, so there is no need to use MINE.

Minor issues:

  • Typo: "DCCS" for DCCA (line 346).
  • App. A.1 lists MNIST and Fashion-MNIST as used datasets; neither appears anywhere in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.