- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Procrustes Similarity with the Reconstructed Space (PSRS), a framework for quantifying the relative information content between two embedding spaces. The approach consists of two main steps: first, a neural network is trained to reconstruct one embedding space from the other; second, Procrustes similarity is used to evaluate the reconstruction fidelity. Interestingly, the resulting score is directional, allowing to investigate asymmetric relationships. A symmetric variant, sPSRS, is also introduced by taking the minimum of the two directional scores. The proposed method is evaluated via synthetic experiments involving independent representations, PCA-based dimensionality reduction, nonlinear coordinate transformations, irrelevant dimensions, and additive Gaussian noise. Its performance is compared with Deep Canonical Correlation Analysis (DCCA) and nonlinear Centered Kernel Alignment (CKA). The authors also present a real-world case study on Matryoshka embeddings.
Critical strengths
A main strength of PSRS is its ability to evaluate representational similarity directionally, providing a potentially useful operational approach going beyond symmetric similarity measures.
Critical weaknesses
My primary concern is the theoretical interpretation of the proposed measure. The authors adopt variance as an information proxy and argue that Procrustes similarity between the reconstructed and original representations quantifies the proportion of information preserved between embedding spaces. However, this interpretation is not supported by a sufficiently rigorous theoretical justification. A related concern arises from the interpretation of Procrustes similarity as analogous to the coefficient of determination. The proposed framework combines two established and independent components, namely neural networks for reconstructing one representation from another, and Procrustes analysis for evaluating similarity. While this combination may be practically useful, the manuscript does not clearly establish what fundamentally new capability it provides. In particular, the experimental evaluation does not include comparisons with sufficiently many alternative baselines, making it difficult to determine whether the proposed framework provides a meaningful methodological improvement. The empirical validation remains relatively narrow for a method presented as a general framework for comparing high-dimensional representations.
Questions for authors
Question 1. Could the authors provide a rigorous mathematical justification for interpreting PSRS as a relative information content? In particular, how does the proposed score relate to the conventional coefficient of determination after centering, normalization, and orthogonal Procrustes alignment? Question 2. Since PSRS reconstructs one embedding space from another using a neural network trained with MSE, why is Procrustes similarity needed to evaluate reconstruction quality instead of a standard out-of-sample multivariate R2? Using the same trained neural network, a comparison between these two evaluation measures would help clarify the specific contribution of the Procrustes step. Question 3. To what extent do the reported results for DCCA and nonlinear CKA depend on their hyperparameter configurations, e.g., the kernel bandwidth used for CKA? Could the authors clarify whether the claimed advantages of PSRS remain consistent under alternative reasonable configurations of hyperparameters? Question 4. Can the authors investigate whether PSRS reliably compares independently learned representations, such as embeddings generated by different neural architectures? This would be needed, as the current experiments involve synthetic transformations or prefixes of the same representation. Question 5. Could the authors provide an analysis of the computational cost and scalability of PSRS? Since the proposed approach requires training a neural network for each directional comparison, reporting training times and computational requirements, alongside those of the competing methods, would help assess its practical applicability.