acceptodds
Under review as a conference paper at ICLR 2027

Population Similarity Enables Statistically Grounded Comparisons of Representation Alignment

10 views

Abstract

Representation similarity metrics are widely used to evaluate neural network alignment. Owing to their quadratic time complexity, these metrics are frequently computed on small samples (), yielding empirical scores that can misrepresent the underlying data-generating process. To capture true alignment, we propose population similarity, i.e., the infinite-sample limit of an empirical score, as the primary target of analysis. We establish the existence of this limit and the consistency of the Triplet and Quadruplet Similarity Indices (TSI and QSI) and Centered Kernel Alignment with an RBF kernel (CKA-RBF), and show that Mutual Nearest Neighbors (MutualNN) admits a population similarity under a mild monotonicity condition. To quantify finite-sample deviations without distribution assumptions, we introduce distribution-free tail bounds, proving non-trivial bounds for TSI, QSI, and CKA-RBF, but not for MutualNN, implying its arbitrarily slow convergence and the failure of empirical scores to reflect its population score. We demonstrate experimentally that the ordering of representation pairs by their finite-sample scores is often inconsistent with their ordering by population similarity, with disagreement rates heavily dependent on the chosen metric and the representation pairs compared. Building on our bounds, we design a one-sided hypothesis test to determine if one population similarity exceeds another by a specified margin, and show its -value can be upper-bounded using only observed scores and sample sizes. These bounds are informative for TSI and QSI at practical sample sizes ( to ). Finally, we showcase the practical utility of our approach with a multimodal case study revealing that CLIP's image and text representations are less aligned at the population level than those of SigLIP2.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Representation Alignment, Similarity Metrics, Population Similarity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WpX8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the statistical reliability of representation similarity metrics under finite samples. The authors take population similarity as the target of inference and analyze consistency and concentration guarantees for TSI, QSI, and CKA-RBF under the stated assumptions. They further develop a one-sided hypothesis test for comparing population similarities. The authors also construct a counterexample showing that MutualNN can be highly unreliable under finite samples. The framework is further evaluated on image-text representations from CLIP, SigLIP, and SigLIP2.

Critical strengths
  1. The paper is clearly written and well organized. It addresses a broadly relevant problem concerning the statistical reliability of representation similarity measures.
  2. The counterexample constructed for MutualNN is particularly interesting and thought-provoking. It reveals a non-obvious limitation of finite-sample similarity estimation and raises interesting questions for further investigation.
Critical weaknesses
  1. Limited theoretical novelty. Statistical testing has already been explored in the context of representation similarity [1]. Standard hypothesis testing alone is insufficient as a contribution to a machine learning conference. Regarding the theoretical results, the concentration bounds for TSI and QSI largely follow from existing U-statistic theory and prior work [2], while concentration results for kernel alignment have also been established [3]. The proposed hypothesis testing framework primarily relies on these existing results and standard probability inequalities, offering limited new theoretical insights.

  2. Insufficient coverage for a benchmarking study. A wide range of representation similarity measures have been proposed and systematically reviewed in prior work [4]. Since this paper introduces neither a new similarity measure nor a novel statistical testing technique, I view it primarily as a benchmarking study of the statistical properties of existing measures. From this perspective, analyzing only four metrics is insufficient (especially given that three of them have already been studied theoretically and most of the results follow from standard concentration inequalities). I believe a comprehensive examination of commonly used measures is necessary, distinguishing those whose statistical properties follow directly from established theory from those that present genuine theoretical challenges, such as MutualNN. Without such a systematic investigation, I find the current contribution insufficient for acceptance.

In summary, the two major weaknesses discussed above would require substantial revisions, and I initially leaned toward a clear rejection. However, given the novelty and thought-provoking nature of the MutualNN counterexample, I am inclined to give this paper a weak rejection. I suggest that the authors build upon this counterexample and existing statistical theory to provide a more comprehensive and systematic analysis of representation similarity measures.

References

[1] Grounding representation similarity through statistical testing. 2021

[2] Scalable and interpretable representation alignment with ordinal similarity. 2026

[3] Algorithms for learning kernels based on centered alignment. 2012

[4] ReSi: A Comprehensive Benchmark for Representational Similarity Measures. 2025

Questions for authors

I have no further technical questions regarding the theoretical analysis and experiments for the four selected similarity measures. My main concerns relate to the novelty and scope of the work, as discussed in the Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer WpX8 (1/2)

We thank Reviewer WpX8 for their insightful comments. In particular, the suggestion to build on 'statistical theory to provide a more comprehensive and systematic analysis of representation similarity measures' led us to derive sufficient conditions for the existence of distribution-free bounds, a substantial theoretical result that applies to broad classes of metrics.

1. On the Novelty and Importance of Population Level Statistical Testing

Limited theoretical novelty. Statistical testing has already been explored in the context of representation similarity [1]. [...]

  • Distinct Scope of Statistical Testing We acknowledge that statistical testing has been explored for representation similarity. The work cited by the reviewer as [1], which we also cite in the related work, tests the connection between representation similarity and functional analysis, a different type of statistical analysis. Our analysis differs from the literature in important ways:
    • Convergence primitives do not rely on Unverifiable Assumptions Our distribution-free tail bounds control how fast an empirical score approaches its population similarity for every data distribution and admissible distance. This is essential because the true distribution is unknown and the distance function varies by application.
    • Hypotheses are at the Population Level The one-sided test asks whether one population similarity exceeds another by a margin t≥0t\ge 0, so the target is the infinite-sample alignment rather than the finite-sample scores. As far as we know, this is the first test of whether one population representation similarity exceeds another, with an analytical finite-sample pp-value bound.
    • We derive pp-value Upper Bounds on an Intractable Quantity The exact tail probability of the empirical gap under the null is intractable. For any metric with distribution-free tail bounds, we can compute an upper-bound to this pp-value using only the observed scores and sample sizes.
    • Analytical Finite-Sample Guarantees rather than Empirical Resampling As noted in the related work, existing analyses mostly rely on bootstrap or permutation procedures. These can be valid for the hypothesis they test, but they do not provide an analytical finite-sample guarantee that holds uniformly over an unknown data-generating distribution and across metrics. Our pp-value bound is of that form.

2. Extensive and Systematic Analysis of Similarity Metrics' Distribution-Free Bounds

From this perspective, analyzing only four metrics is insufficient [...] I believe a comprehensive examination of commonly used measures is necessary, distinguishing those whose statistical properties follow directly from established theory from those that present genuine theoretical challenges, such as MutualNN. Without such a systematic investigation, I find the current contribution insufficient for acceptance.

I suggest that the authors build upon [...] existing statistical theory to provide a more comprehensive and systematic analysis of representation similarity measures.

We thank the reviewer for their suggestions that led us to develop a more general theorem of the existence of distribution-free bounds.

  • Robustness to Outliers and Uniform Asymptotic Unbiasedness imply Distribution-free Bounds New Theorem 1 shows that these two properties are sufficient for a non-trivial distribution-free tail bound (New Theorem 1 L279-311).
    • What the Conditions Require Robustness limits the influence of any single sample by T/NT/N, where NN is the sample size and T>0T>0 is a constant. Uniform asymptotic unbiasedness requires the bias to vanish uniformly as b(N)→0b(N)\to 0.
    • Why these Properties Matter Both are established requirements of a useful similarity metric a score should not be dominated by one point [4], and it should not retain a persistent bias [5].
    • Implication Any metric satisfying them, admits the sub-Gaussian rate uL(ε,N)=uR(ε,N)=exp⁡(−2N[ε−b(N)]+2/T2)u_L(\varepsilon,N)=u_R(\varepsilon,N)=\exp\bigl(-2N[\varepsilon-b(N)]_+^2/T^2\bigr).
    • Resulting pp-value bounds To see that this rate is informative in practice, we instantiate the unbiased case b(N)=0b(N)=0 with T=5T=5. The resulting pp-value upper bounds become useful at practical sample sizes (Figure 4, L378-389).
  • Consequences We follow the reviewer's suggestion to build on existing statistical theory for a more systematic analysis. Rather than deriving a bound separately for each metric, Theorem 1 gives sufficient conditions that cover the whole class. Because the resulting rate is sub-Gaussian, any metric satisfying the two conditions, including TSI\mathrm{TSI}, QSI\mathrm{QSI}, and metrics yet to be formulated, cannot converge arbitrarily slowly, unlike MutualNN\mathrm{MutualNN}. The population test and its pp-value upper bound therefore apply to this whole class, rather than only to the metrics analyzed individually.

(continues ...)

AuthorsOpenReview ↗

Response to Reviewer WpX8 (2/2)

Regarding the theoretical results, the concentration bounds for TSI and QSI largely follow from existing U-statistic theory and prior work [2], while concentration results for kernel alignment have also been established [3].

We agree that the TSI\mathrm{TSI}/QSI\mathrm{QSI} bounds follow from UU-statistic theory [2] and that the CKA-RBF⁡\operatorname{CKA-RBF} bound builds on [3]. These connections were stated in the proofs, and now we also made them explicit in the main text, with a note on how these distribution-free bounds build on existing work, and minor edits elsewhere in the paper (See L322-325).

3. Further Investigation of MutualNN's Arbitrarily Slow Convergence in Euclidean Spaces

The counterexample constructed for MutualNN is particularly interesting and thought-provoking. It reveals a non-obvious limitation of finite-sample similarity estimation and raises interesting questions for further investigation.

Since the MutualNN counterexample proved so illustrative, we also added one with the Euclidean distance, which is widely used in practice and shows that the slow convergence is not limited to handpicked metric spaces (New Appendix G L1998-2198).

4. Linear CKA Can Converge Arbitrarily Slowly to Its Population Similarity

[...] examination of commonly used measures is necessary, distinguishing those [...] that present genuine theoretical challenges, such as MutualNN.

Following the reviewer's suggestion to examine commonly used metrics that pose genuine theoretical challenges, as MutualNN does, we show that CKA with a linear kernel, as opposed to CKA with a RBF kernel, does not admit non-trivial distribution-free tail bounds. (New Appendix H L2200-2344).

5. Not Only a Benchmarking Study

Insufficient coverage for a benchmarking study. [...] I view it primarily as a benchmarking study of the statistical properties of existing measures.

  • Paper Positioning Although, we acknowledge that parts of our work provide a comparative analysis of different similarity metrics, we do not see our work as solely a benchmarking study. Our paper serves other purposes:
    • Establishing Population Similarity as the Target of Representation Alignment We define population similarity as the infinite-sample limit of an empirical score and motivate why that limit, rather than a score on a small batch, is the quantity a comparison should target. Empirical rankings at standard sample sizes can reverse the population ranking. We then prove that any bounded metric that converges in probability, including TSI\mathrm{TSI}, QSI\mathrm{QSI}, and CKA-RBF⁡\operatorname{CKA-RBF}, has a well-defined population similarity and is consistent.
    • Definitions and Theorems that Apply Directly to other Metrics in General The paper is organized as a set of reusable conditions rather than a study of four named metrics. The definition of distribution-free tail bounds, the existence result for population similarity, the new existence result for distribution-free bounds, and the pp-value upper bound are all stated for an arbitrary similarity metric. A new metric is covered once it is checked against these conditions: boundedness and convergence in probability for existence and consistency, outlier robustness and uniform asymptotic unbiasedness for a non-trivial tail bound, and that checked against the tail bound for the population test. No new testing argument is required.

 

We thank Reviewer WpX8 for feedback that nudged us to derive an important theoretical contribution examining sufficient conditions for the existence of distribution-free bounds. We look forward to the discussion.


[1] Ding et al., Grounding representation similarity through statistical testing, NeurIPS 2021

[2] Soares et al., Scalable and interpretable representation alignment with ordinal similarity, ICML 2026

[3] Cortes et al., Algorithms for learning kernels based on centered alignment, JMLR 2012

[4] Davari et al., Reliability of cka as a similarity measure in deep learning, ICLR 2023

[5] Quadrianto et al., Kernelized sorting, NeurIPS 2008

Thank you for the detailed response. The main additions to the revised manuscript are Theorem 1 and two counterexamples. Theorem 1 follows directly from McDiarmid’s inequality. I therefore believe that the paper’s novelty still lies in these counterexamples.

Given the limited novelty, I believe that a thorough discussion of the statistical properties of commonly used representational similarity measures could itself make a valuable contribution. However, the current revision still falls well short in this respect.

I therefore maintain my original recommendation.

We thank Reviewer WpX8 for their feedback.

Theorem 1 follows directly from McDiarmid’s inequality. I therefore believe that the paper’s novelty still lies in these counterexamples.

While the proof of Theorem 1 builds on McDiarmid’s inequality, the importance and novelty of the theorem lies in its generality and practical implications rather than the proof technique itself. The theorem shows that two simple conditions, robustness to outliers and uniform asymptotic unbiasedness, are sufficient to establish a sub-Gaussian distribution-free tail bound. Importantly, these conditions are not specific to any particular similarity metric or data distribution, but apply to broad classes of metrics. This provides a general criterion for establishing when population-level similarity conclusions can be trusted, and gives practitioners a concrete way to assess whether a similarity metric supports statistically reliable comparisons.

Given the limited novelty, I believe that a thorough discussion of the statistical properties of commonly used representational similarity measures could itself make a valuable contribution. However, the current revision still falls well short in this respect.

The framework already covers the two most commonly used metrics, CKA and MutualNN, as well as recent ones such as TSI and QSI [1,2,3,4]. It provides a way to decide when population-level similarity conclusions can be trusted, which is the question alignment studies need answered.

We thank the reviewer again for their comments, which have helped improve our paper.


[1] Kornblith et al., Similarity of neural network representations revisited, ICML 2019

[2] Huh et al., The platonic representation hypothesis, ICML 2024

[3] Soares et al., Scalable and interpretable representation alignment with ordinal similarity, ICML 2026

[4] Schnaus, et al., Shared geometry as a Rosetta stone: Cross-modal alignment without paired data, Arxiv 2026

Reviewer_h7hCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies population limits of representation-similarity scores and uses concentration bounds to test whether one population similarity exceeds another. It develops guarantees for TSI/QSI and CKA-RBF, constructs MutualNN examples without uniform convergence rates, and illustrates the comparison test on CLIP, SigLIP and SigLIP2 image–label representations.

Critical strengths
  1. The comparison theorem turns concentration bounds into a usable conservative test, including when models are evaluated on the same observations. The TSI/QSI derivations and reported p-value calculations are consistent.
  2. The MutualNN constructions make the distinction between pointwise convergence and a rate uniform over distributions concrete. This is a useful caution when interpreting finite-sample scores.
Critical weaknesses
  1. The CKA assumption is substantially stronger than its justification suggests. Assumption 1 (p. 4) requires empirical self-HSIC to exceed a fixed positive m almost surely for every N. For N=2, self-HSIC equals (1-exp(-d(x1,x2)^2/(2 sigma^2)))^2. Ordinary continuous Euclidean distributions give arbitrarily close pairs with positive probability, so they cannot satisfy this condition. The issue therefore extends beyond degenerate representations. A lower bound on population self-HSIC appears to offer a natural repair, but the current CKA guarantee and its claimed scope need revision.

  2. The CKA/MutualNN experiments do not establish the connection to the stated regularity conditions. Section 5.2 uses cosine distance, and the supplied implementation squares 1-cosine inside the RBF. This need not satisfy the squared-distance CND condition in Appendix A, even for four ordinary unit vectors. Repeated class-label embeddings also create MutualNN ties, whereas Appendix A assumes zero tie probability; the code uses a single sorting-based tie rule. These choices leave unclear whether the empirical metric comparisons support the broader practical recommendations. This concern does not invalidate the TSI/QSI comparison theorem.

Questions for authors

Do the Figure 6 conclusions persist with a PSD-compatible RBF kernel and MutualNN results averaged over tie resolutions? This would clarify whether the experimental differences reflect the metrics' convergence behavior or the particular implementation choices.

4 comments
AuthorsOpenReview ↗

Response to Reviewer h7hC (1/2)

We thank Reviewer h7hC for their valuable feedback and thoughtful improvement suggestions. We leverage these suggestions directly and are thus grateful to the reviewer for proposing them.

1. Tighter Non-Degeneracy Assumption

The CKA assumption is substantially stronger than its justification suggests. [...] Ordinary continuous Euclidean distributions give arbitrarily close pairs with positive probability, so they cannot satisfy this condition. The issue therefore extends beyond degenerate representations. A lower bound on population self-HSIC appears to offer a natural repair [...]

  • Changed CKA Assumption We thank the reviewer for raising this issue and follow their suggestion to change the CKA assumption to satisfy two clauses: HSIC(X,X)≠0\mathrm{HSIC}(X,X) \neq 0 and HSIC(Y,Y)≠0\text{HSIC}(Y,Y) \neq 0 almost surely, and HSIC∗(DXX)≥m\mathrm{HSIC}^*(D_{XX}) \ge m together with HSIC∗(DYY)≥m\mathrm{HSIC}^*(D_{YY}) \ge m for a uniform constant m>0m>0 (Assumption 1 in L190-197).
  • Rationale As noted by the reviewer, this offers a natural repair as the first clause guarantees that CKA-RBF⁡\operatorname{CKA-RBF} is always well-defined, and the second ensures that the considered distributions are non-degenerate.
  • Implications for the distribution-free tail bounds The bounds for CKA-RBF⁡\operatorname{CKA-RBF} are unchanged as their proof uses only the uniform lower bound HSIC∗≥m\mathrm{HSIC}^*\ge m on population self-HSIC, which the revised assumption retains.

2. On Cosine Distance and Square Root of Cosine Distance Satisfying CND Condition

[...] Section 5.2 uses cosine distance [...] need not satisfy the squared-distance CND condition in Appendix A, even for four ordinary unit vectors.

  • Cosine Distance does not satisfy CND We thank the reviewer for identifying this inconsistency, which yields kernels that are not PSD. Using a four-unit-vector counterexample, we show that the cosine distance does not satisfy the squared-distance CND condition (New Appendix I, Corollary 7 L2347-2380).
  • Square Root of Cosine Distance satisfies CND Conversely, we show that if we take the square root of the cosine distance as our distance function, then this distance indeed satisfies CND (New Appendix I, Corollary 7 L2347-2380).
  • Replacing Cosine Distance with its Square Root Offers a Natural Alternative As a consequence, a practitioner who wants cosine similarity inside the RBF kernel can use this distance. Because the kernel squares the distance, exp⁡(−d2/(2σ2))\exp\bigl(-d^2/(2\sigma^2)\bigr) reduces to exp⁡(−(1−cos⁡(x,x′))/(2σ2))\exp\bigl(-(1-\cos(x,x'))/(2\sigma^2)\bigr), one simply obtains the cosine distance inside the exponential.

3. Randomized MutualNN in Experimental Settings with Ties

Repeated class-label embeddings also create MutualNN ties, whereas Appendix A assumes zero tie probability; the code uses a single sorting-based tie rule. [...] MutualNN results averaged over tie resolutions

  • Randomized MutualNN We agree that sorting-based tie rules can introduce confounding effects. To remove them while preserving the experimental setting, we follow the reviewer's suggestion and implement a randomized version of MutualNN that breaks ties uniformly at random. This is appropriate because repeated class-label embeddings make several neighbors equally close, so no single sorting order should be preferred.
  • Averaging over Tie Resolutions Averaging these randomized scores then reflects an aggregation across tie resolutions rather than one arbitrary ordering.

4. Re-running Experiments 5.1 and 5.2 with Randomized MutualNN and Square Root of Cosine Distance for CKA-RBF

Do the Figure 6 conclusions persist with a PSD-compatible RBF kernel and MutualNN results averaged over tie resolutions? This would clarify whether the experimental differences reflect the metrics' convergence behavior or the particular implementation choices.

  • Re-running the Experiments To clarify whether the reported differences reflect convergence behavior or implementation choices, as the reviewer suggested, we reran Experiments from Sections 5.1 and 5.2 with two changes. The RBF kernel now uses the square root of the cosine distance, so that it is positive semidefinite, and MutualNN ties are broken uniformly at random, with scores averaged over these resolutions (Figure 5 and Figure 6, L432-461).
  • Conclusions Persist The updated results show no substantial change, except that in Figure 6 the CKA-RBF⁡\operatorname{CKA-RBF} rank disagreement between SigLIP and SigLIP2 is lower at N=800N=800 and N=1.6kN=1.6\mathrm{k}. The conclusions of both figures remain justified.
AuthorsOpenReview ↗

Response to Reviewer h7hC (2/2)

5. New Rebuttal Updates

During the rebuttal process, we also revised the presentation and extended our theoretical results. We report them here to aid the reviewer:

  • Updated Problem Figure Shows a 2D Euclidean Setting where Empirical Orderings Misrepresent Population Orderings (Figure 1 L54-64)
  • Robustness to Outliers and Uniform Asymptotic Unbiasedness imply Sub-Gaussian Distribution-Free Bounds (New Theorem 1 L286-302)
  • pp-value bounds of Robust and Unbiased Similarity Metrics are Informative at Practical Sample Sizes (Figure 4 L378-389)
  • MutualNN Scores Can Converge Arbitrarily Slowly even in Euclidean Spaces (New Appendix G L1998-2198)
  • CKA with Linear Kernel does not admit Non-Trivial Distribution-Free Tail Bound (New Appendix H L2200-2344)

 

We thank Reviewer h7hC for feedback that led to improving the soundness of our assumptions and experimental setting. We look forward to the discussion.

Thank you for the clarifications and revisions. The revised CKA assumptions and PSD-compatible kernel address my main technical concerns. The updated experiments with randomized tie handling also help clarify the empirical findings. I maintain my weak acceptance recommendation.

We thank Reviewer h7hC for their constructive and thoughtful feedback throughout the discussion. We are glad that our clarifications and revisions addressed their main concerns, and we appreciate their positive assessment of our work. With these concerns now resolved, we would kindly ask if the reviewer could consider raising their score.

Reviewer_WsTMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper establishes guarantees of asymptotic convergence for similarity measures, distribution-free tail bounds, and a p-value bound for null hypothesis testing of finite-sample estimates of similarity measures. Centrally, MutualNN and other rank-based methods fail to attain convergence guarantees. Empirical results are presented to verify theoretical predictions and demonstrate an example of the null hypothesis framework applied to the representation similarity of large deep networks.

Importantly, the paper frames its primary contributions as theoretical. For example, only one final sentence in the abstract addresses empirical demonstrations.

Critical strengths

There are two key strengths of this paper:

  1. The authors take the mathematical results very seriously and every assumption is very carefully stated. The results here are sound.

  2. The result in Figure 3, showing that empirical MutualNN scores can converge arbitrarily slowly as a function of sample size is genuinely cool and interesting. It is also very relevant to this field, since MutualNN was used in the well-cited "Plantonic Representation Hypothesis" paper.

Critical weaknesses

The main problem I have with this paper is whether the results are sufficiently novel.

Before addressing novelty, I want to point out that the mathematical notation is overly cumbersome in places. For example, it doesn't seem necessary to drag around the distance functions dXd_X and dYd_Y in the subscripts of the similarity score SdX,dYS_{d_X, d_Y} and its population counterpart SdX,dY∗S_{d_X, d_Y}^*. This clutter is not a big problem once you figure out to look past it, but it does turn off a certain sort of readership.

Theorem 1 is another example. The statement and proof of this result takes up a lot of space (a paragraph of discussion in the main text followed by a corollary), giving the reader the impression that this is a central contribution of the paper. But the result is almost tautological and follows from standard arguments in probability. The result is essentially the bounded convergence theorem and it would be more honest for the authors to state this as a remark than a main novel contribution of the paper.

Corollary 2 is much more interesting! I'm not sure what it is a Corollary of, it would make sense to call this a theorem or a proposition.

My concern with Corollary 2 is that the TSI and QSI convergence bounds—where the constants are decently sharp and interesting—seem to have been already been noted by the original authors of the TSI and QSI measures. See appendix N.4 in Soares et al. 2026.

The CKA-RBF bound is also somewhat of a repackaging of the analysis of Cortes et al. (the original paper the proposed CKA). Furthermore, the constants in this bound are not sharp enough to be practically useful, as the authors themselves note.

That leaves Figure 3 as the main novel contribution. Again, I actually really like that result, but it seems to be a rather small part of the paper and it would require a lot of work to restructure the paper around that result.

The paper also fails to cite and discuss a few prior works that are highly relevant:

Questions for authors

See critical weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer WsTM (1/2)

We thank Reviewer WsTM for a careful and detailed review. In the revised pdf, we have clarified the presentation of our results, and following the reviewer's comments, extended our results. We hope the reviewer will view our contributions as a valuable addition to the representation alignment community.

1. Novelty of our Analysis

The main problem [...] is whether the results are sufficiently novel [...]

To address the central concern of the reviewer directly, we contextualize our contributions within the field of representation alignment:

  • Motivating Conclusions at the Population Level Practitioners often leverage similarity scores to draw conclusions about which representations are more aligned, yet empirical rankings at standard sample sizes can reverse as more samples are drawn. We therefore motivate the study of population similarity, the infinite-sample limit of an empirical score, as the quantity such conclusions should target.
  • Statistical Test of Population Similarity Orderings with Analytical p-value Bounds To our knowledge, this is the first test of whether one population representation similarity exceeds another by a margin t≥0t\ge 0 with an analytical finite-sample p-value bound. Because this bound is distribution-free and depends only on the observed scores and sample sizes, practitioners can directly obtain population rankings from finite-sample evaluations.
  • Guarantees that Hold for any Data Distribution and any Distance Functions Our new definitions of distribution-free tail bounds and p-value upper bounds hold uniformly over every data-generating distribution and admissible distance. This is essential because the true distribution is unknown, and different applications may need distinct distance functions.
  • General Framework that Applies to any Metric As opposed to existing metric-specific analysis, all our definitions and theorems are stated for an arbitrary similarity metric. To analyze a new metric, a practitioner only needs to check simple properties of the metric itself:
    • Existence and Consistency Boundedness and convergence in probability guarantee a well-defined population similarity.
    • Distribution-Free Tail Bounds (New Rebuttal Result) Robustness to outliers and uniform asymptotic unbiasedness guarantee a non-trivial sub-Gaussian bound, and we are the first to identify these sufficient conditions.
    • Population Test Any such bound directly yields the p-value upper bound, with no new testing argument required.
  • First to Show that Widely Used Metrics Can Converge Arbitrarily Slowly (Partly New Rebuttal Result) We are the first to prove that MutualNN, including with the Euclidean distance, and linear CKA admit no non-trivial distribution-free tail bound, resulting in arbitrarily slow convergence to their population scores.
  • Actionable Guidance for Practitioners Although MutualNN and linear CKA are the de facto standard in representation alignment studies, our results show that their empirical rankings cannot be trusted at finite sample sizes without distributional assumptions. By contrast, TSI and QSI support statistically confident population rankings with 1k to 10k samples and gaps above 0.05.

2. Sufficient Conditions for the Existence of Population Similarity

Theorem 1 [...] The result is essentially the bounded convergence theorem [...]

We agree the proof is not technically difficult, as it follows from the bounded convergence theorem. We updated the manuscript to state this explicitly, and renamed it as a corollary (Now Corollary 1 L234-239).

[...] this result takes up a lot of space [...] giving the impression that this is a central contribution of the paper. But the result [...] follows from standard arguments in probability [...]

  • Why this Result is Central This result justifies population similarity as the target of representation alignment. It is well-defined whenever the metric is bounded and its scores converge in probability, conditions we expect similarity metrics to satisfy. Thus, the target extends beyond the specific metrics we consider.
  • Practical Importance is Unrelated to the Proof Difficulty As noted above, the argument follows from the bounded convergence theorem. The space given to the result reflects its role in validating population similarity as the object of analysis, not the technical difficulty of the proof.

3. Presentation Remarks

Presentation: [...] requires substantial effort in places.

To improve the presentation, we updated the problem figure to show a 2D Euclidean setting in which empirical orderings can misrepresent population orderings (Figure 1 L54-64).

[...] it doesn't seem necessary to drag around the distance functions dXd_X and dYd_Y [...]

We acknowledge that the subscripts make reading less convenient. We keep them because dXd_X and dYd_Y define proximity in each space and we believe reporting them highlights their importance.

AuthorsOpenReview ↗

Response to Reviewer WsTM (2/2)

4. Sufficient Conditions for the Existence of Distribution-Free Tail Bounds

[...] Corollary 2 is much more interesting! [...]

  • Generalization to a Broad Class of Metrics Prompted by the remark that Corollary 2 (Now Proposition 1 L314-321) was an interesting result, we analyze which properties already expected of a similarity metric suffice for a distribution-free tail bound. The natural candidates are robustness to outliers [1] and vanishing bias with sample size [2].
    • New Sufficient Conditions In the revised pdf, we show that these two properties suffice for a non-trivial distribution-free tail bound (New Theorem 1 L279-311).
    • What the Conditions Require Robustness limits the influence of any single sample by T/NT/N, where NN is the sample size and T>0T>0 is a constant. Uniform asymptotic unbiasedness requires the bias to vanish uniformly as b(N)→0b(N)\to 0.
    • Implication Any metric satisfying them admits the sub-Gaussian rate uL(ε,N)=uR(ε,N)=exp⁡(−2N[ε−b(N)]+2/T2)u_L(\varepsilon,N)=u_R(\varepsilon,N)=\exp\bigl(-2N[\varepsilon-b(N)]_+^2/T^2\bigr). Hence any such reasonably robust metric, including the ones yet to be formulated, converges to the population similarity at relevant sample sizes.
    • Resulting p-value Bounds Instantiating the unbiased case (b(N)=0,T=5b(N)=0, T=5) yields informative p-value bounds at practical sample sizes (Figure 4, L378-389).

[...] TSI and QSI convergence bounds [...] have already been noted by the original authors [...] CKA-RBF bound is [...] a repackaging of the analysis of Cortes et al. [...] [...] Corollary 2 [...] it would make sense to call this a theorem or a proposition. [...]

  • Better Contextualization of Previous Work We agree that the TSI/QSI bounds follow from U-statistic theory [3] and that the CKA-RBF bound builds on [4]. These connections were stated in the proofs, but in the revision we now also highlight them in the main text (See L322-325).
  • Renaming to Proposition We agree and renamed Corollary 2 to Proposition 1 to better reflect its role (L314-321)
  • Common Distribution-Free Criterion The novelty of Corollary 2 (Now Proposition 1 L314-321) lies in identifying these metrics with a single distribution-free criterion that enables the p-value bounds of Theorem 2, and in proving that MutualNN admits no such bound.

The CKA-RBF bound [...] the constants in this bound are not sharp enough [...]

We agree the constants are too loose to be practically useful. But this is indeed the point, as existing CKA-RBF guarantees do not support population-level comparisons at practical sample sizes, unlike other metrics such as TSI and QSI.

5. Arbitrarily Slow Convergence of Widely Used Metrics

[...] Figure 3 [...] I really like that result, [...]

  • Euclidean MutualNN Construction We strengthened the MutualNN negative result with a Euclidean construction, showing that arbitrarily slow convergence occurs for a standard distance rather than only a custom metric space (New Appendix G L1998-2198).
  • CKA with Linear Kernel does not Admit Non-Trivial Distribution-Free Tail Bounds Motivated by the reviewer's interest in the arbitrarily slow convergence of MutualNN, we investigated whether the same holds for another widely used metric. We show that linear-kernel CKA also exhibits arbitrarily slow convergence, further highlighting the need to study the infinite-sample behaviour of widely used similarity metrics, which lack such guarantees (New Appendix H L2200-2344).

6. Better Contextualization with Related Work

[...] fails to [...] discuss a few prior works [...]

We thank the reviewer for these pointers, and revise the related work section accordingly (Sec. 2 L149-155).

  • Shared Goal Both works are indeed closely related, studying finite-sample estimation errors relative to population similarity (for shape distances [5] and CKA [6]).
  • Fundamental Differences However, both are metric-specific and either rely on unenforceable assumptions (e.g., exact population mean-centering [5]) or target point-estimate bias correction [6]. In contrast, our framework is metric-agnostic and strictly distribution-free.
  • New Insights As kindly detailed in Point 1 above, this enables general concentration conditions, slow-convergence impossibility results, and analytical p-value bounds for population rankings.

[1] Davari et al., Reliability of cka as a similarity measure in deep learning, ICLR 2023

[2] Quadrianto et al., Kernelized sorting, NeurIPS 2008

[3] Soares et al., Scalable and interpretable representation alignment with ordinal similarity, ICML 2026

[4] Cortes et al., Algorithms for learning kernels based on centered alignment, JMLR 2012

[5] Pospisil et al., Estimating shape distances on neural representations with limited samples, ICLR 2024

[6] Chun et al., Estimating neural representation alignment from sparsely sampled inputs and features, CCN 2025

Thanks for the thoughtful and thorough comments. The crux of my concern is what you list under point 4. Namely, that the TSI/QSI results were already given by Soares et al. and the CKA result is more or less a restatement of Cortes et al.

From your response it is still not clear to me what is fundamentally new in this paper, although I appreciate your edits to properly cite the prior works.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.