acceptodds
Under review as a conference paper at ICLR 2027

Mutual neighbor similarity: reliable and rich representational alignment measurements

12 views

Abstract

Every representational similarity measure carries strengths and weaknesses. Mutual kNN (m-kNN) style approaches can report local alignment, but saturate at larger scales. In addition, these ordinal approaches, including kendall's tau, and spearman rank correlation, discard all information-rich geometric details in favor of coarse rank-based descriptions, incurring various geometric failure modes. On the other end, many geometric approaches such as centered kernel alignment (CKA) provide global summary metrics but cannot describe alignment at local or intermediate scales. Furthermore, many of these methods can be easily confounded by factors such as embedding dimension and must be null-calibrated via expensive Monte-Carlo simulations to account for geometric priors and deliver robust estimates of alignment with meaningful baselines. To unify many of these strengths in a single measure while mitigating weaknesses, we introduce mutual neighbor similarity (MNS), a hybrid geometric and ordinal comparison of corresponding embedding space neighborhoods. With an analytically derived and simple to compute null expectation, MNS enables robust comparisons across all scales without saturation at either extreme, while accounting for the structure of point cloud geometries. We construct MNS, describe its invariances, derive its robust form, and run experiments on synthetic data demonstrating its ability to overcome various failure modes of existing approaches while introducing entirely new descriptive abilities. Finally, we present results on large scale real world data suggesting the superiority of our approach for transfer learning tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
representational alignment, neural alignment, computational neuroscience, rsa, cka, transfer learning, knowledge distillation, representation learning, platonic representation hypothesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1vkpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper has a clear goal: to develop a representation similarity measure that captures both geometric structure and neighbor rankings. The method is conceptually simple. Building on mutual -nearest-neighbor matching, it compares similarity profiles under the neighbor orderings induced by the two representations. It then calibrates the resulting scores using their maximum values and a random-permutation baseline. The experiments are fairly comprehensive, covering sensitivity to perturbations, alignment across spatial scales, real-model comparisons, and computational efficiency.

Critical strengths
  1. The idea of blending the geometry and the ranking order is very simple yet effective idea.
  2. The whole paper’s story telling is good, I think all the readers can learned things from the manuscript.
Critical weaknesses
  1. My main concern is whether the sensitivity experiments convincingly demonstrate the advantage of AMNS. Procrustes also captures the expected trend as perturbation strength increases. However, the prescribed perturbation strength does not itself establish how much the alignment score should decrease. Without an independent reference for the appropriate response, it is unclear why Procrustes should be considered overly sensitive while AMNS is judged to respond appropriately.
  2. More downstream applications. Authors list the functional alignment results in the Sec 4.4, which would be great, and I also see that the authors seems be constrained by the computation resource. Beyond transfer learning, I encourage the authors to evaluate AMNS in other downstream settings where representation alignment matters, such as cross-model embedding translation in retrieval systems.
  3. When is AMNS less effective? Different representation-similarity measures encode different invariances and may fail under different conditions. Although the paper discusses the desirable properties of AMNS in detail, it provides limited analysis of its failure modes. I encourage the authors to characterize settings in which AMNS may become unreliable or less informative, and to clarify when it should be preferred over existing measures.
Questions for authors
  1. In the Table 1, I see the AMNS_10 seems better than AMNS_M, could the authors explain why?
  2. In the Figure 2, I see Procrustes, AMNS, and CKA have similar performance, but in the introduction, the authors say they are in the different category, could author explain why?
  3. I see a lot of gaps between paragraphs in the P.7, the ticks in the Figure 3 are quite small.
2 comments

We thank you for a detailed and constructive review. Here, we address your concerns in their original order.

Controlled Perturbation ground truth: Thank you for raising this point! The outlier study in this work was inspired by QSI/TSI which, as is common in representational alignment, builds its analyses off of “desired properties” rather than ground truths, which are rare in this field. However, since controlled perturbations compare points in the same ambient space before and after a transformation, we actually can treat the unperturbed positions as ground truth observations and define “true” alignment, as c=E[cos⁡(xi,perturbation(xi)]c = \mathbb{E}[\cos(x_i,\text{perturbation}(x_i)], as we do in our nonconfoundedness checks in the appendix. We then apply a Cohen’s κ\kappa chance correction/0 to 1 scale normalization: c→c−cos⁡‾1−cos⁡‾c \to \frac{c-\overline{\cos}}{1-\overline{\cos}} to compare to the curves of the tested similarity measures. We vary the anisotropy over 12 gradations from 0 to 0.99 and compute the MAE of the ground truth surface GT=GT(perturbation strength, anisotropy) to each measure’s corresponding surface. We take the mean over 5 seeds. AMNS best tracks the ground truth.

MeasureMAE
AMNS0.0656±0.0011
CKA0.0842±0.0081
SVCCA0.0994±0.0036
PWCCA0.1025±0.0033
RSA0.1182±0.0008
Spearman0.1183±0.0031
Procrustes0.1187±0.0037
Kendall0.1204±0.0030
CKNNA0.1204±0.0012
QSI0.1216±0.0003
TSI0.1235±0.0003
m-kNN0.1237±0.0005

Real world experiments: We are eager to test the utility of AMNS in different representational alignment contexts. At this time we are computationally constrained to experiments requiring no training or inference. All real embeddings used in this paper were publicly available. We have expanded our functional alignment experiments to go beyond simply predicting global performance differences from alignment, to predicting a more informative error consistency measure (Geirhos, Meding & Wichmann, 2020). The results show that AMNS alignment is the best predictor of these functional differences.

MeasureLift
AMNS (AUC)3.47±0.22
AMNS (k=M)3.27±0.21
Spearman ρ3.20±0.21
Kendall τ3.20±0.21
TSI3.15±0.21
QSI3.13±0.23
RSA3.01±0.23
CKA2.81±0.23
AMNS (k=10)2.58±0.25
Procrustes2.25±0.22
CKNNA (AUC)2.20±0.22
m-kNN (k=10)2.11±0.25
CKNNA (k=10)1.90±0.20
SVCCA1.05±0.16
PWCCA1.00±0.15

We believe that our split between analyzing the properties of our measure and real world experimentation matches typical representational alignment methods papers (arXiv:2606.16379, arXiv:2110.14739).

Failure modes: Of the failures we mention in our discussion, we believe the most pertinent is that AMNS is not a mathematical metric. This precludes its usefulness in large scale evaluations of many models where clustering analyses (or other analyses that treat model space as a metric space) are desired. In addition to this and the other outstanding questions mentioned, while we found that AMNS best tracks ground truth alignment in controlled perturbation settings across the anisotropy spectrum, it is not optimal in the extremes (mean cosine = 0 and mean cosine -> 1). While we are not aware of models that have anisotropy outside of the “AMNS optimal” regime, there are certainly low anisotropy settings where it would seem AMNS is overly sensitive to perturbations. This seems reasonable to us given that AMNS exists to be an ordinal measure that is aware of the impact of geometric idiosyncrasies. In settings where there are no such idiosyncrasies, AMNS can be overly sensitive to noise.

Questions:

  1. While we have not characterized this exactly, our best hunch is that local alignment information is simply a more information-rich signal for this prediction task. Whether two models construct the same neighborhoods is more informative of their classification abilities than the large scale structure of those neighborhoods.

  2. AMNS, CKA, and Procrustes shape metric are all geometric, meaning they are aware of the similarity of embeddings, not just relative similarity rankings. As such, they are all sensitive to changes of geometric features of the data such as anisotropy. This explains the results in Figures 1 and 2. AMNS is distinguished from CKA and Procrustes because it does not directly compare pairs of points across spaces. Rather, given an anchor within a space it compares each neighbor to the point that has the same neighbor ranking in the opposite space. By this construction, it does not assume that the two point cloud geometries need to share any geometric structure between them (such as anisotropy).

Dear authors,

First, I am curious why you say "At this time we are computationally constrained to experiments requiring no training or inference"? What kind of equipment you have?

Second, I think my concerns are fully addressed, but I think my score is enough for reflecting my evaluation to this paper.

Best,

Reviewer 1vkp

Reviewer_UzFkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mutual Neighbor Similarity (MNS), a representation alignment measure that combines neighborhood ranking information with geometric similarity. The authors further derive an analytical null expectation to obtain the adjusted measure (AMNS), enabling representation similarity comparisons across different neighborhood scales. The method is evaluated through synthetic experiments, analyses of real-world model representations, and a task predicting performance gaps between models.

Critical strengths
  1. The paper introduces a new measure combining neighborhood ranking information with geometric similarity and derives an analytically computable null expectation. This provides a new approach to multi-scale representation alignment analysis, with theoretical value and potential practical applications.
Critical weaknesses
  1. Lack of qualitative evaluation and comparison.

The paper emphasizes that MNS preserves richer geometric information, identifies shared structures, and offers better interpretability than existing ordinal and similarity measures. However, it lacks qualitative visualizations on real-world data and visual comparisons with baselines such as m-kNN and CKNNA on the same samples. Established visualization techniques are available for nearest-neighbor rankings, semantic relationships, geometric distributions, and hierarchical clustering structures. Statistical curves and numerical results alone are insufficient to clearly demonstrate what meaningful structures MNS captures in real data and what additional information it provides over existing methods, leaving its central claims of descriptive power and interpretability insufficiently validated.

  1. Insufficient validation of practical utility in transfer learning.

The paper highlights the potential applications of MNS in transfer learning and knowledge distillation, but its experiments mainly focus on controlled synthetic data and representation similarity analysis. Its practical evaluation only uses representation similarity to predict ImageNet classification accuracy gaps between pretrained models, without directly evaluating transfer learning or knowledge distillation. Similar classification accuracy between models does not necessarily imply effective knowledge transfer between their representations. Therefore, the current experiments are insufficient to support the claimed practical utility of MNS in transfer learning.

Questions for authors
  1. Qualitative comparison on real-world data.

Can the authors provide qualitative comparisons of MNS with methods such as m-kNN and CKNNA on real-world data, illustrating the differences in the neighborhood relationships, semantic structures, and geometric information captured by these methods?

  1. Practical effectiveness in transfer learning.

Can the authors provide direct transfer learning or knowledge distillation experiments to verify whether AMNS can effectively predict actual knowledge transfer performance, rather than merely the classification accuracy gaps between models?

2 comments

We thank you for a detailed and constructive review. Here, we address your concerns in their original order.

Qualitative comparison: Representational alignment measures do not typically uncover particular shared structures in data in different ways from each other. They all measure pairwise similarities between samples within each space and are identified by how they each transform representational similarity matrices into scalar or vector-valued measures of alignment. There is no neighborhood within a space, or shared neighborhood between spaces, that one measure would identify that another would not. We are unaware of any examples in the representational alignment literature of comparative visualizations between representational similarity measures related to “nearest-neighbor rankings, semantic relationships, geometric distributions, and hierarchical clustering structures”. If there are any existing works you are aware of demonstrating the kind of qualitative comparisons we would be happy to take inspiration from them. We believe we have demonstrated quantitatively, with theoretical arguments and diverse results on both real world and synthetic data, the ways in which AMNS is distinguished from other approaches, both in its construction and what information it leverages to compute a similarity score.

Functional similarity: We agree that performance gap is an insufficient demonstration of the claim on its own. Error consistency (Geirhos, Meding & Wichmann, 2020) is a more information-rich canonical measure of functional similarity than coarse performance gaps. We computed every similarity measure’s ability to predict error consistency and observed superior performance from AMNS. In the results below, we expanded our model pool to 375 models, (64,685 total comparisons after removing all inter-family confounding observations), and simplified our experimental setup by simply thresholding the top pairs by alignment and by error consistency and computing the precision. AMNS also wins in the one-node decision tree setup we used originally.

MeasureLift
AMNS (AUC)3.47±0.22
AMNS (k=M)3.27±0.21
Spearman ρ3.20±0.21
Kendall τ3.20±0.21
TSI3.15±0.21
QSI3.13±0.23
RSA3.01±0.23
CKA2.81±0.23
AMNS (k=10)2.58±0.25
Procrustes2.25±0.22
CKNNA (AUC)2.20±0.22
m-kNN (k=10)2.11±0.25
CKNNA (k=10)1.90±0.20
SVCCA1.05±0.16
PWCCA1.00±0.15

Thank you for the rebuttal and clarifications. I have reviewed the authors' responses and will maintain my original rating.

Reviewer_hozXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Representational alignment is measured in two ways, each with its own set of problems. Geometric measures such as CKA and Procrustes provide global summary scores but assume a linear relation between the two spaces being aligned. They cannot be reliably localised to a chosen spatial scale without some effort. Ordinal measures, such as m-kNN and Spearman coefficient, drop the linearity assumption and admit a scale parameter. These are blind to the geometry of the spaces. The authors argue that practitioners currently stitch together semantically incompatible measurements from both families to measure alignment.

Thus the authors set out to construct a single measure that combines the strengths of both families of measures. They require that the new measure must be semantically consistent at spatial scale, non-saturating at extrema, respect geometry but without linearity assumptions and should not require expensive Monte-Carlo calibrations. The paper introduces Mutual Neighbor Similarity and its calibrated form AMNS, a hybrid between set-based and ordinal alignment measures. They claim that AMNS is (i) well-defined at every k without saturation, (ii) unique multi-scale and (iii) sensitive to geometry, including anisotropy without assuming linearity, (iv) handles ties naturally and (v) is invariant to orthogonal transformations, isotropic scaling, and neuron permutations, and sensitive to arbitrary invertible linear transforms.

Experiments cover four settings: controlled perturbations of synthetic embeddings at varying anisotropy, recovery of shared cluster sizes from critical points of the AMNS curve, multi-scale comparison against CKNNA on the Platonic Representation Hypothesis model suite, and ImageNet accuracy gaps.

Critical strengths
  1. The problem is a well-chosen one. The paper provides a measure specifically designed to answer the question about whether frontier models of different modalities are more aligned locally than globally (Singh et al. 2025, Gröger et al. 2026), across multiple k. The authors isolate the current pain point exactly (stitching of incompatible measures) and setup clear design goals.

  2. The core construction in Section 2 uses X only for ranking πi\pi_i, and Y only for the similarity values sYis^{Y_i}. Similarities are never compared across spaces or kernels. Thus it does not need to assume a linear relation between X's and Y's kernels to be geometrically sensitive. In contrast CKA assumes a shared kernel while m-kNN does not use either side's similarity values.

  3. The AMNS_k curve is non-saturating and consistent. It runs from a first-neighbor statistic at k=1 to a per-anchor Pearson correlation at k=M. There is no saturation ceiling at k=M and this improves over m-kNN. (Appendix B)

  4. The paper derives the max and null mean in closed form as low-cost functions of the Y profile, which is already computed. This avoids per-comparison simulation needed in CKA-style debiasing (Murphy et al., 2024).

Critical weaknesses
  1. In the paper, the authors claim that "even with chance correction, m-kNN converges to 1 as k grows to N - 1". However, if we use ∣A∩B∣=2k−M+∣Ac∩Bc∣|A \cap B| = 2k - M + |A^c \cap B^c|, then cc-mkNN_k is equal to cc-mkNN^{far}_{M - k}. This does not describe global convergence to 1. The large-k end measures agreement amongst the farthest points.

  2. cc-mkNN appears to be R_NX(K) from the co-ranking matrix literature (Lee & Verleysen 2009), and its uniform-in-log-k-AUC comes from Lee, Peluffo-Ordóñez & Verleysen, 2015. It is described as the scalar summary of the R_NX(K) curve. Additionally, trustworthiness (T(K)) and continuity (C(K)) assign a graded penalty to each neighbor that is present in one space but not the other (Venna & Kaski, 2001; Venna et al, 2010). These should address the criticisms aimed at m-kNN in Section 1. Some more measures that the paper should discuss and compare against - (i) LCMC (Chen & Buja, 2009), (ii) rank-biased overlap (Webber, Moffat & Zobel, 2010), (iii) Weighted Kendall-τ\tau (Vigna, 2015).

  3. The perturbation study in 4.1, Figures 1-2 has no meaningful ground truth. The authors describe the outcomes as "graceful" or "catastrophic", but there needs to be measurable quantities for the same. The paper does not give a criterion for what a correct drop in alignment must look like, under a 2% antipodal at θ\theta degrees perturbation. There is also a confounding factor between anisotropy and the effective perturbation size, where a rotation about the origin by angle θ\theta effectively moves a point by 2∣∣z∣∣sin⁡(θ/2)2||z|| \sin(\theta/2). Thus this θ\theta corresponds to a larger Euclidean displacement in an anisotropic cloud where the mass is far from the origin. The paper makes no claims about the rotation being about the origin, eigenvalue structure for anistropy, etc.

Ideally the authors should define what the correct drop should be under perturbation, and then rerun the perturbation with rotation about the origin, so that perturbation size is decoupled from anisotropy. They should specify parameters such as the rotation point and plane, seed count, etc.

  1. Section 4.4, Table 1 has AMNS winning by 17%, but there are issues in these numbers. (i) The target isn't functional similarity. Instead, the authors use ImageNet-1k accuracy gap, which can be a coarse proxy at best (two models can disagree on N number of images). They should use error consistency (Geirhos, Meding & Wichmann, 2020). (ii) Models from the same architecture family at the same scale will have similar representations and accuracy, and it is hard to distinguish whether the method is doing well in this case; this confounding factor needs to be handled. (iii) I didn't see the fraction of straddling pairs that were discarded, the decision tree depth defined, there needs to be more clarity.

The authors ought to rerun with error consistency, stratify by model family and report results both overall and within family, ideally commit to one variant per measure (rather than comparing best measure variant to best measure variant).

  1. The abstract mentions "superiority of our approach for transfer learning tasks" but then Section 4.4 defers this to future work.
Questions for authors
  1. Do you agree with the identity mentioned in weakness 1 and that it doesn't describe global convergence to 1? Can you provide empirical evidence that cc-mkNN "converges to 1 as k grows to N − 1"?

  2. How does AMNS relate to R_NX(K) and the series of papers that follow? Can you distinguish for the sake of novelty?

  3. Can you mention the parameters of your perturbation study more clearly?

2 comments

We thank you for a constructive review and particularly for the connections drawn to the dimensionality reduction/coranking matrix literature. Here, we address your concerns in their original order.

Weaknesses: m-kNN saturation and connection to coranking matrix measures: Thank you for highlighting the connection between “m-kNN” as it is referred to in Huh et al. (2024) and the identical R_NX(K) measure from the coranking matrix literature. Dimensionality reduction and representational alignment can both benefit from such connections. The language you cited is indeed an error on our part. m-kNN definitionally saturates at 1 without chance correction. With Cohen’s κ\kappa chance correction (R_NX(K)), it can be empirically demonstrated that it is overwhelmingly likely to saturate at 0, as the chances of farthest neighbor preservation are low between models. At k=N−2k=N-2, m-kNN increases with the number of samples whose largest neighbor is preserved, and decreases with the total number of samples. We confirmed this by computing m-kNN at k=M-1 on all the data used in this paper.

DatasetNModelsPairsMedianMean95th%%<0.01%<0.05%<0.1
PRH (WIT)1,02425300-0.0010.0060.02291.796.798.7
SEMASIA1,02425031,1250.0010.0070.02981.398.099.4
SEMASIA (large N)16,348581,65300.0020.00696.499.199.6

Other coranking matrix and ordinal measures: We agree that the other measures you cited each alleviate various issues of the binary set inclusion statistic R_NX(K)/m-kNN. However, the common thread of each of them is that they only compute a measure of similarity based on neighbor rankings, without considering geometry. As we discussed and demonstrated (Sec 4.1), this deficiency invites characteristic failure modes, such as anisotropy invariance. This is part of the motivation of AMNS. What distinguishes AMNS from these measures is that it weighs the severity of failures in neighborhood matching using the geometry of the embeddings, a feature absent from strictly ordinal methods.

Controlled Perturbation: Experimental Design: We should clarify that all embeddings in these experiments live on the hypersphere, ∥z∥=1\|z\|=1. This is a reflection of the established theory in representation learning that angular information is representationally privileged over magnitude (Wang & Isola, 2020). Anisotropy is therefore only an angular property of embeddings and angular displacements correspond to constant Euclidean displacements.

Ground truth: Thank you for raising this point! The outlier study in this work was inspired by TSI/QSI which, as is common in representational alignment, builds its analyses off of “desired properties” rather than ground truths, which are rare. However, since controlled perturbations compare points in the same ambient space before and after a transformation, we can treat the unperturbed positions as ground truth observations and define “true” alignment, as c=E[cos⁡(xi,perturbation(xi)]c = \mathbb{E}[\cos(x_i,\text{perturbation}(x_i)], as we do in our non-confoundedness checks in the appendix. We then apply a Cohen’s κ\kappa chance correction/scale normalization c→c−cos⁡‾1−cos⁡‾c \to \frac{c-\overline{\cos}}{1-\overline{\cos}} to compare to the curves of the tested similarity measures. We vary the anisotropy over 12 gradations from 0 to 0.99 and compute MAE of the ground truth surface GT=GT(perturbation strength, anisotropy) to each measure’s surface. We take the mean over 5 seeds. AMNS best tracks the ground truth.

MeasureMAE
AMNS0.0656±0.0011
CKA0.0842±0.0081
SVCCA0.0994±0.0036
PWCCA0.1025±0.0033
RSA0.1182±0.0008
Spearman0.1183±0.0031
Procrustes0.1187±0.0037
Kendall0.1204±0.0030
CKNNA0.1204±0.0012
QSI0.1216±0.0003
TSI0.1235±0.0003
m-kNN0.1237±0.0005

Functional similarity and transfer learning: We agree that error consistency is a more information-rich measure of functional similarity than coarse performance gaps. We computed every similarity measure’s ability to predict error consistency and observed superior performance from AMNS. In the results below, we expanded our model pool to 375 models, (64,685 total comparisons after removing all inter-family confounding observations), and simplified our experimental setup by simply thresholding the top pairs by alignment and by error consistency and computing the precision.

MeasureLift
AMNS (AUC)3.47±0.22
AMNS (k=M)3.27±0.21
Spearman ρ3.20±0.21
Kendall τ3.20±0.21
TSI3.15±0.21
QSI3.13±0.23
RSA3.01±0.23
CKA2.81±0.23
AMNS (k=10)2.58±0.25
Procrustes2.25±0.22
CKNNA (AUC)2.20±0.22
m-kNN (k=10)2.11±0.25
CKNNA (k=10)1.90±0.20
SVCCA1.05±0.16
PWCCA1.00±0.15
AMNS also wins in the one-node decision tree setup we used originally. We agree that these results only suggest the utility of AMNS in transfer learning but large scale direct experimentation is required.
Reviewer_hozXOpenReview ↗

Response to rebuttal

Thank you for the rebuttal. My concerns are addressed for the most part. A few remaining points: (1) error-consistency margin is around ∼0.1σ\sim 0.1\sigma, so your method being the superior predictor needs evidence or change, and (2) AMNS at k=10 drops from first to ninth with the change in target, and the paper needs to discuss this change. The abstract’s transfer-learning claim should also be rephrased to match what Section 4.4 actually claims.

I will raise my score from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.