- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper has a clear goal: to develop a representation similarity measure that captures both geometric structure and neighbor rankings. The method is conceptually simple. Building on mutual -nearest-neighbor matching, it compares similarity profiles under the neighbor orderings induced by the two representations. It then calibrates the resulting scores using their maximum values and a random-permutation baseline. The experiments are fairly comprehensive, covering sensitivity to perturbations, alignment across spatial scales, real-model comparisons, and computational efficiency.
Critical strengths
- The idea of blending the geometry and the ranking order is very simple yet effective idea.
- The whole paper’s story telling is good, I think all the readers can learned things from the manuscript.
Critical weaknesses
- My main concern is whether the sensitivity experiments convincingly demonstrate the advantage of AMNS. Procrustes also captures the expected trend as perturbation strength increases. However, the prescribed perturbation strength does not itself establish how much the alignment score should decrease. Without an independent reference for the appropriate response, it is unclear why Procrustes should be considered overly sensitive while AMNS is judged to respond appropriately.
- More downstream applications. Authors list the functional alignment results in the Sec 4.4, which would be great, and I also see that the authors seems be constrained by the computation resource. Beyond transfer learning, I encourage the authors to evaluate AMNS in other downstream settings where representation alignment matters, such as cross-model embedding translation in retrieval systems.
- When is AMNS less effective? Different representation-similarity measures encode different invariances and may fail under different conditions. Although the paper discusses the desirable properties of AMNS in detail, it provides limited analysis of its failure modes. I encourage the authors to characterize settings in which AMNS may become unreliable or less informative, and to clarify when it should be preferred over existing measures.
Questions for authors
- In the Table 1, I see the AMNS_10 seems better than AMNS_M, could the authors explain why?
- In the Figure 2, I see Procrustes, AMNS, and CKA have similar performance, but in the introduction, the authors say they are in the different category, could author explain why?
- I see a lot of gaps between paragraphs in the P.7, the ticks in the Figure 3 are quite small.
2 comments
Dear authors,
First, I am curious why you say "At this time we are computationally constrained to experiments requiring no training or inference"? What kind of equipment you have?
Second, I think my concerns are fully addressed, but I think my score is enough for reflecting my evaluation to this paper.
Best,
Reviewer 1vkp
We thank you for a detailed and constructive review. Here, we address your concerns in their original order.
Controlled Perturbation ground truth: Thank you for raising this point! The outlier study in this work was inspired by QSI/TSI which, as is common in representational alignment, builds its analyses off of “desired properties” rather than ground truths, which are rare in this field. However, since controlled perturbations compare points in the same ambient space before and after a transformation, we actually can treat the unperturbed positions as ground truth observations and define “true” alignment, as c=E[cos(xi,perturbation(xi)], as we do in our nonconfoundedness checks in the appendix. We then apply a Cohen’s κ chance correction/0 to 1 scale normalization: c→1−cosc−cos to compare to the curves of the tested similarity measures. We vary the anisotropy over 12 gradations from 0 to 0.99 and compute the MAE of the ground truth surface GT=GT(perturbation strength, anisotropy) to each measure’s corresponding surface. We take the mean over 5 seeds. AMNS best tracks the ground truth.
Real world experiments: We are eager to test the utility of AMNS in different representational alignment contexts. At this time we are computationally constrained to experiments requiring no training or inference. All real embeddings used in this paper were publicly available. We have expanded our functional alignment experiments to go beyond simply predicting global performance differences from alignment, to predicting a more informative error consistency measure (Geirhos, Meding & Wichmann, 2020). The results show that AMNS alignment is the best predictor of these functional differences.
We believe that our split between analyzing the properties of our measure and real world experimentation matches typical representational alignment methods papers (arXiv:2606.16379, arXiv:2110.14739).
Failure modes: Of the failures we mention in our discussion, we believe the most pertinent is that AMNS is not a mathematical metric. This precludes its usefulness in large scale evaluations of many models where clustering analyses (or other analyses that treat model space as a metric space) are desired. In addition to this and the other outstanding questions mentioned, while we found that AMNS best tracks ground truth alignment in controlled perturbation settings across the anisotropy spectrum, it is not optimal in the extremes (mean cosine = 0 and mean cosine -> 1). While we are not aware of models that have anisotropy outside of the “AMNS optimal” regime, there are certainly low anisotropy settings where it would seem AMNS is overly sensitive to perturbations. This seems reasonable to us given that AMNS exists to be an ordinal measure that is aware of the impact of geometric idiosyncrasies. In settings where there are no such idiosyncrasies, AMNS can be overly sensitive to noise.
Questions:
While we have not characterized this exactly, our best hunch is that local alignment information is simply a more information-rich signal for this prediction task. Whether two models construct the same neighborhoods is more informative of their classification abilities than the large scale structure of those neighborhoods.
AMNS, CKA, and Procrustes shape metric are all geometric, meaning they are aware of the similarity of embeddings, not just relative similarity rankings. As such, they are all sensitive to changes of geometric features of the data such as anisotropy. This explains the results in Figures 1 and 2. AMNS is distinguished from CKA and Procrustes because it does not directly compare pairs of points across spaces. Rather, given an anchor within a space it compares each neighbor to the point that has the same neighbor ranking in the opposite space. By this construction, it does not assume that the two point cloud geometries need to share any geometric structure between them (such as anisotropy).