acceptodds
Under review as a conference paper at ICLR 2027

A Better Metric Will Not Save You: What You Compare Beats How You Compare It

Abstract

Representational similarity measures (RSA, CKA, Procrustes) are chosen for the transformations they treat as irrelevant, and all of them take the set of conditions as given. That set is fixed by two decisions, which inputs are admitted and how they are grouped, both taken before the matrix a measure operates on exists: no invariance property reaches them. We measure what they are worth where the arrangement the conditions ought to have is known independently of the systems compared, the acoustic organisation of vowels, across fifteen languages and twenty-seven representations. Changing which categories are compared, at a fixed number of them, moves the gap between two representations by more than twice the effect being measured, while changing how many moves it not at all. In this setting rankings of models are identified under these choices, and rankings across languages are not.

Primary area:
interpretability and explainable AI
Keywords:
representational similarity, RSA, CKA, Procrustes analysis, ground-truth evaluation, condition set, self-supervised speech models, vowel space, cross-lingual evaluation, reproducibility, methodological audit, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GuexOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Here, the authors study how input selection and vowel categories affect representational similarity comparisons, using vowel acoustics as an external reference because there is a known underlying structure, which is the typical problem in representational similarity comparisons.

Critical strengths
  • Using vowel acoustics as an external reference is an interesting idea because of its known underlying structure.
Critical weaknesses
  • W1. The main contribution is still unclear to me. That representational similarity depends on the inputs being compared is expected. What I would find useful is an explanation of when this dependence makes a conclusion misleading and how to avoid that problem. However, the paper moves between changes in scores, changes in rankings, and claims about correctness without clearly separating them. Additionally how the paper is written makes it harder to follow, especially in the abstract and introduction, where terms such as "conditions," "admission," and "bifurcations" appear before a concrete explanation of what the authors do.
  • W2. The introduction motivates the problem pretty strongly, but the external reference does not establish that a conclusion that changes across category sets is wrong. For example, one model could recover vowel geometry better for one set of categories, while another performs better for a different set. Both conclusions could be correct for those sets. For me, what the authors demonstrate is sensitivity, but the repeated claims that they make about correctness go further than that.
  • W3. Also, the layer-selection experiment does not isolate the issue of multiple-comparison bias. Moving from the best layer to a fixed relative depth changes which representation is evaluated, and some layers may genuinely capture vowel geometry better than others. For me, it is unclear why the authors argue that the decrease in performance is attributed to selection bias.
  • W4. I also think that some relevant prior work is missing from the discussion. For example, ReSi (ReSi: A Comprehensive Benchmark for Representational Similarity Measures) evaluates similarity measures using explicit groundings of similarity. The authors should discuss prior work more thoroughly and also discuss what the current paper adds relative to these.
  • W5. Lastly, I also find that the practical guidance remains limited. Failure of the tested coverage statistics or filtering-bias measure does not establish that no inexpensive diagnostic can work. I would have appreciated an experiment showing how the proposed recommendations improve the reliability of a comparison.
Questions for authors

See weaknesses.

2 comments

Thank you for the review.

W1–W2. We agree that the dependence itself is expected, and that showing a conclusion changes is not showing that it is wrong. What we measure is how large the dependence is, which aspect of the condition set carries it, and when it reverses a conclusion: on small inventories and for cross-language comparisons. We will replace "wrong" with "not identified": a claim stated without reference to the category set is not identified when its sign depends on which defensible set is used. The revision will keep scores, rankings and identification separate throughout.

W3. Please see our reply to Reviewer y9vG: selecting layers on held-out languages gives a smaller estimate than the reported 46%. We will report it as an upper bound and qualify the simulation.

W4. Thank you for the pointer. ReSi evaluates measures against explicit groundings of similarity; as far as we can tell, it does not vary the set of inputs over which similarity is computed, which is the variable we study. The two are complementary, and the revision will discuss ReSi.

W5. Agreed. The revision will test the recommendations directly: simulated analysts each make one defensible choice per decision, and we measure how often their conclusions agree with and without the recommended reporting. We will not claim that no inexpensive diagnostic can work, only that the two we tested do not.

I thank the authors for their comments. However, the main problems have not been addressed, and the authors do not want to update the revised manuscript despite the reviewers asking for textual or contributional clarifications. Furthermore, even if the authors provide a substantially revised version, including a rewrite of the abstract and introduction, we reviewers have no way to check it. I thus maintain my score of recommending a clear rejection.

Reviewer_LGvpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper evaluates representation similarity measures from a new angle - by altering object set under comparison. It turns out that this intervention changes measures significantly, even more than a typical difference between representations themselves. Experiments are provided with english vowels and a diverse set of representation models (SSL speech models, supervised encoders, neural audio codecs, and non-learned spectral features) across 15 languages from the FLEURS corpus.

Critical strengths
  1. The main research focus of the paper is novel - to study how set of inputs affects relatives changes of representational measures.
  2. The use of extrinsic ground truth (2D space of formants). However, I have concerns about it, see below.
  3. A rigorous experimental protocol in Appendix L.
Critical weaknesses
  1. A limited scope for the main claim. While the paper evaluates 15 languages, two of its most critical structural findings rest almost entirely on English.
  2. Using 2D space of formants as a ground truth is questionable. 2D space encodes physical properties, while neural representations encode statistical properties of vowel or token colocation.
  3. Language of the paper has a large room for improvement. It has lots of syntactically complex, dense structures, or rely on highly specific jargon in ways that obscure their meaning, e.g.: "The three also differ in what they leave to the analyst: RSA fixes neither the distance d nor the statistic comparing the triangles, which CKA and Procrustes fix by definition", "The thesis is therefore best stated by regime", " The premise is correct and the gap nonetheless widens: the reference is itself a coordinate-wise median over formant measurements, and the tokens it discards are not those lying far from a centroid in a spectral space, so matching the name of an estimator does not match its effect. Neither configuration is principled and we report both. ", etc. I recommend to use some AI tool for language polishing.
  4. If metrics agree on rankings, but not on gaps (RSA, CKA, and Procrustes largely agree on the coarse ranking of the 27 representations), maybe a sensitivity of gaps to specific subsets is not a serious issue?
Questions for authors
  1. What do you mean in abstract "In this setting rankings of models are identified under these choices, and rankings across languages are not." ?
  2. What is \rho in Table 1?
  3. The evaluation of similarity measures on various subsets of inputs leads to a combinatorial explosion. Do you see any remedy?
  4. I'm not a experiment in speech processing, but for colours we have different parametrizations: RGB, CMYK, HCV, etc. Are results stable when one takes a different "ground truth" for vowels?

post rebuttal. Thank you for addressing my questions. Since authors will not upload a revision, my score remains unchanged.

1 comment

Thank you for the detailed questions.

Q1. Under the choices we vary, the ordering of models is stable enough to support a conclusion, whereas the ordering of languages changes under either of two unrelated choices (the inclusion threshold and the distance convention). No conclusion about which language's vowel space is best recovered can therefore be drawn from a single pipeline. We will rephrase the sentence.

Q2. ρ is the RSA score: Spearman's correlation between the vectorised dissimilarity matrices of a representation and of the reference (§3, Eq. 1). The caption should have said so.

Q3. We do not enumerate subsets: we draw them at random (500 per cardinality) and report the distribution. The revision will restrict the draws to a declared space of defensible condition sets and show how many draws are needed for the distribution to stabilise.

Q4. We have run the analysis under two parametrisations of the reference, standardised hertz and standardised Bark: the qualitative conclusions hold under both, while magnitudes differ (see our reply to Reviewer y9vG). The revision adds further parametrisations (ERB, with and without F3) and a reference that is not acoustic at all, based on phonological features.

Weaknesses. (i) The revision extends the composition analysis to every language whose inventory allows it, and to a dataset whose generative factors provide an exact reference. (ii) We do not claim that models should encode formants: the reference serves to evaluate a common comparative claim, namely which representation better recovers the acoustic organisation of vowels, and the non-acoustic reference addresses the concern directly. (iii) We will rewrite the paper; thank you for the examples. (iv) Rankings survive at typical inventory sizes but collapse at five categories (rank correlation 0.54 between draws over 27 representations, 0.31 over the trained ones), the size of the Spanish, Mandarin and Polish inventories; cross-language rankings are not identified. These are the cases in which the sensitivity changes conclusions, and the revision will make them explicit.

Reviewer_y9vGOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how choices about what to compare affect conclusions about speech representations. It compares 27 representations across 15 languages against vowel acoustics, then varies the categories and inputs included, how they are aggregated, and which layers are selected. The main English experiment shows that changing the vowel categories can substantially change model comparisons, even when the number of categories stays fixed.

Critical strengths
  • Holding category count fixed is a well-chosen test. The coverage and normalization controls help rule out simpler explanations for the composition effect.

  • The paper gives researchers concrete choices to check when comparing representations. It also distinguishes where its conclusions hold, including differences between small and larger inventories and between within-language and cross-language comparisons.

Critical weaknesses
  • The manuscript and code use three different acoustic references: standardized Bark in Appendix B, standardized Hz in scoring, and unstandardized Bark in the reliability analysis. These can produce different distance rankings. The authors need to identify which reference generated the tables and check the main results and reliability using the same construction. How much this changes the results is still unknown.

  • The simulation doesn't isolate the selection bias behind the reported 46%. Differences across layers may reflect genuine specialization as well as noise. The figure needs a narrower interpretation unless the authors can estimate selection bias using separate data for choosing and evaluating layers.

Questions for authors

Which reference generated the tables, and do the results survive a consistent recipe? Can the 46% be validated independently or qualified as a conditional simulation?

1 comment

Thank you for the close reading, including of the code.

Acoustic reference. You are right. The scores were computed from formants in hertz, standardised on each condition set, while the reliability analysis used unstandardised Bark; Appendix B describes neither accurately. We have recomputed the scores, the reliability analysis and the anchored scale under a single construction (Bark, standardised once per language). The following hold: the sign and order of magnitude of the pre-training gap under all three measures, the separation of composition from the other choices, the effect of granularity, the survival of the model ranking at fourteen categories and its collapse at five, and the non-identification of cross-language rankings. Several magnitudes change, among them the ratio reported in the abstract, which falls from about 2.5 to about 1.6. The revised version will use the single construction and report the other as a sensitivity analysis.

Layer selection. We agree that the simulation conflates selection with genuine layer specialisation. Selecting each model's layer on fourteen languages and evaluating it on the fifteenth attributes 23% of the margin to selection (bootstrap interval 11–38%) under the original construction, and 30% (9–57%) under the single one, against the 46% reported. Since this estimate still includes language-specific specialisation, it is an upper bound; a split by utterance will separate the two. The revision will report the held-out estimate and qualify the simulation accordingly.

Comments on the paper

1 comment
Area_Chair_HQ26OpenReview ↗

Final reminder: discussion closes today (Nov 18 AoE)

The author-reviewer discussion closes today, November 18 (AoE).

Dear reviewers,

None of the reviewers has yet responded to the authors' rebuttal on this paper. Please read the authors' response and post a brief reply here before the deadline: whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Your response is important for the final recommendation.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.