- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Here, the authors study how input selection and vowel categories affect representational similarity comparisons, using vowel acoustics as an external reference because there is a known underlying structure, which is the typical problem in representational similarity comparisons.
Critical strengths
- Using vowel acoustics as an external reference is an interesting idea because of its known underlying structure.
Critical weaknesses
- W1. The main contribution is still unclear to me. That representational similarity depends on the inputs being compared is expected. What I would find useful is an explanation of when this dependence makes a conclusion misleading and how to avoid that problem. However, the paper moves between changes in scores, changes in rankings, and claims about correctness without clearly separating them. Additionally how the paper is written makes it harder to follow, especially in the abstract and introduction, where terms such as "conditions," "admission," and "bifurcations" appear before a concrete explanation of what the authors do.
- W2. The introduction motivates the problem pretty strongly, but the external reference does not establish that a conclusion that changes across category sets is wrong. For example, one model could recover vowel geometry better for one set of categories, while another performs better for a different set. Both conclusions could be correct for those sets. For me, what the authors demonstrate is sensitivity, but the repeated claims that they make about correctness go further than that.
- W3. Also, the layer-selection experiment does not isolate the issue of multiple-comparison bias. Moving from the best layer to a fixed relative depth changes which representation is evaluated, and some layers may genuinely capture vowel geometry better than others. For me, it is unclear why the authors argue that the decrease in performance is attributed to selection bias.
- W4. I also think that some relevant prior work is missing from the discussion. For example, ReSi (ReSi: A Comprehensive Benchmark for Representational Similarity Measures) evaluates similarity measures using explicit groundings of similarity. The authors should discuss prior work more thoroughly and also discuss what the current paper adds relative to these.
- W5. Lastly, I also find that the practical guidance remains limited. Failure of the tested coverage statistics or filtering-bias measure does not establish that no inexpensive diagnostic can work. I would have appreciated an experiment showing how the proposed recommendations improve the reliability of a comparison.
Questions for authors
See weaknesses.
2 comments
I thank the authors for their comments. However, the main problems have not been addressed, and the authors do not want to update the revised manuscript despite the reviewers asking for textual or contributional clarifications. Furthermore, even if the authors provide a substantially revised version, including a rewrite of the abstract and introduction, we reviewers have no way to check it. I thus maintain my score of recommending a clear rejection.
Thank you for the review.
W1–W2. We agree that the dependence itself is expected, and that showing a conclusion changes is not showing that it is wrong. What we measure is how large the dependence is, which aspect of the condition set carries it, and when it reverses a conclusion: on small inventories and for cross-language comparisons. We will replace "wrong" with "not identified": a claim stated without reference to the category set is not identified when its sign depends on which defensible set is used. The revision will keep scores, rankings and identification separate throughout.
W3. Please see our reply to Reviewer y9vG: selecting layers on held-out languages gives a smaller estimate than the reported 46%. We will report it as an upper bound and qualify the simulation.
W4. Thank you for the pointer. ReSi evaluates measures against explicit groundings of similarity; as far as we can tell, it does not vary the set of inputs over which similarity is computed, which is the variable we study. The two are complementary, and the revision will discuss ReSi.
W5. Agreed. The revision will test the recommendations directly: simulated analysts each make one defensible choice per decision, and we measure how often their conclusions agree with and without the recommended reporting. We will not claim that no inexpensive diagnostic can work, only that the two we tested do not.