- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces functional geometry under intervention, a framework for deciding when the geometric structure of a representation is functionally meaningful. The authors hold sentence identity fixed while varying its acoustic realization across clean speech, −3 dB noise, and two speech-enhancement variants that have opposite effects on human comprehension. They characterize sentence manifolds with MFTMA across 429 frozen model-layer representations, and validate the geometry against leave-one-condition-out identity retrieval, 4×4 pairwise condition-transfer matrices, and linear model-to-brain encoding in a 25-participant fMRI experiment. The main findings are that late Whisper encoders develop a coordinated untangling regime whose geometry predicts held-out identity preservation, that pairwise transfer separates representations preserving identity only locally from those generalizing across the full intervention space, and that beneficial enhancement selectively shifts the inferior frontal gyrus toward the clean-speech direction.
Critical strengths
S1: The pairwise-transfer analysis exposes a distinction hidden by aggregate invariance measures. Some models (wav2vec2, WavLM) preserve sentence identity only between acoustically neighboring conditions, while others (Whisper) generalize across the full intervention space. This is a genuinely informative characterization of existing models and gives the community a reusable diagnostic for telling apart "isolated identity" from "discarded information."
S2: The design is methodologically careful. IFG was pre-specified as the cortical target before model-to-brain fitting, all condition-level analyses are strictly train-three/test-one, and the beneficial/harmful intervention is defined by recovered meaning rather than perceived quality. The authors are also explicit about which comparisons are descriptive rather than inferential.
Critical weaknesses
W1: The model-level claims are descriptive, not causal. The local-vs.-general transfer gap confounds architecture, scale, and training objective simultaneously, and the geometry-to-function inference rests on only seven model clusters (21 stage summaries) with wide bootstrap intervals. As it stands, the paper re-describes known models with a more informative lens but does not explain why they differ, which is the main reason my score stays at weak accept.
W2: The core geometric quantities are estimated from only four acoustic realizations per sentence, so radius and effective dimension are a coarse proxy for the true nuisance-subspace complexity. Combined with the small number of independent model clusters, the geometry-to-function claim is based on only 21 data points (7 models × 3 stages).
Questions for authors
Q1: If the fMRI section were removed, what remains as a standalone contribution to the ML community? The local-vs.-general distinction appears to stand on its own, but the paper currently presents the cortical constraint as an independent line of evidence.
Q2: Is there any evidence that could attribute the local-vs.-general gap to the weakly-supervised ASR objective rather than scale or architecture, or is the comparison intended to remain strictly descriptive?
Q3: Adding more acoustic realizations per sentence would make the effective dimension closer to true nuisance dimensionality. You already flag this as a limitation, so I note it only as the natural next step.
4 comments
Response to Reviewer's Question 3
Question 3. Adding more acoustic realizations per sentence would make the effective dimension closer to true nuisance dimensionality.
The four-condition scope () is explicitly stated in Sections 3.3 and 5.2. Within the practical constraints of human fMRI experiments, we made every effort to design and control a meaningful set of acoustic interventions while maintaining sufficient trials per participant. Our priority was to apply the same controlled interventions to human participants and computational speech models, enabling direct comparisons of their representational geometry under matched acoustic conditions. The four conditions were carefully selected to capture distinct acoustic transformations with contrasting effects on human comprehension, balancing experimental feasibility with scientific relevance.
Future work can extend the MFTMA and LOCO framework to additional SNR levels, noise types, and speech-enhancement methods, as well as broader model architectures and participant samples. Such extensions will allow us to evaluate scalability and generalizability beyond the four conditions examined here, while further characterizing manifold geometry across a richer intervention space.
Thank you for the detailed responses. I believe my score correctly reflects the paper's state, hence I maintain it.
Updated Manuscript
Thank you again for your feedback. We would like to inform you that we have uploaded an updated version of our manuscript.
The revised version improves the clarity, organization, and presentation of the Abstract, Introduction, Related Work, Methods, and Discussion. We have also expanded Appendices C.1, C.3.2, and D.7 to provide additional methodological explanations, experimental details, and statistical sensitivity analyses.
Importantly, our core approaches, experimental design, primary analyses, results, figures, and scientific interpretations remain unchanged. These revisions are intended to improve methodological transparency and reproducibility while preserving our original findings and contributions.
Thank you again for your time and consideration.
Response to Reviewer's Question 1 and Question 2
We sincerely thank the reviewer for the careful evaluation, valuable feedback, and encouraging assessment of our work. We particularly appreciate the recognition of our pairwise-transfer analysis as a reusable diagnostic for distinguishing local from broader identity preservation, as well as the methodological rigor of our experimental design, including behaviorally grounded acoustic interventions, held-out evaluation, and the pre-specified IFG analysis. We will clarify the reviewer's key questions and suggestions below.
Question 1. If the fMRI section were removed, what remains as a standalone contribution to the ML community? The local-vs.-general distinction appears to stand on its own, but the paper currently presents the cortical constraint as an independent line of evidence.
Our work makes three standalone contributions to the ML community, independent of the cortical encoding analysis.
1. Functionally grounded manifold evaluation under acoustic interventions. We extend mean-field theoretic manifold analysis (MFTMA) to sentence-level speech representations under four controlled acoustic conditions. Unlike conventional noise-robustness evaluations based primarily on SNR, our framework includes two practical enhancement methods with experimentally demonstrated opposite effects on comprehension: neural enhancement (SEMamba) improves comprehension, whereas classical enhancement (Ephraim–Malah MMSE) reduces it. This provides a behaviorally grounded setting for examining whether changes in manifold geometry accompany meaningful preservation of sentence identity.
2. Linking manifold geometry to held-out functional generalization. We connect geometric descriptors to leave-one-condition-out (LOCO) sentence retrieval, testing whether representations preserve identity under an unseen acoustic condition. Across seven direct-audio encoders at three processing stages, higher manifold capacity, smaller radius, and lower effective dimension are associated with stronger held-out retrieval (ρ=.643,−.604,−.573; FDR-corrected q=.012,.012,.021; Figure 4). Because these descriptors are correlated, our findings support a coordinated geometric regime rather than identifying a single dominant factor. The key contribution is evaluating geometry through functional generalization, rather than treating geometric invariance as sufficient evidence of robustness.
3. Directional pairwise transfer distinguishes local from broader robustness. Our 12 directed condition-to-condition evaluations reveal differences hidden by aggregate LOCO performance (Figure 5; Appendix A.2, Figure A3). Some representations preserve sentence identity between acoustically similar conditions but fail across more challenging interventions, whereas others generalize more broadly. This provides a layerwise diagnostic for identifying condition-specific stability versus broader cross-intervention identity preservation.
Together, these contributions establish a standalone ML framework connecting controlled acoustic interventions, representational geometry, and held-out functional performance. Human fMRI adds a complementary assessment of cortical relevance, but is not required to establish these ML findings.
Question 2. Is there any evidence that could attribute the local-vs.-general gap to the weakly-supervised ASR objective rather than scale or architecture, or is the comparison intended to remain strictly descriptive?
Our study characterizes differences in cross-condition generalization across model families but does not attribute them causally to architecture or training objectives. The models differ in architecture, scale, supervision, and training data, preventing these factors from being isolated.
Accordingly, the local-versus-general distinction is an empirical characterization of representational behavior, not a claim about its architectural cause or universality. We will clarify this limitation in Section 5.2. Future controlled comparisons, such as pretrained versus fine-tuned wav2vec2 or WavLM, could help determine which training factors contribute to the observed differences between local and broader generalization.