acceptodds
Under review as a conference paper at ICLR 2027

Computational Dissection of Sound-to-Meaning Hierarchies in Speech Foundation Models and the Human Brain

Abstract

Whether speech foundation models and the human brain abstract sound into meaning along a convergent route remains unresolved: prior comparisons rely on brain scores, which are hampered by low-level confounds and do not address compositional meaning. We introduce a spoken multi-definition-to-target paradigm in which acoustic, phonetic, and lexical form vary while meaning converges: differently worded definitions, read by multiple speakers, converge on a targeted semantic concept. We dissect eleven speech foundation models spanning masked prediction (HuBERT, WavLM), contrastive prediction (wav2vec 2.0), weak supervision (Whisper), and audio-LLM encoders (Qwen2-Audio, Qwen3-Omni), as well as intracranial recordings from 29 patients performing the same task. Layer-wise decoding and representational geometry analyses in the models reveal cascaded dynamics: speaker identity separates first, followed by surface form and phonetics, each persisting across most layers, whereas lexico-syntactic category and target semantics emerge last, in narrow bands whose depth is set by the training objective, and the semantic target is decodable from held-out definitions. Temporal response function encoding with interpretable speech and linguistic features shows that continuous acoustic and prosodic features trade off against categorical linguistic features across depth, whereas cortex abstracts along a graded, nested hierarchy from core auditory regions to distributed frontal, parietal, and medial hubs, in which part of speech (POS) behaves as a lexical property and constituency structure engages posterior temporal and parietal cortex. Temporal generalization decoding shows that the two systems share dynamics for phonetic and statistical word-level information but diverge for rule-based structure and sentence-level semantic categories. Low-level information is thus spread across depth in the models but localized in cortex, and higher-order structure shows the reverse pattern. By testing speech models for brain-like abstraction rather than brain-like prediction, this work establishes a framework that locates where and how artificial and biological speech networks converge and diverge, and points to the model layers that could serve neuroprostheses decoding meaning as well as speech.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Self-supervised speech models, speech foundational models, speech recognition, semantic representation, human intracranial recordings, encoding models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Qw1KOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether speech foundation models and the human brain share similar representational hierarchies during the transformation from sound to meaning. The authors introduce a multi-definition-to-target paradigm that separates semantic concepts from their surface linguistic forms. They examine eleven speech model checkpoints across different training objectives, together with intracranial recordings from 29 patients. Through layer-wise probing, representational geometry, temporal generalization, and feature-based encoding analyses, the paper identifies both similarities and differences between artificial and biological speech processing. The results suggest that lower-level linguistic representations exhibit more similar dynamics across the two systems, whereas higher-level syntactic and semantic representations show substantial differences in their organization and temporal stability.

Critical strengths
  1. Interesting research question and experimental paradigm. The paper goes beyond conventional brain–model alignment based on neural predictivity and instead investigates the linguistic properties and temporal dynamics of representations within each system. The multi-definition-to-target paradigm is particularly interesting because it evaluates semantic generalization across different surface forms.

  2. Comprehensive empirical analysis. The study covers a diverse collection of speech foundation models, including self-supervised, ASR-supervised, and audio-LLM encoders. The combination of computational models and intracranial recordings from 29 patients provides a valuable empirical foundation for the analysis.

  3. Interpretable findings about representational organization. The combination of probing, temporal generalization, and feature-based encoding offers complementary perspectives on speech representations. The observation that models and cortex show greater similarity for lower-level linguistic information than for higher-order compositional structure is particularly interesting.

Critical weaknesses
  1. Does the semantic decoding task adequately isolate compositional meaning? Although the target word is excluded from the definitions, the task involves only 33 target concepts, each associated with four definitions. Successful cross-definition classification may still rely on lexical or topical associations rather than compositional semantic understanding. Could the authors include a simple lexical baseline, such as a bag-of-words classifier operating on transcripts, or an analysis controlling lexical overlap across definitions? This would help establish how much of the semantic decoding performance reflects abstraction beyond lexical cues.

  2. Comparability of temporal dynamics between models and cortex. The temporal generalization analysis is an important contribution, but the interpretation of the differences between model and neural dynamics requires further clarification. In particular, the evaluated speech encoders generally have access to bidirectional context, whereas the brain processes incoming speech online. Could this difference in available context contribute to the observed temporal generalization patterns? It would be useful to discuss this limitation and, if feasible, examine the robustness of the findings under more closely matched temporal contexts or preprocessing choices.

  3. Interpretation of the different representational hierarchies. The paper argues that models and cortex exhibit substantially different organizations of low- and high-level linguistic information. However, the comparison involves different representational axes (model depth versus cortical anatomy), and the variance-partitioning procedures are not identical across the two systems. How sensitive is the conclusion to these methodological differences? The authors should clarify whether the observed differences reflect distinct representational organizations or, at least in part, the different ways in which the two systems are analyzed.

  4. Statistical robustness of temporal generalization results. Appendix C indicates that some temporal generalization contours use an uncorrected significance threshold. Given the large number of temporal comparisons, could the authors clarify whether the main conclusions remain robust under appropriate multiple-comparison correction?

Questions for authors

See weaknesses.

Reviewer_sRm2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors investigate how 11 speech foundation models and how the human brain (from 29 patients) transform acoustic signals into meaning across various spoken tasks. Quantitative and qualitative analyses show that models encode speaker identity and phonetic information in early layers, while semantic representations appear in later layers. On the other hand, the human cortex characterizes low-level acoustic processing and distributes higher-order linguistic representations across different regions. Meanwhile, models and brains share low-level dynamics but differ in high-level semantics. The framework explores the convergence and divergence of models and brains, serving as a comprehensive study for meaning-decoding speech neuroprostheses.

Critical strengths

(1) This paper includes results of various speech tasks, and comprehensive experiments have been conducted. The results serve as a useful guideline for research in this research direction. (2) This work combines layer-wise decoding, representational geometry, and temporal analyses, aiming to explore where models and brains converge. The analyses provide where and how artificial and biological speech networks converge and diverge.

轉換為 PDF

Critical weaknesses

(1) One major concern we have is that the recordings come from epilepsy patients with clinically determined. Accordingly, potentially uneven electrode placements may limit the conclusions about the organization of language processing throughout the brain. (2) This study investigates correlations between different representations of models and brains. The authors may want to provide some potential applications based on the study, such as the decoding process of different speech tasks. (3) The study provides qualitative and quantitative analyses of different representations in models and brains. Similar studies have been conducted previously, and thus the scientific depth and novelty of this work appear insufficient to meet the standards of a top-tier machine learning conference (ICLR). (4) The experiments did not include the Whisper Large model. Please include the model as well. (5) The 11 models examined in this study differ in architecture, training data, model size, and training objectives, making it difficult to attribute the observed differences to any specific factor.

Questions for authors

(1) Please try to clearly attribute the observed differences to specific factors. (2) Please explain how to exclude line noise, epileptiform activity, or other artifacts from the brain signals. (3) Please describe how to use the findings of this study for speech decoding tasks. (4) Please include the results of using the Whisper Large model.

Reviewer_CcUYOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares linguistic representations in eleven frozen speech-model checkpoints from six families with intracranial high-gamma responses from 29 participants. Its controlled task uses four differently worded definitions for each of 33 concepts, with multiple speakers and without speaking the target word. Layer-wise probes, clustering, feature-based temporal response functions, and temporal generalization are used to separate acoustic, phonetic, lexical, syntactic, and target-related information. Leave-one-definition-out target classification reaches 52-64% accuracy across the models, compared with approximately 3% chance (Table 1). The authors report that lower-level information persists across model depth while higher-level information is concentrated in narrower, objective-associated layer ranges. They further report shared model-brain temporal patterns for phonetic and some word-level information, with differences for constituency and semantic categories, and contrast these model hierarchies with a more distributed cortical linguistic organization. The contribution is an interpretable comparative analysis and controlled stimulus paradigm rather than a new predictive architecture.

Critical strengths
  1. A useful controlled test of generalization across wordings. The multi-definition design, omission of the target word, and definition-disjoint semantic evaluation provide a meaningful improvement over decoding a repeated sentence or correlating unrestricted model features with brain activity. The consistent above-chance target decoding across eleven checkpoints is informative even under the narrower interpretation of cross-wording concept information (Sections 3.1 and 4.1; Table 1).

  2. Rich, interpretable evidence across artificial and biological systems. The combination of 29 intracranial participants, multiple model families, explicit linguistic feature families, and complementary representational and temporal analyses makes this a potentially valuable resource. In particular, asking which linguistic properties explain correspondence is consequential for understanding what conventional aggregate brain scores leave unresolved (Sections 3.2-3.3; Figures 5-8).

Critical weaknesses
  1. The main semantic result does not yet isolate composition from lexical association. Holding out one definition per concept prevents memorization of that exact definition, but related definitions can still share diagnostic content words or correlated word meanings. Above-chance target decoding therefore establishes cross-wording concept information, not by itself composition-sensitive meaning. I could not find an order-insensitive lexical baseline or a control that preserves component words while changing their compositional interpretation in the supplied PDF. This matters because isolating compositional meaning is central to the novelty and to the interpretation of the model-brain comparison (Sections 2, 3.1, and 5). The result remains valuable, but the stronger claim needs an appropriate control or a narrower formulation.

  2. Important model-brain contrasts are not estimated under sufficiently matched analysis choices. For cortex, each feature family is removed from its cumulative tier; for models, it is removed from the full feature model (Section 3.3; Tables A1-A2). These estimate different conditional contributions when features are correlated, so the contrast between cortical nesting and model segregation could partly reflect the partitioning scheme. The temporal comparison also uses HGA smoothed with a 100 ms Gaussian kernel, without an equivalent smoothing step for model states being described (Appendix C). This is particularly relevant to claims about sustained neural versus narrower model temporal-generalization patterns. Neither issue proves that the reported differences are artifacts, but matched-partition and matched-smoothing sensitivity analyses are needed before treating them as evidence for distinct representational organization. These concerns affect the central comparative conclusions and determine my current recommendation.

Questions for authors
  1. Clarification/control affecting the semantic claim: How much target decoding can an order-insensitive lexical representation achieve on the same leave-one-definition-out folds, and how much lexical overlap exists across the four definitions of each concept? Is there evidence in the existing analyses that specifically requires composition rather than lexical association?

  2. Clarification/robustness affecting the cortical comparison: Does the nesting-versus-segregation result remain when the same variance-partition scheme is applied to both systems, ideally reporting both full-model removal and cumulative-tier contrasts?

  3. Clarification/robustness affecting the temporal conclusion: How were TGM similarity and its significance computed, and do the constituency/semantic-category differences persist after matching temporal smoothing and accounting for the reliability of each system's TGM? Please distinguish a demonstrated difference between systems from a nonsignificant similarity estimate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.