acceptodds
Under review as a conference paper at ICLR 2027

The No Free Lunch of Self-Supervised Brain Models: Behavior Prediction, Learning Objective, and Data

Abstract

Deep neural networks provide leading quantitative models of sensory systems. By circumventing the need for massive labeled datasets, self-supervised pretrained models are increasingly viewed as a free lunch for sensory system modeling. In this study, we evaluate this assumption by applying four categories of computational models—four acoustic, three semantic, three supervised, and eight self-supervised models—to predict human auditory cortical responses and perceptual behavior. Our results demonstrate that this lunch is far from free. In auditory cortex, model-class advantages shifted across the cortical hierarchy: supervised models generally showed stronger alignment in primary and secondary regions, whereas selected self-supervised models surpassed the strongest supervised baselines by approximately 25–30% in tertiary regions. Within the self-supervised class, cortical predictivity varied substantially across learning objectives and pretraining datasets, including among closely matched model families. This cortical advantage did not extend to perceptual behavior, where the strongest supervised DNN exceeded the strongest self-supervised model by approximately 54% and 57% in the sound- and word-judgment tasks, respectively. Self-supervised models also tended to peak at earlier network depths. Together, these findings reveal distinct trade-offs between cortical and behavioral alignment and highlight the importance of learning objective and pretraining data in sensory system modeling.

Primary area:
applications to neuroscience & cognitive science
Keywords:
self-supervised learning, representation learning, NeuroAI, brain-model alignment, computational neuroscience, auditory cortex, human perception, learning objective, pretraining data distribution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ucg6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The study explores the differences in model alignment to cortical and behavioral representations, depending on model class, pretraining objective and corpus. The paper employs predictive representational similarity analysis to compare the representations of models to that of 1. fMRI activations in the auditory cortex and 2. human judgement about stimulus (text or sound) similarity. The paper concludes that there is no single recipe that affords optimal alignment across the auditory hierarchy and behavioral targets.

Score increased on Nov. 15th 2026.

Critical strengths

The paper addresses a complex question with a computationally reasonable experiment without extensive pretraining. It also contributes to the emerging field of model interpretability analyses which is valuable in and of itself.

Critical weaknesses

While the paper's focus is on an important problem, it seems rather dispersed, fraught with imprecise terminology.

  1. The writing unclear. For instance, the authors use the term "cortical predictivity" in the last paragraph of the introduction (the one stating contributions). It is unclear, what this term stands for. Is it the capacity of the model to encode cortex activations from stimuli? Is it the capacity of the model to predict a stimulus from region-wise activations? There are myriad phenomena one can designate this way. Later, it is clear that R2 is the measure of cortical predictivity. There is no explicit justification as to why. Then the same concept is designated as "cortical alignment" or "cortical prediction". I suggest the authors either use the terminology established in the field or provide explicit definitions if a novel concept / method is introduced.
  2. Figure 1. is unclear: distinct panels all blend into one, making it difficult to identify the key stages of the analysis. The methods schematic needs to be revised: what is the input and the key transformation stages (split formation, RDM construction, cross-validation, etc.)? How does the final metric (R²cv) follow from the outputs? The three data streams (model activations, fMRI and behavior) and their transformations need to be clearly distinguishable in the schematic.
  3. Panel b of Figure 2 would benefit from a larger size.
  4. The authors do not explain why they include language models in their analysis even though their focus is on auditory processing.
  5. The attempt at dissociating the effect of model class and pretraining corpus is incomplete in the absence of all possible [model x dataset] combinations (Table 1).
  6. The main question of the paper is: "Is self-supervised learning all we need to model sensory systems?", and the abstract states that SSL "is increasingly viewed as a free lunch" but I see no citations of papers adhering to this view. Without fine-tuning (especially in neuroimaging / cognitive neuroscience), one would hardly expect an SSL model to be on par with a supervised model trained from scratch. I suggest the authors reframe their core question such that it aligns better with the methods and key findings.

That said, if the authors address the main questions and weaknesses, I am willing to increase the score.

Questions for authors
  1. Could the authors provide a clear definition of cortical predictivity and provide references to studies employing a similar or adjacent definition?
  2. Which top-down mechanism do you refer to in "These observations are consistent with, but do not establish, a causal role of supervision or a specific top-down mechanism."? Could you also elaborate how these observations are consistent with
  3. Without a full [objective × corpus] design, how do you separate the effect of the objective from that of the corpus?
  4. Could you provide a detailed explanation as to how your analyses address the core question of the paper, i.e., whether SSL alone is sufficient to model sensory systems? If the answer is negative, could you explain how your findings complement the existing papers that demonstrate that fine-tuning (more often than not) improves model prediction of behavior (Muttenthaler et al., 2023; Fu et al., 2023; Binz et al., 2025)?

Muttenthaler, L., Dippel, J., Linhardt, L., Vandermeulen, R. A., & Kornblith, S. (2023). Human alignment of neural network representations. In International Conference on Learning Representations (ICLR). arXiv:2211.01201. Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., & Isola, P. (2023). DreamSim: Learning new dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems, 36 (NeurIPS 2023). Binz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Eckstein, M. K., Éltető, N., Griffiths, T. L., et al. (2025). A foundation model to predict and capture human cognition. Nature, 644(8078). https://doi.org/10.1038/s41586-025-09215-4

8 comments
AuthorsOpenReview ↗

Response to reviewer ucg6 (1/4)

Response to reviewer Ucg6

Dear Reviewer:

Thank you for your comments concerning our manuscript entitled “The No Free Lunch of Self-Supervised Brain Models: Behavior Prediction, Learning Objective, and Data” (Submission Number: 37908). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our researchers. We have studied comments carefully and have made correction which we hope to meet with approval.

Weakness1: Definition of cortical RDM prediction performance

While the paper’s focus is on an important problem, it seems rather dispersed, fraught with imprecise terminology.The writing unclear. For instance, the authors use the term "cortical predictivity" in the last paragraph of the introduction (the one stating contributions). It is unclear, what this term stands for. Is it the capacity of the model to encode cortex activations from stimuli?Is it the capacity of the model to predict a stimulus from region-wise activations? There are myriad phenomena one can designatethis way. Later, it is clear that R2R^2 is the measure of cortical predictivity. There is no explicit justification as to why. Then the sameconcept is designated as "cortical alignment" or "cortical prediction". I suggest the authors either use the terminology established in the field or provide explicit definitions if a novel concept / method is introduced.

Response:

Thank you for pointing this out. We now define cortical RDM prediction performance in the first paragraph of Section 3.3 (“Cross-validated prediction of human dissimilarities”) of the Methods. For a pair of sounds, we predict how dissimilar the evoked multivoxel fMRI response patterns are within a given auditory region of interest (ROI). These distances form the region’s representational dissimilarity matrix (RDM). The predictors are the corresponding distances from components of a computational model. We estimate their linear weights on the training data and evaluate the predictions on held-out sounds and participants. We use the term “cortical RDM prediction performance” in Section 3.3 and in the Figure 2 caption.

We also corrected the scoring procedure. The revised tables report the cross-validated coefficient of determination, Rcv2R^2_{\mathrm{cv}}, on standardized test distances. Its reference prediction assigns the mean observed test distance to every stimulus pair. A positive score therefore indicates that the model predicts the distances for those pairs with lower squared error than the mean prediction. Equation (1) in Section 3.3 gives the formula, and the paragraph immediately following it explains how to interpret positive, zero, and negative scores.

Weakness2:

Figure 1. is unclear: distinct panels all blend into one, making it difficult to identify the key stages of the analysis. The methods schematic needs to be revised: what is the input and the key transformation stages (split formation, RDM construction, crossvalidation, etc.)? How does the final metric (Rcv2R^2_{\mathrm{cv}}) follow from the outputs? The three data streams (model activations, fMRI and behavior) and their transformations need to be clearly distinguishable in the schematic.

Response:

Thank you for pointing this out. We have redrawn Figure 1. Panel a gives each input its own row. Model features yield component RDMs, sound-evoked fMRI patterns yield ROI RDMs, and the merge levels in the sound and word similarity tasks yield behavioral RDMs.

Panel b shows the evaluation splits. For fMRI, each of the 10 participant splits uses two people for training and three for testing. In each of four stimulus folds, 216 sounds are used for training and 72 for testing. Each behavioral task uses 100 splits of 10 training and 10 test participants judging the same stimuli.

Panel c follows one model through fitting and evaluation. Ordinary least squares fits the component weights to the training human distances. We apply the same weights and intercept to the test predictors and compare predicted and observed distances for matching pairs to obtain Rcv2R^2_{\mathrm{cv}} (Eq. 1). Panel d identifies the six auditory ROIs.

Weakness 3:

Panel b of Figure 2 would benefit from a larger size.

Response:

Thank you for this suggestion. We moved the anatomical map showing all six auditory ROIs from Figure 2 to Figure 1d. This gave us room to enlarge all six regional plots in Figure 2 and arrange them in three rows and two columns. The Heschl’s gyrus results, originally shown in Figure 2b, now appear in the enlarged Figure 2a.

AuthorsOpenReview ↗

Response to reviewer ucg6 (2/4)

Weakness 4:

The authors do not explain why they include language models in their analysis even though their focus is on auditory processing

Response:

Thank you for pointing this out. We included GNewsW2V, GloVe, and GUSE as semantic references for the events that generate the sounds. This follows Giordano et al. (2023), who evaluated the same three text-derived representations alongside acoustic models and auditory networks.

Our two behavioral tasks provide a direct reason for the comparison. Separate groups judged the same 80 events by listening to their recordings or reading their verb–noun descriptions. Figure 4d illustrates the distinction. Among the displayed events, “fluttering bird” grouped most closely with “breathing horse” when participants heard the recordings and with “buzzing bee” when they read the descriptions.

Across both tasks, auditory networks always received the recordings and semantic models always received the descriptions. In Figure 4e, VGGish, YamNet, PANNs, and AST each had a higher median behavioral RDM prediction score than all three semantic models on sound judgments. On word judgments, each semantic model scored above all four classifiers. The semantic references thus help identify which distinctions among events are better predicted from audio and which are better predicted from their verbal descriptions.

Weakness5:

The attempt at dissociating the effect of model class and pretraining corpus is incomplete in the absence of all possible [model×dataset] combinations (Table 1).

Response:

Thank you for pointing this out. We evaluated eight of the twelve possible combinations of three model families and four pretraining corpora. We evaluated Wav2Vec2 and HuBERT checkpoints pretrained on LibriSpeech, WenetSpeech, and AudioSet, and WavLM checkpoints pretrained on LibriSpeech and Mix94k. For the remaining four combinations, we could not identify suitable public checkpoints with documented pretraining data. Training these models from scratch was beyond our lab’s computational resources.

Section 4.2 reports two findings based on the available checkpoints. First, the ranking of model families varied with the pretraining corpus (Figure 3a). In pSTG, HuBERT had a higher median cortical RDM prediction score than Wav2Vec2 on WenetSpeech, but Wav2Vec2 scored higher on AudioSet. Second, within a model family, the highest-scoring checkpoint varied by auditory region (Figure 3b). For HuBERT, both the WenetSpeech and AudioSet checkpoints scored higher than the LibriSpeech checkpoint in primary and secondary auditory cortex, while the LibriSpeech checkpoint ranked first in all three higher-order regions.

We also expanded the benchmark from 18 to 30 conditions. This expansion provides a more comprehensive comparison of auditory models across architectures, training tasks, and stages of training.

Added conditionsModelsComparisons enabled
3 supervised modelsPANNs-CNN14, AST, Whisper-baseSound event classification and speech transcription across convolutional and Transformer architectures.
6 self-supervised models for general audioBYOL-A, BEATs, AudioMAE, EAT-Base, ATST-Frame, SSASTGeneral-audio and speech-focused self-supervised models.
3 fine-tuned modelsWav2Vec2-LS-FT, BEATs-FT, AudioMAE-FTEach network before and after supervised fine-tuning, across six auditory ROIs and both behavioral tasks.
AuthorsOpenReview ↗

Response to reviewer ucg6 (3/4)

Weakness 6:

The main question of the paper is: "Is self-supervised learning all we need to model sensory systems?", and the abstract states that SSL "is increasingly viewed as a free lunch" but I see no citations of papers adhering to this view. Without fine-tuning (especially in neuroimaging / cognitive neuroscience), one would hardly expect an SSL model to be on par with a supervised model trained from scratch. I suggest the authors reframe their core question such that it aligns better with the methods and key findings.

Response:

Thank you for your suggestion. We agree that the question should be framed around what our benchmark tests. We ask whether self-supervised auditory models consistently outperform supervised models in predicting cortical RDMs and human similarity judgments. The title’s “No Free Lunch” refers to the finding that the relative performance of these models depends on the auditory region and behavioral task.

To examine fine-tuning directly, we compared three pretrained networks with their counterparts after supervised fine-tuning. Wav2Vec2-LS was fine-tuned for speech recognition, and BEATs and AudioMAE for sound event classification. During evaluation, we froze all network weights and used the same linear readout procedure for each model. After fine-tuning, cortical RDM prediction scores fell for Wav2Vec2-LS in primary auditory cortex and for BEATs in all three higher-order auditory regions. AudioMAE, by contrast, scored higher in all six regions (Appendix C.2, Table 7).

Across the models we evaluated, the advantage that some self-supervised models showed in higher-order auditory cortex did not extend to either behavioral task. Our findings therefore support a narrower conclusion about auditory modeling: whether self-supervised learning offers a predictive advantage depends on which aspect of human hearing the model is asked to predict.

Question 1:

Could the authors provide a clear definition of cortical predictivity and provide references to studies employing a similar or adjacent definition?

Answer:

The first paragraph of Section 3.3 (“Cross-validated prediction of human dissimilarities”) of the Methods defines the prediction target for cortical RDM prediction performance as the pairwise dissimilarities between sound-evoked multivoxel fMRI patterns within an auditory ROI. We jointly fit weights on model-component distances using training participants and sounds, then evaluate predictions on held-out participants and sounds.

Section 3.3, Eq. (1), defines

Rcv2=1−∑i(yi−y^i)2∑i(yi−yˉtest)2,R^2_{\mathrm{cv}} = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y}_{\mathrm{test}})^2},

where yiy_i, y^i\hat{y}_i, and yˉtest\bar{y}_{\mathrm{test}} denote standardized observed test distances, their predictions, and their observed mean. Zero indicates the same squared error as predicting that mean. Negative scores are retained. We average scores over four stimulus folds, then take the median across ten participant splits.

The RDM framework follows Kriegeskorte et al. (2008) and Diedrichsen and Kriegeskorte (2017). The predictive regression approach follows Giordano et al. (2023).

Question 2:

Which top-down mechanism do you refer to in "These observations are consistent with, but do not establish, a causal role of supervision or a specific top-down mechanism."? Could you also elaborate how these observations are consistent with

Answer:

We agree that our original reference to a “top-down mechanism” was too speculative. By “these observations,” we meant that model rankings differed between cortical and behavioral prediction and that behavioral prediction peaked at different layers across models. We did not specify or test a neural feedback process, and these comparisons do not establish a causal role for supervision.

We removed the sentence and revised Section 5 to explain the layer-wise findings in terms of what the networks may represent at different depths. On both behavioral tasks, prediction peaked in the first half of the network for most of the self-supervised speech models we examined. Early layers may retain acoustic differences between sounds that are relevant to perceived similarity, while later layers may increasingly integrate context for the speech pretraining task. In the four supervised sound event classifiers, prediction peaked in middle or late layers. As these networks learn to recognize sound events, distinctions between event categories may become more explicit in those layers. We present this interpretation as a hypothesis about why different layers best predict the same human judgments.

Reviewer_XbjsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses predictive RSA (Giordano et al., 2023) on an existing 7T fMRI dataset (Santoro et al., 2017; N=5) and an existing sound/word similarity-judgment dataset (Giordano et al., 2010) to compare 18 models: acoustic, semantic, 3 supervised CNNs, and 8 base-size Wav2Vec2/HuBERT/WavLM checkpoints. It reports that (i) some SSL models beat the supervised CNNs in tertiary auditory cortex (STG) by about 25-30%, (ii) supervised YamNet/VGGish predict perceptual judgments better, (iii) SSL rankings depend on objective and pretraining corpus, and (iv) behavioral predictivity peaks earlier in depth for SSL models.

Critical strengths
  • Combining cortical and behavioral targets in one framework is a reasonable way to ask whether brain-alignment rankings carry over to behavior. The datasets and the predictive RSA pipeline are established and public.
  • I find interesting the observation that the ordering of acoustic/semantic/supervised/self-supervised neural predictivity flips through the cortical hierarchy
  • The authors are careful about causality. They state several times (Secs. 3.1, 4.2, 5) that corpus differences are confounded and that results are associational.
  • Reproducibility is good: exact HF checkpoints (A.4.4), a detailed CV scheme (B.2-B.3), full median tables (Tables 2-3), and code is promised.
Critical weaknesses
  • Limited novelty The method and both datasets come directly from Giordano et al. (2023). Model-class comparisons on auditory fMRI already exist (Tuckute et al., 2023 etc).
  • Lack of a clear message: the results feel more like an experiment dump than a clear narrative to me: the main interpretation of most figures seems to be that “alignment varies across models, learning objectives, pretraining datasets etc” without clear generalizable trends or reasons for the observed behaviour. The title promises a general "no free lunch" result, but central claims depend on supervised and SSL models that differ in architecture and training domain, and comparisons across two small and unrelated datasets (see below).
  • Confounded comparisons: The supervised vs. SSL contrast is confounded by training domain and architecture. The supervised models are CNNs trained on AudioSet event labels (YamNet, VGGish), which matches the environmental-sound behavioral stimuli. Six of the eight SSL models are transformers trained on speech only. The only domain-matched SSL models (HuBERT-AS, Wav2Vec2-AS) are unofficial "ALM/" checkpoints and are the weakest in Table 3. Key controls are missing: general-audio SSL models (BEATs, AudioMAE, BYOL-A, EAT, ATST); supervised transformers (AST, PANNs, Whisper); most importantly, the same backbone before and after supervised fine-tuning (e.g., wav2vec2-base vs. wav2vec2-base-960h). Without these, "supervision" cannot be separated from "AudioSet labels + CNN".
  • Presentation: Many figures are insufficiently described which makes it hard to extract clear-cut messages. For example, what are supposed to make of figure 3? What are the dots in the MDS plots and what are the RDM matrices representing? Same issue in figure 5 where the layerwise profiles look particularly noisy and the legend of panel a is unclear. Depth is not normalized across architectures (the paper says layers are not matched). No statistic is given for peak location, and exceptions are acknowledged in both classes (Kell, HuBERT-Wenet).
Questions for authors

See above

5 comments
AuthorsOpenReview ↗

Response to reviewer Xbjs (1/3)

Dear Reviewer:

Thank you for your comments concerning our manuscript entitled “The No Free Lunch of Self-Supervised Brain Models: Behavior Prediction, Learning Objective, and Data” (Submission Number: 37908). The additional models you suggested allowed us to test whether our findings held across a wider range of training domains and architectures.

Critical Weaknesses:

Weakness 1: Limited novelty

The method and both datasets come directly from Giordano et al. (2023). Model-class comparisons on auditory fMRI already exist (Tuckute et al., 2023 etc).

Response:

Thank you for raising this point. Our original study extended the comparison to self-supervised auditory models. In response to your suggestions, we further expanded the benchmark from 18 to 30 model conditions. (see below) We added six general-audio self-supervised models, three supervised models spanning CNN and Transformer architectures, and three fine-tuned models (Table 1). These additions allowed us to examine whether the cortical and behavioral ranking patterns held across a broader range of training domains and architectures and to compare the same networks before and after supervised fine-tuning.

Giordano et al. (2023) found that, among the models they evaluated, sound event recognition networks best predicted both superior temporal gyrus responses and perceived sound dissimilarity. Across a broader set of auditory models, we found that an advantage in predicting higher-order auditory cortex does not necessarily carry over to behavior similarity judgments (Figures 2 and 4e). Even for the same behavior judgment, the most predictive representations can occur at different depths in different networks (Figure 5). In Section 5, we discuss how these findings may relate to the sound features that different training tasks emphasize.

Weakness 2: Lack of a clear message

the results feel more like an experiment dump than a clear narrative to me: the main interpretation of most figures seems to be that "alignment varies across models, learning objectives, pretraining datasets etc" without clear generalizable trends or reasons for the observed behaviour. The title promises a general "no free lunch" result, but central claims depend on supervised and SSL models that differ in architecture and training domain, and comparisons across two small and unrelated datasets (see below).

Response:

Thank you for pointing this out. We have revised the Introduction to state the main question, and Sections 4.1 and 4.3 now follow the same models from cortical prediction to behavioral prediction.

Our central question is whether models that lead on one prediction target retain that advantage on another. The five models that ranked highest in all three higher-order auditory regions ranked lower in primary and secondary auditory cortex. On both sound and word similarity judgments, all four sound event classifiers scored above every one of those five models (Figures 2 and 4e). This is what we mean by “No Free Lunch” in this benchmark: stronger prediction of higher-order auditory representations does not necessarily translate into stronger prediction of behavior.

One possible explanation is that models and tasks emphasize different kinds of information about sounds. Speech training may help models capture complex acoustic distinctions that are also reflected in higher-order auditory responses (Section 4.1, final paragraph). The two behavioral tasks also give participants different information about the same events. Hearing a recording and reading its description may place different weights on acoustic properties and knowledge of the sound source (Discussion, paragraph 2).

The same cortical–behavioral ranking pattern held when we expanded the benchmark from 18 to 30 model conditions, covering a wider range of architectures and training domains (Table 1). Its scope is a comparison of model rankings across targets. The fMRI and behavioral studies used different participants, stimuli, and tasks, so they do not support a direct within-participant test of brain–behavior correspondence (Discussion, Limitations).

AuthorsOpenReview ↗

Response to reviewer Xbjs (2/3)

Weakness 3: Confounded comparisons

The supervised vs. SSL contrast is confounded by training domain and architecture. The supervised models are CNNs trained on AudioSet event labels (YamNet, VGGish), which matches the environmental sound behavioral stimuli. Six of the eight SSL models are transformers trained on speech only. The only domain-matched SSL models (HuBERT-AS, Wav2Vec2-AS) are unofficial "ALM/" checkpoints and are the weakest in Table 3. Key controls are missing: general-audio SSL models (BEATs, AudioMAE, BYOL-A, EAT, ATST); supervised transformers (AST, PANNs, Whisper); most importantly, the same backbone before and after supervised fine-tuning (e.g. wav2vec2-base vs. wav2vec2-base-960h). Without these, "supervision" cannot be separated from "AudioSet labels + CNN".

Response:

Thank you for this suggestion. We added the models and fine-tuning comparisons you recommended, expanding the benchmark from 18 to 30 model conditions (Table 1).

  1. General-audio models.

All four supervised sound event classifiers scored higher than every one of the six general-audio self-supervised models on behavior similarity judgments. All ten models were trained on audio spanning multiple sound categories, including environmental sounds. The comparison therefore extends the behavioral finding to self-supervised models trained on general audio (Figure 4e, Table 2).

ModelsArchitectureTraining data
VGGishSup/CNNYouTube audio (an early version of YouTube-8M)
YamNet, PANNs-CNN14Sup/CNNAudioSet
ASTSup/TransformerImageNet, followed by AudioSet
BYOL-ASSL/CNNAudioSet
BEATs, AudioMAE, EAT-BaseSSL/TransformerAudioSet-2M
SSASTSSL/TransformerAudioSet and LibriSpeech
ATST-FrameSSL/TransformerAudioSet
  1. Transformer models.

The supervised sound event classifier AST scored higher than pretrained AudioMAE on both behavioral tasks and in all six cortical regions. Both are spectrogram Transformers trained using AudioSet. AST learned to classify sound events using labels, while AudioMAE learned to reconstruct masked spectrogram patches. This comparison shows that the observed behavioral advantage of sound event classifiers extends to a Transformer architecture (Tables 2–3).

ModelTraining taskSound judgmentsWord judgments
ASTSupervised sound event classification0.25710.1305
AudioMAESelf-supervised masked spectrogram reconstruction0.08710.0838
Whisper-baseWeakly supervised speech transcription and related tasks0.10010.0444

Whisper-base illustrates how the relative ranking of two supervised Transformers can change with the prediction target. It ranked first among all 30 conditions in each higher-order auditory region, yet scored below AST on both behavioral tasks. The AST–AudioMAE comparison addresses the architecture concern, although their different training histories, including AST’s ImageNet initialization, limit what we can attribute to event labels alone.

  1. The same backbone before and after supervised fine-tuning.

Both BEATs and AudioMAE scored higher on the two behavioral tasks after sound event fine-tuning, whereas Wav2Vec2-LS scored lower on both after speech recognition fine-tuning (Appendix C.2, Table 6).

BackboneSupervised fine-tuning taskSound judgments, before → afterWord judgments, before → after
Wav2Vec2-LSSpeech recognition, LibriSpeech 960 h0.1448 → 0.13110.0709 → 0.0587
BEATsSound event classification, AudioSet0.1510 → 0.22490.0814 → 0.1046
AudioMAESound event classification, AudioSet0.0871 → 0.27610.0838 → 0.1452

The two models fine-tuned on sound event classification showed different cortical changes despite improving on both behavioral tasks. BEATs scored lower in all three higher-order auditory regions after fine-tuning, while AudioMAE scored higher in all six cortical regions (Appendix C.2, Table 7).

The expanded model comparisons provide broader support for our main conclusion: self-supervised auditory models do not show a consistent predictive advantage across auditory regions and behavioral tasks.

AuthorsOpenReview ↗

Response to reviewer Xbjs (3/3)

Weakness 4: Presentation

Many figures are insufficiently described which makes it hard to extract clear-cut messages. For example, what are supposed to make of figure 3? What are the dots in the MDS plots and what are the RDM matrices representing? Same issue in figure 5 where the layerwise profiles look particularly noisy and the legend of panel a is unclear. Depth is not normalized across architectures (the paper says layers are not matched). No statistic is given for peak location, and exceptions are acknowledged in both classes (Kell, HuBERT-Wenet).

Response:

Thank you for your comments. We have revised the figures and their captions to make the comparisons easier to follow and the main findings easier to identify.

We have clarified Figure 3 and its interpretation. Panels a–b show how model rankings vary across auditory regions and pretraining corpora. In each of panels c–d, the MDS plots compare human cortical responses with the corresponding model representations. Each point represents a sound. Its color indicates the category, and nearby points indicate greater representational similarity in the projection. The RDMs show the corresponding pairwise dissimilarities between sounds. These visualizations illustrate qualitative similarities in how human cortical responses and model representations organize sounds by category.

We have redesigned Figure 5 as heatmaps, with separate panels for sound and word judgments. Each row represents a network, with its extracted components arranged in order across a normalized horizontal span. Blue intensity indicates median Rcv2R^2_{\mathrm{cv}}. Colored bars identify model groups, and white circles mark the observed peaks. This normalization indicates relative component position within each network, without assuming equivalent computational depth across architectures.

Table 4 now reports peak components, positions, and scores. We present the layerwise pattern as a descriptive tendency, with exceptions explicitly noted, and clarify that differences in peak location were not statistically tested.

Reviewer_LKmHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigated different audio/speech models (especially supervised vs. self-supervised models) on their similarities with humans' cortical activities and external behavior. It was found that supervised models align well with primary and secondary regions, while self-supervised models align better with the tertiary region. As the tertiary region mainly deals with abstract concepts or preferences, authors further examined whether self-supervised models are better predictors for human behaviors (by measuring RSA). Experimental results show that supervised methods actually outperform them.

Critical strengths
  1. The paper is easy to follow. Claims and experiments are clear. Experimental results are abundant and rich comparisons on two axes (types of models & types of tasks) produce some interesting findings, such as supervised models align well with primary and secondary regions while self-supervised models align well with the tertiary region.
  2. In my knowledge, joint comparisons on NNs with neural activities (fMRI) and behavior patterns are underexplored. This may be a valuable direction for establishing deeper understanding of self-supervised acoustic/speech models.
Critical weaknesses
  1. As mentioned by the authors, the fMRI and behavioral data are from different previous studies with different human subjects. Plus the scale is small. These factors undermine the robustness of experimental results.
  2. The findings are observational results that need further investigations. For example, why a high similarity with fMRI data leads to worse predictive power to human behavior? Is this a model feature produced by specific training strategies or biological feature where the mapping from neural activities to human behavior itself requires deeper investigation? Without deeper analysis, the current findings alone are not actionable enough for guiding future model development or relevant studies.
Questions for authors
  1. What is the representational similarity between fMRI data and human behavior? How does it affect your explanations on current results?
  2. Have you tried alternative similarity estimation methods such as CKA? How do they affect the results?
5 comments
AuthorsOpenReview ↗

Response to reviewer LKmH (1/3)

Response to reviewer LKmH

Dear Reviewer:

Thank you for asking what we can learn from comparing fMRI responses and behavioral judgments collected from different participants using different stimuli. We now clarify that our conclusions concern how the same models rank across cortical regions and behavioral tasks. Following your suggestion, we also added a centered kernel alignment (CKA) analysis and report which model rankings held across measures and which changed.

Critical Weaknesses:

1. As mentioned by the authors, the fMRI and behavioral data are from different previous studies with different human subjects. Plus the scale is small. These factors undermine the robustness of experimental results.

Response:

Thank you for raising this concern. We acknowledge both limitations in the Limitations section. Among the public resources we examined, we did not identify a dataset in which a large cohort of participants underwent fMRI while listening to a broad range of natural sounds and also judged the similarity of both those sounds and their corresponding verbal descriptions. Although our fMRI sample included only five participants, each participant heard 288 natural sounds from six categories, with three presentations per sound. We evaluated cortical predictions on held-out participants and sounds, and behavioral predictions on held-out participants in each task.

These datasets allow us to ask whether the models that best predict responses in higher-order auditory cortex also best predict behavior similarity. Giordano et al. (2023) used both datasets for auditory model comparison. The same fMRI recordings were also used to compare models of pitch encoding (De Angelis et al., 2018), and the same sound and word similarity judgments were used to evaluate auditory networks (Esposito et al., 2024). On both behavioral tasks, all four supervised sound-event classifiers outscored each of the five models that ranked highest in all three higher-order auditory regions (Figures 2 and 4e). This finding supports our conclusion that a model’s advantage in predicting higher-order auditory cortex does not necessarily extend to perceptual judgments.

References

Vittoria De Angelis, Federico De Martino, Michelle Moerel, Roberta Santoro, Lars Hausfeld, and Elia Formisano. Cortical processing of pitch: Model-based encoding and decoding of auditory fMRI responses to real-life sounds. NeuroImage, 180:291–300, 2018. DOI: 10.1016/j.neuroimage.2017.11.020

Michele Esposito, Giancarlo Valente, Yenisel Plasencia-Calaña, Michel Dumontier, Bruno L. Giordano, and Elia Formisano. Bridging auditory perception and natural language processing with semantically informed deep neural networks. Scientific Reports, 14:20994, 2024. DOI: 10.1038/s41598-024-71693-9

Bruno L. Giordano, Michele Esposito, Giancarlo Valente, and Elia Formisano. Intermediate acoustic-to-semantic representations link behavioral and neural responses to natural sounds. Nature Neuroscience, 26(4):664–672, 2023. DOI: 10.1038/s41593-023-01285-9

AuthorsOpenReview ↗

Response to reviewer LKmH (2/3)

Critical Weaknesses:

2. The findings are observational results that need further investigations. For example, why a high similarity with fMRI data leads to worse predictive power to human behavior? Is this a model feature produced by specific training strategies or biological feature where the mapping from neural activities to human behavior itself requires deeper investigation? Without deeper analysis, the current findings alone are not actionable enough for guiding future model development or relevant studies.

Response:

Thank you for raising this question. Our finding is that an advantage in predicting higher-order auditory cortex does not necessarily carry over to behavior judgments. Whisper-base ranked first in all three higher-order auditory regions, whereas PANNs ranked first among the auditory networks on both behavioral tasks. Sections 4.1 and 4.3 discuss a possible explanation. Speech training may help models capture complex acoustic distinctions reflected in higher-order cortical responses. Sound-event classification may emphasize distinctions among everyday events that matter when people judge their similarity, whether they hear the sounds or read their descriptions.

The fine-tuning results show that cortical and behavioral prediction can improve together. After sound-event fine-tuning, BEATs and AudioMAE both scored higher on both behavioral tasks. Yet BEATs scored lower in all three higher-order auditory regions, while AudioMAE scored higher in all six cortical regions (Appendix C.2, Tables 6–7). These comparisons show how prediction changed within each network. They leave open how much of the difference between networks is attributable to their training objectives. Testing the biological account would require measuring cortical responses and similarity judgments for the same sounds in the same individuals, as discussed in the Limitations section.

No single model performed best across all the cortical regions and behavioral tasks we evaluated. For model development, this means evaluating both the network and the layer on the intended prediction target. For example, behavioral prediction peaked in the first half of the network for most of the speech-family self-supervised models, but in middle or late layers for all four supervised sound-event classifiers (Figure 5, Table 4). When the goal is to predict perceived similarity, comparing candidate layers on held-out human judgments gives a more direct basis for model selection than relying on higher-order cortical performance alone.

AuthorsOpenReview ↗

Response to reviewer LKmH (3/3)

Questions For Authors:

1. What is the representational similarity between fMRI data and human behavior? How does it affect your explanations on current results?

Answer:

Because the measured fMRI and behavioral representational dissimilarity matrices (RDMs) describe different stimulus pairs, we cannot directly estimate the similarity between fMRI data and human behavior. The fMRI study used 288 sounds, whereas the behavioral study used a different set of 80 events. The studies also recruited different participants, precluding a within-person comparison.

We therefore used the auditory models to ask a related question: do the sound differences that help a model predict cortical responses to one set of stimuli also help it predict human judgments of another?

In Appendix B.2.1, we evaluated 23 auditory network conditions. For each condition, we fitted component weights to the fMRI RDM of each auditory region and applied those weights to the 80 behavioral stimuli. When we combined the six regional predictors, all 23 conditions had positive median transfer scores for both sound and word judgments (Figure 6). This means that, on a typical participant split, the model-derived distances predicted the dissimilarity of individual event pairs more accurately than assigning every pair the mean behavioral dissimilarity.

Some of the sound distinctions captured by cortical prediction therefore transfer to behavior, even though the same network need not perform best on both targets. For example, WavLM-LS predicted responses in all three higher-order auditory regions more accurately than PANNs-CNN14 (Figure 2). Yet when the six regional predictors were combined, PANNs-CNN14 had the highest observed median transfer score of all 23 conditions on both behavioral tasks (Figure 6). This comparison shows how well cortical distinctions captured by a model predict human judgments across stimulus sets. The similarity between the two measured human RDMs remains unknown.

2. Have you tried alternative similarity estimation methods such as CKA? How do they affect the results?

Answer:

Thank you for this suggestion. We added a centered kernel alignment (CKA) analysis in Appendix C.1 (Figure 12, Table 5). CKA gives each model component equal weight when comparing model and human representational structure, whereas the Rcv2R^2_{\mathrm{cv}} analysis fits component weights on training data.

CKA supports our original conclusion that a model’s advantage in higher-order auditory cortex does not necessarily extend to perceptual judgments. Under both methods, Whisper-base ranked first in all three higher-order auditory regions, yet YamNet and AST both ranked above it on both sound and word similarity judgments.

Under both methods, all four supervised sound-event classifiers also ranked above all 14 pretrained self-supervised auditory models in both behavioral comparisons. The behavioral CKA panels cover sound judgments for living sources and word judgments for nonliving sources. The Rcv2R^2_{\mathrm{cv}} analysis includes both source categories in each task.

One difference concerned the ordering of models below Whisper-base in higher-order cortex. Four WavLM and HuBERT speech models ranked second through fifth in all three higher-order regions under Rcv2R^2_{\mathrm{cv}}. Under CKA, all three sound-event CNNs (VGGish, YamNet, and PANNs) ranked above those four models. This change qualifies the cortical ranking of these four speech models while leaving the main conclusion intact: the model that led in higher-order cortex under both methods was outperformed on both behavioral tasks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal and Three Major Contributions

We thank all Reviewers and the Area Chair for their time and constructive feedback. The reviewers’ comments have substantially strengthened our paper and helped us clarify our contributions. We are encouraged that two reviewers have increased their ratings following our rebuttal and expressed positive assessments of the work. As the rebuttal period draws to a close, we would like to reiterate the three main contributions of our study:

  1. A model’s advantage in predicting brain responses can change across the sensory hierarchy.
  2. Auditory models’ ability to predict cortical responses varied with the learning objective and pretraining data.
  3. The advantage in predicting higher-order auditory cortex did not extend to behavioral prediction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.