- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The study explores the differences in model alignment to cortical and behavioral representations, depending on model class, pretraining objective and corpus. The paper employs predictive representational similarity analysis to compare the representations of models to that of 1. fMRI activations in the auditory cortex and 2. human judgement about stimulus (text or sound) similarity. The paper concludes that there is no single recipe that affords optimal alignment across the auditory hierarchy and behavioral targets.
Score increased on Nov. 15th 2026.
Critical strengths
The paper addresses a complex question with a computationally reasonable experiment without extensive pretraining. It also contributes to the emerging field of model interpretability analyses which is valuable in and of itself.
Critical weaknesses
While the paper's focus is on an important problem, it seems rather dispersed, fraught with imprecise terminology.
- The writing unclear. For instance, the authors use the term "cortical predictivity" in the last paragraph of the introduction (the one stating contributions). It is unclear, what this term stands for. Is it the capacity of the model to encode cortex activations from stimuli? Is it the capacity of the model to predict a stimulus from region-wise activations? There are myriad phenomena one can designate this way. Later, it is clear that R2 is the measure of cortical predictivity. There is no explicit justification as to why. Then the same concept is designated as "cortical alignment" or "cortical prediction". I suggest the authors either use the terminology established in the field or provide explicit definitions if a novel concept / method is introduced.
- Figure 1. is unclear: distinct panels all blend into one, making it difficult to identify the key stages of the analysis. The methods schematic needs to be revised: what is the input and the key transformation stages (split formation, RDM construction, cross-validation, etc.)? How does the final metric (R²cv) follow from the outputs? The three data streams (model activations, fMRI and behavior) and their transformations need to be clearly distinguishable in the schematic.
- Panel b of Figure 2 would benefit from a larger size.
- The authors do not explain why they include language models in their analysis even though their focus is on auditory processing.
- The attempt at dissociating the effect of model class and pretraining corpus is incomplete in the absence of all possible [model x dataset] combinations (Table 1).
- The main question of the paper is: "Is self-supervised learning all we need to model sensory systems?", and the abstract states that SSL "is increasingly viewed as a free lunch" but I see no citations of papers adhering to this view. Without fine-tuning (especially in neuroimaging / cognitive neuroscience), one would hardly expect an SSL model to be on par with a supervised model trained from scratch. I suggest the authors reframe their core question such that it aligns better with the methods and key findings.
That said, if the authors address the main questions and weaknesses, I am willing to increase the score.
Questions for authors
- Could the authors provide a clear definition of cortical predictivity and provide references to studies employing a similar or adjacent definition?
- Which top-down mechanism do you refer to in "These observations are consistent with, but do not establish, a causal role of supervision or a specific top-down mechanism."? Could you also elaborate how these observations are consistent with
- Without a full [objective × corpus] design, how do you separate the effect of the objective from that of the corpus?
- Could you provide a detailed explanation as to how your analyses address the core question of the paper, i.e., whether SSL alone is sufficient to model sensory systems? If the answer is negative, could you explain how your findings complement the existing papers that demonstrate that fine-tuning (more often than not) improves model prediction of behavior (Muttenthaler et al., 2023; Fu et al., 2023; Binz et al., 2025)?
Muttenthaler, L., Dippel, J., Linhardt, L., Vandermeulen, R. A., & Kornblith, S. (2023). Human alignment of neural network representations. In International Conference on Learning Representations (ICLR). arXiv:2211.01201. Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., & Isola, P. (2023). DreamSim: Learning new dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems, 36 (NeurIPS 2023). Binz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Eckstein, M. K., Éltető, N., Griffiths, T. L., et al. (2025). A foundation model to predict and capture human cognition. Nature, 644(8078). https://doi.org/10.1038/s41586-025-09215-4
8 comments
Response to reviewer ucg6 (2/4)
Weakness 4:
The authors do not explain why they include language models in their analysis even though their focus is on auditory processing
Response:
Thank you for pointing this out. We included GNewsW2V, GloVe, and GUSE as semantic references for the events that generate the sounds. This follows Giordano et al. (2023), who evaluated the same three text-derived representations alongside acoustic models and auditory networks.
Our two behavioral tasks provide a direct reason for the comparison. Separate groups judged the same 80 events by listening to their recordings or reading their verb–noun descriptions. Figure 4d illustrates the distinction. Among the displayed events, “fluttering bird” grouped most closely with “breathing horse” when participants heard the recordings and with “buzzing bee” when they read the descriptions.
Across both tasks, auditory networks always received the recordings and semantic models always received the descriptions. In Figure 4e, VGGish, YamNet, PANNs, and AST each had a higher median behavioral RDM prediction score than all three semantic models on sound judgments. On word judgments, each semantic model scored above all four classifiers. The semantic references thus help identify which distinctions among events are better predicted from audio and which are better predicted from their verbal descriptions.
Weakness5:
The attempt at dissociating the effect of model class and pretraining corpus is incomplete in the absence of all possible [model×dataset] combinations (Table 1).
Response:
Thank you for pointing this out. We evaluated eight of the twelve possible combinations of three model families and four pretraining corpora. We evaluated Wav2Vec2 and HuBERT checkpoints pretrained on LibriSpeech, WenetSpeech, and AudioSet, and WavLM checkpoints pretrained on LibriSpeech and Mix94k. For the remaining four combinations, we could not identify suitable public checkpoints with documented pretraining data. Training these models from scratch was beyond our lab’s computational resources.
Section 4.2 reports two findings based on the available checkpoints. First, the ranking of model families varied with the pretraining corpus (Figure 3a). In pSTG, HuBERT had a higher median cortical RDM prediction score than Wav2Vec2 on WenetSpeech, but Wav2Vec2 scored higher on AudioSet. Second, within a model family, the highest-scoring checkpoint varied by auditory region (Figure 3b). For HuBERT, both the WenetSpeech and AudioSet checkpoints scored higher than the LibriSpeech checkpoint in primary and secondary auditory cortex, while the LibriSpeech checkpoint ranked first in all three higher-order regions.
We also expanded the benchmark from 18 to 30 conditions. This expansion provides a more comprehensive comparison of auditory models across architectures, training tasks, and stages of training.
| Added conditions | Models | Comparisons enabled |
|---|---|---|
| 3 supervised models | PANNs-CNN14, AST, Whisper-base | Sound event classification and speech transcription across convolutional and Transformer architectures. |
| 6 self-supervised models for general audio | BYOL-A, BEATs, AudioMAE, EAT-Base, ATST-Frame, SSAST | General-audio and speech-focused self-supervised models. |
| 3 fine-tuned models | Wav2Vec2-LS-FT, BEATs-FT, AudioMAE-FT | Each network before and after supervised fine-tuning, across six auditory ROIs and both behavioral tasks. |
Response to reviewer ucg6 (3/4)
Weakness 6:
The main question of the paper is: "Is self-supervised learning all we need to model sensory systems?", and the abstract states that SSL "is increasingly viewed as a free lunch" but I see no citations of papers adhering to this view. Without fine-tuning (especially in neuroimaging / cognitive neuroscience), one would hardly expect an SSL model to be on par with a supervised model trained from scratch. I suggest the authors reframe their core question such that it aligns better with the methods and key findings.
Response:
Thank you for your suggestion. We agree that the question should be framed around what our benchmark tests. We ask whether self-supervised auditory models consistently outperform supervised models in predicting cortical RDMs and human similarity judgments. The title’s “No Free Lunch” refers to the finding that the relative performance of these models depends on the auditory region and behavioral task.
To examine fine-tuning directly, we compared three pretrained networks with their counterparts after supervised fine-tuning. Wav2Vec2-LS was fine-tuned for speech recognition, and BEATs and AudioMAE for sound event classification. During evaluation, we froze all network weights and used the same linear readout procedure for each model. After fine-tuning, cortical RDM prediction scores fell for Wav2Vec2-LS in primary auditory cortex and for BEATs in all three higher-order auditory regions. AudioMAE, by contrast, scored higher in all six regions (Appendix C.2, Table 7).
Across the models we evaluated, the advantage that some self-supervised models showed in higher-order auditory cortex did not extend to either behavioral task. Our findings therefore support a narrower conclusion about auditory modeling: whether self-supervised learning offers a predictive advantage depends on which aspect of human hearing the model is asked to predict.
Question 1:
Could the authors provide a clear definition of cortical predictivity and provide references to studies employing a similar or adjacent definition?
Answer:
The first paragraph of Section 3.3 (“Cross-validated prediction of human dissimilarities”) of the Methods defines the prediction target for cortical RDM prediction performance as the pairwise dissimilarities between sound-evoked multivoxel fMRI patterns within an auditory ROI. We jointly fit weights on model-component distances using training participants and sounds, then evaluate predictions on held-out participants and sounds.
Section 3.3, Eq. (1), defines
where , , and denote standardized observed test distances, their predictions, and their observed mean. Zero indicates the same squared error as predicting that mean. Negative scores are retained. We average scores over four stimulus folds, then take the median across ten participant splits.
The RDM framework follows Kriegeskorte et al. (2008) and Diedrichsen and Kriegeskorte (2017). The predictive regression approach follows Giordano et al. (2023).
Question 2:
Which top-down mechanism do you refer to in "These observations are consistent with, but do not establish, a causal role of supervision or a specific top-down mechanism."? Could you also elaborate how these observations are consistent with
Answer:
We agree that our original reference to a “top-down mechanism” was too speculative. By “these observations,” we meant that model rankings differed between cortical and behavioral prediction and that behavioral prediction peaked at different layers across models. We did not specify or test a neural feedback process, and these comparisons do not establish a causal role for supervision.
We removed the sentence and revised Section 5 to explain the layer-wise findings in terms of what the networks may represent at different depths. On both behavioral tasks, prediction peaked in the first half of the network for most of the self-supervised speech models we examined. Early layers may retain acoustic differences between sounds that are relevant to perceived similarity, while later layers may increasingly integrate context for the speech pretraining task. In the four supervised sound event classifiers, prediction peaked in middle or late layers. As these networks learn to recognize sound events, distinctions between event categories may become more explicit in those layers. We present this interpretation as a hypothesis about why different layers best predict the same human judgments.
Response to reviewer ucg6 (1/4)
Response to reviewer Ucg6
Dear Reviewer:
Thank you for your comments concerning our manuscript entitled “The No Free Lunch of Self-Supervised Brain Models: Behavior Prediction, Learning Objective, and Data” (Submission Number: 37908). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our researchers. We have studied comments carefully and have made correction which we hope to meet with approval.
Weakness1: Definition of cortical RDM prediction performance
While the paper’s focus is on an important problem, it seems rather dispersed, fraught with imprecise terminology.The writing unclear. For instance, the authors use the term "cortical predictivity" in the last paragraph of the introduction (the one stating contributions). It is unclear, what this term stands for. Is it the capacity of the model to encode cortex activations from stimuli?Is it the capacity of the model to predict a stimulus from region-wise activations? There are myriad phenomena one can designatethis way. Later, it is clear that R2 is the measure of cortical predictivity. There is no explicit justification as to why. Then the sameconcept is designated as "cortical alignment" or "cortical prediction". I suggest the authors either use the terminology established in the field or provide explicit definitions if a novel concept / method is introduced.
Response:
Thank you for pointing this out. We now define cortical RDM prediction performance in the first paragraph of Section 3.3 (“Cross-validated prediction of human dissimilarities”) of the Methods. For a pair of sounds, we predict how dissimilar the evoked multivoxel fMRI response patterns are within a given auditory region of interest (ROI). These distances form the region’s representational dissimilarity matrix (RDM). The predictors are the corresponding distances from components of a computational model. We estimate their linear weights on the training data and evaluate the predictions on held-out sounds and participants. We use the term “cortical RDM prediction performance” in Section 3.3 and in the Figure 2 caption.
We also corrected the scoring procedure. The revised tables report the cross-validated coefficient of determination, Rcv2, on standardized test distances. Its reference prediction assigns the mean observed test distance to every stimulus pair. A positive score therefore indicates that the model predicts the distances for those pairs with lower squared error than the mean prediction. Equation (1) in Section 3.3 gives the formula, and the paragraph immediately following it explains how to interpret positive, zero, and negative scores.
Weakness2:
Figure 1. is unclear: distinct panels all blend into one, making it difficult to identify the key stages of the analysis. The methods schematic needs to be revised: what is the input and the key transformation stages (split formation, RDM construction, crossvalidation, etc.)? How does the final metric (Rcv2) follow from the outputs? The three data streams (model activations, fMRI and behavior) and their transformations need to be clearly distinguishable in the schematic.
Response:
Thank you for pointing this out. We have redrawn Figure 1. Panel a gives each input its own row. Model features yield component RDMs, sound-evoked fMRI patterns yield ROI RDMs, and the merge levels in the sound and word similarity tasks yield behavioral RDMs.
Panel b shows the evaluation splits. For fMRI, each of the 10 participant splits uses two people for training and three for testing. In each of four stimulus folds, 216 sounds are used for training and 72 for testing. Each behavioral task uses 100 splits of 10 training and 10 test participants judging the same stimuli.
Panel c follows one model through fitting and evaluation. Ordinary least squares fits the component weights to the training human distances. We apply the same weights and intercept to the test predictors and compare predicted and observed distances for matching pairs to obtain Rcv2 (Eq. 1). Panel d identifies the six auditory ROIs.
Weakness 3:
Panel b of Figure 2 would benefit from a larger size.
Response:
Thank you for this suggestion. We moved the anatomical map showing all six auditory ROIs from Figure 2 to Figure 1d. This gave us room to enlarge all six regional plots in Figure 2 and arrange them in three rows and two columns. The Heschl’s gyrus results, originally shown in Figure 2b, now appear in the enlarged Figure 2a.