acceptodds
Under review as a conference paper at ICLR 2027

Pretraining Domain Shapes Auditory Neural Predictivity Differently Across Species

Abstract

Deep neural networks have successfully captured aspects of neural representation in both humans and nonhuman primates, particularly in vision. However, whether such generalization extends to species-specific domains, such as speech, remains unclear. We tested self-supervised and supervised models spanning training corpora from human speech to animal vocalizations on both animal and human neural responses prediction tasks. Across all animal neural-prediction tasks, models achieved nearly identical best-layer encoding accuracy, however, on human conditions, diverged across domains. To examine whether this convergence versus divergence pattern is already present at the level of model representations, we applied unbiased Centered Kernel Alignment (uCKA) to the activations and observed a pretraining-corpora-dependent input geometry across all datasets. To investigate why this geometry is not fully reflected in encoding accuracy, we introduced a sensitive subspace analysis to the frozen predictor and found pretraining-corpora-dependence in speech models disappeared on animal datasets but preserved on human datasets. Our findings demonstrate a species-dependent effect of pretraining domain and, using speech as a case study, explain how brain filters out domain-specific model representations absent from that species.

Primary area:
applications to neuroscience & cognitive science
Keywords:
NeuroAI, auditory cortex, self-supervised learning, cross-species generalization, pretraining domain, neural encoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RE7aOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how audio pretraining domains affect neural predictivity across zebra finches, squirrel monkeys, and humans. Models achieve similar best-layer performance on animal datasets but diverge more on human datasets. Representation similarity and input-sensitivity analyses suggest that pretraining-related structure is expressed differently in predictors fitted to human versus animal responses.

Critical strengths

Combining prediction accuracy, representation similarity, and input sensitivity offers a useful approach to understanding model–brain correspondence beyond performance rankings. Multiple datasets, subspace-rank checks, and the nonspeech fMRI analysis provide complementary support for the reported patterns.

Critical weaknesses
  1. Species is confounded with recording modality, stimulus distribution, and other dataset characteristics. Animal spiking data versus human EEG/fMRI therefore do not isolate a species-specific effect.
  2. Comparing activation uCKA with input-gradient subspace overlap does not directly demonstrate that speech-related information is “filtered out.” The absence of a speech-model cluster is insufficient evidence of information loss, particularly without uncertainty estimates for the central clustering results.
Questions for authors

What existing controls distinguish species effects from recording modality and other dataset differences? Are the speech-model clustering results robust to subject/stimulus resampling and controls using shuffled neural targets? Was best-layer selection independent of the data used for final performance evaluation?

Reviewer_iCuHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests multiple audio models pretrained on human speech, animal sounds, music, and general sounds. It uses them to predict neural responses in zebra finches, squirrel monkeys, and humans. All models predict animal data about equally well, but they differ on human data. A gradient-based subspace analysis shows speech-specific structure only in the human data.

Critical strengths
  • The question is clear and the 2×2 design (pretraining domain × species) is easy to follow.
  • The models share the same base architecture, so the effect of the training corpus is better isolated.
  • The monkey TIMIT vs human EEG comparative analysis is a nice control.
Critical weaknesses
  • No model is trained only on animal sounds.
  • the paper lacks of statistical tests, and the conclusion goes beyond the evidence.
  • All models may converge in animals because there is little left to predict beyond acoustics
Questions for authors
  • Could you score each model on what it predicts beyond the spectrogram ? Examples are variance partitioning (STRF + DNN vs STRF alone), or regressing out the STRF prediction first. If the domain effect in humans and the lack of it in animals hold (statistically) on this residual, the claim stands. Right now it is not tested.
  • Can you report noise ceilings for each dataset, and compare species on matched data where possible, to show the animal result is not a ceiling effect?
  • No model is trained only on animal sounds. Can you train or find one (even a small one) to make the domain contrast clean?
1 comment

Dear reviewer, Please respond to the authors when you get a moment. (They have just posted an update) Thanks! The AC

Reviewer_hNnqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors tested a breadth of self-supervised and supervised models that were pre-trained using data from different domains. The authors then performed forward passes on particular auditory stimuli and extracted their internal representations. These internal representations were then used two-fold: (1) the authors compared the feature geometry across the pretrained models using uCKA to see consistencies and divergences. (2) The authors trained a neural prediction task observing whether there were consistencies or divergences between the domains of data used here with respect to the neural prediction task performance. Discrepancies between these two analyses are particularly interesting. If the model representation using animal audio data is different compared with the model representation using human speech data, but the performance of the neural prediction task is similar, the differences in internal representations are not an adequate clue as to the downstream performance. It raises the question of which aspects of the internal representations actually contribute to the neural predictions.

The core contributions of the paper are (1) the findings, that different neural representations obtained by using different pretrained models do, in fact, not necessarily cause difference in the down stream neural-prediction task. (2) The analysis using uCKA to show that convergene in animal neural predictivity does not arise from convergence of the underlying model representations. (3) Their analysis of an input-sensitivity subspace of the complete fitted neural predictor to show that speech-corpus-specific structure is lost when moving from a speech-trained model to an animal neural-predictive task, but retained when performing a human neural-predictive task, indicating that the speech-corpus-specific structure might be somehow useful when performing a human neural-predictive task.

The authors validate their findings through usage of five neural-response datasets across three species, comparing models pretrained on different auditory domains. The three main components of analysis are the downstream neural-predcition performance, the uCKA analysis yielding representation similarity, and the input-sensitivity subspace analysis to observe, how the features are used in the neural-predictive tasks, when the pretraining and down-stream task domains align or differ.

Critical strengths

I see two critical strengths from my reading:

(1) The paper demonstrates that models pretrained on different auditory domains can exhibit substantially different internal representations while achieving similar downstream neural-prediction performance, particularly on animal datasets. This highlights an interesting distinction between representational similarity and neural predictivity, suggesting that predictive performance alone may not adequately characterize the relationship between artificial and biological neural representations.

(2) The combination of neural-prediction performance, uCKA, and gradient-based input-sensitivity subspace analysis offers an interesting way to investigate the relationship between pretrained representations and their downstream predictive behavior. In particular, the input-sensitivity analysis provides an additional perspective on which aspects of the auditory input influence the fitted neural predictors, beyond what can be inferred from prediction accuracy and global representation similarity alone.

Critical weaknesses

I see two critical weaknesses from my reading:

(1) The authors attribute differences in neural predictivity and input-sensitivity subspaces to species-specific effects. However, the human and animal datasets also differ substantially in recording modalities, stimulus distributions, and experimental conditions. It remains unclear whether the observed differences genuinely reflect species-specific processing or arise from dataset-dependent factors. This directly weakens one of the paper's central conclusions.

(2) The authors interpret differences in input-sensitivity subspaces as evidence that speech-specific representational structure is preserved in human neural prediction but filtered out in animal neural prediction. However, the analysis measures the sensitivity of fitted prediction pipelines rather than directly identifying biological processing mechanisms. Furthermore, differences in global representation similarity measured by uCKA do not necessarily imply differences in task-relevant information. Therefore, it remains unclear whether the proposed analysis explains the observed discrepancy or merely provides an additional characterization of it.

Questions for authors

On species-specific effects (both clarification and suggestion for future work): Given the substantial differences in recording modalities and experimental conditions between species, how can the authors establish that their findings are genuinely species-specific rather than dataset-dependent? Would the reported patterns remain consistent across independently collected neural-response datasets from the same species?

On selection of pretrained models (clarification): The authors acknowledge the absence of models pretrained exclusively on animal vocalizations. However, existing bioacoustic models such as AudioProtoPNet, BirdNET, and Perch have been trained on vocalizations from large numbers of animal species. Could the authors clarify why such models were not included, and to what extent their absence limits the conclusions regarding the influence of pretraining domain?

On choices of models for downstream analysis (both clarification and suggestion for future work): Given that the supervised and self-supervised models achieve comparable best-layer neural-prediction performance, why were the supervised models excluded from the uCKA and input-sensitivity subspace analyses? Could including them alter the observed clustering patterns or the conclusions regarding speech-specific representations?

On uCKA as a measure for representational similarity (both clarification and suggestion for future work): To what extent do the conclusions depend on the choice of uCKA for measuring representational similarity? Could the observed discrepancy between representational similarity and neural-prediction performance partly reflect the properties of this metric? Would alternative similarity measures or analyses focused on neural-predictive information support the same conclusions?

On the interpretation of input-sensitivity subspaces (clarification): How can the authors distinguish whether the observed differences in input-sensitivity subspaces reflect actual differences in how human and animal brains process auditory information, rather than differences introduced by the fitted neural predictors or the datasets themselves?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the reviewers and the Area Chair for their time and constructive feedback.

Throughout the rebuttal and discussion, we have carefully addressed all questions and concerns raised by the reviewers, providing detailed clarifications, additional experiments, and supporting analyses. These efforts have further strengthened the empirical evidence and clarified the scope of our conclusions.

We are also encouraged by the positive feedback received during the discussion. Reviewer RE7a has consistently supported our work, while Reviewer iCuH described our response as “strong and complete”. We believe this feedback reflects the thoroughness of our rebuttal and the extent to which the reviewers’ concerns have been addressed. We thank the reviewers and the Area Chair again for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.