acceptodds
Under review as a conference paper at ICLR 2027

Beyond Static Brain Alignment: Cross-Condition Neural Transportability In Speech Models

Abstract

Brain-model alignment is typically evaluated within the stimulus regime used to fit the neural mapping, leaving open whether the fitted relationship remains predictive when sensory conditions change. We operationalize cross-condition neural transportability using leave-one-condition-out (LOCO) evaluation, fitting a model-to-brain mapping on three speech conditions and testing it on an unseen fourth. Using fMRI from 25 participants hearing 96 sentences across Clean, Noisy, DNN-enhanced, and classical speech-enhanced conditions, we find that conventional within-condition alignment is broad whereas cross-condition evaluation is more selective. Under strict fold-local layer selection, encoder representations show greater transportability than matched decoder representations in Whisper-Tiny, Whisper-Medium, and SpeechT5, with higher transportability concentrated in encoder and intermediate representations. Architecture-matched controls further show that trained Whisper-Tiny exceeds its random-weight counterpart, providing direct evidence that learned structure can improve neural transportability beyond architectural inductive bias in at least one controlled model family. A complementary descriptive analysis shows representation-class AUROC increasing from under static evaluation to under LOCO, while static alignment and cross-condition transportability remain only moderately associated. Sentence-level LOCO fit does not improve held-out comprehension prediction beyond acoustic condition and Automatic Speech Recognition (ASR) character error rate. Because sentence identity is fixed to condition, LOCO measures joint transfer across acoustic regime and item set rather than the acoustic manipulation alone. Together, these findings show that strong within-condition neural correspondence does not guarantee a transportable model-to-brain mapping, supporting cross-condition neural transportability as a complementary criterion for evaluating brain-aligned speech representations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-model alignment, cross-condition generalization, out-of-distribution evaluation, speech representations, neural encoding models, speech models, fMRI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_hrB8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes leave-one-condition-out evaluation to test whether model-to-brain mappings for speech survive changes in acoustic conditions. It reports that encoders transfer better than decoders and that cross-condition testing separates models better than within-condition testing.

Critical strengths

The question is valuable: brain alignment is almost always tested within one stimulus distribution, and asking whether a fitted mapping generalizes across conditions is a needed evaluation.

Critical weaknesses
  • There's a possible confound - LOCO trains on four times more sentences than the static test, so the main conclusion (LOCO is more discriminative) cannot be separated from sample size. A static baseline with matched training size is required.

  • Similarly, if I'm not mistaken, each condition uses different sentences, with only 24 per test set, so “transfer across acoustic conditions” is also transfer across item sets. An analysis of this effect is needed.

-The raw predictivity (before noise-ceiling normalization), and the computed brain-scores depend on small, unstable noise ceilings. Raw correlations should be reported alongside the normalized result, and be discussed clearly when very low.

Questions for authors

My main questions are

  • Does the static-versus-LOCO difference in discrimination persist when static encoding is trained on the same number of sentences?

  • Does the encoder advantage hold after controlling for decoder recognition errors or with matched extraction? That is, it seems that a control experiment is needed: check whether the encoder advantage persists when recognition is good. For example, test it only on sentences the model transcribed correctly, include character error rate as a covariate, or compare models with very different error rates. Whisper-Large-v3 makes far fewer errors, and it is the one model where the encoder advantage is not significant, which hints that recognition errors may be driving the effect...

2 comments
AuthorsOpenReview ↗

Responses to Size-matched Control and Encoder Advantage & Decoder Recognition Errors

We sincerely thank the reviewer for the feedback and recognition of our study's contribution, particularly the importance of moving beyond conventional brain–model alignment to evaluate whether fitted neural mappings generalize across unseen acoustic conditions. We appreciate the reviewer's critical comments and clarify each concern below.

1. Size-matched Control

To address the sample-size mismatch, we conducted an additional size-matched evaluation using 18 training and 6 test sentences for both protocols, preserving identical feature-extraction procedures.

Our central findings persist after matching sample sizes. Under Static evaluation, no representation-class difference survives FDR correction across the four conditions. In contrast, size-matched LOCO reveals significant encoder advantages for Clean (Δ = 0.289, q = 0.004), Classic-SE (Δ = 0.314, q = 0.004), and Noisy speech (Δ = 0.284, q = 0.004). Representation-class discrimination also remains higher under LOCO (AUROC = 0.853) than Static (AUROC = 0.545).

Within-model comparisons provide convergent evidence. Under size-matched LOCO, encoders significantly outperform their paired decoders in Whisper-Tiny (Δ = 0.150, q = 0.048), Whisper-Medium (Δ = 0.278, q = 0.002), and SpeechT5 (Δ = 0.329, q < 0.001). Whisper-Large-v3 shows a smaller, non-significant difference in the same direction (Δ = 0.069, q = 0.391).

These results show that LOCO's representation-stage selectivity persists under equal training and test sample sizes, ruling out unequal sample counts as a sufficient explanation for the observed pattern. While Static and LOCO intentionally differ in their training-condition distributions, the size-matched findings strengthen our central conclusion: cross-condition neural transportability reveals representational differences that conventional within-condition alignment does not reliably distinguish.

2. Encoder Advantage & Decoder Recognition Errors

We agree that recognition errors may contribute to encoder–decoder differences. However, LOCO is designed to test robustness across speech conditions, including the effects of recognition errors. Encoder representations are extracted directly from audio, while decoder representations depend on generated tokens and may be more affected by recognition failures. Evaluating only correctly transcribed sentences would favor easier stimuli and change the test distribution.

The results suggest that recognition errors alone do not explain the encoder advantage. Whisper-Large-v3 has the lowest mean CER (0.32) and the smallest advantage, consistent with the reviewer's hypothesis. However, Whisper-Tiny has a higher CER than Whisper-Medium (0.61 vs. 0.39) but a smaller encoder advantage (0.150 vs. 0.278), contrary to a simple error-driven explanation (Table G1). Nevertheless, recognition accuracy, model size, and architecture remain difficult to disentangle.

We therefore interpret our findings in terms of predictive robustness: encoder representations show significantly greater neural transportability in three of four architecture-matched comparisons under the evaluated decoding procedures. This supports an encoder advantage in cross-condition generalization without assuming that it is independent of recognition errors.

AuthorsOpenReview ↗

Invitation to Review Our Response and Additional Experimental Results

We sincerely thank the reviewer for the time and effort devoted to evaluating our manuscript and providing detailed feedback.

In response to your questions and concerns, we have conducted additional experiments and analyses to directly address the key methodological issues and provide stronger support for our main findings.

We kindly invite you to revisit our point-by-point responses and consider the new experimental results presented in our rebuttal. We believe these additional findings strengthen our study and clarify its contributions. We would greatly appreciate your further feedback and consideration.

Reviewer_FmNKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This analytical paper investigates whether brain–model relationships observed within a single acoustic condition generalize across different listening conditions. Its research perspective and multi-model comparison are valuable. The current stimulus coverage and experimental design appear insufficient to support strong explanatory conclusions. Further evidence on sentence-level robustness and model language compatibility would strengthen the study.

Critical strengths
  1. Novel research perspective. The study examines fMRI responses across Clean, Noisy, DNN-SE, and Classic-SE speech, providing a valuable perspective on whether brain–model alignment generalizes across acoustic conditions.

  2. Diverse computational models. Comparing multiple model families provides a useful basis for examining differences among computational representations.

Critical weaknesses
  1. Confounding between acoustic condition and sentence identity.

A central concern is that each sentence appears in only one acoustic condition. Consequently, the leave-one-condition-out (LOCO) evaluation simultaneously changes the acoustic regime and the linguistic stimuli. This makes it difficult to determine whether the observed differences reflect genuine cross-condition neural transportability or differences in sentence-level linguistic and acoustic characteristics. Since cross-condition generalization is the paper's primary contribution, this confound directly limits the interpretation of the central findings. Could the authors provide stronger controls for sentence-level differences or evaluate the same sentences under multiple acoustic conditions?

  1. Non-equivalent evaluation protocols for Static Alignment and LOCO.

The static evaluation uses 18 training sentences and 6 test sentences per fold, whereas LOCO uses 72 training sentences and 24 test sentences. Furthermore, the two settings use separately estimated noise ceilings. These differences may affect the reliability, normalization, and relative rankings of Brain Scores. Therefore, the increase in representation-class AUROC from 0.543 to 0.836 cannot be attributed solely to the cross-condition evaluation criterion. Could the authors conduct sample-size-matched analyses and examine whether the reported improvements in model discrimination persist under comparable evaluation settings?

  1. Potentially unfair comparison between encoder and decoder representations.

One of the paper's central findings is that encoder representations exhibit stronger neural transportability than decoder representations. However, the feature extraction procedures differ substantially: Whisper encoder representations retain stimulus-aligned states, whereas decoder representations are extracted from the final non-special token during autoregressive decoding. Differences in temporal aggregation, information availability, and decoding errors could contribute to the observed performance gap. How can the authors distinguish genuine representation-stage differences from artifacts of feature extraction? Additional controls using alternative pooling strategies and decoding procedures would strengthen this conclusion.

Questions for authors
  1. Audio-LLM coverage. The LLM comparisons focus on text-only models. Why exclude audio-input models such as Qwen3-ASR? Including a representative Audio-LLM would be particularly relevant to this task and help assess the generality of the findings.

  2. Stimulus diversity and sample size. Are 96 sentences, with only 24 per condition, sufficiently diverse to support the conclusions? Particular sentences may favor certain models because of their linguistic characteristics. The authors should clarify stimulus selection and assess whether model rankings remain stable across different sentence subsets.

  3. Mandarin compatibility. Do all evaluated checkpoints have appropriate Mandarin capabilities? Limited Mandarin proficiency could affect model representations and confound comparisons, especially for ASR decoders. Please specify the checkpoints, language coverage, and relevant Mandarin performance.

3 comments
AuthorsOpenReview ↗

Responses to Audio-LLM Coverage and Stimulus Diversity & Sample Size

We sincerely thank the reviewer for the thoughtful feedback and encouraging assessment of our study's strengths, particularly the novel perspective on cross-condition brain–model alignment, the comprehensive evaluation across diverse acoustic conditions, and the broad comparison of computational models. We are grateful for the critical suggestions and address the reviewer's questions point by point below.

1. Audio-LLM Coverage

We thank the reviewer for this valuable suggestion. We additionally evaluated Qwen3-ASR-1.7B as a representative audio-input language model, using the same strict fold-local LOCO protocol and decoder feature-extraction procedure as the other sequence-to-sequence models. The results are directionally consistent with our main findings. Encoder representations achieve higher LOCO Brain Scores than decoder representations across all four held-out conditions, with the largest difference under Noisy speech (Δ = 0.15). The overall encoder advantage is smaller (Δ = 0.07, 95% CI = [−0.01, 0.15]), resembling the pattern observed for Whisper-Large-v3.

This additional evaluation extends our comparison to an audio-input language model. Across the evaluated architectures, encoders generally show higher neural transportability than decoders. We will include the Qwen3-ASR results in the revised manuscript.

2. Stimulus Diversity and Sample Size

To address the sample-size mismatch, we conducted an additional size-matched evaluation using 18 training and 6 test sentences for both protocols, preserving identical feature-extraction procedures.

Our central findings persist after matching sample sizes. Under Static evaluation, no representation-class difference survives FDR correction across the four conditions. In contrast, size-matched LOCO reveals significant encoder advantages for Clean (Δ = 0.289, q = 0.004), Classic-SE (Δ = 0.314, q = 0.004), and Noisy speech (Δ = 0.284, q = 0.004). Representation-class discrimination also remains higher under LOCO (AUROC = 0.853) than Static (AUROC = 0.545).

Within-model comparisons provide convergent evidence. Under size-matched LOCO, encoders significantly outperform their paired decoders in Whisper-Tiny (Δ = 0.150, q = 0.048), Whisper-Medium (Δ = 0.278, q = 0.002), and SpeechT5 (Δ = 0.329, q < 0.001). Whisper-Large-v3 shows a smaller, non-significant difference in the same direction (Δ = 0.069, q = 0.391).

These results show that LOCO's representation-stage selectivity persists under equal training and test sample sizes, ruling out unequal sample counts as a sufficient explanation for the observed pattern. While Static and LOCO intentionally differ in their training-condition distributions, the size-matched findings strengthen our central conclusion: cross-condition neural transportability reveals representational differences that conventional within-condition alignment does not reliably distinguish.

AuthorsOpenReview ↗

Responses to Mandarin compatibility

ModelHugging Face checkpoint
wav2vec 2.0TencentGameMate/chinese-wav2vec2-base
WavLMmicrosoft/wavlm-base
Whisper-Tinyopenai/whisper-tiny
Whisper-Mediumopenai/whisper-medium
Whisper-Large-v3openai/whisper-large-v3
SpeechT5microsoft/speecht5_asr
Phi-4-minimicrosoft/Phi-4-mini-instruct
GPT-2 XLuer/gpt2-xlarge-chinese-cluecorpussmall
Mistral-7Bmistralai/Mistral-7B-Instruct-v0.3

Mandarin compatibility varies across checkpoints. Chinese wav2vec 2.0 and GPT-2 XL are pretrained on Mandarin data, while Whisper and Phi-4-mini support multilingual processing, including Mandarin. WavLM-Base has also been used in previous Mandarin speech studies (de la Fuente & Jurafsky, 2024).

We include SpeechT5 for its unified speech and text encoder-decoder architecture, which supports speech recognition, speaker identification, and speech enhancement (Ao et al., 2022), making it particularly relevant to our study of acoustic perturbations.

[1] A. de la Fuente and D. Jurafsky, “A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models,” in Proc. Interspeech, 2024.

[2] J. Ao et al., “SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing,” in Proc. 60th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 5723–5738, 2022.

AuthorsOpenReview ↗

Invitation to Review Our Response and Additional Experimental Results

We sincerely thank the reviewer for the time and effort devoted to evaluating our manuscript and providing detailed feedback.

In response to your questions and concerns, we have conducted additional experiments and analyses to directly address the key methodological issues and provide stronger support for our main findings.

We kindly invite you to revisit our point-by-point responses and consider the new experimental results presented in our rebuttal. We believe these additional findings strengthen our study and clarify its contributions. We would greatly appreciate your further feedback and consideration.

Reviewer_VhpwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates brain–model alignment using a small fMRI dataset recorded from 25 participants hearing 96 stimuli sentences across clean, noisy, DNN-enhanced, and classical speech-enhanced conditions. Encoder and intermediate representations obtained from foundation models including, e.g., Whisper-Tiny, Whisper-Medium, and SpeechT5, were used.

Critical strengths

Brain-model alignment in more realistic, real world scenarios represented by, e.g., presence of ambient noise, is interesting and enlarges the application scope of brain signals facing foundation models.

Critical weaknesses

Limited size of fMRI dataset: Being restricted to only 25 subjects, listening to fewer than 100 sentences each, i.e., under one hour of speech data, it is difficult to access statistical significance of the experimental results (even under the leave one out cross-validation set up), and casts doubts on the veracity of any conclusions that may be drawn.

Limited technical novelty: the RSA based alignment approach is the same as those that were considered in prior researches on similar brain-(DNN)model alignment tasks (e.g., the following). It is unclear as to what extent any novelty can be substantiated in this respect.

Wingfield C, Zhang C, Devereux B, Fonteneau E, Thwaites A, Liu X, Woodland P, Marslen-Wilson W and Su L (2022) On the similarities of representations in artificial and brain neural networks for speech recognition. Front. Comput. Neurosci. 16:1057439. doi: 10.3389/fncom.2022.1057439

Technical soundness: The idea of using speech foundation models encoder layer representations, which are known to produce noisy, and impure speech content representations (e.g., >=30% phoneme clustering impurity observed on HuBERT encoder features, W.-H. Hsu et al. TALSP2021), to conduct RSA based alignment with fMRI signals, is highly questionable and offers no reliable interpretation of brain signals under various speech recording conditions.

Questions for authors

Please address the above issues for possible future submissions.

2 comments
AuthorsOpenReview ↗

Responses to Limited Size of fMRI Dataset and Methodological Novelty

We sincerely thank the reviewer for the feedback and acknowledgment of our study's strengths, particularly the importance of investigating brain–model alignment in realistic listening scenarios and its potential to advance brain-aligned foundation models. We will address each concern point by point below.

1. Limited Size of fMRI Dataset

We acknowledge the modest dataset size (25 participants, 96 sentences) and discuss this limitation in Section 5.4. Nevertheless, our confirmatory analyses treat participants as independent statistical units, using paired sign-flip permutation tests and Benjamini–Hochberg FDR correction. To address potential sample-size confounds, we conducted an additional size-matched evaluation (18 training/6 test sentences for both Static and LOCO). The central findings persist: LOCO reveals significant representation-class differences in three of four conditions (q = 0.004), whereas none survives correction under Static alignment. Representation-class discrimination also remains higher under LOCO (AUROC: 0.853 vs. 0.545). These results demonstrate that the observed representation-stage selectivity is not simply attributable to LOCO's larger training or test sets. While the findings are consistent across participants, we acknowledge that broader generalization across stimuli requires further validation.

2. Methodological Novelty

Our primary analysis is not RSA-based. RSA serves only as a complementary analysis (Section 3.9; Appendix A). Instead, we use ridge-regularized neural encoding with leave-one-condition-out (LOCO) evaluation, fitting a model-to-brain mapping on three acoustic conditions and testing its generalization to an unseen fourth without refitting. We acknowledge prior brain–model alignment work, including Wingfield et al. (2022), and have expanded the related work accordingly. Our contribution is to establish cross-condition neural transportability under speech-specific acoustic perturbations and demonstrate representation-stage differences that conventional Static alignment does not reliably distinguish.

AuthorsOpenReview ↗

Invitation to Review Our Response and Additional Experimental Results

We sincerely thank the reviewer for the time and effort devoted to evaluating our manuscript and providing detailed feedback.

In response to your questions and concerns, we have conducted additional experiments and analyses to directly address the key methodological issues and provide stronger support for our main findings.

We kindly invite you to revisit our point-by-point responses and consider the new experimental results presented in our rebuttal. We believe these additional findings strengthen our study and clarify its contributions. We would greatly appreciate your further feedback and consideration.

Reviewer_isgaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper Summary:

The paper proposes cross-condition neural transportability as a complement to standard within-condition ("static") brain–model alignment. The authors fit a ridge-regression mapping from model features to fMRI responses using three speech conditions. They then freeze this mapping and evaluate it on a fourth, unseen condition using leave-one-condition-out (LOCO) evaluation. It appears that authors conducted an fMRI experiment on 25 native Mandarin listeners hearing 96 sentences, in four conditions: Clean speech, Noisy speech, DNN-enhanced speech (SEMamba), Classical speech enhancement (MMSE estimator). BOLD responses in the Auditory and Language regions were extracted for further analysis. The main emphasis is on comparing Static vs Cross-condition encoder performance. They introduce Leave-one-condition-out (LOCO) evaluation framework. Their results point out that representations taken from encoders outperform those taken from matched decoder representations. The proposed novelty is in the LOCO evaluation approach.

Critical strengths

Critical Strengths

  1. The question is well-posed and experiments and evaluations are in line with the proposed scope.

    • LOCO frame work is compared against within condition evaluation to see the generalization across unseen input conditions.
    • It appears that the fMRI data collected is also a contribution.
  2. Comprehensive and well-thought out experiment design.

    • Layers are reselected inside each LOCO fold using only the three training conditions. The held-out condition is not used for layer selection, hence no leakage is expected.
    • Higher neural transportability is observed in encoder/intermediate representations than in decoder representations.
Critical weaknesses

Critical Weaknesses

  1. The encoder–decoder comparison experiments appear inadequate.

    • The encoder and decoder representations are extracted differently.
    • Encoder features retain stimulus-aligned states, whereas decoder features use the final non-special token state during autoregressive decoding (Section 3.3; Appendix F.1).
    • It appears that the temporal information and aggregation of the features is not balanced across these experiments.
    • Therefore any claims of encoder representations being better need to be tempered down as decoder ones seem to be unfairly pitted against encoder representations.
  2. Model Checkpoint details, Training, test size mismatches and score reliability.

    • Model Checkpoint details: Section 3.3 states that exact checkpoints and extraction details are provided in Appendix F. However, Table F1 reports model names, parameter counts and depths without precise checkpoint identifiers, revisions or task-specific fine-tuning details. Most of these have English/LibriSpeech training. These would introduce potential language mismatches for Mandarin speech, please report the actual model identifiers to make this point clear.

    • Pooling and feature extraction: Prior work by Hadidi et al. (2026), cited by the authors, shows that activation extraction and pooling choices can affect brain predictivity. The different extraction conventions used here therefore need to be controlled.

    • Train/Test Size Mismatch: Static evaluation trains on 18 sentences and tests on 6. LOCO trains on 72 sentences and tests on 24. This changes both the amount of data available to fit the mapping and the number of samples used to calculate the held-out correlation. The key question is whether LOCO reveals a genuine difference in representation generalization or whether part of the observed improvement comes from having more training and test sentences.

Questions for authors

Questions For Authors

====================

  1. Matched extraction (Weakness 1)

Does the encoder advantage persist when encoder and decoder features are extracted using more comparable methods? Would it help if you used teacher-forced decoder states with the ground-truth Mandarin transcripts, while retaining the audio input and decoder cross-attention? If the encoder advantage remains under these controls, I would consider the representation-stage conclusion substantially stronger.

  1. Size-matched controls (Weakness 2)

It appears that within-condition generalization using the same overall training and test sizes as LOCO would give more credibility to the stated claim about LOCO. Please report representation discrimination and encoder–decoder contrasts where train/test set sizes are appropriately matched. Similarly, experiments that separate the effect of condition-held-out transfer (LOCO) from the effect of having more training and test sentences in the Static condition.

  1. Learning claim

How many independent random initializations were used for the random-weight controls? Do the reported confidence intervals include variation across initializations, or only variation across participants for the claims of random-versus-trained comparisons?

It is not clear whether the main findings are confounded by the aggregation choices and nose ceiling normalization choices adopted by the authors. Please clarify.

  1. Dataset contribution and related work

Was the dataset collected specifically for this study? Have any parts of the data or analyses appeared previously, for example, a Clean-versus−3 dB SNR Whisper-Tiny encoding analysis in a workshop paper?

If so, please cite the earlier work in anonymized form, explain what is new in this submission, and clarify whether the data and code will be released.

Please also distinguish the contribution from the following relevant work:

  • Chen et al. (ICML 2026), "Real-World Unsupervised Models Generalize to Predict Brain Responses to Out-of-Distribution Stimuli." This work demonstrates OOD neural-response prediction, including using Mandarin-trained models to predict English-driven auditory brain responses. The present paper should explain what is added by explicitly testing the transfer of a fitted model-to-brain mapping across held-out speech conditions.
  • Madan et al. (NeurIPS 2024), "Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex." This work directly evaluates the OOD generalization of fitted neural encoding models in visual cortex. It is an important methodological precedent for the present evaluation question.

A clearer comparison with these studies would help establish the precise novelty of the proposed LOCO evaluation.

4 comments
AuthorsOpenReview ↗

Responses to Matched Extraction and Size-matched Controls

We sincerely thank the reviewer for the careful review, constructive feedback, and recognition of our study's strengths, particularly the well-motivated research question, rigorous experimental design, valuable findings, and contribution of our fMRI dataset. We appreciate these insightful suggestions and address each concern below.

1. Matched extraction

Our feature-extraction procedure follows brain–model encoding studies (Schrimpf et al., 2021; Goldstein et al., 2025), as described in Section 3.4, preserving methodological consistency with prior benchmarks of static neural alignment. Importantly, we apply the identical feature-extraction procedure to both Static and leave-one-condition-out (LOCO) evaluations, changing only the evaluation framework while keeping the representation construction fixed. This design enables us to assess how conclusions about brain–model correspondence differ between within-condition alignment and cross-condition neural transportability, without introducing feature-extraction differences between the two evaluations.

2. Size-matched controls

To address the sample-size mismatch, we conducted an additional size-matched evaluation using 18 training and 6 test sentences for both protocols, preserving identical feature-extraction procedures.

Our central findings persist after matching sample sizes. Under Static evaluation, no representation-class difference survives FDR correction across the four conditions. In contrast, size-matched LOCO reveals significant encoder advantages for Clean (Δ = 0.289, q = 0.004), Classic-SE (Δ = 0.314, q = 0.004), and Noisy speech (Δ = 0.284, q = 0.004). Representation-class discrimination also remains higher under LOCO (AUROC = 0.853) than Static (AUROC = 0.545).

Within-model comparisons provide convergent evidence. Under size-matched LOCO, encoders significantly outperform their paired decoders in Whisper-Tiny (Δ = 0.150, q = 0.048), Whisper-Medium (Δ = 0.278, q = 0.002), and SpeechT5 (Δ = 0.329, q < 0.001). Whisper-Large-v3 shows a smaller, non-significant difference in the same direction (Δ = 0.069, q = 0.391).

These results show that LOCO's representation-stage selectivity persists under equal training and test sample sizes, ruling out unequal sample counts as a sufficient explanation for the observed pattern. While Static and LOCO intentionally differ in their training-condition distributions, the size-matched findings strengthen our central conclusion: cross-condition neural transportability reveals representational differences that conventional within-condition alignment does not reliably distinguish.

AuthorsOpenReview ↗

Responses to Learning Claim and Dataset Contribution & Related Work

3. Learning claim

Random initializations. Our random-weight controls preserve the architectures of their pretrained counterparts using the default Hugging Face initialization with a fixed seed (2025). Thus, one random initialization was evaluated per architecture, and the reported 95% CIs reflect variability across 25 participants, not across initializations. We therefore restrict our learning-related conclusions to these architecture-matched controls.

Aggregation and noise-ceiling normalization. Following Schrimpf et al. (2021), cross-subject noise ceilings are estimated through voxel-wise asymptotic extrapolation and bootstrap resampling. After excluding voxels with unreliable ceiling estimates (< 5 × 10⁻⁵ or ≥ 0.99), retained voxel estimates are aggregated to obtain a model-independent ceiling for each held-out condition, shared across the 14 ROIs. Brain Scores are computed by dividing ROI-averaged raw predictivity by the corresponding ceiling. GlobalLOCO then averages signed Brain Scores equally across the 14 ROIs and four held-out conditions (Eq. 1).

Importantly, the same retained-voxel masks and brain-side aggregation procedures are applied across models. Because the ceiling is independent of model identity and shared within each condition, normalization cannot reverse the sign of individual within-condition paired contrasts. We acknowledge that condition-dependent normalization can alter the relative weighting of conditions in GlobalLOCO. All inference uses signed participant-level paired differences with Benjamini–Hochberg FDR correction (Section 3.4–3.7; Appendix F).

4. Dataset contribution and related work

The fMRI dataset was collected specifically for this research. We will cite the earlier work in anonymized form and clarify the distinct contributions.

We acknowledge that OOD neural prediction is not new. Madan et al. (NeurIPS 2024) examine generalization under visual distribution shifts, while Chen et al. (ICML 2026) investigate OOD neural prediction, including cross-language settings. Our contribution extends this direction to speech-specific acoustic perturbations and their behavioral consequences.

Across four listening conditions, LOCO tests whether fitted model-to-brain mappings generalize to unseen conditions. Under size-matched evaluation, LOCO provides greater representation-class discrimination than Static alignment (AUROC: 0.853 vs. 0.545), with significant encoder advantages in three of four matched systems. We further find that neural transportability does not improve comprehension prediction beyond acoustic condition and ASR error.

Thus, cross-condition neural transportability provides a complementary evaluation criterion that reveals representation-stage differences beyond Static alignment while remaining distinct from ASR recoverability and human comprehension.

AuthorsOpenReview ↗

Invitation to Review Our Response and Additional Experimental Results

We sincerely thank the reviewer for the time and effort devoted to evaluating our manuscript and providing detailed feedback.

In response to your questions and concerns, we have conducted additional experiments and analyses to directly address the key methodological issues and provide stronger support for our main findings.

We kindly invite you to revisit our point-by-point responses and consider the new experimental results presented in our rebuttal. We believe these additional findings strengthen our study and clarify its contributions. We would greatly appreciate your further feedback and consideration.

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address the concerns raised in my review.

  1. Size-matched evaluation - substantially addressed.

The new size-matched results (18 training / 6 test sentences) strengthen the paper. The continued encoder advantages in three of four matched systems and the higher descriptive LOCO AUROC (0.853 vs. 0.545 for Static) make unequal sample sizes an unlikely sole explanation for the findings. However, please clarify whether the results are stable across repeated sentence subsampling and whether layer selection and ridge tuning remain strictly training-only.

  1. Encoder–decoder comparison - remains unresolved.

My original concern was not whether feature extraction is identical between Static and LOCO, but whether it is comparable between encoders and decoders. As described in Section 3.3 and Appendix F.1, encoder representations retain stimulus-aligned information, whereas decoder representations use the final non-special token state. The comparison changes both representation stage and feature-extraction convention. The response does not resolve this concern. A matched-pooling sensitivity analysis, potentially including teacher-forced decoder states, would strengthen the stage-level interpretation. Alternatively, the conclusions should explicitly acknowledge that the reported advantage applies to the specific representations and extraction procedures evaluated.

  1. Direct evidence for LOCO selectivity.

Significant encoder–decoder differences under LOCO, together with non-significant differences under Static, do not necessarily establish a significant difference between the two evaluation protocols. Could the authors directly compare the encoder–decoder effect under size-matched LOCO versus Static, using paired participant-level contrasts and reporting confidence intervals? This would more directly support the central claim that LOCO reveals representation-stage differences not detected by Static evaluation.

  1. Learning controls, normalization, and novelty.

I appreciate the clarification regarding the single random initialization and noise-ceiling normalization. The learning conclusion should remain limited to the evaluated random-weight instances. Reporting the primary comparisons using raw Pearson correlations would further establish robustness to normalization. The positioning relative to Madan et al. (2024) and Chen et al. (2026) is improved.

Overall assessment

The author comments provides meaningful new evidence and addresses the sample-size concern. However, the encoder-decoder extraction confound and the absence of a statistical comparison of stage effects between Static and LOCO remain important to the paper's conclusions. Considering the originality, significance, and strength of the empirical evidence, I currently maintain my recommendation. I remain open to revising it if these remaining concerns are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.