- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes leave-one-condition-out evaluation to test whether model-to-brain mappings for speech survive changes in acoustic conditions. It reports that encoders transfer better than decoders and that cross-condition testing separates models better than within-condition testing.
Critical strengths
The question is valuable: brain alignment is almost always tested within one stimulus distribution, and asking whether a fitted mapping generalizes across conditions is a needed evaluation.
Critical weaknesses
-
There's a possible confound - LOCO trains on four times more sentences than the static test, so the main conclusion (LOCO is more discriminative) cannot be separated from sample size. A static baseline with matched training size is required.
-
Similarly, if I'm not mistaken, each condition uses different sentences, with only 24 per test set, so “transfer across acoustic conditions” is also transfer across item sets. An analysis of this effect is needed.
-The raw predictivity (before noise-ceiling normalization), and the computed brain-scores depend on small, unstable noise ceilings. Raw correlations should be reported alongside the normalized result, and be discussed clearly when very low.
Questions for authors
My main questions are
-
Does the static-versus-LOCO difference in discrimination persist when static encoding is trained on the same number of sentences?
-
Does the encoder advantage hold after controlling for decoder recognition errors or with matched extraction? That is, it seems that a control experiment is needed: check whether the encoder advantage persists when recognition is good. For example, test it only on sentences the model transcribed correctly, include character error rate as a covariate, or compare models with very different error rates. Whisper-Large-v3 makes far fewer errors, and it is the one model where the encoder advantage is not significant, which hints that recognition errors may be driving the effect...
2 comments
Invitation to Review Our Response and Additional Experimental Results
We sincerely thank the reviewer for the time and effort devoted to evaluating our manuscript and providing detailed feedback.
In response to your questions and concerns, we have conducted additional experiments and analyses to directly address the key methodological issues and provide stronger support for our main findings.
We kindly invite you to revisit our point-by-point responses and consider the new experimental results presented in our rebuttal. We believe these additional findings strengthen our study and clarify its contributions. We would greatly appreciate your further feedback and consideration.
Responses to Size-matched Control and Encoder Advantage & Decoder Recognition Errors
We sincerely thank the reviewer for the feedback and recognition of our study's contribution, particularly the importance of moving beyond conventional brain–model alignment to evaluate whether fitted neural mappings generalize across unseen acoustic conditions. We appreciate the reviewer's critical comments and clarify each concern below.
1. Size-matched Control
To address the sample-size mismatch, we conducted an additional size-matched evaluation using 18 training and 6 test sentences for both protocols, preserving identical feature-extraction procedures.
Our central findings persist after matching sample sizes. Under Static evaluation, no representation-class difference survives FDR correction across the four conditions. In contrast, size-matched LOCO reveals significant encoder advantages for Clean (Δ = 0.289, q = 0.004), Classic-SE (Δ = 0.314, q = 0.004), and Noisy speech (Δ = 0.284, q = 0.004). Representation-class discrimination also remains higher under LOCO (AUROC = 0.853) than Static (AUROC = 0.545).
Within-model comparisons provide convergent evidence. Under size-matched LOCO, encoders significantly outperform their paired decoders in Whisper-Tiny (Δ = 0.150, q = 0.048), Whisper-Medium (Δ = 0.278, q = 0.002), and SpeechT5 (Δ = 0.329, q < 0.001). Whisper-Large-v3 shows a smaller, non-significant difference in the same direction (Δ = 0.069, q = 0.391).
These results show that LOCO's representation-stage selectivity persists under equal training and test sample sizes, ruling out unequal sample counts as a sufficient explanation for the observed pattern. While Static and LOCO intentionally differ in their training-condition distributions, the size-matched findings strengthen our central conclusion: cross-condition neural transportability reveals representational differences that conventional within-condition alignment does not reliably distinguish.
2. Encoder Advantage & Decoder Recognition Errors
We agree that recognition errors may contribute to encoder–decoder differences. However, LOCO is designed to test robustness across speech conditions, including the effects of recognition errors. Encoder representations are extracted directly from audio, while decoder representations depend on generated tokens and may be more affected by recognition failures. Evaluating only correctly transcribed sentences would favor easier stimuli and change the test distribution.
The results suggest that recognition errors alone do not explain the encoder advantage. Whisper-Large-v3 has the lowest mean CER (0.32) and the smallest advantage, consistent with the reviewer's hypothesis. However, Whisper-Tiny has a higher CER than Whisper-Medium (0.61 vs. 0.39) but a smaller encoder advantage (0.150 vs. 0.278), contrary to a simple error-driven explanation (Table G1). Nevertheless, recognition accuracy, model size, and architecture remain difficult to disentangle.
We therefore interpret our findings in terms of predictive robustness: encoder representations show significantly greater neural transportability in three of four architecture-matched comparisons under the evaluated decoding procedures. This supports an encoder advantage in cross-condition generalization without assuming that it is independent of recognition errors.