Beyond Static Brain Alignment: Cross-Condition Neural Transportability In Speech Models

Brain-model alignment is typically evaluated within the stimulus regime used to fit the neural mapping, leaving open whether the fitted relationship remains predictive when sensory conditions change. We operationalize cross-condition neural transportability using leave-one-condition-out (LOCO) evaluation, fitting a model-to-brain mapping on three speech conditions and testing it on an unseen fourth. Using fMRI from 25 participants hearing 96 sentences across Clean, Noisy, DNN-enhanced, and classical speech-enhanced conditions, we find that conventional within-condition alignment is broad whereas cross-condition evaluation is more selective. Under strict fold-local layer selection, encoder representations show greater transportability than matched decoder representations in Whisper-Tiny, Whisper-Medium, and SpeechT5, with higher transportability concentrated in encoder and intermediate representations. Architecture-matched controls further show that trained Whisper-Tiny exceeds its random-weight counterpart, providing direct evidence that learned structure can improve neural transportability beyond architectural inductive bias in at least one controlled model family. A complementary descriptive analysis shows representation-class AUROC increasing from $0.543$ under static evaluation to $0.836$ under LOCO, while static alignment and cross-condition transportability remain only moderately associated. Sentence-level LOCO fit does not improve held-out comprehension prediction beyond acoustic condition and Automatic Speech Recognition (ASR) character error rate. Because sentence identity is fixed to condition, LOCO measures joint transfer across acoustic regime and item set rather than the acoustic manipulation alone. Together, these findings show that strong within-condition neural correspondence does not guarantee a transportable model-to-brain mapping, supporting cross-condition neural transportability as a complementary criterion for evaluating brain-aligned speech representations.