acceptodds
Under review as a conference paper at ICLR 2027

Modality-agnostic brain-tuning of text and speech representations

Abstract

The human brain has largely shared higher-level language representations during reading and listening, despite differences in early sensory processing. Recent "brain-tuning" studies have shown that fine-tuning language models with brain data can improve their semantic representations, but these studies have focused on a single modality. Here we ask whether brain-tuning can help multimodal models learn brain-relevant representations that are shared across modalities. We brain-tune two multimodal text-speech models using fMRI data collected during naturalistic reading and listening. We then evaluate how well the brain-tuned model encodes brain activity, both within and across modalities. We further evaluate the brain-tuned model's representation on several semantic and audio probing tasks. All analyses are conducted across model layers, on held-out participants. Our results show that brain-tuning improves both within- and cross-modality prediction accuracy across language-selective brain regions. These gains emerge primarily in middle and late layers, where brain-tuning also strengthens semantic and audio information. Notably, tuning with both modalities jointly yields greater improvements, for both encoding performance and probing tasks, than tuning with either modality alone. Together, these findings suggest that brain-tuning can strengthen brain-relevant language information shared across modalities, providing a way to ground multimodal language models in human brain representations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-tuning, fMRI, encoding models, language representations, multimodal models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mt3TOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper brain-tunes two multimodal text–speech models (SONAR and CLAP) on fMRI of participants reading and listening to stories. The study demonstrates that joint fine-tuning of text and speech encoders on fMRI data yields better alignment of model internal representations with fMRI signal. Their approach also results in (occasionally) better downstream classification performance on audio and semantic tasks as well as better encoding as measured with correlation. The paper concludes that joint-modality (reading and listening) fine-tuning on fMRI responses improves encoding for both modalities and probing performance of reading and listening stimuli.

Critical strengths

The paper is clearly written. It makes a rather convincing case for bimodal encoder alignment to brain responses during reading and story listening for optimal model performance. It also evaluates the model on two OOD regimes: out-of-story and out-of-subject. The authors provide a null model for comparison via fMRI signal shuffling. The paper presents interesting results and is an overall important contribution.

Critical weaknesses

The paper omits (or does not state clearly) several important methodological details. Also, the findings are only presented as observations, without further interpretation. The details are below.

  1. How was fMRI preprocessed (a short summary in appendix would suffice)?
  2. The authors state that model representations were downsampled via 3-lobe Lanczos interpolation to fMRI-specific sampling rate but they never mention what it is (TR?). The authors do not provide any justification as to why the downsampling is necessary.
  3. During training, fMRI signal is reconstructed in the fsaverage space. However, the test results are reported for voxels. There's no explanation as to this mismatch.
  4. Encoding performance is only measured via correlation. It is well-known that large absolute-value errors and high correlations can co-exist. It would be interesting to see true and reconstructed distributions overlaid.
  5. The main encoding results (Fig. 2) only show relative improvement before vs. after brain fine-tuning. While relevant, this view is hardly informative on its own. Absolute performance would be interesting to see as well as how it compares to the existing encoding models (TRIBE, etc.)
  6. In Fig. 3 the color legend is wrong for the speech encoder column.
  7. There is no explanation / interpretation as to why performance drops for the later layers in Fig. 3.
  8. Similarly, the authors provide no interpretation to a roughly consistent gain in encoding performance in (what looks like) the default mode network and mixed gains/losses everywhere else.
  9. The claim of modality agnosticism is rather overstated: only two stimulus types are featured, and both of them are language-based.
  10. The improvement of model internal representation alignment to brain responses is quantified as improvement in encoding performance. There are no embedding-based methods actually measuring the change in representation geometry: no link is established between the change in embedding geometry and, say, downstream performance.
Questions for authors

Improvement suggestions: In addition to addressing the comments above, the following would strengthen the paper.

  1. A figure showing folding / splits would make it clearer.
  2. I would suggest reformulating the part of the claim to focuses on multimodality. The setting is better characterized as bimodal language-based fine-tuning.
  3. Brain plots currently emphasize flat maps. Larger inflated brain surfaces (in the appendix if not in the main text) would make the paper clearer for a wider readership. In addition, outlines for the seven Yeo networks overlaid on the inflated surfaces and a network-wise ranking in terms of performance gain/loss after fine-tuning would give a better picture of local encoding gains.
  4. A clear comparison of encoding performance for within-story vs. out-of-story evaluation would be interesting.

Depending on the extent to which the concerns are addressed, I am willing to increase the score.

4 comments

Thank you for the constructive feedback! We address each point below:

Weaknesses

  1. We use the preprocessed data released by Deniz et. al 2019. Based on your suggestion, we added a short summary of the acquisition and preprocessing steps to Appendix Section A.

  2. fMRI is sampled once per TR (2.0045 s), whereas model representations are extracted at the token (text) or 500-ms chunk (speech) rate. We downsample the embeddings using 3-lobe Lanczos interpolation to match the two temporal resolutions so the representations can be linearly mapped to the BOLD. We have updated the Methods section (Sec 4.1) to include this.

  3. To clarify: for brain-tuning, each participant’s fMRI responses are mapped from native space to the common fsaverage surface, which lets us brain-tune a single model jointly across participants. For evaluation, we extract embeddings from the brain-tuned models and fit a separate VEM for each held-out participant. VEM results are therefore reported in each participant’s native space (e.g., Fig. 2), which is why they are shown per voxel.

  4. We use Pearson correlation because it is the standard metric in the voxelwise encoding modeling literature. Based on your suggestion, we visualized the distributions and time courses of predicted and recorded responses for example voxels in Fig. 8 (Appendix D). The predicted and recorded responses have similar distributions, and the predicted time courses track the recorded responses.

  5. In the main text, we show the change in encoding performance on the flatmaps (Fig. 2) to see where brain-tuning helps most and to compare directly with prior brain-tuning work [Moussa et al., 2025, Vattikonda et al., 2025, Negi et al., 2025, Vaidya et al., 2026]. Based on your suggestion, we added flatmaps of the absolute encoding performance of the brain-tuned model in Fig. 7 (Appendix D).

  6. Thank you for catching this. The legend in the second column should read “brain-tuned (speech-only)” in violet, as plotted. We have fixed this in the revised manuscript.

  7. The late-layer decline in brain encoding performance coincides with specific, measurable changes in speech representations, rather than a uniform loss of speech information. Three additional analyses support this interpretation:

    • CKA: Brain-tuning increases text–speech representational similarity primarily in speech layers 13–21, with little or no gain in the final layers (more info in point 10).
    • CTC: Connectionist temporal classification, aka character accuracy/probe is largely preserved through layer 20 (losses ≤2.6 points), but declines substantially in layers 21–24 (mean losses of −7.2 points under joint tuning and −8.7 under speech-only tuning). The largest observed decline occurs at layer 23 (−9.9 joint; −16.4 speech-only).
    • Phoneme probing: Phoneme recognition remains preserved (+0.63 points under joint tuning).

    Together, these findings suggest that brain-tuning increases cross-modal alignment in intermediate-to-deep speech layers, while the final layers lose some character-level transcription information but retain phonemic information. This provides a plausible explanation for the late-layer decline in brain encoding performance, rather than indicating a general deterioration of speech representations. We have incorporated these analyses and their interpretation into the revised manuscript (Appendix D.2, Figs. 16 & 17).

  8. The gains are in regions known to represent semantics during language comprehension. The flatmaps (Fig. 2) show improvements in regions similar to those reported by Deniz et al. (2019), the dataset we use, where semantic representations are shared across reading and listening. Effects outside these regions are weaker and more mixed. To quantify this further, we’ve added quantification results across language-selective ROIs (Fig. 14 for SONAR, Fig. 20 for CLAP).

  9. We clarify that “modality-agnostic” is used specifically in the context of text and speech, as explicitly stated in our title, “Modality-Agnostic Brain-Tuning of Text and Speech Representations.” We do not claim generalization to arbitrary sensory modalities. We now added “modality-agnostic” refers specifically to text and speech, to state this explicitly for clarity in the Introduction (line 49-50).

  10. We thank the reviewer for this suggestion. Beyond brain encoding performance, we now directly quantify changes in representation geometry using Centered Kernel Alignment (CKA) between text and speech encoder representations across all 24 × 24 layer pairs of SONAR (Fig. 16, Appendix D.2). Joint brain-tuning increases text-speech CKA from 0.073 to 0.0796 (+9.2%, p = 0.001) (avg. across layers), concentrated in intermediate-to-deep speech layers (13-21) and text layers (14-23), where the encoding (Fig. 3) and probing (Fig. 5) gains also emerge.

Questions

  1. (and 4.) We would like to clarify that all our evaluations are out-of-story. The dataset has 11 stories with independent content. Each story is collected in a separate scanner run. Seven are used for brain-tuning (six stories in training, 1 story in validation). The other four are held out from brain-tuning: three are used to train the voxelwise encoding models, and one is used for testing. To train the encoding models, the ridge parameter is chosen by 5-fold cross-validation over contiguous blocks of concatenated TRs on the training set. Encoding performance is measured on the test story, a story that was never seen during brain-tuning or encoding-model training, so there is no within-story evaluation to compare against. The in-distribution vs. out-of-distribution comparison differs in participants, not stories.

  2. Addressed in limitations 9 above.

  3. Based on your suggestion, we added the encoding results of Fig. 2 on inflated lateral and medial surfaces (Fig. 6, Appendix D). We also added a network-wise quantification of the change in encoding performance across the seven Yeo networks (Fig. 15). The largest gains are for the middle-to-late layers in the dorsal, ventral, default mode, and frontoparietal networks.

Thank you again for your feedback; the above additions make our results more clear, and we hope they also address your concerns. All these updates have been included accordingly in the revised manuscript.

Reviewer_mt3TOpenReview ↗

Response to rebuttal

I thank the authors for the rebuttal, I addresses most of my points. I still think that a figure depicting the evaluation procedure and splits would make it clearer and will gain. While details about the data train / held-out splits is available in multiple places throughout the manuscript, it would be easier for the reader to refer to a single figure in case they need a reminder. The explanations you provided in in your response to W. 7 and 8 would strengthen the paper on the interpretability front if featured in the discussion (if the page limit allows, of course). I will increase my score to 3.

Thank you for your quick response and for raising your score.

Based on your suggestion, we've added a figure visualizing the data splits between brain-tuning and voxelwise encoding evaluation in Fig. 24 (Appendix G). We have updated the Results and Discussion sections to briefly mention this interpretation and point to the relevant appendix.

Reviewer_h8iDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors fine-tune the text and speech encoders of SONAR (and CLAP, in the appendix) end-to-end. The training signal is fMRI from 6 participants of the Deniz et al. (2019) reading/listening dataset, using a negative-Pearson brain-prediction loss plus a cross-modal embedding alignment loss. They report small gains in within- and cross-modality voxelwise encoding on 2 held-out participants, and small gains on linear probes (SentEval, FlashHolmes, Speech Commands).

Critical strengths
  • The question is well motivated. Brain-tuning has so far been single-modality, and the Deniz et al. dataset (same stories, same participants, read and heard) is a good fit for testing transfer across modalities.
  • The end-to-end pipeline is sound: per-modality Lanczos downsampling, FIR delays, separate reading and listening heads, and full backpropagation into both encoders (Sec. 3.3, Fig. 1). It handles the different temporal resolutions of text and speech well.
  • Evaluation uses participants and stories held out from brain-tuning (Sec. 3.1, 5.1). The paper includes text-only, speech-only and shuffled-fMRI controls.
  • The fact that joint brain tuning yields improvements compared to unimodal tuning, but the evidence is rather shallow (see weaknesses)
Critical weaknesses
  • Limited significance. This is perhaps a personal opinion, but I struggle to be sold on the significance of brain tuning. The fact that it improves encoding scores seems trivial, since that is what the optimization task is built for. I really struggle to believe that it can improve downstream tasks, and this claim is not supported by sufficient empirical evidence (like in other papers from the literature), see below.
  • The probe gains are very small, and assessed on limited benchmarks. Results in Figure 4 are reported without variance across probe splits or tuning seeds, and appear very small in most cases (likely within noise). Additionally, these benchmarks appear very outdated (some are pre-2010 such as BioScope), and I do not believe they reflect realistically the performance of modern models such as SONAR: there is no demonstration that brain tuning does not degrade core abilities of the models (e.g. is the WER of SONAR in speech-to-text tasks affected?)
  • Some figures do not convey clear messages. For example it is very hard to compare the four panels quantiatively in figure 2. The authors should provide ROI-summarized barplots to enable quantitative comparisons substantiating claims such as "improvements concentrated in higher-level language and association regions, including AG, IFG, mPFC, and MPC".
  • Statistical support is very thin. Only 2 held-out participants and a single test story are used. This causes very noisy results for the speech-tuning setting (e.g. figure 3 b and d), which the authors interpret as improvement without any adequate statistical tests: "Here too, joint brain-tuning outperforms single modality tuning"
  • The claim of "modality-invariance" is not supported by evidence. No control is applied to show that the fact that text-tuning improves the speech encoder and vice-versa is related to leaning a "modality-invariant" representation: this cannot be inferred from encoding scores alone. To show this, the authors should look at the actual representations themselves: do the speech encoder latents become more similar to the speech encoder ones, and vice versa?
Questions for authors

See above

4 comments

Thank you for the constructive feedback! We address each point below:

  1. Significance of brain-tuning. We agree that higher encoding scores on the training data would be expected. Our evaluations are on held-out participants and held-out stories, so they measure generalization. The cross-modal results are not optimized directly either: brain-tuning the text encoder with reading data improves the prediction of listening responses. Our controls show that these gains come from brain supervision and not from additional training (Fig. 9). The downstream gains are shown in point 2.

  2. Small probe gains, variance, and benchmarks. Based on your suggestion, we now report the probing results across 10 seeds. We’ve updated Figs. 4 and 5 (and Figs. 22 and 23 for CLAP) to report mean ± SD across 10 probe seeds, and added paired tests with Benjamini-Hochberg FDR correction (Appendix D.2 for SONAR and F.2 for CLAP). Joint brain-tuning improves world-model probes (EWoK) by +0.90 ± 0.15 points (6 of 7 tasks; 4 significant after FDR) and linguistic probes by +0.47 ± 0.05 points (6 of 7 tasks; 6 significant). Audio probes improve by +1.87 ± 0.17 points on all 5 tasks, with speaker identification at +7.24 ± 0.76 points (q < 0.0001). Speech-only tuning gives no significant audio gain and declines on Keyword and KeywordHard. On SentEval, the SONAR gains (+0.15 ± 0.11) are small and do not survive FDR correction, while CLAP shows larger gains (+1.28 ± 0.07, p = 2×10⁻⁸). On text probes, joint and text-only tuning perform comparably (10/14 vs. 9/14 significant FlashHolmes tasks).

    Although some underlying datasets are older (e.g., BioScope), they are evaluated here as part of FlashHolmes (Waldis et al., 2024), a recent benchmark that curates such datasets, together with newer ones such as EWoK (Ivanova et al., 2025), to systematically test linguistic and world-knowledge information in frozen representations. Holmes/FlashHolmes is the standard probing suite in recent NeuroAI research (Oota et al., 2026; Merlin and Toneva, 2026) (Section 4.2; Appendix C, Tables 2-4).

    To check if brain-tuning degrades core abilities, based on your suggestion, we checked the WER: [text-only / speech-only] tuning has minimal impact (+0.2/+0.4 points), and joint tuning increases WER by +5.6/+6.4 points. Layerwise CTC (connectionist temporal classification aka character accuracy/probe) analysis localises this cost to the final speech-encoder layers, with earlier layers preserved (Section 5.2; Appendix D.4, Tables 8–10). Further, we evaluated FLORES retrieval, SLURP, cross-modal retrieval, and LibriSpeech ASR. FLORES high-resource retrieval and SLURP intent accuracy are preserved, and text-to-speech retrieval Recall@1 improves from 97.8% to 99.4%.

  3. Figure clarity and ROI summarized bar plots. Based on your suggestion, we’ve now added ROI-summarized bar plots of the change in encoding performance across language-selective ROIs for all four settings (Fig. 14 for SONAR, Fig. 20 for CLAP). Bars show the average change across layers, and dots show individual layers.

  4. Statistical support. We extended the evaluation to two test stories (wheretheresmoke and odetostepfather), refitting the encoding models on the remaining held-out stories, for both OOD participants. Fig. 3 (and CLAP results in Fig. 19) are updated, with the standard error computed across the 4 samples (2 stories × 2 OOD participants); the variations are slightly cleaner now. All results remain consistent with our earlier findings. With only 4 samples per data point, a statistical test is underpowered. Hence, we plot all individual samples alongside the mean and SE.

  5. Modality invariance. We thank the reviewer for this suggestion. We added a Centered Kernel Alignment (CKA) analysis between text and speech encoder representations across all 24 × 24 layer pairs of SONAR (Fig. 16, Appendix D.2). Joint brain-tuning increases text-speech CKA from 0.073 to 0.0796 (+9.2%, p = 0.001) (avg. across layers), concentrated in intermediate-to-deep speech layers (13-21) and text layers (14-23), where the encoding gains also emerge. Brain-tuning gives a significantly larger increase than the shuffled baseline (+0.0067 vs. +0.0006, p = 0.008), so the increase depends on meaningful brain supervision.

Thank you again for your feedback; the above additions make our results more robust, and we hope they also address your concerns. All these updates have been included accordingly in the revised manuscript.

I thank the authors for their response and valuable additions. However, I still struggle to be sold on the significance. The fact that the encoding score gain generalizes is no surprise since that is the goal of the brain tuning. The new experiments do robustify the empirical validation, but also confirm some of my fears: brain tuning degrades the WER of the speech model, which is the core measure of its capability (that is what the model is built for). Also, I feel like the figures showing effect on downstream tasks should show the baseline obtained by removing the neuro loss, keeping on the embedding alignment term, as I suspect this might be yielding the gains. This might be a bit speculative but I simply struggle to believe that supervision by noisy fMRI targets on a very limited dataset could improve some capabilities of such models without breaking others.

1. Significance beyond brain encoding. Beyond encoding gains, we demonstrate cross-modal transfer: reading supervision improves listening-brain prediction on unseen participants and stories. This transfer is not reproduced by shuffled-fMRI or alignment-only controls (Appendix D.1, Fig. 9). Brain-tuning also significantly increases text–speech representational similarity (+9.2% CKA, p=0.001), with a larger increase under real than shuffled fMRI (p=0.008; Appendix D.2). These results go beyond optimizing the brain-prediction objective.

2. Speech-to-text performance. We acknowledge the WER increase under joint tuning (+5.6/+6.4 points). Our additional TIMIT CTC probes show that transcription-related information is largely preserved through layer 12 (changes of about 1 point or less), where the speech encoder’s brain-encoding gains emerge (Fig. 3; Appendix D.4). The largest declines occur in deeper layers and are substantially smaller with brain supervision: at layer 24, alignment-only tuning reduces character and phoneme accuracy by 47.1 and 34.5 points, compared with 7.7 and 9.2 points under joint brain-tuning. Thus, the WER increase reflects deep-layer changes, while brain supervision better preserves transcription information than alignment alone. Because we evaluate SONAR with its frozen pretrained decoder, the WER increase may also reflect, in part, reduced compatibility between the decoder and the tuned final-layer representations, rather than a general loss of speech information.

ProbeLayerPretrainedBrain-tuned ΔSpeech-only ΔAlignment-only Δ
Character878.50−0.29−0.24−0.14
Character1277.97−1.04−1.09−0.86
Character2072.61−4.21−3.61−8.66
Character2462.22−7.66−8.79−47.08
Phoneme887.67−0.21−0.21−0.14
Phoneme1284.60−0.71−0.62−0.35
Phoneme2065.78−2.53−3.57−43.42
Phoneme2451.75−9.24−11.28−34.52

3. Alignment-only downstream baseline. We completed 10-seed alignment-only probing for SONAR and CLAP under the same protocol as the brain-tuned models.

  • FlashHolmes. Alignment-only tuning produces no significant group-level improvements in either model (SONAR: +0.14 world-model, +0.05 linguistic; CLAP: +0.25, +0.19; all p > 0.05), whereas joint brain-tuning yields significant improvements in both (SONAR: +0.90, +0.47; CLAP: +0.87, +1.84). Text-only tuning (without alignment loss) also improves all groups (SONAR: +0.58, +0.47; CLAP: +0.84, +1.69), with gains comparable to joint tuning.
ModelTuning conditionWorld-model Δ (mean±SE)Linguistic Δ (mean±SE)SentEval Δ
SONARBrain-tuning+0.90±0.15 (p = 2e-4)+0.47±0.05 (p = 3e-6)+0.15
SONARBrain-tuning (Text-only)+0.58±0.09 (p = 2e-4)+0.47±0.05 (p = 9e-6)+0.12
SONARAlignment-only+0.14±0.07 (p = 0.07)+0.05±0.04 (p = 0.22)+0.05
CLAPBrain-tuning+0.87±0.22 (p = 0.003)+1.84±0.11 (p = 6e-8)+1.28
CLAPBrain-tuning (Text-only)+0.84±0.21 (p = 0.003)+1.69±0.09 (p = 1e-8)+1.16
CLAPAlignment-only+0.25±0.22 (p = 0.28)+0.19±0.14 (p = 0.21)−1.01
  • SentEval. CLAP improves by +1.28 points under joint brain-tuning but declines by 1.01 points under alignment-only tuning (all five tasks), supporting the contribution of brain supervision beyond embedding alignment. SONAR shows small, nonsignificant gains under joint (+0.15), text-only (+0.12), and alignment-only (+0.05) tuning.

  • Audio probes. The SONAR audio gain is driven primarily by speaker identification (+7.24 points), while alignment-only tuning yields an even larger speaker gain (+23.45 points). Thus, the speaker improvement cannot be attributed specifically to brain supervision. Alignment alone, however, gains speaker information at the expense of lexical content: keyword accuracy falls by 10.88 and 13.61 points, whereas joint brain-tuning preserves it (+0.56 and +0.61). This distinction is important because keyword identification probes linguistic content, and our TIMIT analyses independently show that brain supervision better preserves transcription-related information than alignment-only tuning.

ModelTuning conditionKeyword ΔSpeaker ΔKeywordHard ΔSpeakerHard ΔPhoneme ΔGroup Δ (mean±SE)
SONARBrain-tuning+0.56+7.24*+0.61+0.31+0.63+1.87±0.17 (p=1.3e-6)
SONARBrain-tuning (Speech-only)−0.89*+2.45*−1.08*+0.09+0.47+0.21±0.19 (p=0.31)
SONARAlignment-only−10.88*+23.45*−13.61*+0.96*+4.03*+0.79±0.12 (p=1e-4)
CLAPBrain-tuning−0.26+0.05−0.10−0.67*+0.23−0.15 (p=0.34)
CLAPBrain-tuning (Speech-only)−0.45+0.05+0.35−0.61*+0.23−0.09 (p=0.63)
CLAPAlignment-only+2.43*+0.07*+3.00*−0.73*+0.79+1.11±0.11 (p=4e-6)

4. fMRI supervision. We use the established Deniz et al. (2019) dataset, with six participants for brain-tuning and two held out for evaluation. Despite limited training data, cross-modal transfer and gains absent from alignment-only controls show benefits beyond embedding alignment.

Dear Reviewer h8iD,

Thank you again for your follow-up and for acknowledging that our additional experiments strengthen the empirical validation.

Our response above addresses each of the points you raised, including the requested alignment-only baseline on all downstream probes, the layerwise analysis of the WER increase, and clarification of our claims and the observed trade-offs.

If our responses have satisfactorily addressed your concerns, we would appreciate your consideration of an updated evaluation. We would be happy to address any remaining questions during the discussion period.

Regards,
The Authors

Reviewer_yCvKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper finetunes dual text–speech encoders using reading and listening fMRI, combined with a text–speech alignment loss. It evaluates the resulting representations on brain-response prediction and various probing tasks.

Critical strengths

Using reading and listening responses to the same stories is a thoughtful experimental design. Testing language <-> audio generalization is interesting, and experimenting with both SONAR and CLAP adds thoroughness.

Critical weaknesses
  1. Important controls are missing.

The main comparisons change several things at once. Compared with the pretrained model, brain-tuning adds training on narrative text and audio. Compared with single-modality tuning, joint tuning also adds the alignment loss and updates both encoders. The missing conditions are alignment alone, alignment plus reading fMRI, and alignment plus listening fMRI. Training budgets should be matched too. These would help establish whether brain supervision adds anything beyond further text-speech training, and whether each brain signal contributes.

  1. It is unclear how consistent the gains are.

The main brain evaluation uses two held-out participants and one test story, which is very limited. Some highlighted downstream gains also come from very few examples: the 3.33-point improvement on physical dynamics corresponds to two additional correct answers out of 60 (Figure 4; Appendix B). The confidence intervals in Figure 3 are great, but they are missing in other Figures. They would help establish statistical significance.

Questions for authors

Note that Figure 3 has inconsistent captions.

5 comments

Thank you for the constructive feedback! We address each point below:

  1. Incremental controls. Following your suggestion, we added three more controls with matched training budgets: (i) alignment only, (ii) alignment + reading fMRI, and (iii) alignment + listening fMRI. In the table below, we report the change in encoding performance relative to the pretrained model (mean SE across OOD participants) for the best layer:
SettingBrain-tuned (joint)Brain-tuned (text/speech-only)Alignment onlyAlignment + readingAlignment + listeningShuffled baseline
a) Text encoder predicting reading+4.2% ± 0.0+3.7% ± 0.2-0.2% ± 1.1+3.5% ± 0.5+0.2% ± 1.1+0.6% ± 0.1
b) Speech encoder predicting listening+1.5% ± 1.6+0.2% ± 0.0-12.7% ± 2.2-12.6% ± 1.7+0.8% ± 1.0+0.9% ± 0.8
c) Text encoder predicting listening+3.8% ± 0.5+2.5% ± 0.9-1.2% ± 0.4+2.2% ± 0.8-0.9% ± 0.4+1.4% ± 0.7
d) Speech encoder predicting reading+3.4% ± 1.1+3.2% ± 0.3-16.1% ± 5.3-15.0% ± 6.0+1.8% ± 0.8+1.8% ± 0.5

These show that the gains come from brain supervision and not from further text-speech training, and that each brain signal contributes. Bar graphs visualising this result is added in Appendix Section D.1 and Fig.9 of the updated paper. We’ve also updated the text in Methodology Section 3.3 accordingly.

  1. Participants. As noted in the limitations, we agree that the dataset size is limited. fMRI data is expensive to collect, and this dataset requires both reading and listening conditions for the same participants and stories. For a fair evaluation, we hold out 2 participants and 4 stories from brain-tuning, which is 25% of this dataset. We report our encoding performance evaluation on both in-distribution (ID) and out-of-distribution (OOD) participants. Gains are, as expected, larger for ID participants, and the same trends hold for OOD participants (Figs. 3 and 13).

    Based on your feedback, we extended our analysis to test on two stories (in a leave-one-out fashion), refitting the VEMs on the remaining held-out stories. Fig. 3 (and CLAP results in Fig. 19) are updated, with the standard error computed across the 4 samples (2 stories × 2 OOD participants). All results remain consistent with our earlier findings.

    Downstream gains and Confidence intervals. We thank the reviewer for highlighting the small test set and the need to assess variability. The previously reported +3.33-point improvement in Physical Dynamics was based on a single probe run, corresponding to two additional correct predictions out of 60 examples. We have now repeated the evaluation across 10 paired probe seeds and conducted paired t-tests with Benjamini–Hochberg FDR correction across 18 FlashHolmes tasks.

    The revised improvement is +1.17 percentage points and remains statistically significant (q = 0.042) under joint brain-tuning. Text-only tuning yields a comparable improvement (+1.17 points, q = 0.047). Although the test set remains small, these results establish that the positive effect is not restricted to the originally reported probe run. We have included the seed-level uncertainty and task-level significance results in the revised manuscript (Appendix D, Table 6). All probing results (Fig. 4 and 5, and Fig. 22 and 23 for CLAP) are updated to show the mean ± SE across 10 seeds (1-6, 7, 13 and 42).

  2. Fig. 3 caption. Thank you for catching this! The legend in the second column should read “brain-tuned (speech-only)” in violet, as plotted. We have fixed this in the revised manuscript.

Thank you again for your feedback; the above additions make our results more robust, and we hope they also address your concerns. All these updates have been included accordingly in the revised manuscript.

Thanks for the new analyses! They strengthen the results, but I'm still not fully convinced by the evidence for the consistency of the benefit (as other reviewers have also pointed out) and the broader downstream benefits. I will keep my score unchanged.

Thank you for the follow-up and for acknowledging our additional analyses. Since your comment, we have completed further controls directly addressing consistency and downstream benefits.

1. Consistency of the benefit. Following reviewr's suggestions, we expanded the encoding evaluation from one to two test stories for both held-out participants and included the six in-distribution participants. The text-encoder improvements remain consistent across participants, stories, and within- and cross-modal evaluations, with similar patterns in SONAR and CLAP. The matched-budget shuffled-fMRI and alignment-only controls do not reproduce these improvements (Appendix D.1, Fig. 9). We also repeated downstream probing across 10 seeds, reporting mean±SD and FDR-corrected significance.

2. Downstream benefits. We additionally evaluated alignment-only tuning on all probes using the same 10-seed protocol. World-model and linguistic improvements are significant under brain-tuning but absent under alignment-only tuning in both models:

ModelTuning conditionWorld-model ΔLinguistic Δ
SONARJoint brain-tuning+0.90 ± 0.15 (p = 2e-4)+0.47 ± 0.05 (p = 3e-6)
SONARText-only brain-tuning+0.58 ± 0.09 (p = 2e-4)+0.47 ± 0.05 (p = 9e-6)
SONARAlignment-only+0.14 ± 0.07 (p = 0.07)+0.05 ± 0.04 (p = 0.22)
CLAPJoint brain-tuning+0.87 ± 0.22 (p = 0.003)+1.84 ± 0.11 (p = 6e-8)
CLAPText-only brain-tuning+0.84 ± 0.21 (p = 0.003)+1.69 ± 0.09 (p = 1e-8)
CLAPAlignment-only+0.25 ± 0.22 (p = 0.28)+0.19 ± 0.14 (p = 0.21)

Δ represents the change relative to pretrained (mean±SD across 10 seeds). On CLAP, SentEval also improves by +1.28 points under joint brain-tuning but declines by 1.01 points under alignment-only tuning.

3. Scope of our conclusions. We have clarified that the strongest evidence concerns text-encoder improvements in brain alignment and linguistic/world-model probing. Speech-encoder effects are more variable, and we explicitly report downstream trade-offs, including WER (see our response to Reviewer h8iD).

We have addressed the original concerns through additional controls, expanded encoding evaluations, and 10-seed probing with statistical tests. Given these results, could you please clarify which specific findings you still consider inconsistent, or what further evidence would resolve your concerns? We would appreciate the opportunity to address them during the discussion period.

Thank you again for your constructive feedback and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.