- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper brain-tunes two multimodal text–speech models (SONAR and CLAP) on fMRI of participants reading and listening to stories. The study demonstrates that joint fine-tuning of text and speech encoders on fMRI data yields better alignment of model internal representations with fMRI signal. Their approach also results in (occasionally) better downstream classification performance on audio and semantic tasks as well as better encoding as measured with correlation. The paper concludes that joint-modality (reading and listening) fine-tuning on fMRI responses improves encoding for both modalities and probing performance of reading and listening stimuli.
Critical strengths
The paper is clearly written. It makes a rather convincing case for bimodal encoder alignment to brain responses during reading and story listening for optimal model performance. It also evaluates the model on two OOD regimes: out-of-story and out-of-subject. The authors provide a null model for comparison via fMRI signal shuffling. The paper presents interesting results and is an overall important contribution.
Critical weaknesses
The paper omits (or does not state clearly) several important methodological details. Also, the findings are only presented as observations, without further interpretation. The details are below.
- How was fMRI preprocessed (a short summary in appendix would suffice)?
- The authors state that model representations were downsampled via 3-lobe Lanczos interpolation to fMRI-specific sampling rate but they never mention what it is (TR?). The authors do not provide any justification as to why the downsampling is necessary.
- During training, fMRI signal is reconstructed in the fsaverage space. However, the test results are reported for voxels. There's no explanation as to this mismatch.
- Encoding performance is only measured via correlation. It is well-known that large absolute-value errors and high correlations can co-exist. It would be interesting to see true and reconstructed distributions overlaid.
- The main encoding results (Fig. 2) only show relative improvement before vs. after brain fine-tuning. While relevant, this view is hardly informative on its own. Absolute performance would be interesting to see as well as how it compares to the existing encoding models (TRIBE, etc.)
- In Fig. 3 the color legend is wrong for the speech encoder column.
- There is no explanation / interpretation as to why performance drops for the later layers in Fig. 3.
- Similarly, the authors provide no interpretation to a roughly consistent gain in encoding performance in (what looks like) the default mode network and mixed gains/losses everywhere else.
- The claim of modality agnosticism is rather overstated: only two stimulus types are featured, and both of them are language-based.
- The improvement of model internal representation alignment to brain responses is quantified as improvement in encoding performance. There are no embedding-based methods actually measuring the change in representation geometry: no link is established between the change in embedding geometry and, say, downstream performance.
Questions for authors
Improvement suggestions: In addition to addressing the comments above, the following would strengthen the paper.
- A figure showing folding / splits would make it clearer.
- I would suggest reformulating the part of the claim to focuses on multimodality. The setting is better characterized as bimodal language-based fine-tuning.
- Brain plots currently emphasize flat maps. Larger inflated brain surfaces (in the appendix if not in the main text) would make the paper clearer for a wider readership. In addition, outlines for the seven Yeo networks overlaid on the inflated surfaces and a network-wise ranking in terms of performance gain/loss after fine-tuning would give a better picture of local encoding gains.
- A clear comparison of encoding performance for within-story vs. out-of-story evaluation would be interesting.
Depending on the extent to which the concerns are addressed, I am willing to increase the score.
4 comments
Questions
-
(and 4.) We would like to clarify that all our evaluations are out-of-story. The dataset has 11 stories with independent content. Each story is collected in a separate scanner run. Seven are used for brain-tuning (six stories in training, 1 story in validation). The other four are held out from brain-tuning: three are used to train the voxelwise encoding models, and one is used for testing. To train the encoding models, the ridge parameter is chosen by 5-fold cross-validation over contiguous blocks of concatenated TRs on the training set. Encoding performance is measured on the test story, a story that was never seen during brain-tuning or encoding-model training, so there is no within-story evaluation to compare against. The in-distribution vs. out-of-distribution comparison differs in participants, not stories.
-
Addressed in limitations 9 above.
-
Based on your suggestion, we added the encoding results of Fig. 2 on inflated lateral and medial surfaces (Fig. 6, Appendix D). We also added a network-wise quantification of the change in encoding performance across the seven Yeo networks (Fig. 15). The largest gains are for the middle-to-late layers in the dorsal, ventral, default mode, and frontoparietal networks.
Thank you again for your feedback; the above additions make our results more clear, and we hope they also address your concerns. All these updates have been included accordingly in the revised manuscript.
Response to rebuttal
I thank the authors for the rebuttal, I addresses most of my points. I still think that a figure depicting the evaluation procedure and splits would make it clearer and will gain. While details about the data train / held-out splits is available in multiple places throughout the manuscript, it would be easier for the reader to refer to a single figure in case they need a reminder. The explanations you provided in in your response to W. 7 and 8 would strengthen the paper on the interpretability front if featured in the discussion (if the page limit allows, of course). I will increase my score to 3.
Thank you for your quick response and for raising your score.
Based on your suggestion, we've added a figure visualizing the data splits between brain-tuning and voxelwise encoding evaluation in Fig. 24 (Appendix G). We have updated the Results and Discussion sections to briefly mention this interpretation and point to the relevant appendix.
Thank you for the constructive feedback! We address each point below:
Weaknesses
We use the preprocessed data released by Deniz et. al 2019. Based on your suggestion, we added a short summary of the acquisition and preprocessing steps to Appendix Section A.
fMRI is sampled once per TR (2.0045 s), whereas model representations are extracted at the token (text) or 500-ms chunk (speech) rate. We downsample the embeddings using 3-lobe Lanczos interpolation to match the two temporal resolutions so the representations can be linearly mapped to the BOLD. We have updated the Methods section (Sec 4.1) to include this.
To clarify: for brain-tuning, each participant’s fMRI responses are mapped from native space to the common fsaverage surface, which lets us brain-tune a single model jointly across participants. For evaluation, we extract embeddings from the brain-tuned models and fit a separate VEM for each held-out participant. VEM results are therefore reported in each participant’s native space (e.g., Fig. 2), which is why they are shown per voxel.
We use Pearson correlation because it is the standard metric in the voxelwise encoding modeling literature. Based on your suggestion, we visualized the distributions and time courses of predicted and recorded responses for example voxels in Fig. 8 (Appendix D). The predicted and recorded responses have similar distributions, and the predicted time courses track the recorded responses.
In the main text, we show the change in encoding performance on the flatmaps (Fig. 2) to see where brain-tuning helps most and to compare directly with prior brain-tuning work [Moussa et al., 2025, Vattikonda et al., 2025, Negi et al., 2025, Vaidya et al., 2026]. Based on your suggestion, we added flatmaps of the absolute encoding performance of the brain-tuned model in Fig. 7 (Appendix D).
Thank you for catching this. The legend in the second column should read “brain-tuned (speech-only)” in violet, as plotted. We have fixed this in the revised manuscript.
The late-layer decline in brain encoding performance coincides with specific, measurable changes in speech representations, rather than a uniform loss of speech information. Three additional analyses support this interpretation:
Together, these findings suggest that brain-tuning increases cross-modal alignment in intermediate-to-deep speech layers, while the final layers lose some character-level transcription information but retain phonemic information. This provides a plausible explanation for the late-layer decline in brain encoding performance, rather than indicating a general deterioration of speech representations. We have incorporated these analyses and their interpretation into the revised manuscript (Appendix D.2, Figs. 16 & 17).
The gains are in regions known to represent semantics during language comprehension. The flatmaps (Fig. 2) show improvements in regions similar to those reported by Deniz et al. (2019), the dataset we use, where semantic representations are shared across reading and listening. Effects outside these regions are weaker and more mixed. To quantify this further, we’ve added quantification results across language-selective ROIs (Fig. 14 for SONAR, Fig. 20 for CLAP).
We clarify that “modality-agnostic” is used specifically in the context of text and speech, as explicitly stated in our title, “Modality-Agnostic Brain-Tuning of Text and Speech Representations.” We do not claim generalization to arbitrary sensory modalities. We now added “modality-agnostic” refers specifically to text and speech, to state this explicitly for clarity in the Introduction (line 49-50).
We thank the reviewer for this suggestion. Beyond brain encoding performance, we now directly quantify changes in representation geometry using Centered Kernel Alignment (CKA) between text and speech encoder representations across all 24 × 24 layer pairs of SONAR (Fig. 16, Appendix D.2). Joint brain-tuning increases text-speech CKA from 0.073 to 0.0796 (+9.2%, p = 0.001) (avg. across layers), concentrated in intermediate-to-deep speech layers (13-21) and text layers (14-23), where the encoding (Fig. 3) and probing (Fig. 5) gains also emerge.