- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper explores the interpretability of transformer block activations for encoding of fMRI responses with three input stimulus modalities: speech, text, music and vision. The authors quantify the the importance of transformer block states (internal activations) for voxel-wise encoding performance. This approach is different from previous works that predominantly focused on block outputs rather than their activations. The paper concludes that the most predictive internal state differs with respect to the input modality and stimulus type (early vs. higher-order sensory processing).
UPD Nov 14th 2026: score increased to 3.
Critical strengths
The main question of the paper is clearly stated. The novelty element is clearly present as the authors analyse block activations instead of outputs for multiple modalities. The analyses seem appropriate for the problem addressed.
Critical weaknesses
- There is no null significance testing for encoding performance. Since the main results only report the state/block ranking within models per parcel, it is hard to get an idea of how good the encoding performance is on its own or to which extent it is linked to the model pretraining conditions.
- The MRI data were preprocessed with different pipelines. For the speech vs. music comparison the authors attribute the difference in performance to the properties of model blocks. However, the heterogeneity harbored within the data can alone account for any difference in encoding performance: in LPP the encoding is reported for group average and the preprocessing for LPP and MGND is different. This is a major result validity concern on my end.
- The authors claim that block-wise analyses can guide architectural choices such that sensory processing is captured better. This is a fair statement overall. However, the authors provide no architectural recommendations for future works. It is unclear how the field should benefit from these results concretely.
- My suspicion is that some states are strongly correlated within models.
Questions for authors
- The encoding is done in the voxel space but the results are reported within HCP-MMP parcels. Why this approach? Why not do the encoding on the parcel level to begin with?
- The encoding is done within-subject (only individual runs are held out). Do the authors have an estimation of how much is lost in performance when a whole subject is left out for evaluation?
- Could the authors add a null significance test where the link between the pretrained states and the target (fMRI response) is severed? For instance, one could rerun the analyses on randomly initialized models. If computationally feasible, it would be interesting to see a null distribution from ~100 seeds (a single inference pass per seed).
- What does the blue-red colorbar stand for in winning ratio histograms?
- Could the authors provide absolute encoding accuracies? The aim here would be to have an idea how well an individual state recovers fMRI responses.
4 comments
Response #2
Q5
The encoding is done in the voxel space but the results are reported within HCP-MMP parcels. Why this approach? Why not do the encoding on the parcel level to begin with?
A5. Thank you for this question. We now realize that we should have explained this approach more clearly. Our aim is to characterize neural representations, so we fit the encoding models at the finest spatial unit available in fMRI, the voxel: voxels within a parcel can differ in response selectivity, and averaging them before fitting would discard these differences. For reporting, we then summarize the voxel-wise results within ROIs, which are established functional units of cortex. Summarizing voxel-wise encoding results within ROIs is a widely used approach (e.g., Nishimoto et al., 2011; Güçlü & van Gerven, 2015; Kell et al., 2018; Lescroart & Gallant, 2019). Cortical anatomy and functional organization vary across individuals, making direct voxel-to-voxel comparisons across participants difficult, and HCP-MMP parcels provide a common regional framework for summarizing and comparing results. We have clarified this rationale and added these references in the Discussion.
Q6
The encoding is done within-subject (only individual runs are held out). Do the authors have an estimation of how much is lost in performance when a whole subject is left out for evaluation?
A6. We agree that generalization across participants is important. Our evaluation focuses on prediction of held-out stimuli, following common practice in voxel-wise encoding studies (e.g., Nishimoto et al., 2011; Güçlü & van Gerven, 2015; Kell et al., 2018; Lescroart & Gallant, 2019). Generalization to an entirely new participant additionally concerns how stimulus–response relationships are shared across individuals and how their neural response spaces correspond. These are research questions in their own right, as illustrated by work on functional alignment (e.g., Haxby et al., 2011 and Chen et al., 2015), and extend beyond the scope of our comparison of intermediate representations. We have now explicitly acknowledged this in the Discussion.
Q7
Could the authors add a null significance test where the link between the pretrained states and the target (fMRI response) is severed? For instance, one could rerun the analyses on randomly initialized models. If computationally feasible, it would be interesting to see a null distribution from ~100 seeds (a single inference pass per seed).
A7. Thank you for this suggestion. We added null randomizations that disrupt the correspondence between fixed held-out predictions and measured responses. We showed that prediction accuracy was statistically significant for all states in all datasets and models (Appendix I.5).
We appreciate the suggestion about initialized models, which would help clarify the contribution of pretraining to brain alignment. Our main question concerns how brain alignment varies across intermediate states and conditions in pretrained models. Investigating how these patterns arise through pretraining is an interesting further question that extends beyond the focus of the present study. We have therefore highlighted the proposed comparison across multiple random initializations as a valuable direction for future work in the revised Discussion.
Q8
What does the blue-red colorbar stand for in winning ratio histograms?
A8. We now realize that the color scheme was not sufficiently explained, and we apologize for the confusion. We adopted the blue-to-red color scheme used in Mind’s Transformer for consistency with this relevant prior work. The colors identify states in their displayed within-block order; they do not encode accuracy or statistical significance. Bar length indicates the winning ratio, and solid versus hatched bars distinguish early and higher ROI groups. We have clarified these conventions in the captions and legends of Figure 2.
Q9
Could the authors provide absolute encoding accuracies? The aim here would be to have an idea how well an individual state recovers fMRI responses.
A9. Thank you for this suggestion. Appendix I.5 now reports absolute held-out Pearson r values for the most frequently selected intermediate state in each dataset, model, and principal ROI group, alongside the null significance tests described in A1. Representative r values in early sensory ROIs were 0.371 for LPP/Llama post-rotation Q (99th percentile of the null distribution: 0.095), 0.389 for MGND/MusicGen Q (0.086), and 0.251 for NSD/DeiT III RoPE-Mixed post-rotation Q (0.012). Previous encoding studies have demonstrated the predictive value of language, audio, and vision model representations on these datasets (Chen & Sivakumar, 2026; Denk et al., 2026; Conwell et al., 2024). Our results confirm that this established encoding approach captures meaningful response variation in the present setting, providing a sound basis for our comparisons of intermediate states.
Response to rebuttal
The authors addressed my concerns to an extent. I will raise my score to 3.
I suspect, in the future the number of submissions on applications to cognitive / neuroscience will increase. With that in mind, I'd like to urge the authors to pay more attention to preprocessing and acquisition disparities across datasets as they develop new methods in the future. Also, clear presentation and interpretation of the results in the context of brain science will increasingly be seen as relevant as more reviewers with a background in neuroscience will partake in review processes for major ML conferences. Gaps on these fronts are enough to spark doubt about paper soundness in the neurosci community. I am looking forward to reading the revised article.
Thank you again for your thoughtful reassessment and constructive feedback, which have helped us improve the manuscript.
We have explicitly acknowledged the limitations arising from acquisition and preprocessing differences across datasets and clarified the scope of our interpretations in the revised manuscript. We will continue to pay careful attention to these issues in our future work.
Response #1
We thank the reviewer for the careful evaluation of our paper and for the constructive feedback. We have revised the manuscript and added the analyses described below.
Q1
A1. We agree. We added null significance tests and found that prediction accuracy was statistically significant for all states in all datasets and models (Appendix I.5). As the reviewer notes, our main results concern the ranking of states; in line with this, we slightly adjusted the wording (e.g., in the Abstract) so that statements comparing intermediate states with block output are phrased in terms of this ranking.
Q2
A2. We agree. We therefore added a control in which MGND responses were averaged across participants before fitting the encoding models, bringing its response aggregation closer to LPP. The results were highly similar to those of the participant-wise analysis, and the main findings were reproduced under this control; for example, post-rotation Q/K did not become preferred in the acoustic RoPE model (Appendix I.6). Nevertheless, the observed speech–music difference may still reflect other dataset factors, such as acquisition and preprocessing, which we now note in the Discussion and the Limitations section.
Q3
A3. Thank you for this helpful comment. We now realize that we should have made the practical implications more explicit. For neuroscience, our analyses help relate computational stages within transformers to the functional properties of sensory cortex, as illustrated by the correspondence with independently measured pRFs. For applications in brain-response prediction, the results guide the choice of representations for particular cortical regions—for example, using post-rotation Q/K as features for predicting early visual responses.
For future work aimed at developing more human-like sensory representations, our results suggest possible directions for exploring model design. For example, the visual findings motivate examining how positional encoding shapes spatial representations and their correspondence with human visual cortex. We have revised the Discussion to state these implications and directions for future research explicitly.
Q4
A4. We agree. When correlated states have similar prediction accuracy, small fluctuations can affect which state ranks first. We therefore added 95% bootstrap confidence intervals to the winning ratios (Figure 2 for LPP, Figure A4 for MGND, and Figure A6 for NSD), together with statistical comparisons of winning-ratio differences across states, datasets, and ROI groups (Sections 4.1–4.3; Appendix I.2). These analyses quantify the variability of state preferences and the robustness of the reported differences. For example, the early–higher visual difference in relative Q/K-over-FFN preference was positive in all eight comparisons, ranging from 21.2 to 27.8 percentage points.