- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a whole-brain fMRI decoder that aligns brain activity with CLIP image and text embeddings, using an Information Bottleneck penalty to discard redundant signal in the spirit of efficient coding. The same bottleneck principle is then applied to the frozen decoder to produce per-sample voxel saliency maps, showing which parts of the brain input drive a given decoding decision. On the Natural Scenes Dataset, the authors report competitive retrieval against a whole-brain baseline and some other baselines converted to a whole-brain version, as well as voxel deletion/insertion experiments indicating the highlighted voxels strongly influence the decoder’s alignment.
Critical strengths
- The two core themes of this paper, whole-brain decoding and attribution, both seem worth pursuing. While modern decoding models reach surprising performance, I feel they still are at the level of directly reading primary visual information from the visual cortex, rather than reading deeper thoughts or intents from the brain. For decoding technology to evolve towards user intent and deeper thoughts hidden inside the brain, I believe reading information from various brain regions would be necessary, not just from few select visual regions, and this would lead to the need for whole-brain decoding. As for attribution, the importance would be straightforward, as it can be directly linked with our existing neuroscientific understanding, and potentially be used to further enhance our understanding of brain dynamics.
Critical weaknesses
- Proposed whole-brain decoding results seeem to be substantially weaker than state-of-the-art decoding methods, which use a select few set of voxels. According to their paper, MindEye2 reaches near-perfect performance (98~99%) for the top-1 brain<->image retrieval task, when the model is trained and evaluated using the ROI-selected voxels (nsdgeneral). MindEye2, when adapted to the whole-brain framework, suffers a very significant retrieval performance drop, as the authors show in this paper. Although I believe the drop may be mitigated a bit if the model is better optimized toward the whole-brain input considering the input size is a few order of magnitudes larger and some modules and learning objectives from the original model seem to be excluded in the modified version of this paper. Even the proposed method, having brain<->image top-1 retrieval performances around 50%, is much worse than the nsdgeneral version of Mindeye2. Considering Mindeye2 is published in 2024 and much stronger brain decoding models have been proposed since then, one example being Brain-IT (Beliy, Roman, Amit Zalcher, and Jonathan Kogman. "Brain-IT: Image reconstruction from fMRI via brain-interaction transformer." International Conference on Learning Representations. Vol. 2026. 2026.), it is hard to believe the proposed whole-brain decoding model can compete with state-of-the-art decoding models that use selected voxels as their input in terms of decoding performance.
- The deletion/insertion evaluation measures voxel importance, and since the attribution objective is itself a perturbation objective through that decoder, strong performance on this metric is largely expected. The controls in Fig. 4 and Appendix D may provide some insight but they still all remain to the same proposed decoder. Given the introduction frames the contribution as identifying important voxels for decoding, I would like to see whether the voxel importance transfers to other decoders. The reproduced baselines (Ridge, MindEye, MindEye2, TGBD) already exist, so applying the same deletion/insertion protocol with the same voxel ranking would be not too hard to accomplish during the rebuttal period.
Questions for authors
- How was the "Visual ROIs" method from Section F.1 table 2 implemented? Please provide the details.
- What is the number of valid whole-brain voxels that was used for the input for models such as the adapted Mindeye used in this paper?
- A follow-up to weakness #2: are "important voxels" attributed from one decoding model also important for other decoding models? What happens if voxels attributed from one model are deleted/inserted for another model?
4 comments
Response to W2 (Q3)
W2.1. Same-Decoder Deletion/Insertion Evaluation
Using the same frozen decoder tests fidelity to that decoder's decision. The result is supported by three distinct checks:
- Different intervention: We optimize maps with soft Gaussian perturbations, then evaluate their ranking by hard voxel deletion/insertion (Fig. 4).
- Sample pairing: Correctly paired maps outperform Random and Shuffle. Shuffle keeps broad spatial preferences but uses another stimulus's ranking (Fig. 4).
- Additional controls: Paired maps still outperform Random and Shuffle when voxels are replaced with values from another real fMRI sample (Appendix E, Fig. 9). Image-only maps affect text alignment more strongly than Random and Shuffle even though text was excluded from map optimization. Text-derived maps also affect image alignment, with stronger effects for richer Qwen descriptions than for shorter COCO captions (Appendix E, Fig. 10).
The cross-decoder test requested by the reviewer is reported in W2.2.
W2.2 (Q3). Do Important Voxels Transfer to Other Decoders?
Our method explains the decisions of the decoder on which the map is computed. Cross-decoder agreement is therefore an additional test of whether the ranked voxels also affect other trained decoders, not a requirement that an attribution map must transfer. We applied the same maps from our decoder to hard deletion/insertion in four adapted decoders on the shared MNI voxel space. For comparison, we recalculated our own decoder's AUC with the same endpoint normalization. The table gives eight-subject mean AUCs. Lower deletion and higher insertion indicate a stronger effect of the ranking within each decoder:
| Evaluated decoder | Deletion image/text | Insertion image/text |
|---|---|---|
| Our decoder (map source) | 0.053 / −0.090 | 1.069 / 1.217 |
| Ridge | 0.761 / 0.713 | 1.102 / 1.140 |
| MindEye | 0.245 / 0.150 | 1.078 / 1.139 |
| MindEye2 | 0.134 / 0.125 | 1.160 / 1.133 |
| TGBD | 0.235 / 0.213 | 0.931 / 0.990 |
The results support a model-specific interpretation:
- Shared voxel influence: MindEye and MindEye2 show substantial image-alignment effects with our ranking. Their image insertion AUCs (1.078 and 1.160) are comparable to or higher than ours (1.069). Their deletion AUCs (0.245 and 0.134) are higher than ours (0.053), so the effect is not numerically identical. The agreement nevertheless suggests that voxels important for our decoder can also influence these decoders.
- Why effects differ: Ridge has the weakest image deletion response (0.761) and lower retrieval performance in Fig. 2. A weaker decoder may retain less target-relevant signal for the same voxels to explain. This is one reason we first train a bottleneck decoder with stronger target alignment before attributing its decisions. Moreover, the adapted decoders have different learned brain-feature mappings and internal representations, even though their output targets share the same CLIP space. Their AUCs therefore need not match ours.
These AUCs describe the relative deletion/insertion response within each decoder. They do not measure absolute decoder quality or establish a decoder-independent voxel ranking. Target-decoder Random/Shuffle controls would further test whether the observed cross-decoder effects exceed arbitrary rankings.
Response to the rebuttal
I thank the authors for their detailed response.
My major concern regarding the first weakness point regards the practical utility of the proposed whole-brain decoder itself. I understand the authors are claiming that they propose the best whole-brain approach among the baselines they set up, but if the proposed decoder is clearly inferior in terms of performance compared to models like MindEye2 or Brain-IT, I feel it is very difficult to advocate the use of the proposed decoder or whole-brain decoders in general at this stage. That is why I brought up the comparison against the top-1 brain-image retrieval numbers from the MindEye2 paper. From what I could tell, the retrieval protocol is the same as in this paper (300 candidates, 30 repeats, averaged test repetitions), although I acknowledge the target embedding and the set of subjects differ, as the authors pointed out. Even allowing for those differences, MindEye2 has 98.3% and 98.8% brain->image and image->brain retrieval accuracy, compared to the 57.40% and 48.68% accuracy of the proposed method, and I find this gap too large to be explained by those differences alone. I also note that the authors’ own visual-ROI variant of the same decoder reaches only around 55%, so the gap may be from the proposed decoder itself rather than from the whole-brain input not being effective for brain decoding. Either way, the paper does not show that a whole-brain decoder can approach the level of current models, and it would be difficult to convince practitioners to adopt the whole-brain input when the input format that is currently widely adopted is performing much better.
This concern about the model performance also circles back to the interpretation side. The authors themselves talked about how “a weaker decoder may retain less target-relevant signal for the same voxels to explain,” which I also agree with, and the paper’s limitation section similarly states that an inaccurate decoder may produce misleading saliency maps. If so, considering how the proposed decoder shows a much weaker decoding capability than current models, it makes it difficult to trust the interpretation results it produces.
Due to these reasons, I will maintain my original assessment of the paper.
Reply to Reviewer CKts
Thank you for your reply. We are difficult to believe that a misunderstanding of decoding performance remains your primary concern, even though this paper focuses on decision attribution. We are puzzled by the direct numerical comparison between our method and MindEye2 under different implementation protocols, especially when our paper reports a fair comparison under a consistent protocol.
We certainly recognize MindEye2’s outstanding brain-to-image and image-to-brain retrieval accuracies of 98.3% and 98.8%. However, its implementation differs substantially from ours. MindEye2 uses nsdgeneral voxels in each subject’s native space at 1.8-mm resolution and targets all 256 × 1664 visual tokens from OpenCLIP ViT-bigG/14. Its training also includes a diffusion-prior feature loss and contrastive learning through a retrieval projector. In contrast, our method uses whole-brain voxels in standard MNI 2-mm space and aligns a single 1024-dimensional brain representation with CLIP ViT-H/14 image and text embeddings, without a diffusion prior or its feature loss.
We understand that the motivation for whole-brain input may seem unclear if it does not substantially improve retrieval accuracy. Its purpose here is also to retain a broader set of voxels for decision attribution. We have added experiments in the revised manuscript that demonstrate the contribution of selected voxels outside the predefined visual ROI. We emphasize that this paper asks not only whether the decoder can retrieve the correct target, but also which voxels support its alignment decision.
In fact, in a separate paper awaiting publication, we primarily report the strong performance of this decoder on classification, retrieval, and reconstruction. Under a configuration similar to MindEye2, including a diffusion-prior feature loss and contrastive learning through a projector, it reaches brain-to-image and image-to-brain retrieval accuracies of 99.3% and 98.6%. Those results are outside the scope of the present paper.
We certainly agree that decoder performance can affect attribution quality. However, we do not see why attribution of this decoder’s decisions should require retrieval accuracy above 98%, given that its current performance is already far above the 1/300 chance level. Our attribution experiments directly test the influence of selected voxels on the frozen decoder’s alignment decisions.
Finally, we remain grateful for the effort you have devoted to reviewing our manuscript, even though we have not reached agreement on some points. Your other constructive suggestions have helped us strengthen the paper.
All the best,
Response to W1 (Q1,Q2)
We sincerely thank you for your comments and constructive suggestions. We have already responded to your question below.
W1.1. MindEye2 Performance Gap and Baseline Fairness
The original pretrained MindEye2 reports much higher brain–image retrieval R@1. The two reported scores come from different setups:
nsdgeneralvoxels, cross-subject pretraining, 256×1664 OpenCLIP visual-token targets, and its original training objectives.Our comparison asks whether the proposed whole-brain bottleneck encoder learns effective brain–image/text alignment relative to alternative encoder backbones under a common task and protocol. It is controlled among adapted encoders. The revised manuscript describes these methods as adapted encoder baselines evaluated under a shared retrieval protocol in the main text and Appendix C. We do not claim that this adapted-encoder comparison surpasses the original pretrained MindEye2 system.
W1.2 (Q1). Implementation of Visual ROIs
The “Visual ROIs” condition trains and evaluates the same decoder using only voxels within
nsdgeneral. The originalnsdgeneralROI is defined in each subject's native space and transformed into the common MNI 2-mm space used by our fMRI data. For the whole-brain condition, we instead use the MNI152 2-mm template brain mask (MNI152_T1_2mm_brain_mask). Both settings use the same voxel standardization, decoder, image/text targets, training procedure, and retrieval evaluation. The input mask is the controlled difference.W1.3 (Q2). Number of Valid Whole-Brain Voxels
The whole-brain input used in our comparison is:
The array dimensions include positions outside the brain. 228,483 is the voxel count relevant to the reviewer's question.
W1.4. Comparison with State-of-the-Art Decoders
We have added Brain-IT to Related Work as a relevant image-reconstruction method. Brain-IT evaluates image reconstruction, while our experiments evaluate bidirectional brain–image/text retrieval and attribution of those retrieval decisions. Reconstruction scores and retrieval R@1 measure different outputs, so a numerical ranking between the two systems would not answer the question studied here. Our comparative performance claim concerns the matched retrieval task, not reconstruction benchmarks.