acceptodds
Withdrawn from ICLR 2027

Preserving What Matters: Efficient-Coding-Inspired Information Bottleneck for Whole-Brain Decoding and Voxel Attribution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language pretrained models have greatly advanced brain decoding, but most decoders remain opaque, offering limited evidence about which patterns in fMRI activity influence the decoding decision. We propose an interpretable whole-brain decoding framework inspired by the efficient coding principle of preserving what matters. Concretely, we encourage brain-VLPM alignment through contrastive learning with variational Information Bottleneck regularization to compress redundant ROI-free whole-brain fMRI activity. Built on the whole-brain decoder, our method further explains individual decoding decisions through bottleneck-based attribution, producing voxel-level saliency maps for brain-image/text alignment. Experiments on the Natural Scenes Dataset show strong retrieval performance and quantitatively evaluated interpretability. Through ROI localization and voxel ablation analyses, the highly ranked voxels are shown to strongly influence the trained decoder's alignment scores: removing them rapidly degrades the alignment score, whereas inserting them rapidly restores and even can exceed the full-input alignment score. These results establish our framework as a practical approach to identifying informative voxels supporting individual decoding decisions.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain Decoding, Whole Brain Model, Information Bottleneck, Voxel Attribution, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CKtsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a whole-brain fMRI decoder that aligns brain activity with CLIP image and text embeddings, using an Information Bottleneck penalty to discard redundant signal in the spirit of efficient coding. The same bottleneck principle is then applied to the frozen decoder to produce per-sample voxel saliency maps, showing which parts of the brain input drive a given decoding decision. On the Natural Scenes Dataset, the authors report competitive retrieval against a whole-brain baseline and some other baselines converted to a whole-brain version, as well as voxel deletion/insertion experiments indicating the highlighted voxels strongly influence the decoder’s alignment.

Critical strengths
  • The two core themes of this paper, whole-brain decoding and attribution, both seem worth pursuing. While modern decoding models reach surprising performance, I feel they still are at the level of directly reading primary visual information from the visual cortex, rather than reading deeper thoughts or intents from the brain. For decoding technology to evolve towards user intent and deeper thoughts hidden inside the brain, I believe reading information from various brain regions would be necessary, not just from few select visual regions, and this would lead to the need for whole-brain decoding. As for attribution, the importance would be straightforward, as it can be directly linked with our existing neuroscientific understanding, and potentially be used to further enhance our understanding of brain dynamics.
Critical weaknesses
  • Proposed whole-brain decoding results seeem to be substantially weaker than state-of-the-art decoding methods, which use a select few set of voxels. According to their paper, MindEye2 reaches near-perfect performance (98~99%) for the top-1 brain<->image retrieval task, when the model is trained and evaluated using the ROI-selected voxels (nsdgeneral). MindEye2, when adapted to the whole-brain framework, suffers a very significant retrieval performance drop, as the authors show in this paper. Although I believe the drop may be mitigated a bit if the model is better optimized toward the whole-brain input considering the input size is a few order of magnitudes larger and some modules and learning objectives from the original model seem to be excluded in the modified version of this paper. Even the proposed method, having brain<->image top-1 retrieval performances around 50%, is much worse than the nsdgeneral version of Mindeye2. Considering Mindeye2 is published in 2024 and much stronger brain decoding models have been proposed since then, one example being Brain-IT (Beliy, Roman, Amit Zalcher, and Jonathan Kogman. "Brain-IT: Image reconstruction from fMRI via brain-interaction transformer." International Conference on Learning Representations. Vol. 2026. 2026.), it is hard to believe the proposed whole-brain decoding model can compete with state-of-the-art decoding models that use selected voxels as their input in terms of decoding performance.
  • The deletion/insertion evaluation measures voxel importance, and since the attribution objective is itself a perturbation objective through that decoder, strong performance on this metric is largely expected. The controls in Fig. 4 and Appendix D may provide some insight but they still all remain to the same proposed decoder. Given the introduction frames the contribution as identifying important voxels for decoding, I would like to see whether the voxel importance transfers to other decoders. The reproduced baselines (Ridge, MindEye, MindEye2, TGBD) already exist, so applying the same deletion/insertion protocol with the same voxel ranking would be not too hard to accomplish during the rebuttal period.
Questions for authors
  • How was the "Visual ROIs" method from Section F.1 table 2 implemented? Please provide the details.
  • What is the number of valid whole-brain voxels that was used for the input for models such as the adapted Mindeye used in this paper?
  • A follow-up to weakness #2: are "important voxels" attributed from one decoding model also important for other decoding models? What happens if voxels attributed from one model are deleted/inserted for another model?
4 comments
AuthorsOpenReview ↗

Response to W1 (Q1,Q2)

We sincerely thank you for your comments and constructive suggestions. We have already responded to your question below.

W1.1. MindEye2 Performance Gap and Baseline Fairness

The original pretrained MindEye2 reports much higher brain–image retrieval R@1. The two reported scores come from different setups:

  • Original MindEye2: It uses nsdgeneral voxels, cross-subject pretraining, 256×1664 OpenCLIP visual-token targets, and its original training objectives.
  • Our comparison: The adapted MindEye2 encoder receives standardized MNI 2-mm whole-brain input and predicts one 1×1024 CLIP image–text vector under a bidirectional contrastive objective. All adapted baselines share the same input space, targets, loss, training schedule, and 300-candidate evaluation protocol (Appendix C). Original MindEye2 also uses 300 candidates, so candidate-set size does not explain the score gap.

Our comparison asks whether the proposed whole-brain bottleneck encoder learns effective brain–image/text alignment relative to alternative encoder backbones under a common task and protocol. It is controlled among adapted encoders. The revised manuscript describes these methods as adapted encoder baselines evaluated under a shared retrieval protocol in the main text and Appendix C. We do not claim that this adapted-encoder comparison surpasses the original pretrained MindEye2 system.

W1.2 (Q1). Implementation of Visual ROIs

The “Visual ROIs” condition trains and evaluates the same decoder using only voxels within nsdgeneral. The original nsdgeneral ROI is defined in each subject's native space and transformed into the common MNI 2-mm space used by our fMRI data. For the whole-brain condition, we instead use the MNI152 2-mm template brain mask (MNI152_T1_2mm_brain_mask). Both settings use the same voxel standardization, decoder, image/text targets, training procedure, and retrieval evaluation. The input mask is the controlled difference.

W1.3 (Q2). Number of Valid Whole-Brain Voxels

The whole-brain input used in our comparison is:

  • Space and volume: MNI152 2-mm space, with a 91×109×91 array.
  • Valid brain voxels: 228,483, as defined by the MNI152 2-mm brain mask.
  • Adapted MindEye input: a vector of these 228,483 valid voxels per fMRI sample.

The array dimensions include positions outside the brain. 228,483 is the voxel count relevant to the reviewer's question.

W1.4. Comparison with State-of-the-Art Decoders

We have added Brain-IT to Related Work as a relevant image-reconstruction method. Brain-IT evaluates image reconstruction, while our experiments evaluate bidirectional brain–image/text retrieval and attribution of those retrieval decisions. Reconstruction scores and retrieval R@1 measure different outputs, so a numerical ranking between the two systems would not answer the question studied here. Our comparative performance claim concerns the matched retrieval task, not reconstruction benchmarks.

AuthorsOpenReview ↗

Response to W2 (Q3)

W2.1. Same-Decoder Deletion/Insertion Evaluation

Using the same frozen decoder tests fidelity to that decoder's decision. The result is supported by three distinct checks:

  • Different intervention: We optimize maps with soft Gaussian perturbations, then evaluate their ranking by hard voxel deletion/insertion (Fig. 4).
  • Sample pairing: Correctly paired maps outperform Random and Shuffle. Shuffle keeps broad spatial preferences but uses another stimulus's ranking (Fig. 4).
  • Additional controls: Paired maps still outperform Random and Shuffle when voxels are replaced with values from another real fMRI sample (Appendix E, Fig. 9). Image-only maps affect text alignment more strongly than Random and Shuffle even though text was excluded from map optimization. Text-derived maps also affect image alignment, with stronger effects for richer Qwen descriptions than for shorter COCO captions (Appendix E, Fig. 10).

The cross-decoder test requested by the reviewer is reported in W2.2.

W2.2 (Q3). Do Important Voxels Transfer to Other Decoders?

Our method explains the decisions of the decoder on which the map is computed. Cross-decoder agreement is therefore an additional test of whether the ranked voxels also affect other trained decoders, not a requirement that an attribution map must transfer. We applied the same maps from our decoder to hard deletion/insertion in four adapted decoders on the shared MNI voxel space. For comparison, we recalculated our own decoder's AUC with the same endpoint normalization. The table gives eight-subject mean AUCs. Lower deletion and higher insertion indicate a stronger effect of the ranking within each decoder:

Evaluated decoderDeletion image/textInsertion image/text
Our decoder (map source)0.053 / −0.0901.069 / 1.217
Ridge0.761 / 0.7131.102 / 1.140
MindEye0.245 / 0.1501.078 / 1.139
MindEye20.134 / 0.1251.160 / 1.133
TGBD0.235 / 0.2130.931 / 0.990

The results support a model-specific interpretation:

  • Shared voxel influence: MindEye and MindEye2 show substantial image-alignment effects with our ranking. Their image insertion AUCs (1.078 and 1.160) are comparable to or higher than ours (1.069). Their deletion AUCs (0.245 and 0.134) are higher than ours (0.053), so the effect is not numerically identical. The agreement nevertheless suggests that voxels important for our decoder can also influence these decoders.
  • Why effects differ: Ridge has the weakest image deletion response (0.761) and lower retrieval performance in Fig. 2. A weaker decoder may retain less target-relevant signal for the same voxels to explain. This is one reason we first train a bottleneck decoder with stronger target alignment before attributing its decisions. Moreover, the adapted decoders have different learned brain-feature mappings and internal representations, even though their output targets share the same CLIP space. Their AUCs therefore need not match ours.

These AUCs describe the relative deletion/insertion response within each decoder. They do not measure absolute decoder quality or establish a decoder-independent voxel ranking. Target-decoder Random/Shuffle controls would further test whether the observed cross-decoder effects exceed arbitrary rankings.

Reviewer_CKtsOpenReview ↗

Response to the rebuttal

I thank the authors for their detailed response.

My major concern regarding the first weakness point regards the practical utility of the proposed whole-brain decoder itself. I understand the authors are claiming that they propose the best whole-brain approach among the baselines they set up, but if the proposed decoder is clearly inferior in terms of performance compared to models like MindEye2 or Brain-IT, I feel it is very difficult to advocate the use of the proposed decoder or whole-brain decoders in general at this stage. That is why I brought up the comparison against the top-1 brain-image retrieval numbers from the MindEye2 paper. From what I could tell, the retrieval protocol is the same as in this paper (300 candidates, 30 repeats, averaged test repetitions), although I acknowledge the target embedding and the set of subjects differ, as the authors pointed out. Even allowing for those differences, MindEye2 has 98.3% and 98.8% brain->image and image->brain retrieval accuracy, compared to the 57.40% and 48.68% accuracy of the proposed method, and I find this gap too large to be explained by those differences alone. I also note that the authors’ own visual-ROI variant of the same decoder reaches only around 55%, so the gap may be from the proposed decoder itself rather than from the whole-brain input not being effective for brain decoding. Either way, the paper does not show that a whole-brain decoder can approach the level of current models, and it would be difficult to convince practitioners to adopt the whole-brain input when the input format that is currently widely adopted is performing much better.

This concern about the model performance also circles back to the interpretation side. The authors themselves talked about how “a weaker decoder may retain less target-relevant signal for the same voxels to explain,” which I also agree with, and the paper’s limitation section similarly states that an inaccurate decoder may produce misleading saliency maps. If so, considering how the proposed decoder shows a much weaker decoding capability than current models, it makes it difficult to trust the interpretation results it produces.

Due to these reasons, I will maintain my original assessment of the paper.

AuthorsOpenReview ↗

Reply to Reviewer CKts

Thank you for your reply. We are difficult to believe that a misunderstanding of decoding performance remains your primary concern, even though this paper focuses on decision attribution. We are puzzled by the direct numerical comparison between our method and MindEye2 under different implementation protocols, especially when our paper reports a fair comparison under a consistent protocol.

We certainly recognize MindEye2’s outstanding brain-to-image and image-to-brain retrieval accuracies of 98.3% and 98.8%. However, its implementation differs substantially from ours. MindEye2 uses nsdgeneral voxels in each subject’s native space at 1.8-mm resolution and targets all 256 × 1664 visual tokens from OpenCLIP ViT-bigG/14. Its training also includes a diffusion-prior feature loss and contrastive learning through a retrieval projector. In contrast, our method uses whole-brain voxels in standard MNI 2-mm space and aligns a single 1024-dimensional brain representation with CLIP ViT-H/14 image and text embeddings, without a diffusion prior or its feature loss.

We understand that the motivation for whole-brain input may seem unclear if it does not substantially improve retrieval accuracy. Its purpose here is also to retain a broader set of voxels for decision attribution. We have added experiments in the revised manuscript that demonstrate the contribution of selected voxels outside the predefined visual ROI. We emphasize that this paper asks not only whether the decoder can retrieve the correct target, but also which voxels support its alignment decision.

In fact, in a separate paper awaiting publication, we primarily report the strong performance of this decoder on classification, retrieval, and reconstruction. Under a configuration similar to MindEye2, including a diffusion-prior feature loss and contrastive learning through a projector, it reaches brain-to-image and image-to-brain retrieval accuracies of 99.3% and 98.6%. Those results are outside the scope of the present paper.

We certainly agree that decoder performance can affect attribution quality. However, we do not see why attribution of this decoder’s decisions should require retrieval accuracy above 98%, given that its current performance is already far above the 1/300 chance level. Our attribution experiments directly test the influence of selected voxels on the frozen decoder’s alignment decisions.

Finally, we remain grateful for the effort you have devoted to reviewing our manuscript, even though we have not reached agreement on some points. Your other constructive suggestions have helped us strengthen the paper.

All the best,

Reviewer_BtizOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a whole-brain fMRI decoding model that unifies brain representation learning and voxel attribution via the information bottleneck, enabling assessment of each voxel’s contribution to the decoding task. First, the paper extracts whole-brain fMRI features using a 3D Transformer. Combined with variational information bottleneck regularization and contrastive learning, the brain representations are aligned to the CLIP image and text spaces. The decoder is then frozen, and learnable soft perturbation masks are optimized independently for each sample. This compresses input information while preserving target semantic alignment, and saliency maps are generated based on voxel-wise KL divergence. Experiments on the NSD dataset demonstrate that removing and restoring important voxels substantially affects decoding outcomes. Nevertheless, the method mainly integrates existing techniques. Its implementation suffers from coarse resolution, and experiments only evaluate the relatively simple retrieval task. Therefore, it cannot adequately verify that the selected voxels possess independent functional roles.

Critical strengths
  • Attributing voxel functions based on end-to-end downstream task training is a promising idea.
Critical weaknesses
  • The voxel selection strategy constructed in this paper is essentially a soft perturbation. It mixes random Gaussian noise of varying magnitudes into each voxel, instead of performing decoding on a hard-selected subset of voxels. While this is a viable computational approach, it sacrifices much interpretability from a neuroscience perspective.

  • Although the authors claim that these are voxel-wise masks, in practical implementation, a mask resolution of only 7×8×7 = 392 is adopted for the whole brain, in contrast to hundreds of thousands of voxels across the full brain. The authors apply trilinear interpolation to bridge this substantial resolution gap. However, this interpolation induces strong correlations in mask modeling across distinct voxels, further undermining the interpretability of the method.

  • The attribution calculation proposed in this paper is performed on a per-sample basis. The authors compute a mask for each sample on the test set. However, voxel functions are generally assumed to be fixed. Therefore, a more suitable approach would be to derive an average mask via attribution calculation on the training set and subsequently validate it on the test set.

  • The paper’s experiments focus solely on the retrieval task and omit the reconstruction task that is commonly used across the field.

Questions for authors
  • The retrieval accuracy of the original MindEye/MindEye2 is substantially higher than the values reported in this paper. Did the authors deliberately degrade the performance of this baseline during training? Alternatively, could the performance drop stem from the different loss functions introduced in this work?
  • To ensure a fair comparison, the authors should additionally provide decoding results of voxel selection on the pre-trained MindEye2 using only the nsdgeneral brain region.
  • How large is the variability among masks computed from different samples?
  • Why are 10 perturbations applied to each fMRI sample during attribution computation, and how was this hyperparameter of 10 determined?
2 comments
AuthorsOpenReview ↗

Response to W1 & W2 & W3 (Q3) & W4

We sincerely thank you for your comments and constructive suggestions. We have already responded to your question below.

W1. Soft Perturbation vs. Hard-Selected Voxels

The soft Gaussian gate estimates a voxel ranking. We evaluate that ranking using hard interventions:

  • Deletion: We set the highest-ranked voxels to zero and measure the frozen decoder after each step.
  • Insertion: We restore those voxels to a zero baseline and measure the same decoder.

Deleting paired-map voxels reduces image and text alignment faster, and inserting them restores alignment faster, than Random or across-stimulus Shuffle (Fig. 4). Replacing selected voxels with values from another real fMRI sample preserves the same ordering (Appendix E, Fig. 9). Thus, the continuous gate is an optimization device, while hard interventions test whether the ranked locations influence this decoder's decision.

W2. Correlations Induced by Mask Interpolation

Trilinear interpolation does couple nearby voxel scores. The revised manuscript now makes these resolution limits explicit:

  • Resolution: The gate has 7×8×7 learned parameters, matching the decoder's patch grid. Padding the 91×109×91 volume to 98×112×98 and dividing by 14×14×14 yields this grid. Each valid voxel receives a score, but the scores are not optimized independently.
  • Computational tradeoff: A finer gate requires more optimization variables and memory for repeated perturbations. A full resolution sweep exceeds our available resources.
  • What is tested: Paired maps cause faster alignment degradation and recovery than Random and Shuffle under hard deletion/insertion (Fig. 4). This ordering persists when deleted voxels are replaced with another real fMRI sample (Appendix E, Fig. 9). These tests do not establish resolution invariance or independent voxel-level neural selectivity.

W3.1 (Q3). Variability Among Sample-Specific Masks

Stable functional organization does not require the same voxel ranking for every decoding decision. To quantify the variation, we sampled 1,000 pairs of different test stimuli per subject from the saved maps of all eight subjects (7,674 maps in total). For each pair, we computed Pearson spatial correlation over the 228,483 valid brain voxels and the fraction of the top-ranked 10% voxels shared by both maps. The table reports the mean of each subject's pairwise result, followed by the SD across subjects:

Between-stimulus measureEight subjects, mean ± SD
Spatial Pearson correlation0.276 ± 0.072
Top-10% voxel overlap26.5% ± 4.5 percentage points

The top-10% overlap exceeds the 10% expected for independent random selections, indicating shared spatial structure, while the moderate correlation and overlap show substantial variation across stimuli. Variation alone cannot distinguish stimulus information from measurement noise. The existing Shuffle control addresses whether the pairing matters for this decoder: assigning another stimulus's map to the evaluated sample weakens deletion/insertion effects relative to the paired map (Fig. 4). For a direct comparison, Appendix D, Table 4 now reports matched single-trial Jaccard overlaps for same and different stimuli.

W3.2. Training-Set Average Mask

A training-set average mask identifies voxels generally useful across stimuli. It is a population-level ROI or input-selection tool, not an explanation of which voxels drove the decoder's decision for a particular held-out fMRI–image/text pair. TROI [1], for example, learns a sparse, task-level ROI to select decoder inputs. It addresses voxel selection for decoding, whereas our attribution identifies the evidence for each individual alignment decision. Stable functional organization does not require identical decision-relevant rankings across stimuli. Fig. 4 tests this distinction directly: Shuffle preserves another map's broad visual preference but breaks its pairing with the stimulus, producing weaker image/text deletion and insertion effects than the correctly paired map. Replacing our per-sample maps with a training average would change the research question from decision attribution to global ROI selection.

W4. Evaluation on Reconstruction Tasks

Our evaluated task is bidirectional brain–image and brain–text retrieval and attribution of the corresponding alignment decisions. Image reconstruction would additionally require a generative decoder and reconstruction-specific training and metrics. The absence of that experiment therefore limits the scope of our claims to retrieval and decision attribution. It does not change the four retrieval outcomes or the hard voxel deletion/insertion results reported here. We have make this task boundary explicit in the revised paper.

AuthorsOpenReview ↗

Response to Q1 & Q2 & Q4

Q1.1. Gap to Original MindEye/MindEye2 Performance

No baseline was deliberately degraded. Appendix C defines a controlled adaptation of brain encoders: Ridge, MindEye, MindEye2, and TGBD use the same NSD split, standardized whole-brain inputs, frozen CLIP ViT-H/14 and Qwen targets, training schedule, and 300-candidate retrieval evaluation repeated 30 times. The target representations differ from those of the original systems:

SystemBrain-alignment target
Original MindEye257×768 CLIP ViT-L/14 visual tokens
Original MindEye2256×1664 OpenCLIP ViT-bigG/14 visual tokens
All adapted encoders in our comparisonOne 1×1024 CLIP ViT-H/14 image–text projection vector

The original systems also use selected visual voxels, and MindEye2 includes cross-subject pretraining. Their published scores should therefore not be read as the expected scores of the adapted encoders on our brain–image/text task. The original studies also use 300 retrieval candidates, so candidate-set size does not explain the gap. We have not isolated the numerical effect of any single adaptation. The revised manuscript clarifies that the reported comparison is between adapted encoders under a matched protocol.

Q1.2. Effect of Different Loss Functions

Yes, the loss functions changed when we adapted every encoder to the same retrieval task:

  • Original systems: MindEye/MindEye2 align to dense visual tokens and include objectives such as a diffusion prior and reconstruction losses. MindEye2 derives captions from predicted visual features through a frozen GIT model. It does not directly align brain features with both CLIP image and text targets.
  • Our benchmark: Each adapted encoder outputs one 1×1024 CLIP projection vector and uses equally weighted, bidirectional brain–image and brain–text contrastive losses, with the same 4,096-sample bank and 150-epoch schedule (Appendix C).

This is a matched comparison of adapted encoders. The separate numerical effect of changing the loss has not been isolated by an ablation.

Q2. Voxel Selection on Pretrained MindEye2

Applying our attribution procedure to the original pretrained MindEye2 would require optimizing a new mask against that frozen model, over its nsdgeneral voxel input. The preservation objective and deletion/insertion test would also need to be defined for MindEye2's visual-token predictions, rather than our brain–image/text alignment targets. This requires reproducing the original model's voxel preprocessing and inference pipeline and constitutes a separate attribution study of a different decoder. We therefore limit this paper's attribution claim to the decoder and decisions evaluated here. Testing original pretrained MindEye2 is beyond its scope.

Q4. Why Ten Perturbations per Sample?

Appendix C uses 10 for two different purposes:

  • Independent perturbations: We replicate the same fMRI trial 10 times and draw independent Gaussian noise for each copy. All copies share one learnable mask. Averaging their noisy alignment losses makes each mask update a Monte Carlo estimate of expected alignment, with standard error decreasing approximately as 1/√10 for a fixed mask. The final voxel-wise KL capacity is computed analytically from the optimized mask, so the copies stabilize mask optimization rather than the KL formula itself. Ten is a variance–compute choice, not an empirically established optimum.
  • Optimization steps: We update the gate for 10 Adam steps. Appendix I.4 compares 5, 10, 20, 50, and 100 steps on Subject 01. ROI localization and image/text deletion/insertion change modestly across this range (Figs. 18, 19(c), and 22), although some metrics improve beyond 10 steps. Thus, 10 steps are a practical budget rather than a claim of complete convergence.

The appendix varies optimization steps and noise variance, but does not separately sweep the number of independent perturbations. Appendix C now distinguishes the ten independent perturbation copies from the ten mask-optimization steps.

[1] Ziyu Wang et. al. TROI: Cross-Subject Pretraining with Sparse Voxel Selection for Enhanced fMRI Visual Decoding. ICASSP 2025.

Reviewer_BCHuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an interpretable whole-brain fMRI decoding framework inspired by efficient coding and the Information Bottleneck principle. The method has two components. First, a 3D Transformer maps whole-brain fMRI activity into CLIP's representation space, using contrastive learning for brain-image/text alignment and variational IB regularization to suppress irrelevant information. Second, a perturbation-based attribution method optimizes a sample-specific preservation mask through the frozen decoder to identify voxels that support alignment with a paired image and text representation. Experiments on the Natural Scenes Dataset (NSD), involving eight subjects, show improvements in bidirectional image/text retrieval over adapted baseline architectures. The attribution method is evaluated through visual ROI localization and voxel manipulation experiments. The authors also investigate decoder regularization, attribution hyperparameters, replacement baselines, and cross-modal attribution transfer.

Critical strengths

The method substantially outperforms the adapted retrieval baselines across all four retrieval directions. The authors also compare IB regularization against unregularized and L1-regularized variants of the same decoder, providing evidence that the bottleneck contributes to retrieval performance.

I appreciate the effort to quantitatively evaluate the attribution results rather than relying only on saliency visualizations. The paper compares several attribution methods and includes random and shuffled voxel rankings. Appendix D further examines alternative replacement baselines and cross-modal transfer. These experiments provide useful evidence that the selected voxels are important for the trained decoder's predictions

Critical weaknesses
  • I think the main contribution of the Information Bottleneck is not sufficiently isolated. The framework combines several existing techniques, including contrastive brain-VLPM alignment, variational IB regularization, and perturbation-based IB attribution (Schulz et al., 2020; Wang et al., 2023). While applying these methods to whole-brain fMRI is useful, I feel the methodological novelty is somewhat limited.

    More importantly, Table 1 shows that IB improves Brain-Image R@1 from 56.24% to 57.40%, which is relatively small compared with the improvement over the other decoding baselines. This makes me wonder whether most of the performance gain comes from the decoder architecture or training setup rather than IB itself. Similarly, the paper does not show whether the proposed IB attribution method is better than a simpler mask-optimization approach with conventional sparsity regularization.

  • I am not fully convinced by the evaluation of attribution quality. The attribution masks are optimized to preserve alignment with the paired image and text embeddings, and the main deletion/insertion evaluations measure changes in these same alignment scores. While this is a reasonable way to evaluate the decoder's dependence on selected voxels, it does not necessarily establish that these voxels carry meaningful stimulus-specific neural information.

    The cross-modal experiments in Appendix D provide some additional evidence. However, image and text representations are correlated and share CLIP's embedding space, so I do not think cross-modal transfer offers fully independent validation. Similarly, agreement with the nsdgeneral visual ROI shows that the attribution method identifies broadly visual regions, but tells us relatively little about the accuracy of individual stimulus-specific explanations

  • The benefit of whole-brain decoding is not sufficiently demonstrated. The whole-brain versus visual-ROI comparison is provided only for Subject 01 (Table 2). Although whole-brain decoding performs better, it is unclear whether this improvement is consistent across subjects.

    I also find it interesting that the attribution maps strongly overlap with the visual ROI. This is expected for visual decoding, but it raises the question of how much additional information the model actually extracts from nonvisual regions. I think a more direct comparison between visual-only and whole-brain inputs, or an analysis of the contributions of different brain regions, would make the whole-brain motivation more convincing.

  • The spatial resolution of the attribution maps also needs clarification. According to Appendix C, the masks are learned on a relatively coarse (7 \times 8 \times 7) grid and then interpolated to the original fMRI resolution. Although the final maps assign scores to individual voxels, the model does not independently optimize each voxel. I wonder how much the results depend on this spatial smoothing and whether similar findings would hold with finer-resolution masks.

  • Several relevant studies have investigated interpretability and information bottlenecks in brain decoding, which I think the paper should cite:

    • Towards Interpretable Visual Decoding with Attention to Brain Representations (ICLR 2026).
    • Bridging Brains and Concepts: Interpretable Visual Decoding from fMRI with Semantic Bottlenecks (NeurIPS 2025).
    • BrainBits: How Much of the Brain Are Generative Reconstruction Methods Using? (NeurIPS 2024).
Questions for authors

Q1. What is the specific advantage of IB-based attribution over simpler mask optimization? Could the authors compare their approach with L1, L0, or other conventional sparsity penalties while keeping the decoder, mask resolution, and optimization budget fixed? I think this would help clarify whether IB itself is responsible for the attribution improvements.

Q2. The attribution mask is learned at a relatively coarse (7 \times 8 \times 7) resolution. How sensitive are the results to mask resolution? Also, have the authors considered evaluating attribution consistency across independent fMRI repetitions of the same stimulus? This could provide additional evidence that the maps capture reliable stimulus-specific information rather than measurement noise.

Q3. How much of the decoding improvement comes from IB versus using whole-brain inputs? The improvement from IB within the same architecture is relatively small compared with the difference from other baseline models. Could the authors provide a more systematic comparison of visual-ROI versus whole-brain decoding, with and without IB, across all subjects?

Q4. BrainBits (NeurIPS 2024) uses a bottleneck analysis to show that strong visual reconstruction performance can be achieved using surprisingly limited information extracted from brain recordings. This raises an interesting question about the relationship between decoding performance and the amount of neural information actually used. Since the proposed method also aims to preserve task-relevant information while removing redundancy, how do the authors distinguish effective compression of brain information from simply learning a compact representation sufficient for alignment with CLIP? Could a similar analysis of performance versus bottleneck capacity help clarify how much information the decoder actually uses?

13 comments
AuthorsOpenReview ↗

Response to W1 (Q1) & W2 (Q2.2)

We sincerely thank you for your comments and constructive suggestions. We have already responded to your question below.

W1.1. Contribution of IB vs. Decoder Architecture

We isolate the contribution of IB in two comparisons:

  • Within the same whole-brain decoder: Table 1 holds the architecture and retrieval task fixed. IB improves all four retrieval directions while sharply reducing the estimated information bound.
  • Across input regions: The new eight-subject comparison in W3.1 shows that IB improves all four whole-brain mean scores, while the visual-ROI means decline slightly. This is consistent with IB prioritizing task-relevant information in the broader whole-brain input.

These results identify IB's incremental contribution within our decoder. Schulz et al. (2020) and Wang et al. (2023) motivate the attribution objective. Our extension applies the bottleneck to whole-brain fMRI, preserves both paired image and text alignment through a frozen decoder, and evaluates decision-specific voxel rankings with matched controls. We will clarify this distinction from the prior methods.

W1.2 (Q1). IB Attribution vs. Sparsity Penalties

We thank the reviewer for suggesting this direct comparison. We evaluated IB/KL attribution against L1- and hard-concrete L0-regularized mask optimization using the same frozen decoder, 7×8×7 mask resolution, Gaussian perturbation, 10 optimization steps, and deletion/insertion protocol. Mean AUCs across eight NSD subjects are:

Mask penaltyDeletion, image ↓Deletion, text ↓Insertion, image ↑Insertion, text ↑
IB/KL0.053−0.0901.0691.217
L10.2580.0650.9391.153
L00.2950.1110.9191.127

IB outperforms both sparsity baselines in all four comparisons, supporting the value of the information-bottleneck objective for identifying voxels that influence the decoder's alignment decisions.

W2.1. Attribution Quality Beyond Alignment Scores

Deletion/insertion through the same decoder tests decision fidelity. The evidence does not rely solely on reusing the mask-optimization objective:

  • Different intervention: Attribution optimizes a continuous gate under Gaussian perturbations. Fig. 4 evaluates its ranking by hard voxel deletion/insertion.
  • Controls: Deleting paired-map voxels reduces image and text alignment faster, and inserting them restores alignment faster, than Random or across-stimulus Shuffle (Fig. 4). The pattern persists when deleted voxels are replaced with values from another real fMRI sample (Appendix D, Fig. 9).

These results show that selected voxels influence this decoder's decision for the stimulus. Neural reliability across independent measurements is a separate question.

W2.2. Independence of Cross-Modal Validation

The cross-modal experiment provides a specific, limited test:

  • What it tests: Appendix D evaluates image-only maps on text alignment, and vice versa (Fig. 10). Image-derived rankings affect text alignment more strongly than Random or Shuffle without using text during map optimization. Text-to-image transfer varies with the caption target, with richer Qwen descriptions performing better than shorter COCO captions.
  • What it establishes: The CLIP image and text targets share a representation space. This is cross-target evidence within one decoder, not independent neural ground truth.

W2.3. Accuracy of Stimulus-Specific Explanations

Fig. 3 shows that our maps localize the visual ROI better than the compared methods, but ROI overlap alone cannot establish stimulus specificity. In Fig. 4, Shuffle assigns another stimulus's map to the current fMRI sample, preserving broad spatial preference while breaking the pairing. Paired maps cause faster alignment loss on deletion and recovery on insertion in all four image/text comparisons. Thus, the rankings capture stimulus-specific decoder dependence. W2.4 tests consistency across independent trials.

W2.4 (Q2.2). Consistency Across fMRI Repetitions

We generated 21,118 maps, one per test trial, across eight subjects. For 7,186 stimuli with at least two trials, we measured top-voxel Jaccard overlap between independent repeats of the same image. Single-trial stimuli were excluded only from evaluation. We averaged trial pairs per stimulus and stimuli per subject, then report mean ± SD across subjects. Chance is the expected overlap of two equal-size voxel sets drawn independently and uniformly from 228,483 valid voxels:

Ranked voxelsChance JaccardRepeated-trial Jaccard
Top 1%0.5%3.5%±0.8
Top 10%5.3%13.8%±2.4

For both thresholds, the repeated-trial overlap exceeds chance in every subject. The strongest-ranked 1% of voxels overlaps at seven times chance, and the top 10% overlaps at 2.6 times chance. This consistency across independent fMRI measurements supports the repeatability of the high-attribution voxel subsets identified by our method.

AuthorsOpenReview ↗

Response to W3 (Q3) & W4 (Q2.1) & W5 & Q4

W3.1 (Q3). Whole-Brain vs. Visual ROI, With and Without IB

We thank the reviewer for requesting a systematic comparison. We evaluated the same decoder with visual-ROI or whole-brain input, each with and without IB. The table reports mean R@1 across all eight NSD subjects under the same retrieval task. Here, √ denotes IB and × denotes no IB.

RegionIBBrain–image R@1 (%)Image–brain R@1 (%)Brain–text R@1 (%)Text–brain R@1 (%)
Visual×54.8642.9140.2933.72
Visual√53.9542.1740.1032.87
Whole-brain×56.2445.3042.6636.26
Whole-brain√57.4048.6844.6038.24

The comparison supports two conclusions:

  • Whole-brain input helps: It outperforms the visual ROI in all four mean retrieval metrics with and without IB. This extends the Subject 01 result in Table 2, where whole-brain input also performs better in all four directions.
  • IB depends on input region: It improves all four whole-brain means but slightly lowers all four visual-ROI means. This is consistent with compression suppressing distracting or redundant information from whole-brain input. The preselected visual ROI likely has less such information, so additional compression may remove useful signal.

These are eight-subject averages, not a claim of improvement for every subject or proof that every visual-ROI voxel is task-relevant.

W3.2. Information from Nonvisual Regions

The eight-subject comparison in W3.1 shows a whole-brain retrieval advantage under both IB settings. We also analyzed 7,674 saved attribution maps from eight subjects. For each map, we measured:

  • Top-10% voxel fraction: The share of its highest-ranked 10% of valid voxels lying outside nsdgeneral.
  • Saliency-score fraction: The sum of nonnegative scores outside nsdgeneral divided by the sum across all valid voxels. This measures the share of the map's total score assigned to nonvisual locations.

We averaged each measure within subject. The table gives the mean ± SD across subjects:

Attribution outside nsdgeneralEight subjects, mean ± SD
Fraction of top-10% voxels64.0%±4.8
Fraction of total saliency score72.9%±4.0

88.5% of valid whole-brain voxels lie outside this ROI. The highest-ranked voxels are enriched in the visual ROI relative to its size, yet many lie outside it. Fig. 4 shows through hard deletion/insertion that the ranked voxels influence the frozen decoder's alignment. The new matched input comparison shows that including voxels outside the visual ROI improves mean retrieval in all four directions, both with and without IB. Together, these results support a contribution from nonvisual input to decoding. The deletion/insertion curves do not assign a separate effect size to each nonvisual region.

W4 (Q2.1). Sensitivity to Mask Resolution

The 7×8×7 gate matches the decoder's patch layout. Padding the 91×109×91 fMRI volume to 98×112×98 and dividing it into 14×14×14 voxel patches yields that grid.

  • Tradeoff: A finer gate could improve localization, but would increase optimization variables, memory use, and the cost of repeated perturbations. A full finer-grid sweep exceeds our available computational resources.
  • Interpretation: Interpolation gives each voxel a score, but neighboring scores are coupled. The hard deletion/insertion experiments evaluate that ranking at voxel locations. They do not establish resolution invariance or independently optimized voxel scores.

We will state the grid-to-patch correspondence and this limitation explicitly.

W5. Prior Work on Interpretable Brain Decoding

We will cite Feng et al. [1] on brain-attention interpretation, Cammarota et al. [2] on semantic bottlenecks, and BrainBits [3] on reconstruction bottlenecks. Our method attributes individual brain–image/text retrieval decisions to whole-brain voxels and tests the rankings through paired versus shuffled deletion/insertion. We will clarify this distinction in Related Work.

[1] Feng et al. Towards Interpretable Visual Decoding with Attention to Brain Representations. ICLR 2026.

[2] Cammarota et al. Bridging Brains and Concepts: Interpretable Visual Decoding from fMRI with Semantic Bottlenecks. NeurIPS 2025.

[3] Mayo et al. BrainBits: How Much of the Brain are Generative Reconstruction Methods Using? NeurIPS 2024.

Q4. Bottleneck Capacity vs. CLIP Alignment

Table 1 varies bottleneck strength while fixing the decoder and CLIP target. At α=10⁻⁵, the variational information bound falls 745.31→57.56, while brain–image R@1 rises 56.24→57.40% and the other three retrieval directions improve. Stronger compression (α 10⁻⁴) reduces the bound to 6.02 but lowers brain–image R@1 to 45.77%. This tradeoff identifies a task-dependent capacity optimum despite the fixed CLIP output dimension. The bound estimates dependence on fMRI input, not literal neural bits. BrainBits studies reconstruction, whereas our result concerns brain–image/text alignment.

Reviewer_BCHuOpenReview ↗

Reply to authors

Thank you for the rebuttal and additional experiments. I still have a few reservations that would like you to clarify.

1. Attribution reliability and stimulus specificity.

You report a top-10% Jaccard overlap of 13.8% for repeated presentations of the same stimulus. However, in the response to Reviewer Btiz, you also report 26.5% top-10% overlap between maps from different stimuli. After converting these two measures to a common definition of overlap, the same-stimulus overlap is approximately 24.3%, slightly below the reported between-stimulus overlap of 26.5%. I understand that these analyses may use different samples and trial-averaging procedures, so they are not necessarily directly comparable. I think a matched within-stimulus versus between-stimulus comparison using the same single-trial maps would be much more convincing than comparison against uniformly random voxel sets.

2. Whole-brain decoding and nonvisual information.

The new eight-subject experiments are useful and demonstrate an average retrieval benefit from whole-brain input. However, I do not think the attribution statistics establish how much nonvisual regions independently contribute. You report that 64% of the top-ranked voxels lie outside the visual ROI, but these regions also account for 88.5% of the valid brain voxels. In fact, the highest-ranked voxels are substantially enriched within the visual ROI. A direct ablation of nonvisual regions would provide stronger evidence for the scientific motivation behind whole-brain decoding.

3. Relationship to BrainBits and the interpretation of information compression.

I am still not convinced by the distinction that "BrainBits studies reconstruction, whereas our result concerns brain–image/text alignment." While the final tasks differ, both are brain decoding approaches, and BrainBits explicitly investigates mappings from fMRI to CLIP visual and text representations as part of its reconstruction pipeline. My original question was not about whether you should perform image reconstruction. Rather, I wanted to understand whether your method provides evidence about how much useful information is extracted from brain activity, beyond improving alignment with pretrained CLIP representations.

The new KL-versus-retrieval analysis demonstrates a useful regularization trade-off, but I do not think it fully answers this question. A lower variational information bound does not necessarily indicate how much meaningful neural information is being used. I would be interested in a more direct capacity-performance analysis or an explanation of why the BrainBits perspective is not applicable here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.