- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates which brain regions make critical contributions in fMRI-to-image decoding/reconstruction tasks, and whether certain brain regions can be removed while preserving decoding accuracy. To this end, we construct an fMRI representation model equipped with token selection capability, and optimize both the representation model and the token selector via end-to-end supervised fMRI-image alignment. Experiments are conducted on the NSD dataset. The main result demonstrates that comparable decoding performance can be achieved using only approximately 40% of visual brain regions.
Critical strengths
- The authors have included a detailed appendix to offer thorough explanations
Critical weaknesses
-
The present study adopts a relatively coarse granularity, performing exploration and ablation only at the brain-region level. The experimental results seem to suggest that all brain regions contribute usefully, and there is no absolute redundancy. This conclusion offers little insightful inspiration.
-
As stated by the authors, the selected brain regions vary substantially across different samples. Accordingly, a unified setting of 32 appears unreasonable, and the paper does not analyze the variability in brain-region selection among samples.
-
This paper emphasizes dynamic selection. However, if decoding is performed using the top 32 most relevant brain regions selected based on neuroscientific priors, would comparable performance be attained? If so, the additional training of a model to carry out dynamic selection would lose its merit.
Questions for authors
- Could the dynamics of the scorer’s output scores across the second-stage training be visualized?
- Why did the authors choose DINOv3 features as supervision for training SANE? If CLIP features were adopted, they could be directly interfaced with the image reconstruction model without training an adaptor module.
- Is the adaptor trained solely on images from the NSD dataset? Is this amount of data sufficient for training a representation transformation model that maps DINO features to CLIP features?
- Considering the existence of closely related prior works such as NeuroAdapter, I believe the authors should discuss the distinctions between this work and them within the main manuscript.
8 comments
W3
a. Independent neuroscience prior
There is no established neuroscience prior that identifies the optimal 32 HCP-MMP1 parcels for natural-image decoding. The Top-32 list shown in our appendix is an empirical result of this study. It ranks parcels by how often SANE’s learned Dynamic selector chooses them on the training set and defines our Static-32 baseline.
To test the reviewer’s proposal directly, we ranked parcels by response strength in independent visual localizer data and fixed the top 32 as Prior-32. We evaluated both methods with the same frozen decoder and parcel budget on the same 1,000 held-out images across four NSD subjects.
| Method | DINOv3 cosine ↑ |
|---|---|
| Independent Neuroscience Prior-32 | 0.349 |
| SANE Dynamic-32 | 0.395 |
The tested neuroscience prior yields lower decoding performance than Dynamic-32 under the same frozen decoder. The adjusted paired confidence interval for Prior-32 minus Dynamic-32 is [−0.05092, −0.04072].
b. Static and dynamic selection
Dynamic and Static selection are neither independent nor inherently competing approaches. The value of Dynamic selection is not a universal improvement in mean decoding accuracy at , but its ability to identify reproducible, stimulus-specific neural evidence while maintaining competitive predictive sufficiency. Its predictive advantage becomes more pronounced under tighter evidence budgets.(Appendix D.2.5)
Prediction consistency
| Metric | Dynamic | Static |
|---|---|---|
| DINO Cosine ↑ | 0.395 | 0.398 |
| Image-wise retention variance ↓ | 162.90 | 191.35 |
Dynamic reduces retention variance by 14.87% (95% CI for the difference: [-53.05, -1.05]).
Stimulus specificity and functional importance
| Analysis | Dynamic Advantage | 95% CI / Significance |
|---|---|---|
| Stimulus-matched vs. mismatched masks | +0.008 | — |
| Within-category matched vs. mismatched masks | +0.00511 | [0.00341, 0.00680] |
| Same- vs. different-stimulus selection () | +0.0373 | Adjusted |
| Dynamic vs. Static removal impact | +0.00497 | [0.00143, 0.00862] |
Limited-budget advantages
| Dynamic − Static (DINO Cosine ↑) | |
|---|---|
| 4 | +0.068 |
| 8 | +0.038 |
(d) Learned selection baselines ()
| Method | DINO Cosine ↑ | Removal Impact ↑ |
|---|---|---|
| SANE | 0.39499 | 0.14280 |
| Gumbel-TopK | 0.38964 | 0.16991 |
| Straight-Through TopK | 0.38917 | 0.07781 |
These results demonstrate that Dynamic selection contributes more consistent stimulus-wise retention, reproducible and functionally relevant stimulus-specific evidence, and greater benefits under tighter budgets. SANE also achieves the highest predictive accuracy among the evaluated learned selectors, without claiming superiority on every metric.
The corresponding analyses are reported in Appendix D.2.1–D.2.4.
Q1
We have added visualizations of raw parcel evidence scores and their relative rankings throughout SANE's second-stage training, covering all available parcels across four subjects (Appendix D.2.6, Figure 15 and Table 16).
| Metric | Early (Epochs 0–5) | Late (Epochs 15–20) |
|---|---|---|
| Mean absolute rank change ↓ | 18.63–30.44 | 0.49–1.04 |
| Spearman rank correlation ↑ | — | 0.9975–0.9994 |
These results demonstrate that SANE progressively learns stable anatomical preferences rather than maintaining fluctuating parcel rankings. Importantly, stable aggregate rankings do not imply fixed selection: the scorer continues to select stimulus-specific Top- parcels for each fMRI response.
Q2
We chose DINOv3 as the supervision target because of its image-only self-supervised learning objective and strong visual representation capabilities.
(a) DINOv3 vs. CLIP supervision
To examine whether CLIP provides a better supervision target, we compare both under identical training settings using the same frozen MindEye2 reconstruction pipeline on 50 prespecified test images from Subject 01.
| Metric | Evaluation Aspect | DINOv3 | CLIP | Difference (95% CI) |
|---|---|---|---|---|
| PixCorr ↑ | Pixel-level correspondence | 0.10180 | 0.09677 | +0.00502 [-0.00697, +0.01674] |
| SSIM ↑ | Structural similarity | 0.24132 | 0.24364 | -0.00231 [-0.00825, +0.00334] |
| Alex(2) ↑ | Shallow visual features | 78.367% | 78.612% | -0.245 pp [-2.694, +2.122] |
| Alex(5) ↑ | Deeper visual features | 92.531% | 91.265% | +1.265 pp [-0.408, +2.980] |
| Inception ↑ | High-level features | 83.633% | 85.714% | -2.082 pp [-6.736, +2.245] |
| CLIP ↑ | Semantic identification | 74.041% | 72.571% | +1.469 pp [-1.673, +4.939] |
The additional comparison provides no evidence that switching to CLIP would systematically improve reconstruction fidelity. Our choice of DINOv3 is motivated by the objective of generator-independent visual evidence learning.
(b) Why CLIP supervision does not eliminate adapters
The key distinction is supervision space versus downstream conditioning interface. As described in Appendix E.1 (Eq. 34), our adapter maps SANE's contextualized neural parcel tokens into each backbone's native conditioning representation; it does not convert DINOv3 features into CLIP features.
| Component | Representation / Role |
|---|---|
| DINOv3 supervision | Generator-independent visual learning target |
| CLIP supervision in our comparison | 768-dimensional global image vector |
| SANE transfer input | Contextualized neural parcel tokens |
| MindEye2 conditioning interface | 256 × 1664 intermediate representation |
| Backbone-specific adapter | Aligns neural evidence with the required native conditioning interface |
Replacing DINOv3 supervision with CLIP does not make the resulting neural representation directly compatible with MindEye2 or the other pretrained backbones. The proposed removal of adapters therefore does not follow from changing the supervision target alone.
Q3
Yes, the adapters are trained exclusively on the NSD training split, using frozen SANE neural representations as inputs and backbone-specific intermediate conditioning features as supervision targets, with only the adapters being optimized (Appendix E.1–E.2).
Importantly, the adapters do not perform DINOv3-to-CLIP transformation, but instead align SANE's neural evidence with each backbone's native conditioning space (Appendix E.1, Eq. 34).
The held-out results (Table 1) demonstrate that the available data support functional alignment across all five backbones, although reconstruction fidelity remains architecture-dependent.
Q4
We thank the reviewer for this suggestion. We agree and have updated the Introduction and Related Work sections of the main manuscript, with the additions highlighted in blue.
Unlike NeuroAdapter, which uses regional brain representations through cross-attention for diffusion-based reconstruction, SANE explicitly identifies stimulus-specific anatomical evidence before cross-parcel integration, preserving its anatomical identity.
Unlike Brain Algebra, which constructs concept-level neural perturbations, SANE uses naturally observed fMRI responses for controlled cross-stimulus evidence substitution, directly testing whether the selected evidence carries stimulus-specific visual information.
We have incorporated all revisions into the manuscript, with all changes highlighted in blue.
We thank the reviewer for the valuable and constructive feedback. Below, we address each concern and respond to all questions.
W1
(a) Spatial granularity. We consider parcel-level granularity appropriate for our study. Anatomically defined parcels let us identify where the evidence for visual decoding comes from and compare the cortical regions selected by SANE across stimuli and subjects.
(b) Predictive redundancy. The retention experiment tests whether a stimulus-specific subset can preserve full-input decoding performance (Appendix D.1.2, Table 3).
SANE omits a substantial fraction of parcels per prediction with no observed loss in mean decoding performance, demonstrating predictive redundancy for the evaluated task and selection policy.
W2
We clarify that variation in selected parcels does not imply that a shared evidence budget is unreasonable. SANE explicitly separates two decisions: the budget K determines how much neural evidence to retain, while stimulus-adaptive selection determines which parcels contribute to each prediction. We select a shared budget through empirical ablations to balance predictive performance and computational efficiency, while adapting parcel identities to individual stimuli. This separation of evidence capacity and evidence selection is a deliberate design principle of SANE.
(a) Stimulus-dependent selection variability
SANE learns stimulus-dependent parcel selections rather than relying on a fixed anatomical subset. Appendix D.3 (Figures 15–18) further visualizes evidence scores, cortical selections, and post-selection interactions across 32 stimuli.
(b) Empirical justification of K=32
K=32 is the smallest evaluated budget preserving near-Dense performance, providing an empirically justified accuracy–efficiency operating point rather than an arbitrary restriction.
(c) Sample-level budget variability
K=32 is the modal and median minimum qualifying budget, although individual samples have different requirements. Such variability does not invalidate the shared-budget design, which targets a consistent accuracy–efficiency trade-off rather than per-sample budget optimality.