acceptodds
Under review as a conference paper at ICLR 2027

SANE: Stimulus-Adaptive Neural Evidence for fMRI Visual Reconstruction

Abstract

What part of an fMRI response actually supports a decoded visual prediction? Modern fMRI-to-image systems can reconstruct increasingly rich visual content, yet reconstruction accuracy alone does not reveal which neural measurements support a particular prediction. We introduce Stimulus-Adaptive Neural Evidence (SANE), which scores anatomically defined cortical parcels, selects a fixed-budget stimulus-adaptive subset before cross-parcel contextualization, and tests the selected evidence through retention, removal, and cross-stimulus substitution for sufficiency, functional importance, and content specificity, respectively. On the official 1,000-image NSD shared test set across four subjects, a K=32 evidence budget retains only 39.15% of available cortical parcels while preserving 97.89% of dense DINOv3 decoding performance, reducing neural-side computation from 1.073 to 0.445 GFLOPs. Yet this strong compressibility does not imply interchangeability: removing SANE-selected evidence reduces target cosine similarity by 0.1428, compared with 0.0354 for uniform-random removal and 0.0843 for voxel-count-matched removal, while cross-stimulus substitution produces a donor-directed shift of 0.4523, compared with 0.2226 and 0.3869 for the corresponding controls. The selected evidence also shows a reproducible stimulus-dependent component: repeated measurements of the same image yield more similar selected parcel sets than measurements of different images. Finally, the same frozen evidence mechanism remains usable across five heterogeneous pretrained reconstruction systems through lightweight backbone-specific adapters. These results reveal a striking asymmetry in fMRI visual decoding: dense cortical responses are highly redundant for sufficiency, yet the compact evidence identified by SANE remains functionally structured and stimulus-dependent. SANE thus shifts the question from how much neural data can be discarded to which measured neural signals actually support a decoded visual prediction.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI, visual reconstruction, neural evidence, stimulus-adaptive learning, brain decoding, neural representation, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZzoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates which brain regions make critical contributions in fMRI-to-image decoding/reconstruction tasks, and whether certain brain regions can be removed while preserving decoding accuracy. To this end, we construct an fMRI representation model equipped with token selection capability, and optimize both the representation model and the token selector via end-to-end supervised fMRI-image alignment. Experiments are conducted on the NSD dataset. The main result demonstrates that comparable decoding performance can be achieved using only approximately 40% of visual brain regions.

Critical strengths
  • The authors have included a detailed appendix to offer thorough explanations
Critical weaknesses
  • The present study adopts a relatively coarse granularity, performing exploration and ablation only at the brain-region level. The experimental results seem to suggest that all brain regions contribute usefully, and there is no absolute redundancy. This conclusion offers little insightful inspiration.

  • As stated by the authors, the selected brain regions vary substantially across different samples. Accordingly, a unified setting of 32 appears unreasonable, and the paper does not analyze the variability in brain-region selection among samples.

  • This paper emphasizes dynamic selection. However, if decoding is performed using the top 32 most relevant brain regions selected based on neuroscientific priors, would comparable performance be attained? If so, the additional training of a model to carry out dynamic selection would lose its merit.

Questions for authors
  • Could the dynamics of the scorer’s output scores across the second-stage training be visualized?
  • Why did the authors choose DINOv3 features as supervision for training SANE? If CLIP features were adopted, they could be directly interfaced with the image reconstruction model without training an adaptor module.
  • Is the adaptor trained solely on images from the NSD dataset? Is this amount of data sufficient for training a representation transformation model that maps DINO features to CLIP features?
  • Considering the existence of closely related prior works such as NeuroAdapter, I believe the authors should discuss the distinctions between this work and them within the main manuscript.
8 comments

We thank the reviewer for the valuable and constructive feedback. Below, we address each concern and respond to all questions.

W1

(a) Spatial granularity. We consider parcel-level granularity appropriate for our study. Anatomically defined parcels let us identify where the evidence for visual decoding comes from and compare the cortical regions selected by SANE across stimuli and subjects.

(b) Predictive redundancy. The retention experiment tests whether a stimulus-specific subset can preserve full-input decoding performance (Appendix D.1.2, Table 3).

SettingParcels retainedParcels omittedMean DINOv3 cosine ↑
Dense (full input)100.00%0.00%0.4015
SANE ((K=48))58.72%41.28%0.4016

SANE omits a substantial fraction of parcels per prediction with no observed loss in mean decoding performance, demonstrating predictive redundancy for the evaluated task and selection policy.

W2

We clarify that variation in selected parcels does not imply that a shared evidence budget is unreasonable. SANE explicitly separates two decisions: the budget KK determines how much neural evidence to retain, while stimulus-adaptive selection determines which parcels contribute to each prediction. We select a shared budget through empirical ablations to balance predictive performance and computational efficiency, while adapting parcel identities to individual stimuli. This separation of evidence capacity and evidence selection is a deliberate design principle of SANE.

(a) Stimulus-dependent selection variability

ComparisonShared Parcels
Within-subject, different stimuli21.96 / 32
Within-subject, all 1,000 stimuli0 / 32

SANE learns stimulus-dependent parcel selections rather than relying on a fixed anatomical subset. Appendix D.3 (Figures 15–18) further visualizes evidence scores, cortical selections, and post-selection interactions across 32 stimuli.

(b) Empirical justification of K=32K=32

KKParcels Retained (%)Dense Performance Retained (%)
1619.5775.28
3239.1597.89
6478.30100.69

K=32K=32 is the smallest evaluated budget preserving near-Dense performance, providing an empirically justified accuracy–efficiency operating point rather than an arbitrary restriction.

(c) Sample-level budget variability

Minimum Budget for ≥95% Dense PerformanceObservations (%)
Below 3227.23
3239.62
Above 32 or requiring Dense fallback33.15

K=32K=32 is the modal and median minimum qualifying budget, although individual samples have different requirements. Such variability does not invalidate the shared-budget design, which targets a consistent accuracy–efficiency trade-off rather than per-sample budget optimality.

W3

a. Independent neuroscience prior

There is no established neuroscience prior that identifies the optimal 32 HCP-MMP1 parcels for natural-image decoding. The Top-32 list shown in our appendix is an empirical result of this study. It ranks parcels by how often SANE’s learned Dynamic selector chooses them on the training set and defines our Static-32 baseline.

To test the reviewer’s proposal directly, we ranked parcels by response strength in independent visual localizer data and fixed the top 32 as Prior-32. We evaluated both methods with the same frozen decoder and parcel budget on the same 1,000 held-out images across four NSD subjects.

MethodDINOv3 cosine ↑
Independent Neuroscience Prior-320.349
SANE Dynamic-320.395

The tested neuroscience prior yields lower decoding performance than Dynamic-32 under the same frozen decoder. The adjusted paired confidence interval for Prior-32 minus Dynamic-32 is [−0.05092, −0.04072].

b. Static and dynamic selection

Dynamic and Static selection are neither independent nor inherently competing approaches. The value of Dynamic selection is not a universal improvement in mean decoding accuracy at K=32K=32, but its ability to identify reproducible, stimulus-specific neural evidence while maintaining competitive predictive sufficiency. Its predictive advantage becomes more pronounced under tighter evidence budgets.(Appendix D.2.5)

Prediction consistency

MetricDynamicStatic
DINO Cosine ↑0.3950.398
Image-wise retention variance ↓162.90191.35

Dynamic reduces retention variance by 14.87% (95% CI for the difference: [-53.05, -1.05]).

Stimulus specificity and functional importance

AnalysisDynamic Advantage95% CI / Significance
Stimulus-matched vs. mismatched masks+0.008—
Within-category matched vs. mismatched masks+0.00511[0.00341, 0.00680]
Same- vs. different-stimulus selection (ΔJ\Delta J)+0.0373Adjusted p=0.00050p=0.00050
Dynamic vs. Static removal impact+0.00497[0.00143, 0.00862]

Limited-budget advantages

KKDynamic − Static (DINO Cosine ↑)
4+0.068
8+0.038

(d) Learned selection baselines (K=32K=32)

MethodDINO Cosine ↑Removal Impact ↑
SANE0.394990.14280
Gumbel-TopK0.389640.16991
Straight-Through TopK0.389170.07781

These results demonstrate that Dynamic selection contributes more consistent stimulus-wise retention, reproducible and functionally relevant stimulus-specific evidence, and greater benefits under tighter budgets. SANE also achieves the highest predictive accuracy among the evaluated learned selectors, without claiming superiority on every metric.

The corresponding analyses are reported in Appendix D.2.1–D.2.4.

Q1

We have added visualizations of raw parcel evidence scores and their relative rankings throughout SANE's second-stage training, covering all available parcels across four subjects (Appendix D.2.6, Figure 15 and Table 16).

MetricEarly (Epochs 0–5)Late (Epochs 15–20)
Mean absolute rank change ↓18.63–30.440.49–1.04
Spearman rank correlation ↑—0.9975–0.9994

These results demonstrate that SANE progressively learns stable anatomical preferences rather than maintaining fluctuating parcel rankings. Importantly, stable aggregate rankings do not imply fixed selection: the scorer continues to select stimulus-specific Top-KK parcels for each fMRI response.

Q2

We chose DINOv3 as the supervision target because of its image-only self-supervised learning objective and strong visual representation capabilities.

(a) DINOv3 vs. CLIP supervision

To examine whether CLIP provides a better supervision target, we compare both under identical training settings using the same frozen MindEye2 reconstruction pipeline on 50 prespecified test images from Subject 01.

MetricEvaluation AspectDINOv3CLIPDifference (95% CI)
PixCorr ↑Pixel-level correspondence0.101800.09677+0.00502 [-0.00697, +0.01674]
SSIM ↑Structural similarity0.241320.24364-0.00231 [-0.00825, +0.00334]
Alex(2) ↑Shallow visual features78.367%78.612%-0.245 pp [-2.694, +2.122]
Alex(5) ↑Deeper visual features92.531%91.265%+1.265 pp [-0.408, +2.980]
Inception ↑High-level features83.633%85.714%-2.082 pp [-6.736, +2.245]
CLIP ↑Semantic identification74.041%72.571%+1.469 pp [-1.673, +4.939]

The additional comparison provides no evidence that switching to CLIP would systematically improve reconstruction fidelity. Our choice of DINOv3 is motivated by the objective of generator-independent visual evidence learning.

(b) Why CLIP supervision does not eliminate adapters

The key distinction is supervision space versus downstream conditioning interface. As described in Appendix E.1 (Eq. 34), our adapter maps SANE's contextualized neural parcel tokens into each backbone's native conditioning representation; it does not convert DINOv3 features into CLIP features.

ComponentRepresentation / Role
DINOv3 supervisionGenerator-independent visual learning target
CLIP supervision in our comparison768-dimensional global image vector
SANE transfer inputContextualized neural parcel tokens
MindEye2 conditioning interface256 × 1664 intermediate representation
Backbone-specific adapterAligns neural evidence with the required native conditioning interface

Replacing DINOv3 supervision with CLIP does not make the resulting neural representation directly compatible with MindEye2 or the other pretrained backbones. The proposed removal of adapters therefore does not follow from changing the supervision target alone.

Q3

Yes, the adapters are trained exclusively on the NSD training split, using frozen SANE neural representations as inputs and backbone-specific intermediate conditioning features as supervision targets, with only the adapters being optimized (Appendix E.1–E.2).

Importantly, the adapters do not perform DINOv3-to-CLIP transformation, but instead align SANE's neural evidence with each backbone's native conditioning space (Appendix E.1, Eq. 34).

The held-out results (Table 1) demonstrate that the available data support functional alignment across all five backbones, although reconstruction fidelity remains architecture-dependent.

Q4

We thank the reviewer for this suggestion. We agree and have updated the Introduction and Related Work sections of the main manuscript, with the additions highlighted in blue.

Unlike NeuroAdapter, which uses regional brain representations through cross-attention for diffusion-based reconstruction, SANE explicitly identifies stimulus-specific anatomical evidence before cross-parcel integration, preserving its anatomical identity.

Unlike Brain Algebra, which constructs concept-level neural perturbations, SANE uses naturally observed fMRI responses for controlled cross-stimulus evidence substitution, directly testing whether the selected evidence carries stimulus-specific visual information.

We have incorporated all revisions into the manuscript, with all changes highlighted in blue.

Reviewer_P2TQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SANE, a stimulus-adaptive neural evidence selection framework for fMRI-to-image reconstruction. The method uses anatomically defined cortical parcels, independently scores their importance, and selects the Top-K parcels before cross-parcel contextualization. The authors further introduce retention, removal, and cross-stimulus substitution experiments to evaluate evidence sufficiency, functional importance, and content specificity. Experiments on four NSD subjects demonstrate that selecting 32 parcels preserves 97.89% of dense DINOv3 decoding performance while reducing neural-side computation. The selected representations are also transferred to five pretrained reconstruction backbones.

Critical strengths
  1. The paper investigates which neural measurements contribute to visual decoding rather than focusing solely on reconstruction accuracy. The distinction between sufficiency, functional importance, and content specificity provides a useful perspective.

  2. The experiments include multiple subjects, evidence-budget ablations, intervention-based evaluations, and cross-backbone transfer, providing reasonably extensive validation of the proposed framework.

Critical weaknesses
  1. The central contribution is stimulus-adaptive parcel selection. However, Appendix D.2 shows that at the default setting of K = 32, dynamic selection performs slightly worse than static selection (0.395 vs. 0.398 DINO cosine similarity). Although dynamic selection performs better under smaller budgets, its practical advantage at the main operating point remains unclear. Furthermore, comparisons with stronger learning-based feature selection or attribution methods are insufficient to establish the necessity of the proposed selection mechanism.

  2. Although SANE is evaluated with five reconstruction models, Table 1 shows noticeable performance degradation for most backbones. For example, NeuroFlow's PixCorr decreases from 0.271 to 0.170, while MindEye2 also exhibits reduced reconstruction quality. Moreover, Appendix E.4 reports increased inference latency for some backbones. These results weaken the claimed generalizability and practical effectiveness of the compact neural evidence representation.

Questions for authors
  1. Given that static selection slightly outperforms dynamic selection at K = 32, what is the practical necessity of stimulus-adaptive selection at the proposed operating point? Could the authors provide comparisons with stronger learned selection baselines under identical evidence budgets?

  2. The removal and substitution experiments mainly compare against random and voxel-count-matched controls. Could the authors further compare against parcels matched for anatomical location or neural signal reliability, or against gradient-based attribution methods, to establish whether the selected evidence captures stimulus-specific information rather than generally informative visual regions?

  3. Since SANE reduces reconstruction quality for most pretrained backbones, could the authors clarify the practical trade-off between neural-side efficiency and reconstruction fidelity? In particular, how does SANE compare with a simpler static parcel-selection strategy when integrated into the same reconstruction backbones?

5 comments

We thank the reviewer for the valuable and constructive feedback. Below, we address each concern and respond to all questions.

W1

We clarify that Dynamic and Static selection are neither independent nor inherently competing approaches. As described in Appendix D.2.1, Static selection is derived from the training-set selection frequencies of SANE's learned Dynamic mechanism. The value of Dynamic selection is not a universal improvement in mean decoding accuracy at K=32K=32, but its ability to identify reproducible, stimulus-specific neural evidence while maintaining competitive predictive sufficiency. Its predictive advantage becomes more pronounced under tighter evidence budgets.(Appendix D.2.5)

(a) Prediction consistency

MetricDynamicStatic
DINO Cosine ↑0.3950.398
Image-wise retention variance ↓162.90191.35

Dynamic reduces retention variance by 14.87% (95% CI for the difference: [-53.05, -1.05]).

(b) Stimulus specificity and functional importance

AnalysisDynamic Advantage95% CI / Significance
Stimulus-matched vs. mismatched masks+0.008—
Within-category matched vs. mismatched masks+0.00511[0.00341, 0.00680]
Same- vs. different-stimulus selection (ΔJ\Delta J)+0.0373Adjusted p=0.00050p=0.00050
Dynamic vs. Static removal impact+0.00497[0.00143, 0.00862]

(c) Limited-budget advantages

KKDynamic − Static (DINO Cosine ↑)
4+0.068
8+0.038

(d) Learned selection baselines (K=32K=32)

MethodDINO Cosine ↑Removal Impact ↑
SANE0.3950.14280
Gumbel-TopK0.38960.16991
Straight-Through TopK0.38920.07781

These results demonstrate that Dynamic selection contributes more consistent stimulus-wise retention, reproducible and functionally relevant stimulus-specific evidence, and greater benefits under tighter budgets. SANE also achieves the highest predictive accuracy among the evaluated learned selectors, without claiming superiority on every metric.

The corresponding analyses are reported in Appendix D.2.1–D.2.4, with learned selection baselines addressed in our response to Q1.

W2

We clarify that cross-system transferability does not imply universally improved reconstruction quality or reduced inference latency. Our goal is to assess whether independently learned neural evidence can support reconstruction across heterogeneous pretrained backbones through adapter-only alignment, rather than outperforming their native neural encoders (Section 3.5; Appendix E.1).

(a) Source of reconstruction degradation

To examine whether reconstruction degradation results from the evidence budget, we additionally compare native Brain-IT with SANE using all parcels and Top-32 selection.

MethodPixCorr ↑SSIM ↑Alex(2) ↑Alex(5) ↑Incep ↑CLIP ↑Eff ↓SwAV ↓
Brain-IT†0.375 ± 0.0560.476 ± 0.01398.7 ± 0.9%99.4 ± 0.4%97.3 ± 1.7%96.3 ± 2.1%0.565 ± 0.0540.323 ± 0.030
SANE + Brain-IT (All)0.297 ± 0.0500.463 ± 0.01196.1 ± 2.0%98.1 ± 1.0%94.4 ± 2.8%93.6 ± 2.6%0.634 ± 0.0450.369 ± 0.027
SANE + Brain-IT (Top-32)0.307 ± 0.0590.465 ± 0.01396.6 ± 2.1%98.3 ± 1.0%94.3 ± 3.2%93.5 ± 2.9%0.636 ± 0.0550.369 ± 0.032

Retaining all parcels does not recover native reconstruction fidelity. This indicates that the performance gap is not solely due to evidence compression.

(b) Backbone-dependent efficiency

BackboneNeural-interface parameter reductionBatch-1 latency reductionCLIP retained vs. matched native baseline
MindEye298.28%53.48%94.38%
Brain-IT79.41%41.29%97.10%
NeuroPictor84.12%39.06%93.60%
NeuroAdapter77.92%−1089.28%102.62%
NeuroFlow56.41%−138.74%94.74%

SANE reduces neural-interface parameter counts across all five backbones. Latency gains are backbone-dependent, and NeuroAdapter improves CLIP identification at the cost of higher latency.

(c) Scope of transferability

SANE demonstrates that compact neural evidence can be reused across five heterogeneous reconstruction systems without retraining their pretrained reconstruction components. Its contribution is evidence portability, not universal reconstruction or latency improvements.

The cross-backbone reconstruction results are reported in Section 3.5 (Table 1), with adapter details in Appendix E.1–E.2 and neural-side efficiency in Appendix E.4 (Table 14).

Q1

(a) Why stimulus adaptivity matters at (K=32)

We address the practical value of stimulus-adaptive selection at (K=32) in our response to W1.

(b) Learned baselines at the same evidence budget

We compare SANE, Gumbel-TopK, and Straight-Through TopK (ST-TopK) across four NSD subjects using the same architecture, Dense initialization, 20-epoch training schedule, and (K=32) budget. DINO cosine measures predictive fidelity. Removal impact measures the degradation caused by removing selected parcels from the same frozen Dense decoder.

MethodDINO Cosine ↑Removal Impact ↑
SANE0.3950.1428
Gumbel-TopK0.389640.16991
ST-TopK0.389170.07781

SANE achieves the highest predictive fidelity (Appendix D.1.4).

Q2

We compare all five selection methods at (K=32) using the same frozen Dense decoder. Removal impact measures the drop in target similarity after parcel removal. Substitution shift measures how strongly replacing the selected signals redirects the prediction from the original stimulus toward the donor.

Selection MethodTest Target UsedRemoval ↑Substitution ↑
SANENo0.14280.4523
Anatomy + Reliability MatchedNo0.10060.4061
Strict 1% MatchedNo0.11370.4278
Input × GradientYes-0.02400.2073
Integrated Gradients (IG)Yes0.22990.4516

(a) Anatomy and reliability controls. Both controls match hemisphere, anatomical subdivision, training-estimated signal reliability, and total voxel count. The strict control applies a tighter voxel-count tolerance. Despite substantial parcel overlap, SANE yields stronger removal and substitution effects than both controls (Appendix D.4.4).

(b) Gradient-based attribution. Both gradient methods use the ground-truth test image’s DINOv3 features during parcel selection. IG produces a larger removal effect and a similar substitution shift to SANE. Input × Gradient selects parcels whose removal slightly improves target similarity (Appendix D.4.4, Table 19).

(c) Stimulus specificity. The matched controls show that anatomy, reliability, and voxel count explain only part of the intervention effects. The matched-mask experiments directly test stimulus specificity and show that each stimulus’s own mask outperforms masks from other stimuli, even within the same semantic category (Appendix D.2.2 and D.2.5).

Q3

SANE’s primary goal is to identify and validate stimulus-adaptive neural evidence. The cross-backbone experiments test whether that independently learned evidence can support reconstruction in pretrained systems.

(a) Neural-side efficiency vs. reconstruction fidelity

Our response to W2(a)–(b) shows the practical trade-off between neural-side efficiency and reconstruction fidelity. SANE reduces neural-interface parameter counts across all five backbones, while latency and reconstruction fidelity vary by backbone (Table 1 and Appendix E.4).

(b) Static vs. Dynamic selection in the same MindEye2 pipeline

The controlled comparison uses 50 held-out images from NSD Subject 01, each with three fMRI repetitions. Both variants use the same frozen pipeline, paired generation seeds, and (K=32) budget.

MethodPixCorr ↑SSIM ↑Alex(2) ↑Alex(5) ↑Incep ↑CLIP ↑Eff ↓SwAV ↓
Dynamic0.101800.241320.783670.925310.836330.740410.877020.51370
Static0.092910.235360.768570.920410.817550.744900.878200.51688

The MindEye2 experiment establishes no clear reconstruction advantage for either strategy (Appendix E.2, Static versus Dynamic Evidence Selection in MindEye2). The evidence for stimulus specificity comes from the matched-mask and intervention analyses (Appendices D.2 and D.4).

We have incorporated all revisions into the manuscript, with all changes highlighted in blue.

Reviewer_ZtwfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SANE proposes a framework for identifying which fMRI measurements actually support a decoded visual prediction, rather than simply asking how much data can be discarded. The method scores anatomically defined cortical parcels from the current stimulus response, selects a fixed-budget top-K subset before cross-parcel contextualization, and validates the selection through three intervention tests: retention (sufficiency), removal (functional importance), and cross-stimulus substitution (content specificity). On the NSD 1,000-image test set across four subjects, K=32 retains 39.15% of parcels while preserving 97.89% of dense DINOv3 decoding performance. Removal and substitution tests show that SANE-selected parcels are more functionally important and more content-specific than matched controls. The paper also reports stimulus-dependent reproducibility of selected parcel sets and transferability across five pretrained reconstruction systems via lightweight adapters.

Critical strengths

The core contribution is a well-posed question rather than a new model. The fMRI reconstruction literature has focused almost exclusively on accuracy or dimensionality reduction. SANE instead asks which measurements are causally responsible for a given prediction.

The three-test intervention design (retention, removal, substitution) is principled and goes beyond standard ablation. The distinction between sufficiency, importance, and content specificity is well-motivated: compressibility alone does not establish functional relevance, and the paper correctly identifies this gap.

Removing SANE-selected parcels causes roughly 4x more degradation than random removal, and cross-stimulus substitution produces a strong donor-directed shift. This constrains how we think about information distribution across visual cortex and is not a trivial observation.

Critical weaknesses

The contribution is primarily diagnostic rather than methodological. SANE does not improve reconstruction quality or propose a new learning algorithm. It provides an analytical framework applied post-hoc to existing systems. This raises a venue-fit concern: the work reads more naturally as a neuroscience contribution (NeuroImage, Nature Neuroscience) than as a machine learning advance. The cross-system adapter result is the closest methodological element but is underdeveloped. The authors should articulate more clearly what the learning contribution is.

The stimulus-adaptivity claim lacks rigorous statistical grounding. The paper reports that repeated measurements of the same image produce more similar parcel selections than different images, but the effect size, statistical test, and comparison against null models are not prominent. fMRI parcel-level scores can be dominated by vascular or coil-related variance that is stable within a subject but unrelated to stimulus content. Without permutation tests or appropriate controls for subject-level confounds, it is difficult to distinguish genuine stimulus-dependent selection from stable anatomical or acquisition artifacts.

Questions for authors

What is the overlap between SANE-selected parcels across subjects? If the same parcels are consistently selected regardless of stimulus, the "stimulus-adaptive" claim weakens to "subject-specific but stimulus-invariant." Please report the intersection of selected parcel sets across stimuli within a subject and across subjects.

The removal test compares against uniform-random and voxel-count-matched controls. Have you considered a more informative control: removing the parcels with the highest average activation magnitude? This would test whether SANE is doing something beyond selecting the most active regions.

For the cross-system transferability claim, what is the performance degradation when the frozen evidence mechanism is applied to a reconstruction system it was not selected for? Is the adapter truly lightweight, or does it require retraining on the target system's outputs?

The K=32 budget appears fixed. How sensitive are the conclusions to this choice? Does the asymmetry finding (redundant for sufficiency, non-interchangeable for importance) hold at K=16 or K=64?

3 comments

We thank the reviewer for the valuable and constructive feedback. Below, we address each concern and respond to all questions.

W1

We clarify that SANE introduces a task-driven neural evidence learning framework, rather than a post-hoc diagnostic method or a new Top-K optimization algorithm. Its methodological contribution is to jointly learn which anatomically identifiable neural measurements support a visual prediction and how the selected evidence should be integrated, under an explicit evidence budget.

First, our key architectural principle is identify before integrate (Sections 2.1–2.2). Each parcel is independently encoded and scored before cross-parcel information exchange, preserving the anatomical identity of the selected evidence. Contextual integration then operates on the selected tokens. Crucially, evidence selection participates in end-to-end representation learning; only the subsequent interventions are post hoc.

Second, controlled experiments evaluate the learned selection mechanism.

MethodDINO Cosine ↑Removal Impact ↑
SANE0.39500.14283
Gumbel-TopK0.38960.16991
ST-TopK0.38920.07781

Appendix D.1.4, Table 6.

SANE achieves the highest predictive fidelity among the evaluated selectors, although Gumbel-TopK produces a larger removal impact. A larger removal impact does not necessarily indicate better predictive evidence selection, as the two metrics assess different properties.

Finally, SANE learns anatomically identifiable evidence whose predictive sufficiency, functional importance, and stimulus-specific content can be independently tested through retention, removal, and cross-stimulus substitution (Section 2.3; Figure 5). Moreover, the frozen evidence representation can be transferred to five heterogeneous reconstruction backbones through backbone-specific adapters (Section 3.5).

Thus, evidence identification is the central methodological goal, with visual prediction providing the task-driven learning signal, rather than reconstruction accuracy being the sole objective of the framework.

W2

To distinguish stimulus-dependent selection from acquisition-related confounds, we conducted 10,000 acquisition-blocked stimulus-label permutations across four subjects, preserving session, run, and repetition structure.

AnalysisSame-image JJDifferent-image JJΔJ\Delta JAdjusted pp
All repetitions0.46440.4270+0.03730.00050
Cross-session only0.46010.4267+0.03340.00050

Bonferroni-adjusted permutation tests; the all-repetition effect has a 95% bootstrap CI of [0.0358,0.0389][0.0358, 0.0389].

These results confirm statistically significant stimulus-dependent selection beyond the tested acquisition confounds. Moreover, adaptive selection produces a greater removal-induced DINO cosine drop than static selection (Δ=+0.00497\Delta=+0.00497, 95% CI: [0.00143,0.00862][0.00143,0.00862]), supporting its functional relevance.

The selection consistency analysis is reported in Appendix D.2.3.

Q1

We analyzed the overlap of Top-32 parcel selections across 1,000 shared test images and four anatomically aligned subjects. For each image, parcel scores were averaged across three fMRI repetitions before selecting Top-32 parcels. Shared parcels denote pairwise intersections for two-way comparisons and four-way intersections for the same stimulus across subjects. Unlike the trial-level overlap reported in Appendix D.2.3 (19.0/32), the present analysis uses repetition-averaged stimulus-level masks.

ComparisonShared Parcels
Within-subject, different stimuli21.96 / 32
Cross-subject, same stimulus20.48 / 32
Four subjects, same stimulus13.05 / 32
Within-subject, all 1,000 stimuli0 / 32

These results show that SANE exhibits shared anatomical preferences without selecting fixed, stimulus-invariant parcel sets. The permutation tests in W2 further confirm statistically significant stimulus-dependent selection.

Selection consistency is discussed in Appendix D.2.3.

Q2

To assess whether SANE simply selects highly activated regions, we extend the removal experiment in Fig. 5(a) with activation-based and SANE-based static controls, using the same frozen Dense decoder and Top-32 evidence budget across four subjects. Activation-based Static ranks parcels by training-set mean absolute fMRI activation, whereas SANE-based Static uses training-set mean learned evidence scores; Stimulus-Adaptive SANE selects parcels separately for each test stimulus.

Removal StrategyDINO Cosine Drop ↑
Uniform Random0.03540
Voxel-Count-Matched Random0.08430
Activation-based Static0.03772
SANE-based Static0.13786
Stimulus-Adaptive SANE0.14283
ContributionAdditional Drop ↑Adjusted 95% CI
Learned evidence scoring+0.10013[0.09215, 0.10829]
Stimulus-adaptive selection+0.00497[0.00143, 0.00862]

These results demonstrate that SANE identifies predictively important neural evidence beyond activation magnitude, with learned evidence scoring providing the primary contribution and stimulus-adaptive selection offering an additional statistically supported improvement.

Experimental details are provided in Appendix D.4 (Evidence Removal).

Q3

As reported in Section 3.5 and Table 1, we evaluate the portability of frozen SANE evidence across five pretrained reconstruction systems using backbone-specific conditioning adapters.

BackbonePixCorr Native / SANEΔSSIM Native / SANEΔCLIP (%) Native / SANEΔ (pp)Adapter (M)
MindEye2.284 / .223−.061.373 / .387+.01492.9 / 87.7−5.218.03
NeuroPictor.228 / .157−.071.398 / .414+.01693.5 / 87.5−6.034.57
NeuroAdapter.080 / .095+.015.276 / .300+.02486.7 / 89.0+2.34.44
Brain-IT.375 / .307−.068.476 / .465−.01196.3 / 93.5−2.817.10
NeuroFlow.263 / .170−.093.221 / .204−.01795.0 / 90.0−5.09.47

Δ = SANE − Native; pp = percentage points. Adapter (M) denotes backbone-specific trainable parameters in millions.

Transfer requires backbone-specific adapter training, not zero-shot deployment or backbone retraining. Each adapter learns to align frozen SANE evidence with the target system's conditioning representation, including image-derived latent targets where applicable. Both SANE and the pretrained downstream reconstruction components remain frozen, and no end-to-end image reconstruction loss is optimized (Appendix E.1–E.2).

The results demonstrate cross-architecture portability with explicit fidelity trade-offs. SANE improves NeuroAdapter across the reported metrics, while the other four systems exhibit metric-dependent degradation. Adapter training involves 4.44–34.57M trainable parameters, whereas the complete SANE-based neural interfaces contain 56.41%–98.28% fewer parameters than their native counterparts (Appendix E.4). These reductions concern neural-interface size, exclude frozen downstream generation components, and do not imply universally lower end-to-end inference latency.

This experiment establishes that learned SANE evidence remains usable across heterogeneous reconstruction architectures without retraining their pretrained generative components, rather than claiming lossless or zero-shot transfer.

Q4

We extend the predictive sufficiency, removal, and cross-stimulus substitution experiments to (K=16) and (K=64), using the same evaluation protocol across four subjects.

(K)Parcels Retained (%)Dense Performance Retained (%)Removal (S/R/M)Extra Removal [95% CI]Donor Shift (S/R/M)Extra Shift [95% CI]
1619.5775.280.0418 / 0.0139 / 0.0351+0.0067 [0.0050, 0.0084]0.1929 / 0.0833 / 0.1772+0.0158 [0.0131, 0.0184]
3239.1597.890.1428 / 0.0354 / 0.0843+0.0585 [0.0538, 0.0635]0.4523 / 0.2226 / 0.3869+0.0654 [0.0615, 0.0694]
6478.30100.690.2242 / 0.1269 / 0.2072+0.0170 [0.0139, 0.0199]0.6056 / 0.5343 / 0.5983+0.0073 [0.0061, 0.0086]

Results are averaged across four subjects. S/R/M denotes SANE, Random, and voxel-count-matched random controls, respectively. Extra effects represent SANE minus Matched, with 95% CIs from 10,000 paired bootstrap resamples over shared stimuli.

(a) Predictive sufficiency. The results show a clear budget–performance trade-off. A smaller budget sacrifices predictive sufficiency, whereas a larger budget provides negligible additional performance despite substantially increased parcel retention. This supports (K=32) as an effective operating point rather than an arbitrary fixed choice.

(b) Functional non-interchangeability. Across all three budgets, SANE-selected parcels produce greater removal degradation and stronger donor-directed shifts than matched controls, with consistently positive confidence intervals. These findings establish that the selected evidence remains functionally important and stimulus-specific across different budgets, rather than being freely interchangeable.

(c) Budget-dependent insight. The functional non-interchangeability of selected evidence persists across all tested budgets, while predictive sufficiency depends on K. At smaller budgets, predictive information is insufficiently retained; at larger budgets, the distinction from matched controls becomes less pronounced. K=32 therefore provides a favorable operating point where near-Dense predictive performance coexists with the strongest additional removal and substitution effects.

We have incorporated all revisions into the manuscript, with all changes highlighted in blue.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of rebuttal and revision

Dear AC,

We sincerely thank you for evaluating our work. We also thank all three reviewers for their constructive comments. Their feedback has helped us clarify SANE's methodological contributions and strengthen the experimental evidence through extensive additional experiments.

For your convenience, we summarize the key contributions, major improvements during the rebuttal, and reviewer feedback below.

1. We are encouraged that the reviewers recognized:

  • Novel research perspective and evaluation framework: Distinguishing predictive sufficiency, functional importance, and content specificity through three complementary intervention tests. (Ztwf, P2TQ)
  • Meaningful empirical findings: Revealing that neural evidence is highly redundant for predictive sufficiency but functionally important and stimulus-dependent. (Ztwf, P2TQ)
  • Extensive experimental validation: Providing comprehensive evaluations across subjects, evidence budgets, intervention tests, and reconstruction backbones, with substantial additional experiments acknowledged. (Ztwf, P2TQ, ZzoN)

2. Summary of rebuttal

We conducted substantial additional experiments to address the reviewers' key concerns.

Key ConcernAdditional Evidence and Findings
Methodological novelty and necessity (Ztwf, P2TQ, ZzoN)We clarified SANE's end-to-end neural evidence learning mechanism and its differences from NeuroAdapter and Brain Algebra. We also added Gumbel-TopK and ST-TopK baselines. SANE achieved the highest predictive accuracy, further supporting its methodological value as a task-driven evidence learning framework.
Stimulus adaptivity and selection variability (Ztwf, P2TQ, ZzoN)We conducted 10,000 acquisition-blocked permutation tests and stimulus-matching experiments, confirming significant stimulus dependence (ΔJ = +0.0373, adjusted p = 0.00050). The selected evidence is not simply a fixed anatomical preference but contains a reproducible and functionally meaningful stimulus-specific component.
Dynamic selection and fixed evidence budget (Ztwf, P2TQ, ZzoN)We extended the experiments to K=16/32/64, analyzed sample-level budget requirements, and added independent neuroscience prior comparisons. Dynamic and Static selection are not competing methods. Dynamic selection identifies stimulus-specific neural evidence while maintaining predictive performance, with greater predictive advantages under smaller evidence budgets.
Evidence beyond activation and anatomical priors (Ztwf, P2TQ, ZzoN)We added activation-based, anatomy-matched, reliability-matched, and gradient-based attribution controls. SANE produced a larger removal impact than activation-based selection (+0.1001) and outperformed anatomy- and reliability-matched controls.
Spatial granularity and neuroscientific insights (ZzoN)We added voxel-level predictive sufficiency, repeated-measurement consistency, cross-subject spatial correspondence, and within-parcel voxel attribution analyses. The results reveal reproducible stimulus-specific voxel contributions within parcels. Although voxel selection provides higher predictive efficiency, parcel selection shows significant cross-subject stimulus-specific spatial correspondence, supporting our original view that parcels are more suitable as the basic evidence selection units for neuroscientifically interpretable SANE.
Cross-backbone transferability and efficiency (Ztwf, P2TQ, ZzoN)We provided additional analyses of reconstruction performance, parameter counts, inference latency, and adapter training across five architectures. SANE evidence can be reused across all five heterogeneous backbones, reducing neural-interface parameters by 56.41%–98.28%, while reconstruction quality and inference speed involve backbone-dependent trade-offs.

These additional findings further support SANE's central conclusion: neural signals are highly redundant for predictive sufficiency, but the evidence supporting a specific visual prediction is functionally important, stimulus-dependent, and not arbitrarily interchangeable.

All additional experiments and analyses have been incorporated into the revised manuscript, with changes highlighted in blue.

AuthorsOpenReview ↗

Summary of rebuttal and revision (continued)

3. Reviewer feedback before and after rebuttal

We are encouraged by the reviewers' positive feedback on our additional analyses:

  • Reviewer P2TQ: In the follow-up comment on November 18, the reviewer explicitly confirmed that "The authors have adequately addressed all my concerns. I have no further questions."
  • Reviewer ZzoN: In the follow-up comment on November 17, the reviewer stated that "the authors' response has resolved most of my technical concerns" and suggested that voxel-wise decoding attribution could reveal finer-grained patterns. To address this remaining concern, we conducted systematic voxel-level experiments and further explained why parcels are more suitable as the basic evidence selection units for neuroscientific interpretability in SANE. These results have been added to Appendix F. We have not yet received further feedback on these additional analyses.
  • Reviewer Ztwf: We provided detailed responses and additional experiments addressing methodological novelty, statistical validity of stimulus adaptivity, activation-based controls, cross-backbone transferability, and budget sensitivity. We have not yet received follow-up confirmation.

We believe the rebuttal has made SANE's methodological contribution, experimental validation, and scientific findings clearer.

We sincerely thank the AC and all reviewers again for their time and constructive feedback. We hope this summary helps you evaluate the contributions of our revised manuscript.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.