acceptodds
Under review as a conference paper at ICLR 2027

BrainInterv: In Silico Interventions for Mapping Semantic Selectivity in the Brain

Abstract

Characterizing the selectivity of brain areas is important for understanding their underlying computation. Here we propose an in silico approach that allows us to address this challenge by performing interventions and studying their effect on a downstream decoding task. Brain-tuned visual representations are created by routing and transforming self-supervised visual features into region-of-interest (ROI) tokens, each supervised by functional magnetic resonance imaging (fMRI) responses from a cortical area. The resulting ROI-tuned image tokens support high-accuracy 1000-way semantic retrieval through a lightweight readout to a text-embedding space, reaching . We further extend the decoding task by predicting the ROI tokens from measured fMRI responses using a voxel-to-token mapper. These decoded ROI tokens achieve under the same frozen image-trained semantic readout. We then perform in silico interventions on the model by transplanting ROI tokens across images for the retrieval task and observe that high-level token interventions produce substantially greater donor-semantic transfer than early-visual tokens, whose effects are stronger on lower-level visual probes. This intervention pattern also holds for both ROI-tuned image tokens and decoded tokens, and in different subjects. Overall, the proposed approach provides an intervention-based method to probe the selectivity of different visual brain areas, complementing current encoding- and direct decoding-based methods.

Primary area:
applications to neuroscience & cognitive science
Keywords:
cognitive neuroscience, computational neuroscience, fMRI, brain-aligned representation learning, in silico interventions, semantic retrieval, visual representation learning, neural encoding and decoding, model interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NHb5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for examining how selective replacement of ROI-associated intermediate representations affects semantic decoding. First, frozen vision-model features are transformed into ROI-tuned image tokens, supervised through prediction of the corresponding regional fMRI responses. A semantic readout is then trained to map these tokens to image-caption embeddings. The authors replace tokens associated with selected ROI groups with the corresponding tokens from another image and measure how strongly the output of the frozen readout shifts toward the donor image’s semantics. They additionally train ROI-specific mappers to recover the same token representation from measured fMRI and perform analogous interventions on these decoded tokens. Replacing higher-level visual ROI tokens produces stronger donor-semantic transfer than replacing early visual ROI tokens. Conversely, early visual token interventions have relatively stronger effects on probes of lower-level visual features.

Critical strengths
  • The authors compare predefined ROI groups against random subsets matched for token count, vary the donor mixing coefficient continuously, evaluate alternative original–donor pair sets. The consistent High-level–Early difference across all four participants supports the robustness of the intervention effect within the tested model.
  • Comparing intervention effects across different prediction targets broadens the analysis beyond semantic retrieval alone.
Critical weaknesses

Prior work already investigates regional selectivity through encoding-model weight analyses (Huth et al., 2016), stimulus optimization (BrainDiVE; Luo et al., 2023), and ROI masking during decoding (NeuroAdapter; Feng et al., 2026). Although the paper acknowledges related approaches, it does not clearly identify which unresolved limitation ROI-token transplantation addresses or demonstrate what additional understanding it provides.

The main finding—stronger semantic effects for higher-level tokens and stronger low-level visual effects for early tokens—is broadly consistent with established visual organization. The shared image/fMRI token space is a methodological distinction, but the experiments do not establish a new scientific conclusion or a clear analytical advantage over existing approaches. Reproducing a familiar pattern is useful validation, but does not by itself establish the added value of the proposed framework.

This concern is reinforced by the readout comparison: mean pooling achieves similar retrieval performance but does not preserve the intervention ordering. The authors acknowledge this dependence, which is not itself a methodological flaw. However, it remains unclear what the intervention results reveal about brain-region selectivity beyond how the chosen semantic decoder uses its inputs.

Questions for authors

What additional scientific insight or analytical capability does the proposed method establish?

4 comments
AuthorsOpenReview ↗

Relation to prior work

We thank the reviewers for their comments. Our aim is to provide a region-indexed intervention that tests whether a site’s representation is sufficient to redirect decoded semantics toward a specific donor, under matched controls, on both ROI-tuned image tokens and decoded ROI tokens.

Capability relative to prior tools

ApproachQuestion answeredCannot show
Encoding weights (Huth et al., 2016)Which features predict a voxelWhether that region’s signal redirects a decoder output toward a chosen donor
Stimulus optimization (BrainDiVE; Luo et al., 2023)Which images maximise an ROIWhat the ROI contributes downstream in a fixed readout
ROI masking (NeuroAdapter; Feng et al., 2026)Whether the ROI is necessary (performance drops when removed)Whether the ROI alone is sufficient to transfer identity to a specific donor
ROI-token intervention (ours)Sufficiency + specificity to a donor image, with graded dose and the same protocol on ROI-tuned image tokens, fMRI-reconstructed ROI tokens, and direct fMRI ROI blocks—

Encoding, optimization, and masking remain valuable; they answer different questions. Intervention adds a donor-specific sufficiency test that those tools do not provide.

Three results beyond a generic high-level > early summary

  1. Encoding strength ≠ downstream contribution. On fLoc/streams, early encoding correlations are at least as strong as high-level ones, yet early interventions fall below size-matched random-slot nulls (z −0.6 to −1.9) across atlases, readouts, and token sources. Predictivity of a voxel does not imply that swapping that region rewrites caption identity.

  2. Editable caption semantics concentrate in lateral occipitotemporal cortex, converging across independent atlases (same 1000 pairs; donor R@10 / z vs null):

AtlasSiteROI-tuned image tokensMean-poolReconstruction-only voxel-to-token invertorDirect fMRI
KastnerTO (4/50)0.164 (z=12, 4/4)z=3.1z=5.6z=3.1–3.7
fLocbodies / EBA (~5/40)0.403 (z=9.2, 4/4)z=4.4z=5.9z=4.0–5.3
streamslateral (2/14)0.325 (z=4.7, 4/4)z=3.4z=3.3—
  1. The structure follows anatomical Stage‑1 assignment. Scrambling only the ROI↔voxel supervision keeps encoding and retrieval high, but abolishes the bodies effect (0.403 → 0.028) and collapses the cross-subject fingerprint (ρ 0.99 → 0.63). Slot names and the vision prior alone do not produce the regional pattern.

Readout dependence (NHb5 CW3 / PXXg Q1). When we wrote that mean pooling “changes the intervention ordering,” we meant the fine-grained / absolute per-ROI picture (e.g. TO’s raw donor R@10 shrinks under uniform weights), not the high-level versus early contrast. That coarser ordering survives mean pooling (z 5.1–5.7, 4/4) and also appears in direct fMRI ridge interventions with no learned token weighting. Attention mass likewise fails to predict transfer (parietal mass > TO, yet parietal transfer below null while TO transfers strongly). Full ROI-wise mean-pool results are in our reply to PXXg Q1; the manuscript framing will be elaborated accordingly.

The direct fMRI intervention experiments also help clarify what the ROI-token representation adds beyond conventional decoders. While both approaches recover the broader high-level versus early-visual pattern, the ROI-token framework provides greater sensitivity to interventions in smaller regions, such as TO and body-selective areas. More importantly, it enables comparable interventions on image-derived and fMRI-derived representations through a shared semantic readout, along with more fine-grained analyses of regional contributions (results are in our reply to PXXg Q2).

Relative to prior regional tools, ROI-token interventions provides a donor-specific sufficiency test in a shared image/brain space. Relative to textbook hierarchy, the new content is that downstream contribution is anatomically concentrated, supervision-dependent, and dissociable from encoding strength and from low-level visual transfer.

In light of these results, we hope you might view our work more favorably. Please feel free to let us know if you have any additional questions!

Thank you again for the additional experiments, which substantially strengthen the evidence for robustness. The findings in TO, body-selective regions, and the lateral visual stream, together with the category-stratified analyses, also go beyond a simple high-level versus early-visual comparison.

For work seeking new neuroscientific insights through a new method, careful positioning relative to prior findings is particularly important. Given the extensive neuroscience and machine-learning literature on these regions, especially body-selective cortex, could you clarify what unresolved question these results address and how they refine existing knowledge? It would help to distinguish framework validation from additional neuroscientific insight.

Could you also provide a concrete example of a scientific conclusion or analytical advantage enabled or strengthened by the shared ROI-token representation, beyond what is readily obtainable with simpler approaches such as direct ROI-wise interventions on fMRI inputs?

Thank you for raising these two questions. We agree that recovering known regional selectivity mainly validates the framework. To test what else we could learn, we examined whether replacing tokens from different regions changed decoded posture and setting in scenes that all contained a person. We also compared token and voxel edits using the same images. We fixed the stimulus list in advance and kept the models and readouts frozen.

1. Validation versus additional insight

We asked whether replacing EBA tokens shifted decoded posture (sitting or standing) more than replacing FBA tokens. We also asked whether edits to EBA and PPA preferentially shifted different attributes of the same scene: posture for EBA and setting (indoors or outdoors) for PPA.

We evaluated 40 held-out triplets in four participants. We report means ± SD across participants. Replacing fMRI-derived EBA tokens with image-derived EBA tokens shifted the decoded posture score more than the corresponding FBA edit in all four participants (mean EBA − FBA difference +0.052 ± 0.011). EBA edits preferentially shifted posture, and PPA edits preferentially shifted setting. The region-by-attribute interaction was positive in all four participants (mean +0.118 ± 0.021). Our conclusion concerns how these edits change the frozen decoder’s output.

2. What does the shared ROI-token representation enable?

Prior work has used brain activity to predict caption embeddings. Our contribution is that the image and fMRI pathways produce corresponding tokens for each ROI, and the same frozen readout accepts tokens from either pathway. Starting from an fMRI-derived representation, we can replace one region’s token with its image-derived counterpart while keeping the other tokens and the readout fixed. This lets us test how much that replacement shifts a specific decoded attribute, such as posture or setting. We can take those tokens from different donor images to combine edits, or from the viewed image itself to test how much caption information the replacement recovers.

Combining edits from two donor images. Consider a source scene showing someone sitting indoors. We can take EBA tokens from an image of someone standing indoors and PPA tokens from an image of someone sitting outdoors. Neither donor shows the combined target of standing outdoors. Across the 40 held-out triplets, we scored each edit relative to its intended target. With both edits applied, the average posture and setting scores each shifted in the intended direction in all four participants.

We also tested image-predicted voxel replacements using the same source and donor images. With the equal-PC ridge readout, the voxel edits produced a region-by-attribute interaction in the same direction. The interaction was larger for token edits in all four participants (mean token − voxel difference +0.027 ± 0.009).

Replacing TO tokens from the same viewed image. We replaced four TO tokens decoded from fMRI with the corresponding tokens from the image the participant had viewed. Caption R@10 rose from 0.541 to 0.864. On average, the gain in R@10 was 0.231 larger for TO replacement than for size-matched random-ROI replacement, reproducing the result in Appendix A.17.

For the matched voxel test, we replaced the corresponding TO voxels with calibrated Stage-1 predictions from the viewed image. R@10 changed by +0.002 with raw ridge and −0.004 with equal-PC ridge. Replacing TO voxels with predictions from the wrong image impaired decoding. The voxel decoders were therefore sensitive to the replacement, even though predictions from the correct image yielded little change in retrieval.

We thank the reviewer again for pushing this clarification; we believe it substantially sharpens the paper’s contribution for readers.

Thank you for the additional experiments and clarification. I appreciate the efforts to strengthen the evidence and clarify the value of the proposed framework.

My main concern remains how the findings build on existing neuroscientific knowledge. The response does not yet sufficiently explain what was already known, what remained unresolved, and what new understanding the present work provides. It may help to organize the introduction and discussion around the scientific question being addressed and what answering it adds to existing knowledge.

Although the revisions strengthen the methodological contribution, this central concern remains unresolved. I will therefore maintain my rating of 2.

Reviewer_PXXgOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BrainInterv, a framework for studying the semantic contributions of visual cortical regions through interventions on ROI-supervised representations. The authors construct ROI-specific tokens from visual features using fMRI supervision, perform donor-token replacement to examine changes in semantic retrieval, and extend the analysis to tokens reconstructed from measured fMRI.

Overall, I find this a technically sound and carefully evaluated paper. My main concern is that the scientific findings are largely consistent with existing knowledge, making the overall contribution somewhat incremental.

I am open to revising my rating upward if the authors can satisfactorily address my questions, particularly regarding the readout dependence and direct ROI interventions.

Critical strengths
  1. A coherent and interpretable framework. The paper provides a clear pipeline connecting visual features, ROI-specific fMRI supervision, and semantic decoding.
  2. Thoughtful experimental design and controls. The paper includes size-matched random token interventions, continuous interpolation between original and donor tokens, and separate probes for low-level visual properties. These experiments help characterize the intervention behavior and distinguish semantic effects from effects on lower-level visual features.
  3. Clear presentation and analysis. The methodology is well explained, The quantitative results are complemented by qualitative examples.
Critical weaknesses

Limited novelty of the scientific findings. The main conclusion—that higher-level visual regions contribute more strongly to semantic decoding, while early visual regions are more associated with low-level visual properties—is largely consistent with established knowledge of the visual processing hierarchy. While the experiments validate the proposed framework, I did not find particularly surprising or previously unknown insights into cortical functional organization.

Questions for authors
  1. Dependence on the semantic readout. The paper finds that high-level ROI tokens have a much larger effect on semantic retrieval than early visual tokens. However, the attention readout assigns substantially greater weight to high-level tokens, and the authors report that replacing it with mean pooling changes the intervention ordering despite comparable retrieval performance. This raises the possibility that the observed regional differences are driven primarily by how the readout weights the tokens, rather than by differences in the information represented by those tokens. Could the authors provide the ROI-wise intervention results under mean pooling and clarify how much of the reported regional hierarchy can be attributed to the ROI representations themselves versus the learned readout?

  2. Direct ROI intervention baseline. Since the voxel-to-token mapping consists of independent ROI-specific networks, why not directly perform ROI-wise donor replacement on the fMRI inputs of a conventional decoder? How would the resulting intervention effects compare with those obtained using the proposed ROI-token representation? Such a comparison would help clarify the additional analytical value of the intermediate representation.

4 comments
AuthorsOpenReview ↗

Dependence on the semantic readout

We thank the reviewer for pointing out this precise point in the writing. When we wrote that mean pooling “changes the intervention ordering,” we meant the fine-grained ranking among individual ROI groups (e.g. which of the 50 ROIs stand out in absolute donor transfer), not the coarser high-level versus early contrast that is the paper’s main claim.

What changes under mean pooling. With uniform weights, a small set such as Kastner TO (4/50 slots) receives only ~8% of the pooled mass. Absolute donor R@10 for TO therefore drops sharply relative to the attention readout (ROI-tuned image tokens, Kastner native: 0.164 → 0.029), and TO no longer clearly exceeds early in raw R@10 under that uniform pool. Other small parcels similarly compress. So the per-ROI absolute picture does depend on how Stage‑2 aggregates tokens.

What does not change. The high-level set still moves retrieval toward the donor far more than early visual tokens, on every atlas we tested, with retrieval under mean pooling still high (0.91–0.92 vs attention 0.94–0.95):

AtlasHigh-level donor R@10 (z vs null)Early donor R@10 (z vs null)Lateral site (z)
Kastner (paper, fsaverage)0.415 ± 0.047 (z 4.4–8.3)0.017 (z −1.7 to −0.9)TO 0.028 (z 4.3–6.8, 4/4)
Kastner (native)0.360 ± 0.054 (z=5.7, 4/4)0.029 (z=−1.7)TO z=3.1, 4/4
NSD streams0.273 ± 0.074 (z=5.1, 4/4)0.018 (z=−1.2)lateral z=3.4
NSD fLoc+pRF0.835 ± 0.008 (category)0.028 (z=−1.9)bodies z=4.4, 4/4

Under mean pooling, TO remains above its size-matched null (z=3.1, 4/4) even though its absolute R@10 is small. The high-level − early gap stays positive in 4/4 subjects under attention, mean pooling, and direct fMRI ridge interventions. Content vs routing (brief). Attention mass does not predict transfer: on Kastner(native), parietal mass 0.255 > TO 0.203, yet parietal transfer is 0.039 (z=−0.9) while TO is 0.164 (z=12) under attention. With the Stage‑2 recipe fixed, scrambling only the ROI↔voxel Stage‑1 assignment abolishes the fLoc bodies effect (0.403 → 0.028). So readout weighting scales magnitudes of small groups; the high-level ≫ early sign tracks token content and anatomical supervision.

We will revise the point in the mean-pooling sentence to distinguish (i) changes in fine-grained / absolute per-ROI effects from (ii) the high-level versus early ordering under a uniform readout.

AuthorsOpenReview ↗

Direct ROI intervention

We thank the reviewer for suggesting this direct comparison. We performed the ROI-wise donor replacement on fMRI inputs of a conventional decoder. We then ask what the ROI-token interface adds beyond that baseline, including the reviewer’s follow-up: intervene on the voxel inputs of the voxel-to-token invertor itself.

1. Direct fMRI ROI-block interventions (conventional decoder)

Ridge decoders map measured fMRI → Qwen caption space (no ROI-tuned image tokens, no image-trained Stage‑2). At test time we replace an ROI group’s voxels from a source trial with the donor’s and re-decode. Equal-capacity uses k=32 PCs per ROI; raw swaps literal voxel blocks. Same canonical 1000 pairs; size-matched nulls.

AtlasDecoderBase R@10High-level / category donor R@10Early donor R@10High − early CI > 0
Kastner (paper, fsaverage)equal / raw0.41 / 0.410.161 / 0.1470.024 / 0.0284/4
Kastner (native)equal / raw0.40 / 0.390.158 / 0.1390.026 / 0.0264/4
NSD fLoc+pRFequal / raw0.46 / 0.530.348 / 0.449 (category)0.019 / 0.0224/4
NSD streamsequal / raw0.41 / 0.520.057 / 0.088 (ventral)0.020 / 0.0224/4

Direction matches the token analyses: high-level / category transfers donor identity; early does not (and sits below size-matched nulls). Category-conditioned equal-PC swaps on fLoc further give bodies→person 0.042 (> early in 4/4); places→scene 0.041; faces→face 0.014.

2. What the ROI-token representation adds

Direct fMRI decoderROI-tuned image tokens / decoded ROI tokens
Base retrieval R@100.39–0.530.94–0.95 (image) / 0.51–0.63 (task-aligned voxel-to-token invertor)
Small sitesTO ≈0.033 (z≈3); bodies 0.06–0.11TO 0.164 (z=12); bodies 0.403 (z=9)
Equal capacity per ROIneeds PCA512-d by design
Same intervention on image and brainnoyes (shared Stage‑2)
Dose + low-level probesnot defined on ridge featuresyes

So the conventional decoder confirms the sign of the hierarchy; the ROI-token interface increases sensitivity (~5–7× on TO / bodies) and supports matched image↔brain edits under one frozen readout.

3. Voxel-input interventions

Because each ROI has its own mapper, we also edit the fMRI betas that feed the voxel-to-token invertor, then decode with frozen Stage‑2:

  • Equivalence. Whole-ROI voxel swap = token swap (EBA 0.176 = 0.176).
  • Dose. EBA 10% / 50% / 100% voxels → donor R@10 0.015 / 0.044 / 0.176.
  • Selective voxels. Top 25% person-selective in EBA ≫ random 25% (0.036 vs 0.007, 4/4).
  • Data-driven localizer. Top-N person-selective voxels anywhere land 63–67% inside anatomical EBA and nearly match EBA transfer.
  • Lesion. EBA knock-out ΔR@10 −0.21 to −0.26 (z −6 to −13 vs random lesions).
  • Single-trial donors. EBA still transfers (0.155, z 8.9, 4/4).

The same lateral ≫ early pattern holds for the task-aligned invertor, the reconstruction-pretrained invertor, and brain-native tokens.

Because the voxel-to-token invertor is a set of independent per-ROI MLPs, replacing all voxels of an ROI and then running the invertor is mathematically the same as replacing that ROI’s brain token before Stage‑2. We verified this numerically (e.g. fLoc EBA donor R@10 0.176 either way; Kastner TO 0.052 either way). Whole-ROI voxel→invertor→Stage‑2 interventions therefore match the decoded ROI token intervention column on the same 1000 pairs; what the voxel-input path uniquely adds is analyses that tokens alone cannot define—partial dose, concept-selective voxel subsets, data-driven localizers, virtual lesions, and single-trial donors—while preserving the same lateral ≫ early pattern.

Overall, direct ROI-block interventions reproduce high-level ≫ early without ROI-tuned image tokens. The ROI-token representation is not required for the direction, but it is more sensitive and provides a shared image/brain intervention space. Editing the invertor’s voxel inputs recovers the same anatomy and adds dose, selectivity, localizer, lesion, and single-trial evidence that a plain ridge decoder does not.

We also address the reviewer's concern about novelty in our response to NHb5 below, where we discuss the contributions relative to prior work and highlight the additional findings on lateral visual regions and the shared image–brain intervention space.

We hope our responses addressed the reviewer comments and that you consider a more positive evaluation of our work!

Thank you for the detailed rebuttal and the substantial set of new experiments. I have read the revised manuscript (in particular §5.3, §5.6–5.8, Tables 3–6, and Appendices A.20 and A.25–A.28) together with the responses to the other reviewers. Both of my questions have been addressed, and I am raising my rating from 2 to 3.

Readout dependence. My concern was that the High-level ≫ Early effect might reflect how the attention readout weights tokens rather than what the tokens encode. This is now resolved. The ordering survives mean pooling on all four atlases (Table 27), where every slot shares the same projection. Attention mass does not predict transfer: parietal receives more mass than TO but transfers below its null. The ordering also appears in direct fMRI ridge decoders, which have no token weighting. I read the result as follows: the sign of the High-level vs. Early contrast comes from the representations, while the magnitude of small-group effects (e.g., TO: 0.164 → 0.029) partly comes from the readout. I encourage the authors to frame their claims at this level in the final version.

Direct ROI interventions. Ridge, PCA+ridge, and the image-free 512-d token model all reproduce the ordering in 4/4 subjects without DINOv3 or the image-trained readout (Table 3). This is the cleanest evidence in the revision.

Novelty. The coarse hierarchy remains expected, but two results go beyond it. Early ROIs are encoded well yet have weak intervention effects. The scrambling control abolishes selectivity while preserving encoding and retrieval. Overall, I see BrainInterv as a sound, well-controlled methodological contribution, and I have updated my rating to 3.

We greatly appreciate your positive evaluation of our work and thank you again for your detailed feedback and engagement with our work! We will frame the claims as suggested in the final version.

Reviewer_TZiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BrainInterv as an approach to probe the selectivity of different visual brain areas, which is an interesting direction and meaningful topic given the popularity and advancement of neural encoding & decoding. BrainInterv involves several design and lightweight components for semantic readout and downstream decoding tasks. The model's performance evaluation is comprehensive.

Critical strengths
  1. The paper identifies what can be recovered from a region and what actually drives decoding. It solves the second with interventions, which are borrowed from interpretability (activation patching), and complements standard encoding and decoding analyses in deep learning-related neuroscience research.

  2. The 50 ROI-relevant design, including queries, blocks, and heads, is neuroscience-informed and makes sense. Specifically, the novel part is that they share nothing except the input projection, so each token is shaped only by its own ROI’s loss.

  3. In terms of encoding and decoding performance, 50 ROI tokens keep most of the backbone’s semantic content while still predicting held-out fMRI at high fidelity.

Critical weaknesses
  1. The main analysis uses tokens computed from images, while fMRI only supervises Stage 1. The decoded tokens are trained to predict image tokens and then fine-tuned via the same frozen readout, so the fMRI-side ordering may be inherited from the visual encoder's prior. The mapper control without fine-tuning rules out fine-tuning as the cause, but not the image-trained readout. Swapping ROI voxel blocks in a decoder trained directly on fMRI would be a more direct test.

  2. A group retinotopic atlas is used instead of NSD’s subject-specific pRF/floc ROIs, so "high-level" leaves out much of the category-selective cortex. In addition, some groups are nested (High-level contains TO+LO), so their effects aren’t independent.

  3. Many trained components are only for evaluation or visualization. The core claim needs only the Stage 1 ROI encoder, a fixed readout, and the token swaps with their nulls. On top of these, the paper trains: 1) a CLIP-style caption decoder plus SD generation, used only for figures. Its training is barely described. It is conditioned only on text, so it can’t show the low-level effects central to the dissociation. 2) task-aligned training phase for the voxel-to-token mapper. It raises retrieval performance but isn’t needed for the main ordering, which already holds without it.

Questions for authors

see weaknesses

4 comments
AuthorsOpenReview ↗

Interventions on the fMRI tokens

We thank the reviewer for suggesting this test. Below we provide results for swapping ROI voxel blocks inside a decoder trained directly on fMRI. We agree this is the most direct way to ask whether the high-level ≫ early ordering is truly in the measured signal. We ran that experiment on the paper atlas, provided with statistics for every decoder variant, and added a matched brain-native token path with the same 512-d width as the paper’s image/brain tokens. Atlases. Paper atlas = Kastner 2015 on fsaverage (50 ROIs). For the revision we also re-ran on three native-surface atlases: (1) Kastner 2015 native, (2) NSD subject-specific pRF + fLoc (faces / bodies / places / words; mutually exclusive), (3) NSD streams.

1. Brain-native 512-D tokens on the paper atlas, alongside Ridge and PCA+Ridge baselines

To match the paper’s ROI-tuned image/decoded ROI token width without bringing in DINOv3, we ran a new brain-native path on the same fsaverage Kastner-50 betas used in the submission: z-scored voxels → one linear 512-d token per ROI (no PCA, no image Stage-1, no ROI-tuned image-token targets) → AttnHead trained with caption InfoNCE only. Selection on the train9k holdout; shared1000 test; canonical 1000 pairs; size-matched nulls (n_null=1000).

Path (paper Kastner fsaverage)Feature / token sizeBase R@10High-level donor R@10High-level z vs nullEarly donor R@10Early z vs nullHigh − early CI > 0
Raw ridge#voxels / ROI0.4090.1472.18 – 6.030.028−1.28 – −0.724/4
Equal-capacity ridge32 PCs / ROI0.4070.161 ± 0.0383.0 – 7.00.024−1.58 – −1.004/4
Brain-native linear512-d / ROI0.4960.1782.36 – 4.180.027−1.55 – −1.024/4

On the paper atlas, high-level ≫ early holds for raw voxels, equal-capacity PCs, and equal-width 512-d brain-native tokens.

2. Scramble control: is the pattern just the vision model’s prior?

If the ordering came only from DINOv3 or the readout, changing which voxels supervise which ROI should not matter. So we kept images, frozen DINOv3, architecture, and Stage-2 training fixed, and only scrambled the ROI→voxel assignment used to supervise Stage 1. ROI assignment stay anatomical; each ROI token is supervised by another ROI’s voxels (stem-level derangement). We run this for fLoc, native Kastner, and the paper Kastner fsaverage atlas.

fLoc (native pRF+fLoc):

AssignmentEncoding rReadout R@10Bodies donor R@10 (z)Early-named donor R@10Cross-subject fingerprint ρ
Real0.4210.9470.403 (z=9.2, 4/4)0.0160.99
Scrambled0.4210.9420.028 (z=−0.2, 0/4)0.1550.63

Kastner 2015 (native; same Stage-1 / Stage-2 recipe):

AssignmentEncoding rReadout R@10TO donor R@10 (z)High-level donor R@10 (z)Early-named donor R@10 (z)
Real0.413 ± 0.0480.943 ± 0.0040.164 (z=12.0, 4/4)0.804 (z=3.8)0.018 (z=−0.83)
Scrambled0.412 ± 0.0480.948 ± 0.0050.018 (z=−0.27, 0/4)0.295 (z=0.49)0.139 (z=+0.72)

Kastner 2015 (paper, fsaverage):

AssignmentEncoding rReadout R@10TO donor R@10Early donor R@10
Real0.411 ± 0.0610.942 ± 0.0070.339 ± 0.1670.017 ± 0.002
Scrambled0.413 ± 0.0590.947 ± 0.0080.020 (z=−0.22, 0/4)0.152

Encoding and readout stay essentially unchanged. The selective high-level effects disappear (fLoc bodies; Kastner TO / high-level), and early ROIs gain donor transfer once they are supervised by other regions’ voxels. The same pattern holds on the paper fsaverage Kastner scramble: TO falls to 0.020 (z=−0.22, 0/4) while early ROIs rise to 0.152. So the intervention structure follows the fMRI ROI↔voxel assignment, not the ROI names or the vision prior alone.

Together, these results show that the stronger effect of high-level than early-visual ROI interventions also appears in a decoder trained directly on measured fMRI, without image-derived ROI tokens or an image-trained readout. This provides an additional check on the regional pattern observed in BrainInterv, whose main analyses use the shared representation space.

AuthorsOpenReview ↗

Testing on other Atlases

We appreciate the request for subject-specific NSD ROIs and for clearer, non-nested group reporting. In the revision we treat this as a complementary test of the main claim, not a replacement of the Kastner analysis. High-level/TO/LO in the original paper were predefined union of named ROIs (defined in the text); to avoid any ambiguity about nesting, the revision leads with disjoint sites and adds the atlases below.

Complementary atlases (same Stage-1 / Stage-2 / canonical 1000-pair intervention protocol; native surface):

  1. NSD subject-specific pRF + fLoc (faces / bodies / places / words; mutually exclusive),
  2. Kastner 2015 native,
  3. NSD streams,

ROI-tuned image tokens transplants (attention):

AtlasReadout R@10High-level / category donor R@10Early donor R@10High − early CI > 0
NSD fLoc+pRF0.9470.929 ± 0.004 (category)0.016 ± 0.0044/4
Kastner (native)0.9430.804 ± 0.0460.018 ± 0.0024/4
NSD streams0.9420.752 ± 0.1710.019 ± 0.0034/4

Disjoint sites (primary revision reporting):

AtlasSitekDonor R@10 (z vs null)
KastnerTO4/500.164 (z=12.0, 4/4)
fLocbodies~5/400.403 (z=9.2, 4/4)
streamslateral2/140.325 (z=4.7, 4/4)

Across subject-specific fLoc+pRF, native Kastner, and streams, the same story holds: early swaps do not transfer caption identity; some high-level / category-selective / lateral subset does.

We appreciate the feedback on the framing of the results.

1. Caption decoder + SD generation

We will add more details on how we used the SD to generate the images. They illustrate decoded language / images after a transplant; and are not used to score high-level ≫ early.

2. Task-aligned voxel-to-token

As the reviewer pointed out the task alignemtn does not change the ordering, but we wanted to further show that the retrieval can be improved. We will make this clear in the revision.

We hope our responses addressed all your questions and are happy to address any further questions!

I appreciate the author's effort in providing additional results and clarification during the rebuttal period, and my first two concerns are resolved.

However, my third concern is about framing. It is why the paper trains components irrelevant to its claim, which needs only stage 1, a fixed readout. Additional components trained for evaluation or visualization purposes are a post-hoc methodology, but I don't feel it is consistent with the main focus of the paper. Therefore, I decided to keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all the reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the strengths of BrainInterv, particularly its neuroscience-informed, ROI-specific design, the experimental controls, and the consistency of the intervention effects across participants.

BrainInterv provides a region-indexed framework for studying how visual information associated with different brain regions contributes to semantic decoding. The framework uses representations shaped by region-specific brain activity, allowing us to investigate the semantic effects of visual information encoded along the visual pathway. We intervene on these representations by replacing a selected region's source-image representation with the corresponding donor-image representation and measuring how the decoded output changes. This allows us to compare intervention effects across regions and assess whether they shift the decoded representation toward the donor's semantic representation.

The reviewers raised several important questions about the interpretation of these interventions, particularly whether the observed regional effects are also present in measured fMRI, how well they generalize across anatomical definitions, and to what extent they depend on the learned representations and semantic readout. We also appreciate the suggestions to clarify the relationship to prior work and better distinguish the main findings from the supporting components of the framework. These points helped us strengthen the experimental evaluation and make the scope of our contributions clearer.

In response, we conducted additional experiments and expanded several parts of the manuscript:

  1. Direct interventions on measured fMRI: We added ROI-level interventions using conventional Ridge and PCA+Ridge decoders, alongside an image-free pathway with brain-native 512-dimensional tokens. These experiments provide an independent test of the regional effects without relying on image-derived ROI representations or the original image-trained readout (Table 3).

  2. Generalization across anatomical atlases: We extended the analysis beyond the original Kastner atlas to native-surface Kastner, subject-specific NSD pRF+fLoc, and NSD streams. We also clarified the distinction between individual ROIs and their unions, with greater emphasis on disjoint regions (Table 6; §5.2).

  3. Role of anatomical supervision: We introduced ROI-to-voxel assignment scrambling controls to examine whether the intervention patterns arise from the visual backbone or reflect region-specific fMRI supervision. The selective regional effects disappear under scrambling despite largely preserved encoding and retrieval performance (Table 5). We expanded the Discussion to explain this result and its implications.

  4. Dependence on the readout and training procedure: We further examined mean-pooling interventions and clarified that the readout can affect the magnitude and fine-grained ordering of individual ROI effects, while the broader high-level versus early-visual ordering remains consistent (§5.3). We also clarified the reconstruction-first training procedure and the optional role of task-aligned fine-tuning (§3.3).

  5. Comparison with existing approaches and interpretation of the results: We added a comparison between direct fMRI interventions and ROI-tuned image-token interventions (Table 4), expanded the discussion of related work, and clarified what the intervention framework adds beyond encoding, stimulus optimization, and ROI masking. We also made clearer that caption and image generation are supporting visualizations, while the main findings are evaluated through retrieval and probes (§5.9 and Discussion).

We have incorporated these analyses and clarifications into the revised manuscript and provide further details, including experimental results and controls, in our responses to each reviewer below. For clarity, we have highlighted the changes and additions in the revised manuscript in blue.

We thank the reviewers again and are happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.