- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for examining how selective replacement of ROI-associated intermediate representations affects semantic decoding. First, frozen vision-model features are transformed into ROI-tuned image tokens, supervised through prediction of the corresponding regional fMRI responses. A semantic readout is then trained to map these tokens to image-caption embeddings. The authors replace tokens associated with selected ROI groups with the corresponding tokens from another image and measure how strongly the output of the frozen readout shifts toward the donor image’s semantics. They additionally train ROI-specific mappers to recover the same token representation from measured fMRI and perform analogous interventions on these decoded tokens. Replacing higher-level visual ROI tokens produces stronger donor-semantic transfer than replacing early visual ROI tokens. Conversely, early visual token interventions have relatively stronger effects on probes of lower-level visual features.
Critical strengths
- The authors compare predefined ROI groups against random subsets matched for token count, vary the donor mixing coefficient continuously, evaluate alternative original–donor pair sets. The consistent High-level–Early difference across all four participants supports the robustness of the intervention effect within the tested model.
- Comparing intervention effects across different prediction targets broadens the analysis beyond semantic retrieval alone.
Critical weaknesses
Prior work already investigates regional selectivity through encoding-model weight analyses (Huth et al., 2016), stimulus optimization (BrainDiVE; Luo et al., 2023), and ROI masking during decoding (NeuroAdapter; Feng et al., 2026). Although the paper acknowledges related approaches, it does not clearly identify which unresolved limitation ROI-token transplantation addresses or demonstrate what additional understanding it provides.
The main finding—stronger semantic effects for higher-level tokens and stronger low-level visual effects for early tokens—is broadly consistent with established visual organization. The shared image/fMRI token space is a methodological distinction, but the experiments do not establish a new scientific conclusion or a clear analytical advantage over existing approaches. Reproducing a familiar pattern is useful validation, but does not by itself establish the added value of the proposed framework.
This concern is reinforced by the readout comparison: mean pooling achieves similar retrieval performance but does not preserve the intervention ordering. The authors acknowledge this dependence, which is not itself a methodological flaw. However, it remains unclear what the intervention results reveal about brain-region selectivity beyond how the chosen semantic decoder uses its inputs.
Questions for authors
What additional scientific insight or analytical capability does the proposed method establish?
4 comments
Thank you again for the additional experiments, which substantially strengthen the evidence for robustness. The findings in TO, body-selective regions, and the lateral visual stream, together with the category-stratified analyses, also go beyond a simple high-level versus early-visual comparison.
For work seeking new neuroscientific insights through a new method, careful positioning relative to prior findings is particularly important. Given the extensive neuroscience and machine-learning literature on these regions, especially body-selective cortex, could you clarify what unresolved question these results address and how they refine existing knowledge? It would help to distinguish framework validation from additional neuroscientific insight.
Could you also provide a concrete example of a scientific conclusion or analytical advantage enabled or strengthened by the shared ROI-token representation, beyond what is readily obtainable with simpler approaches such as direct ROI-wise interventions on fMRI inputs?
Thank you for raising these two questions. We agree that recovering known regional selectivity mainly validates the framework. To test what else we could learn, we examined whether replacing tokens from different regions changed decoded posture and setting in scenes that all contained a person. We also compared token and voxel edits using the same images. We fixed the stimulus list in advance and kept the models and readouts frozen.
1. Validation versus additional insight
We asked whether replacing EBA tokens shifted decoded posture (sitting or standing) more than replacing FBA tokens. We also asked whether edits to EBA and PPA preferentially shifted different attributes of the same scene: posture for EBA and setting (indoors or outdoors) for PPA.
We evaluated 40 held-out triplets in four participants. We report means ± SD across participants. Replacing fMRI-derived EBA tokens with image-derived EBA tokens shifted the decoded posture score more than the corresponding FBA edit in all four participants (mean EBA − FBA difference +0.052 ± 0.011). EBA edits preferentially shifted posture, and PPA edits preferentially shifted setting. The region-by-attribute interaction was positive in all four participants (mean +0.118 ± 0.021). Our conclusion concerns how these edits change the frozen decoder’s output.
2. What does the shared ROI-token representation enable?
Prior work has used brain activity to predict caption embeddings. Our contribution is that the image and fMRI pathways produce corresponding tokens for each ROI, and the same frozen readout accepts tokens from either pathway. Starting from an fMRI-derived representation, we can replace one region’s token with its image-derived counterpart while keeping the other tokens and the readout fixed. This lets us test how much that replacement shifts a specific decoded attribute, such as posture or setting. We can take those tokens from different donor images to combine edits, or from the viewed image itself to test how much caption information the replacement recovers.
Combining edits from two donor images. Consider a source scene showing someone sitting indoors. We can take EBA tokens from an image of someone standing indoors and PPA tokens from an image of someone sitting outdoors. Neither donor shows the combined target of standing outdoors. Across the 40 held-out triplets, we scored each edit relative to its intended target. With both edits applied, the average posture and setting scores each shifted in the intended direction in all four participants.
We also tested image-predicted voxel replacements using the same source and donor images. With the equal-PC ridge readout, the voxel edits produced a region-by-attribute interaction in the same direction. The interaction was larger for token edits in all four participants (mean token − voxel difference +0.027 ± 0.009).
Replacing TO tokens from the same viewed image. We replaced four TO tokens decoded from fMRI with the corresponding tokens from the image the participant had viewed. Caption R@10 rose from 0.541 to 0.864. On average, the gain in R@10 was 0.231 larger for TO replacement than for size-matched random-ROI replacement, reproducing the result in Appendix A.17.
For the matched voxel test, we replaced the corresponding TO voxels with calibrated Stage-1 predictions from the viewed image. R@10 changed by +0.002 with raw ridge and −0.004 with equal-PC ridge. Replacing TO voxels with predictions from the wrong image impaired decoding. The voxel decoders were therefore sensitive to the replacement, even though predictions from the correct image yielded little change in retrieval.
We thank the reviewer again for pushing this clarification; we believe it substantially sharpens the paper’s contribution for readers.
Thank you for the additional experiments and clarification. I appreciate the efforts to strengthen the evidence and clarify the value of the proposed framework.
My main concern remains how the findings build on existing neuroscientific knowledge. The response does not yet sufficiently explain what was already known, what remained unresolved, and what new understanding the present work provides. It may help to organize the introduction and discussion around the scientific question being addressed and what answering it adds to existing knowledge.
Although the revisions strengthen the methodological contribution, this central concern remains unresolved. I will therefore maintain my rating of 2.
Relation to prior work
We thank the reviewers for their comments. Our aim is to provide a region-indexed intervention that tests whether a site’s representation is sufficient to redirect decoded semantics toward a specific donor, under matched controls, on both ROI-tuned image tokens and decoded ROI tokens.
Capability relative to prior tools
Encoding, optimization, and masking remain valuable; they answer different questions. Intervention adds a donor-specific sufficiency test that those tools do not provide.
Three results beyond a generic high-level > early summary
Encoding strength ≠ downstream contribution. On fLoc/streams, early encoding correlations are at least as strong as high-level ones, yet early interventions fall below size-matched random-slot nulls (z −0.6 to −1.9) across atlases, readouts, and token sources. Predictivity of a voxel does not imply that swapping that region rewrites caption identity.
Editable caption semantics concentrate in lateral occipitotemporal cortex, converging across independent atlases (same 1000 pairs; donor R@10 / z vs null):
Readout dependence (NHb5 CW3 / PXXg Q1). When we wrote that mean pooling “changes the intervention ordering,” we meant the fine-grained / absolute per-ROI picture (e.g. TO’s raw donor R@10 shrinks under uniform weights), not the high-level versus early contrast. That coarser ordering survives mean pooling (z 5.1–5.7, 4/4) and also appears in direct fMRI ridge interventions with no learned token weighting. Attention mass likewise fails to predict transfer (parietal mass > TO, yet parietal transfer below null while TO transfers strongly). Full ROI-wise mean-pool results are in our reply to PXXg Q1; the manuscript framing will be elaborated accordingly.
The direct fMRI intervention experiments also help clarify what the ROI-token representation adds beyond conventional decoders. While both approaches recover the broader high-level versus early-visual pattern, the ROI-token framework provides greater sensitivity to interventions in smaller regions, such as TO and body-selective areas. More importantly, it enables comparable interventions on image-derived and fMRI-derived representations through a shared semantic readout, along with more fine-grained analyses of regional contributions (results are in our reply to PXXg Q2).
Relative to prior regional tools, ROI-token interventions provides a donor-specific sufficiency test in a shared image/brain space. Relative to textbook hierarchy, the new content is that downstream contribution is anatomically concentrated, supervision-dependent, and dissociable from encoding strength and from low-level visual transfer.
In light of these results, we hope you might view our work more favorably. Please feel free to let us know if you have any additional questions!