BrainInterv: In Silico Interventions for Mapping Semantic Selectivity in the Brain

Characterizing the selectivity of brain areas is important for understanding their underlying computation. Here we propose an in silico approach that allows us to address this challenge by performing interventions and studying their effect on a downstream decoding task. Brain-tuned visual representations are created by routing and transforming self-supervised visual features into region-of-interest (ROI) tokens, each supervised by functional magnetic resonance imaging (fMRI) responses from a cortical area. The resulting ROI-tuned image tokens support high-accuracy 1000-way semantic retrieval through a lightweight readout to a text-embedding space, reaching $R@10=0.942\pm 0.007$. We further extend the decoding task by predicting the ROI tokens from measured fMRI responses using a voxel-to-token mapper. These decoded ROI tokens achieve $R@10=0.541\pm 0.085$ under the same frozen image-trained semantic readout. We then perform in silico interventions on the model by transplanting ROI tokens across images for the retrieval task and observe that high-level token interventions produce substantially greater donor-semantic transfer than early-visual tokens, whose effects are stronger on lower-level visual probes. This intervention pattern also holds for both ROI-tuned image tokens and decoded tokens, and in different subjects. Overall, the proposed approach provides an intervention-based method to probe the selectivity of different visual brain areas, complementing current encoding- and direct decoding-based methods.