MindBank: Data-Efficient Cross-Subject fMRI Decoding by Leveraging Shared Semantic Structure Across Source Subjects

Cross-subject functional magnetic resonance imaging (fMRI) visual decoding aims to adapt decoders pretrained on source subjects to a new target subject using only a limited amount of paired fMRI–image data. However, limited data and large differences in fMRI responses across individuals can cause the target representation to lose semantic information during adaptation. We recognize that despite individual variability in fMRI responses, the underlying structure of visual semantics is shared across individuals. Consistent with this view, we observe that the decoded target representation is more similar to the average decoded representation across source subjects. We propose MindBank, which addresses the challenges by leveraging this similarity and the variability through two complementary mechanisms. (1) MindBank aggregates representations from frozen source decoders into a shared semantic reference and adaptively integrates the reference with target features. This mechanism keeps target adaptation consistent with the semantic structure shared across subjects. (2) MindBank exploits variations across the source representations to generate controlled perturbations of target representations. The perturbations expand the coverage of the target feature space and alleviate the limitations imposed by scarce target data. Experiments on the Natural Scenes Dataset (NSD) under a unified one-hour adaptation protocol demonstrate that Mind Bank uses the fewest trainable parameters, yet outperforms state-of-the-art methods in both retrieval and reconstruction. Code is included in the supplement and will be released.