Which Stimuli to Scan? Small Image-Set Selection for Image-to-fmri Encoder Training

Functional MRI (fMRI) offers a powerful way to study human visual perception. Recently, Image-to-fMRI encoders trained on hundreds of participant-specific image–fMRI pairs were shown to accurately predict the participant's fMRI response to new images. However, collecting such extensive training data is often infeasible due to high scanning costs and the limited availability of participants. Achieving higher accuracy with smaller training sets would substantially broaden the applicability of these encoding models, facilitating both large-scale studies of population variability and studies of how visual responses change with experience or intervention. This raises a fundamental question: If we can measure fMRI responses to only a small number of images, which images should we choose to scan? We show that image-sets selected to provide high coverage of the brain-response space of one subject (with many fMRI scans) generalize well to new subjects and significantly improve their image-to-fMRI encoding in the low-data regime (tens of scanned images). We explore image-set selection both in brain-response space and image representation space, and show that both outperform random image sampling, with brain-response-based selection providing particularly strong gains. Moreover, image-sets selected using NSD neural representations transfer well to other fMRI datasets (e.g., LAION-fMRI, 3T Generic Object Decoding dataset). These results provide a step toward more data-efficient stimulus acquisition for fMRI experiments.