acceptodds
Under review as a conference paper at ICLR 2027

Which Stimuli to Scan? Small Image-Set Selection for Image-to-fmri Encoder Training

Abstract

Functional MRI (fMRI) offers a powerful way to study human visual perception. Recently, Image-to-fMRI encoders trained on hundreds of participant-specific image–fMRI pairs were shown to accurately predict the participant's fMRI response to new images. However, collecting such extensive training data is often infeasible due to high scanning costs and the limited availability of participants. Achieving higher accuracy with smaller training sets would substantially broaden the applicability of these encoding models, facilitating both large-scale studies of population variability and studies of how visual responses change with experience or intervention. This raises a fundamental question: If we can measure fMRI responses to only a small number of images, which images should we choose to scan? We show that image-sets selected to provide high coverage of the brain-response space of one subject (with many fMRI scans) generalize well to new subjects and significantly improve their image-to-fMRI encoding in the low-data regime (tens of scanned images). We explore image-set selection both in brain-response space and image representation space, and show that both outperform random image sampling, with brain-response-based selection providing particularly strong gains. Moreover, image-sets selected using NSD neural representations transfer well to other fMRI datasets (e.g., LAION-fMRI, 3T Generic Object Decoding dataset). These results provide a step toward more data-efficient stimulus acquisition for fMRI experiments.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Image-to-fMRI encoding, Visual neuroscience, Stimulus selection, Brain representations, Data-efficient learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BEbWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the important problem of how to allocate limited neuroimaging scanning budgets to best select visual stimuli that are likely to be most informative for estimating encoding models in new participants. Existing datasets often sample thousands of images over dozens of hours, which is infeasible in the general population and with typical research budgets. The main contributions here are testing a few principled approaches for selecting a limited set of images to show a new participant, and evaluating the extent to which these selections may work in silico by testing within-dataset and across-dataset prediction accuracy under these constraints. No new human participant data is acquired, but the experiments on existing datasets are interesting. The primary conditions tested are stimulus selection objective functions based on image properties (in CLIP/equivalent spaces) and those based on predicted brain responses. Altogether, the authors conclude that optimizing stimulus selection based on predicted brain responses offers a meaningful advantage to previous approaches, and can lead to effective encoding model estimation with substantially less data acquisition.

Critical strengths
  1. The problem addressed by this work is of very high importance, since it is infeasible to acquire large-scale datasets in each new participant, especially when performing ‘traditional’ cognitive neuroscience studies with these methods
  2. The results offer compelling evidence that selecting maximally-distinct images based on their predicted evoked brain activation profiles is an effective strategy for optimizing data collection efficiency, and thorough additional experiments further clarify the boundaries of these results
  3. The paper analyzes several high-quality datasets, providing strong support that their findings are generalizable
Critical weaknesses
  1. The work does not appear to directly compare encoding model performance to what would be expected with a ‘full’ dataset with no acquisition budget (e.g., using a full NSD participant). This ‘baseline’ would more fairly compare to many previous reports/approaches, and would give readers a better sense of the expected cost of reducing their data acquisition. The results are clear that a principled image selection strategy improves model performance over a random one, but this does not speak to the magnitude of the costs incurred.
  2. The experiments focus on encoding performance, but the paper does not address how these methods may be related to image decoding/reconstruction. In principle, improved encoding is necessary for improved decoding/reconstruction, but this is not directly addressed, either in experiments (ideal) or discussion.
Questions for authors
  1. As mentioned above (W1), what is the ‘cost’ to this approach? How much predictive performance is lost by scanning only a small subset of images, as compared to the full available dataset?
  2. The ROI analyses (e.g., Fig. 2B) are quite interesting. However, it appears that these are reporting encoding performance within each ROI based on images selected from a larger superset of voxels. How would optimizing image selection for individual ROIs compare? Perhaps an experiment could be performed limiting voxels used to compute brain space objectives to different ROIs (as compared to a random sample of voxels from ROIs, or as a way to compare the informative images/dimensions between ROIs). This may be outside the scope of the work, but it could be worth addressing in the Discussion.
3 comments

We thank the reviewer for their time, effort, and constructive feedback. Following their suggestions, we conducted additional experiments, revised the manuscript, and respond to each point below. Full experimental details and results are provided in the temporary Appendix E for the reviewer's convenience, with references incorporated into the revised main paper. All new and modified text is highlighted in blue.


W1. The work does not appear to directly compare encoding model performance to what would be expected with a ‘full’ dataset with no acquisition budget (e.g., using a full NSD participant). This ‘baseline’ would more fairly compare to many previous reports/approaches, and would give readers a better sense of the expected cost of reducing their data acquisition. The results are clear that a principled image selection strategy improves model performance over a random one, but this does not speak to the magnitude of the costs incurred.

Response. We agree that a full-data reference point provides a useful measure of the cost of reducing the acquisition budget. We have now evaluated encoding performance using the full available training sets for NSD and LAION-fMRI (Appendix E.8), complementing our existing analysis up to 1,000 images (Appendix A.2). As shown below, on NSD, Brain-Space Coverage achieves a median Pearson correlation of 0.288 with only 20 images, compared with 0.466 using the full training set (approximately 8,300–9,000 images). Similar trends are observed on LAION-fMRI.

DatasetN=20N=100N=1,000Full data
NSD0.2880.3720.4230.466
LAION-fMRI ID0.3220.4330.4940.513
LAION-fMRI OOD0.3340.4200.4780.506

We further evaluated image identification from measured fMRI responses (see W2), a commonly used evaluation task for visual encoding and decoding models. Our results show that encoders trained on small, optimized image sets can already achieve meaningful identification performance. More generally, the practical implications of reduced encoding accuracy depend on the intended downstream application, which should guide the choice of scanning budget.


W2. The experiments focus on encoding performance, but the paper does not address how these methods may be related to image decoding/reconstruction. In principle, improved encoding is necessary for improved decoding/reconstruction, but this is not directly addressed, either in experiments (ideal) or discussion.

Response. Image decoding and reconstruction are important and extensively studied downstream applications, but reconstruction typically relies on additional generative models, making it harder to isolate the contribution of encoding improvements. We therefore added an image identification (retrieval) experiment, a commonly used evaluation task for visual encoding and decoding models, which provides a more direct assessment of the encoder's downstream utility. We also discuss two possible extensions to image reconstruction in Appendix E.6: using encoder-predicted responses to augment reconstruction training data, and optimizing stimulus selection directly for decoding or reconstruction performance, which may require different image sets than those optimized for encoding.

In this experiment, we test whether the trained encoder can identify which image evoked a measured fMRI response. For each held-out test response, we use the encoder to predict fMRI responses to approximately 1,000 candidate images (including the correct one), rank them by noise-ceiling-weighted spatial Pearson correlation with the measured response, and check whether the correct image appears among the top five. We evaluate this across all eight NSD subjects, with chance-level Top-5 accuracy of approximately 0.5% (Appendix E.6). As shown below, Brain-Space Coverage achieves the highest Top-5 identification accuracy across all budgets, with the largest gains in the low-data regime. At N=20, accuracy improves from 21.85% with Random selection to 31.55% with Brain-Space Coverage, a relative improvement of 44.4%.

Selection methodN=20N=40N=60N=80N=100
Random21.85%47.20%61.54%68.27%72.13%
Image-Space29.83%52.28%63.42%69.66%73.06%
Brain-Space Coverage31.55%55.30%65.50%70.45%73.13%

Q1. As mentioned above (W1), what is the ‘cost’ to this approach? How much predictive performance is lost by scanning only a small subset of images, as compared to the full available dataset?

Response. Please see our response to W1.


Q2. The ROI analyses (e.g., Fig. 2B) are quite interesting. However, it appears that these are reporting encoding performance within each ROI based on images selected from a larger superset of voxels. How would optimizing image selection for individual ROIs compare? Perhaps an experiment could be performed limiting voxels used to compute brain space objectives to different ROIs (as compared to a random sample of voxels from ROIs, or as a way to compare the informative images/dimensions between ROIs). This may be outside the scope of the work, but it could be worth addressing in the Discussion.

Response. We thank the reviewer for this very interesting suggestion. Following it, we conducted new experiments investigating ROI-specific stimulus selection and found that focusing selection on individual brain regions provides substantial improvements in low-level visual cortex, while offering more limited benefits in higher-level regions (Appendix E.7).

Specifically, for each ROI, we first restricted the candidate pool to the 10% of images with the highest predicted ROI response magnitude. We then optimized Brain-Space Coverage using only source-subject voxels within that ROI, restricting both selectable images and the coverage reference set to this ROI-focused pool. We compared this approach with random sampling from the same restricted pool and our original unrestricted methods. At N=20, ROI-focused Brain-Space Coverage substantially improved encoding accuracy in the merged low-level visual region (Pearson correlation of 0.3846, compared with 0.2875 for general Brain-Space Coverage and 0.3205 for ROI-focused Random). This advantage persisted at larger budgets. In higher-level regions, improvements were smaller, with ROI-focused Coverage achieving 0.3346 compared with 0.3298 for general Coverage at N=20, and not consistently outperforming it at larger budgets. These results highlight the potential benefits of ROI-focused experimental design, particularly in the early visual cortex.

I appreciate the authors' careful attention to my previous comments. The new additions are welcome and certainly improve the paper. While I will leave my discrete score as "3", this should be considered by the Area Chair as a "high to very high 3"

Reviewer_2UTdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks which images to scan when training an image-to-fMRI encoder under a tight data budget. Images are selected to maximize coverage of a reference subject's brain-response space, then used to train encoders for new subjects. Selection is explored in brain-response space and image representation space. Transfer across datasets is reported.

Critical strengths

The question is well-posed. fMRI scanning cost is a real bottleneck, and treating stimulus selection as a first-class optimization problem rather than an afterthought is the right framing. The finding that brain-response-space coverage outperforms image-space coverage is informative for experimental design.

Cross-dataset transfer is the most consequential result. If a stimulus set selected from one subject's responses generalizes to new subjects and different acquisition protocols, this enables a practical workflow that reduces planning burden for new studies.

Critical weaknesses

The writing is below the standard expected at ICLR. The introduction is verbose and over-motivated, spending excessive space on general fMRI background that the target audience already knows. The problem formulation, which is conceptually simple, is buried under layers of context. The method section does not clearly state the selection algorithm in a single precise sentence.

The paper would benefit from a complete rewrite that states the problem, the method, and the results with minimal preamble. In its current form, the prose obscures rather than clarifies the contribution.

The methodological contribution is thin. Coverage-based subset selection in a fixed feature space is well-studied. Facility location maximization, core-set selection, and submodular optimization all provide principled solutions. The paper does not clearly articulate what is new beyond applying these standard tools to brain-response vectors. The comparison against random sampling is a low bar. No comparison against active learning, uncertainty sampling, or greedy submodular maximization with alternative kernels is provided. Without these, the empirical contribution is difficult to assess.

Questions for authors

What specific selection algorithm is used? Greedy facility location, k-medoids, farthest-point sampling, or something else? How does performance vary across these choices?

What is the absolute encoding accuracy at the smallest budget tested? Is this level sufficient for any downstream application?

How stable is the selected set across different reference subjects?

How does the method compare against iterative active learning strategies that select the next image based on current encoder uncertainty?

3 comments

We thank the reviewer for their time, effort, and constructive feedback. We conducted additional experiments addressing their concerns and revised the manuscript to improve clarity. Full details and results are in Appendix E for the reviewer’s convenience, with key findings referenced in the revised paper. All new and modified text is highlighted in blue.


W1. The writing is below the standard expected at ICLR. The introduction is verbose and over-motivated, spending excessive space on general fMRI background that the target audience already knows. The problem formulation, which is conceptually simple, is buried under layers of context. The method section does not clearly state the selection algorithm in a single precise sentence. The paper would benefit from a complete rewrite that states the problem, the method, and the results with minimal preamble. In its current form, the prose obscures rather than clarifies the contribution.

Response. We thank the reviewer for their suggestions. We have revised the Introduction to more directly present the problem formulation, central hypothesis, and contributions, while reducing repetition and unnecessary background. We also revised the opening of the Methods section to clearly summarize our approach and explicitly identify greedy facility-location maximization in predicted brain-response space as the selection algorithm. We hope these revisions, together with additional clarifications throughout the paper, address the reviewer's concerns about clarity.


W2. The methodological contribution is thin. Coverage-based subset selection in a fixed feature space is well-studied. Facility location maximization, core-set selection, and submodular optimization all provide principled solutions. The paper does not clearly articulate what is new beyond applying these standard tools to brain-response vectors. The comparison against random sampling is a low bar. No comparison against active learning, uncertainty sampling, or greedy submodular maximization with alternative kernels is provided. Without these, the empirical contribution is difficult to assess.

Response: We believe an important contribution of our work is formulating and tackling the practical problem of selecting small, informative stimulus sets for image-to-fMRI encoder training, bringing established subset-selection techniques to a largely unexplored setting. A second key methodological contribution concerns the representation space in which selection is performed, rather than the optimization objective. Specifically, we propose and empirically validate the hypothesis that neural representations from previously scanned subjects can effectively guide stimulus selection for new subjects, without requiring prior measurements from them. Demonstrating that this approach generalizes across datasets with different image distributions and acquisition settings (e.g. 7T and 3T fMRI) further supports its practical applicability. Moreover, following the reviewer's question about downstream utility (Q2), our new experiments demonstrate benefits for image identification. We believe that the problem formulation, the proposed source-guided selection approach, and its extensive empirical evaluation together constitute a meaningful contribution to data-efficient visual neuroscience.

Our paper systematically compares representation spaces and established selection objectives, including greedy facility-location coverage, k-means, k-center/Core-Set, and effective-dimensionality maximization (Section 3.3, Table 1, Appendix A.1). Our setting requires selecting the entire stimulus set before scanning a new subject, without access to any target-subject measurements. This differs from online active learning, which requires repeated cycles of scanning, model adaptation, and stimulus selection. Moreover, online target-adaptive selection introduces repeated model training and uncertainty estimation during scanning, making it less practical for our intended experimental setting.

Nevertheless, we additionally evaluated uncertainty-based selection, including online active learning (Appendix E.2). We estimate uncertainty from disagreement among ten independently initialized encoders trained on the selected images. Starting with five random images, we iteratively select five additional images with the highest uncertainty and retrain the ensemble. We compare Source Iterative Uncertainty, using predicted responses from three previously scanned subjects, with Target Iterative Uncertainty, using target responses revealed iteratively during selection (simulating online active learning). On NSD Subjects 1–2, Brain-Space Coverage outperformed both methods across budgets (e.g., 0.347 vs. 0.329 for Source Iterative and 0.302 for Target Iterative at N=20). Interestingly, Source Iterative also outperformed Target Iterative, providing further evidence of the usefulness of previously collected neural data for stimulus selection.

Q1. What specific selection algorithm is used? Greedy facility location, k-medoids, farthest-point sampling, or something else? How does performance vary across these choices?

Response. Our primary method uses greedy facility-location maximization (Coverage) in predicted brain-response space. At each step, we select the image that most improves how well the selected set represents the entire candidate pool, based on similarity between their predicted fMRI responses. We additionally evaluate three alternative selection algorithms: k-means clustering, k-center/Core-Set (farthest-point sampling), and effective-dimensionality maximization. All methods are described in Section 3.3, with implementation details in Appendix C.3. Performance comparisons are reported in Table 1 and Table T1. Coverage performs best in Brain-Space across all tested budgets, while k-means is generally strongest in Image-Space.


Q2. What is the absolute encoding accuracy at the smallest budget tested? Is this level sufficient for any downstream application?

Response. At the smallest budget of N=20 images, Brain-Space selection achieves a median voxelwise Pearson correlation of 0.288, compared with 0.256 for Image-Space and 0.230 for Random sampling (Table 1). To assess whether this accuracy is sufficient for downstream applications, we additionally evaluated image identification (retrieval), a commonly used benchmark for visual encoding and decoding models.

Specifically, we test whether an encoder trained on a small image set can identify which image evoked a measured, held-out fMRI response. For each test response, we predict the fMRI responses to approximately 1,000 candidate images, rank them by similarity to the measured response, and check whether the correct image appears among the top five. Across all eight NSD subjects, at N=20, Brain-Space selection achieved 31.55% Top-5 identification accuracy, compared with 29.83% for Image-Space and 21.85% for Random sampling (chance level ≈0.5%). At N=60, performance increased to 65.50%, compared with 63.42% and 61.54%, respectively. These results demonstrate that even encoders trained on only 20 selected images can support meaningful image identification, and that optimized stimulus selection improves this downstream performance (see Appendix E.6).


Q3. How stable is the selected set across different reference subjects?

Response. To investigate the stability of selected image sets across reference subjects, we measured the exact-image overlap between sets selected using four different source subjects. Although the overlap was relatively low (8.6–15.1% across budgets), it was substantially higher than random chance (0.3–1.5%). Importantly, as already shown in Appendix A.7 (Table T8), despite selecting different images, individual reference subjects yield reasonably consistent downstream encoding performance, while combining multiple sources achieves the best results. These results suggest that while the exact selected images depend on the reference subject, different selections can provide similarly informative training sets for new subjects.


Q4. How does the method compare against iterative active learning strategies that select the next image based on current encoder uncertainty?

Response. We have now evaluated iterative uncertainty-based active learning, as detailed in our response to W2 and Appendix E.2.

We thank the reviewer again for the time and effort devoted to reviewing our paper and for their feedback. We hope our additional analyses and revisions have addressed your questions and concerns. We would be happy to address any remaining concerns or provide further clarification if needed.

Reviewer_Vn9aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper describes a method for data-efficient training of fMRI visual encoding models for individual subjects. The method is efficient in terms of the amount of fMRI data that has to be acquired for a given subject to fit an encoding model. The method requires fMRI data from other subjects for a large set of stimuli. A stimulus set from the same candidate set or another set is selected for presentation to the new subjects. The authors test heuristic methods for choosing an efficient set of stimuli. The general motivation is that a stimulus is redundant if it elicits a highly similar response pattern as an already included stimulus. In the best performing method, the stimulus set is greedily optimized to “cover” the candidate stimuli’s set of brain response patterns. Using the Natural Scenes 7T and the GOD 3T fMRI data sets, the authors show that this heuristic yield more accurate encoding models when the amount of data from the new subject is severely limited.

Critical strengths
  • Addresses an important problem with a practical method
  • Demonstrates substantial gain in the low-data regime
  • Compares multiple methods using image feature and brain representational model space and multiple heuristic objectives (effective dimensionality, k means, k center, coverage).
  • Demonstrates generalization across subjects, encoding models, and data sets
Critical weaknesses

The objectives explored are not motivated from first principles. The goal is to select stimuli that are optimal for learning the parameters of the encoding model. This suggests maximizing the expected information gain (EIG) about the parameters. In practice, estimating EIG might be difficult, but there are tractable approaches to directly maximizing the reduction of the encoding model's prediction error by the choice of stimuli. A heuristic approach would be to greedily add stimuli for which the gradient descent steps maximally reduce the validation error in the training subjects. To make this efficient, we could compute the gradient vector g_val of the validation loss with respect to the encoding model parameters and greedily pick stimuli whose gradient steps align with g_val.

For the coverage objective, the motivation is to avoid redundant stimuli. However, two stimuli that elicit similar response patterns are not necessarily redundant for learning an encoding model: If the encoding model’s basis features distinguish the stimuli, then the two stimuli with their measured response patterns can serve for the encoding model to learn to disregard the difference and predict responses to both correctly. More generally, the normative perspective shows that the model’s feature space, the brain response space and the encoding model itself all matter for choosing optimal stimuli. This should be discussed. Killamsetty et al., 2021 describe the related GLISTER technique. Normative and heuristic methods for stimulus set optimization for model comparison (here model identification) are described in Golan et al. 2026 (Nat Rev Neurosci).

Questions for authors

Why not use the measured rather than the encoding-model predicted patterns for subset selection? (I understand that using predicted patterns enables selection among arbitrary candidate sets. Still checking for a cost of the encoding-model error seems important.)

What if much larger sets of candidate image were used? I guess there would be a tradeoff: richer options for selection versus larger OOD error of the model.

Is the normalization of the patterns to unit norm necessarily desirable in the context of the coverage maximization and other approaches? Might it hurt performance of stimulus selection?

2 comments

We thank the reviewer for their time, effort, and constructive feedback. We conducted additional experiments addressing their suggestions and responded to each point below. Full experimental details and results are provided in the temporary Appendix E for the reviewer's convenience, with corresponding references added to the revised main paper. All new and modified text is highlighted in blue.


W1. The objectives explored are not motivated from first principles. The goal is to select stimuli that are optimal for learning the parameters of the encoding model. This suggests maximizing the expected information gain (EIG) about the parameters. [...] A heuristic approach would be to greedily add stimuli for which the gradient descent steps maximally reduce the validation error in the training subjects. [...]

Response: Following the reviewer's proposal, we implemented GLISTER-style gradient-alignment selection, alongside uncertainty-based active-learning alternatives. We initially chose an established, computationally efficient facility-location objective to better isolate and emphasize our other contributions: formulating the problem of optimizing small stimulus sets for image-to-fMRI encoder training and proposing that neural representations from previously scanned subjects can guide stimulus selection for new subjects. We implemented Iterative Gradient Alignment (GLISTER-style). For each candidate image, we compute its training-loss gradient using predicted fMRI responses from source subjects and score its alignment with the reference-loss gradient, also computed using predicted source-subject responses over the full candidate pool. Higher alignment indicates a larger expected first-order reduction in this reference loss. The method repeatedly updates source voxel embeddings using previously selected images, recomputes both candidate and reference gradients at the updated parameters, and selects the next five images from 1,024 randomly sampled remaining candidates (see Appendix E.1). Like Brain-Space Coverage, this approach leverages neural-response information from previously scanned subjects, but explicitly incorporates the encoder's learning dynamics. We evaluated the method on all eight NSD subjects.

As shown below, GLISTER-style selection achieves slightly higher performance than Brain-Space Coverage at all budgets, but the differences are small (0.002–0.005 Pearson correlation). This suggests that our simpler Brain-Space Coverage objective already captures much of the useful selection signal, achieving comparable performance without explicitly optimizing the encoder's learning dynamics.

We additionally evaluated uncertainty-based selection, including fixed and iterative source-subject uncertainty and online target-subject active learning that updates using newly acquired target responses. In experiments on NSD Subjects 1–2, Brain-Space Coverage outperformed both uncertainty-based approaches on average across the evaluated budgets (Appendix E.2).

Selection methodN=20N=60N=100
Random0.23240.32700.3611
Image-Space0.25940.33470.3638
GLISTER-style Gradient Alignment0.29080.35540.3761
Brain-Space Coverage0.28810.35330.3716

W2. For the coverage objective, the motivation is to avoid redundant stimuli. However, two stimuli that elicit similar response patterns are not necessarily redundant for learning an encoding model: If the encoding model’s basis features distinguish the stimuli, then the two stimuli with their measured response patterns can serve for the encoding model to learn to disregard the difference and predict responses to both correctly. More generally, the normative perspective shows that the model’s feature space, the brain response space and the encoding model itself all matter for choosing optimal stimuli. This should be discussed. Killamsetty et al., 2021 describe the related GLISTER technique. Normative and heuristic methods for stimulus set optimization for model comparison (here model identification) are described in Golan et al. 2026 (Nat Rev Neurosci).

Response: We agree that similar neural responses do not necessarily imply redundancy for encoder training, and that more effective selection objectives may exist. As discussed in W1, we evaluated a model-aware GLISTER-style approach that jointly incorporates visual features, neural responses, and the encoder's learning dynamics, achieving performance comparable to Brain-Space Coverage. We have also expanded the Related Work to discuss alternative selection approaches, including Killamsetty et al. (2021) and Golan et al. (2026), and the Discussion to acknowledge the limitations of coverage-based selection and discuss our new model-aware experiments.

Q1. Why not use the measured rather than the encoding-model predicted patterns for subset selection? (I understand that using predicted patterns enables selection among arbitrary candidate sets. Still checking for a cost of the encoding-model error seems important.)

Response: We thank the reviewer for this important suggestion. As the reviewer notes, using measured source responses could improve selection, but would restrict the candidate pool to images previously scanned in the source subjects, limiting the method's applicability. To quantify the cost of prediction errors, we conducted a controlled experiment using the 1,000 NSD images shared across subjects (normally used as test stimuli) as a candidate set, with three pools of 800 images, and evaluated on a separate set of 1,000 nonshared target images. We compared selection using measured versus predicted source responses, keeping all other settings identical (Appendix E.3). As shown below, measured-response selection outperformed predicted-response selection, with the largest improvement at N=20 (0.3435 vs. 0.3107). These results demonstrate that source-encoder prediction errors introduce a measurable cost and suggest that leveraging measured responses from previously scanned subjects could further improve stimulus selection when the available images are suitable for the experimental goals.

Selection methodN=20N=60N=100
Random0.28500.41150.4556
Brain-Space (predicted sources)0.31070.41870.4524
Brain-Space (measured sources)0.34350.44010.4618

Q2. What if much larger sets of candidate image were used? I guess there would be a tradeoff: richer options for selection versus larger OOD error of the model.

Response: We examined the effect of candidate-pool size on NSD (see below) and LAION-fMRI (see new Appendix E.4), varying the available candidates from 12.5% to 100% while keeping the scanning budget fixed. Larger pools provided modest improvements, with the largest gains generally appearing at the smallest budget (N=20). NSD is COCO-based and has a limited image distribution, while LAION-fMRI is more diverse but contains fewer images. Selecting stimuli from substantially larger and more diverse image collections, followed by prospective fMRI scanning to evaluate their effectiveness, would be an interesting direction for future work.

NSD Results Across Candidate-Pool Sizes

Candidate poolN=20N=60N=100
12.5%0.28350.34250.3627
25%0.29160.34770.3642
50%0.28750.34500.3643
100%0.29310.34970.3656

Q3. Is the normalization of the patterns to unit norm necessarily desirable in the context of the coverage maximization and other approaches? Might it hurt performance of stimulus selection?

Response: We use cosine similarity because we expected the direction of fMRI response patterns to be more informative for stimulus selection than their magnitude. To test this choice, we additionally compared squared-Euclidean coverage with and without unit normalization, alongside our original cosine-based coverage. Following the paper's ablation protocol on NSD Subjects 1–2, unit normalization improved performance at every budget, particularly at N=20, while the original cosine-based method performed slightly better overall (see new Appendix E.5).

Selection methodN=20N=60N=100
Euclidean Coverage (unnormalized)0.28030.40480.4516
Euclidean Coverage (unit-normalized)0.34520.43150.4592
Brain-Space Coverage (ours, cosine)0.34710.43680.4607

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.