- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the important problem of how to allocate limited neuroimaging scanning budgets to best select visual stimuli that are likely to be most informative for estimating encoding models in new participants. Existing datasets often sample thousands of images over dozens of hours, which is infeasible in the general population and with typical research budgets. The main contributions here are testing a few principled approaches for selecting a limited set of images to show a new participant, and evaluating the extent to which these selections may work in silico by testing within-dataset and across-dataset prediction accuracy under these constraints. No new human participant data is acquired, but the experiments on existing datasets are interesting. The primary conditions tested are stimulus selection objective functions based on image properties (in CLIP/equivalent spaces) and those based on predicted brain responses. Altogether, the authors conclude that optimizing stimulus selection based on predicted brain responses offers a meaningful advantage to previous approaches, and can lead to effective encoding model estimation with substantially less data acquisition.
Critical strengths
- The problem addressed by this work is of very high importance, since it is infeasible to acquire large-scale datasets in each new participant, especially when performing ‘traditional’ cognitive neuroscience studies with these methods
- The results offer compelling evidence that selecting maximally-distinct images based on their predicted evoked brain activation profiles is an effective strategy for optimizing data collection efficiency, and thorough additional experiments further clarify the boundaries of these results
- The paper analyzes several high-quality datasets, providing strong support that their findings are generalizable
Critical weaknesses
- The work does not appear to directly compare encoding model performance to what would be expected with a ‘full’ dataset with no acquisition budget (e.g., using a full NSD participant). This ‘baseline’ would more fairly compare to many previous reports/approaches, and would give readers a better sense of the expected cost of reducing their data acquisition. The results are clear that a principled image selection strategy improves model performance over a random one, but this does not speak to the magnitude of the costs incurred.
- The experiments focus on encoding performance, but the paper does not address how these methods may be related to image decoding/reconstruction. In principle, improved encoding is necessary for improved decoding/reconstruction, but this is not directly addressed, either in experiments (ideal) or discussion.
Questions for authors
- As mentioned above (W1), what is the ‘cost’ to this approach? How much predictive performance is lost by scanning only a small subset of images, as compared to the full available dataset?
- The ROI analyses (e.g., Fig. 2B) are quite interesting. However, it appears that these are reporting encoding performance within each ROI based on images selected from a larger superset of voxels. How would optimizing image selection for individual ROIs compare? Perhaps an experiment could be performed limiting voxels used to compute brain space objectives to different ROIs (as compared to a random sample of voxels from ROIs, or as a way to compare the informative images/dimensions between ROIs). This may be outside the scope of the work, but it could be worth addressing in the Discussion.
3 comments
Q1. As mentioned above (W1), what is the ‘cost’ to this approach? How much predictive performance is lost by scanning only a small subset of images, as compared to the full available dataset?
Response. Please see our response to W1.
Q2. The ROI analyses (e.g., Fig. 2B) are quite interesting. However, it appears that these are reporting encoding performance within each ROI based on images selected from a larger superset of voxels. How would optimizing image selection for individual ROIs compare? Perhaps an experiment could be performed limiting voxels used to compute brain space objectives to different ROIs (as compared to a random sample of voxels from ROIs, or as a way to compare the informative images/dimensions between ROIs). This may be outside the scope of the work, but it could be worth addressing in the Discussion.
Response. We thank the reviewer for this very interesting suggestion. Following it, we conducted new experiments investigating ROI-specific stimulus selection and found that focusing selection on individual brain regions provides substantial improvements in low-level visual cortex, while offering more limited benefits in higher-level regions (Appendix E.7).
Specifically, for each ROI, we first restricted the candidate pool to the 10% of images with the highest predicted ROI response magnitude. We then optimized Brain-Space Coverage using only source-subject voxels within that ROI, restricting both selectable images and the coverage reference set to this ROI-focused pool. We compared this approach with random sampling from the same restricted pool and our original unrestricted methods. At N=20, ROI-focused Brain-Space Coverage substantially improved encoding accuracy in the merged low-level visual region (Pearson correlation of 0.3846, compared with 0.2875 for general Brain-Space Coverage and 0.3205 for ROI-focused Random). This advantage persisted at larger budgets. In higher-level regions, improvements were smaller, with ROI-focused Coverage achieving 0.3346 compared with 0.3298 for general Coverage at N=20, and not consistently outperforming it at larger budgets. These results highlight the potential benefits of ROI-focused experimental design, particularly in the early visual cortex.
I appreciate the authors' careful attention to my previous comments. The new additions are welcome and certainly improve the paper. While I will leave my discrete score as "3", this should be considered by the Area Chair as a "high to very high 3"
We thank the reviewer for their time, effort, and constructive feedback. Following their suggestions, we conducted additional experiments, revised the manuscript, and respond to each point below. Full experimental details and results are provided in the temporary Appendix E for the reviewer's convenience, with references incorporated into the revised main paper. All new and modified text is highlighted in blue.
W1. The work does not appear to directly compare encoding model performance to what would be expected with a ‘full’ dataset with no acquisition budget (e.g., using a full NSD participant). This ‘baseline’ would more fairly compare to many previous reports/approaches, and would give readers a better sense of the expected cost of reducing their data acquisition. The results are clear that a principled image selection strategy improves model performance over a random one, but this does not speak to the magnitude of the costs incurred.
Response. We agree that a full-data reference point provides a useful measure of the cost of reducing the acquisition budget. We have now evaluated encoding performance using the full available training sets for NSD and LAION-fMRI (Appendix E.8), complementing our existing analysis up to 1,000 images (Appendix A.2). As shown below, on NSD, Brain-Space Coverage achieves a median Pearson correlation of 0.288 with only 20 images, compared with 0.466 using the full training set (approximately 8,300–9,000 images). Similar trends are observed on LAION-fMRI.
We further evaluated image identification from measured fMRI responses (see W2), a commonly used evaluation task for visual encoding and decoding models. Our results show that encoders trained on small, optimized image sets can already achieve meaningful identification performance. More generally, the practical implications of reduced encoding accuracy depend on the intended downstream application, which should guide the choice of scanning budget.
W2. The experiments focus on encoding performance, but the paper does not address how these methods may be related to image decoding/reconstruction. In principle, improved encoding is necessary for improved decoding/reconstruction, but this is not directly addressed, either in experiments (ideal) or discussion.
Response. Image decoding and reconstruction are important and extensively studied downstream applications, but reconstruction typically relies on additional generative models, making it harder to isolate the contribution of encoding improvements. We therefore added an image identification (retrieval) experiment, a commonly used evaluation task for visual encoding and decoding models, which provides a more direct assessment of the encoder's downstream utility. We also discuss two possible extensions to image reconstruction in Appendix E.6: using encoder-predicted responses to augment reconstruction training data, and optimizing stimulus selection directly for decoding or reconstruction performance, which may require different image sets than those optimized for encoding.
In this experiment, we test whether the trained encoder can identify which image evoked a measured fMRI response. For each held-out test response, we use the encoder to predict fMRI responses to approximately 1,000 candidate images (including the correct one), rank them by noise-ceiling-weighted spatial Pearson correlation with the measured response, and check whether the correct image appears among the top five. We evaluate this across all eight NSD subjects, with chance-level Top-5 accuracy of approximately 0.5% (Appendix E.6). As shown below, Brain-Space Coverage achieves the highest Top-5 identification accuracy across all budgets, with the largest gains in the low-data regime. At N=20, accuracy improves from 21.85% with Random selection to 31.55% with Brain-Space Coverage, a relative improvement of 44.4%.