acceptodds
Under review as a conference paper at ICLR 2027

Receptive-field-constrained stimulus optimization for human early and intermediate visual cortex

Abstract

An ongoing challenge in sensory neuroscience is to characterize the feature dimensions encoded by cortical populations. Recent approaches probe feature selectivity in a data-driven way, by synthesizing a most-exciting-input (MEI) for a target neural population. While this approach has been successfully applied to human higher visual cortex using fMRI data, generating MEIs for early- and mid-level retinotopic visual areas requires additional modeling constraints due to small receptive field sizes. To address this challenge, we introduce two novel MEI generation frameworks, Receptive Field Diffusion for Visual Exploration (RF-DiVE) and Receptive Field Gradient Optimization (RF-GO). Both methods use a population receptive field (pRF)-constrained voxelwise encoding model; RF-DiVE combines this with a pretrained latent diffusion model, while RF-GO uses regularized gradient ascent. When applied to single voxels in retinotopically defined areas V1-hV4, using data from the Natural Scenes Dataset, we obtain MEIs that exhibit consistent structure within the pRF, suggesting selectivity for local features like contour, color, and texture. We systematically compare MEIs generated by RF-DiVE and RF-GO using two encoding backbones, performing in-silico validation of predicted responses to MEIs using independent encoding models. Across all methods and all visual areas, MEIs elicit higher model-predicted responses than the most activating natural images. We further find that the choice of generation framework and encoding backbone differentially affects MEI properties, including their visual appearance, structural interpretability, and cross-model generalizability. These results offer a new approach for performing data-driven characterization of spatial and feature selectivity across human visual cortex.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain encoder, stimulus optimization, neuroscience, human vision, visual cortex, fMRI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xQwuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors extend MEI approaches to generate population receptive field (pRF)-constrained images. They analyze two frameworks and compare the effect of natural images bias.

Critical strengths

The authors introduce a framework that enable spatially-constrained MEI synthesis, and compare gradient-based and diffusion-based optimization methods for the same brain regions. Their extension can be relevant especially for early and intermediate retinotopic cortex, in which receptive field is narrowed, and their generated images support this localized optimization. They further provide in-silico validation using an independent test brain encoder and show MEIs consistently elicit higher predicted responses than natural-image controls.

Critical weaknesses

I find the novelty and contribution limited. In terms of algorithmic novelty, both the encoding model and the two optimization approaches are largely adapted from prior work. I therefore evaluate the contribution primarily based on whether applying these components to V1–hV4 yields new neuroscientific insights. While the analysis reveal differences within optimization method, the biological contributions remain relatively limited, and most results characterize the behavior of the proposed pipelines rather than establish new biological insights.

Questions for authors

Can you use your framework to provide additional new neuroscientific insights? Since only in-silico validation was provided, can you propose a way to improve the reliability of the results without requiring closed-loop experiments with human participants?

Reviewer_bgLvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes methods for generating images that maximally activate specific brain regions. In this field, related works are previously applied primarily to the high-level visual cortex, now this paper extend to the early and intermediate human visual cortex. Given that these regions possess small receptive fields with high spatial specificity, the authors incorporated receptive field constraints into their encoding models. By combining these constraints with either diffusion models or gradient-based optimization, they generated images tailored to individual fMRI voxels in areas ranging from V1 to hV4. The resulting images exhibit consistent features within the corresponding receptive fields. Across various encoding models, including those tested on independent datasets, these generated images elicited higher predicted neural responses than natural images.

Critical strengths

The authors incorporate a receptive field constraint, using the fact that receptive fields in the early visual cortex are small and highly location-specific, to refine the stimulus generation method from the level of averaged brain regions down to individual voxels. The authors give systematic experiments involving four subjects, four visual areas, two generation methods, and two model backbones.

Critical weaknesses
  1. I think that image characteristics such as color and spatial frequency vary significantly depending on the generation method and model backbone, while current feature descriptions primarily rely on image analysis. I suggest the authors quantitatively compare images of the same voxel generated using different methods and backbones to see if the similarity between them is significantly higher than the similarity between different voxels.

  2. The paper currently uses model predictions to validate the effectiveness of the generated images, without measuring actual brain responses. While directly having participants view the images and record their fMRI responses is difficult to have in the short term, the current validation also has limitations: the independent model used by the authors for evaluation is similar in principle to the model used to generate the images and may have the same biases. So, I recommend the authors to incorporate evaluation models with more significant differences and to compare with more natural images.

Questions for authors

The receptive field of hV4 is obviously larger than that of V1. In the behavior experiment, the hV4 image is judged to have more three-dimensional shape. Is it possible that this is simply because the receptive field is larger, rather than because hV4 itself is more sensitive to three-dimensional shape? Could the authors re-compare using voxels with similar receptive field sizes?

Reviewer_mdXvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes new methods for understanding what optimally drives neurons in the early to mid-level visual cortex. They use an encoding model with a factorized readout that separates spatial tuning ("where") from featural tuning ("what"). These types of encoding models are fairly standard now in primate/human visual neuroscience as well as older work in mice (Klindt et al., NeurIPS 2017). With these encoding models, the maximally exciting images they synthesize for individual voxels show consistent features within the spatial receptive fields but not outside it. They two methods for synthesis differ in whether they leverage a natural image prior or not. The paper also validates those synthesizes images on independent ("test") encoding models and show that the synthesized images indeed drive the test model predicted response to be higher than other top images within natural datasets.

Critical strengths
  • The paper highlights some issues in interpreting featural selectivity of brain voxels using encoding models + MEIs. For example, they show that the choice of encoding model and the synthesis procedure can bias the synthesized images.
  • The idea of validating MEIs in silico using independent test models also seems interesting
Critical weaknesses
  1. The methodological contribution needs clearer justification. RF-DiVE appears to combine BrainDiVE with an established encoding model that separates spatial and feature selectivity. Spatially localized MEIs are therefore expected from the model’s pooling constraint and do not seem like a sufficient additional contribution. An ablation removing spatial weighting while retaining the feature readout would help determine whether the RF constraint changes the recovered features beyond localizing them. This is important as teh spatial receptive field is directly interpretable via the learned spatial mask.

  2. Robustness across encoding models is insufficiently established. Figure 3 directly compares MEIs across backbones for only two hV4 voxels, and their visual agreement is not entirely convincing. Although the appendix provides additional examples, systematic, side-by-side comparisons for the same voxels, accompanied by quantitative measures of feature agreement, are needed to support broader claims of consistency.

  3. The scientific interpretation relies heavily on qualitative observations. The paper relies on selected similarities between MEIs and natural images, such as a yellow railing or pink flowers which as such provide limited evidence that the methods recover consistent tuning properties. Likewise, greater colorfulness in RF-DiVE images could reflect the synthesis prior rather than neural selectivity. Despite the quantitative image statistics and perceptual study, stronger controls linking these properties to voxel-specific tuning are needed for example, comparisons with unguided diffusion samples and tests of whether manipulating the identified features changes responses.

  4. Differences in cross-model generalization warrant more rigorous analysis. The authors acknowledge the attenuation of predicted responses in the independent test encoder but the comparison between RF-GO and RF-DiVE remains largely descriptive. Direct statistical comparisons and analyses across visual areas would help establish the usefulness of a natural image prior for probing teh early visual cortex.

  5. The authors suggest that combining synthesis methods may help identify robust tuning properties. A complementary strategy would use a single synthesis method to optimize responses jointly across multiple encoding models, explicitly discouraging solutions that exploit individual-model idiosyncrasies. I feel like this would be a stronger test and utility of the approach.

Questions for authors

Listed in the weaknesses above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.