acceptodds
Under review as a conference paper at ICLR 2027

A Model-Driven Functional Atlas of Human Visual Cortex under Naturalistic Vision

Abstract

Visual preferences overlap in cortex and recur across objects and scenes. Describing their combinations is a central challenge for a functional atlas of natural vision. We construct a fine-grained atlas by relating sparse, interpretable Transformer features to human fMRI responses. Individual functional modes link image properties to cortical coefficient patterns, resolving distinctions within familiar category-associated regions. Continuous coordinates reveal regional mixtures in which a locally enriched mode contributes only part of the overall profile. Response-matched families show corresponding cortical patterns across the observed participants and independently learned model dictionaries, providing recurrent atlas elements. Condition subatlases organize these elements by both relative preference and common participation: features shared across image categories are retained alongside those that distinguish them. Model responses to individual images select smaller subsets of these repertoires. Together, the results describe visual cortex through overlapping functional patterns whose combinations connect a recurrent atlas to the diversity of natural images.

Primary area:
applications to neuroscience & cognitive science
Keywords:
neural encoding, functional brain atlas, visual cortex, fMRI, transformer

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wiMaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to relate the functional organization of human visual cortex to sparse features learned from vision Transformers. Using the Natural Scenes Dataset (fMRI), the authors derive “functional modes” (FMs) from sparse dictionaries of several pretrained ViTs, map them onto cortex for each subject and model, analyze their arrangement across cortex, and group them into “recurrent families” shared across subjects and models.

The paper claims to contribute the following: a method for linking sparse, multi-layer model features to cortical responses; a set of FM families that recur across subjects and architectures; a low-dimensional description of cortical organization, obtained by applying PCA to the FM profiles of cortical locations, which the authors present as an atlas; and a link between recurrence and compositional structure. In my reading, the novelty and impact relative to prior work remain unclear. As far as I can tell, the paper does not reveal anything new about the organization of visual cortex. It also does not show that the method performs better than existing approaches, and I could not find comparisons to the state of the art. More broadly, the paper is not well written: its goals, motivation, and contributions are not clearly defined.

The validation is mostly qualitative. The main results consist of example images and cortical heatmaps for a single participant and model, shown without color scales or area outlines; PCA axes interpreted through their top-loading image categories; and families defined by a similarity threshold. The paper does not quantify how robustly FMs map onto cortical areas, how reliable they are under cross-validation, or how consistent they are across subjects and models. It also does not compare the approach against baselines, so it remains unclear what the sparse dictionaries add over existing model-based or data-driven methods.

Critical strengths

Relevant question with suitable data: The idea of examining how sparse model features compose across cortex and generalize across participants is interesting and could advance our understanding of the organization of visual cortex. The NSD dataset, with its 1,000 images shared across participants, is well suited to this question. This is the main reason I consider the work worth pursuing. In the current manuscript, however, the question is not addressed with sufficient clarity or validation to support acceptance.

Critical weaknesses

Unclear goals, contributions, and methods: The manuscript does not clearly define its goals and contributions, and central terms such as family and recurrence are left undefined. The methods are unclear from the main text, and critical information for interpreting the results is missing, including how the FMs are obtained and to what degree they depend on neural data.

Insufficient validation and unclear advance: The results are, in part, qualitative heatmaps without color scales or area outlines, focused on a single participant and model. The paper does not quantify how robustly the FMs map onto cortical areas or how reliable they are under cross-validation. This is consequential because it leaves the advance of the paper unclear. As far as I can tell, the advance is not scientific, since the paper does not reveal anything new about visual cortex. If it is methodological, it would need to be supported by comparisons against baselines and the state of the art, which the paper does not provide.

Questions for authors

How exactly are the FMs obtained from the SAE responses? What objective is optimized, and which parameters, if any, are fit to neural data? The answer determines whether the FMs reflect cortical structure or mainly properties of the model dictionaries.

Are results for all participants and models reported anywhere, and do they show the same structure as S5? If this analysis exists, the main claims could rest on more than a single case.

Is there any quantitative evaluation of how well the FMs map onto cortical areas, or of their reliability, for example under split-half cross-validation?

What do the authors consider the main novel contribution, scientific or methodological, relative to prior work linking cortical location to visual preference?

How does the approach perform compared to baselines and existing methods? For example, what do the sparse dictionaries add over dense model features without the SAE, or over a voxel decomposition without a model? What's the predictive accuracy compared to other methods?

Reviewer_syxKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a functional atlas of human visual cortex using sparse features extracted from pre-trained vision Transformers. The authors fit voxel-wise encoding models to obtain cortical maps for individual features, group corresponding features across models and subjects into recurrent families, and construct condition-specific sub-atlases.

The paper explores an interesting direction, but the methodological contribution appears incremental. More importantly, the stability of the proposed functional modes and the interpretation of condition sub-atlases are not sufficiently established.

Critical strengths
  1. The paper addresses an interesting problem of characterizing fine-grained functional organization in human visual cortex under naturalistic stimulation. Moving beyond conventional category-selective ROIs to examine overlapping visual preferences is a worthwhile direction.

  2. The paper includes extensive analyses across multiple vision models, participants, and image conditions. In particular, the investigation of recurrent functional families across models and subjects is interesting, and the authors provide several quantitative analyses and sensitivity checks to support their findings.

Critical weaknesses
  1. Limited novelty and missing baselines. The overall approach is closely related to prior work on model-based cortical mapping and feature decomposition, such as Brain Dissection, BrainSAIL, and SAE-BrainMap. The main novelty seems to lie in grouping features into recurrent families and constructing condition subatlases. However, the paper does not clearly establish what these steps offer beyond existing approaches. Comparisons with simpler decompositions, such as PCA or NMF, would also help clarify whether TopK-SAE is necessary.

  2. Unclear stability of functional modes. The cortical maps are derived from multivariate encoding coefficients, which can be sensitive to correlated features and fitting choices. The paper does not evaluate whether individual FM maps remain stable when the encoding model is refitted on different training subsets. Moreover, cross-participant correspondence is much weaker than cross-model correspondence (median cosine 0.279 vs. 0.769). This raises concerns about how reliably these modes capture shared cortical organization.

  3. Motivation for condition subatlases. The distinction between preference and participation is reasonable, but the value of taking their union is unclear. Among the 589 additional condition–family relationships, 272 have non-positive preference differences. The higher activation coverage of the union is largely expected from including more families. What additional functional insight does this union provide compared with analyzing preference and participation separately?

Questions for authors
  1. How does the proposed framework differ from existing model-based cortical mapping methods beyond the use of recurrent-family grouping? Could the authors clarify the main contribution relative to BrainSAIL and SAE-BrainMap?

  2. Would simpler feature representations, such as PCA or NMF, produce similar cortical maps and recurrent families under the same encoding pipeline?

  3. How stable are individual FM coefficient maps across different training subsets or repeated fits? Could the authors provide a quantitative analysis of their reproducibility?

  4. What explains the large difference between cross-model and cross-participant spatial correspondence? Are certain functional families consistently less reproducible across participants?

  5. What is the motivation for defining condition subatlases as the union of preference and participation? What information is gained compared with keeping these two measures separate?

Reviewer_eZkJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to construct a fine-grained functional map of the human visual cortex using a data-driven approach. The authors extract image features with a pretrained visual model and then train a sparse autoencoder to obtain 256 features, manually constraining each image to activate at most 8 of them. A linear mapping is subsequently adopted to build an encoding model from each feature to voxels, establishing the relationship between images and voxels. Both real and synthetic data are used in this process. The semantic content of images is then leveraged to interpret the 256 features, followed by further analyses. The main results of this work show that category-related brain regions contain multiple overlapping visual preferences; different image categories share some functional patterns while also exhibiting category-specific preferred patterns.

Critical strengths
  • This work investigates fine-grained voxel-wise functional parcellation in detail, which carries clear neuroscientific implications.
  • The analysis design of this paper is well-rounded.
Critical weaknesses
  • This work analyzes the functional map using only a data-driven approach and lacks closed-loop validation through real visual stimulation experiments.
  • All analytical results in this paper are derived based on the 256 features with a top-8 feature retention scheme. The choice of this hyperparameter relies on the reconstruction accuracy of the sparse autoencoder. Therefore, further analysis is required to examine whether the conclusions drawn thus far are contingent on this hyperparameter selection.
  • Under the setting of 256-dimensional features with top-8 retention, most information from the images is discarded. However, voxel activity may well be modulated by this discarded information.
Questions for authors
  • Could the authors provide analytical results under different numbers of features and different top‑k retention thresholds? This is critical for assessing the robustness of the conclusions.

  • I am curious about the performance of the sparse autoencoder. In particular, this encoder involves cross-dataset generalization from ImageNet to NSD (MSCOCO). Currently, only the MSE metric in the representation space is reported. Could the reconstructed representations be mapped back to the pixel space to intuitively evaluate its performance?

  • Could the authors incorporate a voxel-to-image decoding model to analyze the performance of both the sparse autoencoder and the linear voxel encoding model?

  • It also seems worthwhile to explore functional preferences by manually adjusting the top-k response values.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.