- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper aims to relate the functional organization of human visual cortex to sparse features learned from vision Transformers. Using the Natural Scenes Dataset (fMRI), the authors derive “functional modes” (FMs) from sparse dictionaries of several pretrained ViTs, map them onto cortex for each subject and model, analyze their arrangement across cortex, and group them into “recurrent families” shared across subjects and models.
The paper claims to contribute the following: a method for linking sparse, multi-layer model features to cortical responses; a set of FM families that recur across subjects and architectures; a low-dimensional description of cortical organization, obtained by applying PCA to the FM profiles of cortical locations, which the authors present as an atlas; and a link between recurrence and compositional structure. In my reading, the novelty and impact relative to prior work remain unclear. As far as I can tell, the paper does not reveal anything new about the organization of visual cortex. It also does not show that the method performs better than existing approaches, and I could not find comparisons to the state of the art. More broadly, the paper is not well written: its goals, motivation, and contributions are not clearly defined.
The validation is mostly qualitative. The main results consist of example images and cortical heatmaps for a single participant and model, shown without color scales or area outlines; PCA axes interpreted through their top-loading image categories; and families defined by a similarity threshold. The paper does not quantify how robustly FMs map onto cortical areas, how reliable they are under cross-validation, or how consistent they are across subjects and models. It also does not compare the approach against baselines, so it remains unclear what the sparse dictionaries add over existing model-based or data-driven methods.
Critical strengths
Relevant question with suitable data: The idea of examining how sparse model features compose across cortex and generalize across participants is interesting and could advance our understanding of the organization of visual cortex. The NSD dataset, with its 1,000 images shared across participants, is well suited to this question. This is the main reason I consider the work worth pursuing. In the current manuscript, however, the question is not addressed with sufficient clarity or validation to support acceptance.
Critical weaknesses
Unclear goals, contributions, and methods: The manuscript does not clearly define its goals and contributions, and central terms such as family and recurrence are left undefined. The methods are unclear from the main text, and critical information for interpreting the results is missing, including how the FMs are obtained and to what degree they depend on neural data.
Insufficient validation and unclear advance: The results are, in part, qualitative heatmaps without color scales or area outlines, focused on a single participant and model. The paper does not quantify how robustly the FMs map onto cortical areas or how reliable they are under cross-validation. This is consequential because it leaves the advance of the paper unclear. As far as I can tell, the advance is not scientific, since the paper does not reveal anything new about visual cortex. If it is methodological, it would need to be supported by comparisons against baselines and the state of the art, which the paper does not provide.
Questions for authors
How exactly are the FMs obtained from the SAE responses? What objective is optimized, and which parameters, if any, are fit to neural data? The answer determines whether the FMs reflect cortical structure or mainly properties of the model dictionaries.
Are results for all participants and models reported anywhere, and do they show the same structure as S5? If this analysis exists, the main claims could rest on more than a single case.
Is there any quantitative evaluation of how well the FMs map onto cortical areas, or of their reliability, for example under split-half cross-validation?
What do the authors consider the main novel contribution, scientific or methodological, relative to prior work linking cortical location to visual preference?
How does the approach perform compared to baselines and existing methods? For example, what do the sparse dictionaries add over dense model features without the SAE, or over a voxel decomposition without a model? What's the predictive accuracy compared to other methods?