- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes using sparse autoencoders on image features as a window into a more interpretable exploration of visual representations and their functional organization in the brain. The SAE is used to disentangle image representations into visual features, which together with a regression model from image feature space to fMRI responses recorded while people view images, allow the authors to construct a brain map for each feature. They evaluate these maps using preferred images, visual coherence scores, and comparisons with functional localizers.
Critical strengths
-
Understanding visual representations and their organization in the human cortex is an interesting and important problem.
-
Using SAE features learned from visual representations to interpret cortical organization is an interesting idea with potential scientific impact.
Critical weaknesses
- Limited validation of the discovered concepts.
Although the authors claim to discover hundreds of concepts, it remains unclear how many distinct, nonredundant concepts are reliably recovered and independently validated. The separation between concept selection and evaluation needs clarification, as does the characterization of the discovered concepts and their contribution beyond previous methods. Consistency across models and subjects is insufficiently established: direct brain map correspondence across models is reported for only three concepts, while the subject comparison relies on aggregate scores and selected examples from only two subjects (Questions 1 to 5).
- Limited ablations and analysis.
The ablations and analyses are insufficient to establish which components are necessary, the statistical significance of the findings, and the robustness and interpretation of the results. Missing comparisons without SAE decomposition, random controls, the small evaluation pool, the influence of weakly predicted voxels, and the interpretation of decoder steering all require further investigation (Questions 6 to 10).
Questions for authors
-
Concept selection and evaluation. How were the reported hundreds of interpretable concepts selected on the development pool, and how many of those same concepts retain high coherence and SAE to brain correspondence on the test pool? Please clarify the separation between selection and evaluation, particularly given the 10,000 candidates.
-
Characterization of discovered concepts. Can the authors provide descriptions and systematic qualitative and quantitative analyses of the discovered concepts? How many represent distinct properties, rather than redundant features or similar brain maps?
-
Advantage over previous methods. What does this method provide beyond previous large scale approaches using SAEs and the other methods discussed in the related work? Can this advantage be demonstrated experimentally?
-
Consistency across models. The most informative analysis would compare the brain maps themselves, both qualitatively and quantitatively, and establish which concepts are consistently recovered across models and which are not. Can the authors provide this analysis? Why were the three presented concepts selected?
-
Consistency across subjects. Similarly to Question 4, can the authors compare the brain maps across subjects after anatomical alignment, both qualitatively and quantitatively, and establish which concepts are consistently recovered and which are not? Comparing only two subjects is very limited; the authors should include at least the four main NSD subjects.
-
Missing ablation. How do the concept mapping results compare with those obtained using visual features without SAE decomposition?
-
Random controls and statistical significance. How do the results compare with regression from random image features? How do they compare with results derived from random score maps correlated with measured brain activity, or simply from random brain maps? How is statistical significance assessed given the 10,000 candidate maps?
-
Evaluation pool size. Do the findings hold with a larger and more diverse evaluation pool containing similar concepts and challenging distractors? Could the authors reserve more measured images for evaluation, or provide complementary analyses using independent in silico data?
-
Weakly predicted voxels. Are weakly predicted voxels included, and how do they affect the results? Do the findings remain consistent when restricting the analysis to reliably predicted voxels, rather than potentially reflecting projected visual features with limited support from measured responses?
-
Interpretation of steering. To what extent does steering validate the proposed cortical interpretation, rather than demonstrating control over the visual feature space learned by the brain decoder? What evidence distinguishes these explanations?
-
Figure clarity. The figures are generally crowded and difficult to follow, with small fonts and, in some cases, low resolution. Currently, the text is needed to understand the figures, rather than the figures helping explain the text and approach. It is difficult to understand what each panel shows and why. The ordering of Figures 3 and 4 is also inconsistent with the corresponding discussion.
-
Method section clarity. A method overview is missing, making the approach difficult to follow. Section 3 jumps into technical details before clearly explaining the SAE in visual feature space, the regression to fMRI responses, and how these components are used together. The section contains extensive notation and technical explanations, with insufficient intuition or explanation of methodological choices. Despite the detail in some parts, the basic approach remains unclear. In particular, the number of distinct, validated concepts is unclear: SAE features are not necessarily distinct concepts, and several could represent similar properties or produce very similar brain maps. The paper does not clearly establish how many nonredundant concepts are recovered or provide a precise operational definition supporting the claim of hundreds of discovered concepts.
10 comments
Reply 2 / 6 (Q2)
Q2: We are excited to expand the paper's analysis to explore the question of concept diversity, and filtering dictionaries for redundant, duplicate, or split concept atoms. We note that this problem remains an open question in both mechanistic interpretability and neuroscience. Without a standard metric of diversity, we follow a recent approach from neuroAI (Issa et al., 2026) and use perceptual diversity as a heuristic to reduce redundancy in a subselection of features. We again choose DreamSim for similarity comparisons and compute a “diversity score” measuring whether images are more similar to their own MEI set than to those from other feature’s images using leave-one-out comparisons, excluding exact image matches.
We sort all concepts using brain-side odd-one-out interpretability applied to K=8 brain-side MEIs then grow a greedy list by only adding a concept if its brain-side MEIs’ pairwise diversity exceeds 0.8 with every other feature in the growing list. We note that this approach does not take into account the concept’s spatial profile over the image, and thus cannot perfectly capture distinctions between granular features (e.g. “zebra neck” and “zebra head”) that may produce very common sets of MEIs.
A new Appendix G on concept diversity presents the results and an exhaustive analysis using the first 100 features in the greedy list. Figures 21-25 (20 concepts per page) display a posterior view of the concept map, prototypes, and both pairs of MEIs along activation heatmap and quantitative statistics (interpretability, coherence, reactivation) for a thorough, interpretable characterization of each. While this process does result in perceptually diverse MEI sets as discussed below, noticeable from row to row, it does not systematically promote diverse, interpretable, and coherent concepts (consistently interpretable images across a given row, coherent across all column sets). In practice, the overall consistency across the rows of paired MEI sets and prototypes can be quite high:
Several concepts in the resulting “diverse set” display such row-wise image consistency and, to our knowledge, have not previously been isolated in the NSD and reported on. Our joint diagnostics provide more exhaustive concept-level characterization than prior works as well. Among the 100 features characterized in Figures 21-25, using the in-figures concept indices (#) and concept number (c) for referencing, the reviewer can find:
- Individual food items like “pizza” or in groups like “plates of food”; (#02 c4494; #13 c9719)
- Separate kinds of animals like “zebras”, “bears” and “elephants” (#33 c8496, #48 c264; #93 c1704)
- Scene conditions or properties such as “low-light urban scenes” or “white walls”; (#10 c1955; #62 c7312)
- Synthetic image effects like “montage tiling” and “black & white photography”; (#01 c8726; #45 c9930)
- Highly-selective vehicle categories like “airplanes”, “trains”, “buses”; (#19 c9970; #82 c8276; #23 c6119)
- Perceptually-similar but distinct spaces like “grassy outdoors”, “baseball” courts and “tennis courts” (#14 c7203; #64 c2900; #66 c675)
Beyond these examples, we further quantified SAE-brain alignment for concepts in this “diverse set” by correlating the SAE max-patch activations against the spatial correlation between the fMRI response and the concept map. Alignment was positive for 98 of the 100 features with a median Pearson correlation of 0.195 (95% image-bootstrap confidence interval of 0.185-0.214). We further assess that the agreement exceeds chance by shuffling the correspondence 2,000 times and performing a two-sided test for each concept: after Benjamini-Hochberg correction for additional strictness under false discovery rate under 0.05, across 100 tests, 89 features showed a significant positive alignment. While future work on tailored approaches may offer complementary diverse sets, we find a simple greedy DreamSim-based approach can already help us surface diversity among, perceptually coherent features.
Reply 3 / 6 (Q12)
Q12: Echoing Reviewer rMZT, we have rewritten the Methods section to better motivate and introduce each of the elements that go into creating the recruitment matrix. We believe that the new explanations are an improvement, and better guide the reader through the technical details. We thank the reviewer for taking the time to write each of these comments.
As for the number of distinct, validated concepts: we recognize that SAE features are not necessarily distinct concepts, and may suffer from issues like under or oversplitting, or represent concepts mapping to similar cortical patterns, and explore efforts to reduce redundancy in the aforementioned additions to Appendix G.
It should be noted that the authors do not “make the claim of hundreds of discovered concepts”. Our contribution is a “scalable, interpretable mapping” framework to aid the discovery of selective concept maps by providing tools for their interpretation. Thus when we say that we “deliver hundreds of interpretable, distributed concept maps” in the concluding Discussion, we mean that our diagnostics render the selectivity of concept maps subject to analysis and interpretation. This is a crucial distinction between “providing an interpretable feature discovery” and “providing an interpretation of a discovered feature”, and it is consistent with established methods in interpretability, including SAE feature discovery, feature visualization, and local explanation methods. In general, modeling approaches that render representations, mechanics or parameters amenable to human understanding are referred to as “interpretable”, which informs our usage of the term for this method and its outputs.
Throughout our manuscript, we emphasize the need for additional validation of discovered features, and we contribute diagnostic tools for this purpose. Across several examples presented, perceptual similarity of these concepts is clear to the naked eye, and many appear to code for very particular semantics (“baseball courts”, “black-and-white photography”, “kid faces”), which we qualitatively exemplify with tens of concept maps and/or MEIs, as well as our “working interpretation” of them through labels. We do not mean to say that we deliver hundreds of already- and definitively-interpreted or human-validated concepts (to our knowledge, nobody has claimed such an achievement either in neuroscience nor AI) but we do nonetheless find hundreds of concept maps with strong perceptual alignment, and present several psychophysically-validated examples with compelling selectivity. To reveal this selectivity and make it amenable to interpretation, we borrow from neuroAI and MI, constructing measures of perceptual coherence within and across sets and the map effects through independent decoders.
In terms of counting features, Sections 4-5 do offer thresholds that count hundreds of perceptually coherent concepts, but only in narrow terms of perceptual similarity, such as the I>0.8 criterion reported in Section 4 for interpretability or the overlap higher than 5 at K=50 for coherence. We show that hundreds of concepts satisfy such criterions through text and via the histogram counts in Section 5, while reporting hundreds passing each of our chance nulls and controls. As can be inferred from the scatter density in Figure 2 and the histogram in Figure 3 (Figure 4 of the old submission), even more stringent thresholds on the same measures result in hundreds of passing concepts still (i.e. 1,268 concepts with brain-side I>0.975 and over 1,126 concepts with overlap > 10) . We also offered examples sampled at various score percentiles in the figures from Section 4 and Section 5 to provide some intuition as to the correspondence between these numbers and actual psychophysical similarity (as judged by each reader). In the Discussion, we point out that all such thresholds remain descriptive pending a dedicated psychophysics task comparison, which is why we do not center them in our exposition.
In summary, when we say our interpretable mapping framework delivers hundreds of concept maps, and the tools to validate them ahead of additional (i.e. in vivo) verification experiments, it does not imply that hundreds of concepts have been newly discovered, nor that these maps have undergone a finalizing interpretation that we provide. We nonetheless find “strong perceptual coherence on both sides of the mapping” and “strong identity preservation for hundreds of concepts”, while offering several examples that qualitatively reflect our findings. In particular, we have expanded our qualitative and quantitative characterization of NSD concepts we believe to be novel in this rebuttal, as well as the new Section 7 and appendices. Finally, we have also amended the language in the discussion, i.e. “deliver hundreds of interpretable, distributed concept maps” to “deliver hundreds of concept maps (...), with tools to interpret them”, in order to minimize ambiguity.
Reply 1 / 6 (Q5)
We thank the Reviewer eDoE for the extensive, detailed comments and feedback, as they have been instrumental in expanding our experiments, polishing our presentation, and deepening the insights gained from this work. The manuscript has been updated with new content (Section 7), a new appendix on concept diversity (Appendix G). We also enriched our cross-embedding results (Appendix C) and expanded our analysis to the four core NSD subjects (Appendix D).
We are glad that you found the problem important and our approach interesting with potential scientific impact and are happy to provide answers to each of your questions. In what follows, we present our efforts at addressing weaknesses (within answers to Q2, Q4, Q5, Q12) and remaining questions one-by-one in separate replies due to character limits. We include headers indexing replies in hopes that it is helpful.
—
Q5 We thank the reviewer for their suggestion, and have extended our analysis to the four NSD Subjects (S1, S2, S5, S7) as as a result, fitting subject-specific encoders to both subject-native and anatomically-registered fsaverage surfaces using all V=37, 984 in the nsdgeneral ROI. The recipe for these encoder fits and subsequent estimation of the recruitment matrices are identical as in the main analysis of Sections 4. To accommodate the new analysis, we have updated the manuscript with a new Section 7 on anatomical and semantic consistency. Lastly, we rework Appendix D with a complete account of the quantitative findings in Sections 4 for each subject, finding them to recur, as well as additional qualitative and quantitative cross-subject comparisons.
We summarize our main findings on cross-subject consistency below:
For 9,987 valid concepts and all six subject pairings, we calculate the spatial Pearson correlation between matching-concept maps and different concept maps. The matching-concept maps correlated at an average of r=0.246 versus just 0.003 for different concepts, and pairwise means ranged from 0.219 to 0.268 across all subjects. The 95th percentile is r=0.462, meaning several hundred concept maps meet or exceed this value. Visual inspection of highly cross-correlated concept maps reveals clear signed regional overlap of broad areas like FFA, EBA, PPA for semantically-appropriate concepts (faces, bodies, places) as determined by the subject’s brain-side MEIs, showing that mean spatial correlation can surface visibly-aligned signed surfaces.
Examples can be found in Figure 6c, which features apparent cortical overlap of signed regions on the posterior brain, but also highly semantically-aligned MEIs for features that include “faces”, “bodies in motion”, “prepared foods” and “aircrafts in the sky”. We note that matching subjects to a shared surface does not automatically result in functional alignment: for example, the boundaries of regions like FFA will reflect subject-specific differences in the shared space, resulting in partial overlap and making perfect spatial alignment impossible. In spite of this, we see high concept map agreement along broad regions, and find that sorting by vertex correlation alone can help in finding such visibly-aligned cortical surfaces.
We inspect further with a complementary analysis, using an alignment-independent comparison based on MEIs that we retrieve with the reserved “shared 1,000” test image pool. We estimate coherence through Top-K overlap at 50 MEIs: matching concepts share 14.51 of their top 50 images versus just 3.48 for different concepts, and 2.50 for uniformly-sampled image sets as a null comparison (Figure 6). We estimate consistency by deeming a concept “consistent” when it matches its counterpart in another subject better than at least 95% of that subject’s other concepts, and this requirement had to hold for all retrieval combinations for the six subjects.
Using this matching-based criterion, 2,088 brain maps meet the “consistent”. We provide additional extensive characterizations of several concepts from this consistent set in Appendix D for a qualitative characterization. From this set of 50 concepts from the consistent set, we find that the cross-aligned brain-side MEIs correspond to sports and bodies motion (in figure feature indices: 2,3,5,6,9,13,15,28,30,33,35,41,44,50 in Figure 16-17) as well as food or saturated colors (6,23,26,3,46), with the cortical surface showing positive weight (red) along body-, motion-, food- and color-selective areas.
The same appendix includes the aforementioned subject-level findings with recurring results from Sections 4, as well as additional cross-subject details and ablations. We thank the reviewer for their interest in the qualitative side of this evaluation.