acceptodds
Under review as a conference paper at ICLR 2027

Scalable Interpretable Mapping of Hundreds of Visual Concepts in Human Brain Activity

Abstract

Systems neuroscience and mechanistic interpretability share a central challenge: identifying interpretable components within high-dimensional, distributed representations. Recently, sparse autoencoders (SAEs) have been shown to yield rich concept dictionaries that aid in the interpretation of deep embeddings in vision foundation models. This suggests that SAEs might be a similarly powerful tool for interpreting brain activity patterns in the visual system, where a relatively small set of concept maps has been established by decades of dedicated functional localization experiments. However, limited samples and measurement noise complicate the naive application of SAEs to brain recordings. Here we use SAEs to perform scalable, interpretable mapping of visual concepts in human brain activity by following an SAE dictionary derived from a vision foundation model through a fitted voxelwise fMRI encoder. Correlating these maps with measured brain activity retrieves recognizable stimulus sets, including faces, surfers, black-and-white photographs, and tiled montages. Comparison with functional localizers supports consistency with known cortical organization, while measures of visual coherence and model-brain correspondence reveal preservation of feature identity through the fitted mapping. The same maps support intervention as well as interpretation: off-the-shelf brain decoders turn them into concept-reactivating images, and adding them to recorded neural responses produces targeted changes in reconstructed content and appearance. Across large-scale steering experiments, specific target-feature reactivation increases far beyond random map perturbations. We deliver hundreds of concept maps for the visual cortex, expanding the vocabulary of selectivity and validating their interpretation through complementary methods from neuroscience and mechanistic interpretability.

Primary area:
applications to neuroscience & cognitive science
Keywords:
mechanistic interpretability, systems neuroscience, neuroai, sparse autoencoders

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eDoEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes using sparse autoencoders on image features as a window into a more interpretable exploration of visual representations and their functional organization in the brain. The SAE is used to disentangle image representations into visual features, which together with a regression model from image feature space to fMRI responses recorded while people view images, allow the authors to construct a brain map for each feature. They evaluate these maps using preferred images, visual coherence scores, and comparisons with functional localizers.

Critical strengths
  1. Understanding visual representations and their organization in the human cortex is an interesting and important problem.

  2. Using SAE features learned from visual representations to interpret cortical organization is an interesting idea with potential scientific impact.

Critical weaknesses
  1. Limited validation of the discovered concepts.

Although the authors claim to discover hundreds of concepts, it remains unclear how many distinct, nonredundant concepts are reliably recovered and independently validated. The separation between concept selection and evaluation needs clarification, as does the characterization of the discovered concepts and their contribution beyond previous methods. Consistency across models and subjects is insufficiently established: direct brain map correspondence across models is reported for only three concepts, while the subject comparison relies on aggregate scores and selected examples from only two subjects (Questions 1 to 5).


  1. Limited ablations and analysis.

The ablations and analyses are insufficient to establish which components are necessary, the statistical significance of the findings, and the robustness and interpretation of the results. Missing comparisons without SAE decomposition, random controls, the small evaluation pool, the influence of weakly predicted voxels, and the interpretation of decoder steering all require further investigation (Questions 6 to 10).

Questions for authors
  1. Concept selection and evaluation. How were the reported hundreds of interpretable concepts selected on the development pool, and how many of those same concepts retain high coherence and SAE to brain correspondence on the test pool? Please clarify the separation between selection and evaluation, particularly given the 10,000 candidates.

  2. Characterization of discovered concepts. Can the authors provide descriptions and systematic qualitative and quantitative analyses of the discovered concepts? How many represent distinct properties, rather than redundant features or similar brain maps?

  3. Advantage over previous methods. What does this method provide beyond previous large scale approaches using SAEs and the other methods discussed in the related work? Can this advantage be demonstrated experimentally?

  4. Consistency across models. The most informative analysis would compare the brain maps themselves, both qualitatively and quantitatively, and establish which concepts are consistently recovered across models and which are not. Can the authors provide this analysis? Why were the three presented concepts selected?

  5. Consistency across subjects. Similarly to Question 4, can the authors compare the brain maps across subjects after anatomical alignment, both qualitatively and quantitatively, and establish which concepts are consistently recovered and which are not? Comparing only two subjects is very limited; the authors should include at least the four main NSD subjects.

  6. Missing ablation. How do the concept mapping results compare with those obtained using visual features without SAE decomposition?

  7. Random controls and statistical significance. How do the results compare with regression from random image features? How do they compare with results derived from random score maps correlated with measured brain activity, or simply from random brain maps? How is statistical significance assessed given the 10,000 candidate maps?

  8. Evaluation pool size. Do the findings hold with a larger and more diverse evaluation pool containing similar concepts and challenging distractors? Could the authors reserve more measured images for evaluation, or provide complementary analyses using independent in silico data?

  9. Weakly predicted voxels. Are weakly predicted voxels included, and how do they affect the results? Do the findings remain consistent when restricting the analysis to reliably predicted voxels, rather than potentially reflecting projected visual features with limited support from measured responses?

  10. Interpretation of steering. To what extent does steering validate the proposed cortical interpretation, rather than demonstrating control over the visual feature space learned by the brain decoder? What evidence distinguishes these explanations?

  11. Figure clarity. The figures are generally crowded and difficult to follow, with small fonts and, in some cases, low resolution. Currently, the text is needed to understand the figures, rather than the figures helping explain the text and approach. It is difficult to understand what each panel shows and why. The ordering of Figures 3 and 4 is also inconsistent with the corresponding discussion.

  12. Method section clarity. A method overview is missing, making the approach difficult to follow. Section 3 jumps into technical details before clearly explaining the SAE in visual feature space, the regression to fMRI responses, and how these components are used together. The section contains extensive notation and technical explanations, with insufficient intuition or explanation of methodological choices. Despite the detail in some parts, the basic approach remains unclear. In particular, the number of distinct, validated concepts is unclear: SAE features are not necessarily distinct concepts, and several could represent similar properties or produce very similar brain maps. The paper does not clearly establish how many nonredundant concepts are recovered or provide a precise operational definition supporting the claim of hundreds of discovered concepts.

10 comments

Reply 1 / 6 (Q5)

We thank the Reviewer eDoE for the extensive, detailed comments and feedback, as they have been instrumental in expanding our experiments, polishing our presentation, and deepening the insights gained from this work. The manuscript has been updated with new content (Section 7), a new appendix on concept diversity (Appendix G). We also enriched our cross-embedding results (Appendix C) and expanded our analysis to the four core NSD subjects (Appendix D).

We are glad that you found the problem important and our approach interesting with potential scientific impact and are happy to provide answers to each of your questions. In what follows, we present our efforts at addressing weaknesses (within answers to Q2, Q4, Q5, Q12) and remaining questions one-by-one in separate replies due to character limits. We include headers indexing replies in hopes that it is helpful.

—

Q5 We thank the reviewer for their suggestion, and have extended our analysis to the four NSD Subjects (S1, S2, S5, S7) as as a result, fitting subject-specific encoders to both subject-native and anatomically-registered fsaverage surfaces using all V=37, 984 in the nsdgeneral ROI. The recipe for these encoder fits and subsequent estimation of the recruitment matrices are identical as in the main analysis of Sections 4. To accommodate the new analysis, we have updated the manuscript with a new Section 7 on anatomical and semantic consistency. Lastly, we rework Appendix D with a complete account of the quantitative findings in Sections 4 for each subject, finding them to recur, as well as additional qualitative and quantitative cross-subject comparisons.

We summarize our main findings on cross-subject consistency below:

For 9,987 valid concepts and all six subject pairings, we calculate the spatial Pearson correlation between matching-concept maps and different concept maps. The matching-concept maps correlated at an average of r=0.246 versus just 0.003 for different concepts, and pairwise means ranged from 0.219 to 0.268 across all subjects. The 95th percentile is r=0.462, meaning several hundred concept maps meet or exceed this value. Visual inspection of highly cross-correlated concept maps reveals clear signed regional overlap of broad areas like FFA, EBA, PPA for semantically-appropriate concepts (faces, bodies, places) as determined by the subject’s brain-side MEIs, showing that mean spatial correlation can surface visibly-aligned signed surfaces.

Examples can be found in Figure 6c, which features apparent cortical overlap of signed regions on the posterior brain, but also highly semantically-aligned MEIs for features that include “faces”, “bodies in motion”, “prepared foods” and “aircrafts in the sky”. We note that matching subjects to a shared surface does not automatically result in functional alignment: for example, the boundaries of regions like FFA will reflect subject-specific differences in the shared space, resulting in partial overlap and making perfect spatial alignment impossible. In spite of this, we see high concept map agreement along broad regions, and find that sorting by vertex correlation alone can help in finding such visibly-aligned cortical surfaces.

We inspect further with a complementary analysis, using an alignment-independent comparison based on MEIs that we retrieve with the reserved “shared 1,000” test image pool. We estimate coherence through Top-K overlap at 50 MEIs: matching concepts share 14.51 of their top 50 images versus just 3.48 for different concepts, and 2.50 for uniformly-sampled image sets as a null comparison (Figure 6). We estimate consistency by deeming a concept “consistent” when it matches its counterpart in another subject better than at least 95% of that subject’s other concepts, and this requirement had to hold for all retrieval combinations for the six subjects.

Using this matching-based criterion, 2,088 brain maps meet the “consistent”. We provide additional extensive characterizations of several concepts from this consistent set in Appendix D for a qualitative characterization. From this set of 50 concepts from the consistent set, we find that the cross-aligned brain-side MEIs correspond to sports and bodies motion (in figure feature indices: 2,3,5,6,9,13,15,28,30,33,35,41,44,50 in Figure 16-17) as well as food or saturated colors (6,23,26,3,46), with the cortical surface showing positive weight (red) along body-, motion-, food- and color-selective areas.

The same appendix includes the aforementioned subject-level findings with recurring results from Sections 4, as well as additional cross-subject details and ablations. We thank the reviewer for their interest in the qualitative side of this evaluation.

Reply 2 / 6 (Q2)

Q2: We are excited to expand the paper's analysis to explore the question of concept diversity, and filtering dictionaries for redundant, duplicate, or split concept atoms. We note that this problem remains an open question in both mechanistic interpretability and neuroscience. Without a standard metric of diversity, we follow a recent approach from neuroAI (Issa et al., 2026) and use perceptual diversity as a heuristic to reduce redundancy in a subselection of features. We again choose DreamSim for similarity comparisons and compute a “diversity score” measuring whether images are more similar to their own MEI set than to those from other feature’s images using leave-one-out comparisons, excluding exact image matches.

We sort all concepts using brain-side odd-one-out interpretability applied to K=8 brain-side MEIs then grow a greedy list by only adding a concept if its brain-side MEIs’ pairwise diversity exceeds 0.8 with every other feature in the growing list. We note that this approach does not take into account the concept’s spatial profile over the image, and thus cannot perfectly capture distinctions between granular features (e.g. “zebra neck” and “zebra head”) that may produce very common sets of MEIs.

A new Appendix G on concept diversity presents the results and an exhaustive analysis using the first 100 features in the greedy list. Figures 21-25 (20 concepts per page) display a posterior view of the concept map, prototypes, and both pairs of MEIs along activation heatmap and quantitative statistics (interpretability, coherence, reactivation) for a thorough, interpretable characterization of each. While this process does result in perceptually diverse MEI sets as discussed below, noticeable from row to row, it does not systematically promote diverse, interpretable, and coherent concepts (consistently interpretable images across a given row, coherent across all column sets). In practice, the overall consistency across the rows of paired MEI sets and prototypes can be quite high:

Several concepts in the resulting “diverse set” display such row-wise image consistency and, to our knowledge, have not previously been isolated in the NSD and reported on. Our joint diagnostics provide more exhaustive concept-level characterization than prior works as well. Among the 100 features characterized in Figures 21-25, using the in-figures concept indices (#) and concept number (c) for referencing, the reviewer can find:

  • Individual food items like “pizza” or in groups like “plates of food”; (#02 c4494; #13 c9719)
  • Separate kinds of animals like “zebras”, “bears” and “elephants” (#33 c8496, #48 c264; #93 c1704)
  • Scene conditions or properties such as “low-light urban scenes” or “white walls”; (#10 c1955; #62 c7312)
  • Synthetic image effects like “montage tiling” and “black & white photography”; (#01 c8726; #45 c9930)
  • Highly-selective vehicle categories like “airplanes”, “trains”, “buses”; (#19 c9970; #82 c8276; #23 c6119)
  • Perceptually-similar but distinct spaces like “grassy outdoors”, “baseball” courts and “tennis courts” (#14 c7203; #64 c2900; #66 c675)

Beyond these examples, we further quantified SAE-brain alignment for concepts in this “diverse set” by correlating the SAE max-patch activations against the spatial correlation between the fMRI response and the concept map. Alignment was positive for 98 of the 100 features with a median Pearson correlation of 0.195 (95% image-bootstrap confidence interval of 0.185-0.214). We further assess that the agreement exceeds chance by shuffling the correspondence 2,000 times and performing a two-sided test for each concept: after Benjamini-Hochberg correction for additional strictness under false discovery rate under 0.05, across 100 tests, 89 features showed a significant positive alignment. While future work on tailored approaches may offer complementary diverse sets, we find a simple greedy DreamSim-based approach can already help us surface diversity among, perceptually coherent features.

Reply 3 / 6 (Q12)

Q12: Echoing Reviewer rMZT, we have rewritten the Methods section to better motivate and introduce each of the elements that go into creating the recruitment matrix. We believe that the new explanations are an improvement, and better guide the reader through the technical details. We thank the reviewer for taking the time to write each of these comments.

As for the number of distinct, validated concepts: we recognize that SAE features are not necessarily distinct concepts, and may suffer from issues like under or oversplitting, or represent concepts mapping to similar cortical patterns, and explore efforts to reduce redundancy in the aforementioned additions to Appendix G.

It should be noted that the authors do not “make the claim of hundreds of discovered concepts”. Our contribution is a “scalable, interpretable mapping” framework to aid the discovery of selective concept maps by providing tools for their interpretation. Thus when we say that we “deliver hundreds of interpretable, distributed concept maps” in the concluding Discussion, we mean that our diagnostics render the selectivity of concept maps subject to analysis and interpretation. This is a crucial distinction between “providing an interpretable feature discovery” and “providing an interpretation of a discovered feature”, and it is consistent with established methods in interpretability, including SAE feature discovery, feature visualization, and local explanation methods. In general, modeling approaches that render representations, mechanics or parameters amenable to human understanding are referred to as “interpretable”, which informs our usage of the term for this method and its outputs.

Throughout our manuscript, we emphasize the need for additional validation of discovered features, and we contribute diagnostic tools for this purpose. Across several examples presented, perceptual similarity of these concepts is clear to the naked eye, and many appear to code for very particular semantics (“baseball courts”, “black-and-white photography”, “kid faces”), which we qualitatively exemplify with tens of concept maps and/or MEIs, as well as our “working interpretation” of them through labels. We do not mean to say that we deliver hundreds of already- and definitively-interpreted or human-validated concepts (to our knowledge, nobody has claimed such an achievement either in neuroscience nor AI) but we do nonetheless find hundreds of concept maps with strong perceptual alignment, and present several psychophysically-validated examples with compelling selectivity. To reveal this selectivity and make it amenable to interpretation, we borrow from neuroAI and MI, constructing measures of perceptual coherence within and across sets and the map effects through independent decoders.

In terms of counting features, Sections 4-5 do offer thresholds that count hundreds of perceptually coherent concepts, but only in narrow terms of perceptual similarity, such as the I>0.8 criterion reported in Section 4 for interpretability or the overlap higher than 5 at K=50 for coherence. We show that hundreds of concepts satisfy such criterions through text and via the histogram counts in Section 5, while reporting hundreds passing each of our chance nulls and controls. As can be inferred from the scatter density in Figure 2 and the histogram in Figure 3 (Figure 4 of the old submission), even more stringent thresholds on the same measures result in hundreds of passing concepts still (i.e. 1,268 concepts with brain-side I>0.975 and over 1,126 concepts with overlap > 10) . We also offered examples sampled at various score percentiles in the figures from Section 4 and Section 5 to provide some intuition as to the correspondence between these numbers and actual psychophysical similarity (as judged by each reader). In the Discussion, we point out that all such thresholds remain descriptive pending a dedicated psychophysics task comparison, which is why we do not center them in our exposition.

In summary, when we say our interpretable mapping framework delivers hundreds of concept maps, and the tools to validate them ahead of additional (i.e. in vivo) verification experiments, it does not imply that hundreds of concepts have been newly discovered, nor that these maps have undergone a finalizing interpretation that we provide. We nonetheless find “strong perceptual coherence on both sides of the mapping” and “strong identity preservation for hundreds of concepts”, while offering several examples that qualitatively reflect our findings. In particular, we have expanded our qualitative and quantitative characterization of NSD concepts we believe to be novel in this rebuttal, as well as the new Section 7 and appendices. Finally, we have also amended the language in the discussion, i.e. “deliver hundreds of interpretable, distributed concept maps” to “deliver hundreds of concept maps (...), with tools to interpret them”, in order to minimize ambiguity.

Reviewer_WfmcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper maps a vision-model SAE dictionary through a fitted fMRI encoder to obtain cortical concept maps. It evaluates these maps using preferred-image retrieval, functional localizers and image reconstruction. Perturbing recorded responses along selected maps increases the corresponding concept activation in decoded images.

Critical strengths
  1. The construction is simple and useful: each cortical map retains a link to a specific SAE feature, making its preferred images, spatial pattern and decoded appearance available for joint inspection.
  2. Held-out response retrieval and independent functional localizers provide evidence beyond selected visual examples. The second-subject and alternative-embedding results also give some support for the framework’s broader usefulness.
Critical weaknesses
  1. Evidence for preserving specific concept identity is less convincing than evidence for coherent image sets. Section 4.2 reports only 0.2508 shared images in the top-eight sets, with 7,905 concepts sharing none. Limited stimulus coverage can explain part of this, but DreamSim coherence alone does not establish that both sets express the same concept. The localizers validate broad categories. A blind assessment of randomly sampled concepts would better support how many finer concepts actually survive the mapping.
  2. The steering result is convincing as decoder manipulation, but only indirect evidence of cortical selectivity. Concepts are selected from a high-overlap subset and scored using the source SAE. At full spherical alignment, the input follows the concept-map direction, while similarity to the original reconstruction falls to 0.190 (Section 6). Target gains at matched content preservation, assessed with an independent image measure, would provide stronger evidence of selective editing.
Questions for authors
  1. Additional analysis: For concepts chosen without inspecting the test images, what fraction retain the same meaning under a blinded assessment, and how consistent is that fraction across the two subjects?
  2. Additional analysis: Do target-versus-control gains remain at moderate steering strengths when rotation angle and preservation of the original image content are matched?
3 comments

We thank the reviewers for their valuable and insightful feedback. The authors appreciate the comments about the simple and useful construction, a core feature of our contribution. While retaining the link to the specific SAE feature is what unlocks the joint diagnostics of interpretability, our method’s simplicity makes it applicable to off-the-shelf encoders and decoders, using off-the-shelf dictionaries, enabling cross-use across the mechanistic interpretability and neuroscience communities. Furthermore, linking cortical patterns to fine-grained concepts is what enables their comparison as neural features across subjects. To demonstrate this aspect of our contribution and to address feedback provided by Reviewers rMZT & eDoE, we provide additional data by expanding our cross-subject and alternative-embedding analyzes: our quantitative analysis in the main text now reflects all four core NSD subjects (1, 2, 5, 7), and we include additional qualitative examples in the corresponding appendices. The same simple construction is applied across subjects and embedding spaces, demonstrating general utility.

This review remarks that evidence for preserving specific concept identity is not as convincing as that for perceptually coherent image sets. While the majority of concepts do not share images, we note that 76.24% of the concepts feature an image overlap above the reported chance nulls, and that hundreds of these fall under the diagnostic regime at which qualitative interpretability is strong. In terms of odd-one-out interpretability (I), we originally included the same threshold of I>0.8 as in Issa et al., 2026, and reported 8,384 (84%) of concepts meet that criterion. However, we note that this threshold was based on MEI inspection of a different image distribution by a separate group: in our experience, MEIs with I>0.975 are reliably interpretable with our framework, and we find 1,268 concepts satisfying this criterion on brain-side MEIs. The authors noted in the Discussion that such thresholds nonetheless remain descriptive. We provide interpretable concepts by joining several diagnostics together into a single framework, and emphasize that while hundreds of concepts admit high interpretability and coherence, all such thresholds are limited. Even then, the measures introduced in this work for interpretability and coherence allow useful sorting and filtering, simplifying the process of psychophysics validation. We believe it to be important to link several sources of evidence as diagnostics for interpretability, a common practice in the MI community.

The authors agree that limited stimulus coverage can explain the concepts with no coherence across model and brain, and that DreamSim similarity alone cannot establish that both image sets express the same concept. The manuscript emphasized DreamSim as strictly measuring perceptual similarity, and we have indeed flagged examples where DreamSim reports low perceptual similarity yet an underlying and specific correspondence is plausible: one such example is shown in Figure 4c, final row, where the underlying feature triggers on seemingly unrelated images, but in both cases visualizing heatmaps reveal activation along the circular outline of plates (left) and mirrors (right). We have addressed the manuscript text to appropriately reflect this feedback and our responses. We also emphasize that DreamSim is only employed for some diagnostics in this work: Top-K overlap, prototype decoding, and reactivation measurements from prototypes or steering, all introduce additional sources of interpretability and strong evidence of coherence.

With respect to steering interventions, the authors agree that it offers limited evidence of cortical selectivity, which we position as an additional diagnostic to determine the recruited feature’s identity. The review notes that at full alignment, similarity to the base reconstruction decreases substantially. This result at the extreme end of steering is expected, as the endpoint steered response and concept map differ only in their norm. We emphasize that our steering analysis concerns itself with the dose response curve and how the steered feature induces reactivation irrespective of the original content at the end of the process, and do not make claims about specific content retention. The examples in Figure 5 demonstrate that this is nonetheless a property of our method (steering “stop/traffic signs” adds signs onto the tower sample and indoor sample at 10°, behind the animals in the outdoor sample at 20°), but should not be taken as guarantees. Steering may not neatly preserve content, and the measurement of steering interventions enjoys concurrent theoretical and empirical research on steering evaluation could benefit our approach, it remains beyond the scope of this work.

Thank you for your rebuttal. You have extended the quantitative analysis to all four core NSD subjects, which provides stronger evidence of the framework's applicability across subjects. You also acknowledge that DreamSim measures perceptual similarity rather than establishing shared concept identity, and further clarify the limitations of the steering results. These clarifications partially address my previous concerns regarding the scope of the paper's claims.

The proposed method is simple and establishes a connection between sparse autoencoder (SAE) features and cortical maps. Combined with the complementary evidence from retrieval, localizer, and reconstruction experiments, I believe the framework represents a valuable research contribution.

Therefore, I maintain my original rating of 3 (weak acceptance).

The authors thank you once more for your continued engagement with our work and the helpful feedback.

Reviewer_rMZTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies a mechanistic interpretability method to fmri data.

Critical strengths
  • The first application of SAE's to the frmi datasets we have as far as I know.

  • Mostly clear writing (see below though).

Critical weaknesses
  • Exposition is not clear for the core part: ** What is a_s, what is a patch here and how is it embedded? What is s? ** What is beta measuring? It looks like it is mapping to the voxel space. Then why is it an encoder?

  • As we don't have brain activity data for the fine-grained concepts this work is trying to capture, it is hard to say the method actually gives us new insights and methods to study the brain. Although I appreciate the attempt in sec. 4 & 5, it doesn't seem conclusive due to the lack of data. Sure, if you have a latent representation of a signal, make is sparse, and re-project to the primary space, you should get good alignment with the original signal. But this does not say much to me.

  • Overall this looks like an initial experiment and might be better suited for a focused venue.

Questions for authors

Please see above.

2 comments

Reply 1 / 2

We thank the reviewers for the valuable and insightful feedback. The authors appreciate your commentary about the novelty of the work. While other significant efforts have applied sparse autoencoders to fMRI recordings, as noted in the Related Works section, our method does not require supervision or labels, anatomical constraints, or synthetic data, while enabling novel diagnostics for interpretability. We separate our answers in two replies due to character constraints.

As for writing clarity, again we appreciate the recognition for clear writing. In order to address your specific question, as well as commentary from Reviewer eDoE, we have updated the manuscript text with new figure captions and reworked Section 3 into a streamlined “Methods” section. This section offers additional exposition which we hope makes reading it easier and more intuitive, in particular with respect to the introduction of variable names and clarifications around terms.

We reserve s to index patch positions, and a_s refers to an embedding patch of dimension dd, with the experiments using d=768 corresponding to the final residual stream layer of the DINOv2 model, and it’s obtained by feeding the input stimulus to the model off-the-shelf. Beta's (β\beta) are a commonly used shorthand for estimates of the fMRI response amplitude of each voxel to each trial conducted, and here we use “betas” as the plural term. The NSD offers several such beta estimates, and we employ the "betas_fithrf_GLMdenoise_RR" release (Appendix A). You are correct that they correspond to voxel measurements, and that the neural encoder maps to voxel space, i.e., a space where each axis represents the activation of a single voxel. As a result of these requests for clarification, we have now moved clearer data description to the new Methods section for additional clarity. We reserved the term “encoder” to describe mappings from feature spaces to voxel spaces (Naselaris et al., 2011); we have included the term “voxelwise encoder model” and clarified the mapping to avoid confusion in the updated Methods section.

You are correct in identifying the lack of fMRI recordings for fine-grained concepts as a difficulty when it comes to validating these insights and methods. Our SAE uncovers fine-grained concepts for all kinds of visual concepts, including obscure ones like "dog jowl" and "the bottom bun of a hamburger": there are no data sets with stimuli that are specially prepared to isolate so many distinct concepts and measure brain activation in response to the concept alone. Our viewpoint is that running bespoke experiments to map out many hundreds of fine-grained concepts is not tenable or desirable in the first place. A major rationale for this work is to be able to infer the maps for these concepts from activations in a dense, natural scene dataset and alleviate the need for such recordings. This disentanglement gives us virtual labels derived from the feature space itself, allowing us to validate fine-grained concepts even in the absence of bespoke experiments. Note that when using concept maps to retrieve MEIs, we are effectively cross-validating our maps against held out brain activity data. Thus, we contend the "novel" data needed to validate the concept maps are part of the NSD itself.

Reply 2 / 2

To address these concerns about data availability, and echoing Reviewer eDoE’s request for experiments on additional subjects, we have now extended our analysis to the four NSD subjects who completed all three repetitions for all test images in the NSD experiment, computing concept maps in both subject-native voxels and common anatomically-registered fsaverage vertices. The results can be found in the new Section 7. In terms of anatomical alignment, across all six subject pairings, matching-concept maps correlated at an average of r=0.246 versus just 0.003 for different concepts. In terms of semantic coherence, we find that matching concepts across subjects shared 14.51 of their top 50 test images in the brain MEIs versus just 3.48 for different concepts, and 2.40 for uniformly-sampled image sets as a null.

Furthermore, we demonstrate that these diagnostics are useful for filtering incoherent concepts and finding highly aligned concepts. Figure 6 shows concept maps and brain-side MEIs from a set of high coherence and anatomical agreement that demonstrate visual similarity in cortical patterns, as well as very high semantic agreement for granular concepts such as “aircraft in the air” that the authors have not seen previously reported on NSD nor the broader literature. Appendix D has been revamped, extending it from a surface comparison against Subject 2, to a complete Subject-wise account: it includes the quantitative results in Sections 4 for each subject, showing strong recurrence across all of our main findings, and additional quantitative and qualitative cross-subject comparisons. While data availability nonetheless remains a concern relative to the scale used to fit the featurizer and the SAE, it must be noted that NSD remains the largest fMRI dataset of its kind with an established encoding and decoding literature. We present first results on the newer LAION-fMRI dataset as well in Appendix E.

Whether sparsified latent representations need to retain their alignment with the target signal depends on whether the sparse autoencoder’s reconstruction residuals do not carry useful information to predict human neural activity. In the case of DINOv2, Appendix A shows that sparse latent representations incur a relatively small encoding prediction error relative to dense representations, so in this setting the sparse representations carry significant encoding signals. However, this does not guarantee that interpretability is maintained: our framework enables us to reveal to what extent the interpretation given to a feature space direction is preserved by the neural alignment. Such a distinction is useful as practitioners may incorrectly assume that labeled feature space directions (or even sparse features used by their encoder) are faithfully recruited.

To our knowledge, this is the only thorough validation of whether, and to what extent the perceptual identity or interpretation of an SAE concept derived from foundation model embedding is consistent with its image through the mapping of a voxel-wise encoding model. It's important to note that it is not at all guaranteed that a concept will have coherent interpretations on the model- and brain-side. Our results establish that this recruitment is inconsistent in terms of preserving concept identity, and offer several complementary interpretability approaches to diagnose semantic changes through encoders and decoders of brain activity. Besides filtering and sorting the dictionary in order to surface individually-interpretable or coherent concepts, future work built upon our approach can leverage these signals to improve the interpretability and alignment of neural responses and artificial activations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.