acceptodds
Under review as a conference paper at ICLR 2027

Interpreting Brain Responses to Language with Sparse Features from Language Models

Abstract

A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex. Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another. The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically-organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor. Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations. Using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences, we first validate our modeling framework by recovering previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness. We then interpret a previously-uncharacterized (but reliable) voxel population and find that it is tuned to people-related content. Next, we show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, with variable sensitivities to these features. Finally, we show that brain responses are best explained by the features that tend to capture the most general information encoded in LM representations, suggesting a nontrivial correspondence between brain and LM language representation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neuroscience, NeuroAI, Mechanistic Interpretability, Encoding Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gbF3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Augmented Sparse Encoding Models, which combine sparse autoencoder (SAE) features from language models with an explicit surprisal predictor to interpret brain responses to language. Using a 7T fMRI dataset, the authors investigate voxel-level semantic tuning, feature sharing across language regions and participants, and the predictive contributions of different levels of the Matryoshka SAE hierarchy. Additional datasets support several findings. The study addresses an important question, but concerns remain regarding the independence of the voxel-to-voxel generalization analysis and the robustness of the Ghost subgroup findings.

Critical strengths
  1. The framework connects brain-response prediction with interpretable linguistic features, providing more informative explanations than model–brain correlation scores alone.
  2. The paper includes substantial supporting analyses, including nested cross-validation, split-half stability analysis, an additional model layer, and replication of selected findings on independent datasets.
Critical weaknesses
  1. Appendix C states that source-voxel features and their signs are selected using all stimuli before cross-validation on target voxels. This leaves uncertainty about generalization to unseen stimuli, particularly when source and target responses are correlated. Selecting source features within each outer training fold would provide a stronger test of the reported feature sharing.
  2. The criterion for identifying the 32-voxel subgroup in Figure 3A is unclear. The subsequent LTPJ enrichment also depends on several selection thresholds whose sensitivity is not assessed. Clarifying the subgroup-selection procedure and testing reasonable alternative thresholds are important for establishing the robustness of this central finding.
Questions for authors
  1. Please clarify how the 32-voxel Ghost subgroup in Figure 3A was identified: through a predefined clustering rule, a predictivity threshold, or visual inspection of the heatmap. Report the relevant settings and assess the subgroup’s stability.
  2. The LTPJ enrichment analysis uses several thresholds: normalized predictivity above 0.5, agreement with at least seven of ten feature directions, and more than three aligned voxels per participant. Please justify these choices and examine whether the enrichment remains stable under reasonable alternative thresholds. Participant-level counts would also help clarify the reported percentages.
  3. The authors acknowledge that features 44 and 71 track token count through first-token activation and mean pooling. Please test whether the main feature-sharing and hierarchy findings remain consistent after explicitly controlling for token count or changing the pooling procedure.
  4. The feature interpretations rely mainly on manually summarized Neuronpedia examples. Please describe the annotation procedure and provide supporting examples or independent semantic ratings from the study’s own stimuli, particularly for the “people-centered” interpretation.
  5. Several details require correction: the main text states an NCSNR threshold above 0.4, whereas Appendix A allows relaxation to 0.2 for some subtypes; “2028–8192” in Figure 5B should be checked; and the Figure 1B caption should define surprisal as mean negative log probability, consistent with the main text.
  6. Do the voxel-to-voxel generalization findings remain consistent when source-feature selection is performed exclusively within each outer training fold?
  7. Please clarify that “drive” and “suppress” refer to positive and negative regression associations rather than causal neural effects. Given possible correlations among predictors, the stability of the main coefficient signs should also be discussed.
Reviewer_cXhSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The study introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchicallyorganized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor. The proposed Augmented Sparse Encoding Models try to disentangle black-box issues on both brain response and model representation sides: select brain responses of well-characterized voxels and use interpretable SAE features of model representations. The authors first show the validity of the proposed models by recovering former findings about neural encoding of difficulty-driven and content-driven factors. Then they use the proposed models to identify and interpret neural encoding of people-centered content, and characterize the feature basis that drive the language network, leading to novel findings of neural encoding in the human brain.

Critical strengths

1.The authors build an analytical framework that facilitates interpretable and hierarchical SAE features in brain encoding analysis; 2.The authors provide extensive experiments, as well as novel findings about neural encoding of people-centered content and the feature basis that drive the language network.

Critical weaknesses

1.The proposed models are closer to an analytical pipeline than a methodological framework. The conclusions are more neuroscience-oriented than machine learning-oriented--little is novel for model interpretation and diagnosis. These limit the contribution of the present study to narrower audience. 2.The neural responses are constrained to a set of carefully selected brain voxels, which limits the ecological validity of the proposed models. The exploration about Ghost voxels heavily rely on interpretation of SAE features.

Questions for authors

1.The authors emphasize “augmented sparse encoding models”, but what does “augmented” really refer to? Is it the replacement of dense LM hidden states with contextual and hierarchical SAE features? A clear justification would help the readers to better understand the core contribution of the proposed study. As to “models”, my understanding is how to use SAE features more effectively in this study rather than a novel encoding model.
2.The authors state differences of the present study from previous SAE learning to word-level embedding. The motivation of using hierarchical SAE is clear, enabling the identification of which levels of LM feature hierarchy best align with brain responses. However, what is the benefit of using SAEs over contextual LM hidden states? 3.In my opinion, the interpretation of Ghost voxels may heavily rely on annotations of SAE features. Sometimes subjective intuition may necessary. I guess that is the reason why the authors only reported neural encoding of people-centered content. In this case, it would be better to discuss the limitation of the proposed models. 4.Typically, the dimensions of SAE features are much higher than that of the hidden states. However, the limited scale of fMRI dataset may not match the sharply increased encoding model complexity. In this case, how to make sure that encoding models with various model complexity are fairly compared? Would this introduce any bias in result interpretation?

Reviewer_RJwrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The authors propose using SAE features and surprisal as features of linguistic stimuli in encoding models.
  • Their three experiments mainly show the following:
    • The proposed method can interpret the top two principal components of brain responses at a fine-grained level.
    • The proposed method can identify person-related brain regions as voxel populations not captured by the top two principal components of brain responses.
    • There are features shared across the language network as well as features specific to individual participants.
    • The features most common for the LLM best explain BOLD responses.
Critical strengths
  1. The progression from reproducing known features (Section 4) to discovering unknown ones (Section 5) is easy to follow.
  2. The authors confirm that the results replicate across multiple fMRI datasets with different characteristics (Section 6, Appendices H and I). This is valuable for reproducibility.
Critical weaknesses
  1. In Section 5 (Study 2), it is unclear why the proposed method, which is more complex than PCA, is needed. Since ghost voxels are defined simply as voxels not explained by PC1 and PC2, couldn't the same results be obtained by analyzing PC3 or higher principal components without the proposed method?
Questions for authors
  1. The top features shared by the coherent subpopulation of ghost voxels (Figure 3B) are described as "people-centered," but they also include features unrelated to people, such as feature #79 (landscapes) and #44 (short sentences). What proportion of these features are actually related to people?
  2. L482–483 state: "However, other common critiques of SAEs (e.g., lack of causal claims; Wu et al., 2025) do not apply, as we merely require an unsupervised, interpretable feature basis." Could you explain in more detail why the lack of causal claims does not apply to this work? It seems to me that it does apply. This study measures correlations between the SAE features of the stimuli and the BOLD signals of voxels, so the interpretations of the features do not necessarily correspond to "concepts" that the brain causally uses. This limitation should be stated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.