acceptodds
Under review as a conference paper at ICLR 2027

Exploring Divergence Between Brain Responses and Language Model Representations Using Crosscoders

Abstract

To what extent do human brains and language models (LMs) share internal representations of language, and how do these representations differ? Prior work has shown that LM representations can predict brain responses to naturalistic language stimuli, suggesting that the two systems encode common information. However, it has remained unclear which features are shared between brain responses and LM representations and which are specific to one of them. We propose Brain-LM crosscoders, which decompose brain responses and LM representations into a common set of features and label each feature as shared, a brain-specific candidate, or an LM-specific candidate based on its predictive contribution to brain responses and to LM representations. Experiments on fMRI data recorded during naturalistic story listening generate hypotheses about information that only brain responses or LM representations encode and that encoding models ignore. Brain-LM crosscoders move the comparison of brains and LMs from prediction to explanation and could help detect concepts that diverge from those of humans.

Primary area:
interpretability and explainable AI
Keywords:
crosscoders, sparse autoencoders, dictionary Learning, neuroscience, brain, fMRI, language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GA1jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes Brain-LM crosscoders. The proposed method decomposes brain responses and LM representations into a common set of features and labels each feature. Feature categories can be shared, brain-specific candidates, or LM-specific candidates, depending on their predictive contribution to brain responses and LM representations. Experiments show the design advances.

Critical strengths

None

Critical weaknesses
  1. Experimental issues.
  2. Reproduction issues.
Questions for authors
  1. The reasoning in the BRAIN-LM PREDOMINANCE section should be further proofread.
  2. Furthermore, classify features as shared, brain-specific candidates, or LM-specific candidates. Citations should be added. The paper does not prove that its formulated threshold yields reliable three classification assignments.
  3. Go on with cmt 2, the sensitivity analyses are insufficiently validated.
  4. The interpretability is not convincing.
  5. The baseline comparison should add more SOTA methods.
  6. No opensource code.
  7. Fewer implementation details.
3 comments

Thank you for your review. We respond to each comment below.

W1. "Experimental issues."

Please see our responses to Q2–Q5.

W2. "Reproduction issues."

Please see our responses to Q6 and Q7.

Q1. "The reasoning in the BRAIN-LM PREDOMINANCE section should be further proofread."

We wrote this section as clearly as we could. If you tell us which part needs improvement, we will gladly revise it.

Q2. "Classify features as shared, brain-specific candidates, or LM-specific candidates. Citations should be added. The paper does not prove that its formulated threshold yields reliable three classification assignments."Q3. "The sensitivity analyses are insufficiently validated."

We set the threshold of 0.9 for convenience in the case study, and we agree that the threshold should rest on a criterion. We now set the threshold so that fewer than 5% of the features of a control crosscoder trained on circularly shifted data (section 4.6) are classified as shared. This threshold should keep the proportion of features wrongly classified as shared below 5%. We set it separately for each subject–LM pair. Section 3.3 reflects this change.

Q4. "The interpretability is not convincing."

We ask you to consider that this study is exploratory and aims to generate hypotheses. As the introduction states, the explanations "pose hypotheses for future intervention experiments" (L104–106), and rigorous verification is outside the scope of the paper. This is why the paper calls these features specific candidates throughout.

Q5. "The baseline comparison should add more SOTA methods."

To our knowledge, the task of this paper has no established benchmark, so no method is clearly the state of the art. We chose ShIndICA as a comparable method "because its approach is similar to that of crosscoders" (L309–310). If you know of other methods that we should compare, please tell us.

Q6. "No opensource code."

The reproducibility statement (L554–555) already gives the link to our code. We repeat it here: https://anonymous.4open.science/r/brain-lm-crosscoder-dev-67E8

Q7. "Fewer implementation details."

The main text gives priority to an overall understanding of the method over technical details, so the implementation details that our method shares with prior work appear in Appendices B–G (preprocessing, architecture, explained variance, layer selection, detection accuracy, statistical testing).

I will maintain my original rating.

Just to confirm, do you have any remaining concerns? We believe we have addressed all of your comments, but if anything is still unresolved, please let us know it. Thank you!

Reviewer_iciBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Brain-LM crosscoders, which learn sparse latent features that jointly predict fMRI responses to naturalistic stories and corresponding LLM representations. Features are classified as brain-specific, LM-specific, or shared based on their relative explained variance in the brain and LM representations. The authors use LLM-generated feature descriptions to interpret the resulting representations, reporting significantly above-chance explanation accuracy for 41% of evaluated features and outperforming a ShIndICA baseline.

Critical strengths

The paper addresses an interesting question: identifying which aspects of linguistic representations are shared between brains and LLMs, rather than simply quantifying their overall alignment. The proposed framework is reasonably intuitive and clearly presented, and applying crosscoders to this problem is a potentially useful methodological contribution.

Critical weaknesses
  1. Unconvincing scientific interpretations and insufficient validation.

The paper provides little compelling new understanding of either brain or LLM representations. The highlighted examples—empathy as a brain-specific feature, violent impacts as an LM-specific feature, and physical movement sequences as a shared feature—are difficult to interpret as meaningful scientific discoveries. The evidence consists of relative explained variance for jointly learned latent features and automatic descriptions of those features, neither of which directly establishes that the corresponding semantic concepts are differentially represented.

Furthermore, the automatic explanations are validated primarily through above-chance binary detection accuracy, which is a relatively weak criterion for assigning semantic interpretations to learned features. For instance, the showcased brain-specific empathy feature achieves only 0.69 balanced accuracy. The authors do not directly test whether independently defined semantic variables corresponding to their descriptions exhibit the same brain–LM differences as the learned features.

  1. Insufficient validation of the learned decomposition and feature classifications.

The crosscoder jointly infers features from both modalities, but a feature explaining variance in both does not necessarily establish shared information, since a single learned feature can predict components in the brain and LM representations that are independent of each other. Moreover, the threshold for classifying features as shared is permissive ((∣ΔEV∣≤|\Delta EV|\leq 0.9)); the highlighted shared feature has (Δ\Delta EV=0.89), indicating approximately ninefold greater explained variance in the brain.

The explained-variance analyses use leave-one-TR-out cross-validation within training stories, which is vulnerable to temporal autocorrelation and does not establish whether the identified relationships generalize across independent stories. Evaluating feature-specific explained variance on held-out stories would provide substantially stronger evidence for the robustness of the proposed classifications.

Finally, reconstruction-based feature learning prioritizes high-variance components without establishing their semantic or computational importance.

Questions for authors

Do the brain-specific, LM-specific, and shared-feature classifications remain stable when explained variance is computed on entirely held-out stories?

3 comments

Thank you for your valuable comments. They helped us improve the paper.

W1. "The evidence consists of relative explained variance for jointly learned latent features and automatic descriptions of those features, neither of which directly establishes that the corresponding semantic concepts are differentially represented." W2.1.1. "A feature explaining variance in both does not necessarily establish shared information, since a single learned feature can predict components in the brain and LM representations that are independent of each other." W2.3. "Reconstruction-based feature learning prioritizes high-variance components without establishing their semantic or computational importance."

We agree that neither detection accuracy nor explained variance gives decisive evidence by itself, as Section 5.1 states.

We also ask you to consider that this study is exploratory and aims to generate hypotheses. As the introduction states, the explanations "pose hypotheses for future intervention experiments" (L104–106), and rigorous verification is outside the scope of the paper. This is why the paper calls these features specific candidates throughout. We treat these metrics as a filter that selects hypotheses for future testing.

The paper does include one test of these metrics. Section 4.6 shows that ablating the temporal alignment, which makes shared features harder to learn, lowers the sharedness computed from explained variance. Explained variance therefore carries some information about whether a feature is shared or specific.

W2.1.2. "The threshold for classifying features as shared is permissive (0.9)."

We set the threshold of 0.9 for convenience in the case study, and we agree that the threshold should rest on a criterion. We now set the threshold so that fewer than 5% of the features of a control crosscoder trained on circularly shifted data (section 4.6) are classified as shared. This threshold should keep the proportion of features wrongly classified as shared below 5%. We set it separately for each subject–LM pair. Section 3.3 reflects this change.

W2.2. "The explained-variance analyses use leave-one-TR-out cross-validation within training stories, which is vulnerable to temporal autocorrelation [...]. Evaluating feature-specific explained variance on held-out stories would provide substantially stronger evidence for the robustness of the proposed classifications." Q1. "Do the brain-specific, LM-specific, and shared-feature classifications remain stable when explained variance is computed on entirely held-out stories?"

Thank you for this suggestion. We agree that explained variance on stories held out from training tests generalization.

Section 3.3 now computes each EV on the 17 test stories held out from crosscoder training (L225–238). For each feature, we fit the regression from its activation to the brain responses or the LM representations on the training stories and evaluate it on the test stories. A feature whose EV is non-positive for both brain responses and LM representations does not generalize to the test stories, and we label it non-predictive (L246–248). Of the 1,536 evaluated features, 47 are non-predictive. The remaining 1,489 are classified from their EV on the test stories as 216 brain-specific candidates, 316 shared features, and 957 LM-specific candidates (L363–365). All three categories therefore remain when EV is computed on held out stories.

The control of Section 4.6 gives the same result under this EV. Ablating the temporal alignment lowers sharedness for all 12 pairs of subject and LM (L467–469, Figure 5), so explained variance on the test stories still carries information about whether a feature is shared or specific.

Reviewer_iciBOpenReview ↗

Official Comment by Reviewer iciB

Thank you for the detailed rebuttal and substantial revisions. I appreciate the authors' efforts to address my concerns, and I think the paper has been meaningfully strengthened.

In particular, evaluating explained variance on entirely held-out stories addresses my concern regarding leave-one-TR-out cross-validation. The revised threshold calibration is also a clear improvement over the previously arbitrary threshold. The temporal-shift control provides useful evidence that the sharedness measure is sensitive to temporal alignment, and the new brain–brain and LM–LM comparisons provide additional context for interpreting the results. I also appreciate the revised case study, particularly the focus on features that recur across subject–LM pairs, the exclusion of inconsistent interpretations, and the removal of previously highlighted claims that did not survive this analysis.

These are substantial improvements, and I acknowledge the novelty and potential utility of the proposed method. The demonstration that crosscoder features are more readily interpretable than ShIndICA features under the authors' automatic interpretation metric is also a meaningful positive result.

However, my primary concern regarding the scientific informativeness of the resulting interpretations remains unresolved.

While the revised case study establishes greater consistency in the identified features, I remain unconvinced that these findings provide compelling new insights into differences between brain and LM representations. For instance, the observation that features associated with romantic or sexual content or violence consistently explain more variance in LM representations does not, in itself, establish what substantive representational differences exist between the two systems.

More fundamentally, the analysis relies on differences in explained variance for jointly learned features, followed by automatically generated semantic descriptions. It is not clear to me that relative explained variance is a reliable indicator of the semantic importance or granularity of the corresponding information in either system. While the additional controls strengthen confidence in the measured patterns, they do not directly validate the scientific interpretations drawn from them.

I understand and appreciate the authors' emphasis on hypothesis generation rather than definitive scientific conclusions. I agree that exploratory methods can be valuable without fully validating every proposed hypothesis. However, I would still expect a convincing demonstration that the method produces scientifically informative hypotheses about brain–LM differences. Ultimately, while I recognize the methodological contribution and the substantial improvements made during rebuttal, I remain unconvinced that the paper has demonstrated sufficiently informative scientific insights or adequately established the value of its proposed interpretations. I therefore maintain my original rating of 2 (weak reject). Nonetheless, I recognize that others may reasonably place greater weight on the methodological contribution itself, and I would not object to acceptance on that basis.

Thank you for your reply. We are very glad that the additional experiments resolved a lot of your concerns.

We would like to add one point about your remaining concern, the scientific informativeness of our findings. The LM-specific candidates suggest that brain-LM similarity measured with existing encoding models misses differences in features such as named entities, world knowledge, sex, and violence. An encoding model reduces this similarity to a single prediction score, so it cannot show which features diverge. We believe these features give new insight into details that existing methods miss.

Thank you again for recognizing our methodological contribution and for the constructive discussion!

Reviewer_oztaOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CrossCoders to analyze Brain-LLM similarity. The authors implement the method for the LeBel Moth Radio Hour dataset, over 3 subjects and 4 candidate LMs, discovering shared and distinct features between individual brains and LLMs.

Critical strengths
  1. The paper is very clear, masterfully written, and easy to read and understand. The paper did a great job explaining the crosscoder method, such that I understood it on the first try.

  2. The level of statistical rigor is commendable, and the application of crosscoder to brain/LM similarity is novel, and of interest to the NeuroAI community.

Critical weaknesses

While I think this method is strong, and when applied to brain/LLM alignment has great potential, I found the actual insights derived from the method to be somewhat weak (this is the case study section). I have several concrete experimental suggestions to improve the paper, and will gladly increase my score to an Accept if the following concerns are addressed.

  1. The paper currently analyzes shared and divergent brain/LLM features between single brains and single LLMs. While a good starting point, the paper can be made much stronger if it tackles the question for general brains and LLMs. To do this, concretely, it would be great if you can run the controls
  • Brain-brain crosscoder
  • LLM-LLM crosscoder

and then normalize or contrast the brain-LM results to "within-species" EV (see, e.g., Thobani et al., 2025). A qualitative analysis that summarizes the findings over all individual brain-llm pairs is also appreciated.

  1. The method is sold as having potential to discover coherent brain-specific and LM-specific features, however the demonstration and analysis are somewhat weak. An explanation, for instance, is proposed for specific features for UTS03 and Llama (e.g., the point on emotion being more represented in the human brain), but the actual LLM-only feature list in App K seems to contradict this:
  • Internal emotional states, particularly loneliness, longing, and melancholy.
  • Explicit reflection on personal psychological or emotional processing.
  • Immediate physical sensations and visceral internal emotional reactions.

In general, we are lacking more large takeaways that engage with existing hypotheses on brain/LLM differences. The part on emotion is a good example of such a takeaway (but doesn't seem to be supported in the data). These bigger takeaways are quite important for convincing the reader on the usefulness of the method.

  1. The method claims to find different sets of features for brains and LMs, but the qualitative explanations on the different features in App K often overlap. SAE features, moreover, are supposed to be orthogonal. How does one explain this?
  • E.g., brain only: Conversational interactions where the speaker observes multiple people simultaneously engaging in different activities.

  • LM only: (1) Intimate observations about people close to the narrator. (2) Multiple interconnected people introduced through tangential conversational digressions.

Questions for authors

Suggestions:

Missing prior related work on the relation between encoding performance and next-token prediction Cheng et al., ICML 2026 and Proietti et al., ICML 2026. [Did not impact my score.]

Questions:

The feature descriptoins rely on the method by Paulo et al., 2025. When going through them, I did not come away with the impression that they were very specific / significantly differ from each other. How do you know that the qualitative feature descriptions (using Paulo et al., 2025) are not a result of processing by the explainer model? [May impact my score.]

4 comments

Thank you for your thoughtful review. We are glad to receive perspectives that we had not considered.

W1. "The paper can be made much stronger if it tackles the question for general brains and LLMs. [...] A qualitative analysis that summarizes the findings over all individual brain-llm pairs is also appreciated."

Comparing brains and LLMs in general is an interesting experiment. Following your suggestion, we trained crosscoders on pairs of subjects (Brain-brain crosscoders) and on pairs of LMs (LLM-LLM crosscoders) and compared their sharedness, the mean of 1 − |ΔEV|, with that of Brain-LM crosscoders (Appendix L, Figure 8). Within-species crosscoders have higher sharedness than Brain-LM crosscoders, and the LM pairs from the same model family have the highest sharedness.

For the qualitative analysis that summarizes all pairs of subject and LM, we updated the case study (section 4.5). It now analyzes features that recur across crosscoders trained on different pairs of subject and LM.

Among the features with significant detection accuracy, we regard two features as matching if the cosine similarity between the sentence embeddings of their explanations is at least 0.8. We keep the recurring features whose matches all belong to the same category. This procedure yields 54 LM-specific candidates, 4 shared features, and 2 brain-specific candidates (Table 1, Figure 4). The main findings are as follows.

  • LM-specific candidates: the most widespread ones refer to named entities and world knowledge. Others that recur concern sex and violence, which matter for the safety alignment of LMs.
  • Shared features: features that concern time recur.
  • Brain-specific candidates: one activates on concrete sensory or physical details of objects and textures.

W2. "The method is sold as having potential to discover coherent brain-specific and LM-specific features, however the demonstration and analysis are somewhat weak. [...] the actual LLM-only feature list in App K seems to contradict this. [...] we are lacking more large takeaways that engage with existing hypotheses on brain/LLM differences."

We agree. Features whose explanations overlap across categories should have been rejected as hypotheses about differences between brains and LMs.

The case study (section 4.5) now discards recurring features whose matches belong to different categories (L428–429). This step removes from the hypotheses the features found for a single pair of subject and LM and the features whose category is inconsistent. The interpretation you pointed to, "emotion being more represented in the human brain," did not survive this step, and we removed it.

To engage with existing hypotheses, the updated case study adds the following points (L443–453). The brain-specific candidate about concrete sensory or physical details agrees with the finding that LMs fail to capture physical aspects of brain responses (Zhou et al., 2024) and with the lower similarity between humans and LLMs in sensory and motor domains (Xu et al., 2025). The shared features about time suggest that interpretable encoding models (Benara et al., 2024; Singh et al., 2025; Zeng & Gallant, 2026) can rely on temporal features.

W3. "The qualitative explanations on the different features in App K often overlap. SAE features, moreover, are supposed to be orthogonal. How does one explain this?" Q1. "How do you know that the qualitative feature descriptions (using Paulo et al., 2025) are not a result of processing by the explainer model?"

We attribute the overlap you observed to a subset of low quality explanations, and our procedure rejects these explanations as hypotheses about differences between brains and LMs. Two distinct features can receive similar explanations when the explanations describe them poorly. The two explanations you listed under "LM only" illustrate this. "Intimate observations about people close to the narrator" has a detection accuracy of 0.42 and "Multiple interconnected people introduced through tangential conversational digressions" has 0.53, and neither value is significant. The table in Appendix K previously made it hard to see which explanations had significant detection accuracy. It now prints explanations that pass the test in black and the others in light gray, so that readers can see which explanations to adopt as hypotheses.

A more detailed account of the cause would still help future work cover a broader range of hypotheses. We think that these less specific and less diverse explanations result from the granularity of the examples that the explainer LLM receives. Automatic interpretation of LM features gives the explainer a feature activation for every token. In our study, the delay of the fMRI response means that the explainer receives a feature activation for a whole transcript snippet. Brain-LM crosscoders therefore probably tend to learn abstract features more than diverse features at a lower level.

We expect that adding brain responses recorded at a higher temporal resolution, such as MEG, would reduce the overlap among explanations. This experiment is beyond the scope of the present study, and we leave it to future work.

S1. "Missing prior related work on the relation between encoding performance and next-token prediction Cheng et al., ICML 2026 and Proietti et al., ICML 2026."

Thank you! We added both papers to the related work section (L116–117).

Thanks for your response! My questions have been addressed and now the paper looks quite strong. No doubt that this will be a great contribution to the NeuroAI community. I raised my score to accept.

Thank you for raising your score. We are very pleased that you recognize the value of our contributions. Your review have undoubtedly helped us improve the paper.

Comments on the paper

1 comment

We thank all reviewers. We are especially glad that the reviews recognized the clarity of the paper, the interest of the question, and the novelty of the method. Responding to every concern with care took us some time, and we hope the paper is better for it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.