acceptodds
Under review as a conference paper at ICLR 2027

Cognitive Expert Language Models Better Align with the Corresponding Brain Systems

Abstract

Large language models (LLMs) can predict human brain activity across a variety of brain regions during natural language comprehension. Typically, however, LLM–brain alignment is measured using one model for different regions of the brain, and then model performance is summarized across regions. This one-model-fits-all approach ignores the functional specialization of brain regions. In this study, we assess whether a model oriented toward a particular cognitive domain aligns better with the brain system dedicated to that domain. Through prompting and fine-tuning, we first build expert LLM variants for six domains: sensory, spatial, numerical, reasoning, social, and abstract processing. We then examine whether each expert best predicts activity in the brain region associated with the corresponding cognitive domain. Consistent with our hypotheses, each expert's representations align more closely with the brain system most associated with the matching domain than do other experts. This holds under both prompting and fine-tuning, across three base models and three fMRI datasets. In a series of control analyses, we show that this model–brain alignment is specific to cognitive domain interventions; non-cognitive and surface-level interventions do not result in comparable alignment. Specializing models shifts regional alignment while leaving aggregate prediction accuracy largely unchanged, suggesting that summarizing alignment across regions may obscure regional differences in performance for specific models.

Primary area:
applications to neuroscience & cognitive science
Keywords:
LLM–brain alignment, fMRI, voxelwise encoding models, NeuroAI, functional specialization, cognitive domains

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZJcdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a language model that is specialized for one kind of thinking (such as social reasoning or numbers) lines up better with the matching part of the human brain. The authors build six “expert” versions of three open models, using either a short role prompt or light fine-tuning. They then check whether each expert predicts brain activity best in its own brain region, using fMRI data from three datasets.

Critical strengths

The question is clear and interesting, and it goes beyond the usual one-model-for-the-whole-brain approach.

The experiments are broad: three base models, two ways of specializing, three datasets, and many checks.

The control experiments are a real strength. Random prompts, keyword lists, and style-changed text do not produce the same effect, which supports the claim that the effect comes from the cognitive content.

The authors report results honestly, including the weaker cases, and they test whether choosing the best layer could inflate the results.

Critical weaknesses

The brain regions come from an automatic database (Neurosynth), and the mapping from six domains to brain regions is a choice that could be debated. The Pereira mapping to networks is especially loose, as the authors admit.

The paper says that a prompt “shifts representations”, but the main evidence is about brain prediction, not about what the model internally represents. Stronger links to the model’s internal structure would help.

The main claim should be made more modest, or add evidence that the effect is more than a statistical pattern.

Questions for authors

NA

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZJcd (1/2)

We thank the Reviewer for recognizing the question, the breadth of the experiments, and the control analyses, and for the comment about honestly reporting the weaker cases. We have uploaded a revised paper that includes the changes described below. Please find our responses to the raised concerns below:

  1. (Neurosynth and the mapping) We agree with the Reviewer that the use of Neurosynth is one of several possible choices for defining functionally specialized regions/networks. We opted to use Neurosynth because it defines functional regions meta-analytically directly from terms used in thousands of published studies, independently of our models and fMRI data, which the Methods now state explicitly. We visually confirmed that the resulting regions capture trends of functional localization observed in the literature. To test whether the results are robust to different definitions of these regions, we adjusted the regions and scored the result again. First, we removed voxels shared between regions to obtain non-overlapping regions; second, we removed voxels corresponding to a probabilistically defined language network (LanA; Lipkin et al., 2022).

Displacement under different region definitions, over the twelve LeBel and Le Petit Prince combinations.

RegionsMean displacement (0 = chance)Combinations positive
Regions as in the paper+0.6512 of 12
Voxels shared between regions removed+0.6512 of 12
Language network removed (LanA atlas)+0.5712 of 12

Displacement stays positive in both cases, and removing the language network lowers it only slightly. As reported in the paper, the effect also holds with continuous Neurosynth maps and with other region sizes. Since these definitions give the same pattern, we conclude that the result does not depend on one particular mapping. We have added these analyses to the revised paper (Section 4.2 and Appendix C).

The datasets do not include functional localizers for most of these domains, so we have also added a short passage to the Limitations section suggesting that future work could use individual-specific regions defined using such tasks.

We also agree that the Pereira mapping is loose. This point, which another reviewer also raised, led us to a new analysis. In the original submission, we used the version of the Pereira dataset distributed with Brain-Score, which is the one most often used in studies of language models and the brain. This version only includes functional data for voxels inside five functional networks, so the Neurosynth regions could not be mapped onto it, and we translated our domains onto these networks instead. In the revised manuscript, we have obtained the original whole-brain release of the dataset, allowing us to test the same regions as used with the other two datasets. This release includes eight of the paper's ten participants, each in their own brain space, unfortunately without anatomical scans. For these eight participants, we estimated each participant's mapping to standard brain space from the atlas labels included in the release, placed the same six Neurosynth regions as on the other two datasets, and fit all models again. Without anatomical scans this mapping is an approximation, which we note in the revised paper. With the numerical expert now included, displacement on Pereira is positive in all six combinations of model and intervention (mean +0.41, against +0.36 with the networks). We have added this to the revised paper (Section 4.2 and Appendix C).

Finally, we want to emphasize that we do not believe these six cognitive domains are equally important to the brain or privileged in some way over other domains. We selected these domains as capturing high-level cognitive functions that are of interest to both neuroscience and artificial intelligence (and prevalent in both literatures). We have revised the paper to make clear that these six domains are one possible way to divide cognition, not the only one (Section 3.1 and Appendix A).

AuthorsOpenReview ↗

Response to Reviewer ZJcd (2/2)

  1. ("Shifts representations") We thank the Reviewer for raising this point. By this phrase we meant that with a system prompt or an adapter, the same input produces different hidden states in the model. An encoding model is a linear map from those hidden states to brain activity, so when an expert's best predicted region changes, that change comes from its hidden states. In the revised paper, we have changed the wording to say what is measured directly: specialization shifts a model's alignment to better match activity in the matching brain regions (Introduction and Conclusion). The paper also measures these internal changes directly (Section 4.3, Figure 4a, and Appendix G): by centered kernel alignment, fine-tuning moves the six experts' representations 1.4 to 6.8 times further apart than prompting does, increasingly with depth, and the surface-form control changes the representations more than cognitive prompting does without producing the regional alignment.

We agree that these measurements alone do not show what changed inside the model, and we have started a preliminary analysis of it. For each prompted expert, we took the difference between its hidden states and the base model's on the same story, and used PCA to find the directions in which this difference varies most. No brain data is used in this analysis. We then built two models, the base model plus the expert's change along only the top directions, and the expert with that change removed, and scored both like any other model.

Displacement for prompted experts on Le Petit Prince (49 subjects), at each model's Table 1 setting.

ModelLLaMAPhi-4
Full expert+0.42+0.17
Base model plus the 64 directions+0.47+0.24
Expert minus the 64 directions+0.080.00

About 64 directions are enough in both models: adding them to the base model reproduces the effect, and removing them from the expert abolishes it. Fine-tuned experts spread the effect over more directions (about 256 for fine-tuned LLaMA on Le Petit Prince). These results are preliminary, and we plan to develop them further.

  1. (More modest claim) We have made the claim more modest in the revised paper: specializing a model changes which region each expert predicts best relative to the others, without improving overall prediction accuracy, and a new Limitations section states the scope of the result. We also added evidence that the effect reflects the experts' cognitive specialization rather than incidental differences between the models.

First, on average, experts are above the original base model in their own regions (r = 0.0891 against 0.0880, p < .001), while the other experts in those regions overall are not (Section 4.2 and Appendix F).

Second, properties of the fine-tuning data do not explain the effect. The effect is retained after accounting for seven properties of the training sets (p = .001). The effect also remains when experts are retrained on sets with the same inputs, size, format and target length (positive in all nine combinations, and significant in eight; Section 4.4 and Appendix B).

In addition, the preliminary analysis described under point 2 suggests that the effect can be produced and removed by changing the model directly. In a separate preliminary test, the effect also grows with the strength of the change: scaling a fine-tuned LLaMA adapter from 25% to 100% raises displacement from +0.39 to +0.89 on LeBel, and on Le Petit Prince it is significant at every strength.

If there is a specific analysis the Reviewer has in mind, we would be happy to address it.

Thanks for the detailed clarifications, which have addressed my concerns. I decided to update my initial assessments.

We thank the Reviewer for the thoughtful review and for updating the assessment. The comments helped us make our claims more precise and strengthen the paper with additional analyses.

Reviewer_TULrOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors trained LLMs to be experts in six domains and found that brain alignment was strongest with brain regions specialized in each of these domains and that brain alignment was weaker in the other domains.

Critical strengths

Strengths:

  • This is a well-written and interesting paper.
  • Contains a good theoretical background.
  • The methodology is sound, with numerous checks and controls and all including sophisticated statistical tests.
  • The results are interesting and generally well-backed-up with stats.
Critical weaknesses

Weaknesses: I do not see any real weaknesses in this paper. My only concern is that the paper itself (excluding the appendix) didn't clearly state which domains were more clearly aligned with neuroimaging data and which ones were not. It looks like the reasoning domain may be the least aligned, which is an important insight for related fields. I would ask that the authors explain this more clearly in the paper itself.

Questions for authors

Future work: For a longer, more thorough (journal) paper, including mechanistic interpretability techniques and testing causal interventions (e.g., ablating attention heads (or residual stream layers) that strongly align with fMRI activity) could further substantiate your claims. Perhaps something to mention in the discussion.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TULr

We thank the Reviewer for the very positive assessment, especially for the comments about the writing, the theoretical background, and the controls. We have uploaded a revised paper that includes the changes described below. Please find our responses below:

  1. (Which domains align) We agree with the Reviewer, and we now state this result more explicitly in the main text (Section 4.2). The numerical, social, sensory and abstract experts align most clearly with their regions, the spatial expert less clearly, and the reasoning expert least, as the Reviewer mentioned. We note that these differences across experts could result from the quality or brain-alignment of the experts, but could also stem from how well our cognitive domains match genuine brain systems, and the degree to which these domains are anatomically localized.

We next examined why the reasoning expert may underperform. The reasoning region is predicted about as well as the other regions, so this is not because it is hard to predict. The expert that fits it best is the numerical expert, whose training data include many step-by-step math problems. This suggests that this region may relate more to explicit, structured problem solving than to what is broadly labeled reasoning in instruction data. It also connects to the debate on how reasoning in language models relates to human cognition (Mahowald et al., 2023; Fedorenko et al., 2024), and we discuss this in the revised paper (Appendix F).

  1. (Mechanistic and causal analyses) We thank the Reviewer for this suggestion, which matches our own thinking. We now mention ablating attention heads as future work in the Conclusion. We also started on this before the rebuttal and have preliminary results.

For each prompted expert, we took the difference between its hidden states and the base model's on the same story, and used PCA to find the directions in which this difference varies most. No brain data is used. We then built two models, the base model plus the expert's change along only the top directions, and the expert with that change removed, and scored both like any other model.

Displacement for prompted experts on Le Petit Prince (49 subjects), at each model's Table 1 setting.

ModelLLaMAPhi-4
Full expert+0.42+0.17
Base model plus the 64 directions+0.47+0.24
Expert minus the 64 directions+0.080.00

About 64 directions of the model's representation are enough, in both models: adding them reproduces the effect, and removing them abolishes it. This suggests that specialization acts on a small part of the representation, and that this part is what relates to the matching brain system. Fine-tuned experts spread the effect over more directions (about 256 for fine-tuned LLaMA on Le Petit Prince).

These results are preliminary, and we plan to develop them into a fuller analysis. If the Reviewer has a view on which intervention would be most informative, such as attention heads or residual stream layers, we would be glad to hear it.

Reviewer_TULrOpenReview ↗

Thanks, good luck!

Thank you for thoroughly addressing my comments. I already gave you the highest scores, so this stands. Good luck!

We thank the Reviewer for the kind words and for supporting our work.

Reviewer_mVkmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether language models specialized for different cognitive domains align more closely with corresponding functional systems in the human brain. The authors construct six cognitive experts through prompting and LoRA fine-tuning across three base models and evaluate them using three fMRI datasets. The results show that matched experts generally achieve higher regional alignment than other experts, while overall brain prediction accuracy remains largely unchanged. Several non-cognitive control interventions fail to produce comparable effects. The findings suggest that cognitive specialization can alter the regional correspondence between language model representations and neural activity without necessarily improving overall brain prediction performance.

Critical strengths
  1. The study evaluates three base models, two specialization approaches (prompting and LoRA fine-tuning), and three fMRI datasets. This experimental coverage provides evidence that the observed regional alignment patterns are not limited to a single model architecture or neuroimaging dataset.
  2. The authors construct several control families, including random prompts, random LoRA adapters, seed-only fine-tuning, surface-form fine-tuning, and keyword-based controls. These comparisons help distinguish cognitive-domain-specific alignment from more general effects of prompting, fine-tuning, or representational changes.
  3. By distinguishing regional alignment from aggregate brain prediction accuracy, the paper demonstrates that whole-brain evaluation can obscure meaningful differences across functionally specialized brain systems.
Critical weaknesses
  1. The primary metric, displacement, evaluates how highly a cognitive expert ranks relative to other experts in its corresponding brain region. However, this is a relative measure. An expert could achieve a higher rank without improving its own predictive accuracy if competing experts perform worse. Although the authors also report the continuous metric Δr, many improvements in Table 1 are very small, particularly on the LPP dataset, where some values are on the order of 10⁻⁴. Moreover, overall prediction accuracy remains largely unchanged. Therefore, the results provide stronger evidence for relative regional preference than for substantial improvements in neural prediction. Reporting absolute encoding accuracy changes relative to the original base models within each corresponding region would clarify the practical significance of the observed specialization.

  2. The framework assumes a correspondence between six cognitive domains and their associated brain systems. However, cognitive functions such as reasoning, numerical processing, and abstract thinking involve overlapping neural networks rather than entirely separable systems. This issue is particularly apparent in the Pereira dataset, where five existing functional networks are manually mapped to the proposed cognitive categories, and the numerical expert is excluded because no corresponding network is available. Although the authors examine alternative region sizes and continuous Neurosynth maps, these analyses do not fully address the validity of the underlying cognitive-domain mapping. Independent task-based functional localizers or analyses allowing overlapping functional systems would provide stronger evidence for the proposed correspondence.

  3. The domain-specific fine-tuning datasets differ not only in cognitive content but also in dataset size, text length, task format, writing style, and linguistic complexity. Consequently, changes in regional alignment may partly reflect these differences rather than cognitive specialization itself. While the seed-only and surface-form controls help rule out general training effects, they do not fully isolate the influence of domain content from other properties of the fine-tuning corpora. Experiments using domain-specific datasets matched in size, sequence length, instruction format, and linguistic characteristics would provide stronger evidence that the observed effects are specifically driven by cognitive specialization.

Questions for authors

see weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mVkm (1/2)

We thank the Reviewer for the positive assessment, especially for acknowledging the coverage across base models, interventions and datasets, the control families, and the value of separating regional alignment from aggregate accuracy. We have uploaded a revised paper that includes the changes described below. Please find our responses to the comments you have raised:

  1. (Relative measure, comparison with the base model) We agree with the Reviewer that displacement is a relative measure. As the Reviewer suggested, we compared each expert with the original base model inside its own region.

Mean encoding accuracy (r) in each expert's own region, at the Table 1 settings, averaged over prompting and fine-tuning.

ModelBase modelOwn expertOther experts
LLaMA0.09000.09040.0894
Qwen0.08430.08470.0839
Phi-40.08970.09230.0915
All0.08800.08910.0883

Across all combinations, the own expert is above the base model in its region (p < .001), while the other experts overall are not (p = .18). The own expert is above the base model for each of the three models, and above the other experts in its region in all 18 combinations, significantly so for every model (p ≤ .003). The part of the gain specific to the expert's own region, beyond the change shared by all six experts, is positive in 17 of the 18 combinations, and is also significant for every model (p ≤ .013). We have added this comparison to the revised paper (Section 4.2 and Appendix F).

We also agree that the differences in Δr are modest. Part of the reason is that the six experts are versions of the same model, so their predictions are similar. Note that the experts were also trained only on domain text, without any objective related to the brain, because our question was whether specialization shifts regional alignment in a reliable way. The Reviewer's point raises a question we find exciting: what would make this effect larger? Our new results already point in one direction: the effect grows as the strength of the specialization increases. We now mention this question as future work in the Conclusion, and a new Limitations section discusses the size of these differences.

AuthorsOpenReview ↗

Response to Reviewer mVkm (2/2)

  1. (Domain to region mapping) We agree with the Reviewer that these domains share neural resources. Note that in the original submission, we already allowed our functionally-defined regions to overlap, as they do in Neurosynth. To check that the result does not rely on the voxels these regions share, we also tested it with the overlap removed. In addition, we removed the voxels of the language network from every region, using a probabilistic atlas of the language network defined from localizer tasks in 806 participants (LanA; Lipkin et al., 2022).

Displacement under different region definitions, over the twelve LeBel and Le Petit Prince combinations.

RegionsMean displacement (0 = chance)Combinations positive
Regions as in the paper+0.6512 of 12
Voxels shared between regions removed+0.6512 of 12
Language network removed (LanA atlas)+0.5712 of 12

Displacement stays positive in both cases, and removing the language network lowers it only slightly, so the effect does not depend on it. The paper also reports the same pattern with continuous Neurosynth maps and with other region sizes. We have added these analyses to the revised paper (Section 4.2 and Appendix C).

Unfortunately, these datasets do not include individual-specific functional localizers for most of these domains, which is why we originally resorted to using independently-defined meta-analytic maps from Neurosynth. We now highlight this limitation in the revised paper, together with how future work could address it (Limitations).

We also agree that the Pereira mapping is perhaps not the ideal fit, especially given that the numerical expert has no corresponding network in that dataset. This point, which another reviewer also raised, led us to a new analysis. In the original submission, we used the version of the Pereira dataset distributed with Brain-Score, which is the one most often used in studies of language models and the brain. This version only includes functional data for voxels inside five functional networks, so the Neurosynth regions could not be mapped onto it, and we translated our domains onto these networks instead. In the revised manuscript, we have obtained the original whole-brain release of the dataset, allowing us to test the same regions as used with the other two datasets. This release includes eight of the paper's ten participants, each in their own brain space, unfortunately without anatomical scans. For these eight participants, we estimated each participant's mapping to standard brain space from the atlas labels included in the release, placed the same six Neurosynth regions as on the other two datasets, and fit all models again. Without anatomical scans this mapping is an approximation, which we note in the revised paper. With the same regions on all three datasets, and the numerical expert now included, displacement on Pereira is positive in all six combinations of model and intervention (mean +0.41, against +0.36 with the networks). We have added this to the revised paper (Section 4.2 and Appendix C).

  1. (Fine-tuning datasets differ) We agree with the Reviewer that the fine-tuning sets differ in size and text length, although they share one training recipe and are close in word length. We addressed this in two ways.

First, we tested whether these differences explain the result. In a regression, we accounted for seven properties of each set: the number of examples, the input length, the target length, word length, sentence length, type-token ratio, and the share of long-form text. The own-region effect does not change (p = .001), and the largest and smallest fine-tuning sets give roughly the same advantage.

Second, as the Reviewer suggested, we built matched training sets, in which only the domain focus of the target text differs.

The matched fine-tuning sets.

PropertyMatched sets
Training examples3,657 in every domain
Input passagesIdentical in every domain
Input length (words)227 in every domain
Target length (words)112 in every domain
Format and sourceOne format, one teacher model
Sentence length (words)19.7 to 22.6

We trained new experts for all three models on these sets and scored them in the same way as in the paper. Each new expert is still the best of the six on its own domain's text.

Displacement of the new experts trained on the matched sets.

ModelLeBelLe Petit PrincePereira
LLaMA+0.81+0.29+0.34
Qwen+0.96+0.25+0.21
Phi-4+0.80+0.24+0.18

The effect remains with the matched sets: displacement is positive in all nine combinations, and significant in eight of them.

We have added both analyses to the revised paper (Section 4.4 and Appendix B).

Finally, prompting, which uses no training data at all, also gives a positive displacement in all nine of its combinations (Table 1).

The authors have addressed my main concerns through additional experiments. The absolute encoding accuracy comparison clarifies that the regional gains are positive but modest. The analyses using alternative region definitions and the whole-brain Pereira data provide further support for the domain-region correspondence. The matched fine-tuning experiments also help address potential dataset confounds. Some limitations remain regarding effect sizes and functional localization, but the responses have adequately addressed my concerns. I maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer mVkm

We thank the Reviewer for reading our responses and for the positive assessment of the new analyses. We agree that effect sizes and functional localization remain limitations. The revised paper reports standardized effect sizes for every configuration (Appendix F, Figure 26) and discusses both points in the new Limitations section, together with how future work could address them.

We would be grateful to know if there is anything else we could do to strengthen the paper and that would lead the Reviewer to a more positive assessment. If there is a specific analysis the Reviewer has in mind, we would be happy to run it before the end of the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers for their comments, which helped us strengthen the paper with additional analyses. We have tried to address every point raised, both in our replies and in the revised paper we have uploaded. The main points we addressed are:

  • Accuracy relative to the original base model inside each expert's own region (Section 4.2, Appendix F).
  • Which experts align most and least, and why the reasoning expert aligns least (Section 4.2, Appendix F).
  • Robustness to the region definitions, with voxels shared between regions removed and with the language network removed (Section 4.2, Appendix C).
  • The Pereira mapping, now tested with the same Neurosynth regions on the whole-brain release of the dataset (Section 4.2, Appendix C).
  • Differences between the fine-tuning sets, through a regression on seven properties of the sets and new experts trained on matched sets (Section 4.4, Appendix B).
  • The wording of our claims, which we adjusted to follow our findings more closely and make them more accurate, with a new Limitations section stating the scope.
  • What changes inside the models: the representational measures in the paper (Section 4.3, Appendix G) and new preliminary analyses in our replies to Reviewers TULr and ZJcd, with causal tests noted as future work (Conclusion).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.