- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a language model that is specialized for one kind of thinking (such as social reasoning or numbers) lines up better with the matching part of the human brain. The authors build six “expert” versions of three open models, using either a short role prompt or light fine-tuning. They then check whether each expert predicts brain activity best in its own brain region, using fMRI data from three datasets.
Critical strengths
The question is clear and interesting, and it goes beyond the usual one-model-for-the-whole-brain approach.
The experiments are broad: three base models, two ways of specializing, three datasets, and many checks.
The control experiments are a real strength. Random prompts, keyword lists, and style-changed text do not produce the same effect, which supports the claim that the effect comes from the cognitive content.
The authors report results honestly, including the weaker cases, and they test whether choosing the best layer could inflate the results.
Critical weaknesses
The brain regions come from an automatic database (Neurosynth), and the mapping from six domains to brain regions is a choice that could be debated. The Pereira mapping to networks is especially loose, as the authors admit.
The paper says that a prompt “shifts representations”, but the main evidence is about brain prediction, not about what the model internally represents. Stronger links to the model’s internal structure would help.
The main claim should be made more modest, or add evidence that the effect is more than a statistical pattern.
Questions for authors
NA
4 comments
Response to Reviewer ZJcd (2/2)
- ("Shifts representations") We thank the Reviewer for raising this point. By this phrase we meant that with a system prompt or an adapter, the same input produces different hidden states in the model. An encoding model is a linear map from those hidden states to brain activity, so when an expert's best predicted region changes, that change comes from its hidden states. In the revised paper, we have changed the wording to say what is measured directly: specialization shifts a model's alignment to better match activity in the matching brain regions (Introduction and Conclusion). The paper also measures these internal changes directly (Section 4.3, Figure 4a, and Appendix G): by centered kernel alignment, fine-tuning moves the six experts' representations 1.4 to 6.8 times further apart than prompting does, increasingly with depth, and the surface-form control changes the representations more than cognitive prompting does without producing the regional alignment.
We agree that these measurements alone do not show what changed inside the model, and we have started a preliminary analysis of it. For each prompted expert, we took the difference between its hidden states and the base model's on the same story, and used PCA to find the directions in which this difference varies most. No brain data is used in this analysis. We then built two models, the base model plus the expert's change along only the top directions, and the expert with that change removed, and scored both like any other model.
Displacement for prompted experts on Le Petit Prince (49 subjects), at each model's Table 1 setting.
| Model | LLaMA | Phi-4 |
|---|---|---|
| Full expert | +0.42 | +0.17 |
| Base model plus the 64 directions | +0.47 | +0.24 |
| Expert minus the 64 directions | +0.08 | 0.00 |
About 64 directions are enough in both models: adding them to the base model reproduces the effect, and removing them from the expert abolishes it. Fine-tuned experts spread the effect over more directions (about 256 for fine-tuned LLaMA on Le Petit Prince). These results are preliminary, and we plan to develop them further.
- (More modest claim) We have made the claim more modest in the revised paper: specializing a model changes which region each expert predicts best relative to the others, without improving overall prediction accuracy, and a new Limitations section states the scope of the result. We also added evidence that the effect reflects the experts' cognitive specialization rather than incidental differences between the models.
First, on average, experts are above the original base model in their own regions (r = 0.0891 against 0.0880, p < .001), while the other experts in those regions overall are not (Section 4.2 and Appendix F).
Second, properties of the fine-tuning data do not explain the effect. The effect is retained after accounting for seven properties of the training sets (p = .001). The effect also remains when experts are retrained on sets with the same inputs, size, format and target length (positive in all nine combinations, and significant in eight; Section 4.4 and Appendix B).
In addition, the preliminary analysis described under point 2 suggests that the effect can be produced and removed by changing the model directly. In a separate preliminary test, the effect also grows with the strength of the change: scaling a fine-tuned LLaMA adapter from 25% to 100% raises displacement from +0.39 to +0.89 on LeBel, and on Le Petit Prince it is significant at every strength.
If there is a specific analysis the Reviewer has in mind, we would be happy to address it.
Thanks for the detailed clarifications, which have addressed my concerns. I decided to update my initial assessments.
Thank you
We thank the Reviewer for the thoughtful review and for updating the assessment. The comments helped us make our claims more precise and strengthen the paper with additional analyses.
Response to Reviewer ZJcd (1/2)
We thank the Reviewer for recognizing the question, the breadth of the experiments, and the control analyses, and for the comment about honestly reporting the weaker cases. We have uploaded a revised paper that includes the changes described below. Please find our responses to the raised concerns below:
Displacement under different region definitions, over the twelve LeBel and Le Petit Prince combinations.
Displacement stays positive in both cases, and removing the language network lowers it only slightly. As reported in the paper, the effect also holds with continuous Neurosynth maps and with other region sizes. Since these definitions give the same pattern, we conclude that the result does not depend on one particular mapping. We have added these analyses to the revised paper (Section 4.2 and Appendix C).
The datasets do not include functional localizers for most of these domains, so we have also added a short passage to the Limitations section suggesting that future work could use individual-specific regions defined using such tasks.
We also agree that the Pereira mapping is loose. This point, which another reviewer also raised, led us to a new analysis. In the original submission, we used the version of the Pereira dataset distributed with Brain-Score, which is the one most often used in studies of language models and the brain. This version only includes functional data for voxels inside five functional networks, so the Neurosynth regions could not be mapped onto it, and we translated our domains onto these networks instead. In the revised manuscript, we have obtained the original whole-brain release of the dataset, allowing us to test the same regions as used with the other two datasets. This release includes eight of the paper's ten participants, each in their own brain space, unfortunately without anatomical scans. For these eight participants, we estimated each participant's mapping to standard brain space from the atlas labels included in the release, placed the same six Neurosynth regions as on the other two datasets, and fit all models again. Without anatomical scans this mapping is an approximation, which we note in the revised paper. With the numerical expert now included, displacement on Pereira is positive in all six combinations of model and intervention (mean +0.41, against +0.36 with the networks). We have added this to the revised paper (Section 4.2 and Appendix C).
Finally, we want to emphasize that we do not believe these six cognitive domains are equally important to the brain or privileged in some way over other domains. We selected these domains as capturing high-level cognitive functions that are of interest to both neuroscience and artificial intelligence (and prevalent in both literatures). We have revised the paper to make clear that these six domains are one possible way to divide cognition, not the only one (Section 3.1 and Appendix A).