acceptodds
Under review as a conference paper at ICLR 2027

BrainLLM-Bench: Model Rankings Transfer Across Listeners but Not Always Across Stimuli or Readouts

Abstract

Reading a brain-likeness leaderboard as a fact about models assumes its ranking survives a change of readout, listeners or text. On one listener’s MEG, 32 checkpoints get a reliable leaderboard from representational similarity (ranking reliability 0.98) and an equally reliable one from linear encoding (0.96), in opposite orders (ρ = −0.573, family-clustered 95% CI [−0.79, −0.26]). Brain-likeness benchmarks usually fix one readout and resample only participants, so they can see neither this nor whether a ranking survives a change of text. BrainLLM-Bench changes readout, listeners and stimulus one at a time. Listeners barely move a leaderboard: disjoint listener halves agree at median Spearman ρ of +0.76 to +0.96 (two crossed cohorts, both readouts). The reversal, for the readouts as conventionally applied, survives the 48 base checkpoints of the registered test under family-clustered uncertainty (ρ = −0.59, CI [−0.78, −0.12]) but belongs to one cohort, not reproducing on 27-listener MEG or 47-subject English EEG, and feature geometry is not excluded; on the 32 checkpoints it is not determined once static embeddings are also partialled out (ρ = −0.415). Text depends on the cohort: on four narratives heard by the same 27 listeners, similarity leaderboards from different stories do not transfer (mean agreement near zero; exploratory — the registered co-primary supports it, the registered dispersion primary did not), while a fully crossed fMRI cohort (25 narratives × 9 listeners) transfers well (+0.78). The benchmark behind these tests is one protocol over five language cohorts (MEG, EEG and intracranial ECoG; English and German; scalp EEG an uncontrolled probe), six leaderboard models and a readout sweep extended from 32 to 57 checkpoints (48 base) spanning 2019–2026. We prove which feature transformations each readout ignores; this shows the two can disagree without either being wrong, not that this is why they disagree here, and the reversal is a disagreement between measurements, not a refutation of brain-likeness. Model size did not consistently order alignment on a Pythia ladder (70M–2.8B) or across three Qwen generations (up to 72B). A brain-likeness ranking is a claim about a readout, a cohort and a stimulus sample, and should name all three.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-likeness, language models, MEG, EEG, ECoG, encoding models, representational similarity, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f6vgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article explores the brain similarity of the model and proposes BrainLLM-Bench. By changing one factor (either the reading method, the audience, or the stimulus material) separately each time, it assesses the consistency of the model under different conditions. Through numerous experiments, it is shown that maintaining good stability after changing the audience is achieved, while changing the reading method may lead to a reversal.

Critical strengths
  1. The author conducted a study by comparing the similarity between these language models and the human brain. This is an extremely interesting work. In the article, the author mainly discussed the significant differences between the reliability of rankings and the transfer of rankings, and conducted numerous experiments to verify this. Under different measurement conditions, this distinction was made clearer, and the reliability of the rankings was also evaluated.

  2. This article also proposed cross-story experiments and analyses. Although it has not been confirmed whether they support the reversal, from these aspects, a broader interpretation can be supported.

Critical weaknesses
  1. It is somewhat questionable whether the comparison of reversed rankings between the two datasets is fair, as one method uses standardized features in Encoding, while the other RSA calculation relies more on raw geometric features. The authors need to thoroughly justify this difference and demonstrate whether it might affect the results.
  2. It should be considered that Encoding may be significantly influenced by regularization terms; thus, the authors need to conduct more comprehensive experimental analysis to verify whether the same conclusions as presented in the original paper still hold under optimal regularization settings.
  3. In the supplementary materials, after adding C4, the authors have not confirmed any ranking reversal. Therefore, they must provide a detailed explanation and analysis of how the inclusion of C4 alters these variables. Additionally, it remains unclear under what circumstances C1–C3 are applicable.
  4. The ranking reversal mainly occurs in the LibriBrain dataset of a single subject. The failure of cross-story transfer is currently an exploratory finding, but the applicable conditions for each item still need to be specified.
  5. It is suggested that the author provide a summary table, listing the data sets used for each analysis, the reading methods, and the control items, so that the relevant experimental settings can be more clearly understood when reading.
Questions for authors

The author should clarify whether the difference in choosing standardized features and original geometric features would have any impact? Would it lead to inconsistent interpretations of the results? The author regards these as the same brain similarity measures in the paper and suggests explaining the reasons for these two readings, as well as recommending the use of the same feature processing method for a sensitivity analysis experiment.

1 comment

I will maintain my score.

Reviewer_86JyOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BrainLLM-Bench and shows that model brain-likeness rankings depend on readout, listeners, and stimulus. On one LibriBrain listener, RSA and ridge encoding rank 32 checkpoints in opposite orders, but this reversal does not reproduce on MEG-MASC or Brennan EEG. The authors argue a brain-likeness claim must name readout, cohort, and stimulus sample.

Critical strengths

The core idea is valuable because it treats readout, listener, and text as separate sampling axes and shows brain-likeness leaderboards are claims about a specific measurement configuration rather than model truth.

Critical weaknesses

The central reversal rests heavily on a single-subject LibriBrain cohort, and key cross-cohort or cross-story results are exploratory, uncontrolled probe data, or undermined by tokenization defects, so the main claim is thin and not robustly established.

The pipeline has serious computation integrity problems, including physically scaled neural data fed to unit-scale code, unspaced tokenization affecting many checkpoints, and stored-claim reproduction checks that pass while underlying scores are wrong, which weakens trust in every quantitative conclusion.

Questions for authors

Why should a reversal found in one listener with many sessions be treated as a benchmark-level finding when standard group-level brain studies would not generalize from a single participant?

Given that 18 of 25 extension checkpoints were first scored on unspaced text and cross-cohort comparisons also cross tokenization rules, how can any reader trust the readout concordance numbers as properties of models rather than extraction artifacts?

If the registered story-transfer primary failed and the C4-controlled reversal is not determined, what remains of the paper that is both confirmed and not already obvious from prior language-as-fixed-effect and metric-dependence critiques?

The claim-reproduction harness reports green while the scores come from defective code; does this not show the evaluation methodology itself is not ready for acceptance?

Scaling conclusions mix Qwen generations, Pythia ladders, different cohorts, and different ridge grids; under what preregistered rule is any cross-generation size claim actually supported rather than post hoc?

Reviewer_bnfWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This is a paper try to do what its title claims, "Model Rankings Transfer Across Listeners but Not Always Across Stimuli or Readouts", but is not yet properly prepared for what it want to claim.

Critical strengths

I cannot state the strength of this paper in its current state. This paper requires proper attention and effort before it is prepared for a serious review. I provide my current judgement and suggestion in the Critical Weaknesses.

Critical weaknesses

This year, we are seeing the negative impact of AI-driven paper writing flooding into the whole literature, and this paper is one of the iconic one. The writing is not even close to professional academic writing, nor elementary academic writing. This is something like a murmuring of AI, an internal notes of a research group, and not meant to be published in any public space. As a human reviewer, I ask the author (could be human or AI) of this paper to fully revise the whole content. Make it read like a paper meant for human and prepared for human.

My prompt to the author (for the AI author; if you are human, you can skip this and just copy/paste this to your AI):

  1. The writing is awful. Even the abstract is awful. Very hard to read. This is more like an insider’s note, not for public audiences. Even experts in this field cannot understand smoothly on the writing. Information is fragile, partially hidden, and involved. An abstract itself needs to be understandable without reading the full content. Readers can get an overall understanding of the paper, not a bunch of question marks after reading it. This abstract is too puzzling.
    1. So so many numbers here and there. Only AI can read it. Academic papers for human should not be written in this way.
    2. Please completely rewrite the whole abstract. There is no “structure” in the current abstract. The most important contribution of this paper should be the insight and “so what”. In the current writing, my only output of my mind is “so what”? I just see a series of sentences interleaved with numbers, jargons, and no insights, no structures, no reasoning, and no story.
  2. In the introduction, again, too many partially hidden information that is not written for human. “Proposition 1 shows that readouts can disagree, not why ours do; the claim covers the readouts as conventionally applied, with lexical confounds C1–C3 partialled out, and adding C4 leaves the reversal not determined (Appendix J)”. If this sentence appears in conclusion, then this writing is more acceptable. However, it appears in introduction, where nothing about Proposition 1, C1, C2, C3, C4, are introduced or mentioned in any way. Even LLM cannot read this kind of sentence in this context, not to mention a human reader.
  3. In the introduction, even the three ending bullets, which are meant for highlighting the insight and contribution, looks more like a collection of details. For example, in the first bullet, what does the human author of this paper want the reader to see? “Five cohorts, six leaderboard models and a 32-to-57-checkpoint readout sweep (2019–2026) under one protocol and both readouts; a new model enters from frozen features, with routes, result files and numerical checks provided for review (Section 4).” At least, as a human like me, I see no insight in the content of this bullet. The title of this bullet is “A benchmark that re-scores a ranking with one argument changed.”, but the content of this bullet mentions nothing about “re-scores”, “changed”. I am super confused after reading the whole abstract and introduction.
  4. The whole section 3 needs to be rewritten. The variables, equations, definitions are so fragile and hard to read. Many conditions or definitions are not well justified or explained. For example, what is the formal definition of C1, C2, and C3 and why we need them? I only see "C1–C3 are pairwise word-length and log-frequency differences and orthographic edit distance." but no further definition on each and justification why we need these three conditions. It is quite a torture to read and grasp them. As a reviewer, I suffer from the reading, and I don't want the readers of ICLR to suffer from it again.
  5. All the sections have similar issues. Please use your skills as an AI on research or paper-writing to polish the whole paper properly, and prepare it for human readers.

Now, let's start talking about the technical content of this paper. Both the human and AI author of this paper should read this part:

The major issue of this paper is that the whole methodology is not solid. The ROI (which set of electrodes, or channels, or regions of the brains) is not aligned among different datasets (and also poorly documented, I struggle to identify where is the ROI for each dataset used in this paper). My reading is, all the datasets are using their own ROI, and I am so confused why the authors choose the ROI for each dataset in so different and unrelated ways. In the paper, they say "on LibriBrain, left-temporal sensors, broadband, 300–400 ms" in line 156, and then "right temporal sensors on MEG-MASC" and "all 60 channels on Brennan" in line 430. So, now what? I see left temporal region in one dataset, and then right temporal in another, and full brain region in the other? The whole study makes no sense under such setting.

If even the ROI is not aligned among datasets, the result of brain alignment, no matter which one proposed in this work (the RSA or encoding), cannot be read and explained properly, since brain-LLM alignment highly depends on which brain region and which LLM states you are talking about. Such negligence makes all the cross-dataset discussion invalid.

Questions for authors
  1. Please rewrite the section 4 in a structured way, so the reader can easily see the ROI, the dataset scale, the stimuli type and size, etc. The major thing I am expecting is the design of ROI on each dataset. Since these datasets are all brain recording on audio language stimuli, I am expecting to see a focus on audio cortex and language network. Whole brain as ROI can also be chosen, but please justify why choosing such setup in this work.
  2. Current analysis (section 5.1, 5.2, and 5.3) all have focus on cross cohort comparison. However, without aligning the ROI, the current result is not valid to be discussed.
  3. Analysis in 5.2 is not complete. I only see MEG-MASC and ds003020. However, for across-story analysis, how about other datasets' result? Even in the current state, the result looks very divergent, without scientific insight so far.
  4. I only see analysis on ρ(π RSA, π enc). However, no matter we are seeing positive or negative ranking correlation, we must also take a look on the actual ranking. What kind of models are ranked higher, in RSA, and in enc, especially when the correlation is negative. I can see related analysis in 5.4, focusing on the size of the LLM, in "On the largest ladder the ordering is itself readout-dependent: encoding falls with size (72B vs 0.5B, paired p = 0.0005) while similarity rises (p = 0.001)". However, this makes the result weird. Why the one fall and one rise? Explainability is crucial in this kind of study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.