acceptodds
Under review as a conference paper at ICLR 2027

SCRIBE: Scalable Contextual Representations of Interpretable Brain-Grounded Semantics

Abstract

Understanding how the human brain constructs meaning from real-world language remains a fundamental challenge, as semantics depend on context that varies in natural settings. To enable the study of real-world linguistic meaning in the brain, we introduce SCRIBE, an approach for modeling context-sensitive meaning. Grounded in a neuroscience-informed semantic space and developed using an ensemble of open-source large language models, SCRIBE dynamically annotates meaning in context to produce interpretable, word-level semantic representations for an arbitrary text. Experiments using human judgments, controlled synthetic contexts, and brain activity recorded while participants listen to naturalistic stories, show that SCRIBE captures systematic context-dependent semantic shifts and predicts neural responses in language-selective brain regions, significantly outperforming non-contextual semantic representations. Variance partitioning further shows that SCRIBE captures unique brain-predictive information beyond syntactic features, supporting its ability to encode contextual semantic information relevant to the brain. Together, these results establish SCRIBE as a scalable and interpretable foundation for studying natural language semantics in the brain.

Primary area:
applications to neuroscience & cognitive science
Keywords:
representations, semantic processing, large language models, computational neuroscience, interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3H8GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SCRIBE, a method that employs LLMs to annotate the semantic features of Binder et al. (2016) of English words with context involved, in order to get interpretable, contextualized semantic representations for large-scale text. Experiments show that SCRIBE shows high (up to r=0.77) correlation with human notations and considerable (r=0.1837+-0.0144) brain alignment scores. Following analyses indicate the semantic uniqueness of SCRIBE than previous works (e.g. QAEmb, LLaMA embedding).

Critical strengths
  1. The paper implements a simple yet effective sementic annotation method and demonstrates its compliance to human reference. Also, it shows the importance of context in the annotation process.
  2. It analysed the semantic uniqueness of the proposed method than previous methods.
Critical weaknesses
  1. The experiments are insufficient or questionable. First, it does not compare the brain alignment score of SCRIBE and human annotation on either controlled or natural stimuli. Second, when comparing QAEmb and SCRIBE_c, it seems that the paper does not control the two methods to use the same set of words in the stimuli to derive semantic representations and calculate brain alignment scores. Third, the paper compares the brain alignment scores of SCRIBE and LLaMA-2 embeddings in the appendix, but the SCRIBE representation uses LLaMA-3.1 instead of LLaMA-2 as one of the candidates; Why do you use an older and weaker model in this comparison?
  2. The SCRIBE method is after all too simple: it just prompts the LLMs to give ratings in pre-defined dimensions. Given this simplicity of the proposed method, the paper does not raise a clear, valuable research question either, which questions the contribution of the paper.
Questions for authors

None

4 comments

Thank you for the feedback. We have added several new experiments to address your concerns and provide more clarity.

[W1.1] Brain alignment on a human-annotated brain dataset: To the best of our knowledge, there are no publically available fMRI datasets that are already human-annotated with the Binder properties and collecting such human annotations in the time span of the rebuttal is not feasible. The closest dataset is the fMRI data of [R1], which consists of fMRI recordings of viewing line drawings and words corresponding to 60 concrete nouns. 28 of these 60 nouns also occur in the original Binder dataset (Binder et al. 2016) and have corresponding human annotations. We restricted analyses to these 28 nouns and trained encoding models using both the human-annotated Binder properties and the corresponding SCRIBE estimates. We evaluated predictive performance on fMRI data for heldout nouns via Mitchell et al.’s 2v2 accuracy (i.e., 2 heldout fMRI recordings are compared for closer similarity to their true corresponding test data vs. to a distractor): Human ratings reach a 2v2 accuracy of .71 and SCRIBE_O .64 (chance .50; Mitchell’s original features .60): SCRIBE_O approaches human annotation, with no significant difference (one-sided Wilcoxon, p > 0.05). Natural stimuli: no naturalistic fMRI dataset has human contextual ratings on the Binder properties, and collecting them is very costly; SCRIBE offers a scalable alternative. For example, even an fMRI dataset with just 6-word sentences [R2] would require more than 100k participants and ~200k USD, based on 12USD/hr rate and our own experiments showing the need for at least 8 raters per word & property pair for reliable contextual effects (4246 content words across all sentences in the dataset x 65 properties x 8 raters / 20 judgments per participant which take ~10min). [R1] Mitchell et al. 2008, Science 320. [R2] Tuckute et al. 2024, Nat. Hum. Behav. 8.

[W1.2] SCRIBE annotates only content words because the semantic properties describe concepts; QAEmb provides features for every word. The paper therefore compared SCRIBE annotated on fewer words with QAEmb annotated on all words. We added two controls that match word coverage (App. B.3.4; Wilcoxon tests across participants, BH-FDR-corrected p-values across whole brain and four language ROIs): (1) SCRIBE annotating all words, and (2) QAEmb restricted to SCRIBE’s content words. Whole-brain encoding performance (noise-ceiling-normalized r) orders as: SCRIBE_C, all words (.191) > SCRIBE_C, content words (.184) ≈ QAEmb, all words (.178) > QAEmb, content words (.169). Both “>” are significant in all regions: annotating all words improves SCRIBE_C by 4% (p ≤ .047), and restricting QAEmb to content words lowers it by 5.5% (p ≤ .031). The “≈” is the paper’s original comparison, which is not significant.

With matched words, SCRIBE_C outperforms QAEmb in the whole brain, late language, LTC and AG, both on all words (p ≤ .029) and on content words (p ≤ .049; IFG/MFG n.s. in both). It also explains more unique variance (all words, whole brain: 6.3 vs 1.6 x 10⁻³; p=.020 except IFG/MFG), and the two remain complementary (joint > each, p ≤ .039). The original comparison thus favored QAEmb. We keep content words as the main setting.

[W1.3] We initially used Llama-2-7B as an established baseline for this dataset (Antonello et al., 2023). We have now repeated all analyses with Llama-3.1-8B, Qwen2.5-7B and Mistral-7B (App. B.3.7). With the same context length, these models perform similarly to Llama-2-7B (whole brain .243–.250 vs. .240; only Mistral-7B is slightly higher, +.010), so Llama-2-7B is representative, and our conclusions hold for every model: at a longer context window, contextual LLMs outperform SCRIBE_C, which outperforms LLM layer 0 (p ≤ .008); SCRIBE_C retains significant unique variance beyond each LLM (except Qwen in IFG/MFG) and shares less variance with syntax than every LLM in late language, IFG/MFG and LTC (p ≤ .020).

[W2] SCRIBE’s simplicity is a deliberate design choice. Its individual ratings, grounded in the neurocognitive semantic space of Binder et al. (2016), provide interpretable representations that are comparable to human judgments and straightforward to reproduce. Crucially, they allow us to investigate how specific neurocognitively grounded semantic dimensions change with context and contribute to brain responses, questions that existing representations do not readily address. Reviewers 4jaR and BdDb also highlight this interpretability as enabling more targeted hypotheses than LLM embeddings or context-independent feature norms.

Our experiments show that (i) human ratings shift reliably with context (921 participants), (ii) LLMs reproduce them at 75–80% of the noise ceiling, (iii) context improves brain prediction beyond QAEmb, and (iv) this holds under syntactic, low-level and word-coverage controls.

We hope this addresses the concerns and are happy to clarify further.

As the discussion period is ending today, we wanted to ask you whether the additional evidence and clarifications we provided in our response address your main concerns, and if so, to consider adjusting the evaluation of our work accordingly.

Thank you for your reply. I agree that the new results improves the fairness in the comparison. However, there are still some problems:

  1. The claimed motivation of this paper is said to "generate scalable, interpretable, context-dependent semantic representations" to "arbitrary natural language contexts". However, the current experiments only compare SCRIBE with human annotation on isolated words with regard to brain alignment score. This cannot support the advantage of this method in better fitting the human brain's semantic representation in contextualized natural text. The lack of time or money is not a valid reason for experiments that deviate from the main goal of the paper.

  2. Instead of forcing LLMs to Binder-label every word in the corpus, there should be a clear range of words to be labelled. For example, when the word "the" is assessed with "emotion" or "motion", what does it even mean?

  3. If understand it correctly, the novelty of SCRIBE is to use LLM instead of human annotators to produce Binder labels on large-scale text. This seems a engineering approximation instead of scientific finding to me.

Thank you for the follow-up, and for acknowledging that the new controls improve the comparison. Due to the lack of time, we respond briefly to the three remaining points.

Contribution (item 3). The Binder et al. norms are context-independent ratings of isolated words. Our first contribution is empirical and independent of any LLM: using new human ratings (921 participants), we show that these neurocognitive semantic properties shift reliably with context, which to our knowledge has not been shown before. SCRIBE therefore does not approximate an existing resource: it extends this semantic space to words in context, validated against new human data, which we will release. This enables findings that were not previously accessible: context-driven gains in brain prediction are carried mainly by Event, Emotion and Spatial properties (66–77%), individual properties reliably map onto the regions their labels predict, and SCRIBE_C explains unique variance beyond QAEmb and beyond every tested LLM (except Qwen in IFG/MFG).

Contextual evidence (item 1). Our claim about contextualized text does not rest on the isolated-word analysis, which we added only to answer W1.1. It rests on two results in context. (i) In our human study (921 participants), ratings shift reliably with context (ICC(1,8) = .72–.84), and SCRIBE reaches 75–80% of the human noise ceiling on these contextual ratings, collected on controlled sentences. (ii) On naturalistic story fMRI, contextual SCRIBE_C predicts brain responses better than context-independent SCRIBE_O. This gain is essentially unchanged after controlling for low-level features, and SCRIBE_C retains 2–3× more unique variance beyond fastText than SCRIBE_O, indicating that the gain reflects contextual meaning and not lexical or low-level information. We do not claim that SCRIBE fits the brain better than human annotation; we claim it is a scalable alternative, validated against human contextual ratings, for a setting where no human contextual annotations of naturalistic fMRI stimuli exist.

Range of words (item 2). We agree, and this is the paper’s main setting: SCRIBE annotates content words only, selected by POS tag. The all-words variant was run solely to match word coverage with QAEmb, as raised in W1.2, and is reported only in App. B.3.4.

Reviewer_4jaROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose SCRIBE, an interpretable and scalable semantic feature space, especially for use in the field of NeuroAI. SCRIBE uses language models to rate words in context on 65 semantic categories (from Binder et al. 2016). The authors show that SCRIBE embeddings match human ratings out of context (for isolated words), in context (with a new behavioural study), and can predict fMRI activity during a naturalistic language comprehension task better than the no-context variant and existing interpretable QA embeddings.

Critical strengths
  • The method is simple and interpretable. It uses a well-known, brain-based concept space and applies it to create interpretable context-dependent embeddings.
  • The validation of the generated embeddings against human ratings, both in and out of context, is good for their behavioural validity.
  • The new human dataset of context-dependent feature ratings (500 participants) is a useful resource.
  • Encoding modelling is done per participant, and results look consistent.
  • Comparison of the encoding performance of SCRIBE to the already existing QA embeddings.
Critical weaknesses
  • Unclear whether SCRIBE’s brain alignment improvements stem from the Binder feature space or from the LLM ensemble. The Discussion also does not explain why contextual SCRIBE captures more semantic variance than QAEmb.
  • The syntax control is poorly motivated, since sparse concept-level features are unlikely to carry much syntax. A lexical-semantic baseline (e.g., English1000, fastText) would be better and is missing.
  • Low-level features (word rate, phoneme rate, audio envelope) are not included. The embeddings may correlate with these stimulus properties, so it is hard to know where, and by how much, SCRIBE truly beats SCRIBE-O and QAEmb.
  • Table 1’s permutation test is easy to beat. I'd suggest a randomly assignement concept baseline.
  • The paper never shows which of the 65 concepts drive brain alignment or the unique variance, even though interpretability is one of its main selling point.
Questions for authors
  • Why do you think the SCRIBE (contextual variant) captures more semantic variance in brain activity than QA embeddings?
  • How does SCRIBE compare with lexical baselines such as English1000 or fastText?
  • Can authors provide clearer details about the encoding pipeline: ridge parameters,number of iterations, CV style, is a 4s window assumed for all voxels?
  • Is the comparison in Fig 2.a with a joint model or between separately fit models?
  • For the encoding model results, please perform permutation testing and multiple corrections. Do the regional results survive correction for multiple comparisons?
  • Do the overal/regional gains remain the same (both spatially and quantitatively) after accounting for low-level features with variance partitioning?
  • How does SCRIBE compare with lexical baselines such as English1000 or fastText?

Minor:

  • fMRI measures BOLD responses, not neural activity, but the paper refers to it as “neural” throughout.
  • The metric on Figure 3’s y-axis is also not defined. Is it r?
4 comments

Thank you for the feedback.

[New baselines & controls: W1,Q1; W2,Q2/7; W3,Q6] We added fastText (App. B.3.9; English1000 is weaker: r .138, and adds no variance beyond SCRIBE_C) and a low-level control (word rate, phoneme rate, audio envelope; App. B.3.10). We keep syntax as a control for the LLM comparison, since prior work attributes part of LLM-brain alignment to syntax (Caucheteux et al., 2021; Oota et al., 2023). Tests: Wilcoxon across participants, BH-FDR-corrected across whole brain and language ROIs.

Where does SCRIBE_C’s gain come from? Context, not the LLM ensemble or the property space:

  • SCRIBE_C beats SCRIBE_O: With the same LLMs and properties, SCRIBE_C (10 preceding words) beats SCRIBE_O (words in isolation): nc-normalized r .184 vs .150, p ≤ .016 in all regions. Single LLMs nearly match the ensemble (.163–.181; App. B.3.1). We do not claim the 65-d space is the most predictive; we chose it because it is human-normed, which makes its properties testable (W5).
  • SCRIBE_C beats fastText: (.184 vs .152; p = .010, except IFG/MFG). FastText adds no variance beyond it (p ≥ .82), while SCRIBE_C keeps 20–27% of its predictive variance beyond fastText (7/7, p = .008), 2–3× more than SCRIBE_O. SCRIBE_O only matches a static embedding (.150 vs fastText .152, n.s.).
  • Not low-level: The control leaves the context gain intact (unique R² 5.8 vs 5.4 ×10⁻³ before), and it stays significant in every region (p ≤ .016).
  • A compact readout of contextual LLM semantics: LLMs with the same context share 95–97% of SCRIBE_C’s variance and predict slightly better (.20–.21; gap n.s. after the low-level control). Yet SCRIBE_C keeps unique variance beyond every LLM (34/35 tests) and improves each (p ≤ .039).

What does SCRIBE_C capture beyond QAEmb? Encoding performance is comparable (.184 vs .178, n.s.), but SCRIBE_C explains more unique variance in whole brain, late language, LTC and AG (p ≤ .049), also after syntactic and low-level controls. This is not explained by either dimensionality (with 33 properties chosen by QAEmb’s procedure, unique variance is unchanged (5.6 vs 5.4 ×10⁻³) and still exceeds QAEmb’s in whole brain and LTC (p = .039; App. B.3.5)), or by word coverage (on all words, 6.3 vs 1.6 ×10⁻³ (p = .020; App. B.3.4)). After controlling for fastText the advantage is no longer significant, so part of it reflects word-level meaning. This fits SCRIBE’s design: it rates each word on experiential properties in context, whereas QAEmb answers yes/no questions about the preceding text.

[Interpretability: W5] We ran three analyses, following [R1] in decomposing SCRIBE_C’s held-out brain alignment (Pearson r) into per-property contributions (App. B.3.11):

  1. Expected property to region mappings: For seven property→region pairs fixed a priori from the literature (e.g., Landmark/Scene/Path → place network; Number → intraparietal sulcus; HCP atlas regions), the predicted properties are more selective for their region than random property sets of equal size (p < .001), and more selective for their own region than for the other six in 7/7 participants (p = .008).
  2. Contextual gain: Three of Binder et al.’s 14 property groups (Event, Emotion, Spatial) account for 66–77% of the gain from context (SCRIBE_C over SCRIBE_O).
  3. Unique variance beyond QAEmb: Refitting the joint QAEmb+SCRIBE_C model without each group in turn shows the largest unique contributions come from Spatial (31% in AG), Audition (up to 29%), Emotion (11–14%), and Number (6–12%). Vision and Motor contribute no significant unique variance; the remaining variance is shared across other groups.

[R1] Dupré la Tour et al. 2022, NeuroImage 264.

[Statistics: Q5] Block-permutation tests (1000 permutations): all models predict above chance in every ROI (p ≤ .006), and SCRIBE_C’s unique variance beyond syntax, QAEmb and Llama2-7B is above chance in all language ROIs (p ≤ .002). All paired results above are BH-corrected across regions; SCRIBE_C’s unique variance beyond QAEmb also holds beyond syntax (p = .020), and it overlaps less with syntax than every LLM (p ≤ .020). Two AG results did not survive and were removed: the encoding advantage over QAEmb (p = .078) and the lower syntactic overlap than QAEmb (p = .098; holds in late language, p = .039).

[Table 1: W4] We now report our random-concept control explicitly and add two harder ones. SCRIBE (.77) far exceeds pairings with a random word (.12), a same-super-category word (.31) or a same-category word (.52; all p < .001). In context (now 921 participants, ~12 ratings/item), SCRIBE reaches 75-80% of the human noise ceiling, close to Binder’s norms (84%; App. B.2).

[Q3, Q4, minor] Ridge penalty fit per voxel (15 log-spaced values, 10¹-10⁴) with bootstrapped CV on training stories; one 2-8 s window (four delays) for all voxels (App. A.4.5). Fig. 2a compares separately fit models; Fig. 3’s y-axis is unique R². “Neural” is replaced with “brain”/”fMRI”.

As the discussion period is ending today, we wanted to ask you whether the additional evidence and clarifications we provided in our response address your main concerns, and if so, to consider adjusting the evaluation of our work accordingly.

Thank you for the new analysis; the added stats and controls are helpful.

However, I am now more skeptical. After the new controls, the advantage over QAEmb is no longer significant once fastText is included, and two AG results did not survive correction. The abstract and claims should reflect this. The remaining effects are also very small (unique R² ≈ 5–6 ×10⁻³). For transparency, I'd suggest to:

  1. include whole-brain maps for all subjects of SCRIBE-C’s unique variance relative to fastText, the low-level features, and QAEmb. These would show whether the effects form consistent clusters in language regions or just look like salt-and-pepper noise.

  2. include a brain map with a 2D colorbar of the lexical baseline and SCRIBE-O, as they perform about the same (.150 vs .152).

  3. report the number of bootstrap iterations used for the ridge cross-validation.

I keep my overall score for now and reduce the soundness score.

Thank you for your engagement. Your suggestion led us to control for all baselines together (fastText, low-level and syntax) instead of one at a time, which gave us a clearer test of our main result.

Significant unique variance predicted by SCRIBE_C in language regions over QAEmb, when accounting for all considered controls. Beyond fastText, syntactic and low-level features, SCRIBE_C retains significant unique variance over QAEmb in the whole brain and in every language ROI (1.8–3.1 ×10⁻³; 7/7 participants, p = .008, BH-corrected; Table 33). The new whole-brain maps for all participants (Fig. 20) show that this unique variance is located in language regions.

fastText and SCRIBE_O predict largely the same regions. As suggested, we added a 2D-colormap map for all participants (Fig. 17), confirming that the two predict largely the same regions, with no consistent advantage for either in the voxels both predict well. This is in line with our previously reported encoding performance, where they perform similarly (.150 vs .152, n.s.). Even though their encoding performance across the brain is similar, each captures some unique variance: SCRIBE_O beyond fastText in every region, and fastText beyond SCRIBE_O in late language, LTC and AG (App. B.3.9, Table 30).

Bootstrap iterations. We use 3 iterations (App. A.4.5).

We have updated the PDF accordingly.

Reviewer_BdDbOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCRIBE (Scalable Contextual Representations of Interpretable Brain-grounded sEmantics) -- a method for constructing rich input representations for fMRI encoding models that are intended to capture context-sensitive semantic features while remaining interpretable. SCRIBE extends earlier work leveraging context-independent embeddings of semantics by instead using an ensemble of LLMs to generate context-sensitive semantic feature labels.

Through a careful progression of experiments, the authors demonstrate: (1). SCRIBE aligns with human semantics judgements for novel items. (2). SCRIBE tracks shifts in human semantics judgements for the same items across different contexts (demonstrating human-like context-sensitivity). (3). SCRIBE representations outperform a context-independent baseline method (QAemb) at predicting neural responses during a story comprehension task (4). SCRIBE overlaps less with brain regions sensitive to syntax while retaining more unique brain-predictive variance than QAemb in a variance partitioning analysis.

Critical strengths

Clarity of Presentation: The paper is very clearly written and is easy to understand. Its experiments follow an intuitive progression from: first behaviorally validating SCRIBE generated features, to verifying their neural predictivity relative to sensible baselines, before eventually evaluating how the unique brain predictive variance is distributed relative to syntax-sensitive features.

Usefulness as a method: SCRIBE has natural appeal to researchers in NeuroAI that are hoping to better understand what kinds of feature drive brain-model alignment in encoding model analyses of natural language tasks. By supporting the construction of interpretable, expressive, and context-sensitive semantic feature spaces, SCRIBE might allow one to test much more targeted hypotheses about the neural representation of semantics than has been possible with earlier models of semantics that were either uninterpretable (e.g., LLM-representation derived encoding models) or poorly performing (off-the-shelf semantic feature norms).

Critical weaknesses
  • Difficulty assessing qualitative strength of model fits: Given the lack of stronger baselines than QAemb, it is difficult for me to assess the overall quality of SCRIBE encoding model fits. I can appreciate that the purpose of SCRIBE is NOT to achieve a new SoTA BrainScore, but some comparison with the performance of encoding models fit to the same task using representations extracted from a more competitive baseline model (e.g., an LLM) would help provide some additional scale to the results. Including such a baseline would improve my evaluation of Soundness.
Questions for authors

N/A

4 comments

Thank you for the feedback.

As requested, we now provide additional evidence for the quality of SCRIBE_C’s encoding model fits from three extra LLMs (see below). Overall the results show that SCRIBE_C approaches LLM brain predictivity, particularly at matched context lengths, while capturing significant brain-relevant information not contained in LLM embeddings.

Specifically, while the original submission compared SCRIBE with Llama-2-7B, we have now added three newer LLMs: Llama-3.1-8B, Qwen2.5-7B and Mistral-7B (App. B.3.7). Each newer model was evaluated with SCRIBE_C’s 10-word context and with the same 256–512-word context as Llama-2-7B. We selected the model layer either based on prior work with this model and same dataset (layer 15 for Llama-2-7B, Antonello et al., 2023), or by the best performance on the training stories (layer 12 for Llama-3.1-8B and Mistral-7B, layer 16 for Qwen2.5-7B). All p-values are FDR-corrected across brain regions.

(i) Matched 10-word context as SCRIBE (new LLM setting). LLMs, provided with the 10-word context that SCRIBE has access to, reach whole-brain noise-ceiling-normalized r of .204-.207, so SCRIBE_C achieves ~90% of matched-context LLM performance, and the LLM advantage is no longer significant in LTC (nor in AG for Llama-3.1-8B). SCRIBE_C retains significant unique variance beyond every LLM in every region. Once low-level stimulus features (word rate, phoneme rate, envelope; App. B.3.10) are accounted for, the remaining LLM advantage is no longer significant in any region.

(ii) Longer context (the LLM setting from the original submission). LLMs provided with 256-512-word context reach a whole-brain noise-ceiling-normalized r of .240-.250, compared with .184 for SCRIBE_C. SCRIBE_C thus achieves ~75% of LLM performance with only 65 interpretable dimensions. LLMs outperform SCRIBE_C in all regions (p=.008), yet SCRIBE_C retains significant unique variance beyond every LLM in the whole brain and all language ROIs (except for Qwen in IFG/MFG).

We hope this addresses the concerns and are happy to clarify further.

As the discussion period is ending today, we wanted to ask you whether the additional evidence and clarifications we provided in our response address your main concerns, and if so, to consider adjusting the evaluation of our work accordingly.

Thank you for your thoughtful rebuttal. Your analyses addressed my concern regarding the relative strength of your results in relation to other baselines. I will maintain my original positive assessment of your work.

Thank you for your engagement with our work and for taking the time to review our rebuttal.

We’re glad to hear that the additional LLM baselines addressed your concern about the relative strength of our results. These comparisons are now part of the paper, and we believe they make the work stronger.

Thank you again for your constructive feedback and for your continued positive assessment of our work.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. We summarize the main additions below.

Overall, SCRIBE's main results hold under the additional baselines and controls (newer LLMs, lexical embeddings, low-level features, matched words and dimensionality, multiple-comparison correction): contextual annotations near human consistency, a brain-predictive gain from context, and unique variance beyond QAEmb in semantic regions. Moreover, new interpretability analyses show which neurocognitive dimensions drive these effects.

1. Larger human behavioral study of contextual semantics (Sec. 3.2, 4.2, App B.2). We extended the study to 921 participants (~12k ratings) and added a reliability analysis. Human ratings shift reliability with context (ICC(1,8)=.72-.84). SCRIBE reaches 75-80% of the human noise ceiling. For reference, the Binder et al. norms reach 84%.

2. New baselines and controls.

  • Statistics. All paired tests are now BH-FDR corrected across the whole brain and four language ROIs. All the main claims of the paper hold.
  • Newer LLMs (BdDb, 3H8G; App. B.3.7): we addedLlama-3.1-8B, Qwen2.5-7B and Mistral-7B, each with SCRIBE's 10-word context and with the paper's longer context. 10-word context: the LLMs outperform SCRIBE_C by ~.02 in the whole brain, n.s. in LTC. SCRIBE_C retains unique variance beyond every LLM in all regions and shares less variance with syntactic and low-level features (p ≤ .020, all regions but AG). Longer context: the newer LLMs perform similarly to Llama-2-7B and outperform SCRIBE_C. However, SCRIBE_C retains unique variance beyond every LLM (except Qwen in IFG/MFG) and shares less variance with syntactic and low-level features. In both settings, SCRIBE_C outperforms the LLMs’ layer-0 embeddings in all regions.
  • Lexical baselines (4jaR; App. B.3.9): fastText explains no variance beyond SCRIBE_C (p ≥ .82). SCRIBE_C keeps 20-27% of its explained variance beyond fastText (p=0.008). This advantage comes from context: SCRIBE_O performs on par with fastText, and SCRIBE_C retains 2-3x more unique variance beyond fastText than SCRIBE_O. English1000 yields similar results.
  • Low-level features (4jaR; App. B.3.10): after controlling for word rate, phoneme rate and the audio envelope, the gain of SCRIBE_C over SCRIBE_O and its unique variance beyond QAEmb are essentially unchanged, and the gain maps are spatially preserved (r=.89-.95).
  • Word coverage (3H8G; App. B.3.4): with both methods annotating the same amount of words, SCRIBE_C outperforms QAEmb in the whole brain, late language, LTC and AG (p ≤ .049). The original comparison favored QAEmb.
  • Dimensionality (App. B.3.5): with 33 properties (QAEmb's selection procedure), SCRIBE_C’s unique variance is unchanged and still exceeds QAEmb's (whole brain, LTC; p=.039).
  • Random concept-assignment baseline (4jaR; App. B.1): the ensemble’s correlation with human norms (r=.77) is far above a random other word (.12) and a random word from the same category (.52).
  • Human vs SCRIBE brain alignment (3H8G; App. B.3.12): on the Mitchell et al. (2008) fMRI data (28 nouns with Binder norms), human ratings reach 2v2 accuracy .71, SCRIBE_O .64 and the original features .60. The difference between human ratings and SCRIBE_O is not significant (p=.06).

3. Interpretability (4jaR; App. B.3.11). We decomposed the 65-property encoding model into per-property contributions. Properties map onto expected regions: seven a-priori property-region pairs are more selective than random property sets (p < .001), and Landmark/Scene/Path are the most place-selective properties. Event, Emotion and Spatial properties carry 66–77% of the gain from context. In leave-one-property-group-out models, the unique variance beyond QAEmb is concentrated in the Spatial, Audition, Emotion, Number and Gustation domains.

We believe the additional baselines and controls have further strengthened our conclusions, and the new interpretability analyses help improve the impact of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.