- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SCRIBE, a method that employs LLMs to annotate the semantic features of Binder et al. (2016) of English words with context involved, in order to get interpretable, contextualized semantic representations for large-scale text. Experiments show that SCRIBE shows high (up to r=0.77) correlation with human notations and considerable (r=0.1837+-0.0144) brain alignment scores. Following analyses indicate the semantic uniqueness of SCRIBE than previous works (e.g. QAEmb, LLaMA embedding).
Critical strengths
- The paper implements a simple yet effective sementic annotation method and demonstrates its compliance to human reference. Also, it shows the importance of context in the annotation process.
- It analysed the semantic uniqueness of the proposed method than previous methods.
Critical weaknesses
- The experiments are insufficient or questionable. First, it does not compare the brain alignment score of SCRIBE and human annotation on either controlled or natural stimuli. Second, when comparing QAEmb and SCRIBE_c, it seems that the paper does not control the two methods to use the same set of words in the stimuli to derive semantic representations and calculate brain alignment scores. Third, the paper compares the brain alignment scores of SCRIBE and LLaMA-2 embeddings in the appendix, but the SCRIBE representation uses LLaMA-3.1 instead of LLaMA-2 as one of the candidates; Why do you use an older and weaker model in this comparison?
- The SCRIBE method is after all too simple: it just prompts the LLMs to give ratings in pre-defined dimensions. Given this simplicity of the proposed method, the paper does not raise a clear, valuable research question either, which questions the contribution of the paper.
Questions for authors
None
4 comments
As the discussion period is ending today, we wanted to ask you whether the additional evidence and clarifications we provided in our response address your main concerns, and if so, to consider adjusting the evaluation of our work accordingly.
Thank you for your reply. I agree that the new results improves the fairness in the comparison. However, there are still some problems:
-
The claimed motivation of this paper is said to "generate scalable, interpretable, context-dependent semantic representations" to "arbitrary natural language contexts". However, the current experiments only compare SCRIBE with human annotation on isolated words with regard to brain alignment score. This cannot support the advantage of this method in better fitting the human brain's semantic representation in contextualized natural text. The lack of time or money is not a valid reason for experiments that deviate from the main goal of the paper.
-
Instead of forcing LLMs to Binder-label every word in the corpus, there should be a clear range of words to be labelled. For example, when the word "the" is assessed with "emotion" or "motion", what does it even mean?
-
If understand it correctly, the novelty of SCRIBE is to use LLM instead of human annotators to produce Binder labels on large-scale text. This seems a engineering approximation instead of scientific finding to me.
Thank you for the follow-up, and for acknowledging that the new controls improve the comparison. Due to the lack of time, we respond briefly to the three remaining points.
Contribution (item 3). The Binder et al. norms are context-independent ratings of isolated words. Our first contribution is empirical and independent of any LLM: using new human ratings (921 participants), we show that these neurocognitive semantic properties shift reliably with context, which to our knowledge has not been shown before. SCRIBE therefore does not approximate an existing resource: it extends this semantic space to words in context, validated against new human data, which we will release. This enables findings that were not previously accessible: context-driven gains in brain prediction are carried mainly by Event, Emotion and Spatial properties (66–77%), individual properties reliably map onto the regions their labels predict, and SCRIBE_C explains unique variance beyond QAEmb and beyond every tested LLM (except Qwen in IFG/MFG).
Contextual evidence (item 1). Our claim about contextualized text does not rest on the isolated-word analysis, which we added only to answer W1.1. It rests on two results in context. (i) In our human study (921 participants), ratings shift reliably with context (ICC(1,8) = .72–.84), and SCRIBE reaches 75–80% of the human noise ceiling on these contextual ratings, collected on controlled sentences. (ii) On naturalistic story fMRI, contextual SCRIBE_C predicts brain responses better than context-independent SCRIBE_O. This gain is essentially unchanged after controlling for low-level features, and SCRIBE_C retains 2–3× more unique variance beyond fastText than SCRIBE_O, indicating that the gain reflects contextual meaning and not lexical or low-level information. We do not claim that SCRIBE fits the brain better than human annotation; we claim it is a scalable alternative, validated against human contextual ratings, for a setting where no human contextual annotations of naturalistic fMRI stimuli exist.
Range of words (item 2). We agree, and this is the paper’s main setting: SCRIBE annotates content words only, selected by POS tag. The all-words variant was run solely to match word coverage with QAEmb, as raised in W1.2, and is reported only in App. B.3.4.
Thank you for the feedback. We have added several new experiments to address your concerns and provide more clarity.
[W1.1] Brain alignment on a human-annotated brain dataset: To the best of our knowledge, there are no publically available fMRI datasets that are already human-annotated with the Binder properties and collecting such human annotations in the time span of the rebuttal is not feasible. The closest dataset is the fMRI data of [R1], which consists of fMRI recordings of viewing line drawings and words corresponding to 60 concrete nouns. 28 of these 60 nouns also occur in the original Binder dataset (Binder et al. 2016) and have corresponding human annotations. We restricted analyses to these 28 nouns and trained encoding models using both the human-annotated Binder properties and the corresponding SCRIBE estimates. We evaluated predictive performance on fMRI data for heldout nouns via Mitchell et al.’s 2v2 accuracy (i.e., 2 heldout fMRI recordings are compared for closer similarity to their true corresponding test data vs. to a distractor): Human ratings reach a 2v2 accuracy of .71 and SCRIBE_O .64 (chance .50; Mitchell’s original features .60): SCRIBE_O approaches human annotation, with no significant difference (one-sided Wilcoxon, p > 0.05). Natural stimuli: no naturalistic fMRI dataset has human contextual ratings on the Binder properties, and collecting them is very costly; SCRIBE offers a scalable alternative. For example, even an fMRI dataset with just 6-word sentences [R2] would require more than 100k participants and ~200k USD, based on 12USD/hr rate and our own experiments showing the need for at least 8 raters per word & property pair for reliable contextual effects (4246 content words across all sentences in the dataset x 65 properties x 8 raters / 20 judgments per participant which take ~10min). [R1] Mitchell et al. 2008, Science 320. [R2] Tuckute et al. 2024, Nat. Hum. Behav. 8.
[W1.2] SCRIBE annotates only content words because the semantic properties describe concepts; QAEmb provides features for every word. The paper therefore compared SCRIBE annotated on fewer words with QAEmb annotated on all words. We added two controls that match word coverage (App. B.3.4; Wilcoxon tests across participants, BH-FDR-corrected p-values across whole brain and four language ROIs): (1) SCRIBE annotating all words, and (2) QAEmb restricted to SCRIBE’s content words. Whole-brain encoding performance (noise-ceiling-normalized r) orders as: SCRIBE_C, all words (.191) > SCRIBE_C, content words (.184) ≈ QAEmb, all words (.178) > QAEmb, content words (.169). Both “>” are significant in all regions: annotating all words improves SCRIBE_C by 4% (p ≤ .047), and restricting QAEmb to content words lowers it by 5.5% (p ≤ .031). The “≈” is the paper’s original comparison, which is not significant.
With matched words, SCRIBE_C outperforms QAEmb in the whole brain, late language, LTC and AG, both on all words (p ≤ .029) and on content words (p ≤ .049; IFG/MFG n.s. in both). It also explains more unique variance (all words, whole brain: 6.3 vs 1.6 x 10⁻³; p=.020 except IFG/MFG), and the two remain complementary (joint > each, p ≤ .039). The original comparison thus favored QAEmb. We keep content words as the main setting.
[W1.3] We initially used Llama-2-7B as an established baseline for this dataset (Antonello et al., 2023). We have now repeated all analyses with Llama-3.1-8B, Qwen2.5-7B and Mistral-7B (App. B.3.7). With the same context length, these models perform similarly to Llama-2-7B (whole brain .243–.250 vs. .240; only Mistral-7B is slightly higher, +.010), so Llama-2-7B is representative, and our conclusions hold for every model: at a longer context window, contextual LLMs outperform SCRIBE_C, which outperforms LLM layer 0 (p ≤ .008); SCRIBE_C retains significant unique variance beyond each LLM (except Qwen in IFG/MFG) and shares less variance with syntax than every LLM in late language, IFG/MFG and LTC (p ≤ .020).
[W2] SCRIBE’s simplicity is a deliberate design choice. Its individual ratings, grounded in the neurocognitive semantic space of Binder et al. (2016), provide interpretable representations that are comparable to human judgments and straightforward to reproduce. Crucially, they allow us to investigate how specific neurocognitively grounded semantic dimensions change with context and contribute to brain responses, questions that existing representations do not readily address. Reviewers 4jaR and BdDb also highlight this interpretability as enabling more targeted hypotheses than LLM embeddings or context-independent feature norms.
Our experiments show that (i) human ratings shift reliably with context (921 participants), (ii) LLMs reproduce them at 75–80% of the noise ceiling, (iii) context improves brain prediction beyond QAEmb, and (iv) this holds under syntactic, low-level and word-coverage controls.
We hope this addresses the concerns and are happy to clarify further.