- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates how sparse features from language models compare in terms of predicting brain response and predicting the LM’s own hidden state. The authors use fMRI data from 59 participants listening to 2 narrations from the Narratives dataset. They extract features for each token in the narratives using several LMs (different models of Gemma and LLama). They select sparse features from these models using SAEs, varying the number of features and layers. Then, they fit ridge regressions from the sparse features to predict BOLD responses and LM hidden states separately. First, they measure whether they can detect planted correspondence in simulated data. Their analysis is inconclusive, mean detection is very low, regardless of the planted rho. They compare the “self-recovery”, the correlation between the ranking of reliance profiles (changes in held-out R^2 after subtracting values of fitted contribution of each feature). They compare these profiles independently for BOLD and LM prediction, and how they are correlated with each other. Correspondence is low and it is not clear why.
Critical strengths
- The model fitting (ridge regression, stacking delays for BOLD response) and comparison with a baseline use of low-level features (low-level features) is sound
Critical weaknesses
- The paper’s main research question, whether the same sparse features from LMs matter for predicting the LM’s hidden state and fMRI responses, is not clearly motivated.
- The results are not analyzed in a way that’s relevant to neuroscience (e.g., where in the cortex these features do or do not predict) or NLP (e.g., which semantic/syntactic aspects of these features matter for prediction).
- Analyses are mostly inconclusive and inform more on the limitations of the methods than the research question.
- The figures do not convey information very well. Figure 1 reads more like a check-list of the methods and Figure 2 is really unclear. Table captions are quite uninformative. In several Results subsections, it is unclear to which figure/table they refer to.
- Overall writing is quite hard to follow.
Questions for authors
Why do you think fewer features lead to better recovery? Is it because more features are redundant? In any case, it would be interesting to analyze what is represented by these features.