acceptodds
Withdrawn from ICLR 2027

A Cross-Description Test of Sparse Feature Correspondence Between Language Models and Brain Responses

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model features can predict brain responses. Do the features that matter most for this prediction also matter for predicting the model's own hidden state? We compare sparse-autoencoder (SAE) features using fMRI from 59 people listening to two narrations. We select features and fit separate predictors of fMRI responses and the model's final hidden state on one narration, evaluate on the other, and repeat in reverse. We subtract each feature's contribution from the fixed predictions and compare the signed decreases in held-out variance explained. Across six model–SAE pairs and both directions, mean rank correlation is -.056, and no model passes both timing and feature-identity tests in both directions. Simulations show why this null is inconclusive. At 256 features, known positive correspondence is detected in only 1.6%–10.4% of replicates across 64 conditions from two model families, including planted correlation .80. The false-positive fraction is set to 5%. An exploratory sweep recovers median correspondence of .56 at 24 features and .02 at 256, despite the same planted .80. Separate controls recover the fMRI predictor's planted ranking much better with Gaussian predictors of the same shape. These simulations test reliance estimation with fixed predictor matrices, leaving the full transfer procedure's sensitivity unmeasured. The empirical null cannot rule out shared brain–model organization: prediction and repeatability need a separate check that the analysis can recover known feature agreement.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, language models, fMRI, neural encoding, feature importance, statistical power

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WH74OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how sparse features from language models compare in terms of predicting brain response and predicting the LM’s own hidden state. The authors use fMRI data from 59 participants listening to 2 narrations from the Narratives dataset. They extract features for each token in the narratives using several LMs (different models of Gemma and LLama). They select sparse features from these models using SAEs, varying the number of features and layers. Then, they fit ridge regressions from the sparse features to predict BOLD responses and LM hidden states separately. First, they measure whether they can detect planted correspondence in simulated data. Their analysis is inconclusive, mean detection is very low, regardless of the planted rho. They compare the “self-recovery”, the correlation between the ranking of reliance profiles (changes in held-out R^2 after subtracting values of fitted contribution of each feature). They compare these profiles independently for BOLD and LM prediction, and how they are correlated with each other. Correspondence is low and it is not clear why.

Critical strengths
  1. The model fitting (ridge regression, stacking delays for BOLD response) and comparison with a baseline use of low-level features (low-level features) is sound
Critical weaknesses
  1. The paper’s main research question, whether the same sparse features from LMs matter for predicting the LM’s hidden state and fMRI responses, is not clearly motivated.
  2. The results are not analyzed in a way that’s relevant to neuroscience (e.g., where in the cortex these features do or do not predict) or NLP (e.g., which semantic/syntactic aspects of these features matter for prediction).
  3. Analyses are mostly inconclusive and inform more on the limitations of the methods than the research question.
  4. The figures do not convey information very well. Figure 1 reads more like a check-list of the methods and Figure 2 is really unclear. Table captions are quite uninformative. In several Results subsections, it is unclear to which figure/table they refer to.
  5. Overall writing is quite hard to follow.
Questions for authors

Why do you think fewer features lead to better recovery? Is it because more features are redundant? In any case, it would be interesting to analyze what is represented by these features.

Reviewer_9768OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the sparse autoencoder features that predict brain responses to stories are the same ones that drive a language model’s internal representation. Its main empirical result is a null. Its main contribution, though, is a check on the test itself: when the authors plant a known correspondence and run the same procedure, the test usually fails to find it. So the null cannot be read as evidence about the brain or the model.

Critical strengths

The paper’s main strength is its result as a warning signal: a common method for comparing SAE feature rankings across brains and models often fails to recover correspondence even when it is planted. So a null from this kind of analysis cannot be read as evidence that brains and models use different features.

Critical weaknesses
  1. Framing: The introduction and abstract lead with procedure and results rather than with why the question matters to neuroscientists and interpretability researchers. The central claim, that this test is unlikely to detect shared reliance at this size, should come first, and the title should reflect it.

  2. The source of low power is unresolved: he paper does not identify the source of low power. Its bank-size sweep suggests dimensionality, but it uses few seeds and a strong signal; the authors should test whether small banks still work at realistic signal strength.

  3. Identifiability of feature reliance. Each BOLD readout has five lags per feature, so 1,280 predictors for roughly 300 time points per listener. The design is underdetermined: each feature is a linear combination of the others, so leave-one-out reliance is not identifiable and ridge distributes credit somewhat arbitrarily. Stacking listeners does not help, since they share the same stimulus rows. The paper’s own bank-size sweep fits this explanation, with recovery collapsing roughly where the number of predictors exceeds the number of time points. This is likely the main source of low power, and the authors should test it directly (for example, by reporting recovery as a function of predictors per time point).

  4. Feature correlation: Correlated features share credit in leave-one-out importance, which makes each feature’s reliance unstable. The paper reports only typical correlations, which hide the strongly correlated pairs where the problem is worst. The authors should show the full distribution of correlations, test grouping at several thresholds, and simulate clusters of correlated features to show how power degrades.

Questions for authors
  • Does detection recover at smaller bank sizes once the signal is realistic? The answer could separate a method limit (too many features) from a data limit (too little signal or too few TRs).

  • What does the full distribution of feature correlations look like, and do the grouped results hold at thresholds of .70 and .90?

  • Would you state the power finding as the main claim in the title and abstract, with the empirical null as a secondary result?

Reviewer_YnzpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether sparse autoencoder (SAE) features that are important for predicting brain responses are also important for predicting language models' final hidden states. The authors use pretrained SAE features from six language models and fit separate ridge regressions to predict fMRI responses and LLM activations. They train on one narration and evaluate on another, quantifying feature importance through the change in held-out R² when each feature's fitted contribution is removed. The resulting importance rankings are compared using Spearman correlation.

The authors find no consistent correspondence between the rankings (mean ρ = −0.056). However, synthetic experiments with planted correspondence reveal that their feature-reliance estimation procedure has very poor recovery at the primary feature-bank size, making the empirical result inconclusive.

Critical strengths
  1. Attention to statistical validity. The authors evaluate correspondence against temporal and feature-identity nulls, and conduct synthetic experiments to assess whether their analysis can recover known correspondence.

  2. Transparent reporting of negative results. The authors acknowledge their inability to reliably recover planted correspondence, rather than interpreting the empirical null as evidence that brains and LLMs lack shared representational organization.

Critical weaknesses
  1. The central empirical question remains unanswered. The authors find no consistent correspondence between feature-importance rankings, but their simulations demonstrate extremely poor recovery of planted correspondence at the primary feature-bank size, indicating that the central empirical results are inconclusive. While identifying limitations of feature-reliance estimation is useful, I am not convinced that demonstrating these limitations constitutes a sufficiently substantial contribution for ICLR.

  2. The choice of LLM target representation is insufficiently justified. The authors extract SAE features from multiple intermediate layers but exclusively use the final hidden state as the LLM prediction target. This is a questionable choice given that intermediate layers often exhibit greater brain predictivity in language encoding studies. More fundamentally, it is unclear whether the relative importance of SAE features for predicting LLM activations remains consistent across different target layers. If these rankings vary substantially with the chosen target layer, then the proposed correspondence measure may depend heavily on an arbitrary representational choice, further limiting its interpretability.

Questions for authors

Why is the final LLM hidden state chosen as the prediction target, particularly given evidence from previous language encoding studies that intermediate layers are often more brain-predictive? Have the authors investigated whether the LLM-side SAE feature-importance rankings remain consistent across different target layers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.