acceptodds
Under review as a conference paper at ICLR 2027

The Low-Rank Brain: Why fMRI Encoding-Model Predictions Can Mislead Decoding and Localization

Abstract

Predictions from fMRI encoding models are increasingly analyzed as if they were measured brain activity: decoded, correlated, and compared across cortical regions. But a predicted response is a function of stimulus features, and its many vertices need not carry independent or regionally specific information. We examine this with two audio-to-fMRI encoders applied to 8,625 fifteen-second solo-piano clips spanning 12 composers and three stylistic eras. The predictions of the first, TRIBE v2, span 20,484 cortical vertices, yet 6, 9, and 19 standardized principal components explain 90%, 95%, and 99% of their variance. A linear ridge encoder, fit independently to one subject’s natural-speech fMRI and applied out of domain, is lower-rank still (3, 6, and 16 components), suggesting that low rank is not specific to one encoder, although this exploratory comparison varies several factors at once. Era is linearly decodable from TRIBE’s predictions and transfers to held-out composers under leave-one-composer-out (LOCO) cross-validation (balanced accuracy 0.469, chance 0.333; composer-permutation ), whereas the ridge predictions do not decode it (0.280). Yet when era is decoded within each of 22 cortical divisions of the HCP-MMP1.0 atlas, no division beats a size-matched random-vertex null (max-statistic family-wise ): the resulting map looks spatially organized, but no region carries more era information than an arbitrary vertex set of equal size. Encoding-model predictions can thus support decoding without supporting region-level localization. We recommend reporting the effective rank of predicted responses and testing region-level effects against size-matched random-vertex nulls before drawing spatial conclusions. Our claims concern downstream analyses of model predictions, not the validity of encoding models or of measured fMRI.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI encoding models, representation learning, neural decoding, low-rank representations, learned representations, cortical localization, neuroscience

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_grqHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines whether decoding information from model-generated fMRI responses supports claims about regional specialization. Using TRIBE predictions for piano music, the authors find that the responses are low-dimensional and contain musical-era information that transfers across composers, but anatomical regions do not significantly outperform size-matched random collections of cortical vertices. The contribution is a cautionary case study showing that successful decoding from anatomically labeled predictions does not by itself establish localization, and recommending dimensionality checks and random-vertex comparisons before interpreting generated cortical maps.

Critical strengths
  • A demonstration of the distinction between decoding and localization when understanding generated fMRI. None of the anatomical brain regions decoded era across composers significantly better than a random-vertex set. This gives a useful reference for interpreting an otherwise plausible-looking cortical map.
  • Additional analysis on low rank. The unpooled analysis shows that temporal averaging alone does not explain low rank, and composer-level evaluation addresses an important dependence structure.
Critical weaknesses
  • Unclear novelty relative to prior real fMRI work. Prior studies have already shown that successful decoding need not depend on the regions most responsive to a stimulus category. In Haxby et al. (2001), excluding voxels maximally responsive to the categories being compared only reduced pairwise identification accuracy from 96% to 94%. Low-dimensional structure has also been reported in measured auditory responses, as the authors acknowledge. Although these findings do not establish the same random-set equivalence demonstrated here, they make the broader conclusions about decoding and low rank less surprising. The paper would benefit from clarifying what its analysis of generated fMRI adds beyond these established observations, and why that contribution changes how these predictions should be interpreted.
  • Unclear how broadly the findings generalize. The main analyses concern predictions for piano music, outside the model’s naturalistic training distribution. It remains unclear whether the low rank and lack of regional decoding advantage reflect a broader limitation of generated fMRI or this particular stimulus and decoding task. Applying the same analyses to held-out stimuli from a setting where the model predicts measured responses reliably would help distinguish these possibilities. Alternatively, a matched comparison with measured music responses could establish whether the reported behavior also occurs in real fMRI. Although the authors acknowledge these limitations, they leave the broader significance of the findings unresolved.
Questions for authors
  • Checkpoint/training clarification: Which checkpoint and version were used? Section 3.2 describes Algonauts movie training, whereas published TRIBE v2 lists Courtois NeuroMod, BoldMoments, LeBel2023, and Wen2017. If the checkpoint used LeBel2023, does its training overlap with the ridge replication's ds003020 subjects or stories, and should the “unrelated” dataset characterization be qualified? This may be a documentation issue rather than incorrect model use. (TRIBE v2, Table 1.)
  • Interpretation clarification: Is the intended contribution a diagnostic with demonstrated sensitivity to meaningful regional information, or a cautionary example for this music task? How should the result be reconciled with prior empirically validated regional findings from generated responses? Or against prior real fMRI analysis in general?
  • Targeted validation that could change the assessment: Does the rank/localization pattern persist where TRIBE accurately predicts measured group-average responses? One route is validated held-out movie/speech stimuli; another is matched music predictions and recordings using ds003720. Genre decoding would not replicate era decoding exactly, but would help assess whether the spatial benchmark distinguishes measured from predicted representations in a similar music task. Repetitions and matched processing would be needed to avoid interpreting measurement noise or averaging differences as model compression.
1 comment

Thank you. Your checkpoint question uncovered a real error.

Checkpoint and training data. We used the public TRIBE v2 checkpoint (facebook/tribev2, loaded with tribev2 0.1.0). You are right: Table 1 of the TRIBE v2 paper lists LeBel2023, which is ds003020, among its four training datasets, with all eight participants. Our ridge encoder (UTS02, ds003020 stories) therefore shares a subject and story corpus with TRIBE's training data. We removed "unrelated dataset/different subject" throughout. The ridge is now described as fit separately but not on independent data (§3.2, §3.9, §4.2, App. C), and §3.2 now gives TRIBE v2's training data and modality dropout. Thank you for catching this.

Novelty relative to measured-fMRI work. We agree that distributed decodability (Haxby et al., 2001) and low dimensionality (Norman-Haignere et al., 2015) are established for measured fMRI. Both are now cited and discussed (§1 "What this adds", §2). Our object and inference differ. A predicted response is a noise-free function of stimulus features, so any regional difference in it reflects the encoder's readout, not cortex. We quantify rank relative to the encoder's own input (a 15–28× compression) and test random-set equivalence directly. The consequence for interpretation: in measured data, random-set equivalence would be evidence about cortical coding. In predictions, a decoding map supports neither a localization claim nor, without measured data, a claim of distributed coding.

Diagnostic or cautionary example? Primarily a cautionary case study of one out-of-domain setting, plus two inexpensive diagnostics. We now state explicitly that the diagnostics' sensitivity has not been validated with a positive control (§5.5). We also report the localization test's resolution in this setting: a best-parcel advantage of about 0.01 balanced accuracy over random sets would have been flagged (§4.4).

As for reconciling our result with validated regional findings from predictions (TRIBE v2's in-silico localizers; Ratan Murty et al., 2021): those findings are mostly univariate contrasts. With Y^=ZBT+E\hat Y=ZB^T+E, the predicted contrast at vertex vv is bvT(zˉA−zˉB)b_v^T(\bar z_A-\bar z_B). It follows the loadings and can be highly selective even when the rank is small. Low rank limits regional differences in decodability, not in mean response, so the two results do not conflict (§5.2).

Generalization and targeted validation. We agree that the analyses you propose would most directly change the assessment: validated held-out movie or speech stimuli, and ds003720 with genre labels. We could not run them in the discussion period. §5.4–5.5 now specify them, including the repeated presentations and matched processing you suggest for ds003720. Until they are run, we present the results as a case study of one regime (§5.4).

Reviewer_f5iUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

fMRI encoding models are trained to predict neural activity from an input stimulus, such as audio, video or speech. The predictions of those fMRI encoding models are interpreted as standalone brain activity. Follow-up tasks such as decoding from the predictions, analyzing the correlations and investigating the spatial structure of the predictions are common tasks carried out on predictions of novel stimuli, allowing to avoid expensive fMRI scans of unseen stimuli. This study analyzed the predictive capabilities of such encoding models, focusing specifically on the TRIBE model. This encoding model was trained to predict neural responses from video, audio and text, using the Algonauts2025 naturalistic movie data. A linear ridge encoder was fitted independently on a natural-speech fMRI dataset and used as comparison. The authors analyzed solo-piano performance recordings (spanning 8625 clips (15sec), 12 composers and 3 eras) and decoded the era from the model’s predictions. Both models were tested out-of-domain, as one was trained on an audiovisual data corpus (TRIBE), the other on a speech-only one. Era was decodable from the TRIBE’s predictions above chance, although the prediction space was shown to be low-dimensional (19 components explained 99% of the variance from the 20 484-dimensional predictions space). The ridge encoder showed similar low-rank (16 components explained 99% of the variance, out of 94,251 dimensions), but failed to decode the musical era above chance. When testing 22 parcels of the brain map, none of the parcels exceeded a random vertex set of the same size, and hence did not show any evidence of spatial specialization. Furthermore, the best performing parcel was a visual one (Early Visual Cortex), although only relying on audio-based predictions. As an additional test, the authors tested the predictability of era from pure acoustic features and found that performance is just as good as it is for the fMRI predictions, indicating there is no extra information captured in the predictions. Finally, the authors conclude that fMRI encoding models should be used with caution, while this does not bear on the validity of encoding models, but requiring further sanity checks by testing the effective rank of predicted responses and regional effects against a size-matched null.

Critical strengths

(1) The study addresses a useful and important methodological issue: interpreting encoding-model predictions for stimuli without measured brain responses as though they were recorded neural activity. The manuscript provides an informative case study demonstrating limitations of downstream decoding and localization analyses performed on such predictions.

(2) A particular strength is the clear distinction between decoding information from predicted cortical responses and localizing that information in the brain. Anatomical labels on model outputs can make a decoding map appear biologically informative even when different regions provide similar readout of shared stimulus features. Comparing each size region with size-matched random vertex sets is a sensible, accessible diagnostic. The composer-held-out evaluation is also valuable, and the authors appropriately avoid claiming that predicted responses outperform their input representation simply because one permutation test is significant and the other is not.

Critical weaknesses

(1) Low rank alone does not establish a lack of localization. The linear-algebra argument requires regional loading matrices to have sufficient rank and assumes that the shared latent information can be recovered usefully. Conditioning, residual information, regularization, and PCA truncation matter in practice. A low-dimensional representation can still have spatially selective loadings. The paper acknowledges parts of this, but the causal connection between low rank and the observed regional results remains suggestive rather than demonstrated. Per-parcel rank and conditioning analyses would strengthen that connection.

(2) The experimental setting limits generality. Both encoders are evaluated outside their training domain, and TRIBE receives audio alone. Thus, the low rank could partly reflect this particular stimulus set or inference regime. The independent ridge experiment adds evidence, but it shares the audio feature extractor and changes several other factors simultaneously. An in-distribution analysis would be particularly informative (see comments below).

(3) The missing measured-fMRI comparison leaves the biological contrast unresolved. Measured responses can also be low-dimensional and distributed. Applying the same analysis to measured music responses would show whether this diagnostic distinguishes model predictions from actual cortical organization, or identifies a limitation of this decoding task more generally. The authors explicitly acknowledge this, but it remains consequential.

Overall, the work can be taken as a practical warning about interpreting encoding-model outputs. Its strongest contribution is the proposed diagnostic and the demonstration that plausible-looking cortical maps need appropriate controls. As a broader explanation of why encoding predictions mislead localization, it requires stronger evidence.

Questions for authors

Smaller type and language feedback\textbf{Smaller type and language feedback} -P. 5, “All accuracies are balanced accuracy”, better: “All accuracies are reported as balanced accuracy” -P. 9: “and its a-priori component count” → “its a priori component count” -P.13: “principal component on the top-50 audio principal …” → “top 50” -P.8,15: ”under-fit” → “underfit”

Figure formatting\textbf{Figure formatting} -Figure 2(c) x-axis label is cut off on the right, and the observed-value annotations at the top are cramped or clipped. Same for Figure 2(a).

Questions and clarifications\textbf{Questions and clarifications} (1) The description of Figure 1(b) does not fully match the plot or results. The caption says “every parcel sits inside the size-matched random-vertex null band”. However, the lowest parcel appears outside the displayed band, and Section 4.4 states that one parcel exceeds its own size-matched 95th percentile. The caption should distinguish the displayed band from the individual size-matched nulls and the family-wise test.

(2) Two different accuracies are given for a five-component predicted-response representation. On p. 8, Section 4.6 reports 0.420 at five components (line 385), then 0.428 for a five-component projection. Figure 4(a) and Figure 4(b) reproduce these different values. If the procedures differ, that distinction needs explaining; otherwise, one value needs correcting.

(3) The acoustic-baseline gaps are understated in Figure 4’s caption. The caption gives gaps of approximately 0.02–0.05. In panel (b), the displayed acoustic accuracies range from 0.452 to 0.509, compared with 0.428 for predicted fMRI (differences of approximately 0.024–0.081). Panel (a) also includes a gap of 0.063.

(4) Some per-composer results in Appendix B.1 disagree with Figure 2 (b). On p. 14, lines 749–750 give Liszt 99.4% and Haydn 80.3% for audio. The plotted audio values appear to be approximately 99.4% for Liszt and 37% for Haydn, so the Haydn value in particular needs checking.

(5) The reported accuracy difference needs a rounding check. On p.7, the difference between 0.469 and 0.383 is reported as 0.085, although the displayed numbers give 0.086. This could result from using unrounded values - if so, using consistent rounding would avoid confusion.

(6) Statements about human-subjects data are very broad for the ridge replication. Appendix A.3 says “There are no human-subjects data” , and the Broader Impacts statement similarly says none are involved. Yet the ridge encoder uses previously collected human fMRI. A more accurate formulation would be: “No new human-subjects data were collected.”

(7) The absence of a significant regional advantage is sometimes stated too strongly. The results support “we found no evidence that any parcel outperforms the random-vertex baseline.” The authors do not establish that no parcel contains more information, or that all regions are equivalent. Statements such as “no region carries more era information” go beyond what a nonsignificant test establishes. Confidence intervals or a sensitivity analysis would help show what regional advantages the experiment could detect.

(8) Distribution shift limits the interpretation and generality of the findings. TRIBE was trained on multimodal naturalistic movies but is evaluated here on instrumental solo-piano clips using only its audio pathway. This changes both the stimulus domain and the available modalities. Although the model supports missing modalities through modality dropout, this does not establish that its predictions retain the same effective dimensionality or regional specificity in this setting. Consequently, the observed low rank and lack of a detectable regional decoding advantage could partly reflect distribution shift rather than a general limitation of encoding-model predictions. The ridge-encoder replication leaves this concern unresolved: it is trained on spoken narratives and evaluated on piano, while also differing from TRIBE in architecture, training data, subject configuration, and output space. The checks of held-out prediction quality and auditory-cortex enrichment support its validity within the speech domain, but do not establish its suitability for predicting responses to music. The authors acknowledge these limitations, but they remain central to interpreting the results. What motivated the choice of speech data for the ridge replication, and could a baseline be trained using features, training data, and response targets more closely matched to TRIBE? Most importantly, could the effective-rank and regional-decoding analyses be repeated on held-out stimuli matching each encoder’s training distribution, with TRIBE evaluated under both full multimodal and audio-only input? These controls would help distinguish effects of domain and modality shifts from limitations attributable to the encoding stage itself.

2 comments

Thank you for the detailed and constructive review. We addressed every numbered point; revised text is in teal.

W1: low rank vs. localization. We agree. §5.2 now states that the argument gives a sufficient condition and an explanation, not a demonstration. Low rank alone does not rule out localization. Regional differences can still arise from a latent loaded on only one region, from an ill-conditioned BSB_S, or from regularization and PCA truncation. The empirical claim rests on the size-matched null. Per-parcel effective rank and the condition number of BSB_S would test the mechanism directly. We name them in the paper but did not compute them.

W2 and point 8: distribution shift. Please see the new §5.4.

Modality. According to the TRIBE v2 paper, training zeroes each modality with probability 0.3 and includes spoken stories presented without video. Audio-only input is therefore a trained regime, although solo piano is outside the training distribution. Ridge validation. The ridge's held-out-speech checks validate it only on speech, and §4.2 now says so. Choice of ds003020. We chose ds003020 because it offers many hours of naturalistic listening per subject with a documented voxelwise-encoding pipeline. Training overlap. ds003020 turns out to be part of TRIBE v2's training data (see our reply to Reviewer 3). The comparison is therefore weaker evidence of independence than we had stated, and we corrected this. We agree that in-distribution held-out stimuli (TRIBE with full multimodal and with audio-only input) and a linear baseline matched in features and data are the right controls. They are listed in §5.4, but we could not run them in the discussion period.

W3: measured fMRI. Please see §5.5: ds003720 with genre in place of era, repeated presentations, and matched preprocessing.

Specific points

Fig. 1(b). The caption now distinguishes three things. The shaded band is a single summary across size bins, shown only for orientation; Inferior Frontal Cortex (0.449) falls below it. Each parcel is tested against its own size-matched null, shown by the ticks and the percentiles. The family-wise test is in panel (c). The panel title, which repeated the overstatement, is also corrected. 0.420 vs 0.428. These come from different procedures. In Fig. 4(a), the classifier extracts five components from all 20,484 vertices. In Fig. 4(b), the predicted responses enter as a fixed block of their five leading PC scores. This is now explained in §4.6 and in the caption. Gaps. Corrected to 0.032–0.073 in panel (a) and 0.024–0.081 in panel (b), in both the caption and §4.6. Haydn. The 80.3% is correct. Haydn's audio marker (0.803) was hidden behind the legend in Fig. 2(b); the visible marker near 0.37 is the predicted-response value. All legends now sit below the panels. 0.085. This is the difference of the unrounded accuracies; the rounded values give 0.086. We now state this convention in the §4 introduction and in §4.5. Human subjects. Now reads "no new human-subjects data were collected", and notes that the ds003020 data are public and de-identified. Updated in §3.1, the Ethics statement, App. A.3, and Broader impacts. Overstatement. Softened throughout, e.g. "we find no evidence that any region carries more…". A new paragraph in §4.4, "Resolution of the test", reports that the max-statistic null's 95th percentile (0.471) is the smallest best-parcel accuracy the test would have declared significant. That is 0.010–0.013 above the random-set means, or about a tenth of the whole cortex's margin over chance. Smaller regional advantages cannot be excluded, and we say so. Fig. 2(a, c). Legends moved out of the axes, the x-label of (c) is no longer clipped, and the observed values (with p) are now in the legend. Typos. All four are fixed.

I thank the authors for their response and the revisions to the manuscript. While the paper highlights an important issue and serves as a useful practical warning, I am still not convinced by the underlying evidence provided and will maintain my initial score.

Reviewer_ePFcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors investigate whether predicted fMRI responses can be effective for brain decoding and localization. The authors used two audio-to-fMRI models and 8,625 piano clips (from the MAESTRO dataset) and confirmed that predicted responses can be well explained by a few principal components. The experimental results show that musical era can be effectively decoded from one model’s predictions. Moreover, the results show that no cortical region outperforms randomly selected, equally sized vertex sets, showing that apparent spatial patterns do not establish regional specialization. The results also reveal that it is beneficial to check how much independent information the predictions contain and to compare each brain region with random groups of the same size before drawing conclusions about where information is represented.

Critical strengths

(1) The analyses of this study show that successful decoding from predicted fMRI responses can coexist with low dimensionality and no detectable regional specificity. (2) This study serves as guidance for future research. It reports effective dimensionality and regional specificity against random controls. Moreover, the study provides effective ways to improve analyses of predicted brain responses.

Critical weaknesses

(1) The major concern of this study is that the experiments use the MAESTRO dataset, which does not have actual brain recordings for the same piano clips. Therefore, there is no direct ground-truth comparison between the predicted responses and measured cortical data. As a result, the study cannot definitively determine whether the low-rank collapse reflects an artificial failure mode of the encoding models or other factors. (2) The encoders are evaluated on stimuli outside their training domains. Thus, the observed reduction in dimensionality may partly reflect domain mismatch. Please include discussions on how this mismatch may potentially affect the results and their interpretation. (3) In the experiments, only two encoders are examined. Thus, the findings of this study somehow limit the generalization and identification of the underlying cause. (4) The main task is musical-era classification using solo-piano clips. Other stimulus types, decoding tasks, and temporal decoding analyses should be included to test the generalization capability.

Questions for authors

(1) Please consider to use a dataset that has actual brain recordings with audio clips. (2) Addtional analyese shold be conducted to investiagte the factors of domain mismatch. (3) Please include more encoders and adopt another task to investagite the generaltizaion capability.

1 comment

Thank you for the review. We take the points in order.

Measured data (W1, Q1). We agree that a measured comparison is the most important missing piece, and the paper says so. Our central claim does not depend on it, however. The claim is this: when a representation is low-rank relative to its vertex count, decoding from an anatomically labeled region does not by itself establish localization, whatever the source of the low rank. Whether measured cortex behaves the same way is a separate question. §5.5 now specifies the matched analysis: ds003720 with genre in place of era, using repeated presentations and matched preprocessing. We could not run it in the discussion period.

Domain mismatch (W2, Q2). The new §5.4 discusses how this mismatch could affect the results and how to read them:

Modality. With video and text zeroed, every vertex is a function of audio alone, which could lower rank. However, audio-only input is a regime TRIBE was trained on: its modality dropout zeroes each modality with probability 0.3, and its training data include spoken stories presented without video. Stimulus domain. A narrow stimulus class could engage only a few directions of a mapping learned on other data. The compression, however, is measured against the same clips' high-dimensional input (91/195/529 standardized components), so it arises in the mapping. Encoder accuracy. Neither encoder's accuracy on piano has been measured. These factors limit how far the magnitude of the collapse generalizes. They do not limit the logic of the diagnostics, which matter most for stimulus classes that were never scanned. §5.4 lists the analyses that would separate the factors: in-distribution held-out stimuli, multimodal vs audio-only input, and a matched linear baseline.

Two encoders; other tasks and temporal decoding (W3, W4, Q3). We agree. §5.5 now states the scope explicitly: one stimulus class, one task, two encoders, and pooled predictions. We also now note that the two encoders share training data (ds003020), which further limits the ridge comparison as evidence of generality (§3.9, §4.2). We frame the work as a case study. A single well-controlled instance is enough to show that decodable, anatomically indexed predictions need not localize. How often this happens is an open question that we do not claim to answer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.