- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper examines whether decoding information from model-generated fMRI responses supports claims about regional specialization. Using TRIBE predictions for piano music, the authors find that the responses are low-dimensional and contain musical-era information that transfers across composers, but anatomical regions do not significantly outperform size-matched random collections of cortical vertices. The contribution is a cautionary case study showing that successful decoding from anatomically labeled predictions does not by itself establish localization, and recommending dimensionality checks and random-vertex comparisons before interpreting generated cortical maps.
Critical strengths
- A demonstration of the distinction between decoding and localization when understanding generated fMRI. None of the anatomical brain regions decoded era across composers significantly better than a random-vertex set. This gives a useful reference for interpreting an otherwise plausible-looking cortical map.
- Additional analysis on low rank. The unpooled analysis shows that temporal averaging alone does not explain low rank, and composer-level evaluation addresses an important dependence structure.
Critical weaknesses
- Unclear novelty relative to prior real fMRI work. Prior studies have already shown that successful decoding need not depend on the regions most responsive to a stimulus category. In Haxby et al. (2001), excluding voxels maximally responsive to the categories being compared only reduced pairwise identification accuracy from 96% to 94%. Low-dimensional structure has also been reported in measured auditory responses, as the authors acknowledge. Although these findings do not establish the same random-set equivalence demonstrated here, they make the broader conclusions about decoding and low rank less surprising. The paper would benefit from clarifying what its analysis of generated fMRI adds beyond these established observations, and why that contribution changes how these predictions should be interpreted.
- Unclear how broadly the findings generalize. The main analyses concern predictions for piano music, outside the model’s naturalistic training distribution. It remains unclear whether the low rank and lack of regional decoding advantage reflect a broader limitation of generated fMRI or this particular stimulus and decoding task. Applying the same analyses to held-out stimuli from a setting where the model predicts measured responses reliably would help distinguish these possibilities. Alternatively, a matched comparison with measured music responses could establish whether the reported behavior also occurs in real fMRI. Although the authors acknowledge these limitations, they leave the broader significance of the findings unresolved.
Questions for authors
- Checkpoint/training clarification: Which checkpoint and version were used? Section 3.2 describes Algonauts movie training, whereas published TRIBE v2 lists Courtois NeuroMod, BoldMoments, LeBel2023, and Wen2017. If the checkpoint used LeBel2023, does its training overlap with the ridge replication's ds003020 subjects or stories, and should the “unrelated” dataset characterization be qualified? This may be a documentation issue rather than incorrect model use. (TRIBE v2, Table 1.)
- Interpretation clarification: Is the intended contribution a diagnostic with demonstrated sensitivity to meaningful regional information, or a cautionary example for this music task? How should the result be reconciled with prior empirically validated regional findings from generated responses? Or against prior real fMRI analysis in general?
- Targeted validation that could change the assessment: Does the rank/localization pattern persist where TRIBE accurately predicts measured group-average responses? One route is validated held-out movie/speech stimuli; another is matched music predictions and recordings using ds003720. Genre decoding would not replicate era decoding exactly, but would help assess whether the spatial benchmark distinguishes measured from predicted representations in a similar music task. Repetitions and matched processing would be needed to avoid interpreting measurement noise or averaging differences as model compression.
Thank you. Your checkpoint question uncovered a real error.
Checkpoint and training data. We used the public TRIBE v2 checkpoint (facebook/tribev2, loaded with tribev2 0.1.0). You are right: Table 1 of the TRIBE v2 paper lists LeBel2023, which is ds003020, among its four training datasets, with all eight participants. Our ridge encoder (UTS02, ds003020 stories) therefore shares a subject and story corpus with TRIBE's training data. We removed "unrelated dataset/different subject" throughout. The ridge is now described as fit separately but not on independent data (§3.2, §3.9, §4.2, App. C), and §3.2 now gives TRIBE v2's training data and modality dropout. Thank you for catching this.
Novelty relative to measured-fMRI work. We agree that distributed decodability (Haxby et al., 2001) and low dimensionality (Norman-Haignere et al., 2015) are established for measured fMRI. Both are now cited and discussed (§1 "What this adds", §2). Our object and inference differ. A predicted response is a noise-free function of stimulus features, so any regional difference in it reflects the encoder's readout, not cortex. We quantify rank relative to the encoder's own input (a 15–28× compression) and test random-set equivalence directly. The consequence for interpretation: in measured data, random-set equivalence would be evidence about cortical coding. In predictions, a decoding map supports neither a localization claim nor, without measured data, a claim of distributed coding.
Diagnostic or cautionary example? Primarily a cautionary case study of one out-of-domain setting, plus two inexpensive diagnostics. We now state explicitly that the diagnostics' sensitivity has not been validated with a positive control (§5.5). We also report the localization test's resolution in this setting: a best-parcel advantage of about 0.01 balanced accuracy over random sets would have been flagged (§4.4).
As for reconciling our result with validated regional findings from predictions (TRIBE v2's in-silico localizers; Ratan Murty et al., 2021): those findings are mostly univariate contrasts. With Y^=ZBT+E, the predicted contrast at vertex v is bvT(zˉA−zˉB). It follows the loadings and can be highly selective even when the rank is small. Low rank limits regional differences in decodability, not in mean response, so the two results do not conflict (§5.2).
Generalization and targeted validation. We agree that the analyses you propose would most directly change the assessment: validated held-out movie or speech stimuli, and ds003720 with genre labels. We could not run them in the discussion period. §5.4–5.5 now specify them, including the repeated presentations and matched processing you suggest for ds003720. Until they are run, we present the results as a case study of one regime (§5.4).