- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes “recovery profiles.” These measure how much of a reliable brain-response subspace a model’s predictions cover. It builds the reference from repeated fMRI trials and compares models against brain-to-brain differences. On NSD early visual areas, pretrained models score higher than random models with the same accuracy in 25 of 29 pairs.
Critical strengths
- The problem is interesting. Two models with the same accuracy may not capture the same brain signal.
- Using other subjects as a benchmark gives a natural scale for model differences.
- There are good robustness checks (e.g. NSD-synthetic, etc.)
Critical weaknesses
The main test compares only pretrained and random networks. Several previous papers already show that pretraining improves model–brain alignment, so this is an easy contrast. In higher visual areas, the method does not seem to separate trained models beyond brain-to-brain variability. In early visual areas, differences are larger but lack significance tests. These early regions are also tricky: alignment there may reflect low-level stimulus features rather than actual neural processing.
Questions for authors
- Can you test the method on harder contrasts, such as trained models with different architectures or objectives?
- In higher visual areas, the method does not seem to separate trained models beyond brain-to-brain variability. Would more images or repeats fix this, or is the method limited there?
- In early visual areas, can you add significance tests for differences between trained models? And can you show that alignment there reflects neural processing rather than low-level stimulus features?