- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies brain encoding where the standard approach freezes a pretrained backbone and learns only a readout, because neural data is too scarce to tune the backbone. The authors challenge the "frozen" half of that recipe: they fine-tune large multimodal backbones with LoRA on the CNeuroMod/Algonauts 2025 benchmark. To make this affordable they extract features with one causally-masked forward pass instead of sliding windows, and later train hundreds of adapters on a shared backbone with a batched multi-LoRA kernel. Validation: (i) LoRA improves over the frozen backbone on all nine backbones tested; (ii) on the same backbone as the prior SoTA, LoRA + readout outperforms it on both in- and out-of-distribution test sets with far fewer trainable parameters, and full fine-tuning does not beat LoRA; (iii) Shapley attribution over input modalities shows the adapted model keeps a sensible modality-to-region organization; (iv) an ensemble of ~260 adapted models is submitted to the public post-challenge leaderboard.
Critical strengths
(1) The engineering makes an untestable question testable. Without the single-pass causal extraction and the multi-LoRA kernel, adapting backbones of this size on scarce neural data would not have been computable.
(2) The evaluation is more careful than the field's norm.
Critical weaknesses
(1) Little is conceptually new. LoRA, cross-attention readouts, ridge re-fitting, Shapley attribution, and quality-diversity ensemble analysis are all standard; the contribution is their systematic application to a new domain. The one genuinely new observation (adaptation reweights toward audio) is reported but not analyzed.
(2) Evidence is confined to one domain. All training and test data are naturalistic narrative audiovisual content; the transfer tests (held-out subject, NNDb) stay within that domain. The leaderboard-first claim is time-stamped and does not test generalization beyond it.
Questions for authors
-
Beyond ranking, what does the community learn from the 261-model ensemble that it could not learn from the 15-model one? If the answer is only the leaderboard position, is that a scientific contribution or a compute demonstration?
-
Everything is Friends + movies. What would break first at a genuinely different domain (e.g., rest, task, non-narrative stimuli) — the adapter, the readout, or the single-pass extraction?
-
Refer to "Critical Weaknesses"
4 comments
Thank you for the response. I’m not fully convinced that my main concerns have been addressed, and I’ll keep my original score.
Thank you very much for engaging with our rebuttal so quickly! We’d like to make sure we understand which concerns remain open, so that we can address them in the remaining discussion period.
Here is a brief recap of how we addressed each point:
- Methodological novelty: Making general methods scalable is very important [1]. Our main methodological contribution is making PEFT scalable for brain encoding and identifying three scaling directions (scale up/down/out), which required new engineering beyond applying PEFT to a new domain.
- New scientific observations: We use interpretability methods to show adaptation increases audio contribution and analyze it by brain region (gains are largest where audio is least used before adaptation, Fig. 13). We also report increased cross-modal mixing after adaptation (Sec. 5.4) and in silico localization results (General Response, #2).
- Domain coverage: We expand to spoken stories, silent 3s videos, and silent natural videos datasets. Our method generalizes to all of them (General Response, #1). We now show PEFT consistently improves encoding performance across 9 backbones, modality combinations, and datasets, achieving SoTA.
- Model ensemble: We make scaling out the number of adapted models affordable. It also makes per-subject or per-ROI adapters practical for future scientific comparisons.
We'd really appreciate it if you can point out where our response fall short. We’d be glad to run additional analyses in the remaining time. Thank you very much!
Dear reviewer, as the discussion period ends soon, we want to follow up on our previous message. We have uploaded a revised PDF with the new results. We would be grateful if you could let us know which concerns remain, so we can address them in the time left. Thank you again for your time!
We thank the reviewer for highlighting our careful evaluation and noting that our engineering makes an untestable question testable. We respond to the questions below.
On domain coverage
We thank the reviewer for this comment. Naturalistic audiovisual narrative film is a particularly difficult and important domain, and we achieve SoTA performance on it. Below, we have significantly expanded the train/test domains beyond narrative films, including silent 3s videos, longer natural videos, and spoken stories. This is broader in subjects, datasets, and modality combinations coverage than most prior encoding work.
We observe (see the General Response, New Result #1 for details):
Our result generalizes beyond narrative films to new domains, new subjects, new stimuli, and new modality combination on new sites and scanners.
On methodological novelty
General methods that can leverage scaling can often unlock new frontiers in AI and its applications [1]. With limited neural data, we argue that the best lever for scaling brain encoding models is to adapt large pretrained backbones with PEFT.
Our main methodological contribution is therefore to show how we can make PEFT scalable for brain encoding, and identify three concrete directions:
We hope the reviewer can appreciate making scaling feasible for brain encoding and showing it achieves SoTA performance are valuable contributions for this venue (ICLR).
On novel discoveries
We thank the reviewer for this comment and provide examples of how our approach enables novel scientific insights below.
A novel way of understanding what drives brain predictivity. Fine-tuning the backbone opens up a unique opportunity to understand what drives brain predictivity, by comparing the backbone before and after adaptation. Specifically, we show that adapting for brain encoding:
As the reviewer pointed out, these effects cannot be observed without fine-tuning the backbone. Future work can further apply mechanistic interpretability to the before/after comparison to understand what improves brain predictivity.
Our model can serve as an in silico proxy, enabling future large-scale in silico experiments that would be impractical in human subjects. As a proof of concept, we show our model reproduces diverse localization results using various modalities: the ventral visual category map, the language network using audio, and the Lerner temporal-receptive-window hierarchy (General Response, New Result #2).
On the ensemble
We show that we can cheaply scale out the number of adapted models, one axis along which PEFT leverages scale, made affordable by our multi-LoRA kernel. This is part of our main methodological contribution that makes PEFT scalable. In this work, we show that scaling out the number of adapted models can lead to a strong ensemble. However, as we wrote in the discussion (L476-480), the same scalability opens interesting directions for future work to train one LoRA per domain, dataset, subject or even ROI and compare them to gain further scientific insights.
We will update the manuscript to reflect these changes. We hope these results address your comments regarding methodological novelty, novel observations, and domain coverage. We hope you may consider a more positive evaluation of our work!
Reference
[1] Sutton, The Bitter Lesson, 2019.
[2] Naselaris et al., Encoding and decoding in fMRI, NeuroImage, 2011.
[3] Yamins and DiCarlo, Using goal-driven deep learning models to understand sensory cortex, Nature Neuroscience, 2016.
[4] Schrimpf et al., Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence, Neuron, 2020.