acceptodds
Under review as a conference paper at ICLR 2027

Parameter-Efficient Fine-Tuning of Large Multimodal Models for Brain Encoding

Abstract

Brain encoding models aim to predict neural responses to stimuli. The most widely used approach fits readouts from frozen pretrained backbones. Using neural data to train large models from scratch or fully fine-tune them after task-pretraining has proved difficult because neural data are costly and limited. We show that parameter-efficient fine-tuning (PEFT) enables us to effectively adapt very large pretrained backbones for brain encoding with a small number of trainable parameters. To make it computationally practical, we exploit the backbone's causal structure to predict multiple timepoints in a single pass instead of using sliding windows, significantly reducing feature-extraction cost. This enables efficient fine-tuning of multimodal backbones with up to 104B parameters, the largest model adapted for brain encoding to our knowledge. Trained with a low-rank adapter (LoRA) and a lightweight readout, our approach consistently improves brain encoding performance across nine diverse multimodal backbones. On the same Qwen3-Omni backbone as the prior state-of-the-art (SoTA), our method BrainLoRA achieves the best published performance on the Algonauts 2025 Challenge in- and out-of-distribution, using only about as many trainable parameters. We also show the adapted multimodal backbones remain interpretable: modality attribution recovers the canonical sensory topography that prior work obtains by construction from separate unimodal encoders. To scale out the number of adapted models, we develop a multi-LoRA kernel that accelerates joint training of multiple LoRAs on a shared backbone. This enables an ensemble of 200+ adapted models that ranks first on the public Algonauts post-challenge leaderboard at the time of writing, also surpassing unpublished submissions. Together, our results highlight PEFT as an efficient and powerful paradigm for brain encoding with large pretrained models.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain encoding, parameter-efficient fine-tuning, LoRA, fMRI, brain foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yDasOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies brain encoding where the standard approach freezes a pretrained backbone and learns only a readout, because neural data is too scarce to tune the backbone. The authors challenge the "frozen" half of that recipe: they fine-tune large multimodal backbones with LoRA on the CNeuroMod/Algonauts 2025 benchmark. To make this affordable they extract features with one causally-masked forward pass instead of sliding windows, and later train hundreds of adapters on a shared backbone with a batched multi-LoRA kernel. Validation: (i) LoRA improves over the frozen backbone on all nine backbones tested; (ii) on the same backbone as the prior SoTA, LoRA + readout outperforms it on both in- and out-of-distribution test sets with far fewer trainable parameters, and full fine-tuning does not beat LoRA; (iii) Shapley attribution over input modalities shows the adapted model keeps a sensible modality-to-region organization; (iv) an ensemble of ~260 adapted models is submitted to the public post-challenge leaderboard.

Critical strengths

(1) The engineering makes an untestable question testable. Without the single-pass causal extraction and the multi-LoRA kernel, adapting backbones of this size on scarce neural data would not have been computable.

(2) The evaluation is more careful than the field's norm.

Critical weaknesses

(1) Little is conceptually new. LoRA, cross-attention readouts, ridge re-fitting, Shapley attribution, and quality-diversity ensemble analysis are all standard; the contribution is their systematic application to a new domain. The one genuinely new observation (adaptation reweights toward audio) is reported but not analyzed.

(2) Evidence is confined to one domain. All training and test data are naturalistic narrative audiovisual content; the transfer tests (held-out subject, NNDb) stay within that domain. The leaderboard-first claim is time-stamped and does not test generalization beyond it.

Questions for authors
  1. Beyond ranking, what does the community learn from the 261-model ensemble that it could not learn from the 15-model one? If the answer is only the leaderboard position, is that a scientific contribution or a compute demonstration?

  2. Everything is Friends + movies. What would break first at a genuinely different domain (e.g., rest, task, non-narrative stimuli) — the adapter, the readout, or the single-pass extraction?

  3. Refer to "Critical Weaknesses"

4 comments

We thank the reviewer for highlighting our careful evaluation and noting that our engineering makes an untestable question testable. We respond to the questions below.

On domain coverage

We thank the reviewer for this comment. Naturalistic audiovisual narrative film is a particularly difficult and important domain, and we achieve SoTA performance on it. Below, we have significantly expanded the train/test domains beyond narrative films, including silent 3s videos, longer natural videos, and spoken stories. This is broader in subjects, datasets, and modality combinations coverage than most prior encoding work.

datasetstimulusmodalities
trainAlgonauts 2025sitcom + filmsV+A+T
trainBOLD Moments3s short video (silent)V+T
trainLeBel 2023spoken storiesA+T
trainWen 2017natural video (silent)V
testNNDb 2020filmsV+A+T
testNarratives 2021spoken storiesA

We observe (see the General Response, New Result #1 for details):

  • Joint training matches or beats each single-dataset model on its own test data.
  • Joint training improves on held-out OOD datasets.

Our result generalizes beyond narrative films to new domains, new subjects, new stimuli, and new modality combination on new sites and scanners.

On methodological novelty

General methods that can leverage scaling can often unlock new frontiers in AI and its applications [1]. With limited neural data, we argue that the best lever for scaling brain encoding models is to adapt large pretrained backbones with PEFT.

Our main methodological contribution is therefore to show how we can make PEFT scalable for brain encoding, and identify three concrete directions:

  • Scale up: 9 backbones up to 104B, largest models ever adapted for brain encoding, made possible by replacing sliding windows with single causal forward passes
  • Scale down: 5% of the prior SoTA's trainable parameters with PEFT and better than full fine-tuning
  • Scale out: 200+ adapted models, made affordable by our multi-LoRA training kernel

We hope the reviewer can appreciate making scaling feasible for brain encoding and showing it achieves SoTA performance are valuable contributions for this venue (ICLR).

On novel discoveries

We thank the reviewer for this comment and provide examples of how our approach enables novel scientific insights below.

  1. A novel way of understanding what drives brain predictivity. Fine-tuning the backbone opens up a unique opportunity to understand what drives brain predictivity, by comparing the backbone before and after adaptation. Specifically, we show that adapting for brain encoding:

    • increases the audio contribution by 27% (Sec. 5.5, App. P). We analyze the increase by brain region and find that most increases are in regions that use audio less before adaptation (Fig. 13), consistent with the ablation result that audio-only input being the best (Fig. 12);
    • increases cross-modal mixing: the share of the prediction read at another modality's token positions nearly doubles, from 8.9% to 15.3% (Sec. 5.5, Fig. 4d, App. P).

    As the reviewer pointed out, these effects cannot be observed without fine-tuning the backbone. Future work can further apply mechanistic interpretability to the before/after comparison to understand what improves brain predictivity.

  2. Our model can serve as an in silico proxy, enabling future large-scale in silico experiments that would be impractical in human subjects. As a proof of concept, we show our model reproduces diverse localization results using various modalities: the ventral visual category map, the language network using audio, and the Lerner temporal-receptive-window hierarchy (General Response, New Result #2).

On the ensemble

We show that we can cheaply scale out the number of adapted models, one axis along which PEFT leverages scale, made affordable by our multi-LoRA kernel. This is part of our main methodological contribution that makes PEFT scalable. In this work, we show that scaling out the number of adapted models can lead to a strong ensemble. However, as we wrote in the discussion (L476-480), the same scalability opens interesting directions for future work to train one LoRA per domain, dataset, subject or even ROI and compare them to gain further scientific insights.

We will update the manuscript to reflect these changes. We hope these results address your comments regarding methodological novelty, novel observations, and domain coverage. We hope you may consider a more positive evaluation of our work!

Reference

[1] Sutton, The Bitter Lesson, 2019.

[2] Naselaris et al., Encoding and decoding in fMRI, NeuroImage, 2011.

[3] Yamins and DiCarlo, Using goal-driven deep learning models to understand sensory cortex, Nature Neuroscience, 2016.

[4] Schrimpf et al., Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence, Neuron, 2020.

Thank you for the response. I’m not fully convinced that my main concerns have been addressed, and I’ll keep my original score.

Thank you very much for engaging with our rebuttal so quickly! We’d like to make sure we understand which concerns remain open, so that we can address them in the remaining discussion period.

Here is a brief recap of how we addressed each point:

  • Methodological novelty: Making general methods scalable is very important [1]. Our main methodological contribution is making PEFT scalable for brain encoding and identifying three scaling directions (scale up/down/out), which required new engineering beyond applying PEFT to a new domain.
  • New scientific observations: We use interpretability methods to show adaptation increases audio contribution and analyze it by brain region (gains are largest where audio is least used before adaptation, Fig. 13). We also report increased cross-modal mixing after adaptation (Sec. 5.4) and in silico localization results (General Response, #2).
  • Domain coverage: We expand to spoken stories, silent 3s videos, and silent natural videos datasets. Our method generalizes to all of them (General Response, #1). We now show PEFT consistently improves encoding performance across 9 backbones, modality combinations, and datasets, achieving SoTA.
  • Model ensemble: We make scaling out the number of adapted models affordable. It also makes per-subject or per-ROI adapters practical for future scientific comparisons.

We'd really appreciate it if you can point out where our response fall short. We’d be glad to run additional analyses in the remaining time. Thank you very much!

Dear reviewer, as the discussion period ends soon, we want to follow up on our previous message. We have uploaded a revised PDF with the new results. We would be grateful if you could let us know which concerns remain, so we can address them in the time left. Thank you again for your time!

Reviewer_3QCpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies parameter-efficient fine-tuning of large multimodal models for brain encoding. It combines standard LoRA with single-pass feature extraction, lightweight readouts, and closed-form readout refitting. The authors evaluate the approach across nine multimodal backbones, report competitive results on the Algonauts 2025 benchmark, and examine transfer across subjects and datasets.

Critical strengths

The experimental evaluation is relatively comprehensive, covering multiple backbones, component ablations. These results provide useful empirical evidence for the practical effectiveness of parameter-efficient adaptation in large multimodal brain encoding models.

Critical weaknesses

1.The core methodological novelty is limited. The paper primarily uses standard LoRA in combination with existing feature extraction and readout fitting techniques. The method section focuses on model configurations and training procedures, without introducing a clearly novel adaptation mechanism or learning objective. Given that parameter-efficient fine-tuning using brain responses has already been explored, the incremental methodological contribution is limited.

2.The performance improvements are insufficient to establish the research contribution. The manuscript places considerable emphasis on competition results, large-model adaptation, and the effectiveness of model ensembles. These results demonstrate the system’s practical utility but primarily reflect the value of scaling and engineering implementation. Although the experiments are relatively comprehensive, the paper does not sufficiently articulate important scientific findings that advance existing understanding and compensate for the limited methodological novelty.

Questions for authors

Please see the concerns raised in the critical weaknesses section above.

3 comments

We thank the reviewer for finding the evaluation comprehensive across backbones and ablations, and for seeing it as useful evidence that PEFT works for large multimodal brain encoding models. We respond to the comments below.

On novelty

General methods that can leverage scaling can often unlock new frontiers in AI and its applications [1]. With limited neural data, we argue that the best lever for scaling brain encoding models is to adapt large pretrained backbones with PEFT.

Our main methodological contribution is therefore to show how to make PEFT scalable for brain encoding, and identify three concrete directions:

  • Scale up: 9 backbones up to 104B, largest models ever adapted for brain encoding, made possible by causal forward passes
  • Scale down: 5% of the prior SoTA's trainable parameters with PEFT and better than full fine-tuning
  • Scale out: 200+ adapted models, made affordable by our multi-LoRA training kernel

We hope the reviewer will consider reevaluating the novelty in light of this argument, and agree that making scaling feasible for brain encoding and showing it achieves SoTA performance are valuable contributions for this venue (ICLR).

Scientific contribution

We thank the reviewer for this comment and below we provide some examples of how our approach enables scientific insights.

  1. Building the best multimodal whole-brain fMRI encoding model is itself a central scientific goal [2,3,4]. Ours is the best-performing multimodal whole-brain fMRI model, and it generalizes across subjects, domains, and modality combinations (App. M, App. N; General Response, New Result #1).

  2. A novel way of understanding what drives brain predictivity. Fine-tuning the backbone opens up a unique opportunity to understand what drives brain predictivity, by comparing the backbone before and after adaptation. Specifically, we show that adapting for brain encoding:

    • increases the audio contribution by 27%, mostly in regions that use audio least before adaptation (Sec. 5.5, App. P);
    • increases cross-modal mixing: the share of the prediction read at another modality's token positions nearly doubles, from 8.9% to 15.3% (Sec. 5.5, Fig. 4d, App. P).

    Neither effect can be observed without fine-tuning the backbone. Future work can further apply mechanistic interpretability to the before/after comparison to understand what improves brain predictivity.

  3. Our model can serve as an in silico proxy, enabling future large-scale in silico experiments that would be impractical in human subjects. As a proof of concept, we show our model reproduces diverse localization results over various modalities:

    • Ventral visual categories (fLoc, Stigliani 2015): scored against the human category-selective regions of Julian et al. (2012). Places>others selects PPA (AUC 0.94) but not OFA (0.08), while faces>others selects OFA (0.76–0.88) but not TOS (0.04), suggesting clear double dissociations. It also picks out RSC and TOS for places (0.84–0.88), EBA for bodies (0.95) and LOC for objects (0.91).
    • Language network (Scott 2017): intact>degraded speech recovers the probabilistic language atlas (LanA, AUC 0.87–0.91), which includes left temporal cortex plus left IFG and MFG.
    • Temporal receptive windows (Lerner 2011): the same story segment is played after different amounts of its real preceding context. Context sensitivity rises from auditory cortex (0.24–0.32) through STS (0.42–0.47) to TPJ, PFC and precuneus (0.68–0.77), matching the human hierarchy.
    • Effect of the adapter: characters>others finds the visual word form area (VWFA) much better after adaptation (AUC 0.84 vs 0.54 for the frozen backbone).

We will revise the paper to reflect these changes. We hope these results address your comment regarding scientific contributions, and that you may consider a more positive evaluation of our work!

References

[1] R. Sutton, The Bitter Lesson, 2019.

[2] Naselaris et al., Encoding and decoding in fMRI, NeuroImage, 2011.

[3] Yamins and DiCarlo, Using goal-driven deep learning models to understand sensory cortex, Nature Neuroscience, 2016.

[4] Schrimpf et al., Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence, Neuron, 2020.

[5] Stigliani et al., Temporal processing capacity in high-level visual cortex is domain specific, Journal of Neuroscience, 2015.

[6] Julian et al., An algorithmic method for functionally defining regions of interest in the ventral visual pathway, NeuroImage, 2012.

[7] Scott et al., A new fun and robust version of an fMRI localizer for the frontotemporal language system, Cognitive Neuroscience, 2017.

[8] Lipkin et al., Probabilistic atlas for the language network based on precision fMRI data from >800 individuals, Scientific Data, 2022.

[9] Lerner et al., Topographic mapping of a hierarchy of temporal receptive windows using a narrated story, Journal of Neuroscience, 2011.

Thanks to the authors for their clarifications. However, I do not find the contribution and generalizability of this work sufficiently compelling. Accordingly, I keep my original score.

Thank you very much for engaging with our rebuttal! We've uploaded a revised PDF that incorporates the new results. We’d like to understand which concerns remain open so we can address them in the remaining discussion period.

Here is a brief recap of how we addressed each point:

  • Methodological novelty: Making general methods scalable is very important [1]. Our main methodological contribution is making PEFT scalable for brain encoding and identifying three scaling directions (scale up/down/out), which required new engineering beyond applying PEFT to a new domain.
  • New scientific observations: We use interpretability methods to show adaptation increases audio contribution and analyze it by brain region (gains are largest where audio is least used before adaptation, Fig. 13). We also report increased cross-modal mixing after adaptation (Sec. 5.4) and in silico localization results (General Response, #2).
  • Generalizability: We expand to spoken stories, silent 3s videos, and silent natural videos datasets. Our method generalizes to all of them (General Response, #1). We now show PEFT consistently improves encoding performance across 9 backbones, modality combinations, and datasets, achieving SoTA.

We'd really appreciate it if you can point out where our response falls short. We’d be glad to run additional analyses in the remaining time. Thank you very much!

Reviewer_omFAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether parameter-efficient fine-tuning can improve whole-brain fMRI prediction by adapting large pretrained multimodal models to neural data. The proposed approach, BRAINLORA, jointly trains low-rank adapters in the backbone’s attention modules and a lightweight readout that maps video, audio, and transcript representations to responses in 1,000 cortical parcels. To make adaptation computationally practical, the method uses the decoder’s causal structure to extract representations for multiple time points in a single forward pass, replacing repeated sliding-window computation. Joint training is followed by a closed-form ridge refit of the final linear readout. A cross-attention readout variant accesses a richer set of backbone tokens.

The evaluation uses the CNeuroMod/Algonauts 2025 benchmark, with four participants and held-out tests on both Friends episodes and out-of-distribution movies. The authors report improvements from LoRA across nine multimodal backbones spanning approximately 4.7B–104B parameters. Their main Qwen3-Omni model uses approximately 61M trainable parameters and achieves reported mean Pearson correlations of 0.327 and 0.228 on the in-distribution and out-of-distribution tests, respectively. Experiments also compare alternative PEFT methods, full fine-tuning, readout architectures, and input-context designs. Additional analyses examine transfer to participants excluded from adapter training and to a second neuroimaging dataset.

The paper further introduces a batched multi-LoRA training implementation and constructs a 261-member ensemble, reporting correlations of approximately 0.343 and 0.248 on the two held-out benchmarks. Exact modality-subset Shapley analyses recover expected sensory topography and characterize changes in the model’s reliance on audio, video, and text after adaptation. The contribution is primarily empirical and computational: an integrated framework for adapting large multimodal backbones to brain encoding, supported by extensive comparisons, ablations, and transfer analyses.

Critical strengths

1- The strongest result is that LoRA improves encoding performance across all nine tested multimodal backbones, spanning approximately 4.7B–104B parameters. The reported gains over matched frozen-backbone models range from approximately 0.032 to 0.066 in mean Pearson correlation. For the principal Qwen3-Omni configuration, the improvement from 0.2907 to 0.3263 is substantial relative to the reported seed variability. This consistency supports the central premise that neural supervision can improve pretrained stimulus representations beyond what readout training alone achieves.

2- The combination of multi-timepoint feature extraction, low-rank adaptation, temporal readouts, and closed-form refitting makes the approach operationally useful. The main models train approximately 61M parameters—about 5.5% of the quoted MIRAGE readout’s parameter count—while achieving competitive or better reported held-out accuracy. Although the individual components are established, their integration addresses a concrete obstacle to adapting large multimodal models with limited neural data.

3- The frozen and adapted backbone comparisons preserve readout and projection training, step budgets, and the closed-form refit, helping isolate the contribution of backbone adaptation. The paper also examines alternative PEFT methods, adapter rank, context length, readout architecture, and token access. Particularly useful are the findings that closed-form refitting contributes materially to performance and that cross-attention’s benefit primarily reflects access to richer token representations. These experiments explain which design choices matter, rather than presenting only an optimized final system.

4- The authors explicitly acknowledge that extensive development on season 6 introduces selection bias. They nevertheless take meaningful precautions: ensemble weights are cross-fitted by episode, subset selection is nested, and ridge penalties are selected using training data rather than the validation season. Repeated-seed evaluations for the principal configurations and separate held-out ID/OOD results provide additional support. These safeguards make the evaluation more credible than an ensemble assessed on the same observations used to fit its combination weights.

5- Leave-one-subject-out adapter training retains approximately 97% of the improvement obtained with the four-subject adapter, after fitting the held-out participant’s readout. The NNDb experiment is particularly valuable because it transfers the adapted model and readout without additional fitting on that dataset, improving mean correlation from 0.0991 to 0.1223 and benefiting all 20 evaluated participants. This supports the interpretation that adaptation learns useful stimulus representations rather than only accommodating the original participants or acquisition setting.

6- The shifted-target experiment tests whether the adaptation benefit depends on correct stimulus–response alignment, rather than generic properties of fMRI supervision. The modality analysis enumerates all eight input subsets and checks robustness to alternative replacement baselines and independent seeds, providing a substantive account of the model’s changing modality dependence. Reporting unsuccessful modifications—including certain readout, rank, and training-data variations—also improves the scientific value of the study by clarifying the limits of the proposed recipe.

Critical weaknesses
  1. Appendix S reports full fine-tuning of 20.25B parameters using AdamW on a single A800-80GB. A conventional GPU-resident implementation would require approximately 162 GB for weights, gradients, and two optimizer moment buffers even if all four were stored in bf16, before accounting for activations and temporary storage. Gradient checkpointing does not resolve this optimizer-state requirement. Offloading, sharding, or another storage mechanism could explain the configuration, but none is documented. The missing details weaken the reproducibility of an important baseline and should be resolved with explicit tensor placement, precision, memory usage, and timing information.

  2. Figure 7 assigns an ensemble NN times the full cost of one member, including frozen-backbone feature extraction. However, MIRAGE’s published pipeline caches frozen features for reuse across readout training runs. For 15 members sharing that cache, the paper’s own timings imply 700.6+15×1.88=728.8700.6+15\times1.88=728.8 GPU-hours, rather than approximately 10,537 GPU-hours. The exact correction depends on the ensemble’s distinct feature configurations, which should be documented. This accounting can substantially exaggerate the ensemble efficiency advantage and requires correction, independently of the single-model speed comparison.

  3. The reported 29.8× acceleration uses sequence length 128, whereas the brain-encoding pipeline processes approximately 15,000–19,000 tokens per input. The short-sequence result demonstrates improved utilization, but does not establish the practical acceleration achievable with realistic activation-memory requirements and full training windows. The authors acknowledge smaller end-to-end gains without quantifying them adequately. Full-window throughput, feasible adapter counts, peak memory, and comparisons against an appropriate grouped-matmul implementation are needed to substantiate the systems contribution.

  4. Appendix F attributes full-span/prefix feature discrepancies—audio cosines around 0.956–0.958—to bf16 numerics and expert routing, but the presented controls do not isolate that mechanism. A matched-length intervention changing only future content, together with higher-precision or controlled-routing tests, would provide stronger evidence. Separately, the definition cj=9+2.98jc_j=9+2.98j appears to place the first lag block approximately 2–4 seconds after nominal TR jj if stimulus and target clocks share an origin. This may reflect a valid preprocessing convention, but the absolute timestamp mapping is not explicit enough to verify. These are unresolved methodological questions rather than demonstrated leakage.

  5. The minimum-variance weight formula is correct for the stated sum-to-one constraint, but the claim that this constraint is “free” because Pearson correlation is invariant to positive rescaling is not generally valid. Useful coefficient vectors can sum to zero or a negative value, and constrained residual-variance minimization is not equivalent to maximizing correlation. Consequently, the combiner should be presented as a particular optimization choice rather than as a generally unrestricted or Pearson-optimal solution. This does not invalidate its reported out-of-fold improvement, but the mathematical rationale requires correction.

  6. The corrected intersubject-correlation ceiling assumes a homogeneous shared-signal model with independent noise. Unequal participant reliability and subject-specific stimulus responses can invalidate its interpretation as a general encoding ceiling, making comparisons of normalized gains across datasets conditional on unverified assumptions. Appendix Q also conflates within-subject explained variance with cross-subject prediction covariance: a 97% correlation-to-ceiling ratio squares to approximately 94%, whereas the reported 66% describes a different covariance statistic. The raw transfer gains remain informative, but stronger claims about equivalent adaptation benefits and remaining stimulus-driven variance need more careful definitions and sensitivity analyses.

  7. The observed 0.00077 seed spread for one configuration is repeatedly used as a resolution threshold for other methods, although their training variability may differ. Parcel-block intervals also quantify a different source of variation from uncertainty across participants, films, or training seeds. The functional-block construction and resampling procedure need fuller specification. Furthermore, episode-level cross-fitting protects ensemble fitting conditional on the selected model pool; it does not remove the earlier adaptive use of season 6 to design that pool. These limitations chiefly affect small performance differences and claims of superiority between closely matched configurations.

  8. Table 13 lists shifted-LoRA and shifted-frozen scores of 0.1874 and 0.2617, whose difference is −0.0743 rather than the reported −0.0733. Likewise, shifted versus aligned frozen performance differs by −0.0290, not −0.0299. Appendix D’s ridge-fit and holdout counts total 151,200 rows per subject, while the stated 1,429 windows of 100 TRs imply 142,900 rows. These discrepancies may reflect stale table entries or different eligibility masks, but they exceed rounding and require reconciliation against the underlying run artifacts.

Questions for authors

Questions for the Authors

  1. How was full fine-tuning implemented within the reported hardware budget?
    Appendix S specifies a single NVIDIA A800-80GB GPU, bf16 backbone weights, and AdamW, while the full fine-tuning baseline updates approximately 20.25 billion parameters. A conventional GPU-resident implementation would require substantially more memory once gradients and optimizer states are included. Could you provide the actual device count, optimizer-state precision and placement, any offloading or sharding, and peak GPU and host memory? Please also clarify the retained parameter footprint and execution configuration for the 104B backbone. These details are necessary to reproduce the experiments and assess whether the efficiency comparisons use equivalent resource accounting.

  2. Can you clarify the exact temporal alignment and the causality guarantees of the complete prediction pipeline?
    Please provide an explicit mapping between each fMRI target’s acquisition time, the video and audio timestamps, the selected lag windows, and any hemodynamic shift applied to the targets. Under a shared clock origin, the documented indexing rule cj=9+2.98jc_j=9+2.98j and first lag window appear to select nominal frame times approximately 2–4 seconds after TR jj. Is this explained by a target shift or another indexing convention? Separately, the full-sequence versus strict-prefix comparison produces nonidentical audio representations. Could a test that holds sequence length fixed while changing only future content distinguish future-content dependence from numerical and routing effects? Please state how the bidirectional audio chunks affect the temporal guarantee.

  3. How should ensemble costs be calculated when members share frozen features?
    The ensemble cost calculation appears to multiply each member’s complete single-model cost, including frozen feature extraction. Are these features recomputed for every member, or shared across seeds and readout variants? Using the tabulated costs, a 15-member ensemble sharing one feature cache would require approximately 700.6+15×1.88=728.8700.6+15\times1.88=728.8 GPU-hours, compared with 15×(700.6+1.88)=10,537.215\times(700.6+1.88)=10{,}537.2 GPU-hours if extraction is charged separately to every member. Could you provide the member-to-cache mapping and revise the comparison accordingly? A breakdown of feature extraction, member training, model selection, and inference would make clear which efficiency advantages persist under the appropriate amortization.

  4. What speedup does the multi-LoRA implementation achieve under the actual training configuration?
    The reported maximum kernel speedup is measured at sequence length 128, whereas the main experiments use approximately 15,000–19,000 tokens. Could you report a representative benchmark at the production sequence length, including the feasible number of simultaneous adapters, peak memory, and complete forward, backward, and optimizer-step time? Please also describe how numerical agreement with independently trained adapters was verified. This would establish how much the kernel improvement contributes to the reported end-to-end training efficiency.

  5. What optimality claim is intended for the ensemble weighting rule?
    The residual-covariance solution minimizes residual mean squared error subject to 1⊤w=1\mathbf{1}^{\top}w=1. However, positive-scale invariance of Pearson correlation does not make this constraint unrestricted: useful combinations can have zero or negative weight sums, and minimizing constrained residual variance does not generally maximize correlation. Under what additional assumptions is the manuscript’s justification valid? Could you either establish those assumptions or revise the interpretation to match the objective actually optimized? A comparison with an out-of-fold regression-based stacker would also help determine whether the distinction affects the empirical conclusions.

  6. How robust are the noise-ceiling normalization and shared-residual interpretations to their assumptions?
    The ceiling inversion assumes homogeneous participant reliability and independent noise. How sensitive are the normalized cross-dataset comparisons to unequal reliability, uncertainty in the ceiling estimates, and parcels with small ceiling values? Please retain the raw correlation improvements alongside any normalized results. Could you also define the numerators, denominators, and averaging procedures behind the approximately 97%-of-ceiling correlation and 66% shared-covariance figures? Finally, what evidence distinguishes shared residual neural responses to narrative content from shared physiological, acquisition, or preprocessing effects?

  7. What sources of uncertainty and model selection are covered by the reported statistical analyses?
    Could you specify the functional-block definitions, resampling procedure, number of bootstrap replicates, and treatment of participants in the confidence intervals? How broadly is the reported 0.00077 seed variation intended to apply beyond the particular training recipe used to estimate it? Please also distinguish the protection provided by out-of-fold ensemble fitting from the earlier use of the full S6 validation set to develop the candidate pool, and identify which conclusions are supported by the untouched ID and OOD evaluations. For the NNDb readout-refitting experiment, were temporal gaps or other safeguards used between contiguous folds to limit dependence across their boundaries?

1 comment

We thank the reviewer for the careful review, and for recognizing that LoRA improves all nine backbones, that the matched frozen-vs-adapted controls isolate the adapter's contribution, and that the gains transfer to held-out subjects and to new datasets. We address the questions below.

Q1. Hardware. We use only one A800 for Full FT, by putting bf16 weights on the GPU while keeping gradients in CPU memory. Optimizer runs on the CPU with fp32 precision. Peak GPU memory usage is 61.5GB. The 104B backbone is truncated at layer 24 and split over 3 A800s. We will add these details to App. S.

Q2. Alignment and causality. We follow MIRAGE’s alignment. Each window’s media starts 4.47 s before its first TR. We do not claim strict causality, and none of our conclusions depend on it. Our control experiment shows that re-encoding strict prefixes gives highly similar representations (App. F, Tab. 9). Compared with MIRAGE, which can see into future stimuli for up to 145s, our model is far closer to being causal.

Q3. Ensemble cost. The 15-member MIRAGE ensemble can share cached features and thus costs around 729 GPU-h. Our adapted features cannot be shared, so ours cost around 100 GPU-h (15x6.51 hrs). Our advantage is about 7x. We will revise Fig. 7b in App. C.

Q4. Kernel. At the production length (19.8k tokens), fusing 2 or 4 adapters is 1.25× or 1.42× faster than best prior kernel. This effect can be stronger on more powerful GPUs. We will update the paper accordingly.

Q5. Ensemble weighting. The reviewer is correct that minimum variance does not maximize correlation. We will drop the optimality claim from App. S. Note that none of our results rely on this claim.

Q6. Noise ceiling. No conclusion depends on the ceiling: every headline result is raw r, and we already keep raw r beside every normalized value (e.g. NNDb +0.0338 raw, Tab. 15). The 97% is mean r divided by the mean correlation ceiling sqrt(ISC/var) across parcels; the 66% is the share of the ISC that the predictions reproduce across subjects (Tab. 17). Scanner noise cannot correlate across separately scanned subjects, but stimulus-evoked physiology could, so we will update the terms to call it the stimulus-relevant residual.

Q7. Statistics. Bootstrap blocks are 100 k-means clusters of parcels by functional connectivity. The 0.00077 seed spread applies to the LoRA+linear recipe only (L229-230). Model selection used S6 only, while S7 and the OOD films were never used for selection, and all main gains are very large relative to the seed spread and the bootstrap CIs.

We thank the reviewer for the detailed questions, and hope our responses have addressed them all. We will update the manuscript to reflect these clarifications.

Comments on the paper

1 comment

We thank all reviewers for their comments and the AC for overseeing our submission!

We appreciate the reviewers for recognizing that adaptation consistently improves all nine backbones (omFA), that our evaluation is "comprehensive" (3QCp) and "more careful than the field's norm" (yDas), with careful controls that isolate the adapter's contribution and show which design choices matter (omFA).

We include two new results in response to the reviewers' feedback.

New Result #1: expanding train/test domains

We significantly expand the train/test domains beyond narrative films: from the 2 narrative film datasets in the submission to 6 diverse datasets here.

datasetsubjectsstimulusmodalities
trainAlgonauts 20254sitcom + filmsV+A+T
trainBOLD Moments103s short video (silent)V+T
trainLeBel 202310spoken storiesA+T
trainWen 20173natural video (silent)V
testNNDb 202086filmsV+A+T
testNarratives 2021258spoken storiesA

Performance on held-out test data:

trained onAlgonautsBOLD MomentsLeBelWenNNDbNarratives
Algonauts only0.3270.1720.3450.2620.0930.079
BOLD Moments only0.2840.1790.3230.2620.0600.067
LeBel only0.2990.1650.3560.2270.0700.078
Wen only0.2910.1680.3290.2440.0640.069
joint (4 datasets)0.3260.1790.3720.2760.0970.082

We observe:

  • Joint training matches or beats each single-dataset model on its own test data.
  • Joint training improves on held-out OOD datasets.

In short, this suggests a scaling direction along data size as well: one adapter, trained on 27 subjects across 4 datasets and three modality combinations, transfers to new subjects, new stimuli, and new modality combination on new sites and scanners.

This further shows the methodological value of making PEFT scalable for brain encoding as it enables training on diverse datasets from different domains, subjects, scanners, and protocols.

New Result #2: in silico localization

We apply standard human fMRI localizer experiments on our main BrainLoRA model, and compare its predicted localization maps with human maps.

  • Ventral visual categories (fLoc, Stigliani 2015): scored against the human category-selective regions of Julian et al. (2012). Places>others selects PPA (AUC 0.94) but not OFA (0.08), while faces>others selects OFA (0.76–0.88) but not TOS (0.04), suggesting clear double dissociations. It also picks out RSC and TOS for places (0.84–0.88), EBA for bodies (0.95) and LOC for objects (0.91).
  • Language network (EvLab listening localizer, Scott 2017): intact>degraded speech recovers the probabilistic language atlas (LanA, AUC 0.87–0.91), which includes left temporal cortex plus left IFG and MFG.
  • Temporal receptive windows (Lerner 2011): the same story segment is played after different amounts of its real preceding context. Context sensitivity rises from auditory cortex (0.24–0.32) through STS (0.42–0.47) to TPJ, PFC and precuneus (0.68–0.77), matching the human hierarchy.
  • Effect of the adapter: characters>others finds the visual word form area (VWFA) much better after adaptation (AUC 0.84 vs 0.54 for the frozen backbone).

A model trained only on naturalistic movies reproduces the canonical results of vision, language, and temporal-integration localizers. This validates our model as an in silico proxy: hypotheses and experiments can be screened on the model before testing on humans, and the VWFA result shows that adaptation sharpens the model’s functional organization toward the brain’s.

These results highlight how our trained encoding models can contribute to obtaining new neuroscientific insights.

We thank all the reviewers for their constructive feedback. We will revise the paper to reflect the responses to their comments and the new results. We believe these changes has made the presentation of our work stronger.

Reference

  • Stigliani et al., Temporal processing capacity in high-level visual cortex is domain specific, Journal of Neuroscience, 2015.
  • Julian et al., An algorithmic method for functionally defining regions of interest in the ventral visual pathway, NeuroImage, 2012.
  • Scott et al., A new fun and robust version of an fMRI localizer for the frontotemporal language system, Cognitive Neuroscience, 2017.
  • Lipkin et al., Probabilistic atlas for the language network based on precision fMRI data from >800 individuals, Scientific Data, 2022.
  • Lerner et al., Topographic mapping of a hierarchy of temporal receptive windows using a narrated story, Journal of Neuroscience, 2011.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.