acceptodds
Under review as a conference paper at ICLR 2027

BrainWideBench: Benchmarking large-scale pretraining and across-animal transfer in multi-region neural recordings

54 views

Abstract

Advances in large-scale neural recording have made it possible to collect data across many animals and distributed brain regions, raising the question of whether this scale can be exploited to learn general-purpose neural representations transferable across diverse downstream tasks. Yet, progress toward this goal has been limited by fragmented evaluation protocols and a narrow focus on individual task domains. Here, we present BrainWideBench, a benchmark for evaluating across-animal transfer on multi-region neural recordings, built on the International Brain Laboratory Brainwide Map dataset of neural and behavioral recordings spanning 276 brain regions from 139 mice performing a sensory-guided decision-making task. The benchmark is organized around three complementary task suites that evaluate whether learned representations support downstream decoding of behavior, can predict masked or future neural activity, and can recover biologically meaningful anatomical organization. With this benchmark, we systematically evaluate pretraining methods across transfer settings, including finetuning on downstream objectives and zero-shot generalization to unseen animals. Our results confirm pretraining improves performance over matched single-session baselines, but we show current methods exhibit heterogeneity in transfer capabilities: gains depend strongly on the alignment between pretraining objectives and downstream tasks. No single approach performs uniformly well across all three suites, and most methods are designed to only address a subset of them. Together, these findings suggest that learning representations that jointly generalize across behavior, dynamics, and anatomy remains an open challenge. By providing a unified and reproducible evaluation suite, BrainWideBench establishes a framework for measuring progress toward general-purpose models of the mouse brain.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Computational Neuroscience, NeuroAI, Behavioral Decoding, Neural Activity Prediction, Brain Region Identification

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_TbEBOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

Currently, neural activity and behavioral prediction usually require specific decoders for different recording sessions/animals/prediction targets. The state-of-the-art large-scale neural recording technology may make it possible to develop a single model that generalizes across animals/sessions/multiple downstream tasks. The authors introduce BrainWideBench, a benchmark to evaluate whether the learned large-scale neural representations are transferable across diverse tasks using a dataset from IBL. The authors standardize and filter neural activity and behavioral measurements from 139 mice across 276 brain regions, establish quality-control procedures, and define a pretraining set and an evaluation set of 13 held-out mice. The benchmark comprises three task suites: behavioral measurement prediction, neural activity prediction and anatomical region classification. The authors evaluate a range of existing models, including transformer-based, state-space, and self-supervised approaches. Their results show that pretraining generally improves downstream performance. Although, the benchmark is designed to investigate whether learned neural representations can transfer across diverse downstream tasks, the authors mainly focus on across-animal transfer and establishing baseline performance, with limited evaluation of across-task generalization. Overall, the paper provides a useful benchmarking framework while highlighting the limitations of current models in learning general-purpose neural representations that transfer across diverse tasks.

Critical strengths

Very large workload. The paper addresses an important gap by providing a benchmark for evaluating pretrained neural models across sessions/animals/prediction targets. Three task suites cover different aspects of neural representations, including behavior measurement, neural dynamics, and anatomy. The authors put considerable effort in data preprocessing and quality control. In addition, the authors also evaluate a bunch of existing models and provide useful reference for future studies.

Critical weaknesses

I don’t think anatomical region prediction is well-justified downstream task. instead of it may help for decoding behavior or neuron activity. Anatomy may be used to help behavior or neural activity prediction, it is unclear why predicting the brain region from which a neuron was recorded should be considered a core objective. From both neuroscience and engineering perspectives, I do not see a strong motivation for this. The authors should better justify its relevance to evaluating general-purpose neural representations.

Questions for authors
  1. The manuscript is sometimes difficult to follow. In the abstract, the term "task" is used to refer both to the behavioral task performed by the animals and to the prediction tasks defined by the benchmark. This is confusing, particularly given the limited background provided in the abstract. Also, I thought the generalization across tasks indicates the generalization across the three task suites. But much of the evaluation appears to focus across-animal or across tasks within each suite. The authors should clarify which types of generalization the benchmark is primarily designed to evaluate in the introduction. 2.Page 5, Lines 224–227: “For each decoding task, the target is a 1 second window sampled around key trial events” is unclear.
5 comments

We thank the reviewer for their constructive feedback and careful assessment of our work. We appreciate their recognition that BrainWideBench addresses an important gap in evaluating pretrained neural models across animals and tasks, as well as their acknowledgment of our extensive data preprocessing, quality control, and baseline evaluations. Below, we further motivate anatomical identification as a complementary evaluation task, clarify the scope of across-animal transfer and cross-task generalization, and address ambiguities in our task definitions and presentation. We hope these clarifications address the reviewer's concerns and better communicate the motivation and contributions of our benchmark.

1. Motivation for TS3 Anatomical Identification (W1)

“I don’t think anatomical region prediction is well-justified downstream task. instead of it may help for decoding behavior or neuron activity. Anatomy may be used to help behavior or neural activity prediction, it is unclear why predicting the brain region from which a neuron was recorded should be considered a core objective. From both neuroscience and engineering perspectives, I do not see a strong motivation for this. The authors should better justify its relevance to evaluating general-purpose neural representations.”

We thank the reviewer for raising this point and welcome the opportunity to elaborate on the motivation for TS3, discussed in Section 3.3. While anatomical information can indeed support behavioral decoding and neural activity prediction, anatomical identification is also a meaningful downstream objective in its own right, both for its scientific utility and for evaluating general-purpose neural representations.

Anatomical localization is fundamental to interpreting neural recordings, as understanding how the brain processes information requires knowing which regions and circuits the recorded activity originates from. Studies investigating the neural basis of perception, decision-making, and behavior routinely rely on anatomical information to characterize the contributions of different brain regions and their interactions [1–3]. Although experimenters target specific regions during probe insertion, this does not establish the precise anatomical locations of recorded neurons [8]. Accurate localization typically requires costly postmortem histology and expert annotation, limiting precise localization to retrospective analyses and introducing potential annotation variability [4,5]. As discussed in Section 3.3, inferring anatomical identity directly from neural activity could complement these procedures, enabling in vivo localization and validation of anatomical assignments.

Beyond its practical utility, anatomical identification evaluates a complementary capability of general-purpose neural representations. While TS1 and TS2 assess whether representations capture behaviorally relevant information and neural population dynamics, TS3 tests whether they preserve biologically meaningful anatomical structure across animals. This builds on an emerging direction in large-scale neural representation learning, where studies such as NEDS [6] and POCO [7] have investigated the emergence of anatomical organization in learned representations without explicit anatomical supervision. TS3 provides a standardized, quality-controlled protocol for evaluating anatomical prediction in held-out animals and comparing recent methods for extracting neuron-level representations.

Together, these considerations establish the scientific relevance of anatomical identification and its value as a complementary evaluation axis for general-purpose neural representations. We have further clarified this motivation in Section 3.3 of the revised manuscript.

2. Clarifications on terminology, generalization, and task definitions (Q1)

2.1. Distinguishing the behavioral paradigm from benchmark tasks

“In the abstract, the term "task" is used to refer both to the behavioral task performed by the animals and to the prediction tasks defined by the benchmark. This is confusing, particularly given the limited background provided in the abstract.”

We thank the reviewer for pointing out this ambiguity. We now reserve "task" for the benchmark's prediction tasks and task suites, and describe the animals' experimental paradigm as sensory-guided decision-making behavior. Accordingly, in the abstract, "mice performing a sensory-guided decision-making task" now reads "mice engaged in sensory-guided decision-making behavior", with corresponding changes throughout the manuscript (e.g., Sections 2 and 3.1).

2.2. Scope of generalization

“I thought the generalization across tasks indicates the generalization across the three task suites. But much of the evaluation appears to focus across-animal or across tasks within each suite.”

We thank the reviewer for highlighting this distinction. BrainWideBench evaluates two complementary forms of generalization: across-animal transfer, through evaluation on held-out mice, and cross-task generalization, by evaluating pretrained models on downstream prediction objectives that may differ from their pretraining objectives. The latter includes transfer across task suites. For example, NDT-Stitch and MtM, pretrained on neural prediction objectives, are evaluated on behavioral decoding (TS1) and brain region identification (TS3), while POYO+ and POSSM, pretrained for behavioral decoding, are also evaluated on TS3.

“Although, the benchmark is designed to investigate whether learned neural representations can transfer across diverse downstream tasks, the authors mainly focus on across-animal transfer and establishing baseline performance, with limited evaluation of across-task generalization.”

We acknowledge that the current baseline study does not exhaustively characterize transfer across all combinations of task suites, particularly as existing methods are often designed for specific objectives. Our evaluations nevertheless provide an initial characterization of cross-task generalization, while the benchmark establishes a standardized framework for more systematic comparisons. We have revised the introduction to clarify these two dimensions of generalization and the scope of the current evaluation.

2.3. Behavioral decoding windows

“Page 5, Lines 224–227: “For each decoding task, the target is a 1 second window sampled around key trial events” is unclear.”

We thank the reviewer for pointing out this ambiguity. We have revised the description in Section 3.1 to clarify that the 1-second target windows are defined relative to events in the animal's decision-making trial, with their timing depending on the variable being decoded. We also provide concrete examples to illustrate how these windows are used for different prediction targets. The revised text reads:

"For each decoding task, we define a 1-second target window aligned to a relevant event in the decision-making trial (stimulus onset, movement onset, or feedback onset). The timing of these windows depends on the variable being decoded. For example, choice is predicted from neural activity preceding movement onset, whereas licking rate is predicted over the second following feedback. These windows are selected to capture informative neural signals while minimizing confounds from other trial events (see Figure A10)."

The precise definitions of these target windows and their rationale are provided in Appendix D.1 (Table A4 and Figure A10).


References:

[1] Siegle, Joshua H., et al. “Survey of Spiking in the Mouse Visual System Reveals Functional Hierarchy.” Nature, vol. 592, no. 7852, 2021, pp. 86–92.

[2] Steinmetz, Nicholas A., et al. “Distributed Coding of Choice, Action and Engagement across the Mouse Brain.” Nature, vol. 576, no. 7786, 2019, pp. 266–73.

[3] IBL, et al. “A Brain-Wide Map of Neural Activity during Complex Behaviour.” Nature, vol. 645, no. 8079, 2025, pp. 177–91.

[4] Carey, Harry, et al. “DeepSlice: Rapid Fully Automatic Registration of Mouse Brain Imaging to a Volumetric Atlas.” Nature Communications, vol. 14, no. 1, 2023, p. 5884.

[5] Liu, Liu D., et al. “Accurate Localization of Linear Probe Electrode Arrays across Multiple Brains.” ENeuro, vol. 8, no. 6, 2021, p. ENEURO-0241.

[6] Zhang, Yizi, et al. “Neural Encoding and Decoding at Scale.” Forty-Second International Conference on Machine Learning, 2025.

[7] Duan, Yu, et al. “POCO: Scalable Neural Forecasting through Population Conditioning.” The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025.

[8] IBL, et al. “Reproducibility of in vivo electrophysiological measurements in mice.” Elife, 13, RP100840, 2025.

Reviewer_TbEBOpenReview ↗

Update the score of Presentation

The authors have improved their presentation after revision, so I change the score from 2 to 3 in the Presentation.

Reviewer_nVLBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

BrainWideBench addresses the fragmented evaluation of neural models by establishing a unified benchmark based on IBL mouse neural recordings. Pretraining uses 423 sessions from 126 animals, while evaluation uses 29 sessions from 13 held-out animals. The benchmark covers three task families: behavioral decoding, masked and future neural activity prediction, and brain-region identification. Comparisons between pretrained models and single-session baselines show that different methods lead on behavioral decoding, dynamics prediction, and anatomical identification. The main contribution is a shared evaluation framework that highlights the dependence of transfer performance on the pretraining objective: strong performance on one task does not necessarily imply broadly transferable representations.

Critical strengths
  1. The benchmark combines a common data split with three complementary task families. This provides a valuable shared framework for comparing the transfer capabilities of neural models beyond a single downstream task.

  2. The range of baselines and analyses helps reveal how alignment between pretraining objectives and downstream tasks affects transfer. This is a useful contribution to interpreting what strong benchmark performance does and does not demonstrate about a representation.

Critical weaknesses
  1. The aggregation procedure may obscure important differences between methods. Clipping negative R²/D² scores to zero hides the extent of poor performance, and the ranking procedure should not be interpreted as a comprehensive pairwise significance comparison. Since the benchmark's value depends on reliable comparisons, its main conclusions need to be checked against the original, unclipped scores.

  2. Some comparisons do not isolate the effect of pretraining from architectural changes. Certain pretrained models introduce additional output heads or increased model capacity, so the observed gains cannot be attributed entirely to pretraining. This limits the conclusions that can be drawn about the benefit of pretraining itself.

Questions for authors
  1. Robustness of the comparisons: Do the main method rankings and conclusions hold when negative R²/D² values are retained? Can the authors provide the unclipped scores and appropriate pairwise comparisons for the claims central to the paper?

  2. Attribution to pretraining: Which comparisons hold architecture, output heads, model capacity, and downstream training conditions fixed while varying pretraining? What benefit remains under those controls?

3 comments

We thank the reviewer for their thoughtful feedback and positive assessment of our work. We appreciate their recognition of the value of evaluating neural models across complementary task suites, and particularly their observation that our results reveal how the alignment between pretraining objectives and downstream tasks shapes transfer performance. Below, we address their concerns through further analyses of model comparisons, including unclipped scores and more rigorous statistical comparisons, and controlled experiments isolating the benefits of pretraining from architectural differences. We hope these additional results address the reviewer's concerns and provide stronger support for our main conclusions.

1. Robustness of performance aggregation and statistical comparisons (Q1/W1)

1.1. Sensitivity to clipping negative scores

We thank the reviewer for raising this point. R2R^2 and Poisson D2D^2 are bounded above by 1 but unbounded below, so a few strongly negative sessions can dominate the mean and obscure performance on the remaining sessions. We therefore clip each per-session score at 0 before averaging, a design choice to measure whether, and by how much, a model improves on the null model.

"Do the main method rankings and conclusions hold when negative R²/D² values are retained?"

Tables R12 and R13 report, for each model and task, the change in mean score when clipping is removed and the number of negative scores. Removing clipping changes TS1 mean scores by at most 0.05, and the model ranking is identical with and without clipping in both TS1 and TS2, with average ranks changing by at most 0.06. Negative scores are most frequent for the weakest models (e.g., Linear and CEBRA in TS1, the autoencoder in TS2). We included the full unclipped scores (in terms of differences), alongside these tables, in the revised manuscript.

Table R12. Effect of removing clipping on TS1 scores. Each cell shows the change in mean score when clipping is removed (unclipped - clipped) and, in parentheses, the number of negative scores out of 145 session-seed pairs (29 sessions x 5 seeds). Average ranks are shown with clipping, as reported in the paper, and without.

MethodLicks, D2D^2Whisker, R2R^2Wheel, R2R^2RPaw, R2R^2LPaw, R2R^2Avg rank, clippedAvg rank, unclipped
Linear-.025 (18)−.006 (14)−.006 (16)−.012 (38)−.037 (26)7.737.75
MLP−.001 (2)−.025 (11).000 (0).000 (0)−.003 (14)5.465.46
GRU−.033 (15)−.030 (10)−.014 (7)−.002 (5)−.005 (9)3.964.01
CNN−.047 (8)−.009 (5)−.009 (6)−.008 (8)−.007 (14)3.743.78
CEBRA−.031 (13)−.013 (22)−.013 (21)−.009 (17)−.043 (25)7.447.46
NDT−.017 (12)−.034 (14)−.010 (13)−.011 (7)−.002 (11)6.806.81
POYO−.049 (7)−.001 (6)−.001 (4)−.001 (2)−.028 (24)4.804.86
POYO+−.014 (5)−.003 (6).000 (1)−.004 (5)−.003 (11)2.532.52
POSSM.000 (0)−.017 (8).000 (0).000 (0)−.001 (2)3.183.19
NDT-Stitch−.030 (8)−.020 (8)−.007 (10)−.010 (11)−.008 (15)3.583.60
MtM−.021 (9)−.002 (5).000 (3).000 (2)−.002 (8)4.054.08

Table R13. Effect of removing clipping on TS2 scores. Each cell shows the change in mean score when clipping is removed (unclipped - clipped) and, in parentheses, the number of negative scores out of 145 session-seed pairs (29 sessions x 5 seeds). Average ranks are shown with clipping, as reported in the paper, and without.

MethodCo-smoothing, D2D^2Forecasting, D2D^2Avg rank, clippedAvg rank, unclipped
Autoencoder−.080 (35)−.054 (66)4.814.86
LFADS.000 (0).000 (0)2.342.34
NDT−.001 (7).000 (0)2.792.79
NDT-Stitch−.008 (8).000 (0)1.641.64
MtM.000 (0).000 (3)2.572.57

1.2. Pairwise statistical comparisons

“...the ranking procedure should not be interpreted as a comprehensive pairwise significance comparison.” We thank the reviewer for raising this important distinction. We agree that our ranking procedure does not establish comprehensive pairwise statistical significance. Its purpose is to summarize relative model performance across heterogeneous tasks and evaluation sessions, using significance tests to reduce sensitivity to small performance differences. We have modified Appendix E.4 to clarify the scope and make this distinction in the revised manuscript.

“Can the authors provide the [...] appropriate pairwise comparisons for the claims central to the paper?”

To directly assess the statistical support for comparisons we make throughout the text, we additionally conducted paired permutation tests on the original, unclipped scores, exchanging model assignments at the animal level to account for dependencies between sessions and neurons within each animal. We apply Holm correction across model pairs within each evaluation task and report the pairwise results in Appendix I.4.

These analyses broadly support our main findings while providing a more nuanced picture of differences between methods. In TS1, pretrained approaches generally outperform single-session baselines, although differences among leading pretrained models are less consistently significant. In TS2, NDT-Stitch significantly outperforms all other evaluated models on forecasting, while differences among the leading co-smoothing models are not statistically significant under the family-wise correction applied across all model pairs. In TS3, unit-level representations significantly outperform transductive approaches, with NuCLR achieving the highest observed macro-F1 across evaluation settings, although its advantage over NEMO is not statistically significant after correction. We have revised the discussion accordingly and thank the reviewer for prompting this analysis, which has strengthened the statistical evaluation and interpretation of our results.

2. Isolating the Effect of Pretraining (W2/Q2)

“Which comparisons hold architecture, output heads, model capacity, and downstream training conditions fixed while varying pretraining? What benefit remains under those controls?”

We thank the reviewer for raising this concern. Our primary benchmark comparisons evaluate models under their respective architectures and training procedures, while holding downstream data, splits, tasks, and evaluation metrics fixed. The standard spike-based pretrained baselines use backbones of approximately 10M parameters (excluding session-specific embeddings and stitchers). For the closely related NDT/NDT-Stitch and POYO/POYO+ pairs, the underlying model families and decoding mechanisms are shared, although the pretrained variants introduce some architectural differences to accommodate multi-session training. Notably, single-session baselines are extensively tuned per recording over architecture size, depth, and optimization hyperparameters, whereas pretrained models use fixed architectures with limited downstream tuning.

To isolate the effect of pretraining, we additionally compare NDT-Stitch and POYO+ against randomly initialized versions of the same architectures, holding model capacity, downstream output heads, downstream data, splits, and learning-rate search fixed. Across three representative TS1 tasks (whisker motion, licking, and choice), pretrained models consistently outperform their random-initialized counterparts across recordings (Table R14). Similarly, on TS2 tasks (co-smoothing and forecasting), pretrained NDT-Stitch and MtM outperform their randomly initialized counterparts (Table R15).

Table R14. TS1 test performance across 29 recordings (mean ± SEM over 5 seeds), comparing pretrained versus random initialization.

TaskModelRandom-initPretrained
WhiskerNDT-Stitch0.304 ± 0.0020.384 ± 0.003
WhiskerPOYO+0.264 ± 0.0040.380 ± 0.003
LickingNDT-Stitch0.336 ± 0.0050.525 ± 0.002
LickingPOYO+0.364 ± 0.0090.678 ± 0.003
ChoiceNDT-Stitch0.622 ± 0.0020.674 ± 0.001
ChoicePOYO+0.623 ± 0.0020.689 ± 0.003

Table R15. TS2 test performance across 29 recordings (mean ± SEM over 5 seeds), comparing pretrained versus random initialization.

TaskModelRandom-initPretrained
co-smoothingNDT-Stitch0.108 ± 0.0020.157 ± 0.001
co-smoothingMtM0.157 ± 0.0000.191 ± 0.001
forecastingNDT-Stitch0.153 ± 0.0000.177 ± 0.000
forecastingMtM0.099 ± 0.0000.131 ± 0.001

These controlled results demonstrate that the observed benefits cannot be explained solely by architectural differences or increased capacity. We have added this analysis to the revised manuscript (Appendix I.5), along with clarifications regarding the comparisons throughout the text.

AuthorsOpenReview ↗

Follow-up on Rebuttal

Dear Reviewer nVLB,

We would like to follow up on our rebuttal and revised manuscript, and thank you again for your thoughtful feedback. Your review raised important questions about the robustness of our comparisons and the extent to which performance gains can be attributed to pretraining. In response, we conducted additional analyses using unclipped scores and pairwise statistical tests, along with controlled experiments comparing pretrained models against randomly initialized counterparts. These results reinforce our central findings while providing a more nuanced interpretation of the differences between methods.

We appreciate your suggestions, which we believe have helped us meaningfully improve the rigor and clarity of our work. We hope our responses have addressed your concerns, and would be grateful if you could consider increasing your overall score. We would also be happy to address any remaining questions.

Best regards,

The authors

Reviewer_BYk7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BrainWideBench, a benchmark for evaluating large-scale pretraining and across-animal transfer in multi-region neural recordings. It uses the IBL Brainwide Map dataset, with recordings from 139 mice of which 126 are used for pretraining and 13 are held out for evaluation. The benchmark includes three task suites: behavioral decoding, neural activity reconstruction and forecasting, and region classification. The authors evaluate several pretrained models and single-session baselines, and show that their performance depends on the prediction task, transfer setting, and whether the pretrained model's objective is aligned with the specific evaluation task suite. The paper also provides standardized data processing and QC, an evaluation setup, and baseline implementations for studying neural representations across animals.

Critical strengths

I appreciate the effort to build a common evaluation framework on this IBL multi-region dataset. These benchmarks are not common in neural recordings, and having a standardized setup helps with easing model evaluation and benchmarking, as discussed by the authors. The authors have chosen a range of pretrained models and single-session classical models as baselines, and compared them meticulously in their evaluation setup for the three task suites.

Critical weaknesses
  1. My main concern is what the benchmark currently tells us about practical transfer, and the limited scope of generalization. In TS1, the capability to transfer decoders with little or no supervision on target session is not investigated. Instead, the results mainly evaluate the value of pretraining as an initialization for supervised fine-tuning on each new session. Thus, few-shot and zero-shot behavioral decoding remain unevaluated, even though the task and behavior targets are already shared across animals. This could be done using the same fine-tuning and evaluation setup and train/test splits, with smaller amounts of training data. This would not require a new dataset or a different task suite, and the authors state that TS1 supports these settings, yet this is not evaluated in the benchmark. It is therefore unclear how many labeled minutes or trials are needed for useful decoding on a new animal, and how much data pretraining saves compared with training from scratch.

  2. The behavioral task coverage in the benchmark is also very limited, and generalization to different tasks is not evaluated. This limits what the benchmark can establish about general-purpose neural representations as pretraining and evaluation are restricted to the same wheel-turning task. While the authors acknowledge this limitation, other task and stimulus conditions could be considered. Within IBL, spontaneous recordings could provide a test of neural prediction outside the task periods (IBL et al., 2025). Other relevant large-scale mouse datasets from the Allen Institute include the visual change detection task (Bennett et al., 2026) and recordings of mice responding to a battery of visual stimuli (Siegle et al., 2021), which could provide complementary evaluations beyond the current wheel-turning task. IBL also provides LFP and widefield calcium imaging data during the same behavioral task, but neither is considered for pretrained model evaluation in the benchmark.

References:

  1. Bennett, C. et al, (2026). Map of spiking activity underlying change detection in the mouse visual system. Cell.
  2. Siegle, J. H. et al. (2021). Survey of spiking in the mouse visual system reveals functional hierarchy. Nature.
Questions for authors
  1. During MtM adaptation for TS3, are the intra-region and inter-region masks constructed using held-out animals' anatomical labels? If so, could this introduce leakage of the evaluation-region labels in TS3, since the masking strategy is derived from the labels used for evaluation, even if the prediction loss itself does not use those labels?

  2. Does choice decoding use the 1-second window before or after movement onset? Table A4 conflicts with Figure A9 and the text. Also, it seems like the sessions or subjects used to select the 5-minute block duration for TS2 may be the same subjects use in the main evaluation cohort of the benchmark. Can the authors clarify that?

  3. Why are negative D2 and R2 scores clipped to zero? Does removing clipping change the model comparisons?

10 comments

We thank the reviewer for their careful assessment and constructive feedback. We appreciate their recognition of the value of a standardized evaluation framework for neural recordings and our comprehensive comparison of baselines across the three task suites. Below, we address their concerns with new experiments evaluating data-efficient transfer and generalization beyond trial-aligned behavior, alongside additional analyses and corrections that strengthen the rigor of our evaluation protocols. We hope these results and clarifications address the reviewer's concerns and further strengthen the contributions of our work.

1. Transfer under limited supervision (W1)

We thank the reviewer for highlighting the importance of evaluating transfer under limited supervision. Our existing results demonstrate the benefits of pretraining in downstream predictive performance, while the benchmark itself is designed to support a broader range of transfer settings. We extend our evaluation with decoding experiments under limited (few-shot) and zero supervision, across four representative models. In particular, we also add SPINT [1] as an initial baseline for transfer without finetuning through gradient-free calibration.

1.1. Few-shot decoding

Sample-efficient adaptation to new recording sessions is an important goal in neural decoding, as highlighted by benchmarks such as FALCON [2]. To evaluate how pretraining affects sample efficiency, we vary the amount of labeled training data available on each held-out session while keeping the validation and test splits fixed. Specifically, we use 25%, 50%, 75%, or 100% of each session's training split, corresponding on average to approximately 1.1, 2.2, 3.4, and 4.5 labeled minutes (67, 134, 202, and 269 trials), respectively. We evaluate four representative models: CNN and POYO+, the strongest single-session and pretrained models by average rank, and NDT and NDT-Stitch, which allow us to examine the effect of pretraining within the same backbone family. Tables R1-R3 report results across three representative decoding targets: whisker motion energy (continuous), licking rate (rate), and choice (discrete). We will extend this analysis to the remaining TS1 decoding tasks in the camera-ready version.

Table R1. TS1 reduced supervision: whisker motion energy (R2R^2) vs. fraction of held-out training data used for supervision. Results are reported as mean ± SEM over 5 seeds.

Model0.250.500.751.00 (paper)
CNN0.290 ± 0.0050.338 ± 0.0020.366 ± 0.0030.389 ± 0.004
NDT0.216 ± 0.0020.268 ± 0.0030.288 ± 0.0030.303 ± 0.002
POYO+0.233 ± 0.0020.303 ± 0.0020.353 ± 0.0030.380 ± 0.003
NDT-Stitch0.251 ± 0.0070.302 ± 0.0040.345 ± 0.0040.384 ± 0.003

Table R2. TS1 reduced supervision, licking rate (D2D^2) vs. fraction of held-out training data used for supervision. Results are reported as mean ± SEM over 5 seeds.

Model0.250.500.751.00 (paper)
CNN0.393 ± 0.0060.487 ± 0.0080.513 ± 0.0030.549 ± 0.008
NDT0.216 ± 0.0030.262 ± 0.0020.280 ± 0.0040.314 ± 0.001
POYO+0.476 ± 0.0030.589 ± 0.0050.636 ± 0.0040.678 ± 0.003
NDT-Stitch0.315 ± 0.0120.412 ± 0.0040.462 ± 0.0070.525 ± 0.002

Table R3. TS1 reduced supervision, choice (balanced accuracy) vs. fraction of held-out training data used for supervision. Results are reported as mean ± SEM over 5 seeds.

Model0.250.500.751.00 (paper)
CNN0.543 ± 0.0030.565 ± 0.0030.590 ± 0.0040.621 ± 0.002
NDT0.544 ± 0.0020.580 ± 0.0030.599 ± 0.0030.616 ± 0.003
POYO+0.611 ± 0.0040.662 ± 0.0020.681 ± 0.0030.689 ± 0.003
NDT-Stitch0.570 ± 0.0050.620 ± 0.0030.632 ± 0.0020.674 ± 0.001

Results. Pretraining can substantially reduce the amount of labeled data needed to achieve competitive decoding performance on new animals. Using only 50% of the data (2.2 min or 134 trials), POYO+ outperforms CNN trained on the full training split (~4.5 min or ~269 trials) for both choice decoding (0.662 vs. 0.621) and licking rate (0.589 vs. 0.549). Similarly, NDT-Stitch matches NDT trained on the full split using only 25% of the data for licking (0.315 vs. 0.314) and 50% for whisker motion energy (0.302 vs. 0.303). However, these benefits vary across models and tasks: CNN remains stronger than POYO+ on whisker motion energy, while NDT-Stitch provides limited gains for choice under low supervision. This may partly reflect differences in how well pretraining objectives align with downstream targets, consistent with our findings in the main benchmark. Together, these results illustrate how BrainWideBench enables evaluation of sample-efficient transfer, revealing both the strengths and limitations of current approaches across different decoding tasks.

1.2. Decoding transfer with no supervision

We additionally evaluate behavioral decoding on held-out animals without any target-session behavioral labels or gradient-based finetuning. Since POYO+ and POSSM were pretrained on all eight TS1 decoding tasks, we directly evaluate their pretrained behavioral readouts on new sessions, initializing unseen unit and session embeddings with the mean pretrained embedding. We compare these against SPINT [1], which supports gradient-free calibration using only unlabeled neural activity from new sessions.

Table R4. Zero-shot behavioral decoding on held-out TS1 sessions, with supervised baselines for reference. POYO+ and POSSM use mean pretrained embeddings for unseen units and sessions. SPINT performs gradient-free calibration using only unlabeled neural activity preceding the test split of each held-out session. R2R^2/D2D^2 are clipped at 0 per session, as in the main results. Zero-shot results use one seed.

ModelChoiceContrastRewardWheelWhiskerL. pawR. pawLicking
Supervised reference
Linear (single-session)0.6140.2190.8130.2190.1750.0840.0910.151
CNN (single-session)0.6210.2050.8870.3300.3890.1940.2250.549
POYO+ (finetuned)0.6890.2670.8980.3620.3800.2000.2490.678
Zero-shot (new)
POYO+0.5010.2130.5000.0340.0030.0080.0010.000
POSSM0.4990.2000.5000.0000.0000.0000.0000.000
SPINT0.5290.2570.8740.2060.2030.1070.1050.105

Results. As shown in Table R4, POYO+ and POSSM perform near chance on classification tasks and achieve near-zero R2R^2/D2D^2 on continuous and rate targets, despite their strong performance after supervised finetuning. In contrast, SPINT achieves above-chance performance on reward and contrast, and positive average R2R^2 across all four continuous movement targets. Notably, its zero-shot reward decoding (0.874) approaches that of finetuned POYO+ (0.898), while its contrast decoding is comparable (0.257 vs. 0.267). However, licking remains challenging, with SPINT failing to outperform the null model on 14 of 29 sessions, partly due to difficulties estimating session-specific mean lick rates. Together, these results highlight several important directions for future investigation, including the amount and type of calibration data needed for transfer and the role of model architecture in enabling zero-shot generalization. More broadly, these findings, now reported in Appendices I.1.1 and I.1.2, illustrate how BrainWideBench provides a framework for evaluating different transfer capabilities and identifying the strengths and limitations of current approaches.


References:

[1] Le, Trung, et al. “SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding.” The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025.

[2] Karpowicz, Brianna M., et al. “Few-Shot Algorithms for Consistent Neural Decoding (FALCON) Benchmark.” The Thirty-Eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.

2. Scope of generalization and additional modalities (W2)

We thank the reviewer for highlighting the importance of generalization across behavioral paradigms and recording modalities. BrainWideBench is designed to evaluate across-animal transfer and generalization across downstream objectives under a controlled experimental setting. The IBL Brainwide Map provides a suitable foundation for this purpose, combining large-scale, brain-wide recordings across 139 animals with standardized behavioral protocols. Holding the behavioral paradigm fixed allows us to compare transfer capabilities without simultaneously introducing variation in experimental protocols, behavioral targets, and recording conditions. We agree that generalization across behavioral paradigms and recording modalities are important complementary directions, and extending the benchmark along these axes will require evaluation protocols that account for differences in experimental conditions, available signals, and downstream tasks. Below, we present additional evaluations of neural activity prediction outside task-aligned periods and pretrained LFP representations, and discuss opportunities for extending the benchmark to additional datasets and modalities.

2.1. Neural prediction outside task-aligned periods

"spontaneous recordings could provide a test of neural prediction outside the task periods" We thank the reviewer for this suggestion. To assess generalization beyond task-aligned periods, we additionally evaluate TS2 co-smoothing and forecasting on inter-trial intervals within the held-out test blocks. We retain the same model architectures, hyperparameters, held-out neurons, and 1 s prediction windows as in Table 2. As shown in Table R5, performance decreases across all models on both tasks, but the relative model rankings remain unchanged. These results suggest that the main comparisons are robust to the choice of evaluation windows, while highlighting the additional challenges of predicting neural activity outside task-aligned periods. We have included this analysis in Appendix I.2.2 of the revised manuscript.

Table R5. Spiking activity outside the task-aligned windows. Clipped Poisson D², mean ± SEM across 3 seeds.

ModelCo-smoothing, standardCo-smoothing, between trialsForecasting, standardForecasting, between trials
LFADS0.176 ± 0.0000.131 ± 0.0010.144 ± 0.0010.116 ± 0.001
NDT0.132 ± 0.0000.094 ± 0.0010.158 ± 0.0000.136 ± 0.000
NDT-Stitch0.157 ± 0.0010.119 ± 0.0010.177 ± 0.0000.150 ± 0.000
MtM0.191 ± 0.0010.151 ± 0.0010.131 ± 0.0010.105 ± 0.001

We also evaluate the same models on the post-task spontaneous epoch of IBL’s passive protocol (26 of 29 held-out sessions; the remaining 3 have no spontaneous epoch). As shown in Table R6, all models show little to no improvement over the null model. Since this holds for single-session and pretrained models alike, a likely factor is the shift in neural dynamics between task-engaged and spontaneous states, rather than a limitation specific to pretraining. We have added this evaluation as a benchmark setting in Appendix I.2.2.

Table R6. Spontaneous-epoch evaluation (post-task passive protocol, 26 sessions). Clipped Poisson D2D^2, mean SEM across 3 seeds. Number of sessions with negative unclipped D2D^2 in parentheses.

ModelCo-smoothingForecasting
LFADS0.021 0.001 (16/26)0.000 0.000 (26/26)
NDT0.011 0.000 (20/26)0.003 0.000 (22/26)
NDT-Stitch0.017 0.000 (17/26)0.005 0.000 (22/26)
MtM0.019 0.001 (19/26)0.000 0.000 (26/26)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion period

Dear Area Chair,

We thank you and the reviewers for the time and effort devoted to our submission. The reviewers agreed that BrainWideBench "addresses an important gap" (TbEB) and offers "a valuable shared framework for comparing the transfer capabilities of neural models beyond a single downstream task" (nVLB), noting that "having a standardized setup helps with easing model evaluation and benchmarking" (BYk7). They also recognized the "considerable effort in data preprocessing and quality control" (TbEB), the breadth of baselines "compared [...] meticulously" across the three task suites (BYk7), and the benchmark’s value for "interpreting what strong benchmark performance does and does not demonstrate about a representation" (nVLB). As the discussion period concludes, we summarize its outcome and the resulting improvements to the manuscript.

Outcome of the discussion. Two reviewers increased their overall ratings following our rebuttal, bringing the final scores to 3, 3, and 4, all in favor of acceptance:

  • Reviewer BYk7 (2 → 3) acknowledged that our additional experiments "address [their] main concerns about the missing transfer evaluations and make the benchmark more informative." They particularly appreciated the new few-shot and zero-shot evaluations and spontaneous-activity analyses. We subsequently clarified how these transfer settings will be supported by the evaluation pipeline and planned leaderboard.
  • Reviewer TbEB (3 → 4) recognized our improved motivation for anatomical identification (TS3) and clarification of the benchmark's generalization objectives, noting that we had "done a good job" addressing these concerns. They also increased their Presentation score from 2 to 3.
  • Reviewer nVLB (rating: 3) raised concerns about the robustness of model comparisons and attribution of performance gains to pretraining. We addressed these through additional statistical analyses and controlled experiments isolating the effects of pretraining. We have not yet received a follow-up, but these results and revisions are detailed in our rebuttal and incorporated into the manuscript.

Manuscript updates. In response to the reviewers' feedback, we have substantially extended the empirical evaluation and strengthened the rigor and clarity of the benchmark:

  • Few-shot and zero-shot transfer (BYk7; Appendix I.1): Added behavioral decoding evaluations under reduced supervision and without target-session behavioral labels. These experiments reveal both the sample-efficiency benefits of pretraining and the substantial limitations of current models in zero-shot transfer.
  • Generalization beyond task-aligned activity (BYk7; Appendix I.2.2): Extended neural prediction evaluations to inter-trial intervals and spontaneous activity, revealing important generalization gaps while preserving model rankings in the inter-trial setting.
  • Additional pretrained LFP baselines (BYk7; Table 3): Incorporated Lfp2vec and BrainBERT, demonstrating improvements over hand-crafted LFP features for anatomical identification.
  • Robustness and fairness of comparisons (nVLB, BYk7; Appendices E.4–E.5, I.4–I.5): Verified that removing score clipping does not change the overall model rankings, added animal-level paired statistical comparisons with multiple-comparison correction, and conducted architecture-matched experiments demonstrating gains attributable to pretraining.
  • Evaluation protocol corrections (BYk7; Table 3, Appendices D and G): Eliminated anatomical label leakage during MtM adaptation, independently validated the TS2 temporal block duration, and corrected ambiguities in decoding windows.
  • Motivation and presentation (TbEB; Sections 1 and 3.3): Strengthened the neuroscience motivation for anatomical identification and clarified the distinction between across-animal transfer and cross-task generalization, along with improvements to task definitions and presentation.

We believe the reviewers' feedback has made BrainWideBench a substantially stronger and more rigorous benchmark, while sharpening its ability to identify both the capabilities and limitations of current pretrained neural models. We sincerely appreciate their constructive suggestions and your consideration of our revised submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.