- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3
Read review
Paper summary
Currently, neural activity and behavioral prediction usually require specific decoders for different recording sessions/animals/prediction targets. The state-of-the-art large-scale neural recording technology may make it possible to develop a single model that generalizes across animals/sessions/multiple downstream tasks. The authors introduce BrainWideBench, a benchmark to evaluate whether the learned large-scale neural representations are transferable across diverse tasks using a dataset from IBL. The authors standardize and filter neural activity and behavioral measurements from 139 mice across 276 brain regions, establish quality-control procedures, and define a pretraining set and an evaluation set of 13 held-out mice. The benchmark comprises three task suites: behavioral measurement prediction, neural activity prediction and anatomical region classification. The authors evaluate a range of existing models, including transformer-based, state-space, and self-supervised approaches. Their results show that pretraining generally improves downstream performance. Although, the benchmark is designed to investigate whether learned neural representations can transfer across diverse downstream tasks, the authors mainly focus on across-animal transfer and establishing baseline performance, with limited evaluation of across-task generalization. Overall, the paper provides a useful benchmarking framework while highlighting the limitations of current models in learning general-purpose neural representations that transfer across diverse tasks.
Critical strengths
Very large workload. The paper addresses an important gap by providing a benchmark for evaluating pretrained neural models across sessions/animals/prediction targets. Three task suites cover different aspects of neural representations, including behavior measurement, neural dynamics, and anatomy. The authors put considerable effort in data preprocessing and quality control. In addition, the authors also evaluate a bunch of existing models and provide useful reference for future studies.
Critical weaknesses
I don’t think anatomical region prediction is well-justified downstream task. instead of it may help for decoding behavior or neuron activity. Anatomy may be used to help behavior or neural activity prediction, it is unclear why predicting the brain region from which a neuron was recorded should be considered a core objective. From both neuroscience and engineering perspectives, I do not see a strong motivation for this. The authors should better justify its relevance to evaluating general-purpose neural representations.
Questions for authors
- The manuscript is sometimes difficult to follow. In the abstract, the term "task" is used to refer both to the behavioral task performed by the animals and to the prediction tasks defined by the benchmark. This is confusing, particularly given the limited background provided in the abstract. Also, I thought the generalization across tasks indicates the generalization across the three task suites. But much of the evaluation appears to focus across-animal or across tasks within each suite. The authors should clarify which types of generalization the benchmark is primarily designed to evaluate in the introduction. 2.Page 5, Lines 224–227: “For each decoding task, the target is a 1 second window sampled around key trial events” is unclear.
5 comments
2. Clarifications on terminology, generalization, and task definitions (Q1)
2.1. Distinguishing the behavioral paradigm from benchmark tasks
“In the abstract, the term "task" is used to refer both to the behavioral task performed by the animals and to the prediction tasks defined by the benchmark. This is confusing, particularly given the limited background provided in the abstract.”
We thank the reviewer for pointing out this ambiguity. We now reserve "task" for the benchmark's prediction tasks and task suites, and describe the animals' experimental paradigm as sensory-guided decision-making behavior. Accordingly, in the abstract, "mice performing a sensory-guided decision-making task" now reads "mice engaged in sensory-guided decision-making behavior", with corresponding changes throughout the manuscript (e.g., Sections 2 and 3.1).
2.2. Scope of generalization
“I thought the generalization across tasks indicates the generalization across the three task suites. But much of the evaluation appears to focus across-animal or across tasks within each suite.”
We thank the reviewer for highlighting this distinction. BrainWideBench evaluates two complementary forms of generalization: across-animal transfer, through evaluation on held-out mice, and cross-task generalization, by evaluating pretrained models on downstream prediction objectives that may differ from their pretraining objectives. The latter includes transfer across task suites. For example, NDT-Stitch and MtM, pretrained on neural prediction objectives, are evaluated on behavioral decoding (TS1) and brain region identification (TS3), while POYO+ and POSSM, pretrained for behavioral decoding, are also evaluated on TS3.
“Although, the benchmark is designed to investigate whether learned neural representations can transfer across diverse downstream tasks, the authors mainly focus on across-animal transfer and establishing baseline performance, with limited evaluation of across-task generalization.”
We acknowledge that the current baseline study does not exhaustively characterize transfer across all combinations of task suites, particularly as existing methods are often designed for specific objectives. Our evaluations nevertheless provide an initial characterization of cross-task generalization, while the benchmark establishes a standardized framework for more systematic comparisons. We have revised the introduction to clarify these two dimensions of generalization and the scope of the current evaluation.
2.3. Behavioral decoding windows
“Page 5, Lines 224–227: “For each decoding task, the target is a 1 second window sampled around key trial events” is unclear.”
We thank the reviewer for pointing out this ambiguity. We have revised the description in Section 3.1 to clarify that the 1-second target windows are defined relative to events in the animal's decision-making trial, with their timing depending on the variable being decoded. We also provide concrete examples to illustrate how these windows are used for different prediction targets. The revised text reads:
"For each decoding task, we define a 1-second target window aligned to a relevant event in the decision-making trial (stimulus onset, movement onset, or feedback onset). The timing of these windows depends on the variable being decoded. For example, choice is predicted from neural activity preceding movement onset, whereas licking rate is predicted over the second following feedback. These windows are selected to capture informative neural signals while minimizing confounds from other trial events (see Figure A10)."
The precise definitions of these target windows and their rationale are provided in Appendix D.1 (Table A4 and Figure A10).
References:
[1] Siegle, Joshua H., et al. “Survey of Spiking in the Mouse Visual System Reveals Functional Hierarchy.” Nature, vol. 592, no. 7852, 2021, pp. 86–92.
[2] Steinmetz, Nicholas A., et al. “Distributed Coding of Choice, Action and Engagement across the Mouse Brain.” Nature, vol. 576, no. 7786, 2019, pp. 266–73.
[3] IBL, et al. “A Brain-Wide Map of Neural Activity during Complex Behaviour.” Nature, vol. 645, no. 8079, 2025, pp. 177–91.
[4] Carey, Harry, et al. “DeepSlice: Rapid Fully Automatic Registration of Mouse Brain Imaging to a Volumetric Atlas.” Nature Communications, vol. 14, no. 1, 2023, p. 5884.
[5] Liu, Liu D., et al. “Accurate Localization of Linear Probe Electrode Arrays across Multiple Brains.” ENeuro, vol. 8, no. 6, 2021, p. ENEURO-0241.
[6] Zhang, Yizi, et al. “Neural Encoding and Decoding at Scale.” Forty-Second International Conference on Machine Learning, 2025.
[7] Duan, Yu, et al. “POCO: Scalable Neural Forecasting through Population Conditioning.” The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025.
[8] IBL, et al. “Reproducibility of in vivo electrophysiological measurements in mice.” Elife, 13, RP100840, 2025.
Update the score of Presentation
The authors have improved their presentation after revision, so I change the score from 2 to 3 in the Presentation.
We thank the reviewer for their constructive feedback and careful assessment of our work. We appreciate their recognition that BrainWideBench addresses an important gap in evaluating pretrained neural models across animals and tasks, as well as their acknowledgment of our extensive data preprocessing, quality control, and baseline evaluations. Below, we further motivate anatomical identification as a complementary evaluation task, clarify the scope of across-animal transfer and cross-task generalization, and address ambiguities in our task definitions and presentation. We hope these clarifications address the reviewer's concerns and better communicate the motivation and contributions of our benchmark.
1. Motivation for TS3 Anatomical Identification (W1)
We thank the reviewer for raising this point and welcome the opportunity to elaborate on the motivation for TS3, discussed in Section 3.3. While anatomical information can indeed support behavioral decoding and neural activity prediction, anatomical identification is also a meaningful downstream objective in its own right, both for its scientific utility and for evaluating general-purpose neural representations.
Anatomical localization is fundamental to interpreting neural recordings, as understanding how the brain processes information requires knowing which regions and circuits the recorded activity originates from. Studies investigating the neural basis of perception, decision-making, and behavior routinely rely on anatomical information to characterize the contributions of different brain regions and their interactions [1–3]. Although experimenters target specific regions during probe insertion, this does not establish the precise anatomical locations of recorded neurons [8]. Accurate localization typically requires costly postmortem histology and expert annotation, limiting precise localization to retrospective analyses and introducing potential annotation variability [4,5]. As discussed in Section 3.3, inferring anatomical identity directly from neural activity could complement these procedures, enabling in vivo localization and validation of anatomical assignments.
Beyond its practical utility, anatomical identification evaluates a complementary capability of general-purpose neural representations. While TS1 and TS2 assess whether representations capture behaviorally relevant information and neural population dynamics, TS3 tests whether they preserve biologically meaningful anatomical structure across animals. This builds on an emerging direction in large-scale neural representation learning, where studies such as NEDS [6] and POCO [7] have investigated the emergence of anatomical organization in learned representations without explicit anatomical supervision. TS3 provides a standardized, quality-controlled protocol for evaluating anatomical prediction in held-out animals and comparing recent methods for extracting neuron-level representations.
Together, these considerations establish the scientific relevance of anatomical identification and its value as a complementary evaluation axis for general-purpose neural representations. We have further clarified this motivation in Section 3.3 of the revised manuscript.