acceptodds
Withdrawn from ICLR 2027

Fishbank: A Zebrafish Benchmark for Cross-Individual Whole-Brain Activity Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Forecasting whole-brain neural activity provides a quantitative test of brain dynamics models. We investigate the predictability of future activity from recent observations across spatial and temporal scales, individuals, and stimulus conditions. To support this investigation, we introduce FishBank, a multi-individual zebrafish forecasting benchmark comprising calcium imaging recordings of 189 zebrafish spanning three broad sensory categories. Spatial realignment, blockwise aggregation, and activity standardization transform heterogeneous recordings into common block-level activity sequences without requiring one-to-one neuronal correspondence, enabling systematic evaluation across model families, spatiotemporal settings, and training-data compositions. Our evaluation reveals three main findings. First, all learned predictors outperform naive baselines, while five deep neural networks performs closely. Second, forecasting performance depends on the spatial and temporal definition of the target. Third, multi-individual training yields remarkably lower error than the average error of single-recording models, whereas mixed-stimulus settings achieve slight improvement. FishBank provides a unified framework for characterizing the predictability of whole-brain activity, its spatiotemporal dependence, and the benefits and limitations of learning across individuals. We will publicly release FishBank, together with the associated preprocessing and benchmarking code, to facilitate future research.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Zebrafish、Whole-brain activity prediction、Cross-individual generalization、Multi-sample benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jrCAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a neuronal prediction benchmark for zebrafish by aggregating data from previously published works and matching the neuronal activity of the animals across different recording settings. 6 baselines are provided for the benchmark, starting from simple linear models and including recurrent, convolutional and transformer-like architectures.

Critical strengths
  • Elegant aggregation across space: The way to unify the spatial resolution across different recoding settings by matching the volumes of the brains and computing the average activity in the voxels is an elegant way to compare across different subjects and recording settings
  • Two evaluation modes: Paper suggests to test generalization across held-out time and held-out animals.
  • Baseline coverage: The paper provides both naive (copying and mean) baselines as well as more recent baselines like Crossformer, SimVPv2, PredRNN, etc, covering various architecture types
Critical weaknesses
  1. Webpage for benchmark: It is not clear if the benchmark is going to be a separate webpage where people can upload their predictions to be compared or not. If not - how is it a benchmark?
  2. No held-out test set: It seems like the benchmark is composed from fully publicly released datasets, which means that the test set is still publicly available and it is possible to overfit for the test set, which makes the benchmark a bit compromised
  3. Statistical rigor: No error bars on any of the evaluations, no statistical significant tests, which is a big limitation for a benchmark.
  4. Lack of ensembled baselines: Not all baselines are covered - the obvious way to beat current baselines is to ensemble a few of baselines across seeds. This is a common way of gaining a few percentage in ML competitions and i would expect this to be covered in a benchmark.
  5. Non-unified sampling frequencies: While the paper emphasises unifying the recodings across space, the time component is left out. It could be fine if the models take temporal resolution as input but it does not seem to be the case.
  6. Clarity of the results: The paper says to use 2 evaluation strategies on held-out time data and held-out animals. However, its not clearly stated in the results figures and tables - for which specific regime are they done. Jargons like "fictive behaviour" also might be better explained for a machine learning venue submission
  7. Clarity of the text: Several things appear in the text before they are explained - for example, mixed-stimulus settings appear in the abstract and introduction without explanations and are only explaine din the very end. Same for the SimVPv2 which first appears at line 263 and is only explained in line 289.
Questions for authors
  1. Please comment on weaknesses 1, 2, 5, 6
    1.1. Will the be a separate webpage to upload submissions?
    1.2. Is the private test set not available on the internet? (such that the sources are also not available and could not be reconstructed via repeated your preprocessing steps)
    1.3 Why is there no resampling across time? Right now a model is required to learn patterns across different timescales and since there are no temporal encodings in most of the baselines - the model cannot distinguish between 1Hz and 4Hz recordings.
    1.4 Please provide a list for which table and figures which evaluation set up was used (held-out time or help-out animals)
  2. Is any of your baselines also among winners of ZAPBench (Lueckmann et al 2025) competition? Why have not you considered POCO as a baseline (Duan 2025)?
  3. How does your competition relate to other existing neuroscience competitions like Algonauts, BrainScore, Sensorium? (apart from a different model organism)? What about ZAPBench? are there other fish-based competitions?
  4. How have you split the data for 80% train and 20% test parts? Are there repeats in the stimuli - have you considered them? Are there some noticeable time drifts across sessions? have you stratified the split in some fashion (across repeats or across time or somehow else?) How the 4 held-out animals were chosen?
  5. is MAE really the best metric? what could be the alternatives and why don't you use them? Why not to use correlation, which used in other benchmarks (Algonauts, Sensorium)?
  6. What are "Samples" and "Frames" in Table 1?
  7. Why is forecasting a good focus for the competition? Especially for the sensory brain areas, predicting much (2-4 seconds) in the future might not be the best idea is the stimulus changes (e.g. the image shown is changed, hence, a large part of visual activity would be expected to change as well).
Reviewer_4kahOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a neural forecasting benchmark based on Zebrafish data collected across disjoint studies. The benchmark is created by first standardizing the spatial coordinates of neurons across the studies and then block-wise aggregating along spatial axes. The benchmark supports two modes: within-recording evaluation, and across-individual evaluation. The paper evaluates 4 recent models, 2 simple learned baselines, and 2 trivial baselines (copy and mean). It also studies the benefit of multi-recording pretraining in the across-individual evaluation setting.

Critical strengths
  • The benchmark standardizes multiple datasets across labs into a meaningful coordinate system.
  • The benchmark is aimed at the "cross-individual" regime, which is a hard, important, and understudied challenge in the field.
Critical weaknesses
  • The paper severely under-specifies technical details of the benchmark (see Questions For Authors).
  • There is no measure of confidence intervals or error bars in the paper. This, in my opinion, is quite bad for a benchmark paper. In my opinion, the function of a benchmark is to provide researchers a standardized playground to develop and compare models. If the benchmark does not supply a reasonable way of measuring confidence intervals, then it is not providing any means to rank different models reliably. I would suggest doing multi-seed training and reporting standard-error-of-mean across seeds.
Questions for authors
  1. Please specify details related to creating the train/test splits for "Within-recording forecast": What is the size of a temporal block? Are the blocks assigned randomly to train and test splits, or is chronology maintained and the blocks near the end of the recording are assigned to test?
  2. Which study is used as the data source in a "held-out individual" setting? Why only use one study for this? Perhaps each study could have had its own "held-out individual" split?
  3. Could the authors provide a rationale for choosing C=P=4 (context and prediction lengths)?
  4. I see that C=P=4 was used for most experiments. I don't understand why C=32 was used in the P-sweep experiment of Fig 5a? It seems inconsistent and I would suggest the authors redo the sweep with C=4. Also, I believe this paper should also sweep C while keeping P fixed (preferably at 4 to match the main setting).
  5. How were the 4 evaluation subjects chosen in held-out individual setting? The benchmark should measure how sensitive the results are to this split to provide confidence that model rankings will remain stable with different splits.
  6. Could the authors evaluate POCO on this benchmark?

Errata:

  • Abstract: "five deep neural networks performs closely" -> "five deep neural networks perform closely"
Reviewer_EP3FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FishBank, a benchmark for forecasting whole-brain activity in larval zebrafish. It brings together 189 calcium-imaging recordings from nine published studies (visual, auditory, chemosensory and vestibular paradigms) and makes them comparable without matching neurons across fish: coordinates are converted to micrometers, each brain is centered, neuron activity is averaged within fixed 3D blocks (40×40×20 µm by default), and each recording is z-scored. The task is to predict the next frames of block activity from the previous ones, scored by MAE and compared with two simple baselines (repeating the last frame, or averaging the recent frames). Six existing models (linear, fully connected, three video-prediction models and a time-series transformer) all beat these baselines, and the nonlinear models perform similarly. The authors also report multiple expected results including that larger blocks are easier to predict.

Critical strengths

The main strengths are:

  1. Grouping the datasets is probably crucial to improve cross-individual performance and general representation learning: Learning across individuals is hard in zebrafish because every fish has different neurons, and studies differ in resolution, brain coverage and frame rate. Averaging activity within fixed spatial blocks is a simple and practical way to put very different recordings into a common format without matching neurons, and the block size can be changed to study different spatial scales.
  2. A potentially valuable resource, presented honestly. Gathering 189 recordings from nine studies into one benchmark could be very useful for the community if it is released and well documented. Results are always compared with simple baselines, and the authors are careful about what their experiments do not establish, such as a ranking of architectures or the separate effects of data size and individual diversity.
Critical weaknesses

The main weaknesses are:

  1. It does not seem that this is yet evaluated as a benchmark. Although the paper presents 189 recordings, they are never used together: each experiment relies on a small subset (21–27 recordings, mostly from a single study), the cohort used for the main model comparison is not described, and no split tests generalisation to new studies, which is what a multi-study benchmark should enable. There are no error bars or repeated runs. The manuscript feels slightly unfinished.
  2. The common format is not validated. Brains are only centered, without rotation, scaling or registration to a reference atlas, so the same block can correspond to different anatomy in different fish.
  3. The comparisons are too weak to support the conclusions. Some zebrafish forecasting models (POCO and the ZAPBench baselines) are not included, and the "linear" model is a network with a 100,000-unit hidden layer rather than a standard ridge or autoregressive baseline. The main findings (larger blocks are easier, longer horizons are harder, more data helps) are therefore expected, and they do not yet show how challenging the benchmark is for current methods. Maybe additional tasks would help distinguish between the models.
  4. No new method is introduced.
Questions for authors

Clarifications

  1. Which recordings are used in each experiment, especially the main model comparison, and how are the train/validation/test splits defined?
  2. How are blocks without any neuron handled in the input, the loss and the error metric? What fraction of blocks inside the brain is empty at each block size?
  3. Will the data, splits and code be released (an anonymized link would help the review), and do the licenses of the nine source datasets allow redistribution?
  4. can you precise the advantages and cons of the different models and what a better model could be?

Suggestions 5. Add a simple linear baseline (ridge or per-block autoregressive model), and report results over several seeds. 6. Quantify how well brains are aligned across fish (e.g., overlap of anatomical regions) and, where possible, register them to a common atlas. 7. Define splits over all 189 recordings, including held-out studies, and compare mixed and single-stimulus training on the same test fish, with a pooling control that keeps the total amount of training data fixed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.