- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a neuronal prediction benchmark for zebrafish by aggregating data from previously published works and matching the neuronal activity of the animals across different recording settings. 6 baselines are provided for the benchmark, starting from simple linear models and including recurrent, convolutional and transformer-like architectures.
Critical strengths
- Elegant aggregation across space: The way to unify the spatial resolution across different recoding settings by matching the volumes of the brains and computing the average activity in the voxels is an elegant way to compare across different subjects and recording settings
- Two evaluation modes: Paper suggests to test generalization across held-out time and held-out animals.
- Baseline coverage: The paper provides both naive (copying and mean) baselines as well as more recent baselines like Crossformer, SimVPv2, PredRNN, etc, covering various architecture types
Critical weaknesses
- Webpage for benchmark: It is not clear if the benchmark is going to be a separate webpage where people can upload their predictions to be compared or not. If not - how is it a benchmark?
- No held-out test set: It seems like the benchmark is composed from fully publicly released datasets, which means that the test set is still publicly available and it is possible to overfit for the test set, which makes the benchmark a bit compromised
- Statistical rigor: No error bars on any of the evaluations, no statistical significant tests, which is a big limitation for a benchmark.
- Lack of ensembled baselines: Not all baselines are covered - the obvious way to beat current baselines is to ensemble a few of baselines across seeds. This is a common way of gaining a few percentage in ML competitions and i would expect this to be covered in a benchmark.
- Non-unified sampling frequencies: While the paper emphasises unifying the recodings across space, the time component is left out. It could be fine if the models take temporal resolution as input but it does not seem to be the case.
- Clarity of the results: The paper says to use 2 evaluation strategies on held-out time data and held-out animals. However, its not clearly stated in the results figures and tables - for which specific regime are they done. Jargons like "fictive behaviour" also might be better explained for a machine learning venue submission
- Clarity of the text: Several things appear in the text before they are explained - for example,
mixed-stimulus settingsappear in the abstract and introduction without explanations and are only explaine din the very end. Same for theSimVPv2which first appears at line 263 and is only explained in line 289.
Questions for authors
- Please comment on weaknesses 1, 2, 5, 6
1.1. Will the be a separate webpage to upload submissions?
1.2. Is the private test set not available on the internet? (such that the sources are also not available and could not be reconstructed via repeated your preprocessing steps)
1.3 Why is there no resampling across time? Right now a model is required to learn patterns across different timescales and since there are no temporal encodings in most of the baselines - the model cannot distinguish between 1Hz and 4Hz recordings.
1.4 Please provide a list for which table and figures which evaluation set up was used (held-out time or help-out animals) - Is any of your baselines also among winners of ZAPBench (Lueckmann et al 2025) competition? Why have not you considered POCO as a baseline (Duan 2025)?
- How does your competition relate to other existing neuroscience competitions like Algonauts, BrainScore, Sensorium? (apart from a different model organism)? What about ZAPBench? are there other fish-based competitions?
- How have you split the data for 80% train and 20% test parts? Are there repeats in the stimuli - have you considered them? Are there some noticeable time drifts across sessions? have you stratified the split in some fashion (across repeats or across time or somehow else?) How the 4 held-out animals were chosen?
- is MAE really the best metric? what could be the alternatives and why don't you use them? Why not to use correlation, which used in other benchmarks (Algonauts, Sensorium)?
- What are "Samples" and "Frames" in Table 1?
- Why is forecasting a good focus for the competition? Especially for the sensory brain areas, predicting much (2-4 seconds) in the future might not be the best idea is the stimulus changes (e.g. the image shown is changed, hence, a large part of visual activity would be expected to change as well).