- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Nethobench, a benchmark to evaluate models on forecasting neural activity. The benchmark evaluates models on five metrics; distribution, temporal-spectral, relational, geometry and state dynamics. With these metrics, the authors show that models with similar point-wise errors represent varying behaviours and distributions, and that structural composites may better of models with worse error fits. The results on the comparison a variety of methods show that there is no general pattern of which architectures or loss functions are superior on this benchmark.
Critical strengths
Forecasting neural activity involves several important aspects other than just minimizing point-wise error. The benchmark serves well to cover a large variety of metrics, testing models for their forecasting capabilities, distributional and temporal matching and structural alignment. The strength of this expansive evaluation is shown in Figures 2e+f, where methods obtain very different scores on these metrics, while achieving similar point-wise errors, and in Figure 3b the inverse relation between forecast MAE and structural composite is presented.
The metrics are evaluated in-depth on various settings to investigate how sensitive each component is to certain settings. These investigations cover time-shuffling, perturbation severity, mean-preserving variance contraction and ablation studies on OmniMouse. This shows the effectiveness of having several metrics, as each metric represents different qualities of models that are sensitive to the problem, architecture and optimization methods.
Critical weaknesses
There are many different results in Figures 2-4, which makes reading it convoluted and dense, while the results/figure captions insufficiently clarify the presented material. Furthermore, the interpretation of the results is a bit shallow when considering the point of this benchmark is to evaluate methods on several aspects on forecasting neural activity and investigate whether structural properties are preserved. For example, in Figure 4a many methods are compared on the five metrics and the corresponding (short) paragraph highlights some scores, although it would be interesting to investigate what these results mean with respect to the evaluated methods. Why does SSM obtain such the best score on four metrics, but performs poorly on the Relational metric? What does it mean when the simple baseline VAR outperforms a method on a certain metric? Additionally, the range of the metrics is unintuitive and it is difficult to properly compare the methods.
In the introduction, the second and third contributions are too detailed for an introduction, and refer to figures that are far into the paper (Figure 4 is on page 8). The contributions should be rephrased to highlight more high-level findings of the results, because the reader can find the details in the results section later. Also, some of the related work that is in Appendix A should be in the introduction, or a related work section in the main paper.
Questions for authors
What is the motivation for the non-uniform distribution of the weights in Eq. 1? In Appendix C.1, the metrics are defined in more detail (which would preferably happen somewhat in the main paper too), but still it is unclear how a family score is computed. Could this be expanded, for example by stating that the subcomponents are averaged (if this is the case)?
3 comments
I thank the authors for their responses to my concerns and question. The proposed reorganisation of the validation section sounds promising, as they would provide a clearer presentation and interpretation of the results, but I feel many changes are needed and I cannot assess the revised manuscript. I will therefore maintain my score at this time.
Thank you for your feedback. The revised manuscript is now uploaded. We invite the reviewer to check the following parts specifically:
- Introduction: we now discuss related work and closes with a shorter, higher-level statement of our contributions.
- Metric definitions and weighting: Section 2 defines the five families with more details. The paragraph "Aggregation and reporting" explains how component scores are averaged within each family and identifies the weights in Eq. 1 as design choices. Figure S1c-e and Appendix C.3 show that the three checkpoints retained the same mean ranking across 5000 alternative weightings.
- Figures and model interpretation: Figure 2 now presents the metric controls, and Figure 3 presents the diagnostic comparisons. For your widefield example, Section 3.1 and Figure 3d explain why the composite favors the SSM while the relational score favors the teacher-forced Transformer for inter-regional dependencies. Figure 3d includes real-real and shuffled references to help interpret the scale. Table 10 gives the component-level results for VAR and the other models.
- Scientific utility: Section 3.2 reports the new IBL experiment testing how closely forecasts reproduce stimulus-side decoding accuracy in recorded hippocampal activity.
Rebuttal
We thank the reviewer for acknowledging the strength of our approach and for noting that the work we did to demonstrate how subscores are sensitive to different settings and corruptions is convincing.
As we understand, the reviewer’s main concerns relate more to the presentation and interpretation of our results in the manuscript itself than to the scientific significance of our work.
As a result, we are currently working to improve the manuscript, particularly regarding the density of Figures 2–4, the clarity of their captions, the interpretation of our experimental results (primarily regarding the widefield sequence), the introduction, and the definition of each metric. We will upload a new version shortly.
In any case, this has allowed us to rethink how we present our Validation section in particular (Section 3). The other reviewers asked us to conduct an additional experiment which results, we believe, allow us to frame a more comprehensive narrative. We invite you to read the section Scientific Utility of Nethobench of our rebuttal to ewAV to learn more about this additional experiment.
Overall, here is the outline for Section 3:
3.1 Validating the diagnostic utility of the metrics
3.1.1. Do the metrics respond to known structural changes?
Sensitivity to controlled alterations (Fig. 2)
3.1.2. What do the metrics reveal about forecasts from trained models?
3.2 Validating the scientific utility of the metrics
Regarding your question about the reason for the non-uniform distribution of the weights in Eq. 1. These coefficients were fixed design choices after several controls in which we varied them, and we tested how much our conclusions depend on them. Across 5,000 weight draws—including equal weighting (20% per family), omission of one family, and doubling of one family’s weight—the three training checkpoints retained the same ordering by mean composite score across 15 seeds (epoch 30 > 100 > 10; Fig. S5c–e, Appendix C.3). This supports the robustness of that comparison while leaving room for other weightings when a scientific question places particular importance on one family. As per the subcomponents, we mention in Section 2.1 that “components are mapped to agreement scores in [0, 1] and averaged within each family.”
[1] Willeke et al, “OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens”, ICLR, 2026