acceptodds
Under review as a conference paper at ICLR 2027

Nethobench: Forecasting design and conditioning shape the structural realism of generated neural dynamics

Abstract

Forecasting models are increasingly used to model neural population activity, yet existing benchmarks typically evaluate them by pointwise predictive fidelity to held-out observations. In stochastic, partially observed systems, agreement with one sampled realization is distinct from preserving the statistical and dynamical structure of the process. We introduce Nethobench, a benchmark that evaluates structural agreement across five metric families: distributional, temporal spectral, relational, geometric, and state-dynamical. We validate Nethobench using targeted perturbations and stochastic systems with known dynamics, and apply it to mouse calcium imaging, mouse Neuropixels recordings, and human intracranial recordings, using empirical references to calibrate interpretation. Nethobench detects targeted structural failures and assigns high scores to independent samples from the correct synthetic process despite trajectory mismatch. It tracks training progress and structural changes under domain shift. Model comparisons reveal how forecasting formulation, architecture, training regime, and loss shape structural profiles, while input ablations demonstrate structural improvements from stimulus and behavioral conditioning. These results motivate reporting calibrated family-level structural metrics alongside pointwise accuracy, proper probabilistic scores, and task-specific endpoints when evaluating neural forecasts across species and recording modalities.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neural forecasting, neural population dynamics, neural realism, structural evaluation, generative models, stochastic dynamics, benchmarking.

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_FYXAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Nethobench, a benchmark to evaluate models on forecasting neural activity. The benchmark evaluates models on five metrics; distribution, temporal-spectral, relational, geometry and state dynamics. With these metrics, the authors show that models with similar point-wise errors represent varying behaviours and distributions, and that structural composites may better of models with worse error fits. The results on the comparison a variety of methods show that there is no general pattern of which architectures or loss functions are superior on this benchmark.

Critical strengths

Forecasting neural activity involves several important aspects other than just minimizing point-wise error. The benchmark serves well to cover a large variety of metrics, testing models for their forecasting capabilities, distributional and temporal matching and structural alignment. The strength of this expansive evaluation is shown in Figures 2e+f, where methods obtain very different scores on these metrics, while achieving similar point-wise errors, and in Figure 3b the inverse relation between forecast MAE and structural composite is presented.

The metrics are evaluated in-depth on various settings to investigate how sensitive each component is to certain settings. These investigations cover time-shuffling, perturbation severity, mean-preserving variance contraction and ablation studies on OmniMouse. This shows the effectiveness of having several metrics, as each metric represents different qualities of models that are sensitive to the problem, architecture and optimization methods.

Critical weaknesses

There are many different results in Figures 2-4, which makes reading it convoluted and dense, while the results/figure captions insufficiently clarify the presented material. Furthermore, the interpretation of the results is a bit shallow when considering the point of this benchmark is to evaluate methods on several aspects on forecasting neural activity and investigate whether structural properties are preserved. For example, in Figure 4a many methods are compared on the five metrics and the corresponding (short) paragraph highlights some scores, although it would be interesting to investigate what these results mean with respect to the evaluated methods. Why does SSM obtain such the best score on four metrics, but performs poorly on the Relational metric? What does it mean when the simple baseline VAR outperforms a method on a certain metric? Additionally, the range of the metrics is unintuitive and it is difficult to properly compare the methods.

In the introduction, the second and third contributions are too detailed for an introduction, and refer to figures that are far into the paper (Figure 4 is on page 8). The contributions should be rephrased to highlight more high-level findings of the results, because the reader can find the details in the results section later. Also, some of the related work that is in Appendix A should be in the introduction, or a related work section in the main paper.

Questions for authors

What is the motivation for the non-uniform distribution of the weights in Eq. 1? In Appendix C.1, the metrics are defined in more detail (which would preferably happen somewhat in the main paper too), but still it is unclear how a family score is computed. Could this be expanded, for example by stating that the subcomponents are averaged (if this is the case)?

3 comments

We thank the reviewer for acknowledging the strength of our approach and for noting that the work we did to demonstrate how subscores are sensitive to different settings and corruptions is convincing.

As we understand, the reviewer’s main concerns relate more to the presentation and interpretation of our results in the manuscript itself than to the scientific significance of our work.

As a result, we are currently working to improve the manuscript, particularly regarding the density of Figures 2–4, the clarity of their captions, the interpretation of our experimental results (primarily regarding the widefield sequence), the introduction, and the definition of each metric. We will upload a new version shortly.

In any case, this has allowed us to rethink how we present our Validation section in particular (Section 3). The other reviewers asked us to conduct an additional experiment which results, we believe, allow us to frame a more comprehensive narrative. We invite you to read the section Scientific Utility of Nethobench of our rebuttal to ewAV to learn more about this additional experiment.

Overall, here is the outline for Section 3:


3.1 Validating the diagnostic utility of the metrics

3.1.1. Do the metrics respond to known structural changes?

Sensitivity to controlled alterations (Fig. 2)

3.1.2. What do the metrics reveal about forecasts from trained models?

  • Widefield: trade offs between models hidden by similar composite and pointwise
  • Brain Treebank: retrieved activity can preserve population structure even when it is poorly matched to the current history
  • OmniMouse: adding observed behavior to video improves all five family means, finding aligned with the findings reported in [1]

3.2 Validating the scientific utility of the metrics

  • OmniMouse as a transition between 3.1 and 3.2: the observation that the scores increase when the model has access to biologically relevant information might suggest that Nethobench metrics might carry a neuroscientific utility, on top of their diagnostic utility
  • IBL: Nethobench helps recover an independently evaluated neural population property

Regarding your question about the reason for the non-uniform distribution of the weights in Eq. 1. These coefficients were fixed design choices after several controls in which we varied them, and we tested how much our conclusions depend on them. Across 5,000 weight draws—including equal weighting (20% per family), omission of one family, and doubling of one family’s weight—the three training checkpoints retained the same ordering by mean composite score across 15 seeds (epoch 30 > 100 > 10; Fig. S5c–e, Appendix C.3). This supports the robustness of that comparison while leaving room for other weightings when a scientific question places particular importance on one family. As per the subcomponents, we mention in Section 2.1 that “components are mapped to agreement scores in [0, 1] and averaged within each family.”

[1] Willeke et al, “OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens”, ICLR, 2026

I thank the authors for their responses to my concerns and question. The proposed reorganisation of the validation section sounds promising, as they would provide a clearer presentation and interpretation of the results, but I feel many changes are needed and I cannot assess the revised manuscript. I will therefore maintain my score at this time.

Thank you for your feedback. The revised manuscript is now uploaded. We invite the reviewer to check the following parts specifically:

  • Introduction: we now discuss related work and closes with a shorter, higher-level statement of our contributions.
  • Metric definitions and weighting: Section 2 defines the five families with more details. The paragraph "Aggregation and reporting" explains how component scores are averaged within each family and identifies the weights in Eq. 1 as design choices. Figure S1c-e and Appendix C.3 show that the three checkpoints retained the same mean ranking across 5000 alternative weightings.
  • Figures and model interpretation: Figure 2 now presents the metric controls, and Figure 3 presents the diagnostic comparisons. For your widefield example, Section 3.1 and Figure 3d explain why the composite favors the SSM while the relational score favors the teacher-forced Transformer for inter-regional dependencies. Figure 3d includes real-real and shuffled references to help interpret the scale. Table 10 gives the component-level results for VAR and the other models.
  • Scientific utility: Section 3.2 reports the new IBL experiment testing how closely forecasts reproduce stimulus-side decoding accuracy in recorded hippocampal activity.
Reviewer_ewAVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Nethobench, an evaluation framework and widefield forecasting benchmark for generated neural population activity. Eighteen components measure distributional, temporal–spectral, relational, geometric, and state-dynamical agreement, complementing pointwise accuracy and probabilistic evaluation. Validation includes independent stochastic-process realizations, targeted and matched-MSE perturbations, eight widefield forecasters, and applications to mouse and human recordings.

Critical strengths
  1. The paper addresses a relevant evaluation gap: pointwise accuracy does not fully characterize generated neural dynamics. Independent-process samples and matched-MSE controls substantiate this distinction.
  2. Family-level profiles reveal structural trade-offs between models that aggregate scores can obscure. Applications across mouse and human recordings support the framework’s potential as a reusable diagnostic tool.
Critical weaknesses
  1. The scientific utility of the proposed structural metrics remains insufficiently validated. The paper demonstrates score differences between models, but does not establish whether these differences translate into more reliable recovery of behaviorally relevant effects or experimentally defined dynamical changes.
  2. The incremental value of the temporal and state-dynamical metrics over established methods such as DSA remains unclear, as no matched empirical comparison is provided.
Questions for authors
  1. Can benchmark-guided model selection improve recovery of a known behavioral or perturbational effect compared with selection using pointwise error and a compact structural baseline?
  2. Could the authors compare these metrics with DSA (and other DSA variations) on matched synthetic controls and real forecasts, identifying which structural failures each approach uniquely detects?
6 comments
AuthorsOpenReview ↗

Rebuttal I - Scientific Utility 1

We thank the reviewer for raising two questions: one about the scientific utility of Nethobench and one about the incremental value of some of its metrics compared to published methods, and we will address the 2 iteratively. We hope that they will lead the reviewer to reconsider their score.

1. Scientific utility of Nethobench

As the reviewer notes in the “Critical Strengths” section of their review, Nethobench’s primary value lies in its diagnostic utility. Like structural benchmarks introduced in other fields, such as WeatherBench2 [1], Nethobench is primarily designed to investigate what each model uniquely preserves or loses compared to other models that achieve the same pointwise error as it does. We validate this extensively in Section 3.1, as well as in Section 3.3.

Nevertheless, in a biological context, the strength of such an evaluation framework cannot be reduced solely to the fact that it allows one to understand what a generator mathematically preserves from a process; rather, such dynamic preservation should ideally result in better downstream performance or more accurate conditional neural dynamics in order to be scientifically useful.

In addition to demonstrating that our metrics can reproduce the rankings of published models, our experiment on Omnimouse pre-trained models provides an example of how sensitive our scores are to behavioral conditions. We decided to take it a step further and examine whether a model selected by maximizing its Nethobench score on validation data subsequently retains the ability of neurons to encode certain stimuli when applied to test data.

To do this, we trained models to predict the responses of hippocampal neurons, whose population responses have been shown to allow decoding whether a visual stimulus was presented on the right or left side of the visual field (Figure 9 of the eLife paper; IBL data previously used in our Figures S18–19 [2]). In other words, using spike data from neurons in the hippocampus, we can determine whether the presented visual stimulus was on the right or left. We replicated these findings across 7 sessions from different laboratories:

SessionClassificationTrialsq FDR
NR_002481,5 %2760,0029
NR_002776,0 %4220,0029
NR_001974,0 %2650,0029
CSHL05971,9 %3380,0029
NR_003167,7 %1770,0309
UCLA03764,8 %2760,0156
CSH_ZAD_02262,2 %2860,0467

This provides us with a good framework for training models to predict neural responses, by feeding them the spiking data from a preceding window and the stimulus they are presented with ¹. We trained a set of models for each session and selected three of them based on three criteria that compare the actual spiking data with the predicted data:

  • the model that minimizes the mean absolute error (MAE)
  • the model that minimizes the Poisson log loss
  • the model that maximizes Nethobench
SessionMAENethobenchPoisson
NR_0024GRU h64 - epoch 480Stimulus-conditioned mean + history - α=1 000GRU h64 - epoch 60
NR_0027GRU h64 - epoch 260Stimulus-conditioned mean + history - α=10GRU h64 - epoch 100
NR_0019GRU h64 - epoch 240Stimulus-conditioned mean + history - α=100GRU h16 - epoch 80
CSHL059GRU h64 - epoch 480Stimulus-conditioned mean + history - α=1GRU h64 - epoch 60
NR_0031GRU h64 - epoch 180Stimulus-conditioned meanVAR α=10 000
UCLA037GRU h16 - epoch 360Stimulus-conditioned mean + history - α=1GRU h64 - epoch 120
CSH_ZAD_022GRU h32 — epoch 440Stimulus-conditioned mean + history - α=100GRU h32 - epoch 60

It is interesting to note that the choice of criterion affects the selected model family; in 6 out of 7 sessions, Nethobench selected the stimulus-conditional mean with neural history correction.

Now, to determine which criterion selects a model that most faithfully reproduces the population decodability observed in the real data, we can decode the stimulus side, but this time using the predicted neural sequences. To do this, we generated spike sequences for test trials using the models selected by each criterion for each session:

SessionTest trialsReelMAE-chosenNethobench-chosenPoisson-chosen
NR_00245572,8 %*90,9 %*100 %*98,2 %*
NR_00278467,7 %*88,0 %*73,4 %*96,3 %*
NR_00195355,8 %100 %*69,2 %100 %*
CSHL0596768,7 %82,1 %*73,1 %*98,5 %*
NR_00313564,3 %94,0 %*100 %*91,7 %*
UCLA0375549,3 %76,0 %*79,3 %*90,3 %*
CSH_ZAD_0225754,3 %89,6 %*66,7 %96,4 %*

* q FDR < 0,05

What stands out is that Nethobench is the criterion that on average most faithfully reproduces the level of decodability observed in the real data.

AuthorsOpenReview ↗

Rebuttal II - Scientific Utility 2

Selection criterion on validationAverage absolute deviation from actual decodability on the test set
Nethobench18.4 points
MAE26.8 points
Poisson34.1 points

In this experiment, the choice of criterion therefore affects the selected family of models, and this choice has a measurable impact on a downstream analysis that is independent of the selection, thereby demonstrating the scientific value of Nethobench.

We emphasize, however, that while these results are descriptive and encouraging, we continue not to claim to have introduced a method that can accurately predict the recovery of a biological process. Even though we have empirically demonstrated that there may be a correlation between a high Nethobench score and the preservation of biological observations across two experiments.

In conclusion, thanks to this additional experience, the usefulness of Nethobench has now been validated from both a diagnostic and a (neuro)scientific perspective. We will update the manuscript accordingly.

AuthorsOpenReview ↗

Rebuttal III - Comparing Temporal and State Dynamical families with DSA

2. Comparison with angular DSA

Given the existence of established methods to evaluate the temporal structure that 2 systems share - such as DSA - the reviewer is right to guide us towards comparing our Temporal and State-Dynamical families with it. Before going in that comparison, and even if that hasn't been pointed explicitly by the reviewer, we first want to emphasize that, based on neuroscience literature, other statistical aspect of neural data than their temporal structure, are relevant to characterize neural realism. Hence why, we included relational, distribution and geometry scores, on top of Temporal and State-Dynamical. So, even if our Temporal and State-Dynamical scores didn't show an incremental value compared to DSA, that wouldn't make our framework irrelevant.

But that was a digression. We compared angular DSA with Nethobench’s temporal–spectral (T) and state-dynamics (S) families on controlled synthetic data and real neural forecasts.

On synthetic data, we altered the temporal structure of the signals by perturbating the frequency and the amplitude of oscillations. If angular DSA and T and S families have different reactions to such perturbations, we have an empirical proof of incremental value of temporal and state dynamical metrics over DSA. And that's precisely what we observed ²:

ControlDSA distance ↓T ↑S ↑
Unchanged realization0.003520.8400.965
Frequency ×0.50.054970.6550.579
Frequency ×20.072200.7500.795
Amplitude ×0.50.003520.5830.390
Amplitude ×20.003520.5570.424

As expected, halving or doubling the frequency increase DSA distance and reduce Nethobench T and S scores overall. However, doubling or halving amplitudes increase DSA distance with the same magnitude, while halving amplitude has a stronger effect than doubling it on the State Dynamical family and on the Temporal Spectral family to a certain extent. This may be an empirical evidence that an incremental value of these families over DSA lies in that they are sensible to evolution of reference-defined coordinates.

On real data, we focused on the experiment on widefield calcium imaging. On top of comparing forecasts and references using pointwise fidelity and Nethobench scores, we additionally can compare them with angular DSA. We do not have ground truth knowledge of what structural properties each of these model learns. However, comparing the correlation between Nethobench metrics and angular DSA on these models ³ can shed light on at least 2 things:

  • Do the T and S families reward the same models as DSA?
  • Does the rest of the families actually reward models on structural properties different from what DSA rewards? In other words, is dynamical similarity a sufficient proxy for structural similarity?
ComparisonTS(T+S)/2(D+R+G)/3Composite
All eight model means0.8100.7140.833−0.0480.429

The results are eloquent: T and S families are highly correlated with angular DSA, whereas the rest of the families are very poorly correlated, resulting in a middle range correlation at the level of Nethobench composite. To a certain extent T and S families capture similar properties as DSA (even if on these unlabeled predictions, it is hard to pinpoint what dynamical feature one captures that the other doesn't). But Nethobench as a whole gives one a wider structural profile than DSA allows to.

[1] Rasp et al, "WeatherBench 2: A benchmark for the next generation of data-driven global weather models", 2024

[2] IBL et al, "Reproducibility of in vivo electrophysiological measurements in mice", eLife, 2025

¹ context: -100 ms to +100 ms and prediction from +100 ms to +1000 ms. We trained the GRU with different values of h; the VAR with different values of alpha; conditional mean (just the stimulus); conditional mean + history: starting from the average response to the stimulus and learning to adjust based on the observed activity for each trial -> regularized linear correction

² same realization as the one used in Figure 2 for the reference (one generated dataset with 100 sequences)

³ For each of the eight models, we averaged scores over 200 sequences. We then computed Spearman’s ρ across the eight model means using the negative angular DSA distance, so positive correlations indicate agreement in model rankings.

Reviewer_FvdiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work present a benchmark called Nethobench, which aims to evaluate the structure of neural dynamics. Paper proposed to evaluate it across five metric families: distributional, temporal–spectral, relational, geometric, and state-dynamical. The proposed metrics are tested on generated data with controlled perturbations test with known failure modes, also forecasting tasks.

Critical strengths

Conceptually, it is interesting that the paper frame the neural activity learning/evaluation using a rubric-ish-based approach that categorize the evaluation in several different ways.

Critical weaknesses

The major weakness is that all results are synthetic, making it difficult to argue that the method has any practical utility. There is no strong correlation demonstrating that any of the proposed metrics or categories are associated with the real-world performance of the data or model, so everything remains hypothetical. It is also unclear why the rubrics are designed in this particular way: the weights in Eq. 1 appear arbitrary and lack both empirical and theoretical grounding. Since all data are synthetic and all metrics are self-defined, it is difficult to validate any part of the method.

Questions for authors

NA

6 comments
AuthorsOpenReview ↗

Rebuttal I - Results are not all obtained on synthetic data

We would like to thank the reviewer for taking the time to review our work. We would like to clarify a few points regarding the reviewer's concerns:

  • First of all, the data underlying the main figures are not all synthetic.

The reviewer is right that the data underlying Figure 2 are synthetic, and this is justified by the goal of that specific experiment: show that each metric captures the structural property it is designed to capture. To that end, it is only natural to start with signals with known dynamical properties - hence synthetic -, to perturbate them along one axis (one property) and to show that the corruption had the intended effect of on intended metric.

However, we didn't stop there! The analyses in Sections 3.2, 3.3, and Appendices S16, S17–S18 were performed on real-world data from various brain imaging modalities: ranging from different types of electrophysiological data (Neuropixel probes and intracranial EEG) to different types of calcium imaging data (widefield calcium and two-photon calcium). The analyses performed on this real-world data show that Nethobench’s metrics generalize to different types of data and are therefore calibrated not to reward a specific modality for its realism, but rather to capture the realism of neural communication at different scales.

Analyses using synthetic data and real data are therefore complementary, with each serving distinct purposes: on the one hand, to demonstrate sensitivity to controlled alterations in the metrics we introduce, and on the other hand, to demonstrate their applicability to the full range of data modalities that the scientific community uses to study the brain.

AuthorsOpenReview ↗

Rebuttal II - Justification of practical utility 1

  • Regarding the practical utility of Nethobench, we demonstrate through various experiments that it can take two forms:
  1. It primarily serves as a diagnostic tool. Like the structural benchmarks introduced in other fields, such as WeatherBench2 [1], Nethobench is primarily used to investigate what each model uniquely preserves or loses compared to other models that achieve the same pointwise error as it does. We discuss this extensively in Section 3.1, as well as in Section 3.3.

  2. It then takes the form of a scientific utility, in the sense of being biologically relevant.

While Nethobench can demonstrate that a forecast preserves the structural properties of a reference signal, such preservation should ideally result in better downstream performance or more accurate conditional neural dynamics in order to be scientifically useful.

As for the transfer from more accurate conditioning to a high score, our experiment with Omnimouse demonstrates that behavior and stimulus conditioning result in higher Nethobench scores. As for the presumed transfer from a high score to better downstream performance, we still had some work to do!

We therefore designed an experiment to investigate this. We trained models to reproduce the responses of neurons that have been shown to encode a specific stimulus. In the IBL data published in [1] and previously used in S18-19, mice are exposed to visual stimuli on the left or right. The study shows strong decodability of the side of the visual stimulus in the hippocampus (Figure 9 of the eLife paper). In other words, using hippocampal neuron spike data, we can determine whether a given stimulus was on the right or left. We replicated these findings across 7 sessions from different laboratories:

SessionClassificationTrialsq FDR
NR_002481,5 %2760,0029
NR_002776,0 %4220,0029
NR_001974,0 %2650,0029
CSHL05971,9 %3380,0029
NR_003167,7 %1770,0309
UCLA03764,8 %2760,0156
CSH_ZAD_02262,2 %2860,0467

This provides us with a good framework for training models to predict neural responses, by feeding them the spiking data from a preceding window and the stimulus they are presented with ¹. We trained a set of models for each session and selected three of them based on three criteria that compare the actual spiking data with the predicted data:

  • the model that minimizes the mean absolute error (MAE)
  • the model that minimizes the Poisson log loss
  • the model that maximizes Nethobench
SessionMAENethobenchPoisson
NR_0024GRU h64 - epoch 480Stimulus-conditioned mean + history - α=1 000GRU h64 - epoch 60
NR_0027GRU h64 - epoch 260Stimulus-conditioned mean + history - α=10GRU h64 - epoch 100
NR_0019GRU h64 - epoch 240Stimulus-conditioned mean + history - α=100GRU h16 - epoch 80
CSHL059GRU h64 - epoch 480Stimulus-conditioned mean + history - α=1GRU h64 - epoch 60
NR_0031GRU h64 - epoch 180Stimulus-conditioned meanVAR α=10 000
UCLA037GRU h16 - epoch 360Stimulus-conditioned mean + history - α=1GRU h64 - epoch 120
CSH_ZAD_022GRU h32 — epoch 440Stimulus-conditioned mean + history - α=100GRU h32 - epoch 60

It is interesting to note that the choice of criterion affects the selected model family; in 6 out of 7 sessions, Nethobench selected the stimulus-conditional mean with neural history correction.

Now, to determine which criterion selects a model that most faithfully reproduces the population decodability observed in the real data, we can decode the stimulus side, but this time using the predicted neural sequences. To do this, we generated spike sequences for test trials using the models selected by each criterion for each session:

SessionTest trialsReelMAE-chosenNethobench-chosenPoisson-chosen
NR_00245572,8 %*90,9 %*100 %*98,2 %*
NR_00278467,7 %*88,0 %*73,4 %*96,3 %*
NR_00195355,8 %100 %*69,2 %100 %*
CSHL0596768,7 %82,1 %*73,1 %*98,5 %*
NR_00313564,3 %94,0 %*100 %*91,7 %*
UCLA0375549,3 %76,0 %*79,3 %*90,3 %*
CSH_ZAD_0225754,3 %89,6 %*66,7 %96,4 %*

* q FDR < 0,05

What stands out is that Nethobench is the criterion that on average most faithfully reproduces the level of decodability observed in the real data.

AuthorsOpenReview ↗

Rebuttal III - Justification of practical utility 2

Selection criterion on validationAverage absolute deviation from actual decodability on the test set
Nethobench18.4 points
MAE26.8 points
Poisson34.1 points

In this experiment, the choice of criterion therefore affects the selected family of models, and this choice has a measurable impact on a downstream analysis that is independent of the selection, thereby demonstrating the scientific value of Nethobench.

We emphasize, however, that while these results are descriptive and encouraging, we continue not to claim to have introduced a method that can accurately predict the recovery of a biological process. Even though we have empirically demonstrated that there may be a correlation between a high Nethobench score and the preservation of biological observations across two experiments.

In conclusion, thanks to this additional experience, the usefulness of Nethobench has now been validated from both a diagnostic and a (neuro)scientific perspective. We will update the manuscript accordingly.

Reviewer_ELr4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Nethobench, a benchmark for evaluating the structural realism of generated neural activity. The authors argue that traditional forecasting metrics, such as MSE and MAE, do not fully capture whether models preserve important properties of neural activity. Nethobench introduces 18 metrics grouped into five categories: distributional, temporal-spectral, relational, geometric, and state-dynamical.

The benchmark is evaluated using synthetic data and controlled perturbations, as well as real neural recordings from mice and humans. The authors also compare different forecasting models, training strategies, and conditioning inputs. The results show that models with similar prediction errors can have different structural scores, highlighting the importance of evaluating neural dynamics beyond prediction accuracy.

Critical strengths
  • The paper addresses an important limitation of traditional forecasting metrics, showing that low prediction error does not necessarily imply realistic neural dynamics.
  • Nethobench considers multiple metrics that capture different aspects of neural activity, providing a more comprehensive evaluation of neural forecasting models.
Critical weaknesses
  • The main experiments lack comparisons with recent neural forecasting models such as POCO. Furthermore, the primary architecture comparison is restricted to a single widefield dataset, with splits at the recording rather than animal level.
  • The paper does not establish whether higher structural realism leads to better downstream performance or more accurate conditional neural dynamics.
Questions for authors
  • Can the authors demonstrate the practical utility of Nethobench scores, for example, by showing whether higher structural scores correlate with better downstream performance, such as behavioral decoding?
  • How does Nethobench distinguish models that learn to predict future neural activity from models that simply generate realistic neural activity? Would a model that ignores the input history still achieve a high structural score?
  • Are the model rankings consistent across different datasets or held-out animals? Since the main architecture comparison uses only one widefield dataset with recording-level splits, I am unsure whether the rankings would remain the same across different datasets or animals.
  • Would the improvements from adding behavioral and video information still hold if the model only had access to information available before the prediction interval?

My main concerns are the practical utility of the proposed metrics and the limited evaluation of model comparisons across datasets. I would be willing to increase my score if the authors adequately address these concerns.

5 comments
AuthorsOpenReview ↗

Rebuttal I - Weakness 1

We thank the reviewer for their review. We have addressed each concern below and hope that the clarifications and additional experiments will lead the reviewer to reconsider their score.

Critical Weaknesses

  1. Model comparisons and generalization

We do not, in fact, formally compare our models (of Section 3.3) with the “standard” architectures in neural forecasting (POCO, TSMixer, NetFormer,...). This is primarily because we are not proposing a new model architecture here, the superiority - or at least the uniqueness - of which we would then have to defend relative to the SOTA. However, this paper introduces a new criterion for evaluating such models. A criterion that, as you pointed out, “addresses an important limitation of traditional forecasting metrics.” Our task in this paper is therefore to demonstrate that our framework effectively addresses this limitation.

We have chosen not to focus our demonstration on showing that it replicates the leaderboard among the SOTA neural forecasting models, since we argue here that the leaderboard itself may be flawed (precisely because it relies on pointwise error comparisons).

Instead, we decided to compare several models trained on a standardized task, with architectures that incorporate components from SOTA models, but which performances are comparable on a one-to-one basis because they share similar sizes, training budgets, and backbones (for the Transformers). For example, like POCO, we used blocks combining multi-head auto-attention; like TSMMixer and NetFormer, we used cross-region information. An extension of this paper could indeed propose a current ranking of neural forecasting models, based on Nethobench scores, MAEs, and data-specific structural scores.

Regarding the applicability of Nethobench beyond widefield calcium imaging, we conducted additional analyses to demonstrate the robustness of the metrics to other modalities, including 2p imaging, intracranial EEG, and electrophysiological data, as shown in Section 3.2, Figs. S17–S19. Nor did we consider it essential to perform a check using splits at the animal level to demonstrate generalizability across animals within the context of the widefield experiment, as we had already conducted an even stricter version of such a check when we demonstrated partial conservation of Nethobench scores across labs in IBL repeated-site experiments (Figures S18-S19).

In conclusion, we agree with the reviewer that a new evaluation criterion should demonstrate:

  • Its relevance to SOTA design choices. Hence, we evaluate architectures that share design choices with POCO, TSMixer, and NetFormer.
  • Its applicability across recording modalities. Hence, we apply Nethobench to widefield and 2p calcium imaging, intracranial EEG, and Neuropixels recordings, as well as to six synthetic dynamical systems and a synthetic BOLD observation model.
  • Its applicability across animals. Hence, in the IBL repeated-site experiment, the SSM consistently scored higher than the GRU on temporal-spectral structure, and the GRU consistently scored higher on state dynamics, across within-session, held-out-session, and held-out-laboratory evaluations (Fig. S18e).
AuthorsOpenReview ↗

Rebuttal II - Weakness 2

  1. Scientific utility

We agree with the reviewer that, in a biological context, the strength of such an evaluation framework cannot be reduced solely to its ability to help understand what a generator mathematically preserves from a process; rather, such dynamic preservation should ideally result in better downstream performance or more accurate conditional neural dynamics in order to be scientifically useful.

As for the transfer from more accurate conditioning to a high score, our experiment with Omnimouse demonstrates that behavior and stimulus conditioning result in higher Nethobench scores. As for the presumed transfer from a high score to better downstream performance, we still have some work to do!

We therefore designed an experiment to investigate this. We trained models to reproduce the responses of neurons that have been shown to encode a specific stimulus. In the IBL data published in [1] and previously used in S18-19, mice are exposed to visual stimuli on the left or right. The study shows strong decodability of the side of the visual stimulus in the hippocampus (Figure 9 of the eLife paper). In other words, using hippocampal neuron spike data, we can determine whether a given stimulus was on the right or left. We replicated these findings across 7 sessions from different laboratories:

SessionClassificationTrialsq FDR
NR_002481,5 %2760,0029
NR_002776,0 %4220,0029
NR_001974,0 %2650,0029
CSHL05971,9 %3380,0029
NR_003167,7 %1770,0309
UCLA03764,8 %2760,0156
CSH_ZAD_02262,2 %2860,0467

This provides us with a good framework for training models to predict neural responses, by feeding them the spiking data from a preceding window and the stimulus they are presented with ¹. We trained a set of models for each session and selected three of them based on three criteria that compare the actual spiking data with the predicted data:

  • the model that minimizes the mean absolute error (MAE)
  • the model that minimizes the Poisson log loss
  • the model that maximizes Nethobench
SessionMAENethobenchPoisson
NR_0024GRU h64 - epoch 480Stimulus-conditioned mean + history - α=1 000GRU h64 - epoch 60
NR_0027GRU h64 - epoch 260Stimulus-conditioned mean + history - α=10GRU h64 - epoch 100
NR_0019GRU h64 - epoch 240Stimulus-conditioned mean + history - α=100GRU h16 - epoch 80
CSHL059GRU h64 - epoch 480Stimulus-conditioned mean + history - α=1GRU h64 - epoch 60
NR_0031GRU h64 - epoch 180Stimulus-conditioned meanVAR α=10 000
UCLA037GRU h16 - epoch 360Stimulus-conditioned mean + history - α=1GRU h64 - epoch 120
CSH_ZAD_022GRU h32 — epoch 440Stimulus-conditioned mean + history - α=100GRU h32 - epoch 60

It is interesting to note that the choice of criterion affects the selected model family; in 6 out of 7 sessions, Nethobench selected the stimulus-conditional mean with neural history correction.

Now, to determine which criterion selects a model that most faithfully reproduces the population decodability observed in the real data, we can decode the stimulus side, but this time using the predicted neural sequences. To do this, we generated spike sequences for test trials using the models selected by each criterion for each session:

SessionTest trialsReelMAE-chosenNethobench-chosenPoisson-chosen
NR_00245572,8 %*90,9 %*100 %*98,2 %*
NR_00278467,7 %*88,0 %*73,4 %*96,3 %*
NR_00195355,8 %100 %*69,2 %100 %*
CSHL0596768,7 %82,1 %*73,1 %*98,5 %*
NR_00313564,3 %94,0 %*100 %*91,7 %*
UCLA0375549,3 %76,0 %*79,3 %*90,3 %*
CSH_ZAD_0225754,3 %89,6 %*66,7 %96,4 %*

* q FDR < 0,05

What stands out is that Nethobench is the criterion that on average most faithfully reproduces the level of decodability observed in the real data.

Selection criterion on validationAverage absolute deviation from actual decodability on the test set
Nethobench18.4 points
MAE26.8 points
Poisson34.1 points

In this experiment, the choice of criterion therefore influences the selected model family, and this choice has a measurable impact on a downstream analysis independent of the selection process.

We emphasize, however, that while these results are descriptive and encouraging, we continue not to claim to have introduced a method that can accurately predict the recovery of a biological process. Even though we have empirically demonstrated that there may be a correlation between a high Nethobench score and the preservation of biological observations across two experiments.

AuthorsOpenReview ↗

Rebuttal III - Questions

Questions

  1. cf response to Weakness 2
  2. Yes. A model can score highly on Nethobench even if it ignores neural history. In our stimulus-side experiment, a conditional-mean model that predicts the average response to each stimulus ranked second by Nethobench in six of seven sessions and first in the seventh. In those six sessions, adding a correction based on the trial’s neural history yielded the highest score. Nethobench measures how well forecasts preserve the structure of recorded activity. A high score alone cannot establish that a forecast is specific to the observed history.
  3. As discussed in the response to Weakness 1, the purpose of Section 3.3 was not to create a leaderboard of models that generalize across animals, but Nethobench’s capabilities in this regard were investigated in Figures S17–19.
  4. As described in Section 3.2 and the caption of Fig. 3a, the video and behavioral inputs include observations from the prediction interval. We used this setting to test whether Nethobench detects differences in the neural structure preserved under different conditioning inputs. We did not test whether the gains persist when only pre-interval observations are available. Does the reviewer think that there is a specific aspect of Nethobench’s evaluation that it would help clarify?

[1] IBL et al, "Reproducibility of in vivo electrophysiological measurements in mice", eLife, 2025

¹ context: -100 ms to +100 ms and prediction from +100 ms to +1000 ms. We trained the GRU with different values of h; the VAR with different values of alpha; conditional mean (just the stimulus); conditional mean + history: starting from the average response to the stimulus and learning to adjust based on the observed activity for each trial -> regularized linear correction

Comments on the paper

2 comments

Dear Reviewer,

Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at the authors' rebuttal and share your thoughts in the discussion?

Many thanks for your continued help!

Dear Reviewers and Area Chair,

We would first like to thank you again for engaging so thoughtfully with us throughout the discussion. Your feedback substantially improved the manuscript (now uploaded!).

It allowed us to complement our work with 3 new validation experiments and helped us rethink the story, to cleanly present the work so that each experiment motivates the next. The flow and structure of the Section 3 has notably been updated, so that validation experiments serve the purpose of justifying the utility of Nethobench, which we claim should be both diagnostic and scientific. To this end, Section 3.1 asks which properties of recorded activity the metrics detect and when those properties change how we rank models. Section 3.2 follows with a question about scientific utility: when the selection criterion favors a different model, does its forecast more faithfully reproduce stimulus-related activity?

Here are the 3 new experiments and where to find them in the revised manuscript:

  • Comparison with Dynamical Similarity Analysis: Appendix C.2.1 compares angular DSA with Nethobench’s temporal-spectral and state-dynamical scores on controlled oscillations and forecasts from eight widefield models.
  • Model rankings across datasets and animals: Section 3.1 and Table 1 compare rankings by MAE and Nethobench for 3 published architectures and 2 baselines across the 4 datasets used throughout the paper. Section 3.1 also reports a ranking reversal that recurred in 15 of 17 held-out IBL mice.
  • Stimulus-side decoding in IBL: Section 3.2 and Table 2 test how closely forecasts reproduce the decodability of stimulus side in recorded hippocampal activity across 23 sessions. Appendix B.4.1, Table 5 reports the trial-reassignment control.

If the revisions have addressed your concerns, we would be grateful if you'd consider raising your scores. We appreciate the time and care you have given the paper and wish you all a fruitful discussion in the next stage of review!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.