- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces IRCB '26, a standardized benchmark for evaluating multi-region neural communication models. The gap they address is that modles published in this field all introduce thier own syntehtic dataset tailored to their specific model, preventing systematic comparison and potentially hiding failure modes. The benchmark contributes 42 synthetic datasets organized into three families of data-generating networks (DGNs): DELAYED-MEMORY, RELAY-DECISION, and a COGNITIVE TASK SUITE of 40 networks trained on 20 cognitive tasks adapted from Yang et al. 2019. Each DGN is a set of interconnected vanilla RNNs where each RNN represents a brain region. The benchmark also introduces a suite of evaluation metrics that separately assess neural activity reconstruction, effectome recovery, and communication fidelity and specificity. The authors demonstrate the benchmark on two communication models, reduced-rank regression (RRR) and MR-LFADS, revealing distinct limitations of each. Datasets, code, and evaluation scripts are publicly released.
Critical strengths
-
Addresses a genuine and important gap. There is currently no standardized benchmark for multi-region communication models, and this fragmentation is a real problem for the field. The paper is well-placed in the literature and the motivation is compelling.
-
The separation of evaluation metrics into activity reconstruction, effectome recovery, and communication fidelity/specificity is conceptually important and practically useful. The paper demonstrates convincingly that these can dissociate, which means existing single-metric evaluations may be misleading.
Critical weaknesses
-
All DGNs are continuous-time vanilla RNNs. Models like mp-srSLDS assume the underlying dynamics switch between discrete linear regimes — a vanilla RNN DGN won't naturally expose that model's failure modes under regime changes. Gaussian process models like DLAG, MRM-GP1 and ADM2 make very different assumptions about temporal smoothness. The benchmark may be inadvertently tailored to the assumptions of continuous nonlinear dynamical models (like MR-LFADS) than to GP-based or switching models, which could bias which models look good on this benchmark.
-
The claim is too broad. The paper claims to provide evaluation metrics that "standardize evaluations" but the benchmark specifically targets three failure modes. It does not cover:
- Failure modes specific to Gaussian process or switching dynamical systems models
- Non-stationarity and session-to-session drift
- Robustness to model hyperparameter choices
- Scaling to many regions
- Closed-loop sensorimotor dynamics (acknowledged in limitations)
- Heterogeneous region sizes
- Models' ability to handle different timescales across regions The benchmark is a useful step toward standardization, but calling it a standardization of evaluations generally is an overreach. A more accurate framing would be "a benchmark targeting specific classes of failure modes in specific communication models."
- Evaluation metrics: A correlation coefficient or normalized RMSE might be more informative rather than cosine similarity for effectome. Also the measure for communication fidelity is linear and a non-linear measure could be more useful here. a model that encodes the communication content in a highly nonlinear representation could look artificially poor. The related work section mentions the main model families but does not systematically go through each model's synthetic dataset and say "this model used X type of dataset and evaluated with Y metric, and our benchmark covers/doesn't cover this." That kind of structured comparison would be expected in a benchmark paper and would also justify the claim that existing testbeds are insufficient. The limitations section touches on this but it's not done at the level of rigor I would expect. The paper covers most of what the community uses and adds the effectome framework. One thing notably absent from the benchmark is subspace alignment (principal angles between inferred and ground-truth communication subspaces), which is central to how RRR and DLAG-family models think about communication.
In my opinion a strong benchmark paper should:
- Apply the benchmark to a representative set of existing models (not just 2)
- Show which models fail on which dataset conditions and explain why based on their design choices
- Compare the DGN structures to those used in each prior model's own paper and show the benchmark is a superset or a more challenging version
Minor weaknesses:
-
There are some missing related work from other family of models that authors did not mention: MRM-GP1, ADM2, CREIMBO3, CTAE4
-
The authors should mention that their data is solely spike rate and counts, real neural recordings often include LFP, multiunit activity, and calcium imaging signals, none of which are represented here.
-
Missing citations for restrictions of existing synthetic datasets, around line 57. Authors need to cite specific papers whose datasets exemplify these limitations.
References
1: MRM-GP (ICML 2024): Multi-Region Markovian Gaussian Process: An Efficient Method to Discover Directional Communications Across Multiple Brain Regions
2: ADM (ICML 2025): Learning Time-Varying Multi-Region Brain Communications via Scalable Markovian Gaussian Processes
3: CREIMBO (ICLR 2025): Cross-Regional Ensemble Interactions in Multi-view Brain Observations
4: CTAE (ICLR 2026): Coupled Transformer Autoencoder for Disentangling Multi-Region Neural Latent Dynamics
Questions for authors
-
Why were DLAG, mDLAG, mp-srSLDS, and MR-SDS not benchmarked? What prevented their inclusion?
-
How does the benchmark scale to larger numbers of regions, and was computational cost a factor in the design choices?
-
How do the authors justify that vanilla RNN DGNs expose failure modes relevant to switching or GP-based models, given these models have fundamentally different inductive biases?
7 comments
(B) The claim is too broad
The Reviewer argued that our claim, that our evaluation metrics support “standardized evaluations,” is too broad. We now replace “standardized” with “consistent” in this usage. Elsewhere, we use “standardized” in its conventional sense for machine-learning benchmarks: a fixed evaluation protocol that makes results comparable across models, as in RobustBench [8], the Neural Latents Benchmark [9] and BrainWideBench [10], rather than coverage of every condition a model may face.
We appreciate the Reviewer’s point that our benchmark does not yet cover many challenging conditions, and we thank the Reviewer for identifying them. Acknowledging that no benchmark can possibly cover all conditions that may be encountered, we now expand our benchmark to cover many of the conditions that the Reviewer surfaced.
- We are fitting the benchmarked communication models to brain-wide electrophysiology recordings in mice performing a sensorimotor memory-guided decision task [1] (also in response to Reviewer g3JZ), and we will release a preprocessing pipeline that transforms these publicly available data into the same format as our original released datasets, so that every model fit to them can be evaluated using our benchmark evaluation pipeline. These data bring:
- Non-stationarity
- Sensorimotor dynamics
- Heterogeneous region sizes
- Different timescales across regions
- Because ground-truth communication is not available in real recordings, evaluations on these data focus on reconstruction quality. To bring ground-truth communication to these data, the data-driven DGNs described in (A) add:
- GP and switching dynamical models
- Following Reviewer vAUC’s suggestion, we have trained a new multitask DGN that performs all 20 Yang et al. [11] cognitive tasks, and we will add multitask DGN variants that vary in which regions receive inputs and produce readouts, the number of regions, the number of units in each region, the recurrent unit type and the noise level. These variants add:
- Robustness to (DGN) model hyperparameter choices
- Heterogeneous region sizes
We also appreciate the Reviewer’s suggestion that an appropriate framing would be “a benchmark targeting specific classes of failure modes in specific communication models.” We will adopt this description.
Timeline. Revised paper (at close of discussion period): the Reviewer’s framing, fits to the real recordings with reconstruction results, and the new multitask DGN. Camera-ready version: the data-driven DGNs and multitask DGN variants (fitting and tuning now seven benchmarked models to these new datasets requires more compute than the discussion period allows).
(C) Evaluation metrics
The Reviewer suggested a correlation coefficient to complement cosine similarity for effectome evaluation, a nonlinear measure of communication fidelity, and subspace alignment. Following these suggestions, we are adding three metrics:
- Effectome recovery: Pearson correlation, alongside cosine similarity.
- Nonlinear communication fidelity: Nonlinear decoding of the true messages, fit and scored on separate trials, as a secondary metric, comparing kernel ridge regression and a small multilayer perceptron as the decoder. We agree that a model encoding the right content in a nonlinear form “could look artificially poor” under a linear measure. Linear decoding remains the primary score, because the form of a message matters as well as its content: the same information arriving as a pulse or as a step drives the receiving region differently, so a model that recovers the content but not the form may infer different dynamics in that region.
- Subspace alignment: Principal angles between inferred and ground-truth communication subspaces, computed within the high-variance subspace of source-region activity, starting with the models whose messages are linear projections of that activity (observed neurons for RRR [12], inferred rates for MR-LFADS [13]). For switching latent-state models, whose messages are functions of source-region latents, we will map the message into source-neuron space through the model’s emissions: for mp-srSLDS [2] the message is linear, giving an exact subspace per discrete state; for MR-SDS [3] it is nonlinear, so we will use the dominant subspace of its Jacobian, averaged over inferred states. For GP-family models [4–6], no explicit messages are defined, but, as for RRR, the fitted parameters specify the source-neuron subspace that predicts target-region activity.
Timeline. Revised paper (at close of discussion period): all three metrics for the benchmarked models, with subspace alignment for RRR and MR-LFADS. Camera-ready version: subspace alignment for the remaining models.
(D) Prior testbeds
The Reviewer asked for a structured comparison of the synthetic datasets and metrics used to validate each prior model, and for the limitations of those existing approaches. We agree that this would be valuable, and to address this, we are adding appendix tables providing this information, showing that our benchmark is largely a superset of prior testbeds. In particular, prior testbeds typically lack two features central to ours: dynamics that perform behaviorally relevant computations, and unobserved influences, such as external inputs and unrecorded regions, that can confound recovery of communication structure. We will summarize these tables in the main text alongside the requested citations.
Timeline. Revised paper (at close of discussion period): the appendix tables, their summary in the main text, and the citations.
(E) Expand benchmarked models
The Reviewer asked why DLAG, mDLAG, mp-srSLDS and MR-SDS were not benchmarked (Q1). In response, we are extending the set of benchmarked communication models to include mp-srSLDS [2], MR-SDS [3], ADM [4], DLAG [5] and mDLAG [6]. We do not plan to include MRM-GP [14] because its released codebase supports only two regions. Our benchmark code will allow MRM-GP to be benchmarked in future work should its authors or others further develop MRM-GP to support >2 regions.
Timeline. Revised paper (at close of discussion period): results on four core datasets, namely Delayed-Memory, Relay-Decision, a new multitask network trained to perform all 20 Yang et al. [11] cognitive tasks (per Reviewer vAUC), and the public brain-wide Neuropixels electrophysiology recordings of Chen et al. [1] (per Reviewer g3JZ). Camera-ready version: results on all benchmark datasets (fitting and tuning now seven benchmarked models to more than 40 datasets requires more compute than the discussion period allows).
(F) Scaling and computational cost
The Reviewer asked how the benchmark scales to more regions and whether computational cost shaped the design (Q2). Computational cost did shape the design: we kept to four or fewer regions in our DGNs so that every communication model could be fit and appropriately tuned within a practical compute budget. The deep learning models are the main bottleneck, with MR-LFADS and MR-SDS taking 24–72 hours to fit a single model on a given benchmark dataset, and hyperparameter tuning often meant fitting tens or hundreds of variants of each communication model on each dataset.
The benchmark itself is not limited to four regions: the DGN training, communication model training, data format and evaluation pipeline accommodate any number of regions. However, the cost of fitting communication models does scale with the number of regions. The new multitask DGN variants described in (B) include DGNs with more regions (likely 6–8), which will show how each model’s accuracy and fitting cost scale.
Timeline. Revised paper (at close of discussion period): timing and tuning details for every benchmarked model. Camera-ready version: scaling information enabled by multitask DGN variants with >4 regions (fitting and tuning now seven benchmarked models to these new datasets requires more compute than the discussion period allows).
We thank the Reviewer for the detailed and expert review. We were encouraged that the Reviewer found that our paper “addresses a genuine and important gap” and that our benchmark is “conceptually important and practically useful.” Below we respond sequentially to each of the Reviewer’s points. Following the Reviewer’s suggestions, we are benchmarking additional communication models, adding evaluation metrics, analyses and data-generating networks, and systematically comparing our benchmark with prior testbeds. Some of this work is still under way as we provide an initial response to Reviewers to encourage discussion. We will report conclusively on these ongoing threads at the end of the discussion period.
(A) Model-class bias
The Reviewer was concerned that, because every data-generating network (DGN) is a vanilla RNN, the benchmark may be inadvertently tailored to continuous nonlinear models such as MR-LFADS, and asked (Q3) how we might expose failure modes of other communication model classes built on switching or Gaussian-process (GP) dynamics. We agree that the benchmark should include DGNs that match the assumptions of switching and GP models, and we are therefore adding data-driven DGNs of both kinds. We are now fitting these models to real brain-wide electrophysiology recordings in mice performing a sensorimotor memory-guided decision task [1]. We will then use each fitted model as a DGN to create synthetic datasets that capture features of real neural data, with communication known by construction:
Original rationale for RNN DGNs. RNNs make relatively few assumptions (e.g., compared to switching linear or GP models) and are known to be universal function approximators [7]. As such, RNNs can be trained to reproduce the dynamics of a switching linear network or a network with GP latents. The reverse is not true in general.
Empirically, we indeed observe switching behavior in our Cognitive Task Suite DGNs. These DGNs’ dynamics switch operating regimes at the go cue, when transitioning from the delay to response epochs. For example, in DGNs trained on multisensory decision-making, the leading principal axis of the DGN dynamics rotates by about 60° across the go cue, and linear dynamics fit within one phase (R2>0.83) transfer poorly to the other (cross-phase R2 as low as 0.28).
Timeline. Revised paper (at close of discussion period): the switching and GP DGNs fit to the Chen et al. recordings, with their fitting procedures and reconstruction quality, and the empirical switching analysis. Camera-ready version: synthetic datasets sampled from these DGNs, with code and benchmark results (fitting and tuning now seven benchmarked models to these new datasets requires more compute than the discussion period allows).