acceptodds
Under review as a conference paper at ICLR 2027

Scryer: A scalable framework for forecasting neural population activity

Abstract

Forecasting objectives have become a central component of modern large-scale models in language and video, as they enable the learning of structure in complex temporally evolving data. However, applying this paradigm to neural recordings from the brain remains challenging because neural activity is measured over varying populations of neurons and exhibits high-dimensional, sparse, and nonstationary dynamics. In this work, we introduce Scryer, a scalable framework for neural forecasting designed to learn transferable representations from heterogeneous neural recordings, built around a hierarchical multiscale architecture with learnable channel embeddings and population-conditioned retrieval. We evaluate Scryer across both electrophysiology and calcium imaging datasets spanning multiple brain regions and species. Finally, we show that Scryer's unit-identity embeddings, learned purely from the forecasting objective, support brain-region decoding, substantially exceeding an identity representation derived from other methods. Together, these results support forecasting-driven architectures, independent of any single auxiliary training objective, as a viable approach for learning representations from neural population activity.

Primary area:
applications to neuroscience & cognitive science
Keywords:
neural population dynamics, spiking activity, Neuropixels, time series forecasting, cross-attention, population-conditioned retrieval, calcium imaging, neuroscience

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DnjgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SCRYER, a neural population forecasting framework that combines multiscale temporal decomposition, neuron-level attention, and retrieval-conditioned candidate forecast selection.

  • Method: Each neuron's history is decomposed into multiple temporal-frequency components to generate candidate forecasts. Population context, obtained through attention over neuron identity and activity embeddings, is used to weight these candidates and produce an additive correction.
  • Experiments: The authors evaluate SCRYER on IBL electrophysiology and calcium-imaging datasets from mice, C. elegans, and zebrafish, reporting consistent forecasting improvements over existing baselines.
  • Biological analysis: The authors demonstrate brain-region decoding from learned embeddings and investigate neurons' relative dependence on individual history versus population activity.
Critical strengths

1. Broad experimental evaluation with consistent improvements.

The paper evaluates SCRYER across multiple species, recording modalities, and established baselines. The model consistently achieves competitive or superior forecasting performance.

2. Potential biological insights from forecasting representations.

The analysis of history-driven and population-driven neurons provides a potentially interesting perspective on neural population dynamics.

Critical weaknesses

The main weaknesses are stated below, and the authors can check the Questions For Authors part for the details.

1. Incremental performance and insufficient validation of the key contributions.

The proposed architecture combines established techniques, including temporal decomposition, neuron identity embeddings, self-attention, and weighted forecast aggregation. Similar neuron-level attention mechanisms have already been explored in previous work, such as NetFormer.

The forecasting improvements over the strongest baselines are relatively modest, generally around 1–5%.

More importantly, the central contribution of retrieval-conditioned candidate selection is not directly validated. The current ablations do not sufficiently isolate the contributions of candidate generation, population-conditioned selection, and the additive correction network.

2. Limited novel biological insights and insufficient controls.

  • Figure 3: Brain-region decoding has been extensively investigated in previous work and is not a novel task. More importantly, the simple ISI-based baseline reported in BrainWideBench (Table 3) achieves better brain-region classification performance than SCRYER. Although the evaluation settings are not identical, this raises questions about whether deep forecasting representations provide additional biological information beyond conventional spike-train statistics.
  • Figure 5: The history-driven versus population-driven analysis is potentially interesting, but similar masking experiments could be performed using existing forecasting models, such as POCO. The masking procedures are not symmetric and may introduce distribution shifts. Furthermore, the reported dependence on history versus population correlates with firing statistics, making it unclear whether the observed patterns reflect biological properties or model-specific effects.

Overall, the experiments do not sufficiently demonstrate novel biological discoveries uniquely enabled by SCRYER.

3. Insufficient evidence supporting the scalability claim.

Scalability is emphasized in the title, yet the paper provides limited evidence of meaningful scaling behavior across data, model capacity, and computational resources.

  • Data scaling: Figure A2 shows that increasing the Allen training data from 200 to 1,334 sessions (6.7×) reduces MSE by only 0.003 (0.32%). Although statistically significant, this improvement is practically marginal. Figure 4 shows more substantial gains on IBL, but comparisons with the scaling behavior of existing methods are missing.
  • Model scaling: The paper does not investigate whether increasing model capacity leads to further performance improvements.
  • Computational scalability: There is no systematic evaluation of training/inference time or memory consumption as neuronal population size increases. This is particularly relevant because neuron-level attention has quadratic complexity in the number of neurons.

Furthermore, the zebrafish experiments operate on 512 PCA components rather than the original individual-neuron recordings. While this follows the existing POCO benchmark protocol, it does not directly establish scalability of neuron-level retrieval to large neuronal populations.

Overall, the current evidence is insufficient to convincingly substantiate the scalability claim in the title.

Questions for authors

Major Questions

Q1. Contribution of candidate generation and selection.

Could the authors provide additional ablations to isolate the contributions of candidate generation and selection in Figure 1(d)?

Specifically:

  1. Correction-only: Directly predict using the existing correction MLP (δ\delta), without the candidate branch.
  2. Single MLP: Replace the multiple candidate forecasting heads with a single MLP using the same temporal features, while retaining the correction network.
  3. Uniform selection: Retain the candidate forecasts and correction network, but use uniform weights instead of population-conditioned weights.
  4. Energy-based selection: Retain the candidate forecasts and correction network, but use only the existing energy-based SE weights, without population-conditioned selection.

These comparisons would clarify whether the proposed candidate generation and population-conditioned selection mechanisms independently contribute to forecasting performance.

Q2. Contribution and interpretation of neuron-level retrieval.

Could the authors provide population-level analyses of the learned attention patterns, including:

  • Self-attention weights and attention entropy across neurons.
  • Whether different target neurons attend to different source neurons.
  • The relationship between attention weights and neuronal activity correlations.

These analyses would provide stronger evidence of what the proposed neuron-level retrieval mechanism actually learns, beyond the illustrative examples in Figures 2 and A5.

Q3. Novel biological insights beyond brain-region decoding.

Brain-region decoding has already been extensively studied, and the ISI-based baseline in BrainWideBench Table 3 achieves better classification performance than SCRYER under its reported evaluation setting.

Beyond brain-region classification, could the authors demonstrate biological findings that are uniquely enabled by SCRYER or are substantially stronger than those obtainable using existing methods?

Such results would help establish the biological significance of the proposed approach beyond incremental improvements on an established decoding task.

Q4. Robustness and model dependence of Figure 5.

Could the authors perform similar history-only and population-only analyses using existing forecasting models, particularly POCO?

Additionally, could the authors investigate whether the conclusions remain consistent under alternative masking strategies that reduce distribution shift, and clarify how neurons with near-zero full-model predictive performance are handled?

These analyses would help determine whether the reported history-driven and population-driven patterns are robust neural properties or model-dependent observations.

Q5. Scalability and computational efficiency.

Given the emphasis on scalability in the title, could the authors provide systematic scaling experiments that evaluate:

  • Forecasting performance as the number of training sessions or recorded neurons increases.
  • Training/inference time, GPU memory consumption, and model parameter counts.
  • Comparisons with existing methods, particularly POCO, under comparable computational budgets.

In particular, POCO has already evaluated forecasting at single-cell resolution on the original zebrafish recordings, whereas SCRYER only reports experiments using 512 PCA components. Could the authors clarify whether SCRYER can achieve comparable population-level scalability without PCA-based dimensionality reduction?

Minor Question

1. Figure 3: Interpretation of the LDA visualization.

The LDA projection uses brain-region labels to construct the discriminant axes. However, the held-out neurons in Figure 3(b) do not appear to exhibit clear anatomical separation, despite being projected onto the same axes as Figure 3(a).

The authors should clarify how strongly this visualization supports cross-animal anatomical generalization, rather than relying primarily on qualitative clustering.

2. Line 49: Unclear description of previous methods.

The statement "mixing of neurons early before training" is inaccurate or at least misleading.

In POCO, neuronal information is aggregated through a learned encoder during the forward pass, rather than being mixed before training. The authors should clarify this statement and more precisely distinguish SCRYER's retrieval mechanism from previous population-conditioned forecasting architectures.

3. Related work and additional baselines.

Recent related work for neural forecasting, including OmniMouse (ICLR2026) and CalM (ICML2026), is not discussed. Could the authors clarify how SCRYER differs from these approaches and consider additional forecasting comparisons where datasets and evaluation protocols can be matched?

4. Calcium forecasting metrics and preprocessing.

The calcium-imaging results are evaluated mainly using MSE and MAE. Could the authors also report per-neuron Pearson correlation to assess the prediction of temporal activity fluctuations? Moreover, calcium traces are z-scored using statistics from the full recording before temporal train/validation/test splitting. Could the authors clarify the implications of using information from held-out periods, and ideally report results with normalization fitted on training data only?

5. Reproducibility.

Appendix C.2 reports the POCO architecture used for electrophysiology but does not clearly specify how its output parameterization and training objective were adapted to spike counts. Does POCO use the same log-rate output and Poisson loss as SCRYER? Could the authors complete run configurations for SCRYER to support reproducibility?

The reviewer would update the score if reasonable answers are provided.

9 comments

Response to Reviewer Dnjg

We thank the reviewer for the detailed and constructive review. We have conducted additional experiments to address your feedback, including:

  • Ablations of candidate selection (Q1, W1)
  • Analysis of attention weights (Q2)
  • Retrieval initialization and reward-state attention (Q3, W2)
  • Masking analysis on POCO (Q4)
  • Data and model scaling analysis (Q5, W3)
  • Comparisons with CalM and OmniMouse (Minor 3)
  • Per-neuron correlation and a train-only normalization check (Minor 4)

1. Validation of the key contributions (W1, Q1–Q2)

While SCRYER builds on established components, including the neuron-level embeddings of POYO, its main architectural innovation is the coupling of neuron-level retrieval with forecast selection. Each target neuron attends to individual co-recorded neurons and uses the retrieved information to select among candidate forecasts derived from its own multiscale history. POCO and OmniMouse instead compress population activity into shared latents, which prevents direct neuron-to-neuron querying. NetFormer also attends between neurons, but mixes their activity directly into the forecast and is trained separately for each recording. SCRYER thus enables population-informed forecast selection and direct analysis of neuron-to-neuron attention within a single model shared across animals.

Contribution of forecast candidate generation and selection (Q1)

We thank the reviewer for these suggestions. We ran the four suggested ablations (correction only, single MLP, uniform selection, energy weights only) on the 29 held-out sessions of the IBL Brain-wide benchmark. The full model performs best (D² 0.166) and every ablation lowers it. The largest drop was for the correction-only ablation (0.150), followed by the single MLP (0.159) and uniform weights (0.159), and then the energy weights only (0.161). The gain therefore comes from generating candidate forecasts and selecting among them with population-conditioned weights.

Candidate weighting (IBL, from scratch)Full modelCorrection onlySingle MLPUniform weightsEnergy weights only
Mean D² (↑)0.1660.1500.1590.1590.161

Contribution and interpretation of neuron-level retrieval (Q2)

We thank the reviewer for their suggestions. We performed all suggested analyses on the 29 held-out IBL recordings. Most neurons attend to a large fraction of the population but sharpen their focus on a small subset, and that subset changes over the recording.

Self-attention weights and attention entropy across neurons. We found that the distribution of attention in most recordings is heavy tailed. The four most-attended neurons receive 18% of a target's attention, 4.5 times its attention to the remaining neurons. The median normalized entropy is 0.89 and 0.91 for the two layers (1 for uniform attention), so most targets attend broadly yet concentrate on a few sources. Results in this direction would allow choosing most appropriate graph types for the models of brain neural networks (previously the contributions here were largely theoretical) and building heterogeneous models of neural activity that use subgraphs representing different cell types.

Whether different target neurons attend to different source neurons. To show that each target has its own attention pattern, we measured the effective rank of each recording's attention matrix after removing uniform attention. If all targets attended to the same neurons the rank would be 1. Instead, the effective rank is 28 and 33 for the two attention layers in a typical recording of 102 neurons. Part of the attention is still shared, since one component common to all targets explains 66–79% of the matrix, reflecting sources that every target favours, such as fast-firing neurons. The remaining 21–34% differs between targets, and most of it lies in four further components, so the target-specific part is structured rather than noise.

The relationship between attention weights and neuronal activity correlations. To show that attention maps of fixed target neurons change over time, we compared its four most-attended sources between two one-second windows drawn at random from the test period. Their Jaccard index (sources in both sets divided by sources in either set, 1 for identical sets and about 0.02 for random sets) is 0.16, so one source is kept and three change with the ongoing activity. To test whether attention merely tracks correlation, we removed at test time either a target's four most-attended sources or its four most-correlated sources. The drop in D² was 0.047 (95% CI 0.027–0.072) for the attended sources, 0.011 for the correlated sources and 0.003 for random sources. The gap remains at a matched firing rate (0.035 vs 0.013 on the 281 targets where both sets fire at the same rate). Retrieval therefore selects sources that matter for the forecast beyond what correlation or firing rate explain.

2. Novel biological insights beyond brain-region decoding (Q3, W1)

2.1. "Could the authors demonstrate biological findings that are uniquely enabled by SCRYER or substantially stronger than those obtainable with existing methods?"

Thank you for raising this point. While brain region decoding has been investigated in prior works, we want to clarify that the main goal of that set of experiments is to show that our model has learned a meaningful representation of the anatomy using its forecasting loss alone. In contrast, the methods in the Inductive task setting in Brainwidebench have been trained and extensively tuned for the problem of brain region decoding, so it is an inherently different problem setting. Under the same setting as ours, POCO and MtM learn significantly less information about the brain region from their forecasting and masking losses, respectively.

Additionally, due to SCRYER’s ability to directly query information from individual neurons in the population (i.e., neuron-to-neuron attention), we can quantify predictive information that comes from a source neuron to forecast a target neuron’s future activity. This produces a graph which encodes the neuron-to-neuron attention that depends on the neuron identity and activity of the neurons in the context window. Building such graphs to study functional connectivity is an outstanding problem in neuroscience and we believe that SCRYER provides a step in this direction.

We note that this neuron-to-neuron attention analysis is not possible with models like POCO and OmniMouse, where the population state is compressed into a set of latent tokens and thus the queries are across the entire population rather than individual neurons.

Validation against the mesoscale connectome. We first tested how the learned attention maps in SCRYER align with the known anatomical connectome. We computed the target-specific attention between brain regions (each source's attention normalized by the mean attention it receives from all targets, which removes the component shared by all targets) and compared it with the mesoscale connectome of the Allen Mouse Brain Connectivity Atlas, an independent measurement of axonal projections made by viral tracing across a large cohort of mice (Oh et al., 2014, Knox et al., 2019). In our analysis, region pairs that are more densely connected attend to each other more (Spearman 0.47 over 2,104 region pairs, permutation p < 0.001), and the agreement remains after controlling for distance, shared probes, parent region and firing rate (0.28). As a baseline we computed jitter-corrected cross-correlograms (CCG, Siegle et al., 2021), the standard spike-timing estimate of connectivity, on the same recordings and the same region pairs. After the same controls, SCRYER's learned attention agrees with the connectome better than CCG at the fine level (0.28 against 0.22) and at the coarse level of ten major brain divisions (0.47 against 0.23, where CCG is not significant). Neither method recovers the direction of projections, so the agreement concerns how strongly regions are connected rather than which way they project.

Studying state-dependent maps of predictive information. Next, we analyzed the state-dependent properties of the attention across neurons and whether the attention patterns could be used to differentiate rewarded or non-rewarded states. We compared attention in the second after reward with the second after an error (no reward). We used the 29 evaluation recordings from 13 mice never seen in pretraining, each with its finetuned model, and analyzed only time the model never trained on. If attention were a fixed map, it would not change between the two states. Instead, attention between regions separates them in 97% of recordings, and in 90% after regressing out firing rates, compared with 48% for activity correlation. The change is dense rather than confined to a reward network. About 80% of neurons switch their most-attended sources, in every major region (50–98% per region), and the neurons that switch fire faster and change their rate more between the two states. The largest increases after reward are in attention to thalamic and midbrain sources, and the reward-minus-error pattern is shared across mice (r 0.33, p = 0.02). The 423 pretraining recordings from 126 mice show the same pattern on held-out time (79% of recordings, against 22% for activity correlation).

2.2. "Do deep forecasting representations provide additional biological information beyond conventional spike-train statistics?"

To address this question, we evaluated the advantage of SCRYER's representation over spike-train statistics (inter-spike interval, ISI, histograms). To isolate what SCRYER's architecture adds, we initialized the embeddings of held-out neurons from ISI-based spike-train fingerprints, finetuned them with SCRYER (embeddings only, network frozen), and decoded brain region from the result with the same linear probe used for the ISI features, so the two scores differ only by what SCRYER adds on top of the ISI information. With this initialization, region decoding improved for every pretrained model, and the mean single-unit macro-F1 rose from 0.240 to 0.273, above ISI alone (0.262). The gap is even larger at the finer level of 44 Beryl regions (0.335 vs. 0.244 for ISI, chance 0.075). SCRYER's representation therefore carries more region information than spike-train statistics, suggesting that SCRYER learns additional biological information beyond firing statistics.

3. Robustness and model dependence of Figure 5 (Q4)

3.1. "Could the authors perform similar history-only and population-only analyses using existing forecasting models, particularly POCO?"

To address this question, we applied the same masking to POCO. This analysis measures how much each neuron depends on the population, not which neurons it uses, so any model that forecasts each neuron can be tested. The two models agree on which neurons and regions depend most on the population (Spearman 0.51 across neurons and 0.66 across regions). POCO, however, uses the population less effectively. From the population alone, SCRYER predicts a neuron twice as accurately (median D2 0.073 vs 0.032), because each target draws on individual neurons rather than on one compressed summary.

3.2. "Could the authors investigate whether the conclusions remain consistent under alternative masking strategies that reduce distribution shift?"

To show that the map does not depend on the masking scheme, we replaced the mean with an in-distribution mask, in which the masked input is the same neuron's activity from another random test window, so the model sees only real activity. The ranking of neurons remains nearly unchanged (Spearman with the original Δ 0.85 for SCRYER and 0.90 for POCO), and the two models still agree (0.44 across neurons and 0.80 across regions, against 0.51 and 0.66 with the mean).

3.3. "Could the authors clarify how neurons with near-zero full-model predictive performance are handled?"

To show that excluding poorly predicted neurons does not shape the attention map, we varied the exclusion threshold. Δ is divided by the full-model D2 and is unstable near zero, so we exclude neurons with full-model D2 ≤ 0.05 (657 of 3,196 for SCRYER) and 13 neurons whose D2 is undefined. If the excluded neurons shaped the map, changing the threshold would change which neurons appear history-driven or population-driven. With thresholds of 0, 0.02 and 0.10, however, the agreement between SCRYER and POCO stays between 0.46 and 0.53, and SCRYER's median Δ changes close to zero by less than 0.04, so the balance between the two groups is also unchanged.

Spearman of ΔSCRYER, two masksPOCO, two masksSCRYER vs POCO, mean maskSCRYER vs POCO, in-dist. mask
Neurons0.850.900.510.44
Regions0.900.920.660.80
Reviewer_c1HjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new framework for neuron-level activity forecasting, aimed at learning underlying dynamics of a neuron’s behaviour. The authors introduce an architecture, SCRYER, that allows forecasting with direct access to the activity of neighbouring neurons in the population. It combines learned identity embeddings with current activity features, learns how to weight that information across the population and then forecasts the target neuron’s activity by additionally weighting different portions of the target’s own historical activity.

The architecture is evaluated on two types of datasets: The IBL Brain-wide Map (using BrainWideBench; multi-region electrophysiology data) and calcium imaging recordings (Allen Brain Observatory sessions, in mice, C. elegans and zebrafish).

The authors evaluate their architecture across four main aspects: (1) How forecasting performance compares to other methods that do not query population neurons individually, (2) whether anatomical information can be derived from the learned representations, (3) the effect of pretraining and its scale on overall performance and (4) which activity source - the target’s own history or the co-recorded population history - contributes more heavily to the forecast.

SCRYER achieves state-of-the-art performance across all tested architectures on both datasets. Notably, the learned representations organize around anatomical brain regions without the model ever receiving anatomical labels during training. This anatomical structure successfully transfers to unseen recordings, where SCRYER outperforms all transductive baselines. The scaling analysis demonstrates that the performance benefits from a larger pretraining corpus, improving the D2D^2 performance by 11.4%. Finally, the model reveals a continuum in neural dynamics: history-driven neurons tend to fire faster and more regularly, while population-driven neurons fire more sparsely.

Critical strengths

(1) SCRYER addresses a relevant gap in neural forecasting by combining shared representations across recordings with direct neuron-to-neuron querying. Giving each target neuron direct access to individually represented source neurons is a highly relevant and effective contribution to the neural forecasting problem, avoiding the early-mixing limitations of standard latent tokenization approaches.

(2) The empirical evaluation spans multiple datasets, species, and recording modalities. SCRYER consistently outperforms the compared models on the reported forecasting metrics. The finding that learned representations also retain anatomical information, despite receiving no anatomical labels during forecasting training, provides an additional and interesting demonstration of their utility.

(3) The ablation studies investigate the contributions of the architecture’s individual components and support the benefits of combining temporal structure, population retrieval, and identity conditioning. They strengthen the evidence for the proposed design.

Critical weaknesses

(1) The SCRYER architecture introduces the ability to learn retrieval weights across neighbouring neurons, but the paper currently only provides qualitative, hand-picked examples of these interactions (Fig. 2 and Fig. A5). The manuscript would highly benefit from a systematic, qualitative analysis of these attention matrices at the population level. Without analyzing whether these weights exhibit spatial/anatomical structure, or how they differ between history-driven and population-driven neurons, the paper misses a major opportunity to extract biological and functional connectivity insights form its own architecture.

(2) The SCRYER architecture restricts population retrieval strictly to neurons co-recorded within the exact same session. While this aligns with the physical reality of a single recording, it leaves the target neuron vulnerable to the specific anatomical sampling of that session. The authors rightly identify extending retrieval across many sessions as a natural next step in their Discussion. However, the manuscript would benefit from a brief discussion on the feasibility and technical barriers of this approach. Specifically, what prevents the current architecture from querying the broader pretrained corpus during inference? Is it primarily a computational constraint (scaling attention across thousands of neurons), or is it a fundamental issue with aligning session-specific embeddings? Would it be feasible to explore this as an additional analysis in the main body of the manuscript? Expanding on this would provide valuable context for future work.

Overall, these concerns primarily concern the depth of analysis and discussion, rather than undermining the forecasting results.

Questions for authors

Formal comment\textbf{Formal comment} Submitted abstract in openreview does not match the abstract in the manuscript

Smaller typos and language feedback\textbf{Smaller typos and language feedback} -Inconsistent use of neighbour/neighbor -Inconsistent: Mice-200 vs Mice 200 -Fig 1(c): “... Retrieve interaction of individual neurons”, better to use information instead of interaction? -Fig 1(d): “Weighted candidate selection”, removing the plural in candidates -P.4, Methods, Section 3.1: “...and a dataset identifier d”, the ‘a’ is missing. -P.6, Results, Section 4.1: “29 held-out sessions”, hyphen missing -P. 8, Results, Section 4.5: “masked forecast retains, negative”, replace comma by colon or period. -P. 11, citations, “Towards a” universal translator”, correct -P. 12, Appendix, “the number and resolution of bands is unchanged”, is → are -P.13, Appendix, A.4: “To give each neuron a summary”, grammar not correct, e.g., “Our goal is to give each neuron …” -P.13, Appendix, A.4: “it is permutation-invariant, and its” → permutation-equivariant? -P.14, Appendix, A.5: “renormalized to one sum”, → renormalize to sum to one -P. 19: “filter band width” → filter bandwidth -P. 21: Citation formatting “by locomotion and arousal Fu et al. (2014)” -P. 22: Figure A2: “The scaling results with session counts for 200, 800 and 1334 on …”. Sentence structure? Better: “Scaling results for 200, 800, and 1,334 sessions from the Allen mouse dataset.” -Several citations in Appendix B and C are inserted directly after noun phrases without parentheses. For example: p. 15, line 784: “from the IBL Brain-Wide Map IBL et al. (2025) using BrainWideBench Andre et al. (2026)” → “from the IBL Brain-Wide Map (IBL et al., 2025) using BrainWideBench (Andre et al., 2026)”. -The same issue occurs on p. 15, line 799; p. 16, lines 815, 824–825, and 834–835; p. 18, line 957; and p. 19, lines 977, 985, and 998.

Clarifications\textbf{Clarifications} -P. 12 and 14, lines 636–637 and 727 vs. P. 17–18, lines 917–918: Flavell is described as using 6 bands / 42 candidates, but its configuration specifies 5 Haar bands, which would give 35 candidates. -Fig 1(b): temporal encode (maybe better to call it “temporal encoder”?): Coloring is misleading, as the green, orange and blue color were used to distinguish between different neurons? Doesn’t the temporal encoder decomposes the history of one specific neuron (for example the green one) in this figure? -“Co-recorded neurons” maybe vertically to separate as label? -Coloring of Fig 1(c) is also a bit unclear to me, especially the e_{c’} visualization. The dark color was used before for identity embedding, are the dark orange/blue/purple ones identity embeddings only? Or complete embeddings (identity + activity). Also the purple color only is introduced in Fig 1(c), better to have it already in Fig. 1(b). -What is the red coloring representing in Fig 1(d)? Higher weighting?

Additional comments and questions\textbf{Additional comments and questions}

  • Structure of population retrieval. Do the learned retrieval weights exhibit systematic spatial or anatomical organization? For example, do neurons preferentially attend to neurons within their own region or to particular other regions? Do history-driven and population-driven neurons differ in attention concentration, diversity, or the anatomical distribution of their retrieved sources? While the authors correctly note that these weights do not prove causal interactions, a quantitative analysis of the attention matrices could reveal valuable insights into how the model structures its predictive context.
  • Variation in forecasting performance across neurons. Section 4.5 presents an interesting continuum between history-driven and population-driven prediction. How does absolute forecasting performance vary along this continuum? Are the most accurately predicted neurons concentrated in particular anatomical regions, and do they exhibit distinctive retrieval patterns? I there a link to history-driven or population-driven neurons? An analysis accounting for firing rate and firing regularity would help connect these findings.
  • Anatomical information in activity embeddings. The emergence of anatomical structure in the representations (Section 4.3) is a strong result, but they focus solely on learned identity embeddings. Can anatomical regions also be decoded from the activity embeddings alone? Comparing identity-only, activity-only, and combined representations under the same evaluation protocol would help distinguish anatomical information available from current temporal dynamics from information accumulated in the fixed identity parameters.
  • Extension to calcium-imaging datasets. Are anatomical or cell-type annotations available for the calcium-imaging datasets that would support an analogous representation analysis? Such an evaluation would help establish whether the anatomical information observed on IBL extends across recording modalities and datasets.
  • Scaling behavior. What motivated the choice of 40, 100, and 423 recordings for the IBL scaling analysis? It would be interesting to see if there is a constant increase in performance up to 423, or whether it converges at a smaller number of recordings. The reported points demonstrate improvement, but leave the shape and potential saturation of the scaling curve unresolved. Could the authors provide additional intermediate points, or discuss whether existing experiments indicate diminishing returns? Same comment for the Allen scaling result in Fig. A2.
  • Feasibility of cross-session retrieval. What would be required for a target neuron to query information from the broader pretraining corpus at inference time? Is the main obstacle computational cost, alignment of session-specific representations, or defining relevant population context across recordings that are not temporally aligned? A brief clarification would help assess this proposed future direction.
7 comments

Response to Reviewer c1Hj

We are grateful to the reviewer for an unusually thorough and thoughtful review. The questions on the population-level structure of the retrieval weights, on cross-session retrieval and on where anatomical information lives in the representation each led to a new analysis, and together they made the paper considerably stronger. We respond to every point below.

W1; Q5. Population-level analysis of retrieval weights

(W1) The paper shows only hand-picked examples of retrieval weights (Fig. 2, A5). A systematic population-level analysis of the attention matrices, their spatial/anatomical structure and how they differ between history-driven and population-driven neurons, would reveal biological and functional connectivity insights.

(Q5) Do the retrieval weights show spatial or anatomical organization, for example preferential attention within a region or to particular regions? Do history-driven and population-driven neurons differ in attention concentration, diversity or anatomical distribution of sources?

As suggested, we have performed a population-level analysis of the attention matrices, covering all neuron-to-neuron weights in the model pretrained on 423 recordings from 126 mice and in the models finetuned on the 29 held-out recordings, and we compare them with the mouse connectome, the axonal connectivity between brain regions measured in the Allen Mouse Brain Connectivity Atlas.

Anatomical structure. First, we evaluated the average attention by brain region, comparing how much neurons in one region attend to another with an even spread. We found that neurons attend preferentially to their own brain region by 7–65% in every region and in 97–99% of mice, although the model never sees region labels. Furthermore, clustering regions by whom they attend separates the forebrain from the midbrain, hindbrain and cerebellum, and within the hippocampus, its subregions, such as CA1, CA3 and the dentate gyrus, attend preferentially to one another. Region pairs that attend to each other more also have more similar neuron embeddings (Spearman 0.82). To show that this preference is not a matter of distance, we regressed the attention between every pair of neurons on their distance in each recording, controlling for shared region, activity correlation and firing rate, and the effect of distance did not differ from zero across the 29 recordings (p = 0.16 and 0.49 for the two attention layers), so attention and neuron identity capture the anatomy instead of distance.

Row → column, log2 vs uniformOLFHPFCNUCtxTHCBMBHB
OLF+0.40+0.21−0.01−0.07−0.38−0.94−0.43−0.39
HPF−0.02+0.72+0.04+0.11−0.21−0.59−0.23−0.38
CNU−0.16+0.26+0.27−0.01−0.23−0.42−0.20−0.44
Ctx−0.17+0.10−0.04+0.44−0.11−0.35−0.13−0.34
TH−0.17+0.05−0.09+0.11+0.10−0.36+0.02−0.25
CB−0.39+0.02−0.03−0.04−0.07+0.15+0.18−0.06
MB−0.43−0.13−0.09−0.04−0.06+0.05+0.22−0.04
HB−0.51−0.21−0.22−0.10−0.11−0.04+0.00+0.19

Cross-attention, averaged over 126 pretraining mice. Each cell is how much neurons in the row region attend to neurons in the column region, relative to an even spread (log2). Ctx, isocortex.

History-driven versus population-driven neurons. Second, we looked into attention maps of history-driven vs. population-driven neurons. We found that the two groups distribute their attention in the same way. They do not differ in same-region attention, and their small difference in attention concentration disappears once firing rate and regularity are controlled. They differ in how much they rely on the population, not in how they attend to it.

Attention reveals state-dependent coupling. Third, we evaluated the relation between the attention maps (as a proxy of the functional connectivity) and the behavioral states of the animals. We found that, in held-out mice, attention between regions differs between the second after reward and the second after an error in 90% of recordings, even after regressing out firing rates, and the reward-minus-error pattern is shared across mice. Attention therefore captures interactions that change with the animal's state, beyond fixed anatomy.

Comparison with the connectome. Lastly, we compared the model's attention (based purely on forecasting) to the mesoscale connectome of the Allen Mouse Brain Connectivity Atlas, an independent measurement of axonal projections made by viral tracing across a large cohort of mice. We computed the target-specific attention between brain regions (each source's attention normalized by the mean attention it receives from all targets, which removes the component shared by all targets). We found that brain region pairs with denser axonal connections attend to each other more in SCRYER as well (Spearman 0.47 over 2,104 region pairs, permutation p < 0.01), and the agreement remains after controlling for distance, shared probes, parent region and firing rate (0.28). As a baseline we computed jitter-corrected cross-correlograms (CCG, Siegle et al., 2021), the standard spike-timing estimate of connectivity, on the same recordings and region pairs. After the same controls, SCRYER's attention agrees with the connectome better than CCG at the fine level (0.28 against 0.22) and at the coarse level of ten major brain divisions (0.47 against 0.23, where CCG is not significant). Neither method recovers the direction of projections, so the agreement concerns how strongly regions are connected rather than which way they project.

Together, the population-wide analysis of the retrieval weight have shown that attention matrices recover the brain's anatomical organization and its connectome without any anatomical supervision, apply equally to history-driven and population-driven neurons, and in addition reveal coupling that changes with the animal's state. We included this figure at the end of Appendix, see Figure A6.

W2, Q10. Retrieval across sessions

(W2) Retrieval is restricted to co-recorded neurons, leaving the target vulnerable to the session's anatomical sampling. What prevents querying the pretrained corpus at inference, computational cost or alignment of session-specific embeddings? Could this be explored as an additional analysis?

(Q10) What would be required for cross-session retrieval? Is the main obstacle compute, embedding alignment, or defining population context across recordings that are not temporally aligned?

The main barrier for cross-session retrieval is the temporal alignment of neuronal signals recorded in different sessions. Retrieval mechanism requires the simultaneous activity of the sources, and neurons from another session have no activity in the target's window. Extending retrieval across sessions requires a way to align activity across recordings.

To test whether such alignment is feasible, we ran a new pilot experiment. We described each neuron by its spike statistics (log-ISI histogram, autocorrelogram, power spectrum and firing rate), retrieved for each neuron of a new animal the most similar pretraining neuron among 63,382 by cosine similarity, and used the retrieved neuron's identity embedding as the new neuron's starting point. The retrieved neuron shares the new neuron's brain region in 28.8% of cases against 12.7% by chance, and starting from it raises held-out forecasting D² from 0.170 to 0.175 and region-decoding macro-F1 from 0.240 to 0.273. Cross-session matching therefore positively transfers information, and with trial-aligned data such as IBL, where the same task is run in every session, the retrieved neuron's activity at the same task phase can supply the current activity it lacks.

Q1. Abstract mismatch The OpenReview abstract does not match the manuscript.

Thank you. We have made the OpenReview abstract identical to the manuscript abstract.

Q2. Typos, wording and citation format A list of typos, inconsistent spellings (neighbour/neighbor, Mice-200/Mice 200), wording and grammar fixes, and citations in Appendices B and C that lack parentheses.

Thank you for the careful reading, we implemented the suggested corrections.

Q3. Flavell band count Flavell is described as using 6 bands / 42 candidates, but its configuration specifies 5 Haar bands, which would give 35.

Thank you for catching this. The configuration is correct and the text is not. The Flavell models use five Haar bands, so the two-level tree with the root segment gives J = (2^{D+1} − 1)K = 35 candidates rather than 42. We corrected the text.

Q4. Figure 1 colors and labels Fig. 1(b): rename "temporal encode" to "temporal encoder"; coloring by neuron is misleading since the encoder decomposes one neuron's history; place "Co-recorded neurons" vertically. Fig. 1(c): unclear whether dark tokens are identity-only or full embeddings; purple appears only here. Fig. 1(d): what does red represent?

Thanks for your suggestions and we have revised Figure 1 accordingly. We rename the block "temporal encoder" and color its decomposition by band rather than by neuron, since it acts on the history of one neuron. We have placed the "Co-recorded neurons" label vertically to separate it from the target. In Fig. 1(c) the dark tokens are identity embeddings, identity plus activity become the full tokens, and we draw the two parts separately and introduce the purple neuron already in Fig. 1(b). In Fig. 1(d) the red coloring marks higher candidate weights, which we have stated in the caption.

Reviewer_EQL2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a forecasting method that works like this: The inputs are the neuron’s past activity and all of the co-recorded neurons. The model learns attention weights over the co-recorded weights and outputs a forecast. The input is cut up into segments of varying scales. Each segment is represented as a vector of frequencies. These segments are weighed and combined to get the final input representation. This is done to handle the multi-scale nature of the input. The inputs for many neurons are attention-scored to create a context, which is used for forecasting. Forecasting is done per scale and then combined to create a final forecast.

Critical strengths
  • The discussion around context-driven and history-driven neurons is very interesting and seems to offer a unique insight made possible by the paper’s trained model.
  • The forecasting is stronger than the baselines
  • The performance scales with the pre-training data
Critical weaknesses
  • Lines 221: This seems like a very odd choice to do candidate forecasting per scale, where the inputs are all different scales, and the output is always the same scale. Why not use a network that ingests all features at all scales and makes a final forecast in an end-to-end fashion? At the very least, it seems odd to weight the different segments to create a single input (equation 1) and then weight the segments to create a single forecast (equation 5). Why not simply do one?

  • Could you compare against off-the-shelf forecasting models like chronos-2 and Moirai? Some studies are showing they are even more effective https://openreview.net/forum?id=5Xwm8e6vbh than neural-data-specific models.

  • Line 50: Conceptually, it’s not clear what the older models were lacking. The text says that these models “do not allow for individual neurons to query each other’s history”. But this does not hold for many multi-variate time series models like Chronos-2 https://arxiv.org/pdf/2510.15821 which allow for attention between different time patches.

  • I’m not sure if there is a significant machine learning contribution here. There are already time series foundation models that can handle multi-variate data and incorporate cross-attention between variates. What makes this a conceptually new approach, and why is a conceptually new approach required for the task of predicting neuron activity? What are the parts of this approach that are specific to the fact that neural data is being handled?

  • The model takes a session identity, and unit identity. But these don’t transfer between pre-training and fine-tuning. Not a big weakness, but probably points to where pretraining could be even more effective.

Questions for authors
  • Line 46: Could you please define acronyms when they are first used? Like NDT, MtM, etc.?

  • Line 177: Why is vjv_j only indexed by jj? If it corresponds to one temporal segment and frequency band, shouldn’t it have two indices?

  • Line 246: why predict the log expected spike count? Why not the spike count? Is this something specific to spike rate data?

5 comments

Response to Reviewer EQL2

We thank the reviewer for the detailed and constructive review.

W1. Candidate forecasting per scale. "Why forecast candidates per scale, with the output always at the same scale, instead of one end-to-end network over all features at all scales? Weighting segments at the input (Eq. 1) and again at the forecast (Eq. 5) seems redundant — why not just one?"

Thank you for the question. The two design components separate two sources of a neuron's future activity.

In our work, the first source of predictive signal is the neuron's own dynamics. From its own history, SCRYER builds one candidate forecast per time scale and band, and Eq. 1, which sees only the target's own signal, decides which bands are passed down and split further. The second source is the influence of other neurons. Each target attends to every co-recorded neuron individually, and through Eq. 5 the retrieved context sets how the target's own candidates are weighted. This design is rooted in the structure of neuronal connections in the brain, combining neuron’s intrinsic dynamics with input from specific other neurons. For every neuron and window, the candidate weights show which time scale is used, and the attention shows which neurons set that choice. The candidates forecast the same horizon so that they can be compared and combined, as in N-BEATS and N-HiTS, where each block sees part of the input and forecasts the full horizon. Each of the two components contribute to the improved accuracy of the forecast. To show that the selection step is needed, we retrained SCRYER on the 29 held-out IBL recordings with one part changed at a time. Fixing the Eq. 5 weights to uniform or to the energy prior lowers D² from 0.166 to 0.159 and 0.161, and replacing the candidates and both weightings with one parameter-matched MLP, the end-to-end alternative, lowers it to 0.159.

Held-out D² (↑), 29 recordings, retrainedFull modelSingle MLP (parameter-matched)Uniform weightsEnergy weights only
Mean D²0.1660.1590.1590.161

W2. Off-the-shelf forecasting models. "Could you compare against off-the-shelf models like Chronos-2 and Moirai? Some studies show they are even more effective than neural-data-specific models."

Thank you for the suggestion. We ran Chronos-2 and Moirai-2 zero-shot with their official weights on all five datasets, and reported the results below. We found that SCRYER outperforms both on every dataset.

ZimmerFlavellZebrafishAllenIBL
SCRYER0.341 / 0.3960.546 / 0.52338.3 / 4.000.931 / 0.5810.185
POCO0.350 / 0.4020.600 / 0.55639.7 / 4.100.990 / 0.5900.169
MLP0.358 / 0.4070.554 / 0.52444.6 / 4.320.946 / 0.5960.034
Chronos-2 (ZS)0.386 / 0.4130.599 / 0.53252.4 / 4.691.057 / 0.596−0.15
Moirai-2 (ZS)0.402 / 0.4230.617 / 0.54958.0 / 4.870.960 / 0.587−0.19
Persistence0.552 / 0.5000.766 / 0.60070.8 / 5.571.744 / 0.8280.03

W3. Line 50: what earlier models lack. "It is unclear what the older models were lacking. 'Do not allow individual neurons to query each other's history' does not hold for multivariate models like Chronos-2, which allow attention between time patches."

We would like to clarify that the sentence refers to the neural population models cited just before it (NDT, MtM, POCO), which merge neurons into shared tokens or latents before any target-specific computation, and we now stated this scope explicitly. We thank the reviewer for pointing to Chronos-2 and Moirai-2, whose group and any-variate attention do let series attend to one another. The difference lies in what the attention is conditioned on. In these models the variates are anonymous, so the relation between two series has to be inferred again from the context of every window, whereas SCRYER attends over neurons with learned identities shared across 423 recordings and uses the retrieved sources to choose among each target's candidate forecasts. This matters for neural data. Removing retrieval lowers IBL D2 from 0.166 to 0.141 (Table A4), and zero-shot Chronos-2 and Moirai-2, which do use cross-series attention, stay behind SCRYER on every calcium dataset and below a constant-rate forecast on IBL.

W4. Machine-learning contribution and neural-specific parts. "Foundation models already handle multivariate data with cross-variate attention. What is conceptually new here, why is it required for predicting neural activity, and which parts are specific to neural data?"

The central contribution of SCRYER is a transferable target-specific retrieval mechanism that operates on forecasts derived from each neuron's own history. For each neuron, we generate a set of candidate forecasts, such that there is one candidate forecast per each time-frequency component of the neuron’s past activity. The target then retrieves information from individual co-recorded neurons to determine how much each candidate contributes to its prediction. Thus, population activity primarily influences the selection among a neuron's own forecasts, rather than being compressed into a shared population representation. Our ablations directly support this distinction. Removing the additive correction changes the held-out IBL (D^2) by less than 0.001, whereas the correction-only model drops from 0.166 to 0.150. Replacing the selection mechanism with a single MLP, uniform weights, or energy-only weights reduces performance. This formulation also makes it possible to quantify the relative predictive contributions of a neuron's own history and its co-recorded population, as examined in Figure 5.

Existing approaches contain related components but differ in how population information enters each neuron's forecast. General time-series foundation models such as Chronos-2 and Moirai-2 model interactions among variates without recording-specific neuron identities or selection among forecasts derived from an individual neuron's history. POCO and OmniMouse aggregate population activity into shared latent representations, while NetFormer models interactions among neurons within a fixed population. SCRYER instead combines target-specific multiscale candidate forecasts with direct retrieval from individual source neurons. Each neuron is represented through its identity, session context, and current activity, enabling retrieval that depends on both the target and the observed population state. The retrieval and forecasting network is shared across 423 recordings with different neurons, and adaptation to an unseen animal requires learning only new neuron embeddings.

This design addresses two challenges specific to neural recordings. Different sessions contain different sets of neurons, requiring a shared model that can adapt to new identities, and neural population analysis benefits from distinguishing predictions supported by a neuron's own history from those that depend on surrounding activity. The empirical results support both goals. Removing retrieval reduces IBL (D^2) from 0.166 to 0.141 (Table A4). Increasing the number of pretraining recordings improves SCRYER from 0.166 to 0.185, whereas POCO shows no comparable consistent scaling trend. Zero-shot Chronos-2 and Moirai-2 underperform SCRYER across the calcium-imaging datasets and fall below a constant-rate forecast on IBL (Weakness 2). Neuron-, session-, and dataset-specific embeddings, adaptation through new identity embeddings, and a Poisson log-rate output further accommodate the heterogeneity and spike-count statistics of neural recordings.

W5. Identity embeddings do not transfer. "Session and unit identities do not transfer between pre-training and fine-tuning — not a big weakness, but it points to where pretraining could be even more effective."

In line with the Reviewer's comment, in the current version of the model, unit and session embeddings belong to their recording, so new neurons need new embeddings, fitted with the forecasting objective, while the shared network stays fixed. This rows-only adaptation is also what recovers region information in held-out mice. As an alternative, pretrained identities can already be transferred by retrieval. For example, this can be done by initializing each new neuron's embedding with the embedding of the pretraining-set neuron with the most similar spike-train fingerprint (interspike-interval histogram, autocorrelogram, power spectrum and firing rate). In our pilot experiment on the 63,382 pretraining neurons, such a matching initialization raised the held-out Poisson deviance explained (D2) from 0.170 to 0.175, and region decoding from the new embeddings from 0.240 to 0.273 macro-F1. As yet another alternative, an inductive identity computed from a neuron's own activity, as in NuCLR (Contrastive Learning of Neuron Representation, Arora et al., 2025) or NEMO (Neuronal Embeddings via MultimOdal Contrastive Learning, Yu et al., 2025), which obtain a neuron's embedding from an encoder network applied to its own data rather than from a learned lookup table and therefore need no fitting on a new recording, would remove this step, and can be implemented in future work.

Q1. "Please define acronyms (NDT, MtM, ...) at first use."

Thank you for pointing this out. We have defined every acronym at first use in revised paper, for example NDT (Neural Data Transformer) and MtM (multi-task masking), and list brain-region abbreviations in the appendix.

Q2. "Line 177: why is the candidate indexed by a single index if it corresponds to one temporal segment and one frequency band?"

The single index jj enumerates (segment, band) pairs, which we flattened for brevity. We have stated this explicitly where jj is introduced and give JJ for each dataset in Appendix C.1.

Q3. "Line 246: why predict the log expected spike count rather than the spike count? Is this specific to spike data?"

As the Reviewer has pointed out, it is specific to spike counts. The Poisson mean must be positive, so the network outputs η = log λ and the loss is the Poisson negative log-likelihood of the observed counts under λ = exp(η), the canonical link of a Poisson generalized linear model (GLM), also used by LFADS (latent factor analysis via dynamical systems), NDT (Neural Data Transformer) and BrainWideBench. Regressing counts directly with squared error would ignore that they are non-negative integers whose variance grows with the mean, and BrainWideBench scores forecasts by Poisson deviance and bits per spike, which require a rate. For calcium data the model predicts the z-scored trace directly with an MSE loss.

Comments on the paper

1 comment

AC response

We thank the reviewers and AC for their time and thoughtful feedback. We are happy that the reviewers noted a number of strengths that include:

  • Relevance and importance of problem. "SCRYER addresses a relevant gap in neural forecasting by combining shared representations across recordings with direct neuron-to-neuron querying" (c1Hj)
  • Novel analyses and insights. "The discussion around context-driven and history-driven neurons is very interesting" (EQL2)
  • Extensive evaluations. "broad experimental evaluation" (Dnjg) that "spans multiple datasets, species, and recording modalities" (c1Hj)

In response to their questions, we conducted the following additional experiments and analyses:

  • Dissecting retrieval and forecast selection (Dnjg, Q1–Q2; EQL2, W1). Removing candidate forecasts, replacing them with a parameter-matched MLP, or fixing their selection weights reduced IBL forecasting performance from D2=0.166 to 0.150–0.161. Removing the four most-attended source neurons caused a fourfold larger performance drop than removing the four most-correlated sources (0.047 vs. 0.011), even with firing rates matched. These results support the contributions of both retrieval and selection.
  • Testing the biological information captured by attention (c1Hj, W1). Attention identified target-specific sources of predictive information. Aggregating attention across recordings produced a map spanning 196 brain regions that aligned with the independently measured Allen connectome, including after controlling for distance, shared region, firing rate, and activity correlation (partial Spearman r=0.16).
  • Evaluating scalability and model size (Dnjg, W3 and Q5; c1Hj, Q9). Across nested subsets of 40–423 recordings, SCRYER's pretraining gain increased from 0.005 to 0.019 (D2). A smaller SCRYER with 44% of POCO's parameters still outperformed POCO. We also demonstrated improved forecasting on zebrafish recordings containing 53,180–97,766 individual neurons without PCA and quantified training cost as population size increased.
  • Expanding baseline comparisons (Dnjg, M3; EQL2, W2). We evaluated the recently released CalM and OmniMouse models, added IBL baselines, and compared them against Chronos-2 and Moirai-2. SCRYER outperformed these neural and time-series foundation models on every calcium dataset tested.
  • Checking robustness and representation quality (Dnjg, Q3–Q4 and M4; c1Hj, Q7). We added matched-probe region-decoding comparisons, tested the history-versus-context dependence map across masking schemes and models, evaluated calcium forecasts using per-neuron correlation, and examined anatomical information in identity and activity embeddings.

Together, these experiments clarify where SCRYER's gains arise, strengthen the biological interpretation of its retrieval mechanism, and extend the evidence for robustness and scalability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.