acceptodds
Under review as a conference paper at ICLR 2027

SpikeDiT: Generative neural system identification at spike-train resolution

11 views

Abstract

In populations of biological neurons, repeated presentations of the same stimulus evoke different spike trains, making neural responses fundamentally a conditional distribution. Yet, neural encoding models typically predict only the stimulus-conditioned mean response, either ignoring trial-to-trial variability or imposing it by an assumed noise distribution. Here, we introduce SpikeDiT: a stimulus-conditioned generative model of population activity that combines a frozen spike tokenizer with joint flow matching to generate spike trains at a native 1 ms resolution. Conditional mean responses, marginal firing statistics, fine temporal structure, and shared noise correlations all emerge from training and sampling from a single model. The framework is modular with respect to the underlying stimulus encoder and makes no a priori distributional assumptions about the form of neural variability. We evaluate the approach on two datasets that differ in sensory modalities, species, and recording technologies. Existing models capture either the conditional mean or trial-to-trial variability well whereas SpikeDiT matches or exceeds the strongest baselines on both and, without retraining, uses observed neurons to improve single-trial prediction of unobserved ones. By modeling population responses directly in spike space with a flexible generative framework, our approach provides a unified and scalable foundation for predicting and generating complete neural population responses.

Primary area:
applications to neuroscience & cognitive science
Keywords:
computational neuroscience, neuroscience, ai for science, flow matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vwTvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a stimulus-conditioned generative model of population spiking activity at 1 ms resolution. It treats neural encoding as conditional generation in the latent space of a frozen spike tokenizer, a per-unit VAE developed in a concurrent submission. That tokenizer maps each unit binarized spike train to one continuous token per 100 ms period. A latent flow-matching transformer, conditioned on stimulus features through per-unit cross-attention queries, learns the joint distribution of tokens across units and time. Samples are then decoded deterministically back to spike trains, so all trial-to-trial variability comes from the initial noise of the flow and happens in the latent. The result is a stochastic emulator of the stimulus-conditioned response distribution p(Y | x) [Y being the spike trains and x the stimulus], rather than of the mean response alone or than a assumed-variability structure. I emphasize on the fact that the model is non-autoregressive, it generates the full window jointly (i.e. acausal generation within a window).

The model is evaluated on two public datasets that differ in species, modality, and recording technology. The first is macaque V4 responses to natural images, using a jointly fine-tuned DINOv3 encoder. The second is ferret auditory cortex responses to natural sounds, using a convolutional encoder trained from scratch on cochleagrams.
Evaluation covers four axes: conditional-mean prediction, single-unit and shared variability (Fano factors and noise correlations), spike timing at 1 ms (ACG and CCG), and completion of partially observed responses. The baselines are an independent Poisson encoder, a temporal extension of FlowFA, and a reimplementation of another latent-state model of Schmidt et al. (2025). All baselines operate on 20 ms count bins.

A (small in my opinion) ablation study is presented. An auxiliary projected CRPS loss is introduced specifically for the proposed model (called the SpikeDiT) in order to correct the over-dispersion of spike counts that the authors observe when training with flow matching alone, and to calibrate both single-unit variability and pairwise covariance on the decoded spikes rather than only in the latent space.

Critical strengths
  • Monte Carlo convergence is analysed explicitly and the metrics are careful and well defined.

    • Because the model is defined only through samples, every reported statistic is a Monte Carlo estimate. The authors study convergence with the number of draws on both datasets. They also use the analytic limit of the Poisson baseline as a sanity check, which shows that the reported numbers are stable.
  • Evaluation goes well beyond the mean.

    • Besides conditional-mean prediction (which is a sanity check more than a result, in my opinion), the model is assessed on single-neuron dispersion, on both the pattern and the magnitude of shared variability, and on spike timing.

    • The completion experiment includes a well-designed control. It shows that the gain comes from trial-specific shared variability rather than from additional stimulus information.

    • (Weak point: see weaknesses on pedagogy.)

  • Generality across two very different datasets.

    • The two datasets differ in species (macaque vs ferret), sensory modality, brain area (V4 vs auditory cortex), recording technology, population size and response duration.
  • Figure quality.

    • The figures are clear, well designed, and carry the paper message effectively. Figure 1 gives an immediate overview of both the method and the main results, and Figure 2 makes the pipeline easy to follow (even if I would swap left and right panels). The results figures are informative.
  • The latent-regularization ablation is a valuable experiment.

    • It is consistent with the broader observation of the field that a well-regularized latent space, not just accurate reconstruction, is important for latent generative modeling.
  • The limitations section is honest and well thought out.

    • The authors name the specific costs of their design: no tractable likelihood, no interpretable latent state, approximate conditioning, dependence on the tokenizer, acausal stimulus context. Each is framed against baselines that do not share it, which tells the reader when a latent-variable model would still be preferable.
Critical weaknesses

I will separate the weaknesses into two categories reflecting my personal assessment: (1) non-decisive points, which I would welcome being addressed but which will not affect my score, and (2) decisive points, which, if addressed, would lead me to increase my score.

(1) NON-DECISIVE POINTS

  • The paper could explain things more pedagogically e.g. why going beyond the mean matters.

    • The introduction motivates the full distribution mostly by citation, without a concrete question that only a distributional model can answer (e.g., the impact of noise correlations on coding, or planning trial and neuron counts). The most compelling use cases, in silico experiments relying on variability, appear only at the end of the discussion and would be stronger in the introduction.

    • Space for this could come from the implementation and training paragraphs (lines 276 to 299). The optimizer settings, betas, learning-rate schedule, warm-up steps, gradient clipping and flow-time distribution are already listed in Table S1. A one or two sentence summary pointing to Appendix C/F would be enough in the main text.

  • Sensitivity of the CRPS term is not explored.

    • The dose hyperparam.  differs between datasets (0.05 vs 0.01). A sensitivity analysis would show how robust the calibration is.
  • Minor presentation issues.

    • Figure 2 might read more naturally with the left and right panels swapped.
  • ~Line 80, it is said that no binning is done, i disagree with this formulation, and I think that "no coarse" binning would be more honest.

(2) DECISIVE POINTS

  • No baseline for spike timing or for completion.

    • All baselines work with 20 ms counts, so the ACG/CCG comparison is won partly by construction. A baseline that models timing explicitly, such as a GLM with spike-history filters (Pillow et al., 2008), would make the timing result more informative. The independent Poisson model is a useful sanity check, but a correlated Poisson model or similar would be a stronger baseline.

    • LDNS is discussed at length as the closest work but is not compared empirically.

      • I would like the authors either to explain why this comparison is not possible or not relevant (I may be missing an obvious point here, and I am genuinely curious), or to add it.

      • Since other baselines were adapted to fit the framework and ensure a fair comparison, the same could be done for LDNS if needed. Notably, the companion paper already implements (I think) LDNS and LDNS-Perceiver as population tokenizers, so this comparison should be feasible with limited effort.

  • Strong dependence on an unreleased tokenizer from a concurrent submission.

    • I partially read the concurrent paper in the appendix to better understand the submission. My understanding is that the present paper contribution is the latent generative pipeline built on top of that tokenizer. The tokenizer choice should therefore be discussed, and I think an ablation is important: either generating directly in spike (or count) space, or using another published encoder/decoder.

    • The sentence "For neural data, spike tokenizers now provide such a latent space by encoding each neuron's spike train into continuous tokens (Anonymous et al., 2026)" should be tempered and the citations broadened (e.g., LDNS also learns a continuous latent space for spikes, at the population level).

Questions for authors

See the (2) DECISIVE POINTS of the weaknesses.

Reviewer_uWdHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

A pretrained encoder-decoder (binary sequence to sequence of Bernoulli distribution parameters) from another study is paired with a transformer. The transformer is trained with a flow-matching objective to generate latents (of the first model) conditional on contextual information. Generation is joint across neurons and time. The latents are decoded by the decoder of the first model, with output as Bernoulli distribution parameters at 1ms resolution. The proposed approach is compared against 3 baselines using a number of metrics.

Critical strengths

The paper shows that a stimulus-conditioned transformer trained with a flow-matching objective on the latents of a pretrained spike encoder can be used for predicting neuronal spikes.

Critical weaknesses

1. Experiment design does not isolate contributions or the source of gains

An example: the authors emphasize the value of their approach outputting spike trains as opposed to spike counts within intervals. This point is stated, but the experiment design does not isolate this factor. Notably, the baselines differ from SpikeDiT along many dimensions simultaneously. One thing that could help support the claim is to compare performance between different output heads for the same stem model.

2. Some claims/contributions are overstated

Again taking the spike train output as an example. Authors present the 1ms output resolution as an important contribution. The proposed model can be described as outputting an array of scalars with range [0,1][0, 1] each of which parameterize a Bernoulli distribution (L1093). Outputting Bernoulli distributions rather than count distributions is conventional; see for example Modeling the impact of common noise inputs on the network activity of retinal ganglion cells by Vidne et al. (2012). Authors also claim they avoid binning (L080) and don't output firing rates (L974); however, they do bin to 1ms, and thus they lose resolution within 1ms bins, and the Bernoulli parameters can be interpreted as a firing rates within those 1ms intervals, as 1ms ensures that no bin will realistically contain 2 or more spikes. The thresholding used can be understood as mode/MAP prediction from a Bernoulli distribution.

You differentiate your approach to spike train generation from a number of others (L118-132) but none of these approaches are compared to in the experiments. The claim "our tokenizer compresses each neuron separately letting the flow learn both dependencies among neurons and each neuron’s own dispersion" may be true, but there are no controlled experiments to isolate this design choice and determine how important it is.

3. Baselines are not convincing as benchmarks

The proposed model utilizes an encoder-decoder pretrained on a corpus of 20 billion spikes.The encoder-decoder ablation (L441-444) suggests that the proposed model's performance is highly dependent on the details of the pretrained encoder-decoder. In comparison, the other baselines do not utilize this seemingly powerful encoder-decoder.

The baselines do not seem to be given a best-effort shot at the problem. For example, the baselines are given a fixed 20 ms output interval which, despite bin widths being a very important hyperparameter, are not tuned to the data, so it is hard to accept that the baselines are operating optimally.

Rather than making the baselines stronger, it may be even more valuable if the experiments are designed to specifically target certain claims, rather than simply being models used from other works.

Minor suggestions

Some suggestions, although these are minor in comparison to the above issues.

  • Sharing a learning rate can be intended as a control; however when comparing very different architectures the control should be that all models have an appropriately chosen learning rate (lr sweep, or manual tuning). Results can easily be skewed by having a shared learning rate that benefits one model over the others.
  • explaining the meaning of the error bars in the figures would improve the readability of the main text (not relying soley on appendix).
  • if training budget allows, variation between training runs could be reported.
  • it would be easier to read some of the mathematical expressions if the symbols were more clearly defined (for example the expression on L107).
  • it would help to have the implementations of the baselines descibed more precisely.
  • it would be nice to be able to see the code.
Questions for authors

Purely for my curiosity:

  • any reason for flow matching over say a variant of diffusion?
  • have you tried including previous spikes as conditioning information?
Reviewer_v8PXOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats sensory encoding as conditional generation of population spike trains. SpikeDiT pairs a frozen single-neuron spike tokenizer with a stimulus-conditioned flow-matching transformer and samples binary spikes at 1 ms. Conditional means, Fano factors, noise correlations, and millisecond timing statistics are all computed from those samples. A DINOv3 image encoder and a cochleagram convnet are used as interchangeable stimulus front-ends. Poisson, Temporal FlowFA, and a Schmidt-style latent model share that front-end and the cross-attention readout. On macaque V4 and ferret auditory cortex, averaged samples match or slightly exceed the strongest encoding baseline on the mean, and match covariance magnitude and per-neuron dispersion better than the latent-variable baselines. A frozen model can also complete held-out neurons or a later time window at test time.

Critical strengths
  1. The framing is right and precisely stated. Poisson-loss encoders fix variance to the mean and thereby exclude both noise correlations and sub-bin timing; the paper names this failure mode and then declines to substitute another hand-specified covariance structure. Reusing the image-generation recipe is a defensible answer.

  2. Per-neuron dispersion is fit from data rather than fixed by a count family or a shared latent. On visual cortex the Fano-factor correlation is 0.87, against 0.60 and 0.34 for the latent-variable baselines, and the off-diagonal covariance magnitude is 0.983 ± 0.019 (Section 4.2, Table 1). The same Fano ordering holds on auditory cortex (Figure S4).

Critical weaknesses
  1. The millisecond-timing comparison is tied to output resolution. SpikeDiT emits 1 ms spikes. The baselines emit 20 ms counts, and the 1 ms metrics place those counts uniformly or at random inside the bin. The autocorrelogram gap (median 0.580.58 vs. at most 0.210.21) is therefore largely a property of the representation. The cross-correlogram gap is smaller (0.800.80 vs. at most 0.700.70). The protocol is reported, but the abstract and the contribution list still present millisecond timing as a result of competing with these baselines.

  2. The noise-correlation claim is strong for magnitude and for the Schmidt baseline, and weak for pattern agreement with the strongest latent model. On visual sessions the pattern correlations are 0.6450.645 (SpikeDiT), 0.6190.619 (Temporal FlowFA), and 0.4370.437 (Schmidt). The paper does not test the 0.6450.645 vs. 0.6190.619 gap across sessions. Covariance magnitude and the per-neuron Fano pattern (correlation with recorded Fano factors 0.870.87 vs. 0.600.60 and 0.340.34) carry the claim that a prescribed low-rank covariance is unnecessary. Absolute dispersion remains high: mean Fano factor about 1.431.43 against about 1.221.22 in the recordings.

Questions for authors
  1. Please compare temporal structure on the native 20 ms counts, or add a 1 ms baseline such as an independent Poisson process at 1 ms or a point process with spike history. That separates the representation from timing the model has learned.

  2. Were tokenizer architecture and hyperparameters frozen before any reconstruction numbers on Cowley and Wingert were inspected?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the discussion and final revision

We thank the reviewers for a discussion that strengthened both the experiments and their presentation. We have uploaded our final discussion-period revision, now titled “Stimulus-conditioned flow matching captures shared variability in population spike trains”. The paper asks the same scientific question throughout: can a single encoding model predict the mean response to a stimulus accurately while also learning how neurons vary together across repeats, at 1 ms resolution? This final reorganization retains the main figures and all results from the preceding revision. We changed the order of the results so that the controls added during rebuttal are now better integrated in the paper.

Here is a summary of the changes:

  • Title, abstract and Introduction. The title now names stimulus-conditioned flow matching. The abstract and Introduction now better lead with the scientific question of mean prediction and shared variability.
  • Main result and controls. Section 4.2 presents the central result across macaque visual cortex and ferret auditory cortex. Section 4.3 and Table 2 then ask which modeling choices support it. We compare generators operating on the same spike representation. Section 4.3 also tests how much the results depend on the data used to learn that representation (Appendix K.1). There is less emphasis on the importance of the tokenizer trained on the large corpus, in light with what the rebuttal findings showed.
  • Timing and completion. Sections 4.4 and 4.5 follow the learned population dependencies into two further questions: how accurately do the generated spikes capture the timing of shared fluctuations, and does observing some neurons help predict others on the same trial?
  • Opening and Conclusion. We added a short opening to Section 4 and a Conclusion that returns to what repeated trials let us measure.

This re-organization was directly shaped by the rebuttal’s experiments, for which we are grateful to the reviewers for suggesting:

  • Baseline tuning and statistical support. We swept learning rates for SpikeDiT and the main encoding baselines, retrained the count baselines at finer bin widths, and added paired session-level tests of noise-correlation pattern agreement (Appendices I and L). SpikeDiT is still the best generalist model.
  • Spike timing. We added 1 ms Poisson, spike-history, and coupling baselines and compared correlograms on a common 20 ms grid. These comparisons led us to narrow the timing claim and distinguish timing within a neuron from timing between neurons (Appendix I).
  • Prior methods and completion. We added a stimulus-conditioned LDNS-style comparison on four sessions and evaluated both latent-variable baselines on single-trial completion (Appendices K.3 and G).

In the final discussion, Reviewer vwTv judged the scientific content to meet the ICLR standard and suggested giving the joint generator greater prominence. Reviewer uWdH also recognized the improvement and raised their recommendation, while encouraging a similar shift in emphasis. We took that feedback seriously, and have spent the last day of rebuttal to go in that direction. The paper still retains its scientific question and main figures, but now places the controls added during rebuttal directly after the result they help explain, leading, in our opinion, to an overall better and more coherent presentation.

AuthorsOpenReview ↗

Closing summary of our contributions

We would like to close the discussion by summarising what joint generation adds to an encoding model. A deep encoder can accurately predict each neuron’s average response to a sound. If it samples neurons independently, it misses which pairs tend to fire above their own averages on the same trials. The latent-variable baselines model these correlations through a shared state whose structure is specified in advance. We asked whether a flow could more flexibly learn which neurons fluctuate together while preserving accurate mean predictions.

SpikeDiT represents each neuron’s spikes with a frozen tokenizer. Given features from an image or sound encoder, the flow generates tokens for all neurons and time windows together. Decoding one draw gives a population spike train on a 1 ms grid. We train on individual trials without repeat labels or a target covariance matrix. Across repeats, we can then compare the mean responses and noise-correlation patterns in generated and recorded trials.

ModelMean-response R², V4 / auditoryNoise-correlation pattern r, V4 / auditory
SpikeDiT0.627 / 0.4800.890 / 0.729
Temporal FlowFA0.544 / 0.4650.662 / 0.454
Schmidt model0.569 / 0.4900.546 / 0.177
Poisson (20 ms)0.590 / 0.496−0.002 / −0.017
Coupling GLM (1 ms)0.498 / 0.4710.293 / 0.289

Means are from Table 1 in the paper, over sessions with repeated stimuli.

SpikeDiT gives the strongest mean prediction in V4 and comes within 0.016 R² of the best auditory baseline. Its clearest advantage is in which neuron pairs fluctuate together. Noise-correlation pattern agreement exceeds both (previously state of the art) latent-variable baselines on both datasets in paired session-level tests on all sessions on held-out stimuli (Appendix I, Table S7, Holm-adjusted p ≤ 0.0049).

What explains that combination? The new controls in Section 4.3, and improved ablation Table 2 are particularly informative. With the tokenizer fixed, independent per-neuron flows retain most of the joint model’s mean-prediction accuracy (R² of 0.625 versus 0.638 across sessions), while noise-correlation pattern agreement falls from 0.890 to 0. A joint low-rank Gaussian over the same tokens recovers part of the pattern (r = 0.694), but predicts the mean less accurately and overestimates single-neuron dispersion. Removing the CRPS term also degrades calibration. By contrast, fitting the tokenizer on visual cortex training spikes gives similar downstream performance, including the same pattern agreement (r = 0.890). Together, these controls show what the generator and its training objective contribute, and why the shared tokenizer can serve as a reusable representation, even though it is not required for our flexible method to work.

The stronger comparisons also now clarify the limits of our approach better. Spike-history GLMs match or outperform SpikeDiT on within-neuron timing. Both latent-variable baselines benefit from observing part of the population during completion in V4, although SpikeDiT gives more accurate single-trial predictions.

In summary, the reviewers identified two plausible explanations for the central result: the large-corpus tokenizer might account for the gains, or the baselines might benefit from more controls and tuning. The target-dataset tokenizer gave similar performance, and after re-tuning, SpikeDiT still predicted mean responses competitively and recovered the noise-correlation pattern more accurately than both latent-variable baselines in visual and auditory cortex. At the same time, the stronger comparisons also showed where narrower or more precise claims could be made. In particular, spike-history GLMs matched or exceeded SpikeDiT on within-neuron timing, and we now are clear about this in the paper. We welcome the Area Chair’s independent assessment of the contribution in light of these results. We believe the evidence supports acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.