acceptodds
Under review as a conference paper at ICLR 2027

Why Do SNNs Convert but Not Train?

Abstract

Why does a spiking neural network (SNN) converted from a trained Transformer nearly retain its accuracy, but one trained from scratch fall behind a Transformer of the same size? If spikes set the limit, conversion and its long integration windows are the better route; if training sets it, the limit has to be located first. We study this question through the release identity of the spiking neuron. At the non-leaky endpoint of conversion a neuron integrates linearly, whereas in the leaky regime of training its release depends on when past inputs arrived, a dependence to be learned. By the chain rule, a past token reaches the prediction, and its gradient the parameters, only through four links, in which it is written into a membrane, released, read out and credited. For each link we derive why gradient training leaves it closed and which temporal prior opens it, and we build these priors into a spiking language model, NeuronSpark. Trained from scratch at four scales, NeuronSpark reaches the level of mainstream architectures of the same size in training loss and BabyLM evaluations. Register-transfer-level estimates give it the lowest energy per token, with an advantage that grows with scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
spiking neural networks, spiking language models, ANN-to-SNN conversion, training from scratch, surrogate gradients, temporal credit assignment, temporal priors, energy efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QeA9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why SNNs converted from pretrained ANNs can retain competitive performance, while directly trained SNNs often struggle. The authors analyze this gap through a release identity and four mechanisms: write, release, readout, and credit. Based on these insights, they propose NeuronSpark, which achieves performance comparable to Transformers across several model scales, with potential energy-efficiency benefits.

Critical strengths
  1. The paper addresses an interesting and important question in SNN research. The theoretical analysis provides a useful perspective on the challenges of training SNNs directly.
  2. The experiments are carefully organized, with mechanism-level analyses, ablation studies, and comparisons across multiple model scales. The detailed appendix is also appreciated.
Critical weaknesses
  1. While the four-link analysis is insightful, its implications for the full architecture could be better justified. NeuronSpark includes additional memory and residual pathways, and it is not entirely clear how these fit into the proposed explanation.
  2. The contribution of the spiking mechanisms to the final performance remains somewhat unclear. Appendix A.10 reports that many spiking FFN branches remain inactive, particularly at larger scales. This raises questions about how much of the improvement comes from the spiking components versus the additional memory and skip connections.
  3. The energy-efficiency results are encouraging, but they rely on RTL-based estimates that exclude memory traffic. Some caution may be needed when interpreting these results as practical energy savings.
Questions for authors
  1. Could the authors further clarify how the four-link framework applies to the complete NeuronSpark architecture, particularly its memory pathways?
  2. Would it be possible to include a more direct comparison with a non-spiking variant to better isolate the contribution of the spiking mechanisms?
  3. Could the authors comment on how memory-access costs might affect the estimated energy advantages?
4 comments
AuthorsOpenReview ↗

How the memory and residual pathways fit the four links

We thank the reviewer for reading the appendix carefully. Each question led to an addition to the paper.

1. "NeuronSpark includes additional memory and residual pathways, and it is not entirely clear how these fit into the proposed explanation."

Thank you for raising this; these pathways fit inside the same four links, the memory pathways as the readout link. Every pathway that carries a past token is either a write path or part of the readout link, and the residual stream and the skip connections add no path across tokens of their own, since a skip passes only the current release of its input neurons. The readout link is where the synapse pool reads the residual stream and carries a token to the prediction (Section 4.2), and Proposition 4.6 gives its requirement. Every pathway and the factor it enters:

Pathway (Appendix A.1, Figure 11)Crosses tokensFactor it enters
block current Winsq→IqW_{\mathrm{in}} s_q \to I_qyeswrite AtqA_{tq} (retention σ(Wdecays)\sigma(W_{\mathrm{decay}} s), threshold WthsW_{\mathrm{th}} s, soft reset) → release Ψt′\Psi_t' → readout diag(σ(Wgatest))Wout\mathrm{diag}(\sigma(W_{\mathrm{gate}} s_t)) W_{\mathrm{out}} → credit (the transposed product)
block skip WskipstW_{\mathrm{skip}} s_tno path of its ownpasses sts_t, the current release of the input neurons (A.3.1: "the skip term contains sts_t only")
block input neurons (token-projection gate)yesthe PLIF units themselves: write (ηt\eta_t), release (their release sts_t is the only signal entering the block), credit
Hebbian recall trace MtM_tyesan additional write path, with kernel k^t⊤diag(λ)t−sk^s\hat k_t^{\top} \mathrm{diag}(\lambda)^{t-s} \hat k_s in place of the retention product; release and readout unchanged; its readout gate contributes nothing when St=0S_t = 0 (A.3.4)
power-law history currentyesan additional write path, with kernel [(t−s)α−(t−s−1)α]/tα[(t-s)^{\alpha} - (t-s-1)^{\alpha}] / t^{\alpha} (A.3.4)
synapse pool: store mjm_j, read ∑jwt,jτjmj\sum_j w_{t,j} \tau_j m_jyesreadout (Proposition 4.6, constant share cc); reads the residual stream; the tag τj=σ(gj+bh−2)\tau_j = \sigma(g_j + b_h - 2) is set by the release of a bipolar PLIF gate neuron
value residual λn⊙mj(0)\lambda_n \odot m_j^{(0)}yes, through the poolreadout: an anchored path from the first memory layer's contents to the deep wide heads (Design 4.6)
extra patterns, content product termyes, through the poolreadout of the wide head (Table 7)
causal depthwise convolution (width 4) after the synaptic projectionsa fixed 3-token windowpart of the readout link, a fixed filter on what the pool stores and compares
centered input (running mean of the row)uniform averageonly the logits of the write gate ηt\eta_t and of the value-residual gate λn\lambda_n; not a content path
spiking gated FFNthrough its input neurons (slow time constants)input neurons: write and release; the gate × up product: release (the exact zero-gradient state of A.10); the linear skip Wskip′st′W_{\mathrm{skip}}' s_t' passes their current release, as the block skip does
residual stream, embedding, LM head, RMSNormnowithin a token

We thank the reviewer for this question. The paper did not spell this out pathway by pathway, and the revision adds this table to Appendix A.3, which makes the scope of the lemma explicit.

AuthorsOpenReview ↗

The spiking mechanisms contribute on their own

2. "…how much of the improvement comes from the spiking components versus the additional memory and skip connections."

We trained the non-spiking variant the reviewer suggested in two forms, and the spiking mechanisms contribute on their own, since the final loss rises by 0.051 nats when every neuron is removed and by 0.058 nats when every membrane is kept and only its threshold and reset are removed. The setting is the paper's 200M model and protocol with every matrix kept. The first twin removes the neurons with their thresholds and retention gates. The second keeps each membrane and its retention gate as a hidden state and passes the membrane on without threshold or reset, so the loss it shows cannot come from a missing state. With a one-sided release the same removal changes nothing, so the neurons help once their release link is open. The result is below:

Three seeds, mean ± standard deviationFull model, bipolar ↓\downarrowOne-sided release ↓\downarrowNo-neuron twin ↓\downarrowMembrane-only twin ↓\downarrowNo-neuron − fullMembrane-only − fullNo-neuron − one-sided
Final loss2.883 ± 0.0412.936 ± 0.0432.934 ± 0.0382.941 ± 0.041+0.051 ± 0.002+0.058 ± 0.006−0.002 ± 0.006

We also followed the FFN branches that Appendix A.10 reports as inactive through a longer run, and their gate neurons release more as training continues, with a firing rate of 0.16 at 10B tokens against 0.03 at 2B. The setting is a 2.0B-parameter NeuronSpark trained on MiniCPM's open pretraining data (Ultra-FineWeb, Wang et al., 2025; UltraData, Wang et al., 2026), with the firing rate read at every validation point. The result is below (share of neuron–token pairs above threshold):

Tokens2B4B6B8B10B
FFN gate neurons0.0300.0560.0990.1350.158

We are grateful for this suggestion. The comparison isolates the contribution of the spiking mechanisms more directly than anything in the submission, and the revision adds it to a new subsection at the end of Section 5.

AuthorsOpenReview ↗

Memory access leaves the energy advantage almost unchanged

3. "Could the authors comment on how memory-access costs might affect the estimated energy advantages?"

They leave the advantage almost unchanged, with NeuronSpark at 0.38–0.72 of the Transformer's energy per token once memory access is implemented in VHDL for all three models, against 0.39–0.73 with operators only. The setting is the paper's procedure extended to memory. Every memory unit is verified bit by bit against a software reference in simulation and synthesized into logic cells, and an access costs the cell count of the storage bank that serves it, as an operation costs the cell count of its unit. The result is below (words moved per token and energy relative to the Transformer; each column serves every access from storage banks of the stated size):

Scale, contextModelWeight words (M) ↓\downarrowState words (M) ↓\downarrowEnergy, operators only ↓\downarrow64-word banks ↓\downarrow8,192-word banks ↓\downarrow
200M, 1,024Transformer199.725.2111
200M, 1,024Mamba-2199.59.20.9490.9350.926
200M, 1,024NeuronSpark144.617.50.7320.7250.720
2.0B, 1,024Transformer1,984.9102.9111
2.0B, 1,024Mamba-21,980.983.81.0100.9970.989
2.0B, 1,024NeuronSpark895.932.10.4550.4480.445
2.0B, 8,192Transformer1,984.9822.2111
2.0B, 8,192Mamba-21,980.983.80.7510.7400.733
2.0B, 8,192NeuronSpark895.9176.60.3890.3840.381

The advantage persists because NeuronSpark moves fewer words than the Transformer in each kind of storage. It reads only the weights of non-zero releases, and its pool and membrane potentials move fewer words per token than the key-value cache. The advantage does not rest on the FFN branches of Appendix A.10 staying inactive, since counting every one of them as fully active raises the bound by at most 0.023. The result is below (NeuronSpark relative to the Transformer; the last two columns are upper bounds of the energy ratio, as measured and with every FFN branch fully active, and hold whatever a memory access costs, provided it costs both models the same):

ScaleOperators (1,024 / 8,192 tokens)Weight wordsState and activation words (1,024 / 8,192 tokens)Largest of the three: upper bound of the energy ratioUpper bound with every FFN branch fully active
200M0.732 / 0.6760.7240.691 / 0.6150.730.75
500M0.603 / 0.4550.6210.303 / 0.2330.620.64
1.3B0.488 / 0.4190.4850.360 / 0.2540.490.51
2.0B0.455 / 0.3890.4510.318 / 0.2140.460.48

We thank the reviewer for this question, which led to the memory analysis that the energy section lacked. The revision adds it to Appendix A.9.5.

Indeed, repeating at the other scales, with several seeds, the experiments that ran at one scale would help us show that the results hold at every scale, and we would very much like to provide them. The ablations, the two twins and the other-SNN runs at 200M and the pretraining to 10B tokens at 2.0B, each with several seeds, would take about 3,400 hours on one B300 GPU to repeat, which we cannot finish within the discussion period.

We are here for an in-depth discussion and further feedback. We would be glad to answer and address any issue you raise, and to keep improving the paper together with you.

Dear Reviewer QeA9,

We greatly appreciate your constructive feedback, which has given the paper a pathway-by-pathway account of how the four links apply to the complete NeuronSpark architecture and its memory pathways, two non-spiking variants that isolate the contribution of the spiking mechanisms, evidence that the inactive spiking FFN branches become active with longer training, and an energy analysis that includes memory-access costs.

As the Author–Reviewer Discussion period closes in a few hours, on November 18 AoE, we would be grateful if you could review our latest responses and the revised paper, and update your assessment if your remaining concerns have been adequately addressed.

We hope that the additional evidence and clarifications provide sufficient grounds for a favorable reconsideration.

Sincerely,

The Authors

Reviewer_T7kWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper starts from the release mechanism of spiking neurons and asks why converted SNNs can usually preserve the performance of ANNs, while SNNs trained from scratch often fall behind Transformers of the same size. The authors break the influence of past information on the final prediction into four links: writing, release, readout, and credit assignment. They then design a corresponding temporal prior for each link and build these ideas into a spiking language model called NeuronSpark. Across four model sizes, NeuronSpark reaches performance close to that of Transformers and Mamba-2 on training loss and BabyLM evaluations, while achieving lower estimated energy per token. However, it still performs worse on previously unseen target types.

Critical strengths

The paper combines theoretical derivations, mechanism-level measurements, component ablations, and experiments at multiple model scales.
Its analysis provides a systematic explanation of the gap between ANN-to-SNN conversion and direct SNN training by linking it to four specific gradient-flow mechanisms.

Critical weaknesses

The energy results are based on register-transfer-level and hardware-parameter estimates, rather than measurements on an actual neuromorphic chip.
The four links and their corresponding priors are derived within the specific NeuronSpark architecture. Whether the same explanation applies to all directly trained SNNs remains to be demonstrated.

Questions for authors

The energy results are based on register-transfer-level and hardware-parameter estimates, rather than measurements on an actual neuromorphic chip.
The four links and their corresponding priors are derived within the specific NeuronSpark architecture. Whether the same explanation applies to all directly trained SNNs remains to be demonstrated.

5 comments
AuthorsOpenReview ↗

Energy beyond RTL simulation, with FPGA results to come

We thank the reviewer for a summary that states the paper's logic as intended. Both points led to additions to the paper.

1. "The energy results are based on register-transfer-level and hardware-parameter estimates, rather than measurements on an actual neuromorphic chip."

Indeed, a chip measurement would help us show that the energy advantage holds not only in simulation but also on real hardware, and we would very much like to provide it. We are implementing the same VHDL on an FPGA, which takes about five days, and before the discussion period ends we expect to post the FPGA energy of every unit and, from it, the energy per token of the three models at all four scales and both context lengths. In fact, comparisons made at RTL are known to hold on measured hardware (Govindan et al., 2009), so the FPGA does not change the conclusion of the VHDL simulation. Each model does the same work there, and the FPGA only changes how much energy one multiplication or one memory access takes. That energy is the same for all three models, which use the same multiply, lookup and memory units, and NeuronSpark does less work of every kind than the Transformer. The current VHDL results therefore stand, and NeuronSpark uses at most 0.73 of the Transformer's energy per token. The advantage does not rest on the FFN branches of Appendix A.10 staying inactive, since counting every one of them as fully active raises the bound by at most 0.023. The result is below (NeuronSpark relative to the Transformer; the last two columns are upper bounds of the energy ratio, as measured and with every FFN branch fully active, and hold whatever a memory access costs, provided it costs both models the same):

ScaleOperators (1,024 / 8,192 tokens)Weight wordsState and activation words (1,024 / 8,192 tokens)Largest of the three: upper bound of the energy ratioUpper bound with every FFN branch fully active
200M0.732 / 0.6760.7240.691 / 0.6150.730.75
500M0.603 / 0.4550.6210.303 / 0.2330.620.64
1.3B0.488 / 0.4190.4850.360 / 0.2540.490.51
2.0B0.455 / 0.3890.4510.318 / 0.2140.460.48

Other reviewers asked about memory, and the advantage also holds once it is implemented in VHDL for all three models, with NeuronSpark at 0.38–0.72 of the Transformer's energy per token against 0.39–0.73 without it. The setting is the paper's procedure extended to memory. Every memory unit is verified bit by bit against a software reference in simulation and synthesized into logic cells, and an access costs the cell count of the storage bank that serves it, as an operation costs the cell count of its unit. The result is below (energy per token relative to the Transformer at 1,024 / 8,192 tokens; each column serves every access from storage banks of the stated size):

ScaleModel, Transformer = 1 ↓\downarrowOperators only (1,024 / 8,192 tokens)64-word banks1,024-word banks8,192-word banks
200MMamba-20.949 / 0.5320.935 / 0.5220.927 / 0.5170.926 / 0.517
200MNeuronSpark0.732 / 0.6760.725 / 0.6720.721 / 0.6690.720 / 0.669
500MMamba-20.984 / 0.6000.967 / 0.5880.957 / 0.5810.956 / 0.581
500MNeuronSpark0.603 / 0.4550.596 / 0.4500.592 / 0.4470.592 / 0.447
1.3BMamba-21.017 / 0.7361.003 / 0.7240.995 / 0.7180.994 / 0.717
1.3BNeuronSpark0.488 / 0.4190.481 / 0.4140.478 / 0.4110.478 / 0.411
2.0BMamba-21.010 / 0.7510.997 / 0.7400.989 / 0.7330.989 / 0.733
2.0BNeuronSpark0.455 / 0.3890.448 / 0.3840.445 / 0.3810.445 / 0.381

We thank the reviewer for this point, which led us to show that the advantage over the Transformer holds whatever memory costs. The revision adds this bound and the memory traffic to Appendix A.9.5.

AuthorsOpenReview ↗

The same explanation holds in four other directly trained SNNs

2. "Whether the same explanation applies to all directly trained SNNs remains to be demonstrated."

We agree that this remained to be demonstrated, and we now demonstrate it on SpikeGPT (Zhu et al., 2024), SpikingSSMs (Shen et al., 2025), LongSpike (He et al., 2026) and P-SpikeSSM (Bal & Sengupta, 2025), four other directly trained SNNs that each violate at least two of the four requirements and reach a lower loss in every cell once the corresponding prior is added. The lemma itself is proved for any soft-reset LIF or PLIF block whose membrane carries information across tokens (Appendix A.3). SpikeLM (Xing et al., 2024), SpikingBERT (Bal & Sengupta, 2024) and SpikeBERT (Lv et al., 2023) fall outside the lemma, because their membranes integrate T repeated steps of one token and carry nothing across tokens, the regime of conversion (Proposition 4.1). Which requirement each published design violates:

ArchitectureWrite: token-dependent retentionRelease: bipolarCredit: slope matched to the membrane spreadReadout: continuous value path across layers
SpikeGPTno: fixed τ; token-independent RWKV decayno: one-sided binaryyes: ATan α = 2, within 10% of the matched 1.80yes: continuous WKV output
SpikingSSMsno: τ = 0.125 fixedno: one-sided binaryno: α = 1 vs the matched 0.37no: every SSM output passes through a binary neuron
LongSpikeno: τ fixedno: one-sided binaryno: α = 1 fixedno: as SpikingSSMs
P-SpikeSSMretention is the method's own state matrixno: one-sided Bernoulli samplernot fixed by the design: testedno: binary spikes before the mixer

The setting is the paper's protocol (BabyLM, one epoch, three seeds), with each published design rebuilt from its public code, sized to 200M parameters with a tied embedding, and each prior added to it. P-SpikeSSM was published for classification, so it takes our token embedding and LM head, and LayerNorm replaces its BatchNorm, which would read future tokens in a causal model. The release requirement is met in two forms, once with the spikes kept binary, as +1 or −1, and once graded, and the loss falls in both. The result is below (seed-paired change in final loss against the published design, mean ± standard deviation over three seeds; negative is better; paired noise floor 0.007 nats, Appendix A.9.6):

ModelWriteRelease, binary spikesRelease, graded spikesCreditReadoutAll together
SpikeGPT−0.030 ± 0.001−0.034 ± 0.004−0.150 ± 0.006——−0.163 ± 0.004
SpikingSSMs−0.014 ± 0.005−0.084 ± 0.006−0.172 ± 0.004−0.005 ± 0.004−0.052 ± 0.007−0.229 ± 0.008
LongSpike−0.017 ± 0.004−0.088 ± 0.003−0.173 ± 0.007−0.008 ± 0.003−0.057 ± 0.006−0.231 ± 0.008
P-SpikeSSM—−0.299 ± 0.008—−0.075 ± 0.002−0.071 ± 0.002−0.335 ± 0.007

A dash marks a prior that does not apply, either because the design already meets the requirement or because meeting it would replace the method's own contribution.

The same explanation also holds inside these trained models, where each violation is measurable and is relieved by the same prior that lowers the loss, as the retention gate, fixed before, varies across tokens with a standard deviation of 0.205 to 0.239, the units that never release and the starved channels fall, and the added readout path stays in use. A starved channel has a mean surrogate gradient below 0.05. The result is below (final checkpoints, mean ± standard deviation over three seeds; published design → with the prior):

ModelWrite: standard deviation of the retention gate across tokensRelease, binary spikes: units that never releaseRelease, graded spikes: units that never releaseCredit: starved channelsReadout: coefficient of the added path (launched at 0.5)
SpikeGPT0 → 0.239 ± 0.00535 ± 9 → 0 ± 1 of 17,85635 ± 9 → 2 ± 2 of 17,856——
SpikingSSMs0 → 0.214 ± 0.00315 ± 4 → 0 ± 1 of 20,44015 ± 4 → 0 ± 1 of 20,4401.2 ± 0.0% → 0.8 ± 0.0%none → 0.51 ± 0.00
LongSpike0 → 0.205 ± 0.00314 ± 3 → 0 ± 0 of 20,44014 ± 3 → 1 ± 1 of 20,4401.5 ± 0.1% → 1.0 ± 0.1%none → 0.51 ± 0.00
P-SpikeSSM—184 ± 4 → 0 ± 0 of 20,700—13.1 ± 0.2% → 1.7 ± 0.1%none → 0.56 ± 0.00

This request gave the paper its first evidence outside NeuronSpark, for which we thank the reviewer. The revision adds it to a new subsection at the end of Section 5.

AuthorsOpenReview ↗

Experiments at further scales and further discussion

Indeed, repeating at the other scales, with several seeds, the experiments that ran at one scale would help us show that the results hold at every scale, and we would very much like to provide them. The ablations, the two twins and the other-SNN runs at 200M and the pretraining to 10B tokens at 2.0B, each with several seeds, would take about 3,400 hours on one B300 GPU to repeat, which we cannot finish within the discussion period.

We are here for an in-depth discussion and further feedback. We would be glad to answer and address any issue you raise, and to keep improving the paper together with you.

Reviewer_Z5aLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper relates conversion and direct SNN training through the neuron release dynamics, then uses a four-link analysis to motivate the components of NeuronSpark.
Critical strengths
  • The mechanism tests are the most interesting part for me. The paper connects specific failure modes to interventions and checks several predictions directly, rather than relying only on final benchmark scores. The shared training protocol and three seeds are also helpful.
Critical weaknesses
  • The evidence is too narrow to support the broad explanation of why directly trained SNNs lag behind converted models. All model scales are trained for one epoch on the same BabyLM corpus. Reaching the Transformer's loss in this setting does not establish parity after substantial pretraining, or show that the same four bottlenecks explain failures in other directly trained SNN architectures. This limits the main claim, rather than simply leaving a larger-scale experiment for future work.

  • The causal attribution is also incomplete: the bipolar-release result in Table 7 is bundled with other implementation changes. In addition, the energy estimate omits memory traffic, so it does not establish a deployment-energy advantage for the synapse-pool architecture. The mechanism tests are interesting, but the current evidence is insufficient for the paper's broader conclusions. I recommend weak rejection.

Questions for authors
  • Could the authors report an isolated bipolar-release ablation and paired confidence intervals for the loss differences? Being within the Transformer's seed spread is not itself an equivalence test.
6 comments
AuthorsOpenReview ↗

Directly trained SNNs lagging behind conversion is a consensus

We thank the reviewer for the careful reading. Each point below led to a concrete addition to the paper.

1. "The evidence is too narrow to support the broad explanation of why directly trained SNNs lag behind converted models."

We apologize for citing only a few works for this claim; the lag of directly trained SNNs behind converted ones is a consensus in SNN research. It has been raised repeatedly at major venues, in the papers below:

VenuePaperReported evidence
Frontiers in Neuroscience 2017Rueckauer et al., conversionspike-based training "yet to mature to the state where training spiking architectures of the size of VGG-16 becomes possible"; converted Inception-v3 74.6% ImageNet top-1
Frontiers in Neuroscience 2018Pfeiffer & Pfeil, survey"Network conversion has set most benchmark records in terms of accuracy for SNNs"
Nature 2019Roy, Jaiswal & Panda, survey"conversion approaches yield state-of-the-art accuracy … supervised learning … has not been able to"
IEEE Signal Processing Magazine 2019Neftci, Mostafa & ZenkeSNNs are less used "primarily due to the algorithmic issue of trainability"
AAAI 2019Wu et al., direct training (STBP)"SNNs have not shown competitive performance compared with artificial neural networks"
ICLR 2020Rathi et al., hybridconversion followed by spike-based fine-tuning, because training from scratch does not reach it
CVPR 2020Han, Srinivasan & Roy, conversion"error backpropagation over time has limited their scalability for SNNs beyond 9–11 layers"; converted VGG-16 73.09% vs ANN 73.49%
Neural Networks 2020Deng et al.the same CNN on CIFAR-10: ANN 78.16%, converted SNN 76.81%, directly trained SNN 63.19%
ICLR 2021Deng & Gu, conversionSNNs are "difficult to train directly from scratch … difficulty in convergence for large networks such as VGG and ResNet"
ICML 2021Li et al., conversion"training SNN from scratch still lacks the scalability to obtain an effective SNN on the ImageNet dataset"
AAAI 2021Zheng et al., direct training (tdBN)directly trained ResNet-34 63.72% vs converted ResNet-34 71.61% ImageNet top-1
NeurIPS 2021Fang et al., direct training (SEW)SEW-ResNet-34 67.04% vs converted Spiking ResNet-34 74.61%
Nature Machine Intelligence 2021Stöckl & Maass, conversion"direct training of deep spiking neural networks is difficult … has not been able to reach similar classification accuracy"
IEEE TPAMI 2022Wu et al., tandem learningBPTT's "applicability to deep SNNs with more than 10 hidden layers remains elusive"
AAAI 2022Bu et al., conversion"The most effective way to train deep SNNs is through ANN-to-SNN conversion"
ICLR 2022Deng et al., direct training (TET)the spiking mechanism is "failing the surrogate gradient methods to achieve comparable accuracy as for ANNs"
ICLR 2023Zhou et al., SpikformerSpikformer 74.81% vs an ANN Transformer of the same design 80.80% ImageNet top-1
Proceedings of the IEEE 2023Eshraghian et al., tutorial"ANN-to-SNN conversion currently takes the crown for static image classification tasks on complex datasets"
ICML 2024You et al., SpikeZIP-TFconverted Transformer 83.82% vs directly trained Spikformer V2 82.35%
IEEE TPAMI 2025Yao et al., E-SpikeFormer"the performance gap between SNNs and ANNs … intrinsic flaws in spiking neurons caused by binary firing mechanisms"
AAAI 2026Wang et al., conversionWikiText-103: converted GPT-2 23.41 ppl vs directly trained SpikeGPT 39.75 (ANN 22.65)
TMLR 2024Zhu et al., SpikeGPTSNNs' "performance lags behind modern deep learning"; WikiText-103 39.75 ppl
ICLR 2023Lv et al., spiking text classification"it is hard to train spiking neural networks for language tasks directly … the conversion step is indispensable"
AAAI 2024Bal & Sengupta, SpikingBERTdistilled from BERT; "without using the proposed KD, there is at least 4% to 5% drop in accuracy across all datasets"
ICML 2024Xing et al., SpikeLMthe LIF-BERT baseline trails BERT by 28.3% on GLUE, SpikeLM by 6.7%
AAAI 2025Shen et al., SpikingSSMsWikiText-103: Transformer 20.51, S4 20.95 vs SpikeGPT 39.75, SpikingSSM 33.94 ppl
TMLR 2026Huang et al., SpikingMambadistilled from Mamba-2 because "recovering accuracy typically requires full pretraining"

We thank the reviewer for pointing this out. The revision cites these papers in the first paragraph of the introduction, which makes clear what the paper takes as given.

AuthorsOpenReview ↗

Parity holds after pretraining to 10B tokens

2. "Reaching the Transformer's loss in this setting does not establish parity after substantial pretraining."

The one-epoch BabyLM budget is a standard data budget in language-model research, and parity holds after substantial pretraining. It is the budget of the BabyLM shared task, now in its fourth edition (Warstadt et al., 2023; Hu et al., 2024; Charpentier et al., 2025; Choshen et al., 2026), and SpikingSSMs (Shen et al., 2025) and LongSpike (He et al., 2026) train on a corpus of the same order (WikiText-103, about 103M tokens). It is enough to measure language modeling, since language models "require only about 10M to 100M words to learn to reliably encode most syntactic and semantic features" (Zhang et al., ACL 2021).

We also demonstrate the result at 61× this budget. The setting is one run each, the most our compute allows at this scale, of a 2.0B-parameter NeuronSpark and a 2.0B-parameter Transformer trained on the same pipeline to 10B tokens of MiniCPM's open pretraining data (Ultra-FineWeb, Wang et al., 2025; UltraData, Wang et al., 2026). The result is below (validation loss, NeuronSpark / Transformer):

Tokens2B4B6B8B10B
Mean over the 7 training sources ↓\downarrow2.573 / 2.7382.320 / 2.4012.198 / 2.2582.150 / 2.2042.116 / 2.164
Mixture-weighted (held-out training loss) ↓\downarrow3.000 / 3.1762.732 / 2.8212.601 / 2.6672.548 / 2.6102.508 / 2.563

NeuronSpark has the lower loss on each of the 15 validation sources, 7 from the training mixture and 8 never trained on:

Validation source (10B tokens)In the training mixtureNeuronSpark ↓\downarrowTransformer ↓\downarrow
Ultra-FineWeb, Englishyes2.8142.868
Ultra-FineWeb, recent Englishyes2.3212.369
Ultra-FineWeb, Chineseyes3.1533.226
UltraData-Math L1yes2.4342.483
UltraData-Math L2yes2.1832.223
UltraData-Code L2yes0.7720.811
UltraData-Stackyes1.1321.165
Ultra-FineWeb-L3, English question-answer rewritesno2.3412.365
Ultra-FineWeb-L3, English style rewritesno2.6732.728
Ultra-FineWeb-L3, Chinese question-answer rewritesno2.8242.867
Ultra-FineWeb-L3, Chinese style rewritesno3.3053.385
UltraXno3.0793.140
UltraData-Math L3no1.1611.193
UltraData-Code L3no1.4921.527
InfLLM long-context mixno1.9862.019

A power-law fit of each loss curve, L(D)=E+AD−αL(D) = E + AD^{-\alpha}, puts the Transformer's asymptote above NeuronSpark's:

Fitted asymptote EENeuronSpark ↓\downarrowTransformer ↓\downarrow
Mean over the 7 training sources1.8801.964
Mixture-weighted2.2312.327
Validation sources where the Transformer's asymptote is higher15/15

We are grateful for this comment. It prompted this comparison, which shows that the result is not an artifact of the BabyLM budget. The revision adds it to a new subsection at the end of Section 5 and states the one-epoch scope in the abstract and introduction.

AuthorsOpenReview ↗

The four bottlenecks appear in four other directly trained SNNs

3. "…or show that the same four bottlenecks explain failures in other directly trained SNN architectures."

We agree that this needed evidence beyond NeuronSpark, and we now have it from SpikeGPT (Zhu et al., 2024), SpikingSSMs (Shen et al., 2025), LongSpike (He et al., 2026) and P-SpikeSSM (Bal & Sengupta, 2025), four other directly trained SNNs that each violate at least two of the paper's four requirements and reach a lower loss in every cell once the corresponding prior is added. SpikeLM (Xing et al., 2024), SpikingBERT (Bal & Sengupta, 2024) and SpikeBERT (Lv et al., 2023) fall outside the lemma, because their membranes integrate T repeated steps of one token and carry nothing across tokens, the regime of conversion (Proposition 4.1). Which requirement each published design violates:

ArchitectureWrite: token-dependent retentionRelease: bipolarCredit: slope matched to the membrane spreadReadout: continuous value path across layers
SpikeGPTno: fixed τ; token-independent RWKV decayno: one-sided binaryyes: ATan α = 2, within 10% of the matched 1.80yes: continuous WKV output
SpikingSSMsno: τ = 0.125 fixedno: one-sided binaryno: α = 1 vs the matched 0.37no: every SSM output passes through a binary neuron
LongSpikeno: τ fixedno: one-sided binaryno: α = 1 fixedno: as SpikingSSMs
P-SpikeSSMretention is the method's own state matrixno: one-sided Bernoulli samplernot fixed by the design: testedno: binary spikes before the mixer

The setting is the paper's protocol (BabyLM, one epoch, three seeds), with each published design rebuilt from its public code, sized to 200M parameters with a tied embedding, and each prior added to it. P-SpikeSSM was published for classification, so it takes our token embedding and LM head, and LayerNorm replaces its BatchNorm, which would read future tokens in a causal model. The release requirement is met in two forms, once with the spikes kept binary, as +1 or −1, and once graded, and the loss falls in both. The result is below (seed-paired change in final loss against the published design, mean ± standard deviation over three seeds; negative is better; paired noise floor 0.007 nats, Appendix A.9.6):

ModelWriteRelease, binary spikesRelease, graded spikesCreditReadoutAll together
SpikeGPT−0.030 ± 0.001−0.034 ± 0.004−0.150 ± 0.006——−0.163 ± 0.004
SpikingSSMs−0.014 ± 0.005−0.084 ± 0.006−0.172 ± 0.004−0.005 ± 0.004−0.052 ± 0.007−0.229 ± 0.008
LongSpike−0.017 ± 0.004−0.088 ± 0.003−0.173 ± 0.007−0.008 ± 0.003−0.057 ± 0.006−0.231 ± 0.008
P-SpikeSSM—−0.299 ± 0.008—−0.075 ± 0.002−0.071 ± 0.002−0.335 ± 0.007

A dash marks a prior that does not apply, either because the design already meets the requirement or because meeting it would replace the method's own contribution.

The same four bottlenecks are also measurable on the trained models of these architectures, and each is relieved by the same prior that lowers the loss, as the retention gate, fixed before, varies across tokens with a standard deviation of 0.205 to 0.239, the units that never release and the starved channels fall, and the added readout path stays in use. A starved channel has a mean surrogate gradient below 0.05. The result is below (final checkpoints, mean ± standard deviation over three seeds; published design → with the prior):

ModelWrite: standard deviation of the retention gate across tokensRelease, binary spikes: units that never releaseRelease, graded spikes: units that never releaseCredit: starved channelsReadout: coefficient of the added path (launched at 0.5)
SpikeGPT0 → 0.239 ± 0.00535 ± 9 → 0 ± 1 of 17,85635 ± 9 → 2 ± 2 of 17,856——
SpikingSSMs0 → 0.214 ± 0.00315 ± 4 → 0 ± 1 of 20,44015 ± 4 → 0 ± 1 of 20,4401.2 ± 0.0% → 0.8 ± 0.0%none → 0.51 ± 0.00
LongSpike0 → 0.205 ± 0.00314 ± 3 → 0 ± 0 of 20,44014 ± 3 → 1 ± 1 of 20,4401.5 ± 0.1% → 1.0 ± 0.1%none → 0.51 ± 0.00
P-SpikeSSM—184 ± 4 → 0 ± 0 of 20,700—13.1 ± 0.2% → 1.7 ± 0.1%none → 0.56 ± 0.00

This request gave the paper its first evidence outside NeuronSpark, for which we thank the reviewer. The revision adds it to the new subsection of Section 5.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of the revision

We thank all reviewers for their valuable comments. Following them, the revision tests the four links on four other directly trained SNNs, isolates the release rule and the spiking mechanisms, replaces the seed-spread criterion with an equivalence test, extends the comparison to 10B tokens, adds memory traffic to the energy estimate and measures it on an FPGA, maps every pathway of the full network to the four links, and states the scope of the claims and the prior evidence they build on. The revised paper is uploaded. Every change is marked in blue, Figures 1–16 and Tables 1–7 keep their numbers, the new results are in Figures 17–18 and Tables 8–18, and Table 2 adds rows for the new tests (lines 899–909). The additions and where they are:

AdditionRaised byMain textAppendix
(a)Four other directly trained SNNs (SpikeGPT, SpikingSSMs, LongSpike, P-SpikeSSM), each with the priors addedZ5aL, T7kWSection 5.1, Table 8(a, c); lines 481–506A.11.1, Tables 14–16; lines 2349–2420
(b)The release rule alone (bipolar release, only the release rule changed)Z5aLSection 5.1, Table 8(b); lines 486–499, 507–509A.10, new row of Table 7; lines 2313–2329. A.11.2, Table 17; lines 2422–2449
(c)Equivalence test, in place of "within the Transformer's seed spread"Z5aLSection 5; lines 445–447A.9.4, Table 10; lines 2023–2036
(d)Non-spiking twinsQeA9Section 5.1, Table 8(b); lines 486–499, 507–511A.11.2, Table 17; lines 2422–2449
(e)Pretraining to 10B tokensZ5aLSection 5.1, Figure 17, Table 8(d); lines 486–499, 511–524A.11.3, Figure 18, Table 18; lines 2451–2500
(f)Firing rates of the inactive FFN branches over the 10B runQeA9A.10; lines 2271–2274
(g)Memory traffic in the energy estimate and the bound on the energy ratioZ5aL, T7kW, QeA9Section 5; line 481A.9.5, Tables 11–12; lines 2044, 2064–2105
(h)Pathways of the full networkQeA9A.3.5, Table 9; lines 1134–1169. Scope in A.3.6; lines 1176–1181
(i)One-epoch BabyLM scopeZ5aLAbstract, Sections 1 and 6; lines 19–20, 78, 536–537A.9.6, the BabyLM data budget; lines 2147–2151
(j)Evidence that direct training lags conversion; spiking state-space modelsZ5aL, T7kWSections 1 and 2; lines 42–47, 113–114
(k)FPGA measurement of the energy and time per tokenT7kWA.9.5, Table 13; lines 2117–2139
(l)Scope of the four links in the conclusionZ5aLSection 6; lines 537–538

We are here for an in-depth discussion and would be glad to answer any further question.

AuthorsOpenReview ↗

Looking forward to reviewers' feedback before the discussion period ends

Dear Reviewers,

We thank you again for your time and your careful reviews. We have replied to each of your points and uploaded the revised paper, and our Summary of the revision lists every addition, who raised it and where it is in the paper. As the discussion period is coming to an end, we would be grateful if you could let us know whether our replies address your concerns and, if they do, whether you would consider revisiting your rating.

We are here for an in-depth discussion and would be glad to answer any further question.

AuthorsOpenReview ↗

Final summary: what the paper achieves

We thank the reviewers and the area chair for their time and careful reading. The paper asks why an SNN converted from a trained Transformer keeps its accuracy while one trained from scratch falls behind, and finds that the limit lies in what training must learn, not in what spikes can represent. The table follows this answer step by step.

QuestionWhat the paper establishesEvidenceAsked by
Does training from scratch lag behind conversion?Yes, in vision and language, as 27 papers from 2017 to 2026 reportSection 1Z5aL
Why?A neuron integrates linearly only at the non-leaky endpoint of conversion, and with leak its release depends on when past inputs arrived, a dependence that training from scratch must learnProp. 4.1; Fig. 3
Through which paths?A past token reaches the prediction only through four links, write, release, readout and credit. Their product misses none of 21,656 Jacobians against finite differences, their closed forms match simulation within 0.007, and every pathway of the full network falls inside themLemma 4.2; Figs. 4–5; Table 9QeA9
Where does the gap form?Over past tokens, not the current one. On new targets a network without the write, credit and readout priors matches the Transformer with one token of context and falls 0.166 nats behind with 32Fig. 6(a)
What closes the write link, and what opens it?A static retention gate, whose gradient cancels across tokens so that 4–7% survives a batch. A token projection gate opens it and recalls a token past 32 distractors at 0.996, where static and element-wise gates stay at chanceProp. 4.3; Fig. 6(b, c)
And the release link?A one-sided release, which discards the negative half of the drive and leaves units that no gradient reaches. A bipolar release opens it and cuts the units that never release from 1,413 to 23Prop. 4.4; Fig. 7(a, b)
And the credit link?A surrogate too steep for slow membranes, which starves 43% of the channels at slope 96. A slope matched to the membrane noise opens it, starving none at slope 8 and lowering the final lossProp. 4.5; Fig. 7(c, d)
And the readout link?Deep wide heads whose readout is 18–62% constant. An anchored value residual opens it, and replacing the contents it carries costs the wide heads 0.35 of 0.40 natsProp. 4.6; Fig. 8
Does opening the links close the gap?Yes. Each prior lowers the loss by 0.002–0.116 nats, and NeuronSpark, the spiking language model built with all of them, is equivalent to the Transformer within ±0.033 nats at every scale from 200M to 2.0B in two one-sided tests, as close as Mamba-2, with a BabyLM average within 0.1 points of the Transformer's at 200M and above it at the larger scalesFigs. 9–10; Tables 1 and 10Z5aL
Is this specific to NeuronSpark?No. The links hold for any SNN whose membranes carry information from one token to the next, and in SpikeGPT, SpikingSSMs, LongSpike and P-SpikeSSM, each violating at least two requirements, every prior lowers the loss, by 0.163–0.335 nats all together, and relieves its bottleneckTables 8(a, c) and 14–16Z5aL, T7kW
Do the spiking mechanisms contribute?Yes, once the release link is open. The release rule alone lowers the loss by 0.053 ± 0.005 nats (paired 95% CI [−0.066, −0.041]), and with every matrix kept, removing the neurons costs 0.051 nats and removing only threshold and reset 0.058Tables 8(b) and 17Z5aL, QeA9
Does the parity last beyond one epoch?Yes. Trained to 10B tokens at 2.0B, 61× one BabyLM epoch, NeuronSpark has the lower loss on every logged training batch, at every checkpoint, on all 15 validation sources (8 never trained on) and in the fitted asymptote of every sourceFigs. 17–18; Tables 8(d) and 18Z5aL
Does NeuronSpark save energy?Yes. It uses the least energy per token of the three at 1,024 tokens, and the gap widens with scale. With memory traffic in VHDL it uses 0.38–0.72 of the Transformer's energy, at most 0.73 whatever a memory access costs, provided it costs both models the same, or 0.75 with every FFN branch fully active. On an FPGA from 200M to 1.3B it uses 0.48–0.73 at 1,024 tokens and 0.42–0.67 at 8,192Fig. 13; Tables 11–13Z5aL, T7kW, QeA9
What is left?NeuronSpark leads the Transformer on copy targets and ties on seen ones, and the gap left on new targets lies in the silent spiking FFN branches, which open with longer training as their gate neurons' firing rate rises from 0.030 at 2B tokens to 0.158 at 10BFig. 10(c); A.10QeA9

Each row marked with a reviewer gained evidence from that reviewer's question, for which we thank them. We hope these results will help the reviewers and the area chair in the final assessment of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.