- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why SNNs converted from pretrained ANNs can retain competitive performance, while directly trained SNNs often struggle. The authors analyze this gap through a release identity and four mechanisms: write, release, readout, and credit. Based on these insights, they propose NeuronSpark, which achieves performance comparable to Transformers across several model scales, with potential energy-efficiency benefits.
Critical strengths
- The paper addresses an interesting and important question in SNN research. The theoretical analysis provides a useful perspective on the challenges of training SNNs directly.
- The experiments are carefully organized, with mechanism-level analyses, ablation studies, and comparisons across multiple model scales. The detailed appendix is also appreciated.
Critical weaknesses
- While the four-link analysis is insightful, its implications for the full architecture could be better justified. NeuronSpark includes additional memory and residual pathways, and it is not entirely clear how these fit into the proposed explanation.
- The contribution of the spiking mechanisms to the final performance remains somewhat unclear. Appendix A.10 reports that many spiking FFN branches remain inactive, particularly at larger scales. This raises questions about how much of the improvement comes from the spiking components versus the additional memory and skip connections.
- The energy-efficiency results are encouraging, but they rely on RTL-based estimates that exclude memory traffic. Some caution may be needed when interpreting these results as practical energy savings.
Questions for authors
- Could the authors further clarify how the four-link framework applies to the complete NeuronSpark architecture, particularly its memory pathways?
- Would it be possible to include a more direct comparison with a non-spiking variant to better isolate the contribution of the spiking mechanisms?
- Could the authors comment on how memory-access costs might affect the estimated energy advantages?
4 comments
The spiking mechanisms contribute on their own
2. "…how much of the improvement comes from the spiking components versus the additional memory and skip connections."
We trained the non-spiking variant the reviewer suggested in two forms, and the spiking mechanisms contribute on their own, since the final loss rises by 0.051 nats when every neuron is removed and by 0.058 nats when every membrane is kept and only its threshold and reset are removed. The setting is the paper's 200M model and protocol with every matrix kept. The first twin removes the neurons with their thresholds and retention gates. The second keeps each membrane and its retention gate as a hidden state and passes the membrane on without threshold or reset, so the loss it shows cannot come from a missing state. With a one-sided release the same removal changes nothing, so the neurons help once their release link is open. The result is below:
| Three seeds, mean ± standard deviation | Full model, bipolar | One-sided release | No-neuron twin | Membrane-only twin | No-neuron − full | Membrane-only − full | No-neuron − one-sided |
|---|---|---|---|---|---|---|---|
| Final loss | 2.883 ± 0.041 | 2.936 ± 0.043 | 2.934 ± 0.038 | 2.941 ± 0.041 | +0.051 ± 0.002 | +0.058 ± 0.006 | −0.002 ± 0.006 |
We also followed the FFN branches that Appendix A.10 reports as inactive through a longer run, and their gate neurons release more as training continues, with a firing rate of 0.16 at 10B tokens against 0.03 at 2B. The setting is a 2.0B-parameter NeuronSpark trained on MiniCPM's open pretraining data (Ultra-FineWeb, Wang et al., 2025; UltraData, Wang et al., 2026), with the firing rate read at every validation point. The result is below (share of neuron–token pairs above threshold):
| Tokens | 2B | 4B | 6B | 8B | 10B |
|---|---|---|---|---|---|
| FFN gate neurons | 0.030 | 0.056 | 0.099 | 0.135 | 0.158 |
We are grateful for this suggestion. The comparison isolates the contribution of the spiking mechanisms more directly than anything in the submission, and the revision adds it to a new subsection at the end of Section 5.
Memory access leaves the energy advantage almost unchanged
3. "Could the authors comment on how memory-access costs might affect the estimated energy advantages?"
They leave the advantage almost unchanged, with NeuronSpark at 0.38–0.72 of the Transformer's energy per token once memory access is implemented in VHDL for all three models, against 0.39–0.73 with operators only. The setting is the paper's procedure extended to memory. Every memory unit is verified bit by bit against a software reference in simulation and synthesized into logic cells, and an access costs the cell count of the storage bank that serves it, as an operation costs the cell count of its unit. The result is below (words moved per token and energy relative to the Transformer; each column serves every access from storage banks of the stated size):
| Scale, context | Model | Weight words (M) | State words (M) | Energy, operators only | 64-word banks | 8,192-word banks |
|---|---|---|---|---|---|---|
| 200M, 1,024 | Transformer | 199.7 | 25.2 | 1 | 1 | 1 |
| 200M, 1,024 | Mamba-2 | 199.5 | 9.2 | 0.949 | 0.935 | 0.926 |
| 200M, 1,024 | NeuronSpark | 144.6 | 17.5 | 0.732 | 0.725 | 0.720 |
| 2.0B, 1,024 | Transformer | 1,984.9 | 102.9 | 1 | 1 | 1 |
| 2.0B, 1,024 | Mamba-2 | 1,980.9 | 83.8 | 1.010 | 0.997 | 0.989 |
| 2.0B, 1,024 | NeuronSpark | 895.9 | 32.1 | 0.455 | 0.448 | 0.445 |
| 2.0B, 8,192 | Transformer | 1,984.9 | 822.2 | 1 | 1 | 1 |
| 2.0B, 8,192 | Mamba-2 | 1,980.9 | 83.8 | 0.751 | 0.740 | 0.733 |
| 2.0B, 8,192 | NeuronSpark | 895.9 | 176.6 | 0.389 | 0.384 | 0.381 |
The advantage persists because NeuronSpark moves fewer words than the Transformer in each kind of storage. It reads only the weights of non-zero releases, and its pool and membrane potentials move fewer words per token than the key-value cache. The advantage does not rest on the FFN branches of Appendix A.10 staying inactive, since counting every one of them as fully active raises the bound by at most 0.023. The result is below (NeuronSpark relative to the Transformer; the last two columns are upper bounds of the energy ratio, as measured and with every FFN branch fully active, and hold whatever a memory access costs, provided it costs both models the same):
| Scale | Operators (1,024 / 8,192 tokens) | Weight words | State and activation words (1,024 / 8,192 tokens) | Largest of the three: upper bound of the energy ratio | Upper bound with every FFN branch fully active |
|---|---|---|---|---|---|
| 200M | 0.732 / 0.676 | 0.724 | 0.691 / 0.615 | 0.73 | 0.75 |
| 500M | 0.603 / 0.455 | 0.621 | 0.303 / 0.233 | 0.62 | 0.64 |
| 1.3B | 0.488 / 0.419 | 0.485 | 0.360 / 0.254 | 0.49 | 0.51 |
| 2.0B | 0.455 / 0.389 | 0.451 | 0.318 / 0.214 | 0.46 | 0.48 |
We thank the reviewer for this question, which led to the memory analysis that the energy section lacked. The revision adds it to Appendix A.9.5.
Indeed, repeating at the other scales, with several seeds, the experiments that ran at one scale would help us show that the results hold at every scale, and we would very much like to provide them. The ablations, the two twins and the other-SNN runs at 200M and the pretraining to 10B tokens at 2.0B, each with several seeds, would take about 3,400 hours on one B300 GPU to repeat, which we cannot finish within the discussion period.
We are here for an in-depth discussion and further feedback. We would be glad to answer and address any issue you raise, and to keep improving the paper together with you.
Dear Reviewer QeA9,
We greatly appreciate your constructive feedback, which has given the paper a pathway-by-pathway account of how the four links apply to the complete NeuronSpark architecture and its memory pathways, two non-spiking variants that isolate the contribution of the spiking mechanisms, evidence that the inactive spiking FFN branches become active with longer training, and an energy analysis that includes memory-access costs.
As the Author–Reviewer Discussion period closes in a few hours, on November 18 AoE, we would be grateful if you could review our latest responses and the revised paper, and update your assessment if your remaining concerns have been adequately addressed.
We hope that the additional evidence and clarifications provide sufficient grounds for a favorable reconsideration.
Sincerely,
The Authors
How the memory and residual pathways fit the four links
We thank the reviewer for reading the appendix carefully. Each question led to an addition to the paper.
1. "NeuronSpark includes additional memory and residual pathways, and it is not entirely clear how these fit into the proposed explanation."
Thank you for raising this; these pathways fit inside the same four links, the memory pathways as the readout link. Every pathway that carries a past token is either a write path or part of the readout link, and the residual stream and the skip connections add no path across tokens of their own, since a skip passes only the current release of its input neurons. The readout link is where the synapse pool reads the residual stream and carries a token to the prediction (Section 4.2), and Proposition 4.6 gives its requirement. Every pathway and the factor it enters:
We thank the reviewer for this question. The paper did not spell this out pathway by pathway, and the revision adds this table to Appendix A.3, which makes the scope of the lemma explicit.