acceptodds
Under review as a conference paper at ICLR 2027

Async-SNN: Efficient Spike-Driven Large Language Models with Asynchronous Computing Paradigm

Abstract

Large language models (LLMs) are increasingly deployed in edge scenarios where power supply can be strict, intermittent, and time-varying. Spiking neural networks (SNNs), with their event-driven and asynchronous computation paradigm, offer a promising path toward power-adaptive LLM inference. However, existing spiking LLMs largely inherit synchronization-intensive operators from conventional Transformers, such as RMSNorm and Softmax Attention, whose global statistical dependencies fundamentally prevent true neuron-level asynchronous execution. In this work, we propose Async-SNN, a framework for constructing fully spiking LLMs with asynchronous computing capability at the billion-parameter scale. We first build a ready-to-convert Async-LLM by replacing RMSNorm with Hardtanh Dynamic Normalization (HDN) and Softmax Attention with Normalized Linear Attention (NLA), thereby eliminating major sources of synchronization. We then introduce a coarse-to-fine ANN-to-SNN conversion pipeline that combines calibration, supervised fine-tuning based quantization-aware training, and equivalent ST-BIF neuron conversion to preserve task performance. Finally, we propose greedy adaptive firing (GAF), which dynamically reallocates neuron firing budgets across layers and time-steps under instantaneous power constraints. Experiments on 0.6B and 1.7B models show that Async-SNN consistently outperforms existing spiking LLM baselines and maintains strong perplexity and zero-shot accuracy even when only 10% of neurons are allowed to be active per time-step.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Large Language Models, Asynchronous Computing Paradigm

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F9cbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Async-SNN, a framework for LLMs that supports asynchronous execution by replacing normalization and attention operators—which typically rely on global statistics—with components compatible with quantization-aware training and spiking neuron conversion. It employs a greedy adaptive firing strategy to dynamically allocate the neuron-firing budget. Experiments demonstrate that the model maintains performance levels comparable to unconstrained settings even when limiting active neurons to 10% per timestep. Hardware simulations indicate a reduction in energy consumption of approximately 33% for a 1.7B-parameter model compared to a quantized ANN. While this approach offers a viable pathway for low-power LLM inference, its performance under real-world hardware constraints and fluctuating power supply conditions remains to be validated.

Critical strengths

This paper reports hardware simulation results obtained using Eyeriss for ANN models and SANA-FE/Loihi for SNN models.

Critical weaknesses
  1. SpikeZIP-TF [1] already addresses an exact mapping from quantized ANNs to SNNs, and Dynamic Tanh [2] already replaces normalization with element-wise functions. The paper merely combines existing techniques and has limited incremental novelty.

  2. The authors' method employs pre-training on 100 billion tokens and SFT on 10 billion tokens, whereas some competing spiking LLM methods [3] used for comparison rely on training-free approaches. Such a comparison is unfair. Furthermore, the extensive training involved runs counter to the original intent behind ANN-to-SNN conversion.

[1] SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN.

[2] Transformers without Normalization.

[3] SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking.

Questions for authors
  1. The paper provides simulations comparing Eyeriss with SANA-FE/Loihi, which strengthens the efficiency evaluation. However, the two systems employ different hardware architectures, and the current description is insufficient to reproduce the model mapping, weight storage, communication, and scheduling overheads. Could the authors further provide the simulation configurations and clarify the accounting assumptions?

  2. The main text emphasizes that freezing only alters the timing of output generation without changing the cumulative output. However, the specific operation in Algorithm 1 involves setting unselected spikes to zero, without illustrating how the membrane potential and spike-tracer states are updated. Could the authors explain this part in more detail?

Reviewer_qsYeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies asynchronous execution of spike-driven language models under a limited neuron-activity budget. It replaces normalization-dependent components with HDN and NLA, trains the resulting Async-LLM, and converts it into an SNN through calibration and quantization-aware fine-tuning. It also introduces Greedy Adaptive Firing (GAF), which allocates a global firing budget across modules.

Experiments on 0.6B and 1.7B models show that GAF preserves much of the unrestricted SNN’s performance at a 10% activity budget. Hardware simulations also report lower energy consumption and latency. The main unresolved issues are the conditions needed for correct asynchronous state updates, the connection between activity limits and physical power constraints, and the extent to which the gains come from additional training or better budget utilization.

Critical strengths
  1. The paper addresses a concrete obstacle to asynchronous SNN execution. Using spikes in linear layers does not remove the global dependencies introduced by normalization. The authors take this issue seriously and design the architecture, conversion procedure, and firing scheduler together. This gives the work a clear motivation beyond model compression alone.

  2. The fixed-budget results are encouraging. For the 1.7B model, average accuracy changes from 55.95% without a firing constraint to 55.93% with 10% GAF. The corresponding drop for 0.6B is also small. The budget sweeps and hardware simulations provide useful additional evidence. In particular, the reported simulation energy decreases from 11.83 to 9.76 mJ/token when GAF is added to the same 1.7B SNN configuration.

Critical weaknesses
  1. Correctness under delayed firing is not sufficiently established. Algorithms 1–2 suppress unselected outputs, but do not clearly specify when the membrane potential and spike tracer are updated. If these states are updated as though a spike had been transmitted before its output is suppressed, the accumulated information may not be preserved. The paper needs to state the actual update order and explain how pending charge is handled when execution stops.

The equivalence argument also needs clearer conditions. Eventual agreement after the network has finished emitting pending spikes does not establish agreement at the fixed inference horizon used in the experiments. The HDN derivation requires explicit scale and sign constraints, and local activation equivalence does not by itself establish correctness of attention products under asynchronous arrivals. These omissions do not demonstrate that the implementation is wrong, but they leave a central claim insufficiently supported.

  1. The experiments do not yet establish power-adaptive operation or fully isolate the source of the gains. A bound on firing counts is not automatically a bound on physical power: memory access, fan-out, communication, and scheduling have different costs. The energy and latency simulations are useful, but do not show instantaneous power or behavior under time-varying supply constraints.

The attribution of the performance gains also needs stronger controls. The conversion pipeline includes substantial SFT QAT, while comparable training budgets for the main baselines are not fully documented. In the scheduling comparison, 1.7B GAF performs about 1.95 times as many SOPs as SRF. This supports better use of the available budget, but does not establish that shallow-layer priority is better at the same total computation. These distinctions matter to both the power-adaptation claim and the claimed scheduling benefit.

Questions for authors
  1. When a neuron is not selected to fire, how are its membrane potential and spike tracer updated? Is its output delayed or discarded, and how is pending charge handled at the end of the fixed inference horizon? Please provide the state-update equations used in the implementation.

  2. What initialization, scale, rounding, and termination conditions are required for the claimed ANN–SNN equivalence? How are the HDN constraints enforced during QAT? Can output agreement be checked under random delays and pause–resume schedules, both at the reported inference horizon and after pending activity has completed?

  3. How is the firing budget mapped to physical power in the hardware model? Do the reported costs include memory, communication, state updates, and GAF arbitration? Evidence of power traces and constraint violations under a changing budget would directly affect my assessment of the power-adaptation claim.

  4. Are the main conversion baselines matched in initialization, training data, and optimization budget? What benefit remains over QAT-only or an equal-token floating-point fine-tuning control? For GAF, does the advantage remain against a work-conserving scheduler, or when cumulative SOPs or energy are matched?

3 comments
AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear Reviewer qsYe,

As the discussion deadline is approaching, we kindly remind that we have answered your remaining concerns carefully with additional experiments and clarification.

Could you please check our response and consider improving the rating if we successfully addressed you concerns?

Any feedback will be greatly appreciated and we are glad to provide any additional clarifications that you may need to bring a fruitful discussion.

Warm regards, The Authors

Thank you for the clarifications and additional experiments. The rebuttal has largely addressed my concerns about the freezing mechanism, and I appreciate the contribution of the integrated asynchronous design. I have two remaining questions: 1, R2-Table 2 provides useful evidence of convergence under delays and pauses. Does similar convergence hold under the evaluated 10% GAF budget? A small comparison between the outputs at T=16T=16 and after pending firing completes would help establish the effect of the fixed cutoff. 2, please clarify what the two ST-BIF operations per spike represent and which neuron-update costs, including membrane subtraction, spike-tracer updates, and reset, are included in the hardware energy estimates. How much would any excluded neuron-update or spike-generation costs affect the reported savings? Overall, I view the contribution more positively and would be willing to reconsider my rating in light of these clarifications.

AuthorsOpenReview ↗

Additional Response to Reviewer qsYe

Thank you very much for taking the time to read our response and providing positive feedback. We greatly appreciate your thoughtful follow-up. Below, we address the remaining questions.

[Q1] R2-Table 2 provides useful evidence of convergence under delays and pauses. Does similar convergence hold under the evaluated 10% GAF budget? A small comparison between the outputs at (T=16) and after pending firing completes would help establish the effect of the fixed cutoff.

R2-Table 7. Maximum absolute logit error relative to the converged unrestricted SNN. During pause–resume, firing is suspended at steps 5–8, yielding identical errors at T=4T=4 and T=8T=8.

ModelScheduleT=0T=0T=4T=4T=8T=8T=12T=12T=16T=16T=20T=20
Async-SNN-0.6BRandom delays8.4124.3181.9410.4170.0110.000
Async-SNN-0.6BPause–resume8.4124.5204.5201.4120.0780.000
Async-SNN-0.6BGAF (10%)8.4124.4232.1780.4820.0180.000
Async-SNN-1.7BRandom delays8.3714.3081.8960.3990.0090.000
Async-SNN-1.7BPause–resume8.3714.2564.2561.3920.0970.000
Async-SNN-1.7BGAF (10%)8.3714.5562.2790.4760.0140.000

[Output agreement under delays, pauses, and GAF (10%).] Similar convergence holds under the 10% GAF budget. As tabulated in R2-Table 7, the maximum absolute logit errors relative to the converged unrestricted SNN are 0.018 and 0.014 at T=16T=16 for Async-SNN-0.6B and 1.7B, respectively. Both errors decrease to 0.000 at T=20T=20. These results support output convergence under budget-constrained firing, while indicating that the fixed cutoff at T=16T=16 retains small, nonzero output differences. We will incorporate this comparison into the revised manuscript.

[Q2] Please clarify what the two ST-BIF operations per spike represent and which neuron-update costs, including membrane subtraction, spike-tracer updates, and reset, are included in the hardware energy estimates. How much would any excluded neuron-update or spike-generation costs affect the reported savings?

The two ST-BIF operations per spike represent (1) membrane-potential reset, Vt=Ut−OtV_t=U_t-O_t, where Ut=Vt−1+VtinU_t=V_{t-1}+V_t^{\mathrm{in}} and Ot=VthrstO_t=V_{\mathrm{thr}}s_t; and (2) spike-tracer update, St=St−1+stS_t=S_{t-1}+s_t, where st∈{−1,0,+1}s_t\in\lbrace -1,0,+1\rbrace denotes the signed spike output, with zero indicating no spike emission.

The SANA-FE (with a 32-chip Loihi [4] configuration) simulation [2] includes neuron-update costs, including membrane integration, membrane subtraction, spike-tracer updates, and threshold comparison. As specified in R2-Table 3, the SANA-FE (with a 32-chip Loihi [4] configuration) simulation in our paper excludes output spike-generation energy.

R2-Table 8. Effect of including output spike-generation energy. All energies are in mJ/token. Parenthesized percentages indicate reductions relative to unrestricted Async-SNN-1.7B in the same column.

ConfigurationEexcludeE_{\mathrm{exclude}} (mJ/token)ΔEspike\Delta E_{\mathrm{spike}} (mJ/token)EincludeE_{\mathrm{include}} (mJ/token)
Async-SNN-1.7B11.830.735912.5659
Async-SNN-1.7B + GAF (20%)11.18 (5.49% ↓)0.6687 (9.13% ↓)11.8487 (5.71% ↓)
Async-SNN-1.7B + GAF (15%)10.49 (11.33% ↓)0.4918 (33.17% ↓)10.9818 (12.61% ↓)
Async-SNN-1.7B + GAF (10%)9.76 (17.50% ↓)0.3057 (58.46% ↓)10.0657 (19.90% ↓)

However, SANA-FE provides an explicit interface for accounting for output spike-generation energy through its energy_spike_out parameter. We incorporate this previously excluded cost using 69.3 pJ per emitted spike, as specified in SANA-FE’s Loihi configuration [2]:

ΔEspike=NemitNtokens×69.3×10−9mJ/token.\Delta E_{\mathrm{spike}}=\frac{N_{\mathrm{emit}}}{N_{\mathrm{tokens}}}\times69.3\times10^{-9}\quad\mathrm{mJ/token}.

Einclude=Eexclude+ΔEspike.E_{\mathrm{include}}=E_{\mathrm{exclude}}+\Delta E_{\mathrm{spike}}.

Here, EexcludeE_{\mathrm{exclude}} and EincludeE_{\mathrm{include}} denote energy per token excluding and including output spike-generation energy, respectively. NemitN_{\mathrm{emit}} counts all emitted positive and negative spikes over T=16T=16, and Ntokens=4096N_{\mathrm{tokens}}=4096.

As tabulated in R2-Table 8, including output spike-generation energy yields total energy savings of 5.71%, 12.61%, and 19.90% under the 20%, 15%, and 10% GAF budgets, respectively. At the 10% budget, the saving increases from 17.50% to 19.90%, indicating that GAF’s energy advantage is preserved after accounting for the previously excluded spike-generation cost. We will incorporate these accounting clarifications and updated energy estimates into the revised manuscript.

Reviewer_tiyWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Async-SNN, a framework for enabling fully spiking and asynchronous LLM inference under dynamic power constraints. To eliminate synchronization bottlenecks in conventional Transformers, the authors replace RMSNorm and Softmax Attention with Hardtanh Dynamic Normalization (HDN) and Normalized Linear Attention (NLA), respectively. A coarse-to-fine ANN-to-SNN conversion pipeline is introduced to preserve model performance, while Greedy Adaptive Firing (GAF) dynamically allocates neuron firing budgets according to available power. Experiments on 0.6B and 1.7B models demonstrate that Async-SNN outperforms existing spiking LLM baselines and maintains competitive performance even when only 10% of neurons are active per time-step, highlighting its potential for power-adaptive edge deployment.

Critical strengths
  1. This paper is in good written and easy to follow.

  2. This paper focuses on SNN, a bio-inspired method, which is an interesting topic to save energy.

  3. In the experimental results, this paper improves results in Qwen3-0.6B and Qwen3-1.7B in PPL and zero-shot acc. compared with the baselines.

Critical weaknesses
  1. The proposed coarse-to-fine ANN-to-SNN conversion pipeline could be compared more thoroughly with existing ANN-to-SNN conversion methods. Since each component appears to rely on previously established techniques, the authors could further clarify the novelty of the proposed approach and highlight its key differences from prior work.

  2. The effectiveness of SNN-LLM on modern long-context tasks deserves further investigation. For example, modern LLMs have demonstrated strong reasoning capabilities. However, this work primarily focuses on PPL and zero-shot tasks and lacks further exploration of more challenging long-context / response tasks, particularly those requiring complex reasoning. This can be written in the limitations.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear Reviewer tiyW,

As the discussion deadline is approaching, we kindly remind that we have answered your remaining concerns carefully with additional experiments and clarification.

Could you please check our response and consider improving the rating if we successfully addressed you concerns?

Any feedback will be greatly appreciated and we are glad to provide any additional clarifications that you may need to bring a fruitful discussion.

Warm regards, The Authors

AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear Reviewer tiyW,

As the discussion deadline is approaching, we kindly remind that we have answered your remaining concerns carefully with additional experiments and clarification.

Could you please check our response and consider improving the rating if we successfully addressed you concerns?

Any feedback will be greatly appreciated and we are glad to provide any additional clarifications that you may need to bring a fruitful discussion.

Warm regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.