acceptodds
Under review as a conference paper at ICLR 2027

SpikeHENA: Spiking Hybrid Encoding and Nonlinear Approximation for Training-Free LLM-to-SNN Conversion

Abstract

Spiking neural networks (SNNs) are emerging as a promising alternative to conventional Transformer-based large language models (LLMs) because of their spike-driven computation and sparse communication. However, existing ANN-to-SNN conversion methods face a difficult trade-off among representation fidelity, spike activity, and the required number of time steps, while key nonlinear operations such as Softmax, RMSNorm, and SiLU remain difficult to realize with signed binary spikes. These limitations hinder high-fidelity conversion of pretrained LLMs. In this work, we address the conversion problem from both spike encoding and nonlinear computation perspectives. For spike encoding, we propose a Two-Stage Hybrid Encoding Neuron that maintains a single membrane state across both stages, using population-activity-constrained BinarySpike encoding for large-magnitude activation components and SingleSpike encoding for the remaining residuals. For nonlinear computation, we design dedicated spiking units for square root, exponentiation, and division, and compose them into spike-driven implementations of Softmax, RMSNorm, and SiLU. Based on these designs, we develop SpikeHENA, a training-free conversion framework that directly reuses pretrained ANN parameters. Experiments on BERT and four LLMs show that, at , the converted LLMs incur only 0.12–0.64 percentage-point drops in average zero-shot accuracy and small perplexity increases on WikiText-2 and C4, while their theoretical synaptic-computation energy is estimated at 31.2%–32.5% of the corresponding ANN estimates.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Spiking Transformer, ANN-SNN Conversion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_UWf5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

SpikeHENA is a training-free framework for converting pretrained Transformers into SNNs. It combines activity-constrained BinarySpike encoding with SingleSpike residual encoding and introduces spiking arithmetic units for Softmax, RMSNorm, and SiLU. Experiments on BERT and four LLMs show small accuracy and perplexity changes after conversion. The paper also provides encoding and nonlinear-component ablations, local error analyses, and theoretical energy estimates.

Critical strengths
  1. The paper tackles the two real bottlenecks in LLM-to-SNN conversion jointly, encoding and nonlinear operators, instead of improving only one side. The framework preserves near-ANN performance across several architectures and model scales supported by both task accuracy and perplexity.

  2. The hybrid encoding design is well motivated, and the ablations and local error analyses provide useful support for its fidelity–spike-activity trade-off.

Critical weaknesses
  1. Table 1 provides a useful comparison with LAS and other conversion methods on BERT. The BERT comparisons are useful, but the LLM experiments primarily compare SpikeHENA with its source ANNs. Consequently, extending this comparison to representative LLMs would help assess whether the relative benefits also hold for larger generative models.

  2. The scope of the LLM efficiency evaluation is limited. The reported energy ratios cover only synaptic computation, excluding encoding, nonlinear operations, and memory and communication costs. The authors clearly acknowledge this limitation and provide a broader analysis for BERT. Nevertheless, a similar breakdown for one LLM would help determine whether the estimated savings remain substantial under more complete accounting.

Questions for authors
  1. Could you clarify whether other conversion methods could also be evaluated on LLMs under matched conditions? Such a comparison would help assess whether the relative benefits observed on BERT extend to larger generative models.

  2. Table 2 reports synaptic-only energy estimates, whereas Table 4 provide broader accounting for BERT. Do you have an analysis of how the LLM energy ratios would change after including neuron updates, encoding/control, and nonlinear operations under the same cost model?

  3. How frequently does exponentiation clipping occur in the evaluated models, and how sensitive are the results to the clipping range?

Reviewer_X5YWOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a training-free framework for converting pretrained LLMs into SNNs, built from a two-stage hybrid spike encoder and spike-based implementations of the nonlinear operations. The encoder runs on a single evolving membrane state: an initial phase with exponentially decreasing thresholds and a cap on total population spike activity, then a second phase with linearly spaced thresholds that permits at most one residual spike per neuron. The nonlinear path works at two levels: elementary spiking units compute division, square root and exponentiation, and those units are composed into the layer-level operations the models actually use: Softmax, RMSNorm and SiLU. The conversion reuses the pretrained weights unchanged, with no retraining, fine-tuning or post-hoc optimization. The authors evaluate a converted BERT on GLUE and four converted models up to 72B parameters on six zero-shot tasks and two perplexity sets, and reports task-level differences from the source networks together with theoretical operation-energy estimates.

Critical strengths
  1. The hybrid encoder combines two existing encodings that fail in opposite directions. BinarySpike covers a wide activation range in a short window, but its thresholds shrink exponentially and activity climbs toward the end. SingleSpike keeps activity low and needs a window several times longer for the same range. Table 5 shows the consequence at T = 44: SingleSpike sits far above the ANN on diagnostic perplexity, and BinarySpike reaches similar perplexity only at much higher activity. The hybrid caps population activity in the first stage and spends the second stage on the residual held in the membrane, reaching ANN-level perplexity at about half the BinarySpike activity. The threshold schedules, the activity budget and the stage transition are specified closely enough to reimplement.
  2. The paper covers nonlinear operations that earlier conversion work leaves out, and it reports where the cost moves. Earlier training-free conversions kept RMSNorm and SiLU in floating point, or approximated nonlinearities with multi-valued neurons. This paper builds division, square root and exponentiation from signed spikes, threshold comparison and accumulation, then composes them into Softmax, RMSNorm and SiLU. The component table separates the nonlinear terms. The exponential tree and the division unit each cost more than the floating-point operations they replace, and the nonlinear total still falls because native GELU is the largest term on the ANN side.
Critical weaknesses
  1. The energy headline reports the narrowest slice of the paper's own accounting. The abstract quotes a synaptic-computation ratio of about a third of the ANN. That figure covers multiply-accumulate against accumulate and excludes the rest of the pipeline. Section 3.3 gives a larger component-level estimate for BERT once neuron updates, encoding and the nonlinear units are counted, and the memory-augmented estimate in Appendix A.10, which adds SRAM, DRAM, routing and synchronization, reaches roughly half. For the four LLMs the paper reports only the synaptic ratio, although weight and activation movement dominates at that scale. The figure in the abstract is the smallest of the three, and the ratio grows as more of the pipeline is counted.
  2. The fidelity and timestep tradeoff are not compared at matched budgets. Table 1 places the converted BERT slightly below the ANN at T = 16, with LAS a little further below and the gap between the two methods under a point. FAS stays within half a point at T = 4, using a quarter of the time steps, and this paper does not report its own method at that budget. On the LLM side, Table 2 compares only against each model's own ANN, and no prior conversion method appears at any of the four scales. The design claim is a balance among fidelity, activity and time steps, and the evidence for it rests on one baseline at one operating point.
Questions for authors
  1. Section B.1 says the configuration scans use a balanced subset sampled from the six zero-shot datasets reported in the main paper. Is that subset disjoint from the evaluation items used in Table 2? If the default N, M and activity budget were selected on data drawn from the reported test tasks, the accuracy figures are tuned on the evaluation distribution, and a held-out diagnostic split would settle the question.
  2. What does the method achieve at T = 4 or T = 8? FAS reaches a similar GLUE loss at T = 4. A low-step result for this design would show how much of the gain over LAS comes from the encoding and how much from the added time steps.
  3. The square root and exponential units depend on lookup values. How are the shipped approximation tables built, and do they depend on the activation ranges of a particular model, or are they fixed across the five converted checkpoints?
3 comments
AuthorsOpenReview ↗

Low time-step results and method comparisons

W2.

The fidelity and timestep tradeoff are not compared at matched budgets. […] On the LLM side, Table 2 compares only against each model's own ANN, and no prior conversion method appears at any of the four scales.

Q2.

What does the method achieve at T = 4 or T = 8? […] A low-step result for this design would show how much of the gain over LAS comes from the encoding and how much from the added time steps.

A: Among the time steps reported by LAS, SpikeHENA scores higher at T=10T=10 and T=11T=11 by 29.85 and 4.33 percentage points respectively; at T=4T=4 and T=8T=8, our exploratory evaluations yield mean scores of 45.68 and 66.02 respectively, substantially below the source ANN's 85.55. LAS does not report results at T=4T=4 and T=8T=8. SpikeHENA's TT counts both stages of hybrid encoding; the configuration for each exploratory setting was selected on a separate subset of 64 SST-2 training samples.

Table R3: BERT results across seven GLUE tasks at different time steps.

MethodTTQQPMNLI-mSST-2QNLIRTEMRPCSTS-BAvg.
SpikeHENA463.1832.4149.6650.5447.2968.38/81.221.92/1.8245.68
SpikeHENA881.0971.0478.5677.4755.2331.62/0.0082.99/82.9666.02
LAS1068.6432.9649.0849.7347.2968.63/81.2920.58/26.5349.46
SpikeHENA1089.0281.9688.0788.4364.6257.11/55.2487.08/86.6679.31
LAS1186.2971.4179.3686.1447.6576.96/82.0679.56/82.5175.91
SpikeHENA1189.3381.8889.2288.7263.9061.76/62.1486.87/86.4280.24
LAS1390.6984.1192.5590.9166.0684.31/89.1288.78/88.6085.68
SpikeHENA1390.0883.5991.6388.8363.1881.62/85.9888.67/88.4184.24
LAS1690.6383.5892.0990.6563.9083.82/88.7088.64/88.4385.09
SpikeHENA1690.5283.8892.7890.1065.3484.07/88.0788.68/88.4885.32

We recomputed the LAS [1] mean scores at T=10T=10, 1111 and 1313, first averaging the two metrics within each of MRPC and STS-B, then taking the equally weighted mean across all seven tasks in LAS Table 6.

At T=13T=13 the published LAS score is 1.44 percentage points higher, and at T=16T=16 SpikeHENA scores 0.23 percentage points higher in our evaluation. These comparisons align only the time steps, so they do not establish equal computational budgets, nor do they isolate the contribution of the encoder. To isolate the effect of encoding, we performed an ablation experiment on 24 tensors from the Qwen1.5-MoE-A2.7B SNN: at T=44T=44, under comparable spike activity budgets, compared with BinarySpike-only encoding, hybrid encoding reduces aggregate relative L2L_2 reconstruction error by 78.04%, with 0.78% fewer spikes.

In comparing with FAS, we also need to consider differences in the training pipeline. FAS first replaces the activation functions, performs full-parameter fine-tuning, then transfers the weights, and applies layer-wise, neuron-wise calibration; SpikeHENA directly reuses pretrained ANN parameters, without retraining, fine-tuning or additional post-conversion optimization. Therefore, even when the number of time steps is matched, the comparison still reflects differences arising from fine-tuning and calibration.

On OPT-2.7B, at T=16T=16 SpikeHENA achieves 61.39% mean accuracy across seven zero-shot tasks, compared with 61.55% reported for LAS; SpikeHENA uses N=10N=10, M=6M=6, α=0.9\alpha=0.9.

Table R4: OPT-2.7B zero-shot accuracy (%).

TaskANN†LAS, T=16T=16†SpikeHENA, T=16T=16
PIQA73.7873.6171.98
ARC-Easy60.7361.2459.97
OpenBookQA25.0024.4028.20
WinoGrande61.3360.6261.64
COPA77.0078.0075.00
WSC27378.0278.0279.12
RTE55.2554.9753.79
Avg.61.5961.5561.39

† The ANN and LAS values are from LAS Table 1, and the SpikeHENA values are from our evaluation. We will add the BERT and OPT-2.7B comparisons in Section 3.2, and provide the evaluation settings and encoding ablation in the appendix.

Reference

[1] Long Chen, Xiaotian Song, and Yanan Sun. LAS: Loss-less ANN-SNN Conversion for Fully Spike-Driven Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(3), 1730–1738, 2026.

Q1.

Section B.1 says the configuration scans use a balanced subset sampled from the six zero-shot datasets reported in the main paper. Is that subset disjoint from the evaluation items used in Table 2? If the default N, M and activity budget were selected on data drawn from the reported test tasks, the accuracy figures are tuned on the evaluation distribution, and a held-out diagnostic split would settle the question.

**A: The 48-example diagnostic subset overlaps with the Table 2 evaluation data. We took the first eight examples from the preprocessed validation or development split of each of the six tasks and evaluated them with zero-shot prompts.

Appendix B.1 incorrectly describes these scans as selecting the default configuration. We fixed the default (N=24, M=20, α=0.9) before the main experiments. Then we ran the scans afterward solely to assess sensitivity to the total number of time steps T, the allocation between N and M, and the spike-rate threshold α\alpha. We will revise Appendices B.1 and B.6 to clarify this sequence and describe the diagnostic protocol and its overlap with the Table 2 evaluation data in Section 3.1.

Q3.

The square root and exponential units depend on lookup values. How are the shipped approximation tables built, and do they depend on the activation ranges of a particular model, or are they fixed across the five converted checkpoints?

A: The five converted checkpoints follow the same construction rules: the lookup values are computed directly from the functional form and encoding configuration, without fitting to the activation distribution of any individual checkpoint.

For exponentiation, the input is first clipped to [−10,10][-10,10], the default grid spacing is 2−112^{-11}, and the table stores 40,961 values Lj=exp⁡(−10+j 2−11)L_j=\exp(-10+j\,2^{-11}), j=0,…,40960j=0,\ldots,40960. As defined in Section 2.3, the Stage I spike sequence guides the temporal-tree lookup (Exp-Tree) to obtain the exponential of the coarse component ex0e^{x_0}; Stage II then uses the residual δ\delta for a first-order correction, yielding ex0(1+δ)e^{x_0}(1+\delta). Once the Exp encoding configuration (including the number of time steps) is fixed, the lookup grid, table values and residual correction rule remain unchanged and do not depend on the actual spike activity or activation distribution.

For square root: let aa be the accumulated nonnegative operand, and let t0t_0 be the time step of the first nonzero spike. This spike is positive, so the initial estimate is x0=st0ϑt0=ϑt0x_0=s^{t_0}\sqrt{\vartheta^{t_0}}=\sqrt{\vartheta^{t_0}}, followed by two Newton–Raphson updates xk+1=12(xk+a/xk)x_{k+1}=\frac{1}{2}(x_k+a/x_k), k=0,1k=0,1. For a given threshold schedule, the lookup table only needs one entry per time step, storing ϑt\sqrt{\vartheta^t}. For BinarySpike thresholds of the form ϑt=2m\vartheta^t=2^m, the corresponding entries are 2m/2=(2)m2^{m/2}=(\sqrt{2})^m. The first nonzero spike time t0t_0 selects the initial estimate from the table.

I appreciate your thoroughly rebuttal. It is a great work and I will raise my rating.

Reviewer_1HHWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpikeHENA, a training-free framework for converting pretrained Transformer-based language models into spiking neural networks (SNNs). To address the trade-off between representation fidelity, spike activity, and the number of time steps, the authors present a two-stage hybrid encoding scheme that combines population-activity-constrained BinarySpike encoding with SingleSpike residual encoding while maintaining a shared membrane state. The paper also develops spike-based approximations for division, square root, and exponentiation, which are used to implement Softmax, RMSNorm, and SiLU. Experiments on BERT and four LLMs with up to 72 billion parameters show relatively small performance degradation compared with their ANN counterparts. At 44 time steps, the converted LLMs exhibit average zero-shot accuracy drops of 0.12–0.64 percentage points, while their estimated theoretical synaptic-computation energy is 31.2%–32.5% of that of the corresponding ANNs.

Critical strengths
  1. Compelling empirical results obtained using different architectures and model sizes. The paper evaluates the performance of SpikeHENA using BERT and four large language models (LLMs) with up to 72 billion parameters. The converted LLMs show relatively small decreases in average zero-shot accuracy (0.12–0.64 percentage points) and limited increases in perplexity without retraining or fine-tuning. These results provide valuable empirical evidence that training-free conversion of large-scale LLMs into SNNs is feasible.

  2. A practical approach to balancing representation fidelity and spike activity. The proposed hybrid encoding combines the advantages of BinarySpike and SingleSpike, while the spike-based nonlinear approximation units extend the framework to key Transformer operations, including Softmax, RMSNorm, and SiLU. The ablation results in Table 5 show that the framework can maintain relatively high conversion fidelity while consuming significantly less theoretical synaptic-computation energy than BinarySpike alone. Table 6 further evaluates the contributions of the spike-based nonlinear units. This trade-off is important for the practical development of efficient spiking language models.

Critical weaknesses
  1. Insufficient experimental comparisons with existing methods for converting LLMs to SNNs. Although the paper shows relatively high conversion accuracy across different LLM architectures, the experiments involving four LLM models primarily compare SpikeHENA with the original ANN models. The advantage of SpikeHENA over LAS when using BERT, as shown in Table 1, is relatively small, and no direct comparisons with existing LLM-to-SNN conversion methods are provided for the four LLMs. It would be important to include experiments with the same model architectures, evaluation tasks, and computational resources to demonstrate whether SpikeHENA offers a significant advantage over existing methods.

  2. Insufficient experimental evidence regarding efficiency gains. The presented data on LLM energy savings are largely based on the theoretical energy of synaptic computations, without considering the overall cost of inference. Although the paper presents a more detailed theoretical analysis of BERT's energy consumption, it is unclear whether similar advantages exist for larger LLMs. Further experiments that measure execution latency, memory footprint, and total energy consumption, or that provide a more detailed hardware-aware evaluation of at least one LLM, would help to determine the practical significance of the proposed system.

Questions for authors
  1. Numerical stability of the exponential approximation. The proposed exponential approximation uses ex0(1+δ^)e^{x_0}(1+\hat{\delta}). However, if Stage I terminates with a large residual, the approximation may yield a negative value when δ^<−1\hat{\delta}<-1. Is there a mechanism in the actual implementation to prevent this? Furthermore, can the authors provide experimental results showing the impact of clipping the input to [−10,10][-10,10] on the accuracy of Softmax for different sequence lengths?

  2. Alignment between spike encoding and linear computation. The paper reconstructs activations using time-dependent thresholds, but the formula for synaptic computation does not explicitly include these weights. Can the authors explain how these time-dependent weights are applied in the actual implementation, and how the converted linear layers preserve the computations of the original ANN?

  3. Comparison with existing LLM-to-SNN methods. Current LLM experiments primarily compare SpikeHENA with the original ANN models. Can the authors provide a direct comparison with at least one existing LLM-to-SNN conversion method, using the same model, evaluation tasks, and computational budget? This would help clarify whether the proposed method provides a significant advantage over existing methods.

  4. Overall inference efficiency. The paper provides a more comprehensive theoretical analysis of energy for BERT, but the reported energy efficiency for LLMs primarily considers synaptic computations. Can the authors provide a more comprehensive energy breakdown for at least one LLM, which includes memory access, neuron updates, nonlinear operations, and other inference overhead? For the AWQ model, how is the difference in numerical precision handled in the energy comparison?

  5. Contribution of components to hybrid encoding. Can the authors provide additional ablation experiments to disentangle the effects of two-stage encoding and the population-activity constraint, preferably under the same spike activity budget? Furthermore, the reported Full results in the nonlinear ablation experiments seem to correspond to different values of α\alpha, rather than the default value of α=0.9\alpha=0.9. Can the authors explain the parameter settings used in these experiments?

Reviewer_ZTx7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors propose a Two-Stage Hybrid Encoding Neuron, maintaining a single membrane state across both stages. The proposed method uses population-activity-constrained BinarySpike encoding for large-magnitude activation components and SingleSpike encoding for the remaining residuals. To address the non-linear operations, they introduce spiking units for square root, exponentiation, and division, and compose them into spike-driven implementations of Softmax, RMSNorm, and SiLU. Empirically their proposed ANN-SNN conversion framework achieves comparable performance compare to ANN counterparts while theoretical achieving better energy efficiency.

Critical strengths

The motivation of the paper to develop spiking LLM is highly relevant at this current era of ML. Energy-efficiency is a key consideration when deploying model to edge-based devices, so I believe the paper dives into relevant topics for the community. Furthermore, the two staged encoding mechanism proposed in the paper seems interesting.

Critical weaknesses
  1. Primary weakness of the paper lies in the fact that the efficiency numbers are all theoretical. Practical implementations of such models on neuromorphic chips can actually demonstrate their efficiency.
  2. There is no analysis being done on the overhead of ANN-SNN conversion, latency of inference and how it compares to the baselines.
  3. Comparison to other efficient LLMs such as those with quantization, pruning mechanisms are not done.
Questions for authors
  1. What is the overhead of ANN-SNN conversion? Also, what is the actual wall clock time of inference compared to baselines.
  2. How does the work compare to other similar spiking llm work such as [1]
  3. Why use the proposed spiking llm over using a quantized or pruned variant?

Reference [1] Xing, Xingrun, Boyan Gao, Zheng Liu, David Clifton, Shitao Xiao, Wanpeng Zhang, Li Du, Zheng Zhang, Guoqi Li, and Jiajun Zhang. "Spikellm: Scaling up spiking neural network to large language models via saliency-based spiking." In International Conference on Learning Representations, vol. 2025, pp. 32880-32899. 2025.

1 comment
AuthorsOpenReview ↗

Measurements of BERT Inference Throughput and GPU Board Energy

Thank you for raising questions about hardware measurements and wall-clock inference time in W1, W2, and Q1. We supplement the BERT GPU latency results reported in Table R2 of our previous response with measurements of full-model inference throughput and GPU board energy for SpikeHENA and the FP32 ANN. Both implementations used the same fine-tuned BERT checkpoint and the full SST-2 validation set (872 samples), with the samples partitioned identically into batches of up to 32. We measured both implementations on the same NVIDIA RTX 3090 GPU.

In SpikeHENA’s linear projections, each channel’s spike value at a given time step is +1+1, 00, or −1-1. The multiply-accumulate operation for each connection can therefore be expressed as adding or subtracting the connection’s weight according to the spike sign. Channels that do not fire contribute zero. On the RTX 3090, we perform this accumulation in blocks. For each weight bit plane, we count the channels that fire and have a 1 in that plane. Within each bit plane, subtracting twice the negative-spike count from the total spike count gives the positive-spike count minus the negative-spike count. We weight each resulting count by the positional value of its weight bit and sum across bit planes to obtain the synaptic current for that time step. We then take a weighted sum of the currents across time steps using the encoding’s temporal coefficients and add the bias once at the output. Thus, the per-connection synaptic computation can be implemented using additions and subtractions, while the temporal coefficients act on the accumulated result for each time step.

For each implementation, we report the median across three steady-state measurement windows. In each window, we ran inference on the full validation set multiple times using CUDA Graph replay. We calculated throughput from elapsed wall-clock time and obtained energy per sample by dividing the increase in cumulative GPU board energy reported by NVML by the number of samples processed. For SpikeHENA, the measurements covered the full-model forward pass, including online spike packing and nonlinear operations. We completed tokenization, weight preparation, and graph capture before measurement began.

BERT implementationThroughput (samples/s)GPU board energy (J/sample)SST-2 accuracy
SpikeHENA1,954.50.171192.66% (808/872)
ANN1,750.10.191892.66% (808/872)

For one pass over all 872 validation samples, the measured throughputs imply steady-state wall-clock times of approximately 0.446 s for SpikeHENA and 0.498 s for the FP32 ANN. On this workload, SpikeHENA achieved 11.7% higher throughput and used 10.8% less GPU board energy per sample than the FP32 ANN.

We expect neuromorphic chips with native support for event-triggered accumulation, low-bit-width spike communication, and local membrane state updates to make fuller use of SpikeHENA’s spike sparsity and potentially yield greater energy savings. These measurements quantify the cost of BERT inference on the RTX 3090. The magnitude of any energy savings on neuromorphic platforms remains to be measured on the corresponding hardware.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions to SpikeHENA

Dear Reviewers and Area Chair,

Thank you for taking the time to review our work and for your constructive feedback. We have uploaded a revised manuscript with the changes highlighted in blue.

1. Comparisons with LAS and SpikeLLM. At 16 time steps on OPT-2.7B, SpikeHENA averages 61.39% across seven zero-shot tasks, compared with 61.55% calculated from the published LAS results (Table 7). We also report BERT results on GLUE at shorter time steps (Table R, Appendix B.9) and explain how SpikeHENA and SpikeLLM differ in encoding and calibration (Appendix D.2).

2. Theoretical energy and measured GPU latency. For BERT, using the same workload specifications and energy model, the theoretical totals including memory access and communication are 86.79 mJ/sample for SpikeHENA and 195.79 mJ/sample for the official LAS implementation (Table D, Appendix A.10). On PIQA, under the mapping assumptions in Appendix A.11, the corresponding totals are 47.74% of the ANN baseline for Qwen1.5-MoE-A2.7B and 54.62% for Qwen2.5-72B-Instruct-AWQ (Tables H and I). For the AWQ model, the computation subtotal is 100.76% of the ANN estimate, and most of the estimated savings come from dynamic SRAM accesses under the weight-reuse mapping.

In the BERT GPU measurements, SpikeHENA's median steady-state inference latency is 77.4% lower at batch size 1 and 62.9% lower at batch size 32 than that of the official LAS implementation. The ANN is fastest at both batch sizes (Table U, Appendix E). Direct validation on neuromorphic hardware remains future work. SpikeHENA directly reuses the pretrained ANN parameters, with zero training or optimization overhead for conversion.

3. Default configuration and ablations. We clarify that the default LLM configuration was fixed before the main experiments (§3.1). The subsequent sensitivity scans used 48 diagnostic examples that overlap with the evaluation data in Table 2 (Appendix B.1). Under a common spike-count cap, hybrid encoding yields 78.04% lower reconstruction error with 0.78% fewer spikes than BinarySpike-only encoding on 24 tensors sampled from Qwen1.5-MoE-A2.7B (Table P, Appendix B.6). Table 6 also specifies the settings held fixed within each model across all nonlinear ablations, including the Full configuration.

4. Linear-layer weighting and numerical stability. We explain how the converted linear layer weights each time step's synaptic contribution by its input reconstruction threshold (§2.1). We describe how the square-root and Exp lookup tables are constructed and show that the Exp residual correction factor stays positive under the default configuration (§2.3 and Appendices C.4–C.5). In the paired diagnostic with 128 valid keys, the mean L1L_1 error of Spiking Softmax is higher at C=8C = 8 and similar at C=10C = 10 and C=12C = 12 (Table T, Appendix C.7).

We hope the revised submission meets your expectations and highlights the value of our work.

AuthorsOpenReview ↗

Summary of Reviews and Responses

Dear Area Chair,

We thank the reviewers for their careful assessments and constructive feedback. Our additional method comparisons, GPU measurements, and expanded energy analyses further strengthen the evidence for SpikeHENA’s conversion fidelity and computational efficiency. We summarize the contributions recognized by the reviewers and our main responses below.

Strengths Recognized by Reviewers

  • Reviewers X5YW and UWf5 appreciated our design jointly addressing the two bottlenecks of spike encoding and nonlinear computation, and the hybrid encoder’s trade-off between representation fidelity and spike activity.
  • Reviewers 1HHW and UWf5 recognized the conversion results across architectures and model scales. Without retraining, fine-tuning, or post-conversion optimization, SpikeHENA maintains high conversion fidelity on BERT and four LLMs with up to 72B parameters. At 44 time steps, the four LLMs incur only 0.12–0.64 percentage-point drops in mean zero-shot accuracy.

Main Concerns and Responses

1. Comparisons with existing conversion methods.
On OPT-2.7B, SpikeHENA achieves 62.29% mean accuracy across seven tasks at (T=16,α=16)(T=16, \alpha=16), versus 61.46% for our local ANN. Compared with a LAS port using the publicly released BERT configuration, SpikeHENA achieves slightly higher accuracy on the full COPA validation set (79% versus 78%), with only 43.0% of the port’s estimated total computation energy. Across the sampled batches from seven tasks, this ratio is 43.2%–43.8%. Even when the first-stage activity constraint is nonbinding, SpikeHENA maintains lower theoretical computation costs in these configurations, further supporting its computational efficiency advantage.

2. Measured GPU efficiency.
In measurements on the full SST-2 validation set using an RTX 3090, the original FP32 ANN, an ANN using the same 8-bit truncated weights, and the full SpikeHENA model all achieve 92.66% accuracy. Relative to the ANN using the same truncated weights, SpikeHENA increases throughput by approximately 12.5% and reduces GPU board energy per sample by approximately 4.6%. Thus, the current BERT implementation achieves both higher throughput and lower board energy with the same weights and matched task accuracy.

3. Expanded energy accounting.
We included encoding, neuron updates, nonlinear operations, memory accesses, and communication, and further examined tiled INT4 weight streaming for the 72B AWQ model. With 4× reuse of projection inputs for both methods and each SpikeHENA weight tile reused across all 44 time steps, estimated energy savings remain 5.93%–8.01% with the original event encoding and 6.97%–9.42% with the shared-sign scheme. These theoretical results under the specified mappings show that the estimated energy advantage persists under streaming schedules with tiled weight reads and dequantization.

4. Independent contribution of hybrid encoding.
On 24 diagnostic tensors, hybrid encoding reduces local reconstruction error by 78.04% relative to BinarySpike-only under comparable spike-activity budgets, while using 0.78% fewer spikes. This further supports the synergy between the population-activity constraint and residual compensation in balancing fidelity and spike cost.

Additional Clarifications

We clarified the use of temporal coefficients to weight each time step’s contribution in linear-layer accumulation, the positivity of the Exp correction factor under the default configuration, and lookup-table construction rules that require no fitting to individual models’ activation distributions. The default configuration was fixed before the main experiments; the 48 diagnostic examples overlapping with the evaluation data were used for subsequent sensitivity analysis.

We thank you and the reviewers for your time and effort throughout the review and discussion process, and for your careful assessments and constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.