- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work integrates a multi-threshold spiking model with a rotation calibration strategy to propose an ANN-SNN conversion framework tailored for Large Language Models (LLMs). The proposed approach is empirically validated on benchmarks related to language modeling and reasoning.
Critical strengths
- Building upon the conventional multi-threshold spiking model, the authors have implemented relevant optimizations regarding parallelization and surrogate gradient consistency computation.
Critical weaknesses
-
Lack of substantial academic contribution to the SNN domain: Although the paper superficially appears to focus on ANN-SNN conversion learning, the proposed EPMTN model is essentially equivalent to applying bit-width compression directly to the activation layer. In essence, the overall methodology reduces to a straightforward application of established post-training quantization (PTQ) algorithms—typically developed for LLMs—solely to the activation layers, lacking genuine novelty in SNN research.
-
Failure to address core architectural challenges: Regarding the deployment of SNNs on Transformer architectures, this work fails to propose a genuine solution to effectively circumvent the floating-point multiplications inherent in operations such as LayerNorm and Softmax. For instance, it does not explore replacing these with BatchNorm and spiking neurons, or other SNN-friendly computational modules, as demonstrated in prior works like Spiking Vision Transformers. Consequently, the converted spiking LLMs exhibit no substantial architectural or computational differences from standard low-bit quantized LLMs.
Questions for authors
Baseline Comparisons: Regarding the experimental evaluation, the primary baselines for comparison should be quantized LLMs rather than full-precision ANNs. The authors must further substantiate whether the spiking LLMs derived from conversion learning offer any genuine advantages over standard quantized LLMs in terms of both performance and power consumption (energy efficiency).
4 comments
W2
Weakness 2
We thank the reviewer for raising the issue of floating-point operations in Transformer nonlinear modules. Our method focuses on conversion accuracy with few timesteps through a framework based on learnable rotations and MTNs. This framework is independent of, and can be combined with, neuron-based approximations of nonlinear modules. First, these nonlinear modules account for a small fraction of the operation count in the evaluated configuration. We perform an operator-level analysis of a complete dense forward pass of Llama2-7B with batch size 1 and sequence length 2048. The analysis covers weight-activation products, attention matrix multiplications, nonlinear function evaluation, and operations associated with rotations, encoding, gating, and residual connections. RMSNorm, Softmax, and SiLU together account for 0.0730% of the operations and special-function calls in the original EPMTN implementation, compared with 0.0759% in ANN. Most computation remains concentrated in linear projections and attention matrix multiplications, which are therefore the primary focus of our framework.
Supplementary Table W2-a: operations and special-function calls (millions per token)
| Operation category | ANN | Original EPMTN |
|---|---|---|
| Linear projections and LM head | 13214.1548 | 13214.1548 |
| Attention QK and AV | 1073.7418 | 1073.7418 |
| RMSNorm | 1.0651 | 0.7988 |
| Softmax | 8.3876 | 8.3876 |
| SiLU | 1.4090 | 1.4090 |
| Rotations, encoding, and other operations | 5.6095 | 209.4881 |
| Total | 14304.3677 | 14507.9802 |
| Combined share of the three nonlinear modules | 0.0759% | 0.0730% |
Second, retaining floating-point function evaluation does not imply that our accuracy gains require exact evaluation of these functions. To test this directly, we fix T=4, the original rotations learned through CE calibration, FP16 weights, and all Q/A/SiLU encoding locations. We then use populations of threshold neurons to approximate the square and reciprocal square root in RMSNorm, the exponential and reciprocal in Softmax, and the scalar SiLU function. We also evaluate replacing all of these functions simultaneously.
A neuron population approximates a scalar function using fixed thresholds and weighted spike outputs:
Here, K is the number of additional spatial thresholds per scalar function, and H emits a spike when its input strictly exceeds the threshold. Thresholds and output weights are constructed offline using predefined grids, without fitting or selection on the test set. An additional threshold is used in the negative tail of the exponential to preserve zero outputs for causally masked positions.
Supplementary Table W2-b: Neuron-based approximation of nonlinear functions for Llama2-7B at T=4
| Configuration | WT2 PPL ↓ | Pile PPL ↓ |
|---|---|---|
| Original method with nonlinear function evaluation | 6.2086 | 5.6993 |
| Approximate only the square and reciprocal square root in RMSNorm | 6.2135 | 5.7140 |
| Approximate only the exponential and reciprocal in Softmax | 6.2054 | 5.6931 |
| Approximate only SiLU | 6.2121 | 5.7002 |
| Replace all of the above functions | 6.2814 | 5.8165 |
When all functions are replaced simultaneously, WikiText-2 and Pile PPL are 6.2814 and 5.8165, corresponding to relative increases of 1.17% and 2.06%. These results show that, for the evaluated model and settings, the performance of the original rotation-calibrated method is largely preserved when nonlinear functions are approximated by neurons.
For temporal execution, we draw on the nonlinear state accumulation and output compensation mechanism in DCGS [1]. Its differential graded units still evaluate the original function F. For example, Equations (15) and (16) in Section 4.1.1 of the published version are:
This mechanism specifies how nonlinear modules are incorporated into temporal computation.
[1] Differential Coding for Training-Free ANN-to-SNN Conversion, Huang et al., ICML 2025.
Thank you again for your time and thoughtful feedback. Your comments prompted us to clarify the contribution of EPMTN beyond activation quantization and provide additional evidence addressing your concerns. Specifically, our rebuttal includes direct comparisons with QuaRot in terms of perplexity and estimated energy consumption, as well as experiments evaluating neuron-based approximations of Transformer nonlinear functions.
We would greatly appreciate hearing whether these clarifications and additional results address your concerns, or whether any questions remain that we could help clarify. If you find that our responses resolve some of the issues raised, we would be grateful if you could reconsider your overall assessment in light of the additional evidence.
I acknowledge the authors' response to the reviews. After thoroughly evaluating the rebuttal and the perspectives shared by other reviewers, I will maintain my original recommendation.
W1&Q1
Weakness 1
We thank the reviewer for highlighting the connection between EPMTN and activation quantization. However, EPMTN is not constructed by directly imposing an arbitrary bit-width constraint on activations. It is derived from the adopted MTN dynamics as a parallel formulation that is exactly equivalent to the time-averaged output. This equivalence addresses end-to-end MTN calibration by preserving the original forward response while avoiding temporal unrolling and directly defining a surrogate gradient for the complete response. The controlled experiments in Table 1 distinguish forward equivalence from differences in surrogate gradients. Table 2 shows that this design reduces calibration time and memory consumption while improving final perplexity. We therefore view the contribution as extending beyond discrete activation representation. It establishes an equivalent calibration formulation for multi-timestep spiking dynamics and validates its training efficiency and optimization benefits. We acknowledge and appropriately cite the prior quantization research underlying rotation calibration. We do not regard differences in numerical representation or execution alone as evidence of superiority over quantized LLMs. Performance and energy advantages relative to standard quantized models must be assessed through direct experiments.
Question 1
We thank the reviewer for the suggestion. We add a direct comparison between quantized LLMs and quantized EPMTN using the same base model and evaluation data within each model. We compare the FP16 ANN baseline and W4 configurations in terms of perplexity and estimated energy per token.
Supplementary Table 5: QuaRot and EPMTN on Llama3-8B and Qwen3-8B
On Llama3-8B, our method achieves lower WikiText-2 PPL than QuaRot, while Pile PPL is close but slightly higher. Its estimated energy subtotal is approximately 1.66% higher. On Qwen3-8B, our method achieves lower PPL on both datasets, with an estimated energy subtotal approximately 0.40% lower. Relative to FP16 ANN, our method reduces estimated energy by 84.93% on Llama3-8B and 83.82% on Qwen3-8B, trading an increase in PPL for a substantial reduction in cost. Relative to QuaRot, an ANN quantization method, our method reduces WikiText-2 PPL by 6.45% on Llama3-8B while maintaining nearly identical Pile PPL. On Qwen3-8B, it reduces WikiText-2 and Pile PPL by 10.06% and 4.61%, respectively. These results show that our method retains effective conversion accuracy after weight quantization and improves selected performance metrics at estimated costs comparable to those of an established ANN quantization method.