acceptodds
Withdrawn from ICLR 2027

QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) achieve strong performance across many tasks but rely on dense multiply-accumulate (MAC) operations during inference, resulting in high energy cost. Spiking neural networks (SNNs) offer an event-driven alternative in which synaptic integration uses lightweight accumulation. However, spike-driven LLM inference remains difficult because outlier-heavy activations typically require long firing windows or auxiliary non-spiking paths. We propose QuantaSpike, a short-window spike-driven quantization framework for LLMs built around Logarithmic Ternary Integrate-and-Fire (LTIF) neurons. LTIF uses ternary events with power-of-two membrane-response quanta, improving the information represented by each firing step while retaining shift-ACC-compatible computation. QuantaSpike combines this neuron with group-adaptive gain and selective outlier admission: normal values use residual LTIF steps, whereas admitted outliers receive one additional onset spike before entering the same residual dynamics. Across OPT and Llama-2, QuantaSpike achieves state-of-the-art or competitive perplexity and zero-shot accuracy among spike-driven LLM quantization methods. It also transfers to newer dense LLMs, remaining close to the FP16 reference on Llama-3-8B and Qwen3-8B under the same four-step firing window. Analytical linear-energy projections show that QuantaSpike reduces the energy of one linear transformation by about on OPT models and on Llama-2 models relative to SpikeQuant, providing an accurate and energy-efficient spike-driven path for LLM inference.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Large Language Models, Activation Quantization, Energy Efficient

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ki4bOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a method centered on Logarithmic Ternary Integrate-and-Fire (LTIF) neurons, achieving a spiking representation of large-model activations within a maximum of four time steps through ternary spike firing, membrane potential responses based on powers of two, grouped adaptive scaling, and selective encoding of outliers. However, the paper submission violates the submission guidelines: https://iclr.cc/Conferences/2027/AuthorGuidelines

Critical strengths
  1. This paper proposes a short-window spike-driven quantization aiming to preserve both normal values and outliers without long firing windows or auxiliary non-spiking computation paths.
  2. This paper introduces QuantaSpike, a post-training framework centered on the LTIF neuron. LTIF combines ternary spike decisions with power-of-two membrane-response quanta within a four-step firing window.
Critical weaknesses
  1. The paper lacks an "AI Use Statement," which violates the ICLR 2027 submission guidelines (https://iclr.cc/Conferences/2027/AuthorGuidelines).
  2. There is a lack of validation regarding energy consumption and inference efficiency on actual hardware. The paper's claims of energy efficiency advantages are primarily based on theoretical computational cost estimates; actual latency, throughput, and energy consumption were not measured on neuromorphic hardware.
  3. The model incorporates non-linear operators such as Softmax, RMSNorm/LayerNorm, and SiLU. The authors need to clarify which operators implement spike-driven computation versus those that still rely on floating-point operations, as well as the actual spiking ratio of the entire model.
  4. The ablation studies are insufficient; specifically, there is a lack of ablation experiments regarding factors such as grouping size and time steps.
Questions for authors
  1. How is the "group-wise scaling" described in the paper stored online or implemented?
  2. Does the encoding scheme in the paper require identifying every outlier and encoding each one individually?
Reviewer_FGw6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ACT-SNN, a two-stage LLM-to-SNN conversion framework. In the first stage, basis transformations and channel-wise thresholds are jointly calibrated while keeping the pretrained model weights frozen, balancing reconstruction quality against spiking activity. The second stage enables time-step execution through BA-SIF neurons, stimulus–relaxation inference, and differential operators. The results demonstrate favorable trade-offs between accuracy and estimated energy consumption across multiple models.

Critical strengths
  1. The research direction is meaningful. The paper integrates basis transformations, an exterior penalty method, and differential conversion into an LLM conversion framework designed for low spiking activity and time-step execution.
  2. The evaluation is extensive in both scale and model coverage, and the method achieves state-of-the-art accuracy.
  3. The authors use pipelined execution to control latency and prove that its output matches the quantized output of the first stage.
Critical weaknesses
  1. The first-stage quantization largely reuses RTN. The forward transformation xP introduces additional multiplications during inference, but the authors omit this cost from their energy estimates. These multiplications may also complicate deployment on neuromorphic chips.
  2. Both energy consumption and latency are evaluated theoretically, without validation on actual neuromorphic hardware. Although this practice is common in SNN research, it weakens the evidence for the method’s practical feasibility.
Questions for authors
  1. How much training time does the inference calibration stage require?
  2. After external input stops, the network can eventually become quiescent within a finite number of steps, bringing the output of pipelined execution closer to the first-stage quantized output. However, the authors do not prove that the required number of steps is necessarily bounded by the default T_R = 8. Could this introduce additional error?
Reviewer_yKLgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes QuantaSpike, a post-training LLM quantization framework using signed power-of-two spike responses within a four-step firing window. It combines group-wise scaling with selective onset events for outliers and evaluates accuracy alongside analytical linear-energy estimates.

Critical strengths
  1. The paper addresses the relevant challenge of preserving LLM accuracy under a short firing window. Its post-training design avoids gradient-based retraining and operates on pretrained checkpoints, making the approach potentially useful for studying efficient deployment of existing language models.

  2. The evaluation covers OPT, Llama-2, Llama-3, and Qwen3, using both language-modeling perplexity and zero-shot tasks. The reported results show that a four-step firing window can retain competitive accuracy on several tested models, providing evidence of applicability across different model families.

Critical weaknesses
  1. The neuronal novelty of LTIF is insufficiently established relative to the Multi-Threshold Neuron of Huang et al. [1]. Their neuron already uses signed geometrically spaced response levels, at most one active level per timestep, and subtractive reset. With symmetric base thresholds and appropriate normalization, its response can be expressed as a ternary sign multiplied by a power-of-two magnitude, closely resembling LTIF. The manuscript does not discuss this work or establish a substantive distinction between the two coding mechanisms. The authors should clarify what is novel at the neuron level beyond onset gating and group-dependent calibration.

  2. Equations (6) and (16) define input-dependent gains, whereas Section 4.3 and Equation (18) claim that these gains are folded into weight columns offline. Input-dependent gains cannot generally be absorbed into fixed weights for all inputs. The authors should clarify whether the gains are static or dynamic and specify any required runtime scaling and numerical precision. This inconsistency affects both the claimed shift-ACC execution and its energy accounting assumptions.

  3. The reported ablations remove the onset event or magnitude gate, but do not isolate LTIF’s contribution from activation scaling and outlier handling. A controlled comparison should replace LTIF with an existing multi-threshold coding mechanism in the same LLM pipeline, under matched scaling, outlier treatment, and firing budgets. This matters because multiple response levels increase the information represented by each event. Without this control, the four-step results cannot clearly attribute the improvement to a distinct neuron mechanism rather than richer event encoding or the surrounding quantization strategy.

  4. The Uniform baseline on Qwen3-8B is reported with a zero-shot average of 70.01 in Table 3 but 69.01 in Table 7. The five reported task scores in Table 7 average 69.014, supporting the latter value. This inconsistency changes whether QuantaSpike’s average of 69.41 outperforms Uniform and must be resolved to make the comparison reliable.

[1] Zihan Huang et al. Towards High-performance Spiking Transformers from ANN to SNN Conversion. ACM Multimedia, 2024.

Questions for authors

please check the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.