acceptodds
Under review as a conference paper at ICLR 2027

ACT-SNN: Activity-Constrained Two-Stage Conversion for Efficient Spiking Large Language Models

Abstract

Large language models (LLMs) incur substantial computational and energy costs during inference, motivating their conversion into energy-efficient spiking neural networks (SNNs). Although existing LLM-to-SNN methods have achieved high conversion fidelity, this does not necessarily guarantee efficient inference: accurate discrete representations can require high spike activity and thus higher energy consumption, while layer-wise temporal accumulation leads to serial inference latency. In the first stage, our Basis-Augmented Multi-Threshold Neuron (BA-MTN) jointly learns basis transformations and channel-wise thresholds under an explicit activity constraint, decoupling reconstruction optimization from spike activity control while enlarging the feasible space for a favorable reconstruction–activity trade-off. In the second stage, the Basis-Augmented Symmetric Integrate-and-Fire (BA-SIF) neuron provides an equivalent temporal realization with causal step-by-step spike propagation, enabling pipelined cross-layer execution without layer-wise temporal accumulation; relaxation steps and differential operators further compensate for temporal propagation delays and nonlinear operations. Experiments across LLaMA-2, LLaMA-3, and Qwen2.5 models ranging from 7B to 70B parameters demonstrate that ACT-SNN achieves a strong accuracy–energy trade-off under both full-precision and 4-bit weight settings. In particular, under 4-bit weights, ACT-SNN attains energy ratios of –, reducing estimated inference energy by –% over representative W4A4 QANN baselines while maintaining competitive zero-shot accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Spiking Neural Networks, ANN-to-SNN Conversion, Spike Activity, Energy-Efficient Inference, Neuromorphic Computing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qZjpOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ACT-SNN, a two-stage ANN-to-SNN conversion method designed for Large Language Models (LLMs). The first stage parameterizes the basis following FlatQuant and learns channel-wise thresholds separately; it employs an L1-activity-based budget penalty to reduce the estimated spike count while maintaining reconstruction accuracy. The second stage decomposes integer responses into unit positive and negative spikes and utilizes a combination of differential operators and a "stimulus-relaxation" mechanism to enable progressive propagation, thereby laying the groundwork for cross-layer pipelined execution. Experiments covering models ranging from 7B to 70B parameters demonstrate a favorable trade-off between accuracy and estimated arithmetic energy consumption, though the actual energy savings and latency benefits on hardware remain to be verified.

Critical strengths

The experimental results are good.

Critical weaknesses
  1. Lack of innovation.
  • The Stage-I BA-MTN is a FlatQuant-style quantizer [1]. The substantive delta is the activity-budget formulation. The paper should state what is conceptually new beyond (i) reusing FlatQuant's parameterization and (ii) adding an activity regularizer.
  • Early SNN research [2] already sought to reduce energy consumption through activity penalties and target SynOps budgets.
  • Existing research [3] addresses temporal transformation using differential encoding.
  1. The key assumptions underlying the energy consumption conclusions are not sufficiently detailed. The appendix explicitly states that only a portion of the arithmetic costs is estimated, excluding overheads such as memory access, communication, and non-linear operations. More critically, while Equation 104 uniformly assigns INT4 costs to the remaining MAC operations, the actual precision levels for basis transformation, membrane potential, and accumulators are not adequately specified.

[1] FlatQuant: Flatness Matters for LLM Quantization.

[2] Optimizing the Energy Consumption of Spiking Neural Networks for Neuromorphic Applications.

[3] Differential Coding for Training-Free ANN-to-SNN Conversion.

Questions for authors
  1. Pipeline latency analysis relies on sufficient hardware parallel resources, yet there is currently a lack of experiments involving actual hardware deployment or hardware simulations. Do the authors plan to supplement their work with experiments based on hardware deployment or simulations?

  2. The ablation study is incomplete. A more comprehensive validation should involve a four-group comparison: "Standard L1 or Budget-based Penalty" crossed with "With or Without PC-Grad."

8 comments
AuthorsOpenReview ↗

Reply to Reviewer qZjp (Part 1/3)

We thank the reviewer for highlighting the relation to prior quantization and SNN-conversion techniques. We clarify below the substantive contributions of both stages and provide an expanded energy analysis.

W1.1. Stage-I novelty beyond FlatQuant and activity loss

Beyond the activity constraint, Stage I is the tractable surrogate that connects a frozen LLM to step-by-step signed 1-bit inference. Instead of optimizing the full layer-by-time BA-SIF graph, ACT-SNN separates the accumulated event representation to learn from the dynamics that emit it. Stage I learns the completed signed response with single-step BA-MTN; Stage II realizes that response temporally with BA-SIF.

Because the backbone is frozen, the necessary optimization freedom must lie in the neuronal representation. FlatQuant's efficient Kronecker parameterization inspired our tractable basis transform: PP selects the coordinates in which signed event levels are assigned, while θ\theta defines static, positive, per-channel physical membrane thresholds. Unlike token-dependent quantization scales, these thresholds transfer directly to BA-SIF. Algorithm 1 first establishes stable thresholds while adapting the basis, then jointly refines PP and θ\theta under reconstruction and activity constraints.

Unlike conversion methods that fine-tune the source ANN around a prescribed spiking activation, ACT-SNN calibrates only {P,θ}\{P,\theta\} through local reconstruction on unlabeled data. Its calibration graph has no temporal dimension, so memory and backward cost do not scale with the inference horizon. Physical-threshold calibration, basis adaptation, and activity control therefore serve one purpose: learning a low-activity signed representation that BA-SIF can realize without temporal retraining.

W1.2. Difference from prior SynOps penalties

Although both methods penalize activity, they optimize different problems: Reference [2] matches a target SynOps value while fine-tuning network weights, whereas ACT-SNN enforces an upper activity bound while calibrating only the neuronal representation. Specifically, Reference [2] estimates the SynOps of a nonnegative QReLU CNN as

S(W)=∑ℓfoutℓ∑iaiℓ(W),S(W)=\sum_{\ell} f_{\mathrm{out}}^{\ell}\sum_i a_i^{\ell}(W),

and optimizes

min⁡W C(W)+α(S(W)−S0)2.\min_W\ \mathcal C(W)+\alpha\big(S(W)-S_0\big)^2.

Here, SS is a fan-out-weighted estimate of total SynOps and S0S_0 is a target operating point. The penalty contribution is symmetric around S0S_0: its gradient pushes SS downward above the target and upward below it. Reference [2] therefore fine-tunes model weights by trading supervised task loss against deviation from a prescribed SynOps value.

ACT-SNN instead asks a constrained representation-learning question while keeping the LLM weights fixed:

\quad\mathrm{s.t.}\quad \mathcal A(P,\theta)\leq\tau, \qquad \mathcal A=\frac{1}{M}\left\|\frac{xP}{\theta}\right\|_{L_1}.$$ Here, $\mathcal A$ is the average magnitude of the continuous signed-level surrogate, while $\tau$ is an admissible upper bound rather than a target to be matched. The objective selects the lowest-reconstruction-error neuronal representation among those satisfying the activity budget. Any representation with $\mathcal A<\tau$ remains feasible and is not pushed back toward the boundary. Moreover, the optimization variables are the basis and physical thresholds $\{P,\theta\}$ rather than the pretrained model weights. Accordingly, $\mathcal L_{\mathrm{act}}$ is not an unconditional L1 regularizer. It is the quadratic exterior penalty of the constrained problem: $$\mathcal L_{\lambda}(P,\theta) =\mathcal L_{\mathrm{rec}}(P,\theta) +\frac{\lambda}{2}\big[\mathcal A(P,\theta)-\tau\big]_+^2.$$ For $\Phi=\{P,\theta\}$, define the implicit multiplier $\mu_\lambda=\lambda[\mathcal A-\tau]_+$. The update then has the form $$\nabla_\Phi\mathcal L_{\lambda} =\nabla_\Phi\mathcal L_{\mathrm{rec}} +\mu_\lambda\nabla_\Phi\mathcal A, \qquad \mu_\lambda\geq0.$$ This directly corresponds to the stationarity condition of the constrained problem, whose KKT conditions are $$\nabla_\Phi\mathcal L_{\mathrm{rec}}+\mu^* \nabla_\Phi\mathcal A=0, \quad \mu^* \geq0, \quad \mathcal A\leq\tau, \quad \mu^* (\mathcal A-\tau)=0.$$ Constraint violations therefore induce a nonnegative multiplier proportional to their magnitude, while the multiplier is zero in the feasible region. Under standard constraint qualifications, feasible limit points of penalty minimizers as $\lambda\rightarrow\infty$ satisfy the constrained optimum's KKT conditions. Thus, $\tau$ specifies an upper activity budget and $\lambda$ enforces it numerically; this differs both in meaning and optimization behavior from target matching in [2].
AuthorsOpenReview ↗

Reply to Reviewer qZjp (Part 2/3)

W1.3. Difference from prior differential coding

We do not claim the differential identity itself as novel. Within ACT-SNN, Stage II uses it to provide a parameter-preserving realization of the learned activity-constrained representation: BA-SIF directly inherits PP and θ\theta, emits each BA-MTN integer response as signed unit events, and uses stimulus-relaxation execution to propagate residual events under causal cross-layer scheduling. Thus, our claim is the integrated BA-MTN-to-BA-SIF construction, not generic differential coding or relaxation in isolation.

W2. Operator precision and omitted system costs

We now specify every questioned precision and expand the boundary beyond MAC/AC counting. The analytical model covers computation, on-chip SRAM access, and local communication for one Transformer block. Its precision assumptions are:

ComponentPrecision
xPxP inputs and Kronecker factorsFP16
xPxP accumulation / outputFP32 accumulation / FP16 output
BA-SIF dense input, membrane, and thresholdFP16
Signed spikes1 bit
Sparse linear weightsINT4
Sparse linear accumulation / outputINT32 accumulation / FP16 fused epilogue
Differential QK/AV operands1-bit events with INT4 cumulative states
Differential QK/AV accumulated matrix stateINT32
RMSNorm, Softmax, and SiLUFP16 inputs, outputs, and analytical operator estimates
SRAM and local communicationCharged according to the stored/transmitted bit width

All methods use the same 45-nm coefficient set, sequence length 2048, and 16 total SNN steps. The revised per-step activity rates are applied consistently to all spike-gated terms; the QK/AV estimate is based on the differential computation formula for binary operators presented in Appendix C.3, which is implemented as sparse spike-driven ACs according to Eq. (78). The model uses Horowitz 45-nm arithmetic/SRAM costs together with a 45-nm local-movement estimate. It includes neuron-state updates, threshold comparisons, soft resets, weight and state accesses, spike delivery, all online xPxP transforms, two RMSNorms, Softmax, SiLU, and the differential QK/AV operations. The resulting per-block on-chip dynamic-energy estimates are:

ModelConventional SNNACT-SNN
LLaMA-2-7B1.342 J1.085 J
LLaMA-2-70B5.815 J3.929 J

These totals include approximately 0.323 J/1.122 J for xPxP and 0.038 J/0.081 J for repeated nonlinear operators. ACT-SNN retains its advantage because xPxP scales as O(bn3/2)O(bn^{3/2}), whereas lower spike activity reduces the dominant O(bn2)O(bn^2) and O(b2n)O(b^2n) terms.

AuthorsOpenReview ↗

Reply to Reviewer qZjp (Part 3/3)

Q1. Pipeline latency under finite hardware parallelism

We will include ACT-SNN deployment on SpiNNaker2 in future work. SpiNNaker2 directly supports the main requirements of our model: its programmable multicast router and custom neuron kernels support event-driven BA-SIF execution; its matrix accelerator has been demonstrated on Transformer operations; its ARM Cortex-M4F and exp/log accelerator can execute the retained nonlinear operators; and its INT8/INT16 matrix path together with the ARM floating-point path supports heterogeneous-precision execution.

Although the actual inference latency depends on hardware-specific constants, the scheduling advantage of our step-by-step execution is independent of these constants: it eliminates the layer-wise temporal barrier and enables execution overlap under both full and partial spatial parallelism. We formalize this dependency-level latency below.

Let D=5LD=5L be the number of causally ordered spiking stages and S=T+TRS=T+T_R the temporal horizon. Assuming one temporal update per resident stage per cycle, layer-by-layer execution requires

Clayer=DSC_{\mathrm{layer}}=DS

stage-cycles. If all DD stages are resident concurrently, step-by-step execution forms a complete spatial pipeline: after an initial fill of D−1D-1 cycles, one temporal step exits per cycle, giving

Cfull=S+D−1=T+TR+5L−1.C_{\mathrm{full}}=S+D-1=T+T_R+5L-1.

The same principle applies when hardware resources cannot hold the full network. If at most RR consecutive stages are mapped concurrently, the network is partitioned into G=⌈D/R⌉G=\lceil D/R\rceil pipeline segments. Executing each segment as a local step-by-step pipeline gives

=\sum_{g=1}^{G}(S+D_g-1) =GS+D-G, \qquad \sum_gD_g=D.$$ This interpolates between layer-by-layer execution ($R=1$) and full pipelining ($R\geq D$). For $S>1$ and $R>1$, adjacent stages overlap, reducing temporal synchronization windows from $D$ to $G$. # Q2. Ablation of the activity objective and PCGrad We completed the requested \$ 2\times2\$ ablation under identical model, calibration data, initialization, optimization steps, and temporal-inference settings, with $\lambda=\tau=1.0$. Standard L1 uses $\mathcal L_{\mathrm{rec}}+\lambda\mathcal A$, whereas the budget objective uses $\mathcal L_{\mathrm{rec}}+\frac{\lambda}{2}[\mathcal A-\tau]_+^2$. PCGrad is applied to the reconstruction gradient when it conflicts with the activity gradient. | Activity objective | PCGrad | WT2 $\downarrow$ | C4 $\downarrow$ | Avg.5 $\uparrow$ | Activity $\downarrow$ | |---|:---:|---:|---:|---:|---:| | Standard L1 | No | 8.77 | 15.01 | 68.13 | 1.44 | | Standard L1 | Yes | 11725.99 | 8551.39 | 29.77 | **0.95** | | Budget-based penalty | No | 8.19 | 13.90 | 71.26 | 1.53 | | Budget-based penalty | Yes | **8.05** | **13.78** | **71.97** | 1.42 | The theoretical difference follows the constrained formulation in W1.2. Standard L1 has gradient $$\nabla\mathcal L_{\mathrm{rec}}+\lambda\nabla\mathcal A,$$ so its activity multiplier is the fixed constant $\lambda$ and remains active even after activity is already acceptable. A fixed coefficient matches the constrained KKT solution only if it happens to equal the unknown optimal dual multiplier for an active constraint; otherwise it under-enforces the budget or over-regularizes reconstruction. In contrast, the budget loss has the adaptive multiplier $\mu_\lambda=\lambda[\mathcal A-\tau]_+$: it is proportional to constraint violation and becomes exactly zero in the feasible region, matching the KKT complementary-slackness structure. The budget thus reduces activity only when necessary and otherwise leaves reconstruction unbiased by further activity pressure. This distinction also explains the interaction with PCGrad. Because PCGrad modifies only the reconstruction gradient, standard L1 leaves an always-active activity gradient unopposed and can become activity-dominated; the resulting activity of 0.95 is accompanied by complete accuracy collapse. With the budget objective, activity pressure switches off once feasible, so PCGrad resolves conflicts only while the constraint is active. Empirically, budget + PCGrad achieves the best WT2, C4, and Avg.5 with activity 1.42, and Pareto-dominates standard L1 without PCGrad. The ablation therefore supports the complementary roles of the one-sided constraint and gradient surgery.
Reviewer_VmUZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ACT-SNN, a two-stage LLM-to-SNN conversion framework. In the first stage, basis transformations and channel-wise thresholds are jointly calibrated while keeping the pretrained model weights frozen, balancing reconstruction quality against spiking activity. The second stage enables time-step execution through BA-SIF neurons, stimulus–relaxation inference, and differential operators. The results demonstrate favorable trade-offs between accuracy and estimated energy consumption across multiple models.

Critical strengths
  1. The research direction is meaningful. The paper integrates basis transformations, an exterior penalty method, and differential conversion into an LLM conversion framework designed for low spiking activity and time-step execution.
  2. The evaluation is extensive in both scale and model coverage, and the method achieves state-of-the-art accuracy.
  3. The authors use pipelined execution to control latency and prove that its output matches the quantized output of the first stage.
Critical weaknesses
  1. The first-stage quantization largely reuses RTN. The forward transformation xP introduces additional multiplications during inference, but the authors omit this cost from their energy estimates. These multiplications may also complicate deployment on neuromorphic chips.
  2. Both energy consumption and latency are evaluated theoretically, without validation on actual neuromorphic hardware. Although this practice is common in SNN research, it weakens the evidence for the method’s practical feasibility.
Questions for authors
  1. How much training time does the inference calibration stage require?
  2. After external input stops, the network can eventually become quiescent within a finite number of steps, bringing the output of pipelined execution closer to the first-stage quantized output. However, the authors do not prove that the required number of steps is necessarily bounded by the default T_R = 8. Could this introduce additional error?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer VmUZ

We thank the reviewer for the positive assessment and constructive questions. Below we provide an extended energy analysis, calibration costs, and additional finite-relaxation results.

W1. Inference cost of the forward transformation xPxP

We now include the online xPxP cost explicitly. RTN is used only for the reparameterized INT4 weights; the BA-MTN activation representation is learned through PP and the physical thresholds θ\theta.

The expanded model covers computation, on-chip SRAM access, local communication, all online Kronecker transforms, and floating-point nonlinear operators. The fused xPxP transform uses FP16 factors with FP32 accumulation and scales as O(bn3/2)O(bn^{3/2}), compared with O(bn2)O(bn^2) for the dominant projections. At sequence length 2048, xPxP contributes approximately 0.323 J and 1.122 J per block for LLaMA-2-7B and LLaMA-2-70B. The resulting estimates are:

ModelConventional SNNACT-SNN
LLaMA-2-7B1.342 J1.085 J
LLaMA-2-70B5.815 J3.929 J

ACT-SNN therefore reduces the modeled per-block energy by 19.2% and 32.4%, respectively. For deployment, xPxP maps to a local mixed-precision preprocessing kernel before the neuron. Under the fused-kernel dataflow assumed in the model, intermediate matrices remain in registers and the transformed result feeds the neuron directly, avoiding an additional SRAM write/read round trip.

W2. Absence of hardware energy and latency measurements

The present evidence is analytical rather than a hardware measurement. To make it more deployment-relevant, W1 replaces arithmetic-only counting with a common 45-nm on-chip model covering heterogeneous-precision computation, SRAM access, local communication, xPxP, and nonlinear operators. Under this broader boundary, ACT-SNN retains a 19.2%/32.4% modeled advantage over the conventional SNN on the 7B/70B blocks.

For latency, our claim concerns the dependency-limited critical path under pipelined execution: step-by-step propagation changes the schedule from O(5L(T+TR))O(5L(T+T_R)) layer-wise stage-steps to O(T+TR+5L)O(T+T_R+5L) with full spatial pipelining. Although the actual inference latency depends on hardware-specific constants, the scheduling advantage of our step-by-step execution is independent of these constants: it eliminates the layer-wise temporal barrier and enables execution overlap under spatial parallelism.

Q1. Calibration time

Using the unified settings in Table 5 and Sec. 4.1 on one NVIDIA A800-SXM4-80GB GPU, the calibration costs are:

ModelCalibration time
Qwen2.5-7B3.69 h
Qwen2.5-32B8.63 h
LLaMA-2-7B4.02 h
LLaMA-2-13B5.36 h
LLaMA-2-70B21.15 h
LLaMA-3-8B3.87 h

All pretrained LLM weights remain frozen; calibration updates only the basis transformations and channel-wise thresholds using 128 WikiText-2 sequences. This cost is incurred once and does not affect inference latency.

Q2. Error introduced by the fixed relaxation budget TR=8T_R=8

TR=8T_R=8 can leave residual temporal activity; it selects an accuracy--latency operating point rather than guaranteeing complete silence. As new rebuttal evidence, we measured the full termination process on Qwen2.5-7B and WikiText-2 with T=8T=8. At total step 75, activity remained only in the final Transformer layer; from total step 76 onward, all monitored inputs and outputs were zero. Thus, the empirical quiescence point is TR∗=68T_R^*=68.

Increasing relaxation to complete quiescence does not reduce the discrepancy to BA-MTN. The mean MSE across layer inputs and outputs is 0.1944 at TR=8T_R=8 and 0.2069 after termination, and the corresponding 28-layer profiles have a correlation of 0.9983. Since no temporal activity remains in the latter case, the residual is consistent with finite-precision accumulation rather than insufficient relaxation. Thus, TR=8T_R=8 already reaches the practical error floor. Consistently, Fig. 3a is nearly converged by eight relaxation steps, and Tables 1 and 2 use this uniform low-latency setting.

The author's response addressed my concerns; however, validation on real hardware is also crucial. I am willing to maintain my weak acceptance rating.

Reviewer_XXvKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes ACT-SNN, a two-stage framework for LLM-to-SNN conversion. The main ingredients include Basis-Augmented Multi-Threshold Neuron (BA-MTN), which discretizes the original floating-point representation, and Basis-Augmented Symmetric Integrate-and-Fire (BA-SIF), which provides an equivalent temporal realization of the discrete representation. Experiments show that the proposed method yields a favorable accuracy-energy trade-off compared to W4A4 QNNs on LLaMA-2, LLaMA-3, and Qwen2.5 models from 7B to 70B.

Critical strengths
  • This work demonstrates a large-scale SNN conversion pipeline for modern LLMs, which is of practical interest.
  • Explicitly constraining spike activity in conversion is interesting.
Critical weaknesses
  • Non-linear operators are still treated as floating-point operations that need to be called repeatedly; this temporal realization itself is not a spike-driven computation and seems to increase the computational overhead, even compared to the unconverted ANN.
  • The activity loss is essentially an L1 regularization on activation, which does not correspond to minimizing the number of spikes when considering the cancellations between +1 and -1 spikes.
  • The energy consumption calculation assumes 4-bit MAC and AC, but many transformations and/or nonlinear operations may not be 4-bit, e.g., x(t)Px(t)P, softmax, RMSNorm, SiLU, etc.
  • The main idea of BA-MTN of learning an invertible transformation PP is already proposed by FlatQuant, and the differential operator has been introduced by Huang et al.(2024). In general, the technical novelty of the paper is a bit limited.
  • Theoretical results only guarantee infinite-time equivalence between BA-SIF and BA-MTN, which is not very meaningful for low-latency SNN conversion (e.g., 8 stimulus steps and 8 relaxation steps as in the implementation).
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer XXvK (Part 1/2)

We thank the reviewer for the thoughtful comments. Below we clarify the differential computation, activity objective, energy accounting, technical contributions, and finite-step behavior of ACT-SNN.

W1. Repeated floating-point nonlinear operations

RMSNorm, Softmax, and SiLU remain FP16 operations and are evaluated at each temporal step, so their operator count is 16 times that of the ANN at T=TR=8T=T_R=8. We make this accuracy--energy trade-off deliberately: their differential forms preserve the accumulated ANN response while permitting immediate cross-layer propagation, whereas replacing them with spiking approximations would introduce an additional source of conversion error.

The expanded energy model in Q3 counts all repeated nonlinear evaluations. At 8 stimulus and 8 relaxation steps, the two RMSNorms, Softmax, and SiLU together contribute 0.038 J for LLaMA-2-7B and 0.081 J for LLaMA-2-70B, or 3.5% and 2.1% of ACT-SNN's estimated per-block on-chip dynamic energy. The dominant O(bn2)O(bn^2) projections and O(b2n)O(b^2n) attention operations still benefit from sparse spike-driven updates. Thus, the repeated floating-point operators incur a small modeled overhead in exchange for preserving conversion fidelity.

W2. The activity loss does not exactly equal the emitted-spike count

The activity objective is not the exact BA-SIF spike count when opposite-sign events cancel. It is a differentiable surrogate for the magnitude of the BA-MTN level, which lower-bounds the realized event count. For a completed BA-SIF realization, let N+N_+ and N−N_- be the positive and negative spike counts. The corresponding BA-MTN integer level is q=N+−N−q=N_+-N_-, whereas the emitted-event count is K=N++N−K=N_++N_-. Hence,

K≥∣q∣.K\geq |q|.

The activity loss minimizes the continuous pre-discretization counterpart of ∣q∣|q|; after temporal unfolding, ∣q∣|q| is exact when no cancellation occurs and otherwise remains a lower bound on KK. Tables 3 and 4 verify this surrogate against the quantity of interest: activity is measured directly on the unfolded BA-SIF network and includes both signs. Adding LactL_{\mathrm{act}} reduces measured activity from 2.32 to 1.42 (38.8%) in Table 3, and tightening τ\tau from 2.0 to 0.5 reduces it from 1.58 to 1.16 in Table 4.

W3. Precision of transformations and nonlinear operators

We now explicitly model these operators at their stated precisions. The ER in the submission follows the arithmetic-only MAC/AC convention; the expanded analysis estimates on-chip dynamic energy for one Transformer block, including computation, on-chip SRAM access, and local communication.

The extended model uses FP16 inputs, membrane states, thresholds, nonlinear inputs/outputs, and Kronecker factors; FP32 accumulation for the Kronecker transform; 1-bit signed spikes; INT4 linear weights; and INT32 accumulation for sparse linear operations. All methods use the same 45-nm coefficient set. The fused transform xPxP is counted as FP16-by-FP16-to-FP32 MACs, while RMSNorm, Softmax, and SiLU use FP16 analytical estimates.

For a sequence length of 2048, the resulting per-block estimates are:

ModelConventional SNNACT-SNN
LLaMA-2-7B1.342 J1.085 J
LLaMA-2-70B5.815 J3.929 J

These estimates include xPxP and all repeated floating-point nonlinear operations. ACT-SNN reduces the modeled per-block energy by 19.2% on LLaMA-2-7B and 32.4% on LLaMA-2-70B because lower activity in the O(bn2)O(bn^2) and O(b2n)O(b^2n) terms outweighs the O(bn3/2)O(bn^{3/2}) basis-transform overhead.

AuthorsOpenReview ↗

Reply to Reviewer XXvK (Part 2/2)

W4. Relation to FlatQuant and prior differential operators

The FlatQuant-style Kronecker factorization and prior differential formulas are enabling components of ACT-SNN, not our standalone novelty claims. Our contribution is their integration into an activity-constrained representation-to-dynamics conversion for frozen LLMs, with three core elements:

  1. Activity-constrained reconstruction. Stage I does not merely quantize activations. It learns a BA-MTN representation that minimizes reconstruction error subject to an explicit activity budget, while keeping all pretrained LLM weights frozen. Low spike activity is therefore part of the conversion objective rather than a post-hoc property.
  2. Physical membrane-threshold calibration. BA-MTN jointly optimizes the basis PP and static, positive, per-channel thresholds θ\theta. Unlike token-dependent numerical quantization scales, θ\theta is directly deployable as the BA-SIF membrane threshold. Algorithm 1 stabilizes and jointly refines these physical neuron parameters under the reconstruction and activity objectives.
  3. Temporal unfolding. Stage II transfers the learned representation to step-by-step inference without temporal retraining: BA-SIF directly inherits PP and θ\theta and unfolds each BA-MTN integer response into signed unit events. Stimulus-relaxation execution then enables causal cross-layer propagation while converging to the calibrated BA-MTN response.

These components form a single pipeline: Stage I learns what low-activity signed response should be represented, and Stage II specifies how the same response is emitted over time. The core contribution is therefore the activity-constrained representation and its parameter-preserving temporal unfolding, rather than the borrowed parameterization or differential identity in isolation.

W5. Infinite-time theory versus low-latency finite-step inference

TR=8T_R=8 is not a worst-case bound for exact equivalence; it is the selected accuracy--latency operating point. The theorem guarantees convergence without imposing a uniform stopping time, while each finite input has an input-dependent finite termination time under the stated relaxation dynamics.

As new rebuttal evidence, we measured this termination process on Qwen2.5-7B and WikiText-2 with T=8T=8. At total step 75, only the final Transformer layer remained active; from total step 76 onward, every monitored module had zero input and output activity. The network therefore reaches complete quiescence at TR∗=76−8=68T_R^*=76-8=68.

Increasing relaxation to complete quiescence does not reduce the measured discrepancy with BA-MTN: the mean MSE across layer inputs and outputs is 0.1944 at TR=8T_R=8 and 0.2069 after termination, and the two 28-layer MSE profiles are nearly identical (r=0.9983r=0.9983). Because no temporal activity remains in the latter case, this residual is consistent with finite-precision accumulation rather than insufficient relaxation. Thus, TR=8T_R=8 already reaches the practical error floor. Tables 1 and 2 use this TR=8T_R=8 setting, and Table 3 shows that it improves Avg.5 from 59.12 to 71.97 relative to no relaxation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Reply: Expanded System-Level Energy Analysis

Several reviewers asked whether the original MAC/AC-based energy ratio omits the costs of basis transformations, floating-point nonlinearities, memory access, and communication. We therefore provide an expanded analytical estimate for one standard Transformer block. The same model and coefficients are applied to both the conventional SNN and ACT-SNN.

Scope and Assumptions

We estimate on-chip dynamic energy

=E_{\mathrm{compute}}+E_{\mathrm{SRAM}}+E_{\mathrm{communication}}.$$ The estimate includes neuron-state updates, threshold comparisons, soft resets, sparse linear operations, weight/state access, local spike delivery, online Kronecker transformations, differential QK/AV operations, two RMSNorms, Softmax, and SiLU. We use the following common assumptions: - sequence length $b=2048$ and $T=16$ SNN steps ($T=T_R=8$ for ACT-SNN); - FP16 inputs, membrane potentials, thresholds, nonlinear inputs/outputs, and Kronecker factors; - FP32 accumulation for Kronecker transformations; - 1-bit spikes, INT4 linear weights, and INT32 sparse accumulation; - a single measured per-step activity $p_i$ for all spike-gated terms of each method; - joint QK/AV activity approximated by $p_i^2$; - fused RMSNorm, Softmax, SiLU, and Kronecker kernels; Kronecker intermediates remain in registers and are not written back to SRAM. The Horowitz 45-nm coefficients are: | Primitive | Energy | Assumption | |---|---:|---| | FP16 add / compare / reset | 0.40 pJ/op | Compare and reset use the FP16-add proxy | | FP16 $\times$ FP16 $\rightarrow$ FP32 MAC | 2.00 pJ/MAC | 1.10 pJ multiply + 0.90 pJ add | | INT4 + INT4 $\rightarrow$ INT32 AC | 0.10 pJ/AC | INT32-add proxy | | 8-kB SRAM read/write | 0.15625 pJ/bit | 5 pJ per 32-bit access | | Local inter-PE movement | 0.40 pJ/bit | 45-nm local-movement estimate | | FP16 exp / reciprocal / rsqrt | 6.0 / 3.7 / 4.8 pJ/scalar | Analytical nonlinear-operator proxies | # Per-Block Dominant Terms Let $n$ be the hidden size, $d=rn$ the MLP intermediate size, and $H$ the number of attention heads. For one RMSNorm, Softmax, and fused SiLU operation, respectively, define $$\begin{aligned} R(n)&=11.2bn+6.3b,\\\\ S(b,H)&=13.3Hb^2+3.7Hb,\\\\ G(d)&=19.8bd. \end{aligned}$$ The dominant per-module terms are: | Module | Conventional SNN | ACT-SNN | |---|---:|---:| | RMSNorm1 | $TR(n)$ | $TR(n)$ | | QKV | \$ 3.375p_1Tbn^2\$ | \$ 3.375p_2Tbn^2+12Tbn^{3/2}\$ | | QK | $T(2.25p_1+0.65625p_1^2)b^2n$ | $T(2.25p_2+0.65625p_2^2)b^2n$ | | Softmax | $TS(b,H)$ | $TS(b,H)$ | | AV | $T(2.25p_1+0.65625p_1^2)b^2n$ | $T(2.25p_2+0.65625p_2^2)b^2n$ | | O | \$ 1.125p_1Tbn^2\$ | \$ 1.125p_2Tbn^2+4Tbn^{3/2}\$ | | RMSNorm2 | $TR(n)$ | $TR(n)$ | | Up / Gate | \$ 2.25p_1Tbnd\$ | \$ 2.25p_2Tbnd+4Tbn^{3/2}\$ | | SiLU | $TG(d)$ | $TG(d)$ | | Down | \$ 1.125p_1Tbdn\$ | \$ 1.125p_2Tbdn+4Tbd^{3/2}\$ | Collecting the $O(bn^2)$ linear, $O(bn^{3/2})$ Kronecker, and $O(b^2n)$ attention terms gives $$\boxed{ E_{\mathrm{Conv}}^{\mathrm{main}} =1.125p_1Tb(4+3r)n^2 +T(4.5p_1+1.3125p_1^2)b^2n }$$ and $$\boxed{ E_{\mathrm{ACT}}^{\mathrm{main}} =1.125p_2Tb(4+3r)n^2 +4Tb(5+r^{3/2})n^{3/2} +T(4.5p_2+1.3125p_2^2)b^2n. }$$ The second term of $E_{\mathrm{ACT}}^{\mathrm{main}}$ is the full online Kronecker-transform overhead. The repeated nonlinear operators contribute $$E_{\mathrm{nonlinear}} =T[2R(n)+S(b,H)+G(d)]$$ to both methods. # Numerical Results For LLaMA-2-7B, we use $$n=4096,\quad d=11008,\quad H=32,\quad p_1=0.149,\quad p_2=0.083.$$ The dominant matrix terms are $$E_{\mathrm{Conv}}^{\mathrm{main}} =8.697p_1+0.361p_1^2 \approx1.304\ \mathrm{J},$$ $$E_{\mathrm{ACT}}^{\mathrm{main}} =8.697p_2+0.361p_2^2+\underbrace{0.323}_{xP} \approx1.047\ \mathrm{J}.$$ The repeated nonlinear operators contribute approximately 0.038 J, yielding 1.342 J for the conventional SNN and 1.085 J for ACT-SNN. For LLaMA-2-70B, we use $$n=8192,\quad d=28672,\quad H=64,\quad p_1=0.149,\quad p_2=0.071.$$ The corresponding dominant terms are $$E_{\mathrm{Conv}}^{\mathrm{main}} =38.345p_1+0.721p_1^2 \approx5.734\ \mathrm{J},$$ $$E_{\mathrm{ACT}}^{\mathrm{main}} =38.345p_2+0.721p_2^2+\underbrace{1.122}_{xP} \approx3.848\ \mathrm{J}.$$ The repeated nonlinear operators contribute approximately 0.081 J, yielding 5.815 J for the conventional SNN and 3.929 J for ACT-SNN. | Model | Conventional SNN | ACT-SNN | Reduction | |---|---:|---:|---:| | LLaMA-2-7B | 1.342 J | 1.085 J | 19.2% | | LLaMA-2-70B | 5.815 J | 3.929 J | 32.4% | Thus, after explicitly including $xP$, repeated FP16 nonlinearities, on-chip SRAM access, and local communication, ACT-SNN retains a clear energy advantage. The gain increases with model size because the added Kronecker cost scales as $O(bn^{3/2})$, whereas activity reduction affects the dominant $O(bn^2)$ and $O(b^2n)$ terms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.