- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ACT-SNN, a two-stage ANN-to-SNN conversion method designed for Large Language Models (LLMs). The first stage parameterizes the basis following FlatQuant and learns channel-wise thresholds separately; it employs an L1-activity-based budget penalty to reduce the estimated spike count while maintaining reconstruction accuracy. The second stage decomposes integer responses into unit positive and negative spikes and utilizes a combination of differential operators and a "stimulus-relaxation" mechanism to enable progressive propagation, thereby laying the groundwork for cross-layer pipelined execution. Experiments covering models ranging from 7B to 70B parameters demonstrate a favorable trade-off between accuracy and estimated arithmetic energy consumption, though the actual energy savings and latency benefits on hardware remain to be verified.
Critical strengths
The experimental results are good.
Critical weaknesses
- Lack of innovation.
- The Stage-I BA-MTN is a FlatQuant-style quantizer [1]. The substantive delta is the activity-budget formulation. The paper should state what is conceptually new beyond (i) reusing FlatQuant's parameterization and (ii) adding an activity regularizer.
- Early SNN research [2] already sought to reduce energy consumption through activity penalties and target SynOps budgets.
- Existing research [3] addresses temporal transformation using differential encoding.
- The key assumptions underlying the energy consumption conclusions are not sufficiently detailed. The appendix explicitly states that only a portion of the arithmetic costs is estimated, excluding overheads such as memory access, communication, and non-linear operations. More critically, while Equation 104 uniformly assigns INT4 costs to the remaining MAC operations, the actual precision levels for basis transformation, membrane potential, and accumulators are not adequately specified.
[1] FlatQuant: Flatness Matters for LLM Quantization.
[2] Optimizing the Energy Consumption of Spiking Neural Networks for Neuromorphic Applications.
[3] Differential Coding for Training-Free ANN-to-SNN Conversion.
Questions for authors
-
Pipeline latency analysis relies on sufficient hardware parallel resources, yet there is currently a lack of experiments involving actual hardware deployment or hardware simulations. Do the authors plan to supplement their work with experiments based on hardware deployment or simulations?
-
The ablation study is incomplete. A more comprehensive validation should involve a four-group comparison: "Standard L1 or Budget-based Penalty" crossed with "With or Without PC-Grad."
8 comments
Reply to Reviewer qZjp (Part 2/3)
W1.3. Difference from prior differential coding
We do not claim the differential identity itself as novel. Within ACT-SNN, Stage II uses it to provide a parameter-preserving realization of the learned activity-constrained representation: BA-SIF directly inherits and , emits each BA-MTN integer response as signed unit events, and uses stimulus-relaxation execution to propagate residual events under causal cross-layer scheduling. Thus, our claim is the integrated BA-MTN-to-BA-SIF construction, not generic differential coding or relaxation in isolation.
W2. Operator precision and omitted system costs
We now specify every questioned precision and expand the boundary beyond MAC/AC counting. The analytical model covers computation, on-chip SRAM access, and local communication for one Transformer block. Its precision assumptions are:
| Component | Precision |
|---|---|
| inputs and Kronecker factors | FP16 |
| accumulation / output | FP32 accumulation / FP16 output |
| BA-SIF dense input, membrane, and threshold | FP16 |
| Signed spikes | 1 bit |
| Sparse linear weights | INT4 |
| Sparse linear accumulation / output | INT32 accumulation / FP16 fused epilogue |
| Differential QK/AV operands | 1-bit events with INT4 cumulative states |
| Differential QK/AV accumulated matrix state | INT32 |
| RMSNorm, Softmax, and SiLU | FP16 inputs, outputs, and analytical operator estimates |
| SRAM and local communication | Charged according to the stored/transmitted bit width |
All methods use the same 45-nm coefficient set, sequence length 2048, and 16 total SNN steps. The revised per-step activity rates are applied consistently to all spike-gated terms; the QK/AV estimate is based on the differential computation formula for binary operators presented in Appendix C.3, which is implemented as sparse spike-driven ACs according to Eq. (78). The model uses Horowitz 45-nm arithmetic/SRAM costs together with a 45-nm local-movement estimate. It includes neuron-state updates, threshold comparisons, soft resets, weight and state accesses, spike delivery, all online transforms, two RMSNorms, Softmax, SiLU, and the differential QK/AV operations. The resulting per-block on-chip dynamic-energy estimates are:
| Model | Conventional SNN | ACT-SNN |
|---|---|---|
| LLaMA-2-7B | 1.342 J | 1.085 J |
| LLaMA-2-70B | 5.815 J | 3.929 J |
These totals include approximately 0.323 J/1.122 J for and 0.038 J/0.081 J for repeated nonlinear operators. ACT-SNN retains its advantage because scales as , whereas lower spike activity reduces the dominant and terms.
Reply to Reviewer qZjp (Part 3/3)
Q1. Pipeline latency under finite hardware parallelism
We will include ACT-SNN deployment on SpiNNaker2 in future work. SpiNNaker2 directly supports the main requirements of our model: its programmable multicast router and custom neuron kernels support event-driven BA-SIF execution; its matrix accelerator has been demonstrated on Transformer operations; its ARM Cortex-M4F and exp/log accelerator can execute the retained nonlinear operators; and its INT8/INT16 matrix path together with the ARM floating-point path supports heterogeneous-precision execution.
Although the actual inference latency depends on hardware-specific constants, the scheduling advantage of our step-by-step execution is independent of these constants: it eliminates the layer-wise temporal barrier and enables execution overlap under both full and partial spatial parallelism. We formalize this dependency-level latency below.
Let be the number of causally ordered spiking stages and the temporal horizon. Assuming one temporal update per resident stage per cycle, layer-by-layer execution requires
stage-cycles. If all stages are resident concurrently, step-by-step execution forms a complete spatial pipeline: after an initial fill of cycles, one temporal step exits per cycle, giving
The same principle applies when hardware resources cannot hold the full network. If at most consecutive stages are mapped concurrently, the network is partitioned into pipeline segments. Executing each segment as a local step-by-step pipeline gives
=\sum_{g=1}^{G}(S+D_g-1) =GS+D-G, \qquad \sum_gD_g=D.$$ This interpolates between layer-by-layer execution ($R=1$) and full pipelining ($R\geq D$). For $S>1$ and $R>1$, adjacent stages overlap, reducing temporal synchronization windows from $D$ to $G$. # Q2. Ablation of the activity objective and PCGrad We completed the requested \$ 2\times2\$ ablation under identical model, calibration data, initialization, optimization steps, and temporal-inference settings, with $\lambda=\tau=1.0$. Standard L1 uses $\mathcal L_{\mathrm{rec}}+\lambda\mathcal A$, whereas the budget objective uses $\mathcal L_{\mathrm{rec}}+\frac{\lambda}{2}[\mathcal A-\tau]_+^2$. PCGrad is applied to the reconstruction gradient when it conflicts with the activity gradient. | Activity objective | PCGrad | WT2 $\downarrow$ | C4 $\downarrow$ | Avg.5 $\uparrow$ | Activity $\downarrow$ | |---|:---:|---:|---:|---:|---:| | Standard L1 | No | 8.77 | 15.01 | 68.13 | 1.44 | | Standard L1 | Yes | 11725.99 | 8551.39 | 29.77 | **0.95** | | Budget-based penalty | No | 8.19 | 13.90 | 71.26 | 1.53 | | Budget-based penalty | Yes | **8.05** | **13.78** | **71.97** | 1.42 | The theoretical difference follows the constrained formulation in W1.2. Standard L1 has gradient $$\nabla\mathcal L_{\mathrm{rec}}+\lambda\nabla\mathcal A,$$ so its activity multiplier is the fixed constant $\lambda$ and remains active even after activity is already acceptable. A fixed coefficient matches the constrained KKT solution only if it happens to equal the unknown optimal dual multiplier for an active constraint; otherwise it under-enforces the budget or over-regularizes reconstruction. In contrast, the budget loss has the adaptive multiplier $\mu_\lambda=\lambda[\mathcal A-\tau]_+$: it is proportional to constraint violation and becomes exactly zero in the feasible region, matching the KKT complementary-slackness structure. The budget thus reduces activity only when necessary and otherwise leaves reconstruction unbiased by further activity pressure. This distinction also explains the interaction with PCGrad. Because PCGrad modifies only the reconstruction gradient, standard L1 leaves an always-active activity gradient unopposed and can become activity-dominated; the resulting activity of 0.95 is accompanied by complete accuracy collapse. With the budget objective, activity pressure switches off once feasible, so PCGrad resolves conflicts only while the constraint is active. Empirically, budget + PCGrad achieves the best WT2, C4, and Avg.5 with activity 1.42, and Pareto-dominates standard L1 without PCGrad. The ablation therefore supports the complementary roles of the one-sided constraint and gradient surgery.
Reply to Reviewer qZjp (Part 1/3)
We thank the reviewer for highlighting the relation to prior quantization and SNN-conversion techniques. We clarify below the substantive contributions of both stages and provide an expanded energy analysis.
W1.1. Stage-I novelty beyond FlatQuant and activity loss
Beyond the activity constraint, Stage I is the tractable surrogate that connects a frozen LLM to step-by-step signed 1-bit inference. Instead of optimizing the full layer-by-time BA-SIF graph, ACT-SNN separates the accumulated event representation to learn from the dynamics that emit it. Stage I learns the completed signed response with single-step BA-MTN; Stage II realizes that response temporally with BA-SIF.
Because the backbone is frozen, the necessary optimization freedom must lie in the neuronal representation. FlatQuant's efficient Kronecker parameterization inspired our tractable basis transform: P selects the coordinates in which signed event levels are assigned, while θ defines static, positive, per-channel physical membrane thresholds. Unlike token-dependent quantization scales, these thresholds transfer directly to BA-SIF. Algorithm 1 first establishes stable thresholds while adapting the basis, then jointly refines P and θ under reconstruction and activity constraints.
Unlike conversion methods that fine-tune the source ANN around a prescribed spiking activation, ACT-SNN calibrates only {P,θ} through local reconstruction on unlabeled data. Its calibration graph has no temporal dimension, so memory and backward cost do not scale with the inference horizon. Physical-threshold calibration, basis adaptation, and activity control therefore serve one purpose: learning a low-activity signed representation that BA-SIF can realize without temporal retraining.
W1.2. Difference from prior SynOps penalties
Although both methods penalize activity, they optimize different problems: Reference [2] matches a target SynOps value while fine-tuning network weights, whereas ACT-SNN enforces an upper activity bound while calibrating only the neuronal representation. Specifically, Reference [2] estimates the SynOps of a nonnegative QReLU CNN as
S(W)=∑ℓfoutℓ∑iaiℓ(W),
and optimizes
minW C(W)+α(S(W)−S0)2.
Here, S is a fan-out-weighted estimate of total SynOps and S0 is a target operating point. The penalty contribution is symmetric around S0: its gradient pushes S downward above the target and upward below it. Reference [2] therefore fine-tunes model weights by trading supervised task loss against deviation from a prescribed SynOps value.
ACT-SNN instead asks a constrained representation-learning question while keeping the LLM weights fixed:
\quad\mathrm{s.t.}\quad \mathcal A(P,\theta)\leq\tau, \qquad \mathcal A=\frac{1}{M}\left\|\frac{xP}{\theta}\right\|_{L_1}.$$ Here, $\mathcal A$ is the average magnitude of the continuous signed-level surrogate, while $\tau$ is an admissible upper bound rather than a target to be matched. The objective selects the lowest-reconstruction-error neuronal representation among those satisfying the activity budget. Any representation with $\mathcal A<\tau$ remains feasible and is not pushed back toward the boundary. Moreover, the optimization variables are the basis and physical thresholds $\{P,\theta\}$ rather than the pretrained model weights. Accordingly, $\mathcal L_{\mathrm{act}}$ is not an unconditional L1 regularizer. It is the quadratic exterior penalty of the constrained problem: $$\mathcal L_{\lambda}(P,\theta) =\mathcal L_{\mathrm{rec}}(P,\theta) +\frac{\lambda}{2}\big[\mathcal A(P,\theta)-\tau\big]_+^2.$$ For $\Phi=\{P,\theta\}$, define the implicit multiplier $\mu_\lambda=\lambda[\mathcal A-\tau]_+$. The update then has the form $$\nabla_\Phi\mathcal L_{\lambda} =\nabla_\Phi\mathcal L_{\mathrm{rec}} +\mu_\lambda\nabla_\Phi\mathcal A, \qquad \mu_\lambda\geq0.$$ This directly corresponds to the stationarity condition of the constrained problem, whose KKT conditions are $$\nabla_\Phi\mathcal L_{\mathrm{rec}}+\mu^* \nabla_\Phi\mathcal A=0, \quad \mu^* \geq0, \quad \mathcal A\leq\tau, \quad \mu^* (\mathcal A-\tau)=0.$$ Constraint violations therefore induce a nonnegative multiplier proportional to their magnitude, while the multiplier is zero in the feasible region. Under standard constraint qualifications, feasible limit points of penalty minimizers as $\lambda\rightarrow\infty$ satisfy the constrained optimum's KKT conditions. Thus, $\tau$ specifies an upper activity budget and $\lambda$ enforces it numerically; this differs both in meaning and optimization behavior from target matching in [2].