- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes UNISPIKE, a hybrid SNN training framework aiming to improving accuracy while preserving two important properties: addition-only computation and step-by-step inference. The framework first introduces Unified Clip as an ANN activation function to replace ANN operators that are not compatible with SNNs (e.g., softmax, LayerNorm, and GeLU). A SNN-friendly Transformer architecture called Uniformer is then built. The full training pipeline involves ANN pretraining, ANN quantization-aware training, ANN2SNN conversion, and BPTT-based SNN finetuning. Experiments on image classification tasks demonstrate competitive accuracy among SNNs with addition-only and step-by-step properties and low theoretical energy consumption during inference.
Critical strengths
- The motivation is sound. The importance of addition-only computation and step-by-step inference in SNNs are well justified.
- UNISPIKE presents a comprehensive training pipeline integrating pretraining, quantization, conversion and finetuning. It achieves good performance on image classification benchmarks.
- Training cost and early-exit strategies are carefully evaluated. The energy analysis is relatively comprehensive as it considers memory.
Critical weaknesses
- The equivalence between Unified Clip and ST-BIF in Equations (5) - (8) is not sufficiently justified, as the temporal unevenness error [1] is not considered here. Consider input and . Here, , but (assuming firing threshold and quantization step are both 1 ).
- Insufficient ablation. Figure 6 provides a partial comparison of neuron models, and Appendix A3 analyzes training costs. However, the individual contributions of the hybrid training pipeline and the UniFormer architecture to classification accuracies and energy consumption are not clearly disentangled.
[1] Bu et. al., Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks, ICLR 2022.
Questions for authors
- (Weakness 1) Please clarify the required conditions under which the equivalence between Unified Clip and ST-BIF holds.
- (Weakness 2) What's the individual contribution of the UniFormer architecture and the hybrid training pipeline on classification accuracy, training cost and theoretical energy consumption? It's better to compare UniFormer trained directly with Uniformer trained through the hybrid training pipeline.
- The memory cost of QKFormer seems to be exaggerated. Appendix A4 assumes FP32 numerical activations for non-binary values and no operand-cache reuse. Under these assumptions, QKFormer's 48.09 GMACs alone imply approximately 240 mJ of activation reads, accounting for nearly all of the reported 247.7 mJ activation memory energy. However, practical implementations may exploit activation reuse and lower-precision representations for the integers introduced by SEW connections. Please clarify these assumptions and evaluate their impact on the energy comparison.
- Minor: in Equation (2) is not defined and not used elsewhere. Its relationship with and the quantization scale should be clarified.
- Minor: Page 4, Line 200, "for similarity" should be "for simplicity". should be "input current". A careful language revision would improve clarity.
6 comments
Response to Reviewer qtXE (Part 2/4)
[Question 2] What's the individual contribution of UniFormer architecture and the hybrid training pipeline on classification accuracy, training cost, and theoretical energy consumption?
We provide controlled ablation as reviewer acquired.
1) Ablation for UniFormer:
As shown in Table R2, progressively replacing the original ANN operators results in an overall accuracy drop of only 0.5% (82.8% to 82.3%). Specifically, replacing GeLU with ReLU reduces accuracy by 0.4% (82.8% to 82.4%), while further replacing LayerNorm with ReLU causes an additional 0.5% drop (82.4% to 81.9%). Interestingly, replacing Softmax with Unified Clip recovers 0.4% accuracy (81.9% to 82.3%), suggesting that Unified Clip is better suited to the unified activation design in the attention block.
2) Ablation of Hybrid Training:
Accuracy Analysis: As shown in Table R3, hybrid training achieves the highest accuracy of 80.8%, outperforming direct training (54.3%) and conversion alone (66.5%) by 26.5% and 14.3%, respectively. Direct training suffers from poor random initialization and inaccurate surrogate-gradient estimation during BPTT, hindering effective optimization. ANN-to-SNN conversion provides a better initialization, while subsequent BPTT fine-tuning further improves accuracy, demonstrating the effectiveness of the proposed hybrid training pipeline.
Energy and Training Cost: Higher accuracy requires more neuronal firing, resulting in the highest SOPs and inference energy in hybrid training. Direct training takes 327.8 hours, as 4 time-step SNN training incurs approximately 4× the computation per epoch of ANN training. In contrast, hybrid training requires only 10 epochs of SNN fine-tuning, reducing the total training time to 65.00 hours (80.2% reduction).
Table R2: The Ablation Study for operator replacement in UniFormer architecture on ANN pre-training stage. All the models are trained from scratch for 100 epochs under the same training configuration, which differs from the 300-epoch setting used in the main paper.
| Accuracy | |
|---|---|
| Naive | 82.8% |
| +Replace GeLU | 82.4% |
| +Repalce Layernorm | 81.9% |
| +Replace Softmax (UniFormer) | 82.3% |
Table R3: The Ablation Study for the hybrid training pipeline. Training Time is estimated on 8 RTX 4090 GPUs. We use torch.compile to further accelerate the training in the direct training method.
| Model | Time-Step | SOP(G) | Energy(mJ) | Epoch of Stage1/Stage2/Stage3 | Training Time (h) | Accuracy | |
|---|---|---|---|---|---|---|---|
| Direct Training | UniFormer | 4 | 16.011 | 121.9 | 300/0/0 | 327.8 | 54.3% |
| Conversion | UniFormer | 4 | 18.377 | 136.0 | 300/40/0 | 51.34 | 66.5% |
| Hybrid Training | UniFormer | 4 | 19.009 | 138.2 | 300/40/10 | 65.00 | 80.8% |
Response to Reviewer qtXE (Part 3/4)
[Question 3] The memory cost of QKFormer seems to be exaggerated.
We thank the reviewer for the careful examination. We profile QKFormer's SEW-shortcut activations and find that their values range from 0 to 16, requiring only 5-bit unsigned integers (UInt5). With this reduced precision, the activation memory energy decreases from 247.7 mJ to 50.36 mJ, reducing total energy from 726.3 mJ to 528.9 mJ (Table R4). Nevertheless, QKFormer still consumes substantially more energy than UniFormer (138.2 mJ), with weight memory and computation accounting for 90.3% of its total energy.
Table R4: Energy breakdown of SDT-V2, QKFormer, QKFormer with 5-bit SEW-shortcut activations, and UniFormer.
| Energy(mJ) | SDT-V2 | QKFormer | QKFormer with INT5 SEW-ShortCut Activation | UniFormer |
|---|---|---|---|---|
| Weight Memory | 282.2 (75.9%) | 254.1 (35.0%) | 254.1 (48.0%) | 90.90 (65.8%) |
| Activation Memory | 37.99 (10.2%) | 247.7 (34.1%) | 50.36 (9.52%) | 26.99 (19.5%) |
| Neuron State Memory | 1.030 (0.28%) | 0.971 (0.13%) | 0.971 (0.18%) | 3.194 (2.31%) |
| Computation | 50.68 (13.6%) | 223.5 (30.8%) | 223.5 (42.3%) | 17.11 (12.4%) |
| Total | 371.9 (100%) | 726.3 (100%) | 528.9 (100%) | 138.2 (100%) |
Regarding activation reuse, the same optimization can be applied to all SNN architectures; thus, it would benefit all compared architectures similarly. Moreover, after INT5 optimization, activation memory accounts for only 9.52% of QKFormer's total energy, while weight memory and computation remain the dominant costs. Therefore, excluding activation reuse has limited impact on our energy comparison.
Response to Reviewer qtXE (Part 1/4)
[Question 1] Please clarify the required conditions under which the equivalence between Unified Clip and ST-BIF holds.
We thank the reviewer for the careful analysis. The output of quantized Unified Clip is equivalent to the accumulated output of its converted ST-BIF neuron once the SNN reaches equilibrium, i.e., when no further input spikes are received and no additional output spikes are generated. As shown in Table R1, the L1 error falls below 1e-5 when the inference time-step exceeds 42. At this point, the SNN produces virtually no further output, with the L1 norm of its per-step output being zero.
Table R1: Conversion results of UniSpike before BPTT training. L1 Error denotes the average L1 distance between the SNN output at each time-step and the corresponding QANN output for the same input, averaged over 1,000 randomly selected ImageNet-1K test images.