acceptodds
Under review as a conference paper at ICLR 2027

Spike Regularization and Curriculum Based Direct Training of Spiking Neural Networks for Energy-Efficient Inference

Abstract

Spiking neural networks emit binary events that can be exploited for higher energy efficiency on neuromorphic hardware. A spiking neuron that does not fire costs no energy, and hence, spike count is an important objective to optimize. We propose SRSC (Spike Regularization with Spike Curriculum) that applies a time-weighted penalty on every layer's spike tensor, held negligible until the validation accuracy crosses a threshold, and then ramped geometrically to a ceiling. This does not incur additional parameters or architectural changes, and only requires the sparsity ceiling to be tuned to obtain a desirable accuracy-sparsity trade-off. %We also study a per-layer variant of the penalty, allocated in proportion to each layer's gradient sensitivity, against the uniform coefficient used above. We validate the proposed method on VGG, ResNet, and transformer-based architectures across CIFAR-10, CIFAR-100, DVS-CIFAR10, and ImageNet datasets. SRSC matches state-of-the-art accuracy ( for ResNet-19 on CIFAR-10, for VGGSNN on DVS-CIFAR10), while reducing the firing rate by 49% and 44%, and synaptic operations (SOPs) by 24% and 16%, respectively, relative to state-of-the-art methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Spiking Neural Networks, Neuromorphic computing, energy efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_M3owOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SRSC, a training framework for improving the energy efficiency of spiking neural networks (SNNs) through spike regularization and curriculum-based training. The method introduces a time-weighted spike penalty whose strength gradually increases after the validation accuracy reaches a predefined threshold. Experiments on several image classification benchmarks demonstrate reduced firing rates and synaptic operations with limited accuracy degradation. The authors also evaluate the potential energy savings using a hardware accelerator model.

Critical strengths
  1. The experimental results are promising. The paper demonstrate reductions of firing rates and SOPs, while maintaining model performance across various classification datasets.

  2. The paper goes beyond software evaluation and energy estimate. The authors evaluate the energy efficiency of the proposed method on a hardware accelerator.

Critical weaknesses
  1. The major limitation of this paper is novelty and substantial overlap with existing literature. The core idea of reducing neuronal activity in SNN through explicit spike regularization has been extensively studied in previous works, including Allred et al. (2020), Pellegrini et al. (2021), Deng et al. (2021), Yan et al. (2022), and Suetake et al. (2023) [1-5]. For example, Allred et al. (2020)[1] already explored progressively increasing the spike regularization strength during training, which is similar to the core algorithmic contribution of this paper. The authors should provide specific clarification.

[1] Allred et al. (2020). Explicitly Trained Spiking Sparsity in Spiking Neural Networks with Backpropagation. [2] Pellegrini et al. (2021). Low-activity Supervised Convolutional Spiking Neural Networks Applied to Speech Commands Recognition. IEEE SLT. [3] Deng et al. (2021). Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization. IEEE TNNLS. [4] Yan et al. (2022). Backpropagation With Sparsity Regularization for Spiking Neural Network Learning. Frontiers in Neuroscience. [5] Suetake et al. (2023). Synaptic Interaction Penalty: Appropriate Penalty Term for Energy-Efficient Spiking Neural Networks. TMLR.

  1. Another weakness of this paper is that the motivation for penalizing spikes at later timesteps has not been explained. Since the computational cost of a spike does not inherently depend on when it occurs, the proposed temporal weighting appears heuristic rather than directly motivated by energy efficiency.

  2. The results show subtle improvements compare to exisiting sparse SNN training methods. For example, SRSC even exhibits higher estimated energy consumption and lower accuracy than UniSparse in DVS-CIFAR10. Overall, the result table shows that the paper seems to trade more accuracy for less energy and fail to demonstrate an improved accuracy–energy Pareto frontier.

Questions for authors

See in weaknesses

Reviewer_7UiaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SRSC, a spike regularization method, to reduce neural activity in SNNs. The regularization method adds an auxiliary L1 magnitude loss for each layer and penalizes more for later time-steps. To balance accuracy and firing rate suppression, an accuracy gate is proposed that enables regularization when training accuracy exceeds the threshold. Across multiple tasks, SRSC matches state-of-the-art accuracy (±0.5% for ResNet-19 on CIFAR-10, ±0.75% for VGGSNN on DVS-CIFAR10), while reducing the firing rate by 49% and 44%, and synaptic operations (SOPs) by 24% and 16%, respectively, relative to state-of-the-art methods

Critical strengths

1) The experiment is complete. The authors evaluate SRSC across static benchmarks, including ImageNet, CIFAR10, and CIFAR100, and event-based benchmarks, including CIFAR10-DVS. Furthermore, the authors provide energy estimation on LoaS, further enhancing the convincingness of the results.

Critical weaknesses

1) The method is naive and straightforward. Using L1 magnitude regularization to suppress neuron activity is a simple solution to prune neural connections or reduce spikes. The authors claim that the dynamic scheduling (accuracy gate, if I understand correctly) during training is the main difference between SRSC and other SNN compression methods. However, I think the novelty of the scheduling is not enough, since the scheduling is widely used in previous ANN pruning works, such as [1][2].

2) The comparison works are incomplete. The publication year of most of the works, except UniSparse, compared with SRSC is before 2024. Other neural activity pruning methods such as [3] should be compared.

[1] Fischer F, Birk A, Somers P, et al. FeaSel-Net: A recursive feature selection callback in neural networks[J]. Machine Learning and Knowledge Extraction, 2022, 4(4): 968-993.

[2] Wauthier S T, De Boom C, Çatal O, et al. Model Reduction Through Progressive Latent Space Pruning in Deep Active Inference[J]. Frontiers in Neurorobotics, 2022, 16: 795846.

[3] Bu T, Shi X, Yu Z. Activity pruning for efficient spiking neural networks[J]. Advances in Neural Information Processing Systems, 2026, 38: 171134-171153.

Questions for authors

A lot of hyperparameters are set in Sections 4.3 and 4.4. Such as NrampN_{\rm ramp}, α\alpha, β\beta, kk, which are set manually before training. How are these parameters set? The setting of these hyperparameters weakens the generality of this method.

1 comment

Thanks for the reviewer's reply. I remain concerned that the novelty of penalty scheduling (the penalty's starting point) is insufficient. Therefore, I maintain my score.

Reviewer_msosOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SRSC, which introduces a temporally weighted spike regularization term into direct SNN training, imposing larger penalties on spikes emitted later. The regularization strength is kept very small during early training. Once the validation accuracy reaches a threshold or training reaches a specified epoch, it is gradually increased geometrically to a preset upper bound to balance classification accuracy and spike sparsity.

Critical strengths

1.The motivation and proposed method is very simple. 2.The experiments cover multiple datasets and architectures.

Critical weaknesses

1.Using the firing rate as a loss term is not novel. Therefore, the core contribution of this paper is limited to the temporal weighting and training schedule of the regularization strength. 2.Limited performance improvements. Although the firing rate is reduced, accuracy is also slightly lower than that of the baseline methods on most tasks. In some cases, SRSC even achieves lower accuracy and energy efficiency than the pruned model baseline (UniSparse). 3.The authors aim for the network to form effective predictions as early as possible and therefore impose larger penalties on later spikes. However, later spikes may also carry important information, and the paper lacks a theoretical analysis justifying this design. Corresponding ablation experiments are also missing. 4.Table 17 in the appendix states that accuracy is taken from the peak validation accuracy at an earlier epoch, whereas NAS, SOPs, firing rate, and weight sparsity are taken from the final epoch. I do not understand the purpose of this comparison. If the aim is to highlight the firing-rate advantage at the final epoch, the model’s accuracy at that epoch should be compared directly, rather than its best historical accuracy. 5.Eq. (6) claims that the classification loss encourages all neurons to increase their firing, which is clearly incorrect; otherwise, all models would reach a firing rate of 100%. 6.The experiments mainly use a single random seed and lack repeated runs. 7.The different activation schemes include an “accuracy-triggered” approach. However, the model’s final accuracy is not known in advance, necessitating additional baseline training costs.

Questions for authors

1.Would post-training an already trained model with a spike penalty yield better performance than the existing model?

Comments on the paper

1 comment
Area_Chair_fZTdOpenReview ↗

Please Engage in the Author–Reviewer Discussion

Dear Reviewers,

As the author–reviewer discussion period is approaching its deadline (November 18, AoE), I would like to kindly remind you to review the authors' rebuttals and actively engage in the discussion as soon as possible.

Please address any outstanding questions or concerns and update your assessments and ratings if appropriate.

Thank you very much for your time and efforts!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.