acceptodds
Under review as a conference paper at ICLR 2027

Structural Regularization for Spiking Neural Networks

Abstract

Spiking Neural Networks have gained increasing popularity as a biologically plausible alternative to traditional Artificial Neural Networks (ANNs). SNNs are both cost-efficient and deployment-friendly, as they process input in a spatiotemporal manner using binary spikes. It is widely believed that increasing the number of timesteps enables SNNs to capture more features, thereby improving accuracy. However, we observe that as the number of timesteps grows, the risk of overfitting also increases, leading to diminishing returns in accuracy improvement. To handle this issue, we propose a structural regularization technique for SNNs, called DropTime. Specifically, during training, we randomly remove information from certain timesteps. We also provide a theoretical justification showing that this simple operation effectively mitigates overfitting. Experimental results demonstrate that our method consistently improves SNN accuracy on both widely used static non-spiking and neuromorphic datasets. We offer a straightforward yet effective approach for training high-accuracy SNNs.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Network, Structural Regularization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tgrJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims to use regularization methods to alleviate the overfitting problem of spiking neural networks (SNNs) when using a large number of timesteps. In particular, the contributions of the paper are summarized as follows.

  1. Algorithm. This paper proposes DropTime, which uses the average of randomly masked timestep logits as the outputs during training time. All timesteps are used during test time.

  2. Theory. This paper proves that DropTime is unbiased and introduces an implicit regularization term proportional to the variance of temporal features, supporting the efficacy of DropTime.

  3. Experiments. This paper verifies the superiority of DropTime over many sota methods on several datasets and observes that SNN with DropTime benefits from larger timesteps (Figure 2).

Critical strengths
  1. DropTime is well motivated to alleviate the overfitting problem of SNNs when using a large number of timesteps.

  2. DropTime is a simple method and does not increase the time and space complexity significantly. It can be directly used for most SNNs.

  3. DropTime is verified to be effective over many baselines on many datasets.

Critical weaknesses
  1. Some details of DropTime are not clear. For example, in Eq. (5), the 1-norm of m occurs in the denominator. It is unclear how to deal with the case where the 1-norm of m equals 0.

  2. On line 275, the authors claim that DropTime has a lower variance than Temporal Dropout. However, this claim is not proven. In fact, here seems to be a counterexample: T=2, p=0.9, y1=1, y2=-1.

  3. Some of the explanations of theories are overclaimed. For example, on line 282, the authors claim that the unbiased property can "ensure the accuracy of our method". However, the theories are not relevant to accuracy. In fact, biased estimators can sometimes increase the performance of an algorithm. Furthermore, Temporal Dropout is also unbiased, and this is not the key difference between DropTime and Temporal Dropout.

  4. Some of the theories are not correct. For example, in Eq. (14) of Theorem 3.2, the variance term is a matrix, while the other two loss terms are scalars. This occurs due to an incorrect derivation on line 352, where an incorrect equality Tr(AB) = A Tr(B) is used.

  5. In experiments, the authors do not report the standard deviation of the accuracy.

Questions for authors

See the weakness part.

Reviewer_h2wqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DropTime, a simple yet effective structural regularization method designed specifically for SNNs. The core idea is to randomly mask the information from some timesteps during training, while keeping all temporal features at inference. The paper provides solid theoretical proofs showing that DropTime keeps the expected output consistent between training and inference, and that it introduces an implicit regularization term to relieve temporal overfitting. Extensive experiments on static vision datasets (CIFAR-10, CIFAR-100, ImageNet) and a neuromorphic dataset (CIFAR10-DVS) show that DropTime consistently improves SNN classification accuracy.

Critical strengths
  1. The paper points out a key problem in the SNN field: overfitting caused by increasing timesteps, and proposes DropTime for it. The mechanism is simple, easy to implement, and can be plugged directly into existing SNN training pipelines.

  2. The proposed method delivers stable and significant accuracy gains across all tested datasets and network architectures.

  3. The paper is well-structured, logically coherent, and clearly written.

Critical weaknesses
  1. In the discussion comparing DropTime with Dropout, the authors state that DropTime acts on the output side of the SNN. But the paper does not explain why the same idea cannot be applied to the intermediate layers of the network. I would suggest the authors clarify whether this is a deliberate design choice, a technical limitation, or a scheme obtained from experiments.

  2. Insufficient ablation studies. The ablation studies are not enough to demonstrate the effectiveness of the proposed DropTime method. Does the conclusion drawn from Figure 2 also hold on other datasets and other SNN architectures? If it does not hold, an explanation is needed; if it does, it must be verified through experiments.

Questions for authors
  1. Why is DropTime applied only at the output end of the network? Have you tried masking timesteps at intermediate layers? If yes, what were the results? If not, what prevents this?

  2. Does the timestep-induced overfitting shown in Figure 2 (accuracy peaking at 4 timesteps and then dropping) also appear on other datasets? In the main results these datasets are only tested with one or two timestep settings, so it is hard to tell whether the phenomenon itself exists there.

Reviewer_xu5zOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents DropTime, a temporal regularization method for spiking neural networks. During training, the model randomly selects the final outputs from partial time steps and averages them according to the number of actually retained time steps. All time-step outputs are adopted during inference. This method preserves the internal neuronal dynamics and introduces no extra operations at the inference stage. Experiments are conducted on CIFAR-10, CIFAR-100, ImageNet and CIFAR10-DVS, including ablation studies on the number of time steps, retention probability and different normalization schemes.

Critical strengths

The proposed method is simple, low-cost, and introduces no additional inference operations. Specifically, DropTime only modifies output aggregation during training and requires no extra networks, auxiliary losses, or new neuronal states, making it easy to integrate into existing SNN training pipelines. Such usability constitutes a key advantage for a training regularization technique. If the performance gains can be stably reproduced, this method has the potential to serve as a standard training option. The paper includes extensive experiments and demonstrates effectiveness across multiple datasets.

Critical weaknesses
  1. Theorem 3.1 does not handle the case where all time steps are dropped. Under independent Bernoulli sampling, the probability of this event is (1−p)T(1-p)^T, which leads to a zero denominator in the aggregation formula. The proof directly states that the sum of probabilities from k=1k=1 to TT equals 1, omitting this event.

  2. Theorem 3.2 writes the second-order Taylor approximation as a strict equality and employs an invalid matrix trace transformation at the end of its proof. A sound second-order expression should retain the trace of the product between the Hessian and the output covariance, instead of directly decomposing it into the current form.

Questions for authors
  1. How is the all-zero mask handled? Is the mask resampled, reverted to full temporal averaging, or governed by other rules? Are masks sampled independently per sample or shared across the entire minibatch? Please provide definitions consistent with the unbiasedness proof.

  2. Please correct the matrix trace manipulation in Theorem 3.2, and could the paper explicitly state the conditions required for the second-order approximation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.