- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper aims to use regularization methods to alleviate the overfitting problem of spiking neural networks (SNNs) when using a large number of timesteps. In particular, the contributions of the paper are summarized as follows.
-
Algorithm. This paper proposes DropTime, which uses the average of randomly masked timestep logits as the outputs during training time. All timesteps are used during test time.
-
Theory. This paper proves that DropTime is unbiased and introduces an implicit regularization term proportional to the variance of temporal features, supporting the efficacy of DropTime.
-
Experiments. This paper verifies the superiority of DropTime over many sota methods on several datasets and observes that SNN with DropTime benefits from larger timesteps (Figure 2).
Critical strengths
-
DropTime is well motivated to alleviate the overfitting problem of SNNs when using a large number of timesteps.
-
DropTime is a simple method and does not increase the time and space complexity significantly. It can be directly used for most SNNs.
-
DropTime is verified to be effective over many baselines on many datasets.
Critical weaknesses
-
Some details of DropTime are not clear. For example, in Eq. (5), the 1-norm of m occurs in the denominator. It is unclear how to deal with the case where the 1-norm of m equals 0.
-
On line 275, the authors claim that DropTime has a lower variance than Temporal Dropout. However, this claim is not proven. In fact, here seems to be a counterexample: T=2, p=0.9, y1=1, y2=-1.
-
Some of the explanations of theories are overclaimed. For example, on line 282, the authors claim that the unbiased property can "ensure the accuracy of our method". However, the theories are not relevant to accuracy. In fact, biased estimators can sometimes increase the performance of an algorithm. Furthermore, Temporal Dropout is also unbiased, and this is not the key difference between DropTime and Temporal Dropout.
-
Some of the theories are not correct. For example, in Eq. (14) of Theorem 3.2, the variance term is a matrix, while the other two loss terms are scalars. This occurs due to an incorrect derivation on line 352, where an incorrect equality Tr(AB) = A Tr(B) is used.
-
In experiments, the authors do not report the standard deviation of the accuracy.
Questions for authors
See the weakness part.