- Rating
- 1
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The standard spiking neural network architecture with leaky-integrate and fire neurons is modified in two ways. The first modification replaces the scalar LIF temporal dynamics with an expansion to dimensions that is followed by a linear transformation back to . The second modification replaces the threshold firing rule with the sampling of a Bernoulli random variable whose parameter is determined by the membrane potential. A number of experiments are carried out to test the performance of these two modifications.
Critical strengths
The paper presents an understandable treatment of two interesting modifications.
Critical weaknesses
I suggest reject primarily due to issues with the experiments. The paper would be strengthened if it can isolate the effects of the modifications. It is likely that the modifications are not always desirable, and so identifying the situations where the modifications are and are not beneficial would be valuable. If comparing to prior works, their methods should be implemented, which gives the opportunity for confounds such as architecture size to be controlled and will allow for a shared training and testing infrastructure.
1. Experiments do not isolate the proposed changes (e.g. parameter count confound)
The two proposed modifications that come under the paper's PND label can be dropped in to any architecture that supports it. Section 4.3 does some work to isolating their effect, but it is insufficient. Section 4.3 tests a PND applied to a single base architecture compared to PND w/o probabilistic firing and PND w/o temporal expansion. The natural fourth condition (LIF dynamics with fixed firing threshold) is missing. Furthermore, a baseline that has its parameter counts match those of PND (which has increased learned parameters due to the temporal expasnion) should be included, as currently there is no control for parameter count. Lastly, an ablation that covers various base architecture sizes would be valuable. Without these comparisons, it is very hard to understand the effect of the modifications.
2. Weakness of comparison to other models (e.g. training and testing is not controlled)
Table 1 compares models from other works with the base architecture (from appendix E.1) modified with the PND modifications (the latter I'll refer to as the PND model). It appears that many hyperparameters of the PND model are tuned per-dataset, including the expansion dimension . The baseline model performances look to be copied from other papers (also per-dataset tuned). A weakness of such a comparison is that one can "beat" another model by simply performing a more thorough hparam search. The response to this criticism may be: "this is what is done in the literature", and that may be true, but that does not change the fact that the comparison does not give the reader much understanding of the two proposals presented by this paper.
Furthermore, by copying the results from other papers, rather than re-running the baseline models, it is possible that the training and eval setup differences may be the causes of score differences. Consider the AAN dataset: the paper describes concatenation along the batch dimension (presumably for efficient GPU utilization), but the code (from the accompanying zip) concatenates them along the feature dimension. This suggests that the PND model is likely trained and evaluated differently to the other models and the results cannot be compared apples-to-apples to the other baselines' scores. A stronger comparison would have the authors implement the models and train and test them together.
Related to the tuning for Table 1, results in Figure 2 are also created using a base architecture specific to those tests (with different layer counts and widths). There is nothing wrong with this per se, but there is no justification given for it.
3. Test set used for checkpoint selection
You say: "IMDB has no separate validation split and reports the best test accuracy recorded during training" (L957). Performance on the test set should not be used for choosing the model checkpoint. A lack of a designated validation set is common, and practitioners should create their own validation set from the training set. By not specifying a fixed validation set, the dataset supports things such as k-fold cross-validation. Again, it is not a justification that prior work has also made this mistake. This is another reason why all baselines should be implemented and trained-tested on the same setup: so that you do not inherit flaws of previous works.
4. Variability quantification
It looks like there are just single runs for the results in Table 1. Some of the accuracies are quite similar, and it's not clear if stochasticity in training or evaluation can be ruled out as explanations for some of the differences.
Some issues/suggestions
The following are more minor concerns.
-
Terms used in the paper should be defined on first use, even when standard. Mention that is the input history, and specify its type (sequence of reals) (L157). In the appendix, terms like , \tau_m, \tau $ are not defined, but are assumed to be known.
-
"temporal mode" is a fundamental concept to the paper's argument and is used multiple times before any definition.
-
Appendix A1 uses the same u(t) symbols to refer to both the continuous and discrete functions.
-
The in is not defined. It appears to be a real, but it is easily confused with an attempt to represent a small change in .
-
LIF uses symbol $ u_t $ but also the (undefined) .
-
For consistency with later used indexing, equation 3 could be written as:
\qquad K_\tau=\alpha^\tau.$$ Which puts K in the LHS and uses the same notation that is used elsewhere. -
Equation 6 depends on 7, so swap the order, or combine onto one line as one numbered equation.
-
The fact that the paper concerns spiking neural networks should be mentioned in the abstract or title
-
The differences to related work are mentioned in Section 2; however, statements like "These mechanisms enrich temporal behavior, but state dimension alone does not establish the independence of the resulting responses or how they combine into a scalar membrane trajectory" (L134-136) and "Their conditional event distributions alone do not establish prediction consistency under repeated sampling, which requires empirical evaluation" (L141-142) are stated without any supporting argument or evidence.
Questions for authors
Why present the two quite separate modifications together? I imagine that characterizing one of these in detail would create enough material for a single paper. Is the interaction between both important?