- Rating
- 3
- Confidence
- 4(+2)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper concerns architecture search for spiking neural networks, where the differentiable formulation couples the architecture update to spatio-temporal backpropagation and therefore requires an unrolled backward graph. The authors decouple the two: shared weights are trained with STBP, while routing probabilities over candidate edges and operations are updated from spike timing statistics collected in a forward pass, with no backward path through the architecture parameters. The update takes a multiplicative STDP form, which the authors characterize as entropy mirror ascent on the node-wise simplices, driven by the gradient of an explicit concave potential at a frozen network state. Each iteration alternates a supervised lookahead step, a gradient-free routing update on held-out data, and a weight update under the revised routing. Stability, convergence and regret results are stated with their assumptions, including the caveat that the convergence result does not cover the AdamW optimizer used in the experiments. Experiments span CIFAR-10, CIFAR-100, ImageNet-1K, DVS-CIFAR-10 and DVS-Gesture, plus a speech dataset used only in the ablation; accuracy on the two CIFAR benchmarks is close to the strongest published SNN-NAS results, and search cost is lower than SNN-DARTS. In the timing ablation, replacing the timing rule with a Hebbian or rate-only rule produces architectures composed almost entirely of pooling and skip connections.
Critical strengths
- The timing ablation is the strongest part of the paper, and it is set up so that it could have failed. Replacing the STDP utility with a Hebbian or a rate-only rule, holding the rest of the pipeline fixed, drops CIFAR-10 retraining accuracy from 96.51 to 87.90 and 91.17. The reason is visible in what the search returns rather than only in the final number: both alternative rules produce cells made almost entirely of average pooling and skip connections, while the timing rule retains convolutions. I have not seen the rate-versus-timing contrast demonstrated this concretely for SNN architecture search, and it is a mechanism-level result rather than another benchmark entry.
- The search-cost reduction is measured against a matched baseline, and the paper reports where it disappears. With identical search hyperparameters, DNASNet uses roughly a fifth less peak memory and between a quarter and a half less wall-clock search time than SNN-DARTS on CIFAR-10 and CIFAR-100. On DVS-Gesture the paper states that peak memory is the same for every method, because at long sequence lengths the temporal backward graph is the bottleneck rather than the architecture update. Reporting the negative case next to the positive one is what makes the efficiency claim credible.
Critical weaknesses
- The accuracy claims either fall within run-to-run noise or rest on unmatched comparisons. The CIFAR-10 margin over NeuEvo is 0.08 points, smaller than the spread across the authors' own runs. The ImageNet gain comes from retraining a transferred CIFAR-10 cell against baselines that were searched on ImageNet, so it does not isolate the search method. On DVS-CIFAR-10 the method trails NeuEvo by 2.64 points, and SNN-DARTS was not searched on that dataset at all. The accuracy table is therefore not what supports the performance claim.
- The abstract claims near-state-of-the-art performance on DVS-Gesture, which the paper does not support. That benchmark appears only in the rule ablation and a retraining curve, and every number for it is measured against SNN-DARTS or the authors' own rule variants. No external baseline is listed for it anywhere in the submission, and the configuration used there is deliberately small. Either supply external baselines under a stated protocol or drop the claim.
Questions for authors
- The ImageNet result comes from retraining the best CIFAR-10 cell rather than from searching on ImageNet, while the baselines appear to be searched results, and the SEW-ResNet entry is quoted from NeuEvo. Can the authors say what the comparison looks like under matched protocols, or restate the claim as one about transfer? This is the number I would otherwise weigh most heavily, and at present I cannot tell how much of the 4.53-point gap comes from the search method and how much from the transfer setup.
- On CIFAR-10 the margin over NeuEvo is 0.08 points and the proposed method runs vary by 0.10. Do the authors regard the two methods as separated by these results? If not, how would you describe the accuracy contribution, as opposed to the search-cost contribution?
- SNN-DARTS was not searched on DVS-CIFAR-10, so the only in-house comparison there is a random-architecture baseline, and the remaining entries are published numbers from other protocols. Would it be feasible to search SNN-DARTS on that dataset, or to mark the comparison explicitly as cross-protocol in the table?
3 comments
I thank for the authors' rebuttal. I will raise my confidence after this rebuttal.
Thank you for your feedback!
Thank you very much for taking the time to review our rebuttal and for raising your confidence in our work! We greatly appreciate your constructive suggestions during the review process, which have helped us improve the clarity and rigor of the paper.
We appreciate your valuable feedback.
To Q1/W1:
We agree with the reviewer and have explicitly framed the ImageNet-1K experiment as evaluating cross-dataset cell transferability from CIFAR-10 in the revised PDF (Section 4, lines 326–329).
All search baselines on ImageNet in Table 1 report results obtained by transferring and retraining the best cell searched on CIFAR-10. Following the standard cell-transfer protocol established in DARTS and adopted by SNN-NAS baselines such as SpikeDHS and NeuEvo, we search for the cell topology on CIFAR-10, scale up the network depth and channels(Appendix C.3 shows the retrained network configuration after extension on ImageNet), and train the resulting target architecture from scratch on ImageNet-1K (whereas the remaining ImageNet-1K entries in Table 1 are handcrafted architectures without search).
Under this matched CIFAR-10-to-ImageNet-1K cell-transfer protocol, DNASNet (73.27%, 22M) outperforms both NeuEvo (68.74%, 26M) and SpikeDHS (68.64%, 58M) by more than 4.5 percentage points with fewer parameters. Note that the NeuEvo paper (from which the SEW-ResNet-34 baseline accuracy of 67.23% is cited) does not list the parameter count for SEW-ResNet-34.
To Q2/W1:
We agree with the reviewer: given the ±0.10% run-to-run variation, we regard DNASNet and NeuEvo as comparable on CIFAR-10 (we reworded this contribution in the revised edition, lines 067–069) rather than "performance gain". Rather, we characterize DNASNet's accuracy contribution by its substantial margins over NeuEvo on CIFAR-100 (+0.85 points) and ImageNet-1K transfer (+4.53 points), as well as its consistent gains over SNN-DARTS within the matched search space.
To Q3/W1:
We ran an additional SNN-DARTS search on DVS-CIFAR-10 using the T=10 protocol described in Appendix C.4, and retrained the selected architecture under the same evaluation protocol. In this search run, the normal cell contains only skip connections (visualization results are supplemented in Figure 7(e), lines 1542–1552):
The retraining results are as follows:
This skip-dominated outcome is consistent with the DVS-Gesture example in Appendix Figure 7(c)–(d), where six feedforward operations selected by SNN-DARTS are skip connections. That model reaches 89.76%, compared with 93.92% for Random and 97.05% for DNASNet.
One possible explanation is that surrogate-gradient propagation over longer temporal dependencies weakens architecture gradients for parameterized branches, favoring skip connections in the competition. DNASNet instead updates routing using timing statistics collected during forward propagation and avoids that architecture-gradient path.
We have added the new SNN-DARTS retraining result to the revised PDF (Section 4.2, Table 1, line 414) and updated Appendix C.4 to reflect that this additional search has now been performed (Appendix C.4.1, lines 1435–1437; Figure 7(e), lines 1542–1552).
To W2:
Thanks to the reviewer for pointing out this formulation problem, we removed it in the revised draft (Abstract, p.1, lines020-021). However, the scale was not "deliberatly reduced". DVS-Gesture was trained using a network of similar size (0.13M) in SEW-ResNet. They obtained 97.92% accuracy at T=16. DNASNet (0.15M, T=10, Accuracy=97.05%) still showed competitive performance.