acceptodds
Under review as a conference paper at ICLR 2027

Morphing with Spikes

Abstract

Net2net techniques are used to increase the capacity of ANNs without any loss in accuracy during training. This paper extends those techniques to spiking neural networks. We show that net2widernet is exact on SNNs but net2deepernet is not when the inserted layers receive non-binary inputs. We also provide a method of mitigating the accuracy loss introduced by net2deepernet on spiking networks that optimizes the inserted layer’s gain via gridsearch. We demonstrate both afore- mentioned claims with experiments on image classification datasets CIFAR-10, CIFAR-100, CIFAR-10-DVS, and the RL task Cartpole, where our gain-searched spiking morphism techniques achieve substantially higher prediction agreement than raw insertion, allowing future work to build stronger and more efficient neu- ral architecture search algorithms for spiking neural networks

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
spiking neural networks, neuromorphic computing, network morphism, net2net

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rSGQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper extends Net2Net growth operators to spiking neural networks with LIF and ALIF neurons. Widenning duplicates channels with their BN, weight, and neuron states, and deepening inserts and identity conv and an LIF layer. The paper scales the identity kernel by a gain chosen per layer by grid search. Experiments use a 5-layer conv SNN on CIFAR-10, CIFAR-100 and CIFAR-10-DVS and an R-STDP on CartPole, reporting agreement at the growth point and accuracy curves afterwards.

Critical strengths

Problems are clear, and proofs are correct. Negative results are reported openly. Both BP and R-STDP training are covered.

Critical weaknesses

The technical step beyond net2net is small. The widening proof holds for any deterministic stateful unit and uses nothing specific to SNN.

Copied channels get identical gradients. The introduction says so itself, but the method didn't solve this problem.

On CartPole, agreement is measured with neuron noise turned off, so it does not show that the stochastic policy is preserved.

The Experiments do not show that growth helps training. The base stops training at the growth point, and the from-scratch model gets only half the budget.

Questions for authors

How do the methods compare with a base trained for 30 epochs or a grown model trained from scratch for 30 epochs?

Reviewer_JeWGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends Net2Net techniques, a method to accelerate the training of a larger neural network using function-preserving transformations, to the SNN realm. The authors discuss the feasibility of Net2WiderNet and Net2DeeperNet when applying them to widen or deepen SNNs and provide a mitigation method to handle the accuracy loss caused by Net2DeeperNet. They evaluate the method on several image classification datasets and an RL task.

Critical strengths
  1. The paper discusses Net2Net techniques on spiking neural networks.
  2. Both image classification datasets (e.g., CIFAR-10) and an RL task were evaluated.
Critical weaknesses
  1. The presentation of this paper is not clear. No workflow plot or pseudocode is provided to help the readers better understand the authors’ claims about their method. Additionally, the proofs provided in Sections 2.2 and 2.3 are textual narratives rather than rigorous proofs. Such insufficient methodology illustration makes this paper more like a simple report rather than an academic paper.
  2. The investigation of previous work is incomplete. In Section A.1, the authors mention several function-preserving growth studies such as NAS methods and claim that “Each of these methods, however, only applies to ANNs. Thus, this paper extends them to the realm of SNNs”. However, NAS has already been well-studied as an efficient way to find better SNN architectures [1].
  3. The novelty of this paper is very limited. The main contribution is transplanting existing Net2Net methods from ANNs to SNNs. Literally, Net2WiderNet is naturally applicable to SNNs without the need to consider specific properties of SNNs, while Net2DeeperNet only makes a difference in two situations: adaptive LIF, for which it is unsuitable, and the pooling operation. So the potential contribution is already limited. However, the authors even discard further exploration of adaptive LIF and only target the pooling problem. The subsequent gain search to relieve the pooling problem is also a commonly seen engineering calibration approach in previous SNN studies.
  4. The experimental validation of the method is limited, given the fact that Net2Net [1] was validated on the Inception architecture and the ImageNet dataset. Besides, SNNs have also achieved great performance on ImageNet in a line of previous studies. The proposed method in this paper is only tested on 5-layer VGG networks and an MLP. Meanwhile, only the results on CIFAR are reported, with a limited scale of image tasks.

Reference: [1] Kim, Youngeun, et al. "Neural architecture search for spiking neural networks." European conference on computer vision. Cham: Springer Nature Switzerland, 2022. [2] Chen, Tianqi, Ian Goodfellow, and Jonathon Shlens. "Net2net: Accelerating learning via knowledge transfer." arXiv preprint arXiv:1511.05641 (2015).

Questions for authors

Please refer to the weeknesses.

Reviewer_C1jFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates network morphism for spiking neural networks (SNNs), extending the classical Net2Net framework to support function-preserving network growth. The authors propose widening and deepening operations for LIF- and ALIF-based networks, together with a gain-search strategy to mitigate the performance degradation caused by deepening. Experiments on CIFAR-10, CIFAR-100, CIFAR-10-DVS, and CartPole demonstrate the effectiveness of the proposed methods. While the paper addresses an interesting problem and provides useful theoretical analysis, several aspects of the technical contributions and experimental validation need further improvement.

Critical strengths
  1. The paper addresses an interesting problem of function-preserving structural growth in SNNs, which has received relatively limited attention compared with conventional ANN architecture optimization.

  2. The theoretical analysis of the widening operation is clear and provides a reasonable explanation of why neuron duplication can preserve the temporal dynamics of LIF and ALIF networks.

Critical weaknesses
  1. The paper addresses an interesting problem of function-preserving structural growth in SNNs, which has received relatively limited attention compared with conventional ANN architecture optimization.

  2. The theoretical analysis of the widening operation is clear and provides a reasonable explanation of why neuron duplication can preserve the temporal dynamics of LIF and ALIF networks.1. The paper addresses an interesting problem of function-preserving structural growth in SNNs, which has received relatively limited attention compared with conventional ANN architecture optimization.

  3. The theoretical analysis of the widening operation is clear and provides a reasonable explanation of why neuron duplication can preserve the temporal dynamics of LIF and ALIF networks.

Questions for authors

1.Can the authors provide more evidence that function-preserving growth offers significant advantages over random growth in terms of convergence speed, training efficiency, and final performance?

2.How sensitive is the proposed gain-search strategy to the calibration-set size and the predefined search range? Can the selected gain generalize well to unseen samples?

3.Can the authors further validate the proposed methods on larger-scale datasets or more complex SNN architectures to demonstrate their scalability and generalization ability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.