acceptodds
Under review as a conference paper at ICLR 2027

Exact and Trainable Growth of Stateful Spiking Neural Networks

Abstract

Spiking neural networks (SNNs) typically rely on fixed architectures whose ca- pacity is chosen before training. Insufficient width limits representation, whereas excessive width introduces unnecessary parameters and computation. This problem is especially challenging in SNNs because structural changes can disrupt their temporal dynamics. We introduce an exact and trainable growth mechanism for stateful SNNs. The proposed paired basis adds new neurons to a partially trained network while preserving the downstream temporal computation and output logits at every simulation step. Unlike conventional zero readout, which preserves the current output but blocks the immediate task gradient to the incoming parameters of newborn neurons, our construction uses identical neuron pairs whose forward contributions cancel exactly while their backward signals remain nonzero and opposing. Growth is implemented as an atomic transaction that expands the model, migrates Adam state, verifies the required invariants, and resumes surrogate gradi- ent training. We establish temporal function preservation, conditional immediate trainability, preservation of the optimizer state for existing parameters, and layer- wise composition, providing a principled mechanism for expanding stateful SNNs without discarding their learned temporal computation.

Primary area:
optimization
Keywords:
Spiking Neural Networks, Neural Network Growth, Function-Preserving Expansion, Adaptive Model Capacity, Surrogate Gradient Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wdQ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies physical width expansion of partially trained feed-forward LIF spiking networks. Each growth event selects donor neurons by the norm of their effective readout, copies each donor's incoming parameters and membrane state into a pair of newborn neurons, and connects the pair to the next layer through a difference coordinate with nonzero weight and a sum coordinate with zero weight. Because the clones spike identically, the pair adds nothing to the logits at birth, while its incoming parameters receive equal and opposite gradients. Growth is wrapped in a transaction that migrates Adam state with zero moments for new coordinates and accepts the expanded model only after checking logits, parameters, and optimizer state. Theorems cover temporal function preservation, conditional immediate trainability, optimizer-state preservation, and layerwise composition. Experiments include a matched 128→256 intervention against zero readout and width-matched uneven splitting, repeated growth on SpiralHard (99.48% test accuracy versus 99.40% for the best fixed width, with 60.8% fewer width-epochs), a two-hidden-layer variant, four growth policies, UCI HAR with post-hoc physical compaction, and DVS Gesture.

Critical strengths
  1. Exact, verifiable preservation under a strict no-rewrite constraint. The paired construction leaves every existing parameter and Adam moment bitwise unchanged and preserves the temporal logits exactly. The paper verifies this for every reported event, including 24 layerwise events in the two-hidden-layer model and 65 policy-driven transactions, all with zero recorded logit deviation (Appendices E.1 and E.2). The width-matched uneven-splitting control (Table D.2), which changes donor readout coordinates by about 1.0–1.1 and leaves float32 logit deviations of up to 1.43×10−61.43\times10^{-6}, makes the value of the constraint concrete. This matters to my assessment because exactness without rewriting existing coordinates is the property that distinguishes the method, and it is established beyond doubt.
  2. Careful mechanism controls and candid reporting. The matched intervention fixes the checkpoint, optimizer state, donors, and data order (Table D.1). The optimizer control shows that the opposed gradients exist before any update and that plain SGD also separates each pair, and the measured 3.149× effect of the inherited Adam clock matches the predicted 3.151× (Table D.3). The paper also reports results that do not favor it, such as lower DVS Gesture Macro-F1 than the fixed references in two of three seeds (Table F.2), and states that final-accuracy gains are not claimed. This matters because it makes the mechanism claims easy to verify and keeps the conclusions within what was tested.
Critical weaknesses
  1. The core construction and its guarantees add little beyond known techniques, and the problem it addresses lasts one update. With U=0U=0, the pair readouts c(V+U)c(V+U) and c(−V+U)c(-V+U) give the two clones outgoing weights +cV+cV and −cV-cV. This is the antisymmetric pairing used to obtain zero-output yet trainable initializations (e.g., the doubling trick in Chizat et al., 2019), applied here to clones of a trained neuron. Under the reported initialization V=γer/cV=\gamma e_r/c (Eq. B.5), the clones' effective readouts are ±γer\pm\gamma e_r, and since they share the donor's trajectory, Eq. 13 makes their incoming gradients exactly ±γ\pm\gamma times the donor's own gradient. The newborn neurons therefore start by moving along and against the donor's current descent direction, close to the neuron-splitting schemes the paper cites (Wu et al., 2019; 2020). Theorems 1 and 2 follow from the clones being deterministic copies and use no property of LIF dynamics beyond determinism, so the same argument holds for any recurrent unit. Growth methods that preserve the function while explicitly keeping nonzero gradients for new units, such as GradMax (Evci et al., 2022), are neither discussed nor compared. Finally, the limitation of zero readout in Eq. 7 lasts a single update: once WnewW_{\mathrm{new}} receives its own nonzero gradient, the incoming parameters do too. The paper does not show that this one-step delay, or the donor readout change made by uneven splitting, measurably harms training. This is consequential because the novelty of the paper rests on the construction and on the importance of immediate trainability.
  2. The experiments do not show that immediate trainability improves growth outcomes, and the downstream comparisons are weakly controlled. When the growth schedule and final architecture are matched, the benefit is small or mixed: paired-basis and cloned-zero-outgoing growth differ by 0.07 points in the two-hidden-layer study (99.800% vs. 99.733%, Table E.1), and the matched single intervention gives final-accuracy differences of −0.15, +1.80, and +0.70 points (Table D.1). The larger gap in Table 2 (99.48% vs. 98.40%) comes from runs in which the controller grew the two methods to different widths (554.7 vs. 896). The zero-readout control is also degenerate. It attaches two clones of the same donor, whose gradients then differ only by the positive ratio of their importance coefficients; Adam's per-coordinate normalization largely removes such a ratio, so each pair can stay nearly identical long after birth. Standard symmetry-breaking baselines (one clone per donor, Net2Net-style cloning with noise, uneven splitting, GradMax) are absent from the repeated-growth experiments, and the fixed-width baselines lack the position-dependent importance scaling of the readout (Eqs. B.2–B.4) that both growing methods use. The remaining evidence is thin. The main benchmark is a 2-D synthetic task, and the 0.08-point margin over Fixed-best is well below the 0.25-point seed standard deviation of paired-basis itself (Table D.4). On UCI HAR, the growing models select width 128 for two of three seeds, i.e., checkpoints taken before any growth event. On DVS Gesture, the only event-based temporal task, growth trails the fixed references (−0.0116 Macro-F1, lower in two of three seeds). This is consequential because the method is motivated by better training after growth, and the current evidence supports exactness but not that benefit.
Questions for authors
  1. [Requested analysis] Could you add symmetry-breaking baselines to the repeated-growth experiments under matched growth schedules, namely zero readout with one clone per donor, Net2Net-style cloning with small noise, uneven splitting, and GradMax? For the current zero-readout control, please report the distance between the two clones of each donor at the end of training. The first-update separation in Table 1 cannot answer this, since its value in every seed (6.302×10−36.302\times10^{-3}) equals 2η2\eta times the inherited Adam factor of 3.151 and therefore reflects the optimizer step size.
  2. [Clarification] Which growth controller produced Table 2, and why did zero readout grow to width 896 while paired-basis stopped at 554.7? Do Fixed-best and Fixed-matched use the importance-scaled readout of Eqs. B.2–B.4, and how does paired-basis perform without it?
  3. [Requested analysis] Is there a setting in which the one-step delay of zero readout, or the donor readout change of uneven splitting, measurably hurts final accuracy or training cost? Results on standard temporal SNN benchmarks such as SHD, SSC, or CIFAR10-DVS, with convolutional or recurrent SNNs and more seeds, would show whether exact preservation matters beyond 2-D tasks.
  4. [Clarification] Growth happens between mini-batches, where each sequence starts from the temporal initialization, so copying the donor's membrane state (Eq. 8) appears to copy only the initial state. Where does the stateful setting make growth harder than in a non-spiking network, and is there an experiment showing that a standard function-preserving method disrupts the temporal dynamics?
  5. [Suggestion for future work] Recurrent SNNs, in which newborn neurons also project back into their own layer, would test the construction where preservation requires cancellation in the recurrent input as well, and would better support the claim of handling stateful computation.
Reviewer_1Aq4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper validates the key properties of the proposed pairwise-birth growth mechanism using synthetic classification tasks with controllable capacity requirements and the UCI HAR time-series benchmark as the primary experimental settings, supplemented by experiments in the event-driven DVS Gesture scenario in the appendix. Multiple control baselines, including zero-readout and non-uniform splitting, are employed. The experiments cover several aspects, including mechanism isolation, progressive growth, multilayer expansion, strategy comparison, and physical compression. The rigorous comparisons clearly isolate the effects of the proposed mechanism and provide strong support for its core claims of exact preservation of temporal computation and immediate trainability of newly generated neurons. However, the overall dataset scale is relatively limited, with experiments primarily conducted on synthetic tasks and conventional time-series classification benchmarks. Validation on mainstream large-scale visual spiking benchmarks, such as CIFAR10-DVS, is absent. Consequently, the applicability of the method to convolutional SNNs and complex visual tasks, as well as its practical performance gains in these settings, remains unclear, somewhat limiting the work's case for engineering applicability and generalizability.

Critical strengths
  1. The first major strength is the proposed pairwise-birth exact growth mechanism. Through a novel construction based on symmetric neuron pairs and difference-and-sum coordinate decomposition, the method simultaneously achieves three objectives in stateful spiking neural networks: exactly preserving temporal computation at the moment of growth, leaving all existing parameters and optimizer states unchanged, and immediately providing non-zero input-side learning gradients to newly generated neurons. This elegantly resolves the inherent trade-off in conventional growth strategies, where zero-readout expansion preserves the output but blocks gradient flow, while non-uniform splitting enables gradient flow at the cost of modifying existing parameters. Combined with rigorous formal theoretical proofs and an atomic transactional implementation, the method establishes a complete connection between mathematical principles and engineering implementation. It represents a solid methodological advance in structural learning for spiking neural networks and constitutes the paper's central innovative contribution, directly underpinning its academic value.

  2. The second major strength is the rigorous and comprehensive experimental validation framework, which follows a hierarchical evaluation strategy encompassing mechanism isolation, progressive expansion, scenario extension, and practical implementation. Multiple carefully designed baselines, including zero-readout expansion and non-uniform splitting, are employed to provide precise comparisons. Beyond validating the exactness and trainability of the growth mechanism itself, the experiments systematically investigate compatibility with multilayer expansion, adaptability to different growth strategies, and the feasibility of post-training physical compression. The conclusions are consistently supported by controlled comparisons, lending them high credibility. This rigorous experimental design demonstrates the robustness of the core findings rather than relying on potentially incidental performance gains on specific tasks, providing substantial support for the method's practical applicability and the paper's publication merit.

Critical weaknesses
  1. Limited empirical evaluation across SNN architectures and tasks. The experiments primarily focus on fully connected feedforward SNNs with LIF neurons. Although the paper includes experiments on DVS Gesture, it does not evaluate the proposed growth mechanism on more complex visual benchmarks such as CIFAR10-DVS or on convolutional and recurrent SNN architectures. Its applicability to diverse network structures, neuron models, and more challenging spatiotemporal tasks therefore remains insufficiently established. In addition, the method focuses on network expansion without considering how growth could be integrated with neuron pruning or other capacity-control mechanisms, leaving its potential for more general dynamic network adaptation unexplored.

  2. Limited analysis of the sensitivity to growth-related design choices. The method uses the L2L_2 norm of each neuron's effective downstream readout to select donor neurons, while growth decisions and capacity allocation rely on validation-based criteria. Although the paired-basis initialization is theoretically motivated and supported by formal analysis, the paper provides limited systematic comparisons of alternative donor-selection strategies and growth schedules. It remains unclear how sensitive the method's performance and capacity allocation are to these choices, and whether the proposed strategies are consistently effective across different tasks and network configurations.

Questions for authors

1. On the necessity of preserving existing optimizer states.

The proposed method emphasizes preserving all existing parameters and optimizer states during network expansion. Could the authors clarify whether preserving the optimizer states is essential for maintaining training stability and subsequent convergence, or primarily an implementation-level design choice? Have the authors observed any meaningful differences when existing optimizer states are reinitialized after growth?

2. On the relationship between immediate trainability and subsequent learning dynamics.

The theoretical analysis establishes that newborn neuron pairs receive nonzero and opposing gradients immediately after growth. However, does this immediate symmetry breaking necessarily translate into sustained functional differentiation and improved learning efficiency during subsequent training? Could the authors clarify whether the newborn neurons continue to develop distinct temporal representations, rather than becoming functionally redundant after several optimization steps?

3. On the practical computational overhead of physical growth.

Although the paper reports reductions in cumulative parameter allocation and inference computation proxies, the proposed atomic growth transaction involves parameter expansion, optimizer-state migration, and invariant verification. Could the authors clarify the computational and memory overhead of these operations relative to the overall training process? This would help determine whether the reported capacity-efficiency advantages translate into practical training efficiency.

Reviewer_V8KWOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed to grow networks during learning by adding pairs of neurons with identical inputs, but outputs that cancel each other. This allows the gradient to flow backward through the outputs and simultaneously leaves the learned network structures and the layer outputs unchanged.

Critical strengths

The idea of network growth by pair-based addition of neurons that clone existing neurons, keep up the gradient flow and mutually cancel their outputs such that the output is preserved, is nice and valuable.

Critical weaknesses

None.

Questions for authors

A to some extent related idea seems to be put forward for non-spiking networks in the preprint Sareen,.., Ravanbakhsh The role of symmetry... 2026, Eqs. 15, 16 and accompanying text on p. 22. It may be good to discuss this article.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.