- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies physical width expansion of partially trained feed-forward LIF spiking networks. Each growth event selects donor neurons by the norm of their effective readout, copies each donor's incoming parameters and membrane state into a pair of newborn neurons, and connects the pair to the next layer through a difference coordinate with nonzero weight and a sum coordinate with zero weight. Because the clones spike identically, the pair adds nothing to the logits at birth, while its incoming parameters receive equal and opposite gradients. Growth is wrapped in a transaction that migrates Adam state with zero moments for new coordinates and accepts the expanded model only after checking logits, parameters, and optimizer state. Theorems cover temporal function preservation, conditional immediate trainability, optimizer-state preservation, and layerwise composition. Experiments include a matched 128→256 intervention against zero readout and width-matched uneven splitting, repeated growth on SpiralHard (99.48% test accuracy versus 99.40% for the best fixed width, with 60.8% fewer width-epochs), a two-hidden-layer variant, four growth policies, UCI HAR with post-hoc physical compaction, and DVS Gesture.
Critical strengths
- Exact, verifiable preservation under a strict no-rewrite constraint. The paired construction leaves every existing parameter and Adam moment bitwise unchanged and preserves the temporal logits exactly. The paper verifies this for every reported event, including 24 layerwise events in the two-hidden-layer model and 65 policy-driven transactions, all with zero recorded logit deviation (Appendices E.1 and E.2). The width-matched uneven-splitting control (Table D.2), which changes donor readout coordinates by about 1.0–1.1 and leaves float32 logit deviations of up to , makes the value of the constraint concrete. This matters to my assessment because exactness without rewriting existing coordinates is the property that distinguishes the method, and it is established beyond doubt.
- Careful mechanism controls and candid reporting. The matched intervention fixes the checkpoint, optimizer state, donors, and data order (Table D.1). The optimizer control shows that the opposed gradients exist before any update and that plain SGD also separates each pair, and the measured 3.149× effect of the inherited Adam clock matches the predicted 3.151× (Table D.3). The paper also reports results that do not favor it, such as lower DVS Gesture Macro-F1 than the fixed references in two of three seeds (Table F.2), and states that final-accuracy gains are not claimed. This matters because it makes the mechanism claims easy to verify and keeps the conclusions within what was tested.
Critical weaknesses
- The core construction and its guarantees add little beyond known techniques, and the problem it addresses lasts one update. With , the pair readouts and give the two clones outgoing weights and . This is the antisymmetric pairing used to obtain zero-output yet trainable initializations (e.g., the doubling trick in Chizat et al., 2019), applied here to clones of a trained neuron. Under the reported initialization (Eq. B.5), the clones' effective readouts are , and since they share the donor's trajectory, Eq. 13 makes their incoming gradients exactly times the donor's own gradient. The newborn neurons therefore start by moving along and against the donor's current descent direction, close to the neuron-splitting schemes the paper cites (Wu et al., 2019; 2020). Theorems 1 and 2 follow from the clones being deterministic copies and use no property of LIF dynamics beyond determinism, so the same argument holds for any recurrent unit. Growth methods that preserve the function while explicitly keeping nonzero gradients for new units, such as GradMax (Evci et al., 2022), are neither discussed nor compared. Finally, the limitation of zero readout in Eq. 7 lasts a single update: once receives its own nonzero gradient, the incoming parameters do too. The paper does not show that this one-step delay, or the donor readout change made by uneven splitting, measurably harms training. This is consequential because the novelty of the paper rests on the construction and on the importance of immediate trainability.
- The experiments do not show that immediate trainability improves growth outcomes, and the downstream comparisons are weakly controlled. When the growth schedule and final architecture are matched, the benefit is small or mixed: paired-basis and cloned-zero-outgoing growth differ by 0.07 points in the two-hidden-layer study (99.800% vs. 99.733%, Table E.1), and the matched single intervention gives final-accuracy differences of −0.15, +1.80, and +0.70 points (Table D.1). The larger gap in Table 2 (99.48% vs. 98.40%) comes from runs in which the controller grew the two methods to different widths (554.7 vs. 896). The zero-readout control is also degenerate. It attaches two clones of the same donor, whose gradients then differ only by the positive ratio of their importance coefficients; Adam's per-coordinate normalization largely removes such a ratio, so each pair can stay nearly identical long after birth. Standard symmetry-breaking baselines (one clone per donor, Net2Net-style cloning with noise, uneven splitting, GradMax) are absent from the repeated-growth experiments, and the fixed-width baselines lack the position-dependent importance scaling of the readout (Eqs. B.2–B.4) that both growing methods use. The remaining evidence is thin. The main benchmark is a 2-D synthetic task, and the 0.08-point margin over Fixed-best is well below the 0.25-point seed standard deviation of paired-basis itself (Table D.4). On UCI HAR, the growing models select width 128 for two of three seeds, i.e., checkpoints taken before any growth event. On DVS Gesture, the only event-based temporal task, growth trails the fixed references (−0.0116 Macro-F1, lower in two of three seeds). This is consequential because the method is motivated by better training after growth, and the current evidence supports exactness but not that benefit.
Questions for authors
- [Requested analysis] Could you add symmetry-breaking baselines to the repeated-growth experiments under matched growth schedules, namely zero readout with one clone per donor, Net2Net-style cloning with small noise, uneven splitting, and GradMax? For the current zero-readout control, please report the distance between the two clones of each donor at the end of training. The first-update separation in Table 1 cannot answer this, since its value in every seed () equals times the inherited Adam factor of 3.151 and therefore reflects the optimizer step size.
- [Clarification] Which growth controller produced Table 2, and why did zero readout grow to width 896 while paired-basis stopped at 554.7? Do Fixed-best and Fixed-matched use the importance-scaled readout of Eqs. B.2–B.4, and how does paired-basis perform without it?
- [Requested analysis] Is there a setting in which the one-step delay of zero readout, or the donor readout change of uneven splitting, measurably hurts final accuracy or training cost? Results on standard temporal SNN benchmarks such as SHD, SSC, or CIFAR10-DVS, with convolutional or recurrent SNNs and more seeds, would show whether exact preservation matters beyond 2-D tasks.
- [Clarification] Growth happens between mini-batches, where each sequence starts from the temporal initialization, so copying the donor's membrane state (Eq. 8) appears to copy only the initial state. Where does the stateful setting make growth harder than in a non-spiking network, and is there an experiment showing that a standard function-preserving method disrupts the temporal dynamics?
- [Suggestion for future work] Recurrent SNNs, in which newborn neurons also project back into their own layer, would test the construction where preservation requires cancellation in the recurrent input as well, and would better support the claim of handling stateful computation.