- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work proposes NeuroProp, a memory-efficient training algorithm for spiking Transformers. It partitions the network into gradient-isolated units, each performing STBP with local supervision. To coordinate these units, AdaIP modulates neuronal input with semantic information. Experiments demonstrate significant memory reduction and improved training efficiency under pipeline parallelism with acceptable accuracy degradation compared to traditional STBP .
Critical strengths
- Reducing training memory cost of deep SNNs is meaningful. NeuroProp achieves considerable memory savings while maintaining relatively competitive performance.
- Extensive experiments are conducted across different tasks and diverse spiking transformer architectures, with comparison of training memory cost and throughput.
Critical weaknesses
- Two distinct conditioning strategies are proposed (Section 3.4), but their motivations are not sufficiently justified. Also, no controlled comparison is provided to show their effects on training and inference.
- The manuscript contains so many presentation issues that hurt readability. To name a few of them,
- The membrane decay coefficient is denoted as in Figure 4 and in Equation (7).
- The notation is unnecessarily confusing.
- ImageNet accuracy using Spikformer$_2$ is reported as 70.81% on Page 7, Line 355. However, in Table 1, the reported accuracy is 71.30%.
- in Equation (6) is not defined.
- Figure 4 use for semantic condition, while Equation (6) uses .
- There are several missing citations. To name a few of them,
- SpikingJelly is used without citing. [1]
- Gradient checkpointing with spike compression is compared but not cited. [2]
[1] Fang et al., Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence, Science Advances, 2023.
[2] Huang et al., Towards Lossless Memory-efficient Training of Spiking Neural Networks via Gradient Checkpointing and Spike Compression, ICLR 2026.
Questions for authors
- (Weakness 1) Why are both teacher-conditioned and prediction-conditioned training necessary? Could the authors clarify the motivation for each strategy and compare their performance under identical experimental settings?
- (Weakness 2) Please carefully revise the manuscript, notations, figures and tables for presentation issues.
- (Weakness 3) Please add necessary citations.
- Can NeuroProp be applied to other SNN architectures, such as convolutional SNNs (spiking ResNets, spiking VGG, ...)?
- How is AdaIP implemented in SDTv3, which uses stateless MultiSpike activations rather than LIF dynamics assumed in Equation (7)–(8)?
- Please provide the implementation details of pipeline parallelism in the appendix, including forward-backward scheduling, stage partitioning, microbatch configuration, communication, and the parallelization strategy of the global-STBP baseline.
5 comments
To Reviewer pd2X (Continued)
Q5: Stateless MultiSpike in SDTv3.
AdaIP is inspired by adaptive neuronal excitability. In LIF backbones it affects membrane integration and firing, and in stateless SDTv3 we adapt this principle through semantic-dependent feature modulation. Both guide representation formation, with different implementations and dynamical interpretations.Block-specific gains/biases include modulation strength and broadcast across time and space: Modulation precedes the token mixer, attention, and MLP. The first MultiSpike response is where W is the first convolution and Changing U(c) can cross fixed boundaries (0.5,1.5,2.5,3.5), altering output levels. Q and its stateless nature remain unchanged; the first two convolutional units are unmodulated. Eqs. (7)-(8) apply to LIF only. We have explicitly described this MultiSpike-specific implementation of AdaIP in Section A.2 of the Appendix in the revised manuscript.
Q6: Pipeline implementation.
Thank you for the suggestion. The table below summarizes the requested implementation details for Figure 8. We have also included it in the revised manuscript (Table 10 of Appendix).
| Item | Implementation |
|---|---|
| Forward/backward scheduling | Each NeuroProp unit performs local forward, backward, and update per microbatch, then forwards detached outputs. Different GPUs process different microbatches concurrently. |
| Stage partitioning | Logical-unit depths grouped by GPU: 16 blocks/4 GPUs: [1,2]; [2,2]; [2,2]; [2,3]. 24 blocks/4 GPUs: [1,1,2]; [2,2,3]; [2,2,3]; [2,2,2]. 32 blocks/6 GPUs: [2] on GPU0; [2,2,2] on each remaining GPU. |
| Microbatch configuration | NeuroProp loader batch: 4096; training/evaluation microbatches: 512/1024. Local optimizers update per training microbatch, without accumulation over 4096. |
| Communication | NCCL asynchronous point-to-point sends with bounded buffers. Training payloads: detached activations, labels, and metadata; inference payloads: activations and semantic predictions. No cross-unit gradient transmission. |
| Global-STBP baseline | DDP with one full model replica per GPU and synchronized gradients. For 16/24/32 blocks, GPU counts are 4/4/6, per-GPU batches 64/48/36, and global update batches 256/192/216. |
Dear Reviewer,
We hope this message finds you well.
We are writing to kindly follow up on our previous responses to your comments. If there are any remaining concerns or points that need clarification, please feel free to let us know. We would be happy to provide additional explanations.
Thank you very much for your time and effort in reviewing our manuscript. We sincerely appreciate your valuable comments and suggestions.
Best regards,
Authors
To Reviewer pd2X
Thank you for your thorough evaluation and constructive suggestions on our work. We believe we have addressed all the concerns you raised. Our responses to your concerns and questions are as follows.
Thank you for this insightful question. To address this issue, we have added new experimental results and analysis in Section A.2 and Table 9 of the Appendix in the revised manuscript.
Teacher‑conditioned and prediction‑conditioned training are alternative approaches for building local semantic conditions, not two stages that must be combined. On ImageNet‑1K, the performance gap is highly dependent on the conditioning choice: prediction conditioning yields a 1.40% gain for the four‑block single‑step model, rising to 14.36% and 14.81% for the two four‑step setups. We thus report results for models trained with prediction conditioning.
To further investigate the differences among distinct conditions, we have also added an experiment on CIFAR‑100, which shows a performance of 71.03 ± 0.41% under prediction conditioning. These results support prediction conditioning as a useful first choice for more demanding training configurations, though the strategy should ultimately be selected through matched experiments rather than by dataset size alone. We have included these comparisons and clarified the motivation and protocol choices in the revised manuscript.
Thank you for identifying these issues. We agree that notation inconsistencies and numerical errors hindered readability. In the revision, we have checked and corrected the text, equations, figures, and captions. We use λ for membrane decay, γ(g) for modulation gain, g solely as the unit index, and c(g) for semantic conditions, and explicitly define D as the feature embedding dimension. We corrected the eight-block SpikFormer's ImageNet-1K accuracy to 71.30%, consistent with Table 1, and expanded captions to explain symbols, operations, and information-flow directions. We hope the revised manuscript is now clear enough.
Thank you for identifying these omissions. We have added SpikingJelly [1] to the implementation description and [2] to the checkpointing/spike-compression baseline description and comparison table. We also checked the other baseline and implementation descriptions and added citations for the supplementary experiments: SEW-ResNet [3], SLTT [4], and NDOT [5].
References
[1] Fang, W., et al. "SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence." Science Advances 9(40): eadi1480, 2023.
[2] Huang, Y., et al. "Towards Lossless Memory-efficient Training of Spiking Neural Networks via Gradient Checkpointing and Spike Compression." ICLR, 2026.
[3] Fang, W., et al. "Deep Residual Learning in Spiking Neural Networks." NeurIPS 34: 21056-21069, 2021.
[4] Meng, Q., et al. "Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks." ICCV, 2023.
[5] Jiang, H., et al. "NDOT: Neuronal Dynamics-based Online Training for Spiking Neural Networks." ICML, 2024.
We greatly appreciate your constructive suggestions, which may significantly broaden the impact of our work. Our answer is Yes! NeuroProp only requires a hierarchical network and is not a dedicated method designed exclusively for Transformer architectures. In convolutional SNNs, AdaIP generates channel-wise gains/biases broadcast over both spatio-temporal direction, applied before residual-branch LIFs in SEW-ResNet and convolutional LIFs in spiking VGG. The added experimental results on CIFAR-100 are as follows:
Overall, both residual and sequential convolutional SNNs achieve substantial memory savings. The accuracy approaches that of STBP on SEW‑ResNet34, slightly surpasses STBP on VGG11, and delivers a more notable advantage on SEW‑ResNet18. Supplementary experimental settings and detailed results are provided in Section F and Table 12 of the Appendix in the revised manuscript.