acceptodds
Under review as a conference paper at ICLR 2027

NeuroProp: Bio-Inspired Local STBP for Memory-Efficient Spiking Transformers

Abstract

Spatiotemporal backpropagation (STBP) is among the strongest methods for directly training deep spiking neural networks (SNNs), yet its globally coupled graph retains neuronal states across both depth and time. We introduce NeuroProp, a bio-inspired framework that reduces this memory burden by localizing, rather than replacing, STBP. NeuroProp partitions a spiking Transformer into consecutive units, stops gradients at unit boundaries, and performs full surrogate-gradient STBP over the blocks and time steps within each unit. This converts the depth-wise activation graph from a whole-network dependency to one governed by the largest unit. To coordinate gradient-isolated units, we propose adaptive intrinsic plasticity (AdaIP), inspired by contextual neuromodulation of neuronal excitability. AdaIP broadcasts a semantic condition and transforms it into channel-wise synaptic gain and membrane bias, equivalently adjusting the effective firing threshold and equilibrium potential of LIF neurons. Detached inter-unit interfaces also admit pipeline-parallel execution. NeuroProp reduces peak memory by on SpikFormer and on SDTv3 in controlled CIFAR-100 experiments, while AdaIP recovers part of the accuracy lost by naive localization. Without global-STBP fine-tuning, NeuroProp reaches ImageNet-1K top-1 accuracy. These results establish local STBP with adaptive neuronal modulation as a practical accuracy–memory trade-off for scaling spiking Transformers.

Primary area:
learning theory
Keywords:
Spiking Transformer, Bio-inspired learning, Neuroscience.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pd2XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes NeuroProp, a memory-efficient training algorithm for spiking Transformers. It partitions the network into gradient-isolated units, each performing STBP with local supervision. To coordinate these units, AdaIP modulates neuronal input with semantic information. Experiments demonstrate significant memory reduction and improved training efficiency under pipeline parallelism with acceptable accuracy degradation compared to traditional STBP .

Critical strengths
  1. Reducing training memory cost of deep SNNs is meaningful. NeuroProp achieves considerable memory savings while maintaining relatively competitive performance.
  2. Extensive experiments are conducted across different tasks and diverse spiking transformer architectures, with comparison of training memory cost and throughput.
Critical weaknesses
  1. Two distinct conditioning strategies are proposed (Section 3.4), but their motivations are not sufficiently justified. Also, no controlled comparison is provided to show their effects on training and inference.
  2. The manuscript contains so many presentation issues that hurt readability. To name a few of them,
    • The membrane decay coefficient is denoted as τ\tau in Figure 4 and λ\lambda in Equation (7).
    • The notation g(g)g^{(g)} is unnecessarily confusing.
    • ImageNet accuracy using Spikformer$_2$ is reported as 70.81% on Page 7, Line 355. However, in Table 1, the reported accuracy is 71.30%.
    • DD in Equation (6) is not defined.
    • Figure 4 use zz for semantic condition, while Equation (6) uses c(g)c^{(g)}.
  3. There are several missing citations. To name a few of them,
    • SpikingJelly is used without citing. [1]
    • Gradient checkpointing with spike compression is compared but not cited. [2]

[1] Fang et al., Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence, Science Advances, 2023.

[2] Huang et al., Towards Lossless Memory-efficient Training of Spiking Neural Networks via Gradient Checkpointing and Spike Compression, ICLR 2026.

Questions for authors
  1. (Weakness 1) Why are both teacher-conditioned and prediction-conditioned training necessary? Could the authors clarify the motivation for each strategy and compare their performance under identical experimental settings?
  2. (Weakness 2) Please carefully revise the manuscript, notations, figures and tables for presentation issues.
  3. (Weakness 3) Please add necessary citations.
  4. Can NeuroProp be applied to other SNN architectures, such as convolutional SNNs (spiking ResNets, spiking VGG, ...)?
  5. How is AdaIP implemented in SDTv3, which uses stateless MultiSpike activations rather than LIF dynamics assumed in Equation (7)–(8)?
  6. Please provide the implementation details of pipeline parallelism in the appendix, including forward-backward scheduling, stage partitioning, microbatch configuration, communication, and the parallelization strategy of the global-STBP baseline.
5 comments
AuthorsOpenReview ↗

To Reviewer pd2X

Thank you for your thorough evaluation and constructive suggestions on our work. We believe we have addressed all the concerns you raised. Our responses to your concerns and questions are as follows.

W1 + Q1: Why two conditioning strategies?

Thank you for this insightful question. To address this issue, we have added new experimental results and analysis in Section A.2 and Table 9 of the Appendix in the revised manuscript.

Teacher‑conditioned and prediction‑conditioned training are alternative approaches for building local semantic conditions, not two stages that must be combined. On ImageNet‑1K, the performance gap is highly dependent on the conditioning choice: prediction conditioning yields a 1.40% gain for the four‑block single‑step model, rising to 14.36% and 14.81% for the two four‑step setups. We thus report results for models trained with prediction conditioning.

Time stepsTransformer blocksTeacher-conditioned top-1 (%)Prediction-conditioned top-1 (%)
1461.3262.72
4456.2870.64
4856.4971.30

To further investigate the differences among distinct conditions, we have also added an experiment on CIFAR‑100, which shows a performance of 71.03 ± 0.41% under prediction conditioning. These results support prediction conditioning as a useful first choice for more demanding training configurations, though the strategy should ultimately be selected through matched experiments rather than by dataset size alone. We have included these comparisons and clarified the motivation and protocol choices in the revised manuscript.

W2 + Q2: Presentation.

Thank you for identifying these issues. We agree that notation inconsistencies and numerical errors hindered readability. In the revision, we have checked and corrected the text, equations, figures, and captions. We use λ\lambda for membrane decay, γ(g)\gamma^{(g)} for modulation gain, gg solely as the unit index, and c(g)c^{(g)} for semantic conditions, and explicitly define DD as the feature embedding dimension. We corrected the eight-block SpikFormer's ImageNet-1K accuracy to 71.30%, consistent with Table 1, and expanded captions to explain symbols, operations, and information-flow directions. We hope the revised manuscript is now clear enough.

W3 + Q3: Citations.

Thank you for identifying these omissions. We have added SpikingJelly [1] to the implementation description and [2] to the checkpointing/spike-compression baseline description and comparison table. We also checked the other baseline and implementation descriptions and added citations for the supplementary experiments: SEW-ResNet [3], SLTT [4], and NDOT [5].

References

[1] Fang, W., et al. "SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence." Science Advances 9(40): eadi1480, 2023.

[2] Huang, Y., et al. "Towards Lossless Memory-efficient Training of Spiking Neural Networks via Gradient Checkpointing and Spike Compression." ICLR, 2026.

[3] Fang, W., et al. "Deep Residual Learning in Spiking Neural Networks." NeurIPS 34: 21056-21069, 2021.

[4] Meng, Q., et al. "Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks." ICCV, 2023.

[5] Jiang, H., et al. "NDOT: Neuronal Dynamics-based Online Training for Spiking Neural Networks." ICML, 2024.

Q4: Convolutional SNNs.

We greatly appreciate your constructive suggestions, which may significantly broaden the impact of our work. Our answer is Yes! NeuroProp only requires a hierarchical network and is not a dedicated method designed exclusively for Transformer architectures. In convolutional SNNs, AdaIP generates channel-wise gains/biases broadcast over both spatio-temporal direction, applied before residual-branch LIFs in SEW-ResNet and convolutional LIFs in spiking VGG. The added experimental results on CIFAR-100 are as follows:

BackboneSTBP acc. (%)NeuroProp acc. (%)STBP mem. (GiB)NeuroProp mem. (GiB)Reduction
SEW-ResNet1871.91 ± 0.2566.32 ± 0.3529.1413.5553.50%
SEW-ResNet3469.26 ± 1.1168.47 ± 0.2048.8217.7963.56%
Spiking VGG1165.69 ± 0.3166.32 ± 0.178.134.9439.24%

Overall, both residual and sequential convolutional SNNs achieve substantial memory savings. The accuracy approaches that of STBP on SEW‑ResNet34, slightly surpasses STBP on VGG11, and delivers a more notable advantage on SEW‑ResNet18. Supplementary experimental settings and detailed results are provided in Section F and Table 12 of the Appendix in the revised manuscript.

AuthorsOpenReview ↗

To Reviewer pd2X (Continued)

Q5: Stateless MultiSpike in SDTv3.

AdaIP is inspired by adaptive neuronal excitability. In LIF backbones it affects membrane integration and firing, and in stateless SDTv3 we adapt this principle through semantic-dependent feature modulation. Both guide representation formation, with different implementations and dynamical interpretations.Block-specific gains/biases include modulation strength and broadcast across time and space: X~=γ(c)⊙X+β(c).\widetilde X=\gamma(c)\odot X+\beta(c). Modulation precedes the token mixer, attention, and MLP. The first MultiSpike response is U(c)=BN⁡(W∗X~),S(c)=Q(U(c)),U(c)=\operatorname{BN}(W*\widetilde X),\qquad S(c)=Q(U(c)), where W is the first convolution and Q(u)=round⁡(clip⁡(u,0,4))/4.Q(u)=\operatorname{round}(\operatorname{clip}(u,0,4))/4. Changing U(c) can cross fixed boundaries (0.5,1.5,2.5,3.5), altering output levels. Q and its stateless nature remain unchanged; the first two convolutional units are unmodulated. Eqs. (7)-(8) apply to LIF only. We have explicitly described this MultiSpike-specific implementation of AdaIP in Section A.2 of the Appendix in the revised manuscript.

Q6: Pipeline implementation.

Thank you for the suggestion. The table below summarizes the requested implementation details for Figure 8. We have also included it in the revised manuscript (Table 10 of Appendix).

ItemImplementation
Forward/backward schedulingEach NeuroProp unit performs local forward, backward, and update per microbatch, then forwards detached outputs. Different GPUs process different microbatches concurrently.
Stage partitioningLogical-unit depths grouped by GPU: 16 blocks/4 GPUs: [1,2]; [2,2]; [2,2]; [2,3]. 24 blocks/4 GPUs: [1,1,2]; [2,2,3]; [2,2,3]; [2,2,2]. 32 blocks/6 GPUs: [2] on GPU0; [2,2,2] on each remaining GPU.
Microbatch configurationNeuroProp loader batch: 4096; training/evaluation microbatches: 512/1024. Local optimizers update per training microbatch, without accumulation over 4096.
CommunicationNCCL asynchronous point-to-point sends with bounded buffers. Training payloads: detached activations, labels, and metadata; inference payloads: activations and semantic predictions. No cross-unit gradient transmission.
Global-STBP baselineDDP with one full model replica per GPU and synchronized gradients. For 16/24/32 blocks, GPU counts are 4/4/6, per-GPU batches 64/48/36, and global update batches 256/192/216.

Dear Reviewer,

We hope this message finds you well.

We are writing to kindly follow up on our previous responses to your comments. If there are any remaining concerns or points that need clarification, please feel free to let us know. We would be happy to provide additional explanations.

Thank you very much for your time and effort in reviewing our manuscript. We sincerely appreciate your valuable comments and suggestions.

Best regards,

Authors

Reviewer_chgbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NeuroProp, a local training method for spiking Transformers. The network is divided into units with separate classification heads, and gradients are stopped between units while STBP is retained within each unit. The authors also introduce AdaIP, which uses semantic information to adjust neuronal excitability. Experiments show reduced training memory and improved throughput, with some loss in accuracy compared to global STBP.

Critical strengths
  1. The memory savings are clear. NeuroProp reduces peak memory by 48.29% on SpikFormer and 79.41% on SDTv3, making larger models and batch sizes feasible.
  2. The experiments cover several datasets, including ImageNet-1K, and show that local training can retain reasonable accuracy.
Critical weaknesses
  1. Limited novelty. Splitting a network into modules, stopping gradients, and adding local classification heads are already common in local learning methods. NeuroProp mainly applies this idea to spiking Transformers. I am not convinced this represents a substantial methodological advance for SNNs. Reducing training memory is useful, but the paper does not clearly show what new insight it offers about SNN learning.
  2. The contribution of AdaIP is not well established. The proposed gain and bias modulation resembles standard feature-wise affine conditioning. It is unclear why this particular mechanism is necessary or better suited to SNNs. Moreover, the local classification heads also receive semantic information, making it difficult to determine whether the improvement comes from neuronal modulation or simply from the additional conditioning.
Questions for authors
  1. How does AdaIP compare with standard feature-wise modulation (e.g., FiLM) or simply providing semantic information to the local heads?
  2. The teacher-conditioned training protocol uses label information, while inference uses predictions. How much does this mismatch affect performance?
4 comments
AuthorsOpenReview ↗

To Reviewer chgb

We are greatly encouraged that you recognize the value of our memory-efficient local learning method. We also appreciate your feedback highlighting the need to better clarify the contributions and novelty of our work, which will help improve the readability and impact of this manuscript. We address all the concerns you raised point by point below and have revised the manuscript accordingly.

W1: Novelty and Biological Motivation.

Thank you for this constructive comment, which helps improve the readability of our manuscript. We agree that network partitioning, gradient isolation, and local heads are established ideas. Our original presentation did not make our contribution clear enough. We have acknowledged this contribution in the revised version to further clarify the motivation and novelty of our work.

NeuroProp builds upon two complementary biological principles: distributed local adaptation and contextual neuromodulation. It integrates local STBP with AdaIP, which maps task semantics onto neuronal modulation. This enables semantic context to govern how individual modules form representations, allowing gradient-isolated modules to learn toward a shared predictive objective. We hope this work offers a new perspective on SNN training: task-related neuronal regulation can complement local supervision in deep SNNs. Our semantic ablations, FiLM comparison, and modulation analyses provide evidence for this design (Please see the response below).

AuthorsOpenReview ↗

To Reviewer chgb (Continued)

W2 + Q1: Contribution of AdaIP beyond FiLM and head conditioning.

We thank the reviewer for pointing this question, which allows us to further clarify the underlying mechanism of AdaIP.

Comparison with FiLM

We acknowledge that the channel-wise affine modulation of AdaIP may seem similar to FiLM [1]. Below, we compare the two methods from both mechanistic and experimental perspectives to further clarify the contributions of our work.

(1) Mechanistic perspective: In SpikFormer, AdaIP acts before selected LIF neurons, changing their input currents, membrane integration, and spiking responses. Therefore, the semantic condition influences representation formation in Transformer modules. The local learning signal reaches the NAGT controller of AdaIP through Lg⟶ug⟶fg⟶mg⟶ψg,\mathcal L_g\longrightarrow u_g\longrightarrow f_g\longrightarrow m_g\longrightarrow\psi_g, where ugu_g is the local prediction, fgf_g the representation, mgm_g the generated gains and biases, and ψg\psi_g the NAGT controller parameters. Thus, the local task loss supervises AdaIP through its effect on the representation. Indeed, this backpropagation pathway positions AdaIP as a module that leverages semantic information to adaptively optimize representation formation and supply more reliable information to subsequent task heads. The head's direct semantic input is independent of ψg\psi_g and cannot provide a shortcut to the controller that bypasses this path.

(2) Experimental perspective: We replaced AdaIP with FiLM and retrained NeuroProp, keeping the semantic inputs, generator MLP, three pre-LIF modulation sites, local heads, and training budget unchanged. Both methods apply I~=γ(c)⊙I+β(c)\widetilde I=\gamma(c)\odot I+\beta(c), but generate their coefficients differently:

\text{AdaIP:}\quad &\gamma=s_\gamma\odot\sigma(\alpha r_\gamma), \qquad \beta=\alpha s_\beta\odot r_\beta,\\ \text{FiLM:}\quad &\gamma=1+r_\gamma, \qquad \beta=r_\beta. \end{aligned}$$ Here, $r_\gamma,r_\beta$ are the respective MLP outputs, $s_\gamma,s_\beta$ are learnable channel-wise scales, and $\alpha$ controls modulation strength. Compared with FiLM, AdaIP first constrains fluctuations in the generated coefficients, and its learnable scales then adjust modulation strength. This distinction is meaningful for LIF neurons: modifying the input current can alter whether a neuron spikes. A better bounded initial response thus helps preserve useful spiking activity while the model learns task-dependent modulation. In our experiments, the accuracy of AdaIP **69.65 ± 0.64%** outperforms **51.25 ± 0.96%** for the FiLM replacement. | Modulation in NeuroProp | Accuracy: mean ± std (%) | Max (%) | |:--|--:|--:| | FiLM replacement | 51.25 ± 0.96 | 52.78 | | AdaIP | **69.65 ± 0.64** | **70.43** | **Contribution of Semantic Information** To evaluate whether AdaIP equipped with semantic information yields an independent performance gain beyond the local task head, we have conducted a more detailed ablation study. When we zero out the semantic input to these two components separately, the performance drops for both models. This confirms that semantic-conditioned neuronal processing provides extra benefits for training with NeuroProp. | Semantic conditioning | Local head | AdaIP | Accuracy: mean ± std (%) | Max (%) | |:--|:--:|:--:|--:|--:| | AdaIP conditioning only | ✗ | ✓ | 68.76 ± 0.38 | 69.27 | | Head conditioning only | ✓ | ✗ | 69.14 ± 0.16 | 69.28 | | Both pathways (full model) | ✓ | ✓ | **69.65 ± 0.64** | **70.43** | [1] Perez, E., et al. "FiLM: Visual Reasoning with a General Conditioning Layer." *AAAI*, 2018.
AuthorsOpenReview ↗

To Reviewer chgb (Continued)

Q2: Effect of the teacher/prediction conditioning mismatch.

We thank the reviewer for this important question. We acknowledge this difference between the training and inference procedures. To further investigate the effect of this mismatch, we evaluated prediction-conditioned training on the CIFAR-100 dataset to align the source of semantic conditions with inference. We have also added these results and detailed analysis in Section A.2 of the Appendix in the revised version. Overall, prediction conditioning improves the mean accuracy by approximately 1.38% on CIFAR-100. Its gains on ImageNet-1K are 1.40% for the single-step model and 14.36% / 14.81% for the two four-step experiments. These results demonstrate that prediction conditioning is an effective way to further boost training accuracy, especially when the model and dataset are sufficiently large.

DatasetTime stepsTransformer blocksTeacher-conditioned (%)Prediction-conditioned (%)
CIFAR-1004469.65 ± 0.6471.03 ± 0.41
ImageNet-1K1461.3262.72
ImageNet-1K4456.2870.64
ImageNet-1K4856.4971.30

Dear Reviewer,

We hope you are doing well.

We would like to follow up on our previous replies to your comments. If you have any further questions or suggestions regarding our responses, please feel free to let us know. We would be glad to provide more details or clarifications if needed.

We sincerely thank you for your time and effort in reviewing our manuscript. Your valuable comments and suggestions are greatly appreciated.

Best regards,

Authors

Reviewer_JAVHOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper proposes NeuroProp, a local learning framework that partitions a network into consecutive subnetworks and blocks gradient propagation between adjacent subnetworks. To improve coordination among these gradient-isolated subnetworks, the authors further introduce adaptive intrinsic plasticity (AdaIP). Experiments on several spiking Transformer architectures demonstrate that NeuroProp can substantially reduce GPU memory consumption compared with spatio-temporal backpropagation (STBP)-based training methods.

Critical strengths
  1. The substantial GPU memory consumption associated with BPTT/STBP remains a major bottleneck in training large-scale spiking neural networks (SNNs). Local learning offers a promising direction for addressing this challenge, particularly from the perspective of biologically inspired learning algorithms.

  2. Applying local learning to Transformer-based architectures remains relatively underexplored, especially in the context of SNNs. This work extends local learning to spiking Transformers, which is a valuable research direction.

  3. The experimental results demonstrate substantial GPU memory savings. Specifically, NeuroProp reduces GPU memory consumption by 48.29% on SpikFormer and 79.41% on SDTv3 compared with STBP.

  4. In addition to classification datasets, the authors evaluate NeuroProp on other applications, including optical flow estimation and continual learning, which helps demonstrate the broader applicability of the proposed framework.

Critical weaknesses
  1. Although AdaIP is proposed to improve coordination among gradient-isolated subnetworks, its underlying mechanism is not very clear. In particular, it remains unclear how AdaIP facilitates coordination when gradient information cannot propagate across subnetworks.

  2. The specific role and contribution of the causal semantic propagation component are not adequately isolated or explained. More detailed ablation studies would help establish its effectiveness and clarify how it contributes to NeuroProp.

Questions for authors
  1. In Eq. (7), it is unclear which LIF neurons are affected by NAGT. How does this mechanism improve coordination among subnetworks when gradient propagation between them is blocked?

  2. How is the throughput reported in Table 3 measured? Is pipeline parallelism implemented in the experiments?

  3. In Figure 2(a), if the red arrows represent gradient flow, their direction should reflect backward propagation through the network, rather than forward propagation. In Figure 2(b), what does "FF" represent? I suggest to add more explanations in the caption to explain how NeuroProp enhances the coordination between gradient-isolated modules.

  4. Could the authors provide a more detailed explanation of Figure 1 in the caption to improve its clarity and readability?

  5. Have the authors considered combining NeuroProp with existing online learning methods for SNNs? Since NeuroProp primarily reduces memory consumption by decoupling gradient propagation across network depth, combining it with online learning approaches that reduce temporal dependencies during training could potentially provide additional memory savings through spatio-temporal decoupling.


Rebuttal Update:

The authors’ responses adequately addressed my concerns and questions. I find the idea of leveraging global semantic information to enhance coordination among local learning modules without requiring gradient propagation across modules to be a novel and valuable contribution to the local learning literature. The extensive experimental results further demonstrate the effectiveness of the proposed approach. Therefore, I believe this work makes a meaningful contribution to the research community and recommend acceptance.

7 comments
AuthorsOpenReview ↗

To Reviewer JAVH

Thank you for recognizing the value of memory-efficient local learning and your valuable comments. We address the mechanism, ablation studies, measurement protocol, and presentation below point by point.

W1 + Q1: How does AdaIP coordinate gradient-isolated modules, and which LIF neurons are modulated?

Thank you for pointing out this critical issue. We realize that the mechanism of AdaIP was not described clearly enough in our original manuscript. We present these validations as follows.

How does AdaIP coordinate gradient-isolated module?

Theoretically, global STBP trains all Transformer modules using the final task loss, which drives the entire model to learn toward a unified objective. In contrast, NeuroProp severs this cross-module gradient pathway by decoupling connections between Transformer modules. We therefore introduce the AdaIP mechanism to preserve consistent task objectives across modules. Specifically, AdaIP maps semantic information onto neuronal modulation, enabling task context to shape representation learning within each Transformer module. Since this semantic information is globally consistent, AdaIP effectively coordinates these gradient-isolated modules. To guarantee the efficacy of AdaIP, we validate two key claims: (1) the optimization of AdaIP within local learning is confined to a semantic translator; and (2) more reliable semantic information yields larger performance gains for AdaIP. We present these validations below.

(1) The local learning chain illustrates how this modulation is trained. For a single local training module, forward propagation is described as:

m&=M_\psi(\bar c),\qquad f=F_\theta(\bar f_{\mathrm{prev}},m),\\ u&=H_\phi(f,\bar c),\qquad \mathcal L=\ell(u,y). \end{aligned}$$ Here, $M_\psi$ is the NAGT controller, $m$ encodes its gains and biases, $F_\theta$ is the spiking Transformer module, $f$ is its resulting representation, and $H_\phi$ is the local prediction head. Bars denote stop-gradient. By the chain rule, the NAGT controller receives its local learning signal through $$\widetilde{\nabla}_\psi\mathcal L= \left(\frac{\partial m}{\partial\psi}\right)^\top \left(\widetilde{\frac{\partial f}{\partial m}}\right)^\top \left(\frac{\partial u}{\partial f}\right)^\top \nabla_u\ell.$$ The tilde denotes surrogate gradients through spiking activations. **Thus, the local task loss trains AdaIP through its effect on representation formation.** This optimization procedure establishes AdaIP’s role as a semantic translator for representation modulation. (2) We tested the sensitivity of AdaIP to semantic information with a trained model on CIFAR-100. We changed only the semantic conditioning information in AdaIP from zero to the ground-truth one-hot label (more reliable): $$c_{\mathrm{AdaIP}}^{(\rho)}=(1-\rho)c_{\mathrm{base}}+\rho e_y.$$ The test accuracy increased from 70.03% to 70.71%, 71.80%, 74.88%, and 80.99% as $\rho$ increased through 0, 0.25, 0.5, 0.75, and 1; cross-entropy decreased from 2.812 to 1.291 (**Please see Fig. 7 of Appendix in the revised version**). These results verify that **AdaIP is capable of converting semantic information into performance gains, with larger gains attained from more reliable semantic signals**. These results complement **Appendix Section C, Fig. 6**, which shows class-dependent modulation and greater accuracy degradation when strongly modulated channels are masked. **Which LIF neurons are modulated??** In SpikFormer, each block's NAGT controller generates channel-wise gains and biases, applied immediately before three LIF populations: (i) after attention aggregation, (ii) after the attention output projection and BN, and (iii) after the second MLP linear transformation and BN. The modulation is shared across time steps and token positions. It changes neuronal input currents, membrane integration, and spiking responses, thereby affecting representation formation before the local head reads out the features.
AuthorsOpenReview ↗

To Reviewer JAVH (Continued)

W2: Contribution of semantic conditioning.

We appreciate the reviewer’s suggestion to further isolate the contribution of each component. We also believe that the detailed ablation analysis can help to further clarify the novelty of our work. In the revised manuscript, we have added new results and discussions in Section 4.4 and Table 3.

In our NeuroProp method, semantic information is applied to two components: the local task heads and the AdaIP mechanism. We therefore add a more detailed ablation study, where we zero the semantic input of each component separately and zero the semantic inputs of both components at the same time. These results confirm that semantic conditioning contributes to both individual components and the overall performance of NeuroProp.

VariantLocal headAdaIPAccuracy: mean ± std (%)Max (%)
No semantic conditioning✗✗67.34 ± 0.3867.65
AdaIP conditioning only✗✓68.76 ± 0.3869.27
Head conditioning only✓✗69.14 ± 0.1669.28
NeuroProp (full)✓✓69.65 ± 0.6470.43

Q2: Throughput measurement and pipeline parallelism.

Thank you for this important question regarding experimental fairness. To ensure fairness, the training for all methods was conducted on a single A800 GPU. Accordingly, the reported throughput advantage of NeuroProp does not rely on extra GPU resources. Results for parallel training are provided in Section 4.7 of the main text and Section D of the Appendix.

Q3: Clarity of Figure 2.

We appreciate this important suggestion, which will enhance the readability of our paper. We have reorganized Figure 2 and revised its caption to avoid potential confusion and increase readability. Specifically, we have aligned the descriptions of symbols in the figure with the equations in the method section, modulated the presentation of gradient propagation, and optimized the caption to help readers understand the coordination in NeuroProp. We hope that the revised version is now sufficiently clear.

Q4: Caption of Figure 1.

Thanks for your suggestion. We have revised the caption of Figure 1 to clarify the comparisons presented in its panels and their intended interpretation, making the figure more self-contained. The new caption: (a) Peak training memory of STBP and NeuroProp across model sizes and batch sizes. Solid, dashed, and dotted lines indicate feasible boundaries under 80, 40, and 24,GB memory budgets, respectively. (b) Test accuracy versus training runtime using one or four GPUs. NeuroProp achieves total-runtime speedups of 1.26×1.26\times and 1.74×1.74\times in the single-GPU and four-GPU settings, respectively.

AuthorsOpenReview ↗

To Reviewer JAVH (Continued)

Q5: Combining NeuroProp with online learning methods.

We are grateful for this suggestion, which may significantly strengthen the impact of our work. We have carried out additional experiments combining NeuroProp with two online learning strategies, SLTT [1] and NDOT [2]. The related results and analysis are provided in Section 4.9 and Table 7 in the revised manuscript. Overall, the peak-memory reduction increases from 48.29% with NeuroProp alone to 84.95% with NeuroProp+SLTT and 78.21% with NeuroProp+NDOT. NeuroProp+SLTT additionally improves mean accuracy, while NeuroProp+NDOT trades a 1.65% accuracy decrease. These results demonstrate that NeuroProp can be combined with online learning methods to obtain additional accuracy–memory trade-offs.

MethodAccuracy: mean ± std (%)Max (%)Peak memory (GiB)Memory reduction
Global STBP72.21 ± 0.1572.4173.81—
NeuroProp69.65 ± 0.6470.4338.1748.29%
NeuroProp + NDOT68.00 ± 0.6368.7416.0878.21%
NeuroProp + SLTT70.89 ± 0.0771.0111.1184.95%

References

[1] Meng, Q., et al. "Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks." ICCV, 2023.

[2] Jiang, H., et al. "NDOT: Neuronal Dynamics-based Online Training for Spiking Neural Networks." ICML, 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Response

Dear AC and Reviewers,

We sincerely thank AC for handling our manuscript and acknowledge the reviewers for their constructive comments and suggestions, which have helped us significantly improve the clarity and completeness of this manuscript. We also appreciate the reviewers’ positive assessment of NeuroProp as a promising approach to reducing GPU memory consumption in spiking neural networks. Here, we briefly clarify the core contributions of our work and summarize the revisions made in response to the reviewers’ concerns. We would be grateful if the reviewers could review our responses at their convenience and, if necessary, offer further comments to help us strengthen the paper.


Core contributions

Training deep SNNs involves a critical trade-off: the widely used STBP method retains neuronal states across depth and time, resulting in high memory consumption. Isolating gradients across depth offers an opportunity to reduce memory usage. However, training performance may degrade due to the loss of the end-to-end learning signal that coordinates intermediate representations.

In human brain, diffuse neuromodulatory signals regulate neuronal excitability while local synaptic processes operate in distributed circuits. Motivated by this observation, we introduce AdaIP to address the above-mentioned conflict:

  • Local STBP with bounded memory: NeuroProp partitions the entire model into multiple consecutive units, which receive detached upstream information flow and train with local task head. The memory savings come from decoupling the computational graph across units.

  • Task coordination through AdaIP: We construct a semantic information with globally consistance. AdaIP mechanism translates it into neuronal dynamic modulation and guides the formation of task-dependent representations. Our experimental results demonstrate that (1) AdaIP produces distinct modulation patterns across classes and (2) more reliable semantic input to AdaIP improves local prediction accuracy. These findings support AdaIP’s role in promoting coordination across modules.

The main insight for the SNN community is that task-dependent neuronal regulation can complement local synaptic learning to support coordination across gradient-isolated modules. This neuromodulation-inspired strategy enables efficient training and representation optimization for SNNs under resource constraints and also offers valuable insights for other SNN research.


Revisions

In response to the Reviewers’ main concerns, we revised the manuscript and added experiments mainly in three areas.

1. Mechanism and contribution of AdaIP

  • To address JAVH’s and chgb’s questions about coordination and novelty, we clarified the biological motivation for AdaIP, identified the LIF neurons it modulates, and explained how the local loss trains AdaIP through its effect on representations while gradients remain blocked at module boundaries (Section 1, Figures 2, 4 and Section 3.3).
  • We added ablations that isolate the roles of semantic conditioning in the local heads and AdaIP, along with a controlled experiment that varies the reliability of AdaIP’s semantic input (Section 4.4 and Appendix C). We also compared AdaIP with a FiLM replacement at the same modulation sites (Response W2 + Q1 to Reviewer chgb).

2. Additional experiments and extensions

  • Conditioning strategies. To address pd2X’s and chgb’s questions about the difference between training and inference, we compared teacher-conditioned and prediction-conditioned training on CIFAR-100 and ImageNet-1K (Appendix A.2).
  • Online learning and convolutional SNNs. Following JAVH’s and pd2X’s suggestions, we integrated SLTT and NDOT into NeuroProp to evaluate their accuracy–memory trade-offs. We also extended NeuroProp to convolutional architectures (SEW-ResNet and spiking VGG) and compared it with BELL on Net2 under a shared training budget (Section 4.9 and Appendix F).
  • AdaIP under global STBP. We added a preliminary experiment using context derived from upstream features to evaluate AdaIP beyond local learning (Appendix G).

3. Efficiency, reproducibility, and presentation

  • We clarified the measurement protocols and software environments for the single-GPU throughput comparisons and documented the scheduling and implementation of the separate multi-GPU pipeline experiments (Appendix D).
  • We revised Figures 1 and 2 and their captions, standardized notation, corrected numerical inconsistencies, and added missing citations. The main revisions are highlighted in blue, and the code is included in the supplementary materials.

We thank all reviewers for their very valuable comments and suggestions. We hope our responses and revisions address the reviewers’ concerns, and we are happy to make further improvements based on your guidance. Please do not hesitate to contact us if any additional clarification is needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.