- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper revisits an important issue in surrogate gradient (SG) training for spiking neural networks (SNNs): the surrogate gradient applied at the spike nonlinearity differs from the Effective Surrogate Gradient (ESG) actually received by the network after temporal backpropagation through LIF neuron dynamics. Through theoretical analysis, the authors reveal that conventional ESGs may exhibit discontinuities and asymmetry around the firing threshold. Based on these findings, they propose , which redesigns temporal gradient propagation to improve credit assignment. Experiments on multiple neuromorphic classification tasks demonstrate the effectiveness of the proposed method.
Critical strengths
- A novel and theoretically valuable perspective. This paper shifts the focus of surrogate gradient research from local SGs to the ESGs resulting from temporal backpropagation. The proposed gradient propagation mechanism is motivated by this analysis, establishing a clear connection between the theoretical findings and the method design.
- Experimental results support the effectiveness of the method. The authors systematically analyze the ESGs produced by different surrogate gradients and backward propagation rules, and validate the proposed method across multiple neuromorphic datasets and network configurations, with particularly notable improvements in deeper networks.
Critical weaknesses
The proposed involves multiple design components, including gradient selection around threshold crossing and adjustments to gradient propagation during silent steps. Additional ablation studies on these components could help clarify their individual contributions.
Questions for authors
The analysis in this paper primarily focuses on LIF neurons with hard detached reset. Could the authors briefly discuss whether the proposed ESG analysis and method could be extended to other reset mechanisms (e.g., soft reset) or other neuron models?
3 comments
Thanks for the response. In the additional experiments in the appendix, the authors discuss the results under soft reset. We can see that the proposed method does not significantly improve the gradient issues in this setting, possibly because the gradient discontinuity is already small. In addition, the authors committed to providing complete ablations, and I hope these can be added to the paper soon. This does not change my view that this is a valuable work, and I maintain my original rating.
Follow up reviewer hfpi (extended & ablation results)
Thank you for your reply and for maintaining your rating (our new public comment gives an overview of the revised and extended results). We agree with your reading of the soft reset. In simulation the conventional effective surrogate shows little distortion there when neurons fire often (Appendix D). The empirical results across all four reset variants show the same: cdx-t⋆ is ahead of or level with the conventional adjoint, and pooled over datasets its advantage is smallest where the reset leaves little to correct (Figure 19 and Table 22 in Appendix F.2). As promised, the main matter now summarises these reset results (Sections 3.2 and 3.3).
You noted the connection between our theoretical findings and the method design. The analysis traces the defect of the conventional adjoint to the gradient that time steps without a spike pass back to earlier steps. States just below threshold collect this gradient. Our simulations show this as a jump at threshold with more gradient below it (Sections 2.4 and 3.1). This points to three components of an adjoint. cdx-t⋆ adjusts all three:
- What a time step without a spike passes back in time.
- How much of its own gradient it sends to earlier layers.
- At which time step the surrogate of a spike is read.
The first two are the adjustments at time steps without a spike you named, and the third is the gradient selection around the threshold crossing.
The complete ablations of Appendix E cover 13 adjoint designs that change these components individually and in combination, across the experimental grid of Section 3.2 with ten seeds each. Figure 16 in Appendix E illustrates the structure of these designs. Of the adjustments at time steps without a spike, passing back only the gradient of the spike carries the gain in out-of-sample accuracy. Alone it already raises accuracy on every dataset. Sending part of a step's own gradient to earlier layers remains necessary for most runs to learn. The gradient selection around the threshold crossing changes accuracy little. cdx-t⋆ is consistently strong among the ablated designs, with none ahead of it on more than one dataset (Figure 17 and Table 15 in Appendix E). Because its ablation maps all three components alone and in combination, cdx-t⋆ serves as a reference point for the joint design of applied surrogate and adjoint. As you hoped, the paper now includes these ablations and builds its discussion on them (Sections 3.2 and 3.3).
In short, the connection you noted between our theoretical findings and the method design becomes measurable. Overall, the effective-surrogate analysis is the core contribution of our work. It diagnoses the conventional adjoint, separates the components an adjoint can change and indicates which change pays off. cdx-t⋆ adjusts all three, and its ablation measures what each component contributes.
We hope these additions to the manuscript answer your remaining points. Thank you again for your constructive engagement.
Response reviewer hfpi
Thank you for your careful assessment and your constructive suggestions. We are glad that you see the effective-surrogate perspective as novel and theoretically valuable, and we address your two points below.
On ablations. We agree that ablations make the contribution clearer. We intend to ablate the components that define cdx-t⋆ across the full experimental grid to isolate the source of its gains. We aim to report each variant next to the full method.
On resets. Our main study considers spiking LIF neurons with a hard reset. For resets the discontinuity has two sources: detaching the reset in the backward pass and the discontinuous reset itself in the forward pass. A hard reset sets the membrane to a fixed value after a spike, while a soft reset subtracts a fixed amount from it. In the backward pass a detached hard reset cuts the credit at each spike, while a detached soft reset carries it across spikes. A new Appendix D therefore covers hard and soft resets, each detached and attached. The effective-surrogate analysis there reveals that the discontinuity persists across these settings and that cdx-t⋆ keeps the credit more balanced than the conventional adjoint, most clearly under the hard reset. The effective surrogate thus shows how each choice of reset shapes the credit a neuron receives. We intend to summarise these reset results in the main matter of the revision.
On other neuron models. Our work studies the effective surrogate gradient that arises in spiking LIF neurons. Within this family our formulation nests binary neurons at β = 0 and approaches integrate-and-fire neurons as β tends to 1. In these neurons the applied surrogate covers only the step from the spike St to the membrane Ut. The effective surrogate instead captures how the full adjoint transforms this gradient. Under the theoretical framework of §2.3 this analysis extends to any unit with a recurrent hidden state ht. A surrogate is needed only for the non-differentiable spike. For smooth units such as the GRU the analysis yields the effective gradient through the full adjoint. In both cases the effective (surrogate) gradient becomes
Geff(h)=E[λtsht=h].
Here λts is the gradient that the unit passes upstream through its input. Mapping it requires a good exploration of the distribution of ht. The analysis also carries over to multi-dimensional states such as the two states of an oscillatory spiking neuron or the cell and hidden state of an LSTM. However, for such multi-dimensional adjoints the exploration becomes harder through the curse of dimensionality. The effective (surrogate) gradient thus extends to other neuron models as a tool for analysis. By contrast, our proposed cdx-t⋆ uses this tool to design an adjoint specifically for LIF neurons. We intend to describe this wider use of the analysis in the revision.
We hope this answers your questions. During the remaining rebuttal period we hope to share further insight into what drives the gains of cdx-t⋆.