Applied but Not Received: Designing Effective Surrogate Gradients for Spiking Neural Networks

Surrogate gradients are the standard tool for training spiking neural networks (SNNs), yet their true effect on the learning signal remains poorly understood. We show that the surrogate gradient (SG) applied at the spike nonlinearity is not the effective surrogate gradient (ESG) the network receives once it is propagated through the complete adjoint of the leaky integrate-and-fire (LIF) dynamics under backpropagation-through-time. Characterizing the ESG analytically, our analysis shows how common applied surrogate gradient definitions engineered to be smooth and symmetric yield effective surrogate gradients that are neither: discontinuous at threshold and asymmetric around it. These findings are consequential: the discontinuity inflates gradient variance, and the asymmetry biases learning toward non-spiking states, revealing an unintended mismatch between the forward and backward models. We turn this analysis of ESG dynamics into a design principle: rather than reshaping the applied surrogate alone, we design for the effective surrogate directly—shaping the applied surrogate and the LIF adjoint together—so that the effective gradient the network receives is continuous at threshold and restores intended credit assignment between silent and spiking regimes. The resulting design enables more stable and performant training of deep SNNs, demonstrating design of the effective surrogate gradient as a principled route to improved optimization.