acceptodds
Under review as a conference paper at ICLR 2027

Quadratic integrate-and-fire neurons exhibit less fragmented loss landscapes and outperform leaky integrate-and-fire neurons in spike-based gradient descent

Abstract

The ability to train spiking neural networks is essential for modeling biological neural networks as well as for neuromorphic computing. However, for the extensively used leaky integrate-and-fire (LIF) neurons, arbitrarily small parameter changes can induce spike (dis)appearances that disrupt subsequent activity, leading to unstable neural representations and permanently silent neurons during exact spike-based gradient descent. Recent work shows that a class of neuron models, which includes the quadratic integrate-and-fire (QIF) neuron, avoids these discontinuities and enables continuous and even smooth spike-based gradient descent. However, it remains unclear whether these advantages translate into practice. Here, we demonstrate that they do so when learning the popular Spiking Heidelberg Digits dataset, via a controlled comparison between networks of LIF and QIF neurons. Specifically, in a first step, we perform a thorough hyperparameter search to optimize both models, revealing a clear performance advantage of QIF neurons. In a second step, we visualize the loss and gradient landscapes and quantify their roughness. Consistent with their inferior performance, we find that the loss landscapes of LIF neurons, which are discontinuous, are rougher and appear more fragmented, and the related gradients are more erratic. An analysis of the landscapes of single samples indicates that these features arise from changes in the temporal order of spikes, which often cause disruptive spike (dis)appearances. Overall, our results advocate, for gradient descent training where precise spike timing matters, to replace LIF neurons with neuron models exhibiting continuous spiking dynamics, such as QIF neurons.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
spiking neural networks, gradient descent, loss landscapes, leaky integrate-and-fire neurons, quadratic integrate-and-fire neurons, Spiking Heidelberg Digits

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fQPxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the superiority of the Quadratic Integrate-and-Fire (QIF) model over the vanilla Leaky Integrate-and-Fire (LIF) model from the perspective of loss and gradient landscape continuity. The proposed approach is empirically validated on the SHD classification task under the Time-To-First-Spike (TTFS) coding scheme.

Critical strengths
  • The authors effectively analyze the advantages of the QIF model through the lens of gradient continuity, and provide comprehensive, well-designed visualization-based analytical experimental results to support their claims.
Critical weaknesses
  • The justification for the superiority of the QIF model is currently constrained to a highly specific setup: the TTFS coding scheme, relatively simplistic downstream tasks, and basic SNN architectures. To robustly establish its efficacy, further empirical validation is warranted in more complex scenarios. Specifically, the authors should evaluate the QIF model using alternative spike coding schemes, on large-scale datasets (e.g., ImageNet-1k), and with advanced architectures such as Spiking CNNs and Spiking ViTs.

  • Additionally, the paper lacks comparisons with other recently proposed advanced spiking neuron models. Benchmarking the QIF model against these state-of-the-art alternatives is crucial to explicitly demonstrate its relative advantages.

Questions for authors
  • Under the Spatio-Temporal Back-propagation (STBP) framework, which relies on surrogate gradients and discrete time-steps, does the QIF model still maintain its superiority over the vanilla LIF model?

  • Regarding the intra-layer computations of spiking neurons, does the QIF model introduce higher computational overhead compared to the vanilla LIF model during the SNN inference phase?

4 comments

We thank the reviewer for their feedback. In the following, we address the critical weaknesses stated by the reviewer.

Our focus on time-to-first-spike coding is intentional since it relies on the precise timing of spikes. Spike count coding methods rely much more on spike rates than on precise timings. Thus, we expect the theoretical findings of [1] to prove particularly useful when a spike time-based coding scheme is considered. Apart from that, time-to-first-spike coding allows inference as soon as the first output spike appears and does not require counting spikes over a full trial, leading to a reduced latency. Another advantage of spike time-based coding schemes is that individual spikes carry more information and thus, to store the same information, less spikes are required, at least in principle.

In addition, we deem a thorough hyperparameter search necessary for a fair comparison between models. For our existing results presented in the paper, we needed 70 single GPU-days (not including experimental and ineffective runs). Performing a similarly thorough hyperparameter search on more complex structures or datasets would likely require even more resources. While considering such settings would certainly be interesting, performing the necessary simulations is unfortunately presently not possible for us for practical reasons. We are aware of this limitation of our study and note that we mention it in the Discussion section. Finally, we also note that besides a simple performance comparison, a focus of our study is a detailed analysis of the training process. We will emphasize the latter by introducing a new paragraph to the Discussion section.

We chose the LIF neuron for our comparisons since it is the most widely used spiking neuron model. Our approach is to focus on a single setting, carefully optimize both QIF and LIF networks on it and analyze them in detail. Unfortunately, performing a similarly careful optimization and analysis on a broad class of models is practically hard because of the conceptual difficulties of a fair model comparison and because of the high computational cost. We are aware of this limitation of our study and note that we mention it in the Discussion section.


Next, we answer the reviewers questions.

We agree that a comparison between exact spike-based gradient descent and surrogate gradient descent is an important, largely unaddressed problem. Here, the goal is a comparison of QIF and LIF neurons in a relevant practical application. For this, we choose to use exact spike-based gradient descent. Our goal is not a comparison between exact spike-based gradient descent and surrogate gradient descent. It may be that QIF neurons also outperform LIF neurons when both are trained with surrogate gradient descent. To keep its focus, our study should stick to a single gradient-based training method. Performing an equally thorough comparison using a different method or dataset would be a further study, due to the conceptual difficulty of performing a fair comparison (surrogate gradient descent has, for example, as an additional hyperparameter, the shape of the threshold function surrogate) and due to the large computational demand for the thorough hyperparameter tuning that will be necessary for a fair comparison between methods.

We will add a paragraph to the Discussion in which we will emphasize the focus of our study and the importance of a comparison between exact spike-based gradient descent and surrogate gradient descent.

Processing a single spike requires in our experience practically the same amount of time for QIF and LIF neurons. The main contributor to the training and inference time is the number of spikes that have to be simulated. This highly depends on the hyperparameters (e.g. higher input drives I_0 lead to more spikes). Since we did not focus on minimizing the number of spikes, but considered the accuracy to be our only target, we cannot make reliable claims about how many spikes are actually needed to achieve a good accuracy. For example, there might be hyperparameter configurations with a slightly smaller mean test accuracy that require significantly fewer spikes. A comparison of the number of spikes and training times between our best LIF and QIF networks would thus not be fair. Hence, we cannot make any claims on which model is more computationally efficient in practice.

We agree that improving efficiency, especially when applied in a neuromorphic device, will be an important subject for further studies and we will emphasize this point in the revised Discussion.

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review authors' responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

I acknowledge the authors' response to the reviews. After thoroughly evaluating the rebuttal and the perspectives shared by other reviewers, I will maintain my original recommendation.

We thank the reviewer for their response and kindly ask them to consider the changes made in our revision document and specifically the extended Discussion section.

Reviewer_zpMwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the training of spiking neural network with quadratic integrate-and-fire (QIF) neurons. Specifically, this paper analyzes the training progress between QIF and LIF-based SNN. Benefiting from the quadratic nonlinearity, QIF provides smooth spiking dynamics and loss landscape, enabling better performance than LIF-based SNN on SHD dataset.

Critical strengths
  • By analyzing the optimization behavior of SNN, it was confirmed that QIF can avoid the discontinuity of LIF dynamics and achieve smooth spike-based gradient descent, providing a surrogate gradient-free solution for SNN modeling and training.
  • Impressive analysis and systematic experiments to demonstrate the QIF-based SNN has smoother training progress than LIF-based models, leading to a better performance.
Critical weaknesses
  • The core component of the paper, QIF neurons and their spike based gradient descent, has been studied in [1]. This paper experimentally verifies that QIF outperforms LIF neurons in actual SNN training, but the overall method does not exceed the QIF framework.
  • The experiment is limited to time-to-first spike encoding. The effectiveness of other encoding methods, such as spike count coding used in many direct training SNNs, needs to be verified; The network structure and dataset are too singular. Although the current structure performs better than LIF on SHD, it is more convincing to validate on more complex structures (such as ResNet, SpikFormer [2]) and datasets (CIFAR10/100, CIFAR10-DVS, ImageNet)
  • The current performance comparison occurs within the spike-based gradient descent framework, lacking a comparison with surrogate gradient training like spike-temporal backpropagation [3].

[1] Smooth Exact Gradient Descent Learning in Spiking Neural Networks. PRL, 2025.

[2] Spikformer: When spiking neural network meets transformer. ICLR, 2023.

[3] Direct training for spiking neural networks: Faster, larger, better. AAAI, 2019.

Questions for authors

Please refer to "Critical Weaknesses".

Overall, this work provides a detailed analysis of the QIF-based SNN training process based on the landscape of loss and gradient, and further confirms that the nonlinear smoothness introduced by QIF can outperform LIF-based SNN. It indicates that QIF neurons have the potential to replace the surrogate gradient-based backpropagation for SNN training, thereby avoiding the mismatch gradient problem and improving SNN performance. But the current results are still limited to small SNNs and simple datasets. If the author can address the questions I raised in the 'weaknesses' section, I will consider increasing my rating score.

4 comments

We thank the reviewer for their feedback. In the following, we address the critical weaknesses stated by the reviewer.

We believe that our experimental verification that QIF outperform LIF neurons in a widely used benchmark setting is critically important to establish the practical relevance of the results of [1]. Given that currently LIF neurons are very popular in gradient-based training of SNNs, our results of superior performance of a different, similarly simple neuron model, in that no further dynamical variables are introduced, may prove highly influential.

Further, we use, and partially introduce, a range of analysis techniques for SNN training. This includes in particular the visualization of gradient landscapes, the quantification of the local roughness of loss and gradient landscapes and the analysis of how informative the gradient is along the training trajectory. We could imagine that these analysis techniques will also be useful for other researchers. To emphasize this methodological contribution, we will add a paragraph to the Discussion section.

Our focus on time-to-first-spike coding is intentional since it relies on the precise timing of spikes. Spike count coding methods rely much more on spike rates than on precise timings. Thus, we expect the theoretical findings of [1] to prove particularly useful when a spike time-based coding scheme is considered. Apart from that, time-to-first-spike coding allows inference as soon as the first output spike appears and does not require counting spikes over a full trial, leading to a reduced latency. Another advantage of spike time-based coding schemes is that individual spikes carry more information and thus, to store the same information, less spikes are required, at least in principle.

In addition, we deem a thorough hyperparameter search necessary for a fair comparison between models. For our existing results presented in the paper, we needed 70 single GPU-days (not including experimental and ineffective runs). Performing a similarly thorough hyperparameter search on more complex structures or datasets would likely require even more resources. While considering such settings would certainly be interesting, performing the necessary simulations is unfortunately presently not possible for us for practical reasons. We are aware of this limitation of our study and note that we mention it in the Discussion section. Finally, we also note that besides a simple performance comparison, a focus of our study is a detailed analysis of the training process. As mentioned in the answer to the previous comment, we will emphasize the latter by introducing a new paragraph to the Discussion section.

We agree that a comparison between exact spike-based gradient descent and surrogate gradient descent is an important, largely unaddressed problem. Here, the goal is a comparison of QIF and LIF neurons in a relevant practical application. For this, we choose to use exact spike-based gradient descent. Our goal is not a comparison between exact spike-based gradient descent and surrogate gradient descent. It may be that QIF neurons also outperform LIF neurons when both are trained with surrogate gradient descent. To keep its focus, our study should stick to a single gradient-based training method. Performing an equally thorough comparison using a different method or dataset would be a further study, due to the conceptual difficulty of performing a fair comparison (surrogate gradient descent has, for example, as an additional hyperparameter, the shape of the threshold function surrogate) and due to the large computational demand for the thorough hyperparameter tuning that will be necessary for a fair comparison between methods.

We will add a paragraph to the Discussion in which we will emphasize the focus of our study and the importance of a comparison between exact spike-based gradient descent and surrogate gradient descent.

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review authors' responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

Thank you for your response. After reading the author's rebuttal and other reviewers' comments, I maintain the original rating.

We thank the reviewer for their response and kindly ask them to consider the changes made in our revision document and specifically the extended Discussion section.

Reviewer_Vf65OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares QIF and LIF networks trained with exact spike-based gradients on the SHD classification task, finding that QIF networks achieve higher accuracy and larger output spike-time margins. By combining loss landscape visualizations with analyses of actual training trajectories, the authors provide evidence that QIF neurons yield more informative gradients and improved training performance.

Critical strengths
  1. The paper addresses a clear question grounded in existing theory and fills a valuable empirical gap by evaluating the practical benefits of QIF continuity on a standard benchmark.
  2. The experimental design is sound, with a thorough hyperparameter search and evaluation of the main final results across 10 random seeds.
  3. The analysis spans multiple scales, from broad views of training trajectories and loss landscapes to detailed examinations of discontinuities and their underlying mechanisms. Measurements along actual training trajectories connect landscape properties to training behavior.
  4. The publicly available code and detailed appendices support reproducibility.
Critical weaknesses
  1. Althrough the landscape analysis offers a plausible explanation for QIF's advantage, it does not rule out contributions from other dynamical properties, such as its infinite firing threshold and phase transfer curve, through mechanisms beyond continuity.
  2. The main experiments focus on a single dataset, SHD. Although QIF exhibits more favorable loss landscapes than LIF in this setting, it remains unclear whether the proposed approach offers advantages over baselines such as LIF networks trained with surrogate gradients.
  3. The paper does not provide a direct comparison of training time and computational efficiency between QIF and LIF networks. These are important practical considerations in neuromorphic computing.
  4. The noise-robustness claims lack direct experimental support. The authors use distinctness as a proxy for robustness, without conducting explicit noise-injection experiments.
  5. Several numerical differences and instances of symbol reuse require clarification. For example, Appendix C reports a QIF test accuracy of 90.8% with an optimal τ\tau of 0.04s whereas Table 1 reports 90.1% with τ=20 ms\tau=20\ ms. In addition, α\alpha and β\beta denote both the two-dimensional projection coordinates and the learning rate and logit scale, respectively, which may cause confusion.
Questions for authors
  1. How does this work advance beyond Wan et al.'s "From LIF to QIF" ?
  2. Could you test another small dataset and compare performance and loss landscapes with surrogate-trained LIF networks?
  3. Could you directly test noise robustness through noise injection?
  4. Could the authors quantify and compare the computational efficiency of QIF and LIF networks, including training time and simulation costs?
  5. Please check the hyperparameter configurations for consistency.
3 comments

We thank the reviewer for their feedback and answer their questions:

Our work differs from Wan et al.'s in several aspects. First, we perform a much more focused comparison of QIF and LIF neurons in that we train both neuron models with exact spike-based gradient descent and in that we keep the methodology as consistent between the models as possible. In Wan et al., the methodology differs substantially between the models. For example, networks of QIF neurons are trained with exact spike-based gradient descent using event-based simulations, networks of LIF neurons are either trained with surrogate gradient descent using timestep-based simulations or obtained via an ANN-to-SNN conversion. Second, we consider a standard benchmark task in neuromorphic computing, while Wan et al. consider a rather novel application for SNNs, namely scientific machine learning. Third, we perform a careful analysis of loss and gradient landscapes and of the training process, while Wan et al. focus on model performance without such analysis. Fourth, we perform a thorough, documented hyperparameter optimization. The details of hyperparameter optimization used by Wan et al. are unclear.

In summary, Wan et al. and we address related, but different questions. We address the question how QIF neurons compare to LIF neurons in the same setting (exact spike-based gradient descent). Wan et al. address the question how the combination of QIF neurons with exact spike-based gradient descent compares to the combination of LIF neurons with surrogate gradient descent or with ANN-to-SNN conversion. Hence, in our approach there are fewer factors that may affect the performance difference aside from the targeted discontinuity-avoiding QIF dynamics. In Wan et al., different factors like the use of timestep-based simulations may play a significant role in the observed performance difference. We will add a short summary of these differences to our existing mentioning of Wan et al. in our Related work section (Section 1.2).

We agree that a comparison between exact spike-based gradient descent and surrogate gradient descent is an important, largely unaddressed problem. Here, the goal is a comparison of QIF and LIF neurons in a relevant practical application. For this, we choose to use exact spike-based gradient descent. Our goal is not a comparison between exact spike-based gradient descent and surrogate gradient descent. It may be that QIF neurons also outperform LIF neurons when both are trained with surrogate gradient descent. To keep its focus, our study should stick to a single gradient-based training method. Performing an equally thorough comparison using a different method or dataset would be a further study, due to the conceptual difficulty of performing a fair comparison (surrogate gradient descent has, for example, as an additional hyperparameter, the shape of the threshold function surrogate) and due to the large computational demand for the thorough hyperparameter tuning (we needed 70 single GPU-days for our existing results) that will be necessary for a fair comparison between methods.

We will add a paragraph to the Discussion in which we will emphasize the focus of our study and the importance of a comparison between exact spike-based gradient descent and surrogate gradient descent.

We will perform this test and provide results in a follow-up answer.

Processing a single spike requires in our experience practically the same amount of time for QIF and LIF. The main contributor to the training and inference time is the number of spikes that have to be simulated. This highly depends on the hyperparameters (e.g. higher input drives I_0 lead to more spikes). Since we did not focus on minimizing the number of spikes, but considered the accuracy to be our only target, we cannot make reliable claims about how many spikes are actually needed to achieve a good accuracy. For example, there might be hyperparameter configurations with a slightly smaller mean test accuracy that require significantly fewer spikes. A comparison of the number of spikes and training times between our best LIF and QIF networks would thus not be fair. Hence, we cannot make any claims on which model is more computationally efficient in practice.

We thank the reviewer for pointing out this inconsistency. The correct hyperparameter configuration is the one listed in Table 1. We will correct the text in the revision: "The highest mean test accuracy is achieved by two different configurations, both with α = 3 * 10^-5 and β = 300. The first configuration, with τ = 0.04s and I_0 = 0.255, achieves 90.8(3)%, while the second configuration, with τ = 0.02s and I_0 = 0.252, achieves 90.8(11)% accuracy. For our analyses we choose the second one." After training the chosen hyperparameter configuration for 500 instead of 200 epochs (and with 10 different seeds), the final accuracy is 90.1(9)% (the value in Table 1). We will make this clearer in the text.

Thanks for your response. However, the experiments are limited to a single dataset and a simple network, and it remains unclear whether the advantages of QIF generalize to other tasks and training methods. I will therefore maintain my current rating for now.

We thank the reviewer for their response. We agree that further research is needed to test how the advantages of QIF networks generalize to other datasets and that it is interesting question how they perform in other training methods. However, the goal of this study is not to provide a reliable benchmark for QIF networks for general datasets but, for a simple example task, to analyze their properties, training behavior, and the effect of their continuous dynamics compared to the discontinuous LIF networks, specifically by inspecting the loss and gradient landscapes.

We added a noise injection analysis to the revision (Section L). What we see is that, as expected from their distinctness values, the best LIF model is more sensitive than the asynchronous LIF model to spike time noise, additive synaptic noise and synaptic shot noise. For spike loss and multiplicative synaptic noise, their is no significant difference in sensitivity. Thus, the choice of model depends on the specific hardware implementation and its noise properties. Synaptic noise is hard to compare between LIF and QIF neurons because of their different weight scales. Under spike time noise and random spike removals the QIF model outperforms both LIF models at most noise strengths. For large spike time noise (sigma > 50ms) and large removal probabilities (p > 50%) the models are roughly on par and converge towards chance level for even stronger noise.

We kindly ask the reviewer to consider the changes made in our revision document, specifically the extended Discussion section and section L.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.