acceptodds
Under review as a conference paper at ICLR 2027

Towards Characterizing Spiking Neural Networks: An Empirical Study of Loss Landscapes and Diffusion Dynamics

Abstract

Understanding the interplay between loss landscape geometry and weight dynamics is crucial for elucidating the generalization mechanisms of Spiking Neural Networks (SNNs). However, how distinct plasticity rules shape these geometric and dynamical properties remains underexplored. We present a unified framework integrating time-averaged mean squared displacement (TAMSD) with surrogate-based curvature measurements to systematically characterize three plasticity paradigms: global backpropagation (GP), local Hebbian learning (LP), and hybrid plasticity (HP). While all paradigms exhibit a transition from early super-diffusion to late sub-diffusion, they display fundamentally distinct late-stage signatures. GP sustains irregular exploration within sharp minima; LP exhibits nearly lag-independent displacement with rough trajectories and intermediate performance; crucially, HP achieves an intermediate effective displacement-scaling exponent, balancing exploration and stabilization. This dynamical behavior correlates with HP converging to significantly flatter minima, yielding superior test accuracy and robustness to input corruptions compared to GP and LP. Our results provide complementary geometric and dynamical evidence for an implicit regularization effect inherent to hybrid plasticity, where local updates modulate the optimization trajectory without explicitly penalizing curvature. These findings underscore the value of jointly analyzing approximate geometry and empirical training trajectories to uncover the mechanistic principles governing learning in SNNs.

Primary area:
learning theory
Keywords:
SNN, Training Dynamics, Diffusion Dynamics, Loss Landscape

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cXQeOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the effects of different learning rules on the training dynamics, loss landscape, and generalization performance of spiking neural networks (SNNs). The authors compare three learning approaches, GP, LP, and HP, and analyze their optimization behaviors using time-averaged mean squared displacement (TAMSD), diffusion exponents, loss landscape visualization, and Hessian curvature analysis. The experimental results show that all three methods exhibit a transition from superdiffusive behavior in the early stages of training to subdiffusive behavior in later stages. In the main experiments, HP exhibits a late-stage diffusion exponent between those of GP and LP, while also achieving lower Hessian curvature and better noise robustness. The authors suggest that the Hebbian updates in HP may act as an implicit regularizer, encouraging convergence toward flatter regions of the loss landscape and thereby improving generalization performance.

Critical strengths
  1. The research question addressed in this paper is meaningful. Rather than focusing solely on classification accuracy, the authors further investigate weight dynamics and loss landscapes under different learning rules, providing a different perspective on understanding the optimization process of SNNs.
  2. The paper employs multiple analytical techniques, including TAMSD, diffusion exponents, PCA visualization, and Hessian curvature analysis. These methods examine the training process from different perspectives, making the experimental analysis relatively comprehensive.
  3. Beyond comparing classification performance, the authors also evaluate noise robustness and provide additional experiments in the appendix involving different network architectures, EventProp, and weight norm constraints. These supplementary analyses make the overall empirical investigation more complete.
Critical weaknesses
  1. The current experiments primarily focus on Fashion-MNIST and fully connected SNNs. Including additional datasets or network architectures would help strengthen the generality of the findings and make the conclusions more convincing.
  2. Although the paper demonstrates connections among diffusion behavior, loss landscape properties, and generalization performance, the relationships among these phenomena could be investigated more thoroughly to provide stronger support for the authors' proposed interpretation.
  3. Some experimental results are based on single runs. Conducting experiments with multiple random seeds and reporting the variability of the results would help establish the stability and reliability of the conclusions.
  4. Some results in the appendix differ from those reported in the main experiments. For example, the diffusion characteristics of HP exhibit certain variations across different network architectures. A more detailed discussion of these differences and their potential causes would provide a more complete understanding of the findings and help clarify the extent to which the main conclusions generalize across experimental settings.
Questions for authors

see weakness

Reviewer_SU8HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates how global, local, and hybrid plasticity (GP, LP, and HP) shape learning dynamics in spiking neural networks, and whether these dynamics help explain differences in generalization and robustness. It studies a 4 layer, FC-SNN on Fashion-MNIST, combining time-averaged mean-squared displacement (TAMSD) of synaptic weights with loss-landscape visualizations and surrogate-based curvature measurements. Its main empirical contribution is the observation of early superdiffusive and late subdiffusive weight dynamics across the three learning paradigms, extending a phenomenon previously reported in conventional ANNs to the tested SNN settings. The paradigms exhibit different late-stage dynamics: GP retains higher diffusion exponents, LP shows pronounced stagnation in hidden-layer weights, and HP exhibits intermediate exponents in the main architecture. The authors associate HP’s dynamics and lower measured curvature with its favorable test loss and noise robustness, and propose a theoretical explanation separating movement within basins from transitions between basins. Validation includes classification and input-perturbation experiments, a shallower network, fixed-weight-norm experiments, and an early-training comparison with exact-gradient EventProp. These checks broaden the empirical evidence but also reveal architecture- and constraint-dependent behavior. The results support a comparative characterization of the tested learning rules; universality and the proposed causal connection between diffusion, flatness, and performance remain insufficiently established.

Critical strengths

The paper’s strongest contribution is its empirical characterization of synaptic-weight dynamics across global, local, and hybrid plasticity in SNNs. The reported early-superdiffusive to late-subdiffusive transition extends an observation previously established in conventional ANNs to the tested SNN settings, while the comparison reveals differences between learning rules that classification accuracy alone would not capture. This provides potentially useful knowledge about how these rules shape optimization.
The supplementary experiments broaden the empirical characterization: the shallower network also exhibits the reported early-to-late diffusion transition, while the modified fixed-weight-norm setting shows that subdiffusive scaling can coexist with increasing measured curvature. These results help identify which observations persist across settings and which proposed relationships require qualification.

Critical weaknesses

Two unresolved issues undermine the proposed explanation connecting diffusion, loss geometry, and performance. First, Appendix B attributes non-unit diffusion exponents to inter-basin dynamics without adequately excluding single-basin alternatives. Directed relaxation toward the minimum of a fixed quadratic basin can produce finite-lag exponents above one, while subsequent confinement produces a linear-to-plateau crossover with fitted exponents below one. Thus, the observed transition alone does not establish basin hopping; the theoretical baseline must be evaluated using the same displacement measure and fitting procedure as the empirical analysis. Second, the comparative-flatness conclusion lacks a common basis for comparing curvature across paradigms. Appendix A uses different objectives for GP and LP, without establishing their comparability, and Eq. (9) adds branch-specific second derivatives without identifying a clear underlying objective and parameterization. Because both HP branches influence the shared neuronal activity and predictions, separate update rules do not establish the claimed decomposition. These issues are consequential because they leave the paper’s central mechanistic explanation of HP’s performance unsupported, even if the descriptive diffusion observations remain valid.

Questions for authors
  1. Appendix B argues that positive-curvature intra-basin dynamics yield normal diffusion and therefore attributes non-unit diffusion exponents to inter-basin dynamics. Could the authors clarify how this inference excludes two counterexamples within a single fixed quadratic basin?
  • Late-stage exponents below one: noise-driven fluctuations near the minimum produce a crossover from short-lag linear displacement growth to a plateau. Fitting across this crossover can yield an exponent below one without any basin transitions.
  • Early-stage exponents above one: starting away from the minimum produces directed relaxation toward it. Over a finite lag range where this drift dominates noise and the velocity changes little, squared displacement grows approximately quadratically, yielding an exponent above one despite remaining within the same basin. Together, these effects can produce an apparent early-superdiffusive to late-subdiffusive transition without basin hopping. How does the analysis distinguish this possibility from the proposed inter-basin explanation? Could the authors evaluate a single-basin control using the same TAMSD estimator, training windows, and lag-fitting ranges as the empirical analysis, and clarify what additional evidence supports inter-basin transitions?
  1. Both HP weight branches contribute to the same neuronal activity and predictions: changing GP weights can alter the activity driving LP updates, while changing LP weights can alter the error driving GP updates. Thus, separate update rules do not imply independent dynamics. Given this, what objective and parameterization does Eq. (9) represent, and what justifies its additive Hessian decomposition despite this coupling? Appendix A uses different objectives for GP and LP, how are their scales and parameterizations made comparable so that these Hessians—and the HP construction—support conclusions about classification-loss flatness?

  2. Do the early-to-late transition and the main experiment’s late exponent ordering persist across independent runs and reasonable choices of fitting range? Please report variability and fitting-range sensitivity, particularly where the local slope changes substantially with lag. Figure 3(b–c) shows exactly the sharp changes in the local scaling exponent near a lag of 600, corresponding to one training epoch. What causes this feature? Does it persist across random seeds and choices of analysis window and numerical differentiation, and could it reflect epoch-related correlations or processing of the recorded trajectories? Please clarify its effect on the fitted diffusion exponents and their interpretation.

Minors:

  • Training time vs. waiting time Fig. 4(b)
  • Figure 5 should be a table, and its caption is not matching with the content.
  • Figure 10, 11 have Layer 1 (W1) lines missing.
  • the list is not exhaustive...
1 comment

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review their responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

Reviewer_geDEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares SNN training dynamics under global error-driven learning (GP), local Hebbian learning (LP), and hybrid learning (HP). It analyzes time-averaged mean squared displacement (TAMSD) and surrogate-based curvature, relating them to accuracy and robustness to input noise. Experiments on fully connected SNNs trained on Fashion-MNIST show a transition from early superdiffusion to late subdiffusion. The authors attribute HP’s robustness to implicit regularization induced by local updates.

Critical strengths

Joint analysis of training trajectories and curvature. Beyond final accuracy, the paper tracks differences in weight dynamics and curvature across learning rules. These observations provide useful insight into the behavior of GP, LP, and HP.

Critical weaknesses
  1. The curvature definitions and diffusion interpretations underlying the proposed mechanism are insufficiently justified. Appendix A uses cross-entropy curvature for GP, Hebbian-loss curvature for LP, and their sum for HP. However, linear superposition of weights does not by itself imply the Hessian decomposition in Eq. (9). The authors should clarify which objective or update dynamics this operator represents and justify whether comparing the largest eigenvalues of these different operators establishes that HP reaches flatter solutions under the classification loss. Appendix B argues that intra-basin dynamics cannot explain the observed anomalous diffusion. However, mean squared displacement can saturate within a single positive-curvature basin, yielding an estimated exponent (\alpha<1) over a finite lag range. Thus, fitting the full lag range in Eq. (6) does not establish a connection between subdiffusion and inter-basin transitions. Saturation must be distinguished from power-law scaling. Moreover, the one-step TAMSD used to estimate noise strength below Eq. (18) includes both mean drift and stochastic fluctuations. It therefore does not directly establish that Hebbian updates regulate noise covariance. The results show associations among curvature, weight dynamics, and robustness, but do not adequately establish the claimed causal mechanism.

  2. The contribution beyond existing observations and the generality of the conclusions are insufficiently validated. The transition from early superdiffusion to late subdiffusion was already reported in DNNs by Chen et al. (2022), which the manuscript cites. The additional contribution therefore lies in explaining how SNN learning rules alter training dynamics and generalization beyond reproducing this observation. The experiments are limited to Fashion-MNIST and shallow models. The main dynamics figures present representative runs, without reporting variability across runs or confidence intervals for the diffusion exponents. The key relationships should be validated on deeper models and additional datasets, and the stability of the estimates should be assessed.

Minor Comment

  • The title registered in the submission system differs from the title in the manuscript file.
Questions for authors
  1. What operator does Eq. (9) define? Is it the Hessian of an effective HP objective, the Jacobian of the update vector field, or a separate curvature measure? Please also clarify the relationship between the LP update and the gradient of Eq. (8).

  2. Was noise covariance directly estimated? If so, please describe the estimator and how mean drift was removed. If only one-step TAMSD was used, how was covariance modulation distinguished from changes in mean drift?

  3. Which GP configuration and results underpin the EventProp comparison? Appendix C.2 claims matched settings but reports a learning rate of (0.01), whereas Table 1 specifies (0.001) for GP. Please also clarify how the reference value (\alpha_{\mathrm{GP}}\approx1.05) relates to the early GP exponent of (1.2892) in Figure 5.

1 comment

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review their responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.