acceptodds
Under review as a conference paper at ICLR 2027

FGKDSNN: Learning from Firing Activity via Firing-Guided Knowledge Distillation for Spiking Neural Networks

Abstract

Spiking neural networks (SNNs) provide an energy-efficient alternative to artificial neural networks (ANNs) through sparse spike-based computation, yet a substantial performance gap remains under ultra-low time-steps. Knowledge distillation (KD) from pretrained ANNs has emerged as an effective approach to improve SNN training. Nevertheless, existing SNN distillation methods uniformly transfer feature- and logits-level knowledge, without considering the firing activity of the student SNN. In this work, we reveal that student firing activity is closely related to knowledge transfer at both feature- and logits-levels: the firing activity is highly concentrated across feature channels, while higher sample-wise firing ratios are associated with larger differences between teacher and student non-target distributions. Based on the observations above, we propose firing-guided knowledge distillation for SNNs (FGKDSNN), consisting of firing-guided feature-based knowledge distillation (FGFKD) and firing-guided logits-based knowledge distillation (FGLKD). At the feature level, FGFKD leverages channel-wise firing activity to guide feature masking and restoration, strengthening teacher feature transfer to more active student channels. At the logits level, FGLKD reweights non-target distillation according to sample-wise firing activity. Extensive experiments across CNN- and Transformer-based SNNs demonstrate that FGKDSNN consistently outperforms existing SNN distillation methods (e.g., +1.04% on ImageNet), achieving state-of-the-art performance on static and neuromorphic image classification, object detection, and semantic segmentation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Knowledge Distillation, Spiking Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BqT3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FGKDSNN, a firing-guided knowledge distillation framework for SNNs. It introduces two components, FGFKD, which adjusts feature masking probabilities based on channel-wise firing rates, and FGLKD, which reweights non-target knowledge distillation using sample-wise firing rates. Experiments demonstrate improvements across image classification, object detection, and semantic segmentation.

Critical strengths

The proposed method is simple and can be integrated into existing SNN architectures without modifying their main network structures. The authors conduct extensive experiments, covering multiple datasets and downstream tasks.

Critical weaknesses

1.The authors assign higher masking probabilities to high-firing channels without sufficiently explaining why this strategy is preferable to masking low-firing channels. A comparison with low-firing-prioritized masking would strengthen the motivation. 2.The authors directly use the firing rate as the non-target KL divergence weight, but the observed correlation does not justify this relationship. Comparisons with alternative weighting functions or further theoretical analysis are needed. 3.FGFKD and FGLKD appear to be two largely independent mechanisms connected only through firing activity. The paper does not sufficiently explain how these two components interact or complement each other. 4.The paper lacks an analysis of the final firing rates, spike counts, and energy consumption. Since energy efficiency is a key motivation for SNNs, it is important to investigate whether the proposed method affects firing activity as you are using the firing rate to guide the distillation. 5.The experimental comparisons are incomplete. Recent methods such as Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment (ICML 2025) are not included, making it difficult to assess the competitiveness of the proposed approach. 6.Most experiments are conducted at a fixed number of time steps, typically 4. Evaluating performance across different time steps would provide stronger evidence of the method's robustness and generalizability. 7.Although the paper reports improvements on CIFAR-10 and CIFAR-100, CIFAR-10 is a relatively saturated benchmark. I don't recommend to do ablation only on Cifar data.

Questions for authors

See weakness

7 comments
AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear Reviewer BqT3,

As the discussion deadline is approaching, we kindly remind that we have answered your remaining concerns carefully with additional experiments and clarification.

Could you please check our response and consider improving the rating if we successfully addressed you concerns?

Any feedback will be greatly appreciated and we are glad to provide any additional clarifications that you may need to bring a fruitful discussion.

Warm regards, The Authors

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review their responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

Thanks for your reply. I reran the idea compared with temporal-wise method, showing FGKD cannot surpass theirs. I am skeptical of the authenticity of the offered results. Could the author provide the training log? Thanks.

Reviewer_jjJpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes firing-guided knowledge distillation for SNNs (FGKDSNN), which uses firing rates to adjust the strength of feature masks and the loss weights for logit distillation in SNN knowledge distillation. This paper presents experiments on static and neuromorphic datasets that demonstrate the effectiveness of its method.

Critical strengths

This paper considers the firing rate of SNNs and uses it to adjust the strength of the feature mask and the distillation loss weights in knowledge distillation.

Critical weaknesses
  1. This paper focuses on the impact of SNNs' firing rates on their feature distributions and the distillation of non-target class outputs. However, Figure 4 alone is insufficient to demonstrate this focus's validity and generality.
  2. Even if the motivation holds, the proposed method uses a stronger mask on regions with high firing rates. This can easily destroy the key semantics of the SNN spike map, leading to degradation. The authors set the feature distillation weights to an extremely low value of 2×10−32 \times 10^{-3}, which further illustrates that this problem exists. Additionally, applying larger distillation weights to samples with high spike firiing rates may cause the SNN to overfit to difficult-to-learn samples, or it may suggest that the SNN and the ANN teacher differ significantly.
  3. The core of the proposed method relies on feature masking and reconstruction distillation based on the BKDSNN workflow. This indicates that the paper offers only incremental improvements rather than significant innovation.
  4. The experimental results show that the proposed method does not significantly improve performance compared to other SNN distillation methods. In fact, it only increases accuracy on CIFAR-10 by 0.28%. In particular, the method described in this paper achieved only 83.2% accuracy on CIFAR10-DVS, which is lower than that of many SOTA SNN methods. The authors should include comparison results with the latest SNN work.
  5. The proposed method involves a large number of hyperparameters, significantly increasing its complexity. Therefore, the authors should analyze the impact of these hyperparameters and provide guidance on how to configure them.
  6. The proposed method is based on adaptive adjustments to mask intensity and logit distillation weights using spike firing rate. Further evidence is needed to support this method, such as visualizations of the spike features and their masks, and of the samples and their weights.
Questions for authors

The author is requested to explain in detail the innovative aspects of their method compared to existing KD methods and SNN-oriented KD methods.

2 comments
AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear Reviewer jjJp,

As the discussion deadline is approaching, we kindly remind that we have answered your remaining concerns carefully with additional experiments and clarification.

Could you please check our response and consider improving the rating if we successfully addressed you concerns?

Any feedback will be greatly appreciated and we are glad to provide any additional clarifications that you may need to bring a fruitful discussion.

Warm regards, The Authors

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review their responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

Reviewer_Ye8uOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Firing-Guided Knowledge Distillation method (FGKDSNN) for Spiking Neural Networks (SNNs), aiming to improve knowledge transfer from Artificial Neural Networks (ANNs) to SNNs under low time-step conditions. The method includes two distillation strategies: (1) Firing-Guided Feature-based Knowledge Distillation (FGFKD): Dynamically adjusts feature masking probabilities based on the firing rates of different channels in the student SNN, and achieves knowledge transfer through feature recovery and alignment with teacher features. (2) Firing-Guided Logits-based Knowledge Distillation (FGLKD): Dynamically weights the non-target class knowledge distillation loss based on the sample-level firing rates of the student SNN, adjusting the distillation supervision strength for different samples.
The experiments were conducted on image classification, object detection, and semantic segmentation tasks, reporting some performance improvements over existing SNN distillation methods.

Critical strengths

The author have conducted experiments on multiple datasets and tasks, including image classification, object detection, and semantic segmentation. The results show some performance improvements over existing SNN distillation methods, which suggests that the proposed Firing-Guided distillation strategies may have some effectiveness. However, due to the lack of following critical issues, it is still difficult to convincingly demonstrate the uniqueness and necessity of the proposed Firing-Guided distillation mechanism.

Critical weaknesses
  • W1. Innovation and novelty are limited, and the paper lacks sufficient justification for the originality of the proposed methods. Specifically, FGFKD is highly similar to BKDSNN[1] (ECCV 2024), with the only change being the replacement of uniform random masking probability with an adaptive probability based on the firing rate of student channels. Therefore, the main technical change in FGFKD is focused on the masking sampling strategy rather than a new feature distillation mechanism. FGLKD is highly similar to DKD[2] (CVPR 2022), as it follows the same basic decomposition approach and KL divergence optimization objective, with only the fixed weight in non-target class distillation replaced by a dynamic weight determined by the average firing rate of samples. This essentially constitutes a sample-level adaptive reweighting of existing distillation losses. The main contribution lies in incremental adjustments to existing mechanisms rather than proposing a new knowledge transfer paradigm.

  • W2. The core experiments lack key combination baselines, weakening the independent contribution of the proposed mechanism. Since this paper mainly introduces a firing rate-guided mechanism on existing feature distillation and decoupled logit distillation frameworks, the experimental design should focus on verifying the independent benefits brought by these new mechanisms compared to the original methods and their direct combinations. The paper lacks the following critical comparisons: (1) BKDSNN[1] (Uniform Mask) + standard DKD[2]; (2) FGFKD + standard DKD; (3) BKDSNN (Uniform Mask) + FGLKD. These sets of experiments are crucial for further evaluating the unique contributions of the Firing-Guided distillation mechanism. Additionally, if the adaptive knowledge distillation mechanism based on spike firing activity is the main innovation of this paper, the authors should further validate its generality and transferability by introducing it into other existing SNN feature distillation frameworks (such as CKDSNN[3], LaSNN[4], MEOM[5], and Efficient ANN-Guided Distillation[6]), or adapting and combining it with other general ANN knowledge distillation strategies (such as CAM-KD (ICCV 2025)[7], EA-KD (ICCV 2025)[8], WKD (NeurIPS 2024)[9], SDD (CVPR 2024)[10], and SKD (CVPR 2026)[11]). This should be done while maintaining the original framework and fair experimental settings to analyze whether the Firing-Guided mechanism can also bring stable performance improvements, rather than being limited to improvements on BKDSNN. The lack of these key experiments makes it insufficient to demonstrate that the Firing-Guided mechanism has universal value and substantial technical contributions independent of specific distillation frameworks.

  • W3. The sparsity and redundancy of spike firing activity are not new research findings, and the paper lacks sufficient justification for its novel scientific insights. The sparsity, non-uniformity, and redundancy of spike firing in SNNs have already been systematically analyzed in IRSNN (ICCV 2023)[12], which discussed the relationship between firing activity, network performance, and computational efficiency. Therefore, the phenomenon shown in Figure 4(a) of this paper, where firing activity is concentrated in a few channels, while providing specific empirical statistics, does not introduce a novel concept of sparsity or redundancy. The authors need to further explain what new patterns this observation reveals compared to existing analyses of SNN firing activity. It is particularly noteworthy that previous research has indicated that spike firing activity may contain redundancy and noise, and firing frequency alone cannot be directly equated with information value or feature importance. Therefore, the concentration phenomenon in Figure 4(a) is insufficient to directly support the proposed knowledge distillation strategy. If the authors consider these observations as significant scientific contributions of the paper, they need to further clarify their differences from existing research and provide theoretical analysis.

  • W4. The method design is mainly a heuristic adjustment of existing techniques, lacking in-depth analysis of the effectiveness and generalizability of the new mechanisms. If Firing Activity can indeed effectively guide knowledge distillation, then other monotonic nonlinear mappings, such as Sigmoid, power functions, exponential functions, or normalized mappings based on firing rate ranking, could theoretically achieve differentiated supervision for high firing activity channels or samples. These functions have different sensitivities to different firing rate intervals, which may result in different distributions of knowledge transfer intensity. The authors did not explain why they chose a simple truncated linear mapping and lacked systematic comparisons of different function forms. Therefore, it is still unclear whether the performance improvement mainly comes from the effective information contained in Firing Activity or relies on specific heuristic mappings and their hyperparameter configurations. Additionally, this paper limits the mask probability of FGFKD to [0.25, 0.75] and the distillation weight of FGLKD to [0.25, 1.0]. However, the authors did not sufficiently explain the rationale for these boundary values or report the actual distribution of firing rates and mapped weights. It is particularly noteworthy that when a large number of channels or samples have firing rates below 0.25, the corresponding mask probabilities or distillation weights will be uniformly truncated to 0.25, thereby weakening or even eliminating the adaptive differences between different channels or samples. Therefore, the authors should report the statistical distribution of firing rates under different architectures, the proportion of weights concentrated at truncation boundaries, and conduct sensitivity experiments on mask probability boundaries, distillation weight boundaries, and mapping function shapes to verify whether the new mechanisms truly play their intended role.

References

[1] Xu Z., You K., Guo Q., Wang X., He Z. BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation. ECCV, 2024. Link.

[2] Zhao B., Cui Q., Song R., Qiu Y., Liang J. Decoupled Knowledge Distillation. CVPR, 2022. Link.

[3] Liu X., Xia N., Zhou J., Xu J., Guo D. A Closer Look at Knowledge Distillation in Spiking Neural Network Training. AAAI, 40(3): 1946–1954, 2026. Link.

[4] Hong D., Qi Y., Wang Y. LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks. Neurocomputing, 656: 131351, 2025. Link.

[5] Sun K., Duan P., Huang Y., Gong N., Kuhlmann L. Many Eyes, One Mind: Temporal Multi-Perspective and Progressive Distillation for Spiking Neural Networks. ICLR, 2026. Link.

[6] Yang S., Yu C., Liu L., Ma H., Wang A., Li E. Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement. CVPR, 2025. Link.

[7] Lan Q., Tian Q. ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillation. ICCV, 2025. Link.

[8] Su C.-P., Tseng C.-H., Pu B., Zhao L., Yang J., Chen Z., Lee S.-J. EA-KD: Entropy-based Adaptive Knowledge Distillation. ICCV, 2025. Link.

[9] Lv J., Yang H., Li P. Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation. NeurIPS, 2024. Link.

[10] Wei S., Luo C., Luo Y. Scaled Decoupled Distillation. CVPR, 2024. Link.

[11] Jeong H.-J., Lee H.-J., Choi S.-H. Streamlined Knowledge Distillation. CVPR, 2026. Link.

[12] Yao M., Hu J., Zhao G., Wang Y., Zhang Z., Xu B., Li G. Inherent Redundancy in Spiking Neural Networks. ICCV, 2023. Link.

Questions for authors
  • Q(1) Why should high firing rate channels be assigned higher masking probabilities? High firing rates may correspond to important discriminative features, but they may also contain redundant or noisy activity. How do you establish a reliable connection between firing rate and feature knowledge value, and how do you justify why these channels should be masked rather than prioritized for retention or directly enhanced with feature supervision?

  • Q(2) How does more frequent masking of high firing channels help enhance knowledge transfer? Feature masking and recovery can indeed constitute an effective self-supervised regularization mechanism, but this does not necessarily imply that increasing the masking probability of high firing channels will promote effective transfer of teacher knowledge. Can you explain how this strategy affects feature reconstruction, information retention, and gradient optimization processes?

  • Q(3) Why does a larger non-target class distribution discrepancy imply that higher distillation weights should be assigned? You should clarify why supervision strength should be adjusted based on firing rate rather than directly based on teacher-student distribution discrepancy, prediction uncertainty, or sample difficulty.

  • Q(4) Does the correlation between firing rate and distribution discrepancy imply a direct mechanistic link between the two? Higher firing rates may co-vary with factors such as sample difficulty, prediction confidence, inter-class margins, or teacher-student representation differences. The authors have not sufficiently ruled out these potential confounding factors, so it is unclear whether firing rate itself provides independent and necessary guidance for distillation.

  • Q(5). The relationship between Figure 4(b) and the design of FGLKD. Figure 4(b) only shows the statistical correlation between Firing Rate and Non-target Distribution Discrepancy as well as classification sensitivity. How do the authors demonstrate that high firing samples indeed require stronger Non-target KD? Could these phenomena be explained by sample difficulty, prediction confidence, or Logit Margin? Since the authors have already calculated the distribution discrepancy between teacher and student, why not directly use that discrepancy to adjust distillation weights instead of indirectly adjusting through Firing Rate?

  • Q(6). Regarding the sufficiency of experimental comparisons and the reliability of performance improvements. For performance improvements of only about 0.3–0.6 percentage points, can the authors provide mean, standard deviation, and statistical analysis from multiple random seed experiments to demonstrate the stability of these gains? Additionally, can the detection and segmentation tasks supplement more comprehensive evaluation metrics and independent module ablation?

3 comments
AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear Reviewer Ye8u,

As the discussion deadline is approaching, we kindly remind that we have answered your remaining concerns carefully with additional experiments and clarification.

Could you please check our response and consider improving the rating if we successfully addressed you concerns?

Any feedback will be greatly appreciated and we are glad to provide any additional clarifications that you may need to bring a fruitful discussion.

Warm regards, The Authors

Dear Reviewer,

This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review their responses and provide any further comments before the discussion period closes.

Thank you for your time and effort!

AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear Reviewer Ye8u,

As the discussion deadline is approaching, we kindly remind that we have answered your remaining concerns carefully with additional experiments and clarification.

Could you please check our response and consider improving the rating if we successfully addressed you concerns?

Any feedback will be greatly appreciated and we are glad to provide any additional clarifications that you may need to bring a fruitful discussion.

Warm regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional Response to Reviewer jjJp

Thank you very much for taking the time to read our response. We greatly appreciate your thoughtful follow-up. Below, we address the remaining unresolved concerns.

[Concern 1] Motivation, methodological basis, and novelty require further clarification.

FGFKD addresses the concentration of firing activity across channels. Section 4.2 and Figure 4(a) show that activity is concentrated in a subset of channels. Masking nearly silent channels introduces little perturbation. For fixed student features, the expected activity removed from channel dd is defined as follows:

EM[(1−Mb,d)sb,d]=pb,dsb,d.\mathbb{E}_{M}\left[(1-M_{b,d})s_{b,d}\right] = p_{b,d}s_{b,d}.

Uniform masking assigns the same masking probability to all channels, irrespective of their firing activity. FGFKD leverages pb,d=clip⁡(sb,d,pmin⁡,pmax⁡)p_{b,d}=\operatorname{clip}(s_{b,d},p_{\min},p_{\max}) to concentrate the reconstruction challenge on active channels, encouraging recovery of teacher features from the remaining student responses. Masking applies to the auxiliary restoration branch; classification uses unmasked features. The masking ablations in Table 6 and R2-Table 2 support the effectiveness of firing-guided reconstruction, consistent with the improved localization of SNN feature responses to task-relevant regions in Figures 1(b), 5(a), A2, and A3.

FGLKD addresses firing-associated discrepancies in non-target distributions. Figure 4(b) shows that higher-firing samples exhibit larger teacher–student non-target discrepancies and greater sensitivity to competing-class logit perturbations. Since non-target distributions encode inter-class relations, these observations motivate stronger supervision of those relations. Accordingly, wb=clip⁡(rb,wmin⁡,wmax⁡)w_b=\operatorname{clip}(r_b,w_{\min},w_{\max}) increases non-target supervision with firing activity while retaining uniform target-class supervision. The weighting ablations in Table 6 and R2-Table 3 support the effectiveness of firing-guided non-target supervision, while R2-Table 5 further demonstrates improved validation accuracy and reduced training–validation gaps on high-firing samples under full FGKDSNN.

Our contribution is to condition channel-wise masking and sample-wise non-target supervision on the student’s intrinsic firing activity, distinguishing FGKDSNN from the uniform masking and WSLD objective adopted by BKDSNN. The controlled ablations assess this correspondence and its prioritization direction.

[Concern 2] Consistency of the SDT-M results across R2-Tables 1–3.

The three tables use the same student architecture but evaluate different combinations of distillation components. Their correspondence with R2-Table 6 is as follows:

Reported resultFGFKDFGLKDSDT-M top-1 accuracy (%)
R2-Table 1: FGKDSNN, overall✓✓81.56
R2-Table 2: Firing-guided masking✓✗81.03
R2-Table 3: Firing-guided reweighting✗✓80.72

R2-Table 1 reports the complete FGKDSNN method. R2-Table 2 disables FGLKD to isolate the comparison between masking strategies, whereas R2-Table 3 disables FGFKD to isolate the comparison between sample-wise reweighting strategies. All three accuracy values exactly match the corresponding component configurations in R2-Table 6.

We acknowledge that the captions of R2-Tables 2 and 3 do not state these component settings sufficiently explicitly. We will label them as FGFKD-only and FGLKD-only, respectively, and distinguish them from full FGKDSNN in R2-Table 1. We will also incorporate the methodological explanations above into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.