acceptodds
Under review as a conference paper at ICLR 2027

A Sparse Frequency-Aware Spiking Transformer with Hierarchical Wavelet Pruning and Adaptive Dual-Path Routing

Abstract

Spiking neural networks (SNNs) offer significant energy efficiency advantages, yet existing compression methods perform uniform pruning on temporal feature maps, failing to exploit the substantially differentiated redundancy structures among different frequency components. Inspired by the M/P pathway frequency segregation mechanism in biological visual systems, this paper proposes the Sparse Frequency-Aware Transformer (SFAT), which achieves frequency-aware structured sparsity through hierarchical Haar wavelet decomposition: coefficient-wise soft thresholding distinguishes low/high-frequency subbands, while channel-wise energy gating adaptively prunes the four quadrant subbands. To compensate for high-frequency information loss induced by pruning, we design the Dual-Path MLP Routing mechanism (DPMR), comprising a frequency compensation path with subband-specific experts and global gating, supporting both a Parallel mode for high-precision scenarios and a Lightweight mode for edge deployment. Experiments on CIFAR-100, Tiny-ImageNet, and neuromorphic datasets demonstrate that SFAT(Parallel) achieves 79.70% on CIFAR-100 and 66.44% on Tiny-ImageNet, outperforming existing Spiking Transformer methods while reducing energy consumption by approximately 60%; SFAT(Lightweight) achieves comparable accuracy with nearly half the parameters, validating the synergistic optimization between frequency decomposition and structural compression. Code will be made available upon acceptance.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Spiking Transformer, Frequency-Aware Sparse Attention, Hierarchical Wavelet Pruning, Dual-Path MLP Routing, Model Compression, Energy-Efficient Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_s69KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Sparse Frequency-Aware Transformer, which reduces redundant spikes through staged Haar wavelet decomposition, coefficient thresholding, and channel energy gating, and designs a frequency compensation branch to provide both a parallel mode that balances accuracy and a lightweight mode that reduces parameters. In experiments, the parallel mode achieves 79.70% and 66.44% accuracy on CIFAR-100 and Tiny-ImageNet, respectively, while reducing estimated energy consumption. The lightweight mode further reduces parameters and computational cost, but at the cost of accuracy. Its main value lies in exploring the combination of frequency information and spiking neural network compression. Currently, its performance on neuromorphic datasets is still weaker than strongest reported baselines.

Critical strengths

The direction of frequency-selective sparsity is reasonable.

Critical weaknesses
  1. The core definition of "structured pruning" is invalid. The channel-wise gating weight in Eq. (10) is always between 0.4 and 0.6, and the gating itself does not set the channel to zero. Subsequent thresholds can reduce activity, but the paper does not explain how to form pruning entire channels or how to actually skip the corresponding computation.

  2. The innovation and controlled ablation studies are relatively weak. The original SWFormer paper already included three-branch frequency mixing, ternary spikes, and axial Haar transform. The novelty of this paper mainly lies in the sparsity and compensation modules, but it lacks direct comparisons with ordinary grouped MLP, frequency-agnostic pruning with shared thresholds, and compression methods such as SparseSpikFormer.

Questions for authors

The "high-frequency compensation" mechanism lacks supporting evidence. The FCP divides the input channel into four groups, which are then referred to as the four wavelet subbands. However, in Figure 2, the input has already undergone inverse transform and other branch fusions. Why do these four groups still have clear frequency identities? Currently, ablation can only support the idea that "adding grouping branches helps," but it cannot fully support the idea that "lost high-frequency information is recovered."

Reviewer_WscjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Sparse Frequency-Aware Transformer (SFAT). It first uses wavelet decomposition to separate information at different frequencies, applies stronger sparsification to the more redundant parts, and then uses a lightweight MoE module to compensate for the loss of detail.

Critical strengths
  • Based on wavelet decomposition, this paper performs sparsification in the frequency domain, achieving a better trade-off between performance and computational overhead for SNNs.
Critical weaknesses
  • The novelty of this paper is limited. The proposed SFAT is merely an incremental improvement on the existing SWFormer. SWFormer has introduced the Haar wavelet frequency branches, the ternary spike representations, and a block-diagonal weight structure within the frequency branches. Based on this, the proposed SFAT adds frequency-based sparsification and the gated lightweight MoE module. The novelty of this improvement is limited.
  • Although parallel SFAT achieves higher classification accuracy and lower SOPs than SWFormer, this comes at the cost of more parameters. Meanwhile, although lightweight SFAT has fewer parameters than SWFormer, its classification accuracy is also significantly lower than that of SWFormer. Therefore, the experimental results only demonstrate that SFAT achieves a different trade-off between efficiency and performance compared to SWFormer.
  • The experiments in this paper are insufficient. The proposed SFAT is only evaluated on static image classification benchmarks CIFAR-100 and Tiny-ImageNet, and neuromorphic vision classification benchmarks CIFAR10-DVS and DVS-Gesture. It lacks experimental validation on large-scale datasets such as ImageNet, and other vision tasks such as object detection and instance segmentation on COCO.
  • The low-frequency-biased assumption is not well-suited for event data. On CIFAR10-DVS, the Parallel accuracy is 78.7%, lower than 80.9% of Spikformer. On DVS-Gesture, it achieves 97.9% accuracy, lower than 99.3% of SDformer. The authors also acknowledge that high-frequency suppression may compromise the dynamic information in event data.
Questions for authors

Please provide a more detailed comparison between SFAT and SWFormer, including the novelty of the methods, their performance on additional benchmarks, and the trade-offs between performance, computational cost, and the number of parameters.

Reviewer_h5mdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes the Sparse Frequency-Aware Spiking Transformer (SFAT), which aims to reduce the computational cost of Spiking Transformers through frequency-aware sparsification. The method consists of two main components: HSWL employs hierarchical Haar wavelet decomposition, coefficient-wise thresholding, and energy-driven channel gating to achieve sparsification; DPMR introduces a lightweight frequency compensation pathway to balance accuracy and computational efficiency. Experiments on CIFAR-100, Tiny-ImageNet, and neuromorphic datasets report competitive classification accuracy, along with reductions in SOPs and estimated energy consumption.

Critical strengths

The paper addresses a practically relevant problem and presents a reasonably organized approach. The inclusion of ablation studies and evaluations across different data modalities provides some useful empirical evidence.

Critical weaknesses

1. Limited methodological novelty. SWFormer (Fang et al., ECCV 2024) has already introduced Haar wavelet decomposition into Spiking Transformers and employed FATM for frequency-domain feature processing. This paper mainly builds upon that work by incorporating coefficient-wise thresholding, channel gating, and a lightweight grouped-convolution pathway. The independent novelty of these modifications appears relatively limited.

2. Insufficient support for the central motivation. The authors motivate their method by arguing that conventional self-attention leads to the loss of high-frequency information. However, avoiding this issue primarily relies on the frequency-domain FATM already introduced by SWFormer, whereas the newly proposed HSWL mainly performs further sparsification on existing wavelet features. The paper lacks direct evidence that HSWL can preserve task-relevant high-frequency information more effectively. Therefore, the connection between the central motivation and the proposed methodological contributions remains insufficiently established. I would suggest that the authors further clarify and refine the paper's main claims.

3. Incomplete energy evaluation, insufficient to support a core contribution. The paper presents energy reduction as one of its main contributions. However, Section 5.1 only specifies the unit energy costs of MACs (4.6 pJ) and ACs (0.9 pJ), and the reported Energy values are approximately proportional to SOPs, suggesting that the estimates may primarily rely on arithmetic operation counts. Nevertheless, previous SNN hardware studies [1, 2] have shown that memory accesses and data movement can account for a substantially larger proportion of energy consumption than arithmetic computation. Practical SNN inference also involves weight accesses, membrane-potential state reads and writes, data movement, and on-chip communication, but the paper does not clarify whether these costs are included in the reported Energy metric. Therefore, a reduction in SOPs does not directly establish an approximately 60% reduction in overall energy consumption. A more comprehensive hardware-aware energy evaluation would help substantiate this core claim.

References

[1] S. Narayanan, K. Taht, R. Balasubramonian, E. Giacomin, and P.-E. Gaillardon. SpinalFlow: An Architecture and Dataflow Tailored for Spiking Neural Networks. In Proceedings of the 47th Annual International Symposium on Computer Architecture (ISCA), pp. 349–362, 2020. https://doi.org/10.1109/ISCA45697.2020.00038

[2] G. Tang et al. SENECA: Building a Fully Digital Neuromorphic Processor, Design Trade-offs and Challenges. Frontiers in Neuroscience, 17:1187252, 2023. https://doi.org/10.3389/fnins.2023.1187252

Questions for authors

Regarding the high-frequency compensation mechanism: Appendix B.3 illustrates the attenuation of high-frequency information caused by channel pruning, but it remains unclear how FCP specifically compensates for this information loss. Could the authors further explain the underlying mechanism and provide a spectral comparison before and after introducing FCP, to verify that its effect goes beyond general feature enhancement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.