acceptodds
Withdrawn from ICLR 2027

Local Neuronal Dynamics as Temporal Inductive Biases for Spiking Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spiking neurons maintain internal states that evolve over time, providing an intrinsic source of temporal dynamics for spiking neural networks. However, existing spiking Transformers often underutilize this intrinsic temporal structure, either treating timesteps independently or introducing explicit temporal interactions through temporal attention. In this work, we investigate whether local neuronal dynamics can serve as an effective temporal inductive bias for spiking Transformers. We propose SNN-native Temporal Modeling (SNTM), a framework that incorporates temporal information at two stages of spiking computation. Multi-threshold Spiking Patch Splitting (MT-SPS) enriches spike representations by aggregating responses from multiple fixed firing thresholds during the initial spike projection. Event-driven Value Modulation (EDVM) captures local temporal variations through the current spike activity and the first-order difference between adjacent value features, and injects these cues into the value pathway. Across four spiking Transformer backbones and six static or event-based benchmarks, SNTM improves the corresponding baselines. On ImageNet, it improves SDT, QKFormer, and Spikformer by 1.75, 1.46, and 2.07 accuracy points, respectively. Mechanism studies on CIFAR10-DVS show that disrupting temporal adjacency in EDVM reduces accuracy from 78.23% to 76.47%, whereas reversing the temporal reference has little effect, consistent with EDVM relying on local temporal contrast rather than temporal direction. SNTM adds only three learned scalars per enabled attention block and at most 0.09G MACs in our measured settings. These results demonstrate that local neuronal dynamics provide a practical and low-overhead temporal inductive bias for spiking Transformers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spiking Neural Networks, Spiking Transformers, Temporal Modeling, Neuronal Dynamics, Temporal Inductive Bias

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Zv4zOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper enhances the design of Spiking Transformers to address their intrinsic temporal modeling issue. The authors introduce multi-threshold spiking neurons (MTSN) in the embedding module to generate different responses to the same input. In the attention module, they further construct a modulation mechanism for the value branch, termed EDVM, by leveraging the variance across time steps.

Results on multiple benchmarks show that the proposed method can achieve certain improvements over the baseline. However, the choice and reproduction of the baselines raise concerns about the reliability of the experimental results. The degree of coupling between MTSN and EDVM is also unclear. In addition, the hardware friendliness of the multi-level integer response mechanism in MTSN has not been sufficiently evaluated, and the analysis of some ablation experiments is overly coarse-grained. Overall, I believe that the paper still has considerable room for improvement.

Critical strengths
  1. The paper improves model performance from the perspective of temporal characteristics, and the proposed enhancement is model-agnostic.

  2. The proposed method does not introduce additional parameters and therefore avoids excessive computational overhead or complex operations.

  3. EDVM considers different modulation strategies for event-based and static temporal characteristics, making it adaptable to cases with either pronounced or less significant temporal dynamics.

Critical weaknesses
  1. The baselines in the paper (except for SDT) are all taken from STAtten. Why are the results not cited from their corresponding original papers? For some models, such as Spikformer and SpikingResFormer, even with the improvement brought by SNTM, their performance still fails to reach the results reported in the original papers.

  2. The coupling between EDVM and MTSN has not been analyzed. As shown in Tab. 2, the improvement brought by EDVM is relatively significant, whereas MTSN only brings a 0.17 improvement on SDT.

  3. The ablation experiments are limited to component-level analysis. The sensitivity to different thresholds in MTSN has not been investigated. Similarly, it remains unclear why EDVM adopts first-order adjacent temporal differences instead of higher-order temporal differences.

  4. The modulation results on the value branch lack visualization and more specific analysis.

Questions for authors

see Weakness.

Reviewer_o42TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SNN-native Temporal Modeling (SNTM) for spiking Transformers, aiming to exploit intrinsic neuronal dynamics as a lightweight temporal inductive bias. SNTM combines Multi-threshold Spiking Patch Splitting (MT-SPS) with Event-driven Value Modulation (EDVM), which uses adjacent value differences to modulate the value pathway. Experiments across four spiking Transformer backbones and six event-based benchmarks show consistent improvements with limited reported computational overhead.

Critical strengths
  1. SNTM is lightweight and architecture-compatible, producing consistent improvements across four different spiking Transformer backbones and both static and event-based datasets.
  2. The paper includes useful mechanism analyses, particularly the shuffle experiments showing that EDVM depends on local temporal adjacency rather than temporal direction.
Critical weaknesses
  1. The current mechanism analysis shows that adjacent temporal differences are useful, but does not fully establish that the gain specifically comes from intrinsic neuronal dynamics. A stronger control, such as resetting neuronal states across timesteps while keeping the EDVM computation unchanged, would more directly isolate this claim.
  2. The efficiency claim is mainly based on MACs and parameter counts. MT-SPS produces integer-valued multi-threshold responses rather than standard binary spikes, while EDVM introduces real-valued differences, nonlinearities, and multiplications. Their actual effect on spike sparsity, activation precision, and neuromorphic energy efficiency is not evaluated.
Questions for authors
  1. Can the authors provide an experiment that resets or removes neuronal state evolution across timesteps to directly verify that SNTM’s gains originate from intrinsic neuronal dynamics?
  2. How does MT-SPS affect spike sparsity, firing activity, and activation precision compared with the original binary-spike representation?
  3. Can the authors provide energy or hardware-level efficiency measurements beyond MACs, especially given the multi-threshold activations and real-valued EDVM operations?
Reviewer_f7ErOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SNN-native Temporal Modeling (SNTM) to better utilize temporal information in Spiking Transformers. The method consists of two components. MT-SPS uses multiple firing thresholds to enrich spike representations, while EDVM combines spike activity at the current timestep with value feature differences between adjacent timesteps to modulate the value pathway. Experiments on four Spiking Transformer architectures and six datasets show improvements in classification accuracy with relatively small reported computational overhead.

Critical strengths
  1. The paper evaluates the proposed method on multiple Spiking Transformer architectures and datasets, including ImageNet, demonstrating its potential generalizability.

  2. The paper provides ablation studies and additional analyses to investigate the contributions of different components and temporal information.

Critical weaknesses
  1. EDVM mainly relies on temporal feature differences and value modulation. I believe its novelty over existing temporal modeling methods could be better explained.

  2. According to Table 2, EDVM alone already achieves most of the performance gains, while adding MT-SPS brings only a small additional improvement. This raises the question of whether MT-SPS is necessary.

  3. Although the method introduces few additional parameters and MACs, its actual inference efficiency remains unclear. I would like to see comparisons of inference latency and energy consumption.

Questions for authors
  1. What are the main differences between MT-SPS and existing methods such as I-LIF? How would the model perform if I-LIF were used instead of MT-SPS?

  2. Could the authors provide additional experiments to demonstrate the necessity of combining MT-SPS with EDVM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.