Local Neuronal Dynamics as Temporal Inductive Biases for Spiking Transformers

Spiking neurons maintain internal states that evolve over time, providing an intrinsic source of temporal dynamics for spiking neural networks. However, existing spiking Transformers often underutilize this intrinsic temporal structure, either treating timesteps independently or introducing explicit temporal interactions through temporal attention. In this work, we investigate whether local neuronal dynamics can serve as an effective temporal inductive bias for spiking Transformers. We propose SNN-native Temporal Modeling (SNTM), a framework that incorporates temporal information at two stages of spiking computation. Multi-threshold Spiking Patch Splitting (MT-SPS) enriches spike representations by aggregating responses from multiple fixed firing thresholds during the initial spike projection. Event-driven Value Modulation (EDVM) captures local temporal variations through the current spike activity and the first-order difference between adjacent value features, and injects these cues into the value pathway. Across four spiking Transformer backbones and six static or event-based benchmarks, SNTM improves the corresponding baselines. On ImageNet, it improves SDT, QKFormer, and Spikformer by 1.75, 1.46, and 2.07 accuracy points, respectively. Mechanism studies on CIFAR10-DVS show that disrupting temporal adjacency in EDVM reduces accuracy from 78.23% to 76.47%, whereas reversing the temporal reference has little effect, consistent with EDVM relying on local temporal contrast rather than temporal direction. SNTM adds only three learned scalars per enabled attention block and at most 0.09G MACs in our measured settings. These results demonstrate that local neuronal dynamics provide a practical and low-overhead temporal inductive bias for spiking Transformers.