- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper constructs a Softmax-free, spike-driven Transformer language model using Winner-Take-All (WTA) attention. The core idea is to replace Softmax normalization with sparse, binary WTA selection, thereby creating an attention mechanism that better aligns with the spike-driven computing paradigm. The authors evaluated the model across 16 datasets covering natural language understanding, question answering, and commonsense reasoning. Experimental results demonstrate that Hard WTA attention achieves performance comparable to Softmax attention while eliminating the need for exponential and normalization operations.
Critical strengths
- The motivation is sound and meaningful: recognizing that visual SSA is unsuitable for language modeling tasks, the authors propose an attention scheme tailored for spiking language modeling.
- The method is simple yet effective, and the core experimental results are compelling. The comparison with Softmax under the same architecture is particularly significant; Hard WTA generates one-hot attention and eliminates the need for Softmax normalization during inference while retaining the vast majority of predictive performance.
- The experiments are comprehensive and provide evidence of scalability. The paper covers 16 downstream datasets and investigates various factors, including WTA variants, time steps, surrogate gradients, model sizes, and pre-training data volumes. The failure of traditional visual spiking attention to converge during causal pre-training further underscores the need for a specialized attention mechanism for language tasks. Scaling the decoder model from 0.4B to 1B parameters and increasing pre-training data from 0.1B to 0.5B tokens consistently yielded performance improvements, holding positive implications for the future development of large-scale spiking language models.
Critical weaknesses
Compared to modern language models, the scale and pre-training data of current models remain limited; therefore, conclusions regarding scalability should be qualified by the scope of the actual evaluations conducted in the paper. Minor issue: Figures lack sufficient clarity. Please refine the color palette to maintain uniform styling throughout the paper.
Questions for authors
1.During training, the gradient from the proposed method was replaced with the Softmax gradient. What are the differences and connections between this training approach and the surrogate gradient scheme used for LIF neurons?
2. In Table 4.1, the model adopting T‑LIF achieves evidently higher accuracy. However, T‑LIF is not an innovation of this paper. Why is it still compared with NI‑LIF?
3. In Figure 1, WTA appears to only pass data whose threshold is greater than or equal to 5. Could this be replaced with a neuron with a threshold of 5? Has this been attempted, and what would the performance be?
2 comments
The author has addressed my concerns. I will keep my score unchanged and raise my confidence level.
Response to reviewer vHBv
We sincerely thank the reviewer for the constructive feedback. We have carefully considered the concerns and provide our detailed responses below.
R: Thank you for this constructive suggestion. We agree that our scalability claims should be limited to the evaluated model sizes and pretraining data scales. Due to computational resource constraints, 1.0B parameters is the largest model we have evaluated through end-to-end training, and we will explicitly acknowledge this limitation.
Regarding scalability, our experiments (Figure3, Table 6,7) demonstrate consistent improvements within the evaluated range. With 0.1B pretraining tokens, increasing WD-SpikingFormer from 0.4B to 1.0B parameters improves average accuracy from 28.4% to 29.0% on QAT and from 43.2% to 43.6% on CRT. For the 1.0B model, increasing the pretraining data from 0.1B to 0.5B tokens further improves accuracy to 31.9% on QAT and 44.1% on CRT. These results suggest modest but consistent performance gains with increasing model size and pretraining data within the tested range.
Regarding figure clarity, we will improve the figures by adopting a consistent color palette, enhancing contrast, standardizing typography and line widths, and refining labels and legends throughout the manuscript.
R: Thank you for this insightful question. Both methods share the same surrogate-gradient principle: discrete operations are preserved in the forward pass, while smooth gradients are used for backpropagation without affecting inference. However, they approximate different discontinuities. For LIF neurons, surrogate gradients approximate the derivative of threshold-based spike firing. In contrast, Hard WTA performs competitive selection across multiple attention scores, whose exact gradient is zero almost everywhere. We therefore adopt the Softmax Jacobian as a surrogate to provide smooth relative-ranking gradients, motivated by Hard WTA being the zero-temperature limit of Softmax.
Thus, LIF surrogate gradients approximate threshold firing, whereas WTA surrogate gradients approximate competitive token selection.
R: Thank you for this insightful question. We compare T-LIF with NI-LIF not to claim novelty in neuron design, but to evaluate the impact of neuron choice and demonstrate the compatibility of WTA attention with different spiking neurons. As shown in Table 8, under the same backbone and T=4, T-LIF achieves a slightly higher GLUE score (66.3% vs. 65.7%) but requires substantially longer pretraining (121 vs. 26 hours on eight RTX 4090 GPUs). Thus, NI-LIF achieves comparable accuracy with approximately 4.7× faster pretraining, making it our default choice for efficient spiking simulation.
R: Thank you for this insightful question. Figure 1 is a schematic illustration, where 5 represents the maximum score rather than a predefined threshold. Hard WTA performs relative competition, selecting the maximum element regardless of its absolute value: yi=1[i=argmaxjaj]. In contrast, a neuron with a fixed threshold of 5 performs element-wise threshold firing (yi=1[ai≥5]), which may activate zero, one, or multiple positions and cannot guarantee one-hot selection. Therefore, fixed-threshold firing is not equivalent to WTA and was not evaluated as a direct replacement. Implementing WTA through spiking neurons would require adaptive competition or lateral inhibition across candidate inputs, which is essentially another implementation of WTA.