acceptodds
Under review as a conference paper at ICLR 2027

Softmax-Free Spiking Transformers for Language Modeling via Winner-Take-All Attention

Abstract

Spiking Transformers, which combine the scalability of Transformers with the sparse, energy-efficient property of Spiking Neural Networks (SNNs), have achieved impressive results in neuromorphic and vision tasks and attracted increasing attention. Existing spiking language models often rely on softmax-based self-attention, whose exponentiation and normalization operations increase computational costs and complicate neuromorphic deployment. To address this issue, we introduce Winner-Take-All (WTA) mechanisms into spiking transformers and propose two novel softmax-free, spike-driven self-attention modules: WTA Spiking Self-Attention (WSSA) and Causal WTA Spiking Self-Attention (CWSSA). Based on them, we design WTA-based Encoder-only Spiking Transformer (WE-Spikingformer) for masked language modeling and WTA-based Decoder-only Spiking Transformer (WD-Spikingformer) for causal language modeling, systematically exploring softmax-free, spike-driven transformer architectures trained end-to-end for natural language processing tasks. Extensive experiments on 16 datasets spanning natural language understanding, question-answering tasks, and commonsense reasoning tasks validate the effectiveness of our approach and highlight the promise of spiking transformers for general language modeling and energy-efficient artificial intelligence.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Winner-Take-All, Spiking Transformers, Language Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vHBvOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper constructs a Softmax-free, spike-driven Transformer language model using Winner-Take-All (WTA) attention. The core idea is to replace Softmax normalization with sparse, binary WTA selection, thereby creating an attention mechanism that better aligns with the spike-driven computing paradigm. The authors evaluated the model across 16 datasets covering natural language understanding, question answering, and commonsense reasoning. Experimental results demonstrate that Hard WTA attention achieves performance comparable to Softmax attention while eliminating the need for exponential and normalization operations.

Critical strengths
  1. The motivation is sound and meaningful: recognizing that visual SSA is unsuitable for language modeling tasks, the authors propose an attention scheme tailored for spiking language modeling.
  2. The method is simple yet effective, and the core experimental results are compelling. The comparison with Softmax under the same architecture is particularly significant; Hard WTA generates one-hot attention and eliminates the need for Softmax normalization during inference while retaining the vast majority of predictive performance.
  3. The experiments are comprehensive and provide evidence of scalability. The paper covers 16 downstream datasets and investigates various factors, including WTA variants, time steps, surrogate gradients, model sizes, and pre-training data volumes. The failure of traditional visual spiking attention to converge during causal pre-training further underscores the need for a specialized attention mechanism for language tasks. Scaling the decoder model from 0.4B to 1B parameters and increasing pre-training data from 0.1B to 0.5B tokens consistently yielded performance improvements, holding positive implications for the future development of large-scale spiking language models.
Critical weaknesses

Compared to modern language models, the scale and pre-training data of current models remain limited; therefore, conclusions regarding scalability should be qualified by the scope of the actual evaluations conducted in the paper. Minor issue: Figures lack sufficient clarity. Please refine the color palette to maintain uniform styling throughout the paper.

Questions for authors

1.During training, the gradient from the proposed method was replaced with the Softmax gradient. What are the differences and connections between this training approach and the surrogate gradient scheme used for LIF neurons?
2. In Table 4.1, the model adopting T‑LIF achieves evidently higher accuracy. However, T‑LIF is not an innovation of this paper. Why is it still compared with NI‑LIF?
3. In Figure 1, WTA appears to only pass data whose threshold is greater than or equal to 5. Could this be replaced with a neuron with a threshold of 5? Has this been attempted, and what would the performance be?

2 comments
AuthorsOpenReview ↗

Response to reviewer vHBv

We sincerely thank the reviewer for the constructive feedback. We have carefully considered the concerns and provide our detailed responses below.

Weaknesses 1: Compared to modern language models, the scale and pre-training data of current models remain limited; therefore, conclusions regarding scalability should be qualified by the scope of the actual evaluations conducted in the paper. Minor issue: Figures lack sufficient clarity. Please refine the color palette to maintain uniform styling throughout the paper.

R: Thank you for this constructive suggestion. We agree that our scalability claims should be limited to the evaluated model sizes and pretraining data scales. Due to computational resource constraints, 1.0B parameters is the largest model we have evaluated through end-to-end training, and we will explicitly acknowledge this limitation.

Regarding scalability, our experiments (Figure3, Table 6,7) demonstrate consistent improvements within the evaluated range. With 0.1B pretraining tokens, increasing WD-SpikingFormer from 0.4B to 1.0B parameters improves average accuracy from 28.4% to 29.0% on QAT and from 43.2% to 43.6% on CRT. For the 1.0B model, increasing the pretraining data from 0.1B to 0.5B tokens further improves accuracy to 31.9% on QAT and 44.1% on CRT. These results suggest modest but consistent performance gains with increasing model size and pretraining data within the tested range.

Regarding figure clarity, we will improve the figures by adopting a consistent color palette, enhancing contrast, standardizing typography and line widths, and refining labels and legends throughout the manuscript.

Questions 1: During training, the gradient from the proposed method was replaced with the Softmax gradient. What are the differences and connections between this training approach and the surrogate gradient scheme used for LIF neurons?

R: Thank you for this insightful question. Both methods share the same surrogate-gradient principle: discrete operations are preserved in the forward pass, while smooth gradients are used for backpropagation without affecting inference. However, they approximate different discontinuities. For LIF neurons, surrogate gradients approximate the derivative of threshold-based spike firing. In contrast, Hard WTA performs competitive selection across multiple attention scores, whose exact gradient is zero almost everywhere. We therefore adopt the Softmax Jacobian as a surrogate to provide smooth relative-ranking gradients, motivated by Hard WTA being the zero-temperature limit of Softmax.

Thus, LIF surrogate gradients approximate threshold firing, whereas WTA surrogate gradients approximate competitive token selection.

Questions 2: In Table 4.1, the model adopting T‑LIF achieves evidently higher accuracy. However, T‑LIF is not an innovation of this paper. Why is it still compared with NI‑LIF?

R: Thank you for this insightful question. We compare T-LIF with NI-LIF not to claim novelty in neuron design, but to evaluate the impact of neuron choice and demonstrate the compatibility of WTA attention with different spiking neurons. As shown in Table 8, under the same backbone and T=4T=4, T-LIF achieves a slightly higher GLUE score (66.3% vs. 65.7%) but requires substantially longer pretraining (121 vs. 26 hours on eight RTX 4090 GPUs). Thus, NI-LIF achieves comparable accuracy with approximately 4.7× faster pretraining, making it our default choice for efficient spiking simulation.

Questions 3: In Figure 1, WTA appears to only pass data whose threshold is greater than or equal to 5. Could this be replaced with a neuron with a threshold of 5? Has this been attempted, and what would the performance be?

R: Thank you for this insightful question. Figure 1 is a schematic illustration, where 5 represents the maximum score rather than a predefined threshold. Hard WTA performs relative competition, selecting the maximum element regardless of its absolute value: yi=1[i=arg⁡max⁡jaj]y_i=\mathbf{1}[i=\arg\max_j a_j]. In contrast, a neuron with a fixed threshold of 5 performs element-wise threshold firing (yi=1[ai≥5]y_i=\mathbf{1}[a_i\geq5]), which may activate zero, one, or multiple positions and cannot guarantee one-hot selection. Therefore, fixed-threshold firing is not equivalent to WTA and was not evaluated as a direct replacement. Implementing WTA through spiking neurons would require adaptive competition or lateral inhibition across candidate inputs, which is essentially another implementation of WTA.

The author has addressed my concerns. I will keep my score unchanged and raise my confidence level.

Reviewer_zK5rOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates Softmax-free spiking Transformers for language modeling. The authors propose Winner-Take-All Spiking Self-Attention (WSSA) for encoder architectures and Causal Winner-Take-All Spiking Self-Attention (CWSSA) for causal decoder architectures. Based on these mechanisms, the paper constructs WE-Spikingformer for masked language modeling and WD-Spikingformer for causal language modeling.

Critical strengths
  1. This work addresses a key challenge in extending Spiking Transformers from the visual domain to language modeling. Softmax operations—involving exponentiation, normalization, and division—are difficult to implement efficiently on neuromorphic hardware, and standard vision -based Spiking Self-Attention (SSA) performs poorly in language tasks. Consequently, exploring new attention mechanisms tailored for spiking language modeling is a well-founded approach.
  2. The proposed methods are concise and effective. WSSA and CWSSA naturally extend the Winner-Take-All (WTA) mechanism to both bidirectional and causal attention. This mechanism aligns seamlessly with the sparsity and event-driven nature of Spiking Neural Networks (SNNs). CWSSA is particularly valuable for autoregressive language modeling, as existing Softmax-free Spiking Transformers have primarily focused on vision tasks or encoder-only architectures.
  3. The ablation studies are comprehensive. The paper compares Hard WTA, Softmax Attention, traditional Spiking Self-Attention, Top-K WTA, and Sparsemax, facilitating an isolated analysis of the proposed mechanisms' effects. While producing extremely sparse attention outputs, Hard WTA achieves results comparable to Softmax across GLUE, question answering, and commonsense reasoning tasks.
  4. The experimental scope is extensive. The paper evaluates both encoder and decoder architectures across 16 language datasets. Scaling experiments with models up to 1 billion parameters demonstrate that the architecture benefits from increases in both model size and pre-training data.
Critical weaknesses
  1. Vision-based SSA performs poorly when directly applied to language modeling—a fact primarily evidenced by experimental results rather than a deep analysis of the underlying mechanisms. Furthermore, the paper would gain depth if the authors provided a more thorough analysis explaining why WTA outperforms SSA in language modeling tasks.
  2. Energy consumption figures are based on computational complexity estimates; measurements on actual neuromorphic hardware have not yet been conducted.
Questions for authors
  1. Is the WTA (Winner-Take-All) mechanism in the attention component of the paper applied across the sequence length LL? Will the WTA modeling approach still work if the sequence length increases?
2 comments
AuthorsOpenReview ↗

Response to reviewer zK5r

We sincerely thank the reviewer for the constructive feedback. We have carefully considered the concerns and provide our detailed responses below.

Weaknesses1: Vision-based SSA performs poorly when directly applied to language modeling—a fact primarily evidenced by experimental results rather than a deep analysis of the underlying mechanisms. Furthermore, the paper would gain depth if the authors provided a more thorough analysis explaining why WTA outperforms SSA in language modeling tasks.

R: Thank you for this valuable suggestion. We have expanded our analysis to explain why WTA is more suitable for language modeling than vision-oriented Spiking Self-Attention (SSA). (1) Limitations of SSA in language modeling. SSA computes SN(QK⊤V⋅s)\mathrm{SN}(QK^\top V\cdot s), where all keys with overlapping spikes contribute to the output without explicit token-wise competition or normalization. Such accumulation can be effective in vision tasks, where image patches often exhibit substantial spatial correlation and redundancy. However, language modeling requires distinguishing discrete, semantically diverse tokens and capturing specific syntactic or semantic dependencies, including long-range relationships. Without competitive selection, SSA may accumulate weakly relevant token representations, introducing interference and increasing the input magnitude to downstream spiking neurons as the number of active keys grows. (2) Advantages of WTA attention. In contrast, WTA explicitly performs competition along the key-sequence dimension for each query token: Aij=1[j=arg⁡max⁡m(qi⊤km)].A_{ij}=\mathbf{1}\left[j=\arg\max_m(q_i^\top k_m)\right]. By selecting the strongest query–key association before aggregating values, WTA suppresses contributions from weakly matched tokens and limits the number of selected keys. This provides a sparse attention mechanism that may improve the stability of information aggregation. Moreover, Hard WTA can be viewed as the zero-temperature limit of Softmax, preserving its relative token-selection property without requiring exponential or normalization operations in the forward pass. During training, a Softmax surrogate gradient enables optimization despite the discrete WTA selection. (3) Experimental evidence. Our ablation results support this interpretation. On GLUE, Hard WTA improves the average score from 55.4% (SSA) to 66.3%, approaching Softmax attention (66.8%). For causal language modeling, SSA fails to converge on QAT and CRT, whereas Hard WTA achieves 28.4% and 43.2%, respectively, closely matching Softmax (28.7% and 43.3%). We will incorporate this mechanistic analysis into the revised manuscript.

Weaknesses2: Energy consumption figures are based on computational complexity estimates; measurements on actual neuromorphic hardware have not yet been conducted.

R: Thank you for this important comment. Our reported energy consumption is theoretically estimated based on MAC/AC operation counts, following the established evaluation protocol adopted in prior Spiking Transformer studies, including Spikformer [1] and Spike-driven Transformer v2/v3 [2,3]. Specifically, we account for firing rates and simulation time steps, using the reference energy costs of 4.6 pJ per MAC and 0.9 pJ per AC under 45-nm technology. This protocol provides a consistent basis for comparing estimated arithmetic energy across models under the same assumptions. We acknowledge that these values represent theoretical estimates rather than direct measurements on neuromorphic hardware. We have explicitly stated this limitation in the manuscript.

[1] Spikformer: When spiking neural network meets transformer, ICLR 2023

[2] Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips, ICLR 2024.

[3] Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training, TPAMI 2025.

Questions 1: Is the WTA (Winner-Take-All) mechanism in the attention component of the paper applied across the sequence length? Will the WTA modeling approach still work if the sequence length increases?

R: Thank you for this insightful question. Similar to standard Softmax attention, our WTA mechanism operates along the key-sequence dimension of the attention map for each query token. Specifically, WSSA performs token selection across the entire sequence, whereas CWSSA restricts selection to causally valid preceding tokens through masking. In our experiments, Hard WTA performs well with a maximum sequence length of 512. The mechanism is also applicable to longer sequences, although selecting a single key token may limit information aggregation as the context length increases. Our additional observations indicate that Top-K WTA outperforms Hard WTA on longer sequences by preserving multiple relevant tokens while maintaining sparse, Softmax-free attention.

I appreciate the authors' detailed response. My concerns are mostly resolved. I would like to raise my score to 4.

Reviewer_XuryOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper replaces softmax attention with binary winner-take-all selection in spiking encoder and decoder language models. Training uses a softmax surrogate gradient, while inference uses hard selection.
Critical strengths
  • The matched attention ablation is more convincing to me than the cross-model comparisons: WTA comes close to softmax within the spiking architecture and improves substantially over the vision-oriented SSA baseline. Testing both masked and causal modeling is useful.
Critical weaknesses
  • The headline energy result mixes model sizes and relies on operation-count estimates rather than measured execution. The matched attention ablation supports replacing softmax with WTA within this spiking architecture, but does not establish the overall quality-energy advantage. An equal-size, equal-token comparison should account for QK score computation, winner selection, memory access and spike time steps; removing softmax does not remove quadratic score computation.

  • The encoder evaluation protocol also needs clarification: Section 4.1 says the models are fine-tuned on the GLUE dev set, leaving the training, selection and evaluation splits unclear. These limitations affect the central empirical claims. I recommend weak rejection.

Questions for authors
  • Can the authors reconcile the 66.3 Hard-WTA GLUE result in Table 4 with 65.7 in Table 10, specifying the neuron variant for each? Repeated runs would also help interpret the small WTA-versus-softmax differences.
2 comments
AuthorsOpenReview ↗

Response to reviewer Xury (1/2)

We sincerely thank the reviewer for the constructive feedback. We have carefully considered the concerns and provide our detailed responses below.

Weaknesses 1: The headline energy result mixes model sizes and relies on operation-count estimates rather than measured execution. The matched attention ablation supports replacing softmax with WTA within this spiking architecture, but does not establish the overall quality-energy advantage. An equal-size, equal-token comparison should account for QK score computation, winner selection, memory access, and spike time steps; removing softmax does not remove quadratic score computation.

R: Thank you for this important comment. Our reported values are theoretical energy consumption estimates following the MAC/AC-counting-based computational power consumption, consistent with the protocol adopted in prior Spiking Transformer studies, including Spikformer [1] and Spike-driven Transformer v2/v3 [2,3]. Specifically, the estimation accounts for Linear, QK computation, firing rates, and simulation time steps, using 45-nm reference energy costs of 4.6 pJ/MAC and 0.9 pJ/AC.

Energy–accuracy trade-off. To reduce the model-size confound, Tables 6 and 7 in the appendix of the manuscript additionally report a near-scale, equal-token comparison between WD-Spikingformer-1.0B and Qwen-1.5B. Both models use the same 0.1B-token pretraining budget and are evaluated under the same downstream pipeline:

ModelParametersTEnergy (mJ)QAT Avg. (%)CRT Avg. (%)
Qwen-1.5B1.5B-3398.333.244.3
WD-Spikingformer-1.0B1.0B4574.729.043.6

WD-Spikingformer-1.0B achieves comparable CRT accuracy (43.6% vs. 44.3%) with only 16.9% of Qwen's estimated computational energy, while exhibiting a 4.2-point accuracy gap on QAT. These results demonstrate a promising energy–accuracy trade-off within the evaluated regime. Due to computational resource constraints, 1.0B parameters is the largest scale evaluated through end-to-end training, and we will explicitly acknowledge this limitation.

We agree that neither Softmax nor WTA eliminates the quadratic attention-score computation: both retain the same $QK^{T} $ term, but they impose fundamentally different requirements on the underlying hardware.

Neuromorphic deployability. Our primary motivation for replacing Softmax is to enable spike-driven, neuromorphic-friendly attention. As discussed in Spikformer [1], Softmax introduces exponentiation, global summation, and division, which are not native to typical event-driven neuromorphic hardware and may require additional arithmetic units or approximation circuits. Moreover, its dense, real-valued output disrupts the spike-driven computation paradigm.

In contrast, Hard WTA replaces Softmax normalization with comparison-based competition and binary one-hot selection. Its operations can potentially be implemented through neuromorphic primitives such as lateral inhibition and event routing, allowing the selected value representations to be propagated sparsely. Thus, although WTA retains quadratic attention-score computation, it eliminates Softmax-specific exponentiation and normalization, providing a more hardware-compatible mechanism for spike-driven language modeling.

[1] Spikformer: When spiking neural network meets transformer, ICLR 2023

[2] Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips, ICLR 2024.

[3] Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training, TPAMI 2025.

AuthorsOpenReview ↗

Response to reviewer Xury (2/2)

We sincerely thank the reviewer for the constructive feedback. We have carefully considered the concerns and provide our detailed responses below.

Weaknesses 2: The encoder evaluation protocol also needs clarification: Section 4.1 says the models are fine-tuned on the GLUE dev set, leaving the training, selection and evaluation splits unclear. These limitations affect the central empirical claims. I recommend weak rejection.

R: Thank you for pointing out this ambiguity. The phrase “fine-tune them on the GLUE dev set” in Section 4.1 is a typographical error, rather than a description of our actual evaluation protocol. Following the standard practice of BERT [4], SpikeBERT [5], and SpikeLM [6], we independently fine-tune each pretrained model on the official GLUE training split and evaluate it on the corresponding development split. No development-set examples are used for gradient updates. We have corrected this expression and clarified the evaluation protocol in the revised manuscript. The corrected sentence reads: “We pretrain WE-Spikingformer and WE-Spikingformer$^\dagger$ on English Wikipedia using masked language modeling across eight GPUs. We then independently fine-tune each pretrained model on the official training split of each GLUE task and evaluate its performance on the corresponding development split.”

[4] Bert: Pre-training of deep bidirectional transformers for language understanding, NAACL-HLT 2019.

[5] SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation, Neural Networks 2025.

[6] SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms, ICML 2024.

Questions 1: Can the authors reconcile the 66.3 Hard-WTA GLUE result in Table 4 with 65.7 in Table 10, specifying the neuron variant for each? Repeated runs would also help interpret the small WTA-versus-softmax differences.

R: Thank you for raising this question. These values correspond to two different neuron variants rather than inconsistent results for the same configuration. The 66.3% result is obtained by WE-Spikingformer$^\dagger$ with T-LIF, whereas the 65.7% result is obtained by WE-Spikingformer with NI-LIF; both use Hard WTA with T=4.

Table 1 and Section 4.1 describe these two variants in detail: the caption specifies that WE-Spikingformer uses NI-LIF and WE-Spikingformer$^\dagger$ uses T-LIF, and Section 4.1 discusses their performance-efficiency trade-off. In our setup, T-LIF achieves a slightly higher score but requires 121 hours of pretraining, whereas NI-LIF achieves 65.7% with only 26 hours, motivating NI-LIF as the default because of its substantially faster spiking simulation.

Nevertheless, we agree that the neuron variant is not sufficiently explicit in the later ablation tables. We will again highlight the neuronal types used in the model results in Tables 4 and 10.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.