acceptodds
Under review as a conference paper at ICLR 2027

SpikeMoE: Brain-Inspired Competitive Routing for Flexible Spiking Mixture-of-Experts

Abstract

Spiking Neural Networks (SNNs) enable event-driven computation through biologically inspired dynamics at the neuronal scale, while Mixture-of-Experts (MoE) perform conditional computation through expert selection at the model scale. Integrating their strengths offers potential for flexible neural architectures. A key challenge, however, lies in designing an expert selection mechanism based on spiking activity. To address this, we introduce a spike-based k-WTA Router inspired by competition-inhibition observed in the hippocampal CA1 region. The router incorporates lateral inhibition and refractory period to select Top-K experts according to discrete spike counts. Building on this, we present SpikeMoE, a framework that integrates neuronal-scale spiking dynamics with model-scale expert selection. To address incomplete multisensory inputs in multimodal tasks, we further equip SpikeMoE with a two-stage missing-modality modeling module that combines empirical prototypes from an observed-modality pool with modality-specific learnable embeddings to construct missing-modality representations. Experiments on vision, language, and multimodal benchmarks demonstrate that SpikeMoE achieves state-of-the-art performance among the SNN baselines, matches or exceeds the performance of ANN counterparts, and maintains robustness across diverse missing-modality conditions. These results demonstrate a favorable trade-off between performance and energy efficiency, validating the integration of spiking dynamics with sparse expert computation and highlighting SpikeMoE as a promising approach to energy-efficient brain-inspired computing.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WCbROpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SpikeMoE, a spiking mixture-of-experts architecture that selects a small number of experts using spike counts from a k-winner-take-all router. For incomplete multimodal inputs, the authors also introduce a two-stage representation-completion procedure using observed-modality features and learnable embeddings. I found the integration of spike-based neuronal competition with conditional expert computation informative. The paper is also useful in showing how this design behaves across visual, language, and incomplete-modality settings.

Critical strengths

The strongest aspect of the paper is its empirical breadth. The method improves results on several spiking Transformer backbones and is also examined on language and incomplete-modality benchmarks. The gains over the closest alternatives vary in size, but the repeated positive results make the architectural idea worth considering. I also appreciate that the paper investigates inhibition and refractoriness rather than treating the router as a black box. Taken together, the study offers a useful contribution to the design of conditional computation in SNNs.

Critical weaknesses

Previous work has already explored mixtures of experts in spiking Transformers, and the manuscript does not yet provide a sufficiently controlled comparison to show when competitive spiking dynamics are preferable to a simpler sparse router. The use of an auxiliary balancing objective also complicates the explanation of why expert allocation improves. A more focused comparison would be more informative than extending the evaluation to additional datasets.

Questions for authors
  1. How does the proposed router compare with a conventional sparse or Top-K router when the experts, training objective, and computational budget are kept comparable? In particular, it would be helpful to distinguish the effect of competitive spiking dynamics from that of the load-balancing objective already used in training.
  2. Which aspects of the proposed routing design are most important for the observed gains over earlier spiking-MoE work? A clearer discussion of the closest prior methods and of the conditions under which the gains are largest would help position the contribution.
3 comments

Dear Reviewer WCbR,

Thank you for your careful review. We address your questions with controlled comparisons and ablation studies. We will include these additional experiments in the revised manuscript.

Weakness 1 & Weakness 2 & Question 1

Comparison with conventional routers under a matched compute budget

To test when competitive dynamics improve sparse routing, we hold the SNN backbone, expert architecture and capacity, active parameter count, training objective, and compute budget fixed. We use the same weight for Lbalance\mathcal{L}_{\mathrm{balance}} across all variants. We compare floating-point Softmax Top-K, Linear Top-K without exponential normalization, Spike-count Top-K without competition, and k-WTA variants with lateral inhibition and refractoriness.

MethodF1AUCGini↓
Softmax42.42 ± 0.1266.05 ± 0.670.14
Linear Top-K41.42 ± 0.1764.09 ± 0.440.22
Spike-count Top-K39.52 ± 0.2157.45 ± 0.320.34
+lateral inhibition42.50 ± 0.3466.12 ± 0.560.42
+refractory period41.37 ± 0.7166.02 ± 0.490.35
Full k-WTA43.54 ± 0.0367.30 ± 0.710.12

The Gini coefficient [1] measures expert-load imbalance across the dataset. Lower values indicate more balanced expert use.

Backbone and routerParams (M)Active params (M)Estimated energy (mJ)Top-1 (%)
SDT-8-384+Dense FFN16.8116.813.9072.28
SDT-8-384+Linear Top-K16.0311.803.6672.56
SDT-8-384+Softmax Top-K16.0311.803.7874.35
SDT-8-384+k-WTA (ours)16.0311.803.5474.52
Spikingformer-8-384+Dense FFN16.8116.814.6972.45
Spikingformer-8-384+Linear Top-K16.0311.804.3872.63
Spikingformer-8-384+Softmax Top-K16.0311.804.5574.22
Spikingformer-8-384+k-WTA (ours)16.0311.804.2774.37
  • Gains from simple sparse routing. Linear Top-K yields small gains over Dense FFN. It improves ImageNet-1K accuracy by 0.18–0.28 percentage points and CMU-MOSI F1 by 0.13 points. Its dataset-level expert-load Gini on CMU-MOSI is 0.22.
  • Gains from competitive dynamics. Spike-count Top-K alone yields an F1 of 39.52 and a Gini of 0.34. Adding lateral inhibition and refractoriness improves on both Linear Top-K and Spike-count Top-K. Full k-WTA also outperforms Softmax Top-K without floating-point exponentiation. It achieves the lowest estimated inference energy at 11.80M active parameters on both vision backbones and the most balanced expert use on CMU-MOSI, with a Gini of 0.12. Furthermore, it can be observed that introducing lateral inhibition alone is insufficient to resolve the issue, which is consistent with the conclusion we reach below.

Reference:

[1] Chen, Lizhang, et al. "ϕ\phi-Balancing for Mixture-of-Experts Training." Forty-third International Conference on Machine Learning.

Separating competitive dynamics from the load-balancing loss

We conduct a 2×22\times2 factorial ablation and remove each component individually on CMU-MOSI at 50% missingness. The first four rows vary competitive dynamics and Lbalance\mathcal{L}_{\mathrm{balance}} independently. The last two rows remove one component at a time while retaining the loss.

DynamicsLbalance\mathcal{L}_{\mathrm{balance}}F1AUCGini↓
√√43.5467.300.12
√×41.1065.850.28
×√39.5257.450.34
××30.2755.600.59
w/o lateral inhibition√41.3766.020.35
w/o refractory period√42.5066.120.42
  • Lateral inhibition and refractoriness act within the internal routing window tmax⁡t_{\max}. Without refractoriness, the neuron with the highest initial membrane potential can fire at every simulation step and dominate the spike count for that sample. Even with Lbalance\mathcal{L}_{\mathrm{balance}}, removing refractoriness raises Gini to 0.42 and lowers F1 to 41.50. Competitive dynamics alone yield a lower Gini and higher F1 than the loss alone (0.28 vs. 0.34 and 41.10 vs. 39.52). Removing lateral inhibition causes a larger drop in F1, while removing the refractory period causes a larger increase in expert-load Gini. Keeping lateral inhibition may help distinguish candidate experts by their responses to the input. This could explain the smaller F1 drop when the refractory period is removed. Without a refractory period, the first expert to fire can fire again sooner and continue to suppress its peers through lateral inhibition. This can concentrate spike counts in the dominant expert. Together, the two mechanisms preserve competition while limiting repeated firing by the same expert within each routing window.
  • Lateral inhibition sharpens competition within a sample, while some experts may still be selected more often across samples. The load-balancing loss addresses this imbalance at the aggregate level. With competitive dynamics enabled, adding the loss reduces Gini from 0.28 to 0.12. The two mechanisms thus improve expert use at complementary scales.

Weakness 1 & Question 2

Relation to prior work. Zhou et al. introduced expert mixtures into Spiking Transformers through SEMM. It uses spike sequences to gate expert outputs element by element, allocating computation across attention heads and feed-forward channel groups. SpikeMoE assigns a routing neuron to each candidate expert. Lateral inhibition and refractoriness shape the accumulated spike counts. The router then selects the Top-K experts and uses their normalized counts to weight their outputs. Our contribution is to use explicit competitive dynamics for expert selection and weighting in spiking MoE.

Roles of lateral inhibition and refractoriness. In the CMU-MOSI ablation with the load-balancing loss enabled, removing either component reduces performance and increases Gini. Removing refractoriness has the larger effect. F1 falls from 43.54 to 41.50, and Gini rises from 0.12 to 0.42. The ablation measures the effect of each component in this setting. The comparison with SEMM measures the difference between the two complete designs.

Configurations with larger gains. Among the six vision configurations in Table 1, we focus on the gains on Spikformer-8-384 and Spikingformer-8-384. To control for hyperparameter differences and assess run-to-run variation, we compare k-WTA with SEMM on ImageNet-1K across three seeds (42, 43, and 44). We match the expert architecture, parameter count, loss, and training protocol.

BackboneSEMM Top-1 (%)k-WTA Top-1 (%)Gain (pp), (t-test, p<0.005)
Spikformer-8-38472.86 ± 0.1274.12 ± 0.06+1.26
Spikingformer-8-38473.58 ± 0.1074.37 ± 0.01+0.79

Dear Reviewer WCbR,

Thank you again for your detailed feedback and suggestions. We have uploaded the revised manuscript. Appendix A.6 (Tables 11–13) presents controlled comparisons with conventional routers on ImageNet-1K and CMU-MOSI. Sec. 5.3 (Tables 5–6) examines the roles of lateral inhibition and refractoriness and separates the effect of competitive dynamics from that of the load-balancing loss. Sec. 2 explains how k-WTA differs from SEMM, and our rebuttal reports the three-seed ImageNet-1K comparison with SEMM.

Reviewer_3H7FOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The authors introduce an expert selection module based on competitive inhibition, with learned inhibition weights between experts. Expert neurons are run for some number of internal timesteps, producing spikes with some refractory period. Each spike inhibits other experts. Then, the top K experts are selected based on spike rate, and weights are assigned based on the spike rate.
  • They also introduce a strategy to deal with missing modality in a multimodality scenario, in two stages:
    • stage 1: the missing modality is replaced by some average of values randomly sampled from the training set.
    • stage 2: when stage 1 cannot be applied (the exact criteria are unclear to me, see question below), the missing modality is replaced by a learned embedding.
  • They perform experiments and compare accuracy and energy efficiency, showing that they outperform existing baselines.
Critical strengths
  • The expert selection module with competitive inhibition is an interesting idea. It may hold potential for future biologically plausible or energy efficient models.
  • Results and analyses are complete. Model comparison and ablations are interesting.
Critical weaknesses
  • related works: explanation of related works is missing, including spiking transformer and variants. For instance, how does their k-WTA router differ from the expert selection mechanism from zhou et al, "Spiking Transformer with Experts Mixture"? It would also be good to cite older spiking WTA works, e.g. Nessler et al 2013 PLOS CB or other works from the Maass group.
  • wording: weak transitions, the text is hard to parse. - transition: "A central challenge is therefore to perform effective expert selection directly based on spiking activity. At the perception level, multisensory inputs in real-world scenarios are frequently incomplete..." - hard to parse: "We propose a two-stage missing modality modeling strategy (Figure 3) that handles missing modalities through the adaptive fusion of a learnable modality pool and learnable embeddings." - acronym WTA used many time before explaining that it means winner-take-all.
  • inconsistencies, notation - eq 4 should sum to m right? - eq 11: sum of t=1 to t? - what is n in eq 13? - Dimension of X: what are N and C? It seems like they only explain it in the appendix
  • explanations - stage 2 explanation: - "When the modality pool cannot provide a suitable imputed modality (e.g., when the pool is empty, data quality is poor, or no matching contextual is available), a learnable embedding is used as the imputed modality." what criterion exactly? what do they mean by "matching contextual"? - following sentences: what do they mean by number of modalities in the modality pool? It seems that they refer to the number of possible combinations of modalities. What do they mean by cascaded embedding?
  • figure captions - figure 1: multi-head spiking self attention? - figure 4: unclear to me, see my question
Questions for authors
  • what does figure 4 show exactly? What is the router mask? Router of 0? Dynamic sparsity?
  • what criteria determine when stage two is applied in section 4.2? I find the explanation a bit vague.
  • how does it the proposed router functionally compare to softmax in terms of resulting expert weights? I'm not asking for additional analyses, but some intuitions.
  • in the model results indicated with a star in tables 1 and 2 (indicating your reproduced results), how do they compare with original results? Also, what explains the large difference between LIF-BERT and LIF-BERT* in Table 2 (54.9 vs 34.6)?
4 comments

Dear Reviewer 3H7F,

Thank you for your careful review. We address your comments below with ablation results and implementation details. We will include the additional experiments and mathematical clarifications in the revised manuscript.

Weakness 1

We will expand Related Work to cover Spiking Transformers and their variants, including Spikformer, Spike-driven Transformer, Spikingformer, and SEMM. We will also cite foundational studies on spiking winner-take-all circuits and lateral inhibition, including Nessler et al. (PLOS Computational Biology, 2013) and Maass (Neural Computation, 2000). These references will clarify the biological and theoretical basis of our routing mechanism.

SEMM and SpikeMoE differ in how they assign experts.

MethodRouting and expert assignment
SEMMThe router and experts produce spike sequences that interact through element-wise gating. EMSA and EMSP assign experts at the attention-head and channel-group levels, respectively. Sparsity varies with the router’s spike activity.
SpikeMoEEach candidate expert has a corresponding routing neuron. Lateral inhibition and refractoriness shape spike counts within an internal simulation window. The router selects the Top-K experts by count and aggregates their outputs using normalized counts.

Our contribution is to use competitive dynamics for expert assignment in spiking MoE. On ImageNet-1K, we match the backbone, expert capacity (four experts, Top-K = 2), and active parameter count (11.80M). k-WTA achieves 74.52% on SDT and 74.37% on Spikingformer, compared with 73.93% and 73.58% for SEMM. It also has lower estimated energy costs.

Weakness 2

We agree that the introduction should explain how missing modalities affect routing. We will revise the transition as follows.

At the perception level, missing modalities reduce the input information available to the spiking router and can affect expert selection. We therefore complete missing modality representations before routing. For each missing modality, we estimate a prototype from training samples in which that modality is observed. When enough samples are available, we fuse the prototype with a modality-specific learnable embedding. Otherwise, we use the learned embedding alone.

We will define k-Winner-Take-All (k-WTA) at first use and replace the ambiguous phrase "a learnable modality pool" with "an observed-modality pool constructed from training samples". We will also explain how we construct the pool and learn the modality embeddings and fusion weights.

Weakness 3

Thank you for checking the notation and indices. We will make the following changes in the revised manuscript.

  • Summation in Eq. 4. The original equation sums over the KK experts selected for the current input, but its notation could suggest a fixed set of experts indexed from 1 to KK. We will write y(x)=∑e=1Ewe(x)fe(x)y(x)=\sum_{e=1}^{E}w_e(x)f_e(x), where EE is the total number of candidate experts, denoted by mm in the original text. Unselected experts have zero weights, so the sum is equivalent to summing over the selected set.
  • Time index in Eq. 11. We use ss for the internal step index and TWTAT_{\mathrm{WTA}} for the number of internal steps. The spike count of expert ee for routing unit uu is cu,e=∑s=1TWTASu,e[s]c_{u,e}=\sum_{s=1}^{T_{\mathrm{WTA}}}S_{u,e}[s]. We retain TT for the number of backbone time steps.
  • Sample index in Eq. 13. In the revised manuscript, ii indexes modalities and nn indexes samples. We set oi,n=1o_{i,n}=1 if modality ii is missing from sample nn and oi,n=0o_{i,n}=0 if it is observed. Each sample must have at least one observed modality. We therefore revise Eq. (14) to require ∑i=1M(1−oi,n)≥1\sum_{i=1}^{M}(1-o_{i,n})\geq 1 for every sample nn. We construct the pool for modality ii from training samples jj with oi,j=0o_{i,j}=0.
  • Input dimensions NN and CC. At the first definition of X∈RT×B×N×CX\in\mathbb{R}^{T\times B\times N\times C} in Sec. 4.1, we will specify that TT is the number of backbone time steps, BB is the batch size, NN is the number of tokens or image patches, and CC is the number of feature channels per position.

Weakness 4 & Question 2

Thank you for pointing out the ambiguity in Sec. 4.2. We have clarified how we construct the modality pool, when the model uses or combines the two stages, and how we concatenate the resulting representations.

Modality pool and conditions for using Stage 2. We have clarified the phrase “matching contextual” by defining the modality pool as the set of representations from training samples in which the target modality is observed. Let ii index modalities and nn index samples. We set oi,n=1o_{i,n}=1 when modality ii is missing from sample nn, and oi,n=0o_{i,n}=0 when it is observed. The pool for modality ii is Pi=Mi,j:j∈Dtrain,oi,j=0,\mathcal P_i=\\{M_{i,j}:j\in\mathcal D_{\mathrm{train}},\\ o_{i,j}=0\\}, where Mi,j∈RdM_{i,j}\in\mathbb R^d is the observed representation of modality ii in training sample jj. When ∣Pi∣≥K|\mathcal P_i|\geq K, we randomly sample KK entries with training sample indices j1,…,jKj_1,\ldots,j_K and compute the empirical prototype Mˉi=1K∑k=1KMi,jk\bar M_i=\frac{1}{K}\sum_{k=1}^{K}M_{i,j_k}.

For a missing modality, the model uses the modality-specific learnable embedding elearnie_{\mathrm{learn}}^i when ∣Pi∣<K|\mathcal P_i|<K. When ∣Pi∣≥K|\mathcal P_i|\geq K, it combines the Stage 1 prototype with the Stage 2 embedding using a coefficient αi∈(0,1)\alpha_i\in(0,1) learned end to end. Eq. (15) of the revised manuscript gives the resulting representation

\begin{cases} \mathrm{Encoder}^i(n), & \text{if } o_{i,n}=0,\\\ e_{\mathrm{learn}}^i, & \text{if } o_{i,n}=1 \text{ and }|\mathcal P_i|<K,\\\ \alpha_i\bar M_i+(1-\alpha_i)e_{\mathrm{learn}}^i, & \text{if } o_{i,n}=1 \text{ and }|\mathcal P_i|\geq K. \end{cases}$$ Stage 1 provides an empirical prior for the target modality, and the learned gate controls how much the model relies on this prior during imputation. **Number of modality combinations.** Your interpretation is correct. The original text referred to the possible patterns of modality availability. With $M$ modalities, there are $2^M-1$ nonempty patterns, of which $\sum_{k=1}^{M-1}\binom{M}{k}=2^M-2$ are incomplete. The revised manuscript defines the pool $\mathcal P_i$ for each modality and uses its size $|\mathcal P_i|$ to determine when Stage 2 is used alone. **Meaning of “cascaded embeddings”.** In the revised manuscript, $h_n=[e_n^1,\ldots,e_n^M]$ concatenates the modality representations of sample $n$ along the sequence dimension. The sequence includes encoded representations of observed modalities and imputed representations of missing modalities from Eq. (15). Multi-head spiking self-attention (MSSA) then models interactions across this sequence. We have replaced “cascaded embeddings” with “concatenated modality embeddings.” ## Weakness 5 & Question 1 **Figure 1.** MSSA stands for Multi-head Spiking Self-Attention. Each head applies attention to its spiking Q, K, and V representations. We then concatenate and project the head outputs. We will spell out MSSA in the caption and refer readers to Appendix A.2.2 for the equations. The revised caption will also identify Fig. 1(a) as the overall architecture and Fig. 1(b) as a comparison between a standard spiking neuron and a routing expert neuron. **Figure 4.** The figure shows patch-level expert assignments for the same image across backbone time steps. The router selects two of four candidate experts for each routing decision. For the expert (e) shown in a panel, the binary mask is $m_{t,n,e}=\mathbf{1}[e\in\mathcal S_{t,n}]$. A black patch has mask value 0, meaning that it is not assigned to that expert. A patch shown in its original colors has mask value 1. “Dynamic sparsity” means that the selected experts and their spatial coverage can vary across patches and backbone time steps. Each decision selects exactly ($K=2$) experts. We will label the columns with backbone time steps, identify the experts shown in each row, and specify the layer being visualized. We will also replace “router of 0” with “binary routing-mask value of 0” to distinguish the mask from the continuous aggregation weights.

Question 3

Both routes can set the weights of unselected experts to zero after Top-K selection. They differ in how they compute expert scores and assign relative weights to selected experts.

Softmax uses continuous logits zez_e. For two selected experts, the weight ratio is exp⁡(zi−zj)\exp(z_i-z_j). k-WTA first generates spike counts cec_e through inhibition and refractoriness. It then normalizes the selected counts, giving a weight ratio of (ci+ε)/(cj+ε)(c_i+\varepsilon)/(c_j+\varepsilon).

For example, if two selected experts fire six and two times, their k-WTA weights are approximately 0.75 and 0.25 for a small ε\varepsilon. Competition can change the accumulated counts, which can change both the expert ranking and the weight ratios. The input and routing dynamics determine how concentrated the weights are.

Question 4

Thank you for pointing out that the asterisks in both tables could be misinterpreted as the same type of reference marker. We will clarify their respective meanings:

  • Table 1. The asterisks in the SEMM rows appear in the energy column. Zhou et al.’s ImageNet-1K table provides the reference Top-1 accuracies. We estimated the starred energy values using the standard SNN energy estimation protocol.
  • Table 2. The scores of 34.6 for LIF-BERT*, 34.7 for PSN-BERT*, and 54.9 for LIF-BERT match the corresponding results in Table 3 of SpikeLM. Appendix A.1 of SpikeLM compares the two LIF-BERT implementations directly. The 34.6 result comes from a SpikingJelly implementation that uses the original neuron settings and an arctangent surrogate gradient. The 54.9 result comes from Xing et al.’s [1] LIF implementation, which uses a straight-through estimator (STE) for backpropagation. Both use binary LIF spikes in the forward pass. The 20.3-point gap is the difference between the two configurations reported in SpikeLM. We will label these rows “LIF-BERT (arctan surrogate)” and “LIF-BERT (STE)” to make the distinction clear.

Reference: [1] Xing et al. Spikelm: Towards general spike-driven language modeling via elastic bi-spiking mechanisms

Dear Reviewer 3H7F,

Thank you again for your detailed feedback and suggestions. We have uploaded the revised manuscript.

  • Sec. 1 defines k-WTA at first use and explains how missing modalities affect routing. Sec. 2 discusses SEMM and earlier spiking WTA work.
  • Sec. 3.2 revises Eq. (4) to clarify the expert summation. Sec. 4.1 (Eqs. 11–12) describes spike counting and expert weights, and Sec. 4.2 (Eqs. 13–15) clarifies the modality pool, Stage 2 fallback, and embedding concatenation.
  • The updated captions for Figs. 1 and 4 clarify the architecture and routing masks. Appendix A.2.2 gives the MSSA equations. Table 2 (Sec. 5.2.1) now labels the two LIF-BERT variants by their backpropagation methods. Our rebuttal explains the starred energy entries in Table 1.
Reviewer_DqEVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SpikeMoE, a framework that combines spiking neural networks (SNNs) with mixture-of-experts (MoE) architectures. Its main contribution is a spike-based k-Winner-Take-All (k-WTA) router inspired by biological competition-inhibition mechanisms. The router uses spike counts for expert selection and incorporates lateral inhibition and refractory dynamics to mitigate expert collapse. Additionally introducing a two-stage missing-modality modeling strategy based on a modality pool and learnable embeddings. Experiments on ImageNet-1K, GLUE, and several multimodal benchmarks show improvements over existing SNN backbones and competitive performance against selected ANN baselines.

Critical strengths

Combining neuron-level event-driven computation with expert-level conditional computation is a promising direction for energy-efficient neural architectures. And the use of lateral inhibition and refractory dynamics provides an interpretable alternative to conventional softmax-based expert selection. Additionally, the authors evaluate the framework on vision, language, and multimodal benchmarks, demonstrating its potential applicability beyond a single task.

Critical weaknesses
  1. My primary concern is that the experimental comparisons do not adequately isolate the contribution of the proposed k-WTA router. In Table 1, the authors primarily compare standard SNN backbones, including Spikformer, Spike-driven Transformer, and Spikingformer, against their corresponding variants with the proposed MoE module. However, replacing a dense FFN with an MoE layer changes the network architecture and parameterization. Consequently, the observed accuracy improvements cannot be attributed solely to the proposed spike-based competitive routing mechanism. Crucially, the paper lacks a direct comparison against a conventional MoE router implemented within the same SNN backbone. Such a comparison is necessary to determine whether the improvements arise from the MoE architecture itself or from the proposed k-WTA mechanism. At minimum, I would expect the following controlled comparisons:
  • ANN Transformer with dense FFN and conventional MoE.
  • SNN Transformer with dense FFN.
  • SNN Transformer with conventional Top-K/softmax MoE routing.
  • SNN Transformer with the proposed k-WTA routing. These comparisons should control for backbone architecture, expert capacity, active parameters, computational budget, and training configuration. Although the multimodal experiments include FuseMoE and other ANN baselines, these methods differ substantially in architecture and implementation. They therefore do not provide a controlled comparison of routing mechanisms. Without these experiments, the paper demonstrates that the proposed module can improve SNN backbones, but does not convincingly establish that its routing mechanism is superior to conventional MoE approaches.
  1. A central motivation of the paper is that lateral inhibition and refractory dynamics mitigate expert collapse and promote balanced expert utilization. However, Appendix A.2.6 introduces an explicit load-balancing loss, which is included in the overall training objective. This raises a fundamental attribution issue: to what extent is balanced expert utilization achieved through the proposed biological competition mechanisms, rather than through conventional load-balancing regularization? Table 5 demonstrates that removing lateral inhibition or the refractory period degrades performance, but does not isolate the independent contribution of these mechanisms from that of the balancing objective. I would expect a factorial ablation varying both the competition mechanisms and the load-balancing loss, together with quantitative measurements of expert utilization, routing entropy, and load variance. Furthermore, the refractory period primarily constrains repeated spike firing within a routing simulation. It does not necessarily prevent the same expert from being repeatedly selected across different inputs. Therefore, the claim that refractory dynamics mitigate expert collapse requires more rigorous dataset-level evidence.

  2. Energy efficiency is a major motivation of SpikeMoE, but the presented evidence is largely based on theoretical operation and energy estimates rather than measurements on actual neuromorphic hardware. Moreover, the proposed router introduces additional simulation time steps, recurrent membrane-state updates, and learnable expert-to-expert inhibition. These operations may introduce nontrivial overhead, particularly as the number of experts increases. Although the appendix reports GPU latency measurements, the evaluation does not sufficiently establish the end-to-end efficiency advantage of the proposed approach against matched ANN-MoE and SNN-MoE architectures. In addition, the inhibition matrix scales with the number of experts, potentially introducing quadratic computational or memory overhead under a dense implementation. A more convincing evaluation would report measured inference latency, energy consumption, and memory overhead under controlled hardware settings, while explicitly accounting for the additional routing simulation and expert-selection operations. The paper should also distinguish theoretical spike-operation savings from practical end-to-end energy savings.

  3. The two-stage missing-modality modeling strategy appears relatively independent of the proposed spike-based routing mechanism. In particular, Stage 1 constructs a missing-modality representation by averaging samples from a modality pool. However, the paper does not clearly explain how the sampled representations are conditioned on the observed modalities or how semantically compatible representations are selected. As a result, it is unclear whether this procedure truly captures cross-modal dependencies or simply provides an empirical prior over missing representations. Similarly, the transition between the modality-pool mechanism and the learnable embedding fallback is insufficiently specified. The authors should compare this strategy against simpler imputation baselines, including zero filling, learned missing-modality tokens, and unconditional mean embeddings, under the same backbone and routing configuration. More importantly, the paper would benefit from a clearer explanation of how missing-modality modeling interacts with the proposed competitive router, rather than presenting two largely separate contributions within one framework.

Questions for authors
  1. Can the authors compare k-WTA against conventional Top-K/softmax MoE routing within exactly the same SNN backbone, controlling for expert counts, active parameters, and training budgets?
  2. What are the independent contributions of lateral inhibition, refractory dynamics, and the explicit load-balancing loss? Can the authors provide ablations with and without the balancing objective?
  3. Is expert selection performed per token, per sample, or across temporal sequences? How are the routing scores computed from the input tensor?
  4. How does end-to-end inference latency and measured energy scale with the number of experts and WTA simulation steps?
  5. How are modality-pool samples selected to ensure semantic compatibility with the observed modalities, and what determines when Stage 2 is used?
4 comments

Dear Reviewer DqEV,

Thanks for your careful review and constructive feedback. We address your questions below, and these points will be incorporated into the revised manuscript.

Weakness 1 & Question 1

We conducted controlled experiments by keeping the backbone, expert capacity, loss function, and hyperparameters consistent:

BackboneRouterTotal Params (M)Active Params (M)Energy (mJ)Acc (%)
SDT-8-384Dense FFN16.8116.813.9072.28
SDT-8-384Linear Top-K16.0311.803.6672.56
SDT-8-384Softmax16.0311.803.7874.35
SDT-8-384SEMM16.0511.823.7573.93
SDT-8-384k-WTA16.0311.803.5474.52
Spikingformer-8-384Dense FFN16.8116.814.6972.45
Spikingformer-8-384Linear Top-K16.0311.804.3872.63
Spikingformer-8-384Softmax16.0311.804.5574.22
Spikingformer-8-384SEMM16.0511.824.4773.58
Spikingformer-8-384k-WTA16.0311.804.2774.37

The table above compares different MoE routers across two mainstream backbones (all with 4 Experts, Top-k = 2).

ModelRouterF1AUC
ANN-TFDense FFN42.34±0.1064.57±0.31
ANN-TFSoftmax43.40±0.1366.24±0.15
SNN-TFDense FFN41.29±0.5363.03±0.22
SNN-TFSoftmax42.42±0.1266.05±0.67
SNN-TFLinear Top-K41.42±0.1764.09±0.44
SNN-TFSpike-count Top-K39.52±0.2157.45±0.32
SNN-TFk-WTA43.54±0.0367.30±0.71

The table above fixes the architecture, completion strategy, expert number (6 Experts, Top-k = 2), and training pipeline.

  • ANN Transformer with dense FFN and conventional MoE: Replacing Dense FFN with Softmax in the ANN-Transformer improves F1 and AUC on CMU-MOSI by 1.06 and 1.67, respectively. This establishes a baseline for conditional computation and multi-expert gains in floating-point models, while avoiding confounding factors from encoder or scale differences in external baselines.
  • SNN Transformer with dense FFN: In vision experiments, both the total and active params of the MoE variants are lower than those of the SNN-FFN baseline, accompanied by reduced energy consumption. This proves that performance gains stem from sparse expert activation rather than parameter expansion.
  • SNN Transformer with conventional Top-K/softmax MoE routing: Under the same backbone, Linear Top-K yields only marginal improvements over Dense FFN, showing that merely introducing an MoE structure does not guarantee ideal gains. In contrast, while Softmax achieve higher performance, it introduces floating-point exponential and division operations—which are incompatible with SNNs—thereby leading to increased energy consumption.
  • SNN Transformer with the proposed k-WTA routing: Under identical param budgets, the k-WTA Router achieves optimal performance. Removing the proposed dynamics degrades it into Spike-count Top-K, which suffers from performance drops due to single-expert monopoly. With these dynamics enabled, the k-WTA Router not only outperforms SEMM on vision tasks, but also surpasses the Softmax Router of the same backbone across both vision and multimodal benchmarks.

Weakness 2 & Question 2

These two mechanisms operate complementarily at different scales:

  • Temporal dynamics act within the routing time-steps. Without refractory period, the neuron with the highest initial membrane potential fires continuously, causing spike counts to be monopolized by a single expert. The refractory period forces fired neurons to reset, allowing sub-optimal experts to activate in subsequent time steps and ensuring collaborative representations among Top-K experts.
  • Dataset-level load balancing: Since lateral inhibition sharpens competition within a single sample, frequent selection of the same expert can still occur across different samples. The load-balancing loss (Lbalance\mathcal{L}_{balance}, Eq. 39) explicitly constrains the variance of expert activation probabilities and routing entropy, maintaining global expert utilization balance at the batch and dataset levels.
DynamicLbalance\mathcal{L}_{balance}F1AUCGini↓
√√43.5467.300.12
√×41.1065.850.28
×√39.5257.450.34
××30.2755.600.59
ConfigF1AUCGini↓
Full SpikeMoE43.5467.300.12
w/o Lbalance\mathcal{L}_{balance}41.1065.850.28
w/o lateral inhibition41.3766.020.35
w/o refractory period42.5066.120.42
  • The results show that removing the load-balancing loss while retaining neural dynamics causes only a slight performance drop, and the Gini coefficient remains significantly better than the configuration with the load-balancing loss alone but no dynamics. This proves that neural dynamics provide a structural inductive bias that complements the load-balancing loss.
  • With Lbalance\mathcal{L}_{balance} enabled, removing lateral inhibition causes the larger drop in F1, while removing the refractory period causes the larger increase in the Gini coefficient of expert load. One possible explanation is that lateral inhibition helps distinguish candidate experts based on their responses to the input. This may preserve some routing selectivity and limit the drop in F1. Without a refractory period, the first expert to fire can fire again sooner and continue to suppress other experts. This can concentrate spikes in the dominant expert. Combining both mechanisms preserves competition among experts while limiting repeated firing by the same expert within each routing window.

Weakness 3 & Question 4

  • Complexity analysis of the lateral inhibition matrix: Although L∈RE×EL \in \mathbb{R}^{E \times E} and M∈RE×EM \in \mathbb{R}^{E \times E} scale with the number of experts in theory, the number of experts EE (4,6,84, 6, 8) is far smaller than the N×CN \times C. Thus, L⊙ML \odot M and subsequent inhibition involve only E2E^2 scalar operations (16 to 64 operations per step), triggered event-driven by sparse spike vectors S[t−1]S[t-1], yielding negligible compute and memory overhead.
  • Measured latency scaling (Appendix A.5 and Figure 11): Evaluated on a single NVIDIA RTX 4090 GPU (BS=64):
    • Expert count scaling: As experts increase from 4 to 6 and 8, since Top-K activation fixes active experts to 2, backbone and expert layer latencies remain constant. The k-WTA router accounts for only 3.7% to 4.4% of total latency (inhibitory operations add only ~0.1 ms).
    • Time step scaling: Increasing internal time steps tmaxt_{\text{max}} from 8 to 20 increases router execution time by only ~1 ms, keeping total latency stable. Moreover, lateral inhibition-induced spike sparsity reduces downstream expert activation frequency, further reducing SOPs and energy consumption compared to the baseline.
  • Theoretical energy and hardware efficiency: As noted in Appendix A.2.4, energy figures in Tables 1, 2, and 6 follow standard SNN protocols (Yao et al., 2023), estimating theoretical energy based on MAC and AC on 45nm CMOS. We provide hardware comparison results on a single RTX 4090 GPU (BS=64):
ModelLatency (ms/batch)Peak VRAM (MB)Power Consumption (W)
ANN-Softmax-MoE58.628420315.4
SpikeMoE (Softmax)51.407610268.2
SpikeMoE (k-WTA Router)46.807580252.5

Reference: Yao et al. Spike-driven Transformer.

Weakness 4 & Question 5

Thank you for your feedback. We clarify and supplement the following:

  • Modality pool and use of Stage 2. For each target modality ii, we build a training pool Pi={Mi,j:j∈Dtrain, oi,j=0}\mathcal P_i=\{M_{i,j}:j\in\mathcal D_{\mathrm{train}},\ o_{i,j}=0\}, where oi,j=0o_{i,j}=0 indicates that modality ii is observed in training sample jj. When ∣Pi∣≥K|\mathcal P_i|\geq K, Stage 1 randomly samples KK representations and averages them to form the empirical prototype Mˉi=1K∑k=1KMi,jk\bar M_i=\frac{1}{K}\sum_{k=1}^{K}M_{i,j_k}. For a missing modality, the model uses the Stage 2 embedding elearnie_{\mathrm{learn}}^i alone when the pool contains fewer than KK entries. Otherwise, it combines the prototype and embedding using a coefficient αi∈(0,1)\alpha_i\in(0,1) learned end to end. Eq. (15) in the revised manuscript gives the resulting representation.
\begin{cases} \mathrm{Encoder}^i(n), & \text{if } o_{i,n}=0,\\\ e_{\mathrm{learn}}^i, & \text{if } o_{i,n}=1 \text{ and }|\mathcal P_i|<K,\\\ \alpha_i\bar M_i+(1-\alpha_i)e_{\mathrm{learn}}^i, & \text{if } o_{i,n}=1 \text{ and }|\mathcal P_i|\geq K. \end{cases}$$ - **Modality sample selection and cross-modal semantic compatibility:** Stage 1 randomly samples and computes the mean prototype $\bar{M}_{n,m}$ from the modality pool, providing an empirical distributional prior. Fine-grained cross-modal dependencies are learned via the MSSA layer after concatenating imputed and observed representations. UMAP visualizations (Fig.5 and Fig.9) confirm that imputed and observed modalities overlap closely in the feature manifold. - **Comparison with simplified baselines:** Figure 7 compares the full two-stage fusion method with three ablations at a 50% modality missing rate. The ablations use zero filling (both stages removed), a learned modality-specific embedding alone (Stage 2 only), or a sampled mean prototype alone (Stage 1 only). The Stage-1-only variant averages $K$ representations randomly sampled from the training pool for the target modality. A comparison with a fixed mean computed over the entire pool remains open.The full method achieves the highest F1 and AUC on both CMU-MOSI and CMU-MOSEI. Compared with the strongest ablation (Stage 1 only), it improves F1 and AUC by **3.90** and **3.14** percentage points on CMU-MOSI and by **3.29** and **3.60** percentage points on CMU-MOSEI, respectively. Figure 10(b) in the appendix shows the same trend on UrbanSound8K-AV. In the tested settings, combining the sampled prototype with the learned embedding yields better performance than using either representation alone.

Dear Reviewer DqEV,

Thank you again for your detailed feedback and suggestions. We have uploaded the revised manuscript.

  • Appendix A.6 (Tables 11–13) presents the controlled router comparisons, and Sec. 5.3 (Tables 5–6) examines the effects of competitive dynamics and load balancing.
  • Sec. 4.1 and Appendix A.7 explain how routing scores and expert weights are computed. Appendix A.5 reports latency scaling, and Appendix A.2.4 describes the energy estimates. The additional GPU comparison is in our rebuttal.
  • Sec. 4.2 (Eq. 15) clarifies modality-pool sampling and the Stage 2 fallback. Sec. 5.3 (Fig. 7) and Appendix A.4.2 (Fig. 10b) report the imputation ablations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response: Additional Experiments and Revisions

Dear Area Chair and Reviewers,

Thank you again for your detailed feedback and suggestions. We added controlled experiments and clarified several aspects of SpikeMoE during the rebuttal.

  1. Contribution of competitive routing. We compared k-WTA with Linear Top-K and Softmax Top-K under matched SNN-MoE configurations. On CMU-MOSI at 50% modality missingness, k-WTA improved F1 by 1.12 points and AUC by 1.25 points over Softmax Top-K. On ImageNet-1K, it improved top-1 accuracy and reduced estimated energy on both tested backbones at the same active parameter count. We also clarified how k-WTA and SEMM differ in expert selection and weighting.

  2. Competitive dynamics and load balancing. A 2×2 ablation varied competitive dynamics and the load-balancing loss independently. With the loss disabled, enabling competitive dynamics reduced expert-load Gini from 0.59 to 0.28. Adding the loss reduced it further to 0.12. These results show that both mechanisms improve expert-load balance in the tested setting.

  3. Inference overhead. We reported how measured GPU latency varies with the number of experts and internal routing steps. In a direct RTX 4090 comparison, k-WTA took 46.80 ms per batch, compared with 51.40 ms for the SNN Softmax router. We also reported peak GPU memory and power, and explained how we obtained the operation-based energy estimates.

  4. Missing-modality handling. We clarified how we sample a prototype from training examples where the target modality is observed, fuse it with a learned embedding, and use the embedding alone when too few examples are available. At 50% modality missingness, the two-stage method outperformed zero filling and either stage alone on both CMU-MOSI and CMU-MOSEI.

  5. Notation and presentation. We clarified the expert aggregation equation, spike-count weighting, and the interpretation of patch-level routing masks. We also distinguished the two LIF-BERT implementations in the GLUE comparison.

We have uploaded the revised manuscript to OpenReview. The controlled router comparisons are presented in Appendix A.6, and the ablations that separate the effects of competitive dynamics and the load-balancing loss are presented in Section 5.3. Thank you for taking the time to consider these updates. We would be happy to clarify any remaining questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.