- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SpikeMoE, a spiking mixture-of-experts architecture that selects a small number of experts using spike counts from a k-winner-take-all router. For incomplete multimodal inputs, the authors also introduce a two-stage representation-completion procedure using observed-modality features and learnable embeddings. I found the integration of spike-based neuronal competition with conditional expert computation informative. The paper is also useful in showing how this design behaves across visual, language, and incomplete-modality settings.
Critical strengths
The strongest aspect of the paper is its empirical breadth. The method improves results on several spiking Transformer backbones and is also examined on language and incomplete-modality benchmarks. The gains over the closest alternatives vary in size, but the repeated positive results make the architectural idea worth considering. I also appreciate that the paper investigates inhibition and refractoriness rather than treating the router as a black box. Taken together, the study offers a useful contribution to the design of conditional computation in SNNs.
Critical weaknesses
Previous work has already explored mixtures of experts in spiking Transformers, and the manuscript does not yet provide a sufficiently controlled comparison to show when competitive spiking dynamics are preferable to a simpler sparse router. The use of an auxiliary balancing objective also complicates the explanation of why expert allocation improves. A more focused comparison would be more informative than extending the evaluation to additional datasets.
Questions for authors
- How does the proposed router compare with a conventional sparse or Top-K router when the experts, training objective, and computational budget are kept comparable? In particular, it would be helpful to distinguish the effect of competitive spiking dynamics from that of the load-balancing objective already used in training.
- Which aspects of the proposed routing design are most important for the observed gains over earlier spiking-MoE work? A clearer discussion of the closest prior methods and of the conditions under which the gains are largest would help position the contribution.
3 comments
Weakness 1 & Question 2
Relation to prior work. Zhou et al. introduced expert mixtures into Spiking Transformers through SEMM. It uses spike sequences to gate expert outputs element by element, allocating computation across attention heads and feed-forward channel groups. SpikeMoE assigns a routing neuron to each candidate expert. Lateral inhibition and refractoriness shape the accumulated spike counts. The router then selects the Top-K experts and uses their normalized counts to weight their outputs. Our contribution is to use explicit competitive dynamics for expert selection and weighting in spiking MoE.
Roles of lateral inhibition and refractoriness. In the CMU-MOSI ablation with the load-balancing loss enabled, removing either component reduces performance and increases Gini. Removing refractoriness has the larger effect. F1 falls from 43.54 to 41.50, and Gini rises from 0.12 to 0.42. The ablation measures the effect of each component in this setting. The comparison with SEMM measures the difference between the two complete designs.
Configurations with larger gains. Among the six vision configurations in Table 1, we focus on the gains on Spikformer-8-384 and Spikingformer-8-384. To control for hyperparameter differences and assess run-to-run variation, we compare k-WTA with SEMM on ImageNet-1K across three seeds (42, 43, and 44). We match the expert architecture, parameter count, loss, and training protocol.
| Backbone | SEMM Top-1 (%) | k-WTA Top-1 (%) | Gain (pp), (t-test, p<0.005) |
|---|---|---|---|
| Spikformer-8-384 | 72.86 ± 0.12 | 74.12 ± 0.06 | +1.26 |
| Spikingformer-8-384 | 73.58 ± 0.10 | 74.37 ± 0.01 | +0.79 |
Dear Reviewer WCbR,
Thank you again for your detailed feedback and suggestions. We have uploaded the revised manuscript. Appendix A.6 (Tables 11–13) presents controlled comparisons with conventional routers on ImageNet-1K and CMU-MOSI. Sec. 5.3 (Tables 5–6) examines the roles of lateral inhibition and refractoriness and separates the effect of competitive dynamics from that of the load-balancing loss. Sec. 2 explains how k-WTA differs from SEMM, and our rebuttal reports the three-seed ImageNet-1K comparison with SEMM.
Dear Reviewer WCbR,
Thank you for your careful review. We address your questions with controlled comparisons and ablation studies. We will include these additional experiments in the revised manuscript.
Weakness 1 & Weakness 2 & Question 1
Comparison with conventional routers under a matched compute budget
To test when competitive dynamics improve sparse routing, we hold the SNN backbone, expert architecture and capacity, active parameter count, training objective, and compute budget fixed. We use the same weight for Lbalance across all variants. We compare floating-point Softmax Top-K, Linear Top-K without exponential normalization, Spike-count Top-K without competition, and k-WTA variants with lateral inhibition and refractoriness.
The Gini coefficient [1] measures expert-load imbalance across the dataset. Lower values indicate more balanced expert use.
Reference:
[1] Chen, Lizhang, et al. "ϕ-Balancing for Mixture-of-Experts Training." Forty-third International Conference on Machine Learning.
Separating competitive dynamics from the load-balancing loss
We conduct a 2×2 factorial ablation and remove each component individually on CMU-MOSI at 50% missingness. The first four rows vary competitive dynamics and Lbalance independently. The last two rows remove one component at a time while retaining the loss.