- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SpikeWeave, a spiking self-attention architecture for long-range sequence modeling. The key idea is to align neuronal time directly with sequence positions, avoiding an additional simulation-time dimension. To preserve long-term information, the model introduces a multiscale dendritic exponential moving average (DEMA) module with multiple learnable timescales, whose states persist across somatic resets. SpikeWeave further combines this memory with chunkwise causal spiking self-attention. The method is evaluated on sequential MNIST variants, SHD, and five Long Range Arena tasks. The results are competitive with existing spiking sequence models, and intervention experiments show that resetting dendritic states at chunk boundaries substantially degrades performance, suggesting that cross-chunk dendritic memory is functionally important.
Critical strengths
1.Aligning neuronal time with sequence positions provides a natural alternative to conventional direct encoding, and the proposed DEMA mechanism is well matched to the goal of retaining long-range temporal context.
2.Resetting dendritic states at chunk boundaries causes a large performance drop, providing direct evidence that the model relies on cross-chunk dendritic memory rather than only local attention.
Critical weaknesses
-
The contribution of DEMA is not sufficiently isolated.
DEMA is essentially a bank of learnable leaky/EMA states with multiple timescales, but the paper does not compare it against representative multi-timescale or multi-compartment spiking neurons under the same backbone. Comparisons with standard LIF, TC-LIF, TS-LIF, or related temporal neuron designs would help determine whether the gains come specifically from DEMA rather than from the overall SpikeWeave architecture. -
The practical efficiency claim is insufficiently validated.
The paper repeatedly emphasizes the reduction from to neuronal updates, but no wall-clock comparison is provided. Since DEMA maintains multiple recurrent branches per feature and the model also introduces additional gating and projection operations, fewer neuronal updates do not necessarily translate into lower latency or higher throughput on modern hardware. -
No training/inference throughput or memory comparison is reported.
The paper would be substantially stronger with training speed, inference latency/throughput, and peak GPU memory comparisons against representative direct-encoding Spiking Transformers and recurrent/SSM baselines. Without such measurements, it is difficult to assess the practical significance of the claimed efficiency advantage. -
The mechanistic analysis is performed on a model variant that differs from the main architecture.
The appendix indicates that the reference checkpoint used for the intervention experiments differs from the main model in normalization and FFN/SSA details. Therefore, the intervention results do not directly establish that the same mechanism is responsible for the performance of the main benchmark model. -
The comparison between local attention and long-range memory could be better disentangled.
Cross-chunk information is propagated through recurrent dendritic states rather than direct cross-chunk attention. Therefore, SpikeWeave is closer to a combination of local attention and recurrent multiscale memory than to global long-range attention. Additional ablations would help clarify the relative contribution of these two components. -
The meaning of the “Parallel” designation is not sufficiently clear.
DEMA itself is recurrent across sequence positions. The paper does not clearly explain whether this recurrence is implemented sequentially, via parallel scan/prefix operations, or through custom kernels. This matters for interpreting both the scalability and efficiency claims. -
Figure readability is poor in several places.
The fonts and annotations in several architecture figures are too small, making them difficult and unpleasant to read. This significantly hurts presentation quality.
Questions for authors
-
Could the authors provide controlled comparisons against standard LIF and representative multi-timescale/multi-compartment neurons such as TC-LIF and TS-LIF, while keeping the same attention backbone, model width, and training recipe?
-
Could the authors report practical efficiency measurements, including training throughput, inference latency/throughput, and peak GPU memory, against representative direct-encoding Spiking Transformers and recurrent/SSM baselines?
-
How is the DEMA recurrence implemented in practice: sequential recurrence, parallel scan/prefix computation, or a custom fused kernel? This is important for understanding the practical meaning of the “Parallel” designation.
-
Can the authors repeat the mechanistic intervention experiments on the exact model used for the main benchmark results, rather than on the separate reference variant described in the appendix?
-
Can the authors provide a more direct ablation separating the contribution of chunkwise attention from that of the recurrent dendritic memory? In particular, how much of the final performance comes from local SSA versus DEMA-based cross-chunk state propagation?
-
Could the authors improve figure readability? The fonts and annotations in several figures are too small and are difficult to read at the current scale.
9 comments
Rebuttal for W6 & Q3 (Implementation of the DEMA recurrence)
We thank the reviewer to raise thiis question. Below we explain how we implement SpikeWeave in detail.
The DEMA recurrence state update and readout are shown in Eqs. 15–17 in the manuscript: Here denotes the sigmoid and the elementwise product. The input is the ternary event at position . The learned parameters are , , and : and determine the retention , and scales the input. The state is dendritic branch , is the number of branches, and is that branch's readout coefficient. The coefficient scales the direct input, and is the dendritic readout passed to the soma. The somatic reset does not appear in these equations, so the recurrence remains linear. Unrolling it gives the causal kernel in Eq. 20, Here is the lag in positions, and is the response of the dendritic readout to an input positions earlier. Our implementation evaluates this kernel with one FFT convolution over the full sequence. That computation has no sequential dependence across positions. Placing it in dendritic compartments, so that spike generation can use the integrated summary without erasing it, is the spiking adaptation.
Intra-chunk SSA is a batched masked matrix product and is parallel within a chunk. The binary and ternary LIF populations are separate from the dendritic recurrence. They use subtractive reset, Here is the input current, is the membrane before reset, and is the membrane after reset. We use rather than to distinguish it from the dendritic state above. The decay is , the threshold is , is the Heaviside step, and is the emitted spike. Because is a threshold function of , this dependence is nonlinear and cannot be written as a convolution kernel or removed by a parallel scan. For efficient training, all ten populations share one fused Triton kernel. Each program handles 64 neurons and loops over the positions inside the kernel. The membrane remains in registers, and the backward pass is a reverse-time loop in a second kernel. Fusion removes the cost of launching one kernel per position. It does not make positions independent.
On an RTX 5090, inference for a ternary population of shape takes 2.95 ms with the fused kernel and 303.9 ms with the Python loop. The forward-backward times are 7.54 ms and 1,215 ms. Across the measured shapes, fusion is 46–267× faster.
The Parallel designation is therefore partial. DEMA is parallel through the FFT convolution, intra-chunk SSA is parallel within a chunk, and the resetting LIF populations remain a fused sequential scan. We will state this distinction in the revised manuscript.
Rebuttal for W4 & Q4 (Mechanistic analysis experiment on the main architecture)
We thank the reviewer for this question. Reviewer 1 notes the same mismatch in the second weakness, and Reviewer 2 asks in Q3 for the interventions to be repeated on the reported architecture.
During paper preparation we ran the interventions on a ListOps model with D=80 rather than the Table 4 width of 128, so that they would finish sooner because of limited computing resources we have. We have now repeated both fixed-weight interventions on a Text model (D=d_v=d_ff=256). The evaluation covers all 25,000 IMDB examples. In the LRA release this file is also the validation split, and the checkpoint was selected by accuracy on it, so the split is not an independent test set.
| Condition | Acc. (%) | (pp) [95% CI] |
|---|---|---|
| Baseline | 88.676 | — |
| Reset dendrites at chunk boundaries | 87.980 | −0.696 [−0.968, −0.424] |
| Mask fastest 25% | 89.024 | +0.348 [+0.080, +0.616] |
| Mask slowest 25% | 88.068 | −0.608 [−0.912, −0.316] |
| Random mask, 25% (20 trials) | 88.612–88.968 | mean +0.124 (−0.064 to +0.292) |
Clearing the dendritic state at each chunk boundary lowers accuracy by 0.70%, and the interval excludes zero. The four shortest branches, when masked, raise accuracy by 0.35%. The four longest lower it by 0.61%. The direction agree with the ListOps intervention. However, we also observe that the random masks behave differently from Table 3. On ListOps, the random mask gave lower accuracy than suppressing the slowest branches. On Text, all 20 random masks give higher accuracy than suppressing the slowest branches: they change accuracy by −0.06 to +0.29 points relative to the baseline.
Below we also provide the effect of the boundary reset by input length, one group per 512-position chunk as in Figure 4.
| Input length | ListOps n | ListOps [95% CI] | Text n | Text [95% CI] |
|---|---|---|---|---|
| ≤512 | 6 | 0 (no prediction changes) | 2,639 | 0 (no prediction changes) |
| 513–1,024 | 280 | −19.64 [−25.00, −14.29] | 10,833 | +0.01 [−0.36, +0.38] |
| 1,025–1,536 | 156 | −23.08 [−30.77, −16.03] | 5,038 | −0.77 [−1.41, −0.14] |
| 1,537–2,048 | 70 (>1,536) | −21.43 [−34.29, −8.57] | 2,562 | −1.83 [−2.81, −0.90] |
| 2,049–2,560 | — | — | 1,462 | −0.68 [−2.05, +0.68] |
| 2,561–3,072 | — | — | 883 | −1.81 [−3.74, +0.11] |
| 3,073–3,584 | — | — | 566 | −3.00 [−5.65, −0.35] |
| >3,584 | — | — | 1,017 | −4.52 [−6.59, −2.46] |
All three multi-chunk ListOps groups lose 19.6–23.1%. In contrast, on Text the accuracy in the 513–1,024 group is merely unchanged, and the largest losses occur in the two groups longer than 3,072 positions (3.00% and 4.52%).
We state that the smaller effect on Text may reflect a difference between the two tasks. IMDB is sentiment classification of movie reviews: cues for the label are spread throughout the review, so averaging features computed within each chunk may be enough to classify most reviews. ListOps consists of prefix expressions whose root operator appears at the start of the sequence. The answer requires nested evaluation of arguments spread across the whole sequence, which averaging chunk-local features cannot provide.
Rebuttal for W2 & W3 & Q2 (Efficiency measurements)
We thank the reviewer for raising this point. It is also raised by Reviewer 1, in the efficiency part of the second weakness and in Q2, and by Reviewer 2, in the first weakness and in Q1. We report the shared measurements here: training step time, inference latency and throughput, and peak memory at N=4000, against direct-encoding Spiking Transformers and spiking sequence models. We agree that fewer neuronal updates need not produce lower latency or higher throughput. The measurements below test that directly at N=4000 on an H100 MIG 3g.40gb slice (60 of 132 SMs, 39.5 GiB available). The baselines are the direct-encoding SNN Transformers and the SSM-based SNN models requested.
Training. Step time and peak memory are listed below.
The memory result favors SpikeWeave over the two SNN Transformers. At batch 4 it uses 8.17 GiB, compared with 17.94 GiB for SDT. Its largest batch is 20, against 8 for SDT and 2 for Spikformer. Both baselines repeat the input for T=4 simulation steps, and both compute attention over the full sequence. Spikformer's attention activations also grow with N2.
The two SSM-based models remain faster and use less memory. At batch 4, SpikeWeave is 2.9× slower than SpikingSSMs and 4.1× slower than Binary S4D. At batch 16 those ratios are 1.6× and 3.3×. We are obligated to point out that SpikingSSMs and Binary S4D are faster for a certain reason. SpikingSSMs predicts its membrane with a pretrained convolution and does not backpropagate through that prediction. Binary S4D applies one binarization per position to the S4D output, without a membrane reset or a temporal recurrence. Their memory is a linear convolution. SpikeWeave instead keeps a resetting LIF recurrence in the backward pass.
Inference. Under the same hardware setting:
Peak memory is again the quantity that SpikeWeave improves just as the results shown in training process above. SpikeWeave uses 0.24 GiB at batch 1, roughly 1/6 of SDT and one nineteenth of Spikformer. At batch 16 it uses 1.43 GiB, against 21.62 GiB for SDT; Spikformer does not fit. Its batch-16 memory is also below SpikingSSMs (3.91 GiB) and Binary S4D (1.64 GiB).
Latency and throughput do not improve in the same way. At batch 1, 51.7 ms is better only than Spikformer's 63.5 ms. It is 3.1× the latency of SDT, 2.9× that of SpikingSSMs, and 3.9× that of Binary S4D. At batch 16, 85.1 seq/s is 1.10× the throughput of SpikingSSMs, 0.90× that of SDT, and 0.28× that of Binary S4D.
The batch-1 cost is the length of the reset recurrence, not the width. Six layers and ten LIF populations give 60 sequential scans. They account for 38.7 ms, or 80.4% of kernel time, averaging 0.64 ms each. One program is launched per 64 neurons, so a batch-1 scan has only 2–8 programs on this 60-SM slice and leaves most SMs idle. Changing width from D=80 to D=256 changes latency only from 50.6 ms to 51.7 ms. SDT's recurrence has length T=4. Spikformer's extra cost is its T=4 full-sequence matrix products, which take 53.7% of its kernel time.
The same scans scale better than the baselines once the batch fills the slice. Increasing the batch from 1 to 16 increases SpikeWeave's forward time by 3.6×, from 51.7 ms to 188.0 ms. Each scan increases by 2.2×, from 0.64 ms to 1.42 ms, while its program count increases from 8 to 128. SpikingSSMs, MEGA-chunk, and SDT increase by 10–12× over the same change. Binary S4D increases by 3.9×, because constructing its convolution kernel does not depend on the batch.
At last, we ensure that the revised manuscript will include these measurements and restrict the efficiency statements to what they show.