acceptodds
Under review as a conference paper at ICLR 2027

SpikeWeave: Spiking Self-attention across Thousands of Timesteps

Abstract

Spiking neural networks promise energy-efficient sequence modeling through sparse, event-driven computation. However, existing spiking self-attention architectures rely on direct encoding, which repeatedly presents a static input over multiple timesteps. This introduces a simulation dimension decoupled from the input sequence and multiplies computational cost. Neuronal dynamics are thus spent on re-encoding a static input rather than on modeling temporal structure. Whether intrinsic neuronal dynamics can instead drive long-range sequence modeling remains largely unexplored. We propose SpikeWeave, a temporal spiking self-attention architecture that aligns each sequence position with a single neuronal time step. The network thus consumes successive elements natively through its own dynamics. To retain context over long horizons, we introduce a multiscale dendritic exponential moving average (DEMA) mechanism that maintains parallel leaky integrations at distinct decay rates. A separate somatic compartment converts the dendritic readout into spikes for query and key generation. Crucially, somatic resets leave dendritic states intact, decoupling spike generation from memory retention. For scalable training, we develop a chunkwise spiking self-attention scheme in which dendritic states carry context across chunk boundaries. We evaluate SpikeWeave on long-range sequence modeling benchmarks, where it delivers strong performance among spiking architectures while preserving sparse, event-driven computation. We believe SpikeWeave takes a step toward scalable, energy-efficient sequence modeling with spiking neural networks.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Neural Networks, Long-Sequence Modeling, Spiking Self-attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_V6FZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpikeWeave, a spiking self-attention architecture for long-range sequence modeling. The key idea is to align neuronal time directly with sequence positions, avoiding an additional simulation-time dimension. To preserve long-term information, the model introduces a multiscale dendritic exponential moving average (DEMA) module with multiple learnable timescales, whose states persist across somatic resets. SpikeWeave further combines this memory with chunkwise causal spiking self-attention. The method is evaluated on sequential MNIST variants, SHD, and five Long Range Arena tasks. The results are competitive with existing spiking sequence models, and intervention experiments show that resetting dendritic states at chunk boundaries substantially degrades performance, suggesting that cross-chunk dendritic memory is functionally important.

Critical strengths

1.Aligning neuronal time with sequence positions provides a natural alternative to conventional direct encoding, and the proposed DEMA mechanism is well matched to the goal of retaining long-range temporal context.

2.Resetting dendritic states at chunk boundaries causes a large performance drop, providing direct evidence that the model relies on cross-chunk dendritic memory rather than only local attention.

Critical weaknesses
  1. The contribution of DEMA is not sufficiently isolated.
    DEMA is essentially a bank of learnable leaky/EMA states with multiple timescales, but the paper does not compare it against representative multi-timescale or multi-compartment spiking neurons under the same backbone. Comparisons with standard LIF, TC-LIF, TS-LIF, or related temporal neuron designs would help determine whether the gains come specifically from DEMA rather than from the overall SpikeWeave architecture.

  2. The practical efficiency claim is insufficiently validated.
    The paper repeatedly emphasizes the reduction from O(TN)O(TN) to O(N)O(N) neuronal updates, but no wall-clock comparison is provided. Since DEMA maintains multiple recurrent branches per feature and the model also introduces additional gating and projection operations, fewer neuronal updates do not necessarily translate into lower latency or higher throughput on modern hardware.

  3. No training/inference throughput or memory comparison is reported.
    The paper would be substantially stronger with training speed, inference latency/throughput, and peak GPU memory comparisons against representative direct-encoding Spiking Transformers and recurrent/SSM baselines. Without such measurements, it is difficult to assess the practical significance of the claimed efficiency advantage.

  4. The mechanistic analysis is performed on a model variant that differs from the main architecture.
    The appendix indicates that the reference checkpoint used for the intervention experiments differs from the main model in normalization and FFN/SSA details. Therefore, the intervention results do not directly establish that the same mechanism is responsible for the performance of the main benchmark model.

  5. The comparison between local attention and long-range memory could be better disentangled.
    Cross-chunk information is propagated through recurrent dendritic states rather than direct cross-chunk attention. Therefore, SpikeWeave is closer to a combination of local attention and recurrent multiscale memory than to global long-range attention. Additional ablations would help clarify the relative contribution of these two components.

  6. The meaning of the “Parallel” designation is not sufficiently clear.
    DEMA itself is recurrent across sequence positions. The paper does not clearly explain whether this recurrence is implemented sequentially, via parallel scan/prefix operations, or through custom kernels. This matters for interpreting both the scalability and efficiency claims.

  7. Figure readability is poor in several places.
    The fonts and annotations in several architecture figures are too small, making them difficult and unpleasant to read. This significantly hurts presentation quality.

Questions for authors
  1. Could the authors provide controlled comparisons against standard LIF and representative multi-timescale/multi-compartment neurons such as TC-LIF and TS-LIF, while keeping the same attention backbone, model width, and training recipe?

  2. Could the authors report practical efficiency measurements, including training throughput, inference latency/throughput, and peak GPU memory, against representative direct-encoding Spiking Transformers and recurrent/SSM baselines?

  3. How is the DEMA recurrence implemented in practice: sequential recurrence, parallel scan/prefix computation, or a custom fused kernel? This is important for understanding the practical meaning of the “Parallel” designation.

  4. Can the authors repeat the mechanistic intervention experiments on the exact model used for the main benchmark results, rather than on the separate reference variant described in the appendix?

  5. Can the authors provide a more direct ablation separating the contribution of chunkwise attention from that of the recurrent dendritic memory? In particular, how much of the final performance comes from local SSA versus DEMA-based cross-chunk state propagation?

  6. Could the authors improve figure readability? The fonts and annotations in several figures are too small and are difficult to read at the current scale.

9 comments
AuthorsOpenReview ↗

Rebuttal for W2 & W3 & Q2 (Efficiency measurements)

We thank the reviewer for raising this point. It is also raised by Reviewer 1, in the efficiency part of the second weakness and in Q2, and by Reviewer 2, in the first weakness and in Q1. We report the shared measurements here: training step time, inference latency and throughput, and peak memory at N=4000N=4000, against direct-encoding Spiking Transformers and spiking sequence models. We agree that fewer neuronal updates need not produce lower latency or higher throughput. The measurements below test that directly at N=4000N=4000 on an H100 MIG 3g.40gb slice (60 of 132 SMs, 39.5 GiB available). The baselines are the direct-encoding SNN Transformers and the SSM-based SNN models requested.

Training. Step time and peak memory are listed below.

ModelStep time, B=4 (s)Step time, B=16 (s)Peak memory, B=4 (GiB)Peak memory, B=16 (GiB)
SpikeWeave0.2720.4968.1729.88
Spikformer (T=4T=4)OOMOOMOOMOOM
SDT (T=4T=4)0.099OOM17.94OOM
SpikingSSMs0.0930.3093.3011.71
Binary S4D0.0660.1526.8313.73
MEGA-chunk0.0980.3125.1715.30

The memory result favors SpikeWeave over the two SNN Transformers. At batch 4 it uses 8.17 GiB, compared with 17.94 GiB for SDT. Its largest batch is 20, against 8 for SDT and 2 for Spikformer. Both baselines repeat the input for T=4T=4 simulation steps, and both compute attention over the full sequence. Spikformer's attention activations also grow with N2N^2.

The two SSM-based models remain faster and use less memory. At batch 4, SpikeWeave is 2.9×2.9\times slower than SpikingSSMs and 4.1×4.1\times slower than Binary S4D. At batch 16 those ratios are 1.6×1.6\times and 3.3×3.3\times. We are obligated to point out that SpikingSSMs and Binary S4D are faster for a certain reason. SpikingSSMs predicts its membrane with a pretrained convolution and does not backpropagate through that prediction. Binary S4D applies one binarization per position to the S4D output, without a membrane reset or a temporal recurrence. Their memory is a linear convolution. SpikeWeave instead keeps a resetting LIF recurrence in the backward pass.

Inference. Under the same hardware setting:

ModelLatency, B=1 (ms)Peak memory, B=1 (GiB)Batch time, B=16 (ms)Throughput, B=16 (seq/s)Peak memory, B=16 (GiB)
SpikeWeave51.70.24188.085.11.43
Spikformer (T=4T=4)63.54.62OOMOOMOOM
SDT (T=4T=4)16.61.39169.794.321.62
SpikingSSMs17.60.33205.877.73.91
Binary S4D13.30.5352.4305.51.64
MEGA-chunk9.90.28111.5143.52.74

Peak memory is again the quantity that SpikeWeave improves just as the results shown in training process above. SpikeWeave uses 0.24 GiB at batch 1, roughly 1/6 of SDT and one nineteenth of Spikformer. At batch 16 it uses 1.43 GiB, against 21.62 GiB for SDT; Spikformer does not fit. Its batch-16 memory is also below SpikingSSMs (3.91 GiB) and Binary S4D (1.64 GiB).

Latency and throughput do not improve in the same way. At batch 1, 51.7 ms is better only than Spikformer's 63.5 ms. It is 3.1×3.1\times the latency of SDT, 2.9×2.9\times that of SpikingSSMs, and 3.9×3.9\times that of Binary S4D. At batch 16, 85.1 seq/s is 1.10×1.10\times the throughput of SpikingSSMs, 0.90×0.90\times that of SDT, and 0.28×0.28\times that of Binary S4D.

The batch-1 cost is the length of the reset recurrence, not the width. Six layers and ten LIF populations give 60 sequential scans. They account for 38.7 ms, or 80.4% of kernel time, averaging 0.64 ms each. One program is launched per 64 neurons, so a batch-1 scan has only 2–8 programs on this 60-SM slice and leaves most SMs idle. Changing width from D=80D=80 to D=256D=256 changes latency only from 50.6 ms to 51.7 ms. SDT's recurrence has length T=4T=4. Spikformer's extra cost is its T=4T=4 full-sequence matrix products, which take 53.7% of its kernel time.

The same scans scale better than the baselines once the batch fills the slice. Increasing the batch from 1 to 16 increases SpikeWeave's forward time by 3.6×3.6\times, from 51.7 ms to 188.0 ms. Each scan increases by 2.2×2.2\times, from 0.64 ms to 1.42 ms, while its program count increases from 8 to 128. SpikingSSMs, MEGA-chunk, and SDT increase by 1010–12×12\times over the same change. Binary S4D increases by 3.9×3.9\times, because constructing its convolution kernel does not depend on the batch.

At last, we ensure that the revised manuscript will include these measurements and restrict the efficiency statements to what they show.

AuthorsOpenReview ↗

Rebuttal for W6 & Q3 (Implementation of the DEMA recurrence)

We thank the reviewer to raise thiis question. Below we explain how we implement SpikeWeave in detail.

The DEMA recurrence state update and readout are shown in Eqs. 15–17 in the manuscript: p=σ(δ),ρ=1−p⊙σ(α),p = \sigma(\delta), \quad \rho = 1 - p \odot \sigma(\alpha), mt=ρ⊙mt−1+(p⊙β)⊙et,m_t = \rho \odot m_{t-1} + (p \odot \beta) \odot e_t, rt=1K∑k=1Kγ(k)⊙mt(k)+ω⊙et.r_t = \frac{1}{\sqrt{K}} \sum_{k=1}^{K} \gamma^{(k)} \odot m_t^{(k)} + \omega \odot e_t. Here σ(⋅)\sigma(\cdot) denotes the sigmoid and ⊙\odot the elementwise product. The input ete_t is the ternary event at position tt. The learned parameters are δ\delta, α\alpha, and β\beta: δ\delta and α\alpha determine the retention ρ\rho, and β\beta scales the input. The state mt(k)m_t^{(k)} is dendritic branch kk, KK is the number of branches, and γ(k)\gamma^{(k)} is that branch's readout coefficient. The coefficient ω\omega scales the direct input, and rtr_t is the dendritic readout passed to the soma. The somatic reset does not appear in these equations, so the recurrence remains linear. Unrolling it gives the causal kernel in Eq. 20, Kd[j]=1K∑k=1Kγ(k)⊙p(k)⊙β(k)⊙(ρ(k))j.K_d[j] = \frac{1}{\sqrt{K}} \sum_{k=1}^{K} \gamma^{(k)} \odot p^{(k)} \odot \beta^{(k)} \odot (\rho^{(k)})^j. Here jj is the lag in positions, and Kd[j]K_d[j] is the response of the dendritic readout to an input jj positions earlier. Our implementation evaluates this kernel with one FFT convolution over the full sequence. That computation has no sequential dependence across positions. Placing it in dendritic compartments, so that spike generation can use the integrated summary without erasing it, is the spiking adaptation.

Intra-chunk SSA is a batched masked matrix product and is parallel within a chunk. The binary and ternary LIF populations are separate from the dendritic recurrence. They use subtractive reset, ut=λvt−1+It,st=H(ut−θ),vt=ut−θst.u_t = \lambda v_{t-1} + I_t, \quad s_t = H(u_t - \theta), \quad v_t = u_t - \theta s_t. Here ItI_t is the input current, utu_t is the membrane before reset, and vtv_t is the membrane after reset. We use vtv_t rather than mtm_t to distinguish it from the dendritic state above. The decay is λ\lambda, the threshold is θ\theta, H(⋅)H(\cdot) is the Heaviside step, and sts_t is the emitted spike. Because sts_t is a threshold function of utu_t, this dependence is nonlinear and cannot be written as a convolution kernel or removed by a parallel scan. For efficient training, all ten populations share one fused Triton kernel. Each program handles 64 neurons and loops over the LL positions inside the kernel. The membrane remains in registers, and the backward pass is a reverse-time loop in a second kernel. Fusion removes the cost of launching one kernel per position. It does not make positions independent.

On an RTX 5090, inference for a ternary population of shape 4×4000×2564\times4000\times256 takes 2.95 ms with the fused kernel and 303.9 ms with the Python loop. The forward-backward times are 7.54 ms and 1,215 ms. Across the measured shapes, fusion is 46–267× faster.

The Parallel designation is therefore partial. DEMA is parallel through the FFT convolution, intra-chunk SSA is parallel within a chunk, and the resetting LIF populations remain a fused sequential scan. We will state this distinction in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal for W4 & Q4 (Mechanistic analysis experiment on the main architecture)

We thank the reviewer for this question. Reviewer 1 notes the same mismatch in the second weakness, and Reviewer 2 asks in Q3 for the interventions to be repeated on the reported architecture.

During paper preparation we ran the interventions on a ListOps model with D=80 rather than the Table 4 width of 128, so that they would finish sooner because of limited computing resources we have. We have now repeated both fixed-weight interventions on a Text model (D=d_v=d_ff=256). The evaluation covers all 25,000 IMDB examples. In the LRA release this file is also the validation split, and the checkpoint was selected by accuracy on it, so the split is not an independent test set.

ConditionAcc. (%)Δ\Delta (pp) [95% CI]
Baseline88.676—
Reset dendrites at chunk boundaries87.980−0.696 [−0.968, −0.424]
Mask fastest 25%89.024+0.348 [+0.080, +0.616]
Mask slowest 25%88.068−0.608 [−0.912, −0.316]
Random mask, 25% (20 trials)88.612–88.968mean +0.124 (−0.064 to +0.292)

Clearing the dendritic state at each chunk boundary lowers accuracy by 0.70%, and the interval excludes zero. The four shortest branches, when masked, raise accuracy by 0.35%. The four longest lower it by 0.61%. The direction agree with the ListOps intervention. However, we also observe that the random masks behave differently from Table 3. On ListOps, the random mask gave lower accuracy than suppressing the slowest branches. On Text, all 20 random masks give higher accuracy than suppressing the slowest branches: they change accuracy by −0.06 to +0.29 points relative to the baseline.

Below we also provide the effect of the boundary reset by input length, one group per 512-position chunk as in Figure 4.

Input lengthListOps nListOps Δ\Delta [95% CI]Text nText Δ\Delta [95% CI]
≤51260 (no prediction changes)2,6390 (no prediction changes)
513–1,024280−19.64 [−25.00, −14.29]10,833+0.01 [−0.36, +0.38]
1,025–1,536156−23.08 [−30.77, −16.03]5,038−0.77 [−1.41, −0.14]
1,537–2,04870 (>1,536)−21.43 [−34.29, −8.57]2,562−1.83 [−2.81, −0.90]
2,049–2,560——1,462−0.68 [−2.05, +0.68]
2,561–3,072——883−1.81 [−3.74, +0.11]
3,073–3,584——566−3.00 [−5.65, −0.35]
>3,584——1,017−4.52 [−6.59, −2.46]

All three multi-chunk ListOps groups lose 19.6–23.1%. In contrast, on Text the accuracy in the 513–1,024 group is merely unchanged, and the largest losses occur in the two groups longer than 3,072 positions (3.00% and 4.52%).

We state that the smaller effect on Text may reflect a difference between the two tasks. IMDB is sentiment classification of movie reviews: cues for the label are spread throughout the review, so averaging features computed within each chunk may be enough to classify most reviews. ListOps consists of prefix expressions whose root operator appears at the start of the sequence. The answer requires nested evaluation of arguments spread across the whole sequence, which averaging chunk-local features cannot provide.

Reviewer_VG6SOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpikeWeave, a spiking self-attention architecture for long-range sequence modeling. Unlike conventional Spiking Transformers that maintain a separate simulation-time dimension, SpikeWeave advances neuronal states directly along sequence positions, reducing the number of neuronal updates from (O(TN)) to (O(N)). To retain long-range context, the method introduces a multiscale dendritic exponential moving average (DEMA) module with multiple learnable decay timescales and a separate somatic compartment for spike generation. The resulting somatic events are used to construct query and key spikes for chunkwise causal spiking self-attention, while dendritic states are carried across attention chunks. The method is evaluated on sequential MNIST, SHD, and Long Range Arena tasks, together with fixed-weight interventions intended to analyze cross-chunk dendritic memory.

Critical strengths
  1. A clear and well-motivated sequence-time formulation for spiking self-attention. The distinction between simulation time and sequence time is clearly presented, and binding neuronal updates directly to sequence positions is a conceptually appealing way to avoid repeated temporal encoding of the same token-level representation.

  2. An interpretable integration of multiscale memory and spiking attention. The separation between persistent dendritic states and somatic spike generation provides a coherent mechanism for retaining long-range context while still producing sparse query/key events. The cross-chunk intervention provides useful evidence that the analyzed checkpoint indeed relies on the carried dendritic state.

Critical weaknesses
  1. The central efficiency claims are not supported by direct computational measurements. The paper motivates SpikeWeave as a scalable and energy-efficient alternative to direct-encoding Spiking Transformers and provides asymptotic reductions in neuronal updates and attention complexity. However, the empirical evaluation does not report wall-clock latency, throughput, peak memory, operation counts, or energy-related measurements against matched baselines. Since practical efficiency is one of the main motivations, complexity arguments alone are insufficient to establish the claimed benefit.

  2. The empirical evidence is statistically limited, and the mechanistic analysis does not use the main reported architecture. The appendix states that each task is trained with a single seed. This is particularly problematic because several reported differences against competing methods are very small. In addition, the fixed-weight dendritic interventions are conducted on a selected ListOps checkpoint whose normalization and FFN structure differ from the architecture described in the main method section, and the analysis is performed on a validation subset rather than an independent test set. These choices make it difficult to determine how robustly the mechanistic conclusions transfer to the main reported model.

Questions for authors

Could the authors provide matched efficiency measurements against direct-encoding Spiking Transformers and representative spiking sequence models, including runtime/throughput, peak memory, and ideally spike/SOP or energy-related proxies? How much practical improvement is obtained beyond the asymptotic reduction from O(TN)O(TN) to O(N)O(N) neuronal updates?

Could the authors report multiple-seed results for the main benchmarks? In particular, several differences in Table 1 and Table 2 are very small, and it is currently unclear whether they exceed run-to-run variability.

Why is the mechanistic analysis performed on a checkpoint whose normalization and FFN structure differ from the main SpikeWeave architecture? Can the cross-chunk reset and branch-suppression experiments be reproduced on the exact main architecture and preferably on an independent test set?

Could the authors provide a controlled same-backbone ablation that replaces DEMA with simpler temporal-memory alternatives, such as a single-timescale LIF/EMA or another recurrent state, while retaining the same chunkwise SSA? This would help isolate whether the improvement comes specifically from multiscale dendritic memory rather than from sequence-time processing and chunked attention more generally.

3 comments
AuthorsOpenReview ↗

Rebuttal for W1 & Q1 (Matched runtime, throughput, and peak memory)

We sincerely thank the reviewer for raising this issue. Below we will respond to the questions raised in detail.

a. Training We measured the training latency and peak memory at N=4000N=4000 on an H100 MIG 3g.40gb slice (60 of 132 SMs, 39.5 GiB available)

ModelLatency@B=4 (s/step)Latency@B=16 (s/step)Peak Memory@B=4 (GiB)Peak Memory@B=16(GiB)
SpikeWeave0.2720.4968.1729.88
Spikformer (T=4T=4)OOMOOMOOMOOM
SDT (T=4T=4)0.099OOM17.94OOM
SpikingSSMs0.0930.3093.3011.71
Binary S4D0.0660.1526.8313.73
MEGA-chunk0.0980.3125.1715.30

Compared with other SNN Transformer architectures, SpikeWeave significantly save the training memory. At batch 4, SpikeWeave uses 8.17 GiB, against 17.94 GiB for SDT. It fits 20 sequences; SDT fits 8, and Spikformer fits 2. The memory difference from Spikformer and SDT has two sources in the timed models: they repeat the sequence along a simulation axis of length T=4T=4, and their attention is computed over the full sequence. Spikformer's attention activations grow with N2N^2.

Compared with other SSM-based SNN architectures, we do not find a training advantage. At batch 4, SpikeWeave is 2.9×2.9\times slower than SpikingSSMs and 4.1×4.1\times slower than Binary S4D, and it uses more memory than both. At batch 16 the time ratios fall to 1.6×1.6\times and 3.3×3.3\times. We need to point out that SpikingSSMs and Binary S4D are faster for a different reason. SpikingSSMs predicts the membrane with a pretrained convolution and does not backpropagate through it. Binary S4D binarizes the S4D output once per position and has no membrane reset or temporal recurrence. Their temporal memory is a linear convolution. In contrast, SpikeWeave is a Spiking Transformer: it retains a resetting LIF recurrence and backpropagates through it, while removing the extra simulation-time axis and restricting attention to chunks.

b. Inference Using the same setting above, we measured the inference latency, thoroughput and peak memory on an H100 MIG 3g.40gb slice.

ModelLatency@B=1 (ms)Peak Memory@B=1 (GiB)Latency@B=16 (ms/batch)Thoroughput@B=16 (seq/s)Peak Memory@B=16 (GiB)
SpikeWeave51.70.24188.085.11.43
Spikformer (T=4T=4)63.54.62OOMOOMOOM
SDT (T=4T=4)16.61.39169.794.321.62
SpikingSSMs17.60.33205.877.73.91
Binary S4D13.30.5352.4305.51.64
MEGA-chunk9.90.28111.5143.52.74

Same as training process, peak inference memory is the clearest gain in SpikeWeave. SpikeWeave uses 0.24 GiB: about 1/61/6 of SDT and 1/191/19 of Spikformer. At batch 16 it uses 1.43 GiB, against 21.62 GiB for SDT, while Spikformer occurs OOM. It also uses less memory than SpikingSSMs (3.91 GiB) and Binary S4D (1.64 GiB) at this batch. However, we do not observe the advantages in latency. At batch 1, SpikeWeave takes 51.7 ms. It is faster only than Spikformer (63.5 ms). It is 3.1×3.1\times slower than SDT, 2.9×2.9\times slower than SpikingSSMs, and 3.9×3.9\times slower than Binary S4D. At batch 16, its throughput is 85.1 seq/s: 1.10×1.10\times SpikingSSMs, 0.90×0.90\times SDT, and 0.28×0.28\times Binary S4D.

The batch-1 latency comes from 60 sequential scans: 6 layers times 10 LIF populations. These scans take 38.7 ms, 80.4% of kernel time, or 0.64 ms each. The fused kernel launches one program per 64 neurons. At batch 1, a scan has only 2–8 programs on a slice with 60 SMs, so most SMs are idle. The latency is set by the length of the chain, not by width: 50.6 ms at D=80D=80 and 51.7 ms at D=256D=256. SDT recurs for only T=4T=4 steps. Spikformer is slower than SpikeWeave because its T=4T=4 full-sequence matrix products take 53.7% of its kernel time.

From batch 1 to 16, SpikeWeave's forward pass grows 3.6×3.6\times, from 51.7 ms to 188.0 ms, while the batch grows 16×16\times. Each scan grows only 2.2×2.2\times, from 0.64 ms to 1.42 ms, as the number of programs rises from 8 to 128. Over the same change, SpikingSSMs, MEGA-chunk, and SDT grow 1010–12×12\times. Binary S4D grows 3.9×3.9\times because its largest cost, constructing the convolution kernel, does not depend on batch size.

Thus the practical improvement beyond the change from O(TN)O(TN) to O(N)O(N) is lower memory and a larger feasible batch than the timed T=4T=4 SNN Transformers. We will add these measurements to the revised manuscript and qualify the efficiency claims to match them.

AuthorsOpenReview ↗

Rebuttal for W2 & Q3 (Cross-chunk reset and branch suppression experiment on the main architecture)

Because of limited compute while preparing the paper, we ran the mechanistic analysis on a ListOps model smaller than the Table 4 preset (D = 80 instead of 128), so that results could be obtained faster. We now repeated both fixed-weight interventions on a Text checkpoint whose architecture follows Section 4.1 and the Text preset in Table 4 (D = d_v = d_ff = 256). We evaluate on all 25,000 IMDB examples. In the LRA release, the IMDB validation and test files are identical, and the checkpoint was selected by accuracy on them, so this evaluation is not on an independent test set.

ConditionAcc. (%)Δ\Delta (pp) [95% CI]
Baseline88.676—
Reset dendrites at chunk boundaries87.980−0.696 [−0.968, −0.424]
Mask fastest 25%89.024+0.348 [+0.080, +0.616]
Mask slowest 25%88.068−0.608 [−0.912, −0.316]
Random mask 25% (20 trials)88.612–88.968mean +0.124 (−0.064 to +0.292)

Same as the results preference in ListOps, Resetting the dendritic states at every chunk boundary lowers Text accuracy by 0.70%. Suppressing the four slowest branches lowers accuracy by 0.61%, while suppressing the four fastest raises it by 0.35%. However, we also observe that the random masks behave differently from Table 3. On ListOps, the random mask gave lower accuracy than suppressing the slowest branches. On Text, all 20 random masks give higher accuracy than suppressing the slowest branches: they change accuracy by −0.06 to +0.29 points relative to the baseline.

Below we provide the effect of the boundary reset by input length, one group per 512-position chunk as in Figure 4.

Input lengthListOps nListOps Δ\Delta [95% CI]Text nText Δ\Delta [95% CI]
≤51260 (no prediction changes)2,6390 (no prediction changes)
513–1,024280−19.64 [−25.00, −14.29]10,833+0.01 [−0.36, +0.38]
1,025–1,536156−23.08 [−30.77, −16.03]5,038−0.77 [−1.41, −0.14]
1,537–2,04870 (>1,536)−21.43 [−34.29, −8.57]2,562−1.83 [−2.81, −0.90]
2,049–2,560——1,462−0.68 [−2.05, +0.68]
2,561–3,072——883−1.81 [−3.74, +0.11]
3,073–3,584——566−3.00 [−5.65, −0.35]
>3,584——1,017−4.52 [−6.59, −2.46]

All three multi-chunk ListOps groups lose 19.6–23.1%. In contrast, on Text the accuracy in the 513–1,024 group is merely unchanged, and the largest losses occur in the two groups longer than 3,072 positions (3.00% and 4.52%).

We state that the smaller effect on Text may reflect a difference between the two tasks. IMDB is sentiment classification of movie reviews: cues for the label are spread throughout the review, so averaging features computed within each chunk may be enough to classify most reviews. ListOps consists of prefix expressions whose root operator appears at the start of the sequence. The answer requires nested evaluation of arguments spread across the whole sequence, which averaging chunk-local features cannot provide.

AuthorsOpenReview ↗

Rebuttal for Q4 (controlled ablation on DEMA using simpler temporal-memory alternatives)

We trained same-backbone controls, denote as SpikeWeave_st (single timescale) on PS-MNIST from random initialization with three seeds (131071, 1, 2). All models share L=2,D=dv=dff=64,dq=32,nc=49,N=784L = 2, D = d_v = d_ff = 64, d_q = 32, n_c = 49, N = 784 and the same chunkwise SSA; only the dendritic memory differs. Training uses AdamW (peak learning rate 3e-3, global batch 128, 5,000 updates). The checkpoint is selected on a 6,000-image validation split taken from the MNIST training set, and we report accuracy on the 10,000-image MNIST test set. SpikeWeave_st(τ\tau) keeps one dendritic branch per channel. All channels share one time constant τ\tau. We test τ∈\tau \in {2, 32, 1024}.

ModelBranches per channelTest acc. (%)Δ\Delta vs. SpikeWeave
SpikeWeave1690.41 ± 0.28—
SpikeWeave_st (τ\tau = 2)183.49 ± 0.55−6.92 ± 0.74
SpikeWeave_st (τ\tau = 32)187.63 ± 0.72−2.78 ± 0.57
SpikeWeave_st (τ\tau = 1024)190.29 ± 0.76−0.12 ± 0.91

Among the models above, SpikeWeave achieves the highest mean accuracy: (90.41), against (83.49)–(90.29) for SpikeWeave_st, demostrating the proposed DEMA method is effective in sequence modeling.

Reviewer_DFxpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpikeWeave processes one input sequence position per neuronal timestep, combining persistent multiscale dendritic EMA states with a separate spike-generating soma. The dendritic state is not reset when the soma spikes. Somatic events condition queries, keys, and modulation signals, while causal spiking attention is restricted to fixed-length chunks. Recurrent states carry compressed history across chunk boundaries. The architecture also uses gated residual updates and a spiking FFN with a continuous content branch. Evaluation covers sequential/permuted MNIST, SHD, and five LRA tasks. A separate ListOps reference variant is studied with fixed-weight boundary-reset and branch-suppression interventions.

Critical strengths
  1. The separation of persistent dendritic memory from a resetting soma is a coherent interface for spiking attention. It allows event generation to depend on a temporal summary without directly resetting that summary. The paper clearly specify the recurrence, readout, and cross-chunk carry; Proposition 1 gives a valid scaled-EMA interpretation.

  2. The method achieves competitive predictive performance across several modalities, including 82.40% average LRA accuracy and 87.64% on Text (Tables 1-2). The fixed-weight reset study is also meaningful evidence that the analyzed checkpoint uses its dendritic carry: accuracy falls from 57.42% to 36.72%, with a paired interval excluding zero. The paper carefully states that this does not isolate query/key addressing, and distinguishes checkpoint-conditional uncertainty from training-seed variability.

Critical weaknesses
  1. The closest architectural precursor is missing, making the novelty insufficiently delimited. MEGA (Ma et al., ICLR 2023) already combines multidimensional damped EMA, EMA-conditioned queries/keys, gating, and fixed-length chunked attention. SpikeWeave's linear dendritic core in Equations 15-17 matches MEGA's EMA parameterization, including the learned retention/input factors, branch readout scale, and direct-input contribution. The additional spiking soma and event interfaces are meaningful differences, but the EMA-plus-chunked-attention design should not be treated as new without this comparison. Neither MEGA nor a closely matched continuous control appears in the discussion or experiments. This leaves the scientific increment and its accuracy-computation tradeoff unclear.

  2. The experiments do not isolate the benefit of the proposed spiking memory-attention combination. There is no trained memory-only/no-attention, single-timescale, or matched non-spiking control. The mechanism study uses a different ListOps variant: width 80, chunk size 512, Pre-LN, an extra FFN content LIF, and no SSA RMS normalization; the main preset has width 128, chunk size 128, Post-LN, and SSA RMS normalization. Its reset drop establishes dependence in that checkpoint, but does not establish that attention or multiscale memory causes the main architecture's performance. The efficiency side of the tradeoff is also unmeasured: there are no whole-model operation counts, runtime, peak-memory, or energy comparisons. The reported soma event rates cover one population in the reference variant; DEMA, normalization, and the continuous gated FFN output projection retain real-valued computation. The asymptotic attention bound is valid, but it does not quantify the practical benefit of the spiking adaptation.

Questions for authors
  1. Please explain the relationship to MEGA/MEGA-chunk and identify precisely which components are new beyond its damped EMA and gated chunkwise attention. Can a matched continuous MEGA-style control establish the cost and accuracy effects of the spiking soma and event interfaces?

  2. Please provide whole-model operation accounting or runtime/peak-memory measurements at the evaluated sequence lengths, including continuous DEMA decay/readout, normalization, and the FFN output projection. What implementation justifies the Parallel checkmark in Table 1 for the resetting LIF populations? This requests evidence for existing efficiency claims, not deployment on a new neuromorphic platform.

6 comments
AuthorsOpenReview ↗

Rebuttal for W1 & Q1

We sincerely thank the reviewer for raising this issue. Below we will respond to the questions raised one by one.

a. Relationship between our method and MEGA, and more precise delimitation the contribution of this work.

Here we clarify the relationship between our method and MEGA, and more precisely delimit the contribution of this work.

Relationship. SpikeWeave is an SNN realization for sequence tasks. Our motivation is that existing Spiking Transformers rely on direct encoding, so neuronal dynamics evolve along a simulation-time axis independent of the sequence. We instead want each sequence position to correspond to one neuronal update while still retaining long-range context, thereby avoiding the redundant direct encoding and attention computation of existing SNN Transformers on long sequences. During the initial design, we became aware of MEGA (Ma et al., ICLR 2023). Its multidimensional damped EMA, EMA-parameterized query and key generation, and restriction of attention to fixed-length chunks achieve strong long-sequence performance under limited memory. SpikeWeave builds on this design and extends it to SNNs.

What is new:

  1. We analyze MEGA’s multiscale EMA as dendritic computation under a somatic reset, and show that placing its recurrence in dendritic compartments lets spike generation use the integrated summary without erasing it.
    Sequence modeling requires long-range memory that persists across positions, which conflicts with the reset mechanism of SNN neuron layers. This conflict is especially severe when sequence positions are aligned with neuronal update timesteps: every spike then occurs at a timestep where memory should be retained, and the reset directly erases the sequence summary being integrated. We therefore place this EMA in dendritic compartments on which the somatic reset does not act, and use the soma only to convert the dendritic readout into spike events. As the reviewer points out, event generation can then depend on an integrated temporal summary without reset. The linear recurrence and parameterization of the multiscale EMA follow MEGA and are not a contribution of this work.

  2. We present a spike-driven temporal SNN Transformer architecture connecting chunked SSA with cross-chunk dendritic memory.
    Its overall partition into local attention and cross-chunk recurrent memory follows MEGA-chunk and is not a contribution of this work. Dendritic states accumulate temporal information across chunks; the soma converts the dendritic readout into spikes, which are then used for causal SSA within each chunk. In SpikeWeave, the Q,K,V, update gate, and context gate are binary or ternary events, so the projections and attention require only accumulations. The architecture also parallelizes well. Dendrite-soma separation keeps the memory recurrence linear after the spiking adaptation, so this part can still be computed efficiently in parallel: dendritic states are processed over the whole sequence by one causal FFT convolution, and intra-chunk SSA is parallel within a chunk. Only the somatic membrane and the other LIF populations require a sequential scan along the sequence.

In the revised Intro, Related Work, and Sec. 4.2, we will state explicitly that DEMA's linear recurrence and parameterization follow MEGA, that chunked attention follows MEGA-chunk, and that we offer the dendrite–soma separation as an interpretation of how an existing EMA function can coexist with spike generation.

b. Matched continuous control: accuracy and computation.

We trained the original SpikeWeave and a control model SpikeWeave_cont on PS-MNIST experiment using a RTX 5090 GPU. In the control model, the EMA recurrence and parameterization are unchanged and only the spiking soma and event interfaces are replaced by continuous activations. Both models receive the same pixel sequences and the same fixed permutation. Both use seed=131071, L=2, D=d_v=d_ff=64, d_q=32, K=16, n_c=49, the same causal chunk mask, α=1/64, no softmax. The EMA update is

p(k)=σ(δ(k)), ρ(k)=1−p(k)⊙σ(α(k))p^{(k)}=\sigma(\delta^{(k)}),\ \rho^{(k)}=1-p^{(k)}\odot\sigma(\alpha^{(k)}),

mt(k)=ρ(k)⊙mt−1(k)+(p(k)⊙β(k))⊙xtm^{(k)}_{t}=\rho^{(k)}\odot m^{(k)} _{t-1}+(p^{(k)}\odot\beta^{(k)})\odot x_t,

rt=K−1/2∑k=1Kγ(k)⊙mt(k)+ω⊙xtr_t=K^{-1/2}\sum_{k=1}^{K}\gamma^{(k)}\odot m^{(k)}_t+\omega\odot x_t.

N=784SpikeWeaveSpikeWeave_cont
Param.90,43288,128
Acc./%86.5894.28
MACs + ACs per token10,959 + 17,16787,608 + 1,984
Train/Infer. time (batch 32)/ms26.7 / 9.011.2 / 3.0
Peak training memory/MB1,087542

Accuracy. SpikeWeave cost 7.70 percentage points under this protocol (86.58% versus 94.28%).

Computation. The replacement cuts MACs from 87,608 to 10,959 per token, while ACs rise from 1,984 to 17,167, reducing total arithmetic operations from 89,592 to 28,126. The SpikeWeave is 2.4×/3.0x slower in training/inference, and uses 2.0× more peak memory at N=784. A 4.6 pJ/MAC and 0.9 pJ/AC energy estimate gives 66 versus 405 nJ/token.

AuthorsOpenReview ↗

Rebuttal for W2 & Q2

We sincerely thank the reviewer for raising this issue. Below we will respond to the questions raised one by one.

a. Performance ablation on attention/multiscale memory

We trained each model below on PS-MNIST from random initialization with three seeds (131071, 1, 2). We report the test accuracy on a 6,000-sequence validation split (mean ± std over seeds). Δ is the same-seed difference to SpikeWeave (mean ± std). All models share L=2,D=dv=dff=64,dq=32,K=16L = 2, D = d_v = d_{ff} = 64, d_q = 32, K = 16 (except SpikeWeave_st), nc=49n_c = 49 and N=784N = 784. They are trained with AdamW (peak 3e-3, global batch 128, 5,000 updates).

  • SpikeWeave_st (single timescale) keeps one dendritic branch per channel (K = 1). All 64 channels share one dendritic time constant τ\tau, which stays fixed during training. We test τ∈\tau \in{2, 32, 1024}: the two ends and the middle of the K = 16 initialization grid.

  • SpikeWeave_diagSSA (no within-chunk attention) replaces the chunk mask PbP_b of Eq. 21 by the identity, so position ii only reads its own value: $y_i = α(q_i·k_i)v_i. $

  • SpikeWeave_cont (matched continuous control) is the model described in our response to W1 & Q1. TSN_in becomes the identity, the soma is removed so that Q/K read the DEMA readout, and the other spiking neurons become SiLU or sigmoid.

ModelTypeAcc. (%)Δ\Delta (mean ±\pm std)
SpikeWeaveSNN90.41±0.2890.41 \pm 0.28—
SpikeWeave_st (τ=2\tau = 2)SNN83.49±0.5583.49 \pm 0.55−6.92±0.74-6.92 \pm 0.74
SpikeWeave_st (τ=32\tau = 32)SNN87.63±0.7287.63 \pm 0.72−2.78±0.57-2.78 \pm 0.57
SpikeWeave_st (τ=1024\tau = 1024)SNN90.29±0.7690.29 \pm 0.76−0.12±0.91-0.12 \pm 0.91
SpikeWeave_diagSSASNN85.27±1.6285.27 \pm 1.62−5.14±1.35-5.14 \pm 1.35
SpikeWeave_contANN hybrid93.73±0.1393.73 \pm 0.13+3.31±0.41+3.31 \pm 0.41

Among the spiking models, SpikeWeave achieves the highest mean accuracy: 90.4190.41, against 85.2785.27 for SpikeWeave_diagSSA and 83.4983.49–90.2990.29 for SpikeWeave_st.

b. Per-layer event rates and MACs

We evaluate the layer—wise firing rate of SpikeWeave and its OPs/token on LRA Text task. The architecture follows the Appendix B.2: L=6L=6, D=dv=dff=256D=d_v=d_{ff}=256, dq=64d_q=64, K=16K=16, nc=512n_c=512, Pre-LN. We run all 25,000 test sequences (31,714,156 valid positions) in eval mode and FP32.

Event rates (mean over the 6 layers):

PopulationMeanMin–max across layers
TSNinTSN_{in}0.3940.334–0.565
DEMA soma0.0340.009–0.050
QQ0.4800.381–0.614
KK0.5080.419–0.638
VV0.4440.291–0.511
Context gate RR0.6100.318–0.713
Update gate UU0.2220.058–0.412
Context event0.3210.224–0.386
FFN input0.2580.161–0.316
FFN gate density gGg_G0.1290.092–0.165

Operation counts and energy estimation:

Count per tokenMAC(M)AC(M)Energy(uJ)
SpikeWeave0.150.761.37

Details are as follows listed:

ComponentMACAC
DEMA decay24,5760
DEMA readout24,5760
Normalization25,34416,896
FFN output projection W2W_250,7970
DEMA input09,685
DEMA direct term0605
Q/K projections06,577
V projection0154,966
Chunk QK⊤QK^\top029,589
Scores ×\times V0159,543
Modulation projections039,459
Attention output projection0126,375
FFN input projections0202,729
LIF membrane updates21,5044,549
SSA score scaling1,4290
Residual and candidate additions04,608
Six-layer total148,226755,581
Final LayerNorm, pooling, and classifier820768
Full model149,046756,349

Using an arithmetic estimate at 45 nm (0.9 pJ/AC and 4.6 pJ/MAC) gives 1.37uJ/token. MACs contribute 50.2%.

AuthorsOpenReview ↗

Rebuttal for Q2 (continued)

c. Parallel implementation claim

We sincerely thank the reviewer for pointing out this issue. In SpikeWeave, the components parallel across sequence positions are DEMA and intra-chunk SSA. The binary and ternary LIF populations use a fused sequential scan.

Each LIF population uses subtractive reset, so the membrane at position tt depends on whether position t−1t-1 fired. This dependence is nonlinear. Neither an FFT nor a parallel scan removes it. For efficient training, all ten populations share one fused Triton kernel. Each program handles 64 neurons and loops over the LL positions inside the kernel. The membrane remains in registers, and the backward pass is a reverse-time loop in a second kernel. Fusion removes the cost of launching one kernel per position. It does not make positions independent.

On an RTX 5090, inference for a ternary population of shape 4×4000×2564\times4000\times256 takes 2.95 ms with the fused kernel and 303.9 ms with the Python loop. The forward-backward times are 7.54 ms and 1,215 ms. Across the measured shapes, fusion is 46–267× faster.

SpikeWeave is therefore partial parallel: DEMA and intra-chunk SSA are parallel, while the ten resetting LIF neuron layers use a fused sequential scan. We will further correct this distinction in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.