acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Pairwise Token Interaction in Spiking Transformers

Abstract

Spiking Transformers inherit token interaction mechanisms from conventional Transformers, yet their sparse binary representations fundamentally alter how token-to-token communication is established. In particular, spike-based query–key matching produces highly sparse and input-dependent interaction patterns, coupling information propagation to the instantaneous availability of matching spike events. This motivates a different interaction paradigm in which long-range communication does not rely solely on pairwise spike coincidence. We therefore propose **Gated Spike Axial Propagation (GSAP)**, a spike-native token interaction mechanism that decouples information propagation from context selection. Instead of directly determining communication through query–key matching, GSAP first propagates spike-based context along the horizontal and vertical axes, allowing information to reach distant tokens through structured sequential propagation. A receiver-conditioned gate then determines how much of the propagated context is incorporated at each token, while a lightweight local pathway preserves fine-grained neighborhood information. In this way, GSAP reformulates token interaction as a propagate-then-select process, enabling structured long-range communication while retaining the sparse event-driven nature of spiking representations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spiking Transformer, Spiking Attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CM3ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the concern that sparse query–key matching in Spiking Transformers may restrict long-range information exchange. It proposes Gated Spike Axial Propagation (GSAP), which uses axial propagation for global contextual interaction and receiver-conditioned gating to regulate context utilization. GSAP further incorporates a local feature pathway and an input residual connection to integrate local and global information while preserving spiking representations and temporal dynamics. Experiments demonstrate improvements over the baseline models on image classification, event-based recognition, and semantic segmentation. Ablation studies and token-perturbation analyses further investigate the contributions of individual components and the information propagation properties of the proposed mechanism.

Critical strengths
  1. The paper evaluates GSAP on multiple static-image classification, event-based recognition, and semantic segmentation benchmarks, demonstrating its applicability across different tasks and backbone architectures.

  2. The paper identifies a potential limitation of communication based on sparse query–key coincidences and presents a clearly motivated design that separates spatial context propagation from receiver-conditioned context selection.

Critical weaknesses
  1. Insufficient comparison with stronger ImageNet baselines, such as QKFormer and SDT-v3 [1].

  2. GSAP is structurally highly similar to ParCNetV2 [2]. Both employ large one-dimensional convolutional kernels along two spatial axes to aggregate information, a local feature branch, a gating-like branch, and related feature fusion operations.

  3. The ablations are restricted to relatively small datasets, and several reported performance differences are below 0.2 percentage points. Without results across multiple random seeds or uncertainty estimates, it is difficult to determine whether these differences reflect reliable improvements or training variability. Moreover, the local-only variant performs worse than the baseline, leaving the standalone benefit of the local pathway unclear.

[1] Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training. TPAMI 2025.

[2] ParCNetV2: Oversized Kernel with Enhanced Attention. ICCV 2023.

Questions for authors
  1. The paper states that SSA is severely affected by sparsity, and Figure 1 shows that SSA is sparser than GSAP. Why, then, does SSA consume more energy than GSAP?

  2. What are the core methodological differences between GSAP and ParCNetV2, beyond the introduction of spiking neurons? Which experiments demonstrate the benefits of these differences?

  3. How does GSAP affect temporal spike patterns?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CM3Z (1/2)

Thank you for noting GSAP's similarity to ParCNetV2 and asking about energy and spike timing. Below we address these questions and stronger ImageNet baselines, and give full three-seed ablations and additional temporal analyses.

1. Stronger ImageNet Architectures

We compared QKFormer and its GSAP variant at 10–384 under identical settings.

ModelImageNet Top-1 (%)ParamsMixer GFLOPs/imageMixer SOP (G/image)Energy (mJ/image)
QKFormer74.2916.47M7.01343.25952.9413
QKFormer + GSAP74.4615.69M5.12100.75420.7020

GSAP saves 0.78M parameters and about 76.1% attention energy, with an observed Top-1 gain of +0.17 percentage points (pp), showing successful adaptation to the stronger QKFormer with energy and performance gains. We plan to report QKFormer + GSAP results at more model scales in future versions.

SDT-V3 mainly optimizes SDT-series neuronal firing patterns. Like Spikingformer, it is a single-stage spiking Transformer. We hope to explore compatibility between GSAP and SDT-V3's neuronal mechanisms in future work.

2. ParCNetV2 and Component Benefits

(a) Structure and Direct Comparison

Both mixers use large axial kernels, local features and gating, with these structural differences:

Mixer designParCNetV2GSAP
Local/global sourcesShared projected featuresLocal: input spikes; global: separately projected features
Inter-axis connectionSequential axial convolutionRow shortcut before spike re-encoding and column propagation
Gating scopeLocal and global features jointlyPropagated global messages
FusionGated local and global featuresIdentity + local features + gated global messages

Based on the official implementation, our ParCNetV2 spiking adaptation in Spikingformer retains shared projected features and joint gating, maintaining binary synaptic inputs. Three-seed results:

MethodCIFAR-100CIFAR10-DVSN-Caltech101
Spiking adaptation of ParCNetV280.07 ± 0.1880.43 ± 0.4685.53 ± 0.56
GSAP80.22 ± 0.0781.17 ± 0.6485.54 ± 0.85

GSAP's mean accuracy gains over the spiking adaptation are 0.15, 0.74, and 0.01 pp, respectively. On CIFAR-100, its estimated energy is 12.39% lower: 0.080613 versus 0.092018 mJ/image under the same counting method.

(b) Three-Seed Ablations and Local Branch

Three-seed ablations examine the components and propagation path:

MethodCIFAR-100CIFAR10-DVSN-Caltech101
SSA baseline79.97 ± 0.1880.97 ± 1.0085.01 ± 0.18
Local branch only78.83 ± 0.2080.77 ± 0.6185.50 ± 0.88
Global branch only79.49 ± 0.2180.53 ± 0.4085.38 ± 0.31
Large-kernel 2D convolution80.62 ± 0.2080.63 ± 0.3585.70 ± 0.50
Parallel axial implementation79.85 ± 0.0880.10 ± 1.0185.49 ± 0.49
Without gating79.56 ± 0.3280.47 ± 0.5585.58 ± 0.37
GSAP80.22 ± 0.0781.17 ± 0.6485.54 ± 0.85

GSAP's mean gains over SSA are 0.25, 0.20, and 0.53 pp, respectively. Its mean accuracy is also higher than that of the parallel axial variant tested on each dataset.

The local branch supplies neighborhood detail; the global branch aggregates long-range information. On CIFAR-100, local-only yields 78.83%, below SSA's 79.97%; both branches yield 80.22%. Full GSAP improves global-only mean accuracy by 0.73, 0.64, and 0.16 pp, respectively, supporting effective joint local–global modeling.

The 2D variant replaces both axial operators, the inter-axis LIF and the row shortcut, so we compare the performance and cost of the full paths. Its mean accuracy is higher on CIFAR-100 and N-Caltech101; GSAP is better on CIFAR10-DVS. At k=15k=15, axial kernels use 30C30C weights versus 225C225C for 2D, giving GSAP 86.7% fewer propagation kernel parameters. This illustrates the trade-off between accuracy and propagation kernel cost.

3. Sparse Matching and Energy

(a) Sparse Matching and Energy in SSA

Q–K matching sparsity and neuronal firing sparsity measure different properties: zero-inner-product token-pair rates versus zero-spike rates in neuronal outputs. For binary Q/K in one head and step, only co-firing in corresponding channels, (Qi,c,Kj,c)=(1,1)(Q_{i,c},K_{j,c})=(1,1), contributes nonzero values to QiKj⊤Q_iK_j^\top.

For Qi=(1,0)Q_i=(1,0) and Kj=(0,1)K_j=(0,1), both groups fire but their inner product is zero: an unmatched token pair does not mean its tokens did not fire. Projections have already occurred; these spikes may match other tokens. To estimate total arithmetic energy, we sum costs across input projection, spatial aggregation, output projection, fusion and other stages, accounting for actual spike activity, operator scale and execution method.

AuthorsOpenReview ↗

Response to Reviewer CM3Z (2/2)

Tab.8's network-wide structural comparison uses a uniform effective activity rate of ρ=0.1\rho=0.1. These supplementary Spikingformer token mixer estimates use actual spike activity from trained checkpoints:

DatasetMixerGFLOPseMAC (M)SOP (B)Energy (mJ)
ImageNetSSA31.4453.8743.8353.4697
ImageNetGSAP22.80311.5193.1752.9109
CIFAR-100SSA1.3090.3160.094480.086485
CIFAR-100GSAP0.9370.9400.084760.080613

Both use T=4T=4: ImageNet uses 8–768 and 1,000 validation images; CIFAR-100 uses 4–384 and 10,000 test images. GFLOPs counts dense computation; eMAC is MAC-equivalent floating-point arithmetic. Energy coefficients are 4.6 pJ/eMAC and 0.9 pJ/SOP.

Relative to SSA on ImageNet and CIFAR-100, GSAP reduces arithmetic energy by 16.10% and 6.79%, SOPs by 17.21% and 10.28%, and dense computation by 27.48% and 28.43%, respectively. Savings from fewer spike accumulations outweigh the added floating-point arithmetic costs.

(b) Savings Mainly Come from Input Projection

The mixer energy breakdown (mJ/image) identifies the savings:

Energy sourceImageNet SSAImageNet GSAPCIFAR-100 SSACIFAR-100 GSAP
Input projection2.81171.71980.0728700.048306
Spatial aggregation0.00940.02820.0007740.001584
Output projection0.63071.10990.0113860.026398
Scaling and fusion0.01780.05300.0014550.004325
Total3.46972.91090.0864850.080613

SSA's Q/K/V projections consume 81.04% and 84.26% of mixer arithmetic energy on ImageNet and CIFAR-100. GSAP's feature and gating projections save about 1.0919 and 0.0246 mJ/image, covering higher spatial aggregation, output projection, scaling and fusion costs, for net savings of 0.5588 and 0.005872 mJ/image, respectively. Lower input projection cost is the main source of savings on both datasets.

Under the same CIFAR-100 counting method, GSAP and the spiking adaptation of ParCNetV2 use 0.080613 and 0.092018 mJ/image of mixer arithmetic energy, respectively, a 12.39% reduction for GSAP.

4. Temporal Spike Patterns

(a) Firing and Temporal Persistence

We measured per-step firing before Block 8's output projection for Spikingformer-8-768 and GSAP-8-768 on the same 1,000 ImageNet validation images (T=4T=4). To assess sustained early-information propagation, we cleared GSAP's inter-axis LIF membrane state each step and tracked remote firing changes after removing first-step central-source spikes.

StepSSA firing rateGSAP firing rateRemote response: native GSAPRemote response: state cleared
13.0%18.9%49.7%49.7%
28.5%39.2%50.1%33.0%
310.1%36.3%46.1%26.3%
411.3%42.0%41.7%22.4%

GSAP has higher firing rates at every step. State clearing leaves first-step remote responses unchanged but weakens later responses, from 41.7% to 22.4% at step 4. Top-1 remains close (78.4% vs 78.2%, a gap of 0.2 pp). Thus, the inter-axis membrane state helps sustain early information's effects on later remote firing.

(b) Prediction Timing and Online Learning

Each model's original classification head uses the mean of the spatially pooled features over the first t simulation steps:

StepsSSA (%)GSAP (%)Gain and 95% interval (pp)
153.363.5+10.2 [7.1, 13.3]
270.975.6+4.7 [2.0, 7.2]
373.076.8+3.8 [1.3, 6.1]
474.978.4+3.5 [1.2, 5.7]

GSAP leads by 10.2 pp at step 1 and stays ahead thereafter. SSA correctly classifies 50.6% of images at every step (1–4), versus 60.9% for GSAP, showing that GSAP makes correct predictions earlier and maintains them. The 95% intervals use paired image resampling at fixed checkpoints.

Training results are from the original Fig. 4 online-learning experiment with the GSAP propagation variant without receiver gating, using T=6T=6, 200 epochs and seed 2022.

DatasetOTTT-A: SSA → propagation variant (gain)OTTT-O: SSA → propagation variant (gain)
CIFAR-1082.51 → 83.97 (+1.46)89.30 → 91.19 (+1.89)
CIFAR-10051.48 → 62.07 (+10.59)63.05 → 68.49 (+5.44)

Accuracy is in %; gains in parentheses are in pp. OTTT-A accumulates gradients across steps before updating; OTTT-O updates each step. The propagation variant is more accurate under both. Fig. 4's mean cross-step gradient cosine similarity rises from 0.164 to 0.311, indicating more consistent cross-step gradient directions.

Thanks again for suggestions that improved our analysis and presentation. We hope this addresses your questions. We will add the relevant results and analyses in future versions for clearer, more complete arguments.

Thank you again for your thoughtful review. We wanted to follow up on your question about ParCNetV2. In the same Spikingformer backbone, GSAP and the spiking adaptation of ParCNetV2 achieved similar mean accuracy across three seeds, with lower estimated mixer arithmetic energy for GSAP on CIFAR-100.

We would be happy to clarify any remaining questions about the comparison or the other results in our rebuttal. If the additional evidence addresses your main concerns, we would appreciate it if you would reconsider your overall assessment.

Reviewer_xZ6XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gated Spike Axial Propagation (GSAP) as an alternative to pairwise query-key matching in Spiking Transformers. It combines sequential axial spike propagation, receiver-conditioned gating, and local feature fusion. Experiments cover image classification, event-based recognition, and semantic segmentation.

Critical strengths

The paper addresses a relevant limitation of sparse spike-based attention and presents a clear alternative interaction mechanism. The improvements across different backbones and tasks are encouraging, particularly on ImageNet and semantic segmentation. The component analyses also provide useful support for the proposed design.

Critical weaknesses

The distinction between spike-based axial propagation and conventional spatial mixing could be better explained. Some ablations change multiple components together, and a few experimental claims need more precise wording.

Questions for authors
  1. Please clarify how the proposed spike-based propagation differs from conventional large-kernel spatial mixing beyond the use of spiking neurons.

  2. The 2D-kernel variant in Figure 3 also removes the intermediate LIF neuron and residual connection; please make this distinction clearer and avoid attributing the accuracy difference solely to the propagation operators.

  3. Figure 1 reports Q–K no-match rates for a selected example; could the authors clarify how representative these rates are?

  4. The scaling factor ss in Eq. (2) is not defined in the accompanying text; please specify its meaning.

  5. Table 4 shows that VOC aAcc decreases from 73.73 to 72.59 after adding GSAP; please acknowledge this exception when discussing segmentation improvements.

4 comments
AuthorsOpenReview ↗

Response to Reviewer xZ6X (1/2)

Thank you for recognizing the novelty of our work and our experimental results. We address your questions in turn.

1. How GSAP differs from conventional large-kernel spatial mixing

(a) Architecture and mixer comparison

GSAP uses large-kernel depthwise convolution for propagation. The local branch receives input spikes directly, while the global branch takes independently projected features. A row shortcut and spike re-encoding connect the two axes. The receiver gate selects only global messages. Local and identity features enter fusion independently. This replaces interactions that require source–receiver Q–K co-occurrence with axial propagation and receiver-side selection.

ParCNetV2 also uses large axial kernels, a local branch and gating. Its local and global branches share projected features and are gated jointly. Our spiking adaptation of ParCNetV2 in Spikingformer preserves this structure and keeps synaptic inputs binary.

Token mixerCIFAR-100CIFAR10-DVSN-Caltech101
SSA79.97 ± 0.1880.97 ± 1.0085.01 ± 0.18
Spiking adaptation of ParCNetV280.07 ± 0.1880.43 ± 0.4685.53 ± 0.56
Large-kernel 2D convolution80.62 ± 0.2080.63 ± 0.3585.70 ± 0.50
Parallel axial implementation79.85 ± 0.0880.10 ± 1.0185.49 ± 0.49
GSAP80.22 ± 0.0781.17 ± 0.6485.54 ± 0.85

Entries are the mean ± sample standard deviation (%) of the highest test accuracy per run (best-on-test) across three seeds. In table order, GSAP's reported mean gains are 0.25, 0.20 and 0.53 percentage points (pp) over SSA and 0.15, 0.74 and 0.01 pp over spiking ParCNetV2. These higher means across datasets demonstrate GSAP's effectiveness as a Spiking Transformer token mixer.

(b) Long-range propagation

At step 1 we removed only the spikes one source sent to the global branch (its V in SSA or its propagation features in GSAP). All other computations were unchanged. We used the same 1,000 ImageNet validation images, each model's native stem and fixed 8–768 checkpoint, and four steps. Remote receivers share neither the source's row nor its column and are more than two tokens away in Chebyshev distance. A firing change in any channel of the LIF before output projection counts as a response.

Remote response coverageSSA, step 1GSAP, step 1SSA, step 4GSAP, step 4
Block 432.8%32.8%4.0%24.2%
Block 812.6%45.5%1.3%38.4%

Averaging over images and three fixed source positions, we see GSAP's Block 4 coverage advantage mainly at later steps. In Block 8, responses are broader and persist longer. GSAP also elicits a response at 45.1% of Block 8 positions where SSA lacks an effective first-step path from that source. Thus GSAP can reach these positions and sustain the source's influence across steps.

2. What the 2D-kernel ablation in Fig. 3 compares

The 2D variant replaces both axial operators and removes the inter-axis LIF and row shortcut, so Fig. 3 compares entire propagation paths. We will clarify these changes in §5.1 and the caption and interpret the accuracy differences accordingly.

PathSpatial operatorInter-axis LIFRow shortcutPropagation kernel weights
GSAPSequential horizontal and vertical depthwise convolutionsRetainedRetained2Ck2Ck
2D variantSingle 2D depthwise convolutionRemovedRemovedCk2Ck^2

In the original single runs (Fig. 3), the 2D variant and GSAP reached 96.12% and 95.95% on CIFAR-10 and 80.43% and 80.21% on CIFAR-100, respectively. Across three seeds, 2D has higher mean accuracy on CIFAR-100 and N-Caltech101, while GSAP leads on CIFAR10-DVS. At k=15k=15, the 2D and axial kernels use 225C225C and 30C30C weights, respectively. GSAP uses 86.7% fewer propagation kernel parameters. We will discuss this accuracy–parameter trade-off in §5.1.

To test the inter-axis state with weights fixed in one GSAP checkpoint, we cleared only the Block 8 inter-axis LIF membrane state before each step. We retained binary firing, the row shortcut and all other computations. We measured remote responses after removing the central source at step 1.

StepNative GSAPInter-axis state cleared each step
149.7%49.7%
250.1%33.0%
346.1%26.3%
441.7%22.4%

Coverage is identical at step 1 and diverges from step 2 onward. Retaining the state raises step-4 coverage from 22.4% to 41.7%. Four-step Top-1 is 78.4% with the native state and 78.2% after clearing. The confidence interval for the difference crosses zero. The fixed-weight comparison shows that the inter-axis membrane state prolongs the source's influence on later propagation.

AuthorsOpenReview ↗

Response to Reviewer xZ6X (2/2)

3. Representativeness of sparse matching in Fig. 1

For Fig. 1, we sampled one ImageNet validation image per class (1,000 images, 224×224, T=4T=4). For each head and step, the zero-pair rate is the fraction of source–receiver pairs with zero Q–K inner product.

LayerZero-pair rateRange, middle 80% of imagesTokens with no source match, one head/step (%)Pairs unmatched across all heads/steps (%)
Block 267.8%66.0–69.4%0.19%<0.01%
Block 479.8%78.1–81.4%2.37%3.40%
Block 687.8%86.1–89.3%2.39%9.90%
Block 885.3%84.0–86.6%4.40%2.67%

Narrow ranges within each layer show that instantaneous sparsity is representative across images. Heads and steps provide complementary paths. Across eight heads and four steps, every token in the eight evaluated blocks matches at least one source, though some pairs remain unmatched. GSAP addresses the dependence of current-step source–receiver interactions on spike co-occurrence.

We raised the Q, K and V LIF inference thresholds in a trained SSA checkpoint, fixing weights and all other neuronal thresholds. We tested 1,000 images per dataset.

QKV LIF thresholdCIFAR-100 QK match rate (%)CIFAR-100 Top-1 (%)ImageNet QK match rate (%)ImageNet Top-1 (%)
1.0030.0079.9020.4974.90
1.2522.5376.9015.5373.00
1.5014.7969.7012.2965.20
1.758.7061.608.1553.00
2.004.7957.004.5038.60

On both datasets, sparser QKV firing accompanies lower matching rates and accuracy. Since the intervention changes Q–K matching, V activity and aggregated current together, it measures SSA's overall sensitivity to QKV spike sparsity.

On the same ImageNet images, each model's original classification head used the mean of spatially pooled features from the first tt steps.

StepsSSA Top-1 (%)GSAP Top-1 (%)Gain and 95% interval (pp)
153.363.5+10.2 [7.1, 13.3]
270.975.6+4.7 [2.0, 7.2]
373.076.8+3.8 [1.3, 6.1]
474.978.4+3.5 [1.2, 5.7]

GSAP leads throughout steps 1–4, with a 10.2 pp advantage at step 1. The share of images classified correctly at every step increases from 50.6% to 60.9%, showing that GSAP makes correct predictions earlier and maintains its accuracy lead. Intervals use paired image resampling with fixed checkpoints.

In Fig. 4 we tested the GSAP propagation variant without receiver gating in online learning (T=6T=6, 200 epochs).

Dataset / training methodSSA (%)Ungated propagation variant (%)Gain (pp)
CIFAR-10 / OTTT-A82.5183.97+1.46
CIFAR-10 / OTTT-O89.3091.19+1.89
CIFAR-100 / OTTT-A51.4862.07+10.59
CIFAR-100 / OTTT-O63.0568.49+5.44

OTTT-A updates after accumulating gradients across steps. OTTT-O updates each step. The ungated variant outperforms SSA under both methods on both datasets, gaining 10.59 and 5.44 pp on CIFAR-100. Together with prediction timing, this shows that propagation-based interactions can improve recognition and online learning in Spiking Transformers.

4. Experimental clarifications

(a) Scaling factor ss

ss scales the aggregated current entering the attention LIF, Jt=s(QtKt⊤)VtJ_t=s(Q_tK_t^\top)V_t. Our SSA baseline uses the fixed, non-learned scalar s=0.125=1/8s=0.125=1/8. We will add its definition and value after Eq. (2).

(b) Segmentation metrics and results

mIoU is our primary segmentation metric. It averages per-class intersection-over-union between predicted and true regions. This measures overlap with equal class weights and accounts for false positives and false negatives. Our secondary metric, aAcc, measures the fraction of valid pixels predicted correctly and can be dominated by classes with more pixels. Table 4 shows all changes.

DatasetBackbonemIoUmAccaAcc
ADE20KQKFormer32.63 → 36.90 (+4.27)43.33 → 47.88 (+4.55)75.56 → 77.63 (+2.07)
ADE20KSpikingformer31.64 → 33.82 (+2.18)41.85 → 43.63 (+1.78)74.99 → 76.49 (+1.50)
VOCQKFormer60.76 → 62.23 (+1.47)90.44 → 91.13 (+0.69)73.73 → 72.59 (−1.14)

Values are baseline → GSAP (%), with changes in pp in parentheses.

mIoU gains 4.27, 2.18 and 1.47 pp in the three settings, and mAcc improves in each. VOC aAcc drops from 73.73% to 72.59% (−1.14 pp). In §4.3 we will highlight the segmentation quality gains reflected by mIoU and report all aAcc changes.

Thank you for your constructive comments. We hope this answers your questions and look forward to further discussion.

Thank you again for your constructive feedback. We wanted to follow up and see whether any points in our response, including the matching analysis in Fig. 1 or the ablation study in Fig. 3, would benefit from further clarification. We have also provided additional explanations of certain details of the paper and conducted further experiments to validate our findings. We would be happy to address any remaining questions you may have.

Thank you for the detailed response and additional experiments, which have addressed several of my concerns. I appreciate the clarifications and will maintain my original rating.

Reviewer_TAcROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an interesting attempt to reconsider token interaction mechanisms in Spiking Transformers. The proposed GSAP achieves promising improvements across multiple visual recognition tasks, and the experimental results demonstrate its practical effectiveness.

However, I have several concerns regarding the novelty of the proposed mechanism, the justification of its central motivation, and the adequacy of the experimental comparisons. In particular, the distinction between GSAP and existing convolution-based spatial mixing mechanisms is not sufficiently established. Moreover, the current results do not fully demonstrate that the improvements originate from addressing the limitations of sparse pairwise spike matching.

Overall, I lean toward a weak reject at this stage. Nevertheless, I acknowledge the empirical improvements and the potential value of this research direction. I am open to raising my score if the authors can convincingly address the above concerns during the rebuttal period, particularly those concerning novelty, motivation, and controlled comparisons.

Critical strengths
  1. The paper identifies an important limitation of conventional pairwise query–key matching under sparse binary spike representations and provides a fresh perspective on token interaction in Spiking Transformers. By decoupling global information propagation from receiver-conditioned context selection, the proposed GSAP offers a meaningful alternative to traditional Spiking Self-Attention and better accommodates the sparse and temporal characteristics of spiking computation.

  2. The proposed GSAP features a clear and well-structured design that combines sequential horizontal–vertical spike propagation, receiver-conditioned gating, and local feature refinement. This design enables structured long-range communication without explicitly computing pairwise token affinities, while preserving local spatial information and maintaining a relatively lightweight architecture.

  3. The paper presents extensive experimental evaluations across static image classification, event-based recognition, and semantic segmentation benchmarks, demonstrating consistent improvements across different Spiking Transformer architectures. Furthermore, component ablations, token-perturbation analyses, different input encoding strategies, and online learning experiments provide additional evidence of the effectiveness, adaptability, and generality of the proposed approach.

Critical weaknesses
  1. Limited novelty of the proposed interaction mechanism.

GSAP primarily combines large-kernel depthwise axial convolutions, spiking neurons, receiver-conditioned gating, and local feature refinement. While this combination is effective, its distinction from existing convolution-based token mixers and gated spatial aggregation mechanisms is not sufficiently discussed. It remains unclear whether the main contribution lies in a fundamentally new token interaction paradigm or in adapting established spatial mixing operations to Spiking Transformers.

  1. Insufficient evidence supporting the motivation.

Figure 1 reports high fractions of tokens without Q–K matches for a selected example. However, the absence of spike coincidences does not necessarily imply ineffective information propagation or reduced recognition capability. The paper does not establish a direct relationship between Q–K sparsity and performance degradation. Further analysis across different samples, layers, and spike activity levels would help substantiate the proposed motivation.

  1. Incomplete isolation of the proposed mechanism's contributions.

Although the authors evaluate individual components, it remains unclear whether the improvements arise specifically from the proposed propagate-then-select paradigm or from introducing large receptive fields and additional convolutional spatial mixing. Comparisons with parameter-matched large-kernel depthwise convolution, parallel axial mixing, or simpler gated convolutional alternatives would make the contribution more convincing. Moreover, the 2D-kernel ablation also removes the intermediate spiking neuron and residual connection, making it difficult to attribute the observed differences solely to the propagation structure.

  1. Limited computational and energy-efficiency advantages.

According to Table 8, replacing QKFormer with GSAP reduces FLOPs by only 0.35%–0.52% and normalized energy by 0.24%–0.51% across the evaluated datasets. Furthermore, the energy estimates assume a common effective firing rate and exclude several practical costs, including memory access and neuron-state updates. Therefore, the claimed efficiency benefits would benefit from more careful qualification, particularly regarding deployment on neuromorphic hardware.

  1. Limited statistical reliability and mechanism-specific evidence.

Several improvements are relatively small, particularly on CIFAR-10 and event-based datasets, but results across multiple random seeds or confidence intervals are not reported. Additionally, the perturbation analyses mainly provide qualitative evidence that distant tokens can influence the output. They do not sufficiently establish that the learned propagation mechanism captures content-dependent long-range relationships more effectively than simpler convolutional alternatives.

Questions for authors

see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer TAcR (1/2)

Thank you for recognizing GSAP’s task performance and research value.

1. Novelty and sparse spike interactions

(a) Spatial interaction

Nearly all Spiking Transformer mixers use explicit Q–K matching as in SSA or its variants. GSAP replaces interactions requiring source–receiver Q–K co-occurrence with stateful axial propagation and receiver-side selection. It addresses performance and deployment barriers from sparse matching. Large axial kernels are one implementation. Its aim differs from transferring an established ANN mechanism to SNNs.

As reviewer CM3Z noted, ParCNetV2 [3] also uses large axial kernels, a local branch, and gating. Its branches share projected features and joint gating. In GSAP, the local branch processes input spikes. The global branch uses independently projected features and keeps the row shortcut during inter-axis re-encoding. The gate selects only global messages. Local and identity features enter fusion independently.

We adapted ParCNetV2 to SNNs in Spikingformer, retaining its feature sources and joint gating and inserting an inter-axis LIF for binary synaptic inputs.

MethodCIFAR-100CIFAR10-DVSN-Caltech101
ParCNetV280.07 ± 0.1880.43 ± 0.4685.53 ± 0.56
GSAP80.22 ± 0.0781.17 ± 0.6485.54 ± 0.85

We also calculated energy on CIFAR-100.

MethodEnergy consumption (mJ/image)
SSA0.086485
GSAP0.080613
ParCNetV20.092018

GSAP consistently improves performance over spiking ParCNetV2 and uses less energy on CIFAR-100, making it better suited as a new Spiking Transformer mixer.

(b) Sparse matching, performance, and deployment

In current spiking attention, a match requires Q and K to fire together, as also observed in α\alpha-XNOR [1] and ACF-SLI [2]. To test the prevalence of this pattern, we extended Fig. 1 to 1,000 ImageNet validation images, one per class, with the same input parameters as in training.

LayerZero Q–K pair rateRange for the middle 80% of images
Block 267.8%66.0–69.4%
Block 479.8%78.1–81.4%
Block 687.8%86.1–89.3%
Block 885.3%84.0–86.6%

Using a trained CIFAR-100 Spikingformer checkpoint, we manually varied only Q, K, and V LIF firing thresholds to control QKV spike sparsity. All weights and other thresholds stayed fixed. Sparser firing accompanies a sharp fall in Q–K match rate and a collapse in accuracy.

QKV LIF thresholdAcc@1 (%)ΔAcc@1 (pp)QK matching rate (%)
1.0079.90+0.0030.00
1.2576.90−3.0022.53
1.5069.70−10.2014.79
1.7561.60−18.308.70
2.0057.00−22.904.79

Deployed SNNs often face online learning, which typically requires gradients at every step (Sec. 5.2). In SSA, sparse Q–K matches produce sparse linear signals and cut off gradients for some weights. The gradients also become less stable across steps, preventing rapid, effective learning.

Fig. 4 shows SSA’s gradient instability and how the GSAP propagation variant without receiver gating substantially alleviates the online-learning bottleneck under OTTT-A and OTTT-O.

MethodOTTT-AOTTT-O
SSA51.4863.05
GSAP propagation variant without receiver gating62.0768.49
Gain (percentage points)+10.59+5.44

These gains show the value of propagation for online learning and the limits of SSA in current Spiking Transformers.

2. Component ablations and mixer comparisons

Our added ablations and mixer comparisons give mean accuracy ± sample standard deviation (%) over three seeds.

MethodCIFAR-100CIFAR10-DVSN-Caltech101
SSA baseline79.97 ± 0.1880.97 ± 1.0085.01 ± 0.18
Local branch only78.83 ± 0.2080.77 ± 0.6185.50 ± 0.88
Global branch only79.49 ± 0.2180.53 ± 0.4085.38 ± 0.31
Large-kernel 2D convolution80.62 ± 0.2080.63 ± 0.3585.70 ± 0.50
Parallel axial implementation79.85 ± 0.0880.10 ± 1.0185.49 ± 0.49
Without gating79.56 ± 0.3280.47 ± 0.5585.58 ± 0.37
GSAP80.22 ± 0.0781.17 ± 0.6485.54 ± 0.85

GSAP exceeds the tested parallel implementation on all three datasets. The large 2D kernel scores higher on CIFAR-100 and N-Caltech101, while GSAP scores higher on CIFAR10-DVS. At k=15k=15, the 2D kernel has 225C225C weights and the two axial kernels have 30C30C, 86.7% fewer propagation-kernel weights.

These results show the need for each GSAP component and consistent gains across benchmark types. Even where the large 2D kernel scores higher, GSAP’s accuracy–cost trade-off better suits efficient, low-power SNN inference.

AuthorsOpenReview ↗

Response to Reviewer TAcR (2/2)

3. Computational and energy advantages

Table 8 estimates whole-network structural costs at a common effective activity rate of ρ=0.1\rho=0.1. For QKFormer, normalized energy falls by 0.24%–0.51%. We also estimated SSA and GSAP mixer costs in Spikingformer using actual spike activity from trained checkpoints.

DatasetMixerGFLOPseMAC (million)SOP (billion)Energy consumption (mJ)
ImageNetSSA31.4453.8743.8353.4697
ImageNetGSAP22.80311.5193.1752.9109
CIFAR-100SSA1.3090.3160.094480.086485
CIFAR-100GSAP0.9370.9400.084760.080613

Both evaluations use T=4T=4. ImageNet uses the 8–768 model and 1,000 validation images. CIFAR-100 uses the 4–384 model and 10,000 test images. GFLOPs denotes the dense operation count and eMAC denotes MAC-equivalent floating-point arithmetic. We estimate energy with coefficients commonly used for SNNs, 4.6 pJ/eMAC and 0.9 pJ/SOP.

On ImageNet and CIFAR-100, GSAP reduces arithmetic energy by 16.10% and 6.79%, SOP by 17.21% and 10.28%, and dense computation by 27.48% and 28.43%, respectively. Spike accumulations account for most of the energy in the table. Their reduction saves more energy than GSAP’s extra floating-point arithmetic costs.

These results show that GSAP outperforms SSA in performance, parameter count, and energy consumption.

4. Statistical variation and long-range learning

Across three random seeds, GSAP achieves higher mean accuracy than the rerun SSA baseline on all three datasets.

DatasetSSA baselineGSAPMean gain (percentage points)
CIFAR-10079.97 ± 0.1880.22 ± 0.07+0.25
CIFAR10-DVS80.97 ± 1.0081.17 ± 0.64+0.20
N-Caltech10185.01 ± 0.1885.54 ± 0.85+0.53

The results show both positive mean gains across datasets and training variability. The CIFAR-100 gain uses our reproduced three-seed baseline. We also quantified long-range perturbation responses across the same 1,000 ImageNet validation images. With each 8–768 checkpoint fixed, we removed only one source’s spikes entering the global branch at step 1. Remote receivers lie outside the source’s row and column, with the larger of the row and column distances exceeding two tokens. A response occurs if any LIF channel before the output projection changes its firing. We averaged response coverage (%) over images and three fixed source positions.

LayerSSA step 1GSAP step 1SSA step 4GSAP step 4
Block 432.832.84.024.2
Block 812.645.51.338.4

In Block 4, the difference emerges mainly at later steps. In Block 8, GSAP has broader remote influence at both steps 1 and 4. Of positions where SSA lacks an effective first-step path from the source, 45.1% respond under GSAP in Block 8. Together with the fixed-weight state-clearing results, these statistics extend individual visualizations of propagation paths and temporal retention to quantitative evidence.

To compare predictions across steps, we averaged the spatially pooled features from the first tt steps and applied each model’s original classification head.

Steps usedSSA Top-1 (%)GSAP Top-1 (%)Gain and 95% interval (percentage points)
153.363.5+10.2 [7.1, 13.3]
270.975.6+4.7 [2.0, 7.2]
373.076.8+3.8 [1.3, 6.1]
474.978.4+3.5 [1.2, 5.7]

The evaluated GSAP checkpoint correctly classifies more images at step 1 and maintains its advantage thereafter. The intervals come from paired image resampling with fixed checkpoints and reflect variation across evaluation images.

In these settings, GSAP achieves higher mean recognition accuracy and broader, more persistent remote responses than SSA. These quantitative results support GSAP as a more effective token interaction module for Spiking Transformers.

References

[1] Xiao, Yichen, et al. Rethinking spiking self-attention mechanism: Implementing α-xnor similarity calculation in spiking transformers. CVPR 2025.

[2] Zhao, Dongcheng, et al. Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer. arXiv

[3] Xu, Ruihan, et al. Parcnetv2: Oversized kernel with enhanced attention. ICCV 2023.

Thank you again for your constructive comments. We hope these responses address your concerns and questions, and we welcome further discussion.

Thanks to the authors for the added experiments and explanations. Most of my concerns have been addressed. I've increased the score

Thank you for taking the time to carefully consider our rebuttal and update your assessment. We sincerely appreciate your thoughtful feedback and the effort you have devoted to reviewing our work.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your thoughtful reviews and constructive feedback. During the rebuttal period, we conducted an extensive body of new experiments across multiple datasets, backbones, random seeds, and diagnostic settings to directly and conclusively resolve every major concern raised by the reviewers. We summarize these efforts below to assist the Area Chair's final assessment.

1. Methodological Distinction & Broad Mixer Comparisons (Reviewers TACR, xZ6X, CM3Z) Beyond clarifying GSAP's spike-native paradigm and structural differences from ParCNetV2, we newly implemented a spiking ParCNetV2 baseline alongside multiple alternative spatial mixers and benchmarked them across three static and event-based datasets. These controlled comparisons conclusively validate GSAP's superior accuracy–efficiency trade-off.

2. Large-Scale Motivation & Mechanistic Verification (Reviewers TACR, xZ6X, CM3Z) We carried out a comprehensive suite of quantitative diagnostics—spanning large-scale multi-layer matching statistics, cross-dataset sparsity sensitivity tests, remote perturbation analyses, membrane-state interventions, step-wise prediction evaluations, and online-learning analyses. Collectively, these results decisively confirm our sparse-matching motivation as well as GSAP's persistent long-range propagation and training stability.

3. Stronger Backbone Scaling & Full Multi-Seed Ablations (Reviewers TACR, xZ6X, CM3Z) We expanded our evaluation to a stronger ImageNet backbone (QKFormer) and conducted complete multi-seed baseline comparisons and component ablations across multiple benchmarks. These extensive runs verify consistent gains, local–global synergy, and clear pathway-level attributions.

4. Cross-Dataset Actual-Activity Efficiency Profiling (Reviewers TACR, CM3Z) We clarified the distinction between matching sparsity and firing sparsity, and performed comprehensive actual-activity profiling with stage-wise energy breakdowns across multiple datasets and mixers. Strictly following widely adopted energy estimation protocols, these new measurements conclusively confirm GSAP's significant energy savings.

5. Manuscript Clarifications (Reviewer xZ6X) We clarified the ambiguities raised regarding specific metrics and notations in the paper, and will revise the corresponding descriptions in the final manuscript.

Taken together, this substantial body of new experimental evidence thoroughly addresses the reviewers' core concerns. Notably, Reviewer TACR confirmed that most of their concerns were resolved and raised their score accordingly. We will incorporate all new experiments and clarifications into the revised manuscript and greatly appreciate the Area Chair's comprehensive evaluation.

Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.