acceptodds
Under review as a conference paper at ICLR 2027

Robustness from Interaction: Event-Structure-Aware Attention for Event-Based Spiking Transformers

Abstract

Robustness in event-based vision is often addressed before the network via event denoising, corruption-aware training schemes, or robust feature representations. We pose a complementary question: after noisy events have already been discretized and tokenized, can we improve robustness by controlling how event tokens interact with one another? Unlike generic sequence tokens, event tokens still encode sensing-imposed spatial and polarity structure inherited from the event generation process. We introduce Spiking Event-Consistent Attention (SECA), which maps this token-aligned structure into a bounded multiplicative modulation of learned query–key correlations, adding only two learnable scalars per block while keeping spiking dynamics intact. Without any corruption training (trained only on clean data), SECA maintains clean accuracy across three Spiking Transformer families and cuts the performance drop under the strongest polarity flip on DVS128-Gesture from to points, with consistent improvements under background activity and hot-pixel noise. Controlled violations of the assumed priors indicate that retaining event structure provides benefits beyond a generic bounded modulation, especially under polarity corruption. A test-time gate ablation further shows that much of the robustness remains even when the gate is turned off at inference, suggesting the structural constraint primarily shapes the learned solution rather than serving only as test-time suppression. Overall, these findings position interaction-level robustness as a complementary design principle for event-based Spiking Transformers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spiking Neural Networks, Event-based Vision, Spiking Transformers, Attention Mechanisms, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fYhQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Spiking Event-Consistent Attention (SECA) to improve the robustness of event-based spiking transformers. The method uses spatial proximity and event polarity to construct a bounded gate on query-key correlations, introducing two learned scalars per block. Models trained on clean data are evaluated under several event corruptions and across multiple backbones.

Critical strengths
  1. The method is simple and addresses a meaningful problem. The bounded gate provides an interpretable way to incorporate event structure, and the improvement under strong polarity-flip noise is substantial. The component ablation also supports using both spatial and polarity information.

  2. This paper provides detailed analysis. The authors compare genuine, shuffled, random, and misaligned priors, and show that much of the improvement remains after removing the gate at inference. This result is interesting because it suggests an effect on the learned parameters.

Critical weaknesses
  1. The necessity of the structural gate at inference is unclear. Table 8 shows that disabling the gate in the same trained model yields a numerically higher PFN accuracy (89.028% vs 87.986%) without retraining. Although this slightly reduces clean accuracy, the robustness gain is largely retained, leaving the benefit of the additional inference-time modulation uncertain.

  2. The polarity descriptor captures only relative ON/OFF imbalance, discarding absolute event counts and failing to distinguish meaningful activity from noise with similar polarity statistics.

  3. The practical overhead of SECA is insufficiently evaluated. Adding only two learnable scalars per block establishes a small parameter overhead, but does not quantify computational or memory costs.

Questions for authors

see weaknesses

2 comments

Dear Reviewer fYhQ,

Thank you for these helpful comments. We clarified the gate-removal evidence and the polarity descriptor's role, and added measured computational and memory costs across all three backbone families. We welcome further discussion.

Weakness 1: Necessity of the inference-time gate.

Active inference-time gating is not necessary to retain PFN robustness in this experiment. In Table 8, full SECA and gate-off share the same trained checkpoint without retraining. Gate-on has higher clean accuracy; gate-off has a numerically higher PFN mean over five paired corruption realizations. These summaries do not establish statistical superiority. Removing the gate preserves learned parameters and does not test training without the prior. The abstract, Sec. 4.6, and Appendix E.4 clarify our contribution: event-derived structure guides token interactions under clean-only training, with observed PFN robustness persisting beyond instantaneous inference-time modulation.

Table 8. Inference-time gate removal on DVS128-Gesture at ηPFN=0.10\eta_{\mathrm{PFN}}=0.10. PFN mean$\pm$SD over five paired corruption realizations; SECA views share one checkpoint.

ModelClean Acc. (%)PFN Acc. (%)Δ\DeltaAcc (p.p.)
SSA97.22268.819±1.65468.819\pm1.65428.403±1.65428.403\pm1.654
Full SECA97.91787.986±1.19387.986\pm1.1939.931±1.1939.931\pm1.193
SECA gate-off (G≡1G\equiv1)97.22289.028±1.03089.028\pm1.0308.194±1.0308.194\pm1.030

Weakness 2: Polarity descriptor and absolute activity.

The descriptor summarizes relative ON/OFF imbalance, ρ=(a(0)−a(1))/(a(0)+a(1)+ϵ)\rho=(a^{(0)}-a^{(1)})/(a^{(0)}+a^{(1)}+\epsilon), with ϵ=10−6\epsilon=10^{-6} (Appendix A). Absolute counts remain available at the feature-branch input through the original two-channel frames (Fig. 2). An empty region and a nonempty region with balanced polarities both give ρ=0\rho=0, but different feature inputs; prior entries involving either token vanish, leaving its interactions without additional structural modulation. Polarity balance alone cannot distinguish signal from noise with matching statistics. Sec. 3.2 and Appendix B therefore define polarity/proximity as soft compatibility cues. Table 4 supports their utility in the evaluated setting: at ηPFN=0.10\eta_{\mathrm{PFN}}=0.10, accuracy is 68.403% for SSA, 78.819% for polarity-only modulation, and 88.542% for joint spatial–polarity modulation.

Weakness 3: Practical computational and memory costs.

Sec. 4.7 and Table R2 quantify costs beyond parameter count. Each pair uses the same L20 and all 288 clean DVS128-Gesture sequences (T=16T=16, FP32, TF32 disabled), averaging five full passes after 50 warm-up forwards. CUDA-synchronized timing includes forward inference and spike reset, excluding data loading and host-to-device transfer. Dense MACs count only Conv/Linear/matrix products; memory is peak allocated GPU memory. SDT's separable spatial contractions add 0.403 G MACs per sequence. Spikformer and SDT-V3 retain their existing matrix products; their additional elementwise modulation is excluded from this count but included in measured runtime and peak memory.

Table R2. Inference cost on DVS128-Gesture. Entries: baseline / SECA.

BackboneDense MACs (G/sequence)Latency (ms), B=1B=1Throughput (seq/s), B=16B=16Peak memory (MiB), B=16B=16
Spikformer-2-2566.057 / 6.05731.74 / 31.10360.71 / 361.432098.04 / 2098.20
SDT-8-38496.259 / 96.662105.54 / 107.2940.88 / 37.9212361.13 / 13173.35
SDT-V3-2-36024.338 / 24.3385.82 / 6.06168.16 / 167.643677.60 / 3677.62

For SDT, B=1B=1 latency increases by about 1.7% (105.54 to 107.29 ms). At B=16B=16, throughput decreases from 40.88 to 37.92 seq/s, corresponding to about 7.8% higher batch latency (Lbatch=1000B/RL_{\mathrm{batch}}=1000B/R ms); peak memory increases by 6.57%. Spikformer/SDT-V3 add 0.16/0.02 MiB at B=16B=16. Reversing Spikformer's execution order reverses its small timing difference, providing no evidence of acceleration. The measured overhead therefore depends on backbone and batch size. Table R2 and Table R3 (Appendix E.7) were profiled in separate sessions with different cuDNN settings and execution-order protocols; cuDNN TF32 is disabled in Table R2 and enabled in Table R3. Absolute timings are comparable only within each table.

I appreciate the authors' efforts in providing additional experiments and explanations, which have addressed most of my concerns. So I will maintain my original score as 3.

Reviewer_nXxNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spiking Event-Consistent Attention (SECA) to improve the robustness of event-based Spiking Transformers against data corruptions. SECA introduces a spatial–polarity prior to modulate attention interactions through a lightweight multiplicative gate, without relying on corruption-aware training at the preceding feature extraction stage. Experiments demonstrate that SECA improves robustness against various event corruptions while largely preserving classification accuracy on clean data. Ablation studies suggest that the robustness gains may primarily stem from SECA's influence on representation learning during training, rather than direct noise suppression at inference time.

Critical strengths

Simple and lightweight design. SECA introduces a spatial–polarity prior into spiking attention with only two additional learnable scalars per block. The method is easy to integrate and does not require corruption-aware training. Comprehensive experimental analysis. The paper evaluates multiple Spiking Transformer backbones and event datasets, demonstrating substantial robustness improvements in several settings. The ablation studies, controlled prior violations, and inference-time gate removal provide useful insights into the proposed mechanism.

The paper evaluates SECA across multiple Spiking Transformer backbones and event-based datasets, showing substantial robustness improvements in several settings. In addition, the authors conduct ablation studies, controlled prior violations, and inference-time gate removal experiments, providing a relatively thorough analysis of the method's effectiveness and underlying mechanisms.

Critical weaknesses
  1. The inference-time gate-removal experiment shows that robustness remains largely intact, and is even slightly improved, without the proposed modulation. This raises the question of whether the gains arise from event-consistent interactions or merely from a more general regularization effect during training. Stronger comparisons with alternative training-time regularization strategies are needed to clarify this.

  2. For SDT-based backbones, SECA replaces the original SDSA operator with a pairwise attention formulation. As a result, the observed improvements may stem in part from the change of attention operator rather than from the proposed spatial–polarity prior. Additional controlled comparisons are needed to isolate the contribution of SECA.

Questions for authors
  1. SECA maintains, and even slightly improves, its robustness when the attention gate is removed at inference time. This raises the question of whether the gains stem primarily from the training process rather than from attention modulation itself. Could the authors compare SECA with other regularization methods to better identify the source of the robustness gains?

  2. In the SDT experiments, the baseline employs the original SDSA operator, whereas SECA adopts a different pairwise attention operator. It is therefore unclear whether the improvements should be attributed to the proposed spatial–polarity prior or simply to the change of attention operator. Could the authors add a baseline that uses the same pairwise attention operator but without the prior?

3 comments

Dear Reviewer nXxN,

Thank you for highlighting training regularization and attention-operator confounding. We added regularization controls and clarified the backbone-specific computations and their prior-free limits. We welcome further discussion.

Weakness 1 / Q1: Training regularization versus inference gating.

Removing the gate from SECA-trained weights preserves the outcome of training; it does not evaluate training without the prior (Sec. 4.6; Appendix E.4). Table R1 compares always-G=1G=1 training and attention dropout (p=0.1p=0.1, applied to attention scores only during training) on shared corrupted inputs. Gate-on/off use the same SECA checkpoint.

Table R1. Regularization and gate ablation on DVS128-Gesture (%). ρBAN=2.0\rho_{\mathrm{BAN}}=2.0, ηPFN=0.10\eta_{\mathrm{PFN}}=0.10, dHPN=0.10d_{\mathrm{HPN}}=0.10; corruption seed 2026.

MethodCleanBANPFNHPN
Always G=1G=196.87592.01478.81994.444
SECA (gate-on)97.91793.75088.54295.139
SECA (gate-off)97.22293.75088.54295.139
Attention dropout97.56993.05677.08394.444

At the evaluated checkpoints, both SECA views attain 88.542% PFN accuracy, versus 78.819% for always-G=1G=1 and 77.083% for attention dropout. SECA uses clean-selected weights and the controls use final-epoch weights, so Table R1 is a descriptive checkpoint comparison. Equal corruption accuracies for seed 2026 do not imply identical predictions; Table 8 reports five paired PFN realizations with gate-on/off means of 87.986/89.028%, without establishing statistical superiority. Table 5 provides complementary structural evidence: at identical 97.917% clean accuracy, PFN degradation increases from 9.375 p.p. with the true prior to 18.056/24.653 p.p. with shuffled/misaligned priors. The tested dropout setting does not reproduce SECA's PFN accuracy, and the structural ablations support the relevance of event-aligned interactions, without identifying a unique training mechanism.

Weakness 2 / Q2: Attention-operator confounding.

Thank you for identifying the ambiguity in our submitted phrase, “replace the corresponding attention operator.” This clarification changes no baseline implementation, SECA operator, or previously reported result. Table 3 of the submission labeled the SDT-V3 baseline “SDSA”; its evaluated implementation already uses the QK⊤QK^\top-based attention of SDT-V3 (Yao et al., 2025), denoted E-SDSA here to distinguish it from SDT's native SDSA. The revised Table 3, Sec. 3.3, and Appendices C–D clarify this distinction.

In SDT, SECA retains native SDSA key-value aggregation, LIF, and elementwise query gating. With uj=kj⊙vju_j=k_j\odot v_j and Φij=ρiκijρj\Phi_{ij}=\rho_i\kappa_{ij}\rho_j, it computes

\sum_j u_j+\lambda\sum_j\Phi_{ij}u_j \right).$$ Setting $\Phi=0$ recovers native SDSA aggregation; no $QK^\top$ content-attention matrix is introduced. In SDT-V3, SECA modulates the baseline's existing query–key correlations while retaining native scaling, value aggregation, spiking, and projection: $$A=QK^\top,\qquad Z_{\mathrm{native}}=cAV,\qquad Z_{\mathrm{SECA}}=c\left[A\odot(\mathbf{1}+\lambda\Phi)\right]V.$$ Setting $G\equiv1$ recovers native E-SDSA. The native SDSA/E-SDSA baselines were trained without the prior (Tables 1 and 3); the SDT-V3 baseline therefore already provides the requested pairwise formulation without structural modulation. At shared weights, prior-free forward checks match the native implementations on the tested inputs; Appendix C reports the verification scope and gradient diagnostics.

Dear Reviewer nXxN,

Thank you again for your careful review and constructive questions. We have added regularization controls and clarified the attention operators without changing the evaluated implementations. When you have a chance, we would appreciate your feedback on whether these address your concerns and would be happy to provide further clarification.

Best regards,
Authors

Thank you for the authors' detailed response and additional experiments. My concern regarding Q2 has been fully addressed, while the new results have partially alleviated my concern regarding Q1. Therefore, I maintain my original rating of 3

Reviewer_o6qROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at making Spiking Transformers robust against event-stream corruptions (specifically BAN, PFN, and HPN) by altering self-attention rather than preprocessing events. The proposed SECA module multiplies the query-key attention scores by a bounded gate G=1+λΦG = 1 + \lambda \Phi, where Φ\Phi is an event-polarity and spatial-distance prior, tuned with just two learnable scalar parameters per layer. Experiments are mostly centered on DVS128-Gesture with Spikformer, with some additional evaluations on SDT backbones and CIFAR10-DVS/HARDVS.

Critical strengths

(1) Instead of tacking on heavy convolutional filters or disrupting discrete spike communication, using a closed-form spatial-polarity matrix with just two learnable scalars per block keeps the parameter overhead near zero. So, the formulation is quite lightweight.

(2) I appreciate the ablation effort in Table 5 and Figure 6. The authors actually test perturbed versions of the prior (shuffled, inverted, random) to verify whether the structural information matters, which is definitely better than treating the module as a pure black box.

Critical weaknesses

(1) The test-time behavior in Table 8 is really confusing and feels contradictory to the main premise. When testing under polarity flip noise (ηPFN=0.10\eta_{PFN}=0.10), turning off the gate completely at inference (G≡1G \equiv 1) actually yields 89.03% accuracy—which is higher than keeping SECA active (87.99%). If SECA works by using physical sensing priors to filter out bad tokens during inference, why does shutting it off give better results? On top of that, Fig 6a shows the learned modulation is vanishingly small (E[∣λΦ∣]≈10−4\mathbb{E}[|\lambda \Phi|] \approx 10^{-4}). Modulating attention weights by 0.01% seems far too tiny to physically re-route token interactions; it feels more like the module acted as a slight regularizer during training rather than an effective physical gate at test time.

(2) The claimed robustness under polarity noise does not hold up across models and datasets. The paper leans heavily on Spikformer results where PFN error drops significantly, but looking at the other tables paints a different picture: on SDT-V3 (Table 1), SECA actually degrades PFN robustness compared to the baseline (mean drop increases from 0.03 to 1.81); similarly on CIFAR10-DVS (Table 6), the PFN accuracy drop is worse with SECA than without it (6.14 vs 4.99). If the polarity prior is fundamental to event consistency, why does it backfire on these other settings?

(3) Even though the paper argues against input-level filtering, there are no empirical comparisons against standard event denoising filters (like background activity filters or refractory period checks) or even simple training-time data augmentations (e.g. random polarity jitter). We have no way of knowing whether this attention-level modification actually beats a 2-line preprocessing script.

Questions for authors

(1) Could you explain why setting G=1G=1 at test time in Table 8 outperforms the full SECA model under PFN (89.03% vs 87.99%)?

(2) Why does SECA hurt PFN robustness on SDT-V3 (Table 1) and SDT on CIFAR10-DVS (Table 6)?

(3) Given that the learned modulation scale is on the order of 10−410^{-4} (Fig 6a), what is the physical intuition for how this tiny shift provides a 20% robustness gain on Spikformer?

(4) How does SECA compare in compute overhead and clean/corrupted accuracy against a standard input event filter (e.g. standard nearest-neighbor or BA filter)?

2 comments

Dear Reviewer o6qR,

Thank you for the time and care you devoted to reviewing our work. Your comments helped us clarify the gate's role and add regularization, filtering, and cost comparisons.

Q1 / Weakness 1: Gate removal and regularization.

Gate removal retains PFN robustness in SECA-trained weights; it does not test training without the prior. In Table 8, gate-on/off share one checkpoint: PFN means are 87.986/89.028% over five paired realizations, with clean accuracies of 97.917/97.222%. Table R1 tests always-G=1G=1 training and train-only attention dropout (p=0.1p=0.1). Neither control matches SECA's PFN accuracy at these checkpoints; SECA is clean-selected and controls are final-epoch, so the comparison is descriptive. Table 5 adds structural evidence: at 97.917% clean accuracy, true/shuffled/misaligned priors give PFN drops of 9.375/18.056/24.653 p.p.

Table R1. Regularization and gate ablation on DVS128-Gesture (%). BAN/PFN/HPN severities: 2.0/0.10/0.10; corruption seed 2026.

MethodCleanBANPFNHPN
Always G=1G=196.87592.01478.81994.444
SECA (gate-on)97.91793.75088.54295.139
SECA (gate-off)97.22293.75088.54295.139
Attention dropout97.56993.05677.08394.444

Q2 / Weakness 2: PFN regressions.

Mean/max-severity PFN drops increase from 0.03/1.16 to 1.81/2.55 p.p. for SDT-V3 (Table 1), and 4.99/5.93 to 6.14/7.80 for SDT on CIFAR10-DVS (Table 6). The polarity prior is a soft compatibility cue and does not enforce flip invariance. SDT on DVS128-Gesture improves from 7.35/15.97 to 5.13/9.38 p.p. (Table 1), showing gains beyond Spikformer. These results show that PFN robustness benefits vary across backbones and datasets, as discussed in Sec. 4.1 and Appendix E.1.

Q3 / Weakness 1: Small modulation, large PFN gain.

At ηPFN=0.10\eta_{\mathrm{PFN}}=0.10, Fig. 6a reports M1=9.651×10−4M_1=9.651\times10^{-4} and M2=3.530×10−4M_2=3.530\times10^{-4}, averaging absolute gate modulation over token pairs and inputs, including distant pairs with small kernels. These are inference-time averages, not uniform per-pair amplitudes or training trajectories. Table 7 gives 4.448%/2.079% pairwise bounds. The 19.167 p.p. PFN gain (87.986% - 68.819%, Table 8) compares separately trained models rather than instantaneous gating of baseline weights.

Q4 / Weakness 3: Filtering, augmentation, and cost.

Table 2 retains original severities and compares unchanged checkpoints on paired inputs. The frame-domain BA-style filter uses polarity-agnostic, nonrecursive four-neighbor support in the current or preceding bin (Appendix D). SECA exceeds filtered SSA at the highest listed severities. Filtering improves SECA at HPN 0.10 and 0.20 but lowers clean/PFN accuracy. At HPN 0.20, the combination reaches 79.167%, versus 62.847% for SECA and 62.500% for filtered SSA, suggesting complementary effects of input filtering and token-interaction modulation in this setting. Our clean-only comparisons do not establish superiority over polarity-jitter training.

Table 2. Spikformer-2-256 clean/corruption accuracies (%) on DVS128-Gesture; corruption seed 2026.

MethodCleanBAN 0.20BAN 1.00BAN 2.00PFN 0.01PFN 0.05PFN 0.10HPN 0.02HPN 0.10HPN 0.20
SSA97.22296.87594.09788.88995.48690.97268.40395.83384.72257.986
SSA + BA-style filter95.83396.87594.79290.62595.83388.19472.56995.13993.05662.500
SECA (Ours)97.91796.87594.44493.75096.52896.52888.54296.87595.13962.847
SECA + BA-style filter96.87597.22295.13993.40395.48692.70884.37595.83396.52879.167

Table R3 (Appendix E.7) uses one L20, 288 clean sequences, T=16T=16, FP32: 50 warm-ups and five full passes in each of two reversed orders (mean$\pm$SD over ten passes). CUDA-synchronized timing includes optional filtering, inference, and spike reset, excluding data loading and host-to-device transfer; memory is peak allocated. Filtering adds 32 MiB at B=16B=16; timing differences reverse with order, providing no acceleration evidence. Tables R2 and R3 use separate sessions with different cuDNN settings and execution-order protocols; cuDNN TF32 is off in Table R2 and on in Table R3, while AMP/matmul TF32 are off in both. Compare absolute timings only within each table.

Table R3. DVS128-Gesture inference costs. Filter-only memory includes inputs and intermediates.

MethodLatency (ms), B=1B=1Throughput (seq/s), B=16B=16Peak memory (MiB), B=16B=16
SSA31.13±0.2531.13\pm0.25361.22±2.43361.22\pm2.432098.04
SSA + BA-style filter31.16±0.2031.16\pm0.20360.20±3.03360.20\pm3.032130.04
SECA31.46±0.2531.46\pm0.25358.19±2.95358.19\pm2.952098.20
SECA + BA-style filter31.49±0.0531.49\pm0.05358.58±2.08358.58\pm2.082130.20
BA-style filter only0.164±0.0030.164\pm0.00347,123±1,32347{,}123\pm1{,}32392.00

Dear Reviewer o6qR,

Thank you again for your time and thoughtful feedback. We have added regularization controls and filtering comparisons and clarified the gate-removal evidence. When you have a chance, we would appreciate your feedback on whether these address your concerns and would be happy to clarify any remaining points.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.