acceptodds
Under review as a conference paper at ICLR 2027

SpikeRED: Fully Spike-Driven RGB-Event Object Detection with Event-Guided Fusion and Box-Guided Query Refinement

Abstract

Spiking neural networks (SNNs) offer substantial potential for energy-efficient visual detection. However, existing spiking object detectors primarily rely on unimodal inputs, while existing multimodal spiking approaches typically retain ANN components, limiting their energy-efficiency advantages. To address this issue, we propose SpikeRED, a fully spike-driven RGB-event object detection framework. Its Asymmetric Spiking Fusion (ASF) module enables more effective Event-to-RGB interaction with lower energy overhead. The Box-Guided Spiking Decoder further enhances the representation capability of spiking object queries through reference-box guidance. Experiments on DSEC-Det and PKU-DAVIS-SOD demonstrate that SpikeRED achieves performance comparable to leading multimodal ANN detectors while substantially reducing theoretical inference energy. To our knowledge, SpikeRED is the first fully spike-driven query-based multimodal object detection framework.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spiking Neural Networks, RGB–Event Fusion, Object Detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rKtdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SpikeRED, a spiking object detection framework for RGB images and event streams. The framework employs a dual-branch spiking backbone to extract features from both modalities and an Asymmetric Spiking Fusion (ASF) module to incorporate event information unidirectionally into RGB features. The decoder uses reference bounding boxes to guide multi-scale local feature sampling and enhances object queries through residual updates, progressively refining detection results across decoder layers. Experiments on DSEC-Det and PKU-DAVIS-SOD demonstrate that SpikeRED-L achieves mAP scores of 30.7% and 32.2%, respectively. The paper also reports theoretical inference energy consumption and presents ablation studies on fusion direction, value quantization, and decoder architecture.

Critical strengths
  1. ASF enriches RGB features with complementary event information, improving detection accuracy on both datasets while reducing the theoretical inference energy consumption of the entire network compared with the fusion module used in EOLO.

  2. Enhancing spiking queries with multi-scale features from object regions improves mAP by 6.8 and 6.9 percentage points on DSEC-Det and PKU-DAVIS-SOD, respectively, compared with a standard spiking attention decoder, with only a small increase in theoretical energy consumption.

Critical weaknesses
  1. If the paper aims to emphasize visual tasks in spiking neural networks (SNNs), representative segmentation and tracking algorithms should also be included. Section 2.1 would benefit from broader coverage and a more comprehensive review of the relevant literature.

  2. Figure 1 contains several substantial issues. First, the origins of the three scales of features fed into the ASF module are unclear, and Figures 1 and 2 appear inconsistent in this regard. Second, “fusion Feat1” and “fusion Feat2” in Figure 1 appear to have identical spatial dimensions, while the feature dimensions in Figure 2 are not consistently aligned with those in Figure 1. These inconsistencies create considerable confusion and make the proposed architecture difficult to follow. Third, the addition operator in the lower-right diagram has only one input. Together with other serious errors, this renders the diagram difficult to interpret. For example, the origin and role of the neuron depicted between the convolution (Conv) and batch normalization (BN) layers are unclear.

  3. The illustrations on the left side of Figure 2 contain an error. In the rightmost image of the first subfigure, the red squares should exhibit different color intensities rather than a uniform shade.

  4. Figure 2 illustrates concepts that are already widely known in the SNN literature and do not constitute a contribution of this work. Its inclusion therefore adds little value. Moreover, empirical experience suggests that representing queries, keys, and values (Q, K, and V) entirely as spikes does not lead to substantial performance degradation. Overall, I do not consider this issue a worthwhile research topic, nor does the proposed treatment even qualify as an implementation trick.

  5. State-of-the-art accuracy results and other key values in the tables should be highlighted in bold according to a consistent criterion to improve readability. The figures convey very limited information.

  6. Overall, the paper exhibits insufficient novelty and lacks adequate analysis of the experimental results. The presentation could also be more concise. I recommend strengthening the methodological contributions and experimental analysis, followed by further revision to improve clarity and conciseness.

Questions for authors

None.

1 comment

Dear authors, the AC has reminded us reviewers to respond to your rebuttal. If there is anything you would like to clarify or discuss, please leave a comment here, and I will respond as soon as possible.

Reviewer_QvDTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SpikeRED as a fully spike-driven RGB-event object detection framework featuring an Asymmetric Spiking Fusion module and a Box-Guided Spiking Decoder. It evaluates three model sizes on DSEC-Det and PKU-DAVIS-SOD, reporting detection accuracy and operation-based theoretical energy estimates. Ablation studies examine the contributions of box-guided query refinement and the fusion design, showing accuracy improvements under the evaluated settings.

Critical strengths
  1. The paper presents a fully spike-driven, query-based framework for multimodal object detection. Experiments on DSEC-Det and PKU-DAVIS-SOD demonstrate competitive detection performance with theoretical energy efficiency.

  2. The proposed Asymmetric Spiking Fusion and Box-Guided Spiking Decoder are designed to address the challenges of multimodal fusion and query refinement in spiking neural networks. The authors provide corresponding ablation studies to evaluate the effectiveness of these two modules.

Critical weaknesses
  1. The paper omits several recent methods, including SPFD (CVPR 2026). On DSEC-Det, SPFD achieves an mAP₅₀ of 56.7 and an mAP of 34.6, outperforming the proposed method (50.8 and 30.7, respectively). This comparison raises concerns about the competitiveness of the proposed method and weakens the paper's claims regarding its detection performance.

[1] Wang K, Liu S, Shi H, et al. Beyond Duality: A Hybrid Framework of Leveraging Shared and Private Features for RGB-Event Object Detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 4415-4424.

  1. The qualitative analysis is relatively limited. Although the paper presents visual comparisons of sampling patterns and detection results, the examples cover only a few selected scenarios, and the comparisons are restricted to a limited set of baselines. In particular, qualitative comparisons with conventional ANN-based detectors and other recent methods are missing.
Questions for authors
  1. In Eq. (6), the size of reference box s_i is multiplied by the offset O_ij to determine the sampling locations. Since these quantities appear to involve continuous-valued operations, could the authors clarify how this computation is implemented within the proposed fully spike-driven framework? Specifically, does this operation introduce traditional continuous-value computations, thereby undermining the fully spike-driven contribution claimed in this paper?

  2. The paper introduces the Box-Guided Spiking Decoder to address the limited information carried by discrete queries under low time-step settings. However, its effectiveness is evaluated only under a fixed configuration of T=1 and D=4. The lack of ablation studies across different time-step and firing-level settings makes it difficult to determine whether the benefits of box-guided refinement are specifically attributable to the representational limitations of SNNs. Moreover, it remains unclear whether applying the same refinement strategy to a conventional ANN-based decoder would yield comparable improvements.

Reviewer_WWeZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SpikeRED, to the authors' knowledge the first fully spike-driven query-based object detector for aligned RGB and event streams. Two Spike-Driven Transformer v3 backbones extract features from each modality. The Asymmetric Spiking Fusion module performs one-directional Event-to-RGB fusion with factorized linear cross-attention, spiking the query and key branches while keeping the projected event values continuous to preserve fine-grained event cues. The Box-Guided Spiking Decoder samples multiscale features around the current reference boxes, gates them with spiking weights, and injects them into the object queries as a residual correction before each spiking decoder layer. On DSEC-Det and PKU-DAVIS-SOD, SpikeRED-L reaches 30.7 and 32.2 mAP, matching the ANN multimodal detector SFNet within 0.3 mAP at roughly 15% of its theoretical energy, and outperforming the hybrid EOLO by 3+ mAP at about half its energy. Ablations cover the decoder design, value quantization, fusion direction, and modality choice, with an operator-level theoretical energy breakdown in the appendix.

Critical strengths
  1. The system-level contribution is clear and the "first" claim is substantiated: spike-driven computation covers both backbones, fusion, and the query decoder, which no prior RGB-event detector does. The operator-wise energy breakdown in Appendix A.3 is thorough and makes the energy accounting inspectable.
  2. The design choices are supported by targeted ablations rather than assertion: keeping event values continuous costs almost no energy (0.2 mJ) while gaining 1.6 mAP, reversed fusion loses 7.0 mAP, bidirectional fusion adds energy without accuracy, and the box-guided decoder beats plain and deformable spiking attention by 6.8 and 8.2 mAP.
  3. The work delivers a practical accuracy-energy point: parity with a strong ANN multimodal detector at ~15% theoretical energy, and consistent gains over the hybrid EOLO across both benchmarks.
Critical weaknesses
  1. All efficiency evidence is theoretical (45 nm MAC/AC accounting). No latency, throughput, or measured energy on any hardware is reported — not even FPS — although the motivation is deployment on energy-constrained platforms and a six-layer query-based decoder is the latency-critical part of the pipeline.
  2. The headline margins depend on the chosen comparison. The ">85% energy reduction" is against SFNet, the most energy-demanding ANN baseline. Against the RGB-only ANN YOLOv11 (30.3/30.9 mAP, 156.7/66.1 mJ), SpikeRED-L's advantage shrinks to 0.4/1.3 mAP with only 15-25% energy saving, which changes the practical story considerably.
  3. The value of the event branch is not analyzed where the motivation says it should matter. On PKU-DAVIS-SOD, moving from the RGB-only SpikeRED-L* (31.1 mAP, 28.9 mJ) to the full model (32.2 mAP, 49.9 mJ) costs 72% more energy for 1.1 mAP, and no per-condition breakdown (low light, high speed, tunnel exits) shows when events actually pay off.
  4. The multimodal spiking and hybrid methods discussed in Related Work (EGSTNet, LOD-Drive) are absent from Table 1, so the positioning against the nearest existing systems is not empirically demonstrated.
Questions for authors
  1. What fraction of SpikeRED-L's total theoretical energy comes from non-spike dense-MAC operations (first convolutions, continuous-value products, continuous readouts)? Is the accuracy-energy trade-off preserved when NI-LIF's D=4 multilevel activations are mapped to neuromorphic hardware that natively supports only binary spikes?
  2. Since energy scales linearly with D x R x T, report accuracy and theoretical energy for at least D in {2, 8}, ideally also T in {1, 2}, so the reader can see how the operating point was chosen.
  3. Provide a per-condition breakdown on PKU-DAVIS-SOD (e.g., day vs. low light, low vs. high ego-speed) comparing the RGB-only and full models, to show the regimes where event fusion is worth its energy cost.
  4. The OpenReview listing title (SpikeResNet, Event-Guided Fusion) differs from the PDF (SpikeRED, Asymmetric Spiking Fusion). Please align the two.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.