acceptodds
Under review as a conference paper at ICLR 2027

Efficient RGB–Event Fusion for Moving Object Detection in Autonomous Driving

Abstract

In autonomous driving, low-light conditions and motion blur degrade RGB appearance representations, thereby impairing detection accuracy. RGB–Event fusion introduces event-based motion information to complement RGB appearance information when RGB imaging is degraded. However, camera ego-motion generates substantial background event responses, which interfere with target representation during cross-modal fusion. Notably, although existing multi-window methods improve detection accuracy, they struggle to balance accuracy and efficiency, ultimately constraining the reliability and real-time performance of autonomous driving perception systems. To address these challenges, we propose a single-window RGB–Event fusion network for object detection in autonomous driving. The network comprises hierarchical cross-modal fusion and multi-window-to-single-window knowledge distillation (M2S-KD). Hierarchical cross-modal fusion enhances complementary information through bidirectional feature modulation and improves deep feature fusion through Bidirectional Attention. M2S-KD transfers detection knowledge from a multi-window teacher to a single-window student through cross-head distillation, while only the student network is retained during inference to improve efficiency. Experimental results on DSEC-MOD show that the proposed method outperforms the second-best method by 2.55 percentage points in Frame [email protected], while efficiency experiments demonstrate a 115.41% increase in inference speed compared with the multi-window model. These results provide a practical approach to balancing detection accuracy and deployment efficiency for object detection in autonomous driving.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Event Cameras, RGB-Event Fusion, Knowledge Distillation, Object Detection, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1vnjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a framework for fusing event camera data with RGB for moving object detection for autonomous driving applications. The paper introduces two ways of fusing the features from event and RGB cameras. The bidirectional feature enhancement uses the response of other feature to modulate its own feature. The deep feature calibration module uses cross attention to fuse the data from the two modalities. The results show minor improvement in accuracy on the DSEC-MOD dataset, and compares with methods from 2024 and earlier.

Critical strengths

The paper extends the idea from RENet paper and adds to it new techniques of fusing the features obtained from event and RGB cameras. It also adds a knowledge distillation path to the architecture that enhances the accuracy.

Critical weaknesses

The novelty of the paper is limited. The ideas presented in this paper are very similar to the RENet paper. The experimental results only include comparisons with papers published in 2024 and earlier. FlexEvent (https://github.com/DylanOrange/flexevent) reports 48.64% mAP on DSEC-mod datset which is quite higher than the 38.99 presented in this paper.

Questions for authors

Questions:

  1. Can you provide some intiution behind the design of "bidirectional feature enhancement" which uses the response of one feature to modulate the other feature. Aren't these modalities complementary and there may be situations where response of one is weaker while the other is stronger?
  2. Can you provide more discussion on equation (17), why did you align the features like that. What happens when you do not align the features and use the student features directly with the teacher network?
  3. Why did you choose E_{30} as input to the student network. Why not E_{50} ? Suggestions:
  4. Provide results on DSEC-Det dataset also.
  5. Do more experiments with additional dataset - PKU-DAVIS-SOD (https://www.pkuml.org/research/pku-davis-sod-dataset.html)
  6. Compare with newer methods
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1vnj

Thank you for your constructive comments. Our responses are below.

1. BFE design.

Why use the response of one feature to modulate the other? Aren't these modalities complementary?

RGB and events are complementary, which motivates BFE. A strong response in one modality does not necessarily indicate a detection-relevant feature. Information from the other modality guides feature selection and response adjustment before fusion, allowing complementarity to support mutual guidance rather than only final aggregation.

Can one modality have a weaker response while the other has a stronger response?

BFE does not require both modalities to respond strongly. When a scene favors RGB, learning tends to yield stronger RGB responses and weaker event responses, reflecting their complementarity.

2. Alignment in Eq. (17).

Why align the features in this way?

Teacher and student features have compatible dimensions, but differ in their statistics. Mean and standard-deviation alignment adjusts offsets and scales while preserving relative spatial responses within each channel. Unlike a learnable convolutional mapping, it adds no learnable parameters or adaptation-layer optimization.

What happens without alignment when student features are used directly with the teacher network?

Student features can still enter the frozen teacher head, but it was trained on teacher features with different channel-wise means and standard deviations. Prediction discrepancies may therefore reflect numerical scale mismatch as well as detection-information differences, interfering with distillation supervision.

On paired samples, the average absolute differences between teacher features and unaligned student features at their detection-head inputs were 0.1044 in channel-wise means and 0.2797 in standard deviations. This indicates feature-statistics discrepancies and supports the alignment strategy.

We further conducted an ablation by removing the alignment in Eq. (17) while retaining the remaining model components. As shown below, alignment improves all three metrics, supporting its effectiveness.Ablation settings are provided in Appendix A.8.

Feature alignment[email protected] (%)[email protected] (%)[email protected] (%)
Without alignment34.3617.014.28
With alignment38.9925.087.52

3. Student event window.

Why choose E30E_{30} rather than E50E_{50}?

A longer window is not necessarily better: short windows may yield incomplete object structures due to event sparsity, while long windows may accumulate more background responses and motion trails. We choose 30 ms to balance information completeness and interference. As shown below, 30 ms achieves the highest [email protected] among the three evaluated windows, supporting this choice.

Event accumulation window[email protected] (%)
15 ms34.18
30 ms38.99
50 ms35.93

4. DSEC-Det experiments.

Provide results on DSEC-Det.

We added experiments using the SFNet-released eight-class annotations and split.For a fair comparison, baselines use the same input resolution and training epochs as our model; details are provided in Appendix A.9. Differences from original studies in input scales, augmentation and training settings may contribute to discrepancies in absolute accuracy.

MethodAP50 (%)
FAOD17.10
FPN-Fusion10.73
FRN21.83
RENet25.10
EOLO28.16
SFNet32.44
ACGR19.13
SPFD34.20
Ours33.14

Our method achieves 33.14% AP50, 8.04 points above RENet and close to SPFD (34.20%), supporting its applicability to general traffic object detection beyond moving-object detection.

5. PKU-DAVIS-SOD experiments.

Provide additional experiments on PKU-DAVIS-SOD.

Following your suggestion, we added experiments on the low-light subset of PKU-DAVIS-SOD using Ours, the RENet teacher, and SPFD, the runner-up method in our main comparison. We compare these methods under the same protocol described in Appendix A.9, with results shown below:

MethodAP50 (%)
RENet23.11
SPFD30.10
Ours28.45

6. Comparisons with recent methods.

Compare with more recent methods.

Following your suggestion, we have added comparisons with ACGR (CVPR 2025) and SPFD (CVPR 2026) to the updated comparison table, extending our evaluation to more recent RGB-event object detection methods.

MethodDSEC-MOD [email protected] (%)DSEC-MOD [email protected] (%)DSEC-MOD [email protected] (%)PKU-DDD17-Car [email protected] (%)DSEC-Det AP50 (%)
ACGR (CVPR 2025)19.368.782.8372.6119.13
SPFD (CVPR 2026)36.8023.686.8487.7834.20
Ours38.9925.087.5290.5033.14

We sincerely thank the reviewer for the valuable feedback and constructive suggestions.

Thanks for the revisions and addressing some of my comments. I maintain my rating of 2 for this paper.

Reviewer_Gd4jOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an RGB/Event camera fusion system for recognizing moving objects in traffic. Fusion includes modules Bidirectional Feature Enhancement and Deep Feature Calibration. These modules include borrows from prior work like channel sharing and style transfer. Multi-window processing is converted to single-window capability via a knowledge distillation framework, this enhancement makes the proposed network efficient. The research employs datasets DSEC-MOD and PKU-DDD17-Car for training and evaluation. The validation results show outperformance over SOTA, and significant speed improvement.

Critical strengths

The speed improvement is good. Using a distillation model, these results are not surprising. Leaning towards accept because it appears a well engineered system.

Critical weaknesses

I learned not much from the paper in terms a novel loss function, a metric, or a theoretical insight.

Questions for authors

As suggestion for future work, I would like to have seen additional theory on fusing RGB and Event streams..

1 comment
AuthorsOpenReview ↗

Response to Reviewer Gd4j

Thank you for your constructive comments and for recognizing the practical value and efficiency of our framework.

1. Future theoretical analysis.

We agree that a deeper understanding of RGB–event fusion is valuable. We identify two directions for future work. First, we aim to analyze the contributions of RGB and events to center localization, motion-offset prediction, and size regression, and explore task-dependent fusion and distillation. Second, we aim to study how illumination degradation, event sparsity, and noise affect fusion, extending our cross-modal modulation with explicit reliability estimation to understand when to enhance complementary information and when to limit unreliable information propagation.

2. Scope and contribution.

We acknowledge the reviewer’s concern and recognize room for further contributions in loss functions, evaluation metrics, and theoretical analysis. However, these aspects are not the primary focus of this work. Our focus is the coordinated design of fusion and distillation through BFE, DFC, and M2S-KD to balance detection accuracy and inference efficiency. Building on related research, we will further explore loss functions and evaluation approaches tailored to RGB–event complementarity, alongside a deeper theoretical analysis of cross-modal fusion.

3. Additional experiments.

Additionally, we have expanded the empirical evaluation in three aspects.

DSEC-Det. We added experiments covering 55,490 annotated frames and eight traffic-object categories, extending evaluation beyond moving-object detection. Our method achieves 33.14% AP50, exceeding RENet by 8.04 percentage points and approaching SPFD (34.20%). The experimental settings are detailed in Appendix A.9.

MethodDSEC-Det AP50 (%)
FAOD17.10
FPN-Fusion10.73
FRN21.83
RENet25.10
EOLO28.16
SFNet32.44
ACGR19.13
SPFD34.20
Ours33.14

Recent methods. We added comparisons with ACGR (CVPR 2025) and SPFD (CVPR 2026). The results below further contextualize our framework against recent RGB–event detection methods.

MethodDSEC-MOD [email protected] (%)DSEC-MOD [email protected] (%)DSEC-MOD [email protected] (%)PKU-DDD17-Car [email protected] (%)
ACGR19.368.782.8372.61
SPFD36.8023.686.8487.78
Ours38.9925.087.5290.50

BFE and DFC analysis. We constructed a 704-frame high-global-motion proxy subset from DSEC-MOD using RGB motion outside annotated object regions. On this subset, jointly using BFE and DFC improves [email protected] by 5.65 percentage points over Base+KD. At approximately 50% recall, the full model also produces fewer total and background-like false positives.

Configuration[email protected] (%)FP/frameBackground-like FP/frame
Base+KD38.973.8302.663
+BFE42.852.0801.398
+DFC41.311.9291.287
Full44.621.4770.977

These additions broaden the evaluation and provide further evidence of the framework’s effectiveness and the complementary contributions of BFE and DFC.

We sincerely thank the reviewer for the valuable feedback and constructive suggestions.

Reviewer_rsFvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an RGB-Event fusion framework for moving object detection in autonomous driving, consisting of Hierarchical Cross-Modal Fusion and M2S-KD. The fusion module includes BFE and DFC to improve cross-modal interactions, while M2S-KD transfers knowledge from a multi-window teacher to a single-window student. Experiments on DSEC-MOD and PKU-DDD17-Car demonstrate improvements in detection accuracy and inference efficiency. However, the methodological novelty and the effectiveness of the proposed mechanisms in addressing the stated challenges remain insufficiently established.

Critical strengths

This article discusses the trade-off between accuracy and efficiency in RGB event object detection. The experiment covered two datasets, including component ablation, robustness evaluation, and inference efficiency comparison.

Critical weaknesses
  1. The proposed BFE and DFC are both traditional feature fusion modules with incremental innovation. The comparison methods are all from 2024 and before, and there is no comparison or analysis of the latest research. This article does not fully determine the novelty of the methods for these components, nor does it explain why specific combinations of them are needed.

  2. This article aims to suppress the background event response caused by camera self motion, but neither BFE nor DFC modules explicitly model self-motion and cannot distinguish foreground events from background response; It is difficult to achieve effective background suppression solely through global channel modulation and cross modal attention. Meanwhile, the robustness experiment only perturbs RGB input and fixed event input, without evaluating the effects of event noise and self-motion background interference. Existing experiments are insufficient to support the core conclusion.

  3. The motivation is unclear, and there are too few targeted experiments and visual analyses to support the paper's claim of suppressing background event responses under different self movement conditions and distinguishing foreground motion from background interference. And only two datasets were used, both of which were small-scale and similar scenes.

Questions for authors
  1. How exactly do BFE and DFC suppress background event responses caused by camera ego-motion? Could the authors provide quantitative evidence or control experiments to demonstrate their effectiveness in distinguishing foreground motion from background responses?
  2. Under the same teacher-student architecture, how does M2S-KD compare with conventional logic-based and feature-based extraction methods?
1 comment
AuthorsOpenReview ↗

Response to Reviewer rsFv

Thank you for your constructive comments. Our responses are below.

1. BFE and DFC under strong background-motion responses.

Can you provide quantitative evidence or controlled comparisons to evaluate BFE and DFC under strong background motion?

We used RGB global motion as a proxy for background-motion responses and compared accuracy and false positives at matched recall.

Subset construction. We estimated motion outside annotated objects using optical-flow tracks and RANSAC partial-affine fitting on all 2,819 DSEC-MOD test frames. The top approximately 25% of reliable frames per video formed a 704-frame subset. Construction and evaluation details are in Appendix A.7.

Detection accuracy. BFE and DFC raise AP50 from 38.97% to 42.85% and 41.31%; together they achieve 44.62%, a 5.65-point gain.

Recall-controlled comparison. At approximately 50% recall, Full reduces total FP/frame from 3.830 to 1.477 and background-like FP/frame from 2.663 to 0.977.

ConfigurationAP50 (%)FP/frameBackground-like FP/frame
Baseline(Base+KD)38.973.8302.663
+BFE42.852.0801.398
+DFC41.311.9291.287
Full44.621.4770.977

These results support improved accuracy and false-positive/recall trade-offs under strong background-motion responses.

2. Comparison with logic-based and feature-based distillation.

Under the same teacher-student architecture, how does M2S-KD compare with conventional logic-based and feature-based extraction methods?

Logic-based KD supervises the student head with labels and teacher predictions, which may conflict. M2S-KD instead passes aligned student features through the frozen teacher head for distillation, leaving the student head supervised by labels to potentially alleviate this conflict.

Feature-based KD directly matches features, potentially overconstraining a single-window student learning from a multi-window teacher. M2S-KD aligns channel-wise means and standard deviations via Eq. (17), then matches predictions through the teacher head.

Both ablations retain the architectures and matched configured training settings (Appendix A.8). M2S-KD performs best on all three metrics.

Distillation strategy[email protected] (%)[email protected] (%)[email protected] (%)
Logic-based KD37.3923.896.91
Feature-based KD34.9321.226.26
M2S-KD38.9925.087.52

3. Comparisons with recent methods.

The comparisons only include methods published in 2024 or earlier, without comparisons with or analysis of recent studies.

We added ACGR (CVPR 2025) and SPFD (CVPR 2026), extending comparisons to recent RGB-event detection methods.

MethodDSEC-MOD [email protected] (%)DSEC-MOD [email protected] (%)DSEC-MOD [email protected] (%)PKU-DDD17-Car [email protected] (%)DSEC-Det AP50 (%)
ACGR19.368.782.8372.6119.13
SPFD36.8023.686.8487.7834.20
Ours38.9925.087.5290.5033.14

4. Dataset coverage.

We added DSEC-Det with 55,490 annotated frames and eight traffic-object categories, extending evaluation beyond moving-object detection. We also evaluated Ours, the RENet teacher, and SPFD on the PKU-DAVIS-SOD low-light subset for challenging illumination. All methods are compared under the same experimental settings; details are in Appendix A.9.

DSEC-Det

MethodAP50 (%)
FAOD17.10
FPN-Fusion10.73
FRN21.83
RENet25.10
EOLO28.16
SFNet32.44
ACGR19.13
SPFD34.20
Ours33.14

PKU-DAVIS-SOD (low-light subset)

MethodAP50 (%)
RENet23.11
SPFD30.10
Ours28.45

5. Rationale for combining BFE and DFC.

Why are BFE and DFC used together?

BFE uses cross-modal global context to modulate feature responses across all six levels. Deep RGB and event features may differ in spatial correspondence and response distribution. DFC complements BFE at Levels 4-6 through Bidirectional Attention and feature-statistics-based calibration, combining response modulation with deep cross-modal calibration. This rationale is described in Overall Architecture and supported by Ablation Studies.

Joint use achieves 38.99% [email protected] versus 35.58% for BFE and 34.40% for DFC, supporting their complementary contributions.

Fusion[email protected][email protected][email protected]
BFE35.5824.496.47
DFC34.4019.385.76
BFE+DFC38.9925.087.52

We sincerely thank the reviewer for the valuable feedback and constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.