acceptodds
Under review as a conference paper at ICLR 2027

SACFusion: From Semantic Compatibility to Conditional Complementarity in RGB-Event Object Detection

Abstract

RGB-Event object detection leverages complementary information to improve robust perception under challenging and highly dynamic conditions, particularly for autonomous driving. However, substantial differences between RGB and Event modalities in semantic representation, observation density, and information reliability often lead to modality bias during fusion. To address this issue, we propose SACFusion, a two stage RGB-Event object detection framework that progresses from semantic compatibility to conditional complementarity. In Stage I, a Temporal Polarity Aggregator explicitly models the temporal, polarity, and local spatial structures of Event data, while Scale Aware Local Semantic Alignment and Cross Path Semantic Preservation adapt Event features to the pretrained RGB semantic space without reconstructing RGB appearance. In Stage II, an RGB dominant asymmetric fusion mechanism uses a spatially varying Event Retrieval Prior to guide RGB queries toward more informative Event regions at corresponding feature scales. This design preserves stable RGB representations while reducing interference from sparse or unreliable Event observations. By separating representation adaptation from complementary interaction, SACFusion enables Event information to be exploited more selectively according to local scene conditions. Experiments on DSEC-Det and PEOD demonstrate consistent improvements under diverse and challenging conditions, achieving relative mAP gains of 11.1% and 16.5%, respectively, over the strongest competing methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RGB-Event object detection, event based vision, cross modal fusion, semantic compatibility, conditional complementarity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NMb5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SACFusion is a two-stage framework for RGB–Event object detection. The first stage adapts event features to a pretrained RGB semantic space through temporal–polarity aggregation, local semantic alignment, and cross-path semantic preservation. The second stage uses RGB-guided asymmetric retrieval and a learned event retrieval prior to incorporate complementary event information. Experiments on DSEC-Det and PEOD report consistent improvements over the compared methods, supported by component ablations and feature visualizations.

Critical strengths
  1. The two-stage design is well motivated. Separating event representation adaptation from cross-modal fusion is a useful approach to modality imbalance. Local alignment accommodates small spatial mismatches, while RGB-guided retrieval provides a clear way to incorporate auxiliary event information without symmetric feature exchange.
  2. The improvements are substantial across both datasets. SACFusion exceeds the strongest reported baselines by 3.6 mAP points on DSEC-Det and 5.3 points on PEOD Overall, with a 4.0-point gain on PEOD Challenge. Its parameter count is also comparable to SPFD. These results provide good evidence for the effectiveness of the complete method.
Critical weaknesses
  1. The ablations do not isolate the main contributions. Table 2 evaluates SLSA and CP together and introduces asymmetric retrieval together with the learned prior. The final 0.73 AP gain therefore does not establish the prior’s individual contribution.
  2. There is also no comparison between one-stage and two-stage training using the same architecture and matched training budgets, despite the additional 30k iterations in Stage I. This leaves the benefit of the proposed training sequence insufficiently established.
  3. The claim of reducing interference from unreliable events needs more direct evidence. In Eq. (11), assigning the same low prior to all sampled candidates leaves the softmax weights unchanged. The prior therefore provides relative selection rather than directly controlling the amount of event information injected, consistent with its stated role. Figure 6 demonstrates spatial selectivity, but does not establish robustness when event observations are uniformly weak or noisy. Comparisons against RGB-only and prior-free retrieval under such conditions would better support this claim.
Questions for authors
  1. With the same architecture and a matched total training budget, how does one-stage joint training compare with the proposed two-stage schedule?
  2. How much does the Event Retrieval Prior contribute when the aligned features and RGB-to-event deformable retrieval are otherwise unchanged?
  3. When all sampled event candidates are weak or noisy, what limits their contribution? Can results grouped by event density or quality show whether the method avoids degrading the RGB-only baseline?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Comments on Training Strategy, Event Retrieval Prior, and Robustness Analysis

We thank the reviewer for the detailed assessment. We have added further experiments and clarified the training strategy and retrieval mechanism. We hope the following responses address your concerns.

  1. We compared our two-stage training with two single-stage alternatives using the same detection architecture and COCO-pretrained DINO-R50 initialization. One Stage (Detection) optimizes only the detection objective, while One Stage (Joint) additionally incorporates SLSA and CP. Under the same 20k-iteration detection training settings, they achieve 35.03 and 34.54 AP, respectively, compared with 35.89 AP for our method (Table 4). We also observed signs of overfitting when extending single-stage training beyond 20k iterations, suggesting that simply increasing the training budget would not necessarily improve their performance. Although our method includes an additional 30k-iteration Stage I, these results support the benefit of learning semantic compatibility before detection optimization. Further details are provided in Appendix A.5.
  2. We added an ablation that removes only the learned Event Retrieval Prior while retaining the aligned features and RGB-guided deformable retrieval. This variant achieves 34.88 AP, compared with 35.89 AP for the full model (Table 3), demonstrating a 1.01 AP improvement from the prior. We also evaluated its inputs separately. Density Only and Relation Only achieve 34.87 and 34.36 AP, respectively. We believe these additional results provide clearer evidence of the contribution of the Event Retrieval Prior.
  3. The observation regarding Eq. (11) is correct. When all sampled prior values are identical, the additive term cancels under softmax. Therefore, the prior adjusts the relative importance of sampled Event candidates rather than directly controlling the total Event contribution. To further examine its effectiveness under different observation conditions, we divided DSEC-Det into three Event density groups (Table 6). SACFusion improves over RGB-only detection by 2.84, 2.71, and 2.45 AP under low, medium, and high Event densities, respectively. The corresponding improvements over retrieval without the prior are 0.81, 2.05, and 0.31 AP. These results show that the proposed method consistently benefits from Event information across different density levels. We have also clarified in the manuscript that Event density reflects observation activity rather than absolute reliability, and that the prior performs relative candidate weighting rather than explicit suppression of uniformly unreliable Event observations.

Thank you again for your valuable comments and suggestions. If you have any further questions or concerns, please feel free to let us know. We would be happy to provide additional clarification.

Reviewer_Etw1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies modality bias in RGB-Event object detection. The authors propose SACFusion, a two-stage framework. Stage I, semantic compatibility learning, uses a Temporal Polarity Aggregator (TPA) to encode temporal and polarity structure, Scale-Aware Local Semantic Alignment (SLSA) to align RGB and Event features within local neighborhoods, and Cross-Path Semantic Preservation (CP) to keep the Event backbone close to the RGB semantic space. Stage II, conditional complementarity fusion, uses RGB features as queries to retrieve information from corresponding-scale Event features. An Event Retrieval Prior adjusts the attention weights of candidate locations, and the retrieved Event context is added to RGB features through a residual connection before DINO detection. The paper reports improvements over competing methods on DSEC-Det and PEOD.

Critical strengths
  1. Clear problem decomposition. This motivates the “compatibility before complementarity” design, and the proposed stages broadly follow that rationale.
  2. A plausible fusion strategy. Using RGB as the query source and Event as auxiliary evidence is well suited to settings where the modalities have different levels of reliability.
Critical weaknesses
  1. The contribution of individual components is not fully established. SLSA and CP are ablated together as “Semantic Alignment,” so their separate effects are unclear.
  2. The interpretation of the retrieval prior is supported mainly by indirect evidence. The paper describes it as estimating which Event regions are useful, but provides no independent labels or quantitative metric to validate this interpretation. Similarity scores and visualizations are suggestive.
Questions for authors
  1. Adding more detailed, module-by-module ablation experiments would provide stronger evidence for the effectiveness of each module.
  2. More recent baseline methods should be included in Table 1 to better demonstrate the competitiveness of the proposed method.
  3. The abstract gives relative mAP gains of 11.1% on DSEC-Det and 16.5% on PEOD, while the absolute differences reported in the main text and Table 1 are not consistent with those.
  4. Could the authors provide more detailed implementation details, such as the value choices for hyperparameters?
  5. What does “Rackbone” in Figure 2 refer to? Should it be “Backbone”?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Comments on Component Analysis, Baseline Comparisons, and Implementation Details

We thank the reviewer for the helpful suggestions. We have added further ablation studies, recent baseline comparisons, and more detailed implementation settings. We hope the following responses address your concerns.

  1. We added separate component ablations in Table 3. Removing CP while retaining SLSA reduces AP from 35.89 to 35.20. Removing the Event Retrieval Prior while retaining asymmetric deformable retrieval yields 34.88 AP, indicating a 1.01 AP improvement from the learned prior. We also evaluated the two prior inputs separately. Density Only and Relation Only achieve 34.87 and 34.36 AP, respectively. These results clarify the individual contributions that were not distinguished in the original Table 2.
  2. We added a quantitative analysis of the prior under different Event densities (Table 6). Compared with retrieval without the prior, SACFusion improves AP by 0.81, 2.05, and 0.31 points in the low-, medium-, and high-density groups, respectively. Section 3.3 further clarifies that the prior adjusts the relative importance of sampled Event candidates rather than estimating absolute Event reliability. Figure 6 illustrates the spatial variation of the learned responses, while Tables 3 and 6 provide additional quantitative evidence of the prior's contribution to detection. We believe these results provide stronger support for the effectiveness of the proposed mechanism.
  3. We included DAGr (Nature 2024) and FlexEvent (NeurIPS 2025) in Table 1, with reproduced results on DSEC-Det and PEOD. Some results differ from those reported in the original papers, partly due to differences in dataset processing and evaluation protocols. Specifically, we use the SFNet-processed version of DSEC-Det, while the original evaluation implementations of DAGr and FlexEvent exclude certain small ground-truth boxes. To ensure a consistent comparison, we evaluate all methods using the same dataset annotations and COCO-style evaluation protocol, without method-specific ground-truth filtering. Although the reproduced results may differ from the published values, we have followed the available implementations and training configurations as closely as possible. The reproduction settings are further described in Appendix A.3.
  4. We corrected the abstract to report absolute improvements of 3.6 AP points on DSEC-Det and 5.3 AP points on PEOD Overall, rather than the previously reported relative percentages. These values are consistent with Table 1 and Section 4.2.
  5. We have added the key implementation details in Appendix A.1–A.3, including parameter initialization, loss weights, attention configurations, training resolutions, and evaluation settings. We will also release the complete source code, training configurations, and model weights upon acceptance to facilitate reproducibility and further research.
  6. We sincerely apologize for this oversight and thank the reviewer for pointing it out. The typo "Rackbone" in Figure 2 has been corrected to "Backbone". We also revised the figure annotations and related descriptions to improve clarity.

Thank you again for your valuable comments and suggestions. If you have any further questions or concerns, please feel free to let us know. We would be happy to provide additional clarification.

Thank you for your reply. I will keep my score.

AuthorsOpenReview ↗

Thank You for Your Response and Request for Further Feedback

Thank you for your response. I respect your decision. I would also greatly appreciate any further specific suggestions or clarification on the remaining weaknesses of our work, as they would be very helpful for improving the paper.

Reviewer_vKCdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a two-stage fusion-based detection method called SACFusion. The first stage adapts events to an RGB backbone through temporal–polarity aggregation, local semantic alignment, and cross-path semantic preservation. The second stage performs RGB-dominant asymmetric fusion using a spatially varying Event Retrieval Prior to guide RGB queries toward more informative Event regions. Experiments on DSEC-Det and PEOD demonstrate the proposed method’s improvements over state-of-the-art methods.

Critical strengths
  • The proposed separation of representation adaptation and selective cross-modal interaction is an interesting idea. Also, combining semantic alignment with prior-guided asymmetric retrieval offers a potentially useful approach to cross-modal fusion.
  • The method shows significant improvements on two benchmarks.
Critical weaknesses
  • The proposed method is composed of a lot of new modules, while the motivation for each design remains largely vague. For example, Section 3.2.3 states that local alignment may cause the event backbone to drift from the pretrained RGB semantic space, but does not explain clearly why this occurs despite the RGB-directed alignment objective, or demonstrate how CP specifically resolves it. Moreover, the Event Retrieval Prior combines several feature-relation terms and event density, but how these operations are motivated is not explained clearly.
  • There are some strong but unsupported arguments. For example, in Line 87, it is stated that “We therefore argue that semantic compatibility and conditional complementarity should be treated as two distinct but progressive learning objectives.” The results show that the proposed pipeline is effective, but do not demonstrate that compatibility should be learned before complementarity.
  • Insufficient ablation study. Although Table 2 provides some results on the effectiveness of each module, SLSA and CP are evaluated together and does not separate asymmetric retrieval from the learned retrieval prior. It remains unclear how much improvement comes from each mechanism.
  • The results are sometimes inconsistent. The RGB-only baseline (DINO-R50) differs between Tables 1 and 2 (32.3 versus 31.84 AP) without explanation.
  • The writing and figures are sometimes difficult to follow.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer Comments on Methodological Clarifications and Additional Ablation Studies

We thank the reviewer for the constructive comments. We have revised the method descriptions and added several experiments to address the concerns. We provide point-by-point responses below and hope these revisions address your concerns.

  1. SLSA encourages local semantic correspondence between RGB and Event features, but does not necessarily preserve the Event backbone's pretrained RGB representations during Event-specific training. Previous works, such as SFNet and MCFNet, have shown that Event information can be transformed into representations with RGB-like structural characteristics. This motivates us to preserve the semantic knowledge acquired from large-scale RGB pretraining, which we believe can benefit the subsequent fusion detection stage. We therefore introduce CP to constrain the Event backbone during adaptation. Specifically, CP processes identical RGB images through both backbones and aligns the Event backbone's outputs with those of the frozen RGB backbone. As shown in Table 5, incorporating CP increases cross-path cosine similarity from 0.66/0.60/0.52 to 0.99/0.98/0.97 across res3–res5, while improving AP from 35.20 to 35.89. We have clarified this motivation in Section 3.2.3.
  2. The Event Retrieval Prior combines feature relations with Event density because they provide complementary information. Feature differences and products characterize cross-modal discrepancies and agreement, while Event density reflects local Event activity. We evaluate their contributions separately in Table 3. Density Only achieves 34.87 AP and Relation Only achieves 34.36 AP, compared with 35.89 AP for the complete prior. We have also clarified the motivation for these inputs in Section 3.3.1.
  3. To examine the proposed training sequence, we compared two single-stage alternatives using the same final detection architecture. One Stage (Detection) achieves 35.03 AP, while One Stage (Joint), which jointly optimizes detection, SLSA, and CP, achieves 34.54 AP. Our two-stage approach obtains 35.89 AP (Table 4). These results support the benefit of learning semantic compatibility before detection optimization under the evaluated training settings.
  4. We added separate ablations for CP and the Event Retrieval Prior in Table 3. Removing CP while retaining SLSA reduces AP to 35.20. Removing the prior while retaining asymmetric deformable retrieval reduces AP to 34.88, compared with 35.89 for the full model. These results help clarify the respective contributions of semantic preservation and prior-guided asymmetric retrieval.
  5. We apologize for this oversight. The RGB-only result in Table 2 has been corrected to 32.29 AP, consistent with the rounded value of 32.3 AP in Table 1. Both tables now report the same baseline.
  6. We have revised the descriptions of CP and the Event Retrieval Prior, corrected figure annotations, and improved the consistency of terminology and experimental explanations throughout the manuscript. We hope these revisions make the paper clearer and easier to follow.

Thank you again for your valuable comments and suggestions. If you have any further questions or concerns, please feel free to let us know. We would be happy to provide additional clarification.

Thank you for the response and additional experiments. They clarify each module’s contribution to AP, but the motivation still remains largely heuristic. For example, Figure 5 evaluates SLSA and CP together, and the rebuttal’s CP analysis measures agreement on identical RGB inputs, while it is unclear what the CP actually corrects. And there are more small details like this that are unsupported.

Moreover, this paper talks a lot about the modalities bias, while the specific definitions of these biases, how each module addresses the corresponding bias, and the concrete results beyond performance improvements are mostly subjective and vague.

AuthorsOpenReview ↗

Clarification on Method Motivations and Supporting Evidence

Thank you for your follow-up comments. We would like to further clarify the motivations behind our design choices and how the existing experiments support the proposed mechanisms, particularly the role of CP and the two forms of modality bias.

  1. The motivation for CP comes from previous RGB and Event fusion studies, including SFNet and MCFNet, which demonstrate the effectiveness of transforming Event data into image-like representations and leveraging RGB pretrained backbones for fusion detection. These findings suggest that pretrained RGB representations provide a useful semantic foundation for Event learning. Considering the representation imbalance discussed in our paper, we introduce CP alongside SLSA to preserve this foundation during Event adaptation. SLSA encourages local semantic correspondence between RGB and Event features, while CP constrains the Event backbone to retain its pretrained representation capability. This design is also supported by Cao et al. (2026), which introduces capability preservation for a similar purpose.
  2. The use of identical RGB inputs in Table 5 is intentional, as it directly measures how well the Event backbone retains its pretrained RGB responses. Without CP, the cross-path similarities are 0.66/0.60/0.52 across res3–res5, compared with 0.99/0.98/0.97 when CP is applied. Detection AP also improves from 35.20 to 35.89. Figure 5 evaluates the overall RGB and Event semantic alignment achieved by Stage I, while Table 5 separately examines the contribution of CP. These results provide complementary evidence for the proposed design.
  3. Regarding modality bias, the Introduction explicitly distinguishes representation bias from interaction bias. Representation bias arises from differences in sensing mechanisms, feature distributions, and pretrained semantic capabilities. Interaction bias concerns potential interference caused by indiscriminate information exchange between modalities with unequal reliability. These two issues motivate our framework. Stage I improves semantic compatibility through Event representation adaptation and preservation, while Stage II uses RGB dominant asymmetric retrieval to selectively incorporate complementary Event information. The corresponding mechanisms and objectives are detailed in Sections 3.2 and 3.3.
  4. We also conduct targeted analyses to better understand the effects of the proposed mechanisms. Figure 5 quantifies feature similarity between RGB and Event representations, Table 5 evaluates pretrained semantic preservation, and Table 4 compares different training strategies. Table 3 isolates the contributions of CP and the retrieval prior, while Figure 6 and Table 6 examine the spatial behavior of the learned prior and its effectiveness under different Event density conditions. These analyses provide quantitative support for the proposed mechanisms from different perspectives.

Our work focuses on developing and empirically validating an effective RGB and Event detection framework. The design motivations are supported by prior research, explicit method formulations, and targeted experimental analyses. While a more formal theoretical investigation of modality bias and semantic preservation could offer further insights, we consider it a valuable direction for future work rather than the primary objective of this study.

Comments on the paper

1 comment

Dear Reviewers, as the discussion deadline is approaching, please review the authors’ responses and join the discussion. Please share whether your concerns have been addressed and update your assessments as appropriate. Thank you for your time and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.