acceptodds
Withdrawn from ICLR 2027

RETDet: Degradation-Aware Text as a Detachable Prior for RGB-Event Object Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

By fusing RGB images and event data, multimodal object detection improves accuracy and robustness in complex scenes, yet attention-dominated fusion modules incur high computational cost, and under degraded conditions, the representation capability of both modalities declines. High-level semantic priors describing the degradation itself could compensate for this loss, but querying a multimodal large language model (MLLM) at every inference step is prohibitively expensive. We propose RETDet, an RGB-Event-Text detection framework in which degradation-aware text acts as a detachable prior. During training, MLLM-generated degradation descriptions are mapped into layer-wise modulation parameters and randomly replaced by a learned null embedding, so that the detector internalizes degradation priors into its visual pathway while remaining able to consume explicit descriptions. The trained model supports two inference modes: a prior mode taking only RGB and event inputs, invoking neither an MLLM nor a text encoder at deployment, and a conditioned mode accepting scene descriptions for higher accuracy. Without any textual input at inference, prior mode already outperforms SOTA RGB-Event methods on DSEC, PKU-DDD17-Car, TUMTraf, and EventKITTI, and conditioned mode yields further gains concentrated on the most degraded subsets, decoupling the semantic benefit of MLLM supervision from its inference cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Obeject detection, multimodal fusion, vision language model, event camera

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sqgYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a multi-modal object detection method with a combination of RGB images, event data, and text. This paper claims that high-level semantic priors describing the image degradation could compensate for the information loss, and then designs a new method that conditions the RoI features on the MLLM outputs. Experiments on four datasets demonstrate the improvements the proposed method.

Critical strengths
  • The proposed text-guided architecture is new (not sure about the fusion part). It is an interesting attempt to see how the text-guided MLLM outputs change the RGB-Event object detection performance.
  • Extensive experiments on four representative datasets show a clear improvement of different modules, demonstrating state-of-the-art performance.
Critical weaknesses
  • The presentation of the method is not clear enough. It is unclear where the null embedding comes from and how it is learnt, what the embedding is composed of, and how it is passed through CLIP to modulate the results.
  • Utilizing the text information to enhance Event-RGB object detection performance is a bit counter-intuitive. First, text usually has large domain gaps with RGB and Event data, although an MLLM has been leveraged to bridge the modality gap, the MLLM has never seen the event data so the domain gap remains, which in most cases would degrade the performance. On the other hand, I am wondering what additional information the text brings to the model that benefits the detection performance, as making the model aware of the degradation semantics does not mitigate the degradation itself. Regarding these concerns, more analysis and explanation are required to see what mechanism drives the improvement.
  • Using text information to enhance object detection performance also introduces two strong assumptions: there should always be a lot of degradations in the data, and the degradation context is always known when capturing images. However, these assumptions do not hold in real-world applications.
  • The main improvements of the model come from the fusion strategy, while the text conditioning plays a relatively minor role compared to the fusion module, which can be seen from the ablation study. Therefore, the contribution of the text may be somewhat exaggerated in this study.
Questions for authors

None

Reviewer_fRtwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses several key challenges in RGB-Event object detection, including performance degradation under adverse conditions, the computational overhead of complex fusion mechanisms, and the high latency associated with online multimodal large language model (MLLM) inference. To tackle these challenges, the authors propose RETDet, a framework that treats degradation-aware text as a detachable prior. By incorporating random text dropout and a learnable null embedding during training, the framework supports both text-free inference and explicit text-conditioned inference within a unified model. In addition, RETDet integrates lightweight cross-modal feature fusion and proposal refinement mechanisms, achieving competitive detection performance across four benchmark datasets. Overall, the proposed framework presents an interesting and practically relevant approach, supported by encouraging experimental results. However, several aspects warrant further investigation, particularly the independent contribution of text supervision, the additional benefits of the conditioned inference mode, the robustness of the learned prior under distribution shifts, and the fairness of comparisons with existing methods.

Critical strengths
  1. The core idea is novel and practically valuable. The paper introduces degradation-aware text as a detachable prior for RGB-Event object detection. By incorporating a conditioning dropout mechanism during training, the model can perform detection without requiring explicit textual descriptions at inference time. This design effectively adapts the existing conditioning dropout strategy to degradation-aware RGB-Event detection, establishing a meaningful connection between semantic-guided training and efficient inference. The proposed formulation demonstrates methodological novelty and practical relevance.
  2. The dual inference modes offer flexibility for practical deployment. RETDet supports two inference modes within a unified framework. The prior mode eliminates the need for text generation and encoding during inference, while the conditioned mode allows explicit scene descriptions to be incorporated when additional guidance is beneficial. Both modes share the same model parameters and differ only in the source of the conditioning embedding. This design provides flexibility in balancing detection performance and inference overhead across different deployment scenarios.
  3. The lightweight fusion module achieves a favorable balance between accuracy and efficiency. The proposed Parameter-Free Enhanced Module (PFEM) performs cross-modal feature enhancement and statistical alignment through neuron energy modeling and feature distribution alignment without introducing additional learnable parameters. Experimental results demonstrate the effectiveness of the overall framework in terms of both detection accuracy and computational efficiency. As reported in Table 3, RETDet achieves 13.0 FPS and an mAP of 0.507 in prior mode on the PKU-DDD17-Car dataset, outperforming the compared baselines in both metrics. These results highlight the potential of the proposed approach for efficient multimodal detection.
  4. The experimental evaluation provides substantial evidence of the framework's effectiveness. The authors evaluate RETDet on four public benchmark datasets and conduct several ablation studies, including component-level analysis, inference cost comparisons, and evaluations of different conditioning sources. These experiments provide useful evidence supporting the effectiveness of the overall framework and its key components. Nevertheless, further investigation would be beneficial to more clearly isolate the contribution of text-based semantic supervision and establish its causal relationship with the observed performance improvements.
Critical weaknesses
  1. The independent contribution of text-based semantic supervision is not sufficiently demonstrated. The central claim of RETDet is that degradation-aware semantic knowledge can be internalized into the visual representations during training, allowing the model to retain the benefits of textual priors without requiring explicit text input at inference time. However, the experiments in Table 2 primarily demonstrate the effectiveness of the learned null embedding and the associated modulation architecture, without fully isolating the contribution of meaningful degradation-aware semantic supervision from that of additional modulation structures and training mechanisms. More rigorous controlled experiments, such as training with randomly shuffled degradation descriptions, fixed non-informative text, or a learnable null embedding without explicit semantic supervision, would help determine whether the observed performance improvements genuinely originate from the learned degradation semantics.
  2. The robustness of the prior mode under distribution shifts remains unclear. The authors interpret the learned null embedding as capturing the marginal degradation distribution of the training data. However, the paper does not systematically evaluate the robustness of this prior under distribution shifts. In particular, when test samples contain previously unseen degradation types, combinations of degradation factors, or more severe degradation levels, the learned prior may not generalize effectively. Cross-dataset evaluations or controlled experiments involving unseen degradation conditions would help assess the generalization capability of the prior mode and determine whether its performance advantages over conventional RGB-Event detectors can be maintained under distribution shifts.
  3. The additional performance gains of the conditioned mode are relatively limited. As shown in Table 4, the conditioned mode provides only modest improvements over the prior mode, with mAP gains ranging from 0.002 to 0.009 across the three full test sets and reaching 0.012 on the degraded TUMTraf subset. Although these results suggest that the model may retain much of the benefit of text-assisted training without explicit descriptions, the current experiments do not clearly establish the scenarios in which conditioned inference provides substantial additional value. A more detailed analysis across different degradation types and severity levels would help clarify the practical benefits and applicability of the conditioned mode.
  4. The fairness and comprehensiveness of comparisons with existing methods require further improvement. Although RETDet achieves competitive overall detection performance across four benchmark datasets, the experimental configurations of the compared methods are not entirely consistent. As described in Section 4.1, most baseline results follow the evaluation protocol of CAFR, whereas the results for FlexEvent are taken directly from its original publication and are reported only on DSEC. Furthermore, differences in backbone architectures and detection frameworks may also contribute to the observed performance gaps, making it difficult to isolate the improvements attributable to the proposed components from those arising from the underlying model architecture. More controlled comparisons using consistent backbone architectures and training configurations would strengthen the validity of the experimental conclusions. In addition, although RETDet achieves the highest overall mAP on the reported datasets, it does not consistently outperform competing methods across all object categories. Therefore, the performance claims should be stated more precisely.
  5. The impact of textual description quality on the framework's performance has not been sufficiently investigated. The degradation descriptions used during training are generated exclusively by Qwen-VL-Max. However, the accuracy, granularity, and consistency of these descriptions may directly influence how effectively the model learns degradation-related semantic knowledge. Although the authors acknowledge the potential impact of inaccurate descriptions, the paper does not provide a quantitative analysis of the model's sensitivity to variations in description quality. Controlled experiments involving vision-language models with different capabilities, deliberately incorrect degradation labels, varying levels of description granularity, or injected textual noise would provide a clearer understanding of the framework's dependence on description quality and its robustness to imperfect semantic supervision.
  6. The statistical assumptions underlying Bayesian proposal aggregation lack sufficient empirical validation. TBRM treats highly overlapping proposals as conditionally independent Gaussian observations of the same underlying object and uses objectness scores as proxies for localization reliability. However, in crowded or occluded scenes, highly overlapping proposals may correspond to different objects, while objectness confidence does not necessarily correlate with localization accuracy. Although the authors acknowledge these limitations, their practical impact has not been quantitatively evaluated. Additional experiments on crowded scenes, together with comparisons against alternative weighting strategies based on predicted IoU or localization quality estimates, would help assess the reliability and robustness of the proposed proposal aggregation mechanism.
Questions for authors
  1. Regarding the independent contribution of text-based semantic supervision: Table 2 demonstrates the performance advantages of the learned null embedding over a random embedding and a model without TPN. However, these comparisons do not fully disentangle the contribution of degradation-aware semantic supervision from that of the additional modulation architecture. Have the authors considered training with randomly shuffled degradation descriptions, fixed non-informative text, or only a learnable null embedding without explicit semantic supervision? Such controlled experiments would help clarify whether the observed performance gains primarily arise from learning degradation-related semantic knowledge.
  2. Regarding the robustness of the prior mode under distribution shifts: The authors interpret the learned null embedding as capturing the marginal degradation distribution of the training data. If test samples contain previously unseen degradation types, combinations of degradation factors, or more severe degradation levels, can the prior mode maintain its performance advantages over conventional RGB-Event detectors? Could the authors provide cross-dataset evaluations or controlled experiments involving unseen degradation conditions to further assess its generalization capability?
  3. Regarding the additional value of the conditioned mode: Table 4 shows that the conditioned mode provides relatively modest improvements over the prior mode. Could the authors further analyze the performance differences across degradation types and severity levels? In particular, under what conditions can explicit degradation descriptions provide meaningful additional information that cannot be adequately captured by the learned prior?
  4. Regarding the individual contributions of PFEM components: PFEM combines SimAM-inspired parameter-free feature enhancement with AdaIN-inspired statistical alignment. However, the existing ablation studies primarily evaluate PFEM as a whole. Could the authors separately assess the contribution of each component and clarify whether the two mechanisms provide complementary benefits?
  5. Regarding the reliability of Bayesian proposal aggregation: TBRM uses objectness scores as proxies for proposal localization reliability, although objectness confidence does not necessarily correlate with localization accuracy. Have the authors considered alternative weighting strategies based on predicted IoU or other localization quality estimates? Furthermore, could the IoU-based connected-component clustering incorrectly merge proposals corresponding to different objects in crowded or heavily occluded scenes?
  6. Regarding robustness to textual description quality: The degradation descriptions used during training are generated by Qwen-VL-Max. How sensitive is the prior mode to the quality of these descriptions, particularly when using a less capable vision-language model or introducing incorrect degradation labels, severity estimation errors, or textual noise? Could the authors conduct controlled perturbation experiments to quantify the impact of description quality on detection performance?
Reviewer_Zgu4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RETDet, an RGB-Event detector that uses MLLM-generated degradation descriptions as a detachable prior. During training, these descriptions are mapped to layer-wise modulation parameters and randomly replaced by a learned null embedding, so the deployed prior mode requires neither an MLLM nor a text encoder. The practical motivation is clear, but the central claim is not well supported. Most of the reported improvement appears to come from a substantially different detector, while the explicit text prior contributes only marginal and partly circular gains.

Critical strengths

The paper reports results on DSEC, PKU-DDD17-Car, TUMTraf, and EventKITTI, and separates prior mode from conditioned mode.

Critical weaknesses

Refer to the questions.

Questions for authors
  1. Conditioned mode improves prior mode by only +0.002, +0.003, and +0.009 on the three full test sets, and by +0.012 on the selected TUMTraf subset. On TUMTraf, removing the text-conditioned pyramid reduces mAP only from 0.543 to 0.523. These changes are small relative to the gaps over the RetinaNet baselines, so the paper does not establish that MLLM supervision is the source of the reported improvement.
  2. PFEM combines SimAM-style energy attention with AdaIN-style moment alignment, while the proposal update is explicitly the confidence-weighted averaging of Weighted Boxes Fusion. The aggregation itself does not use text. Section 4.4 does not separate the two PFEM operations or test the reversed content-style assignment, and replacing the aggregation with NMS or Soft-NMS changes TUMTraf mAP only from 0.543 to 0.537 or 0.539.
  3. RETDet underperforms existing methods on DSEC large vehicles, TUMTraf cars and buses, and EventKITTI cars; the EventKITTI average gain is only +0.010. Prior mode requires 306.6 ms per 640×480 frame, or 3.26 FPS, which is not real-time driving perception. The 13.0 FPS result is measured at 346×260, while the model has 98.1M parameters and 425.1 GFLOPs. The impact statement’s claim of suitability for resource-limited real-world systems is therefore not supported.
  4. Appendix C.1 presents detections absent from the annotations as correctly recovered targets. Without independent verification, these outputs may simply be false positives, so the figures do not substantiate the degradation-robustness claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.