- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a multi-modal object detection method with a combination of RGB images, event data, and text. This paper claims that high-level semantic priors describing the image degradation could compensate for the information loss, and then designs a new method that conditions the RoI features on the MLLM outputs. Experiments on four datasets demonstrate the improvements the proposed method.
Critical strengths
- The proposed text-guided architecture is new (not sure about the fusion part). It is an interesting attempt to see how the text-guided MLLM outputs change the RGB-Event object detection performance.
- Extensive experiments on four representative datasets show a clear improvement of different modules, demonstrating state-of-the-art performance.
Critical weaknesses
- The presentation of the method is not clear enough. It is unclear where the null embedding comes from and how it is learnt, what the embedding is composed of, and how it is passed through CLIP to modulate the results.
- Utilizing the text information to enhance Event-RGB object detection performance is a bit counter-intuitive. First, text usually has large domain gaps with RGB and Event data, although an MLLM has been leveraged to bridge the modality gap, the MLLM has never seen the event data so the domain gap remains, which in most cases would degrade the performance. On the other hand, I am wondering what additional information the text brings to the model that benefits the detection performance, as making the model aware of the degradation semantics does not mitigate the degradation itself. Regarding these concerns, more analysis and explanation are required to see what mechanism drives the improvement.
- Using text information to enhance object detection performance also introduces two strong assumptions: there should always be a lot of degradations in the data, and the degradation context is always known when capturing images. However, these assumptions do not hold in real-world applications.
- The main improvements of the model come from the fusion strategy, while the text conditioning plays a relatively minor role compared to the fusion module, which can be seen from the ablation study. Therefore, the contribution of the text may be somewhat exaggerated in this study.
Questions for authors
None