acceptodds
Desk rejected at ICLR 2027

Open-Vocabulary Object Detection for Low-Altitude Scenarios Using RGB-Infrared Data: A Benchmark and A New Method

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-altitude UAV perception plays an important role in applications such as urban management, environmental monitoring, and emergency response. However, conventional object detectors rely on a predefined category set and struggle to recognize emerging object categories in real-world environments. Open-vocabulary object detection (OVD) provides a promising solution by transferring visual-language knowledge to enable recognition beyond fixed categories. Nevertheless, applying OVD to low-altitude scenarios remains challenging due to two critical issues: existing UAV datasets are mainly designed for closed-set detection and lack multimodal, large-scale benchmarks for open-vocabulary learning; meanwhile, current OVD methods are predominantly trained on natural images or remote sensing images, which exhibit substantial domain gaps from low-altitude aerial imagery characterized by small objects, arbitrary orientations, complex backgrounds, and significant illumination variations. To address these challenges, we construct UAVOVD, the first visible-infrared open-vocabulary object detection dataset specifically designed for low-altitude scenarios. UAVOVD contains 25K registered multimodal image pairs, 88 object categories, and 350K rotated bounding box annotations, with a seen-unseen category split to support open-vocabulary generalization evaluation. Based on this dataset, we propose LAOVD, a multimodal open-vocabulary rotated object detection framework for low-altitude environments. Furthermore, we introduce a Structure-Guided Asymmetric Gated Fusion (SGAGF) module, which leverages infrared modality as a stable structural prior and adaptively integrates visible modality details according to cross-modal reliability, thereby improving object localization and semantic transferability under complex imaging conditions. Extensive experiments on UAVOVD, DroneVehicle, and ATR-UMOD demonstrate that the proposed dataset provides a challenging benchmark for low-altitude open-vocabulary detection. Moreover, LAOVD consistently outperforms existing methods in unseen-category detection and cross-domain generalization, validating the effectiveness of multimodal open-vocabulary learning for low-altitude intelligent perception.

Primary area:
datasets and benchmarks
Keywords:
Open-vocabulary object detection, Multimodal object detection, Cross-modal fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_h3aoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UAVOVD, a visible–infrared open-vocabulary oriented object detection benchmark for low-altitude scenarios, and LAOVD, a multimodal open-vocabulary oriented object detection framework. A Structure-Guided Asymmetric Gated Fusion (SGAGF) module is designed to fuse infrared and visible modalities. LAOVD achieves superior performance on several benchmarks.

Critical strengths
  1. The paper introduces the first visible–infrared OVD benchmark for low-altitude scenarios, with more object categories than previous datasets.
  2. The paper is well written and easy to read.
Critical weaknesses
  1. The motivation and necessity for dual-modality open-vocabulary object detection are not explored in sufficient depth. The modality fusion design is not inherently connected to open-vocabulary object detection. The architectural innovations do not stem from the specific difficulties and challenges of open-vocabulary object detection. Instead, the paper reads more like a combination of modality fusion, open-vocabulary object detection, and a new dataset. It does not sufficiently explain why multimodal fusion is necessary for open-vocabulary object detection, and the connection between the motivation and the design is insufficient.
  2. The related work section lacks sufficient discussion of RGB–IR methods.
  3. The experimental settings in Section 5.2 are incomplete.
  4. The main results table compares too few methods and should include more OVD methods.
  5. The reasoning supporting the claims about model size and generalization in L411 is insufficient.
Questions for authors

See weakness, especially W1, 4.

2 comments
AuthorsOpenReview ↗

Responses on Method Motivation and Experimental Evidence

(1) Motivation for Dual-Modal Open-Vocabulary Detection

This work focuses on open-vocabulary detection in low-altitude UAV scenes. Its primary contribution is the UAVOVD dataset, on which the subsequent methods and experiments are built. In complex low-altitude scenes, low light, shadows, and glare can reduce the visibility of object contours and textures in visible images. This affects oriented object localization and weakens the matching between candidate regions and novel-class text descriptions. When thermal contrast is present, infrared images provide complementary structural cues, while visible images retain the texture and appearance details needed to distinguish semantically similar categories.

Therefore, the problem addressed in this work involves not only aggregating information from the two modalities, but also organizing that information to support novel-class detection when novel-class annotations are unavailable. To address this, SGAGF adopts an asymmetric design in which infrared structural information is dominant and visible details provide complementary information. The infrared branch and compensation path preserve infrared object evidence, while the joint gate regulates the injection of visible information. Region–text alignment then associates the fused region representations with category semantics. This design aims to preserve both object structure and fine-grained semantic information.

(2) Related Work on Visible–Infrared Detection

The related work section focuses more on open-vocabulary detection methods and visible–infrared datasets. We will expand the related work to cover UA-CMDet, which uses uncertainty-aware learning; TSFADet, which performs cross-modal spatial alignment; and ICAFusion, which uses cross-attention fusion. These methods primarily study how to use cross-modal information for predefined categories, whereas our work focuses on localizing and recognizing novel objects when detection supervision is available only for base classes.

To the best of our knowledge, UAVOVD is the first RGB–IR bimodal open-vocabulary oriented object detection benchmark for low-altitude scenes. We also propose the corresponding LAOVD framework. Its SGAGF module prioritizes infrared structural information and uses gated fusion to incorporate visible semantic details, followed by region–text alignment. As shown in Table 4, SGAGF improves novel-class mAP50 by 2.03 percentage points over visible-dominant fusion, providing experimental evidence for a fusion design aimed at novel-class detection.

(3) Experimental Settings in Section 5.2

We use the AdamW optimizer with an initial learning rate of 5 × 10⁻⁴, a final learning rate of 5 × 10⁻⁶ (a learning-rate factor of 0.01), and a weight decay of 10⁻⁴. Models are trained for 100 epochs with a batch size of 144 on four RTX 5090 GPUs.

(4) Additional Open-Vocabulary Detection Baselines

Existing open-vocabulary detection methods are evaluated on datasets with horizontal bounding-box annotations. Adapting each method to oriented object detection requires changes to its data-loading pipeline and detection head, which makes the implementation more involved. We have selected Grounding DINO, ViLD, and CastDet as additional baselines. Experiments are currently underway.

(5) Evidence for the Claim in Line 411

The claim in Line 411 is supported by two comparisons: scaling up the model and changing the fusion strategy while keeping the network settings the same. In Table 2, scaling the model from small to medium improves novel-class mAP50 by only 0.29 percentage points. In Table 4, keeping the backbone and training settings fixed while changing from visible-dominant gating to infrared-dominant gating improves novel-class mAP50 by 2.03 percentage points. This improvement results from the change in fusion strategy and cannot be explained by an increase in model size.

I thank the authors for their response. After reading the rebuttal, my concern on additional OVD baselines is not fully resolved. Therefore I maintain my initial rating.

Reviewer_yPoyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UAVOVD, a visible–infrared dataset for open-vocabulary oriented object detection in low-altitude UAV scenes, and proposes LAOVD. LAOVD extracts features from the two modalities with two separate YOLO26 backbones, fuses them with an infrared-dominant gated module (SGAGF), and performs region–text alignment with a frozen CLIP text encoder. Experiments compare LAOVD with single-modality YOLO-World and DOSOD on UAVOVD and evaluate zero-shot transfer on DroneVehicle and ATR-UMOD.

Critical strengths

It combines paired visible–infrared images, oriented bounding boxes, and 88 fine-grained categories, and it covers day and night, different weather conditions, flight altitudes, and viewing angles. Few existing UAV datasets offer this combination. Most novel classes are also close to a base class, which makes the split a better test of fine-grained open-vocabulary recognition than a random split.

Critical weaknesses
  1. LAOVD uses a YOLO26 backbone, whereas all baselines use YOLOv8. LAOVD also has two independent backbones, so its backbone cost is roughly twice that of the baselines, yet the paper reports no parameter counts, FLOPs, or inference speed. In addition, every baseline takes only a single modality as input. The gains in Table 2 therefore cannot be attributed to SGAGF. At minimum, the authors should add (a) single-modality LAOVD with the same YOLO26 backbone, (b) a comparison in which LAOVD and the baselines share the same backbone, and (c) late fusion (NMS/WBF) of two single-modality models.
  2. In Table 4, plain element-wise addition already reaches 0.1686 novel-class mAP50, higher than every single-modality baseline (best: 0.156). Of the 3.3-point gain over the strongest baseline, about 1.3 points are thus obtained without SGAGF. Meanwhile, SGAGF gives the worst base-class result among the four fusion strategies (0.5652 vs. 0.5823–0.5991). The paper also lacks an ablation of the individual components of SGAGF.
Questions for authors

The dataset is a useful contribution, but the experiments do not yet show that the gains come from SGAGF, and the method itself is incremental. I rate the paper weak rejection and would consider raising my score if the controlled comparisons requested in W1 and W2 are provided.

3 comments
AuthorsOpenReview ↗

Controlled Comparisons, Computational Efficiency, and SGAGF Ablations

(1) Controlled Comparisons and Computational Efficiency

We have added the three requested controlled comparisons, as shown in Table R1.

  • (a) Single-modality LAOVD with the same backbone. With YOLO26s, infrared-only and visible-only LAOVD-S achieve novel-class mAP50 scores of 0.1044 and 0.1300, respectively.
  • (b) LAOVD with a matched YOLOv8m backbone. We replaced the backbone of LAOVD-M with YOLOv8m, kept the rest of the architecture unchanged, and retrained the model. The resulting novel-class mAP50 of 0.1691 exceeds those of the YOLOv8-based baselines but remains below the 0.1890 achieved by LAOVD-M with YOLO26m. This comparison indicates that the backbone accounts for part of the performance gain.
  • (c) Late fusion of two single-modality models. WBF and NMS achieve novel-class mAP50 scores of 0.1198 and 0.1283, respectively. Both are below the 0.1861 achieved by LAOVD-S with SGAGF. The models use the same YOLO26s backbone type, and both late-fusion configurations use predictions from the two modalities. These results support the effectiveness of the proposed fusion design for novel-class detection.

Table R1. Detection results on UAVOVD

MethodInput / FusionBackboneBase mAP50Base mAP95Novel mAP50Novel mAP95
YOLO-World v1-sInfraredYOLOv8s0.43540.31920.11570.0841
YOLO-World v2-sInfraredYOLOv8s0.43690.32530.10200.0773
DOSOD-sInfraredYOLOv8s0.42400.30660.10250.0740
YOLO-World v1-mInfraredYOLOv8m0.48340.36450.12040.0905
YOLO-World v2-mInfraredYOLOv8m0.41950.30930.12850.0893
DOSOD-mInfraredYOLOv8m0.42470.30800.11190.0824
YOLO-World v1-sVisibleYOLOv8s0.53070.41160.14630.1134
YOLO-World v2-sVisibleYOLOv8s0.50020.38780.14200.1105
DOSOD-sVisibleYOLOv8s0.51770.39860.13660.1013
YOLO-World v1-mVisibleYOLOv8m0.55600.43340.15600.1240
YOLO-World v2-mVisibleYOLOv8m0.52910.41030.15560.1215
DOSOD-mVisibleYOLOv8m0.55710.44330.14660.1166
LAOVD-S (Ours)Visible + Infrared (SGAGF)YOLO26s0.47680.36780.18610.1359
LAOVD-M (Ours)Visible + Infrared (SGAGF)YOLO26m0.56520.44950.18900.1450
LAOVD-M (Ours)Visible + Infrared (SGAGF)YOLOv8m0.58690.47370.16910.1347
LAOVD-S (Ours)Infrared onlyYOLO26s0.48180.36320.10440.0756
LAOVD-S (Ours)Visible onlyYOLO26s0.53380.41180.13000.1006
LAOVD-S (Ours)Visible + Infrared (WBF late fusion)YOLO26s0.57620.43920.11980.0895
LAOVD-S (Ours)Visible + Infrared (NMS late fusion)YOLO26s0.54790.41670.12830.0974

We also report parameter counts, computational cost, and inference speed in Table R2. The dual-backbone design adds parameters and computation. Both LAOVD variants achieve real-time inference, with 92.76 FPS for LAOVD-S and 83.98 FPS for LAOVD-M.

Table R2. Model complexity and inference speed

MethodParams (M)GFLOPsInference Time (ms)FPS
LAOVD-S (Ours)22.2456.710.7892.76
LAOVD-M (Ours)48.60191.911.90783.98
DOSOD-s11.6230.74.16240.37
DOSOD-m26.7182.95.05198.05
YOLO-World v1-s13.6772.45.81172.06
YOLO-World v1-m29.63133.56.76147.85
YOLO-World v2-s13.0451.95.27189.71
YOLO-World v2-m28.93112.76.20161.20

(2) Effectiveness of SGAGF and Component Ablations

The existing baselines use single-modality images. Our higher novel-class scores demonstrate the value of dual-modal input for open-vocabulary recognition. We agree that combining the two modalities already accounts for part of the gain over the single-modality baselines. Within the same dual-modal architecture, SGAGF further improves novel-class mAP50 from 0.1686 with element-wise addition to 0.1890, a gain of 2.04 percentage points. This comparison shows that the choice of fusion strategy can further improve novel-class recognition with dual-modal input.

We also ablated each SGAGF component in LAOVD-M. The full module achieves the highest novel-class mAP50 across these settings, and removing any of the three gates reduces this score. These results show that each gate contributes to novel-class detection in the YOLO26-based model.

Table R3. Component ablation of SGAGF in LAOVD-M

ConfigurationBase mAP50Base mAP95Novel mAP50Novel mAP95
Without the infrared structural gate (SG)0.34210.23240.07180.0515
Without the visible detail gate (DG)0.50060.38590.12710.0989
Without the joint reliability gate (JG)0.48390.38220.11690.0954
Full SGAGF0.56520.44950.18900.1450

I appreciate the authors' response. However, it does not resolve my concerns: 1)The method itself appears incremental. 2)According to Table 2 in the authors' response, LAOVD-S achieves 92.76 FPS and LAOVD-M achieves 83.98 FPS, which show no obvious advantage over DOSOD, an ICRA 2025 work. Specifically, LAOVD-S compared with DOSOD-m suffers a much lower accuracy while being more than 2x slower; LAOVD-S compared with DOSOD-M gains only a marginal improvement in accuracy while again being more than 2x slower. Where exactly lies the significance of these improvements?

Response to Question 2

DOSOD uses a single visual modality, whereas LAOVD is a dual-modality open-vocabulary detector. Fusing two visual modalities adds computational overhead and contributes to LAOVD’s lower inference throughput. Although LAOVD-S achieves lower AP than DOSOD-S on base categories, it achieves substantially higher AP on novel categories. Generalization to novel categories is a central objective of open-vocabulary detection, and this gain is the main benefit demonstrated by LAOVD-S.

Reviewer_ayrAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces UAVOVD, a low-altitude RGB–infrared benchmark for open-vocabulary oriented detection with 25,292 registered image pairs, 88 categories, and a 68/20 base–novel split. It also proposes LAOVD, a dual-stream YOLO26 detector that classifies regions against CLIP text embeddings and fuses modalities through an infrared-dominant gated module, SGAGF. LAOVD reaches 0.189 novel-class mAP50 on UAVOVD, against 0.156 for the best single-modality YOLO-World or DOSOD baseline, and is further evaluated zero-shot on ATR-UMOD and DroneVehicle.

Critical strengths

UAVOVD combines registered RGB–infrared pairs, oriented boxes, day and night capture, and a fine-grained 88-class taxonomy, a combination no existing UAV dataset offers, so it could become a useful resource if released with a sound protocol. Pairing multimodal fusion with open-vocabulary oriented detection is a meaningful setting for low-altitude perception.

Critical weaknesses

The experiments do not support SGAGF as the source of the gains. In Table 4, simple element-wise addition already reaches 0.169 novel mAP50, above every single-modality baseline, and beats SGAGF on base classes; SGAGF adds only 2.0 novel mAP50, is selected on the test set, and the baselines also differ in backbone (YOLOv8 vs. YOLO26) and input modality, so the improvement cannot be credited to the proposed module.

The benchmark protocol is not reliable enough for a dataset contribution. Frames are sampled at 1 fps from videos with no stated sequence- or scene-level split, most novel classes are fine-grained subtypes of base classes whose labeling during training is unspecified, several novel classes have only a few dozen instances in total, and the data are not available for review.

Questions for authors

Please see the Critical Weaknesses.

1 comment
AuthorsOpenReview ↗

SGAGF Effectiveness and Benchmark Evaluation Protocol

(1)

Effectiveness of SGAGF. The comparisons in Table 2 show that the multimodal architecture outperforms the single-modality architectures. Table 4 compares different fusion strategies within the same multimodal architecture. SGAGF further improves novel-class mAP50 to 0.1890 (+2.04 percentage points), while base-class mAP50 decreases from 0.5823 to 0.5652 (−1.71 percentage points). Given this trade-off, we consider SGAGF superior to the other fusion strategies.

Differences in input modality. Existing open-vocabulary oriented object detection methods generally support only single-modality input, whereas our method uses dual-modal images for open-vocabulary object detection. As shown in the table below, the multimodal model outperforms the single-modality baselines, which is an advantage of our method.

Differences in backbone. As shown in the last row of the table, we replaced the LAOVD-M backbone with YOLOv8m and retrained the model. With the YOLOv8m backbone, our method still outperforms the other methods.

MethodInput ModalityBackboneBase mAP50Base mAP95Novel mAP50Novel mAP95
YOLO-World v1-sInfraredYOLOv8s0.43540.31920.11570.0841
YOLO-World v2-sInfraredYOLOv8s0.43690.32530.10200.0773
DOSOD-sInfraredYOLOv8s0.42400.30660.10250.0740
YOLO-World v1-mInfraredYOLOv8m0.48340.36450.12040.0905
YOLO-World v2-mInfraredYOLOv8m0.41950.30930.12850.0893
DOSOD-mInfraredYOLOv8m0.42470.30800.11190.0824
YOLO-World v1-sVisibleYOLOv8s0.53070.41160.14630.1134
YOLO-World v2-sVisibleYOLOv8s0.50020.38780.14200.1105
DOSOD-sVisibleYOLOv8s0.51770.39860.13660.1013
YOLO-World v1-mVisibleYOLOv8m0.55600.43340.15600.1240
YOLO-World v2-mVisibleYOLOv8m0.52910.41030.15560.1215
DOSOD-mVisibleYOLOv8m0.55710.44330.14660.1166
LAOVD-S (Ours)Visible + InfraredYOLO26s0.47680.36780.18610.1359
LAOVD-M (Ours)Visible + InfraredYOLO26m0.56520.44950.18900.1450
LAOVD-M (Ours)Visible + InfraredYOLOv8m0.58690.47370.16910.1347

(2)

We first annotated the entire dataset and then divided the 88 categories into 68 base classes and 20 novel classes. We assigned every visible–infrared image pair containing at least one novel-class instance to the test set and all remaining pairs to the training set. Accordingly, no image pair is assigned to both sets, the base and novel classes have distinct category definitions and labels, and the training images contain no novel-class instances. We retained some semantically related base and novel classes to assess whether the model can distinguish fine-grained categories that share appearance or structural features. We also retained a small number of low-frequency novel classes to cover the long tail of categories found in real-world aerial scenes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.