acceptodds
Withdrawn from ICLR 2027

AerialDojo-200K: A Large-Scale Benchmark Suite for Open-World Aerial Object-Goal Search

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-world aerial object-goal search is a foundational yet challenging task, requiring aerial agents to autonomously explore large-scale, unstructured three-dimensional environments and reach target objects specified by semantic descriptions or reference images, rather than following route-specific instructions. However, research in this task remains at a nascent stage and relies on small, environment-specific benchmarks with heterogeneous action spaces and data formats. These limitations hinder large-scale training and cross-benchmark evaluation, constraining the scalability and generalizability of aerial agents. To address this problem, we propose AerialDojo-200K, a large-scale benchmark suite for open-world aerial object-goal search, with (times) as many scenes and as many task instances as the largest existing benchmark for this task. Specifically, we construct 42 simulation scenes spanning four scene families and 21 scene types, including 18 urban, 12 natural, six infrastructure, and six disaster scenes. To ensure data quality, 12 annotators spent two months manually annotating 109 landmarks, 2099 target objects, and 2099 object anchors across these scenes. We further construct 205,732 task instances, comprising over 100K semantic-goal and over 100K image-goal instances across Base, Standard, and Long-Horizon settings. Each task instance includes a collision-free reference trajectory and corresponding multi-view video recordings. We also develop a unified evaluation framework with a scene partition comprising 21 in-distribution scenes and 21 out-of-distribution scenes. Finally, our evaluation of five open-source and four closed-source multimodal large language models reveals that there is still a long way to go toward achieving general-purpose aerial agents.

Primary area:
datasets and benchmarks
Keywords:
Aerial Agent, Large-Scale, Object-Goal Search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_FSwyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. A large-scale dataset is proposed for object-goal UAV search.
  2. Several LLMs are evaluated on the proposed dataset.
Critical strengths
  1. The scale of the dataset is much more bigger compared with existin datasets.
  2. More scenarios are used when consturcting such a datast.
Critical weaknesses
  1. The paper proposes a large-scale benchmark for object-goal search, but the value of its scale is not fully demonstrated. Compared with the existing benchmarks listed in Table 1, what specific benefits does the increased scale bring? The authors should provide further analysis to justify the necessity and significance of such a large-scale dataset.

  2. The quality of Figure 2 needs to be improved. If AI-generated content was used, the authors should carefully verify its accuracy and consistency, as AI-generated images may contain unrealistic or unreasonable elements.

  3. The evaluation metrics, including SR, OSR, DTS, SPL, and CR, are relatively standard and do not sufficiently reflect the unique characteristics of the proposed benchmark or its task design. Beyond the increase in dataset scale, what distinguishes this benchmark from existing ones, and how can its added value be quantitatively evaluated? The authors should consider introducing more task-specific metrics or analyses to better demonstrate its contributions.

Questions for authors
  1. What specific benefits does the increased scale bring?
  2. What distinguishes this benchmark from existing ones, and how can its added value be quantitatively evaluated?
Reviewer_G1LyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AerialDojo-200K, a large-scale benchmark for aerial object-goal search (AerialOGS). The authors place manually annotated target objects in 42 simulation scenes, and they build about 200K tasks in which the goal is given by text or by an image. Each task includes a reference trajectory and a video. The authors evaluate several off-the-shelf MLLMs, and the best success rate is only 3.20% even in the easiest setting.

Critical strengths

The authors build a large-scale dataset with 42 scenes and about 200K tasks. This is much larger than existing AerialOGS benchmarks. The dataset has the following good points, and it can be useful as a common foundation for the community: (1) the diversity of environments, which includes not only urban and natural scenes but also infrastructure and disaster scenes; (2) the definition of success based on anchors placed outside the objects, so that the goal is physically reachable.

Critical weaknesses

This paper has some value as a resource that expands the number of scenes, the number of tasks, and the environment coverage for aerial object search. However, as a benchmark/dataset paper, it has critical problems.

First, the paper does not show any new findings that can be obtained only with this benchmark. The reported finding (the low zero-shot performance of off-the-shelf MLLMs) can also be obtained with existing benchmarks. In addition, OOD generalization and adaptation with a small amount of data are defined as evaluation protocols, but they are not used in the experiments.

Second, the authors claim that the small scale of existing benchmarks is a bottleneck, but they do not show any findings obtained by increasing the scale. The experiments include only zero-shot evaluation, and most of the training data is not used.

Third, the authors claim high-quality data based on manual annotation, but there is no quality assurance. Regarding the quality, the paper only describes that 12 people worked for two months. It does not report inter-annotator agreement or error rates.

Fourth, human performance is not reported. Therefore, it is not verified whether the tasks are properly designed to be solvable, and we cannot judge whether the 3.20% success rate of MLLMs shows the difficulty of the benchmark or a flaw in its design.

Questions for authors

As written above, this paper has several critical problems as a benchmark/dataset paper. Please address the issues listed in the Weaknesses section.

Reviewer_hT82OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AerialDojo-200K, a large-scale benchmark for open-world aerial object-goal search (AerialOGS). Built on Unreal Engine and ProjectAirSim, it contains 42 scenes with 109 manually annotated landmarks and 2,099 target objects together with their anchors. From these, the authors generate 205,732 task instances with semantic or image goals, grouped into Base, Standard, and Long-Horizon settings. Each task includes a Hybrid A* reference trajectory and four-view RGB-D video recordings. The paper also provides an ID/OOD scene partition and a unified closed-loop evaluation framework, and evaluates nine open-source and closed-source MLLMs.

Critical strengths

Strengths  The 42 scenes cover four families (urban, natural, infrastructure, and disaster), giving broad scene coverage at a large scale.  The manual annotation pipeline is clearly described and well designed.  The dataset provides diverse data modalities and supports both training and evaluation.

Critical weaknesses

Weaknesses  The data distribution is highly imbalanced, and the OOD split is relatively weak.  The paper motivates the benchmark by arguing that existing benchmarks are too small to support large-scale training, and it provides a 113K-instance training split. However, no model is trained on this training split.  The evaluation does not use multi-view inputs, and the prompt does not specify action step sizes. As a result, the evaluation setup may underestimate model capabilities.  The experimental analysis is shallow and offers limited insight to the community.  The writing is somewhat disorganized and needs further polishing.

Questions for authors

Questions  How do the authors validate the value of the proposed training split?  How is annotation quality verified? Have the authors measured the ambiguity rate of the goal descriptions or inter-annotator agreement?  Could the authors provide an analysis of failure modes?

Reviewer_a1DbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AerialDojo-200K is an aerial object-goal search benchmark with 42 scenes and 205,732 goal-conditioned instances, corresponding to 102,866 navigation tasks with semantic and image variants. It provides manual annotations, Hybrid A* reference trajectories, RGB-D recordings, three difficulty settings, and ID/OOD splits with optional adaptation. Nine multimodal large language models are evaluated.

Critical strengths
  • Broad environmental coverage. The 42 scenes span 21 types across urban, natural, infrastructure, and disaster families, making the suite a useful testbed for studying aerial search across varied environments.
  • Unified, well-specified evaluation protocol. A shared action space, collision-checked reference paths, fixed action budgets, and five complementary metrics make methods directly comparable, addressing the heterogeneous setups that motivate the paper.
Critical weaknesses
  • The dataset’s training utility remains unvalidated. The paper emphasizes the need for large-scale training data but does not show that using AerialDojo improves an agent. A before-and-after training comparison on held-out ID and OOD tasks would substantiate its learning and transfer value.
  • Benchmark difficulty is insufficiently calibrated. Only general-purpose MLLMs are evaluated, with near-floor success rates. Constructing a task-oriented search agent or providing a human baseline on a representative subset would help distinguish genuine task difficulty from limitations of the evaluated controllers and calibrate attainable performance.
  • The analysis offers limited new understanding of search failures. The results describe performance differences but do not explain the mechanisms behind exploration, recognition, or stopping failures. A targeted failure analysis would clarify what limits current agents and provide insights that guide the development of better search methods.
Questions for authors

Please address the questions and clarification requests in the Weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.