acceptodds
Withdrawn from ICLR 2027

Can Vision-Language Models Control Drones? Diagnosing Viewpoint-Aware Spatial Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) are increasingly viewed as promising interfaces for aerial systems, where they could translate human goals and visual observations into drone commands. However, it remains unclear whether they can reliably map image-grounded spatial relations to directions defined in the aircraft movement frame. To address this gap, we introduce DroneBench, a diagnostic benchmark for one-step command prediction from an aerial image and a result-oriented task goal. DroneBench contains 1,350 paired images and tasks across four operational scenarios and two viewpoints, with each command decomposed into speed, aircraft-centered direction, and action for component-wise evaluation. Experiments with 15 VLMs show that models generally recognize task-relevant targets and choose plausible operational actions, yet remain substantially less reliable at direction grounding, especially for top-down views. We further evaluate five models on 50 matched forward-facing/top-down pairs collected during physical drone flights and observe the same failure pattern on onboard imagery. These results identify viewpoint-aware coordinate conversion as a central obstacle to reliable aerial command selection and establish DroneBench as a practical benchmark for advancing spatial reasoning in aerial VLMs.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, drone/UAV control, benchmark, spatial grounding, viewpoint awareness, aerial perception

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EAQdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DroneBench, a benchmark for predicting a one-step drone command from an aerial image and a result-oriented task. It contains 1,350 examples across four scenarios and two viewpoints, with commands decomposed into speed, aircraft-centered direction, and action. Experiments on 15 VLMs identify direction as the weakest component, which is also verified in self-collected real-world data.

Critical strengths
  1. The problem is practically important for aerial active perception. Translating high-level task goals and visual observations into aircraft-centered commands is an important step toward bridging the gap between high-level planning and low-level execution. By examining speed, direction, and action separately, the benchmark offers a useful diagnostic perspective on this interface, particularly whether VLMs can select movement directions appropriate to the observed viewpoint.

  2. The paper proposes a component-wise evaluation exposing failures hidden by aggregate command scores. Separating speed, direction, and action provides a useful diagnostic perspective. Direction is the weakest component for 12 of 15 models, and the analysis identifies cases where direction alone prevents an otherwise correct command. Reporting both graded scores and full-command exact match makes the practical significance of component errors clearer.

Critical weaknesses
  1. Limited closed-loop validation Although translating high-level goals into drone commands is practically valuable, DroneBench evaluates structured one-step predictions within a fixed command space without closed-loop execution. The real-flight extension uses onboard images but does not execute the predicted commands or assess their consequences. While this is consistent with the benchmark’s diagnostic scope, it leaves unclear whether higher benchmark scores translate into better task completion during actual flight, limiting the evidence for its practical contribution.

  2. Insufficient validation of single-reference commands Section 3.4 states that annotators select a “single best command” and do not record a set of acceptable alternatives. However, the paper does not establish whether the selected command is uniquely preferable to other feasible choices for the same image and goal. Although alternatives can receive partial credit, an equally valid command may still be penalized for differing from the reference. This makes it difficult to distinguish genuine model errors from disagreements with annotation preferences and weakens the interpretation of the reported command-prediction failures.

  3. Insufficient analysis of evaluator bias The benchmark relies on GPT-4o to assign component-wise scores, but the paper does not report agreement with independent human judgments of model predictions or sensitivity to alternative evaluators. Reliance on a single judge may introduce evaluator-specific biases, particularly when assessing the usefulness of non-exact commands.

Questions for authors
  1. How do benchmark scores relate to actual task success in closed-loop drone operation? Have the authors executed predicted commands in simulation or physical flights to assess whether higher scores correspond to better task completion, beyond offline prediction on onboard images?

  2. How do the authors establish that the selected reference command is preferable to other feasible commands? What annotation or adjudication procedure distinguishes equally valid alternatives from genuinely inferior choices, and how are equally valid commands scored?

  3. How robust are the graded scores and model comparisons to the choice of evaluator? Have GPT-4o’s judgments, particularly partial-credit decisions, been compared with independent human assessments or alternative judges to examine potential evaluator bias?

  4. What are the sources of the 1,350 images in the main benchmark, and how to verify their quality?

Reviewer_LtM2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DroneBench, a diagnostic benchmark that tests whether VLMs can map an aerial image and a result-oriented task goal to a one-step drone command.

Critical strengths
  1. The paper presents a substantial evaluation, with 1,350 curated examples, 15 VLMs, and a real-flight extension of 50 matched image pairs, and it yields an insightful finding: VLMs can usually choose plausible speeds and actions but struggle to ground movement direction in the aircraft frame, especially from top-down views.

  2. The paper is well written and easy to follow.

Critical weaknesses
  1. The setup of the direction labels is problematic. The viewpoint rules are designed and defined by the authors themselves. For general-purpose MLLMs, could the low direction scores simply mean that the models do not know these author-defined rules, rather than that they fail at spatial grounding?

  2. I want the authors to include comparisons with VLMs specifically designed or trained for spatial reasoning not just the general ones.

  3. The validation of the conclusions lacks multiple runs, making it difficult to assess the statistical significance of the reported findings.

  4. Finally, the formatting of the references has notable issues and should be carefully revised.

Questions for authors
  1. Could you please provide the comparisons with the VLMs designed for spatial reasoning?

  2. How can the authors demonstrate that the poor performance of MLLMs stems from a genuine lack of spatial grounding ability, rather than from not having learned to respond in the expected format or label convention?

Reviewer_DYtvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark to evaluate LLMs on action choice for UAVs. Three different tasks are designed for speed, direction, and specific actions. Typical LLMs are evaluated and compared.

Critical strengths
  1. A new QA benchmark for UAV action choice, which contains 1350 images and corresponding questions and answers.
  2. 15 LLMs are evaluated on three tasks.
Critical weaknesses
  1. The paper should better justify whether speed, direction, and action can be reliably determined from a single image and a task description. Multiple valid commands may exist for the same task, raising concerns about the use of a single reference command.
  2. The benchmark contains only 1,350 images, which is relatively small for a comprehensive UAV benchmark. The authors should clarify the proportion of synthetic and real-world images and consider expanding the dataset with more diverse real-world scenarios.
  3. Despite the title, the experiments only evaluate single-frame command prediction rather than actual UAV control. Closed-loop experiments involving real embodied tasks are needed to assess control performance and error accumulation over long horizons.
Questions for authors
  1. The paper should clarify the relationship between the image coordinate system and the UAV body frame. Since the camera can rotate independently of the UAV heading, how can the model determine the correct movement direction without knowing the UAV orientation and camera-to-body transformation? Could this ambiguity affect the validity of the direction labels and evaluation results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.