- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DroneBench, a benchmark for predicting a one-step drone command from an aerial image and a result-oriented task. It contains 1,350 examples across four scenarios and two viewpoints, with commands decomposed into speed, aircraft-centered direction, and action. Experiments on 15 VLMs identify direction as the weakest component, which is also verified in self-collected real-world data.
Critical strengths
-
The problem is practically important for aerial active perception. Translating high-level task goals and visual observations into aircraft-centered commands is an important step toward bridging the gap between high-level planning and low-level execution. By examining speed, direction, and action separately, the benchmark offers a useful diagnostic perspective on this interface, particularly whether VLMs can select movement directions appropriate to the observed viewpoint.
-
The paper proposes a component-wise evaluation exposing failures hidden by aggregate command scores. Separating speed, direction, and action provides a useful diagnostic perspective. Direction is the weakest component for 12 of 15 models, and the analysis identifies cases where direction alone prevents an otherwise correct command. Reporting both graded scores and full-command exact match makes the practical significance of component errors clearer.
Critical weaknesses
-
Limited closed-loop validation Although translating high-level goals into drone commands is practically valuable, DroneBench evaluates structured one-step predictions within a fixed command space without closed-loop execution. The real-flight extension uses onboard images but does not execute the predicted commands or assess their consequences. While this is consistent with the benchmark’s diagnostic scope, it leaves unclear whether higher benchmark scores translate into better task completion during actual flight, limiting the evidence for its practical contribution.
-
Insufficient validation of single-reference commands Section 3.4 states that annotators select a “single best command” and do not record a set of acceptable alternatives. However, the paper does not establish whether the selected command is uniquely preferable to other feasible choices for the same image and goal. Although alternatives can receive partial credit, an equally valid command may still be penalized for differing from the reference. This makes it difficult to distinguish genuine model errors from disagreements with annotation preferences and weakens the interpretation of the reported command-prediction failures.
-
Insufficient analysis of evaluator bias The benchmark relies on GPT-4o to assign component-wise scores, but the paper does not report agreement with independent human judgments of model predictions or sensitivity to alternative evaluators. Reliance on a single judge may introduce evaluator-specific biases, particularly when assessing the usefulness of non-exact commands.
Questions for authors
-
How do benchmark scores relate to actual task success in closed-loop drone operation? Have the authors executed predicted commands in simulation or physical flights to assess whether higher scores correspond to better task completion, beyond offline prediction on onboard images?
-
How do the authors establish that the selected reference command is preferable to other feasible commands? What annotation or adjudication procedure distinguishes equally valid alternatives from genuinely inferior choices, and how are equally valid commands scored?
-
How robust are the graded scores and model comparisons to the choice of evaluator? Have GPT-4o’s judgments, particularly partial-credit decisions, been compared with independent human assessments or alternative judges to examine potential evaluator bias?
-
What are the sources of the 1,350 images in the main benchmark, and how to verify their quality?