- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ActiveFly-Bench, a benchmark for language-guided aerial embodied perception that bridges high-level question answering and low-level UAV control. The benchmark comprises three hierarchical tasks, namely Aerial Embodied Question Answering (EQA), Observation Behavior Planning (OBP), and Fine-grained Language-Guided UAV Control (FLUC). It includes approximately 10K trajectories and 3.6K embodied QA samples collected from both simulated and real-world environments. Furthermore, the authors develop ActiveFly, a closed-loop UAV system that integrates VLM-based reasoning with VLA-based action execution. Experiments involving OpenVLA, Pi-0.5, and several VLMs, together with real-world UAV deployment, demonstrate the feasibility of the proposed framework while highlighting substantial limitations in the planning and control capabilities of existing agents.
Critical strengths
The paper presents a meaningful benchmark that addresses an important gap between high-level embodied question answering and low-level UAV action execution. By introducing a hierarchical formulation, the proposed benchmark provides a unified framework for evaluating reasoning, observation planning, and fine-grained control in active perception tasks.
Critical weaknesses
-
Potential misalignment between evaluation metrics and task objectives. The end-to-end success metric requires correct intermediate observation planning and oracle trajectory success, in addition to correct question answering. However, different observation strategies may provide equally valid visual evidence for completing the same task. The substantial number of escape cases reported in the paper further suggests that the current metric may underestimate successful active perception and may not fully reflect task-level completion.
-
Insufficient validation of the key design choices and generalization capabilities. The paper lacks ablation studies that isolate the respective contributions of OBP and 5-DoF control. Furthermore, OBP is primarily evaluated through multiple-choice questions rather than open-ended plan generation, while the random trajectory-level split adopted in simulation may not adequately assess generalization to unseen scenes. These limitations weaken the empirical evidence supporting the central claims of the proposed benchmark.
Questions for authors
- Could the authors provide a task-oriented success metric that directly evaluates whether the agent acquires sufficient visual evidence and correctly answers the question, without requiring agreement with a single reference observation plan or target pose? How would the adoption of such a metric affect the main conclusions?
- Given that OBP is evaluated as a multiple-choice task, how are OBP instructions generated during real-world deployment? Could the authors provide an open-ended planning evaluation and conduct ablation studies comparing the complete framework with variants that exclude OBP or gimbal pitch control?
3 comments
Q2:
Given that OBP is evaluated as a multiple-choice task, how are OBP instructions generated during real-world deployment? Could the authors provide an open-ended planning evaluation and conduct ablation studies comparing the complete framework with variants that exclude OBP or gimbal pitch control?
Response:
Thanks for your suggestion. In real-world deployment, OBP follows the same multiple-choice formulation as in simulation. We additionally evaluate free-form OBP and ablations without OBP or gimbal pitch control on the real UAV system.
Baselines:
- Free-form OBP: We follow the same free-form planning setting described in our response to Q1, applied to the real UAV system.
- No OBP: We directly provide the embodied question, current observation, and UAV state to OpenVLA, which outputs actions without an intermediate observation plan. We fine-tune OpenVLA on the real-world dataset for this setting.
- No Gimbal Pitch: The VLM planner generates free-form instructions without gimbal pitch commands, following the same instruction format as the Free-form OBP baseline. These instructions are then provided to OpenVLA.
All baselines use GPT-5.6-Sol to answer the embodied questions based on the collected videos. We evaluate the baseline on Real-world Datasets.
Results:
| Ablations | EQA Accuracy (%) |
|---|---|
| Free-form OBP | 47.3 |
| No OBP | 22.4 |
| No Gimbal Pitch | 46.4 |
| Reference OBP (Ours) | 77.3 |
Free-form OBP and the variant without gimbal pitch control successfully complete some tasks, but their EQA accuracy is substantially lower than that of Reference OBP. The No OBP baseline answers most questions incorrectly. These results suggest that:
-
Directly mapping embodied questions to actions remains challenging. The low accuracy of No OBP, even after fine-tuning, suggests that the semantic gap between questions and executable actions makes this mapping difficult to learn.
-
Reference OBP provides useful human priors for active perception. Its higher accuracy compared with Free-form OBP supports the value of reference observation plans in helping the model learn effective observation behaviors.
Q3:
while the random trajectory-level split adopted in simulation may not adequately assess generalization to unseen scenes. These limitations weaken the empirical evidence supporting the central claims of the proposed benchmark.
Response:
Thank you for this comment. For the real-world dataset, we evaluate the model through online rollouts with perturbations to the UAV’s initial position and object placements. This evaluation is similar in spirit to perturbation-based settings in embodied benchmarks such as LIBERO, RoboTwin, and RoboDojo. Our current setting assesses the model’s robustness to changes in spatial configurations and visual observations within the evaluated environments. These evaluations demonstrate that the model can generalize to changes in spatial configurations and visual observations within the evaluated environments.
We appreciate your thoughtful feedback and valuable suggestions.
We are writing to follow up on our rebuttal, which includes additional experiments and analyses addressing the concerns you raised. We would be grateful if you could review these updates when your schedule permits.
If any questions or concerns remain, please let us know. We would be glad to clarify or elaborate as needed.
Thank you for your time and attention.
Q1:
Response:
Thanks for your comments! To evaluate whether an agent acquires sufficient visual evidence, we introduce an automated visual judge that assesses whether its observations contain the semantic cues needed to answer the EQA question. And conduct ablation studies on free-form planning instructions.
Visual judge design:
We use GPT-6-Sol with a tailored prompt to determine whether the input observations provide sufficient evidence to identify the answer. For a positive judgment, the judge must identify the supporting frames or timestamps and describe concrete visible facts. Otherwise, the observations are classified as not providing confirmed sufficient evidence.
We randomly sampled 100 judgments for human verification and obtained 93% agreement between the visual judge and human annotations, supporting the reliability of the judge on the audited sample.
Metrics:
Baselines:
No exploration: the VLM answerer directly answers the embodied question only rely on the initial frame.
Free-form OBP: the VLM planner directly generates observation instructions instead of selecting from predefined options.
Reference OBP: baselines using our reference observation plans.
Fairness of the free-form OBP comparison:
To preserve the VLM planner’s freedom to make planning decisions while helping to reduce the potential influence of instruction-distribution differences on VLA execution, we prompt the planner to generate instructions with wording and sentence structures similar to those of the reference instructions. We assess syntactic similarity using FastKASSIM [1] and style-distribution similarity using MMD [2] over StyleDistance [3] embeddings before reporting the EQA performance of the free-form OBP baselines.
Datasets:
simulated datasets.
Results:
The FastKASSIM scores suggest comparable syntactic similarity to the reference-split baseline. The MMD score for free-form instructions is closer to the score for reference splits. The results indicate that our free-form OBP preserves the planner’s freedom to make decisions while mitigating the influence of instruction-style differences on the downstream VLA model, helping its execution more faithfully reflect the upstream planner’s decisions.
All baselines use GPT-5.6 Sol as the EQA answerer, and exploration-based baselines use the same model as the OBP planner.
These results support the following interpretation:
[1] Chen et al. (2023). FastKASSIM: A Fast Tree Kernel-Based Syntactic Similarity Metric. EACL.
[2] Gretton et al. (2012). A Kernel Two-Sample Test]. JMLR, 13:723–773.
[3] Patel et al. (2025). StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples NAACL.