acceptodds
Under review as a conference paper at ICLR 2027

ActiveFly-Bench: Aligning Embodied Question Answering with Vision-Language-Action for Aerial Embodied Perception

Abstract

We introduce ActiveFly-Bench, the first benchmark to bridge cyberspace reasoning and physical-world interaction for UAV embodied perception. The benchmark decomposes active perception into three hierarchical tasks: Aerial Embodied Question Answering (Air-EQA), Observation Behavior Planning (OBP), and Fine-grained Language-guided UAV Control (FLUC), explicitly connecting high-level task understanding, behavior planning, and low-level control. The datasets are collected from both real-world and simulated outdoor environments for training and evaluation. We further develop ActiveFly, a closed-loop UAV agent that integrates visual-language reasoning with fine-grained control, and deploy it on a physical UAV platform. Experiments with representative VLMs and VLA models show that current UAV agents still struggle with behavior planning, viewpoint adjustment, and robust task completion in active perception. These results establish ActiveFly-Bench as a new testbed for embodied aerial intelligence.

Primary area:
datasets and benchmarks
Keywords:
UAV, embodied perception, embodied question answering, vision-language-action

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hn1aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ActiveFly-Bench, a benchmark for language-guided aerial embodied perception that bridges high-level question answering and low-level UAV control. The benchmark comprises three hierarchical tasks, namely Aerial Embodied Question Answering (EQA), Observation Behavior Planning (OBP), and Fine-grained Language-Guided UAV Control (FLUC). It includes approximately 10K trajectories and 3.6K embodied QA samples collected from both simulated and real-world environments. Furthermore, the authors develop ActiveFly, a closed-loop UAV system that integrates VLM-based reasoning with VLA-based action execution. Experiments involving OpenVLA, Pi-0.5, and several VLMs, together with real-world UAV deployment, demonstrate the feasibility of the proposed framework while highlighting substantial limitations in the planning and control capabilities of existing agents.

Critical strengths

The paper presents a meaningful benchmark that addresses an important gap between high-level embodied question answering and low-level UAV action execution. By introducing a hierarchical formulation, the proposed benchmark provides a unified framework for evaluating reasoning, observation planning, and fine-grained control in active perception tasks.

Critical weaknesses
  1. Potential misalignment between evaluation metrics and task objectives. The end-to-end success metric requires correct intermediate observation planning and oracle trajectory success, in addition to correct question answering. However, different observation strategies may provide equally valid visual evidence for completing the same task. The substantial number of escape cases reported in the paper further suggests that the current metric may underestimate successful active perception and may not fully reflect task-level completion.

  2. Insufficient validation of the key design choices and generalization capabilities. The paper lacks ablation studies that isolate the respective contributions of OBP and 5-DoF control. Furthermore, OBP is primarily evaluated through multiple-choice questions rather than open-ended plan generation, while the random trajectory-level split adopted in simulation may not adequately assess generalization to unseen scenes. These limitations weaken the empirical evidence supporting the central claims of the proposed benchmark.

Questions for authors
  1. Could the authors provide a task-oriented success metric that directly evaluates whether the agent acquires sufficient visual evidence and correctly answers the question, without requiring agreement with a single reference observation plan or target pose? How would the adoption of such a metric affect the main conclusions?
  2. Given that OBP is evaluated as a multiple-choice task, how are OBP instructions generated during real-world deployment? Could the authors provide an open-ended planning evaluation and conduct ablation studies comparing the complete framework with variants that exclude OBP or gimbal pitch control?
3 comments

Q1:

Could the authors provide a task-oriented success metric that directly evaluates whether the agent acquires sufficient visual evidence and correctly answers the question, without requiring agreement with a single reference observation plan or target pose? How would the adoption of such a metric affect the main conclusions?

Response:

Thanks for your comments! To evaluate whether an agent acquires sufficient visual evidence, we introduce an automated visual judge that assesses whether its observations contain the semantic cues needed to answer the EQA question. And conduct ablation studies on free-form planning instructions.

Visual judge design:

We use GPT-6-Sol with a tailored prompt to determine whether the input observations provide sufficient evidence to identify the answer. For a positive judgment, the judge must identify the supporting frames or timestamps and describe concrete visible facts. Otherwise, the observations are classified as not providing confirmed sufficient evidence.

We randomly sampled 100 judgments for human verification and obtained 93% agreement between the visual judge and human annotations, supporting the reliability of the judge on the audited sample.

Metrics:

  1. EQA accuracy. ; 2. True Positive (TP) proportion among correct answers: the proportion of correctly answered EQA questions for which the judge also identifies sufficient visual evidence, calculated as TP/(TP+FP)

Baselines:

  • No exploration: the VLM answerer directly answers the embodied question only rely on the initial frame.

  • Free-form OBP: the VLM planner directly generates observation instructions instead of selecting from predefined options.

  • Reference OBP: baselines using our reference observation plans.

Fairness of the free-form OBP comparison:

To preserve the VLM planner’s freedom to make planning decisions while helping to reduce the potential influence of instruction-distribution differences on VLA execution, we prompt the planner to generate instructions with wording and sentence structures similar to those of the reference instructions. We assess syntactic similarity using FastKASSIM [1] and style-distribution similarity using MMD [2] over StyleDistance [3] embeddings before reporting the EQA performance of the free-form OBP baselines.

Datasets:

simulated datasets.

Results:

  1. Instruction similarity
ComparisonFastKASSIMMMD + StyleDistance
Free-form OBP vs. reference instruction split A0.590.09
Reference instruction split A vs. split B0.570.02

The FastKASSIM scores suggest comparable syntactic similarity to the reference-split baseline. The MMD score for free-form instructions is closer to the score for reference splits. The results indicate that our free-form OBP preserves the planner’s freedom to make decisions while mitigating the influence of instruction-style differences on the downstream VLA model, helping its execution more faithfully reflect the upstream planner’s decisions.

  1. EQA performance

All baselines use GPT-5.6 Sol as the EQA answerer, and exploration-based baselines use the same model as the OBP planner.

BaselineEQA Accuracy (%)Improvement (pp)TP among Correct Answers (%)
No exploration50.4——
OpenVLA w/ Free-form OBP67.817.475.54
OpenPi w/ Free-form OBP69.419.080.92
OpenVLA w/ OBP (Ours)77.226.878.77
OpenPi w/ OBP (Ours)81.731.387.22

These results support the following interpretation:

  • If the reference-OBP baselines achieve comparable TP proportions, this would indicate that correct answers under both planning approaches are generally accompanied by sufficient visual evidence. This comparison concerns the availability of evidence in the answerer’s inputs.
  • Our reference-OBP baselines achieve higher EQA accuracy than their free-form counterparts, which is consistent with more effective observation guidance. This aligns with our benchmark’s design goal: to evaluate effective evidence acquisition and provide efficient, human-informed observation plans through which models can learn useful observation priors.

[1] Chen et al. (2023). FastKASSIM: A Fast Tree Kernel-Based Syntactic Similarity Metric. EACL.

[2] Gretton et al. (2012). A Kernel Two-Sample Test]. JMLR, 13:723–773.

[3] Patel et al. (2025). StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples NAACL.

Q2:

Given that OBP is evaluated as a multiple-choice task, how are OBP instructions generated during real-world deployment? Could the authors provide an open-ended planning evaluation and conduct ablation studies comparing the complete framework with variants that exclude OBP or gimbal pitch control?

Response:

Thanks for your suggestion. In real-world deployment, OBP follows the same multiple-choice formulation as in simulation. We additionally evaluate free-form OBP and ablations without OBP or gimbal pitch control on the real UAV system.

Baselines:

  • Free-form OBP: We follow the same free-form planning setting described in our response to Q1, applied to the real UAV system.
  • No OBP: We directly provide the embodied question, current observation, and UAV state to OpenVLA, which outputs actions without an intermediate observation plan. We fine-tune OpenVLA on the real-world dataset for this setting.
  • No Gimbal Pitch: The VLM planner generates free-form instructions without gimbal pitch commands, following the same instruction format as the Free-form OBP baseline. These instructions are then provided to OpenVLA.

All baselines use GPT-5.6-Sol to answer the embodied questions based on the collected videos. We evaluate the baseline on Real-world Datasets.

Results:

AblationsEQA Accuracy (%)
Free-form OBP47.3
No OBP22.4
No Gimbal Pitch46.4
Reference OBP (Ours)77.3

Free-form OBP and the variant without gimbal pitch control successfully complete some tasks, but their EQA accuracy is substantially lower than that of Reference OBP. The No OBP baseline answers most questions incorrectly. These results suggest that:

  1. Directly mapping embodied questions to actions remains challenging. The low accuracy of No OBP, even after fine-tuning, suggests that the semantic gap between questions and executable actions makes this mapping difficult to learn.

  2. Reference OBP provides useful human priors for active perception. Its higher accuracy compared with Free-form OBP supports the value of reference observation plans in helping the model learn effective observation behaviors.


Q3:

while the random trajectory-level split adopted in simulation may not adequately assess generalization to unseen scenes. These limitations weaken the empirical evidence supporting the central claims of the proposed benchmark.

Response:

Thank you for this comment. For the real-world dataset, we evaluate the model through online rollouts with perturbations to the UAV’s initial position and object placements. This evaluation is similar in spirit to perturbation-based settings in embodied benchmarks such as LIBERO, RoboTwin, and RoboDojo. Our current setting assesses the model’s robustness to changes in spatial configurations and visual observations within the evaluated environments. These evaluations demonstrate that the model can generalize to changes in spatial configurations and visual observations within the evaluated environments.

We appreciate your thoughtful feedback and valuable suggestions.

We are writing to follow up on our rebuttal, which includes additional experiments and analyses addressing the concerns you raised. We would be grateful if you could review these updates when your schedule permits.

If any questions or concerns remain, please let us know. We would be glad to clarify or elaborate as needed.

Thank you for your time and attention.

Reviewer_fCnHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ActiveFly-Bench for evaluating aerial embodied perception. It contains three related tasks: aerial embodied question answering, observation behavior planning, and fine-grained UAV control. The benchmark includes approximately 10K trajectories, 3.6K questions, and 1K planning pairs from simulated and real environments, with control over both UAV motion and camera pitch. The authors evaluate several VLM–VLA combinations, include human-assisted baselines, and conduct physical UAV experiments. The benchmark addresses a useful problem, although its evaluation criteria and data splits need further justification.

Critical strengths
  1. The benchmark covers a useful aspect of aerial perception. Answering questions about occluded, distant, or poorly oriented objects requires more than reaching a location. Including UAV movement and camera-pitch adjustment provides a practical setting for evaluating whether agents can obtain informative observations.

  2. The connected tasks support more detailed evaluation. Separating question answering, observation planning, and control allows researchers to examine intermediate performance rather than relying only on final answer accuracy. The human-assisted baselines and category-level results add useful diagnostic information.

  3. The data collection includes meaningful real-world effort. The benchmark combines expert demonstrations, manually checked annotations, and real flight data. Physical UAV evaluation further demonstrates that the tasks can be executed outside simulation, which strengthens the benchmark’s practical relevance.

Critical weaknesses
  1. The benchmark’s added coverage over existing datasets needs a more precise explanation. CityEQA already evaluates urban embodied question answering with active observation adjustment, UAV-Flow covers language-guided fine-grained flight control, and EXPRESS-Bench examines the relationship between exploration and answering. ActiveFly’s useful distinction appears to be the combination of aligned task annotations, camera-pitch control, and real UAV evaluation. A benchmark does not need a new algorithm, but it should clearly establish what researchers can measure with it that existing benchmarks cannot. Table 1 gives a broad comparison, but a closer comparison of observation requirements, annotation granularity, and evaluation protocols would better support this contribution. In particular, the claim of bridging reasoning and physical interaction is broader than the specific gap demonstrated here.

  2. The gap between control performance and answer accuracy needs closer investigation. In Table 2, replacing OpenVLA with Pi-0.5 under the same GPT backbone increases EP success from 12.0% to 46.3%, but answer accuracy only rises from 77.2% to 81.7%. This is potentially an interesting finding, yet its meaning remains unclear. An agent may obtain useful evidence from a viewpoint different from the reference, or answer correctly without obtaining the required evidence. The current “escape” analysis groups these possibilities together based on answer correctness and reference-pose success. Figure 4 illustrates the former, but does not establish how common it is. A representative audit separating evidence-supported answers from unsupported correct answers would help determine whether the gap reflects limitations of the control metric, shortcuts in the questions, or both. Without this distinction, it is difficult to know whether the benchmark’s main score and failure analysis accurately reflect active perception ability.

  3. The evidence that the benchmark measures active perception is incomplete. The authors make a useful effort to remove questions answerable from the initial view, including model screening and human review. Nevertheless, initial-frame accuracy remains around 50–54% in a four-choice setting. This does not by itself invalidate the questions, but it leaves unclear how often movement is necessary for a correct answer. Table 3 establishes that additional observations help; it does not establish that task-directed observation is necessary. Budget-matched fixed scanning or question-independent exploration baselines would help assess this. OBP also uses four candidate instructions, with distractors described as physically impossible or unhelpful. A text-only OBP baseline and an analysis of distractor difficulty are needed to check whether this task measures visually grounded planning rather than elimination of implausible options.

  4. Dataset quality is not sufficiently verified. The paper describes manual annotation, VLM screening, and human refinement, but provides limited quantitative evidence of the final annotation quality. Appendix A.3 requires unanimous human agreement, yet the number of independent reviewers, the disagreement rate, and the procedure for resolving disagreements are not reported. These details matter because several annotations require judgment: whether the initial view contains enough evidence, whether the collected observations support the answer, and whether an alternative OBP instruction is also valid. Errors in these labels would directly affect both model rankings and the reported failure categories. An independent audit of a representative sample should report the rates of ambiguous questions, incorrect answers, and invalid or non-unique planning labels, together with the corrections made. Describing the filtering procedure alone does not establish that the released benchmark is reliable.

Questions for authors
  1. Does the end-to-end evaluation use multiple-choice OBP selection or freely generated observation plans? How are alternative plans that reveal the required evidence treated, and are the reported “escape” cases checked for actual evidence acquisition rather than answer correctness alone?

  2. Annotation quality and validation. Was a sample of the final dataset independently checked by annotators who did not create the examples? If so, how many examples were checked, what were the disagreement and error rates for Air-EQA and OBP, and how were disputed labels resolved? In particular, did this audit verify both that the answer is supported by the collected observations and that the OBP question has only one valid option?

  3. Do Tables 2 and 6 use different subsets or averaging procedures? For GPT + OpenVLA, Table 6 implies an OBP accuracy of 561/747 = 75.1% and a FLUC oracle success rate of 103/747 = 13.8%, while Table 2 reports 72.5% and 15.5%. Clarifying the evaluation populations is necessary to interpret the benchmark results and failure breakdown.

4 comments

Thank you for the additional experiments. The random exploration, fixed scanning, and free-form planning results better support the value of question-guided exploration. The clarification of the benchmark’s contribution and the new visual-evidence assessment are also helpful.

However, my main concern about dataset quality remains. The 93% agreement on 100 examples validates the visual judge, not the benchmark annotations. Please clarify whether the final Air-EQA and OBP labels were independently checked, what errors were found, and how ambiguous questions or multiple valid planning options were handled.

The evidence assessment also does not yet explain the original “escape” cases. How many of these cases contain sufficient evidence despite failing the control criterion? This would help determine whether the benchmark incorrectly penalizes valid observation behaviors.

Finally, please clarify whether “Reference OBP” uses multiple-choice selection or the correct reference instruction directly, and resolve the discrepancy between Tables 2 and 6.

These additions are useful, but the concerns about data reliability and evaluation remain. I therefore maintain my weak reject rating.

Question 1:

main concern about dataset quality remains. The 93% agreement on 100 examples validates the visual judge, not the benchmark annotations. Please clarify whether the final Air-EQA and OBP labels were independently checked, what errors were found, and how ambiguous questions or multiple valid planning options were handled.

Response:

Thank you for clarifying this concern. Dataset quality was assessed through a separate annotation and human-review process.

For Air-EQA:

  1. Initial annotation. Four annotators, working independently, created a total of 1,004 distinct EQA questions.
  2. Initial-frame screening. As described in Section 4.2 and Appendix A.3, GPT-5.6-nano, Gemini-2.5-flash, and Qwen3-VL-flash independently answered each question using only the initial frame. We discarded questions answered correctly by all three models, leaving 850 questions.
  3. Independent human review. We divided these 850 questions into three subsets, each reviewed by a different reviewer who had not participated in the initial annotation. Using the task scene and the retained model reasoning, reviewers assessed whether each question required embodied actions to obtain or supplement the visual information needed for an accurate answer. Questions whose answers were already fully available in the initial frame, or that failed this criterion were discarded.
  4. Second human review. The reviewers exchanged their subsets for a second review using the same criterion. Across the two human-review rounds, we obtain 747 EQA questions.

For OBP:

  1. Trajectory and instruction annotation. Five annotators independently collected 1.2k trajectories and annotated the corresponding instructions during collection.
  2. Instruction–trajectory consistency review. The annotators exchanged their trajectories and instructions to check their consistency. Then, they selected trajectories containing sufficiently informative semantic targets to support EQA tasks, retaining 313 trajectories.
  3. EQA construction. These trajectories were then provided to the EQA annotators to construct the associated questions.
  4. OBP option construction. After the corresponding EQA questions were available, two additional annotators who had not participated in EQA annotation, constructed the OBP options using each trajectory’s reference instruction and associated EQA questions. The reference instruction served as the correct option. The three manually written distractors were required to either fail to acquire the information needed to answer the EQA questions or describe physically infeasible movements, such as moving through obstacles.
  5. Cross-review of the options. The two annotators exchanged their annotations for review. We discarded items that violated the distractor criteria or whose options revealed the EQA answer. This review reserve 291 distinct OBP trajectories with their associated options.

Question 2

How many of these cases contain sufficient evidence despite failing the control criterion? This would help determine whether the benchmark incorrectly penalizes valid observation behaviors.

Response

Thank you for this insightful comment. We use the same visual judge to assess whether escaped cases contain sufficient visual evidence. We also compare the mean full-rollout distance of evidence-supported escaped cases with that of OSR=1 cases.

BaselinesEscaped CasesEscape Rate (%)Escaped Cases with Sufficient Evidence(%)Mean Traj. Length of OSR=1 Cases (m)Mean Traj. Length of Evidence-Supported Escaped Cases (m)
GPT-5.6-Sol + OpenVLA43069.6329 (76.5%)10.8413.79 (+27.20%)
GPT-5.6-Sol + Pi33945.4269 (79.4%)20.8730.84 (+47.76%)
Gemini-2.5-Pro + OpenVLA42969.4316 (73.7%)13.2614.06 (+6.04%)
Gemini-2.5-Pro + Pi35647.7253 (71.1%)21.0431.29 (+48.75%)
Qwen3-VL-Max + OpenVLA41867.6288 (68.9%)11.3513.46 (+18.60%)
Qwen3-VL-Max + Pi16842.0133 (79.2%)27.9331.94 (+14.34%)

The results show that 68.9-79.2% escaped cases contain sufficient visual evidence despite failing the control criterion. At the same time, their mean rollout distances are 6.04–48.75% longer than those of OSR=1 cases.

We therefore report evidence sufficiency alongside OSR to distinguish successful evidence acquisition from reaching the reference control target. This is consistent with our benchmark’s intended role: to provide human-informed supervision for efficient active perception while recognizing that alternative observation behaviors can also obtain the information needed to answer the question.


Question 3

Please clarify whether “Reference OBP” uses multiple-choice selection or the correct reference instruction directly,

Response

"Reference OBP" uses instructions from multiple-choice option selected by the VLM planner. The instruction associated with the selected option is passed to the controller. Note that the instruction associated with the correct option is the human-annotated reference instruction described in Response 1. However, this instruction is passed to the controller only if the VLM planner selects the correct option.

Question4

The discrepancy between Table 2 and Table 6.

Response

Thank you for pointing this out. The discrepancy arises from different aggregation units. During dataset construction, we generate multiple EQA questions for a single trajectory. These questions share the same reference OBP instruction and trajectories.

In the simulation setting, identical instructions produce identical rollouts. We believe that repeatedly counting the OBP and OSR outcomes of the same trajectory may obscure the model’s actual performance on the OBP and FLUC tasks. Therefore, Table 2 reports OBP accuracy and FLUC OSR over unique OBP–trajectory pairs. In Table 6, the same OBP and FLUC outcomes are counted repeatedly for each associated EQA question, so that each evaluation case contains one EQA question, one OBP decision, and one FLUC rollout, yielding an EQA:OBP:FLUC ratio of 1:1:1.

We appreciate your thoughtful feedback and valuable suggestions.

We are writing to follow up on our rebuttal, which includes additional experiments and analyses addressing the concerns you raised. We would be grateful if you could review these updates when your schedule permits.

If any questions or concerns remain, please let us know. We would be glad to clarify or elaborate as needed.

Thank you for your time and attention.

Reviewer_RxrPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

ActiveFly-Bench connects aerial QA, observation behavior planning and language-guided UAV control. It constructs aligned Air-EQA, OBP, and FLUC data from 10k real and simulated trajectories. The agent proposed combines VLM-based planning and answering with VLA control and is evaluated in simulation and on a physical UAV.

Critical strengths
  • The benchmark provides aligned supervision for Air-EQA questions, observation behavior planning, and fine-grained UAV control. This allows QA, observation planning, and control execution to be evaluated separately and thus the benchmark provides a more detailed and interpretable assessment of embodied perception.

  • The dataset contains 10k simulated and real trajectories with 5-DoF actions: 3D translation, UAV yaw, and camera-gimbal pitch. Jointly evaluating position and camera orientation is well motivated for active perception, where reaching a location is not enough if the target remains outside the camera view.

  • The benchmark includes real indoor and outdoor flight data, along with physical UAV experiments evaluating the complete pipeline.

Critical weaknesses
  • Table 1 marks CityEQA as lacking "Planning," although its agent explicitly decomposes questions into navigation, exploration, and evidence-collection subtasks. The paper instead distinguishes CityEQA by its lack of reference trajectories. The table therefore appears to conflate planning capability with annotated planning supervision. Also, ActiveFly shares substantial similarities with CityEQA, both in its overall task formulation and its QA categories. In fact, all 5 Air-EQA categories (object recognition, attribute recognition, counting, spatial reasoning, and world knowledge) are already present in CityEQA. Similarly, UAV-Flow has already explored fine-grained language-guided UAV control with real-world deployment. So, while the ActiveFly benchmark does add aligned question plan trajectory supervision, body and gimbal control, and physical evaluation, question-driven aerial exploration is not itself new, and the substantial overlap with prior work deserves a more detailed discussion to clearly establish the paper's novelty.

  • OBP evaluates planning as selecting 1 of 4 predefined instructions rather than generating an observation plan, with three alternatives designed to be infeasible/ineffective. Also, EP requires agreement with the reference observation behavior and target pose, although the escape analysis shows that alternative trajectories can still support correct answers. The benchmark therefore favors reference matching over open ended, outcome-based planning, which is not ideal. This limits how well the benchmark reflects an agent's ability to plan and explore autonomously.

Questions for authors
  • Does "Planning" in Table 1 refer to planning capability or annotated planning supervision? Given that CityEQA already includes hierarchical planning and shares the same five QA categories, could the authors better clarify the novelty relative to CityEQA? Is the main difference the introduction of explicit observation-behavior planning supervision, rather than a fundamentally different planning task?

  • Following the previous point, how does the proposed observation-behavior planning compare with existing approaches, such as CityEQA's hierarchical planning? Could the authors consider a comparison or ablation demonstrating the benefits of their planning formulation? I understand that this is primarily a benchmark but it needs to be properly motivated to justify its adoption by the community.

3 comments

Q1:

Does "Planning" in Table 1 refer to planning capability or annotated planning supervision? Given that CityEQA already includes hierarchical planning and shares the same five QA categories, could the authors better clarify the novelty relative to CityEQA? Is the main difference the introduction of explicit observation-behavior planning supervision, rather than a fundamentally different planning task?

Response:

Thanks for the valuble comments. “Planning” in Table 1 refers to explicit observation-behavior annotations. We will rename the column accordingly. We acknowledge that CityEQA already supports hierarchical planning and observation adjustment. Our main contribution is the alignment of questions, observation intentions, and fine-grained control trajectories, with two benefits:

  1. Connecting question-driven planning with physical execution. Our benchmark evaluates how agents infer appropriate observation behaviors from questions and current observations, then execute UAV-body and gimbal control to obtain answer-relevant views. UAV-Flow evaluates execution of supplied flight instructions; our benchmark additionally evaluates how observation intentions are inferred from EQA questions.
  2. Supporting separate evaluation and diagnosis. Aligned annotations allow us to assess observation planning, instruction execution, and visual answering separately, alongside overall EQA performance. This helps distinguish difficulties in deciding what to observe, reaching the intended viewpoint, and interpreting the acquired observations.

The distinction lies in aligned supervision and diagnostic evaluation, rather than the introduction of hierarchical planning itself.


Q2:

Following the previous point, how does the proposed observation-behavior planning compare with existing approaches, such as CityEQA's hierarchical planning? Could the authors consider a comparison or ablation demonstrating the benefits of their planning formulation? I understand that this is primarily a benchmark but it needs to be properly motivated to justify its adoption by the community.

Response:

The purpose of OBP is to translate the question’s observation requirements into executable instructions for UAV movement and viewpoint adjustment. Its intended benefit is to guide the agent toward observations useful for answering the question. We therefore compare downstream EQA accuracy with CityEQA’s hierarchical planning and two rule-based exploration strategies.

Baselines:

  • Random exploration: Generates a random trajectory for each rollout.
  • Fixed scanning: Follows a predefined left-to-right and top-to-bottom pattern. Both rule-based strategies use 20 steps, matching the average reference trajectory length.
  • CityEQA
  • Reference OBP (Ours)

Results:

Exploration MethodEQA Acc. (%)Improvement over No Exploration (pp)
No exploration50.4—
Random55.95.5
Fixed scanning56.76.3
CityEQA52.42.0
OpenVLA w/ OBP (Ours)77.226.8
OpenPi w/ OBP (Ours)81.731.3
  1. Rule-based exploration provides limited gains. Random exploration and fixed scanning yield only small improvements in EQA accuracy, suggesting that rule-based exploration without semantic guidance struggles to capture the visual information relevant to embodied questions.

  2. Insufficiently precise viewpoint control limits exploration. CityEQA’s VLM-based planning emphasizes bringing the target into view, but may not provide sufficiently fine-grained viewpoint planning for tasks requiring a specific viewing angle, such as observing a billboard from the front. This can lead to failures on such tasks. Its improvement over no exploration is also limited.

  3. Clarification of our OBP. Our benchmark does not aim to introduce a new planner architecture. Instead, we introduce the first alignment of high-level EQA tasks and low-level control through OBP in low-altitude environments. Through human annotation, OBP provides observation plans that reflect human reasoning, allowing models to learn human priors about scene understanding and decision-making. This motivates our benchmark’s aligned evaluation of observation planning, control execution, and EQA, allowing researchers to examine whether a planning decision translates into effective observation behavior and improved answers.

Q3:

OBP evaluates planning as selecting 1 of 4 predefined instructions rather than generating an observation plan, with three alternatives designed to be infeasible/ineffective. Also, EP requires agreement with the reference observation behavior and target pose, although the escape analysis shows that alternative trajectories can still support correct answers. The benchmark therefore favors reference matching over open ended, outcome-based planning, which is not ideal. This limits how well the benchmark reflects an agent's ability to plan and explore autonomously.

Response:

Thanks for your comments. Our benchmark is not intended to encourage models to merely match reference plans. Instead, through human annotation and review, we provide straightforward and effective courses of action for a given scene. We aim to help models learn from human experience in scene understanding and action selection by learning these decision-making patterns. To evaluate the effectiveness of the instructions we provide, we also conduct experiments with free-form instructions.

Baselines:

  • No exploration: the VLM answered directly answer the embodied question only rely on the inital frame.
  • Free-form OBP: the VLM planner directly generates observation instructions instead of selecting from predefined options.
  • Reference OBP: baselines using our reference observation plans.

Fairness of the free-form OBP comparison:

To preserve the VLM planner’s freedom to make planning decisions while helping to reduce the potential influence of instruction-distribution differences on VLA execution, we prompt the planner to generate instructions with wording and sentence structures similar to those of the reference instructions. We assess syntactic similarity using FastKASSIM [1] and style-distribution similarity using MMD[2] over StyleDistance[3] embeddings, before reporting the EQA performance of the free-form OBP baselines.

Datasets:

we evaluatae these baselines on simulated datasets.

Results:

  1. Instruction similarity
ComparisonFastKASSIMMMD + StyleDistance
Free-form OBP vs. reference instruction split A0.590.09
Reference instruction split A vs. split B0.570.02

The FastKASSIM scores suggest comparable syntactic similarity to the reference-split baseline. The MMD score for free-form instructions is closer to the score for reference splits. The results indicating that our free-form OBP preserves the planner’s freedom to make decisions while mitigating the influence of instruction-style differences on the downstream VLA model, helping its execution more faithfully reflect the upstream planner’s decisions.

  1. EQA performance

All baselines use GPT-5.6 Sol as the EQA answerer, and exploration-based baselines use the same model as the OBP planner.

BaselineEQA Accuracy (%)Improvement (pp)TP among Correct Answers (%)
No exploration50.4——
OpenVLA w/ Free-form OBP67.817.475.54
OpenPi w/ Free-form OBP69.419.080.92
OpenVLA w/ OBP (Ours)77.226.878.77
OpenPi w/ OBP (Ours)81.731.387.22

These results support the following interpretation:

  1. If the reference-OBP baselines achieve comparable TP proportions, this would indicate that correct answers under both planning approaches are generally accompanied by sufficient visual evidence. This comparison concerns the availability of evidence in the answerer’s inputs.
  2. Our reference-OBP baselines achieve higher EQA accuracy than their free-form counterparts, which is consistent with more effective observation guidance. This aligns with our benchmark’s design goal: to evaluate effective evidence acquisition and provide efficient, human-informed observation plans through which models can learn useful observation priors.

[1] Chen et al. (2023). FastKASSIM: A Fast Tree Kernel-Based Syntactic Similarity Metric. EACL.

[2] Gretton et al. (2012). A Kernel Two-Sample Test]. JMLR, 13:723–773.

[3] Patel et al. (2025). StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples NAACL.

These new experiments are appreciated and address aspects of my concerns. I raised my score to weak accept, but it is really close.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.