- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CrossFly, a benchmark for evaluating interaction-oriented aerial mobility across heterogeneous environments. To address the limitations of existing benchmarks in continuous indoor-outdoor navigation and the transition from navigation to physical interaction, the authors construct a dataset containing 15,263 trajectories across 23 reconstructed scenes. An Interaction Readiness Rate (IRR) is introduced to evaluate whether terminal navigation states satisfy the geometric conditions required for subsequent manipulation. In addition, a hierarchical VLA agent is developed by integrating high-level language-based planning with low-level action generation. Experimental results demonstrate the increased difficulty of cross-environment navigation and show that the proposed agent outperforms several representative baselines.
Critical strengths
- Practical significance of the research problem. The paper addresses the important connection between long-horizon aerial navigation and subsequent physical interaction, which is not sufficiently considered in conventional navigation benchmarks. The integration of heterogeneous environments and interaction-oriented evaluation provides a meaningful research setting for aerial embodied intelligence.
- Systematic benchmark construction and evaluation. The proposed benchmark incorporates diverse reconstructed environments, structured trajectories, and compositional task instructions. The evaluation across multiple VLA models, together with the controlled comparison of different task settings, provides useful evidence of the challenges associated with cross-environment aerial navigation.
Critical weaknesses
- Limited quantitative evidence supporting the generalizability of IRR. Although the authors derive the IRR thresholds from manipulation outcomes across multiple aerial embodiments, the effectiveness of the resulting metric is not sufficiently evaluated on independent test cases. In particular, the paper does not report quantitative results demonstrating how reliably IRR distinguishes manipulation-ready states from states that lead to manipulation failure. Further evaluation of its predictive performance and cross-embodiment generalizability would strengthen the central contribution of the benchmark.
- Insufficient experimental evidence for the contribution of hierarchical planning. The proposed CrossFly-Agent achieves improved performance over several representative baselines. However, the paper does not provide controlled ablation studies to isolate the contribution of the high-level VLM planner. As the compared methods differ in their underlying architectures and pretrained models, the current results do not clearly establish whether the improvements primarily arise from hierarchical subtask planning. This limits the experimental support for the effectiveness of the proposed agent architecture.
Questions for authors
- Can the authors provide quantitative results evaluating the predictive performance of IRR on held-out manipulation trials? In particular, how do manipulation success rates differ between IRR-satisfied and IRR-violated terminal states, and how well do the derived thresholds generalize across different aerial embodiments?
- Can the authors provide an ablation study comparing the same QwenPi-based policy with and without the high-level VLM planner? This would help clarify the contribution of hierarchical subtask planning to the reported performance improvements.
3 comments
W2: "However, the paper does not provide controlled ablation studies to isolate the contribution of the high-level VLM planner. As the compared methods differ in their underlying architectures and pretrained models..."
A-W2: Thank you for this question. We add controlled ablations to examine the contribution of hierarchical subtask guidance, including the roles of planner fine-tuning and VLA subtask supervision.
(1) Experimental validation.
We compare four configurations on the Outdoor task suite from the seen-scene split, using the same pretrained QwenPi backbone, training trajectories, VLA visual inputs, action supervision, and fine-tuning budget. All configurations use the same evaluation episodes as Table 2, and the full-model scores match its Outdoor results.
| Configuration | High-level Planner | VLA Training Subtask | VLA Inference Subtask | SR (%) ↑ | IRR (%) ↑ |
|---|---|---|---|---|---|
| A. VLA only | — | — | — | 18.1 | 40.4 |
| B. Frozen-planner variant | Frozen pretrained VLM | Ground truth | Planner prediction | 20.7 | 42.5 |
| C. Predicted-subtask variant | Fine-tuned VLM | Planner prediction | Planner prediction | 23.1 | 45.6 |
| D. Full CrossFly-Agent | Fine-tuned VLM | Ground truth | Planner prediction | 25.5 | 50.3 |
(2) Results and mechanism analysis.
Compared with the VLA-only baseline, the full model (A → D) improves SR by 7.4 percentage points and IRR by 9.9 percentage points. The high-level planner gives QwenPi an explicit local objective for target approach and manipulation invocation, connecting the global instruction to the current stage of execution.
With the same trained VLA checkpoint, planner fine-tuning (B → D) produces differences of 4.8 percentage points in SR and 7.8 percentage points in IRR. This comparison isolates the effect of learning to select the active subtask from the global instruction and visual context.
With the same fine-tuned planner, using ground-truth rather than predicted subtasks during VLA training (C → D) produces differences of 2.4 percentage points in SR and 4.7 percentage points in IRR. Ground-truth subtasks connect the local objective with the corresponding flight actions and invocation decisions during training, helping coordinate final pose adjustments with the navigation-to-manipulation handoff.
We include these ablations and the corresponding analysis in the revised manuscript (Lines 1011–1034, Table 11), with the revised text highlighted in red.
The answers to Q1 and Q2 are the same as those provided above.
Looking forward to your further discussion
Dear Reviewer aAeu,
Thank you again for your kind review and comments. We hope our responses and the revised paper have addressed your concerns. If any further clarification would be helpful, we would be glad to provide it. We sincerely hope that we will be able to use the remaining time to engage in further dialogue with domain experts to enhance the quality of our work. We truly appreciate your time and feedback.
Thanks again, Authors
We thank reviewer for the valuable time and positive feedback. We provide point-to-point responses below and have revised the manuscript accordingly (please see the revised PDF in the Rebuttal Revision).
W1-1:"...In particular, the paper does not report quantitative results demonstrating how reliably IRR distinguishes manipulation-ready states from states that lead to manipulation failure..."
A-W1-1: Thanks for highlighting the need for quantitative validation. We address this concern through an experimental validation and mechanism analysis.
(1) Experimental validation.
We evaluate 50 independent test cases across 10 scene categories, comprising 25 IRR-satisfied and 25 IRR-violated terminal states. Results are reported below.
The 68-percentage-point success-rate gap supports IRR’s ability to distinguish interaction readiness beyond target reaching, since both groups already satisfy the navigation success criterion.
(2) Mechanism analysis.
IRR constrains terminal distance and target-facing alignment, both of which affect depth-based target localization. Excessive distance or yaw misalignment can increase target-position errors, impairing downstream manipulation.
Our failure analysis examines terminal distance, yaw offset, and localization error. Figure 6 (§4.4) illustrates this connection, with the same grasping policy succeeding from an IRR-satisfied state and failing from an IRR-violated state.
We include the evaluation protocol, results, and failure analysis in the revised manuscript (Lines 429–469 and 990–995, Figure 6, Table 5), with the revised text highlighted in red.
W1-2: "...Further evaluation of its predictive performance and cross-embodiment generalizability would strengthen the central contribution of the benchmark."
A-W1-2: We agree. To assess the applicability of IRR across embodiments, we selected two representative aerial manipulation platforms for a controlled comparison, the hand-like UAV HI-ARM [1] and a UAV equipped with an SO100 arm [2].
Both platforms use the same visual localization pipeline and each has a dedicated RL grasping policy. Each policy receives the target’s 3D position in the UAV body frame, estimated from the depth at the center of the target’s visible surface (§4.4).
For each platform, we compare grasping success using visually estimated positions and simulator ground-truth positions.
With accurate target positions, grasping success exceeded 99.7% on both platforms. Using visual position estimates reduces grasping success by approximately 50.13 percentage points on HI-ARM and 49.59 percentage points on the SO100-equipped UAV. These decreases indicate that target-localization accuracy is the main bottleneck on both tested embodiments.
Excessive terminal distance or yaw misalignment can increase target-localization error, impairing downstream grasping. The IRR distance and viewpoint criteria provide an embodiment-independent geometric assessment of terminal-state suitability for target localization.
We report this comparison and discuss how the shared visual localization pipeline supports cross-embodiment evaluation in the revised manuscript (Lines 471–485 and 899–936, Table 6), with the revised text highlighted in red.
[1] Hand-like autonomous flying robot for airborne grasping and interaction. In Nature Communications 2026.
[2] LeRobot: State-of-the-art Machine Learning for Real-world Robotics in PyTorch. GitHub. SO100 documentation.