acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

Abstract

Multimodal large language models (MLLMs) are increasingly used to make decisions for drones, but existing evaluations are mostly limited to simulation, to a single drone, and to targets that are visible from the start and do not move. We present DroneCATS, a benchmark in which the MLLM alone controls the drone through four actions, and evaluate ten MLLMs from frontier models to small open models. The benchmark varies target visibility, target motion, scene and simulator, and extends to real drones. Beyond a single drone, it adds a multi-drone suite, where one model drives multiple drones, and a commander + sub-agents suite, where a commander model directs one sub-agent for each of the drones. Across these settings, we find that frontier models achieve the highest success rates, but their latency causes them to lose moving targets, while small open models often do not declare arrival even when they reach the target. Guided by this failure analysis, we fine-tune a 2B model on drone action data synthesized from public RGB-D images; its success rate rises from 0% to 31% in simulation and outperforms the frontier models in real flight. With multiple drones, we find that every zero-shot model succeeds more often as the commander of one sub-agent per drone than when it drives all of them from one context, and that commanding success follows general VLM ability. On real drones, we observe that the results broadly follow those in simulation, while latency and hardware limits matter more. We will fully open-source the benchmark.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Vision-Language-Action Models, Drone, Agents

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_hT8TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DroneCATS, a benchmark for evaluating multimodal models on drone approaching, searching, tracking, and multi-drone coordination through a shared four-action interface. It compares direct control with a commander-and-sub-agents structure and evaluates models in two simulators and real flights. It also fine-tunes a 2B model using action data generated from public RGB-D images and reports improved navigation success.

Critical strengths

1: The benchmark addresses a useful evaluation problem. Bringing target search, target motion, and multi-drone coordination into one framework allows comparisons across settings that are often studied separately.

2: The experimental coverage is substantial. The paper includes two simulators, 118 real-flight episodes, and useful ablations on visual history, thinking, maps, and command structure. These provide a useful basis for studying the strengths and limitations of different models.

Critical weaknesses

1: The arrival evaluation does not clearly support the paper’s interpretation of model behavior. Incorrect completion declarations do not end an episode, and any later qualifying declaration counts as success. Table 9 reports an average of 39.4 declarations per episode for Qwen3.5-27B, so successful completion may still involve many incorrect judgments. The prompt also treats empty margins around the target as evidence that it is over 10 metres away. This rule is unreliable across target sizes, including the egg used in real flights, and does not match the distance-based success criterion. These issues make it difficult to attribute failure to declare arrival to poor instruction following.

2: The main comparative conclusions need stronger support. The hierarchical setup changes the pilot models, map information, and reasoning settings alongside the command structure. The appendix provides useful controls, but the overall gain remains difficult to attribute to organization alone. The real-flight advantage is also based on only 11/20 successful flights for the fine-tuned model versus 9/20 for Gemini. These results support initial effectiveness, but not a stable advantage over frontier models. The paper should distinguish these findings from broader claims and make the benchmark’s added value over existing work clearer, particularly in the abstract.

Questions for authors

1: Arrival evaluation: What are the first-declaration success rate and false-declaration rate for each model? How would the rankings change if the first completion declaration ended the episode?

2: Prompt sensitivity: Does the reported declaration failure persist when the prompt states the actual success threshold and removes the frame-filling rule? Please also distinguish episodes that satisfy both distance and visibility requirements from those that only enter the distance threshold.

3: Multi-drone comparison: How much of the hierarchical advantage remains when the same model is used across roles, both structures receive the same map information, and reasoning settings are matched? Please also report the differences in model-call counts and computational costs.

4: Real-flight evidence: Can you provide uncertainty estimates or additional paired trials for the reported 55% versus 45% success rates? This would help establish how much weight to place on the claimed advantage.

5: Benchmark contribution: Beyond broader task coverage, which findings does DroneCATS enable that existing evaluations such as SPF and MissionBench cannot establish?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the insightful questions, which helped us clarify the evaluation protocol. Our point-by-point responses follow.


What are the first-declaration SR and false-declaration rate? (W1, Q1)

The table reports the first-declaration SR, which assumes that the episode ends at the first completion declaration, and the false-declaration rate, the share of all declarations that do not meet the success condition.

ModelSRFirst-declaration SRFalse-declaration rate
Gemini 3.7 Flash64.449.474
Gemini Robotics-ER 257.548.191
GPT-555.049.463
Claude Opus 546.235.092
Qwen3.5-27B39.45.099
FT 2B31.28.898
Qwen3.5-9B21.219.456
Qwen3.5-4B6.22.592

The four frontier models remain at the top under the first-declaration criterion. By design, an incorrect declaration does not end the episode: what we evaluate is whether the drone eventually reaches the target and confirms its arrival, not whether its first declaration is correct. This also lets the drone keep following a moving target after declaring, and lets a commander redirect a sub-agent in the multi-drone setting, so one rule covers both settings. For Qwen3.5-27B and FT 2B, however, most successes follow many incorrect declarations, which the first-declaration SR and the false-declaration rate make visible. In the revision, we explain in the main text why the episode does not end at the first declaration, and report the first-declaration SR and false-declaration rate (revised: Section 3.1, "Metrics"; Section 4.1, "Recognition and declaration"; Appendix B, Table 5).


Does the failure persist without the margin rule? (W1, Q2)

To test whether the declaration failure is caused by the specific prompt, and in particular by the margin rule you pointed out, we reran three models whose failures come from declaring: Qwen3.5-9B, which rarely declares, and Qwen3.5-27B and FT 2B, which over-declare. Prompt A removes the margin rule and states the actual value of δ. Prompt B adds two few-shot examples to A.

ModelOriginalAB
Qwen3.5-27B39.450.043.8
Qwen3.5-9B21.218.115.6
FT 2B31.228.128.8
Mean30.632.129.4

The declaration failure persists under both prompts. The mean SR of the three models remains nearly unchanged, at 30.6%, 32.1% and 29.4%. Only Qwen3.5-27B improves, with prompt A, while Qwen3.5-9B and FT 2B decline slightly. The declaration failure therefore stems not from the margin rule or a particular prompt, but from the models' ability to follow the declaration rule in the system prompt.

Nor is this a perception failure. We placed the logged drone and target poses of every step of the paper runs back in the simulator and checked whether the target was visible. Qwen3.5-9B entered δ in 63.1% of episodes by distance alone and in 51.9% with the target also in view, yet succeeded in only 21.2%. It reached the target and saw it, but failed to declare arrival.


Are distance and visibility distinguished? (Q2)

To distinguish episodes that only enter the distance threshold from those that satisfy both the distance and visibility requirements, Table 1 reports OSR and SR together. OSR is the share of episodes in which the drone enters δ, and SR is the share in which the drone is inside δ with the target in view at the time of declaration.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

How much of the hierarchical advantage remains under matched settings, and at what cost? (W2, Q3)

For the same-model and matched-reasoning controls, please see the general response (Benefit of the commander structure and confounding factors). Regarding the map usage, we separately compared against commanders run without the map in the map ablation of the appendix (with ER2 sub-agents). Even without the map, the commander structure outperforms multi-drone (73.3 vs 50.0 for Gemini 3.7 Flash, 80.0 vs 46.7 for Claude Opus 5).

We report model calls and cost as per-episode averages on the Three-sign sum task. Because each of the four sub-agents issues its own calls, the commander structure requires more calls and tokens than multi-drone. Compared with the no-commander setting, it requires fewer calls at a similar cost while achieving a higher success rate.

StructureHow the four drones are controlledCalls (Opus / Flash)Tokens (Opus / Flash)Cost (Opus)
Multi-droneOne model controls all four drones64 / 34342k / 233k$1.89
No commanderFour instances of the same model, one per drone, without communication269 / 192726k / 517k$4.28
Commander + same-model sub-agentsOne commander directs four sub-agents of the same model234 / 172738k / 778k$4.15

Does 11/20 versus 9/20 show an advantage on the real drone? (W2, Q4)

We thank the reviewer for this point. The statement that the fine-tuned model outperforms frontier models in real flight could mislead readers, since the overall gap is small: 11/20 (95% interval 34–74%) versus 9/20 (26–66%). We therefore clarify it in the revision. The overall numbers, however, hide a clear difference by task. Both models flew the same targets, and the fine-tuned 2B succeeds in 5 of 10 moving-target flights and Gemini 3.7 Flash in none, while Gemini 3.7 Flash leads on static targets (9/10 vs 6/10). This matches the simulation finding that slow models lose moving targets. In the revision, the abstract and introduction state that the fine-tuned model is comparable to frontier models on the real drone (revised: Abstract; Section 1; Section 5).


What can DroneCATS show that SPF or MissionBench cannot? (W2, Q5)

DroneCATS establishes three findings that existing evaluations cannot. First, whether a model judges its own arrival. SPF ends an episode with an external rule and therefore cannot measure this, whereas in DroneCATS this judgment is the main point of failure. Second, the effect of inference latency. Only a closed-loop evaluation in which the target keeps moving during inference reveals that slow frontier models lose moving targets. When the agent design of MissionBench is transplanted onto our benchmark, it enters δ in only 3 of 20 moving-target episodes and succeeds in none (Table 20). This latency effect also appears on the real drone, flown with the same prompt and actions. Third, the organization of multiple drones. Unlike existing evaluations that cover a single drone, we find that a commander directing sub-agents achieves higher success than a single model controlling all four drones directly. In the revision, we state these findings in the abstract and the introduction (revised: Abstract; Section 1).


We hope these answers address your concerns, and we would be happy to clarify anything further. Thank you again for your valuable time and feedback.

Thank you for the detailed response. Most of my questions have been addressed, but I still have some concerns about the declaration-based success criterion, the additional cost of the hierarchical setup, the limited real-flight evidence, and the benchmark novelty over existing work. For now, I am inclined to keep my current rating.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the follow-up and for the time spent on our responses. Regarding the remaining concerns about the declaration-based success criterion, the additional cost of the hierarchical setup, the real-flight evidence, and the novelty over existing work, we have tried to address each of these points in detail in our previous responses. We would be very grateful if the reviewer could let us know which aspects still seem insufficient, and we would be glad to clarify any specific point further during the remaining discussion period.

Reviewer_YWDHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DroneCATS is a benchmark where an MLLM is the only thing controlling a drone. Each step the model sees one forward-camera frame plus the instruction and its last five actions, and returns one of four JSON actions: go (a pixel and a depth), rotate, think, or finished. A rule-based controller flies it. An episode counts as a success only if the model declares arrival while within a radius of the target with the target in view, so "got close but never said so" is a failure. Ten models are tested on four task types (target visible/hidden, static/moving) in AirSim, plus a smaller Isaac Sim suite, a four-drone suite, and real DJI Tello flights. The best model (Gemini 3.7 Flash) succeeds 64% of the time. Frontier models lose moving targets because they are slow; small open models reach the target but don't declare. A 2B model fine-tuned on synthetic data from public RGB-D images goes from 0% to 31%, and does best on the real drone. With four drones, every model does better as a commander giving text orders to per-drone sub-agents than driving all four itself.

Critical strengths
  • The model is the whole pilot. No planner or verifier in the loop, so model comparisons are clean.
  • Several models flown on a real drone, same prompt and actions as in sim. Prior work shows demos, not comparisons.
  • The failure analysis is the best part. Separating "reached the target" from "declared arrival" shows small models fail at the protocol, not at flying (Qwen3.5-9B: 63% reached, 21% succeeded).
  • Noise is reported. Every sim cell is flown twice (Table 10), and the paper says which differences are within noise.
Critical weaknesses
  1. Samples are small for the size of the claims. 20 episodes per cell in sim, 5 on the real drone, 28 in Isaac Sim flown once. The authors say within-tier order is noise, yet the abstract says real flight follows sim (ρ = 0.57 over cells of 5 flights) and that the fine-tuned 2B beats frontier models in the air (55% vs 45%, 20 episodes a side). Isaac Sim flips several mid-table orderings.
  2. Latency is a serving property, not a model property, but it drives the moving-target results. API models go over the network; the 2B models run locally. Isaac Sim, where the sim runs 11x slower than wall clock, removes the effect and the fast-model edge vanishes. Report real-time and paused-during-inference numbers separately, and split model time from transport time.
  3. The declaration failure is mostly "did it follow this one system prompt." The 0.95 SysBench correlation is the paper's own evidence. No alternative prompt or few-shot example is tried for the small models, so we can't tell protocol failure from perception failure.
  4. Commander vs multi-drone has a strong-sub-agent confound. Multi-drone = one model alone; commander = that model plus four Gemini Robotics-ER 2 pilots. The "no commander" control exists for one model only (Table 14), and commanders run with reasoning on while multi-drone runs at the lowest level.
  5. Thin elsewhere too. Multi-drone: 10 episodes per level, one map per level, so 80% vs 40% is 8/10 vs 4/10. Real flights are judged by one observer by eye, no second judge.
  6. Title oversells. The benchmark is "fly to a named thing and say so." No obstacles, no collision metric, no altitude tasks. "Generalist VLA agents" promises more than four actions deliver.
  7. Missing citations. Two directly related benchmarks from the same line of work are absent from Table 16: Towards Realistic UAV VLN (ICLR 2025; OpenUAV platform, UAV-Need-Help object-search benchmark, MLLM navigator) and UAV-Flow Colosseo (NeurIPS 2025 D&B; 30k real flight trajectories, real-UAV VLA deployment, and an explicit treatment of inference latency with stop-and-infer vs continuous-motion schemes). The second one bears directly on weakness 2.
Questions for authors
  1. Rerun track / search + track with the simulator paused during inference. How much of the frontier-model gap survives?
  2. Split latency into model time and transport time. Did the local 2B models get a speed advantage from sharing a machine with the simulator?
  3. Try one alternative prompt or two few-shot examples for the Qwen models. Does the declaration failure go away?
  4. Run the no-commander and multi-drone controls for two more models, at the same reasoning setting the commander uses.
  5. How many judges scored the real flights? Any second-judge check?
  6. Any collisions in sim or on the Tello? A crash inside δ that declares would count as success now.
  7. UAV-Flow Colosseo proposes stop-and-infer vs continuous-motion schemes for exactly the latency problem in Sec. 4.1. Can you position your controller against those and cite both papers above?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We are grateful to the reviewer for the thorough and detailed review. We address each weakness and question below; points shared with other reviewers are answered in the general response.


Are the claims too strong for the small samples? (W1)

We agree that more samples make the results more robust. To address this, we checked how stable and robust the results are across repeated runs. In AirSim, every cell was flown twice under identical settings, and the two runs give nearly the same results: they differ by only 5.0 points on average, which is smaller than the 6 to 8 point standard deviation that 40 runs per cell have by chance alone (Appendix, Run-to-run variance). In both runs, frontier models lead but lose moving targets, and small open models fail to declare arrival.

Real flights and Isaac Sim are costly to run: each real episode needs a person to move the target and another to judge on site, and one Isaac Sim episode can take up to about 55 minutes, so we use them only to check whether these trends carry over.

We therefore revised the abstract, introduction and conclusion to state that the fine-tuned 2B model is comparable to frontier models in real flight, not better, and toned down the Isaac Sim description (revised: Abstract; Section 1; Section 5; Appendix G).


How much of the gap is latency, and did local models benefit from sharing a machine? (W2, Q1, Q2)

Please see the general response (Latency). On whether the local 2B models gained from sharing a machine with the simulator: with the same FT 2B checkpoint, controller and task configuration, moving-target SR is 27.5% when served locally on one L40S (0.57–0.66 s per call) and 30.0% when served remotely on one A100 (0.41–0.44 s per call). The A100 call time includes network transfer, yet the result barely changes.


Does the declaration failure persist with another prompt? (W3, Q3)

As suggested, we reran three models whose failures come from declaring, with an alternative prompt and with two few-shot examples: Qwen3.5-9B, which rarely declares, and Qwen3.5-27B and FT 2B, which over-declare. Prompt A removes the margin rule and states the actual value of δ. Prompt B adds two few-shot examples to A.

ModelOriginalAB
Qwen3.5-27B39.450.043.8
Qwen3.5-9B21.218.115.6
FT 2B31.228.128.8
Mean30.632.129.4

The declaration failure persists under both prompts. The mean SR of the three models remains nearly unchanged, at 30.6%, 32.1% and 29.4%. Only Qwen3.5-27B improves, with prompt A, while Qwen3.5-9B and FT 2B decline slightly. The declaration failure therefore stems not from a particular prompt, but from the models' ability to follow the declaration rule in the system prompt.

Nor is this a perception failure. We placed the logged drone and target poses of every step of the paper runs back in the simulator and checked whether the target was visible. Qwen3.5-9B entered δ in 63.1% of episodes by distance alone and in 51.9% with the target also in view, yet succeeded in only 21.2%. It reached the target and saw it, but failed to declare arrival.


Is the commander advantage confounded? (W4, Q4)

Please see the general response (Benefit of the commander structure and confounding factors).

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Are 10 episodes per level enough, and who judged the real flights? (W5, Q5)

Our multi-drone comparison is based on the total over the three levels, 30 episodes per model, rather than on a single level of 10 episodes. On this total, the commander structure outperforms multi-drone for all nine zero-shot models, by 6.7 to 46.7 points. The hard level is where this trend is most visible.

In preliminary flights, two authors judged each declaration together and never disagreed, since with δ = 1.5 m indoors it is clear whether the drone is within δ and whether the target is in view. The remaining flights were therefore split between two authors, with one author judging each flight in real time. For distance, when the drone declared arrival it hovered in place while the judge measured the distance between the drone and the target with a tape measure, and the declaration counted only if it was within δ = 1.5 m. This time is excluded from the episode time. For visibility, another author, blind to the first judgment, re-judged every declaration in the 100 single-drone real flights (177 in total) from the saved frames. The two judgments agree on 97.7% of declarations, and the 4 disagreements all occurred with the drone outside δ. With the re-judged visibility, the outcome of all 100 flights stays the same.


Does "generalist" overstate the scope? (W6)

Please see the general response (The term "generalist" in the title).


Should a declaration after a collision count as a success? (Q6)

DroneCATS is goal-oriented: it scores whether the instruction is completed and does not score collision avoidance separately, so a declaration after a collision inside δ currently counts as a success. In the moving-target tasks, 11.1% of successful episodes had a collision inside δ before the declaration, mostly contact with the target. Counting them as failures lowers SR by 5.2 points on average, while the two weakest models remain at the bottom. On the Tello, 22 of 100 flights ended in a collision, and only one of them was counted as a success. We agree that collisions deserve discussion beyond task completion, and add a short discussion of collisions to the failure cases in the appendix (revised: Section 3.1, "Metrics"; Appendix D).


How does the controller relate to UAV-Flow Colosseo? (W7, Q7)

Our controller follows the continuous-motion scheme of UAV-Flow Colosseo (NeurIPS 2025 D&B): the simulation keeps running during inference. The paused-during-inference rerun in the general response (Latency) corresponds to stop-and-infer, and we now report both. We add UAV-Flow Colosseo and Towards Realistic UAV VLN (ICLR 2025) to Table 19 and cite both (revised: Section 4.1; Appendix I, Table 19).


We hope these answers address your concerns, and we would be happy to clarify anything further. Thank you again for your valuable time and feedback.

Reviewer_YWDHOpenReview ↗

Post-rebuttal Comments.

Thanks for the thorough rebuttal. The paused-simulator rerun and the alternative prompts answer my two main concerns directly, and the same-model commander control and second-judge check settle the rest. I’m keeping my score at 3.

One request from the other reviews: please put first-declaration SR and false-declaration rate in Table 1. With 98-99% false declarations for FT 2B and Qwen3.5-27B, the 0% to 31% result should be described as learning to declare, not yet learning to judge arrival.

AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the helpful suggestions on the details of the paper. We have reflected both requests in the revision.

  • We added the first-declaration SR (1st SR), the SR if the first declaration ended the run, and the false-declaration rate (FDR) to Table 1 (lines 271–292).
  • In Section 4.1, we now describe the fine-tuning result as the model learning to declare arrival (line 348).

We would be happy to answer any further questions.

Reviewer_HpSjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a simulation based benchmark for drone control with pre-trained VLM agents. It provides scenes, tasks tiered by difficulty, a simple API interface for high-level drone control, and task scoring for comparing the abilities of various VLMs in this task. It includes multi-agent orchestration tasks, which are an interesting and unique task domain. The paper presents results of the benchmark with 10 pre-trained VLM models also presents a fine-tuned version of Cosmos3-Edge-2B for this task.

Critical strengths

This goal of this paper is quite timely and laudable. There are still many open challenges in drone autonomy that are not solved with current VLA or VLM agentic approaches, and are still mostly attempted with classical approaches. Drones have unique use cases, motion capabilities, and severe timing, weight, power, and form-factor limitations that cannot be addressed with existing learning-based policy frameworks for manipulation, humanoids, and AVs. I also very much appreciate the multi-drone orchestration tasks, as I see this as a significant blind spot, not only in drone autonomy research, but generally in the robot learning literature.

Critical weaknesses

While the motivation of this work is excellent, my concern is with the realism and difficulty of the tasks chosen. My concerns are listed below in decreasing order of importance:

  1. The abstraction chosen for the simulation benchmark essentially removes most of the elements that make drones challenging. The tasks involve moving to a target that is either in view, or can be brought into view after an initial yaw motion. This task family can already be well solved with existing light-weight modules, e.g,, Yolo or SAM, together with basic navigational commands. The fact that the VLMs struggle on this task just shows that these VLMs are not appropriate. It would be interesting to see frontier VLMs, like Opus 5.5 or Astra, that can build code as policies and use tools like Yolo and control scripts to solve this problem. I imagine they would do quite a good job given the simplicity of the tasks. These models would have high latency, but could solve the task by building a reactive control loop that has very low latency at runtime, so offline precompute may be slow, the there is no need for the large VLM to be in the feedback loop.

  2. The paper mentions tracking a moving target, but I could not find details on how much the targets move, whether they eventually stop, and whether they can move through and around occlusions---ultimately these are the factors that make target following difficult. The success checker asks if the drone reaches within a radius of the target. But clearly this is not sufficient if the target is moving. Indeed, the task doesn't have a simple completion criterion if the target is dynamic. The drone should continually fly within some proximity of the target, not just reach that proximity once and call it a day.

  3. What about collision avoidance? Indeed, the primary concern of drone autonomy is navigating in 3D space without hitting obstacles. It seems this tasks does not deal with this challenge, which is more primal than tracking targets. Before one can track targets, one must stay alive in flight and not hit things. Indeed, all targets seems to have a clear line of sight from drone to target (given an appropriate yaw angle). Also the hardware experiments appear to be in a blank open space, with no risk of collisions.

  4. Drone tasks can be much richer than just following targets. Drones can also build 3D maps, navigate through complex indoor environments, explore unknown 3D environments zero shot, look for trule hidden targets using a memory-based search, and track truly dynamic targets through complex 3D geometry. While these are all topics of current research in drone autonomy, there do exist classical or modular approaches that accomplish these tasks today, running on real drone hardware, flying under real conditions. So this call into questions the value of this benchmark given SOTA drone autonomy capabilities.

Questions for authors

My questions are implied by my concerns above.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We appreciate the reviewer's thoughtful comments and the positive remarks on the clarity and timeliness of the paper and on the multi-drone tasks. We respond to each weakness below.


Why not YOLO, SAM or code policies? (W1)

Detector-and-script controllers fit a narrower problem. They need the target class in advance and a hand-written stopping rule, which may work for approaching a known object. They do not directly cover search, where the drone must decide where to look for a target described in free form before it is in view, or the multi-drone tasks, where the drones must be assigned and coordinated. Our question is whether general-purpose MLLMs that were not trained for drones can act as closed-loop pilots from the camera view and the instruction alone, judging arrival on their own. In this setting the tasks are not yet solved: the best model reaches 64.4%, and small open models reach 0–21%.

The finding that VLMs struggle also does not end with the conclusion that VLMs are unsuitable. We separate the failures into two causes and show that both can be fixed. Slow models lose moving targets: when the simulator is paused during inference, Gemini 3.7 Flash rises from 45.0% to 87.5%. Small models reach the target but fail to declare arrival: fine-tuning on synthesized data raises a 2B model from 0% to 31%. Showing where and why models fail is the role of this benchmark.


How do targets move, and when does a moving-target episode succeed? (W2)

As described in the appendix, targets move along one-way routes at 0.3 m/s outdoors and 0.15 m/s indoors and do not stop during the episode. The main question of this work is whether an MLLM can directly serve as a basic drone controller, and since MLLMs already struggle on the current approach, track, search and search + track tasks, we did not include occlusion or obstacle avoidance. Success is not measured by entering the radius alone: the model must declare arrival while within δ with the target in view. We use this criterion for all four tasks so that they are compared under one general metric. To check sustained following, as you suggest, we measured for successful tracking runs how long the drone remained within δ after first entering it. Averaged over the seven models, 90.1% of successful runs remained within δ for at least 5 consecutive seconds, indicating that most successes reflect sustained following rather than a transient pass through δ.


Why is collision avoidance not covered? (W3)

DroneCATS is goal-oriented: SR measures whether the instruction is completed, so a collision that keeps the drone from completing the task already counts against the model. Successes that follow a collision are uncommon. In simulation, counting successes after a collision inside δ as failures lowers SR by 5.2 points on average, and on the real drone only one of the 22 flights that ended in a collision was counted as a success. To discuss collisions beyond task completion, we add a short analysis to the failure cases in the appendix (revised: Section 3.1, "Metrics"; Appendix D).


What does the benchmark add beyond classical drone stacks? (W4)

Classical and modular stacks solve many of these tasks on real drones. However, they do not tell us how a general-purpose MLLM behaves as a pilot. Moreover, many specific drone applications require algorithms or models specialized for each task. Just as a person can be asked to find a hidden target or to fly a drone to explore an unknown environment, we believe that advanced MLLMs will become able to perform such tasks. Current MLLMs, however, still have problems even with simple drone control. We propose DroneCATS to first show these problems and to suggest directions for improvement.


We hope these answers address your concerns, and we would be happy to clarify anything further. Thank you again for your valuable time and feedback.

Thanks for the responses. Unfortunately I do not think they addressed my main concerns. I will be keeping my score as is. My overarching concern is that if the task is to yaw to bring the target into view, then pursue in a straight, obstacle-free line until you reach within a radius of the goal, this is too simplistic for drone autonomy and can already be solved with many approaches both classical and AI based. I wish the authors luck in their work, and encourage them to consider stronger baselines and more realistic and challenging drone tasks.

AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the follow-up. However, we believe the significance of our work may have been misunderstood. We do not claim that MLLMs are the best drone controllers. Our goal is not to show that these tasks cannot be solved by a dedicated pipeline, but to examine whether MLLMs can understand diverse instructions and solve the mission without task-specific manual effort, and to show where even the latest MLLMs stand. A detector-based pipeline does not remove this effort, since it needs hand-written rules for each task, and even with such rules it does not solve the tasks well.

We understand how the reviewer's impression could arise, since the examples in the main text show relatively simple routes. The benchmark, however, also includes harder cases. The routes of the moving targets range from a nearly straight path to an indoor route on which the target climbs the stairs and is occluded by them, which we now show in Figure 7 of the revised appendix (lines 757–773).

To test the reviewer’s concern directly, we implemented the strategy the reviewer described with the open-vocabulary detector YOLO-World and flew it on the same test set as in the paper, without any tuning. We gave YOLO-World a single target name extracted from the instruction (e.g., "black SUV"). YOLO-World only outputs bounding boxes and has no way to declare that the task is complete, so on its own it cannot be used under a task-completion criterion. Still, to compare in favor of the detector, we added a rule that declares arrival on its behalf once the box height exceeds 30% of the frame. YOLO-World also cannot choose actions such as rotating, so it would score zero on Search and Search + Track as is. We therefore also added a rule that rotates 90° when no box is detected and moves 3 m forward after a full turn without detection.

ApproachSearchTrackSearch + TrackSRLat. (s)
YOLO-World + rules30.025.010.010.018.80.07
Gemini 3.7 Flash87.567.555.047.564.43.5

Even with this help, YOLO-World reached an SR of only 18.8%. In 75 of 160 runs (47%), it never produced a single detection, even after scanning a full turn around the drone.

What matters in these tasks is not only detecting the target from the start, but also deciding what to do next when the target has not been found. In runs where the first decision did not point at the target, Gemini 3.7 Flash succeeded in 58%. It judged the situation on its own, explored the surroundings by rotating or moving, and approached the target once it found it. YOLO-World, in contrast, recovered and succeeded in only 11% of such runs, even with the search rule. Once the target is lost, it cannot infer where the target might be and can only rotate or go straight as the rules dictate. Nor is this a matter of speed. YOLO-World takes 0.07 s per step, far faster than the MLLMs, yet reaches only 10% on Track.

Understanding the instruction matters even more with multiple drones. In the multi-drone tasks, the target must be identified by what is written on it (e.g., the license plate 5217, the sign that says GATE 4), which a detector cannot read. Handling such cases would require new hand-written rules for each task, which is the effort MLLMs can remove. We ran YOLO-World on all 30 multi-drone episodes in Table 2. It failed in all 30 episodes: in 28 of them, none of the four drones detected a single box.

We hope this clarifies our work. If anything remains unclear, we will do our best to respond carefully before the end of the discussion period. Thank you again for your time.

Reviewer_1XYVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DroneCATS, a benchmark for evaluating multimodal large language models as high-level vision-language-action (VLA) agents for drone control. The benchmark covers four single-drone tasks—approaching, searching, tracking, and combined searching and tracking—and compares two multi-drone architectures: centralized control by a single model and hierarchical control by a commander model with independent sub-agents. The authors evaluate ten closed-source and open-source models across AirSim, Isaac Sim, and physical DJI Tello drones. Guided by failure analysis, they further fine-tune Cosmos3-Edge-2B using publicly available RGB-D data. The experiments examine how target recognition, arrival declarations, adherence to system prompts, inference latency, and multi-agent organization affect task success rates.

Critical strengths

(1) The paper evaluates models in a closed-loop setting that requires observation, action selection, and termination decisions. Reporting both success rate (SR) and oracle success rate (OSR), which measures whether the drone ever enters the target region, helps identify cases in which a model approaches the target without successfully completing the task. (2) The evaluation spans models of different sizes, two simulators, single-drone and multi-drone tasks, and physical flights. The main AirSim single-drone evaluation includes 80 episodes per model, each repeated twice. The real-world evaluation includes 100 single-drone flights and 18 two-drone collaborative flights. The appendices also examine the effects of the thinking action, visual history, map inputs, and commander configurations. These experiments provide useful context for interpreting the main results.

Critical weaknesses

(1) The authors should clarify whether “generalist” refers to the foundation models’ general-purpose pretraining or to cross-task generalization demonstrated by the evaluation, and calibrate the title and conclusions accordingly. They should also specify the conditions under which particular models or organizational structures offer advantages and substantiate these claims through results stratified by target speed, occlusion, scene complexity, and task horizon. Retaining a stronger generality claim would require additional evaluations involving unseen environments and targets, sustained tracking, together with an explicit account of operating limits and failure conditions. (2) The authors should conduct controlled comparisons across hardware and local versus remote serving while keeping the same open-model checkpoint, controller, and task configuration fixed, and separately report model inference, communication, and end-to-end control latency. To isolate network effects, controlled delay and jitter should be introduced across multiple target speeds, with curves showing moving-target success rates and target-loss rates. The simulation-to-wall-clock time ratio should be controlled or explicitly reported. (3) Limited scale of real-world experiments. The real-drone experiments involve relatively few tasks and repeated trials, and the multi-drone setup differs in scale from the simulation experiments. Therefore, conclusions regarding real-world generalization should be interpreted with some caution.

Questions for authors

(1) The appendix already provides control experiments comparing different multi-drone organizational structures. Could the authors further clarify to what extent factors such as the low-level control model and inference budget may contribute to the observed performance advantages of the hierarchical architecture? (2) For the fine-tuned lightweight model, could the authors discuss how synthetic drone-action data and general instruction data might contribute differently to the performance improvements, and clarify the limitations of the current experiments in attributing these improvements to specific data sources.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the careful reading and the constructive suggestions, in particular the request for a finer-grained analysis. Our responses to each point are given below.


Under which conditions do models fail? (W1)

Following the reviewer's suggestion, we compared the models under the conditions listed below.

Target speed. We ran the tracking task with the target speed set to 0.15, 0.3 and 0.6 m/s. Because this ablation applies the same speed to indoor and outdoor targets, the values differ from those reported in the paper.

Model (call time)0.15 m/s0.3 m/s0.6 m/s
Gemini 3.7 Flash (3.5 s)854530
Claude Opus 5 (2.9 s)805040
Gemini Robotics-ER 2 (2.0 s)805055
GPT-5 (1.4 s)606050
Qwen3.5-27B (1.0 s)806020
Qwen3.5-9B (1.4 s)15100
Qwen3.5-4B (1.0 s)505
Qwen3.5-2B (0.8 s)550
FT 2B (0.6 s)452540
Cosmos3-Edge-2B (0.6 s)000

Higher target speeds degrade the slower models, whereas FT 2B, one of the fastest, is largely unaffected. At 0.6 m/s, the four frontier models and Qwen3.5-27B lose 10 to 60 points, while FT 2B remains nearly unchanged (45 to 40). The four small general-purpose models fail at every speed (0 to 15%).

Occlusion. Success rates remain limited even with a clear line of sight to the target. Tasks involving occlusion, which additionally require path planning, are therefore beyond the scope of this paper, and we leave this extension to future work.

Scene complexity. Because scene complexity is difficult to define consistently, we instead use a clear-cut criterion: whether the target is visible in the initial view (approach) or not (search). The two tasks share the same drone and target positions and differ only in that search requires the drone to rotate to find the target. This difference alone lowers the mean success rate by 13.6 percentage points (Table 1).

Task horizon. We grouped episodes into tertiles by the target size in the initial frame, measured as the fraction of image pixels occupied by the target. This quantity reflects both the distance the drone must travel and the difficulty of recognizing the target. For search and search + track, where the target is not visible initially, we use the values of the corresponding approach and track episodes, which share the same start and target positions. The mean SR is 38.2% for small targets (< 0.20%), 45.1% for medium targets (0.20–0.37%) and 51.3% for large targets (≥ 0.37%), indicating that performance declines as the horizon becomes longer and the target smaller.

Regarding generality, all evaluation environments and targets are unseen during training. The fine-tuned 2B model was trained on general-purpose RGB-D datasets, so no benchmark simulator data were used. We evaluate across three environments with different renderers and dynamics (AirSim, Isaac Sim and a real drone), and within AirSim across indoor and outdoor scenes on two maps.

Regarding sustained tracking, we measured, for successful tracking runs, how long the drone remained within δ after first entering it. Averaged over the seven models, 90.1% of successful runs remained within δ for at least 5 consecutive seconds, indicating that most successes reflect sustained following rather than a transient pass through δ.

Based on these analyses, we identify the following episode conditions that lead to failure, in addition to the two model-related causes analyzed in the paper (latency and the failure of weaker models to declare arrival). We summarize them below.

  1. Fast-moving targets: the mean SR of the six models that achieve at least 45% on slow targets is 71.7%, 48.3% and 39.2% at 0.15, 0.3 and 0.6 m/s.
  2. Targets that require search: the mean SR over the seven models is 55.7% for approach and 42.1% for search (Table 1).
  3. Targets that appear small in the initial frame: the mean SR is 38.2%, 45.1% and 51.3% for small, medium and large targets.

What does "generalist" mean? (W1)

Please see the general response (The term "generalist" in the title).

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

How do serving hardware, network delay and simulation speed affect latency? (W2)

The split of each step into capture time and model call time, and the rerun with the simulator paused during inference, are given in the general response (Latency). In addition, with the same FT 2B checkpoint, controller and task configuration, moving-target SR is 27.5% when served locally on one L40S (0.57–0.66 s per call) and 30.0% when served remotely on one A100 (0.41–0.44 s per call). The A100 call time includes network transfer, yet the result barely changes.

To isolate the effect of network delay, we injected a delay of 0, 1, 2 or 3 s (±50% jitter) after each model call of FT 2B and ran the tracking task at target speeds of 0.15, 0.3 and 0.6 m/s without pausing the simulator (20 runs per cell). Longer delays lower the success rate and raise the target-loss rate, and the effect is larger for faster targets. Without added delay, FT 2B still reaches 40% at 0.6 m/s, but with a 3 s delay this drops to 5% and the target is lost in every run.

Target speed0 s1 s2 s3 s
0.15 m/s45 (70)55 (70)45 (75)35 (80)
0.3 m/s25 (80)30 (75)20 (75)15 (80)
0.6 m/s40 (75)20 (85)15 (85)5 (100)

(SR %; in parentheses, the share of runs in which the target stayed out of view for more than 3 s in a row after first being seen)

The simulation-to-wall-clock ratio of Isaac Sim is already reported in Appendix G. Because the outdoor street scene (Brownstone) is large and runs with collision checking, Isaac Sim runs about 11 times slower than wall clock, while AirSim runs in real time (ratio 1.00) (revised: Appendix A.2).


How should the real flights be interpreted? (W3)

The real flights are designed to check whether the trends observed in simulation carry over to physical hardware, rather than to serve as a standalone benchmark. The real multi-drone flights use two drones instead of the four used in simulation, because each Tello acts as a Wi-Fi access point with a fixed IP address, and more than two drones interfered with each other. Even with two drones, the results follow the same trend as in simulation. As suggested, we tone down the real-world conclusions. We replace the claim in the abstract that the fine-tuned model outperforms frontier models on the real drone with the statement that it is comparable to them, and present the real flights as initial evidence that the trends observed in simulation carry over to the real drone, not as evidence of general real-world performance (revised: Abstract; Section 1; Section 5).

Regarding uncertainty, the overall success rates of the fine-tuned 2B (11/20, 95% interval 34–74%) and Gemini 3.7 Flash (9/20, 26–66%) have overlapping intervals and do not establish an advantage. Both models flew the same targets. By task, the fine-tuned 2B succeeds in 5 of 10 moving-target flights and Gemini 3.7 Flash in none, while Gemini 3.7 Flash leads on static targets (9/10 vs 6/10). This matches the simulation finding that slow models lose moving targets.


Does the commander advantage come from the sub-agent model or the inference budget? (Q1)

For the sub-agent model and the reasoning budget, please see the general response (Benefit of the commander structure and confounding factors), where the commander and the sub-agents use the same model and reasoning is set to the lowest level in every structure. Regarding the compute budget, the commander structure uses more model calls and tokens than multi-drone, since each sub-agent issues its own calls (per episode with Claude Opus 5: 234 vs 64 calls, 4.15 vs 1.89 USD). However, a larger budget alone does not explain the gain: the no-commander setting uses a similar budget (269 calls, $4.28) but succeeds less often than multi-drone (36.7 vs 46.7), whereas the commander structure reaches 70.0.


Which training data contributes to the fine-tuned model? (Q2)

To separate the contributions of the two data sources, we additionally evaluated a model trained on the general instruction data (MAmmoTH) alone, with the drone data removed and the same 400 steps and batch size. Moving toward the target comes from the general data, which alone matches the full mixture (OSR 32.5%, compared with 26.2% before fine-tuning). Declaring arrival, in contrast, comes from the drone data: the model trained on general data alone declares only 0.23 times per episode and reaches an SR of 1.9%, whereas the model trained with the drone data declares 12.3 times per episode and reaches 31.2%. However, we tested a single 10% mixing ratio, so this attribution is limited to the current setting.


We hope these answers address your concerns, and we would be happy to clarify anything further. Thank you again for your valuable time and feedback.

AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer again for the review. As the discussion period is nearing its end, we would like to kindly remind the reviewer of our responses. Our rebuttal addressed the questions raised in the review:

  • the term "generalist", which we removed from the title;
  • conditions that lead to failure (target speed, whether the target requires search, target size);
  • the latency breakdown and the network delay injection experiment;
  • the scope of the real-flight conclusions, which we toned down in the revision;
  • confounding factors in the commander structure;
  • the contribution of each training data source.

We would be happy to discuss anything that needs further clarification.

The authors’ response is generally thorough, and the additional experiments substantially strengthen the credibility of several conclusions. I keep my current rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment

We sincerely thank all reviewers for their detailed reviews and constructive comments on our paper. We are encouraged that the reviewers found the paper supported by a broad set of experiments that aid the interpretation of the results (1XYV, YWDH, hT8T), clear and easy to follow (HpSj, YWDH), and timely (HpSj), and that they found the multi-drone tasks interesting (HpSj). We have revised the paper based on these reviews. Below, we first address the points raised by multiple reviewers.


Benefit of the commander structure and confounding factors (1XYV Q1, YWDH W4 Q4, hT8T W2 Q3)

As the reviewers point out, the original comparison differed not only in the command structure but also in the sub-agent model and the reasoning setting. To separate these factors, we added controls for two models, Claude Opus 5 and Gemini 3.7 Flash. In each run the commander and the sub-agents use the same model, and reasoning is set to the lowest level each API allows in all conditions (off for Opus, zero budget for Flash).

StructureHow the four drones are controlledClaude Opus 5Gemini 3.7 Flash
Multi-droneOne model controls all four drones46.750.0
No commanderFour instances of the same model, one per drone, without communication36.750.0
Commander + same-model sub-agentsOne commander directs four sub-agents of the same model70.060.0

Even when the sub-agents use the same model as the commander, adding a commander raises the success rate (36.7 → 70.0 and 50.0 → 60.0) (revised: Section 4.2, "Commander + sub-agents beats multi-drone"; Appendix H, Table 17).


The term "generalist" in the title (1XYV W1, YWDH W6)

Our work aims to examine how far the general-purpose capabilities of MLLMs transfer directly to drone control as VLA agents across diverse settings: single- and multi-drone control, four tasks and three environments. However, as the reviewers point out, the term "generalist" may introduce a potential misunderstanding that our evaluation covers all VLA tasks for drones. We acknowledge this and have revised the title to "Evaluating Multimodal LLMs as Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching" (revised: title).


Latency (1XYV W2, YWDH W2 Q1 Q2)

We split the time the drone waits in each step, from capturing a frame to receiving the next command, into two parts. The first is frame capture and sending the action to the simulator. It takes 0.58–0.81 s regardless of the model, and sending the action (simulator RPC) is negligible at 0.3 ms (median). The second is the model call, which ranges from 0.61 s (FT 2B) to 3.46 s (Gemini 3.7 Flash) depending on the model. The total time from observation to command is shortest for FT 2B at 1.21 s and longest for Gemini 3.7 Flash at 4.19 s. The latency differences between models therefore come mostly from the model call.

ModelCapture + action transferModel call (inference + network)Total (observation → command)
Gemini 3.7 Flash0.73 s3.46 s4.19 s
Claude Opus 50.63 s2.91 s3.54 s
Gemini Robotics-ER 20.81 s1.98 s2.79 s
Qwen3.5-9B0.65 s1.40 s2.05 s
GPT-50.58 s1.37 s1.95 s
Qwen3.5-27B0.66 s0.99 s1.65 s
FT 2B0.60 s0.61 s1.21 s

We reran the track and search + track tasks with the simulator paused during inference and report these results separately from the real-time results (revised: Section 4.1, "Success rates"; Appendix E, Table 12). With the simulator paused, Gemini 3.7 Flash, the model with the highest latency, improves substantially on moving targets, from 45.0% to 87.5% SR, whereas FT 2B, the model with the lowest latency, barely changes (30.0% to 32.5%). Together, these results confirm that the performance drop of Gemini 3.7 Flash stems from latency.

AuthorsOpenReview ↗

General Comment by Authors

We thank all reviewers again for their time and effort. As the discussion period ends, we summarize the discussion. We are still waiting for a response from one reviewer. Reviewer HpSj has not replied to our additional response.

In the discussion period, we addressed the main concerns of 1XYV (3), YWDH (3) and hT8T (2), who kept their ratings. 1XYV noted that the additional experiments substantially strengthen the credibility of several conclusions. For YWDH, we addressed latency by rerunning the tasks with the simulator paused, and prompt sensitivity by evaluating with alternative prompts. For hT8T, we addressed the declaration criterion by measuring the first-declaration SR, and the commander comparison by running a same-model commander control. At YWDH's request, we also added the first-declaration SR and the false-declaration rate (FDR) to Table 1 (lines 271–292).

HpSj (1) raised four main concerns about whether the tasks can be solved with classical methods, the moving targets and the success criterion, collision avoidance, and the value of the benchmark over existing drone autonomy. We addressed all four in our rebuttal. In the follow-up, HpSj kept the original view that the tasks are too simple and can be solved with classical methods. We posted an additional response on both points, and we hope that it addresses HpSj's concerns.

  • For the first point, as the simple example routes in the main text may have suggested that all tasks take this form, we added Figure 7 (lines 757–773), a route on which the target climbs the stairs and is occluded by them.
  • For the second point, we implemented the yaw-then-straight strategy HpSj suggested, using YOLO-World and hand-written rules. On the single-drone tasks, it reached an SR of 18.8%, compared with 64.4% for Gemini 3.7 Flash, and in 47% of the runs it never produced a single detection. On the multi-drone tasks, it failed in all 30 episodes.

We believe that our work contributes to research on whether general-purpose MLLMs can complete drone missions from diverse instructions without task-specific effort, and that this question matters to the community. We will release the benchmark and continue to incorporate the remaining feedback. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.