- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DroneCATS, a benchmark for evaluating multimodal models on drone approaching, searching, tracking, and multi-drone coordination through a shared four-action interface. It compares direct control with a commander-and-sub-agents structure and evaluates models in two simulators and real flights. It also fine-tunes a 2B model using action data generated from public RGB-D images and reports improved navigation success.
Critical strengths
1: The benchmark addresses a useful evaluation problem. Bringing target search, target motion, and multi-drone coordination into one framework allows comparisons across settings that are often studied separately.
2: The experimental coverage is substantial. The paper includes two simulators, 118 real-flight episodes, and useful ablations on visual history, thinking, maps, and command structure. These provide a useful basis for studying the strengths and limitations of different models.
Critical weaknesses
1: The arrival evaluation does not clearly support the paper’s interpretation of model behavior. Incorrect completion declarations do not end an episode, and any later qualifying declaration counts as success. Table 9 reports an average of 39.4 declarations per episode for Qwen3.5-27B, so successful completion may still involve many incorrect judgments. The prompt also treats empty margins around the target as evidence that it is over 10 metres away. This rule is unreliable across target sizes, including the egg used in real flights, and does not match the distance-based success criterion. These issues make it difficult to attribute failure to declare arrival to poor instruction following.
2: The main comparative conclusions need stronger support. The hierarchical setup changes the pilot models, map information, and reasoning settings alongside the command structure. The appendix provides useful controls, but the overall gain remains difficult to attribute to organization alone. The real-flight advantage is also based on only 11/20 successful flights for the fine-tuned model versus 9/20 for Gemini. These results support initial effectiveness, but not a stable advantage over frontier models. The paper should distinguish these findings from broader claims and make the benchmark’s added value over existing work clearer, particularly in the abstract.
Questions for authors
1: Arrival evaluation: What are the first-declaration success rate and false-declaration rate for each model? How would the rankings change if the first completion declaration ended the episode?
2: Prompt sensitivity: Does the reported declaration failure persist when the prompt states the actual success threshold and removes the frame-filling rule? Please also distinguish episodes that satisfy both distance and visibility requirements from those that only enter the distance threshold.
3: Multi-drone comparison: How much of the hierarchical advantage remains when the same model is used across roles, both structures receive the same map information, and reasoning settings are matched? Please also report the differences in model-call counts and computational costs.
4: Real-flight evidence: Can you provide uncertainty estimates or additional paired trials for the reported 55% versus 45% success rates? This would help establish how much weight to place on the claimed advantage.
5: Benchmark contribution: Beyond broader task coverage, which findings does DroneCATS enable that existing evaluations such as SPF and MissionBench cannot establish?
4 comments
Official Comment by Authors (2/2)
How much of the hierarchical advantage remains under matched settings, and at what cost? (W2, Q3)
For the same-model and matched-reasoning controls, please see the general response (Benefit of the commander structure and confounding factors). Regarding the map usage, we separately compared against commanders run without the map in the map ablation of the appendix (with ER2 sub-agents). Even without the map, the commander structure outperforms multi-drone (73.3 vs 50.0 for Gemini 3.7 Flash, 80.0 vs 46.7 for Claude Opus 5).
We report model calls and cost as per-episode averages on the Three-sign sum task. Because each of the four sub-agents issues its own calls, the commander structure requires more calls and tokens than multi-drone. Compared with the no-commander setting, it requires fewer calls at a similar cost while achieving a higher success rate.
| Structure | How the four drones are controlled | Calls (Opus / Flash) | Tokens (Opus / Flash) | Cost (Opus) |
|---|---|---|---|---|
| Multi-drone | One model controls all four drones | 64 / 34 | 342k / 233k | $1.89 |
| No commander | Four instances of the same model, one per drone, without communication | 269 / 192 | 726k / 517k | $4.28 |
| Commander + same-model sub-agents | One commander directs four sub-agents of the same model | 234 / 172 | 738k / 778k | $4.15 |
Does 11/20 versus 9/20 show an advantage on the real drone? (W2, Q4)
We thank the reviewer for this point. The statement that the fine-tuned model outperforms frontier models in real flight could mislead readers, since the overall gap is small: 11/20 (95% interval 34–74%) versus 9/20 (26–66%). We therefore clarify it in the revision. The overall numbers, however, hide a clear difference by task. Both models flew the same targets, and the fine-tuned 2B succeeds in 5 of 10 moving-target flights and Gemini 3.7 Flash in none, while Gemini 3.7 Flash leads on static targets (9/10 vs 6/10). This matches the simulation finding that slow models lose moving targets. In the revision, the abstract and introduction state that the fine-tuned model is comparable to frontier models on the real drone (revised: Abstract; Section 1; Section 5).
What can DroneCATS show that SPF or MissionBench cannot? (W2, Q5)
DroneCATS establishes three findings that existing evaluations cannot. First, whether a model judges its own arrival. SPF ends an episode with an external rule and therefore cannot measure this, whereas in DroneCATS this judgment is the main point of failure. Second, the effect of inference latency. Only a closed-loop evaluation in which the target keeps moving during inference reveals that slow frontier models lose moving targets. When the agent design of MissionBench is transplanted onto our benchmark, it enters δ in only 3 of 20 moving-target episodes and succeeds in none (Table 20). This latency effect also appears on the real drone, flown with the same prompt and actions. Third, the organization of multiple drones. Unlike existing evaluations that cover a single drone, we find that a commander directing sub-agents achieves higher success than a single model controlling all four drones directly. In the revision, we state these findings in the abstract and the introduction (revised: Abstract; Section 1).
We hope these answers address your concerns, and we would be happy to clarify anything further. Thank you again for your valuable time and feedback.
Thank you for the detailed response. Most of my questions have been addressed, but I still have some concerns about the declaration-based success criterion, the additional cost of the hierarchical setup, the limited real-flight evidence, and the benchmark novelty over existing work. For now, I am inclined to keep my current rating.
Official Comment by Authors
We sincerely thank the reviewer for the follow-up and for the time spent on our responses. Regarding the remaining concerns about the declaration-based success criterion, the additional cost of the hierarchical setup, the real-flight evidence, and the novelty over existing work, we have tried to address each of these points in detail in our previous responses. We would be very grateful if the reviewer could let us know which aspects still seem insufficient, and we would be glad to clarify any specific point further during the remaining discussion period.
Official Comment by Authors (1/2)
We thank the reviewer for the insightful questions, which helped us clarify the evaluation protocol. Our point-by-point responses follow.
What are the first-declaration SR and false-declaration rate? (W1, Q1)
The table reports the first-declaration SR, which assumes that the episode ends at the first completion declaration, and the false-declaration rate, the share of all declarations that do not meet the success condition.
The four frontier models remain at the top under the first-declaration criterion. By design, an incorrect declaration does not end the episode: what we evaluate is whether the drone eventually reaches the target and confirms its arrival, not whether its first declaration is correct. This also lets the drone keep following a moving target after declaring, and lets a commander redirect a sub-agent in the multi-drone setting, so one rule covers both settings. For Qwen3.5-27B and FT 2B, however, most successes follow many incorrect declarations, which the first-declaration SR and the false-declaration rate make visible. In the revision, we explain in the main text why the episode does not end at the first declaration, and report the first-declaration SR and false-declaration rate (revised: Section 3.1, "Metrics"; Section 4.1, "Recognition and declaration"; Appendix B, Table 5).
Does the failure persist without the margin rule? (W1, Q2)
To test whether the declaration failure is caused by the specific prompt, and in particular by the margin rule you pointed out, we reran three models whose failures come from declaring: Qwen3.5-9B, which rarely declares, and Qwen3.5-27B and FT 2B, which over-declare. Prompt A removes the margin rule and states the actual value of δ. Prompt B adds two few-shot examples to A.
The declaration failure persists under both prompts. The mean SR of the three models remains nearly unchanged, at 30.6%, 32.1% and 29.4%. Only Qwen3.5-27B improves, with prompt A, while Qwen3.5-9B and FT 2B decline slightly. The declaration failure therefore stems not from the margin rule or a particular prompt, but from the models' ability to follow the declaration rule in the system prompt.
Nor is this a perception failure. We placed the logged drone and target poses of every step of the paper runs back in the simulator and checked whether the target was visible. Qwen3.5-9B entered δ in 63.1% of episodes by distance alone and in 51.9% with the target also in view, yet succeeded in only 21.2%. It reached the target and saw it, but failed to declare arrival.
Are distance and visibility distinguished? (Q2)
To distinguish episodes that only enter the distance threshold from those that satisfy both the distance and visibility requirements, Table 1 reports OSR and SR together. OSR is the share of episodes in which the drone enters δ, and SR is the share in which the drone is inside δ with the target in view at the time of declaration.