- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes ACTIVE TARGET LOCALIZATION FOR AERIAL VISION-AND-LANGUAGE NAVIGATION, where a UAV must determine whether the accumulated visual context is sufficient to localize a language-described target or whether further observation is required. The authors propose a closed-loop dual-agent framework consisting of a Selective Target Localization Agent, which either predicts a target location or defers localization with a rationale, and a Rationale-Guided Exploration Agent, which uses the rationale and shared spatial context to select the next exploration action. New observations are integrated into a world-aligned spatial memory, allowing localization to be reassessed over time. The localization agent is trained using counterfactual sparse-view supervision. Experiments on CityNav report consistent improvements across seen and unseen splits, achieving 42.57% SR and 30.95% SPL on the original Test Unseen split.
Critical strengths
Strong experimental performance. The proposed method achieves substantial improvements over prior automated approaches across the seen and unseen CityNav splits. The gains are also observed on the revised CityNav dataset, suggesting that the improvements are not limited to a single evaluation protocol. The component ablations further demonstrate the contributions of selective supervision, rationale guidance, and spatial memory.
Effective CSS-based data construction. Counterfactual sparse-view supervision provides a systematic way to construct LOCALIZE/DEFER training samples by varying the observed regions while keeping the scene, instruction, and target fixed. It explicitly includes challenging cases in which the target is visible but the relevant relational or ordinal context remains incomplete. The selective-localization results and human-annotation evaluation indicate that CSS effectively reduces premature localization under partial observation.
Critical weaknesses
Unclear motivation: The author sums up the main problem as: whether to localize from the current visual context or continue observing the environment. However, this problem is mainly tackled by The Selective Target Localization Agent and doesn’t really explain the role of the second exploration agent, nor does it justify the need for a dual-agent framework.
Lack of Novelty: Although the paper presents a complete engineering pipeline and reports substantial improvements on CityNav, its methodological novelty appears limited. The paper's contributions lean more towards task-specific data construction and system integration, rather than new model structures, learning objectives, exploration algorithms, or feedback mechanisms. The most distinctive component, counterfactual sparse-view supervision, is closely tied to CityNav-specific annotations and a limited set of predefined contextual cues. So, the generalizability of the method needs to be discussed further.
Lacking detailed explanation:While the input-output interface between the two agents is formally defined, the actual interaction mechanism remains under-specified. In particular, it is unclear how a free-form deferral rationale is reliably translated into a spatial exploration action by a frozen VLM, and the presented ablations establish the usefulness of rationale content but not the necessity of the dual-agent decomposition. Finally, the framework may require multiple sequential VLM calls over high-resolution visual inputs, yet no inference latency, computational cost or memory consumption is reported.
Questions for authors
-
Is counterfactual sparse-view supervision (CSS) first introduced in this work, or is it related to existing partial-observation, masking, or selective-prediction methods? What is its main technical novelty?
-
Since CSS relies on CityNav-specific annotations and predefined contextual cues, can it generalize to other datasets and scenarios where such information is unavailable?
-
Could the authors clarify how the localization agent’s natural-language rationale is translated into an exploration action by the frozen VLM, including the prompts, output parsing, and handling of invalid or ambiguous outputs?
-
Is the second VLM agent necessary, or could it be replaced by a simpler heuristic or action-selection module?
-
How does the system handle failures, such as uninformative exploration, repeated
DEFERdecisions, or an incorrectLOCALIZEprediction? Is there any mechanism for detecting or recovering from these cases?
7 comments
Part 2/3
Comment 3: Translating deferral rationales into actions, prompts, and parsing
Response: The deferral rationale specifies what evidence is missing. The frozen exploration VLM uses the target instruction, deferral rationale, world-aligned RGB canvas with the UAV position marked, current UAV position and heading, and landmark names and contours to select the next movement direction.
Prompts. The explorer receives the instruction, deferral rationale, spatial canvas, UAV pose, and landmark priors. The prompt asks it to choose one of four heading-relative movements to acquire the missing evidence. For example, in Fig. 3 in the manuscript, the instruction refers to the square lawn beside Forbes Mellon Library. The library is visible, but the lawn extends beyond the observed region, preventing its shape from being determined. The localizer therefore outputs DEFER with the rationale "Lawn only partly visible; square shape unresolved." Along with the canvas and UAV pose, the relevant prompt content is:
Target description: The square green lawn in front of the right side of Forbes Mellon Library.
Named landmarks: Forbes Mellon Library.
Defer reason: Lawn only partly visible; square shape unresolved.
Choose the movement most likely to reveal the missing visual evidence.
Choose exactly one action from {forward, backward, left, right}.
Return only JSON: {"direction":"forward|backward|left|right","reason":""}
Output parsing and execution. In this example, the explorer selects RIGHT to obtain a more complete view of the lawn. In the required JSON format, this action is represented as:
{"direction":"right","reason":"Obtain a better view of the lawn."}
The parser removes Markdown fences, parses the JSON, and reads direction. The system validates right and executes the corresponding movement. The new observation is added to spatial memory, after which the localizer predicts the lawn's bounding box. Only direction controls movement; reason is explanatory text. We use greedy decoding with at most 256 generated tokens. If JSON parsing fails, the parser attempts to recover an action explicitly associated with the direction key.
Invalid or ambiguous outputs. Capitalization and supported directional aliases are normalized. A value containing multiple actions, such as left or right, or an unsupported direction is rejected. If no valid action can be recovered, or the movement would cross the map boundary, the system randomly selects a legal action toward unobserved space. Such fallbacks occur in fewer than 1% of exploration steps in our evaluation. We will include the complete prompt, coordinate conventions, and parsing code in Appendix E.
Part 3/3
Comment 4: Necessity of the second VLM agent vs. simpler heuristics
Response: Our two-agent design separates deciding whether the target can be identified from deciding where to acquire missing evidence. The localization agent predicts a target box or outputs DEFER with a rationale describing what is missing, while the exploration agent uses this rationale and the current spatial context to select the next movement. To test whether the second VLM can be replaced by a simpler module, we compare three heuristic strategies while keeping the localization agent and spatial memory fixed. The strongest alternative, which uses rationales and landmark geometry, achieves 36.67% Test Unseen SR, compared with 42.57% for the exploration VLM (Table R8), supporting its contribution to selecting useful observations.
Specifically, we compare three simple rules defined for this experiment. The first randomly selects one of the four movement directions. The second checks neighboring cells in the order forward, right, backward, and left, and moves to the first unvisited cell; if all have been visited, it selects a direction randomly. The third reads the deferral rationale and either moves toward the unobserved part of a referenced landmark using its known contour or follows an explicitly stated direction. If neither rule applies, it uses the same fixed order. All three use only information available to our framework. We also test the exploration VLM without deferral rationales. When a rationale describes an incomplete vehicle row without specifying a direction, the rules provide no guidance beyond the fixed order, whereas the VLM can use the visible row layout to choose a movement. The results support using the VLM to select actions from both the current image and the deferral rationale. The exploration VLM remains frozen throughout. We will include these rule definitions and comparison results in the revision.
Table R8: Comparison of exploration strategies on the original CityNav Test Unseen split. The localization agent and spatial memory are unchanged across variants.
| Exploration strategy | SR (%) | SPL (%) |
|---|---|---|
| Random movement | 34.83 | 25.57 |
| Fixed-order scan | 35.85 | 27.39 |
| Rationale-aware rule | 36.67 | 27.71 |
| VLM without rationale | 37.09 | 27.95 |
| Rationale-guided VLM (Ours) | 42.57 | 30.95 |
Comment 5: Handling ineffective exploration, repeated DEFER, and wrong LOCALIZE
Response: The current framework does not include a separate mechanism to detect and recover from these errors. After each exploration movement, it updates spatial memory and reassesses localization using the accumulated observations. If the localizer still outputs DEFER, its new rationale guides the next movement, allowing the search direction to change without explicitly detecting whether the previous action was effective. Repeated deferral is bounded by five exploration movements; if no target box is produced, the UAV stops at its current position. Counterfactual sparse-view supervision helps reduce premature localization: on views labeled DEFER, it reduces LOCALIZE predictions from 37.79% for bbox-only SFT to 7.23% (manuscript Table 2). This reduces such errors but does not verify individual predictions. Once the localizer outputs a target box, the system uses it as the terminal navigation goal without an additional verification or recovery step. We will clarify these behaviors and limitations in the revision.
Part 1/3
Comment 1: Novelty of CSS and relation to masking / selective-prediction work
Response: We introduce counterfactual sparse-view supervision in this work to train the localizer to determine whether the observed context supports target identification and explain what information is missing. It builds on existing masking and selective-prediction methods [1–3].
Our specific contribution is to vary the observed region while keeping the scene, instruction, and target fixed, including cases where the target remains visible but instruction-relevant landmarks or structural cues are missing. These examples jointly supervise target localization, deferral, and explanations of missing context. The explanations then guide physical exploration, allowing localization to be reassessed after acquiring additional observations. This supervision supports the paper's broader contribution: a closed-loop two-agent framework that connects selective localization with targeted evidence acquisition and retains spatial context across observations. With identical sparse training views, supervision to defer under insufficient context reduces false localization from 60.89% to 32.78%, while adding rationale supervision further raises navigation SR from 36.28% to 42.57% (Table R6). The full framework also improves performance across seen and unseen splits of both the original and revised CityNav benchmarks (manuscript Table 1). We will clarify these contributions and their relationship to prior work in the revision.
Table R6: Ablation of supervision components on identical sparse training views (CityNav Test Unseen).
[1] Long Chen, Xin Yan, Jun Xiao, et al. Counterfactual Samples Synthesizing for Robust Visual Question Answering. In CVPR, pp. 10797–10806, 2020.
[2] Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, et al. Improving Selective Visual Question Answering by Learning From Your Peers. In CVPR, pp. 24049–24059, 2023.
[3] Tejas Srinivasan, Jack Hessel, Tanmay Gupta, et al. Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning. In Findings of ACL, pp. 12935–12948, 2024.
Comment 2: Generalization of CSS without CityNav-specific annotations
Response: Counterfactual sparse-view supervision can be adapted without CityNav's landmark annotations or predefined contextual cues. We verify this on UAV-ON [4], where it increases SR from 7.10% to 8.20% and reduces DTS from 49.44 m to 44.77 m across all 1,000 official test tasks in 14 scenes (Table R7).
Specifically, we construct training examples from target annotations, descriptions, and 5,099 images collected exclusively from UAV-ON training environments along trajectories with changes in position, altitude, and viewpoint. Qwen3-VL-8B-Instruct helps select observations where target detection changes from failure to success and generate explanations of missing evidence. The supervision distinguishes three cases: the target is unobserved, a candidate is visible but cannot yet be identified, or sufficient evidence is available for identification. The first two receive
DEFERsupervision with a rationale, while the third receives target-box supervision. We fine-tune only the Qwen3-VL-2B-Instruct localization agent for one epoch with LoRA rank 2, leaving the exploration agent frozen. The comparison without fine-tuning uses the same backbone, observations, and execution procedure. Thus, the experiment shows that we can construct and use the supervision from observations in another dataset without CityNav-specific contextual annotations. We will provide the complete data construction procedure, prompts, and examples in the revised appendix.Table R7: Navigation results on all 1,000 official UAV-ON test tasks. Both methods use the same framework. Our method fine-tunes only the localization agent using UAV-ON training observations without CityNav-specific contextual annotations.
[4] Jianqiang Xiao, Yuexuan Sun, Yixin Shao, et al. UAV-ON: A Benchmark for Open-World Object Goal Navigation with Aerial Agents. In Proceedings of the 33rd ACM International Conference on Multimedia (ACM MM), pp. 13023–13029, 2025.