acceptodds
Under review as a conference paper at ICLR 2027

Look Before You Localize: Active Target Localization for Aerial Vision-and-Language Navigation

Abstract

In language-goal aerial navigation, a UAV progressively acquires observations to localize a target described in a language instruction by its visual attributes and relations to surrounding landmarks. A target may already be visible while the relational, ordinal, or landmark context needed to distinguish it from plausible alternatives remains outside the observed region. This creates an important decision during navigation: whether to localize from the current visual context or continue observing the environment. We propose a closed-loop active target localization framework that makes this decision explicit through two complementary agents. The Selective Target Localization Agent either predicts a target location or defers localization based on the accumulated visual context, while providing a rationale associated with the deferral. The Rationale-Guided Exploration Agent conditions on this rationale and the shared spatial context to select the next exploratory movement. New observations update a world-aligned spatial memory, allowing localization to be reassessed using the retained visual context. Experiments on CityNav show consistent improvements across seen and unseen splits. On the original Test Unseen split, our method achieves 42.57% SR and 30.95% SPL, exceeding the best compared automated SR and SPL by 16.67% and 11.32%, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-and-Language Navigation 、UAV Navigation、Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8mP7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes ACTIVE TARGET LOCALIZATION FOR AERIAL VISION-AND-LANGUAGE NAVIGATION, where a UAV must determine whether the accumulated visual context is sufficient to localize a language-described target or whether further observation is required. The authors propose a closed-loop dual-agent framework consisting of a Selective Target Localization Agent, which either predicts a target location or defers localization with a rationale, and a Rationale-Guided Exploration Agent, which uses the rationale and shared spatial context to select the next exploration action. New observations are integrated into a world-aligned spatial memory, allowing localization to be reassessed over time. The localization agent is trained using counterfactual sparse-view supervision. Experiments on CityNav report consistent improvements across seen and unseen splits, achieving 42.57% SR and 30.95% SPL on the original Test Unseen split.

Critical strengths

Strong experimental performance. The proposed method achieves substantial improvements over prior automated approaches across the seen and unseen CityNav splits. The gains are also observed on the revised CityNav dataset, suggesting that the improvements are not limited to a single evaluation protocol. The component ablations further demonstrate the contributions of selective supervision, rationale guidance, and spatial memory.

Effective CSS-based data construction. Counterfactual sparse-view supervision provides a systematic way to construct LOCALIZE/DEFER training samples by varying the observed regions while keeping the scene, instruction, and target fixed. It explicitly includes challenging cases in which the target is visible but the relevant relational or ordinal context remains incomplete. The selective-localization results and human-annotation evaluation indicate that CSS effectively reduces premature localization under partial observation.

Critical weaknesses

Unclear motivation: The author sums up the main problem as: whether to localize from the current visual context or continue observing the environment. However, this problem is mainly tackled by The Selective Target Localization Agent and doesn’t really explain the role of the second exploration agent, nor does it justify the need for a dual-agent framework.

Lack of Novelty: Although the paper presents a complete engineering pipeline and reports substantial improvements on CityNav, its methodological novelty appears limited. The paper's contributions lean more towards task-specific data construction and system integration, rather than new model structures, learning objectives, exploration algorithms, or feedback mechanisms. The most distinctive component, counterfactual sparse-view supervision, is closely tied to CityNav-specific annotations and a limited set of predefined contextual cues. So, the generalizability of the method needs to be discussed further.

Lacking detailed explanation:While the input-output interface between the two agents is formally defined, the actual interaction mechanism remains under-specified. In particular, it is unclear how a free-form deferral rationale is reliably translated into a spatial exploration action by a frozen VLM, and the presented ablations establish the usefulness of rationale content but not the necessity of the dual-agent decomposition. Finally, the framework may require multiple sequential VLM calls over high-resolution visual inputs, yet no inference latency, computational cost or memory consumption is reported.

Questions for authors
  1. Is counterfactual sparse-view supervision (CSS) first introduced in this work, or is it related to existing partial-observation, masking, or selective-prediction methods? What is its main technical novelty?

  2. Since CSS relies on CityNav-specific annotations and predefined contextual cues, can it generalize to other datasets and scenarios where such information is unavailable?

  3. Could the authors clarify how the localization agent’s natural-language rationale is translated into an exploration action by the frozen VLM, including the prompts, output parsing, and handling of invalid or ambiguous outputs?

  4. Is the second VLM agent necessary, or could it be replaced by a simpler heuristic or action-selection module?

  5. How does the system handle failures, such as uninformative exploration, repeated DEFER decisions, or an incorrect LOCALIZE prediction? Is there any mechanism for detecting or recovering from these cases?

7 comments

Part 1/3

Comment 1: Novelty of CSS and relation to masking / selective-prediction work

Response: We introduce counterfactual sparse-view supervision in this work to train the localizer to determine whether the observed context supports target identification and explain what information is missing. It builds on existing masking and selective-prediction methods [1–3].

Our specific contribution is to vary the observed region while keeping the scene, instruction, and target fixed, including cases where the target remains visible but instruction-relevant landmarks or structural cues are missing. These examples jointly supervise target localization, deferral, and explanations of missing context. The explanations then guide physical exploration, allowing localization to be reassessed after acquiring additional observations. This supervision supports the paper's broader contribution: a closed-loop two-agent framework that connects selective localization with targeted evidence acquisition and retains spatial context across observations. With identical sparse training views, supervision to defer under insufficient context reduces false localization from 60.89% to 32.78%, while adding rationale supervision further raises navigation SR from 36.28% to 42.57% (Table R6). The full framework also improves performance across seen and unseen splits of both the original and revised CityNav benchmarks (manuscript Table 1). We will clarify these contributions and their relationship to prior work in the revision.

Table R6: Ablation of supervision components on identical sparse training views (CityNav Test Unseen).

Training supervisionFalse localization (%) ↓\downarrowNav. SR (%) ↑\uparrow
Target visibility only60.8935.38
Context sufficiency32.7836.28
Context sufficiency with rationales (Ours)20.5842.57

[1] Long Chen, Xin Yan, Jun Xiao, et al. Counterfactual Samples Synthesizing for Robust Visual Question Answering. In CVPR, pp. 10797–10806, 2020.

[2] Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, et al. Improving Selective Visual Question Answering by Learning From Your Peers. In CVPR, pp. 24049–24059, 2023.

[3] Tejas Srinivasan, Jack Hessel, Tanmay Gupta, et al. Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning. In Findings of ACL, pp. 12935–12948, 2024.

Comment 2: Generalization of CSS without CityNav-specific annotations

Response: Counterfactual sparse-view supervision can be adapted without CityNav's landmark annotations or predefined contextual cues. We verify this on UAV-ON [4], where it increases SR from 7.10% to 8.20% and reduces DTS from 49.44 m to 44.77 m across all 1,000 official test tasks in 14 scenes (Table R7).

Specifically, we construct training examples from target annotations, descriptions, and 5,099 images collected exclusively from UAV-ON training environments along trajectories with changes in position, altitude, and viewpoint. Qwen3-VL-8B-Instruct helps select observations where target detection changes from failure to success and generate explanations of missing evidence. The supervision distinguishes three cases: the target is unobserved, a candidate is visible but cannot yet be identified, or sufficient evidence is available for identification. The first two receive DEFER supervision with a rationale, while the third receives target-box supervision. We fine-tune only the Qwen3-VL-2B-Instruct localization agent for one epoch with LoRA rank 2, leaving the exploration agent frozen. The comparison without fine-tuning uses the same backbone, observations, and execution procedure. Thus, the experiment shows that we can construct and use the supervision from observations in another dataset without CityNav-specific contextual annotations. We will provide the complete data construction procedure, prompts, and examples in the revised appendix.

Table R7: Navigation results on all 1,000 official UAV-ON test tasks. Both methods use the same framework. Our method fine-tunes only the localization agent using UAV-ON training observations without CityNav-specific contextual annotations.

MethodDTS (m) ↓\downarrowSR (%) ↑\uparrowOSR (%) ↑\uparrowSPL (%) ↑\uparrow
Base VLM49.447.1015.944.02
Ours44.778.2018.374.34

[4] Jianqiang Xiao, Yuexuan Sun, Yixin Shao, et al. UAV-ON: A Benchmark for Open-World Object Goal Navigation with Aerial Agents. In Proceedings of the 33rd ACM International Conference on Multimedia (ACM MM), pp. 13023–13029, 2025.

Part 2/3

Comment 3: Translating deferral rationales into actions, prompts, and parsing

Response: The deferral rationale specifies what evidence is missing. The frozen exploration VLM uses the target instruction, deferral rationale, world-aligned RGB canvas with the UAV position marked, current UAV position and heading, and landmark names and contours to select the next movement direction.

Prompts. The explorer receives the instruction, deferral rationale, spatial canvas, UAV pose, and landmark priors. The prompt asks it to choose one of four heading-relative movements to acquire the missing evidence. For example, in Fig. 3 in the manuscript, the instruction refers to the square lawn beside Forbes Mellon Library. The library is visible, but the lawn extends beyond the observed region, preventing its shape from being determined. The localizer therefore outputs DEFER with the rationale "Lawn only partly visible; square shape unresolved." Along with the canvas and UAV pose, the relevant prompt content is:

Target description: The square green lawn in front of the right side of Forbes Mellon Library.
Named landmarks: Forbes Mellon Library.
Defer reason: Lawn only partly visible; square shape unresolved.
Choose the movement most likely to reveal the missing visual evidence.
Choose exactly one action from {forward, backward, left, right}.
Return only JSON: {"direction":"forward|backward|left|right","reason":""}

Output parsing and execution. In this example, the explorer selects RIGHT to obtain a more complete view of the lawn. In the required JSON format, this action is represented as:

{"direction":"right","reason":"Obtain a better view of the lawn."}

The parser removes Markdown fences, parses the JSON, and reads direction. The system validates right and executes the corresponding movement. The new observation is added to spatial memory, after which the localizer predicts the lawn's bounding box. Only direction controls movement; reason is explanatory text. We use greedy decoding with at most 256 generated tokens. If JSON parsing fails, the parser attempts to recover an action explicitly associated with the direction key.

Invalid or ambiguous outputs. Capitalization and supported directional aliases are normalized. A value containing multiple actions, such as left or right, or an unsupported direction is rejected. If no valid action can be recovered, or the movement would cross the map boundary, the system randomly selects a legal action toward unobserved space. Such fallbacks occur in fewer than 1% of exploration steps in our evaluation. We will include the complete prompt, coordinate conventions, and parsing code in Appendix E.

Part 3/3

Comment 4: Necessity of the second VLM agent vs. simpler heuristics

Response: Our two-agent design separates deciding whether the target can be identified from deciding where to acquire missing evidence. The localization agent predicts a target box or outputs DEFER with a rationale describing what is missing, while the exploration agent uses this rationale and the current spatial context to select the next movement. To test whether the second VLM can be replaced by a simpler module, we compare three heuristic strategies while keeping the localization agent and spatial memory fixed. The strongest alternative, which uses rationales and landmark geometry, achieves 36.67% Test Unseen SR, compared with 42.57% for the exploration VLM (Table R8), supporting its contribution to selecting useful observations.

Specifically, we compare three simple rules defined for this experiment. The first randomly selects one of the four movement directions. The second checks neighboring cells in the order forward, right, backward, and left, and moves to the first unvisited cell; if all have been visited, it selects a direction randomly. The third reads the deferral rationale and either moves toward the unobserved part of a referenced landmark using its known contour or follows an explicitly stated direction. If neither rule applies, it uses the same fixed order. All three use only information available to our framework. We also test the exploration VLM without deferral rationales. When a rationale describes an incomplete vehicle row without specifying a direction, the rules provide no guidance beyond the fixed order, whereas the VLM can use the visible row layout to choose a movement. The results support using the VLM to select actions from both the current image and the deferral rationale. The exploration VLM remains frozen throughout. We will include these rule definitions and comparison results in the revision.

Table R8: Comparison of exploration strategies on the original CityNav Test Unseen split. The localization agent and spatial memory are unchanged across variants.

Exploration strategySR (%) ↑\uparrowSPL (%) ↑\uparrow
Random movement34.8325.57
Fixed-order scan35.8527.39
Rationale-aware rule36.6727.71
VLM without rationale37.0927.95
Rationale-guided VLM (Ours)42.5730.95

Comment 5: Handling ineffective exploration, repeated DEFER, and wrong LOCALIZE

Response: The current framework does not include a separate mechanism to detect and recover from these errors. After each exploration movement, it updates spatial memory and reassesses localization using the accumulated observations. If the localizer still outputs DEFER, its new rationale guides the next movement, allowing the search direction to change without explicitly detecting whether the previous action was effective. Repeated deferral is bounded by five exploration movements; if no target box is produced, the UAV stops at its current position. Counterfactual sparse-view supervision helps reduce premature localization: on views labeled DEFER, it reduces LOCALIZE predictions from 37.79% for bbox-only SFT to 7.23% (manuscript Table 2). This reduces such errors but does not verify individual predictions. Once the localizer outputs a target box, the system uses it as the terminal navigation goal without an additional verification or recovery step. We will clarify these behaviors and limitations in the revision.

Reviewer_YyhcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies language-goal-driven UAV navigation and focuses on one specific problem: the target may already be in view, but the landmark, spatial-relation, or ordering information needed to check it against the instruction is still incomplete. Localizing the target at this point can lead to committing too early to the wrong candidate. To address this, the authors propose a closed-loop system with two agents. The Selective Target Localization Agent uses the instruction and the accumulated visual context to either output a target bounding box or output DEFER with a rationale. The Rationale-Guided Exploration Agent then picks the next viewing direction based on that rationale, spatial memory, the UAV pose, and landmark priors. Each new observation is registered onto a shared world-coordinate canvas, and the localization agent decides again. When it outputs a bounding box, the system converts the box center into world coordinates, flies to that location, and stops.

Critical strengths
  1. The gains in exploration efficiency and success rate come from rationale content tied to the current task. Performance drops clearly when the rationale is replaced with a generic one or shuffled.
  2. The gains in exploration efficiency and success rate come from rationale content tied to the current task. Performance drops clearly when the rationale is replaced with a generic one or shuffled.
  3. The method outperforms the strongest existing baselines by a wide margin. It leads consistently on both the original and corrected CityNav, across seen and unseen splits.
Critical weaknesses
  1. CSS does not show that the model identifies missing semantic evidence: In both training and offline evaluation, the visible area is strongly correlated with the DEFER label. The model may simply learn to defer when little of the scene is visible, without actually determining which piece of context the instruction still lacks.

  2. The Selective Supervision ablation cannot isolate the source of CSS's gains: This comparison changes the input distribution (sparse views to complete views) and the supervision format (LOCALIZE/DEFER with rationales to bbox-only) at the same time. It is therefore unclear whether the improvement comes from the context-sufficiency supervision or from a closer match between training and inference inputs.

  3. The paper evaluates on CityNav with orthographic top-down observations and fixed-step planar movement (§3.4, Appendix E). That setup can't show whether the method works for complex 3D UAV navigation. Experiments on UAV-ON would help test whether it stays effective under changes in altitude and viewpoint, occlusion, and collision constraints.

[1] Uav-on: A benchmark for open-world object goal navigation with aerial agents,ACMMM 2025.

Questions for authors

see weakness

2 comments

Part 1/2

Comment 1: Whether CSS recognizes missing semantic evidence beyond visible area

Response: Our additional experiments support that the localizer's decisions depend on the information required by the instruction, beyond the amount of visible area. We first hold the observation and target fixed and change only the instruction, then verify the required decisions using human judgments. We also compare context-sufficient and context-insufficient views after controlling their visible-area distributions. Our method outperforms bbox-only SFT in all three comparisons.

Changing only the instruction. We test whether the model changes its decision when the same observation provides enough information for one instruction but not another. Each pair contains the same image and physical target, together with two existing human instructions describing that target. For example, one instruction may identify a house through an already observed landmark, while another requires a sequence of buildings that is not yet fully observed. We select pairs whose automatically generated labels specify LOCALIZE for one instruction and DEFER for the other. A deterministic rule based only on visible area would give the same decision to both and therefore cannot correctly classify an entire pair. The resulting set contains 1,324 pairs covering 711 targets, with at most two pairs per target. A pair is counted as correct only when both decisions are correct. Our method achieves 37.99%, compared with 17.45% for bbox-only SFT (Table R2).

Verification using human judgments. We check whether this improvement persists when the required decisions are determined by human judgments rather than automatic labels. We sample 100 pairs covering unobserved landmarks, incomplete landmark relations, and incomplete groups of similar objects. Annotators judge each image and instruction individually in random order, without seeing automatic labels, model predictions, or which examples form a pair. They judge that 42 pairs require opposite decisions, 47 require LOCALIZE for both instructions, and 11 require DEFER for both. This disagreement indicates that the automatic labels are imperfect proxies for whether the available information is sufficient. On the 42 pairs requiring opposite decisions according to human judgments, our method correctly classifies 17 pairs, compared with 8 for bbox-only SFT. These results support instruction-dependent decisions on the independently confirmed subset.

Table R2: Percentage of instruction pairs for which both localization/deferral decisions are correct. Each pair contains the same observation and target but different instructions.

Evaluation setbbox-only SFTOurs
All 1,324 pairs17.45%37.99%
42 pairs confirmed by human judgments19.05% (8/42)40.48% (17/42)

Controlling the visible area. We further test whether the model can decide if the target is identifiable when a similar amount of the scene is visible, but the information required by the instruction is either present or missing. For example, two views may both contain the target house, but only one shows the library needed to identify it as "the house next to the library." Although similar areas are visible, the model should localize in one case and defer in the other. To test this, we group target-visible views with similar visible areas and select equal numbers with sufficient and insufficient information from each group. This produces 3,130 Test Unseen views, with 1,565 of each type. All models use the same inference prompt. Our method improves decision Macro-F1 from 49.31% to 68.46% and reduces incorrect localization when information is insufficient from 71.50% to 20.58% (Table R3). These results indicate that visible area alone does not explain its decisions, although it also defers more often when sufficient information is available.

Table R3: Localization and deferral results after controlling the visible-area distributions. All values are percentages. Metric definitions follow Table 2 of the manuscript.

ModelMacro-F1 ↑\uparrowFLR2\mathrm{FLR}_{2} ↓\downarrowCond. Acc. ↑\uparrowUDR ↓\downarrow
Base VLM43.7647.4836.4464.22
bbox-only SFT49.3171.5063.8324.22
Ours68.4620.5866.6341.79

Together, these experiments support that the localization/deferral decisions respond to instruction-specific information requirements beyond the visible-area statistics considered here. We will include the experiments and their evaluation procedures in the revision.

Part 2/2

Comment 2: Isolating CSS supervision gains from sparse-view distribution matching

Response: To distinguish the effects of supervision from those of training on sparse views, we conduct an additional ablation using identical sparse training views, model initialization, sample weights, and training budget. We first compare training the model to localize whenever the target is visible with training it to defer when the target is visible but the required context is missing. We then test the effect of adding deferral rationale supervision. All variants are trained to defer when the target is unobserved. Table R4 shows that teaching the model to defer under insufficient context reduces false localization from 60.89% to 32.78%, although unnecessary deferral increases and the navigation gain is modest. Adding rationale supervision further improves localization/deferral decisions and raises navigation SR from 36.28% to 42.57%. These results show that both supervision components contribute beyond training on sparse views alone. We will replace the original ablation with this comparison in the revision.

Table R4: Effects of supervision on identical sparse training views. All values are percentages. Offline metrics use the 3,130 Test Unseen views with visible area controlled as described above; navigation metrics use the standard Test Unseen split.

Training supervisionMacro-F1 ↑\uparrowFLR2\mathrm{FLR}_{2} ↓\downarrowUDR ↓\downarrowNav. SR ↑\uparrowNav. SPL ↑\uparrow
Target visibility only58.2860.8918.8535.3828.10
Context sufficiency61.6432.7843.7136.2827.76
Context sufficiency with rationales (Ours)68.4620.5841.7942.5730.95

Comment 3: Effectiveness in complex 3D UAV navigation (UAV-ON)

Response: To test our framework in 3D UAV navigation, we adapt it to UAV-ON and evaluate all 1,000 official test tasks across 14 scenes. We replace CityNav's top-down canvas with front, left, right, and downward RGB-D views. The localizer either predicts a target box, which is converted to a 3D goal using depth and camera parameters, or defers with a rationale that guides further exploration. We construct training examples from 5,099 images collected exclusively in UAV-ON training environments, using Qwen3-VL-8B-Instruct to assist in generating deferral rationales. We compare the same Qwen3-VL-2B-Instruct framework with and without fine-tuning the localizer, keeping the observations, exploration agent, and execution procedure unchanged. As shown in Table R5, fine-tuning increases SR from 7.10% to 8.20% and reduces DTS from 49.44 m to 44.77 m, while also improving OSR and SPL. These results support applying the framework to 3D observations after adaptation. Collision rates remain similar, so the experiments do not show improved collision avoidance. We will include these experiments and implementation details in the revision.

Table R5: Navigation results on the official UAV-ON test set. AOA-F results are reported in the UAV-ON paper. Base VLM and our method use the same framework; only our localization agent is fine-tuned on UAV-ON training observations.

MethodDTS (m) ↓\downarrowSR (%) ↑\uparrowOSR (%) ↑\uparrowSPL (%) ↑\uparrowCollision (%) ↓\downarrow
AOA-F (reported)48.377.3017.504.0665.50
Base VLM49.447.1015.944.0266.54
Ours44.778.2018.374.3466.82
Reviewer_97jCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a framework for active geolocalization in the UAV setting. The idea is to use two interacting "agents" which enable deferred localization. One agent takes in the current state and the instruction (target description) and decides one of two actions: (1) localize the target or (2) defer localization with a given rationale. The other agent reviews the rationale (which includes all accumulated information about the agent and the world) and generates an action. The method is evaluated on the CityNav dataset (but the original and revised version), resulting in significant improvements relative to a solid set of recent baselines.

Critical strengths
  • The method seems to preform well on a dataset that many methods have been evaluated on. So, there are strong baselines in place.
  • The presentation is fairly clear.
  • The two-agent architecture seems to be a natural way to decompose the key elements of addressing the task.
Critical weaknesses
  • I find myself a little uncertain about the implementation details. It's not clear that someone could replicate this work given the provided description.
  • I don't really understand why the method is better than previous methods. The framework and setup seems reasonable, but I don't see much comparison to the methods used in previous work. The performance gains are significant, so I would expect that there's a fundamental shift in approach. The paper doesn't do a very good job of making this clear, so I'm left wondering if there's something else. For example, L493 mentions that they are using existing datasets with "additional manual annotations", but it's not clear what those additional annotations are.
  • I would have liked to have seen some longer agent execution traces, with the agent outputs mapped to the spatial Location. Some single step examples are provided in the appendix, but some longer ones could be illuminating.
Questions for authors
  • I may have missed it, but I couldn't find what LLM was being used. Can you provide some additional details? I see LoRA fine tuning, but not much else.

  • It's not clear why this method is better than the previous results. In table 3, it seems like each version of the model without important components is better / or very similar in performance to the state of the art. Why is the method, even without the stated key innovations, so good? I would think the model would struggle to beat the state of the art without all the components.

  • What are the additional annotations mentioned on L493? Do those impact performance? Does that invalidate the fair comparison to baselines?

  • This isn't really a question, and I'm not sure exactly how to accomplish this, but I'd like to better understand which of the two agents is the weakest link. It's not really clear if the current architecture is performing optimally and so a little more understanding about the respective agents and where they are failing and succeeding seems like it could be interesting.

4 comments

Part 1/2

Comment 1: Base LLM backbone and LoRA fine-tuning details

Response: Both agents use Qwen3-VL-2B-Instruct. We fine-tune only the localization agent with LoRA to learn selective localization and deferral rationales; the exploration agent remains frozen and selects actions through rationale-conditioned prompting. We will add the following implementation details to the main text.

Implementation details. Both agents use Qwen3-VL-2B-Instruct, with only the localization agent fine-tuned using LoRA for one epoch. Training uses 32,000 sparse views constructed from 8,000 training episodes and weighted autoregressive cross-entropy, assigning weight 0.5 to target-unobserved DEFER samples and 1.0 otherwise. We use AdamW with a learning rate of 2×10−52\times 10^{-5}, cosine decay, and a warm-up ratio of 0.03. LoRA uses rank 16, α=32\alpha=32, and dropout 0.05. Training runs on three NVIDIA RTX 3090 GPUs with an effective batch size of 6, bfloat16 precision, gradient checkpointing, and random seed 7. The exploration agent retains the original pretrained weights. Both agents use greedy decoding with a maximum of 256 generated tokens.

Comment 2: Why the method and ablated variants outperform prior SOTA

Response: Our baseline already achieves strong performance by fine-tuning a VLM to ground targets directly in local RGB observations. Previous VLM-based methods, such as FlightGPT and GeoNav, instead predict global-map coordinates or rely on detector-derived scene graphs. Direct local grounding preserves access to fine-grained appearance and spatial relations that can be lost in these intermediate representations, helping distinguish visually similar candidates. We further improve this foundation through counterfactual sparse-view supervision and rationale-guided exploration to address missing instruction-relevant context.

Why the foundation is strong. Our foundation combines direct visual grounding with task-specific fine-tuning. The localizer predicts the target box directly from the instruction and local RGB observations, retaining access to the appearance and spatial relationships needed to distinguish nearby candidates. Fine-tuning adapts this grounding capability to aerial imagery and navigation instructions. In manuscript Table 2, bbox-only SFT increases conditional localization accuracy from 27.22% for the base VLM to 42.81%. The ablated variants therefore retain a strong ability to locate targets when sufficient evidence is available.

What the proposed components add. Building on this strong baseline, we further address a remaining limitation: a visible candidate may still be ambiguous when instruction-relevant context lies outside the observed region. We therefore introduce counterfactual sparse-view supervision to teach the localizer when to defer, rationale-guided exploration to acquire the missing evidence, and world-aligned spatial memory to retain that evidence across observations. Together, these components allow the system to resolve ambiguity through further observation. With the same trained localizer, exploration raises Test Unseen SR from 34.48% at T=0T=0 to 42.57% at T=5T=5 (manuscript Table 7), showing additional gains over an already strong localizer. Manuscript Table 3 further supports the individual contributions of these components.

Comment 3: Purpose and fairness of the additional annotations (line 493)

Response: The annotations mentioned in line 493 do not affect the fairness of the benchmark comparisons. They are human judgments on whether to localize or defer for 500 sparse validation views, used solely for the offline diagnostic evaluation in Appendix A.6. They are not used for training, hyperparameter selection, or the main navigation benchmark. We will revise line 493 to clarify their purpose.

Part 2/2

Comment 4: Which agent is the bottleneck and when each succeeds or fails

Response: The two-agent design separates selective localization from exploration, using deferral rationales to connect the two decisions. The localization agent is trained to predict a target or identify missing evidence, while the exploration agent uses that evidence requirement and the spatial context to select the next observation. To test this separation, we prompt the fine-tuned localization agent to also select exploration actions. This single-agent variant achieves 34.91% Test Unseen SR, compared with 42.57% for the two-agent framework (Table R1), supporting the benefit of assigning exploration to a separate agent.

Within this two-agent framework, the localization agent is the main performance bottleneck. Supplying the ground-truth box whenever it emits LOCALIZE and the target is visible raises SR from 42.57% to 58.36%, while retaining its localize/defer decisions. Replacing the exploration agent's direction selection with oracle movements toward the missing context or target raises SR to 52.38% (Table R1). Each intervention leaves the remaining framework unchanged, indicating greater room for improvement in target grounding. The failure analysis provides a corresponding explanation: the localizer can still confuse similar targets under complex spatial or ordinal constraints after the required context has been observed, whereas the explorer mainly struggles with long-range search across homogeneous urban blocks.

Table R1: Agent design and oracle bottleneck analysis on the original CityNav Test Unseen split.

VariantSR (%) ↑\uparrow
Single-agent variant (localizer also selects exploration actions)34.91
Two-agent framework (Ours)42.57
Two-agent framework + oracle exploration movements52.38
Two-agent framework + oracle target box when visible at LOCALIZE58.36

Comment 5: Longer multi-step execution traces linked to spatial positions

Response: We have added four multi-step rollouts to Appendix C.2 of the revised manuscript (Figs. 6–7), including two successes and two failures. Each covers four or five exploration movements and shows, at every step, the accumulated canvas, localization decision and rationale, exploration action, and UAV position on the spatial trajectory map. The successful rollouts illustrate how rationale-guided exploration accumulates context across views to disambiguate the target. The failure rollouts show where this process can still break down, distinguishing an incorrect final target selection after exploration from unsuccessful context acquisition within the fixed exploration budget.

Thanks for the rebuttal and additional experiments. The response addresses my main lingering concerns, including missing implementation details, the strength of the underlying localizer, how the additional annotations were used, and the relative value of the two agents.

One additional issue which I had considered while reviewing and was surfaced in the other reviews is whether the two-agent decomposition is essential. The new single-agent comparison is suggestive, but the two agents are trained differently: the localizer is fine-tuned for localization/deferral, while the explorer remains a frozen general-purpose VLM. So, the result may partly reflect specialization of the fine-tuned localizer rather than an advantage of using two agents. So, we don't have strong evidence that the two-agent setup is essential and I'm left wondering if there's a way to make a unified agent with explicit two-stage instructions behave similarly (but, I recognize that is something that could be pursued in future work).

Overall, the responses to my questions address my main concerns and, after considering the other reviews and responses, I plan to maintain my original score. However, I can see the argument for rejection if the belief is that the argument in favor of the two-agent architecture is found to be weakly supported. I suspect, in the future, practical solutions that perform well in tasks like this are going to be composed of interacting agents given the engineering / specialization advantages. But, I suppose the ultimate question is whether this contribution a great fit for ICLR or better aimed at a more robotics-focused venue.

Comment 1: Necessity of the two-agent decomposition and methodological fit

Response: To address the training difference in the earlier comparison, we train a unified agent for both localization/deferral and exploration. It uses the same Qwen3-VL-2B-Instruct backbone, LoRA configuration, spatial memory, and task-specific prompts as our framework. The same model first decides whether to localize or defer and, after deferral, selects an exploration action using shared parameters. Training uses the same CityNav split and localization/deferral supervision, with additional exploration-action supervision.

The two-agent framework achieves 42.57% SR, compared with 38.19% for the jointly trained unified agent, and also achieves higher SPL (Table R9). Thus, the advantage persists when the unified agent receives training for both tasks and follows an explicit two-stage procedure. This supports separating localization and exploration under the evaluated training setup.

Table R9: Comparison of the jointly trained unified agent and the two-agent framework on the original CityNav Test Unseen split.

VariantSR (%) ↑SPL (%) ↑
Unified agent, jointly trained on both tasks38.1928.16
Two-agent framework (Ours)42.5730.95

We clarify that our methodological contribution is learning when the available evidence is sufficient to identify a target and using an explanation of missing evidence to guide further observation. Counterfactual sparse-view supervision trains the localization/deferral decision and its rationale, which connects localization to exploration. Our controlled supervision ablations and experiments using different instructions for the same observation support this contribution beyond the comparison between one and two agents. We will make this emphasis explicit in the revision and present the two-agent architecture as an empirically supported design within the proposed approach.

Comments on the paper

1 comment

Summary

To address the remaining concern about the two-agent design, we trained a unified agent for both localization/deferral and exploration using explicit two-stage instructions and shared parameters. The two-agent framework achieves 42.57% SR, compared with 38.19% for this jointly trained alternative, supporting our design under the evaluated settings.

To address localization-error recovery, we reused the frozen exploration agent to verify proposed targets before terminal navigation. Its feedback allows the localizer to revise a prediction or gather another observation within the original movement budget, raising SR to 43.29%.

We also clarify that the methodological contribution is supervision for judging whether available evidence identifies a target and using an explanation of missing evidence to guide further observation. Controlled supervision ablations and tests using different instructions for the same observation support this contribution beyond the agent-count comparison. We will foreground these findings while qualifying the scope of the architectural and recovery claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.