acceptodds
Under review as a conference paper at ICLR 2027

StopWatch: Goal Grounding for Object-goal Navigation

Abstract

Pretrained VLMs bring rich world knowledge and semantic priors to object-goal navigation. Fine-tuning them into navigation policies has become a common recipe. Despite the progress, success rates remain low, and it remains unclear which capability the fine-tuned navigation policies lack. Our diagnosis identifies goal grounding as the dominant failure: the agent explores and remembers well, yet often fails to recognize that the goal is already in view or to estimate how far it remains. To address this failure, we let the agent ground the goal explicitly before each action, stating the semantics in view and the distance to the goal. Our experiments show that explicit grounding substantially improves success, reaching state-of-the-art results on six object-goal navigation benchmarks across three scene collections. Our analysis further shows that the policy nearly solves the task once grounding is correct, leaving the remaining headroom in perception rather than policy learning. Our work establishes goal grounding as the central problem of object-goal navigation and offers a simple yet effective way to address it.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Navigation, VLM

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_TmykOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper examines goal grounding as a core bottleneck in object goal navigation using vision-language models. It focuses on the difficulty of identifying the target object and deciding when to stop navigating. To address this, the authors propose the StopWatch framework. The framework consists of two models that separate object perception from navigation policy learning. A dedicated perception model predicts structured grounding information, including visible object categories and their distances to the goal, and passes it to a separately trained VLM policy model for action prediction. Experiments on six navigation benchmarks show that the method outperforms existing approaches. Ablation studies and failure case analyses further highlight the importance of explicit grounding supervision and accurate object perception in improving navigation success rates.

Critical strengths

None

Critical weaknesses

The methodological novelty is limited relative to existing modular ObjectNav approaches. Systems such as PONI and VLFM have already explored explicit goal recognition, spatial localization, and stopping decisions in depth. The proposed method largely reorganizes these established components by expressing perception outputs as structured text and passing them to a separately trained VLM policy. The combination is effective, but the paper does not convincingly establish a fundamentally new mechanism for grounding or navigation learning.

Questions for authors

The methodological novelty is limited relative to existing modular ObjectNav approaches. Systems such as PONI and VLFM have already explored explicit goal recognition, spatial localization, and stopping decisions in depth. The proposed method largely reorganizes these established components by expressing perception outputs as structured text and passing them to a separately trained VLM policy. The combination is effective, but the paper does not convincingly establish a fundamentally new mechanism for grounding or navigation learning.

The diagnosis is overgeneralized. The paper presents goal grounding as the central bottleneck in object goal navigation, yet recognizing target objects and determining their locations are already core components of the ObjectNav problem, and prior work has studied related difficulties. Moreover, the experiments focus on VLM policies trained with behavior cloning. They mainly expose the limitations of these particular policies. They do not show that goal grounding is the dominant challenge, nor that it applies to ObjectNav methods more broadly.

The performance gains over existing modular systems are not sufficiently attributed. StopWatch substantially outperforms VLFM, but VLFM already includes explicit target detection, spatial localization, and a stopping mechanism. The improvement may therefore come from stronger perception models, additional supervised training, the learned navigation policy, or some combination of these, rather than from the proposed grounding formulation itself.

The need for the complete grounding representation is not well justified. In Table 2, full grounding improves success by 6.1 percentage points over the baseline without grounding, but by only 0.7 points over the best single-field variant. This points to considerable overlap between semantic and distance grounding, particularly in signaling whether the goal is visible and whether the agent has arrived. A simpler predictor of goal visibility or an auxiliary STOP classifier might recover much of the gain.

The paper provides no controlled comparisons supporting the text interface. The paper does not adequately explain why converting explicit perception information into structured text is preferable to conventional representations such as semantic maps, masks, or spatial features. In addition, the advantage of the two-model architecture does not,, by itself,, show that the text interface drives the improvement.

5 comments

We thank the reviewer for the detailed questions. We would like to first point out a misunderstanding of our core contribution. The review assesses the paper as a new mechanism for grounding or navigation, but the paper does not claim one. Our contribution is a finding about why current VLM navigation policies fail, together with a solution that resolves most of this failure. Its three claims are below.

  • Goal grounding is the dominant failure of current VLM navigation policies. They reach the goal but do not know that they have arrived (Section 3.2).
  • Making grounding explicit resolves most of this failure. With ground-truth grounding, our policy reaches 91.6% success on clean HM3D-OVON val_unseen, close to the 96.1% of the expert that generated its demonstrations (Fig. 7).
  • The remaining headroom lies in perception. With the same policy, success rises with the accuracy of the grounding it reads (Fig. 7, Appendix A).

W1 / Q1. The methodological novelty is limited relative to modular methods such as PONI and VLFM.

  • We do not propose a new mechanism, and we do not argue for end-to-end over modular navigation. Our paper asks which capability limits current navigation policies and finds that it is goal grounding.
  • Wrong stops limit even a modular method with explicit goal detection. VLFM detects, localizes, and stops at the goal by explicit rules, yet it ends 37.1% of clean HM3D-OVON val_unseen episodes with a wrong stop (Q2 below).
  • Prior modular work builds grounding into its pipelines, but it does not measure its role. It does not show that grounding is the dominant failure, how much of the failure correct grounding removes, or where the remaining headroom lies. Our analysis measures all three.
  • The ICLR reviewer guide notes that submissions bring value when they "convincingly demonstrate new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc)", and our contribution is empirical knowledge of this kind.

Q2. The diagnosis is overgeneralized, since goal recognition is a known part of ObjectNav and the experiments cover only VLM policies trained by behavior cloning.

  • We credit the observation to prior work. Section 3.2 cites earlier reports that agents fail to stop at the goal (Wasserman et al., 2022; Song et al., 2023). Prior work noted the difficulty but did not show that it is the dominant failure of current policies or offer a way to resolve it, and our paper does both.
  • The finding matters more now than before. Open-vocabulary detectors such as SAM 3 and stronger VLMs might suggest that recognizing the goal is solved. Table 3 shows otherwise. GPT-6 Astra recalls 95.9% of visible goals, yet with our policy it reaches 81.4% success even with ground-truth distance, against 91.6% with full ground-truth grounding.
  • We separate failures of the model from defects in the benchmark. Annotation defects reject valid stops, and we measure them apart from model failures (Section 3.4 and Appendix E).
  • We study VLM policies trained by behavior cloning because they lead this task. They are the dominant approach to object-goal navigation and hold the strongest prior results on most of our benchmarks (Table 1).
  • Wrong stops limit even the modular VLFM, which detects the goal explicitly. VLFM ends 37.1% of episodes with a wrong stop, most of them before it ever reaches the goal. The revision adds its failure flows to Appendix H.
MethodReach goalSuccessWrong stop after reaching
VLFM (modular)64.452.87.4
LightNav-075.760.810.3
Ours77.469.63.6

Q3. The gain over VLFM is not attributed to the grounding formulation rather than stronger perception, more supervision, or the learned policy.

  • The comparison with VLFM shows that our system works, not that one paradigm is better. Our claims do not rest on it.
  • The gain splits into the learned policy and the grounding. Our policy without grounding, trained on the same demonstrations, already exceeds VLFM (Tables 1 and 2). Explicit grounding then adds 6.0 points.
Clean HM3D-OVON val_unseenSuccess (%)
VLFM52.8
Our policy without grounding63.6
+ predicted grounding (ours)69.6
+ ground-truth grounding91.6
  • Stronger perception is how the formulation pays off. The grounding text lets any perception model inform the same policy. With this policy fixed, success rises with the goal recall of the perception model, and GPT-6 Astra exceeds our perception model without retraining the policy (Appendix A). The 22-point gap to ground-truth grounding is the headroom that better perception can claim.

Q4. Full grounding improves little over the best single field, so a simpler visibility predictor or an auxiliary STOP classifier might suffice.

  • We corrected the ablation in the revision (Table 2). The semantic field alone raises success from 63.6% to 68.7%, and full grounding reaches 69.6%.
  • The small margin of the distance field reflects prediction accuracy, not redundancy. Substituting the ground-truth distance whenever both the predicted and the annotated inventory contain the goal raises success from 69.6% to 78.6% (Fig. 7a). Wrong distance also causes 26% of the remaining failures, against 43% for wrong visibility (Section 4.4).
Clean HM3D-OVON val_unseenSuccess (%)
Semantics only68.7
Semantics and distance (ours)69.6
Ours with ground-truth distance78.6
  • A perfect goal-visibility predictor falls short of the object list. Due to the time limit, we ran a smaller comparison than Table 2. On 256 randomly sampled val_unseen episodes, a policy that reads a ground-truth goal-visibility flag reaches 48.8%, and one that reads the predicted list of objects in view reaches 65.2%. Even ground-truth visibility thus helps the policy less than the object list, so a simpler visibility predictor would not suffice. We will add the complete ablation to Table 2 in the camera-ready version.

Q5. No controlled comparison supports the text interface over semantic maps, masks, or spatial features.

  • Semantic patch tokens perform about as well as text, so we chose text as the simpler interface. We passed the semantics in view to the policy as semantic patch tokens, pooled from the semantic masks over image patches, which keeps their spatial layout. Success was similar to that of the category list, but each frame then needs one token per patch instead of a few tokens per category (Appendix F).
  • Text follows how VLMs and VLAs already reason. They generate reasoning text before acting (Wei et al., 2022; Zawalski et al., 2025), and the text interface gives the policy grounding in the form it already reads, without a new encoder (Section 3.3).
  • We do not claim that text is the best interface, and it is not among our main claims. Our claims concern goal grounding, as stated at the top. Text is a simple interface that carries the semantics in view and the distance to the goal, and it suffices to test these claims.
AuthorsOpenReview ↗

Kind Reminder on the Author-Reviewer Discussion

As the author-reviewer discussion period ends soon, we would be grateful if you could look at our response and the revised PDF, and let us know whether they address your concerns. If you have any follow-up questions, please post them soon so that we can reply before the discussion closes.

Thank you again for your time and your review.

Reviewer_EzKoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why VLM-based object-goal navigation policies often reach a target’s vicinity but fail to complete the task. It identifies goal recognition and distance estimation as important bottlenecks and proposes StopWatch, a two-model system in which a perception model predicts structured grounding text and a policy model uses this text alongside visual observations to select actions. Grounding supervision is derived from annotated training scenes. The paper evaluates the approach across six navigation benchmarks from three scene collections, analyzes performance under ground-truth grounding, and investigates annotation and naming inconsistencies in existing evaluations.

Critical strengths
  1. The progression from failure analysis to explicit grounding, followed by semantic/distance ablations and ground-truth substitution experiments, provides a coherent account of where the evaluated systems fail. The experiments that replace predicted grounding with ground truth while retaining the trained policy are particularly useful for identifying opportunities to improve the proposed system. Although stopping difficulties are not new, the analysis offers actionable insight into their role in contemporary VLM-based navigation policies.
  2. The method shows consistent benefits that are not confined to the proposed corrected evaluation. Tab1 reports improvements across the evaluated benchmarks, including under official criteria.The investigation of annotation defects is also potentially valuable to the community, particularly because the paper reports both official and corrected results rather than replacing the former entirely.
Critical weaknesses
  1. Overclaim. Some conclusions about the cause of failure and the benefit of model separation are stronger than the controls establish. Appendix F explains that Fig2 compares two separately trained policies with different inputs, rather than adding grounding to a fixed RGB-only policy. Thus, the improvement establishes the value of privileged grounding under the corresponding training setup, but does not by itself isolate the failure mechanism of the original policy. The later fixed-policy substitution experiments provide stronger evidence, although their conclusions primarily concern the proposed architecture and evaluated settings. In addition, the disjoint system includes a separate 4B perception model with dense supervision; the ablations do not clearly separate architectural benefits from differences in capacity, supervision, and computation. These limitations warrant a narrower formulation of the claim that perception, rather than policy learning, is the central remaining problem.
  2. Experiment analysis. The corrected evaluation needs greater methodological transparency and more consistent reporting. The clean protocol combines vocabulary normalization, episode exclusions, and a revised accepted-target evaluation, making it difficult to attribute score changes to individual corrections. The exclusions are substantial in some settings. More detail is needed on the VLM-assisted audit, human verification, exclusion criteria, and resulting category/difficulty distributions. Numerical discrepancy: Fig 7 reports 68.9% and 91.2% for predicted and ground-truth grounding on HM3D-OVON val_unseen, whereas Tab 1 and Appendix Tab 8 report 69.5% and 90.7%. Perhaps I might have misunderstood. Could the author provide a brief explanation?
Questions for authors
  1. Clarification and additional evidence: Please specify the model sizes, training data, supervision, and training budgets for all variants in Table 2. Is there a capacity- or compute-matched single-model control, or a comparison with a simpler goal-recognition/termination module? This would help determine whether the gains arise specifically from separating perception and policy.
  2. Clarification and additional evidence: How were annotation errors verified, and were the exclusion rules fixed independently of the compared methods’ outcomes? Can the authors separate the effects of vocabulary normalization and sample filtering by reporting the original and revised scoring rules on the same retained episodes? Please also clarify whether held-out navigation goal categories appear as supervised labels in the perception model’s training inventories.
  3. Clarification: What accounts for the differences between Figure 7 and Tables 1 and 8? Are they based on different runs, checkpoints, or episode sets? Please identify the exact conditions and reconcile the corresponding values.
  4. Additional evidence: Even if SPL is considered an imperfect reference for search, can the authors report average action counts and inference time alongside SR? This would establish whether improved stopping accuracy is accompanied by comparable search efficiency and clarify the practical cost of the additional perception model.
  5. GOAT-Bench's instructions were generated by vision-language models, which may introduce ambiguous or inaccurate descriptions. A human-verified benchmark for this task, LangMap [1], is already publicly available. It would be beneficial to include results on this benchmark, along with the evaluation settings, to further strengthen the empirical validation of the proposed method. However, given the limited rebuttal period, I understand if conducting these additional experiments is not feasible.

[1] LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation. arXiv, 2026.

3 comments

We thank the reviewer for the careful reading and the constructive questions. We answer each point below.

W1. Conclusions about the cause of failure and the benefit of separating the models are stronger than the controls establish.

  • Fig. 2 motivates the problem, and our claims do not rest on it alone. Fig. 2 opens Section 3 to show the failure that the paper addresses. Fine-tuned VLM policies reach the goal far more often than they succeed (76% vs 52% on novel goal categories), and a policy trained to read correct grounding succeeds in 89% of the same episodes. We agree that these two policies are trained separately, so Fig. 2 indicates where the failure lies rather than isolating its cause.
  • The ablation (Table 2) and the failure flows (Appendix H) give the complete comparison and support our claims. LightNav-0, an end-to-end VLM policy, reaches the goal as often as our system (75.7% vs 77.4%) but stops in the wrong place after reaching nearly three times as often (10.3% vs 3.6%). Explicit grounding thus improves the decision to stop, the failure that Fig. 2 points to.
  • Grounding helps only when a separate model writes it. The joint policy writes the grounding and then acts, and it falls to 50.7%, below the 63.6% of the same policy without grounding. A separate perception model raises success to 69.6% (Table 2). Q1 shows that capacity does not explain this gain.
  • In our system, the remaining problem is perception rather than policy learning. Replacing predicted grounding with ground truth raises the same policy from 69.6% to 91.6%, close to the 96.1% of the expert that generated its demonstrations (Fig. 7). Table 3 confirms this with other perception models and the same policy. Among VLMs not trained on our task, success rises with goal recall, and GPT-6 Astra exceeds our own perception model.

W2. The corrected evaluation needs more transparency, and Fig. 7 disagrees with Tables 1 and 8.

  • We answer the transparency points under Q2 and the numerical discrepancy under Q3.

Q1. Please give model sizes, training data, supervision, and budgets for every Table 2 variant, and a capacity-matched single-model control or a simpler goal-recognition module.

  • Setup of the variants. Every variant is trained on the same demonstrations from the same three training sources, HM3D-OVON, MP3D, and HSSD. In Table 2, every policy is Qwen3-VL-2B, including the joint policy, and every perception model is Qwen3-VL-4B. The policies learn the expert actions by behavior cloning, the perception model learns grounding text derived from the annotations of the training scenes, and the joint policy learns both. The other training settings, including updates, frames per update, and compute, are in Appendix G (Table 8).
  • Capacity does not explain the gain. Our perception model is Qwen3-VL-4B, the size of LightNav-0, and our policy is Qwen3-VL-2B, the smallest Qwen3-VL model. We tested that with a 2B perception model instead, our system still reaches 68.0% on clean HM3D-OVON val_unseen, above the 60.8% of LightNav-0 (Table 1).
  • A perfect goal-visibility predictor falls short of the object list. Due to the time limit, we ran a smaller comparison than Table 2. On 256 randomly sampled val_unseen episodes, a policy that reads a ground-truth goal-visibility flag reaches 48.8%, and one that reads the predicted list of objects in view reaches 65.2%. Even ground-truth visibility thus helps the policy less than the object list, so a simpler visibility predictor would not suffice. We will add the complete ablation to Table 2 in the camera-ready version.

Q2. How were annotation errors verified, were exclusions fixed independently of outcomes, can the two corrections be separated, and do held-out goal categories appear in perception training?

  • Verification. We manually compared rendered frames with their semantic annotations. We flag a scene/category pair only on repeated evidence of a missing or wrong annotation and exclude episodes whose goal is flagged in their scene (Appendix E).
  • The exclusion rules never use episode outcomes. The flags depend only on scene annotations and rendered frames, and the reachability rule only on the episode and the navigation mesh (Appendix C). One mask applies to every method.
  • The two corrections can be separated. Our criterion changes the accepted goal names and viewpoints, and the clean subset removes flagged episodes. The table reports every combination. Neither correction changes the ranking, and our lead over the strongest baseline is 5.6 to 8.8 points in every column.
MethodOfficial, all 3,000Official, clean 2,309Ours, all 3,000Ours, clean 2,309
DAgRL21.023.324.226.0
Nav-R²27.429.034.034.0
Uni-NaVid32.135.238.941.0
VLFM39.043.849.152.8
LightNav-047.552.657.560.8
Ours53.958.266.269.6
  • The retained episodes keep the original distribution. The clean subset keeps 41 of the 49 goal categories, and at the median 77.9% of a scene's episodes (33.3% to 100% across the 36 scenes). Its median start-to-goal geodesic distance is 4.5 m, against 4.7 m for all episodes, so the exclusions barely change difficulty.
  • Held-out goal categories do appear in perception training, as objects. The perception model learns to list every annotated object in view in the training scenes, which include all held-out categories. HM3D-OVON holds them out only as navigation goals.
  • This does not leak evaluation data. Training and evaluation use disjoint scenes, so no evaluation scene enters training. These are common object names that pretrained VLMs, including the VLM baselines, already know.

Q3. What accounts for the differences between Fig. 7 and Tables 1 and 8?

  • The revision uses the final checkpoints throughout. The submitted Fig. 7 used earlier checkpoints, and it now agrees with Table 1 and the appendix tables. On clean HM3D-OVON val_unseen, predicted grounding now gives 69.6% and ground-truth grounding 91.6%.

Q4. Please report action counts and inference time alongside SR.

  • Path efficiency. Appendix B (Table 4) reports SPL. Its shortest-path reference assumes the agent already knows where the goal is, so we also report SPLFE\mathrm{SPL_{FE}}, which uses the path of our frontier-exploration expert on the same episode, since the expert also has to search.
  • Our system leads every baseline in SR, SPL, and SPLFE\mathrm{SPL_{FE}}.
SR / SPL / SPLFE\mathrm{SPL_{FE}} (%)HM3D-OVON val_unseenMP3D-OVHSSD-OV
VLFM52.8 / 30.2 / 39.830.8 / 16.7 / 27.136.3 / 18.5 / 29.0
LightNav-060.8 / 28.0 / 38.044.3 / 21.4 / 34.932.8 / 16.9 / 25.7
Ours69.6 / 36.5 / 50.749.9 / 24.0 / 40.256.4 / 27.7 / 44.9
  • Inference time. Our perception model takes 1.0 s per step, the same as Qwen3-VL-4B, against 9.3 to 16.6 s for GPT-6 Astra (Appendix A, Table 3).

Q5. Results on LangMap would strengthen the validation.

  • We thank the reviewer for the pointer. Our paper does not evaluate on GOAT-Bench. Our benchmarks give goals as category names, and our audit addresses errors in their scene annotations (Section 3.4).
  • LangMap offers an independent check of our evaluation. Its scene-level tasks give the goal as a category name, as in our task, and its human-verified annotations target the same errors as our audit. Its room, region, and instance goals require language descriptions that our policy is not trained to read.
  • The revision discusses LangMap in the related work (Section 2), and we will add results on its scene-level tasks, with their evaluation settings, in the camera-ready version.
AuthorsOpenReview ↗

Kind Reminder on the Author-Reviewer Discussion

As the author-reviewer discussion period ends soon, we would be grateful if you could look at our response and the revised PDF, and let us know whether they address your concerns. If you have any follow-up questions, please post them soon so that we can reply before the discussion closes.

Thank you again for your time and your review.

Reviewer_Pw1dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a two-model approach to improve robot navigation performance. This two-model design is inspired by the observation that grounding a scene with semantic and distance-to-goal information helps boost performance. The two models include a perception model and a policy model. The perception model predicts grounding information in addition to standard vision representations. The policy model accepts the outputs of the perception model to predict actions. The paper also identifies issues with evaluation methods and proposes improvements. The performance of the proposed method is shown to be better than the chosen baselines across all the test settings and metrics on synthetic datasets.

Critical strengths

The observation that grounding the goal with semantics and distance to the goal is interesting.

The proposed two-stage model delivers consistent improvements over baselines on the chosen datasets and metrics.

Critical weaknesses

Estimating the distance to the goal with only RGB input is typically error-prone. Given that the distance to the goal is an important grounding element, poor distance estimates degrade performance, especially in outdoor environments. Surprisingly, there is no mention of depth in the paper. Even more puzzling is the fact that the synthetic datasets used in this work all contain depth information.

SPL is an important metric. The justification for omitting SPL is not valid.

Validating only on synthetic data is limiting. Most, if not all, recent state-of-the-art agent navigation methods are validated on a real robotic platform.

Questions for authors

Why was depth not used as an input to estimate the distance to the goal?

Can you please report the SPL numbers for all the datasets used in this work?

Was there a limitation that prevented testing on a robot?

2 comments

We thank the reviewer for the comments and address each point below.

W1 / Q1. Estimating distance from RGB is error-prone, and depth is not used although the datasets provide it.

  • The failure we address is recognizing the goal, not measuring distance. The policy reaches the goal but not stop not because of wrong depth estimation, but because it does not recognize the goal (Section 3.2). Distance to the goal matters only after the goal is recognized, and depth cannot recognize it. Depth measures the distance to whatever surface is in view without telling whether that surface is the goal.
  • Our grounding follows this order. The distance field is given only when the goal is visible in the current image (Appendix F), and the ablation shows that the semantic field carries most of the gain (Table 2).
GroundingSR (%)
None63.6
Semantics only68.7
Semantics + distance69.6
  • Our distance field is not a depth value. It is the geodesic distance to the goal, measured along the navigable path to the goal's viewpoints rather than the line of sight. A VLM estimates such distance from RGB once it has found the goal, while depth would not give geodesic distance directly.
  • Depth helps little on ObjectNav. For example, on HM3D ObjectNav, the RGB-only Uni-NaVid outperforms InstructNav, the strongest method with depth input in its comparison (Zhang et al., 2025). PIRLNav also leaves out its available depth sensor, stating "We don't use this sensor as we don't find it helpful" (Ramrakhya et al., 2023). In our evaluation, VLFM reads depth, yet our RGB-only system outperforms it on all six benchmarks (Table 1).

W2 / Q2. SPL is an important metric and should be reported.

  • We report SPL in Appendix B (Table 4) of the revision.
  • SPL normalizes each successful path by the shortest path from the start to the goal, a reference that assumes the agent already knows where the goal is. We therefore also report SPLFE\mathrm{SPL_{FE}}, which replaces this reference with the path that our frontier-exploration expert travels on the same episode, since the expert also has to search before it finds the goal.
  • Our system leads every baseline in SR, SPL, and SPLFE\mathrm{SPL_{FE}}. Both path metrics discount success by path length, and our system leads on both.
SR / SPL / SPLFE\mathrm{SPL_{FE}} (%)OVONMP3D-OVHSSD-OV
VLFM52.8 / 30.2 / 39.830.8 / 16.7 / 27.136.3 / 18.5 / 29.0
LightNav-060.8 / 28.0 / 38.044.3 / 21.4 / 34.932.8 / 16.9 / 25.7
Ours69.6 / 36.5 / 50.749.9 / 24.0 / 40.256.4 / 27.7 / 44.9

W3 / Q3. Validation only on synthetic data is limiting, and recent methods are validated on real robots.

We want to first point out a misunderstanding that our evaluation uses only synthetic data. HM3D and MP3D are 3D reconstructions of real buildings, so four of our six benchmarks render real scenes.

Beyond these real scenes, we expect our system to transfer to real robots, for the two reasons below, and we explain why we did not run real-robot experiments.

  • Policies trained like ours already transfer. LightNav-0 and Uni-NaVid, two of our baselines, are VLM policies that read only RGB and learn navigation from data rendered in simulation, including HM3D and MP3D. Both run zero-shot on real robots, LightNav-0 on humanoid, quadruped, aerial, and wheeled platforms. Our policy is also a VLM fine-tuned on demonstrations rendered in HM3D, MP3D, and HSSD.
  • Our one new component can be replaced by VLMs that already work on real images. The perception model passes its output to the policy as text, so other VLMs can replace it without retraining the policy. With GPT-6 Astra, which is not trained on our task, the same policy even succeeds more often than with our own perception model (Appendix A, Table 3).
  • Real-world ObjectNav has no standard benchmark, and its results are hard to reproduce. Recent real-robot studies each choose their own robot, buildings, and goals, and physical conditions are hard to replicate across experiments and institutions (Savva et al., 2019; Deitke et al., 2020). The standard ObjectNav benchmarks are therefore simulated, and many recent methods report only simulation results, including ImagineNav (ICLR 2025), SG-Nav (NeurIPS 2024), and PIRLNav (CVPR 2023).
  • A real-robot evaluation also does not fit the rebuttal period. A meaningful one requires deploying our system and the baselines on the same robot, across several buildings and goal categories.
AuthorsOpenReview ↗

Kind Reminder on the Author-Reviewer Discussion

As the author-reviewer discussion period ends soon, we would be grateful if you could look at our response and the revised PDF, and let us know whether they address your concerns. If you have any follow-up questions, please post them soon so that we can reply before the discussion closes.

Thank you again for your time and your review.

Reviewer_EHX5OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies a key dominant failure mode, namely the lack of goal grounding, in fine-tuned VLM policies for embodied navigation tasks. The authors address this problem by using a separate perception model to provide explicit semantic information about the current view and the distance to the goal. This approach effectively enables the fine-tuned VLM policy to solve the tasks with a high success rate.

Critical strengths
  1. The paper not only proposes a new VLM policy but also identifies an underlying cause of the low success rates in current VLM-based navigation tasks.
  2. The proposed method is simple and effective, establishing new state-of-the-art performance and demonstrating great applicability.
  3. The paper also proposes a more reasonable evaluation criterion for valid stops, as well as an extended open-vocabulary benchmark.
  4. The proposed method is well supported by extensive experimental evaluations.
Critical weaknesses
  1. The evaluation is mainly conducted in simulated environments. It would be beneficial to see more results from real-world experiments.
  2. As mentioned in the limitations section, the availability of annotated scene data might be a limiting factor. However, I believe this issue should be addressable and the approach should become more scalable as foundation perception models continue to advance.
Questions for authors

None.

4 comments

We thank the reviewer for the positive feedback.

W1. The evaluation is mainly in simulation, and real-world results would strengthen the paper.

We expect our system to transfer to real robots, for the two reasons below, and we explain the difficulties of real-robot experiments.

  • Policies trained like ours already transfer. LightNav-0 and Uni-NaVid, two of our baselines, are VLM policies that read only RGB and learn navigation from data rendered in simulation, including HM3D and MP3D. Both run zero-shot on real robots, LightNav-0 on humanoid, quadruped, aerial, and wheeled platforms. Our policy is also a VLM fine-tuned via behavior cloning.
  • Our one new component can be replaced by VLMs that already work on real images. The perception model passes its output to the policy as text, so other VLMs can replace it without retraining the policy. With GPT-6 Astra, which is not trained on our task, the same policy even succeeds more often than with our own perception model (Appendix A, Table 3).
  • Real-world ObjectNav has no standard benchmark, and its results are hard to reproduce. Recent real-robot studies each choose their own robot, buildings, and goals, and physical conditions are hard to replicate across experiments and institutions (Savva et al., 2019; Deitke et al., 2020). The standard ObjectNav benchmarks are therefore simulated, and many recent methods report only simulation results, including ImagineNav (ICLR 2025), SG-Nav (NeurIPS 2024), and PIRLNav (CVPR 2023).
  • A real-robot evaluation also does not fit the rebuttal period. A meaningful one requires deploying our system and the baselines on the same robot, across several buildings and goal categories.

W2. The need for annotated scene data may limit scalability.

  • We agree that stronger foundation models address this, and our results already show it. Annotated scenes mainly limit the perception model, where our analysis places the remaining headroom. Because the policy reads perception as text, a foundation model can take this role without annotated scenes and without retraining the policy.
  • GPT-6 Astra already takes this role. It is not trained on our task, yet the same policy succeeds more often with GPT-6 Astra than with our perception model trained on annotated scenes (Appendix A, Table 3). As foundation models improve, our system improves with them, without more annotated data.
AuthorsOpenReview ↗

Kind Reminder on the Author-Reviewer Discussion

As the author-reviewer discussion period ends soon, we would be grateful if you could look at our response and the revised PDF, and let us know whether they address your concerns. If you have any follow-up questions, please post them soon so that we can reply before the discussion closes.

Thank you again for your time and your review.

I thank the authors for their response, which has resolved my concerns. The main reason I gave a higher score is that the paper is clearly written, the problem is well-defined, and the proposed method is simple and effective. I personally appreciate the simplicity of the method. Given this, I will maintain my original score.

Thank you for your response and for your support of our work. We are glad that our response resolved your concerns, and we appreciate your recognition of the clarity of the paper, the problem formulation, and the simplicity of the method.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal Response

We appreciate the time and feedback from the reviewers. Reviewers find that the paper identifies an underlying cause of low success in VLM navigation with a simple and effective method (EHX5), and that its gains are not confined to our corrected evaluation (EzKo).

Contribution. Current VLM navigation policies reach the goal but do not know that they have arrived. Making goal grounding explicit resolves most of this failure, and the remaining headroom lies in perception.

Responses to the main concerns.

  • Separation versus capacity (EzKo, Tmyk). A joint policy that writes its own grounding falls below the policy without grounding (50.7% vs 63.6%), while our system with a 2B perception model and a 2B policy outperforms LightNav-0, a single 4B model (68.0% vs 60.8%).
  • Real robots (EHX5, Pw1d). VLM policies trained in simulation already transfer to real robots, and our only new component, perception, can be replaced by VLMs that work on real images.
  • Novelty relative to modular methods (Tmyk). We contribute a finding rather than a new mechanism, and the modular VLFM, which detects the goal explicitly, still ends 37.1% of episodes with a wrong stop.
  • Depth and SPL (Pw1d, EzKo). The goal must be recognized before its distance matters, and our system leads every baseline in SR, SPL, and SPLFE\mathrm{SPL_{FE}}.

All changes in the revised PDF are highlighted. We welcome follow-up questions from the reviewers on any remaining concern.

AuthorsOpenReview ↗

Summary on Author-Reviewer Discussion

We thank the AC and the reviewers for their effort helping us improve the paper. Below we summarize the strengths the reviewers recognize, the main concerns with our answers, and where the discussion stands.

Contributions recognized by the reviewers

  • Insights for the community. Current VLM navigation policies reach the goal but do not know that they have arrived. Our paper shows that goal grounding is their dominant failure, that making grounding explicit resolves most of it, and that the remaining headroom lies in perception. The reviewers find that the paper "identifies an underlying cause of the low success rates" (EHX5), "provides a coherent account of where the evaluated systems fail" with "actionable insight" into stopping failures (EzKo), and makes an "interesting" observation on goal grounding (Pw1d). Its investigation of annotation defects is "potentially valuable to the community" (EzKo).
  • Technical strengths. The method is "simple and effective, establishing new state-of-the-art performance" and "well supported by extensive experimental evaluations" (EHX5). Its gains are consistent across the six benchmarks (Pw1d, Tmyk), "including under official criteria" (EzKo), and the ground-truth substitution experiments are "particularly useful" (EzKo). EHX5 also credits our "more reasonable evaluation criterion for valid stops" and the extended open-vocabulary benchmark. EHX5, EzKo, and Tmyk rate the presentation "clear and easy to follow".

Main concerns and our answers

The concerns fall into three groups, on what the paper contributes, what causes the gain, and whether the evaluation supports the results.

  • The finding is new and holds across system types (Tmyk, EzKo). The reviewers questioned its novelty relative to modular methods, its scope, and whether our evidence isolates the failure.
    • On novelty, we contribute a finding rather than a new mechanism, which Tmyk accepted.
    • The finding is new because prior modular work builds grounding into its pipelines but does not measure its role.
    • On scope and isolation, the failure flows (Appendix H) separate reaching the goal from stopping at it, and show that wrong stops limit both end-to-end VLM policies and modular navigation systems.
  • Ablation studies attribute the gain to grounding and the headroom to perception (EzKo, Tmyk, Pw1d). The reviewers asked whether capacity, supervision, a simpler signal, or depth explains the gain instead.
    • Capacity and supervision do not explain it. A joint policy that writes its own grounding falls below the policy without grounding, and our system with a 2B perception model outperforms the 4B LightNav-0.
    • A simpler signal does not suffice either, since even a ground-truth visibility underperforms predicted goal grounding.
    • Depth does not address the failure, because it cannot tell whether a surface is the goal, and our RGB-only system already outperforms the depth-reading VLFM system on all six benchmarks.
    • The remaining failures come from perception. With the policy fixed, ground-truth grounding raises success from 69.6% to 91.6%, close to the 96.1% of the expert.
  • The results do not rest on our evaluation choices (EzKo, Pw1d, EHX5). The reviewers asked about the corrected evaluation, SPL, and real robots.
    • The corrected evaluation does not create our lead. Its exclusion rules never use episode outcomes, and the lead holds under the official criterion as well as ours, on all and on clean episodes.
    • Our system also leads every baseline in path efficiency, measured by SPL and SPLFE\mathrm{SPL_{FE}} (Appendix B).
    • For real-world deployment, four of the six benchmarks already render scans of real buildings, and VLM policies trained like ours transfer zero-shot on real robots in our compared baselines.
    • Our one new component, perception, can also be replaced by VLMs that already work on real images. This answer resolved the concern for EHX5.

Discussion status

EHX5 (4) confirmed that our response "has resolved my concerns" and kept the clear-acceptance rating.

Tmyk (2) wrote that the VLFM failure flows and our attribution of the gain "resolved some of my concerns", and we answered the four follow-up questions. The reviewer has not yet update their final ratings.

EzKo (3) and Pw1d (2) raised no further concerns after our responses and have not yet update their final ratings.

We have addressed every concern raised in the reviews and the discussion, and all changes are highlighted in the revised PDF. We are confident that the three claims are well supported, and that the paper will inform how the community builds and evaluates navigation policies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.