acceptodds
Under review as a conference paper at ICLR 2027

Specialist Foundation Models as Tools for Generalist Navigation

Abstract

Generalist navigation requires diverse perceptual and reasoning capabilities to support tasks such as instruction following and object search. Recent generalist navigation models typically learn these capabilities within an end-to-end policy, so adding or improving individual capabilities can require further training of the navigation model. We argue that a navigation decision model need not master every capability itself; it can instead use independently pretrained specialist models as external tools to expand navigation capabilities without retraining the decision model. We present Perceive–Imagine–Verify (PIV), a loosely coupled framework that augments a shared decision model trained across navigation tasks with specialist foundation models as external tools. Perception tools recover missing depth and reconstruct panoramic context, foresight tools generate future observations conditioned on candidate moves, and verification tools ground targets to validate the decision model's stopping proposals. This separation offers two benefits. First, compatible tool backends can be replaced or upgraded while keeping the decision model fixed, providing a path to benefiting from advances in specialist models. Second, perception tools enable one checkpoint to operate across panoramic and forward-facing RGB and RGB-D configurations without sensor-specific fine-tuning. PIV achieves state-of-the-art performance in both vision-and-language navigation and object-goal navigation with one shared decision model. Specifically, it attains success rates of 71.6% on R2R-CE val-unseen and 56.6% on HM3D-OVON val-unseen.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Navigation, Vision-Language Navigation, Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_ZZPYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Perceive–Imagine–Verify (PIV), a generalist navigation framework that integrates specialist foundation models as external tools. The framework employs Qwen3-VL as a shared navigation decision model, while leveraging pretrained models for depth estimation, panoramic reconstruction, future observation prediction, and goal verification. The authors demonstrate competitive performance across multiple navigation benchmarks, including VLN, ObjectNav, and OVON, while supporting different sensor configurations with a single decision model.

Critical strengths

Clear and practical motivation. Leveraging specialist foundation models as external tools is a reasonable approach to improving the modularity and flexibility of generalist navigation systems.

Strong experimental results. PIV achieves competitive performance across several navigation benchmarks, demonstrating the effectiveness of integrating specialized capabilities into a shared navigation framework.

Good generalization across tasks and sensor configurations. The ability to support different navigation tasks and RGB/RGB-D configurations with a single decision model is a valuable aspect of this work.

Critical weaknesses

Limited architectural novelty. PIV primarily integrates existing specialist models through a predefined harness. While the resulting performance improvements are reasonable and valuable, they are also somewhat expected. Could the authors further clarify the key architectural novelty compared to conventional tool-augmented navigation pipelines?

Tool replaceability. The paper emphasizes that specialist tools can be replaced without retraining the navigation model. However, the foresight outputs are already incorporated into the training of Qwen3-VL. Have the authors experimented with replacing OmniRoam or other specialist models while keeping the decision model frozen?

Questions for authors

The current ablation studies of the foresight and verification modules are mainly conducted using the Stage 1 checkpoint. Could the authors clarify whether these tools provide similar improvements with the final Stage 2 model?

3 comments

Thank you for your insightful feedback and for recognizing the practical motivation and strong results. Below, we provide replacement experiments and more ablations.

A1. Clarify the architectural novelty beyond conventional modular navigation

PIV is the first to demonstrate both Generalist Learning and Modular Extensibility in a unified navigation architecture: a learned decision policy shared across VLN and ObjectNav that can benefit from independently upgraded specialist foundation models without retraining.

Conventional task-specific modular methods, such as SemExp, VLFM, and LM-Nav, tie their decision logic to predefined perception pipelines rather than independently learning a shared policy across navigation tasks. Their task-specific representations and sensing assumptions can therefore limit direct reuse across tasks and sensor configurations.

Q2. Add ablation of independent specialist upgrades without policy retraining.

Four completed substitutions use the same final Stage 2 checkpoint with zero additional decision-model training. Foresight swaps disable verification; grounding swaps disable foresight.

Table S1. Foresight backend replacement; verification disabled.

Foresight backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
MVSplat36069.5 / 62.555.6 / 32.8
OmniRoam (original)70.8 / 63.555.4 / 32.8
Geometric rendering71.0 / 64.255.8 / 33.0

Table S2. Verification backend replacement; foresight disabled.

Verification backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
Grounding DINO70.1 / 61.955.9 / 31.4
LocateAnything (original)70.5 / 62.456.6 / 31.5
GPT-6 Astra72.2 / 63.458.4 / 32.0

Geometric foresight gives results close to OmniRoam on both benchmarks, while GPT-6 Astra grounding increases measured SR by 1.7 points on R2R-CE and 1.8 points on OVON. MVSplat360 and Grounding DINO also run through the corresponding interfaces, with different accuracy trade-offs. The results demonstrate compatibility for the tested substitutions without policy retraining, rather than identical accuracy across backends.

For foresight tool,we clarify the distinction between training and evaluation. Policy training used simulator-rendered candidate views, while closed-loop evaluation uses candidate views generated online by the foresight tool. No ground-truth future views are supplied to the policy during evaluation. We will make this distinction explicit in §3.5, Appendix B.3, and the corresponding limitation statement.

Q3. Add specialist models ablation with the final Stage 2 checkpoint.

We repeated the complete ablation with the final checkpoint fixed.

Table S1. Foresight and Verification ablations with the final Stage 2 checkpoint.

ForesightVerificationR2R-CE SR / SPLHM3D-OVON Unseen SR / SPL
OffOff69.1 / 62.055.0 / 32.4
OnOff70.8 / 63.555.4 / 32.8
OffOn70.5 / 62.456.6 / 31.5
OnOn71.6 / 63.156.7 / 33.2

Foresight improves R2R-CE SR by 1.7 points, while verification improves OVON SR by 1.6 points. This fits their complementary roles: VLN provides intermediate landmarks for comparing future views, whereas ObjectNav explicitly names a target for Stop verification. Task dependence is useful in PIV because tools can be configured independently without retraining the policy.

Reference

[1] VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation. ICRA 2024.

[2] MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views. NeurIPS 2024.

[3] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV 2024.

We hope our responses address your concerns. If you have any further questions, please feel free to contact us. Thank you again for the constructive suggestions.

AuthorsOpenReview ↗

Sincere Thanks and Follow-up to Reviewer ZZPY

Dear Reviewer ZZPY,

Thank you again for taking the time to review our manuscript and share your feedback. We have responded to your comments and revised the manuscript accordingly. We hope that our responses and revisions address your concerns. If any questions or concerns remain, we would be happy to address them during the discussion period.

Best regards,

The Authors

AuthorsOpenReview ↗

Reminder to reply

Dear Reviewer ZZPY,

Thank you again for your careful and thorough review. As the discussion deadline approaches, we would appreciate your feedback on whether our revisions and additional evidence have addressed the concerns you raised.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_v4btOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PIV combines a shared navigation decision model with external perception, foresight, and verification models. These tools reconstruct observations, predict candidate outcomes, and check proposed stopping decisions through predefined interfaces. The system achieves competitive results across instruction-following and object-goal navigation while reusing one decision checkpoint across several sensor configurations.

Critical strengths
  • Evaluation spans multiple navigation tasks rather than a single benchmark.
  • Reusing one checkpoint across RGB, RGB-D, panoramic, and forward-facing inputs is useful.
  • The tool interfaces and waypoint-based decision process are clearly specified.
  • The appendix discusses cases where reported success does not imply faithful instruction following.
Critical weaknesses
  • Independent tool improvement is asserted but not demonstrated. Since predictive navigation is already established by NavForesee, an important distinction is the ability to benefit from replaceable specialists. Swap tool backends and measure navigation changes with the decision model fixed.
  • The ablations do not establish tool contributions in the final model. They use Stage 1, whereas headline results use Stage 2. Repeat key controls on the final checkpoint; the negligible OVON foresight gain and small R2R verification gain also require uncertainty estimates.
  • The sensing and execution costs are insufficiently accounted for. Forward-facing operation uses accumulated posed observations and active scans, while foresight is precomputed. Report scan actions, sensor assumptions, and full online latency before drawing comparisons with monocular navigation systems.
Questions for authors
  • Which verification and docking configuration is used for each reported benchmark?
  • How are stopping distances calibrated when depth is inferred rather than measured?
4 comments

Thank you for your insightful feedback and constructive suggestions. We have provided additional experiments and clarify the sensing and stopping protocols below.

A1. Add ablation of independent specialist upgrades without policy retraining.

Specialist upgrades can benefit an already-trained navigation policy. We keep the final Stage 2 checkpoint fixed throughout these comparisons, with no further decision-model training. Each replacement is evaluated against its original backend, disabling verification for foresight tests and foresight for verification tests.

Table S1. Foresight backend replacement; verification disabled.

Foresight backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
MVSplat36069.5 / 62.555.6 / 32.8
OmniRoam (original)70.8 / 63.555.4 / 32.8
Geometric rendering71.0 / 64.255.8 / 33.0

Table S2. Verification backend replacement; foresight disabled.

Verification backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
Grounding DINO70.1 / 61.955.9 / 31.4
LocateAnything (original)70.5 / 62.456.6 / 31.5
GPT-6 Astra72.2 / 63.458.4 / 32.0

GPT-6 Astra improves SR over LocateAnything by 1.7 points on R2R-CE and 1.8 points on HM3D-OVON. Geometric rendering achieves comparable or slightly higher scores than OmniRoam, while the other substitutions produce mixed effects. These results demonstrate that specialist upgrades can translate into better navigation without updating the policy, with gains depending on the chosen backend.

A2. Add specialist models ablation with the final Stage 2 checkpoint.

We repeated all four conditions on the final Stage 2 checkpoint, keeping the decision weights fixed.

Table S3. Foresight and Verification ablations with the final Stage 2 checkpoint.

ForesightVerificationR2R-CE SR / SPLOVON Unseen SR / SPL
OffOff69.1 / 62.055.0 / 32.4
OnOff70.8 / 63.555.4 / 32.8
OffOn70.5 / 62.456.6 / 31.5
OnOn71.6 / 63.156.7 / 33.2

Foresight alone improves SR by 1.7/0.4 points on R2R-CE/OVON, and verification by 1.4/1.6 points, respectively. This pattern is consistent with future views helping landmark-guided route selection and grounding helping object-goal confirmation.

On OVON, adding foresight to verification raises SPL by 1.7 points, with only a 0.1-point SR increase. These are point estimates; paired confidence intervals are not available, so we do not claim statistical significance for the small gains.

A3. Clarify sensing assumptions and quantify computational costs.

The forward-facing pipeline uses accumulated posed observations and averages 4.4 active scans per episode. The sensor-adaptation result demonstrates reuse of decision weights under this observation protocol. Foresight is generated online during evaluation; simulator-rendered candidate views are supplied to the policy only during training. We will clarify this distinction in the manuscript.

Online computation per decision. We benchmarked the same R2R input on an RTX 5090, using one warm-up run followed by five timed runs. Each enabled foresight backend generates five candidate views.

Foresight configurationMean computation time / decision (s)
No foresight0.414
Geometric rendering from historical point clouds0.577
MVSplat360, without diffusion refinement0.712
OmniRoam25.058

Timing includes DA3, panorama reconstruction, waypoint proposal, foresight generation, and selection. Observation acquisition, Stop verification, and action execution are excluded. Stop grounding additionally takes 0.130 s per checked view, with a one-time 3.030 s target-parsing cost.

Backbone inference cost across episodes. Across all 1,839 R2R-CE Val-Unseen episodes, recorded invocation counts and separate same-GPU BF16/SDPA profiling yield:

ConfigurationDecision calls / episodeInference time (min)Inference time / episode (s)
PIV: proposer + selector8.4980.022.61
NaVIDA: action prediction29.59529.1317.26

PIV makes 71.3% fewer decision calls. Among the 706 episodes where both methods succeed, the counts remain 6.88 versus 22.59. Each NaVIDA action chunk counts as one inference call. PIV estimates disable perception, foresight, verification tools, execution.

A4. Clarify the verification and docking configuration.

Docking is an optional capability and was disabled in all main benchmark results. These evaluations use grounding-based Stop confirmation: after the policy proposes Stop, Qwen3 extracts the target description and LocateAnything checks the current RGB views to accept or reject the proposal. Verification is disabled in the sensor-adaptation ablations of Table 4(b).

A5. How are stopping distances calibrated with inferred depth?

No depth-based stopping-distance calibration is required in the reported protocol. The selector proposes Stop and grounding checks target presence. DA3 depth normalization supplies the waypoint predictor, not a metric Stop threshold.

Reference

[1] MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views. NeurIPS 2024.

[2] GPT-6 Astra System Card. OpenAI, 2026.

[3] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV 2024.

[4] NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Prediction. CVPR 2026.

[5] NaVIDA: Vision-Language Navigation with Inverse Dynamics Augmentation. arXiv:2601.18188, 2026.

We hope these new experiments and clarifications address your concerns. Thank you again for the careful and constructive review.

AuthorsOpenReview ↗

Sincere Thanks and Follow-up to Reviewer v4bt

Dear Reviewer v4bt,

Thank you again for taking the time to review our manuscript and share your feedback. We have responded to your comments and revised the manuscript accordingly. We hope that our responses and revisions address your concerns. If any questions or concerns remain, we would be happy to address them during the discussion period.

Best regards,

The Authors

AuthorsOpenReview ↗

Reminder to reply

Dear Reviewer v4bt,

Thank you again for your careful and thorough review. As the discussion deadline approaches, we would appreciate your feedback on whether our revisions and additional evidence have addressed the concerns you raised.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_R5ebOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Perceive-Imagine-Verify (PIV), a framework for navigation leveraging pretrained specialist foundation models as tools. The authors finetune Qwen3-VL-4B to act as a decision model to select candidate waypoints while calling external tools for perception, foresight, and verification. First, they use a frozen pretrained RGB-D waypoint predictor to propose candidate waypoints. Then, they imagine what the world would look like at each candidate waypoint using OmniRoam, a panoramic video generation model. When the decision model proposes Stop, the pipeline uses a verification tool (Qwen3 + LocateAnything) that checks if the target from the instruction is in the current view. The authors achieve state-of-the-art performance on VLN and ObjectNav benchmarks, and emphasize the plug and play nature of the framework, so it can leverage newer specialist models.

Critical strengths

The paper is clearly written and the framework description is easy to follow.

Critical weaknesses
  1. The authors claim that the foundation tool models can be replaced. However, there are no experiments showing replacement of any backend foundation models (SEVA, LocateAnything). Also, since the decision model is trained on OmniRoam outputs, it is unclear whether replacing OmniRoam with another video generation model would work without retraining.
  2. OmniRoam appears to be an expensive tool call: it runs 5 30 step video generations per decision step. In the paper, these generations were precomputed offline. For inference, I believe generation may be too costly so I am uncertain of the actual usefulness of the framework.
Questions for authors
  1. Can you swap the backend foundation models? This helps address weakness 1.
  2. What is the wall-clock cost of running the framework?
  3. How were the foresight generations precomputed for the closed loop evaluations (R2R-CE and HM3D-OVON)?
5 comments

Thank you for your insightful feedback and for recognizing the clarity of the framework. Your questions prompted direct replacement experiments, runtime profiling, and a correction to the foresight protocol description.

A1. Add ablation of independent specialist upgrades without policy retraining.

PIV supports independent upgrades of foresight and verification without retraining the decision policy. All six configurations use the same final Stage 2 checkpoint, with the other tool disabled in each comparison.

Table S1. Foresight backend replacement; verification disabled.

Foresight backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
MVSplat36069.5 / 62.555.6 / 32.8
OmniRoam (original)70.8 / 63.555.4 / 32.8
Geometric rendering71.0 / 64.255.8 / 33.0

Table S2. Verification backend replacement; foresight disabled.

Verification backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
Grounding DINO70.1 / 61.955.9 / 31.4
LocateAnything (original)70.5 / 62.456.6 / 31.5
GPT-6 Astra72.2 / 63.458.4 / 32.0

Geometric rendering and GPT-6 Astra improve the reported scores over their respective original backends, demonstrating that specialist upgrades can benefit the fixed navigation policy.

A2. Measure the wall-clock cost of the framework.

The measured wall-clock cost is 0.577 s per decision step with geometric foresight and 25.058 s with one-step OmniRoam. The results for all tested configurations are:

Foresight configurationMean time per step (s)
No foresight0.414
Geometric rendering0.577
MVSplat360;0.712
OmniRoam25.058

Timing includes DA3 depth estimation, monocular panorama reconstruction, waypoint proposal, foresight generation, and waypoint selection, ending when the selector returns an action. Stop verification and action execution are excluded. Stop verification adds 0.130 s per checked RGB view for target grounding.

A3. Clarify foresight training and online evaluation.

For the foresight tool, we clarify the distinction between training and evaluation. Policy training used simulator-rendered candidate views, while closed-loop evaluation uses candidate views generated online by the foresight tool. No ground-truth future views are supplied to the policy during evaluation.

We also optimized OmniRoam inference by reducing denoising steps, caching input-image encodings, and optimizing VAE tiling and output processing. The OmniRoam timing reported in A2 uses the optimized one-step denoising configuration.

We will make these protocols and inference settings explicit in §3.5, Appendix B.3, and the corresponding limitation statement.

Reference

[1] MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views. NeurIPS 2024.

[2] GPT-6 Astra System Card. OpenAI, 2026.

[3] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV 2024.

AuthorsOpenReview ↗

Sincere Thanks and Follow-up to Reviewer R5eb

Dear Reviewer R5eb,

Thank you again for taking the time to review our manuscript and share your feedback. We have responded to your comments and revised the manuscript accordingly. We hope that our responses and revisions address your concerns. If any questions or concerns remain, we would be happy to address them during the discussion period.

Best regards,

The Authors

AuthorsOpenReview ↗

Reminder to reply

Dear Reviewer R5eb,

Thank you again for your careful and thorough review. As the discussion deadline approaches, we would appreciate your feedback on whether our revisions and additional evidence have addressed the concerns you raised.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_9EiLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework for navigation that combines a shared navigation decision model with external specialist foundation models. The framework includes perception tools for depth estimation and panoramic reconstruction, a foresight tool for predicting future observations at candidate waypoints, and a verification tool for validating stopping decisions. The navigation policy is based on Qwen3-VL and is trained across vision-and-language navigation (VLN) and object-goal navigation tasks.

The paper evaluates the approach on a rather diverse benchmark seettings. It reports strong results across these benchmarks using a single decision-model checkpoint, including 71.6% success rate on R2R-CE val-unseen and 56.7% on HM3D-OVON val-unseen. The authors also conduct ablations of the foresight and verification tools and investigate adaptation to different camera and depth-sensing configurations

Critical strengths
  1. The most convincing aspect of the paper is its empirical performance. The proposed approach achieves competitive or leading results across several VLN and ObjectNav benchmarks using a shared navigation checkpoint. The results are particularly encouraging given that these tasks require different forms of reasoning, from following detailed route instructions to locating objects specified by category. This provides useful evidence that a shared decision model can support different navigation objectives.

  2. The separation between navigation decisions and external perception, prediction, and verification capabilities is a reasonable and potentially useful design choice. In particular, the ability to operate with different sensor configurations without additional decision-model fine-tuning is appealing for embodied applications. I also appreciate that the paper evaluates the contributions of different tools rather than relying solely on comparisons with previous methods.

Critical weaknesses
  1. The methodological novelty is not sufficiently differentiated from prior VLM-based navigation frameworks.

A big part of PIV's navigation pipeline resembles ideas already explored in previous work. In particular, [1] formulates navigation as selecting among feasible, visually annotated candidate actions using a VLM. It constructs candidate actions from geometric information, projects action identifiers onto camera observations, and uses a separate mechanism to determine when navigation should terminate.

The proposed approach similarly relies on an external waypoint proposer, overlays candidate identifiers on visual observations, and asks a VLM to select an action or Stop. Although the exact waypoint-generation procedure and stopping verification differ, the overall decision interface seems to me to be similar.

The proposed framework adds specialist modules for depth estimation, panoramic completion, future observation generation, and target grounding. However, the paper does not sufficiently explain what is fundamentally new about the integration mechanism beyond combining existing pretrained components with a trained navigation policy. In particular, the distinction between the proposed tool-based architecture and earlier modular or VLM-based navigation systems should be made more explicit.

  1. The experiments do not sufficiently validate the claimed benefits of modularity and tool replacement.

One of the paper's main claims is that specialist foundation models can be replaced or upgraded independently while keeping the navigation decision model fixed. However, the reported experiments primarily evaluate the effects of enabling or disabling tools, rather than replacing one specialist model with another.

For example, the paper uses OmniRoam for foresight and LocateAnything for target grounding, but does not demonstrate whether alternative models can be substituted successfully, or whether improvements in the underlying specialist models translate into improved navigation performance without retraining the decision model.

The ablation results also suggest that the benefits of individual tools are task-dependent and sometimes relatively small. On R2R-CE, verification alone improves success rate by only 0.46 percentage points over the no-tool setting. On HM3D-OVON, foresight alone provides almost no improvement in success rate (+0.05 percentage points), while the full configuration performs slightly worse than verification alone in success rate.

These results do not invalidate the framework, but they raise questions about whether the additional specialist models consistently justify their complexity. I would like to see a more direct evaluation of tool interchangeability, as well as an analysis of the computational overhead introduced by the additional models.

[1] End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering

Questions for authors

See the questions above/

7 comments

Thank you for your insightful feedback and for recognizing the empirical performance and practical value of the shared decision model. We have added backend replacements and final-checkpoint controls to address your concerns directly.

A1. Clarify the methodological novelty and distinguish PIV from VLMnav.

To the best of our knowledge, PIV is the first to demonstrate both Generalist Learning and Modular Extensibility in a unified navigation architecture: a learned decision policy shared across VLN and ObjectNav that can benefit from independently upgraded specialist foundation models without retraining.

We agree with the reviewer that external candidate proposal, visual candidate annotations, and VLM-based action selection are shared design choices. We do not claim these interface choices as our novelty. The differences between PIVNav and VLMNav are summarized below.

AspectVLMnavPIV
Decision modelOff-the-shelf Gemini Flash without navigation fine-tuningOne decision checkpoint jointly trained for VLN and ObjectNav
Task scopeObjectNav and GOAT object/image/description goalsRoute-instruction following and object-goal search
Observation adaptationRGB-D, RGB-only depth/segmentation alternatives, and FOV experimentsPanoramic/forward-facing RGB and RGB-D configurations with the same decision checkpoint
Evidence for action selectionCurrent observations with annotated feasible actionsCurrent observations plus candidate-conditioned future views
TerminationSeparate VLM stopping queryDecision-model Stop proposal checked through external target grounding

A2. Add ablation of independent specialist upgrades without policy retraining.

PIV supports independent upgrades of foresight and verification without retraining the decision policy. All six configurations use the same final Stage 2 checkpoint, with the other tool disabled in each comparison.

Table S1. Foresight backend replacement; verification disabled.

Foresight backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
MVSplat36069.5 / 62.555.6 / 32.8
OmniRoam (original)70.8 / 63.555.4 / 32.8
Geometric rendering71.0 / 64.255.8 / 33.0

Table S2. Verification backend replacement; foresight disabled.

Verification backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
Grounding DINO70.1 / 61.955.9 / 31.4
LocateAnything (original)70.5 / 62.456.6 / 31.5
GPT-6 Astra72.2 / 63.458.4 / 32.0

Geometric rendering and GPT-6 Astra improve the reported scores over their respective original backends, demonstrating that specialist upgrades can benefit the fixed navigation policy.

For foresight tool,we clarify the distinction between training and evaluation. Policy training used simulator-rendered candidate views, while closed-loop evaluation uses candidate views generated online by the foresight tool. No ground-truth future views are supplied to the policy during evaluation. We will make this distinction explicit in §3.5, Appendix B.3, and the corresponding limitation statement.

A3. Add specialist models ablation with the final Stage 2 checkpoint.

We repeated the complete foresight/verification ablation using the Stage 2 checkpoint, with the same decision weights in all four conditions.

Table S3. Foresight and Verification ablations with the final Stage 2 checkpoint.

ForesightVerificationR2R-CE SR / SPLHM3D-OVON Unseen SR / SPL
OffOff69.1 / 62.055.0 / 32.4
OnOff70.8 / 63.555.4 / 32.8
OffOn70.5 / 62.456.6 / 31.5
OnOn71.6 / 63.156.7 / 33.2

Compared with the no-tool configuration, foresight alone increases SR by 1.7 points on R2R-CE and 0.4 points on OVON; verification alone increases SR by 1.4 and 1.6 points, respectively. A plausible explanation for the task-dependent benefits of individual modules is that VLN’s intermediate landmarks make future views useful for route selection, whereas ObjectNav’s explicit target makes grounding-based Stop confirmation more directly useful.

A4. Add analysis of the computational overhead.

We separately profiled the following components on an RTX 5090, reporting mean latency over 3‒5 runs after warm-up and excluding model loading.

ToolMethodMean latency (s)
Depth recoveryDA30.130
Panorama reconstructionGeometric rendering0.026
ForesightOmniRoam20.3
VerificationLocateAnything0.130

Perception tools run only when required by the sensor configuration, and verification is triggered only after a Stop proposal. OmniRoam introduces the largest computational overhead. A faster alternative, MVSplat360, generates five candidate views in 0.090 s without diffusion refinement. We will further explore efficient foresight backends that balance latency and navigation performance.

Reference

[1] End-to-End Navigation with Vision-Language Models: Transforming Spatial Reasoning into Question-Answering. NeuS 2025.

[2] MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views. NeurIPS 2024.

[3] GPT-6 Astra System Card. OpenAI, 2026.

[4] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV 2024.

Reviewer_3vT6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Perceive–Imagine–Verify (PIV), a generalist navigation framework that combines a shared VLM-based decision model with pretrained specialist foundation models for perception, future observation generation, and goal verification. The decision model is fine-tuned for waypoint selection using aggregated multi-task navigation data and pretrained foresight outputs, while additional perception and verification tools are integrated without further fine-tuning. The method achieves strong performance across R2R-CE, RxR-CE, MP3D ObjectNav, HM3D ObjectNav, and HM3D-OVON benchmarks. Ablation studies further investigate the contributions of individual tools and adaptation across different sensor configurations.

Critical strengths
  1. Strong empirical performance across diverse navigation tasks.

PIV demonstrates competitive performance on both vision-and-language navigation and object-goal navigation benchmarks using a shared decision checkpoint. The extensive evaluation across multiple navigation tasks supports the practical effectiveness of the proposed system.

  1. Adaptation across different sensor configurations.

The framework demonstrates that a single navigation decision checkpoint can operate across panoramic and front-facing RGB/RGB-D configurations without sensor-specific fine-tuning. Although this flexibility is largely achieved through pretrained perception tools and standardized observation representations rather than a novel adaptation mechanism, it offers practical value for deploying navigation policies under heterogeneous sensing conditions.

Critical weaknesses
  1. Limited methodological novelty.

The paper's primary claimed contribution, separating navigation decision-making from specialist capabilities, is essentially a conventional modular robotics paradigm rather than a new methodological insight. The core decision-making component is a VLM fine-tuned for waypoint selection using aggregated multi-task navigation data, an approach that has already been extensively explored in the navigation community. The proposed PIV framework primarily integrates this decision model with existing pretrained specialist models, while several underlying ideas, such as future observation prediction for navigation, have already been explored in prior works including NavForesee and ImagineNav. The framework does not introduce a new learning objective or a principled mechanism for adaptive tool selection and coordination. Consequently, the methodological contribution beyond system-level integration remains unclear.

  1. Insufficient empirical justification for the proposed framework.

Although the paper demonstrates strong overall navigation performance, the ablation studies suggest that the benefits of individual specialist modules are relatively modest and task-dependent. For example, on HM3D-OVON Unseen (Table 5), foresight provides negligible improvement, and combining it with verification does not yield a meaningful additional gain in success rate. Moreover, these ablations are conducted using the Stage 1 checkpoint rather than the final Stage 2 model used for the main benchmark comparisons.

The paper also emphasizes the ability to independently replace or upgrade specialist models without retraining the decision model. However, replacing conventional perception modules through compatible interfaces is already a standard property of modular architectures. For more consequential components, such as the imagination module, the decision model is explicitly trained using outputs from OmniRoam, and the paper provides no experiments demonstrating whether replacing it with another pretrained generator preserves or improves navigation performance without fine-tuning. Therefore, the claimed advantage of independent specialist upgrades is not convincingly established.

Questions for authors
  1. Methodological novelty.

Beyond combining a fine-tuned VLM waypoint selector with existing pretrained specialist models, what is the key methodological contribution of PIV compared with prior modular and generalist navigation frameworks? In particular, what distinguishes the proposed separation of decision-making and specialist capabilities from conventional modular navigation architectures?

  1. Effectiveness of specialist modules.

The ablation studies suggest that the benefits of individual modules are modest and task-dependent. For example, foresight provides negligible improvement on HM3D-OVON Unseen (Table 5). Could you explain this observation? Furthermore, since the ablations use the Stage 1 checkpoint, do the specialist modules provide similar improvements when using the final Stage 2 checkpoint?

  1. Independent specialist upgrades.

The paper claims that specialist models can be independently replaced or upgraded without retraining the decision model. Given that the decision model is trained using OmniRoam-generated future observations, could you demonstrate whether replacing OmniRoam with another pretrained generator preserves or improves navigation performance without fine-tuning the decision model?

6 comments

Thank you for your insightful feedback and for recognizing the cross-task results and sensor adaptability. Your requests motivated final-checkpoint ablations and direct backend-replacement experiments.

A1. Clarify the methodological novelty of PIVNav.

To the best of our knowledge, PIV is the first to demonstrate both Generalist Learning and Modular Extensibility in a unified navigation architecture: a learned decision policy shared across VLN and ObjectNav that can benefit from independently upgraded specialist foundation models without retraining.

Conventional task-specific modular methods, such as VLFM, tie their decision logic to a predefined set of specialist modules rather than independently learning a shared policy across navigation tasks. Their prescribed sensor settings and perception pipelines also limit extensibility.

NavForesee jointly learns future-feature prediction and navigation, coupling foresight to the navigation model and limiting independent replacement. ImagineNav also uses external future-view generation, but its ObjectNav decisions come from a prompted VLM, without an independently trained decision policy shared across navigation tasks.

A2. Add specialist models ablation with the final Stage 2 checkpoint.

We repeated the complete foresight/verification ablation using the Stage 2 checkpoint, with the same decision weights in all four conditions.

Table S1. Foresight and Verification ablations with the final Stage 2 checkpoint.

ForesightVerificationR2R-CE SR / SPLHM3D-OVON Unseen SR / SPL
OffOff69.1 / 62.055.0 / 32.4
OnOff70.8 / 63.555.4 / 32.8
OffOn70.5 / 62.456.6 / 31.5
OnOn71.6 / 63.156.7 / 33.2

Relative to the no-tool configuration, foresight alone increases SR by 1.7 points on R2R-CE and 0.4 points on OVON; verification alone increases SR by 1.4 and 1.6 points, respectively. A plausible explanation why the benifits of individual modules are task-dependent is that VLN’s intermediate landmarks make future views useful for route selection, whereas ObjectNav’s explicit target makes grounding-based Stop confirmation more directly useful. This task dependence is useful in practice, guiding us to replace or upgrade specialists according to each task’s needs and omit those with limited benefit, while keeping the shared decision policy fixed.

A3. Add ablation of independent specialist upgrades without policy retraining.

PIV supports independent upgrades of foresight and verification without retraining the decision policy. All six configurations use the same final Stage 2 checkpoint, with the other tool disabled in each comparison.

Table S2. Foresight backend replacement; verification disabled.

Foresight backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
MVSplat36069.5 / 62.555.6 / 32.8
OmniRoam (original)70.8 / 63.555.4 / 32.8
Geometric rendering71.0 / 64.255.8 / 33.0

Table S3. Verification backend replacement; foresight disabled.

Verification backendR2R-CE SR / SPL (%) ↑HM3D-OVON Unseen SR / SPL (%) ↑
Grounding DINO70.1 / 61.955.9 / 31.4
LocateAnything (original)70.5 / 62.456.6 / 31.5
GPT-6 Astra72.2 / 63.458.4 / 32.0

Geometric rendering and GPT-6 Astra improve the reported scores over their respective original backends, demonstrating that specialist upgrades can benefit the fixed navigation policy.

For foresight tool,we clarify the distinction between training and evaluation. Policy training used simulator-rendered candidate views, while closed-loop evaluation uses candidate views generated online by the foresight tool. No ground-truth future views are supplied to the policy during evaluation. We will make this distinction explicit in §3.5, Appendix B.3, and the corresponding limitation statement.

Reference

[1] VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation. ICRA 2024.

[2] NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Prediction. CVPR 2026.

[3] ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination. ICLR 2025.

[4] MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views. NeurIPS 2024.

[5] GPT-6 Astra System Card. OpenAI, 2026.

[6] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV 2024.

We hope these new results and clarifications address your concerns about the contribution, final-model tool benefits, and independent replacement. Thank you again for the constructive suggestions.

Thank you for the additional experiments and detailed clarifications. I appreciate the final-checkpoint ablations and backend-replacement experiments, and I would appreciate further clarification on two points.

  1. The clarification regarding foresight training appears inconsistent with the original manuscript. Appendix B.3 explicitly states that "Simulator-rendered future views are used only for diagnostic checks, never as substitutes for generated predictions," whereas your response states that policy training uses simulator-rendered candidate views. Furthermore, Appendix E explicitly states that evaluation uses foresight precomputed offline, whereas your response describes online generation. Could you clarify these discrepancies and specify exactly which future observations are used during Stage 1 training, DAgger data collection, Stage 2 training, and evaluation?

  2. The additional results in Table S2 suggest that geometric rendering achieves consistently comparable or better performance than OmniRoam across both benchmarks. Could you clarify the implementation of geometric rendering, including what scene information is available to it? Given these results, what is the practical advantage of using a generative foresight model over conventional geometric rendering, particularly considering their potential differences in computational cost?

Thank you for these follow-up questions.

(1) Foresight training and evaluation protocols. Due to OmniRoam's high generation cost, we used simulator-rendered candidate views for training-data construction in both stages, including DAgger collection. During evaluation, foresight is generated online from the agent's current observations and candidate motions. No ground-truth future observations are provided to the policy at evaluation time. We apologize for the unclear descriptions and will explicitly revise §3.5, Appendix B.3, and Appendix E to reflect the actual protocols.

(2) Geometric rendering versus generative foresight. Our geometric backend unprojects observed RGB-D into a colored point cloud and renders candidate views through depth-ordered point splatting and Navier–Stokes inpainting. It uses only observed geometry, never future ground-truth observations. Geometric rendering preserves observed appearance and may better match the simulator-rendered candidate views used during training. However, it cannot reliably recover previously occluded or unseen regions, as inpainting only fills gaps from nearby pixels. Generative foresight can potentially infer plausible content beyond observed surfaces. We acknowledge that geometric rendering currently offers a better accuracy–efficiency trade-off, while more efficient generative or hybrid foresight remains a promising direction.

We appreciate the reviewer's careful and insightful suggestion and hope these clarifications resolve the concerns.

Comments on the paper

2 comments

Dear Area Chair and Reviewers,

We thank all reviewers for their constructive feedback and recognition of our strong experimental results across navigation benchmarks (3vT6, 9EiL, ZZPY), generalization across tasks and sensor configurations (3vT6, 9EiL, v4bt, ZZPY), practical motivation and modular design (9EiL, ZZPY), and clear framework presentation (R5eb, v4bt). To address the shared concerns, particularly regarding methodological novelty and empirical validation, we conducted additional experiments and provide clarifications summarized below.

1. Core Novelty: Unifying Generalist Learning and Modular Extensibility

To the best of our knowledge, PIV is the first to demonstrate both Generalist Learning and Modular Extensibility in a unified navigation architecture. The key innovation is not merely integrating pretrained tools, but establishing a clear learning boundary between generalist navigation competence and specialist capabilities. PIV learns a shared decision policy across VLN and ObjectNav, while independently trained foundation models provide perception, foresight, and verification through replaceable interfaces. Crucially, this enables a trained generalist navigator to benefit from advances in specialist foundation models without retraining its decision policy. Unlike task-specific modular pipelines (e.g., VLFM), jointly learned predictive navigation (NavForesee), or prompted VLM-based navigation (VLMNav and ImagineNav), PIV unifies cross-task policy learning with independently extensible specialist capabilities. Our new fixed-checkpoint backend-replacement experiments provide direct empirical evidence for this distinction.

2. Specialist Replacement and Independent Upgrades

We evaluated six specialist configurations with the same final Stage 2 checkpoint and no additional policy training. Replacing LocateAnything with GPT-6 Astra improves SR by 1.7 points on R2R-CE and 1.8 points on HM3D-OVON. Alternative foresight backends also operate through the same interface, with geometric rendering achieving slightly better results than OmniRoam. These experiments directly validate PIV's modular extensibility and its ability to benefit from specialist upgrades without policy retraining.

3. Final-Checkpoint Foresight and Verification Ablations

We repeated the complete tool ablations on the final Stage 2 checkpoint. Foresight alone improves R2R-CE SR by 1.7 points, while verification alone improves HM3D-OVON SR by 1.6 points. Combining both achieves 71.6% and 56.7% SR, respectively, confirming their complementary, task-dependent benefits.

4. Computational Efficiency and Protocol Clarifications

We added module-level latency profiling and end-to-end decision-step measurements, and showed that PIV requires 71.3% fewer decision-model calls than NaVIDA. We also clarified the forward-facing sensing protocol, stopping mechanism, and the distinction between simulator-rendered training views and online-generated foresight during evaluation, correcting the corresponding manuscript descriptions.

Together, these additional experiments strengthen our central claim that PIV unifies Generalist Learning and Modular Extensibility, enabling a shared navigation policy to benefit from independently evolving specialist foundation models. We hope our responses adequately address the reviewers' concerns and sincerely welcome any further questions or suggestions during the remaining discussion period. Thank you again for your thoughtful consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair and Reviewers,

We sincerely appreciate your time and efforts throughout the ICLR 2027 review process. Thank you for your constructive feedbacks on PIV. During the rebuttal, we conducted significant additional experiments to address the raised concerns.

Following the rebuttal, three of the five reviewers now recommend weak acceptance. The two reviewers recommending weak rejection also acknowledged the new experimental evidence.

We have revised the manuscript, with additions and substantive changes highlighted in blue in both the main paper and the supplement.

  • Generalist learning and specialist replacement. We clarified how one policy learns navigation decisions across tasks while specialists can be upgraded independently. With the policy fixed and foresight disabled, replacing LocateAnything with GPT-6 Astra as the stop verifier raised the success rate by 1.7 percentage points on R2R-CE and 1.8 points on HM3D-OVON.
  • Tool contributions on the final checkpoint. We repeated the ablations with the model used for the main results. Foresight gave the larger success-rate gain on route following, while stop verification gave the larger gain on object search.
  • Evaluation protocol and computational cost. We clarified that training uses simulator-rendered candidate views and that evaluation generates them online from observations available before each move. We also documented active scans, stopping criteria, and per-decision computation. Across the tested foresight backends, geometric rendering achieved the highest success rate on both benchmarks and was faster than OmniRoam.

We appreciate the reviewers’ suggestions and positive feedback. Thank you for your time and consideration.

We believe the additional experiments solidly validated our claims. We are grateful for the opportunity to improve our work and hope the AC finds these revisions satisfactory.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.