- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ScaleNav, a framework for language-conditioned local navigation that leverages action-free Internet egocentric videos to learn video/navigation priors. Using a three-stage training process (video pretraining, action alignment, and joint fine-tuning) built upon a unified world-action model with a Diffusion Transformer backbone, the method aims to translate Internet-scale visual data into executable robot actions. The authors evaluate ScaleNav on two simulation benchmarks as well as on a real quadruped robot, demonstrating improvements over several baselines.
Critical strengths
An interesting setting may open-up more discussions in agentic era
Critical weaknesses
-
Questionable Problem Setting & Missing SOTA Comparisons on Real Robots The core setting focuses on local navigation where the target is already visible in the initial observation. For such a setting, the visual-to-action mapping should arguably be well-handled by existing SOTA foundation models and recent visual-language-action navigation frameworks. For instance, recent methods like NVIDIA LightNav or similar SOTA approaches likely achieve extremely high performance on real robots even under zero-shot or simpler adaptation settings. The authors fail to compare ScaleNav against these strong, contemporaneous SOTA baselines on real-world hardware, making it difficult to judge the genuine necessity and performance upper bound of the proposed multi-stage training framework. The simulation experiments also rely heavily on outdated baselines or adapters that feel somewhat outdated. Given the rapid progress in embodied AI and foundation navigation models, more recent and competitive SOTA simulation baselines should be included to validate the true efficacy of the proposed design.
-
Limited Methodological Innovation From a technical standpoint, the framework largely consists of stitching together existing techniques—such as combining a pre-trained video diffusion backbone, a standard chunk-wise causal attention mechanism, a basic action alignment stage, and standard flow-matching objectives. There is a lack of deep conceptual or architectural innovation tailored specifically to the unique challenges of local navigation. Furthermore, the current real-robot and simulation results fail to convincingly demonstrate that these combined components yield a decisive, breakthrough advantage over properly tuned existing pipelines.
Questions for authors
Given that the target is already visible in the initial observation, what specific failure modes of existing SOTA methods does ScaleNav fundamentally solve, and to what extent do the individual training stages contribute to overcoming these specific failure modes rather than just providing a brute-force data scaling effect?
5 comments
I thank the authors' for the detailed explanation. After reading the rebuttal, it sounds like this work is a simplified version of SparseVideoNav. SparseVideoNav also leverages egocentric videos for pretraining to generate executable actions without requiring action labels, while handling more complex scenarios, even when the target object is initially out of view. Compared to SparseVideoNav, ScaleNav offers little methodological advantage and operates in a noticeably simpler setting.
Could the authors elaborate more edges compared to SparseVideoNav?
Differences from SparseVideoNav: Video Pretraining and Task Setting
Thank you for highlighting SparseVideoNav, which is closely related to our work and makes valuable contributions to beyond-the-view navigation. SparseVideoNav builds video prediction and inverse-dynamics action learning around a self-collected navigation corpus, with action generation dependent on the quality and consistency of predicted futures. ScaleNav instead learns navigation priors from independently sourced Internet videos and transfers them to robot control through a shared video-action backbone that continues to adapt under action supervision. Our focus is on scalable transfer from Internet video to executable navigation, together with a complementary practical objective: reliable execution of visible-target instructions for direct human-robot interaction and as a reusable low-level capability in agentic systems.
1. Data utilization and training framework
SparseVideoNav uses 140 hours of self-collected navigation videos and manually annotated instructions to train a history-conditioned model for long-horizon sparse future prediction and distill it for deployment. The purpose of this video pretraining is to provide long-horizon visual foresight for subsequent inverse-dynamics action generation. It then generates futures from observations in the same corpus and uses DA3-estimated camera motion from the generated frames to supervise a separate inverse-dynamics action predictor. (Sections 3.2-3.6)
The future predictor used by their inverse-dynamics model is learned during video pretraining and frozen during action learning. Video pretraining must therefore establish reliable scene and motion prediction for subsequent action generation. This places demanding requirements on pretraining video quality, as action supervision cannot further refine the predictor and its errors can propagate into control. Their own failure analysis reports future-prediction errors and visual-action mismatches, including incorrect turns and stops. The authors hypothesize that their decoupled design may limit generalization and identify joint video-action training as future work. (Section 3.6 and Appendix A.1)
Our central idea is to use existing Internet egocentric videos as an independent, scalable source of navigation priors, learning from diverse real-world scenes and the visual dynamics of human navigation.
Our video pretraining aims to learn navigation priors from real-world visual dynamics for subsequent transfer and adaptation to robot control. It therefore need not establish a fixed future predictor that is already reliable enough for action decoding. Videos with varied appearances, camera motions, and recording conditions can contribute transferable visual and motion knowledge. We believe this paradigm is better suited to scaling navigation learning with larger, more diverse Internet-video corpora. Our earlier no-Internet ablation and matched-budget video-source control examine the value of this transfer.
2. Task setting and practical relevance
SparseVideoNav evaluates both instruction-following and beyond-the-view navigation, emphasizing long-horizon foresight for initially unseen targets. Its deployed four-step model reports 25.0% BVN success across six real-world scenes. This demonstrates progress in search and route selection, while reliable completion remains challenging in that setting. (Table 1)
Our setting starts from a practical requirement: dependable execution of local goals in human-robot interaction and agentic systems. A person or high-level agent can specify a visible target through language without supplying metric coordinates or low-level motions. We view this as a useful interface for delegating local navigation.
For people and high-level agents, local navigation must provide dependable execution that subsequent actions and decisions can build on. When this capability is invoked repeatedly, a failed local command can disrupt the overall task, making robustness and repeatability important. Whereas SparseVideoNav emphasizes extending navigation to initially unseen targets through long-horizon foresight, our setting prioritizes reliable completion of specified local goals as a reusable execution capability.
In our evaluations, ScaleNav achieves 94.9% SR on HabitatGS-LN, 86.5% on single-view Short-Horizon OVON, and 83.3% across 60 real-robot trials. These results support ScaleNav's ability to execute local navigation instructions robustly across the evaluated simulation benchmarks and real-world tasks.
Response to Reviewer njUs
Thank you for the detailed feedback. We clarify the challenges of navigation to initially visible targets, add comparisons with recent navigation models, and analyze failure modes and the roles of training and model components. We also clarify the core contribution and provide an update on the ongoing real-robot comparisons.
1. Task difficulty and recent baselines
An initially visible target provides a reference for identifying the instructed object, but does not specify a traversable route, executable motions, or a stopping decision. Viewpoints change and the target may leave the field of view. Our world-action model connects motion generation with predicted visual changes, while the initial-observation anchor preserves the reference needed to interpret the instruction.
To examine recent models’ performance in this setting, we added LightNav, NaVILA, and OmniNav. ScaleNav retains the highest SR among the evaluated single-view methods on both benchmarks. The complete comparison, including three-view OmniNav, is in CQ1 of our Common Questions response.
We are also conducting real-robot comparisons with recent navigation baselines and will provide the results once these experiments are complete.
2. Failure modes and training contributions
To identify the failures reduced by ScaleNav, we manually reviewed and annotated, in random order, all 104 failed rollouts from ScaleNav, its variants without Internet-video training and without the anchor, and fine-tuned StreamVLN and Uni-NaVid on the same 157 HabitatGS-LN tasks. Each rollout is assigned to one primary failure category.
Target grounding and maintenance: persistently approaching the wrong object category, or deviating before arrival without resuming the target approach, judged from the approached object and motion direction. These cases concern grounding across viewpoints.
Wrong instance: approaching an object of the correct category but not the instance specified by the instruction, judged against the initial reference and instruction constraints. These cases require grounding attributes and spatial relations.
Collision or blockage: physical contact with scene obstacles or an inability to move past them. Avoidance requires estimating clearance and generating motion that accounts for the robot’s footprint.
Stopping timing: stopping outside the success range in front of the correct target, or continuing forward after arrival, assessed from stopping records and target distance. These cases concern arrival judgment and termination.
Primary failure counts on the same 157 HabitatGS-LN tasks.
The main improvements are in grounding and stopping. ScaleNav has 2 grounding-related failures versus 25 for fine-tuned Uni-NaVid and 24 for StreamVLN, including 0 wrong-instance errors versus 5 and 7. Stopping errors fall from StreamVLN’s 11 to ScaleNav’s 2.
The ablations help interpret these gains: removing Internet-video training increases target-grounding and maintenance failures from 2 to 7. These results suggest that Internet-video training helps maintain correspondence between the instruction and the target as the viewpoint changes. In a task-wise comparison between the variant without the anchor and ScaleNav, 6 grounding failures and 3 premature stops become successes. With 2 reverse changes, this gives a net gain of 7 successes. We believe the anchor’s preserved initial visual context supports both instruction grounding and arrival judgment; zero-motion supervision directly trains termination after arrival.
The failure analysis links specific improvements to Internet-video training and the anchor. The proposed mechanisms are inferred from these results; a finer attribution to individual training stages remains for further study.
3. Contribution
We agree that several components build on prior work. Our contribution is to establish the value of Internet egocentric videos as a scalable source of navigation experience and provide a framework that uses the learned navigation priors to generate executable actions, without requiring action labels for those videos. ScaleNav performs strongly in simulation and navigates successfully on a real quadruped. CQ2 in our Common Questions response details this contribution.