acceptodds
Under review as a conference paper at ICLR 2027

DR-VLA: Learning Human-Like Quadruped Navigation from Egocentric Web Videos via Domain-Invariant Scene Representations

Abstract

Quadruped robots performing outdoor deliveries must interpret complex scenes, anticipate changes, and navigate in socially compliant ways. Robot demonstrations are costly to collect at the scale needed for generalization. Egocentric web videos offer abundant human navigation experience, but differences in viewpoint and motion make this experience difficult to transfer to robots. We introduce DR-VLA, a Mixture-of-Transformers framework that transfers human navigation experience through a domain-invariant scene representation (DISR). The framework combines vision-language understanding and latent world modeling with a flow-matching action expert. We introduce EgoLynx, comprising 10,000 hours of curated egocentric web video (EgoLynx-Web) and 2,000 clips of teleoperated robot navigation collected during real-world deliveries (EgoLynx-Robot). We pretrain the understanding and world experts on EgoLynx-Web without action labels, then adapt the model to robot observations and train the action expert on EgoLynx-Robot to predict executable trajectories. We also propose the Quadruped Navigation Score (QNS), a pseudo-closed-loop metric combining safety, instruction compliance, comfort, and progress. On the held-out EgoLynx-Robot test split, improves QNS by 5% and reduces open-loop final heading error (FHE) by 14.6%, each relative to the best competing result for that metric. Ablations show that web-video pretraining improves generalization, the model’s key components contribute to performance, and DISR outperforms five alternative visual representations as a future-prediction target. Trained separately on NAVSIM v1 without web pretraining, DR-VLA achieves 92.1 PDMS, suggesting that the framework also applies to autonomous driving.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Model,WAM,Scene understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mgLnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims to transfer human navigation experience from large-scale egocentric web videos to quadruped robots by predicting future scene representations instead of pixels or pseudo-actions. The proposed representation is a latent learned by an autoencoder over segmentation masks and relative depth from off-the-shelf foundation models, and it serves as the prediction target of a world expert pretrained on web videos without robot actions. An action expert is then trained on teleoperated robot data to generate trajectories conditioned on the understanding and world experts. The paper also contributes a curated web-video and robot dataset and a pseudo-closed-loop metric, and reports improvements over VLN and VLA baselines on held-out robot data as well as competitive results on NAVSIM.

Critical strengths
  1. Data. The curated egocentric web corpus and the real-world delivery teleoperation data, which the authors plan to release, would be a valuable resource for learning navigation from human videos.

  2. Formulation. Factorizing the policy into an embodiment-shared scene-evolution predictor and an embodiment-specific action generator is a clean and well-motivated way to use action-free human videos.

  3. The paper presentation is clear and easy to follow.

Critical weaknesses
  1. Evaluation. All quadruped results are open-loop or a non-reactive 1-second replay on a test split drawn from the same teleoperation collection, and open-loop metrics are known to correlate poorly with closed-loop performance in navigation and driving. Evaluation on a closed-loop urban navigation benchmark, such as Wanderland [1], is needed to support the claimed navigation and social-compliance gains.

  2. Performance. Even under the pseudo-closed-loop metric, the gains over the strongest baseline on the most important safety metrics, NC and WAC, are marginal, and web pretraining yields similarly small gains.

  3. Representation. The paper does not explain why an implicit autoencoder latent is preferable to directly predicting the future segmentation and depth as explicit targets, as is common in chain-of-thought-style VLA supervision, and provides no comparison with such explicit targets. The claimed relational and domain-invariant properties are also unverified: the masks and depth remove appearance but not viewpoint, and no experiment shows that the representation transfers across domains or that human and robot representations actually align.

  4. Insights. The alternative prediction targets in Table 2 perform only marginally differently from DISR, and the paper offers no analysis of why DISR outperforms DINOv3, V-JEPA, or depth-augmented DINO features.

  5. Baselines. All baselines are trained only on robot data, so Table 1 conflates the proposed representation with the use of extra web data. Comparisons with navigation methods that learn from web videos via pseudo-actions, such as CityWalker and MBRA [2], as well as standard visual navigation baselines such as NoMaD, are needed to show that DISR is a better way to exploit human videos.

  6. NAVSIM. The driving experiment uses no web pretraining and adds a trajectory scorer, so it does not test the paper's central claim of human-to-robot transfer.

[1] Liu, Xinhao, et al. "Wanderland: Geometrically grounded simulation for open-world embodied AI." CVPR. 2026.
[2] Hirose, Noriaki, et al. "Learning to drive anywhere with model-based reannotation." RA-L. 2025
Questions for authors
  1. The web-video captions are generated from the full clip, including subsequent motion, and describe ego-motion. Do they therefore encode the camera wearer's future movement, and if so, how is this consistent with pretraining "without action labels"?

  2. What are the QNS weights, and how were they chosen?

Reviewer_mak5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DR-VLA learns quadruped navigation by pretraining on 10,000 hours of egocentric web video and adapting on 2,000 robot demonstration clips. It predicts future DISR latents built from segmentation, relative depth and temporal relations, using separate understanding, world and action experts. Evaluation uses offline robot trajectory prediction and non-reactive replay, plus a separately trained NAVSIM model.

Critical strengths
  1. Using semantic and geometric structure as prediction targets is well motivated for transferring from web videos. The segmentation and depth teachers are only needed during training.

  2. The ablations are reasonably thorough. Tables 2 and 4 compare alternative targets, individual DISR components and a parameter-matched backbone.

  3. Table 5 provides useful evidence for data efficiency. Web pretraining with 500 robot clips slightly exceeds robot-only training with 2,000 clips on QNS.

Critical weaknesses
  1. The robot evaluation never tests autonomous execution. One-second predictions against non-reactive pedestrians cannot establish sustained yielding, recovery or successful navigation. The authors acknowledge this limitation, but QNS is also not validated against these behavioral outcomes.

  2. DISR removes RGB appearance, but its image-plane masks and relative depth still depend on viewpoint and camera motion. Table 2 shows that DISR is a useful target; it does not isolate whether the advantage comes from better cross-domain transfer or simply better scene supervision on robot data.

  3. The argument for scene-structure transfer over pseudo-action transfer lacks a direct comparison. The DPVO variant in Fig. 3 receives no robot fine-tuning, whereas DR-VLA does. This matters because CityWalker already uses scale normalization and robot fine-tuning to address the transfer gap.

  4. The benefit of future prediction is not separated from ordinary auxiliary scene supervision. All alternative targets in Table 2 still predict the future, so these comparisons do not show whether anticipation itself is necessary.

  5. Section 3 says instructions do not specify when to stop or detour, but Appendix B scores waiting instructions and instructed avoidance sides. Since instructions are generated from full clips (§4.3), it is unclear how much social behavior must actually be inferred from the scene.

Questions for authors
  1. Do you have autonomous robot or reactive rollout results showing that the QNS improvement translates to better yielding and navigation success?

  2. What are the gains from web pretraining for DISR and the strongest alternative target, each compared with its own robot-only version?

  3. How does the DPVO variant perform after scale normalization and the same robot fine-tuning as DR-VLA?

  4. How much performance remains if the world expert predicts current DISR instead of future DISR, keeping model capacity and training data fixed?

  5. Please show the annotation prompts and actual test instructions. Do the social-navigation gains remain when instructions specify only the route?

Reviewer_nYUVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents DR-VLA, which transfers human navigation experience from YouTube egocentric videos to outdoor delivery robots. The core idea is to have the world expert predict future DISR rather than transfer pixels or pseudo-actions. DISR is produced by an autoencoder from segmentation masks and relative depth, and is used only as a training target. The model is first pretrained on 10k hours of action-free web videos, then trained with the action expert on about 17 hours of teleoperated data. On the self-collected test split, DR-VLA beats eight baselines on QNS, and on NAVSIM v1 it reaches 92.1 PDMS.

Critical strengths

The problem is important, and the question of "at what level of abstraction should we transfer" is well framed. DISR is used only as a training target, so SAM 3 and DA3 are not needed at inference, which keeps deployment cheap. Table 2(a) compares five alternative targets under the same conditions, which makes the ablation solid. The data efficiency gains from web pretraining and the scaling curve that has not saturated are convincing. Releasing EgoLynx is valuable for the community.

Critical weaknesses
  1. No same-condition comparison with pseudo-action transfer. The core claim is that transferring scene structure beats transferring pixels or actions. There is a pixel-side control (Cosmos), but the action side only has the qualitative examples in Fig. 3. Navigation actions are planar waypoints, and human trajectories can be used directly after scale normalization, as in CityWalker. The core claim is therefore not fully supported.

  2. "Domain invariance" has no direct evidence. The only evidence is downstream numbers, with no measurement of whether human and robot DISR are more aligned. The invariance also only holds on the output side: the world expert still takes RGB as input, and the input-side gap is left entirely to Stage 2.

  3. DISR as a representation overlaps substantially with prior structured-target world models. DreamVLA, cited in the paper, also uses future depth and semantic features (DINO/SAM) as world prediction targets, but the related work section only compares DR-VLA to "generic visual embedding" targets. In Table 2(a), the Depth + DINO video target already reaches 0.866, and DISR only adds 0.018 QNS on top of that.

Questions for authors
  1. Does DISR also carry human motion bias? DISR is expressed relative to the observer, so its evolution necessarily contains the observer’s own motion, for example human walking speed, head turns that are unrelated to the direction of travel, and cycling, which can be several times faster than the robot. This is the same kind of problem the authors criticize pseudo-action methods for, and the assumption in Eq. 3 that the scene evolution distribution is shared across embodiments is not tested. Could the authors explain how this bias is handled, for example whether the time interval ΔΔ is normalized by speed, and provide a pretraining ablation that uses only walking videos or removes cycling videos?
  2. Can QNS be reproduced? Eq. 8 does not give the four sub-score weights, and does not say whether QNS is computed per sample and then averaged, or whether sub-scores are averaged first and then combined, so QNS cannot be reproduced. Looking at the sub-scores in Table 1, the DR-VLA row does not look consistent with the way the other rows seem to be computed. Could the authors release the weights and the aggregation rule, and double-check this row?
  3. How is the test split constructed? Could the authors provide results split by location or route, or results on new scenes (e.g. nighttime, a new city), to support the claim that web pretraining improves generalization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.