acceptodds
Under review as a conference paper at ICLR 2027

NavWorld: Scaling Long-Horizon Outdoor Language Navigation with Crowdsourced Street Imagery

Abstract

While language-conditioned navigation has made substantial progress in indoor environments and autonomous driving, long-horizon instruction following for ground robots in outdoor pedestrian environments remains challenging. We present NavWorld, an ecosystem for long-horizon, language-conditioned outdoor navigation built around a simple idea: automatically turning large-scale, geographically diverse street imagery into language-labeled trajectories. Our data engine sources candidate navigation trajectories from Mapillary, a large-scale crowdsourced platform for ground-level imagery spanning diverse geographic regions and capture conditions. It then filters sequences for geometric and perceptual quality and automatically generates grounded high-level instructions, intermediate subtasks, and stopping conditions from visual observations and trajectory geometry. We train a language-conditioned navigation policy that generates language subtasks from high-level instructions, tracks progress across subtasks, and predicts continuous motion from monocular egocentric observations. We evaluate NavWorld in Habitat-GS, a photorealistic outdoor navigation simulator, where it achieves a 43% success rate on NavWorld-labeled instructions, more than 9x that of the navigation Vision-Language-Action (VLA) baselines.

Primary area:
datasets and benchmarks
Keywords:
navigation, egocentric data, VLA, robotics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vHMEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents NavWorld, a framework for long-horizon language-conditioned outdoor navigation. The authors propose an automated data pipeline that converts crowdsourced Mapillary imagery into navigation trajectories with language instructions and hierarchical subtask annotations, resulting in a dataset of over 1.5 million frames across 365 cities. Based on this dataset, they train a navigation VLA model that combines a Qwen3-VL backbone for subtask generation and progress tracking with a flow-matching action expert for continuous motion prediction. Experiments on 400 episodes in Habitat-GS show that NavWorld achieves a 43% success rate with NavWorld-generated instructions, substantially outperforming the evaluated navigation VLA baselines. The authors also investigate the impact of training data scale on navigation performance.

Critical strengths

1. Scalable data construction for outdoor language navigation. The paper proposes an automated pipeline that transforms crowdsourced Mapillary imagery into language-labeled navigation trajectories. By leveraging publicly available data, it constructs a geographically diverse dataset with over 1.5 million frames across 365 cities, reducing the need for costly robot-specific data collection and manual annotation.

2. Promising zero-shot outdoor navigation performance. NavWorld demonstrates substantial improvements over existing baselines on unseen Habitat-GS environments. The evaluation provides evidence that the automatically constructed dataset and trained policy can support effective closed-loop outdoor language navigation.

Critical weaknesses

1. Limited methodological novelty. The methodological novelty of the paper is relatively limited. The proposed NavWorld-VLA largely builds upon existing VLA architectures, combining a pretrained VLM backbone, hierarchical language subtasks, and a flow-matching action expert. While these components are adapted to outdoor navigation, the overall design closely follows established approaches such as π0.5, and the paper does not clearly demonstrate substantial technical innovations beyond their integration. Consequently, the main contribution appears to lie in data construction rather than model development.

2. Insufficient ablation studies to validate the proposed model design. The experimental evaluation primarily focuses on comparisons with existing methods and data scaling, with limited analysis of the proposed model components. In particular, the paper lacks ablations examining the effectiveness of hierarchical subtask generation, subtask-aware attention masking, and the multi-stage training strategy. Without these experiments, it is difficult to determine which design choices contribute to the reported performance improvements or whether the proposed architecture is necessary to achieve them.

Overall, I find the proposed dataset valuable, and the authors demonstrate promising navigation performance by leveraging this data with a relatively standard model architecture. However, I believe the primary contribution of this work still lies in the dataset, while both the problem formulation and the methodological design largely build upon existing approaches. Furthermore, the paper lacks several important ablation studies needed to validate the contributions of individual model components. These concerns limit my assessment of the paper's overall novelty and technical contributions, leading to my final rating.

Questions for authors
  1. Are the automatically generated instructions further filtered or validated after annotation? How do the authors handle potentially incorrect, ambiguous, or visually ungrounded instructions, and how is the overall annotation quality evaluated?
  2. Could the authors provide ablations on hierarchical subtask generation and subtask-aware attention masking? How much does each component contribute to navigation performance?
  3. Could the authors provide ablations on the three-stage training strategy? In particular, how does Stage 3 affect performance across different navigation lengths and numbers of subtasks?
3 comments

Dear Reviewer vHME,

Thank you for your careful review. We are glad you found our data construction scalable and geographically diverse, and our zero-shot outdoor navigation results promising. We respond to your concerns and questions below.

W1: Limited methodological novelty.

We agree that our main contribution is the data engine and dataset, and we designed it that way. We believe a diverse, large-scale outdoor dataset would further improve current VLN policies and help them generalize to outdoor environments. NaVILA's released weights reach 0.50% SR on our Habitat-GS episodes, and with our data, it improves to 16.75% SR. NavWorld provides a scalable way to produce outdoor supervision from crowdsourced imagery: 1.58M frames across 365 cities, with ordered, boundary-aligned subtask instructions.

We propose the NavWorld model to leverage the dataset's structure: it generates the next subtask, tracks progress, and restricts action prediction to the active subtask's context. The matched-data experiments in the open response show that the dataset and this design each contribute:

  • The data transfers to other architectures. Training NaVILA on our dataset raises its SR from 0.50% to 16.75%.
  • The model design matters given the same data. NavWorld, trained on the same trajectories with subtask supervision, reaches 31.25% SR: 14.5 points above NaVILA trained on our data, and 15.0 points above a flat policy trained without subtasks (16.25%).

W2: Insufficient ablation studies to validate the proposed model design.

Q2: Could the authors provide ablations on hierarchical subtask generation and subtask-aware attention masking? How much does each component contribute to navigation performance?

We thank the reviewer for vHME’s advice on showing ablations for hierarchical subtask generation and attention masking. We train a flat policy without subtask supervision or prediction and evaluate it on the same 400 Habitat-GS episodes:

ModelSR ↑SPL ↑OSR ↑NE (m) ↓
Flat policy (no subtasks)16.2514.7038.0010.69
NavWorld (final model)31.2530.3538.508.07

Removing the subtask hierarchy halves SR and SPL, while OSR barely changes (38.00 vs. 38.50). Both policies reach the goal region about equally often, but the flat policy stops there only half as often. The hierarchy's main contribution is therefore progress tracking: knowing which part of the instruction is done and when to stop, which is what subtask generation and the controller are for.

Please also refer to our central thread for more details on this topic.

Q3: Could the authors provide ablations on the three-stage training strategy? In particular, how does Stage 3 affect performance across different navigation lengths and numbers of subtasks?

In our final model, we do not use Stage 3. The new comparisons use a two-stage NavWorld model. They demonstrate that the reported benefits are achievable without the Stage 3 curriculum. To reiterate:

  • Stage 1 adapts the VLM backbone with cross-entropy supervision of subtask text, discrete actions, and transition and termination decisions.
  • Stage 2 initializes from Stage 1 and adds the flow-matching action expert on top of the adapted backbone to enable continuous action predictions.

We hope these results address your concerns, and we're happy to answer any further questions.

Reviewer_vHMEOpenReview ↗

Response to Authors

Thank you for the additional experiments and clarifications. I appreciate the authors' efforts. The new results provide useful evidence for the effectiveness of the dataset and hierarchical design. But I think my main concern about methodological novelty remains, as the overall architecture still largely builds on existing approaches.

However, considering the demonstrated effectiveness of the approach and the paper's primary contribution in dataset construction, I have decided to raise my rating to 3.

Dear Reviewer vHME,

Thank you again for your valuable time and for revisiting your assessment. We’re glad the new results helped clarify the value of the NavWorld dataset and the role of subtask supervision in outdoor navigation.

Reviewer_QaMvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents NavWorld, a framework for long-horizon outdoor vision-language navigation. The authors introduce an automated data engine that converts crowdsourced Mapillary imagery into navigation trajectories with hierarchical language annotations, constructing a large-scale dataset containing over 1.5 million frames across 365 cities. Based on this dataset, they develop a language-conditioned navigation policy that combines hierarchical subtask generation, progress control, and a flow-matching action expert for continuous trajectory prediction. Experiments in Habitat-GS demonstrate promising zero-shot outdoor navigation performance, substantially outperforming the evaluated VLA baselines.

Critical strengths

The paper introduces a large-scale, geographically diverse outdoor navigation dataset with an automated data collection and annotation pipeline.

NavWorld achieves substantial improvements over existing navigation VLA baselines in zero-shot outdoor navigation.

The proposed hierarchical navigation framework is well motivated, and the paper is clearly structured with encouraging data scaling results.

Critical weaknesses
  1. Insufficient validation of the automatically generated dataset.

Dataset construction is a central contribution of this paper. Although the authors introduce a comprehensive filtering and annotation pipeline, the quality of the generated data is not sufficiently validated, particularly regarding trajectory correctness and annotation accuracy. More importantly, the action labels used for training are derived from crowdsourced camera poses, which may contain localization and orientation errors. The authors should provide quantitative evaluations of the generated data to establish the reliability of the navigation supervision.

  1. Concerns regarding the fairness of baseline comparisons.

NavWorld is trained on a large-scale outdoor navigation dataset, while the compared baselines directly use their original pretrained models without adaptation to the same training data. Such comparisons make it difficult to distinguish the benefits of the proposed dataset from those of the model architecture. Furthermore, several baselines exhibit substantial gaps between SR and OSR, suggesting that stopping behavior may be an important factor behind the reported performance differences. This raises additional concerns about whether the large improvements in SR accurately reflect improvements in navigation capability. The authors should provide a more detailed analysis of these results and consider comparisons under matched training conditions.

  1. Insufficient ablation studies on the proposed hierarchical navigation framework.

The proposed model introduces several components, including subtask-aware attention and hierarchical progress control. However, the paper lacks ablation studies that isolate their individual contributions. In particular, the ability to decompose long-horizon instructions into subtasks and manage transitions between them is an interesting aspect of the proposed framework, yet no dedicated experiments are provided to demonstrate its effectiveness. The authors should evaluate the quality of subtask generation and progress control, and investigate how these components contribute to long-horizon navigation performance.

Questions for authors

Can the authors provide quantitative validation of the generated dataset, particularly the accuracy of trajectory-derived action labels, subtask boundaries, and language annotations?

Can the authors compare against baselines adapted to the same outdoor training data and further analyze the substantial SR–OSR gaps to clarify the contribution of navigation execution versus stopping behavior?

Can the authors provide ablations of subtask-aware attention and hierarchical progress control, along with direct evaluations of subtask generation and transition accuracy?

Whether the dataset will be released?

3 comments

Dear Reviewer QaMv,

Thank you so much for your valuable comments and feedback. We are glad you found our paper insightful and easy to follow, and that you acknowledged our model’s performance. We respond to your concerns and questions below:

W1: Insufficient validation of the automatically generated dataset.

Q1: Can the authors provide quantitative validation of the generated dataset, particularly the accuracy of trajectory-derived action labels, subtask boundaries, and language annotations?

We agree with the reviewer that quantitative validation is required to ensure quality. We therefore designed a new verification pipeline that verifies (1) annotation quality and (2) trajectory correctness.

We used GPT-6 Astra to check a 5000-sample subset for the correctness and unambiguity of actions, landmarks, and goals. We also verified score consistency between human and GPT-6 Astra judgments, achieving 92.7% agreement.

The verifier result on annotation quality is presented below:

MetricActionsLandmarksGoalsAll kinds
Correct91.6%96.7%93.8%94.7% [94.4, 95.0]
Unambiguous88.4%90.6%97.7%91.2% [90.8, 91.6]
Both81.1%87.5%91.6%86.3% [85.9, 86.7]

We also asked GPT-6 Astra to check whether the derived action labels match what the view presents. It found that 96.2% of subtasks have matching trajectories and views.

The results above establish that (1) our dataset’s annotation is correct and unambiguous, and (2) our rigorous filtering pipeline produced accurate trajectories.

Please also see the central thread open to everyone for more details.

W2: Concerns regarding the fairness of baseline comparisons.

Q2a: Can the authors compare against baselines adapted to the same outdoor training data…?

We appreciate reviewer QaMv’s feedback on adapting baselines to NavWorld data to better isolate the effects of our NavWorld dataset and proposed model architecture. To address this, we trained NaVILA stage 3 using our 0.5x dataset and evaluated it against the held-out Habitat-GS 400-episode evaluation.

Compared to the released NaVILA model, our dataset improved its success rate (SR) from 0.50% to 16.75%, while the oracle success rate (OSR) remained nearly unchanged. Our full model shows a 31.25% SR. We draw the conclusion that (1) our dataset enables popular VLNs to achieve better outdoor navigation performance; (2) our model, with subtask supervision, further improves it.

W3: Insufficient ablation studies on the proposed hierarchical navigation framework.

We thank the reviewer QaMv for pointing out that further ablation studies would strengthen the contribution of our hierarchical navigation framework. We conducted an ablation study to show how subtask decomposition and hierarchical control improve our final model’s performance by training a flat policy without subtask decomposition and hierarchical control, and evaluated it on the same 400 held-out Habitat-GS episodes:

ModelSR ↑SPL ↑OSR ↑NE (m) ↓
Flat policy16.2514.7038.0010.69
NavWorld model31.2530.3538.508.07

With nearly doubled SR, unchanged OSR, and NE reduced by 2.62 meters, we claim that hierarchical and subtask supervision have significantly improved the model’s ability to track progress and achieve task success.

Q2b: …further analyze the substantial SR–OSR gaps to clarify the contribution of navigation execution versus stopping behavior?

We agree that navigation execution and stopping behavior should be distinguished. Our flat-policy ablation also helps clarify this: the full and flat policies have similar OSR, while the full model nearly doubles the SR. The strongest demonstrated benefit is therefore improved successful completion at the instructed endpoint, which is crucial for showing the policy's understanding of the instruction and view.

Regarding the SR-OSR gap, we believe SR more meaningfully reflects a model's navigation capability and instruction understanding. To show this, we have designed a deterministic policy as below and evaluated it in the same setting on Habitat-GS instruction. The table presents the deterministic policies versus our baselines and our model.

ModelSR ↑SPL ↑OSR ↑NE (m) ↓
Deterministic: Walk straight, without stopping0.000.0041.2516.28
NaVILA0.500.4043.826.33
StreamVLN3.53.333.519.60
InternVLA-N15.04.343.016.87
NavWorld (final model)31.2530.3538.508.07

Note that an instruction-agnostic straight-walking policy achieves 41.25% OSR despite never issuing STOP, demonstrating that substantial OSR can arise without using the instruction.

Q4: Whether the dataset will be released?

Yes, please find the anonymized release of our dataset in the central thread. We will also release the code and model weights soon!

The author’s experiments on Navila have only deepened my concerns. The comparisons made using the author’s original version are inherently unfair—a point also raised by other reviewers. Furthermore, I do not understand why only 0.5x of the data was used.

While I agree that using GPT-6-Astra to verify the data is reasonable for instruction annotations, I am not convinced that it ensures sufficient precision for action labels; determining a vague direction is not the same as achieving accurate action labeling. Consequently, this does not alleviate my concerns regarding the quality of the data.

In summary, I maintain my original assessment: weak reject.

Reviewer_WWLFOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors build a data engine that automatically generates navigation trajectories with language instructions from crowdsourced street-view images in Mapillary. Using this engine, they create the NavWorld dataset, a geographically diverse outdoor navigation dataset with language instructions. It covers 365 cities, more than 3,700 km, and more than 1.5M frames. In addition, the authors propose NavWorld-VLA, a hierarchical VLA model trained on this data.

Critical strengths

The authors develop a data engine that automatically converts crowdsourced Mapillary street-level imagery into language-annotated outdoor navigation trajectories. Using this engine, they construct a large-scale dataset spanning 365 cities and over 1.5M frames.

Critical weaknesses
  • All the baselines are models trained on indoor data, while the proposed method is trained on outdoor data. Therefore, the performance gap can be explained by the domain difference. There is no comparison with baselines trained on the same data, and there is also no ablation of the model components. As a result, I cannot judge whether the improvement comes from the data or from the model.
  • NavWorld instructions are generated with the same splitting rules and the same labeler as the training data. This means that these instructions are in-distribution only for the proposed method. Therefore, this setting is not a zero-shot evaluation, unlike what the paper claims.
Questions for authors
  • Do you have results for the baselines trained on the NavWorld data or on the training split of Habitat-GS?
  • How is the quality of the generated dataset verified?
  • Can you show ablation results for the components of the proposed model?
1 comment

Dear Reviewer WWLF,

Thank you so much for your time and valuable comments. We respond to your concerns below:

W1a: All the baselines are models trained on indoor data, while the proposed method is trained on outdoor data. Therefore, the performance gap can be explained by the domain difference.

We would first like to clarify that NaVILA’s published training data include real-world human touring videos with residential exterior and patio navigation; its data are therefore not exclusively indoor (e.g., their training episode nY8eEMb3i88_1-0). Therefore, we believe our evaluation domain is seen in their training.

W1b: There is no comparison with baselines trained on the same data, and there is also no ablation of the model components. As a result, I cannot judge whether the improvement comes from the data or from the model.

Q1: Do you have results for the baselines trained on the NavWorld data or on the training split of Habitat-GS?

To show (1) our dataset can benefit existing popular VLN models, and (2) our subtask decomposition can improve models' navigation success, we trained NaVILA on the NavWorld dataset:

ModelSR ↑SPL ↑OSR ↑NE (m) ↓
NaVILA, released weights0.500.4043.826.33
NaVILA, trained on our dataset16.7515.9741.509.14
NavWorld (ours), full model31.2530.3538.508.07

Compared with its released weights, training increases the success rate (SR) from 0.50% to 16.75% and SPL from 0.40% to 15.97%. Our full NavWorld model, trained on the same trajectories with additional subtask supervision, achieves 31.25% SR, exceeding the trained NaVILA baseline by 14.50 percentage points.

Please also see our main thread open to all reviewers for more details.

W2: NavWorld instructions are generated with the same splitting rules and the same labeler as the training data. This means that these instructions are in-distribution only for the proposed method. Therefore, this setting is not a zero-shot evaluation, unlike what the paper claims.

We would like to clarify that we report two separate instruction settings in our paper: original Habitat-GS instructions (zero-shot) and instructions generated by our pipeline on the same episodes. The shared-labeler concern therefore applies to the latter setting, but not to the results using the original Habitat-GS instructions. In both settings, our model significantly outperforms the baselines.

Q2: How is the quality of the generated dataset verified?

We agree with the reviewer that a detailed analysis and verification of dataset generation is crucial to enforce quality. Therefore, we conducted a new dataset-quality evaluation of annotated instructions, trajectories, and actions.

We conducted the evaluation with GPT-6 Astra, asking it to check each subtask for the correctness and unambiguity of actions, landmarks, and goal descriptions. We also verified with humans to ensure consistency between the automatic verifier and human judgment, and achieved 92.7% agreement.

The final verifier result on the instruction, on a 5000-sample subset, is presented below:

MetricActionsLandmarksGoalsAll kinds
Correct91.6%96.7%93.8%94.7% [94.4, 95.0]
Unambiguous88.4%90.6%97.7%91.2% [90.8, 91.6]
Both81.1%87.5%91.6%86.3% [85.9, 86.7]

We also asked GPT-6 Astra to evaluate trajectory-view consistency for each sample, which measures whether top-down route geometry matches the motion seen in image frame sequences. It yielded that 96.2% of subtasks have matching trajectory and view.

The results above establish that (1) our dataset’s annotation is correct and unambiguous, and (2) our rigorous filtering pipeline produced accurate trajectories.

Please also refer to the central thread open to all reviewers for more details on this topic.

Q3: Can you show ablation results for the components of the proposed model?

We thank reviewer WWLF for requesting an ablation experiment to show the architectural contribution of our NavWorld model.

To show the importance of subtask supervision and prediction, we trained another flat policy in which no subtask is predicted, and no hierarchical mask is applied; everything else is kept the same as in our proposed NavWorld model. We evaluate it on the same Habitat-GS 400 episodes as all other models. The evaluation result is below:

ModelSR ↑SPL ↑OSR ↑NE (m) ↓
Flat policy16.2514.7038.0010.69
NavWorld model31.2530.3538.508.07

These results show that, with hierarchy and subtask supervision, we nearly doubled SR and reduced navigation error by 2.62m, while OSR changed little; we conclude that subtask decomposition helps the model track progress more precisely and improves overall navigation performance.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback and recognition of our dataset’s scale and diversity (WWLF, QaMv, vHME) and the paper’s clear presentation (QaMv, vHME).

We wish to emphasize that:

  • NavWorld contributes a scalable data engine that converts geographically diverse crowdsourced imagery into outdoor navigation trajectories with aligned task-level and subtask-level language supervision.
  • Our additional experiments establish two practical benefits: the resulting data improves an existing navigation architecture, and using its hierarchical supervision further improves successful task completion.

We have also released the dataset used, and we will release the code soon:

We thank the reviewers again for the feedback and address three shared concerns through new experiments and analysis: baseline fairness and dataset utility, the contributions of subtask-enabled learning and individual model components, and annotation quality and verifier reliability.

Protocol for the new experiments. In the original submission, we constructed overall instructions by concatenating subtask descriptions. For the revised experiments, we use L1 instructions that describe the overall task over a window of subtasks to better show the benefit of subtask supervision to navigation policies.

NavWorld and the adapted NaVILA baseline are trained on the same 0.5× trajectory subset with NavWorld task-level (L1) instructions and evaluated with original instructions on the same 400 held-out Habitat-GS episodes. NavWorld additionally receives L2 subtask supervision during training and predicts these subtasks at inference; NaVILA uses L1 instructions without L2 supervision.

AuthorsOpenReview ↗

Our dataset can help baseline VLN perform better in zero-shot outdoor navigation

We appreciate reviewer WWLF, QaMv’s concern about baseline fairness and how our dataset could help with our baseline models or other VLNs.

To address this, we train NaVILA stage 3 using our 0.5x dataset and evaluate on the same 400 held-out Habitat-GS episodes. Compared with its released weights, training increases success rate (SR) from 0.50% to 16.75% and SPL from 0.40% to 15.97%. Our full NavWorld model, trained on the same trajectories with additional subtask supervision, achieves 31.25% SR, exceeding the trained NaVILA baseline by 14.50 percentage points.

ModelSR ↑SPL ↑OSR ↑NE (m) ↓
NaVILA, released weights0.50.443.826.33
NaVILA, trained on our dataset16.7515.9741.59.14
NavWorld (ours), full model31.2530.3538.58.07

This confirms the usefulness of our dataset by showing that the existing popular model architecture can achieve better performance with our diverse, high-quality data; it also shows that our model outperforms the baseline under the same training data and evaluation settings. We are also excited to see future work extending our sub-task decomposition and showing its value in even longer-horizon tasks.

AuthorsOpenReview ↗

NavWorld’s curation and annotation pipeline produced a high-quality dataset

We thank Reviewers WWLF, QaMv, and vHME for asking how we validate our dataset.

Instruction-View Consistency

We introduce a new data quality evaluation experiment that uses a VLM-based verifier (GPT-6 Astra Medium) to assess correctness and unambiguity in actions, landmarks, and goals referenced in our annotation.

We also evaluated the verifier by comparing against human annotations of 50 randomly sampled subtasks. Across 928 binary judgments, human–verifier agreement was 92.7%.

We then applied the verifier to 5,000 randomly sampled training subtasks, covering 41,922 phrases. The verifier rated 94.7% of phrases as correct, 91.2% as unambiguous, and 86.3% as both.

The tables below provide the evaluation rubric and evaluation results.

Actions (what to do while moving)Landmarks (anything to orient by)Goals (where the subtask ends)
CorrectThe frames show it happening as stated: direction and side (bends under 30° count as straight)Visible in the subtask's frames, and not on the wrong sideDoesn't stop you with ⅓ or more of the subtask left, or after its end
UnambiguousOnly one reasonable way to do it hereA landmark placed on one side doesn't also appear on the otherOnly one place satisfies it
MetricActions (12,298)Landmarks (22,430)Goals (7,194)All kinds (41,922)
Correct91.6%96.7%93.8%94.7% [94.4, 95.0]
Unambiguous88.4%90.6%97.7%91.2% [90.8, 91.6]
Both81.1%87.5%91.6%86.3% [85.9, 86.7]

Trajectory-View Consistency

We additionally evaluated trajectory consistency (top-down route geometry vs. motion seen in image frames) using a VLM verifier. Compared against human annotations of 50 randomly sampled trajectories, the human-verifier agreement was 98%.

Applied to 5,000 randomly sampled training subtasks, trajectory matches the motion in the frames in 96.2% [95.7, 96.7] of subtasks.

This result shows that: (1) the verifier is consistent with human judgment; (2) our dataset’s annotation is correct and unambiguous; (3) after our filtering pipeline, the retained trajectory and action faithfully reflect the motion captured by the frames.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.