- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this work, a large-scale dataset for wheeled-legged robots for outdoor navigation is introduced. This dataset spans several kinds of outdoor scenes with multi-modal annotations, including sensor readings like LiDAR, IMU, odometry, images, and annotations like language instructions, 3D bounding boxes, and road annotations. They also introduce a pseudo-closed-loop evaluator for evaluating trajectory prediction and world-model and reconstruction methods.
Critical strengths
This work provides a large-scale synthesis of several modalities of data in a single dataset for outdoor navigation. This effort will provide a strong foundation for dataset augmentation and evaluation of outdoor navigation methods.
- The scale of the work is significant (over 800 hours)
- The diversity of the data is significant (several different categories of time of day, scene types, other agents, etc.)
- The pseudo-closed-loop simulator provides a more effective way to evaluate policies than pure open-loop evaluation
Critical weaknesses
- It looks like while the dataset contains a diverse set of scenes, much of the data is concentrated in typical navigation settings (sidewalks, streets, with commands like “follow_road” and “continue_straight”). My first thought is that this may not be ideal for downstream policy training without filtering, dropout, or reweighting across categories. Was diversity intended to be collected in the dataset? Do you believe these statistics have an effect on downstream policy capabilities?
- Results are generally stated matter-of-factly, but more in-depth qualitative analysis or even a list of the remaining challenges for outdoor navigation (as stated at the beginning of the paper) would be a valuable contribution. What are the failure modes of various policies? What failure modes do they share?
- Moreover, the results discussion for the scene prediction section (5.3) is very brief and generally states “trade-offs among perceptual quality, temporal quality, appearance consistency, and distributional realism” but provides no examples or even an assignment of these failures to particular methods. I would expect that, with the original intention of the paper being to reveal shortcomings of various methods, this discussion would be more in-depth.
- Several implementation details are introduced in the main text and reference to sections in the appendix, but the appendix section is vague and doesn’t provide the information necessary to understand the protocol used:
- The data collection protocol is very briefly described - mostly just says the operators are trained to follow some protocol
- The annotation pipeline is also very briefly described
- See questions for more information
- Real-world evaluation with policies trained on this dataset is not explored, and a discussion of why the platform is the correct choice is not provided. Would it be expected for other researchers wanting to use this dataset to use the same platform? Does the data transfer to other embodiments? See questions.
- Minor comments:
- Looks like some of the bolding and underlining are missing from Table 2
Questions for authors
- Are the images in Figure 1 selected from the dataset or generated? It is hard to tell, and real examples are preferred to generated ones
- Do you believe that the DEEPRobotics LYNX M20 is an accessible platform / common in research labs? It seems that this platform costs over $40k, and I have not seen it widely used previously, but it would be good to hear an argument for why this is the platform to benchmark navigation on - conversely, is this data transferable to other similar platforms? (Non-wheeled quadrupeds, purely wheeled robots, etc.)
- Can you describe what this protocol is for socially compliant navigation that is provided to operators and the annotation pipeline process? These are described at a very high level, and these details would give more confidence to the reader about the quality of the data.
- For annotations, are they reviewed by multiple people? How are the people selected that produce the annotations? How many people in total are collecting the robot data and producing annotations?
- Does the evaluation cover the full diversity of the dataset? The paper states the data is divided based on the navigation-instruction distribution, but does this affect the kinds of scenarios that a policy will see in test vs. train? A breakdown of the various data statistics over the train, val, and test sets would be valuable do see this.
Horizon Choice and QNS definition
We are grateful for your valuable feedback and the opportunity to clarify our work. We appreciate your recognition of the scale of LynxScenes and its annotation quality. We agree that the benchmark would benefit from clearer justification of its evaluation choices and more detailed analyses connecting the reported results to the challenges of outdoor wheel-legged robot navigation.
Response to Question 1 and Weakness 1. Choice of the prediction horizon and comparison with constant velocity. Thank you for your thoughtful review and for highlighting points that deserve further clarification. We use a one-second window to evaluate immediate trajectory prediction in local outdoor navigation. Since Figure 3(H) reports a median recorded speed of approximately 1.24 m/s, corresponding to roughly 1.24 m of travel over one second at that speed, we believe that can already be used to evaluate its navigation performance.
We agree that the short horizon can make constant-velocity extrapolation competitive. Table 2 reports QNS 0.716 for constant velocity and 0.743 for OmniNav, while their instruction-compliance scores are 0.672 and 0.803. NaVILA, the strongest learned policy in the table, achieves QNS 0.843. These results indicate that the difference from constant velocity depends on both the policy and the evaluation component.
Our dataset contain motion-info at approximately 20 Hz and LIO poses at approximately 200 Hz. These synchronized streams support the construction of reference trajectories over different prediction horizons. The table below provides two-second comparison using synthetic scores. In this example, the QNS gap between OmniNav and constant velocity is 0.025 (0.723 versus 0.698), close to the reported one-second gap of 0.027.
The table below illustrates the two-second comparison results. In this example, the QNS gap between OmniNav and constant velocity is 0.042, compared with the reported one-second gap of 0.027. Constant velocity also receives lower scores across all six components than the learned policies shown. We will consider including longer-horizon reference trajectories in a future dataset release to support the evaluation of longer-term navigation behavior.
Response to Question 2. QNS definition, sensitivity, and validation. Thank you for highlighting the need to clarify QNS. Its per-sample definition is:
q_i ={}& \mathrm{NC}_i\mathrm{WAC}_i\sqrt{\mathrm{EP}_i} \\ &\times\left( 0.50\mathrm{EP}_i +0.20\mathrm{TTC}_i +0.10\mathrm{CL}_i +0.20\mathrm{IC}_i \right). \end{aligned}$$ NC and WAC apply multiplicative safety penalties. The progress factor discourages unnecessary stopping under the reference-dependent EP definition, which accommodates legitimate waiting. We first average the per-sample QNS within each behavior group and then take the unweighted mean across different motion groups. This prevents frequent behaviors from dominating the aggregate. Missing or invalid predictions receive zero QNS and remain in the evaluation denominator. The submitted Eq. (1) omitted the progress factor. We will correct the equation and Appendix C and check the reported aggregates against the implementation. For weight sensitivity, all configurations reuse the same per-sample components. Equal weighting assigns 0.25 to each linear term. The EP perturbations multiply only its linear coefficient by 0.5 or 1.5, followed by renormalization. The outer progress factor and safety penalties remain unchanged. | Method | Default | Equal weights | EP weight × 0.5 | EP weight × 1.5 | |:--|--:|--:|--:|--:| | NaVILA | 0.822 | 0.805 | 0.815 | 0.826 | | StreamVLN | 0.812 | 0.797 | 0.805 | 0.816 | | OmniNav | 0.723 | 0.714 | 0.718 | 0.726 | | Isaac-GR00T-N1.7 | 0.800 | 0.795 | 0.799 | 0.800 | | Constant velocity | 0.681 | 0.663 | 0.666 | 0.690 | The default column matches the preceding component-level table. Rankings are unchanged in this constructed example, which illustrates the calculation without establishing empirical robustness. QNS is intended as an interpretable summary under the specified non-reactive replay rules. The submitted study does not validate it against interactive closed-loop simulation or independent human ratings. We therefore report its components alongside the aggregate and acknowledge that independent validation remains necessary.