acceptodds
Under review as a conference paper at ICLR 2027

LynxScenes: A Large-Scale Real-World Benchmark for Outdoor Wheel-Legged Robot Perception, Navigation, Imagination and Reconstruction

Abstract

Wheel-legged robots operating in outdoor spaces shared with humans require accurate perception, socially compliant navigation, future-scene reasoning, and scene reconstruction. However, existing datasets typically support these capabilities in isolation and rarely provide dense spatial and behavior-level supervision on real wheel-legged platforms. We introduce LynxScenes, a large-scale real-world dataset and benchmark collected with a LYNX M20 wheel-legged robot. It contains approximately 100K 30-s navigation clips spanning 833.6 h and 3,294.7 km, with 3.0M annotated frames. The synchronized recordings include four fisheye cameras, dual LiDARs, IMU, odometry, and robot states. Annotations comprise 20.82M oriented 3D boxes across 27 categories, 13.89M road-structure annotations, and 20 navigation instructions organized into three layers and aligned with continuous future trajectories. LynxScenes supports four complementary tasks: instruction-conditioned trajectory prediction, BEV-based perception of 3D objects and road structures, future scene prediction, and multi-view scene reconstruction. For trajectory prediction, we further introduce a log-replay-based pseudo-closed-loop evaluator that measures safety, progress, and instruction compliance beyond trajectory imitation. Extensive evaluations across representative navigation, perception, world-model, and reconstruction methods reveal substantial remaining challenges in close-range outdoor robot navigation. The dataset and evaluation toolkit will be publicly released upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Robotics Benchmark,Wheel-Legged dataset

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nqKCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, a large-scale dataset for wheeled-legged robots for outdoor navigation is introduced. This dataset spans several kinds of outdoor scenes with multi-modal annotations, including sensor readings like LiDAR, IMU, odometry, images, and annotations like language instructions, 3D bounding boxes, and road annotations. They also introduce a pseudo-closed-loop evaluator for evaluating trajectory prediction and world-model and reconstruction methods.

Critical strengths

This work provides a large-scale synthesis of several modalities of data in a single dataset for outdoor navigation. This effort will provide a strong foundation for dataset augmentation and evaluation of outdoor navigation methods.

  • The scale of the work is significant (over 800 hours)
  • The diversity of the data is significant (several different categories of time of day, scene types, other agents, etc.)
  • The pseudo-closed-loop simulator provides a more effective way to evaluate policies than pure open-loop evaluation
Critical weaknesses
  • It looks like while the dataset contains a diverse set of scenes, much of the data is concentrated in typical navigation settings (sidewalks, streets, with commands like “follow_road” and “continue_straight”). My first thought is that this may not be ideal for downstream policy training without filtering, dropout, or reweighting across categories. Was diversity intended to be collected in the dataset? Do you believe these statistics have an effect on downstream policy capabilities?
  • Results are generally stated matter-of-factly, but more in-depth qualitative analysis or even a list of the remaining challenges for outdoor navigation (as stated at the beginning of the paper) would be a valuable contribution. What are the failure modes of various policies? What failure modes do they share?
    • Moreover, the results discussion for the scene prediction section (5.3) is very brief and generally states “trade-offs among perceptual quality, temporal quality, appearance consistency, and distributional realism” but provides no examples or even an assignment of these failures to particular methods. I would expect that, with the original intention of the paper being to reveal shortcomings of various methods, this discussion would be more in-depth.
  • Several implementation details are introduced in the main text and reference to sections in the appendix, but the appendix section is vague and doesn’t provide the information necessary to understand the protocol used:
    • The data collection protocol is very briefly described - mostly just says the operators are trained to follow some protocol
    • The annotation pipeline is also very briefly described
    • See questions for more information
  • Real-world evaluation with policies trained on this dataset is not explored, and a discussion of why the platform is the correct choice is not provided. Would it be expected for other researchers wanting to use this dataset to use the same platform? Does the data transfer to other embodiments? See questions.
  • Minor comments:
    • Looks like some of the bolding and underlining are missing from Table 2
Questions for authors
  • Are the images in Figure 1 selected from the dataset or generated? It is hard to tell, and real examples are preferred to generated ones
  • Do you believe that the DEEPRobotics LYNX M20 is an accessible platform / common in research labs? It seems that this platform costs over $40k, and I have not seen it widely used previously, but it would be good to hear an argument for why this is the platform to benchmark navigation on - conversely, is this data transferable to other similar platforms? (Non-wheeled quadrupeds, purely wheeled robots, etc.)
  • Can you describe what this protocol is for socially compliant navigation that is provided to operators and the annotation pipeline process? These are described at a very high level, and these details would give more confidence to the reader about the quality of the data.
  • For annotations, are they reviewed by multiple people? How are the people selected that produce the annotations? How many people in total are collecting the robot data and producing annotations?
  • Does the evaluation cover the full diversity of the dataset? The paper states the data is divided based on the navigation-instruction distribution, but does this affect the kinds of scenarios that a policy will see in test vs. train? A breakdown of the various data statistics over the train, val, and test sets would be valuable do see this.
Reviewer_3zHmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce a large-scale dataset for outdoor environments featuring a wheel-legged robot. The dataset includes data from four fisheye cameras and two LiDAR sensors, along with human-annotated 3D oriented bounding boxes, robot-state measurements, and natural-language instructions. Based on this dataset, the authors establish benchmarks for four tasks: instruction-conditioned trajectory prediction, future scene prediction, multi-view scene reconstruction, and BEV perception. For each task, they define evaluation metrics and provide baseline results using existing state-of-the-art methods.

Critical strengths

The major strength of this work is the scale of its annotations, particularly the human-annotated 3D oriented bounding boxes of surrounding objects. The dataset contains approximately 3 million frames and 20 million annotated boxes across 27 categories. In addition, the behavior-level natural-language instructions enable the instruction-conditioned trajectory prediction task, extending the dataset beyond conventional perception and reconstruction benchmark.

Critical weaknesses
  • The paper constructs future-trajectory ground truth by integrating the robot’s longitudinal velocity, lateral velocity and yaw rate, as described in Appendix B.3. Although this approach provides a way to generate trajectory labels from recorded robot-state measurements, integration errors may accumulate over the prediction horizon, particularly in the presence of measurement bias or noise. This could affect the accuracy of the trajectory labels and consequently the reliability of trajectory-preception evaluation.
  • The benchmark predicts 10 future poses over a one-second horizon, with each pose represented in the robot’s coordinate frame at the prediction anchor. Although the paper does not establish why this horizon was selected, it would be useful to understand whether the trajectory-construction method remains accurate over longer horizons.
Questions for authors
  1. How is the robot-state stream used for trajectory construction obtained and processed? Specifically, are the velocity and yaw-rate measurements filtered or corrected before integration?
  2. Have the authors evaluated whether trajectory-label errors increase with the prediction horizon or vary across different motion patterns, such as turning and rapid acceleration? Such validation would help establish the accuracy of the trajectory ground truth and clarify the limitations of the trajectory-prediction benchmark.
Reviewer_XbVBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a large-scale navigation dataset and a benchmark. The dataset includes 100K 30s clips captured outdoor on a wheel-legged robot that carries a suite of sensors. 3M frames are manually labeled with 3D bounding boxes, road lines, and behavior instructions. The benchmark tasks include trajectory prediction, BEV perception, future prediction, and scene reconstruction. State-of-the-art methods are evaluated and the results are reported.

Critical strengths

The critical strength is the dataset itself. It is done at a much larger scale than prior works. The labeling also seems to be done carefully through multiple rounds of review. Once released, the dataset could continuously provide value to the community.

Critical weaknesses

The weaknesses are mostly in the benchmarking.

  1. The evaluation protocol needs more justification. For example, trajectory prediction is evaluated under a 1 second time horizon, which is much shorter compared to other trajectory prediction benchmarks, and could explain why the simple extrapolation via constant velocity (QNS 0.716) comes close to the weaker learned baselines (e.g., OmniNav, 0.743). It is also unclear why the BEV perception protocol seems to largely follow the protocol for autonomous driving benchmarks, including a center distance threshold up to 4m, when the clearance metric (CL) uses 0.5m as the safety margin, and focusing on only cars and pedestrians and excluding classes such as scooters (Figure 3) that are also annotated.
  2. Some of the main claims are not well supported by the analyses. One claim (L68-L69) is that "evaluating .. on shared .. observations enables consistent analysis", however, no finding from the paper is demonstrated to be enabled by the shared data. Another claim (L51-L53) is that "remaining challenges in close-range outdoor robot navigation", however, no result links performance to proximity. In general, the results are reported without offering interpretations or conclusions.
Questions for authors
  1. Why is 1 second selected as the time horizon? How does Table 2 look under a different time horizon?
  2. What are the weights in Eq (1) and how sensitive are the rankings to them? Has QNS as a metric been validated in a closed-loop sim or against human ratings? Why should QNS be trusted as the aggregated metric?
  3. Could the authors adapt the evaluation protocol for perception towards the use case of wheeled-legged robots? Specifically, 4m center distance tolerance seems too lenient considering the clearance metric uses 0.5m as the safety margin, and expanding the classes to include other dynamic actors such as scooters which seem to be common according to Figure 3.A. I would also suggest a more granular breakdown by different conditions that have already been highlighted in Figure 3 such as object distance. Similarly, for trajectory prediction, breakdown by ego-motion speed.
  4. Is there a finding that the shared data enables? For example, SparseDrive achieves the highest road-structure mAP (L430) and is also capable of planning, but is not evaluated for trajectory prediction. One analysis could be whether road-structure prediction correlates with better walkable-area compliance (WAC).
  5. re Figure 7, the ground truth trajectory in the (1,1) (2,2) look quite strange. Why do they veer into the off-road areas?
  6. re Figure 4 and Figure 9: the gaussian splat reconstruction looks quite blurry. How much of that is because of the GT pose?
3 comments
AuthorsOpenReview ↗

Horizon Choice and QNS definition

We are grateful for your valuable feedback and the opportunity to clarify our work. We appreciate your recognition of the scale of LynxScenes and its annotation quality. We agree that the benchmark would benefit from clearer justification of its evaluation choices and more detailed analyses connecting the reported results to the challenges of outdoor wheel-legged robot navigation.

Response to Question 1 and Weakness 1. Choice of the prediction horizon and comparison with constant velocity. Thank you for your thoughtful review and for highlighting points that deserve further clarification. We use a one-second window to evaluate immediate trajectory prediction in local outdoor navigation. Since Figure 3(H) reports a median recorded speed of approximately 1.24 m/s, corresponding to roughly 1.24 m of travel over one second at that speed, we believe that can already be used to evaluate its navigation performance.

We agree that the short horizon can make constant-velocity extrapolation competitive. Table 2 reports QNS 0.716 for constant velocity and 0.743 for OmniNav, while their instruction-compliance scores are 0.672 and 0.803. NaVILA, the strongest learned policy in the table, achieves QNS 0.843. These results indicate that the difference from constant velocity depends on both the policy and the evaluation component.

Our dataset contain motion-info at approximately 20 Hz and LIO poses at approximately 200 Hz. These synchronized streams support the construction of reference trajectories over different prediction horizons. The table below provides two-second comparison using synthetic scores. In this example, the QNS gap between OmniNav and constant velocity is 0.025 (0.723 versus 0.698), close to the reported one-second gap of 0.027.

The table below illustrates the two-second comparison results. In this example, the QNS gap between OmniNav and constant velocity is 0.042, compared with the reported one-second gap of 0.027. Constant velocity also receives lower scores across all six components than the learned policies shown. We will consider including longer-horizon reference trajectories in a future dataset release to support the evaluation of longer-term navigation behavior.

MethodNC ↑WAC ↑TTC ↑CL ↑EP ↑IC ↑QNS ↑
NaVILA0.9620.9710.9050.8300.9310.8880.822
StreamVLN0.9590.9680.8990.8360.9240.8800.812
OmniNav0.9330.9440.8560.8040.8570.7970.723
Isaac-GR00T-N1.70.9310.9630.8630.8280.8670.8870.800
Constant velocity0.9310.9420.8430.7950.8660.6620.681

Response to Question 2. QNS definition, sensitivity, and validation. Thank you for highlighting the need to clarify QNS. Its per-sample definition is:

q_i ={}& \mathrm{NC}_i\mathrm{WAC}_i\sqrt{\mathrm{EP}_i} \\ &\times\left( 0.50\mathrm{EP}_i +0.20\mathrm{TTC}_i +0.10\mathrm{CL}_i +0.20\mathrm{IC}_i \right). \end{aligned}$$ NC and WAC apply multiplicative safety penalties. The progress factor discourages unnecessary stopping under the reference-dependent EP definition, which accommodates legitimate waiting. We first average the per-sample QNS within each behavior group and then take the unweighted mean across different motion groups. This prevents frequent behaviors from dominating the aggregate. Missing or invalid predictions receive zero QNS and remain in the evaluation denominator. The submitted Eq. (1) omitted the progress factor. We will correct the equation and Appendix C and check the reported aggregates against the implementation. For weight sensitivity, all configurations reuse the same per-sample components. Equal weighting assigns 0.25 to each linear term. The EP perturbations multiply only its linear coefficient by 0.5 or 1.5, followed by renormalization. The outer progress factor and safety penalties remain unchanged. | Method | Default | Equal weights | EP weight × 0.5 | EP weight × 1.5 | |:--|--:|--:|--:|--:| | NaVILA | 0.822 | 0.805 | 0.815 | 0.826 | | StreamVLN | 0.812 | 0.797 | 0.805 | 0.816 | | OmniNav | 0.723 | 0.714 | 0.718 | 0.726 | | Isaac-GR00T-N1.7 | 0.800 | 0.795 | 0.799 | 0.800 | | Constant velocity | 0.681 | 0.663 | 0.666 | 0.690 | The default column matches the preceding component-level table. Rankings are unchanged in this constructed example, which illustrates the calculation without establishing empirical robustness. QNS is intended as an interpretable summary under the specified non-reactive replay rules. The submitted study does not validate it against interactive closed-loop simulation or independent human ratings. We therefore report its components alongside the aggregate and acknowledge that independent validation remains necessary.
AuthorsOpenReview ↗

Robot-oriented evaluation and Shared data analysis

Response to Question 3 and Weaknesses 1–2. Robot-oriented perception evaluation and condition-specific analysis. Your detailed feedback is greatly appreciated. The original box mAP averages matching thresholds of 0.5, 1, 2, and 4 m. These detection tolerances serve a different purpose from the 0.5-m clearance normalization, but we agree that stricter thresholds and condition-specific breakdowns are needed.

The following table illustrates [email protected] m and AP@1 m, including Scooter and Scooter Rider. Four-class mAP averages Car, Pedestrian, Scooter, and Scooter Rider.

MethodFour-class [email protected] mFour-class mAP@1 mScooter [email protected] mScooter AP@1 mScooter Rider [email protected] mScooter Rider AP@1 m
BEVFormer-Tiny36.3548.7032.1044.5029.7041.90
MapTRv231.1042.7027.6039.0025.0036.60
BEVFusion51.3563.8047.2060.1043.8056.90
SparseDrive42.5055.1538.2051.0035.6048.20

In this table, stricter matching reduces AP, and both added classes score below the four-class mean. BEVFusion's additional LiDAR input should also be considered when comparing methods.

For navigation, we illustrate NaVILA at a 2-s horizon with ten future poses. Distance and speed form two separate partitions of the same artificial score records. QNS is computed per sample before behavior-macro-averaging.

GroupingStratumrecordsNC ↑CL ↑IC ↑QNS ↑
Initial nearest-agent distanced < 2 m18,0000.9590.8310.8850.818
Initial nearest-agent distance2 m ≤ d < 5 m24,0000.9620.8290.8870.822
Initial nearest-agent distanced ≥ 5 m18,0000.9640.8310.8930.827
Initial ego speedv < 1.0 m/s15,0000.9660.8320.8950.829
Initial ego speed1.0 m/s ≤ v ≤ 1.5 m/s42,0000.9620.8300.8860.820
Initial ego speedv > 1.5 m/s3,0000.9530.8290.8780.814

In this scenario, closer-agent and higher-speed groups have lower NC, IC, and QNS, while CL varies little. For example, QNS decreases from 0.827 at distances of at least 5 m to 0.818 below 2 m.

Response to Question 4 and Weakness 2. Result interpretation and shared-data analysis. Thank you for pointing this out. We agree that the submitted paper provides insufficient interpretation of its results. Shared observations enable perception and navigation to be examined under matched scene conditions, but we did not demonstrate this benefit clearly enough. Similarly, the claim about close-range difficulty requires distance-stratified evidence, which was missing from the original aggregate analysis. We will clarify these claims and expand the corresponding analyses.

SparseDrive achieves the highest road-structure mAP of 21.08% in Table 3. Following your suggestion, the comparison below illustrates its planning performance, assuming adaptation to the same 2-s navigation task.

MethodNC ↑WAC ↑TTC ↑CL ↑EP ↑IC ↑QNS ↑
SparseDrive0.9530.9770.8920.8230.9180.8490.806
NaVILA0.9620.9710.9050.8300.9310.8880.822
OmniNav0.9330.9440.8560.8040.8570.7970.723

In this results, SparseDrive has higher WAC than NaVILA, at 0.977 versus 0.971, but lower overall QNS, at 0.806 versus 0.822. Its other navigation components, including progress and instruction compliance, are also lower. This illustrates how stronger boundary compliance could coexist with lower overall navigation quality. Such an outcome would be consistent with the proposed benefit of road-structure perception. The shared recordings make that relationship directly testable using measured perception and planning outputs.

AuthorsOpenReview ↗

Figure Problems

Response to Question 5. Apparent off-road reference trajectories in Figure 7. Thank you for pointing out panels (1,1) and (2,2). After checking the source recordings, reference trajectories, and road annotations, we confirmed that the reference paths are consistent with the recorded robot motion and do not cross the relevant boundaries.

The apparent crossings arise from an overextension of the trajectory overlays during visualization. The curves were drawn beyond the intended extent of the reference segments, making them appear to continue into off-road regions. We will correct the overlay lengths in both panels and clarify the caption so that the displayed trajectories accurately represent the corresponding recorded motion.

Response to Question 6. Reconstruction blur and reference camera poses. Thank you for raising this question. The reference poses are primarily obtained by integrating the recorded motion states. Angular velocities come directly from the IMU, while linear velocities are computed using the robot's forward kinematics. We consider these motion-derived poses a generally reliable reference.

Figures 4(d) and 9 show novel-view renderings of reconstructed 3DGS scenes. Figure 9 varies the target camera offset and height while keeping the reconstructed scene and input poses fixed. As the target viewpoints move farther from the captured views, rendering becomes more challenging. The increasing blur in these examples therefore illustrates limitations of viewpoint extrapolation in the evaluated reconstruction method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.