- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Odyssey proposes a long-horizon, sensor-level closed-loop benchmark for end-to-end autonomous driving. It curates 100 scenarios from 100-second nuPlan driving logs and provides explicit standard-definition (SD) map routes instead of ambiguous high-level directional commands. The simulator reconstructs dynamic real-world scenes using 3D Gaussian Splatting, improves rendered observations with diffusion-based refinement, and aligns traffic events with ego-vehicle progress using section-based triggering. The evaluation introduces SD Route Compliance (SDC), Pre-Lane Change Accuracy/Score (PLCA/PLCS), and RouteDS to assess route following, lane preparation, and overall driving performance. Experiments on several vision-based and VLA-based planners show that explicit route guidance generally improves route compliance, planner rankings can change as the evaluation horizon increases, and improvements in open-loop metrics do not necessarily transfer to closed-loop driving.
Critical strengths
Practically relevant research problem. The paper addresses important limitations of existing closed-loop driving benchmarks, particularly short evaluation horizons and ambiguous navigation commands. Comprehensive evaluation framework. Odyssey integrates real-world scene reconstruction, SD-map-based navigation, diffusion-based visual refinement, and traffic simulation into a unified framework for continuous long-horizon evaluation. Useful empirical observations. The experiments indicate that planner rankings can change over longer rollouts and that improvements in open-loop performance do not necessarily translate into better closed-loop driving.
Critical weaknesses
Limited methodological novelty and insufficient differentiation from existing benchmarks. The framework primarily combines established techniques, including OmniRe, 3DGS, diffusion-based refinement, and rule-based driving metrics. Prior work such as HUGSIM and DriveArena has explored closed-loop evaluation with realistic sensor feedback, while Bench2Drive has used explicit route or target-point guidance. Odyssey is mainly distinguished by longer real-log rollouts and SD-map routes. However, the paper does not sufficiently establish the incremental evaluation value of these extensions against the closest existing frameworks. Insufficient validation of the necessity of long-horizon evaluation. Figure 8 shows declining RouteDS and changes in planner rankings as the evaluation horizon increases. However, lower scores are partly expected under cumulative multiplicative penalties, and the analysis does not adequately separate longer exposure to infractions from genuinely long-term planning failures. Statistical uncertainty and more targeted analyses of decisions whose consequences emerge only after substantial delays are also missing. Thus, the distinct value of 100-second evaluation remains insufficiently established. Limited realism of counterfactual traffic interactions. Surrounding agents primarily follow recorded trajectories or IDM-based control. Section-based traffic triggering improves the temporal alignment of interactions with ego progress, but does not by itself model how other agents should react to substantial off-log deviations, particularly in complex lateral interactions. The reported traffic-density analysis is insufficient to validate behavioral realism or the reliability of collision-related driving scores under such conditions. Insufficient validation of metric design and experimental controls. SDC is binary, PLCA/PLCS use a fixed 10-meter threshold, and RouteDS relies on manually specified multiplicative penalties, without adequate sensitivity analysis. Moreover, SD-route guidance differs from conventional directional commands in both information content and model integration, potentially confounding the claimed benefit of the route representation. More carefully matched guidance baselines and metric-robustness analyses would strengthen the conclusions.
Questions for authors
Benchmark differentiation and horizon-specific benefits. What evidence demonstrates that Odyssey provides evaluation capabilities beyond extending existing closed-loop simulators? Could the authors conduct controlled comparisons at different horizons (e.g., 20, 50, and 100 seconds), with uncertainty estimates and targeted delayed-decision scenarios, and compare SD-route guidance against navigation representations with comparable information content? Reliability under off-log deviations. When the ego vehicle substantially deviates from the recorded trajectory, how do the authors verify the realism of other agents' responses and the semantic consistency of diffusion-refined observations? Could they quantify the effect of these simulation errors on collision metrics, RouteDS, and planner rankings?
6 comments
Q2. Weakness: Insufficient validation of the necessity of long-horizon evaluation. Figure 8 shows declining RouteDS and changes in planner rankings as the evaluation horizon increases. However, lower scores are partly expected under cumulative multiplicative penalties, and the analysis does not adequately separate longer exposure to infractions from genuinely long-term planning failures. Statistical uncertainty and more targeted analyses of decisions whose consequences emerge only after substantial delays are also missing. Thus, the distinct value of 100-second evaluation remains insufficiently established.
Question: Could the authors conduct controlled comparisons at different horizons (e.g., 20, 50, and 100 seconds), with uncertainty estimates and targeted delayed-decision scenarios.
A2. To assess the statistical reliability of the observed ranking changes, we additionally report 95% confidence intervals (CIs) for RouteDS across different evaluation horizons (20, 50, and 100 seconds). The results indicate that the observed ranking reversals are not merely artifacts of evaluation variability, suggesting that short-horizon evaluations may lead to different conclusions about the relative performance of E2E driving models compared with long-horizon evaluations.
| RouteDS, mean ± 95% CI | 20 s horizon | 50 s horizon | 100 s horizon |
|---|---|---|---|
| DrivoR | 67.7 ± 5.8 | 52.4 ± 5.5 | 35.9 ± 5.2 |
| SafeDrive | 64.6 ± 6.1 | 52.8 ± 5.2 | 39.2 ± 5.7 |
| SafeDrive (IL) | 65.1 ± 5.5 | 52.6 ± 4.9 | 43.1 ± 4.9 |
| DiffusionDrive | 53.9 ± 5.3 | 38.2 ± 4.5 | 29.6 ± 4.5 |
| ReCogDrive | 52.8 ± 5.7 | 34.7 ± 5.1 | 22.3 ± 4.5 |
| ReCogDrive (IL) | 49.7 ± 4.5 | 38.9 ± 4.1 | 31.2 ± 4.1 |
| LTF | 51.9 ± 5.0 | 30.8 ± 4.2 | 19.3 ± 3.7 |
More importantly, we argue that the value of long-horizon evaluation extends beyond the accumulation of infractions over longer driving durations. Long-horizon simulations expose E2E models to sequences of diverse road layouts, curvatures, and topologies, where earlier navigation decisions can influence subsequent driving behaviors and eventually lead to downstream failures. Such long-term navigation capabilities can be assessed using the Pre-Lane Change Score (PLCS), which measures a model's ability to anticipate and prepare for upcoming navigation maneuvers. By design, PLCS requires a sufficiently long evaluation horizon to capture anticipatory navigation behaviors and their downstream consequences, which may not be observable in short-horizon simulations.
To further investigate this aspect, we analyze failure cases attributable to pre-lane-change errors across different evaluation horizons. As shown in the table below, the number of such failures increases with the evaluation horizon, indicating that longer simulations reveal navigation-related failures that may remain undetected in shorter evaluations. Furthermore, the PLCS scores vary across E2E models, highlighting differences in their ability to anticipate and prepare for upcoming road configurations.
| PLCS failures ↓ | 20 s horizon | 50 s horizon | 100 s horizon |
|---|---|---|---|
| DrivoR | 20 | 45 | 58 |
| SafeDrive | 15 | 31 | 50 |
| SafeDrive (IL) | 16 | 32 | 42 |
| DiffusionDrive | 13 | 27 | 32 |
| ReCogDrive | 30 | 57 | 75 |
| ReCogDrive (IL) | 21 | 35 | 43 |
| LTF | 33 | 62 | 72 |
| Mean | 21.1 | 41.3 | 53.1 |
Q3. Weakness: Limited realism of counterfactual traffic interactions. Surrounding agents primarily follow recorded trajectories or IDM-based control. Section-based traffic triggering improves the temporal alignment of interactions with ego progress, but does not by itself model how other agents should react to substantial off-log deviations, particularly in complex lateral interactions. The reported traffic-density analysis is insufficient to validate behavioral realism or the reliability of collision-related driving scores under such conditions.
Question: Reliability under off-log deviations. When the ego vehicle substantially deviates from the recorded trajectory, how do the authors verify the realism of other agents' responses.
A3. Odyssey primarily focuses on evaluating the navigation reasoning capabilities of E2E driving models under interactive traffic conditions, rather than faithfully reproducing complex multi-agent driving behaviors. To this end, we adopt relatively simple traffic models, including log-replay and IDM-based agents, which are also commonly used in closed-loop simulators such as nuPlan, Waymax, and HUGSIM. While these models enable E2E planners to encounter and respond to surrounding traffic, we acknowledge that they do not fully capture realistic agent reactions to substantial off-log deviations, particularly in complex lateral interactions and cooperative negotiations.
To mitigate the limitations of log-replay in long-horizon evaluation, Odyssey employs section-based traffic triggering, which ensures that planners encounter the intended traffic interactions regardless of their driving speed or arrival time. However, this mechanism is designed to preserve exposure to relevant traffic scenarios rather than reproduce fully realistic counterfactual agent responses. Consequently, collision-related scores under substantial off-log deviations should be interpreted with this limitation in mind.
Nevertheless, we believe that the current traffic modeling provides a practical basis for evaluating navigation capabilities under interactive traffic conditions, which is the primary objective of Odyssey. Incorporating more sophisticated reactive multi-agent behavior models would further improve simulation fidelity and the reliability of collision-related evaluations. We have added this limitation and the corresponding future research direction to the Limitations and Future Work sections.
We sincerely thank the reviewer for the thoughtful critique and constructive questions. The suggested analyses provide valuable directions for more clearly demonstrating the benefits of long-horizon evaluation and strengthening Odyssey as a benchmark. We have incorporated these insights into the revised manuscript.
A1. We argue that existing closed-loop benchmarks are insufficient for comprehensively evaluating the navigation capabilities of E2E driving models. While many existing benchmarks primarily focus on driving safety and interactions with surrounding agents within relatively short simulation windows (typically around 20 seconds), Odyssey enables long-horizon evaluation across diverse road geometries, curvatures, and topologies. This allows us to evaluate sequences of navigation decisions and their downstream consequences, which may not be observable in short-horizon evaluations. To explicitly assess these navigation challenges, Odyssey introduces navigation-oriented metrics, including SD Route Compliance and Pre-Lane Change Score, which capture aspects of navigation performance not explicitly measured by existing E2E driving benchmarks.
Furthermore, Odyssey evaluates navigation capabilities using SD-map routes, which are widely available in production vehicles but have received limited attention in existing closed-loop benchmarks. Although Bench2Drive also provides explicit route or target-point guidance, it relies on lane-level route information derived from HD maps. In contrast, Odyssey requires E2E driving models to navigate using coarse SD-map routes without explicit lane-level guidance, making navigation decisions more challenging, particularly at intersections and road splits. This distinction enables Odyssey to evaluate route interpretation and navigation reasoning under less detailed route guidance, addressing an important gap in existing benchmarks.
The distinct evaluation capabilities of Odyssey are further supported by our experimental results. As shown in the table below, the best-performing model on HUGSIM exhibits substantially lower performance on Odyssey, suggesting that strong performance on existing short-horizon benchmarks does not necessarily translate into strong long-horizon navigation performance. Together, these findings demonstrate that Odyssey provides complementary evaluation capabilities beyond simply extending the simulation duration of existing closed-loop benchmarks.