acceptodds
Under review as a conference paper at ICLR 2027

Odyssey: A Closed-Loop Benchmark for Long-Horizon Real-World Driving with Explicit Navigation Routes

29 views

Abstract

Closed-loop evaluation of end-to-end driving requires continuous rollouts that reveal how earlier decisions affect subsequent driving. However, existing benchmarks evaluate only short segments and fail to capture later consequences. Ambiguous directional commands also obscure the intended navigation objective. We introduce Odyssey, a closed-loop benchmark for long-horizon driving comprising 100 scenarios, each reconstructed from a 100-second nuPlan driving log to preserve the context of navigation maneuvers and traffic interactions. To provide a consistent navigation objective, Odyssey replaces directional commands with explicit standard-definition (SD) map routes that specify which roads to follow, while sensor-based planning determines local driving actions. Throughout these rollouts, diffusion-based refinement of 3DGS-rendered images reduces rendering artifacts along the ego trajectory. To assess how effectively planners follow these routes and prepare for upcoming maneuvers, we introduce SD Route Compliance and Pre-Lane Change Score. These assessments are complemented by RouteDS, which extends the Driving Score with penalties for SD-route deviations and failed lane preparation. We adapt state-of-the-art planners, including vision-language-action (VLA) models, and evaluate their navigation performance using these metrics. Odyssey raises questions about route representation and integration in E2E driving stacks. Benchmark code and adapted baselines will be released publicly.

Primary area:
datasets and benchmarks
Keywords:
closed-loop evaluation, end-to-end driving, long-horizon planning, autonomous driving benchmark, route-conditioned navigation, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UFngOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Odyssey proposes a long-horizon, sensor-level closed-loop benchmark for end-to-end autonomous driving. It curates 100 scenarios from 100-second nuPlan driving logs and provides explicit standard-definition (SD) map routes instead of ambiguous high-level directional commands. The simulator reconstructs dynamic real-world scenes using 3D Gaussian Splatting, improves rendered observations with diffusion-based refinement, and aligns traffic events with ego-vehicle progress using section-based triggering. The evaluation introduces SD Route Compliance (SDC), Pre-Lane Change Accuracy/Score (PLCA/PLCS), and RouteDS to assess route following, lane preparation, and overall driving performance. Experiments on several vision-based and VLA-based planners show that explicit route guidance generally improves route compliance, planner rankings can change as the evaluation horizon increases, and improvements in open-loop metrics do not necessarily transfer to closed-loop driving.

Critical strengths

Practically relevant research problem. The paper addresses important limitations of existing closed-loop driving benchmarks, particularly short evaluation horizons and ambiguous navigation commands. Comprehensive evaluation framework. Odyssey integrates real-world scene reconstruction, SD-map-based navigation, diffusion-based visual refinement, and traffic simulation into a unified framework for continuous long-horizon evaluation. Useful empirical observations. The experiments indicate that planner rankings can change over longer rollouts and that improvements in open-loop performance do not necessarily translate into better closed-loop driving.

Critical weaknesses

Limited methodological novelty and insufficient differentiation from existing benchmarks. The framework primarily combines established techniques, including OmniRe, 3DGS, diffusion-based refinement, and rule-based driving metrics. Prior work such as HUGSIM and DriveArena has explored closed-loop evaluation with realistic sensor feedback, while Bench2Drive has used explicit route or target-point guidance. Odyssey is mainly distinguished by longer real-log rollouts and SD-map routes. However, the paper does not sufficiently establish the incremental evaluation value of these extensions against the closest existing frameworks. Insufficient validation of the necessity of long-horizon evaluation. Figure 8 shows declining RouteDS and changes in planner rankings as the evaluation horizon increases. However, lower scores are partly expected under cumulative multiplicative penalties, and the analysis does not adequately separate longer exposure to infractions from genuinely long-term planning failures. Statistical uncertainty and more targeted analyses of decisions whose consequences emerge only after substantial delays are also missing. Thus, the distinct value of 100-second evaluation remains insufficiently established. Limited realism of counterfactual traffic interactions. Surrounding agents primarily follow recorded trajectories or IDM-based control. Section-based traffic triggering improves the temporal alignment of interactions with ego progress, but does not by itself model how other agents should react to substantial off-log deviations, particularly in complex lateral interactions. The reported traffic-density analysis is insufficient to validate behavioral realism or the reliability of collision-related driving scores under such conditions. Insufficient validation of metric design and experimental controls. SDC is binary, PLCA/PLCS use a fixed 10-meter threshold, and RouteDS relies on manually specified multiplicative penalties, without adequate sensitivity analysis. Moreover, SD-route guidance differs from conventional directional commands in both information content and model integration, potentially confounding the claimed benefit of the route representation. More carefully matched guidance baselines and metric-robustness analyses would strengthen the conclusions.

Questions for authors

Benchmark differentiation and horizon-specific benefits. What evidence demonstrates that Odyssey provides evaluation capabilities beyond extending existing closed-loop simulators? Could the authors conduct controlled comparisons at different horizons (e.g., 20, 50, and 100 seconds), with uncertainty estimates and targeted delayed-decision scenarios, and compare SD-route guidance against navigation representations with comparable information content? Reliability under off-log deviations. When the ego vehicle substantially deviates from the recorded trajectory, how do the authors verify the realism of other agents' responses and the semantic consistency of diffusion-refined observations? Could they quantify the effect of these simulation errors on collision metrics, RouteDS, and planner rankings?

6 comments

We sincerely thank the reviewer for the thoughtful critique and constructive questions. The suggested analyses provide valuable directions for more clearly demonstrating the benefits of long-horizon evaluation and strengthening Odyssey as a benchmark. We have incorporated these insights into the revised manuscript.


Q1. Weakness: Limited methodological novelty and insufficient differentiation from existing benchmarks. The framework primarily combines established techniques, including OmniRe, 3DGS, diffusion-based refinement, and rule-based driving metrics. Prior work such as HUGSIM and DriveArena has explored closed-loop evaluation with realistic sensor feedback, while Bench2Drive has used explicit route or target-point guidance. Odyssey is mainly distinguished by longer real-log rollouts and SD-map routes. However, the paper does not sufficiently establish the incremental evaluation value of these extensions against the closest existing frameworks.

Question: Benchmark differentiation and horizon-specific benefits. What evidence demonstrates that Odyssey provides evaluation capabilities beyond extending existing closed-loop simulators?

A1. We argue that existing closed-loop benchmarks are insufficient for comprehensively evaluating the navigation capabilities of E2E driving models. While many existing benchmarks primarily focus on driving safety and interactions with surrounding agents within relatively short simulation windows (typically around 20 seconds), Odyssey enables long-horizon evaluation across diverse road geometries, curvatures, and topologies. This allows us to evaluate sequences of navigation decisions and their downstream consequences, which may not be observable in short-horizon evaluations. To explicitly assess these navigation challenges, Odyssey introduces navigation-oriented metrics, including SD Route Compliance and Pre-Lane Change Score, which capture aspects of navigation performance not explicitly measured by existing E2E driving benchmarks.

Furthermore, Odyssey evaluates navigation capabilities using SD-map routes, which are widely available in production vehicles but have received limited attention in existing closed-loop benchmarks. Although Bench2Drive also provides explicit route or target-point guidance, it relies on lane-level route information derived from HD maps. In contrast, Odyssey requires E2E driving models to navigate using coarse SD-map routes without explicit lane-level guidance, making navigation decisions more challenging, particularly at intersections and road splits. This distinction enables Odyssey to evaluate route interpretation and navigation reasoning under less detailed route guidance, addressing an important gap in existing benchmarks.

The distinct evaluation capabilities of Odyssey are further supported by our experimental results. As shown in the table below, the best-performing model on HUGSIM exhibits substantially lower performance on Odyssey, suggesting that strong performance on existing short-horizon benchmarks does not necessarily translate into strong long-horizon navigation performance. Together, these findings demonstrate that Odyssey provides complementary evaluation capabilities beyond simply extending the simulation duration of existing closed-loop benchmarks.

PlannerHUGSIM (HD-Score ↑)Odyssey (RouteDS ↑)
DrivoR38.336.5
SafeDrive27.844.4

Q2. Weakness: Insufficient validation of the necessity of long-horizon evaluation. Figure 8 shows declining RouteDS and changes in planner rankings as the evaluation horizon increases. However, lower scores are partly expected under cumulative multiplicative penalties, and the analysis does not adequately separate longer exposure to infractions from genuinely long-term planning failures. Statistical uncertainty and more targeted analyses of decisions whose consequences emerge only after substantial delays are also missing. Thus, the distinct value of 100-second evaluation remains insufficiently established.

Question: Could the authors conduct controlled comparisons at different horizons (e.g., 20, 50, and 100 seconds), with uncertainty estimates and targeted delayed-decision scenarios.

A2. To assess the statistical reliability of the observed ranking changes, we additionally report 95% confidence intervals (CIs) for RouteDS across different evaluation horizons (20, 50, and 100 seconds). The results indicate that the observed ranking reversals are not merely artifacts of evaluation variability, suggesting that short-horizon evaluations may lead to different conclusions about the relative performance of E2E driving models compared with long-horizon evaluations.

RouteDS, mean ± 95% CI20 s horizon50 s horizon100 s horizon
DrivoR67.7 ± 5.852.4 ± 5.535.9 ± 5.2
SafeDrive64.6 ± 6.152.8 ± 5.239.2 ± 5.7
SafeDrive (IL)65.1 ± 5.552.6 ± 4.943.1 ± 4.9
DiffusionDrive53.9 ± 5.338.2 ± 4.529.6 ± 4.5
ReCogDrive52.8 ± 5.734.7 ± 5.122.3 ± 4.5
ReCogDrive (IL)49.7 ± 4.538.9 ± 4.131.2 ± 4.1
LTF51.9 ± 5.030.8 ± 4.219.3 ± 3.7

More importantly, we argue that the value of long-horizon evaluation extends beyond the accumulation of infractions over longer driving durations. Long-horizon simulations expose E2E models to sequences of diverse road layouts, curvatures, and topologies, where earlier navigation decisions can influence subsequent driving behaviors and eventually lead to downstream failures. Such long-term navigation capabilities can be assessed using the Pre-Lane Change Score (PLCS), which measures a model's ability to anticipate and prepare for upcoming navigation maneuvers. By design, PLCS requires a sufficiently long evaluation horizon to capture anticipatory navigation behaviors and their downstream consequences, which may not be observable in short-horizon simulations.

To further investigate this aspect, we analyze failure cases attributable to pre-lane-change errors across different evaluation horizons. As shown in the table below, the number of such failures increases with the evaluation horizon, indicating that longer simulations reveal navigation-related failures that may remain undetected in shorter evaluations. Furthermore, the PLCS scores vary across E2E models, highlighting differences in their ability to anticipate and prepare for upcoming road configurations.

PLCS failures ↓20 s horizon50 s horizon100 s horizon
DrivoR204558
SafeDrive153150
SafeDrive (IL)163242
DiffusionDrive132732
ReCogDrive305775
ReCogDrive (IL)213543
LTF336272
Mean21.141.353.1

Q3. Weakness: Limited realism of counterfactual traffic interactions. Surrounding agents primarily follow recorded trajectories or IDM-based control. Section-based traffic triggering improves the temporal alignment of interactions with ego progress, but does not by itself model how other agents should react to substantial off-log deviations, particularly in complex lateral interactions. The reported traffic-density analysis is insufficient to validate behavioral realism or the reliability of collision-related driving scores under such conditions.

Question: Reliability under off-log deviations. When the ego vehicle substantially deviates from the recorded trajectory, how do the authors verify the realism of other agents' responses.

A3. Odyssey primarily focuses on evaluating the navigation reasoning capabilities of E2E driving models under interactive traffic conditions, rather than faithfully reproducing complex multi-agent driving behaviors. To this end, we adopt relatively simple traffic models, including log-replay and IDM-based agents, which are also commonly used in closed-loop simulators such as nuPlan, Waymax, and HUGSIM. While these models enable E2E planners to encounter and respond to surrounding traffic, we acknowledge that they do not fully capture realistic agent reactions to substantial off-log deviations, particularly in complex lateral interactions and cooperative negotiations.

To mitigate the limitations of log-replay in long-horizon evaluation, Odyssey employs section-based traffic triggering, which ensures that planners encounter the intended traffic interactions regardless of their driving speed or arrival time. However, this mechanism is designed to preserve exposure to relevant traffic scenarios rather than reproduce fully realistic counterfactual agent responses. Consequently, collision-related scores under substantial off-log deviations should be interpreted with this limitation in mind.

Nevertheless, we believe that the current traffic modeling provides a practical basis for evaluating navigation capabilities under interactive traffic conditions, which is the primary objective of Odyssey. Incorporating more sophisticated reactive multi-agent behavior models would further improve simulation fidelity and the reliability of collision-related evaluations. We have added this limitation and the corresponding future research direction to the Limitations and Future Work sections.

Reviewer_4itcOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

To address the limitations of short evaluation horizons, ambiguous directional commands, and rendering artifacts in existing closed-loop benchmarks, this paper introduces Odyssey, a photorealistic evaluation framework comprising one hundred 100-second driving sequences reconstructed from real-world logs. The system replaces vague turn commands with explicit standard-definition map routes and enhances 3D Gaussian Splatting with single-step diffusion refinement to sustain visual fidelity during novel-view closed-loop rollouts. Evaluating contemporary end-to-end and vision-language-action planners on newly designed route-compliance and pre-lane change metrics reveals that open-loop gains do not reliably translate to closed-loop driving, and that coarse route guidance alone fails to guarantee proactive lane positioning or traffic safety.

Critical strengths
  1. The benchmark pioneers 100-second closed-loop rollouts with explicit standard-definition map routes, successfully resolving the maneuver ambiguities inherent to short-horizon, command-based evaluations.
  2. The simulation pipeline integrates 3D Gaussian Splatting with fine-tuned single-step diffusion refinement, effectively preserving photorealistic visual quality under significant viewpoint deviations during closed-loop rollouts.
  3. The proposed evaluation protocol introduces well-designed metrics such as RouteDS and Pre-Lane Change Score, uncovering critical disconnects between open-loop scores and long-horizon closed-loop performance across modern end-to-end and vision-language-action planners.
Critical weaknesses
  1. The scenario suite is limited to only 100 scenes with scarce representation of specific challenges, including just 13 traffic light encounters, which restricts statistical diversity and tail-case coverage.
  2. Applying per-frame single-step diffusion refinement without temporal conditioning introduces potential inter-frame flickering that can confuse multi-frame temporal reasoning in end-to-end planners.
  3. Non-ego traffic depends predominantly on pre-recorded playback with basic IDM longitudinal control, failing to model realistic interactive multi-agent behaviors and cooperative negotiations.
  4. Section-based traffic triggering artificially offsets background agents and signal states according to ego arrival time, which compromises global physical coherence and ambient traffic plausibility.
  5. The multiplicative formulation of RouteDS exhibits high sensitivity to individual penalties, allowing a single minor infraction to severely degrade the overall performance score.
  6. Evaluating route progress and compliance via post-hoc HMM map-matching against coarse OpenStreetMap polylines is susceptible to misclassification errors near complex intersections and closely parallel lanes.
Questions for authors

Please refer to the weaknesses.

3 comments

We appreciate the reviewer's recognition of Odyssey's strengths and detailed comments on its limitations. The feedback raises important concerns about the reliability of our simulation and evaluation, which we address through additional experiments and clarifications below. We have incorporated these findings into the revised paper.


Q1. The scenario suite is limited to only 100 scenes with scarce representation of specific challenges, including just 13 traffic light encounters, which restricts statistical diversity and tail-case coverage.

A1. We would like to clarify the scale and composition of Odyssey. Although Odyssey comprises 100 scenarios, it covers 2.8 hours of distinct real-world driving data curated from the 10.4-hour OpenScene test set. This exceeds the total duration of source logs used in nuPlan Test14-random (1.2 hours), HUGSIM (0.4 hours), and NeuroNCAP (0.08 hours). More importantly, our scenario curation prioritizes navigation-relevant events, such as turns and lane changes, rather than uniformly sampling driving sequences. As shown in the table below, Odyssey retains 42% of all turns and lane changes in the OpenScene test set while using only 27% of its total driving duration. Consequently, Odyssey includes 647 navigation-relevant events, averaging one every 15.5 seconds. Unlike existing benchmarks that primarily evaluate isolated maneuvers within short simulation windows, Odyssey evaluates sequences of consecutive navigation maneuvers within a single 100-second closed-loop rollout. This design enables the evaluation of navigation decisions and their downstream consequences across multiple maneuvers.

We acknowledge that the number of traffic-light encounters (13) is relatively limited, reflecting a limitation in the coverage of traffic-light-related scenarios. Nevertheless, existing photorealistic closed-loop benchmarks, including HUGSIM, NeuroNCAP, and DriveArena, do not explicitly support dynamic traffic-light state changes. In contrast, Odyssey incorporates time-varying traffic signals, enabling E2E models to be evaluated under changing traffic-light states.

Navigation-relevant eventFull OpenScene test set (10.4 h)Odyssey (2.8 h)
General turning15375
Right turn229101
Left turn280123
Lane change351124
Stop moving399109
Start moving426115
Total1,838647

Q2. Applying per-frame single-step diffusion refinement without temporal conditioning introduces potential inter-frame flickering that can confuse multi-frame temporal reasoning in end-to-end planners.

A2. Since refinement is applied independently to each frame, it may introduce temporal inconsistencies in the appearance or geometry of scene elements. To assess this effect, we measure the class-wise IoU between Mask2Map predictions from consecutive frames after compensating for ego motion. As shown in the table below, refinement achieves higher temporal consistency across all evaluated map categories than 3DGS-only rendering. These results suggest that refinement improves semantic perception without compromising the temporal consistency of map elements.

ObservationSplitLane centerline IoU ↑Ped. crossing IoU ↑Road boundary IoU ↑mIoU ↑
GT ImageOL77.062.767.669.1
3DGS OnlyOL76.062.266.568.2
+ RefinementOL76.962.967.669.1
3DGS OnlyCL71.955.864.464.0
+ RefinementCL73.356.966.165.4

Our experiments also suggest that such temporal artifacts do not substantially affect the temporal stability of E2E planning. To investigate this, we measured the average L2 distance between trajectories predicted at consecutive time steps by SafeDrive, a multi-frame E2E driving model. As shown in the table below, SafeDrive exhibits comparable temporal stability in its trajectory predictions with and without diffusion-based refinement. These results suggest that potential flickering artifacts introduced by per-frame refinement have a limited impact on the temporal consistency of the evaluated E2E planner. Nevertheless, we recognize that temporal conditioning could further improve visual consistency, and we plan to investigate its integration into diffusion-based refinement in future versions of Odyssey.

ObservationAverage L2 distance of output trajectories between consecutive steps (m) ↓
w/o refinement0.362
w/ refinement0.320

Q3. Non-ego traffic depends predominantly on pre-recorded playback with basic IDM longitudinal control, failing to model realistic interactive multi-agent behaviors and cooperative negotiations.

A3. Odyssey primarily focuses on evaluating the navigation reasoning capabilities of E2E driving models under interactive traffic conditions, rather than faithfully reproducing complex multi-agent driving behaviors. To this end, we adopt relatively simple traffic models, including log-replay and IDM-based agents, which are also commonly used in closed-loop driving simulators such as nuPlan, Waymax, and HUGSIM. While these models have limitations in capturing complex interactions and cooperative behaviors among agents, they allow E2E planners to be evaluated in traffic scenarios involving surrounding vehicles. Furthermore, our section-based traffic triggering ensures that planners encounter the intended traffic interactions regardless of their driving speed or arrival time, which is particularly important for long-horizon navigation evaluation. We believe this design provides a practical balance between simulation fidelity and our primary objective of evaluating navigation reasoning under traffic interactions.

Nevertheless, we acknowledge that our current traffic modeling does not fully capture realistic multi-agent interactions, particularly complex negotiations and reactive behaviors. Incorporating more sophisticated driver behavior models would further improve simulation fidelity and enable more comprehensive evaluation of E2E driving models. We have added this limitation and the corresponding future research direction to the Limitations and Future Work sections.


Q4. Section-based traffic triggering artificially offsets background agents and signal states according to ego arrival time, which compromises global physical coherence and ambient traffic plausibility.

A4. Section-based traffic triggering was introduced to address a limitation of 3DGS-based simulation, where interactions with surrounding agents may be missed when the planner's rollout timing differs from that of the original recording. Although this approach does not fully reproduce realistic traffic behavior and may compromise global physical consistency, it ensures that E2E planners encounter the intended traffic interactions regardless of their driving speed or arrival time. In this respect, section-based triggering improves the realism of traffic encounters compared with simulations without this mechanism, as supported by the improved WOSAC realism scores reported in the table below. While this improvement does not imply full physical consistency of the simulated traffic, it demonstrates the benefit of section-based triggering in preserving relevant traffic interactions during long-horizon rollouts. We therefore believe that this design adequately serves the primary objective of Odyssey: evaluating the navigation capabilities of E2E driving models under interactive traffic conditions.

WOSAC Realism ↑Logged oraclew/o triggerw/ trigger
Kinematic0.650.490.58
Interactive0.740.630.68
All0.710.580.65

Q5. The multiplicative formulation of RouteDS exhibits high sensitivity to individual penalties, allowing a single minor infraction to severely degrade the overall performance score.

A5. RouteDS employs severity-dependent penalty factors, ensuring that individual infractions affect the overall score in proportion to their severity. For example, partial lane preparation incurs a relatively small penalty factor of 0.9, whereas more severe infractions, such as collisions, incur substantially larger penalties with factors of 0.5–0.6 (Appendix C). Therefore, a single minor infraction does not substantially degrade the overall RouteDS. While multiplicative penalties can accumulate over long-horizon rollouts, this formulation intentionally reflects the cumulative impact of driving infractions, with individual penalty factors determined by their severity. Moreover, this severity-weighted multiplicative formulation is consistent with the Driving Score used in widely adopted closed-loop driving benchmarks, including CARLA Leaderboard 2.0 and Bench2Drive.

Q6. Evaluating route progress and compliance via post-hoc HMM map-matching against coarse OpenStreetMap polylines is susceptible to misclassification errors near complex intersections and closely parallel lanes.

A6. We carefully designed our HMM-based map-matching procedure to minimize road misclassification errors. After each rollout, we incrementally construct the ego trajectory and perform HMM-based map matching to identify the road segments traversed by the ego vehicle. To improve matching reliability, we employed the following strategies:

  • (1) Intersection handling: Identifying the correct outgoing road can be ambiguous while the ego vehicle is traversing an intersection. To avoid premature or incorrect road assignments, we defer map matching until the vehicle exits the intersection, when its outgoing road can be identified more reliably.
  • (2) Parallel road disambiguation: Closely spaced parallel roads, such as opposing carriageways of a divided road, can introduce ambiguity in road assignment. To address this issue, we incorporate both the ego vehicle's position and heading into the matching process, allowing us to distinguish between roads with different travel directions.

Furthermore, we manually inspected the map-matching results across all evaluation rollouts reported in the paper to identify potential road assignment errors.

Reviewer_AbiCOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce Odyssey, a closed-loop driving benchmark built from 100 nuPlan scenarios, each covering a 100-second recording. It combines explicit SD-map navigation routes with 3D Gaussian scene reconstruction and diffusion refinement to provide camera observations during simulation. The benchmark evaluates route following, lane preparation, and driving safety. Experiments with several planners show that rankings change over longer rollouts and that improvements on NAVSIM do not necessarily carry over to closed-loop driving.

Critical strengths
  • The benchmark is timely and addresses an important direction for autonomous driving. As gains on NAVSIM become harder to interpret as improvements in actual driving, longer closed-loop evaluation with explicit navigation routes is a valuable next step. The paper supports this motivation with concrete findings: planner rankings change with rollout length, and better NAVSIM scores do not necessarily translate into better closed-loop performance.

  • The full-stack implementation is substantial. The work covers scene reconstruction, diffusion refinement, traffic-light modeling, traffic activation, and route-conditioned adaptations of several planners. The appendix provides detailed implementation choices, and the reconstruction ablations help explain the contribution of individual components.

  • The experiments provide useful findings beyond a leaderboard. The comparisons of route guidance, evaluation horizon, safety scoring, and GRPO reveal where existing approaches fall short.

Critical weaknesses

This is a solid work. I don't have any major concerns on critical weaknesses. One area the authors may consider to improve is as follows:

The effect of diffusion refinement on evaluation reliability could be validated further. The perception and driving results, together with the supplementary visualizations of lane markings, vehicles, and pedestrians, provide useful evidence. Quantitative checks of whether refinement preserves these elements across consecutive frames would further strengthen confidence, since improved visual quality alone does not establish semantic accuracy or temporal consistency.

Questions for authors

Please refer to the weakness section.

1 comment

We thank the reviewer for the thoughtful review and positive assessment of Odyssey. We hope Odyssey can provide a foundation for navigation reasoning research that addresses the practical challenges of deploying autonomous driving systems in the real world.


Q1. The effect of diffusion refinement on evaluation reliability could be validated further.

The perception and driving results, together with the supplementary visualizations of lane markings, vehicles, and pedestrians, provide useful evidence. Quantitative checks of whether refinement preserves these elements across consecutive frames would further strengthen confidence, since improved visual quality alone does not establish semantic accuracy or temporal consistency.

A1. Diffusion refinement improves the recognizability of driving-critical scene elements, bringing perception accuracy closer to that achieved on recorded images. To quantify this improvement, we evaluate perception performance using Mask2Map for map elements and BEVFormer for 3D objects, both along the logged trajectory (OL) and during closed-loop planner rollouts (CL). As shown in the table below (and Figure 11 of the revised Appendix), refinement consistently improves AP across all evaluated categories compared with 3DGS-only rendering.

ObservationSplitLane centerline AP ↑Ped. crossing AP ↑Road boundary AP ↑Det. vehicle AP ↑Det. pedestrian AP ↑Det. traffic cone AP ↑Det. generic object AP ↑
GT ImageOL67.059.665.141.914.219.361.3
3DGS OnlyOL64.156.261.935.712.712.554.0
+ RefinementOL66.359.064.836.413.916.355.1
3DGS OnlyCL45.035.043.529.86.20.812.9
+ RefinementCL47.338.346.130.36.41.813.2

Since refinement is applied independently to each frame, it may introduce temporal inconsistencies in the appearance or geometry of scene elements. To assess this effect, we measure the class-wise IoU between Mask2Map predictions from consecutive frames after compensating for ego motion. As shown in the table below, refinement achieves higher temporal consistency across all evaluated map categories than 3DGS-only rendering. These results suggest that refinement improves semantic perception without compromising the temporal consistency of map elements.

ObservationSplitLane centerline IoU ↑Ped. crossing IoU ↑Road boundary IoU ↑mIoU ↑
GT ImageOL77.062.767.669.1
3DGS OnlyOL76.062.266.568.2
+ RefinementOL76.962.967.669.1
3DGS OnlyCL71.955.864.464.0
+ RefinementCL73.356.966.165.4

Comments on the paper

1 comment

We sincerely thank all reviewers for their thoughtful and constructive feedback. Odyssey is designed to evaluate how planners interpret coarse navigation routes, prepare for upcoming maneuvers, and sustain safe, goal-directed driving over long-horizon closed-loop rollouts. In response to the reviewers' concerns, we conducted additional experiments and analyses to further substantiate the value of long-horizon navigation evaluation and strengthen confidence in Odyssey's simulation and evaluation reliability. We summarize these efforts below, with detailed responses provided under the corresponding reviews.

1. Benchmark Value and Navigation Evaluation

  • Benchmark differentiation [UFng]: Compared Odyssey with existing benchmarks to highlight its ability to evaluate successive navigation decisions and their downstream consequences.
  • Horizon-dependent performance [UFng]: Evaluated 20/50/100-second horizons with 95% confidence intervals, revealing changes in planner rankings.
  • Navigation failure analysis [UFng]: Analyzed lane-preparation failures across horizons, highlighting navigation challenges missed by shorter evaluations.
  • Scenario coverage [4itc]: Quantified 647 navigation-relevant events across 2.8 hours of real-world driving.
  • Navigation guidance ablations [UFng]: Isolated the effects of route representation and integration strategy, demonstrating improvements from SD-route guidance and cross-attention.

2. Simulation and Evaluation Reliability

  • Semantic and rendering fidelity [AbiC, UFng]: Evaluated perception accuracy and rendering quality along logged trajectories and across off-log deviations, demonstrating improvements from diffusion refinement.
  • Temporal consistency [AbiC, 4itc]: Evaluated consecutive-frame map consistency and multi-frame planning stability, finding no substantial degradation from refinement.
  • Traffic realism [4itc]: Demonstrated improved WOSAC realism with section-based triggering while acknowledging limitations in counterfactual interactions.
  • Metric robustness [UFng, 4itc]: Confirmed stable planner rankings across tested distance thresholds and RouteDS penalty factors.
  • Downstream evaluation stability [UFng]: Compared collision counts, RouteDS, and planner rankings with and without refinement, finding largely consistent relative planner performance.

We also clarified the evaluation protocol, including map-matching safeguards, RouteDS penalty design, and limitations of simplified traffic interactions.

Together, these additional analyses provide further evidence for the value of evaluating sustained, route-conditioned navigation beyond isolated driving decisions, while helping clarify the reliability and limitations of Odyssey's evaluation framework. We hope our responses help address the reviewers' concerns and would sincerely appreciate your consideration of this additional evidence in your final assessments. Thank you again for your time and careful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.