acceptodds
Withdrawn from ICLR 2027

Bench2Drive-Plan: Bridging Privileged Planner and End-to-End Autonomous Driving under Shared Protocols

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning-based autonomous driving has developed within two separate communities based on the input information. Privileged planners receive ground-truth object states and HD maps while end-to-end models use raw sensor inputs. Despite sharing many similar modeling ideas, differences in scene inputs, simulation,and metrics make it difficult to compare results and transfer insights across them. In this work, we introduce to bridge this gap by evaluating privileged planners under current end-to-end protocols. On Bench2Drive, we adapt vectorized, diffusion-based, and language-based planning paradigms,including PLUTO, Diffusion Planner and InstructDriver. We also implement two raster-based CNN baselines using RGB semantic BEV images and multi-channel semantic masks, respectively. Comparisons with end-to-end baselines results show that ground-truth scene inputs do not inherently yield a performance advantage for the privileged planners. This finding raises a question about the information needed for learning based driving. Ground-truth boxes and maps provide accurate scene states but may omit useful context from raw sensors. End-to-end models could utilize visual context, but their learned perception may be inaccurate. To examine whether planners could benefit from visual information beyond boxes and maps, we augment privileged planners with camera inputs. On Bench2Drive, extra camera inputs provide no overall benefit. While on NAVSIM, extra camera inputs yield a positive effect, which might indicate that under real world complex environment, scene details beyond boxes could be helpful. Together, these studies show how a shared benchmark can benefit the studies of the two communities and we hope Bench2Drive-Plan could pave the way for future studies about synergistically training planner and end-to-end models.

Primary area:
datasets and benchmarks
Keywords:
Autonomous driving, Benchmark and evaluation, Closed-loop evaluation, Motion planning, End-to-end learning, Bird's-eye-view representation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_KRvtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work extends the existing CARLA-based E2E driving benchmark Bench2Drive to privildged planners. It provides various privlidge input interfaces including vecorized representation, BEV mask, BEV RGB, and language representation, and evaluates planners using these representations on the proposed benchmark. It enables direct comparison between E2E planners and privilidge planners on Bench2Drive. The results show that the GT perception inputs to privilidge planners don't bring advances compared with E2E planners in this evaluation protocol.

Critical strengths

It provides adapters that port nuPlan-style learned planners (PLUTO, Diffusion Planner) and an LLM-based planner to Bench2Drive. If released, these are a useful resource for the planning community.

Critical weaknesses

First, the motivation overlooks existing work that already evaluates privileged planners under E2E protocols. The paper is motivated by stating that privileged planners are commonly evaluated on nuPlan and are therefore hard to compare with E2E methods (line 46, 47). However, privileged planners have long been evaluated alongside E2E planners under the same protocols:

  • on CARLA: rule-based planners (PDM-Lite [1]), RL-based planners (Roach [2], Think2Drive [5]), and learned planners (PlanT [3], PlanT 2.0 [4])
  • on NAVSIM: the privileged PDM-Closed [6] is reported together with TransFuser(TF) and LTF [7]. Notably, the official Bench2Drive training data used here was itself collected by Think2Drive, a privileged RL planner.

Second, the claim that “ground-truth scene inputs do not inherently yield a performance advantage” (line 25) is attributed to missing visual cues (line 370), while several stronger confounders are left uncontrolled:

  • training data: top baselines such as SimLingo are trained on self-collected PDM-Lite data, whereas the privileged planners tested here use the standard Bench2Drive training set with only 950 clips. Table 1 does not indicate the training data of each method.
  • output representation: in Table 1, the only two methods above 65 DS (SimLingo, HiP-AD) are also the only ones with decoupled path/speed-style outputs. All other E2E baselines output temporal trajectories, waypoints, or direct control, and score 38–65 DS, the same range as the tested privileged planners. HiP-AD is trained on standard Bench2Drive training set, which makes it the clearest case. Prior CARLA work (e.g., TF++ [8]) has also argued for path-based outputs over plain trajectories. The privileged planners tested here use plain trajectory outputs without such adaptation, which may account for much of their gap.

Both factors are illustrated by PlanT 2.0 [4], a privileged learned planner. It is trained on PDM-Lite data and uses decoupled path and waypoint outputs, and it reaches 92.4 DS on Bench2Drive, above every E2E method in Table 1. Its ablation further shows that, with identical privileged inputs, replacing plain waypoints with decoupled path and waypoint outputs raises the normalized DS on the CARLA validation routes from 8.1 to 28.6. This suggests that the gap between privileged planner and the best e2e planners in Table 1 likely stems from data and output design rather than from missing visual cues.

Finally, this work introduces no new scenarios, metrics, or data. Its contribution is a set of input adapters on top of the existing Bench2Drive protocol. It is unclear what capability or insight this work enables beyond running existing planners on an existing benchmark.

[1] Sima et al. PDM-Lite: A Rule-Based Planner for CARLA Leaderboard 2.0. Technical report.

[2] Zhang et al. End-to-End Urban Driving by Imitating a Reinforcement Learning Coach. ICCV, 2021.

[3] Renz et al. PlanT: Explainable Planning Transformers via Object-Level Representations. CoRL, 2022.

[4] Gerstenecker et al. PlanT 2.0: Exposing Biases and Structural Flaws in Closed-Loop Driving. arXiv:2511.07292, 2025.

[5] Li et al. Think2Drive: Efficient Reinforcement Learning by Thinking with Latent World Model for Autonomous Driving (in CARLA-v2). ECCV, 2024.

[6] Dauner et al. Parting with Misconceptions about Learning-based Vehicle Motion Planning. CoRL, 2023.

[7] Chitta et al. TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving. IEEE TPAMI, 2023.

[8] Jaeger et al. Hidden Biases of End-to-End Driving Models. ICCV, 2023.

Questions for authors
  • How do the tested privileged planners perform when trained on PDM-Lite data and/or given decoupled outputs? How does PlanT 2.0 perform within your pipeline? If the gap to the strongest E2E methods closes, the finding "ground-truth scene inputs do not inherently yield a performance advantage for the privileged planners"would not hold.
  • Beyond the input adapters, what does Bench2Drive-Plan enable that the existing Bench2Drive protocol does not?
Reviewer_DUgaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Bench2Drive-Plan, which is a unified benchmark to evaluate both privileged planners and sensor-based end-to-end driving agents under a shared closed-loop evaluation protocol. The authors adapt five privileged planners, including PLUTO, Diffusion Planner, InstructDriver, BEV-RGB and BEVMask and compare them with 12 sensor-based end-to-end planners. The experiments show that access to ground-truth scene input does not give privileged planners a performance advantage over sensor-based models. The authors further investigate if incorporating additional visual information into privileged planners could improve the performance. They report that adding visual information gives no overall benefit (and slight degradation) in Bench2Drive, whereas it shows a slight positive effect in real-world scenarios on NAVSIM.

Critical strengths

The idea of establishing a single shared closed-loop benchmark that unifies privileged motion planning models and sensor-based end-to-end driving models under the same protocol is a good idea. It makes it easier to compare model performance between two communities. The specific engineering design can also be beneficial for future work, such as how to extract and adapt to native representations needed by different planners, how to increase scenario and actor behavior diversity, and so on.

Critical weaknesses
  1. The contribution of the paper is incremental and lacks novelty. The authors primarily perform a bunch of engineering integration work by taking existing privileged planners and adapting them to run within the Bench2Drive benchmark. Incorporating camera information into a privileged planner is also an incremental experiment rather than a conceptual contribution. Moreover, the finding that ground-truth representation does not contain important visual cues and that the privileged agent might suffer from distribution shifts in closed-loop evaluation is already presented in previous end-to-end driving literature, e.g., ChauffeurNet, Learning by Cheating.
  2. The methodology of camera information study is not thorough, and analysis is shallow. First, using a single camera view is probably inadequate for autonomous planning, as most modern planners use multi-camera setups. Moreover, the reported performance degradation when adding camera inputs might also be related to suboptimal training settings or naive fusion mechanisms rather than an inherent uselessness of visual information. The authors provide no detailed diagnostic experiments to support their hypothesis.
  3. The presentation quality needs improvement. Figure 4 and Table 3 convey essentially identical data without offering new information. One of them should be removed. And the paper contains some formatting issues, such as missing space after periods, e.g., line 045, 049, and 087.
Questions for authors
  1. Did you analyze whether the performance gap between privileged agents and the best sensor-based end-to-end models comes from closed-loop distribution shifts? Privileged models are trained offline using perfect ground-truth inputs. When evaluated in closed-loop settings, any small control deviation can lead the vehicle into unvisited states outside the training distribution. Providing detailed failure cases or a thorough qualitative/quantitative failure analysis would help clarify this point.
  2. Do you have ablation studies showing how variations in scene representation extraction, such as lane sampling strategies, affect planner performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.