acceptodds
Under review as a conference paper at ICLR 2027

HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving

Abstract

End-to-end autonomous driving has witnessed rapid progress, yet existing benchmarks are increasingly saturated, with state-of-the-art models achieving near-perfect scores on widely used open-loop and closed-loop benchmarks. This saturation does not mean that the problem has been solved; instead, it reveals that current benchmarks remain limited in scenario diversity, object variety, and the breadth of driving capabilities they evaluate. In particular, they lack sufficient long-tail scenarios involving rare but safety-critical objects and fail to assess advanced decision-making such as legal compliance, ethical reasoning, and emergency response. To address these gaps, we propose HiDrive, a new closed-loop benchmark for end-to-end autonomous driving that emphasizes long-tail scenarios and a richer evaluation of driving capabilities. HiDrive introduces a diverse set of rare objects and uncommon traffic situations, and expands evaluation from basic driving skills to more advanced capabilities, including rule compliance, moral reasoning, and context-dependent emergency maneuvers. Correspondingly, we extend previous collision-avoidance-centered metrics into a comprehensive evaluation system that encompasses collision and braking, traffic-rule compliance, and moral-reasoning indicators. Built on a more advanced physics engine, HiDrive provides physically realistic lighting and high-fidelity visual rendering, offering a more challenging and realistic testbed for assessing whether autonomous driving systems can handle the complexity of real-world deployment. The HiDrive software, source code, digital assets, and documentation are available at https://huggingface.co/HiDriveBenchmark/HiDriveBenchmark.

Primary area:
datasets and benchmarks
Keywords:
Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NXRWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces HiDrive, a new CARLA-like closed‑loop benchmark for end‑to‑end autonomous driving. HiDrive constructs 330 routes, with 30 ability categories divided into Basic / Hard / Thorny sets, with long‑tail scaenarios, interactive traffic, legal constraints, and ethical‑norm‑aware scenarios. Together with the benchmark, the authors propose a dedicated metric including Route Completion (RC), Legal Score (LS), Ethics Score (ES), and composite Driving Score (DS). The authors evaluate models trained on Bench2Drive data and show that conventional planners achieve low overall DS, while VLA / knowledge‑augmented methods obtain clear gains, especially on Hard and Thorny sets.

Critical strengths

Practical studied problem: Current popular autonomous‑driving benchmarks are suffering from score saturation. When model scores cluster between 88‑95%, it becomes difficult to tell whether incremental numerical improvements correspond to meaningful real‑world capability gaps. HiDrive targets exactly this pain point, aiming to expose capability differences among existing end‑to‑end planners.

Novel evaluation dimension for high‑order driving capabilities beyond collision & lane‑following: Beyond classic collision avoidance and route completion, HiDrive explicitly models legal and ethical driving scenarios. These scenarios are closely related to daily driving experience and safety requirements. This capability‑oriented design is forward‑looking for end‑to‑end driving research.

Empirical results demonstrate good discriminative power of the benchmark: Experimental results reveal consistent trends: models augmented with external world‑knowledge / VLA components obtain substantial gains, especially over Hard‑Set and Thorny‑Set scenarios. Different families of methods show clearly separated performance patterns, which validates that HiDrive can tell apart models’ high‑level decision‑making abilities.

Critical weaknesses

I did not find critical weaknesses of this paper, but several important open questions that need clarification in the rebuttal/revision:

The value of improved rendering / visual fidelity needs further justification: One of HiDrive's selling points is UE5‑based higher‑fidelity lighting and textures compared to Bench2Drive. However, I am not sure about the motivation of improving visual quality. If the goal is to evaluate generalization toward real‑world driving, even improved CARLA rendering still has large intrinsic sim‑to‑real visual mismatch. It remains ambiguous how much gain is actually brought by enhanced rendering quality for benchmark discrimination. The paper does not provide controlled ablation: keeping scenarios identical while toggling rendering quality, to quantify how much model ranking shifts.

Clarification about scenario construction and metric calibration:

  • For high‑level normative scenarios like Pedestrian‑related ethics (e.g. puddle‑splash avoidance): How are these concrete scenario instantiated? Are they hand‑crafted, or generated from real‑world logs?
  • For ethics‑aware penalties such as PUDDLE_ETHICS_INFRACTION: Table 6 presents penalty coefficients derived from human surveys. However, the paper only lists these coefficients without describing the exact computation pipeline of the criterion itself. Since the overall Driving Score (DS) heavily relies on these coefficients and the breakdown metrics self, clarification over all related metrics is required.
Questions for authors

See weaknesses.

3 comments

We thank the reviewer for their recognition of our work and constructive comments. Below, we address the reviewer's concerns and questions.

W1. The value of improved rendering / visual fidelity needs further justification

UE5 allows certain assets and scenarios to appear more realistic and visually distinguishable. For example, UE4 renders standing water poorly, potentially making it difficult even for human observers to identify accurately, whereas UE5 can reproduce its visual characteristics more realistically through global illumination and specular reflections.

Q2.1. Construction of high-level normative scenarios

These high-level normative scenarios are manually designed.

Drawing on numerous driving knowledge bases and accident records, we designed scenario assets to be as diverse and comprehensive as possible. Many test cases correspond to representative real-world accidents involving injuries or fatalities. Although such cases may rarely appear in routine driving logs, mishandling them can have severe consequences. With Level 2 driving automation now relatively mature, the test cases we introduce are critical for assessing progress toward higher-level driving automation.

Q2.2. Computation pipeline of ethics-aware penalties

The evaluation process first detects the corresponding violation events based on predefined scenario conditions and then applies the associated penalty coefficients to update the relevant scores.

Regarding coefficient calibration, collision penalties follow established protocols adopted by the CARLA Leaderboard and Bench2Drive. Ethical penalties are calibrated with reference to the range of collision penalty coefficients: we invited 14 participants to rank norm-sensitive events according to their relative severity and mapped the resulting rankings to different penalty levels while preserving their relative severity ordering. Therefore, coefficients such as 0.7 and 0.8 reflect different levels of severity.

To investigate the impact of the absolute coefficient values on baseline capability evaluation, we also provide a sensitivity analysis in Table 7 of Appendix A.1, where all penalty coefficients are uniformly set to 0.5. The results show that the trends obtained using our designed penalty coefficients are largely consistent with those obtained using a fixed penalty coefficient of 0.5.

In HiDrive's open-source evaluation framework, users can choose between our predefined set of penalty coefficients and a version in which all penalty coefficients are uniformly set to 0.5.

Reviewer_NXRWOpenReview ↗

Response

I thank the authors for the response.

For W1, the authors have not answered my key question: what are the main benefits of higher rendering fidelity? The visual outputs shown in the paper are still easily distinguishable as simulations rather than real scenes.

I tend to keep my acceptance rating, given the considerable workload and effort behind the simulator development.

Thank you for your reply. Regarding the question of rendering fidelity:

On the one hand, some new instances are difficult to construct at low rendering fidelity. For example, when constructing water puddles in the UE4 engine, we could only achieve an effect similar to that in Figure 1, row 2, column 3. The lack of puddle reflections made it difficult even for humans to determine whether it was a stain on the ground or a puddle. In this case, it was difficult to fairly test whether a model could handle puddles. After increasing the rendering fidelity, we achieved the effect shown in Figure 2, row 2, column 1, and only then did the puddle bear some resemblance to a real puddle.

On the other hand, although our benchmark still has a gap with the real world, our sim-to-real gap is considerably smaller than that of existing benchmarks such as Bench2Drive. UE5 provides tools for lighting, reflections, and so on, so we spent some effort leveraging them to improve rendering fidelity. The main users of simulation benchmarks are academic teams that do not have the conditions to conduct real-vehicle testing. We believe that since it is possible to make the simulation more realistic, we should invest effort to make it as good as possible. This is our working creed when building HiDrive.

Finally, thank you again for your time and for recognizing our work.

Reviewer_zNn4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HiDrive, a new closed-loop benchmark for autonomous driving. Building upon existing benchmarks, HiDrive introduces more realistic and challenging long-tail scenarios, including uncommon obstacles, abnormal vehicle behaviors, emergency vehicles, and situations involving legal and ethical considerations. The authors evaluate representative autonomous driving methods from different paradigms and provide fine-grained, ability-oriented performance analyses. Overall, I consider this an important and meaningful contribution to autonomous driving evaluation.

Critical strengths
  1. Well-motivated. The paper identifies important limitations of existing autonomous driving benchmarks, particularly their insufficient coverage of long-tail scenarios and high-level decision-making capabilities. The proposed benchmark addresses a relevant and meaningful research problem.
  2. Easy to follow. The paper is clearly written and well-organized. The benchmark design, capability taxonomy, evaluation metrics, and experimental results are presented in a structured and understandable manner.
Critical weaknesses
  1. Limited comparison with existing benchmarks. The related work and benchmark comparison could be more comprehensive. Currently, the paper primarily compares HiDrive with nuScenes, NAVSIM, CARLA Leaderboard, and Bench2Drive. Including more recent and relevant benchmarks, particularly those focusing on safety-critical scenarios and high-level decision-making, would help better position HiDrive and clarify its unique contributions.
  2. Missing WAM-based baselines. Although the paper discusses WAMs in the related work, no representative WAM-based approach is included in the experiments. Evaluating such a method would provide a more comprehensive comparison across autonomous driving paradigms and offer valuable insights into which architectures are more promising for handling long-tail scenarios and high-level decision-making tasks.
  3. Validity of legal and ethical evaluation. One of the key contributions of HiDrive is its evaluation of legal compliance and ethical decision-making. However, these capabilities are primarily assessed through predefined violation events and penalty coefficients. It remains unclear to what extent these rule-based metrics can faithfully reflect high-level reasoning and ethical decision-making capabilities. Further validation, such as examining the agreement between benchmark scores and human or expert judgments, would strengthen the credibility of the proposed evaluation framework.
Questions for authors

None.

4 comments

We thank the reviewer for recognizing our work and for the constructive comments. Below, we address the reviewer's concerns and questions.

W1. More comparison with existing benchmarks

Thank you for the suggestion. We have added a discussion and comparison of Safe2Drive, a recent safety-critical driving benchmark. Safe2Drive evaluates safety behaviors on 100 routes covering work zones, jaywalking, and occluded vulnerable road users. HiDrive offers broader scenario coverage and more diverse long-tail objects, extending evaluation to legal compliance, ethical decision-making, and situations involving conflicting rules or abnormal conditions.

We have incorporated this discussion and comparison into the Related Work section; please refer to the latest PDF. We also welcome recommendations for other relevant benchmarks and will incorporate appropriate citations and comparisons based on their relevance to our work.

W2. WAM-based baselines

Thank you for the suggestion. We have added two WAM-based baselines, DA-WAM (Zhong et al., 2026) and Latent-WAM (Wang et al., 2026), to broaden the comparison across driving paradigms. Both methods are evaluated on the same 330 HiDrive test routes, achieving overall driving scores of 24.2 and 36.3, respectively.

We have added the corresponding method descriptions and references to Section 4.1 and included the evaluation results in Table 3. Please refer to the latest PDF.

W3. Validity of legal and ethical evaluation

Our metric design draws on existing closed-loop benchmarks. As you noted, this evaluation is based on planning outcomes and predefined rules. However, designing metrics that measure a specific capability in isolation can be challenging across autonomous driving systems, particularly end-to-end models, whose internal capabilities are difficult to interpret. We would welcome further discussion with you on potentially better metric designs.

Regarding agreement with human judgments, the large number of test scenarios and variation in individuals' value judgments make it challenging to complete a study with sufficient samples within the limited time available. We will conduct such an evaluation in future work and share the results through HiDrive's open-source community.

Dear Reviewer zNn4,

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether we have addressed all your concerns. If any points remain unclear or require further clarification in our response and revision, please do not hesitate to let us know.

We greatly appreciate your time and look forward to your feedback.

Thank you for the detailed response and additional experiments. The expanded benchmark comparison and the two additional WAM-based baselines have addressed my first two concerns.

Regarding the third concern, I acknowledge the challenges in designing metrics to evaluate legal compliance and ethical decision-making. I look forward to seeing future experiments examining the agreement between benchmark scores and human or expert judgments. Overall, I will maintain my current score.

Thank you for your reply.

Regarding your third concern. During the discussion phase, we conducted an experiment. For all baselines tested, in law-and-ethics test scenarios, we sampled 120 driving trajectories (60 were judged by our evaluation code as violations and 60 as compliant with the rules). We invited 10 respondents, each of whom randomly sampled some trajectories and viewed the corresponding sensor videos and BEV trajectories to manually judge whether they were violations. The survey yielded a total of 510 data points, summarized as follows:

Human judges as compliantHuman judges as violation
HiDrive Code judges as compliant2510
HiDrive Code judges as violation3256

In only 3/510 cases did humans and our code make different judgments. We further analyzed and discussed these 3 cases with the respondents: two were due to human oversight during judgment, and one was because humans had different standards for the speed at which one should decelerate when encountering a puddle (the threshold in HiDrive was set to the average of the values determined by 14 respondents). In short, consistency remains relatively high, and we hope this addresses your concern.

We will subsequently add this information to the appendix and conduct further analysis. In addition, our benchmark is currently open-sourced anonymously, and users can audit all the evaluation code.

Reviewer_VTZXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

the paper introduces a calar-based close-loop benchmark for AV evaluation in long-tail and context-sensitive situations. It has 330 short routes with 30 abilities and 94 scenario instantiations. The evaluation contains route completion, legal related event penalities to provide aggregate and ability level evaluation. The paper provides experiment results on 10 driving systems. The main contribution is an expanded evaluation resource and protocol.

Critical strengths
  1. the topic the paper targets is a meaningful gap in the AV evaluation.
  2. concrete, exntensible close-loop resource is provided.
Critical weaknesses
  1. the reported scores do not align with the definition of the score? In Equation 1, DS_i = RC_i x LS_i x ES_i, and all of those metrics are within 0~1. Then why in table 3, ThinkTwice DS > RC?

  2. the claimed capability interpretation is not sufficiently established. The following things are changed at the same time:

  • rendering
  • object appearance and traffic flow distribution
  • scenario demands
  • scoring rules Then poor performance does not clearly establish failure of high-level reasoning, rather than perception/domain-transfer or ordinary control failure.

Likewise, the “ethics” score operationalizes selected event rules and penalties. The 14-person severity-ranking survey helps motivate penalty ordering, but does not independently validate event detection, acceptable behaviors or the interpretation of the score as moral reasoning.

Questions for authors

how were the evaluator's judgements validated for scenarios with multiple reasonable responses, especially dilemma cases?

3 comments

We thank the reviewer for their recognition of our work and constructive comments. Below, we address the reviewer's concerns and questions.

W1. Inconsistency between the reported scores and the metric definition

Thank you for your careful checking. After verification, we found that the Hard Set DS for some methods was computed using an earlier temporary implementation, which incorrectly used ∣IE,s∣|I_{E,s}| instead of NsN_s as the denominator. We have corrected the affected results; please refer to the latest PDF.

This bug is absent from the publicly released evaluation toolkit. Some methods were evaluated using the updated implementation, which matches the released code, and their results were therefore unaffected.

W2. Insufficient evidence for the claimed capability interpretation

Like other closed-loop driving benchmarks, HiDrive evaluates models' end-to-end planning capabilities through their final control outputs. These outputs alone make it difficult to determine which internal capability is deficient, particularly for end-to-end models, whose internal processes may be difficult to interpret.

The statement that "different sets test different capabilities" may be imprecise. Rather, the sets comprise different types of scenarios:

  • Basic Set: Covers fundamental driving scenarios.
  • Hard Set: Introduces scenarios involving traffic laws, ethics, and defensive driving.
  • Thorny Set: Further places models in scenarios involving conflicting rules or exceptional circumstances.

We have revised the corresponding statements in the manuscript. Please refer to the latest PDF for these changes.

W3. Insufficient validation of the ethics score

Similar to the driving score, the ethics and legal scores evaluate the appropriateness of the model's final driving behavior against predefined requirements. We cannot determine whether the model internally understands traffic laws or possesses moral reasoning capabilities, nor can we distinguish deficiencies in perception, reasoning, or execution, particularly for end-to-end models whose internal processes are difficult to interpret.

Q1. Acceptance of multiple reasonable responses and evaluation of dilemma scenarios

Our evaluator uses a penalty-based mechanism: behavior incurs no penalty unless it triggers a predefined violation event. Consequently, any reasonable response can receive full scores.

Regarding dilemma scenarios, we distinguish between fake dilemmas and real dilemmas:

  • Fake dilemmas: Consider an ambulance approaching from behind while the traffic light is red. This scenario admits a clear, legally supported response: crossing the stop line against the red light to give way. Accordingly, running the red light in this scenario incurs no penalty.

  • Real dilemmas: For situations resembling the trolley problem, we assign penalties based on the resulting harm. This design reflects our view that a capable autonomous driving system should, through appropriate speed control, safe following distances, and proactive observation, minimize the likelihood of entering situations where such difficult trade-offs become unavoidable.

Dear Reviewer VTZX,

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether we have addressed all your concerns. If any points remain unclear or require further clarification in our response and revision, please do not hesitate to let us know.

We greatly appreciate your time and consideration and look forward to your feedback.

Reviewer_VTZXOpenReview ↗

response to the authors

thanks for the clarification, i think my concerns are addressed and i plan to change the score to be positive.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. We have revised the manuscript, with the main changes highlighted in red:

  1. Score correction: We corrected Hard Set and overall driving scores in Table 3.
  2. Evaluation interpretation: We revised the wording to emphasize performance across different scenario types and clarify the limits of interpreting internal capabilities.
  3. Benchmark comparison: We added Safe2Drive to the Related Work discussion and benchmark comparison in Table 1.
  4. Additional baselines: We added DA-WAM and Latent-WAM to Section 4.1 and Table 3, with Latent-WAM results obtained from our own reproduction.

Please refer to the revised PDF and our individual responses for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.