- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces HiDrive, a new CARLA-like closed‑loop benchmark for end‑to‑end autonomous driving. HiDrive constructs 330 routes, with 30 ability categories divided into Basic / Hard / Thorny sets, with long‑tail scaenarios, interactive traffic, legal constraints, and ethical‑norm‑aware scenarios. Together with the benchmark, the authors propose a dedicated metric including Route Completion (RC), Legal Score (LS), Ethics Score (ES), and composite Driving Score (DS). The authors evaluate models trained on Bench2Drive data and show that conventional planners achieve low overall DS, while VLA / knowledge‑augmented methods obtain clear gains, especially on Hard and Thorny sets.
Critical strengths
Practical studied problem: Current popular autonomous‑driving benchmarks are suffering from score saturation. When model scores cluster between 88‑95%, it becomes difficult to tell whether incremental numerical improvements correspond to meaningful real‑world capability gaps. HiDrive targets exactly this pain point, aiming to expose capability differences among existing end‑to‑end planners.
Novel evaluation dimension for high‑order driving capabilities beyond collision & lane‑following: Beyond classic collision avoidance and route completion, HiDrive explicitly models legal and ethical driving scenarios. These scenarios are closely related to daily driving experience and safety requirements. This capability‑oriented design is forward‑looking for end‑to‑end driving research.
Empirical results demonstrate good discriminative power of the benchmark: Experimental results reveal consistent trends: models augmented with external world‑knowledge / VLA components obtain substantial gains, especially over Hard‑Set and Thorny‑Set scenarios. Different families of methods show clearly separated performance patterns, which validates that HiDrive can tell apart models’ high‑level decision‑making abilities.
Critical weaknesses
I did not find critical weaknesses of this paper, but several important open questions that need clarification in the rebuttal/revision:
The value of improved rendering / visual fidelity needs further justification:
One of HiDrive's selling points is UE5‑based higher‑fidelity lighting and textures compared to Bench2Drive. However, I am not sure about the motivation of improving visual quality. If the goal is to evaluate generalization toward real‑world driving, even improved CARLA rendering still has large intrinsic sim‑to‑real visual mismatch. It remains ambiguous how much gain is actually brought by enhanced rendering quality for benchmark discrimination. The paper does not provide controlled ablation: keeping scenarios identical while toggling rendering quality, to quantify how much model ranking shifts.
Clarification about scenario construction and metric calibration:
- For high‑level normative scenarios like Pedestrian‑related ethics (e.g. puddle‑splash avoidance): How are these concrete scenario instantiated? Are they hand‑crafted, or generated from real‑world logs?
- For ethics‑aware penalties such as
PUDDLE_ETHICS_INFRACTION: Table 6 presents penalty coefficients derived from human surveys. However, the paper only lists these coefficients without describing the exact computation pipeline of the criterion itself. Since the overall Driving Score (DS) heavily relies on these coefficients and the breakdown metrics self, clarification over all related metrics is required.
Questions for authors
See weaknesses.
3 comments
Response
I thank the authors for the response.
For W1, the authors have not answered my key question: what are the main benefits of higher rendering fidelity? The visual outputs shown in the paper are still easily distinguishable as simulations rather than real scenes.
I tend to keep my acceptance rating, given the considerable workload and effort behind the simulator development.
Thank you for your reply. Regarding the question of rendering fidelity:
On the one hand, some new instances are difficult to construct at low rendering fidelity. For example, when constructing water puddles in the UE4 engine, we could only achieve an effect similar to that in Figure 1, row 2, column 3. The lack of puddle reflections made it difficult even for humans to determine whether it was a stain on the ground or a puddle. In this case, it was difficult to fairly test whether a model could handle puddles. After increasing the rendering fidelity, we achieved the effect shown in Figure 2, row 2, column 1, and only then did the puddle bear some resemblance to a real puddle.
On the other hand, although our benchmark still has a gap with the real world, our sim-to-real gap is considerably smaller than that of existing benchmarks such as Bench2Drive. UE5 provides tools for lighting, reflections, and so on, so we spent some effort leveraging them to improve rendering fidelity. The main users of simulation benchmarks are academic teams that do not have the conditions to conduct real-vehicle testing. We believe that since it is possible to make the simulation more realistic, we should invest effort to make it as good as possible. This is our working creed when building HiDrive.
Finally, thank you again for your time and for recognizing our work.
We thank the reviewer for their recognition of our work and constructive comments. Below, we address the reviewer's concerns and questions.
W1. The value of improved rendering / visual fidelity needs further justification
UE5 allows certain assets and scenarios to appear more realistic and visually distinguishable. For example, UE4 renders standing water poorly, potentially making it difficult even for human observers to identify accurately, whereas UE5 can reproduce its visual characteristics more realistically through global illumination and specular reflections.
Q2.1. Construction of high-level normative scenarios
These high-level normative scenarios are manually designed.
Drawing on numerous driving knowledge bases and accident records, we designed scenario assets to be as diverse and comprehensive as possible. Many test cases correspond to representative real-world accidents involving injuries or fatalities. Although such cases may rarely appear in routine driving logs, mishandling them can have severe consequences. With Level 2 driving automation now relatively mature, the test cases we introduce are critical for assessing progress toward higher-level driving automation.
Q2.2. Computation pipeline of ethics-aware penalties
The evaluation process first detects the corresponding violation events based on predefined scenario conditions and then applies the associated penalty coefficients to update the relevant scores.
Regarding coefficient calibration, collision penalties follow established protocols adopted by the CARLA Leaderboard and Bench2Drive. Ethical penalties are calibrated with reference to the range of collision penalty coefficients: we invited 14 participants to rank norm-sensitive events according to their relative severity and mapped the resulting rankings to different penalty levels while preserving their relative severity ordering. Therefore, coefficients such as 0.7 and 0.8 reflect different levels of severity.
To investigate the impact of the absolute coefficient values on baseline capability evaluation, we also provide a sensitivity analysis in Table 7 of Appendix A.1, where all penalty coefficients are uniformly set to 0.5. The results show that the trends obtained using our designed penalty coefficients are largely consistent with those obtained using a fixed penalty coefficient of 0.5.
In HiDrive's open-source evaluation framework, users can choose between our predefined set of penalty coefficients and a version in which all penalty coefficients are uniformly set to 0.5.