- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Modern end-to-end driving planners perform well on common driving benchmarks, but it remains difficult to evaluate their true safety under rare corner cases. This is primarily constrained by the scarcity of safety-critical events in natural driving logs, the high cost of staged data collection, and the difficulty of existing generative methods in guaranteeing multi-view and cross-modal (LiDAR) geometric consistency. To address this, the paper proposes CornerFactory, an automated, specification-driven pipeline grounded in real-world driving logs. Utilizing 3D Gaussian reconstruction and ray tracing, it achieves multi-view-consistent object insertion, novel-view synthesis from perturbed ego poses, and vehicle-configuration transfer, while strictly providing aligned LiDAR updates. Building on this, CornerBench, a benchmark comprising 230 cases (spanning 3 major categories and 19 concrete scenario types) is constructed. By introducing continuous, severity-aware metrics, it enables a systematic and fine-grained quantitative evaluation of existing models' true robustness and safety response capabilities in long-tail scenarios.
Critical strengths
-
Rigorous Geometric and Cross-Modal Alignment: By leveraging 3D Gaussian reconstruction and ray tracing, it ensures spatial self-consistency across multi-view and RGB-LiDAR inputs. This effectively reduces confounding factors where models might fail simply due to mismatched input interfaces or generation artifacts.
-
Discriminative Evaluation Metrics: The continuous metrics can quantitatively distinguish between "unmitigated collisions" and "attempts to avoid hazards (e.g., significant deceleration)", providing a more fine-grained basis for evaluating a model's safety response strategies, rather than treating all failures identically.
-
Exposing Limitations of Existing Methods: It systematically validates the bottlenecks of current purely data-driven models in long-tail scenarios, providing clear testbeds and strong motivation for introducing new mechanisms, such as causal alignment or experiential memory.
Critical weaknesses
None
Questions for authors
Q1. Open Source and Extensibility: Will the CornerFactory generation pipeline and the CornerBench dataset be fully open-sourced? Furthermore, does the pipeline support users in defining new structured scene specifications (e.g., specific causal intervention logic) to extend and generate novel test cases?
Q2. Fine-Grained Data Export: Does the evaluation script support exporting time-series data of intermediate states ? This would facilitate a more detailed analysis of the model's specific response behaviors at critical moments, rather than relying solely on the final aggregated score.
2 comments
Thank you for the additional clarifications. I will maintain my rating.
Response to Reviewer 8mcc
We sincerely appreciate your insightful questions and we believe they hold significant value for our work. We try to resolve your concerns below.
Thank you for raising this important question about openness and extensibility. We have now publicly released anonymous resources for inspection and reuse:
(1) Defining new scenarios. Users can edit structured specifications or sample parameter ranges with recorded random seeds:
These controls let users define new corner cases grounded in recorded driving scenes. For example, they can vary an obstacle's position or crossing time to create different hazard scenarios. This flexibility enables the community to build customized datasets for model training, evaluation, and other research applications.
(2) Pipeline upgrades and dataset expansion. To provide the community with higher-quality generation tools and a more comprehensive dataset, we have made two updates:
We also compare video quality between the original and upgraded pipelines separately for ego-state perturbations and object insertion, using VBench components [R1]:
Thank you for this suggestion. Our evaluation tooling saves both the predicted trajectory and the simulated ego-vehicle state sequence for each sample, allowing researchers to examine how a planner responds at critical moments. The available outputs support the following analyses:
For example, two planners may both collide yet differ substantially in how much they slow before impact. Their speed traces and collision statistics reveal these differences and help explain the behavior underlying the final score. For a concrete example, Appendix I, Figure 25 (p. 27) of the revised manuscript visualizes the predicted trajectories and simulated speed curves of Drive-JEPA and DiffusionDrive in three collision cases.
We appreciate your constructive feedback and hope our responses have addressed your concerns. We welcome any additional questions or suggestions that could help us further improve our work.
References
[R1] Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR 2024, pp. 21807–21818. https://arxiv.org/abs/2311.17982