- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the limitations of using a single scalar score to evaluate end-to-end driving models. It characterizes driving competence across four dimensions—safety, comfort, route fidelity, and efficiency—and uses partial-order analysis to reveal substantial trade-offs that make many trajectories incomparable. Based on these findings, the paper proposes the Driving Competence Profile (DCP), which combines an admissibility gate, multidimensional competence metrics, and a scalar score to provide a more comprehensive assessment of driving models.
Critical strengths
The paper introduces an order-theoretic approach to multidimensional driving competence evaluation. Unlike conventional scalar rankings, it uses Pareto incomparability and order-dimension analysis to reveal trade-offs among safety, comfort, route fidelity, and efficiency across different driving models.
Critical weaknesses
- Unclear contribution of DCP. PDMS/EPDMS already contain multiple evaluation components, and applying Pareto analysis directly to these components might yield similar findings. The paper lacks this essential baseline and does not distinguish the contributions of redesigned metrics from those of multidimensional reporting. Moreover, the incomparability rate and order dimension are sensitive to axis partitioning, raising questions about the generality of the reported quantitative findings.
- Insufficient validation of DCP's evaluation accuracy. Several key competence axes in DCP lack independent validation. The human comparison depends on the definition of efficiency, and the DCP scalar ranking does not fully agree with closed-loop evaluation results. The current experiments do not adequately demonstrate that DCP reflects actual driving competence more accurately than existing metrics.
- Known evaluation error left uncorrected. The trajectory sampling resolution used in the paper is lower than the official NAVSIM standard, potentially missing violations and affecting evaluation scores. Although the authors acknowledge this issue in the appendix, they do not correct it or recompute the main results. This directly raises concerns about the reliability of the safety assessment and subsequent partial-order statistics.
Questions for authors
- Would applying Pareto analysis directly to the existing PDMS/EPDMS components yield similar findings? Moreover, does the sensitivity of incomparability rates to axis partitioning affect the generality of the partial-order conclusions?
- The human comparison depends on the efficiency definition, and DCP rankings do not fully agree with closed-loop evaluation results. How can the authors demonstrate that DCP reflects actual driving competence more accurately than existing metrics?
- The trajectory sampling resolution differs from the official NAVSIM standard. Could the authors recompute the main results using the official resolution and examine whether the safety scores, incomparability rates, and order dimensions change?