acceptodds
Withdrawn from ICLR 2027

Driving Competence Is Not a Scalar: A Pareto Audit and Profile for End-to-End Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A single number ranks end-to-end driving planners, and NAVSIM's PDMS, built to fix its predecessor, is that number. Each repair so far has adjusted weights inside the number and kept its type. We argue the type is wrong. Driving competence rests on attributes that genuinely trade off, so the object of scoring is a partial order rather than a quantity. On navtest, under a four-axis operationalization whose partition we also sweep, 62–71% of admissible trajectory pairs are Pareto-incomparable, and the per-scene competence poset has certified Dushnik–Miller order dimension ≥3 in ∼91% of scenes and dimension 1 in none. A scalar's own order is total, so on every such pair it must either collapse a real distinction or order it by an undeclared preference, and that contested set is exactly the incomparable set at any positive weighting. Known complaints follow. PDMS separates a human from a strong sensor-based planner by under 0.01 in 46.3% of scenes, its per-scene winner contradicts its aggregate ranking in up to 28.8% of scenes, and on a deployed pair that closed-loop testing separates by six points, the sign of its verdict turns on scorer implementation and weighting. The Driving Competence Profile (DCP) reports competence at its type. It returns an admissibility gate, four graded axes calibrated independently of the evaluated planners, and a leaderboard scalar carried with the profile and a map of which comparisons it arbitrates. Over 13 zoo planners plus current state-of-the-art methods, it ranks the human above a blind no-perception model in 83.4% of scenes against the official PDMS's 71.6%, and it flags evidence consistent with an external Goodhart step, a method whose published PDMS rose from 88.1 to 91.2 while its measured competence fell.

Primary area:
datasets and benchmarks
Keywords:
autonomous driving evaluation, end-to-end planning, NAVSIM, partial order, Pareto incomparability, benchmark metrics, Goodhart's law, measurement theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7pptOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the limitations of using a single scalar score to evaluate end-to-end driving models. It characterizes driving competence across four dimensions—safety, comfort, route fidelity, and efficiency—and uses partial-order analysis to reveal substantial trade-offs that make many trajectories incomparable. Based on these findings, the paper proposes the Driving Competence Profile (DCP), which combines an admissibility gate, multidimensional competence metrics, and a scalar score to provide a more comprehensive assessment of driving models.

Critical strengths

The paper introduces an order-theoretic approach to multidimensional driving competence evaluation. Unlike conventional scalar rankings, it uses Pareto incomparability and order-dimension analysis to reveal trade-offs among safety, comfort, route fidelity, and efficiency across different driving models.

Critical weaknesses
  • Unclear contribution of DCP. PDMS/EPDMS already contain multiple evaluation components, and applying Pareto analysis directly to these components might yield similar findings. The paper lacks this essential baseline and does not distinguish the contributions of redesigned metrics from those of multidimensional reporting. Moreover, the incomparability rate and order dimension are sensitive to axis partitioning, raising questions about the generality of the reported quantitative findings.
  • Insufficient validation of DCP's evaluation accuracy. Several key competence axes in DCP lack independent validation. The human comparison depends on the definition of efficiency, and the DCP scalar ranking does not fully agree with closed-loop evaluation results. The current experiments do not adequately demonstrate that DCP reflects actual driving competence more accurately than existing metrics.
  • Known evaluation error left uncorrected. The trajectory sampling resolution used in the paper is lower than the official NAVSIM standard, potentially missing violations and affecting evaluation scores. Although the authors acknowledge this issue in the appendix, they do not correct it or recompute the main results. This directly raises concerns about the reliability of the safety assessment and subsequent partial-order statistics.
Questions for authors
  • Would applying Pareto analysis directly to the existing PDMS/EPDMS components yield similar findings? Moreover, does the sensitivity of incomparability rates to axis partitioning affect the generality of the partial-order conclusions?
  • The human comparison depends on the efficiency definition, and DCP rankings do not fully agree with closed-loop evaluation results. How can the authors demonstrate that DCP reflects actual driving competence more accurately than existing metrics?
  • The trajectory sampling resolution differs from the official NAVSIM standard. Could the authors recompute the main results using the official resolution and examine whether the safety scores, incomparability rates, and order dimensions change?
Reviewer_TfxfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the evaluation of end-to-end autonomous driving systems. The authors argue that existing scalar metrics cannot fully capture the trade-offs among different driving capabilities. Through Pareto analysis, the paper reveals the widespread incomparability among driving trajectories. To address this issue, the authors propose the Driving Competence Profile (DCP), which evaluates safety, comfort, route fidelity, and efficiency as separate dimensions. Experiments further investigate its advantages over existing metrics such as PDMS and EPDMS.

Critical strengths

1.The paper revisits the widely adopted scalar evaluation paradigm in autonomous driving, providing a valuable perspective on the trade-offs among different driving capabilities.

  1. The authors conduct a systematic analysis of existing evaluation metrics. Through extensive trajectory comparisons, ablation studies, and model comparisons, the paper highlights capability differences that may be obscured by a single scalar score.
Critical weaknesses
  1. whether these redesigned new metrics provide a more reliable assessment of driving competence. Further evidence would help establish the advantages of the proposed metric definitions over their existing counterparts.

  2. Although DCP provides a multidimensional assessment of driving competence, its practical utility remains insufficiently validated, particularly given the inconsistent model rankings between DCP and closed-loop evaluations in Table 3.

3.Minor Comments The manuscript reports different Pareto-incomparability rates (62–71% and approximately 72%) based on different comparison sets. The corresponding sample definitions could be made clearer, particularly in Figure 1. The frequent use of labels such as P1, P2, P5, and P7 makes parts of the discussion difficult to follow. In the Contributions section (Page 2), the expression "four (P7 new)" appears to be an editorial artifact and should be clarified or revised.

Questions for authors

see weakness

Reviewer_7njvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how aggregate scores represent driving performance in autonomous driving evaluation. It uses Pareto comparisons to analyze trade-offs among safety, comfort, route adherence, and efficiency. The proposed Driving Competence Profile (DCP) combines an admissibility gate, four graded metrics, and an aggregate score based on a geometric mean by default. Experiments on NAVSIM compare several planners and examine metric discrimination and ranking sensitivity.

Critical strengths

The paper provides a detailed analysis of the limitations of aggregate driving scores. Concrete examples make the trade-offs between different aspects of driving performance easy to understand. The authors also examine how metric definitions and aggregation choices affect the results. They discuss several limitations of their approach, which helps readers understand the scope of their claims.

Critical weaknesses

The paper's contribution is very limited. Safety, comfort, route adherence, and efficiency are already used across common planning benchmarks such as NAVSIM, Waymax, and nuPlan. DCP mainly changes the metric calculations and parameter settings, without introducing a new evaluation dimension. The paper stresses the need to assess multiple dimensions of driving competence. However, NAVSIM already reports individual metrics such as NC, DAC, TTC, and Comfort alongside PDMS. These metrics already allow researchers to assess different aspects of driving performance. The paper also argues that weighted aggregate scores reflect preference choices, but it does not explain why its geometric mean is better justified than the weighted aggregation used in PDMS. Therefore, DCP appears to be a variant of PDMS with different scoring details, without a clear advance in evaluation goals, dimensions, or accuracy.

Questions for authors
  • What additional value does DCP provide over the existing NAVSIM sub-metrics when the same Pareto analysis and weight-sensitivity checks are applied to them?

  • What justifies the default equal-weight geometric mean over PDMS’s weighted aggregation, given that both methods encode preference choices?

  • What independent evidence shows that the proposed route adherence, comfort, and efficiency scores better reflect driving quality than the existing metrics?

  • How would the dominance and incomparability results change if each metric used a tolerance based on observed variation across runs rather than a shared fixed threshold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.