acceptodds
Withdrawn from ICLR 2027

Beyond Fixed Benchmarks: Measuring Data-Scaling Behavior of Autonomous-Driving Planners on the Physical AI Autonomous Vehicles Dataset

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Public autonomous-driving benchmarks typically compare planners at a single training-data size. This identifies the top-ranked planner at that size but not how performance or rankings change with more data. Using the NVIDIA Physical AI Autonomous Vehicles Dataset (PhysicalAI), we train five planners on common nested subsets ranging from 1 to 64 hours. We evaluate the selected checkpoints on a fixed, training-disjoint PhysicalAI set using three-second average and final displacement errors (ADE3 and FDE3), and on NAVSIM v2 navtest using the closed-loop-relevant Extended Predictive Driver Model Score (EPDMS). The scaling curves reveal three behaviors hidden by a single benchmark point: the top-ranked planner according to ADE3 changes with training-data size; planners achieve different gains after the same data doubling; and lower open-loop error does not always lead to higher EPDMS. Uni-World VLA has the highest post-adaptation EPDMS at every training-data size, yet its score changes by only 0.268 points from 1 to 64 hours. These results separate absolute performance from data-scaling behavior and show why comparisons among public planners should report scaling curves rather than one fixed-size result.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Data Scaling, Closed-loop Performance, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_qmLLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper evaluates five autonomous-driving planners on seven nested subsets of the NVIDIA PhysicalAI dataset, ranging from 1 to 64 hours of training data. The results show that planner rankings and data-doubling gains vary with training-data size, while improvements in open-loop trajectory errors do not necessarily lead to higher EPDMS.

Critical strengths
  1. The paper highlights a limitation of fixed-size benchmarks, which do not reveal how planner performance changes with increasing amounts of training data.
  2. The experiments cover five planners, seven data scales, and multiple evaluation metrics, with a generally clear experimental protocol.
Critical weaknesses
  1. The contribution is limited. The main novelty is a controlled comparison of existing planners across multiple data scales. However, the findings that planner rankings may change with data size and that open-loop metrics may not align with EPDMS are relatively intuitive and have been discussed in prior work. The paper therefore provides limited new scientific insight beyond a broader empirical comparison.

  2. The central conclusions are insufficiently supported. Results are based on a single training run and one subset chain, making it difficult to separate data-scaling effects from subset composition or training randomness. The final reporting set also overlaps with the validation set used for checkpoint selection, potentially introducing selection bias.

Questions for authors
  1. Could the authors clarify which methodological or empirical conclusions are novel relative to prior public studies of data scaling and open-loop/planning-metric mismatch?

  2. Why is the validation subset used for checkpoint selection included in the final reporting set, and would the main conclusions hold on a fully independent evaluation set?

Reviewer_9xANOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates data‑scaling behaviors for end‑to‑end autonomous driving planners. The authors construct nested training subsets ranging from 1 h to 64 h from the PhysicalAI dataset and run controlled training for five representative planner architectures. Models are evaluated with open‑loop trajectory metrics (ADE3 / FDE3) and NAVSIM‑based closed‑loop‑relevant EPDMS score. Key observations include: (1) planner rankings change with training‑data size; (2) open‑loop trajectory improvement does not consistently align with closed‑loop‑relevant EPDMS improvement; (3) different planners exhibit distinct data‑doubling gains.

Critical strengths

Practical research with comprehensive empirical setup: The paper studies data‑scaling laws for end‑to‑end driving planners, an essential yet under‑explored topic in autonomous driving research. The authors design nested 1 h‑to‑64 h training splits and benchmark five diverse planner paradigms (simple MLP, BEV planner, sparse‑token planner, VLA‑based planner, world‑action planner). A rich set of open‑loop and pseudo‑closed‑loop metrics is reported, and supplementary materials provide fine‑grained per‑run measurements, which makes the empirical study substantial.

Well‑structured and well‑written manuscript: The paper clearly motivates the limitation of conventional single‑size benchmarks, states research questions, describes experimental protocols, and presents observed results. Readers can easily follow the studied problem and the discovered empirical patterns.

Critical weaknesses

Lack of scene‑composition analysis for nested subsets weakens the generality of scaling conclusions: Autonomous driving is heavily scene‑dependent. The performance gain when increasing dataset size is strongly determined by what kinds of scenarios are included. For an extreme example: if all added hours only contain simple highway straight‑driving scenes without lane‑change or interactive traffic cases, scaling dataset size may bring almost no real‑world improvement. Similarly, two 1‑hour subsets can produce vastly different scaling implications, depending on whether they cover diverse challenging scenarios or only trivial easy cases. The paper only states that nested subsets are subsampled from the full pool, but provides no analysis of scene composition (e.g., distribution of maneuver types, interaction complexity, weather, geography) across these nested splits. Without controlling or characterizing scene diversity, we cannot tell whether the observed performance trends come purely from data volume or are confounded by shifting scene distribution. This makes the reported scaling observations conditional and limits the generalizability of conclusions.

Limited actionable guidance for autonomous‑driving development (major concern): After reading all results, it remains unclear what concrete take‑aways that can be adopt for real‑world model developement. For instance:

  • Which metric (ADE3 / FDE3 / EPDMS) should engineers prioritize for model selection?
  • Given limited compute and data budget, what is the practical preference among these five planner families?
  • How much data should a production team collect given a target performance requirement?

The paper mostly describes what empirical patterns are observed, but does not translate these patterns into prescriptive guidance for driving‑system development. As an empirical scaling study, higher practical prescriptive value would significantly strengthen its impact.

Minor limitation: experimental uncertainty from single‑run per‑configuration: Each planner‑and‑data‑size combination uses only one training run, without multiple random seeds or alternative subset sampling. As such, we cannot separate true architectural scaling behavior from training noise / random subset fluctuation. The authors themselves acknowledge this point, but it still restricts how firmly we can trust some of the observed rank‑reversal phenomena.

Questions for authors

See weakness.

Reviewer_AixHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that the conventional practice of comparing model performance on benchmarks with a fixed training-data size can yield an incomplete picture of model rankings.

The paper constructs a nested sequence of progressively larger subsets from the NVIDIA Physical AI Autonomous Vehicles Dataset (PhysicalAI), trains multiple autonomous-driving models on these subsets, and compares their performance across training-data sizes.

Specifically, the experiments evaluate AD-MLP, BevAD-lite, Tiny-SSR, TPS-Drive, and Uni-World VLA, reporting open-loop metrics on PhysicalAI and the EPDMS pseudo-simulation metric on NAVSIM v2, and analyzing how their rankings and scores change as the training-data size increases.

Critical strengths

The paper presents detailed experiments comparing how different models respond to increasing training-data sizes and shows that their relative rankings vary across scales; this deserves consideration, given that conventional comparisons typically evaluate models at a single shared training-data size.

Critical weaknesses

The observation that “open-loop improvements do not always translate into EPDMS gains” has already been widely discussed in prior work; although this paper provides quantitative experiments, the results remain descriptive and do not yield a clear conclusion with sufficient practical guidance.

The paper uses only one nested subset sequence for training, and its starting dataset of approximately 20,000 frames is relatively small, making the results potentially sensitive to the sampling distribution; conclusions about these specific models should therefore be interpreted cautiously.

The paper keeps each model’s hyperparameters fixed across all training-data sizes, whereas the optimal hyperparameters can differ substantially across scales, which further weakens the conclusions about these specific models.

Questions for authors

Could you repeat the experiments using multiple independently sampled nested subset sequences and determine whether the same rankings or trends persist?

Reviewer_iapyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study of how self-driving planners perform as training-data scale increases. The authors evaluate five planners across nested NVIDIA PhysicalAI training subsets using two types of metrics: open-loop trajectory L2 errors and planning performance measured by EPDMS in NAVSIM. They observe that planner rankings vary with data scale and that improvements in open-loop trajectory accuracy do not always translate into better EPDMS performance.

Critical strengths
  • The paper studies an important question: how self-driving planner performance changes as training-data scale increasing. Its findings provide useful empirical evidence that planner rankings can vary with different data scale.

  • The paper evaluates five planners on shared nested training subsets using both open-loop prediction L2 error metrics and planning performance. This provides a useful empirical comparison across planners and reveals that improvements in open-loop prediction accuracy do not always translate into better planning performance.

Critical weaknesses
  • The paper reports performance trends but does not sufficiently investigate what causes them. In particular, although the mismatch between open-loop trajectory accuracy and downstream EPDMS is an important observation, the paper does not explain when or why this mismatch occurs. This makes it difficult for the community to draw further conclusions from the results.

  • The paper does not sufficiently translate its empirical observations into practical guidance for model development or data collection. For example, it remains unclear how the findings should inform planner selection or when additional training data are likely to improve downstream performance. This limits the paper's broader contribution beyond just providing a descriptive comparison of planner performance at different data scales.

Questions for authors
  • What explains the observed mismatch between open-loop trajectory metrics and EPDMS?

  • Can the observed trends support practical recommendations or principals for planner selection or data collection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.