- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper evaluates five autonomous-driving planners on seven nested subsets of the NVIDIA PhysicalAI dataset, ranging from 1 to 64 hours of training data. The results show that planner rankings and data-doubling gains vary with training-data size, while improvements in open-loop trajectory errors do not necessarily lead to higher EPDMS.
Critical strengths
- The paper highlights a limitation of fixed-size benchmarks, which do not reveal how planner performance changes with increasing amounts of training data.
- The experiments cover five planners, seven data scales, and multiple evaluation metrics, with a generally clear experimental protocol.
Critical weaknesses
-
The contribution is limited. The main novelty is a controlled comparison of existing planners across multiple data scales. However, the findings that planner rankings may change with data size and that open-loop metrics may not align with EPDMS are relatively intuitive and have been discussed in prior work. The paper therefore provides limited new scientific insight beyond a broader empirical comparison.
-
The central conclusions are insufficiently supported. Results are based on a single training run and one subset chain, making it difficult to separate data-scaling effects from subset composition or training randomness. The final reporting set also overlaps with the validation set used for checkpoint selection, potentially introducing selection bias.
Questions for authors
-
Could the authors clarify which methodological or empirical conclusions are novel relative to prior public studies of data scaling and open-loop/planning-metric mismatch?
-
Why is the validation subset used for checkpoint selection included in the final reporting set, and would the main conclusions hold on a fully independent evaluation set?