- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DriveVal, a dynamic framework that selects synthetic driving data based on training utility rather than intrinsic data properties. It estimates data value using optimizer-aware, second-order effects on real-data loss, penalizes valuation instability, and periodically updates data selection.
Experiments on NAVSIM across five driving models show consistent improvements over random selection and full synthetic-data training, achieving an average PDMS of 84.24 versus 74.17 and 77.23, respectively.
Critical strengths
-
- Well-motivated problem formulation
The paper addresses an important challenge in synthetic data curation by evaluating data based on their training utility. The proposed framework provides a practical approach to dynamically selecting useful synthetic driving data.
- Well-motivated problem formulation
-
- Comprehensive ablation studies and consistent empirical validation
The paper systematically evaluates the contributions of optimizer-aware valuation, second-order estimation, variation penalties, and periodic re-evaluation. Additional analyses establish correlations between estimated training values and observed loss reductions, providing evidence beyond downstream performance gains.
- Comprehensive ablation studies and consistent empirical validation
Critical weaknesses
-
W1. Unclear incremental novelty and insufficient differentiation from prior work.
The paper builds on existing work in dynamic data valuation, including OPUS and In-Run Data Shapley. However, the methodological distinctions and incremental contributions of DriveVal are not sufficiently clarified. Furthermore, the experiments mainly compare against random selection and full synthetic-data training, without competitive data selection baselines, making it difficult to establish the advantages of the proposed approach. -
W2. Lack of end-to-end compute-matched comparisons
The current experiments match training samples and optimizer updates, but DriveVal incurs substantial additional costs from repeated data valuation and selection. Since data curation is an integral part of the proposed framework, end-to-end comparisons under equal total GPU budgets are essential. Without such evaluations, it remains unclear whether DriveVal offers practical advantages over random selection or full synthetic-data training when the entire training pipeline is considered.
Questions for authors
-
Q1. Comparison with existing dynamic data selection methods
The current experiments primarily compare DriveVal against random selection and full synthetic-data training. Could the authors provide comparisons with representative dynamic data selection methods, particularly optimizer-aware approaches such as OPUS, under matched experimental settings? Such comparisons are important for establishing the incremental benefits of DriveVal over existing approaches. -
Q2. Computational cost and efficiency
DriveVal performs 20 valuation stages throughout training, yet reports only 1.27–3.43× the GPU computation of random selection. Could the authors clarify how these costs are measured and provide a breakdown of training, gradient-based valuation, and data selection overhead? In particular, do the reported GPU-hours include the complete end-to-end pipeline, including all repeated valuations and Top-K selection operations? -
Q3. Reference overfitting and out-of-distribution generalization
DriveVal repeatedly selects synthetic data based on gradient alignment with the same 240 real reference samples. Could this adaptive selection introduce overfitting to the reference distribution, potentially limiting generalization to unseen driving scenarios? Beyond the current NAVSIM evaluation, could the authors compare the OOD performance of models trained with real data, full synthetic data, random selection, and DriveVal to determine whether the proposed selection strategy improves generalization rather than primarily optimizing performance on the target distribution?