acceptodds
Withdrawn from ICLR 2027

Which Synthetic Driving Data Are Worth Training On? Dynamic Data Valuation Based on Their Effects on Model Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative models can now produce synthetic driving data at scale, providing a promising way to improve the performance of end-to-end driving models beyond costly real-world data collection. However, not all synthetic data are equally valuable for training, making it important to accurately identify which data are worth training on. Existing methods typically value synthetic data using their properties, such as visual realism and temporal consistency, thereby treating data value as an invariant property of the data. This overlooks two key issues: (i) these data properties do not directly reveal whether synthetic data actually improve model performance; and (ii) the value of synthetic data is not fixed, but varies across training stages and can be sensitive to the data used for valuation. To address these issues, we propose DriveVal, which evaluates synthetic driving data based on their training effects on end-to-end driving model performance and uses the resulting values for data selection. With a small set of real driving data as references, data value is quantified by the dot product between the real-data gradient and the synthetic-data gradient under evaluation. In addition, DriveVal periodically re-evaluates data throughout training while accounting for the current optimizer state at each stage, allowing the valuation to adapt as training progresses. Finally, DriveVal penalizes variations in data value across different real references and synthetic-data combinations, preventing the valuation from being dominated by a particular data choice. Across five end-to-end driving models on NAVSIM, DriveVal selects synthetic training data matched in size to navtrain throughout training. The resulting models achieve 95.2% of the mean PDMS of their corresponding real-data baselines, while improving mean PDMS by 13.6% over size-matched random selection and by 9.1% over training on the full synthetic set. Further experiments demonstrate that DriveVal remains effective across different generation sources and visual conditions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Synthetic Driving Data, End-to-End Autonomous Driving, Dynamic Data Valuation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wUnrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DriveVal, a dynamic framework that selects synthetic driving data based on training utility rather than intrinsic data properties. It estimates data value using optimizer-aware, second-order effects on real-data loss, penalizes valuation instability, and periodically updates data selection.

Experiments on NAVSIM across five driving models show consistent improvements over random selection and full synthetic-data training, achieving an average PDMS of 84.24 versus 74.17 and 77.23, respectively.

Critical strengths
    1. Well-motivated problem formulation
      The paper addresses an important challenge in synthetic data curation by evaluating data based on their training utility. The proposed framework provides a practical approach to dynamically selecting useful synthetic driving data.
    1. Comprehensive ablation studies and consistent empirical validation
      The paper systematically evaluates the contributions of optimizer-aware valuation, second-order estimation, variation penalties, and periodic re-evaluation. Additional analyses establish correlations between estimated training values and observed loss reductions, providing evidence beyond downstream performance gains.
Critical weaknesses
  • W1. Unclear incremental novelty and insufficient differentiation from prior work.
    The paper builds on existing work in dynamic data valuation, including OPUS and In-Run Data Shapley. However, the methodological distinctions and incremental contributions of DriveVal are not sufficiently clarified. Furthermore, the experiments mainly compare against random selection and full synthetic-data training, without competitive data selection baselines, making it difficult to establish the advantages of the proposed approach.

  • W2. Lack of end-to-end compute-matched comparisons
    The current experiments match training samples and optimizer updates, but DriveVal incurs substantial additional costs from repeated data valuation and selection. Since data curation is an integral part of the proposed framework, end-to-end comparisons under equal total GPU budgets are essential. Without such evaluations, it remains unclear whether DriveVal offers practical advantages over random selection or full synthetic-data training when the entire training pipeline is considered.

Questions for authors
  • Q1. Comparison with existing dynamic data selection methods
    The current experiments primarily compare DriveVal against random selection and full synthetic-data training. Could the authors provide comparisons with representative dynamic data selection methods, particularly optimizer-aware approaches such as OPUS, under matched experimental settings? Such comparisons are important for establishing the incremental benefits of DriveVal over existing approaches.

  • Q2. Computational cost and efficiency
    DriveVal performs 20 valuation stages throughout training, yet reports only 1.27–3.43× the GPU computation of random selection. Could the authors clarify how these costs are measured and provide a breakdown of training, gradient-based valuation, and data selection overhead? In particular, do the reported GPU-hours include the complete end-to-end pipeline, including all repeated valuations and Top-K selection operations?

  • Q3. Reference overfitting and out-of-distribution generalization
    DriveVal repeatedly selects synthetic data based on gradient alignment with the same 240 real reference samples. Could this adaptive selection introduce overfitting to the reference distribution, potentially limiting generalization to unseen driving scenarios? Beyond the current NAVSIM evaluation, could the authors compare the OOD performance of models trained with real data, full synthetic data, random selection, and DriveVal to determine whether the proposed selection strategy improves generalization rather than primarily optimizing performance on the target distribution?

Reviewer_6AFzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method to better leverage synthetic data to train generative models for autonomous driving. They identified two key issues in existing methods: 1) the data properties as visual realism and temporal consistency do not reveal whether synthetic data actually improve model performance and 2) the value of synthetic data is not fixed during training, and for different evaluations. They propose DriveVal which evaluates synthetic driving data based on some reference data and periodically choose "good" synthetic data based on gradient updates comparison between syntheitic and reference data across the training. They show that their method works better than training with randomly selected or full synthetic data.

Critical strengths
  1. The problem is relevant but very specific. It could be more interesting if we can show it on other generative modeling tasks, e.g., image and video generation other than autonomous driving.
  2. The method is interesting and makes sense.
  3. The experimental gain is considerable compared to the chosen baselines.
Critical weaknesses
  1. There are missing baselines for experiments. The current baselines are very primitive, i.e., randomly selection or just use the full synthetic data. The motivation of the paper is that the existing methods that evaluate the data properties overlook some aspects, and thus the authors should compare with these existing methods as baselines.

  2. The idea of leveraging scalable synthetic data to train models sounds promising, but the reality seems bitter. The experiments show that using synthetic data cannot beat the models trained with real data. Since real data exist, could the authors give stronger motivations to why use synthetic data to train models a priori.

  3. It's not sufficiently presented in the paper how the synthetic data is selected, for example, how often the re-evaluation is run, how many synthetic data is selected at each re-evaluation, is all synthetic data used during the full training, etc.

  4. The efficiency is not discussed in the paper. To conduct data selection, it needs more computations. In addition, authors argue that we should do data selection per-stage. It would be nice to reveal the computational costs and how this selection slows down the training.

Questions for authors

see weaknesses

Reviewer_cGzTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DriveVal selects synthetic sequences during the training of end-to-end driving models. It estimates their value using gradients on real reference data, the current optimizer state, and a second-order correction. It penalizes variation across reference sets and synthetic-data combinations and refreshes the selected subset every five epochs. Experiments on NAVSIM report improvements over random selection and training on the full synthetic pool across five driving models.

Critical strengths
  1. Selecting useful synthetic data is a relevant problem for autonomous driving. Table 1 reports consistent improvements across five models, with mean PDMS increasing from 74.17 for random selection and 77.23 for the full synthetic pool to 84.24. These comparisons use matched numbers of training updates and processed samples.

  2. The experiments cover several generation sources and include ablations of the optimizer state, variation penalty, and periodic re-evaluation. This provides useful evidence that the observed gains extend beyond a single driving model or generation source.

Critical weaknesses
  1. The methodological novelty is limited, and the incremental contribution is not convincingly established. Reference-guided gradient scoring and optimizer-aware data selection have precedents in LESS and OPUS; OPUS also updates selection throughout training. Equation (5) explicitly follows the second-order attribution form of In-Run Data Shapley. The distinctive additions appear to be the particular optimizer-Jacobian formulation and the penalty for variation across real references and synthetic-data combinations. The paper does not establish what these additions offer over representative existing selectors. Comparisons with Random, the full synthetic pool, and internal variants show that the pipeline works, but leave its advance over prior methods unresolved. Table 9 also does not report a first-order version retaining the full periodic selection schedule and variation penalty. This is a relevant lower-cost comparator given the total costs of 1.27–3.43 times Random in Table 13, including 2.84 times for DiffusionDrive. The paper needs to establish the contribution of the proposed extensions relative to existing selection methods, with both the training budget and total valuation cost accounted for.

  2. The driving-specific behavior of the selector remains poorly characterized. Tables 4 and 7 analyze agreement with local loss reductions, and Figure 2 tracks changes in rankings and subsets. They do not show which scenes are selected or what distinguishes retained from rejected data. This matters because Appendix A.1 pairs generated observations with original targets and auxiliary inputs, while reporting mainly file-alignment and provenance checks. The generation conditions, remaining image–label inconsistencies, and how the selector responds to them are insufficiently described. A breakdown by scene type and representative scored examples would help distinguish artifact filtering, improved scenario coverage, and preference for scenes resembling the small real reference set. The claimed reduction in sensitivity to reference choice also needs evidence about the resulting selections and trained models under different reference draws; the existing bootstrap analysis concerns local loss reductions on the held-out reference set.

Questions for authors
  1. What conditions are used to generate the camera observations, and which scene attributes can change? How do you verify that the original training targets and auxiliary inputs remain valid? Please provide representative paired examples and results of any consistency checks beyond file/token correspondence.

  2. What distinguishes high-value and low-value sequences in practice? Examples of retained and rejected data, including cases whose rankings change during training, would help explain the selection. Does the variation penalty disproportionately remove rare or difficult scenes?

  3. Does adding DriveVal-selected synthetic data improve over training on a fixed real dataset alone? How does this compare with adding randomly selected synthetic data or using the original observations from the same source sequences?

  4. How does DriveVal compare with a simpler dynamic gradient-based selector or an existing optimizer-aware selection method, using the same real reference data and accounting for valuation cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.