acceptodds
Withdrawn from ICLR 2027

Image-Defect Evidence for Task-Fidelity Configuration of Frozen Driving Video Generators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Driving-video generators provide synthetic observations for offline evaluation of autonomous-driving systems, but visually plausible outputs can alter perception, tracking, and planning metrics relative to condition-aligned real observations. Inference-time controls offer a way to reduce these discrepancies without retraining, yet evaluating every configuration with the full driving stack is costly. We present a reference-based configuration workflow that uses image-defect measurements to prioritize downstream evaluation while keeping both the generator and evaluators frozen. Single-factor screening identifies controls that alter task-associated defects, and a lightweight surrogate ranks candidates within the retained search space. Each search round sends only the best-ranked candidate for full downstream evaluation, while a separate multi-task check assesses nominated configurations on held-out scenes. Experiments with the WorldDreamer renderer and a Cosmos-based pipeline on 26 InterHub scenes show that reconstruction and temporal-consistency defects are associated with perception and planning gaps in both pipelines after accounting for scene and generator effects. Held-out evaluation identifies one configuration per pipeline that reduces the composite gap while satisfying task-specific tolerance checks under the frame-level decision rule. The relative reductions are 31.24% and 21.02% for WorldDreamer and Cosmos, respectively, corresponding to absolute reductions of 0.1415 and 0.0952 on the fixed percentile-normalized scale. The WorldDreamer configuration primarily narrows the tracking gap, while the Cosmos configuration narrows both perception and tracking gaps. In a matched single-run comparison, guided search reaches a lower task gap using 41% of the GPU time recorded for random search. These results support image-defect-guided configuration of frozen generators for offline driving evaluation, with image measurements prioritizing candidates and observed task-metric gaps determining which configurations to retain.

Primary area:
generative models
Keywords:
Driving Video Generation, Task Fidelity, Black-Box Models, Inference-Time Control, Autonomous Driving, Configuration Search, Defect Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PHGMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

When using video generators to evaluate autonomous driving systems, one challenge is that the video generator can render visually plausible outputs that still cause perception/planning/tracking algorithms to diverge from how they'd behave in real-world settings. This paper tries to tune a frozen video generator's inference settings such that the AV stack (also frozen) behaves on its renderings in the same manner as it would on real video.

The core idea is to pair real camera frames with generated frames of the same scene, moment, and camera. The same AV stack is run on both and the differences in perception (spotting objects), tracking (following the objects over time), and planning (the distance between where a planner thinks the car should go and where it actually goes) are measured. The generated frames depend on the video model's hyperparameters, and since running the full AV stack on every possible configuration is too expensive, the method instead measures six image-level metrics (L1, PSNR, LPIPS, ProjErr, LPIPS-t, FlowErr) and trains a small model (RBF-SVR) to predict the combined AV gap from them. After determining the metrics that matter, the method runs multiple search rounds. In each round, the small model ranks five videos with different hyperparameter settings, the full AV stack is run on the most promising one, and the setting is only kept if its measured gap improves. Each round's tested setting is then evaluated on held-out validation scenes and accepted if it clearly reduces the overall AV gap without any single task's gap getting worse beyond a certain threshold.

The experiments evaluate their method on DriveArena WorldDreamer and a Cosmos-Transfer1-based pipeline using 26 scenes from InterHub. These scenes are split into 12 screening scenes to train the small image-metric model, 6 search scenes used during the search rounds, and 8 held out scenes. The small model ranks frames decently (rank correlation 0.8338) but predicts the exact AV metric gap only moderately (R^2 0.4999). The overall method proposes 19 candidates to test against the default generator hyperparameters, of which 2 are accepted, 8 are tradeoffs, and 9 are rejected. One accepted setting reduces the composite AV gap by 31.24% relative to WorldDreamer's defaults (mainly by improving tracking) and another reduces the gap by 21.02% relative to Cosmos's defaults. Figure 3 shows an example where tracking briefly loses a car with the WorldDreamer defaults and does not with the method's suggested settings.

Critical strengths

I think the problem setting is interesting and agree that generating pretty visuals does not mean a video generator is well-suited to AV simulation. I also find the authors' problem setting of keeping the generator and AV stack frozen, and portraying the method to be cheap inference-time alignment, to be well-motivated. I found this to be the main benefit of reading the paper.

Critical weaknesses

This submission is extremely dense and difficult to read, with passages such as "we present a reference-based configuration workflow that uses image-defect measurements to prioritize downstream evaluation while keeping both the generator and evaluators frozen" in the abstract and "paired defect vectors and composite task gaps train the ranking surrogate, while residualized associations identify guardrail channels," in Figure 2, Sec 3.2's title "RESIDUALIZED DEFECT–TASK ASSOCIATION", etc. Almost every sentence in the manuscript contains similar unclear jargon, which makes it hard for me to envision the community gaining real benefit from reading this paper and impacts my rating negatively.

I also have big concerns about the experimental protocols and results:

  • The validation sample is very small (only 8 held-out scenes), and the reported intervals treat frames from the same scene as independent.
  • The AV stack the method uses is questionable and performs poorly on real frames. Sec. 4 reports planner L2 of 9.56–32.93 m on real scenes, whereas the same VAD-Tiny model is reported at 0.78 m average error on nuScenes [1]. Figure A4 shows perception F1 of 0.037 on a real-world scene. This implies the "task-fidelity gap" the paper measures is largely meaningless (comparing differences between two failed outputs).
  • There is no real evidence that the image-space metrics are a meaningful proxy for the AV stack performance. The authors themselves note that the surrogate's 0.83 rank correlation "does not independently establish ordering among configurations of one generator" (Sec. 5.4), and the paper has no ablation (Sec. 6). The only baseline is a single 8-frame comparison against random search, where the respective gaps are 0.447 (guided) and 0.453 (random, equal to the default).
  • The paper claims that the real references are from nuScenes images that correspond to InterHub trajectories, but InterHub's released data (nuPlan, Waymo, Lyft, INTERACTION, Argoverse 2) does not contain nuScenes. There is no discussion of how the real frames are aligned with the trajectories used for generation and ground truth.

References:

  • [1] Bo Jiang et al. VAD: Vectorized Scene Representation for Efficient Autonomous Driving. ICCV 2023
Questions for authors

Realistically my concerns are too large to be addressed in a rebuttal - the paper would need to be rewritten in its entirety to address my readability issues alone. With that being said:

  • Can you clarify why the AV stack seems to perform so poorly on real-world scenes?
  • Can you compare your search against the same search without the image-metric surrogate (such as choosing among the five proposals at random), or against a simple grid search over the retained settings (e.g., FRONT_SIGMA_MAX = 60, 65, …, 95) with the same GPU budget, to show that the image metrics are a useful proxy?
  • Can you confirm where the real-world scenes come from? Do you mean nuPlan instead of nuScenes?
Reviewer_VwTDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes an audit–screen–search–confirm workflow for selecting inference-time configurations of frozen driving-video generators for autonomous-driving evaluation. The workflow first measures six image-level defects between condition-aligned real and generated observations, studies their associations with perception, tracking, and planning gaps, and trains an RBF-SVR surrogate to prioritize candidate configurations. A local TPE search then evaluates only the highest-ranked candidate from each round using the full downstream evaluator stack. Finally, selected candidates are compared with the default configuration on held-out scenes using a multi-task acceptance rule.

The method is evaluated on the WorldDreamer renderer and a Cosmos-based pipeline using 26 InterHub scenes. The paper reports one accepted configuration for each pipeline, with composite-gap reductions of 31.24% and 21.02%, respectively, and presents a single-run comparison suggesting lower GPU usage than random search

Critical strengths

Practically relevant problem formulation. The paper correctly distinguishes visual similarity from downstream task fidelity and considers the realistic setting in which both the generator and downstream evaluators remain frozen. Using image defects for candidate prioritization while reserving downstream task metrics for final decisions is a reasonable and practically motivated design.

Explicit separation of screening, search, and confirmation. The scene-disjoint screening/search/confirmation protocol is a meaningful effort to reduce direct selection–confirmation leakage. The authors also appropriately acknowledge that their results do not establish improved closed-loop driving performance.

Critical weaknesses

The statistical uncertainty analysis is insufficient for the claims being made. The confirmation intervals and much of the association analysis use frames as the effective resampling or testing units, although frames from the same scene are temporally correlated and tracking metrics are computed over temporal windows. The paper itself acknowledges that the reported intervals are only nominal frame-bootstrap intervals. However, this is not a peripheral limitation: the reported acceptance of WD-1 and CS-1, as well as the defect–task association claims, directly rely on these intervals and tests. With only eight held-out confirmation scenes, the current evidence does not establish reliable scenario-level generalization.

The surrogate is not validated at the configuration level required by the proposed search. The reported surrogate metrics measure observation-level prediction of composite task gaps from image defects, whereas the actual procedure must rank configurations according to their average downstream gap over a cohort. The paper explicitly notes that its scene-held-out diagnostic does not independently validate configuration-level ranking. There is no complete evaluation of the non-selected candidates from each search round, no configuration-level ranking or regret analysis, and no ablation against simpler alternatives such as random search, grid search, defect-only ranking, or surrogate search without guardrails. Consequently, it is unclear whether the proposed workflow, rather than simple local search or selection effects, is responsible for the reported gains.

Questions for authors

Configuration-level validation: What is the observed configuration-level correlation between surrogate predictions and full downstream task gaps? Can the authors evaluate the four candidates that were not selected in each search round, at least offline, and report top-1 selection accuracy, ranking correlation, or regret?

Generator-specific significance: The implementation appears to apply BH correction to pooled associations while requiring only positive within-generator coefficients, rather than statistically significant associations within each generator. Should the claim that the associations are significant in both pipelines be weakened, or can the authors provide generator-specific uncertainty estimates?

Repeatability margins: How many repeated real–real evaluations and scenes were used to estimate the non-inferiority margins? Why is the HOTA margin exactly 0.0000? Please report the underlying repeatability distribution and explain whether the margin itself has meaningful estimation uncertainty.

Reviewer_KgPhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies inference-time configuration of frozen driving-video generators for reference-based offline evaluation. It measures defects between condition-aligned real and generated observations, screens native controls, ranks candidate configurations with an SVR surrogate, and makes final decisions using held-out perception, tracking, and open-loop planning metric gaps. On 26 InterHub scenes and two generator pipelines, one configuration per pipeline passes the authors’ frame-level multi-task rule.

Critical strengths
  1. The problem is clearly defined and has practical value.
  2. Clear writing and easy to follow
  3. Conducted sufficient experiments on the defined task.
Critical weaknesses
  1. The issue concerns the effective scope of video data. In modular ADS, RGB video enters the system solely as camera input at the perception front-end, and downstream modules have no direct access to the video data. Therefore, the shift in downstream tasks is the result of the shift of perception output propagating through the system. Addressing questions regarding task-level correlations requires analyzing intermediate results. However, this requirement that lies beyond the capabilities of a paradigm limited to manipulating video inputs alone. Downstream module experiments illustrate this issue. BEVFormer extracts representations directly from generated images to produce perception results, thereby reflecting the perception shifts caused by visual inputs. In the case of YOLOv8n+ByteTrack, YOLOv8n first generates detection results from images, which are then temporally associated by ByteTrack; changes in HOTA could stem from detection shifts or from the amplification of these shifts during the association process, but this paper does not make distinction. Similarly, VAD processes images via a BEV encoder, with motion and map representations subsequently influencing the planning Transformer; consequently, variations in planning error cannot be attributed solely to how the planning module itself responds to image imperfections. I am pleased to see that the author is indeed aware of this issue; the text includes several qualifying remarks regarding the interpretation of the metrics. However, these statements do not compensate for the missing intermediate links in the chain of evidence. To substantiate the analysis presented, it would be necessary to deconstruct and examine the components situated between the camera and the target module. But this requirement beyond what this RGB-only endpoint-evaluator paradigm can identify.
  2. The search objective is inconsistent with the final decision rule. Eq5 is based on the predicted average gap across the three tasks, whereas Eq7 requires that the tolerance threshold be met for each individual task.
Questions for authors
  1. Since video affects downstream modules only via perception shifts propagating through the stack, how do you support task-level conclusions without intermediate-result analysis?
  2. Eq. 5 ranks candidates using a predicted composite task gap and defect penalties, whereas Eq. 7 additionally requires each task to satisfy an individual tolerance. Why are task-wise constraints absent from the search objective? How many highly ranked candidates fail the final task-wise check, and would including these constraints change the selected configuration or search efficiency?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.