- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
When using video generators to evaluate autonomous driving systems, one challenge is that the video generator can render visually plausible outputs that still cause perception/planning/tracking algorithms to diverge from how they'd behave in real-world settings. This paper tries to tune a frozen video generator's inference settings such that the AV stack (also frozen) behaves on its renderings in the same manner as it would on real video.
The core idea is to pair real camera frames with generated frames of the same scene, moment, and camera. The same AV stack is run on both and the differences in perception (spotting objects), tracking (following the objects over time), and planning (the distance between where a planner thinks the car should go and where it actually goes) are measured. The generated frames depend on the video model's hyperparameters, and since running the full AV stack on every possible configuration is too expensive, the method instead measures six image-level metrics (L1, PSNR, LPIPS, ProjErr, LPIPS-t, FlowErr) and trains a small model (RBF-SVR) to predict the combined AV gap from them. After determining the metrics that matter, the method runs multiple search rounds. In each round, the small model ranks five videos with different hyperparameter settings, the full AV stack is run on the most promising one, and the setting is only kept if its measured gap improves. Each round's tested setting is then evaluated on held-out validation scenes and accepted if it clearly reduces the overall AV gap without any single task's gap getting worse beyond a certain threshold.
The experiments evaluate their method on DriveArena WorldDreamer and a Cosmos-Transfer1-based pipeline using 26 scenes from InterHub. These scenes are split into 12 screening scenes to train the small image-metric model, 6 search scenes used during the search rounds, and 8 held out scenes. The small model ranks frames decently (rank correlation 0.8338) but predicts the exact AV metric gap only moderately (R^2 0.4999). The overall method proposes 19 candidates to test against the default generator hyperparameters, of which 2 are accepted, 8 are tradeoffs, and 9 are rejected. One accepted setting reduces the composite AV gap by 31.24% relative to WorldDreamer's defaults (mainly by improving tracking) and another reduces the gap by 21.02% relative to Cosmos's defaults. Figure 3 shows an example where tracking briefly loses a car with the WorldDreamer defaults and does not with the method's suggested settings.
Critical strengths
I think the problem setting is interesting and agree that generating pretty visuals does not mean a video generator is well-suited to AV simulation. I also find the authors' problem setting of keeping the generator and AV stack frozen, and portraying the method to be cheap inference-time alignment, to be well-motivated. I found this to be the main benefit of reading the paper.
Critical weaknesses
This submission is extremely dense and difficult to read, with passages such as "we present a reference-based configuration workflow that uses image-defect measurements to prioritize downstream evaluation while keeping both the generator and evaluators frozen" in the abstract and "paired defect vectors and composite task gaps train the ranking surrogate, while residualized associations identify guardrail channels," in Figure 2, Sec 3.2's title "RESIDUALIZED DEFECT–TASK ASSOCIATION", etc. Almost every sentence in the manuscript contains similar unclear jargon, which makes it hard for me to envision the community gaining real benefit from reading this paper and impacts my rating negatively.
I also have big concerns about the experimental protocols and results:
- The validation sample is very small (only 8 held-out scenes), and the reported intervals treat frames from the same scene as independent.
- The AV stack the method uses is questionable and performs poorly on real frames. Sec. 4 reports planner L2 of 9.56–32.93 m on real scenes, whereas the same VAD-Tiny model is reported at 0.78 m average error on nuScenes [1]. Figure A4 shows perception F1 of 0.037 on a real-world scene. This implies the "task-fidelity gap" the paper measures is largely meaningless (comparing differences between two failed outputs).
- There is no real evidence that the image-space metrics are a meaningful proxy for the AV stack performance. The authors themselves note that the surrogate's 0.83 rank correlation "does not independently establish ordering among configurations of one generator" (Sec. 5.4), and the paper has no ablation (Sec. 6). The only baseline is a single 8-frame comparison against random search, where the respective gaps are 0.447 (guided) and 0.453 (random, equal to the default).
- The paper claims that the real references are from nuScenes images that correspond to InterHub trajectories, but InterHub's released data (nuPlan, Waymo, Lyft, INTERACTION, Argoverse 2) does not contain nuScenes. There is no discussion of how the real frames are aligned with the trajectories used for generation and ground truth.
References:
- [1] Bo Jiang et al. VAD: Vectorized Scene Representation for Efficient Autonomous Driving. ICCV 2023
Questions for authors
Realistically my concerns are too large to be addressed in a rebuttal - the paper would need to be rewritten in its entirety to address my readability issues alone. With that being said:
- Can you clarify why the AV stack seems to perform so poorly on real-world scenes?
- Can you compare your search against the same search without the image-metric surrogate (such as choosing among the five proposals at random), or against a simple grid search over the retained settings (e.g., FRONT_SIGMA_MAX = 60, 65, …, 95) with the same GPU budget, to show that the image metrics are a useful proxy?
- Can you confirm where the real-world scenes come from? Do you mean nuPlan instead of nuScenes?