- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces DriveAudit, a benchmark for assessing generated driving scenarios jointly through visual quality, scene consistency, policy response, and testing effectiveness. It contains 550 histories from five sources, including 50 monocular Nexar histories with synthesized additional views. Seven renderers are evaluated with six policies under recorded and generated traffic using repeated-anchor, non-reactive replay. Proposed metrics include trajectory–visual consistency (TVC), traffic realism (TRS), responsibility-aware PDM scoring (RPDMS), and QTES, which multiplies reference-relative quality retention by paired policy degradation.
Critical strengths
-
The paper identifies an important evaluation confound and controls several sources of variation. Renderers share scene anchors, traffic conditions, and policy checkpoints; fixed-RGB state perturbations demonstrate that changing evaluator-side traffic can change the score without changing the policy prediction (Sections 3.3 and 4.3; Appendix C.1). This motivates checking image–state correspondence rather than assuming realistic-looking video is a valid policy test.
-
The benchmark offers broad diagnostic coverage and clearly states its non-reactive scope. The multi-source dataset, seven-renderer comparison, and six-policy panel support analyses beyond a single architecture or image metric (Table 2; Figure 4). Appendices distinguish synthesized views from observations, retain provenance, and explicitly call replay route completion a progress proxy. The TRS corruption and horizon experiments also provide useful diagnostics of geometric overlap and traffic-distribution drift (Figure 5; Appendices B–C).
Critical weaknesses
-
QTES has not been shown to distinguish meaningful policy tests from policy-breaking visual corruption. Multiplicative retention discourages degradation, but it does not provide the prevention claimed in the introduction. For example, under Equations 19–23, a valid case with all four local quality-retention ratios equal to 0.8, TRS = 1, and a policy score reduced to zero has QTES = 100 × 0.8^(3/4) ≈ 84.6, above the unchanged-reference score of 50. Thus moderate measured quality loss can still be rewarded if it breaks the policy. The manuscript does not establish that its quality measurements or validity rules rule out this failure mode. In the reported comparisons, all generators score below the unchanged reference, and QTES falls for every renderer when switching to CAT-K despite larger policy degradation (Table 2; Appendix A.5, Table 3). That is not inherently incorrect, but without independent labels of test usefulness, controlled visual-corruption comparisons, or ranking sensitivity analysis, the central interpretation remains unvalidated.
-
Several foundational measurements and their validation are not reproducible from the supplied description. RPDMS specifies how a responsibility estimate ρ changes the collision gate but does not specify the algorithm that produces ρ, its inputs, or the criteria for valid attribution (Appendix A.4). Figure 7 reports strong ordering agreement, yet the source, sample size, and independence of the ego-/other-dominant labels are not sufficiently described. TVC likewise names support and matching components without enough extraction and normalization detail to reconstruct them (Appendix A.3). These are central quantities, not incidental implementation choices. In addition, the matched view-completion control is described without a numerical result, and two sensor-fusion policies receive null LiDAR without a demonstrated sensitivity control (Appendices B.4 and C.2). Consequently, it is difficult to separate generator effects from metric and input-adaptation effects.
Questions for authors
-
On matched scenarios, does QTES rank independently judged plausible hazards above blur, freeze, visual artifacts, and unchanged-reference controls? Please report sensitivity to the quality exponents and TRS penalty, with source-recording-clustered uncertainty for renderer rankings. This directly tests the principal interpretation of the score.
-
How exactly is ρ computed, and how were the independent responsibility labels in Figure 7 obtained? Please provide the event count, labeling protocol, and estimator details, together with the extractors and normalization rules for all TVC components.
-
What are the numerical results of the matched completion control in Appendix B.4, and do the main rankings persist on native-view data and a camera-only policy subset excluding null-LiDAR adaptations? These controls would clarify whether the benchmark measures generator differences robustly.
10 comments
Detailed Response to Reviewer SLBC
Response 4.1: Interpretation and Empirical Assessment of Quality Weighting
We appreciate the reviewer for raising this excellent and critical point regarding the interpretation of QTES. To address this concern, we first clarify the conditions underlying the numerical example and then examine quality weighting through controlled visual corruption, score decomposition, ablation, parameter sensitivity, and an additional generator comparison. These analyses show that quality retention lowers mean QTES for the evaluated corruptions and changes the preference obtained from policy degradation alone, while the leading generator remains unchanged across the tested quality-weight configurations.
(1) Interpretation of the numerical example
QTES aggregates paired degradation across the evaluated ego-policy panel rather than using the failure of an individual policy. For the policy set , the scenario-level response is
0\leq\mathrm{TE}_{n,a}\leq1.$$ Under the stated quality-retention ratios, $T=1$, and valid evaluation, the calculation $100\times0.8^{3/4}\simeq84.59$ assumes $\mathrm{TE}_n=1$. This requires **all evaluated ego policies to attain $\mathrm{TE}_{n,a}=1$**, with each policy having a reference score at or above the numerical floor and zero mean RPDMS under the generated condition. The example therefore describes maximal degradation across the entire policy panel, rather than the failure of a single policy or an isolated collision. Even if this occurs, credible visual variation that preserves driving semantics and image–state correspondence can still expose meaningful policy weaknesses, extending testing beyond challenges arising solely from traffic interactions. To examine whether visual corruption itself produces additional scoring gains, we therefore conduct the matched interventions below. ### (2) Controlled visual-corruption evidence We apply blur, frame freezing, and local artifacts to matched DreamForge videos under Recorded and CAT-K traffic, holding trajectories, real-video references, policies, and evaluation windows fixed within each condition. Table 4-1 shows lower mean QTES for all three corruptions under both traffic settings. For blur and local artifacts, this reduction occurs despite increased mean policy degradation. These comparisons demonstrate that **quality retention counteracts corruption-induced scoring gains in the evaluated conditions**, rather than rewarding visual degradation solely for impairing policy performance. **Table 4-1. Matched DreamForge corruption controls across completed source recordings. (Appendix A.8.1, Table 17)** | Observation | Mean $\mathrm{TE}$ | Mean $B$ | QTES | $\Delta$QTES | 95% interval for $\Delta$QTES | | --- | --- | --- | --- | --- | --- | | *CAT-K traffic* | | | | | | | Unmodified | 0.5257 | 0.7789 | 38.049 | — | — | | Blur | 0.5364 | 0.6492 | 32.482 | $-5.567$ | $[-6.401,\,-4.656]$ | | Freeze | 0.5241 | 0.7643 | 37.231 | $-0.818$ | $[-1.092,\,-0.547]$ | | Local artifact | 0.5300 | 0.7372 | 36.268 | $-1.781$ | $[-2.437,\,-1.071]$ | | *Recorded traffic* | | | | | | | Unmodified | 0.5040 | 0.7849 | 39.440 | — | — | | Blur | 0.5154 | 0.6566 | 33.893 | $-5.547$ | $[-6.374,\,-4.714]$ | | Freeze | 0.5047 | 0.7743 | 38.977 | $-0.463$ | $[-0.691,\,-0.222]$ | | Local artifact | 0.5133 | 0.7462 | 38.230 | $-1.210$ | $[-2.210,\,-0.135]$ |Detailed Response to Reviewer SLBC
Continued response4.1 to W1&Q1
(3) Quality and policy-response contributions
Component contributions. To explain the cross-condition results, we separate QTES into the quality contribution at a neutral policy response and the additional quality-weighted response increment. Table 4-2 shows that the response increment increases under CAT-K traffic, while reduced quality retention and traffic realism lower the neutral contribution. Their combined effect explains why stronger policy degradation can coexist with lower QTES when the loss of scenario fidelity outweighs the increased response contribution.
Table 4-2. Quality and policy-response contributions to QTES. (Appendix A.5.3, Table 7)
| Renderer | Recorded / | Recorded / | Recorded / QTES | CAT-K / | CAT-K / | CAT-K / QTES |
|---|---|---|---|---|---|---|
| OpenDWM | 43.976 | 0.655 | 44.631 | 40.662 | 3.417 | 44.079 |
| DreamForge | 41.592 | -0.019 | 41.573 | 38.440 | 2.871 | 41.311 |
| HorizonDrive | 39.726 | 0.781 | 40.507 | 36.812 | 3.296 | 40.108 |
| MagicDrive-V2 | 38.803 | 1.065 | 39.868 | 36.106 | 3.403 | 39.509 |
| Panacea | 38.738 | 0.899 | 39.637 | 35.739 | 3.407 | 39.146 |
| BaseDreamer | 38.702 | 0.063 | 38.765 | 35.623 | 2.899 | 38.522 |
| MagicDrive | 36.230 | 1.609 | 37.839 | 33.514 | 3.768 | 37.282 |
Removing quality retention. We remove while retaining the same paired inputs, scoring windows, and remaining score components. Table 4-3 shows that this ablation reverses 15 of 21 pairwise generator orders and changes the leading renderer from OpenDWM to MagicDrive. The comparison demonstrates that quality retention materially changes the generator ordering rather than merely rescaling policy degradation.
Table 4-3. Scores and rankings after removing local quality retention. (Appendix A.8.2, Table 18)
| Models | Full QTES rank | Recorded / | Recorded / | CAT-K / | CAT-K / | Ablated rank |
|---|---|---|---|---|---|---|
| OpenDWM | 1 | 50.740 | 50.740 | 50.977 | 54.205 | 5 |
| DreamForge | 2 | 50.000 | 50.000 | 50.617 | 53.821 | 7 |
| HorizonDrive | 3 | 51.007 | 51.007 | 51.151 | 54.390 | 4 |
| MagicDrive-V2 | 4 | 51.563 | 51.563 | 51.625 | 54.893 | 2 |
| Panacea | 5 | 51.179 | 51.179 | 51.427 | 54.683 | 3 |
| BaseDreamer | 6 | 50.179 | 50.179 | 50.887 | 54.108 | 6 |
| MagicDrive | 7 | 52.935 | 52.935 | 52.962 | 56.315 | 1 |
| Spearman vs. full QTES | — |
(4) Parameter sensitivity and uncertainty
Visual quality and consistency weights. We examine alternative allocations and individual component perturbations while preserving the total local exponent and fixing the TRS contribution. Table 4-4 reports high ranking consistency with the default configuration. Although some lower-ranked positions change, OpenDWM remains first across all 16 tested configurations under both Recorded and CAT-K traffic, supporting the stability of the leading result across alternative quality weights.
Table 4-4. Ranking consistency under alternative local quality exponents. (Appendix A.7.1, Table 10)
| Quantity | Allocations / Default | Allocations / Equal | Allocations / Img.-heavy | Allocations / TVC-heavy | IQA / | IQA / | IQA / | DINO / | DINO / | DINO / | Epi@3 / | Epi@3 / | Epi@3 / | TVC / | TVC / | TVC / |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.167 | 0.188 | 0.200 | 0.100 | 0.094 | 0.225 | 0.273 | 0.188 | 0.150 | 0.136 | 0.188 | 0.150 | 0.136 | 0.200 | 0.143 | 0.125 | |
| 0.167 | 0.188 | 0.200 | 0.100 | 0.188 | 0.150 | 0.136 | 0.094 | 0.225 | 0.273 | 0.188 | 0.150 | 0.136 | 0.200 | 0.143 | 0.125 | |
| 0.167 | 0.188 | 0.200 | 0.100 | 0.188 | 0.150 | 0.136 | 0.188 | 0.150 | 0.136 | 0.094 | 0.225 | 0.273 | 0.200 | 0.143 | 0.125 | |
| 0.250 | 0.188 | 0.150 | 0.450 | 0.281 | 0.225 | 0.205 | 0.281 | 0.225 | 0.205 | 0.281 | 0.225 | 0.205 | 0.150 | 0.321 | 0.375 | |
| , Rec. | 1.000 | 0.893 | 0.857 | 0.929 | 0.964 | 0.964 | 0.893 | 1.000 | 1.000 | 1.000 | 0.964 | 0.893 | 0.893 | 0.857 | 1.000 | 1.000 |
| , CAT-K | 1.000 | 0.893 | 0.857 | 0.929 | 0.964 | 0.893 | 0.893 | 1.000 | 1.000 | 1.000 | 0.929 | 0.893 | 0.893 | 0.857 | 1.000 | 0.964 |
General response to Reviewer SLBC
We thank the reviewer for the rigorous assessment and for recognizing the benchmark’s controlled comparisons and broad diagnostic coverage. Our revisions focus on your concerns about quality weighting, metric reproducibility, and potential input-adaptation effects.
QTES interpretation and empirical evidence. We clarify the policy-panel assumptions underlying the numerical example and provide controlled corruption, decomposition, ablation, and sensitivity analyses. All tested corruptions lower mean QTES, and quality retention reverses the degradation-only preference for the evaluated Cosmos outputs.
Metric reproducibility. We specify responsibility-estimation inputs, aggregation, validity rules, and collision scoring, together with extraction and normalization for all five TVC components. Reference-driver agreement is explicitly distinguished from independent-label accuracy.
Input-adaptation controls. Real versus all-zero LiDAR has model-dependent effects, but excluding both adapted policies preserves the complete generator ordering. Additional Nexar visualizations, manual-review information, and perceptual and temporal diagnostics document the completed views while distinguishing synthesized inputs from recorded observations.
We have incorporated the corresponding revisions into the main manuscript and highlighted the changes in blue. Alongside our point-by-point replies in the OpenReview comments, we have also submitted a response letter as supplementary material to present the additional visualizations and detailed tables in a more readable format. We kindly invite you to consult the supplementary letter alongside our OpenReview replies when reviewing the supporting evidence.