acceptodds
Under review as a conference paper at ICLR 2027

Beyond Visual Realism: Benchmarking Generated Driving Scenarios as Policy Tests

Abstract

End-to-end autonomous vehicles need to be tested on rare, safety-critical situations that real-world driving data collects too slowly and expensively. Generative models offer a scalable fix—synthesizing unbounded, controllable traffic scenarios from recorded histories—but the community still lacks a standard for whether a generated scenario actually works as a driving test. Existing benchmarks assess visual fidelity, temporal consistency, or traffic plausibility, but rarely ask whether it exposes meaningful policy failures, whereas visual corruption naturally lowers policy performance. We introduce **DriveAudit**, a hierarchical benchmark for evaluating generated driving scenarios as policy tests. The multi-source dataset includes 550 scenarios, combining native multi-camera histories with monocular-derived multi-view inputs constructed through generative view completion. Evaluation proceeds through four coupled levels—observation quality, scenario fidelity, policy response, and quality-aware testing effectiveness. Observation quality assesses visual realism and temporal coherence. Scenario fidelity measures image–motion agreement and traffic plausibility through Trajectory–Visual Consistency (TVC) and Traffic Realism Score (TRS). Policy response use Responsibility-aware PDMS (RPDMS), which gates collisions by event-aligned ego attribution, and the headline Quality-aware Testing Effectiveness Score (QTES) multiplies paired policy degradation by reference-relative quality retention, so a visually corrupted scenario cannot be rewarded merely for breaking the policy. Benchmarking seven scenario-generating baselines with six driving policies reveals a mismatch between visual fidelity and testing effectiveness: models that produce more realistic observations do not consistently yield more effective policy tests. DriveAudit thus enable assessing the generated scenarios through joint analysis of visual realism, traffic fidelity, and policy behavior.

Primary area:
datasets and benchmarks
Keywords:
Autonomous Driving, Scenario Generation, Generative Models, World Models, Benchmark, Policy Evaluation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_SLBCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces DriveAudit, a benchmark for assessing generated driving scenarios jointly through visual quality, scene consistency, policy response, and testing effectiveness. It contains 550 histories from five sources, including 50 monocular Nexar histories with synthesized additional views. Seven renderers are evaluated with six policies under recorded and generated traffic using repeated-anchor, non-reactive replay. Proposed metrics include trajectory–visual consistency (TVC), traffic realism (TRS), responsibility-aware PDM scoring (RPDMS), and QTES, which multiplies reference-relative quality retention by paired policy degradation.

Critical strengths
  1. The paper identifies an important evaluation confound and controls several sources of variation. Renderers share scene anchors, traffic conditions, and policy checkpoints; fixed-RGB state perturbations demonstrate that changing evaluator-side traffic can change the score without changing the policy prediction (Sections 3.3 and 4.3; Appendix C.1). This motivates checking image–state correspondence rather than assuming realistic-looking video is a valid policy test.

  2. The benchmark offers broad diagnostic coverage and clearly states its non-reactive scope. The multi-source dataset, seven-renderer comparison, and six-policy panel support analyses beyond a single architecture or image metric (Table 2; Figure 4). Appendices distinguish synthesized views from observations, retain provenance, and explicitly call replay route completion a progress proxy. The TRS corruption and horizon experiments also provide useful diagnostics of geometric overlap and traffic-distribution drift (Figure 5; Appendices B–C).

Critical weaknesses
  1. QTES has not been shown to distinguish meaningful policy tests from policy-breaking visual corruption. Multiplicative retention discourages degradation, but it does not provide the prevention claimed in the introduction. For example, under Equations 19–23, a valid case with all four local quality-retention ratios equal to 0.8, TRS = 1, and a policy score reduced to zero has QTES = 100 × 0.8^(3/4) ≈ 84.6, above the unchanged-reference score of 50. Thus moderate measured quality loss can still be rewarded if it breaks the policy. The manuscript does not establish that its quality measurements or validity rules rule out this failure mode. In the reported comparisons, all generators score below the unchanged reference, and QTES falls for every renderer when switching to CAT-K despite larger policy degradation (Table 2; Appendix A.5, Table 3). That is not inherently incorrect, but without independent labels of test usefulness, controlled visual-corruption comparisons, or ranking sensitivity analysis, the central interpretation remains unvalidated.

  2. Several foundational measurements and their validation are not reproducible from the supplied description. RPDMS specifies how a responsibility estimate ρ changes the collision gate but does not specify the algorithm that produces ρ, its inputs, or the criteria for valid attribution (Appendix A.4). Figure 7 reports strong ordering agreement, yet the source, sample size, and independence of the ego-/other-dominant labels are not sufficiently described. TVC likewise names support and matching components without enough extraction and normalization detail to reconstruct them (Appendix A.3). These are central quantities, not incidental implementation choices. In addition, the matched view-completion control is described without a numerical result, and two sensor-fusion policies receive null LiDAR without a demonstrated sensitivity control (Appendices B.4 and C.2). Consequently, it is difficult to separate generator effects from metric and input-adaptation effects.

Questions for authors
  1. On matched scenarios, does QTES rank independently judged plausible hazards above blur, freeze, visual artifacts, and unchanged-reference controls? Please report sensitivity to the quality exponents and TRS penalty, with source-recording-clustered uncertainty for renderer rankings. This directly tests the principal interpretation of the score.

  2. How exactly is ρ computed, and how were the independent responsibility labels in Figure 7 obtained? Please provide the event count, labeling protocol, and estimator details, together with the extractors and normalization rules for all TVC components.

  3. What are the numerical results of the matched completion control in Appendix B.4, and do the main rankings persist on native-view data and a camera-only policy subset excluding null-LiDAR adaptations? These controls would clarify whether the benchmark measures generator differences robustly.

10 comments
AuthorsOpenReview ↗

General response to Reviewer SLBC

We thank the reviewer for the rigorous assessment and for recognizing the benchmark’s controlled comparisons and broad diagnostic coverage. Our revisions focus on your concerns about quality weighting, metric reproducibility, and potential input-adaptation effects.

  • QTES interpretation and empirical evidence. We clarify the policy-panel assumptions underlying the numerical example and provide controlled corruption, decomposition, ablation, and sensitivity analyses. All tested corruptions lower mean QTES, and quality retention reverses the degradation-only preference for the evaluated Cosmos outputs.

  • Metric reproducibility. We specify responsibility-estimation inputs, aggregation, validity rules, and collision scoring, together with extraction and normalization for all five TVC components. Reference-driver agreement is explicitly distinguished from independent-label accuracy.

  • Input-adaptation controls. Real versus all-zero LiDAR has model-dependent effects, but excluding both adapted policies preserves the complete generator ordering. Additional Nexar visualizations, manual-review information, and perceptual and temporal diagnostics document the completed views while distinguishing synthesized inputs from recorded observations.

We have incorporated the corresponding revisions into the main manuscript and highlighted the changes in blue. Alongside our point-by-point replies in the OpenReview comments, we have also submitted a response letter as supplementary material to present the additional visualizations and detailed tables in a more readable format. We kindly invite you to consult the supplementary letter alongside our OpenReview replies when reviewing the supporting evidence.

AuthorsOpenReview ↗

Detailed Response to Reviewer SLBC

Response 4.1: Interpretation and Empirical Assessment of Quality Weighting

We appreciate the reviewer for raising this excellent and critical point regarding the interpretation of QTES. To address this concern, we first clarify the conditions underlying the numerical example and then examine quality weighting through controlled visual corruption, score decomposition, ablation, parameter sensitivity, and an additional generator comparison. These analyses show that quality retention lowers mean QTES for the evaluated corruptions and changes the preference obtained from policy degradation alone, while the leading generator remains unchanged across the tested quality-weight configurations.

(1) Interpretation of the numerical example

QTES aggregates paired degradation across the evaluated ego-policy panel rather than using the failure of an individual policy. For the policy set A\mathcal A, the scenario-level response is

0\leq\mathrm{TE}_{n,a}\leq1.$$ Under the stated quality-retention ratios, $T=1$, and valid evaluation, the calculation $100\times0.8^{3/4}\simeq84.59$ assumes $\mathrm{TE}_n=1$. This requires **all evaluated ego policies to attain $\mathrm{TE}_{n,a}=1$**, with each policy having a reference score at or above the numerical floor and zero mean RPDMS under the generated condition. The example therefore describes maximal degradation across the entire policy panel, rather than the failure of a single policy or an isolated collision. Even if this occurs, credible visual variation that preserves driving semantics and image–state correspondence can still expose meaningful policy weaknesses, extending testing beyond challenges arising solely from traffic interactions. To examine whether visual corruption itself produces additional scoring gains, we therefore conduct the matched interventions below. ### (2) Controlled visual-corruption evidence We apply blur, frame freezing, and local artifacts to matched DreamForge videos under Recorded and CAT-K traffic, holding trajectories, real-video references, policies, and evaluation windows fixed within each condition. Table 4-1 shows lower mean QTES for all three corruptions under both traffic settings. For blur and local artifacts, this reduction occurs despite increased mean policy degradation. These comparisons demonstrate that **quality retention counteracts corruption-induced scoring gains in the evaluated conditions**, rather than rewarding visual degradation solely for impairing policy performance. **Table 4-1. Matched DreamForge corruption controls across completed source recordings. (Appendix A.8.1, Table 17)** | Observation | Mean $\mathrm{TE}$ | Mean $B$ | QTES | $\Delta$QTES | 95% interval for $\Delta$QTES | | --- | --- | --- | --- | --- | --- | | *CAT-K traffic* | | | | | | | Unmodified | 0.5257 | 0.7789 | 38.049 | — | — | | Blur | 0.5364 | 0.6492 | 32.482 | $-5.567$ | $[-6.401,\,-4.656]$ | | Freeze | 0.5241 | 0.7643 | 37.231 | $-0.818$ | $[-1.092,\,-0.547]$ | | Local artifact | 0.5300 | 0.7372 | 36.268 | $-1.781$ | $[-2.437,\,-1.071]$ | | *Recorded traffic* | | | | | | | Unmodified | 0.5040 | 0.7849 | 39.440 | — | — | | Blur | 0.5154 | 0.6566 | 33.893 | $-5.547$ | $[-6.374,\,-4.714]$ | | Freeze | 0.5047 | 0.7743 | 38.977 | $-0.463$ | $[-0.691,\,-0.222]$ | | Local artifact | 0.5133 | 0.7462 | 38.230 | $-1.210$ | $[-2.210,\,-0.135]$ |
AuthorsOpenReview ↗

Detailed Response to Reviewer SLBC

Continued response4.1 to W1&Q1

(3) Quality and policy-response contributions

Component contributions. To explain the cross-condition results, we separate QTES into the quality contribution at a neutral policy response and the additional quality-weighted response increment. Table 4-2 shows that the response increment increases under CAT-K traffic, while reduced quality retention and traffic realism lower the neutral contribution. Their combined effect explains why stronger policy degradation can coexist with lower QTES when the loss of scenario fidelity outweighs the increased response contribution.

Table 4-2. Quality and policy-response contributions to QTES. (Appendix A.5.3, Table 7)

RendererRecorded / QneutralQ_{\mathrm{neutral}}Recorded / Δresp\Delta_{\mathrm{resp}}Recorded / QTESCAT-K / QneutralQ_{\mathrm{neutral}}CAT-K / Δresp\Delta_{\mathrm{resp}}CAT-K / QTES
OpenDWM43.9760.65544.63140.6623.41744.079
DreamForge41.592-0.01941.57338.4402.87141.311
HorizonDrive39.7260.78140.50736.8123.29640.108
MagicDrive-V238.8031.06539.86836.1063.40339.509
Panacea38.7380.89939.63735.7393.40739.146
BaseDreamer38.7020.06338.76535.6232.89938.522
MagicDrive36.2301.60937.83933.5143.76837.282

Removing quality retention. We remove BnB_n while retaining the same paired inputs, scoring windows, and remaining score components. Table 4-3 shows that this ablation reverses 15 of 21 pairwise generator orders and changes the leading renderer from OpenDWM to MagicDrive. The comparison demonstrates that quality retention materially changes the generator ordering rather than merely rescaling policy degradation.

Table 4-3. Scores and rankings after removing local quality retention. (Appendix A.8.2, Table 18)

ModelsFull QTES rankRecorded / Qw/o BQ_{\mathrm{w/o}\,B}Recorded / QTEQ_{\mathrm{TE}}CAT-K / Qw/o BQ_{\mathrm{w/o}\,B}CAT-K / QTEQ_{\mathrm{TE}}Ablated rank
OpenDWM150.74050.74050.97754.2055
DreamForge250.00050.00050.61753.8217
HorizonDrive351.00751.00751.15154.3904
MagicDrive-V2451.56351.56351.62554.8932
Panacea551.17951.17951.42754.6833
BaseDreamer650.17950.17950.88754.1086
MagicDrive752.93552.93552.96256.3151
Spearman rsr_s vs. full QTES−0.536-0.536−0.536-0.536—

(4) Parameter sensitivity and uncertainty

Visual quality and consistency weights. We examine alternative allocations and individual component perturbations while preserving the total local exponent and fixing the TRS contribution. Table 4-4 reports high ranking consistency with the default configuration. Although some lower-ranked positions change, OpenDWM remains first across all 16 tested configurations under both Recorded and CAT-K traffic, supporting the stability of the leading result across alternative quality weights.

Table 4-4. Ranking consistency under alternative local quality exponents. (Appendix A.7.1, Table 10)

QuantityAllocations / DefaultAllocations / EqualAllocations / Img.-heavyAllocations / TVC-heavyIQA / ×0.5\times0.5IQA / ×1.5\times1.5IQA / ×2.0\times2.0DINO / ×0.5\times0.5DINO / ×1.5\times1.5DINO / ×2.0\times2.0Epi@3 / ×0.5\times0.5Epi@3 / ×1.5\times1.5Epi@3 / ×2.0\times2.0TVC / ×0.5\times0.5TVC / ×1.5\times1.5TVC / ×2.0\times2.0
wIw_I0.1670.1880.2000.1000.0940.2250.2730.1880.1500.1360.1880.1500.1360.2000.1430.125
wDw_D0.1670.1880.2000.1000.1880.1500.1360.0940.2250.2730.1880.1500.1360.2000.1430.125
wEw_E0.1670.1880.2000.1000.1880.1500.1360.1880.1500.1360.0940.2250.2730.2000.1430.125
wTw_T0.2500.1880.1500.4500.2810.2250.2050.2810.2250.2050.2810.2250.2050.1500.3210.375
rsr_s, Rec.1.0000.8930.8570.9290.9640.9640.8931.0001.0001.0000.9640.8930.8930.8571.0001.000
rsr_s, CAT-K1.0000.8930.8570.9290.9640.8930.8931.0001.0001.0000.9290.8930.8930.8571.0000.964
Reviewer_VZsMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DriveAudit, a benchmark for evaluating generated driving scenarios as tests of driving policies. It combines 550 scenario histories from multiple sources with a hierarchical evaluation framework covering observation quality, scenario fidelity, policy response, and testing effectiveness. The proposed metrics include trajectory–visual consistency, traffic realism, responsibility-aware policy scoring, and QTES to combine paired policy degradation with retained scenario quality. Experiments compare seven visual generators and six driving policies under recorded and generated traffic using repeated-anchor non-reactive replay. The results illustrate differences between visual-quality rankings and the proposed assessment of testing effectiveness.

Critical strengths
  1. The paper asks the right question. A generated scene that makes a policy fail is not automatically a good test: the failure may come from rendering artifacts, or from scoring the plan against traffic states that do not match what the policy actually saw. Most existing generation benchmarks do not separate these cases. Here, TVC checks the rendered views against the submitted traffic states, and the fixed-RGB control in Sec. 4.3 makes the point concrete: with the video and DrivoR's predictions held fixed, perturbing only the evaluator-side states already changes RPDMS. The multiplicative quality gate in QTES (Eq. 5) follows the same logic, so a renderer cannot gain credit just by producing corrupted frames. Reporting the four levels separately in Table 2 also shows where a given renderer falls short, instead of hiding it in one aggregate score.
  2. The experimental design is well controlled. All renderers receive the same frozen CAT-K traffic pool and the same scoring anchors, so differences within each block of Table 2 come from rendering, while the recorded vs. generated blocks separate rendering effects from traffic effects. Evaluating six policies on the same grid also pays off: Fig. 4 shows that generated traffic shifts RPDMS and collision rate quite differently across policies, which a single-policy evaluation would miss.
Critical weaknesses
  1. The responsibility estimate ρi\rho_i is central to RPDMS but never actually defined. I could not find what model or rules produce it or what inputs it uses. The validation has the same gap. Since ρi\rho_i enters RPDMS directly and QTES through TEnTE_n, one of the benchmark's main scores currently cannot be reproduced or checked.
  2. QTES is presented as a measure of testing effectiveness, but in practice it mostly reflects scene quality. Table 3 makes this visible: the paired-response term Δresp\Delta resp ranges from about −0.02 to +1.6 under recorded motion and +2.9 to +3.8 under CAT-K, while total QTES spans roughly 37–45. The renderer ranking is therefore set almost entirely by the quality-retention term. The reported mismatch between visual realism and testing effectiveness thus looks more like a disagreement among quality metrics.
Questions for authors
  1. How is ρi\rho_i computed? Please describe the model or rules, its inputs, and other details.
  2. How does the renderer ranking change if QTES is computed without B_n, or under a few alternative exponent settings? This would show how much of the ranking is actually driven by policy response.
  3. The conclusion leaves reactive evaluation to future work. Could a cheaper step be conducted, such as re-rendering from a few anchors where the ego plan diverges most?
9 comments
AuthorsOpenReview ↗

General response to Reviewer VZsM

We thank the reviewer for the detailed comments and for recognizing the importance of image–state consistency and the controlled experimental design. Your questions helped us clarify responsibility computation, the contributions of quality and policy response to QTES, and the effects of policy-driven re-rendering.

  • Explicit responsibility computation. We describe the behavioral evidence, reference-driver fusion, validity conditions, and event-to-window aggregation used to compute ego responsibility and the resulting collision penalty.

  • Quality and policy-response contributions. Exact decomposition makes their respective roles explicit. Quality retention primarily determines the current aggregate ordering, and removing it reverses 15 of 21 pairwise generator orders. In contrast, alternative quality weights retain OpenDWM in first place.

  • Sequential re-rendering. Following your suggestion, we add multi-policy closed-loop evaluation on 20 matched scenarios. Ego-state feedback lowers RPDMS and route completion while preserving 20 of 21 pairwise QTES orders, with a Spearman correlation of rs=0.964r_s=0.964.

We have incorporated the corresponding revisions into the main manuscript and highlighted the changes in blue. Alongside our point-by-point replies in the OpenReview comments, we have also submitted a response letter as supplementary material to present the additional visualizations and detailed tables in a more readable format. We kindly invite you to consult the supplementary letter alongside our OpenReview replies when reviewing the supporting evidence.

AuthorsOpenReview ↗

Detailed Response to Reviewer VZsM

Response 3.1: Computation of Ego Responsibility and Collision Scoring

We apologize for the incomplete definition of ρi\rho_i in the original submission. Responsibility is estimated from explicit behavioral rules and reference-driver checks rather than a separately trained predictor. The procedure scores both collision participants, normalizes the ego score by their combined evidence, and converts the resulting attribution into a window-level collision penalty. We describe (1) behavioral evidence and responsibility estimation, (2) reference-driver checks and evidence fusion, and (3) responsibility-aware collision scoring below. The accompanying comparisons show high consistency with model-derived reference evidence and stable aggregate policy rankings under the tested fusion coefficients.

(1) Behavioral evidence and responsibility estimation

For each collision, we assess both participants using synchronized pre-collision trajectories, headings, participant types and dimensions, relative geometry, and observable traffic rules. These inputs support four evidence components covering rule violations, interaction attribution, avoidability, and pre-conflict maneuvers. Table 3-1 specifies their weights and processing settings. For participant aa in event ee, let ea,e,ke_{a,e,k} denote the component score and Ka,e\mathcal K_{a,e} the set of components supported by valid observations. The participant score and event-level ego-responsibility ratio are

\rho_e=\frac{S_{\mathrm{ego},e}}{S_{\mathrm{ego},e}+S_{\mathrm{other},e}}.$$ Weights are normalized over the available evidence. Attribution is valid only when both participant scores are finite and their sum exceeds the numerical tolerance. Otherwise, it is marked unavailable. Thus, $\rho_e$ expresses the ego vehicle’s share of the combined behavioral evidence, with larger values indicating greater estimated ego responsibility. **Table 3-1. Responsibility evidence and processing settings. (Appendix A.4, Table 6)** | **Component** | **Weight** | **Evidence used** | | --- | --- | --- | | **(a) Responsibility evidence** | | | | Rule violations | 0.30 | One minus normalized compliance with observable traffic rules. | | Interaction attribution | 0.30 | Relative motion and trajectory intrusion in rear-end, cut-in, crossing, and oncoming conflicts. | | Avoidability | 0.25 | Warning and reaction times, braking demand and response, and eligible reference-driver evidence. | | Pre-conflict maneuvers | 0.15 | Hazardous pre-collision acceleration, lateral motion, jerk, and turning. | | **(b) Processing settings** | | | | Behavioral history | | Latest contiguous valid joint observations within 3 s before collision. | | Observation support | | $\geq 2$ synchronized timestamps; positive duration; gaps $\leq 0.5$ s or the stricter native-track limit. | | History diagnostic | | The 0.5 s history-duration threshold is diagnostic, not a uniform exclusion rule. | | Kinematic thresholds | | Time-to-collision warning: 3 s; reaction: 0.75 s; reference deceleration: $4\,\mathrm{m/s^2}$. | | Reference replay | | FSM, CC-JP, RSS; history: 5 s target, 1 s minimum; replay step: 0.1 s. | ### (2) Reference-driver checks and evidence fusion **Avoidability assessment.** We examine alternative braking responses using FSM, CC-JP, and RSS. Each replay replaces only the ego vehicle’s longitudinal response while keeping the other trajectories fixed. Successful avoidance provides evidence of an available braking opportunity under the corresponding reference model. Table 3-2 summarizes the models and their agreement diagnostics. The results show **high consistency between responsibility ordering and model-derived reference evidence, with matching reference judgments on jointly resolved cases**. **Table 3-2. Reference driving models and agreement diagnostics. (Appendix A.6, Table 8)** | Model | Safety criterion | Braking strategy | Ordering | 95% CI | | --- | --- | --- | --- | --- | | *Reference models and responsibility-ordering agreement* | | | | | | FSM | Fuzzy Safety Model: proximity and conflict-criticality assessment | Reaction-delayed, graded braking with jerk limits | 0.978 | 0.890–1.000 | | CC-JP | Japanese Careful and Competent Driver: lane-aware proximity and TTC constraints | Pedal release followed by progressive braking | 0.976 | 0.897–1.000 | | RSS | Responsibility-Sensitive Safety: longitudinal and lateral safe-distance constraints | Reaction-delayed, jerk-limited braking | 0.973 | 0.896–1.000 | | *Reference-model pair* | | | Overall agreement | Resolved agreement | | FSM–RSS | | | 71.8% | 100% | | FSM–CC-JP | | | 73.8% | 100% | | RSS–CC-JP | | | 88.3% | 100% |
AuthorsOpenReview ↗

Detailed Response to Reviewer VZsM

Continued response3.1 to W1&Q1

Fusion and sensitivity. For records eligible for reference-evidence fusion, let vv denote the fraction of valid reference-driver votes indicating successful avoidance. This evidence is combined with kinematic avoidability as eavoid∗=(1−α)eavoidkin+αve_{\mathrm{avoid}}^{*}=(1-\alpha)e_{\mathrm{avoid}}^{\mathrm{kin}}+\alpha v, with α=0.5\alpha=0.5 by default. When only one evidence source is available, that source is retained. To assess dependence on this choice, we vary α\alpha while holding the evidence inputs fixed. Table 3-3 reports small changes in aggregate RPDMS, while the policy ordering remains unchanged across all tested fusion coefficients.

Table 3-3. Sensitivity to reference-evidence fusion; changes relative to α=0.5\alpha=0.5. (Appendix A.7.3, Table 13)

Quantityα=0\alpha=0α=0.25\alpha=0.25α=0.50\alpha=0.50α=0.75\alpha=0.75α=1.00\alpha=1.00
Mean ∣Δρ∣\vert \Delta\rho\vert0.02140.00940.00000.00820.0159
95th percentile ∣Δρ∣\vert \Delta\rho\vert0.08620.03900.00000.03310.0613
Max ∣Δρ∣\vert \Delta\rho\vert0.88010.17030.00000.16150.4613
Max policy ∣ΔRPDMS∣\vert \Delta\mathrm{RPDMS}\vert0.0004080.0001850.0000000.0001520.000280

(3) Responsibility-aware collision scoring

To incorporate event-level attribution into planning-window scoring, each estimate is matched to the evaluated scenario, policy, planning anchor, and collision participants. For a window ii containing multiple collisions, ρi\rho_i is the largest event-level ego responsibility when all estimates are valid. The collision gate RNCi\mathrm{RNC}_i is 11 for collision-free windows, 1−ρi1-\rho_i for windows with valid collision attribution, and 00 when any collision attribution is unavailable. RPDMS combines this gate with the unchanged non-collision planning factor:

B_i=s_i^{\mathrm{dac}}\frac{5s_i^{\mathrm{ttc}}+2s_i^{\mathrm{comf}}+5s_i^{\mathrm{prog}}}{12}.$$ Table 3-4 compares binary and responsibility-aware gating while holding driving outcomes, evaluation windows, and non-collision components fixed. The positive score adjustments across all six policies demonstrate that **responsibility-aware gating differentiates collisions according to estimated ego contribution while preserving the remaining planning-score structure**. **Table 3-4. Matched collision-gate comparison across six driving policies. (Appendix A.6, Table 9)** | Quantity | UniAD | VAD | DrivoR | SparseDrive V2 | DiffusionDrive V2 | GoalFlow | | --- | --- | --- | --- | --- | --- | --- | | Binary-gate score | 0.6001 | 0.5642 | 0.9103 | 0.9293 | 0.7496 | 0.8232 | | Responsibility-gate score | 0.6347 | 0.5970 | 0.9267 | 0.9422 | 0.7826 | 0.8427 | | Score change | +0.0346 | +0.0328 | +0.0164 | +0.0128 | +0.0330 | +0.0195 |
Reviewer_iSvROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DriveAudit, a benchmark for evaluating generated driving scenarios as tests of autonomous driving policies. It combines 550 scenarios from native multi-camera recordings and monocular-derived multi-view histories, and evaluates seven visual generators with six driving policies. The framework assesses observation quality, scenario fidelity, policy response, and testing effectiveness. Its main proposed metrics include responsibility-aware planning scores and QTES, which combines paired policy degradation with scene-quality retention under a repeated-anchor, non-reactive replay protocol.

Critical strengths
  1. A useful shift from visual realism to policy-testing value. The paper addresses a meaningful evaluation problem: visually convincing generation does not by itself establish that a scenario is informative for testing a driving policy. Explicitly recognizing that rendering artifacts can also reduce policy performance is a valuable motivation. This gives the benchmark a potentially useful role in diagnosing driving-world generators, even while the interpretation of its headline effectiveness score needs further validation.

  2. A broad diagnostic framework with controlled paired comparisons. The combination of visual quality, cross-view geometry, image–trajectory agreement, traffic plausibility, and downstream policy response offers a more informative picture than visual metrics alone. Separating recorded-motion and generated-motion rendering, and sharing scenarios, traffic candidates, and scoring anchors across seven renderers and six policies, provides a useful basis for reproducible comparisons. These design choices are substantive strengths and make the paper a close case for acceptance.

Critical weaknesses
  1. The interpretation and calibration of testing effectiveness remain the main uncertainty. QTES combines paired policy degradation with quality-retention factors, which is a reasonable attempt to account for generation quality. The current diagnostics, however, do not yet clearly establish how reliably a high score distinguishes a meaningful driving challenge from residual generation artifacts. The weighting choices, reference-relative quality ratios, and traffic-realism penalty may affect this interpretation; responsibility estimation also affects RPDMS. More explicit evidence about sensitivity, ranking stability across policies and data sources, and agreement of responsibility estimates with independent judgments would help assess reliability. These are validation gaps rather than demonstrated failures of the proposed metrics, but they matter because testing effectiveness is the paper's central claim.

  2. The non-reactive protocol supports a narrower conclusion than sustained driving evaluation. Policy outputs do not change subsequent ego observations or traffic states, so the protocol does not capture action-induced distribution shifts, accumulated errors, or recovery. The authors acknowledge this limitation and identify Route Completion as a replay-progress proxy. This is a defensible controlled diagnostic setting, but the connection from its scores to practical tests of sustained driving is not yet sufficiently established. The paper would benefit from more sharply separating the evidence it currently provides from that broader intended use; selected closed-loop comparisons would be a valuable extension rather than a requirement to replace the full benchmark.

Overall assessment

Weak Rejection (borderline). The paper offers a relevant problem formulation and a useful, carefully structured diagnostic benchmark. I view these as meaningful contributions, and my recommendation is close to the acceptance boundary. At present, the remaining uncertainty about the interpretation of QTES and the practical scope of non-reactive replay slightly outweighs those strengths. I therefore lean toward rejection, while remaining open to clarification or existing validation that more convincingly supports the central testing-effectiveness claim.

Questions for authors
  1. Metric interpretation and reliability. Which existing validation most directly supports interpreting high QTES as a meaningful policy test rather than residual generation artifacts? Please distinguish evidence for metric sensitivity from evidence for that interpretation, and summarize any available checks of weight sensitivity, responsibility-estimation accuracy, and ranking stability across policies or data sources. Controlled challenge-versus-corruption comparisons would be useful additional validation if available.

  2. Scope of the replay protocol. Which claims about practical driving tests are supported by the current non-reactive evaluation, and which require future closed-loop validation? If there is already evidence linking replay scores to sustained driving failures, please highlight it. Otherwise, a clearer statement of the demonstrated diagnostic scope would help calibrate the contribution.

7 comments
AuthorsOpenReview ↗

General response to Reviewer iSvR

We thank the reviewer for the thoughtful assessment and for recognizing the value of moving beyond visual realism through controlled, policy-oriented comparisons. Our revisions focus on your central concerns about the interpretation of QTES and the scope of non-reactive evaluation.

  • Evidence for quality weighting. Matched blur, freezing, and local-artifact interventions reduce mean QTES under both traffic conditions. For blur and local artifacts, this occurs despite increased mean policy degradation. The Cosmos comparison provides complementary evidence of the quality penalty operating on naturally occurring generation defects.

  • Reliability of comparisons. The leading generator remains unchanged across the tested quality weights, while leave-one-policy-out and responsibility-parameter comparisons preserve the complete ordering. Source-wise analyses separately characterize dataset-dependent performance.

  • Closed-loop evidence and evaluation scope. We clarify the role of non-reactive testing in standardized leaderboard evaluation and add multi-policy closed-loop tests on 20 matched scenarios. Feedback exposes additional policy degradation while retaining the leading two QTES positions.

We have incorporated the corresponding revisions into the main manuscript and highlighted the changes in blue. Alongside our point-by-point replies in the OpenReview comments, we have also submitted a response letter as supplementary material to present the additional visualizations and detailed tables in a more readable format. We kindly invite you to consult the supplementary letter alongside our OpenReview replies when reviewing the supporting evidence.

AuthorsOpenReview ↗

Detailed Response to Reviewer iSvR

Response 2.1: Interpretation and Reliability of QTES

We appreciate the reviewer’s emphasis on distinguishing meaningful policy challenges from visual artifacts. To address this concern, we examine (1) controlled corruption and naturally occurring generation defects, (2) scoring parameters and responsibility estimation, and (3) policy panels, data sources, and traffic backends. The visual controls directly examine whether degraded observations gain score by impairing policy performance, while the remaining analyses assess the stability of the reported comparisons. The results show that the evaluated visual corruptions reduce mean QTES, even when blur and local artifacts increase mean policy degradation, and that the leading generator comparisons remain stable across the tested quality weights and policy panels.

(1) Controlled corruption and naturally occurring defects

We use DreamForge as a fixed multi-view generation backend for its relatively low inference cost and apply blur, frame freezing, and local artifacts to matched generated videos. Traffic trajectories, real-video references, policies, and evaluation windows remain fixed within each traffic condition, isolating the effect of visual degradation. Table 2-1 shows that all three corruptions reduce mean QTES under both Recorded and CAT-K traffic. For blur and local artifacts, this reduction occurs despite increased mean policy degradation, demonstrating that quality retention counteracts corruption-induced scoring gains in the evaluated conditions.

Table 2-1. Matched DreamForge corruption controls across completed source recordings. (Appendix A.8.1, Table 17)

ObservationMean TE\mathrm{TE}Mean BBQTESΔ\DeltaQTES95% interval for Δ\DeltaQTES
CAT-K traffic
Unmodified0.52570.778938.049——
Blur0.53640.649232.482−5.567-5.567[−6.401, −4.656][-6.401,\,-4.656]
Freeze0.52410.764337.231−0.818-0.818[−1.092, −0.547][-1.092,\,-0.547]
Local artifact0.53000.737236.268−1.781-1.781[−2.437, −1.071][-2.437,\,-1.071]
Recorded traffic
Unmodified0.50400.784939.440——
Blur0.51540.656633.893−5.547-5.547[−6.374, −4.714][-6.374,\,-4.714]
Freeze0.50470.774338.977−0.463-0.463[−0.691, −0.222][-0.691,\,-0.222]
Local artifact0.51330.746238.230−1.210-1.210[−2.210, −0.135][-2.210,\,-0.135]

Naturally occurring generation defects. Figure 8 (Appendix A.8.3) shows comparatively clear front views but blurred non-front views from the evaluated Cosmos Drive Dreams integration. Table 2-2 compares Cosmos and OpenDWM on all scenarios under Recorded traffic. Removing local quality retention ranks Cosmos above OpenDWM, whereas full QTES reverses this ordering. This complements the controlled corruption experiment with defects arising during generation. Since BnB_n combines multiple quality terms, the score difference cannot be attributed solely to blur. Together, these checks support sensitivity to the evaluated visual defects; they do not independently establish that every high-QTES scene is a meaningful driving challenge.

Table 2-2. Quality weighting for an additional generator. (Appendix A.8.3, Table 19)

MethodMean TE\mathrm{TE}Mean BnB_nWithout BnB_nQTES
Recorded reference0.50001.000050.00050.000
OpenDWM0.50730.879550.72844.622
Cosmos Drive Dreams0.51020.702651.02435.814
AuthorsOpenReview ↗

Detailed Response to Reviewer iSvR

Continued response2.1 to W1&Q1

(2) Parameter sensitivity and responsibility estimation

Visual quality and consistency weights. We examine alternative allocations among the components of BnB_n while preserving their total exponent and fixing the TRS contribution. Table 2-3 reports high ranking consistency with the default configuration. Across all tested allocations, OpenDWM remains first under both Recorded and CAT-K traffic, supporting the stability of the leading comparison under alternative quality weights.

Table 2-3. Ranking consistency under alternative local quality exponents. (Appendix A.7.1, Table 10)

QuantityAllocations / DefaultAllocations / EqualAllocations / Img.-heavyAllocations / TVC-heavyIQA / ×0.5\times0.5IQA / ×1.5\times1.5IQA / ×2.0\times2.0DINO / ×0.5\times0.5DINO / ×1.5\times1.5DINO / ×2.0\times2.0Epi@3 / ×0.5\times0.5Epi@3 / ×1.5\times1.5Epi@3 / ×2.0\times2.0TVC / ×0.5\times0.5TVC / ×1.5\times1.5TVC / ×2.0\times2.0
wIw_I0.1670.1880.2000.1000.0940.2250.2730.1880.1500.1360.1880.1500.1360.2000.1430.125
wDw_D0.1670.1880.2000.1000.1880.1500.1360.0940.2250.2730.1880.1500.1360.2000.1430.125
wEw_E0.1670.1880.2000.1000.1880.1500.1360.1880.1500.1360.0940.2250.2730.2000.1430.125
wTw_T0.2500.1880.1500.4500.2810.2250.2050.2810.2250.2050.2810.2250.2050.1500.3210.375
rsr_s, Rec.1.0000.8930.8570.9290.9640.9640.8931.0001.0001.0000.9640.8930.8930.8571.0001.000
rsr_s, CAT-K1.0000.8930.8570.9290.9640.8930.8931.0001.0001.0000.9290.8930.8930.8571.0000.964

TRS parameters and exponent. We vary the penalty parameters (q,η)(q,\eta) and outer exponent γ\gamma while keeping the remaining scoring components fixed. Table 2-4 shows that these choices affect absolute QTES and cross-condition comparisons. Within each traffic condition, however, TRS acts as a shared multiplier, so the complete generator ordering remains unchanged across the tested settings.

Table 2-4. Sensitivity of QTES to TRS parameters and exponent. (Appendix A.7.2, Table 11)

MetricTRS parametersTRS exponent
CAT-K TRS range0.447–0.9770.782
CAT-K QTES range38.332–46.60036.665–46.870
Recorded QTES44.63144.631
Within-condition Spearman rsr_s1.0001.000

Responsibility-aware collision scoring. To isolate the contribution of responsibility attribution, we compare binary and responsibility-aware collision gates under identical driving outcomes, evaluation windows, and non-collision score components. Table 2-5 reports positive score adjustments across all six policies. This comparison shows that responsibility-aware gating differentiates collisions that would otherwise receive the same binary penalty, according to the estimated ego contribution.

Table 2-5. Matched collision-gate comparison across six driving policies. (Appendix A.6, Table 9)

QuantityUniADVADDrivoRSparseDrive V2DiffusionDrive V2GoalFlow
Binary-gate score0.60010.56420.91030.92930.74960.8232
Responsibility-gate score0.63470.59700.92670.94220.78260.8427
Score change+0.0346+0.0328+0.0164+0.0128+0.0330+0.0195
Reviewer_gg8YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DriveAudit is a hierarchical benchmark for evaluating generated autonomous driving scenarios as policy tests, not just for visual realism. Its contributions are: (1) a four-level evaluation framework covering observation quality, scenario fidelity, policy response, and testing effectiveness; (2) new metrics—TVC, TRS, RPDMS, and QTES—for assessing scenario consistency, traffic plausibility, collision responsibility, and meaningful policy degradation; (3) a multi-source benchmark of 550 scenarios. The authors validate it with seven scenario-generation models and six driving policies, plus controlled experiments on geometric inconsistency, traffic realism, and collision responsibility. The authors validate the framework by evaluating seven scenario generation models and six autonomous driving policies under recorded and generated traffic conditions. Additional controlled experiments examine geometric inconsistencies, traffic realism, and collision responsibility attribution. Results show that higher visual fidelity or greater policy degradation does not necessarily mean a more effective driving test, highlighting the need to jointly evaluate scenario quality and policy behavior.

Critical strengths
  1. A timely and meaningful problem formulation. This paper addresses an important but underexplored question: whether generated driving scenarios truly help evaluate driving policies, rather than merely looking realistic. It clearly distinguishes visual quality, scenario realism, and policy-test effectiveness, offering a valuable perspective.

  2. A well-designed, comprehensive benchmark with paired evaluation. The authors build a multi-source benchmark of 550 scenarios and evaluate seven generation methods and six driving policies. Its hierarchical evaluation framework, combined with a paired non-reactive replay protocol, enables systematic analysis of the relationship among visual generation quality, traffic consistency, and policy response. Diagnostic experiments further show that visual realism and test effectiveness are not necessarily aligned.

Critical weaknesses
  1. The design choices and robustness of QTES could be further justified. Although the motivation behind combining policy degradation with scenario-quality retention is reasonable, some weighting factors and exponents in QTES appear to be manually specified. The paper would benefit from a more comprehensive sensitivity analysis to demonstrate that the conclusions and relative rankings of different generation methods remain stable under alternative reasonable configurations. This would strengthen the justification and reliability of the proposed metric.

  2. RPDMS validation and protocol generality could be strengthened. More implementation details and independent validation of ego-responsibility estimation are needed. The main comparison also relies on a shared CAT-K backend, with SMART and VBD used mainly in traffic-realism analysis; broader traffic-generation methods and reactive closed-loop settings would improve applicability. These limitations do not invalidate the benchmark but narrow the scope of the conclusions.

Questions for authors
  1. Could the authors provide sensitivity analyses to demonstrate that the QTES rankings remain stable under different reasonable choices of weighting factors and exponents?

  2. RPDMS Validation. How is the ego-responsibility ratio estimated in practice, and is its reliability validated against independent ground-truth annotations?

8 comments
AuthorsOpenReview ↗

General response to Reviewer gg8Y

We thank the reviewer for the constructive feedback and for recognizing the importance of evaluating generated scenarios as policy tests and the value of our hierarchical, paired evaluation framework. Your comments motivated us to strengthen the justification of QTES, clarify responsibility estimation, and broaden the evaluation.

  • Parameter robustness. OpenDWM remains first across all 16 tested quality-weight configurations, while leave-one-policy-out evaluation preserves the complete generator ordering. Responsibility-weight variations change QTES by less than 0.100.10 points without altering generator rankings.

  • Responsibility estimation. We specify the behavioral inputs, evidence aggregation, reference-driver checks, validity criteria, and collision-gating procedure. The additional diagnostics show high reference-based consistency and stable aggregate comparisons under the tested weight and fusion variations.

  • Broader evaluation. SMART-based testing retains OpenDWM as the leading generator. Multi-policy closed-loop evaluation on 20 matched scenarios reveals additional policy degradation while preserving the two leading QTES positions.

We have incorporated the corresponding revisions into the main manuscript and highlighted the changes in blue. Alongside our point-by-point replies in the OpenReview comments, we have also submitted a response letter as supplementary material to present the additional visualizations and detailed tables in a more readable format. We kindly invite you to consult the supplementary letter alongside our OpenReview replies when reviewing the supporting evidence.

AuthorsOpenReview ↗

Detailed Response to Reviewer gg8Y

Response 1.1: QTES Parameter Choices and Ranking Stability

We agree that the default QTES parameters require clearer justification. The default allocation balances image-derived fidelity with traffic-related consistency and realism. Specifically, CLIP-IQA+, DINO-T, and Epi@3 jointly receive a total exponent of 1/21/2, equally divided among perceptual, temporal, and cross-view evidence. The remaining 1/21/2 is divided equally between TVC and population-level TRS. This grouping assigns equal total exponents to the two types of evidence rather than determining their relative weighting solely by the number of metrics. To assess the dependence of the reported comparisons on these choices, we examine (1) alternative visual quality weights, (2) TRS parameters and exponent, and (3) policy-panel composition, using matched scenarios and scoring windows. The results show that the leading QTES result is stable across the tested quality weights, and leave-one-policy-out evaluation preserves the complete generator ordering.

(1) Visual quality and consistency weights

We evaluate equal and non-uniform allocations and individual weight perturbations, preserving the total local exponent and fixing the TRS exponent. Table 1-1 shows high ranking consistency with the default configuration. The leading result therefore does not depend on the specific default allocation, although lower positions can change.

Table 1-1. Ranking consistency under alternative local quality exponents. (Appendix A.7.1, Table 10)

QuantityAllocations / DefaultAllocations / EqualAllocations / Img.-heavyAllocations / TVC-heavyIQA / ×0.5\times0.5IQA / ×1.5\times1.5IQA / ×2.0\times2.0DINO / ×0.5\times0.5DINO / ×1.5\times1.5DINO / ×2.0\times2.0Epi@3 / ×0.5\times0.5Epi@3 / ×1.5\times1.5Epi@3 / ×2.0\times2.0TVC / ×0.5\times0.5TVC / ×1.5\times1.5TVC / ×2.0\times2.0
wIw_I0.1670.1880.2000.1000.0940.2250.2730.1880.1500.1360.1880.1500.1360.2000.1430.125
wDw_D0.1670.1880.2000.1000.1880.1500.1360.0940.2250.2730.1880.1500.1360.2000.1430.125
wEw_E0.1670.1880.2000.1000.1880.1500.1360.1880.1500.1360.0940.2250.2730.2000.1430.125
wTw_T0.2500.1880.1500.4500.2810.2250.2050.2810.2250.2050.2810.2250.2050.1500.3210.375
rsr_s, Rec.1.0000.8930.8570.9290.9640.9640.8931.0001.0001.0000.9640.8930.8930.8571.0001.000
rsr_s, CAT-K1.0000.8930.8570.9290.9640.8930.8931.0001.0001.0000.9290.8930.8930.8571.0000.964

(2) TRS parameters and exponent

We vary the penalty parameters (q,η)(q,\eta) and outer exponent γ\gamma while holding the other scoring components fixed. The score ranges in Table 1-2 show that these choices affect absolute QTES and comparisons between traffic conditions. Because TRS acts as a shared multiplier within each condition, however, the complete generator ordering remains unchanged across the tested configurations, including OpenDWM, DreamForge, and HorizonDrive in first, second, and third place, respectively.

Table 1-2. Sensitivity of QTES to TRS parameters and exponent. (Appendix A.7.2, Table 11)

MetricTRS parametersTRS exponent
CAT-K TRS range0.447–0.9770.782
CAT-K QTES range38.332–46.60036.665–46.870
Recorded QTES44.63144.631
Within-condition Spearman rsr_s1.0001.000

(3) Policy-panel composition

To examine dependence on policy selection, we recompute QTES by excluding each policy in turn and by evaluating each policy individually on the same scenarios and scoring windows. As reported in Table 1-3, every leave-one-policy-out evaluation preserves the complete generator ordering, while all single-policy evaluations retain OpenDWM and DreamForge in first and second place. Together, these comparisons show that the leading results remain consistent across the tested policy panels and do not depend on retaining any one policy.

Table 1-3. Ranking consistency under alternative policy-panel compositions. (Appendix A.7.4, Table 15)

Policy-panel configurationConfigurationsRecorded rsr_sCAT-K rsr_sTop-ranked generator in both conditions
Leave one policy out61.0001.000OpenDWM
Use each policy individually60.893–1.0000.893–1.000OpenDWM
AuthorsOpenReview ↗

Detailed Response to Reviewer gg8Y

Response 1.2: RPDMS Responsibility Estimation and Sensitivity

We apologize for the incomplete description of responsibility estimation in the original submission. The estimator combines explicit behavioral evidence with reference-driver checks to assess the relative contributions of both collision participants, without relying on a separate learned predictor. To clarify its computation and influence on evaluation, we describe (1) evidence aggregation, (2) reference-driver assessment, (3) matched collision-gate comparisons, and (4) parameter sensitivity. Together, these analyses show high consistency with model-derived reference evidence and stable aggregate comparisons under the tested evidence-weight and fusion variations.

(1) Behavioral evidence and responsibility estimation

For each collision, we assess both participants using synchronized pre-collision trajectories, headings, types, dimensions, relative geometry, and observable traffic rules. Table 1-4 summarizes the evidence components, weights, and processing settings. For participant aa, let ea,ke_{a,k} denote the component score and Ka\mathcal K_a the set of components supported by valid observations. The participant score and ego-responsibility ratio are

\rho_{\mathrm{ego}}=\frac{S_{\mathrm{ego}}}{S_{\mathrm{ego}}+S_{\mathrm{other}}}.$$ Weights are normalized over the available evidence. Attribution is valid only when both participant scores are finite and their sum exceeds the numerical tolerance. Otherwise, it remains unavailable. The ratio represents relative behavioral attribution rather than calibrated legal liability. **Table 1-4. Responsibility evidence and processing settings. (Appendix A.4, Table 6)** | **Component** | **Weight** | **Evidence used** | | --- | --- | --- | | **(a) Responsibility evidence** | | | | Rule violations | 0.30 | One minus normalized compliance with observable traffic rules. | | Interaction attribution | 0.30 | Relative motion and trajectory intrusion in rear-end, cut-in, crossing, and oncoming conflicts. | | Avoidability | 0.25 | Warning and reaction times, braking demand and response, and eligible reference-driver evidence. | | Pre-conflict maneuvers | 0.15 | Hazardous pre-collision acceleration, lateral motion, jerk, and turning. | | **(b) Processing settings** | | | | Behavioral history | | Latest contiguous valid joint observations within 3 s before collision. | | Observation support | | $\geq 2$ synchronized timestamps; positive duration; gaps $\leq 0.5$ s or the stricter native-track limit. | | History diagnostic | | The 0.5 s history-duration threshold is diagnostic, not a uniform exclusion rule. | | Kinematic thresholds | | Time-to-collision warning: 3 s; reaction: 0.75 s; reference deceleration: $4\,\mathrm{m/s^2}$. | | Reference replay | | FSM, CC-JP, RSS; history: 5 s target, 1 s minimum; replay step: 0.1 s. | ### (2) Reference-driver-based avoidability assessment Similar to CARS, we assess collision avoidability by replacing only the ego vehicle’s longitudinal response while keeping other trajectories fixed. Table 1-5 summarizes FSM, CC-JP, and RSS and their agreement diagnostics. Responsibility ordering is highly consistent with the reference evidence, and judgments coincide on jointly resolved cases. Overall agreement also reflects model applicability and unresolved outcomes. **Table 1-5. Reference driving models and agreement diagnostics. (Appendix A.6, Table 8)** | Model | Safety criterion | Braking strategy | Ordering | 95% CI | | --- | --- | --- | --- | --- | | *Reference models and responsibility-ordering agreement* | | | | | | FSM | Fuzzy Safety Model: proximity and conflict-criticality assessment | Reaction-delayed, graded braking with jerk limits | 0.978 | 0.890–1.000 | | CC-JP | Japanese Careful and Competent Driver: lane-aware proximity and TTC constraints | Pedal release followed by progressive braking | 0.976 | 0.897–1.000 | | RSS | Responsibility-Sensitive Safety: longitudinal and lateral safe-distance constraints | Reaction-delayed, jerk-limited braking | 0.973 | 0.896–1.000 | | *Reference-model pair* | | | Overall agreement | Resolved agreement | | FSM–RSS | | | 71.8% | 100% | | FSM–CC-JP | | | 73.8% | 100% | | RSS–CC-JP | | | 88.3% | 100% |

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers (gg8Y, iSvR, VZsM, and SLBC) for their constructive feedback. DriveAudit evaluates generated driving scenarios through observation quality, scenario fidelity, policy response, and quality-aware testing effectiveness (QTES). We have summarized the main clarifications and implemented additional experiments below for addressing the reviewers’ main comments.

  • QTES interpretation and quality controls (all reviewers). We separate quality and policy-response contributions through exact decomposition and ablation. All tested corruptions reduce mean QTES under both traffic conditions, including blur and local artifacts that increase mean policy degradation. Quality retention also reverses the degradation-only preference for Cosmos Drive Dreams over OpenDWM. Removing the quality term reverses 15 of 21 generator pairs, confirming its substantive role in the reported ordering.

  • Scoring robustness and ranking stability (all reviewers). Leading rankings are robust to the tested scoring and policy choices. OpenDWM remains first across 16 quality-weight configurations, and single-policy evaluations preserve the leading two generators. Leave-one-policy-out, camera-only, responsibility-weight, and reference-fusion comparisons retain the complete generator ordering. Responsibility-weight changes shift QTES by less than 0.10 points. TRS affects score calibration, while source-wise comparisons reveal dataset-specific differences.

  • Metric computation and reproducibility (gg8Y, VZsM, SLBC). We specify the behavioral inputs, evidence aggregation, reference-driver fusion, validity criteria, and collision gates. Responsibility-aware scoring differentiates collision penalties through explicit rules. Matched-gate comparisons show differentiated penalties across all six policies. FSM, CC-JP, and RSS provide reference-based consistency diagnostics. The five TVC components now have explicit extraction, matching, normalization, and missing-evidence rules.

  • Closed-loop testing and traffic-backend expansion (gg8Y, iSvR, VZsM). We implement additonal closed-loop testing, as closed-loop feedback exposes additional degradation while preserving the leading QTES order. Across 20 benchmark scenarios, RPDMS and route completion decrease under feedback for all seven renderers. DreamForge and MagicDrive-V2 remain first and second in QTES, with only third and fourth place exchanged (rs=0.964r_s=0.964). SMART-based leaderboard testing separately retains OpenDWM as the leading generator under an alternative traffic backend.

  • Input-adaptation controls and view-completion diagnostics (SLBC). Null-LiDAR adaptation affects individual policies but does not determine the generator ordering. Real versus all-zero LiDAR changes GoalFlow and DiffusionDriveV2 performance in different directions, while excluding both preserves the complete ordering. For Nexar view completion, additional visualizations, manual review, and CLIP-IQA+ and adjusted DINO-T diagnostics document the completed inputs, with observed and synthesized views explicitly distinguished.

We have incorporated the corresponding revisions into the main manuscript and highlighted them in blue. The supplementary response letter presents the detailed replies alongside the additional visualizations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.