- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that existing driving-world-model benchmarks assess visual fidelity and coarse physical plausibility but never compare generated futures against synchronized ground-truth vehicle states in kinematic and dynamic terms. It introduces VehDyn, built on a CARLA-CarSim co-simulation platform: CARLA renders, CarSim computes a validated multi-body vehicle response, and the two are synchronized on a shared clock. A full factorial design over five vehicle types, four tire-road friction coefficients, three maneuvers, four target speeds, 14 scenes, and three sun angles yields 10,080 configurations. Twelve models (general, physical, and driving world models) are post-trained on VehDyn and benchmarked. Headline findings: trajectory metrics are nearly saturated (10 of 12 models within 20% of ground truth), no model reaches 92% of ground truth on dynamic consistency, and visual-quality metrics correlate only weakly with dynamics fidelity.
Critical strengths
- The benchmark fills a real and specific gap. No prior resource pairs generated driving videos with synchronized, physically controlled vehicle-state ground truth; the full-factorial CARLA-CarSim design provides matched pairs in which the ground-truth response change is known directly (e.g., friction contrast, speed sweep), which is what makes the condition-aware dynamic-consistency analysis possible. The dataset itself, with 100,800 samples and motion-centric captions, is a substantial community asset.
- The measurement framework is well conceived: a hierarchical protocol separating trajectory, kinematics, and dynamics, with the friction-circle compliance rate and cross-condition response tests going beyond any existing benchmark. The finding that feasible motion (high FCCR) does not imply correct response to changed conditions (low CDC) is a useful, non-obvious decomposition.
Critical weaknesses
- State-recovery noise may dominate the signal. Every score is computed from DPVO's monocular recovery of the ego state, and the ground-truth-video ceiling shows the extraction floor is far from zero (the GT video itself scores ADE 0.514 m, pitch RMSE 2.522 deg, yaw RMSE 3.885 deg). Since the best models' kinematic errors approach that floor, the benchmark may be measuring DPVO's robustness to each model's particular video as much as the models' vehicle dynamics.
- All models are post-trained on the VehDyn training split, which confounds model capability with fine-tuning protocol. Rankings and the dynamics ceiling could reflect how each model was fine-tuned (budget, learning rate, conditioning format, LoRA settings) rather than inherent dynamics reasoning. An off-the-shelf evaluation, or a standardized fine-tuning protocol with controlled budgets, is needed to support the conclusions about model abilities.
Questions for authors
- Can you provide an explicit error budget? Quantify DPVO recovery error on ground-truth videos per metric and per maneuver, and report which model differences are statistically above the extraction floor. The GT ceiling row already shows the floor is non-trivial for pitch and yaw.
- How much do the rankings and the dynamic-consistency results depend on the post-training protocol? Would off-the-shelf evaluation (or a single standardized fine-tuning recipe across models) change the conclusions?
- For FCCR and the CDC components, how sensitive are the scores to the assumed friction coefficient and to errors in the recovered acceleration? Is the friction condition given to the model via the caption, and could caption-following rather than physical response drive CDCDir?
5 comments
Response to Reviewer #p19W Part 2
Response to Reviewer #p19W Part 2
Continue to [A3]
CDC sensitivity. CDC compares reconstructed responses across matched conditions in Table 2. With condition pairs held fixed, we perturbed the peak lateral acceleration response while retaining the other responses. Under acceleration gains of 0.8-1.2, the aggregate CDC_Dir remained unchanged, while CDC_Mag changed by an average of 0.274 percentage points and at most 0.522 points. CDC_Dir remains unchanged because a common positive gain does not alter the signs of pairwise response differences. The magnitude component provides complementary information about response amplitude.
Table 2: Sensitivity of the CDC direction, magnitude, and trend scores across 12 models. Entries report absolute score changes in percentage points relative to the default configuration. Mean changes average equally over models and, for grouped rows, tested settings; maximum changes are over individual model–setting combinations.
| Setting | CDC_Dir | CDC_Mag | CDC_Trend | |||
|---|---|---|---|---|---|---|
| Mean ↓ | Max ↓ | Mean ↓ | Max ↓ | Mean ↓ | Max ↓ | |
| Meaningful-change threshold (default: ) | ||||||
| 0.316 | 0.745 | 1.206 | 1.672 | 0.001 | 0.010 | |
| 0.148 | 0.344 | 0.777 | 1.040 | 0.001 | 0.010 | |
| 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
| 0.082 | 0.169 | 0.546 | 0.818 | 0.000 | 0.000 | |
| 0.141 | 0.420 | 1.288 | 1.664 | 0.003 | 0.031 | |
| 0.454 | 1.293 | 4.523 | 5.705 | 0.048 | 0.168 | |
| Acceleration-response and differentiation perturbations | ||||||
| Acceleration gain – | 0.000 | 0.000 | 0.274 | 0.522 | 0.000 | 0.000 |
| Cubic differentiation window 0.4–1.2 s | 0.061 | 0.197 | 0.175 | 0.612 | 0.206 | 0.550 |
| Quadratic differentiation window 0.8 s | 0.136 | 0.322 | 0.926 | 1.390 | 0.270 | 0.788 |
| Finite differences | 0.056 | 0.109 | 0.102 | 0.166 | 0.233 | 0.543 |
Thank you for additional experiments and clarifications. The new results help quantify the reconstruction floor, especially for pitch and yaw. I note that the ranking of top models is relatively stable across DPVO and GEM, while full-set ranking agreement remains moderate. This partially addresses my concern about whether the benchmark is measuring reconstruction pipeline behavior rather than model dynamics.
However, it is still difficult to tell for which individual model comparisons the performance gap is clearly above the extraction noise floor without explicit statistical testing per model pair.
I also encourage the clarification on model inputs and the split of CDC metrics into responses visible / not visible from conditioning. This decomposition is valuable. I look forward to the upcoming zero-shot evaluation results, as they will be critical to disentangling inherent physical reasoning ability from effects of post-training and fine-tuning recipes.
Given the updates, I keep my current rating as Weak Accept.
Response to Reviewer #p19W Part 1
Response to Reviewer #p19W Part 1
[W1+Q1] Can you provide an explicit error budget? Quantify DPVO recovery error on ground-truth videos per metric and per maneuver, and report which model differences are statistically above the extraction floor. The GT ceiling row already shows the floor is non-trivial for pitch and yaw.
[A1] See our Response to Reviewer Phih Part 1 & Part 2, [A1]. Especially, you can refer to the reconstruction-accuracy analysis added to Section 4.2 (“Feature Extraction Analysis”), robustness evaluations added to Appendix A.4.1, and the new “Effect of Attitude Response Amplitude” subsection in Appendix 4.2.
[W2+Q2] How much do the rankings and the dynamic-consistency results depend on the post-training protocol? Would off-the-shelf evaluation (or a single standardized fine-tuning recipe across models) change the conclusions?
[A2] See our Response to Reviewer Phih Part 2 & Part 3, [A2]. To be more specific, the model-input details are clarified in Section 4.1 and the separate CDC analyses with and without input information are demonstrated in Appendix A.4.4.
The zero-shot VehDyn scores and complementary visual evaluation scores will be reported later in Response to Reviewer #p19W Part 3 (the work load is heavy and the experiments are undergoing) and the results will be also added to a new subsection "Zero-Shot Results" in Appendix A.4.6 later.
[Q3] For FCCR and the CDC components, how sensitive are the scores to the assumed friction coefficient and to errors in the recovered acceleration? Is the friction condition given to the model via the caption, and could caption-following rather than physical response drive CDCDir?
[A3] We have added the parameter sensitivity analyses of FCCR and CDC to Appendix A.5.
FCCR sensitivity. We evaluated 22 parameter-perturbation settings on the same valid model-clip pairs across 12 models, using the default scoring configuration as the baseline. These tests cover filtering and differentiation settings, tolerance, assumed friction coefficients, recovered-acceleration gains and magnitude offsets, and joint perturbations, as shown in Table 1. Trajectory calibration and the integration window were held fixed throughout these tests. Across all 22 settings and 12 models, the mean absolute FCCR change was 1.78 percentage points, the maximum change was 5.71 points, and FCCR rankings shifted by at most two places. Filtering and differentiation changes produced a maximum score change of 4.33 points and a maximum rank shift of two places. Varying the tolerance from 0 to 1m/s2 produced a maximum change of 1.99 points and a maximum rank shift of one place. These results indicate limited sensitivity within the ranges of parameters tested under fixed reference-based calibration.
Table 1: Sensitivity of FCCR scores and ranks relative to the default configuration. Mean absolute score changes weight models and settings equally within each row. The maximum is taken over all model–setting combinations in that row. Score changes are in percentage points (pp). Rank shifts are absolute changes in the FCCR ranking of the 12 models.