acceptodds
Under review as a conference paper at ICLR 2027

VehDyn: A Driving World Model Benchmark for Vehicle Dynamics

Abstract

Video world models are emerging as data engines, action planners, and generative simulators for autonomous driving, but existing benchmarks primarily assess visual fidelity and coarse physical plausibility, providing limited evidence on whether generated driving futures obey realistic vehicle kinematics and dynamics. This limitation is further compounded by the lack of datasets in which vehicle, road, maneuver, and speed conditions are independently controlled, and ground-truth vehicle states are recorded in synchrony with videos. We introduce VehDyn, a driving world model benchmark for vehicle dynamics. VehDyn is built on a CARLA-CarSim co-simulation platform where photorealistic rendering is coupled with a validated multi-body dynamics model, and it contains 10,080 configurations from a full factorial design over five vehicle types, four tire-road friction coefficients, three maneuvers, four target speeds, 14 scenes, and three illuminations, each paired with synchronized position, velocity, and attitude sequences. Built on this dataset, VehDyn introduces a hierarchical evaluation framework that measures trajectory alignment, kinematic consistency, and dynamic consistency, and benchmarks 12 state-of-the-art video world models. We further assess the video quality using two established protocols and correlate it with the VehDyn score. Trajectory-level metrics are nearly saturated, with ten of twelve models within 20% of ground truth, while no model reaches 92% of ground truth on dynamic consistency, and visual-quality metrics are only weakly correlated with vehicle-dynamics fidelity. DrivingWorld achieves the highest VehDyn score, followed by Cosmos 3 Nano and LTX-Video 2.5, and the VehDyn score agrees closely with human judgment. VehDyn provides a systematic foundation for developing driving world models that are physically consistent and visually realistic.

Primary area:
datasets and benchmarks
Keywords:
video world model, driving world model, vehicle dynamics, vehicle kinematics, physical consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_p19WOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that existing driving-world-model benchmarks assess visual fidelity and coarse physical plausibility but never compare generated futures against synchronized ground-truth vehicle states in kinematic and dynamic terms. It introduces VehDyn, built on a CARLA-CarSim co-simulation platform: CARLA renders, CarSim computes a validated multi-body vehicle response, and the two are synchronized on a shared clock. A full factorial design over five vehicle types, four tire-road friction coefficients, three maneuvers, four target speeds, 14 scenes, and three sun angles yields 10,080 configurations. Twelve models (general, physical, and driving world models) are post-trained on VehDyn and benchmarked. Headline findings: trajectory metrics are nearly saturated (10 of 12 models within 20% of ground truth), no model reaches 92% of ground truth on dynamic consistency, and visual-quality metrics correlate only weakly with dynamics fidelity.

Critical strengths
  1. The benchmark fills a real and specific gap. No prior resource pairs generated driving videos with synchronized, physically controlled vehicle-state ground truth; the full-factorial CARLA-CarSim design provides matched pairs in which the ground-truth response change is known directly (e.g., friction contrast, speed sweep), which is what makes the condition-aware dynamic-consistency analysis possible. The dataset itself, with 100,800 samples and motion-centric captions, is a substantial community asset.
  2. The measurement framework is well conceived: a hierarchical protocol separating trajectory, kinematics, and dynamics, with the friction-circle compliance rate and cross-condition response tests going beyond any existing benchmark. The finding that feasible motion (high FCCR) does not imply correct response to changed conditions (low CDC) is a useful, non-obvious decomposition.
Critical weaknesses
  1. State-recovery noise may dominate the signal. Every score is computed from DPVO's monocular recovery of the ego state, and the ground-truth-video ceiling shows the extraction floor is far from zero (the GT video itself scores ADE 0.514 m, pitch RMSE 2.522 deg, yaw RMSE 3.885 deg). Since the best models' kinematic errors approach that floor, the benchmark may be measuring DPVO's robustness to each model's particular video as much as the models' vehicle dynamics.
  2. All models are post-trained on the VehDyn training split, which confounds model capability with fine-tuning protocol. Rankings and the dynamics ceiling could reflect how each model was fine-tuned (budget, learning rate, conditioning format, LoRA settings) rather than inherent dynamics reasoning. An off-the-shelf evaluation, or a standardized fine-tuning protocol with controlled budgets, is needed to support the conclusions about model abilities.
Questions for authors
  1. Can you provide an explicit error budget? Quantify DPVO recovery error on ground-truth videos per metric and per maneuver, and report which model differences are statistically above the extraction floor. The GT ceiling row already shows the floor is non-trivial for pitch and yaw.
  2. How much do the rankings and the dynamic-consistency results depend on the post-training protocol? Would off-the-shelf evaluation (or a single standardized fine-tuning recipe across models) change the conclusions?
  3. For FCCR and the CDC components, how sensitive are the scores to the assumed friction coefficient and to errors in the recovered acceleration? Is the friction condition given to the model via the caption, and could caption-following rather than physical response drive CDCDir?
5 comments
AuthorsOpenReview ↗

Response to Reviewer #p19W Part 1

Response to Reviewer #p19W Part 1

[W1+Q1] Can you provide an explicit error budget? Quantify DPVO recovery error on ground-truth videos per metric and per maneuver, and report which model differences are statistically above the extraction floor. The GT ceiling row already shows the floor is non-trivial for pitch and yaw.

[A1] See our Response to Reviewer Phih Part 1 & Part 2, [A1]. Especially, you can refer to the reconstruction-accuracy analysis added to Section 4.2 (“Feature Extraction Analysis”), robustness evaluations added to Appendix A.4.1, and the new “Effect of Attitude Response Amplitude” subsection in Appendix 4.2.

[W2+Q2] How much do the rankings and the dynamic-consistency results depend on the post-training protocol? Would off-the-shelf evaluation (or a single standardized fine-tuning recipe across models) change the conclusions?

[A2] See our Response to Reviewer Phih Part 2 & Part 3, [A2]. To be more specific, the model-input details are clarified in Section 4.1 and the separate CDC analyses with and without input information are demonstrated in Appendix A.4.4.

The zero-shot VehDyn scores and complementary visual evaluation scores will be reported later in Response to Reviewer #p19W Part 3 (the work load is heavy and the experiments are undergoing) and the results will be also added to a new subsection "Zero-Shot Results" in Appendix A.4.6 later.

[Q3] For FCCR and the CDC components, how sensitive are the scores to the assumed friction coefficient and to errors in the recovered acceleration? Is the friction condition given to the model via the caption, and could caption-following rather than physical response drive CDCDir?

[A3] We have added the parameter sensitivity analyses of FCCR and CDC to Appendix A.5.

FCCR sensitivity. We evaluated 22 parameter-perturbation settings on the same valid model-clip pairs across 12 models, using the default scoring configuration as the baseline. These tests cover filtering and differentiation settings, tolerance, assumed friction coefficients, recovered-acceleration gains and magnitude offsets, and joint perturbations, as shown in Table 1. Trajectory calibration and the integration window were held fixed throughout these tests. Across all 22 settings and 12 models, the mean absolute FCCR change was 1.78 percentage points, the maximum change was 5.71 points, and FCCR rankings shifted by at most two places. Filtering and differentiation changes produced a maximum score change of 4.33 points and a maximum rank shift of two places. Varying the tolerance from 0 to 1m/s21 \mathrm{m/s^2} produced a maximum change of 1.99 points and a maximum rank shift of one place. These results indicate limited sensitivity within the ranges of parameters tested under fixed reference-based calibration.

Table 1: Sensitivity of FCCR scores and ranks relative to the default configuration. Mean absolute score changes weight models and settings equally within each row. The maximum is taken over all model–setting combinations in that row. Score changes are in percentage points (pp). Rank shifts are absolute changes in the FCCR ranking of the 12 models.

ParameterPerturbed SettingsMean Change ↓ (pp)Max. Change ↓ (pp)Max. Rank Shift ↓
Scoring perturbations with the integration window fixed at 0.8 s
Cubic differentiation window0.4, 0.6, 1.0, 1.2 s0.391.532
Polynomial degreeQuadratic, same 0.8 s window2.824.332
Differentiation methodFinite differences0.270.980
Tolerance δ\delta0, 0.25, 0.75, 1.0 m/s2\mathrm{m/s^2}1.031.991
Friction coefficient μ\mu×{0.8,0.9,1.1,1.2}\times\{0.8, 0.9, 1.1, 1.2\}2.615.712
Acceleration gainA×{0.8,0.9,1.1,1.2}A\times\{0.8, 0.9, 1.1, 1.2\}2.775.262
Acceleration offsetA±0.5 m/s2A\pm 0.5\,\mathrm{m/s^2}1.001.471
Joint perturbationTighter and looser settings3.445.222
All 22 settings1.785.712
Integration-window ablation with default differentiation
Integration window1.0 s1.732.852
Integration window0.6 s2.574.911
Integration window0.4 s5.4411.242
Integration windowNo integration8.7917.505
AuthorsOpenReview ↗

Response to Reviewer #p19W Part 2

Response to Reviewer #p19W Part 2

Continue to [A3]

CDC sensitivity. CDC compares reconstructed responses across matched conditions in Table 2. With condition pairs held fixed, we perturbed the peak lateral acceleration response while retaining the other responses. Under acceleration gains of 0.8-1.2, the aggregate CDC_Dir remained unchanged, while CDC_Mag changed by an average of 0.274 percentage points and at most 0.522 points. CDC_Dir remains unchanged because a common positive gain does not alter the signs of pairwise response differences. The magnitude component provides complementary information about response amplitude.

Table 2: Sensitivity of the CDC direction, magnitude, and trend scores across 12 models. Entries report absolute score changes in percentage points relative to the default configuration. Mean changes average equally over models and, for grouped rows, tested settings; maximum changes are over individual model–setting combinations.

SettingCDC_DirCDC_MagCDC_Trend
Mean ∣Δ∣\lvert\Delta\rvert ↓Max ∣Δ∣\lvert\Delta\rvert ↓Mean ∣Δ∣\lvert\Delta\rvert ↓Max ∣Δ∣\lvert\Delta\rvert ↓Mean ∣Δ∣\lvert\Delta\rvert ↓Max ∣Δ∣\lvert\Delta\rvert ↓
Meaningful-change threshold (default: α=5%\alpha = 5\%)
α=0%\alpha = 0\%0.3160.7451.2061.6720.0010.010
α=2.5%\alpha = 2.5\%0.1480.3440.7771.0400.0010.010
α=5%\alpha = 5\%0.0000.0000.0000.0000.0000.000
α=7.5%\alpha = 7.5\%0.0820.1690.5460.8180.0000.000
α=10%\alpha = 10\%0.1410.4201.2881.6640.0030.031
α=20%\alpha = 20\%0.4541.2934.5235.7050.0480.168
Acceleration-response and differentiation perturbations
Acceleration gain × 0.8\times\,0.8–1.21.20.0000.0000.2740.5220.0000.000
Cubic differentiation window 0.4–1.2 s0.0610.1970.1750.6120.2060.550
Quadratic differentiation window 0.8 s0.1360.3220.9261.3900.2700.788
Finite differences0.0560.1090.1020.1660.2330.543

Thank you for additional experiments and clarifications. The new results help quantify the reconstruction floor, especially for pitch and yaw. I note that the ranking of top models is relatively stable across DPVO and GEM, while full-set ranking agreement remains moderate. This partially addresses my concern about whether the benchmark is measuring reconstruction pipeline behavior rather than model dynamics.

However, it is still difficult to tell for which individual model comparisons the performance gap is clearly above the extraction noise floor without explicit statistical testing per model pair.

I also encourage the clarification on model inputs and the split of CDC metrics into responses visible / not visible from conditioning. This decomposition is valuable. I look forward to the upcoming zero-shot evaluation results, as they will be critical to disentangling inherent physical reasoning ability from effects of post-training and fine-tuning recipes.

Given the updates, I keep my current rating as Weak Accept.

Reviewer_UFVMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

VehDyn addresses a legitimate and important evaluation gap in driving world models. Its strongest contribution is the combination of controlled vehicle-dynamics simulations, synchronized reference states, and a dynamics-oriented evaluation framework. The topic iteself is very interesting, the provided Carla-Carsim sounds a new dataset to analyze more realistic behaviors.

Critical strengths

Substaintial effort to build the new dataset and benchmarks.

Critical weaknesses

There are some key concerns for this paper as below: (1) There are existing digital twins for traffic+vehicle dynamics co-simulations such as Nvidia PhysX and Chronos+Carla. Carsim+Carla may not be a big contribution. (2) Vehicle dynamics are far more complex than 6 DOF. For example, there could be rolling-over, skiding etc, which is more beyond of scope for the current paper (3) Just measuring 2D trajectories, and corresponding states such as velocity, pitch angle, and roll angle. Analyze the bounding box corners could be much more helpful. (4) Tire friction is super-important for vehicle dynamics, especially considering safety. (5) Transportation system is high-dimensional, including multiple factors. 10080 configures may not be enough to fully represent the whole picture.

Questions for authors

(1) Include substaintial real-world dataset to justify the realism of Carla-Carsim hollistically and seperately are very needed.

5 comments
AuthorsOpenReview ↗

Response to Reviewer #UFVM Part 1

Response to Reviewer #UFVM Part 1

[W1] What does VehDyn contribute beyond existing vehicle-dynamics co-simulation systems such as NVIDIA PhysX and Chrono-CARLA? Is integrating CarSim with CARLA sufficiently novel?

[A1] We clarify in Section 1 that VehDyn builds on established co-simulation infrastructure, with its contributions centered on controlled benchmark design and a hierarchical protocol for evaluating the physical fidelity of video world models. Existing systems, including NVIDIA PhysX, Chrono–CARLA, and the existing CarSim–CARLA integration, provide the underlying capabilities for simulating vehicle motion; VehDyn contributes the controlled dataset and evaluation protocol. As shown in Table 1 in the main paper (Section 1), VehDyn provides a controlled dataset that contains full controlled configurations. There is no other open-source dataset that includes 10,080 factorial configurations and 100,800 video episodes, each paired with synchronized reference vehicle states. Beyond its scale and synchronized ground-truth states, VehDyn systematically covers a controlled scenario-parameter space through a factorial design. This structure enables matched comparisons in which selected physical conditions are varied while other configuration parameters are held fixed, allowing us to evaluate whether generated driving futures reproduce the corresponding changes in vehicle dynamics. In addition, VehDyn introduces a hierarchical evaluation protocol covering trajectory alignment, kinematic consistency, and condition-aware dynamic consistency. None of these depends on the choice of solver beyond the requirement that its ground truth be trustworthy, and the metric pipeline consumes only synchronized 6-DoF states, so the benchmark can be regenerated with any solver that exports them. Unlike a simulation framework alone, VehDyn provides controlled physical interventions, matched reference responses, video-based state extraction, and standardized metrics specifically designed to evaluate generated driving futures.

[W2] Does a 6-DoF representation adequately capture complex vehicle dynamics? How does the benchmark address phenomena such as rollover and skidding?

[A2] We clarify the distinction between the 6-DoF motion representation and the underlying multi-body dynamics model in Section 3, and specify the current evaluation scope and limitations in Section 5 of the main paper. The ground truth is generated by CarSim's multi-body vehicle model, which comprises the sprung mass, tire slip, wheel rotation, suspension motion, tire forces, load transfer, etc., which are all important components of vehicle dynamics. However, our use of 6-DoF refers to the observable rigid-body motion representation, rather than a restriction of the underlying CarSim vehicle model to 6-DoF. CarSim includes additional internal dynamic states, while VehDyn focuses on quantities that can be evaluated from generated RGB videos and compared with synchronized reference trajectories. The current VehDyn benchmark concentrates on emergency-braking and lane-changing maneuvers, and does not comprehensively evaluate extreme phenomena such as rollover, where no video world model is trained on or capable of rendering a camera that inverts, and the monocular state-extraction pipeline cannot recover attitude through such an event. However, skidding is within VehDyn's scope and within the dataset. The lane-changing conditions at high speed on low friction demand lateral accelerations at or beyond the available tire force, so tire saturation and lateral slip occur in a substantial fraction of the samples. The emergency-braking conditions on low friction likewise produce longitudinal slip. Skidding is also evaluated directly: lateral velocity vyv_y, which is non-zero only when the vehicle develops sideslip, is one of the kinematic-consistency channels, the friction-circle compliance rate (FCCR) is defined precisely to detect generated motion whose planar acceleration exceeds the limits, and the dynamic-consistency metrics measure each model's response to friction changes in matched pairs.

[W3] Is evaluating trajectories, velocity, pitch, and roll sufficient? Could tracking vehicle bounding-box corners provide useful complementary geometric information?

[A3] The camera in VehDyn is rigidly attached to the ego-vehicle body. Consequently, the projected corners of the vehicle's rigid bounding box can remain approximately fixed in the camera coordinate frame even when the vehicle changes its orientation relative to the road. Therefore, directly tracking ego-vehicle bounding-box corners in the existing chase-camera videos would not necessarily provide independent measurements of pitch and roll.

AuthorsOpenReview ↗

Response to Reviewer #UFVM Part 2

Response to Reviewer #UFVM Part 2

Continue to [A3]

To assess the reliability of the extracted motion states, we additionally evaluated DPVO, DrivingGen, and GEM on the original simulation videos against the simulator reference, and conducted a complementary evaluation on KITTI in the additional paragraph “Feature Extraction Analysis” added to Section 4.2. The results of VehDyn show that trajectory, velocity, and acceleration are nearly perfect, but pitch, yaw, and roll have higher uncertainty. The results of KITTI show that trajectory, velocity, and acceleration are similar to VehDyn, but the attitude responses are more accurate in real-world driving scenarios, resulting in smaller errors. These results explain that the higher VehDyn score of generated videos than ground truth is not because of the accuracy of feature extraction method, but the rendering limitations of CARLA simulator, which provides more insights into sim-to-real gap in autonomous driving. They do not replace the complementary geometric information that bounding-box analysis could provide. Future work will investigate static scene landmarks, road-plane geometry, and vehicle-to-road spatial relationships as complementary measurements.

[W4] How does VehDyn account for tire-road friction, given its importance to vehicle dynamics and driving safety?

[A4] Tire-road friction is one of the four explicitly controlled physical factors in VehDyn. The current dataset includes four road conditions, independently combined with five vehicle types, three maneuvers, and four target speeds. Our evaluation also includes the friction-circle constraint rate (FCCR), which assesses whether the recovered planar acceleration satisfies an approximate tire-road adhesion bound. Additionally, the condition-aware dynamic consistency metrics examine whether generated vehicle responses change appropriately across different physical conditions. Controlled friction conditions and the corresponding evaluation metrics are described in Sections 2.1 and 3.2 of the main paper, respectively. We added friction-coefficient sensitivity analysis in Appendix A.5. VehDyn focuses on friction-dependent consistency in generated vehicle motion; a comprehensive assessment of stability and driving safety lies beyond its current scope.

[W5] Are 10,080 configurations sufficient to represent the many factors in transportation systems? What are the limits of the benchmark's coverage?

[A5] We clarify in Section 1 that VehDyn is designed as a controlled benchmark for selected ego-vehicle dynamics rather than an exhaustive simulation of all transportation scenarios. The factorial design contains 240 unique physical configurations, formed by five vehicle types, four friction coefficients, three maneuvers, and four target speeds. These are combined with 42 visual conditions, consisting of 14 scenes and three illumination settings, to form 10,080 configurations. The benefit of this design is that the physical factors can be systematically varied while other conditions are held fixed. This enables matched comparisons and isolates the effects of physical interventions, which are difficult to obtain from uncontrolled naturalistic driving datasets. We clarify in Section 5 that future work will extend the benchmark to additional maneuvers and environmental conditions multi-agent interaction, multi-view observation, and closed-loop planning, and will validate the protocol against real-world driving scenarios.

[Q1] Can substantial real-world data validate the realism of CARLA and CarSim, both individually and as a coupled system?

[A6] In the “Feature Extraction Analysis” paragraph added to Section 4.2 of the main paper, we incorporated additional real-world validation using KITTI Odometry [R1], which provides camera recordings and vehicle-motion measurements. The results in Table 1 show that the trajectory, velocity, and acceleration values in real-world data are similar to simulation, but the attitude responses are more accurate in real-world driving scenarios, resulting in smaller errors. These results explain that the higher VehDyn score of generated videos than ground truth is not because of the accuracy of feature extraction method, but the rendering limitations of CARLA simulator, which provides more insights into sim-to-real gap in autonomous driving. This conclusion is also verified by other two latest benchmarks shown in the original Table 2 (Section 4.2), with higher photometric consistency, style consistency, video consistency, and trajectory consistency than ground truth.

AuthorsOpenReview ↗

Response to Reviewer #UFVM Part 3

Response to Reviewer #UFVM Part 3

Continue to [A6]

Table 1: Comparison of trajectory-reconstruction pipelines on VehDyn (top) and KITTI Odometry (bottom). Entries are sequence-level macro means for VehDyn and clip-level macro means for KITTI Odometry, with medians in parentheses. All metrics are errors, so lower is better. The best mean for each metric within each dataset is highlighted.

VehDyn MethodsAll ManeuversEmergency-BrakingSingle-Lane-Changing
ADE ↓ (m)FDE ↓ (m)vxv_x RMSE ↓ (m/s)axa_x RMSE ↓ (m/s²)Pitch RMSE ↓ (deg)vyv_y RMSE ↓ (m/s)aya_y RMSE ↓ (m/s²)Roll RMSE ↓ (deg)Yaw RMSE ↓ (deg)
DPVO0.514 (0.301)0.814 (0.346)0.897 (0.469)2.829 (1.176)2.522 (2.162)0.697 (0.521)1.509 (0.986)0.602 (0.468)3.885 (3.813)
DrivingGen2.202 (1.591)4.536 (2.238)1.572 (1.248)5.275 (4.012)8.660 (6.726)3.552 (2.736)9.204 (6.806)9.141 (5.250)9.095 (6.823)
GEM0.589 (0.283)1.071 (0.346)0.982 (0.288)2.871 (1.829)2.765 (2.037)1.045 (0.661)3.394 (1.529)0.624 (0.496)3.888 (3.863)
KITTI Odometry MethodsTrajectorySpeed and AccelerationAttitude
ADE ↓ (m)FDE ↓ (m)Speed RMSE ↓ (m/s)Acceleration RMSE ↓ (m/s²)Pitch RMSE ↓ (deg)Roll RMSE ↓ (deg)Yaw RMSE ↓ (deg)
DPVO0.602 (0.314)1.092 (0.582)0.504 (0.245)0.567 (0.181)0.160 (0.144)0.149 (0.114)0.200 (0.149)
DrivingGen1.566 (0.889)2.885 (1.586)1.076 (0.667)2.030 (1.279)1.406 (1.067)1.398 (0.711)3.002 (2.224)
GEM0.901 (0.334)1.861 (0.749)1.007 (0.258)1.633 (0.244)0.183 (0.151)0.218 (0.129)0.385 (0.178)

To further investigate the validity of VehDyn's simulation-based evaluation, we have added robustness evaluations to Appendix A.4.5 comparing the visual and temporal-consistency metrics obtained on VehDyn with previously published results from WorldScore and DrivingGen, which evaluate world models on different real-world or heterogeneous visual data distributions. For DrivingGen, seven models have complete results for three shared metrics: FVD, Video Consistency, and Trajectory Consistency. An exploratory equal-weight aggregation of their metric-wise rankings yields a Spearman's rank correlation of ρ=0.7182\rho=0.7182 between the original benchmark and VehDyn. Notably, the same three models LTX-Video, Wan 2.2-I2V, and SkyReels-V2-I2V occupy the top three positions in both evaluations. The individual metrics exhibit weaker correlations, indicating that some aspects of model performance are sensitive to the evaluation domain. For WorldScore, the available reference results allow a direct comparison of two overlapping model families across four quality metrics. Their relative ordering is preserved in three of the four metrics, providing limited additional evidence of consistency in visual-quality assessment.

Overall, these observations suggest that VehDyn reproduces certain relative model-performance patterns observed on established evaluation datasets despite the differences between simulated and real-world or heterogeneous visual domains. These results support the utility of VehDyn as an evaluation environment. Establishing the physical realism of the simulated vehicle dynamics requires separate validation against measured real-world vehicle responses; ranking consistency alone does not establish it. The original values are also reported. We recognize that naturalistic datasets generally do not provide controlled measurements of all relevant physical factors, particularly tire-road friction and extreme loss-of-control events. Therefore, the conclusions from this validation will be restricted to the variables and operating conditions directly supported by the available measurements.

[R1] Geiger, A., Lenz, P., & Urtasun, R. (2012, June). Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition (pp. 3354-3361). IEEE.

Reviewer_PhihOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VehDyn evaluates whether generated driving videos reproduce realistic vehicle motion. It combines CARLA rendering with CarSim dynamics to construct 100,800 episodes spanning 10,080 configurations of vehicle type, road friction, maneuver, speed, scene, and illumination. The evaluation uses DPVO to recover vehicle states and assesses trajectory alignment, kinematic accuracy, friction-circle compliance, and responses to changes in physical conditions. Results across 12 models suggest that visual quality alone is a poor indicator of vehicle-dynamics fidelity.

Critical strengths
  1. The controlled dataset is a useful contribution. The CARLA–CarSim setup provides synchronized videos and vehicle states while systematically varying physical conditions. This enables comparisons that are difficult to make using observational driving data, particularly for the effects of friction, speed, and vehicle properties.

  2. The evaluation asks a meaningful question beyond visual realism. Assessing how motion changes across physical conditions is more informative than checking trajectory alignment or smoothness alone. The distinction between physically feasible motion and a correct response to changing conditions is particularly valuable.

Critical weaknesses
  1. Reconstruction uncertainty makes the physical interpretation of the scores unclear. In Table 2, several generated-video results have lower attitude errors than the ground-truth videos processed through the same pipeline. Section 4.2 also reports a kinematic score above the ground-truth reference. This raises a concern that reconstruction difficulty influences the scores substantially. Some attitude responses in Figure 2 are only a few degrees, making it important to establish whether the pipeline can reliably resolve these differences. Acceleration-based evaluation requires further care because differentiation can amplify pose-estimation noise. The main paper does not establish whether the differences between models exceed this measurement uncertainty.

  2. Different conditioning inputs complicate the comparison. General and physical models receive an initial image and a caption, whereas driving models receive trajectories or poses. If those inputs already specify the motion changes evaluated by CDC, a high score may reflect accurate rendering of supplied motion rather than prediction of vehicle dynamics. Conversely, an image and caption may leave the initial state or subsequent controls underspecified. The paper needs a clearer account of what each model must infer and what it is explicitly given before the overall ranking can support conclusions about dynamics capability.

Questions for authors
  1. How accurately can the DPVO-based pipeline detect controlled changes in pitch, roll, velocity, and acceleration? Please explain why generated videos can obtain lower attitude errors than ground-truth videos and whether the model rankings remain stable under reconstruction uncertainty.

  2. Which future states are included in the trajectory/pose inputs supplied to driving models? Do these inputs contain the response quantities evaluated by CDC? Would the main conclusions hold under matched conditioning?

  3. How is metric scale recovered from monocular estimates? Is the reference trajectory used for alignment? How sensitive are FCCR scores and rankings to filtering, differentiation, and tolerance settings?

  4. Can episodes from the same factorial configuration appear in both training and test sets? If so, are results available for held-out physical configurations or combinations?

6 comments
AuthorsOpenReview ↗

Response to Reviewer #Phih Part 1

Response to Reviewer #Phih Part 1

[W1+Q1] How accurately can the reconstruction pipeline recover pitch, roll, velocity, and acceleration, particularly for small attitude changes and noise-sensitive acceleration estimates? Why can generated videos score better than ground-truth videos? Do model differences and rankings remain reliable under reconstruction uncertainty?

[A1] In the original Table 10 (Appendix A.4.1), we reported the comparative performance of trajectory-reconstruction pipelines on simulation platform within VehDyn dataset. In the “Feature Extraction Analysis” paragraph added to Section 4.2, we reported axa_x RMSE in emergency-braking scenarios and aya_y RMSE in single-lane-changing scenarios, together with an additional evaluation of the trajectory-reconstruction methods on the real-world KITTI Odometry dataset [R1] (Table 1). The results of VehDyn show that trajectory, velocity, and acceleration are nearly perfect, but pitch, yaw, and roll have higher uncertainty. The results of KITTI show that trajectory, velocity, and acceleration are similar to VehDyn, but the attitude responses are more accurate in real-world driving scenarios, resulting in smaller errors. These results explain that the higher VehDyn score of generated videos than ground truth is not because of the accuracy of feature extraction method, but the rendering limitations of CARLA simulator, which provides more insights into sim-to-real gap in autonomous driving. This conclusion is also verified by other two latest benchmarks shown in the original Table 2 (Section 4.2), with higher photometric consistency, style consistency, video consistency, and trajectory consistency than ground truth.

Table 1: Comparison of trajectory-reconstruction pipelines on VehDyn (top) and KITTI Odometry (bottom). Entries are sequence-level macro means for VehDyn and clip-level macro means for KITTI Odometry, with medians in parentheses. All metrics are errors, so lower is better. The best mean for each metric within each dataset is highlighted.

VehDyn MethodsAll ManeuversEmergency-BrakingSingle-Lane-Changing
ADE ↓ (m)FDE ↓ (m)vxv_x RMSE ↓ (m/s)axa_x RMSE ↓ (m/s²)Pitch RMSE ↓ (deg)vyv_y RMSE ↓ (m/s)aya_y RMSE ↓ (m/s²)Roll RMSE ↓ (deg)Yaw RMSE ↓ (deg)
DPVO0.514 (0.301)0.814 (0.346)0.897 (0.469)2.829 (1.176)2.522 (2.162)0.697 (0.521)1.509 (0.986)0.602 (0.468)3.885 (3.813)
DrivingGen2.202 (1.591)4.536 (2.238)1.572 (1.248)5.275 (4.012)8.660 (6.726)3.552 (2.736)9.204 (6.806)9.141 (5.250)9.095 (6.823)
GEM0.589 (0.283)1.071 (0.346)0.982 (0.288)2.871 (1.829)2.765 (2.037)1.045 (0.661)3.394 (1.529)0.624 (0.496)3.888 (3.863)
KITTI Odometry MethodsTrajectorySpeed and AccelerationAttitude
ADE ↓ (m)FDE ↓ (m)Speed RMSE ↓ (m/s)Acceleration RMSE ↓ (m/s²)Pitch RMSE ↓ (deg)Roll RMSE ↓ (deg)Yaw RMSE ↓ (deg)
DPVO0.602 (0.314)1.092 (0.582)0.504 (0.245)0.567 (0.181)0.160 (0.144)0.149 (0.114)0.200 (0.149)
DrivingGen1.566 (0.889)2.885 (1.586)1.076 (0.667)2.030 (1.279)1.406 (1.067)1.398 (0.711)3.002 (2.224)
GEM0.901 (0.334)1.861 (0.749)1.007 (0.258)1.633 (0.244)0.183 (0.151)0.218 (0.129)0.385 (0.178)

We further investigated the robustness of VehDyn's rankings by comparing three trajectory-reconstruction pipelines (Table 2), with all the detailed results and analyses added to Appendix A.4.1. Across all 12 models, DPVO and GEM exhibit moderate ranking agreement, with Kendall's (τ=0.5455\tau=0.5455) and Spearman's (ρ=0.6154\rho=0.6154). Notably, both pipelines identify DrivingWorld as the highest-ranked model and preserve two of the top three models. We additionally perform an exploratory analysis on the six highest-ranked models under the original DPVO evaluation. Within this subset, DPVO and GEM achieve substantially stronger agreement, with Kendall's (τ=0.8667\tau=0.8667) and Spearman's (ρ=0.9429\rho=0.9429). In contrast, DrivingGen exhibits considerably lower agreement on the same subset. These findings suggest that the ranking of leading models is more consistent between DPVO and GEM, although differences remain across the complete benchmark.

AuthorsOpenReview ↗

Response to Reviewer #Phih Part 2

Response to Reviewer #Phih Part 2

Continue to [A1]

Table 2: Robustness of VehDyn's rankings across different trajectory reconstruction pipelines. Kendall's τ\tau and Spearman's ρ\rho measure ranking agreement, and Top-3 overlap measures the proportion of shared top-three models.

MethodKendall’s τ\tau ↑Spearman’s ρ\rho ↑Top-3 Overlap ↑
DPVO vs. GEM (Top-6)0.86670.942966.7%
DPVO vs. GEM (Full)0.54550.615466.7%
DPVO vs. DrivingGen (Full)0.33330.496533.3%
GEM vs. DrivingGen (Full)0.30300.39860.0%

To further verify the reconstruction accuracy, we tested the trajectory-reconstruction methods under more aggressive driving behaviors in simulation and reported this analysis in the new subsection “Effect of Attitude Response Amplitude” in Appendix A.4.2. We quantified response strength by the temporal standard deviation σGT\sigma_{\mathrm{GT}} of each GT angle and divide the sequences into low-, middle-, and high-response tertiles within each maneuver and axis. Table 3 reports median σGT\sigma_{\mathrm{GT}} and normalized RMSE NRMSE=100∗RMSE/σGT\mathrm{NRMSE}=100*\mathrm{RMSE}/\sigma_{\mathrm{GT}}. The high-response results show lower NRMSE than the low-response results in all five comparisons, supporting an overall association between larger attitude responses and smaller relative reconstruction errors.

Table 3: Attitude-response strength and normalized reconstruction error in extreme GT-video DPVO outputs. Entries report the median temporal ground-truth standard deviation (σGT\sigma_{\mathrm{GT}}, degrees) and median normalized root mean squared error (NRMSE, %), computed separately within each axis/maneuver response tertile.

Axis / ManeuverLow ResponseMiddle ResponseHigh Response
σGT\sigma_{\mathrm{GT}} ↓ (deg)NRMSE ↓ (%)σGT\sigma_{\mathrm{GT}} ↓ (deg)NRMSE ↓ (%)σGT\sigma_{\mathrm{GT}} ↓ (deg)NRMSE ↓ (%)
Pitch / Emergency-Braking0.187820.90.390478.20.778402.4
Roll / Left Lane-Changing0.026336.10.361129.70.994103.6
Roll / Right Lane-Changing0.026321.60.218155.90.844103.3
Yaw / Left Lane-Changing2.192170.12.301155.44.046135.5
Yaw / Right Lane-Changing1.629143.82.239185.22.856138.3

[R1] Geiger, A., Lenz, P., & Urtasun, R. (2012, June). Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition (pp. 3354-3361). IEEE.

[W2+Q2] Which future states are included in the trajectory/pose inputs supplied to driving models? Do these inputs contain the response quantities evaluated by CDC? Would the main conclusions hold under matched conditioning?

[A2] Section 4.1 now specifies the inputs of each model family as follows.

(1) Driving World Models. DrivingWorld, Epona, GEM, and Vista receive the first frame and a sequence of forward and lateral displacements every 0.5s0.5 s, the same interval as the ego trajectory in the captions of the general and physical world models: increments for DrivingWorld and Epona, and waypoints relative to the start of each generation window for GEM and Vista. The inputs contain no yaw, vertical motion, pitch, roll, velocity, acceleration, or control signals.

(2) General and Physical World Models. The caption of the general and physical world models gives the scene, vehicle, surface condition, target speed, maneuver type, and the ego trajectory (time, forward displacement, lateral displacement). It states no friction coefficient, braking distance, braking duration, lateral acceleration, or maneuver timing. The friction coefficient can nevertheless be inferred from the road surface condition, which is described in the caption and visible in the first frame, together with the ego trajectory, since a lower friction coefficient yields, for example, a longer braking distance. The driving world models have access to the same information through the first frame and the trajectory. An example of a complete lane-changing caption is shown below:

AuthorsOpenReview ↗

Response to Reviewer #Phih Part 3

Response to Reviewer #Phih Part 3

Continue to [A2]

“A straight residential road with yellow center marking, wooden houses, streetlights and a traffic light ahead, guardrail and wooded slope alongside, dry asphalt with a fine visible grain, light grey, scattering light diffusely with no mirror reflections, the sun partway up the sky, shadows neither stretched nor compressed, highlights on the vehicle roof and road edges, a small dark two-door city car, short-tailed, the vehicle travels at 72km/h72 km/h and swings out to the left in a single smooth lane change, then settles back level, speed unchanged throughout, filmed from a third-person chase camera rigidly mounted behind and slightly above the vehicle. Ego trajectory every 0.5s0.5 s (t (ss), forward (mm), left (mm)): (0.0, 0.0, 0.0); (0.5, 10.0, 0.0); (1.0, 20.0, 0.0); (1.5, 29.9, 0.5); (2.0, 39.8, 1.9); (2.5, 49.6, 3.1); (3.0, 59.6, 3.4); (3.5, 69.6, 3.5); (4.0, 79.5, 3.5); (4.5, 89.5, 3.5); (5.0, 99.5, 3.5)."

All numerical values are computed by a script from the CarSim ground-truth vehicle states recorded at 30 Hz. To keep the caption within the input length limit of the text encoders, the ego trajectory is down-sampled from these states to 2Hz2 Hz (every 0.5s0.5 s). The driving world models' inputs are down-sampled in the same way, so that all models receive the same trajectory.

Accordingly, the nine CDC responses measure two different abilities, which we reported and analyzed separately in in the new subsection “CDC Analysis by Input Visibility” in Appendix A.4.4 according to whether the response information is supplied in the inputs. Braking distance, braking duration, and lateral acceleration, together with their changes between the two clips of a matched pair, can be approximately recovered from the trajectory given to all models, and CDC on these three responses measures whether the generated motion follows the supplied trajectory across conditions. The six yaw, pitch, and roll responses are not given to any model: the driving inputs contain no heading, pitch, or roll, and the captions describe these motions only qualitatively, with identical wording within a matched pair, so CDC on these six responses measures whether the model predicts the vehicle response to the changed condition. The split distinguishes trajectory-conditioned consistency from the consistency of response values not explicitly supplied. DrivingWorld ranks first in both groups. CogVideoX ranks fourth on visible responses but eleventh on invisible responses. GEM moves from fifth to third, and Vista from tenth to fifth. These differences show that the aggregate CDC ranking can conceal different performance across the two response groups. They describe generated behavior under the supplied conditioning and do not establish a model's internal physical mechanism.

The general and physical world models were post-trained and tested with the captions described in (2), which allowed them to learn the correspondence between the ego trajectory in the caption and the ego motion in the video, and the driving world models were conditioned on the trajectory described in (1). All models receive the same motion information and no additional post-training was performed for this rebuttal.

[Q3] How is metric scale recovered from monocular estimates, and is the reference trajectory used for alignment? How sensitive are FCCR scores and rankings to filtering, differentiation, and tolerance settings?

[A3] We added feature extraction details to Section 3.1 of the main paper. We first fit a ground plane to the monocular DPVO point cloud using RANSAC. We estimated the distance h^\hat h from the first camera center to that plane and multiply the trajectory translations by 2.5m/h^2.5 \mathrm{m}/\hat h, using the calibrated camera height. We then fit a per-clip Sim(3) transformation to the time-matched positions along the full CARLA reference trajectory. This second step estimates rotation, translation, and a scale factor. The velocities and accelerations used by FCCR inherit that fitted scale. Additionally, We used Sim(3) alignment to compare monocular trajectories, following the DPVO evaluation protocol. The fitted scale also rescales the acceleration used by FCCR. The reported FCCR reflects friction consistency after reference-based calibration. Aligning the estimated trajectory with the reference trajectory is necessary.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Chairs and Reviewers

General Response

We thank all the reviewers for their constructive feedback. Here, we summarize our responses to the shared concerns and the main revisions to the manuscript.

Shared Concerns

  • Accuracy of Trajectory-Reconstruction Methods (Phih, UFVM, p19W). We added a_x RMSE in emergency-braking scenarios and a_y RMSE in single-lane-changing scenarios. We tested the trajectory-reconstruction methods in one real-world dataset KITTI Odometry. We also investigated the robustness of VehDyn model rankings by comparing three trajectory-reconstruction pipelines. We further generated more aggressive driving behaviors in simulation to get larger attitude responses and evaluated the estimation errors.

  • Inputs of Different World Models (Phih, p19W). We clarified the specific inputs of driving world models, general world models, and physical world models. Separately, we evaluated CDC with input information and CDC without input information. These comparisons examine the different abilities of motion-following, context-understanding and dynamics-prediction. We then reported the rankings of the zero-shot models.

  • Parameter Sensitivity Analysis (Phih, p19W). We evaluated 22 parameter-perturbation settings on FCCR across 12 models. We also tested the sensitivity of meaningful-change threshold, acceleration-response and differentiation perturbations on CDC across 12 models.

Main Revisions to the Manuscript

  • Section 3.1: We added some details about the trajectory-reconstruction method (Phih, UFVM, p19W).

  • Section 4.1: We supplemented the specific inputs of different world models and the data split strategy (Phih, p19W).

  • Section 4.2: Feature Extraction Analysis. We added the comparison of trajectory-reconstruction pipelines on the VehDyn dataset and KITTI Odometry (Phih, UFVM, p19W).

  • Appendix A.4.1: We tested the robustness by comparing three trajectory-reconstruction pipelines, comparing original WorldScore and DrivingGen benchmark scores, and adding more aggressive driving behaviors (Phih, UFVM, p19W).

  • Appendix A.4.3 & A.4.4: We reported the VehDyn score and complementary visual evaluation score of zero-shot models, and separated CDC with input information and without input information (Phih, p19W).

  • Appendix A.5: We evaluated different perturbations on FCCR and CDC across 12 models (Phih, p19W).

Detailed explanations and quantitative results are provided in our response to each reviewer.

Dear Reviewers,

The rebuttal deadline is approaching. I am leaving this friendly reminder to encourage active discussion. Thank you for your valuable time and engagement!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.