acceptodds
Under review as a conference paper at ICLR 2027

Toward Physically Consistent Driving Video World Models under Challenging Trajectories

Abstract

Video generation models have shown strong potential as world models for autonomous driving simulation. However, existing approaches are primarily trained on real-world driving datasets, which mostly contain natural and safe driving scenarios. As a result, current models often fail when conditioned on challenging or counterfactual trajectories—such as imperfect trajectories generated by simulators or planning systems—producing videos with severe physical inconsistencies and artifacts. To address this limitation, we propose PhyGenesis, a world model designed to generate driving videos with high visual fidelity and strong physical consistency. Our framework consists of two key components: (1) a physical condition generator that transforms potentially invalid trajectory inputs into physically plausible conditions, and (2) a physics-enhanced video generator that produces high-fidelity multi-view driving videos under these conditions. To effectively train these components, we construct a large-scale, physics-rich heterogeneous dataset. Specifically, in addition to real-world driving videos, we generate diverse challenging driving scenarios using the CARLA simulator, from which we derive supervision signals that guide the model to learn physically grounded dynamics under extreme conditions. This challenging-trajectory learning strategy enables trajectory correction and promotes physically consistent video generation. Extensive experiments demonstrate that PhyGenesis consistently outperforms state-of-the-art methods, especially on challenging trajectories.

Primary area:
generative models
Keywords:
Autonomous driving, video generation, physical AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UA1yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PhyGenesis, a driving video world model designed to handle challenging and physically implausible trajectory conditions. Specifically, a physical condition generator converts invalid trajectories into corrected motions, while a physics-enhanced video generator synthesizes multi-view videos conditioned on the rectified trajectories. The paper also constructs a physics-rich dataset using CARLA, incorporating collision and off-road scenarios to improve the diversity of training data. Extensive experiments demonstrate performance improvements, particularly in challenging scenarios.

Critical strengths
  1. The problem is relevant and clearly motivated. Models trained predominantly on nominal driving data may struggle with imperfect trajectory inputs. Decoupling trajectory correction from video generation provides a reasonable approach to addressing this issue.

  2. The extensive experiments, ablation studies, and qualitative results provide useful insights into the contributions of the proposed dataset and individual modules.

Critical weaknesses
  1. The paper lacks discussion and comparison with some closely related recent approaches. For instance, GenieDrive [a] also investigates physics-aware driving world modeling. A comparison with GenieDrive, at least at the conceptual or methodological level, would help clarify the novelty and advantages of the proposed physics-enhanced video generator.

    [a] GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation

  2. The demonstrated rectification capability is narrower than the claimed applicability to arbitrary challenging trajectories. The paper constructs corrupted trajectories by extrapolating pre-collision velocity after impact, and uses the same construction for the main CARLA evaluation. It remains unclear whether the proposed method generalizes to other types of planning errors or complex interaction scenarios. Furthermore, correcting an input trajectory inherently modifies the behavior being simulated. The paper should clarify how the method balances physical plausibility with preservation of the intended maneuver, particularly given its potential applications in planning and safety evaluation.

Questions for authors
  1. How do the video generator baselines perform when conditioned on the same trajectories rectified by the physical condition generator? This comparison would help isolate improvements from trajectory correction versus those from the proposed video generator.

  2. Since video generator is trained with ground-truth trajectories but receives predicted rectified trajectories at inference, how sensitive is its performance to trajectory prediction errors? Would training with predicted or perturbed trajectories improve robustness?

4 comments

W1. How does PhyGenesis differ from GenieDrive?

PhyGenesis and GenieDrive introduce physical information through different intermediate representations. GenieDrive predicts future 4D occupancy from ego motion to guide video generation, while PhyGenesis corrects all vehicles’ input trajectories into 6-DoF motions with PCG. These corrected motions then guide a video generator trained on real driving and simulated physical interactions.

We compared both methods on 150 CARLA-Ego clips with the same stylized first frames and references. Given the rebuttal time limit, we generated one sample per clip for each method, rather than the four samples used in the main experiments. For GenieDrive, we used FlashOCC [1] to estimate initial occupancy and supplied the extrapolated ego trajectory. PhyGenesis achieves lower FID and FVD. GenieDrive’s freedom to predict surrounding-vehicle motion may help avoid collisions and explain its higher PHY than other baselines, though its score remains below PhyGenesis. We appreciate your suggestion to compare with GenieDrive. Section 4.3 now points to Appendix L, where we report and discuss the comparison (Table 8).

MethodFID ↓FVD ↓PHY (%) ↑
GenieDrive26.12153.5869.00
PhyGenesis9.9085.3270.17

[1]: Yu, Zichen, et al. "FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugin." arXiv preprint arXiv:2311.12058 (2023).

W2. Which challenging maneuvers does PCG preserve?

Thank you for pointing out that “arbitrary challenging trajectories” is too broad. We have revised this wording to “arbitrary physics-violating trajectories” in the manuscript to clarify that our goal is to rectify physically infeasible inputs into physically consistent motion. One of the clearest physical violations occurs when a trajectory carries a vehicle through another vehicle or a roadside obstacle after contact. For other challenging maneuvers, it is harder to define when correction is warranted. We therefore focus on collisions, where penetration identifies an infeasible trajectory and the simulator records the post-impact motion that serves as the correction target. We will explore other forms of physically infeasible motion in future work.

We additionally tested 16 nuScenes clips with sustained close encounters or sharp speed changes. These recorded maneuvers are feasible and should be preserved. The comparison below contrasts their preservation with correction of physically infeasible inputs on CARLA-Ego. ADE is measured against the recorded trajectory over 3 s. PCG makes similarly small adjustments to challenging and ordinary nuScenes trajectories (0.18 m and 0.20 m), supporting its ability to preserve feasible motion even in more complex interactions. These adjustments are about 20% of the 1.00 m reduction in trajectory error on CARLA-Ego, where the inputs require correction.

SettingMagnitude of ego ADE change after PCG (m)PHY after PCG ↑
nuScenes: close encounters / sharp speed changes (16 clips)0.180.984
nuScenes: remaining clips (134 clips)0.200.972
CARLA-Ego: physically infeasible extrapolated trajectories (150 clips)1.000.705

Q1. Does the same trajectory correction help other video generators?

We compared extrapolated 2D trajectories with their PCG-corrected counterparts on the same 150 CARLA-Ego clips, keeping the stylized first frames and seed fixed for each baseline. Given the rebuttal time limit, we generated one sample per clip per condition rather than the four used in the main experiments.

As the table shows, PCG leaves FID nearly unchanged but improves FVD across all three baselines, consistent with motion closer to the recorded CARLA interactions. Adding PCG does not consistently improve PHY across the baselines. One likely reason is that collisions are rare in their training data, so they can struggle to render vehicle contact even with physically plausible trajectories. Figure 9, added to Section 4.3 of the revised manuscript, illustrates this limitation under ground-truth collision trajectories.

These results support the complementary roles of our two components: PCG corrects the motion conditions, while training on simulated physical interactions teaches the video generator to render the resulting dynamics. The benefits of this training are also illustrated in Figure 8 and the component ablations in Table 4.

BackboneFID ↓FVD ↓Front-view PHY ↑
MagicDrive-V249.13 → 49.10234.07 → 214.0254.33% → 52.17%
DiST-4D20.31 → 20.36196.82 → 149.4741.50% → 40.33%
UniMLVG39.76 → 40.10280.12 → 258.0748.67% → 52.83%
PhyGenesis10.73 → 10.05132.21 → 86.0463.33% → 69.17%

Arrows show the change from extrapolated inputs to PCG-corrected inputs for each generator. We compare changes within each row.

Q2. How sensitive is the video generator to trajectory prediction errors?

Thank you for the suggestion. We tested lateral trajectory perturbations on 150 CARLA-Ego clips. Given the rebuttal time limit, we generated one sample per clip for each perturbation level rather than the four samples per clip used in the main experiments. At 0.30 m, all three metrics remain close to their unperturbed values. Even at 1.61 m, FID changes little (10.56 to 10.87) and PHY shows a modest decline (71.00% to 67.33%). FVD rises from 71.09 to 82.08, likely because the shift in ego motion changes how the scene moves across frames relative to the original reference videos. Training with perturbed ego trajectories may improve robustness. However, shifting the ego trajectory can move vehicles away from their annotated positions in the original camera views. Training on such mismatched pairs could weaken trajectory control, so we leave training with matched video and box annotations to future work.

Thank the authors for the detailed response. My concerns have been addressed with the revised claim and additional experiments. I will keep my previous score.

Thank you again for taking the time to share your detailed comments and suggestions. We truly appreciate it.

Reviewer_Y5BmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why existing driving video world models often fail when conditioned on challenging or counterfactual trajectories and proposes PhyGenesis, a novel framework for generating high-quality, physically consistent multi-view driving videos even under such conditions. The authors identify two main reasons for these failures. First, trajectories from simulators or planners can themselves be physically invalid, for example, showing one vehicle passing through another. Current generators tend to follow their conditioning blindly without assessing its physical plausibility, which leads to unrealistic interactions. Second, these generators are typically trained on nominal driving data and have not learned to render rare but physically valid events such as collisions or off-road departures. PhyGenesis addresses both issues using a two-stage pipeline. A Physical Condition Generator first transforms potentially invalid 2D agent trajectories into physically plausible 6-DoF motions. The rectified motions are then projected into per-camera layout conditions for a Physics-Enhanced Multi-view Video Generator, which is trained on a mixture of nominal real-world data and physically challenging scenarios from CARLA and generates the corresponding multi-view videos. The method is compared against prior controllable driving video generators under both physics-violating and ground-truth trajectories. It is evaluated on visual quality, physical consistency, and human preference, with ablations on its two main components.

Critical strengths
  1. The paper addresses an important problem. Generating physically consistent driving videos from imperfect and challenging trajectories is crucial for using world models in simulation and safety testing. Although not demonstrated, the proposed pipeline could potentially be integrated into a closed-loop system, where the physical condition generator corrects implausible trajectories by a planner, and the video generator subsequently renders photorealistic, physically consistent multi-view videos.
  2. Combining real-world nominal data with simulated data covering more challenging scenarios has been shown to improve coverage of safety-critical, long-tail events (ReSim, PhyGenesis). Towards this direction, the authors carefully collected a CARLA dataset, comprising 31 hours of multi-view driving data with challenging trajectories, which could benefit the research community if publicly released.
Critical weaknesses
  1. The selected baselines are not the most relevant comparisons. UniMLVG, MagicDrive-v2, and DiST-4D are general-purpose controllable driving video generators that were not specifically trained to handle physically challenging scenarios. In contrast, PhyGenesis is explicitly trained for this task, so its advantage over them is expected. More relevant comparisons would be with closely related works that also target safety-critical scenario generation, such as SafeMVDrive, Challenger, ReSim, and Ctrl-Crash (even in single-view setup). Evaluating these methods under e.g., the same input trajectories would provide stronger empirical evidence that PhyGenesis generates more physically consistent videos.
  2. The evaluation does not convincingly support the paper’s main claim of improved physical consistency. The original WorldModelBench benchmark evaluates physical adherence (e.g., mass, impenetrability) and commonsense (temporal-level and frame-wise quality) as separate dimensions. However, PhyGenesis combines these metrics into a single PHY score, even though only two of them directly relate to physical consistency. Therefore, it is hard to judge whether improvements in PHY are actually because of better physical adherence or just because of better visual quality. This may explain some counterintuitive results. For example, MagicDrive-v2 and DiST-4D each score higher PHY under physics-violating trajectories (Table 2) compared to ground-truth trajectories (Table 1). In general, a geometric evaluation (e.g., running a multi-view 3D detector and tracker as done in the baselines and computing metrics such as 3D box overlap) would be more reliable compared to a VLM judge.
  3. The controllability evaluation is also not fully convincing. CtrlErr only measures how accurately the ego camera pose follows its predefined trajectory, which suits camera-controlled generators but not a model conditioned on the motion of more than one agent in the scene. Prior methods (e.g., MagicDrive-v2, UniMLVG) evaluate controllability by applying pretrained 3D perception or tracking modes to the generated videos and comparing their predictions against the conditioning layouts using standard 3D detection and tracking metrics. Adopting a similar evaluation protocol would better reflect controllability.
Questions for authors
  1. The paper mentions ReSim’s limitation in controlling multiple agents (L207), but PhyGenesis itself is configured with only N = 2 agents. Can the method scale beyond this number, for example, to simulate more complex, multi-agent traffic scenarios?
  2. The results are limited to a narrow set of scenarios: the ego or another vehicle colliding with the leading car, or driving off-road and then hitting a static obstacle. Can the framework handle challenging trajectories other than crashes? If not, what limits it to these types of crashes?
  3. In the supplementary videos, outputs under counterfactual inputs (e.g., “Nearby accident”, “Ego offroad”, “Ego collusion”) look more like CARLA renderings than photorealistic. Were they generated from style-transferred or raw CARLA input frames? How does PhyGenesis perform on the original CARLA videos without style transfer?
  4. In the training data, collision layouts appear only in CARLA clips, so the generator could learn a shortcut that associates physically challenging layouts with CARLA’s visual domain and dynamics. Did the authors observe a drift towards CARLA-like appearance or motion when stress testing on nuScenes?
6 comments

W1. Comparison with safety-critical video generators

SafeMVDrive and Challenger use trajectory simulators to generate adversarial motions, then render them with UniMLVG and MagicDrive-V2, respectively. Our task starts from a prescribed trajectory, and we already evaluate both video generators under this setting. At the time of evaluation, ReSim had not publicly released its non-expert CARLA training data or the corresponding model weights. We therefore compared PhyGenesis with Ctrl-Crash, which uses sequences of 2D vehicle bounding boxes to control the collision and generates video from a single viewpoint. We evaluated both methods on the front view of the same 150 CARLA-Ego clips using identical stylized first frames, with one sample per clip per condition rather than the four used in the main experiments.

MethodExtrapolated 2D · FID ↓Extrapolated 2D · FVD ↓Extrapolated 2D · PHY ↑Recorded GT · CtrlErr ↓
Ctrl-Crash48.45282.710.8130.182
PhyGenesis22.75151.010.7050.053

PhyGenesis achieves lower FID and FVD than Ctrl-Crash and follows the recorded ego trajectories more accurately. We attribute part of the FID/FVD gap to the poor image quality of Ctrl-Crash’s dashcam training footage, which highlights the value of high-quality simulated collision data. Ctrl-Crash achieves a slightly higher PHY score than PhyGenesis. Its collision training helps it render impacts, while weaker trajectory control can let it miss an intended collision and still pass the deformation and penetration checks.

W2. Physical consistency and visual quality metrics

We average four WorldModelBench items in PHY because a collision can reveal the same failure in both vehicle shape and visual quality. In the upper-left example of Figure 8, the model trained without heterogeneous data renders the car as a blurred, distorted shape at impact. The artifact affects the quality of the frame and makes it difficult to recognize the same vehicle across frames. We therefore include frame quality and temporal consistency alongside the checks for deformation and penetration. PHY summarizes these complementary observations.

Regarding your observation that both baselines score slightly higher on PHY with extrapolated trajectories, we think this outcome is plausible. Both models were trained mainly on nominal driving data, where collisions are rare, so even an accurate trajectory does not guarantee that the generator will render collision realistically. This highlights the importance of including simulated collision data when training the video generator. We added Figure 9 to Section 4.3 of the revised manuscript to show how each model renders the same collision given the ground-truth trajectories. At impact, the lead vehicle fades from view in MagicDrive-V2 and its right rear stretches in DiST-4D, while its outline remains visible in PhyGenesis.

Following your suggestion, we also measure whether a 3D tracker can follow the same car through collision. We used PubTracker with StreamPETR detections to track the collision target in each of the 56 CARLA-Ego clips generated under extrapolated 2D trajectories. The Collision Consistency Score (CCS) is the percentage of clips in which that car retains the same track ID across the two keyframes spanning impact, 0.5 seconds apart. PhyGenesis retained the target car’s ID across impact 12.5 percentage points more often than DiST-4D, the strongest baseline on this measure. This suggests that PhyGenesis preserves the collision target’s visual continuity more reliably across impact.

MethodCCS (%) ↑
UniMLVG17.9
DiST-4D19.6
MagicDrive-V216.1
PhyGenesis32.1

W3. Controllability of surrounding agents

CtrlErr measures how closely the ego camera follows its prescribed trajectory. To assess whether surrounding agents appear where the input layout places them, we add StreamPETR detector to each method. We compared its predicted 3D boxes with the annotated boxes used to condition generation on 900 matched keyframes from 150 nuScenes clips. We report the results in Section 4.3 of the revised manuscript (Table 5).

PhyGenesis scores 2.04 mAP points and 2.00 car AP points higher than MagicDrive-V2, the strongest baseline on both measures. These results complement CtrlErr by evaluating agreement with the object layouts. For temporal consistency across collision, please see the Collision Consistency Score (CCS) results in our response to W2.

MethodmAP ↑NDS ↑Car AP ↑
UniMLVG17.2434.8630.39
DiST-4D24.2938.2737.65
MagicDrive-V227.9439.9443.80
PhyGenesis29.9842.0045.80

Q1. Trajectory control and interactions involving more than two vehicles

ReSim provides trajectory annotations only for the ego vehicle. Its data alone therefore cannot directly supervise a driving world model to follow a specified trajectory for each surrounding vehicle. We constructed our dataset with trajectory annotations for every vehicle. Our PCG takes a 2D trajectory for each vehicle in the scene layout and predicts a corrected 6-DoF trajectory for each one. These trajectories let us control all vehicles in the layout when generating the six camera views.

As a first study of joint trajectory correction and multi-view driving video generation, we focused on pairwise physical interactions. Thus, N=2N=2 means that at most two vehicles participate directly in each physical event we constructed. The PCG can process a variable number of vehicles jointly. We have clarified the scope of our dataset in Sections 3.2 and 4.1. In future work, we will extend our dataset to explore physical interactions involving three or more vehicles and evaluate how well the PCG models their joint motion.

Q2. Challenging trajectories beyond crashes

PhyGenesis focuses on collisions because contact marks a clear physical event and the simulator records the resulting motion. For non-collision maneuvers, it is harder to define when a trajectory becomes physically infeasible and to simulate a plausible outcome, so we leave these cases for future work. From 150 nuScenes clips, we selected eight where the ego vehicle stayed 0.2–2.0 m from the same moving vehicle at two consecutive keyframes and eight others where its absolute longitudinal acceleration reached at least 2.0 m/s². With the original 2D trajectories, mean front-view PHY was 0.984 on these clips and 0.968 on the remaining 134 (0.970 overall). This suggests that PhyGenesis maintains similarly high visual quality and physical consistency in these feasible close encounters and sharp speed changes as in the rest of the evaluation set.

Q3. Conditioning frames and performance on raw CARLA

All three counterfactual videos on our submission page used style-transferred CARLA first frames. To test whether PCG also helps without style transfer, we evaluated PhyGenesis with both raw and style-transferred first frames on the same 150 CARLA-Ego clips. Given the rebuttal time limit, we generated one sample per clip for each setting rather than the four used in the main experiments. As the table below shows, PCG lowered both FID and FVD on raw CARLA inputs and raised PHY from 0.52 to 0.59. We observed the same pattern with style-transferred inputs.

First frame and FID/FVD referenceTrajectory inputFID ↓FVD ↓Front-view PHY ↑
Raw CARLAExtrapolated 2D18.58128.540.52
Raw CARLAPCG-corrected 2D16.4775.510.59
Style-transferred CARLAExtrapolated 2D10.73132.210.63
Style-transferred CARLAPCG-corrected 2D10.0586.040.69

Q4. Appearance and motion under trajectory stress on nuScenes

In our nuScenes stress test, the generated videos retained the photographic appearance of the real first frame, with no visible shift toward CARLA’s game-like visual style. We attribute this in part to fine-tuning the pretrained Wan2.1 image-to-video model, which helps preserve the lighting and textures of the real first frame. In Figure 10 of the original submission, the white van remains recognizable as the ego vehicle approaches it, and the tree-lined road retains the daylight appearance of the first frame. We added Figure 13 to the revised appendix to compare videos generated from the original 2D trajectory and the PCG-corrected stress trajectory using the same real first frame. In both examples, the PCG outputs retain the photographic look of nuScenes seen in the original 2D generations, with no obvious shift toward a game-like CARLA rendering style.

AuthorsOpenReview ↗

Follow up our rebuttal

Dear Reviewer Y5Bm,

Thank you again for your valuable feedback and for taking the time to review our paper.

As the discussion period is approaching its end, with approximately two days remaining, we would like to kindly follow up on our response submitted three days ago. We hope our responses have adequately addressed your questions and concerns.

If there are any remaining concerns or points that require further clarification, we would greatly appreciate your feedback and would be happy to provide additional explanations before the discussion deadline.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_CJ63OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PhyGenesis, a driving world model designed to generate high-fidelity, physically consistent multi-view videos even when conditioned on challenging or physics-violating trajectories (e.g., collisions, off-road). The authors identify two key limitations in existing models: lack of physical awareness for trajectory feasibility, and lack of physics-consistent generation due to training on nominal, safe data. To address this, PhyGenesis proposes a two-stage framework: (1) a Physical Condition Generator that rectifies invalid 2D trajectories into physically plausible 6-DoF motions, and (2) a Physics-Enhanced Multi-view Video Generator (PE-MVGen) built on Wan2.1, which synthesizes the videos. The authors also construct a heterogeneous dataset combining real-world nuScenes data and simulated extreme scenarios from CARLA. Experiments demonstrate that PhyGenesis outperforms baselines in visual quality, physical consistency, and controllability, particularly under challenging trajectory conditions.

Critical strengths
  1. Addressing a Critical Bottleneck: The paper tackles a highly practical and critical problem in driving world models, handling physics-violating trajectories generated by imperfect planners or simulators. This is a significant step toward using world models for closed-loop evaluation and safety-critical scenario synthesis.

  2. Novel Methodological Design and Dataset Curation: The two-stage framework is well-designed. The Physical Condition Generator, equipped with a Time-Wise Output Head to capture sudden collision impulses, is a clever addition. Furthermore, the construction of the heterogeneous dataset and the counterfactual trajectory corruption strategy provide a robust foundation for learning physical priors.

  3. Comprehensive Empirical Validation: The experiments are extensive and convincing. The authors evaluate on three different datasets and use multiple metrics. The results consistently show large improvements over strong baselines, especially on challenging CARLA datasets.

Critical weaknesses
  1. Evaluation of Style Transfer for CARLA: To ensure a fair comparison, the authors use a style-transfer model to translate CARLA clips into the nuScenes visual style for FID/FVD evaluation. While practical, this introduces a synthetic domain gap. The style-transferred "ground truth" might not perfectly reflect true physical dynamics, potentially introducing noise or bias into the visual quality metrics.

  2. Reliance on Controllability Metrics: The controllability error relies on extracting camera poses using ViPE from generated videos. While standard, this adds another layer of potential error, and it would be beneficial to provide more direct geometric consistency metrics or downstream task evaluations to further validate the utility of the generated data.

Questions for authors

See Weakness

3 comments

W1. Style-transferred CARLA references for FID and FVD

Thank you for raising this concern. We use style transfer to bring CARLA references closer to the target visual domain. Since the transfer model saw no collisions during training, it may alter details at impact. We use FID and FVD to assess overall similarity to the target visual domain. Because these scores summarize features across frames or clips, a small artifact confined to the moment of impact may have little effect on either score [1] [2].

We therefore assess physical consistency directly on the generated videos using PHY, complemented by human judgments of overall visual quality that can also reflect visible physical inconsistencies. Both evaluations use only the generated videos, without the style-transferred CARLA references. As Table 2 shows, PhyGenesis achieves the highest scores for both PHY and human preference on CARLA-Ego.

MethodOverall PHY ↑Human preference ↑
UniMLVG0.550.13
MagicDrive-V20.600.06
DiST-4D0.390.10
PhyGenesis0.710.71

[1]: Jayasumana, Sadeep, et al. "Rethinking FID: Towards a Better Evaluation Metric for Image Generation." CVPR 2024.

[2]: Ge, Songwei, et al. "On the Content Bias in Fréchet Video Distance." CVPR 2024.

W2. Geometric consistency metrics and downstream evaluation

Thank you for suggesting a downstream evaluation of geometric consistency. We applied StreamPETR detector to all methods and compared its predictions with the original 3D annotations on 900 matched keyframes from 150 nuScenes clips. This evaluation uses the dataset’s camera calibration and is independent of ViPE pose estimates. Across the configurations shown, PhyGenesis improves over MagicDrive-V2 by 2.04 mAP, 2.06 NDS, and 2.00 car AP points. We added this downstream detection evaluation to Section 4.3 of the revised manuscript (Table 5).

MethodmAP ↑NDS ↑Car AP ↑
UniMLVG17.2434.8630.39
DiST-4D24.2938.2737.65
MagicDrive-V227.9439.9443.80
PhyGenesis29.9842.0045.80

To complement our evaluation of geometric consistency, we introduce the Collision Consistency Score (CCS) to measure whether the same car remains trackable across impact. We used PubTracker with StreamPETR detections to track the collision target in each CARLA-Ego clip. CCS is the percentage of clips in which that car retains the same track ID across the two keyframes spanning collision onset, 0.5 seconds apart. PhyGenesis retained the target car’s ID across impact 12.5 percentage points more often than DiST-4D, the strongest baseline on this measure.

MethodCCS (%) ↑
UniMLVG17.9
DiST-4D19.6
MagicDrive-V216.1
PhyGenesis32.1
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer CJ63,

Thank you again for your valuable feedback and for taking the time to review our paper.

As the discussion period is approaching its end, with approximately two days remaining, we would like to kindly follow up on our response submitted three days ago. We hope our responses have adequately addressed your questions and concerns.

If there are any remaining concerns or points that require further clarification, we would greatly appreciate your feedback and would be happy to provide additional explanations before the discussion deadline.

Thank you again for your time and consideration.

Best regards, The Authors

Dear Reviewer Cj63,

With about three and a half hours remaining in the discussion period, we would appreciate hearing whether our additional results and clarifications have addressed your remaining concerns. If so, we would be grateful if you could consider updating your assessment and score accordingly. If anything remains unclear, we would be happy to clarify.

Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.