- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PhyGenesis, a driving video world model designed to handle challenging and physically implausible trajectory conditions. Specifically, a physical condition generator converts invalid trajectories into corrected motions, while a physics-enhanced video generator synthesizes multi-view videos conditioned on the rectified trajectories. The paper also constructs a physics-rich dataset using CARLA, incorporating collision and off-road scenarios to improve the diversity of training data. Extensive experiments demonstrate performance improvements, particularly in challenging scenarios.
Critical strengths
-
The problem is relevant and clearly motivated. Models trained predominantly on nominal driving data may struggle with imperfect trajectory inputs. Decoupling trajectory correction from video generation provides a reasonable approach to addressing this issue.
-
The extensive experiments, ablation studies, and qualitative results provide useful insights into the contributions of the proposed dataset and individual modules.
Critical weaknesses
-
The paper lacks discussion and comparison with some closely related recent approaches. For instance, GenieDrive [a] also investigates physics-aware driving world modeling. A comparison with GenieDrive, at least at the conceptual or methodological level, would help clarify the novelty and advantages of the proposed physics-enhanced video generator.
[a] GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation
-
The demonstrated rectification capability is narrower than the claimed applicability to arbitrary challenging trajectories. The paper constructs corrupted trajectories by extrapolating pre-collision velocity after impact, and uses the same construction for the main CARLA evaluation. It remains unclear whether the proposed method generalizes to other types of planning errors or complex interaction scenarios. Furthermore, correcting an input trajectory inherently modifies the behavior being simulated. The paper should clarify how the method balances physical plausibility with preservation of the intended maneuver, particularly given its potential applications in planning and safety evaluation.
Questions for authors
-
How do the video generator baselines perform when conditioned on the same trajectories rectified by the physical condition generator? This comparison would help isolate improvements from trajectory correction versus those from the proposed video generator.
-
Since video generator is trained with ground-truth trajectories but receives predicted rectified trajectories at inference, how sensitive is its performance to trajectory prediction errors? Would training with predicted or perturbed trajectories improve robustness?
4 comments
Q1. Does the same trajectory correction help other video generators?
We compared extrapolated 2D trajectories with their PCG-corrected counterparts on the same 150 CARLA-Ego clips, keeping the stylized first frames and seed fixed for each baseline. Given the rebuttal time limit, we generated one sample per clip per condition rather than the four used in the main experiments.
As the table shows, PCG leaves FID nearly unchanged but improves FVD across all three baselines, consistent with motion closer to the recorded CARLA interactions. Adding PCG does not consistently improve PHY across the baselines. One likely reason is that collisions are rare in their training data, so they can struggle to render vehicle contact even with physically plausible trajectories. Figure 9, added to Section 4.3 of the revised manuscript, illustrates this limitation under ground-truth collision trajectories.
These results support the complementary roles of our two components: PCG corrects the motion conditions, while training on simulated physical interactions teaches the video generator to render the resulting dynamics. The benefits of this training are also illustrated in Figure 8 and the component ablations in Table 4.
| Backbone | FID ↓ | FVD ↓ | Front-view PHY ↑ |
|---|---|---|---|
| MagicDrive-V2 | 49.13 → 49.10 | 234.07 → 214.02 | 54.33% → 52.17% |
| DiST-4D | 20.31 → 20.36 | 196.82 → 149.47 | 41.50% → 40.33% |
| UniMLVG | 39.76 → 40.10 | 280.12 → 258.07 | 48.67% → 52.83% |
| PhyGenesis | 10.73 → 10.05 | 132.21 → 86.04 | 63.33% → 69.17% |
Arrows show the change from extrapolated inputs to PCG-corrected inputs for each generator. We compare changes within each row.
Q2. How sensitive is the video generator to trajectory prediction errors?
Thank you for the suggestion. We tested lateral trajectory perturbations on 150 CARLA-Ego clips. Given the rebuttal time limit, we generated one sample per clip for each perturbation level rather than the four samples per clip used in the main experiments. At 0.30 m, all three metrics remain close to their unperturbed values. Even at 1.61 m, FID changes little (10.56 to 10.87) and PHY shows a modest decline (71.00% to 67.33%). FVD rises from 71.09 to 82.08, likely because the shift in ego motion changes how the scene moves across frames relative to the original reference videos. Training with perturbed ego trajectories may improve robustness. However, shifting the ego trajectory can move vehicles away from their annotated positions in the original camera views. Training on such mismatched pairs could weaken trajectory control, so we leave training with matched video and box annotations to future work.
Thank the authors for the detailed response. My concerns have been addressed with the revised claim and additional experiments. I will keep my previous score.
Thank you again for taking the time to share your detailed comments and suggestions. We truly appreciate it.
W1. How does PhyGenesis differ from GenieDrive?
PhyGenesis and GenieDrive introduce physical information through different intermediate representations. GenieDrive predicts future 4D occupancy from ego motion to guide video generation, while PhyGenesis corrects all vehicles’ input trajectories into 6-DoF motions with PCG. These corrected motions then guide a video generator trained on real driving and simulated physical interactions.
We compared both methods on 150 CARLA-Ego clips with the same stylized first frames and references. Given the rebuttal time limit, we generated one sample per clip for each method, rather than the four samples used in the main experiments. For GenieDrive, we used FlashOCC [1] to estimate initial occupancy and supplied the extrapolated ego trajectory. PhyGenesis achieves lower FID and FVD. GenieDrive’s freedom to predict surrounding-vehicle motion may help avoid collisions and explain its higher PHY than other baselines, though its score remains below PhyGenesis. We appreciate your suggestion to compare with GenieDrive. Section 4.3 now points to Appendix L, where we report and discuss the comparison (Table 8).
[1]: Yu, Zichen, et al. "FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugin." arXiv preprint arXiv:2311.12058 (2023).
W2. Which challenging maneuvers does PCG preserve?
Thank you for pointing out that “arbitrary challenging trajectories” is too broad. We have revised this wording to “arbitrary physics-violating trajectories” in the manuscript to clarify that our goal is to rectify physically infeasible inputs into physically consistent motion. One of the clearest physical violations occurs when a trajectory carries a vehicle through another vehicle or a roadside obstacle after contact. For other challenging maneuvers, it is harder to define when correction is warranted. We therefore focus on collisions, where penetration identifies an infeasible trajectory and the simulator records the post-impact motion that serves as the correction target. We will explore other forms of physically infeasible motion in future work.
We additionally tested 16 nuScenes clips with sustained close encounters or sharp speed changes. These recorded maneuvers are feasible and should be preserved. The comparison below contrasts their preservation with correction of physically infeasible inputs on CARLA-Ego. ADE is measured against the recorded trajectory over 3 s. PCG makes similarly small adjustments to challenging and ordinary nuScenes trajectories (0.18 m and 0.20 m), supporting its ability to preserve feasible motion even in more complex interactions. These adjustments are about 20% of the 1.00 m reduction in trajectory error on CARLA-Ego, where the inputs require correction.