- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VeRA, a world modelling method that first learns physical simulation and then use a renderer for RGB rendering. The authors first train a object-centric simulator using geometric and visual features to output accelerations. Then it uses point splatting or video model for synthesizing the final videos. Experiments on physical-parameter generalization, video generations and robot tasks are performed.
Critical strengths
- Decouping simulation from rendering is reasonable and sound.
- The evaluations are comprehensive, the authors evaluates unseen physical parameters, object counts, and rollout lengths, etc.
Critical weaknesses
- The overall pipeline largely follows prior works PhysCtrl, which trains a trajectory transformer first and use a video model for rendering. I didn't see clear difference in terms of high-level idea.
- The visual results are quite limited, the videos in the supp show noticeable blurrness and artifacts.
- Figure 9 shows that the diffusion renderer can violate predicted trajectories, but the paper does not clearly distinguish evaluation of simulation and rendering in its physics benchmarks.
Questions for authors
Are the PISA-Bench and WorldBench results computed from predicted trajectories, point-splatted videos, or diffusion-rendered videos?
2 comments
Dear Reviewer myD9,
Thank you for your thoughtful review and comments on improving our work. As the discussion period comes to a close, we wanted to kindly follow-up to ask whether our response has addressed your comments and questions. We have responded with additional details on how our work differs from PhysCtrl, details on visual quality limitations and evaluation procedures, and an evaluation of whether our generated videos match the predicted trajectories. If any details of our work remain unclear or if you have any further questions or comments, we would be very grateful for the opportunity to discuss them with you before the end of the discussion period.
Thank you for your time and thoughtful consideration!
Thank you for taking the time to review our work and for your thoughtful comments and feedback. We have addressed them individually below:
The overall pipeline is similar to PhysCtrl. What are the key differences?
While our work tackles a similar problem to PhysCtrl, there are a few key differences which set our work apart:
Visual quality is limited
While artifacts and blurriness in the final videos are undesirable and affect the quality of the outputs, they are a result of the quality of the DiffusionAsShader model which we use without modification in this work. Additionally, Table 1b shows that videos from our model outperform prior video models such as Cosmos 3 and Wan 2.2 on video quality metrics (PSNR, LPIPS, FVD).
There is no evaluation to show rendered videos accurately represent the predicted trajectories
Thank you for raising this important point. In order to judge how well generated videos follow the input trajectories, we run CoTracker3 on generated DiffusionAsShader videos and then compare the output tracks with the 2D projections of the input 3D trajectories and calculate the pixel space error. On the synthetic eval set, which covers both our generated data and data taken from Physion, we find the error is 8.3 ± 4.1 pixels (1 std dev), which tends to be a fraction of the overall motion of each trajectory. In addition, we perform a test on the set of friction variation videos: for each video, we use CoTracker3 and DepthAnything3 to estimate the 3D trajectory of the objects in each video. We can then compare the video stopping distance with the input trajectory stopping distance to track whether the generated videos accurately match the input trajectories. The results are shown below. Across all 10 friction levels, the video distance deviates by an average of 2.71 centimeters and tracks the input trajectory closely, showing that the final videos do preserve predicted trajectories.
Are PISA-Bench and WorldBench results from trajectories, point-splats, or diffusion videos?
The PISA-Bench and WorldBench results are all computed from DiffusionAsShader videos using the estimation techniques from the original works.
Please let us know if you have any further questions. Thank you again for taking the time to review our work!