- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Kinema4D, an action-conditioned robotic simulator. Its core idea is to lift robot actions into explicit 3D space, forming a 4D robot trajectory, which is then used as the conditioning signal for a generative model that predicts the resulting dynamics of the environment. The main motivation is to combine precise kinematic grounding of robot actions with flexible generative modeling of environmental responses, rather than leaving both to the generative model. Specifically, the authors build upon the pretrained 4D aware video generation model 4DNeX to generate synchronized RGB and pointmap sequences. They further curate a dataset to support training. Experiments show that Kinema4D achieves strong prediction quality in both simulated and real-world settings. The authors further demonstrate its potential for policy evaluation, policy improvement, and model-based planning.
Critical strengths
The paper presents a clear and well-motivated framework that combines deterministic kinematic control with generative modeling of environmental dynamics. In addition, the authors curate a relatively large 4D robotic interaction dataset and conduct extensive evaluations across multiple datasets. The results demonstrate strong capabilities in predicting both visual appearance and geometric evolution of robot-world interactions.
Critical weaknesses
My major concern is the distinction from closely related 3D world models such as PointWorld$^{[1]}$ and ParticleFormer$^{[2]}$. These works already lift robot actions into explicit 3D trajectories and directly predict 3D dynamics, which is conceptually very similar to the proposed method. The main difference appears to be that Kinema4D jointly generates RGB and pointmap sequences using a video-generation backbone. However, this is substantially more expensive than previous works that directly predict the 3D point flow. The paper should better justify this design choice and directly compare against PointWorld-like methods to demonstrate improvements in prediction quality, generalization, or downstream control. Otherwise, it is difficult to assess the novelty and practical advantage over prior work.
Furthermore, the MPC evaluation is limited to a Push-T task. Broader model-based control experiments on more diverse manipulation tasks, ideally including real-world settings, would strengthen the paper.
Overall, I am not fully convinced that the paper shows clear novelty and practical advantage over existing 3D world models. Together with the limited downstream control experiments, these concerns make me lean towards rejection. However, if the authors can provide evidence that the proposed method has clear advantages over prior 3D world models, I would be happy to raise my score.
[1] W. Huang et al., PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation, CVPR 2026.
[2] S. Huang et al., ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation, CoRL 2025.
Questions for authors
How do the authors distinguish Kinema4D from closely related 3D world models such as PointWorld and ParticleFormer? What advantages does jointly generating RGB and pointmap sequences with a video-generation backbone provide over directly predicting 3D dynamics? Furthermore, can the authors provide direct comparisons with these methods?
The current MPC evaluation is limited to a Push-T-style task. Can the authors provide broader model-based control results on more diverse manipulation tasks, ideally including real-world settings?
I am happy to raise my score after these questions are addressed.
5 comments
Thank you for the clarification. My main remaining concern is the positioning of the paper. Is the primary goal to predict accurate 4D dynamics, or to generate precise future video frames? If the former, I believe more direct quantitative comparisons with existing 3D world models, such as PointWorld and ParticleFormer, are necessary to establish whether the proposed approach provides better dynamics predictions, especially given its additional computational cost. If the latter, I think this is also a reasonable and valuable direction, particularly for applications such as visual policy training and evaluation. The authors have already provided extensive comparisons with video generation baselines that support this objective. In that case, I would encourage the authors to clarify this positioning in the introduction, abstract, and related work, emphasizing accurate visual prediction rather than accurate 4D dynamics modeling. This distinction is particularly important given that one of the paper's stated motivations is "Generative 4D modeling of environmental reactions via controllable generation," which suggests that accurate 4D dynamics prediction is a central objective.
I would be happy to reconsider my score if the authors can further clarify the paper's primary objective and position its contributions accordingly.
P.S. I noticed that the best LPIPS score in Table 1 is not bolded. Please correct the formatting.
Response to Reviewer yYjk
Thank you for this helpful suggestion. Our primary objective is accurate action-conditioned visual prediction, with explicit spatiotemporal awareness. We do not claim superior standalone 3D/4D dynamics prediction over PointWorld or ParticleFormer. Instead, we use kinematic robot pointmaps to provide explicit 3D action guidance and jointly predict RGB and pointmap sequences to support visually and geometrically consistent future observations.
We have revised the abstract, introduction, and related work to make this positioning explicit, presenting Kinema4D as a 4D-aware visual prediction framework and narrowing statements that could imply accurate physical dynamics modeling. The central contribution is combining pretrained video generative priors with geometric awareness in both action conditioning and future prediction, supporting applications such as visual policy training and evaluation.
We have also corrected Table 1 to bold the best LPIPS score. Thank you for pointing out this formatting issue.
Response to Reviewer yYjk
We thank the reviewer for the constructive assessment. We address the concerns below.
W1/Q1: Distinction from PointWorld and ParticleFormer; why jointly generate RGB and pointmaps?
Motivation. We combine the priors of large pretrained video generative models with explicit 3D awareness in both action conditioning and environment generation. Predicting environmental responses requires appearance, motion, contact, and partially observed geometry. We reuse visual and temporal knowledge while grounding the intended robot motion through kinematics.
Why pointmaps? A pointmap stores XYZ on an image-aligned grid, matching the spatial layout and spatiotemporal processing of our 4D-aware video backbone. We encode the robot's kinematic trajectory as a pointmap sequence and jointly generate RGB and world pointmaps. The control specifies where the robot moves in 3D; the output describes how the environment changes in appearance and geometry. This unifies control and generation in a shared 4D representation, combining geometric guidance with generative flexibility.
Table Y1. Inference interfaces and modeling choices. Checks indicate a feature present or required; dashes indicate otherwise.
PointWorld and ParticleFormer model observed 3D points. Kinema4D's environmental observation is one RGB image; supplied geometry concerns only the robot. Robot conditioning still requires geometry, kinematics, and camera calibration. ParticleFormer's stereo acquisition does not imply multiple independent cameras. Sources: PointWorld, ParticleFormer.
Generalization. In our tests, PointWorld generalized poorly to scenes outside its training distribution, struggling to predict unfamiliar interactions reliably. This supports our motivation to combine video generative priors with explicit 3D action conditioning.
Table Y2. Output ablation from submission Table 4.
Joint modeling improves PSNR by 2.43 dB and reduces CD-L1 by 32.7%. RGB supports visual policies and value models, while synchronized pointmaps provide geometric predictions. We acknowledge the additional computation; our design targets applications requiring both visual and geometric outcomes under explicit 3D control.
W2/Q2: Broader model-based control, including real-world tasks.
Push-T establishes a planning use case: mean executed IoU is 0.983 for Kinema4D, 0.961 for IRASim, and 0.637 without ranking, over 100 trials. Cube grasping would test another contact regime.
Table Y3. Hypothetical cube-grasping comparison, not measured results. Counts, Wilson intervals, and timing below are illustrative; Kinema4D rows use proposed accelerated variants.
At each decision, the proposal policy supplies K=50 candidates. Controls are constructed before execution; only the selected segment is executed, followed by a fresh observation and replanning. Offline training recordings are not candidate outcomes available to the planner.
The comparison will match initial states, proposal policy, sampling, candidate budget, time limits, and training data. RGB planners will share a frozen task-specific value model R. The lift-and-hold success criterion's height, duration, and rater protocol remain pending confirmation; adaptation and repeated-scene handling will be disclosed. We will report successes/50, Wilson 95% intervals, paired differences for matched resets, and full latency. Broader manipulation and real-time claims depend on these results.