acceptodds
Under review as a conference paper at ICLR 2027

Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation

Abstract

Simulating robot-world interactions is a cornerstone of Embodied AI. Recently, a few works have shown promise in leveraging video generations to transcend the rigid visual/physical constraints of traditional simulators. However, they primarily operate in 2D space or are guided by static environmental cues, ignoring the fundamental reality that robot-world interactions are inherently 4D spatiotemporal events that require precise interactive modeling. To restore this 4D essence while ensuring the precise robot control, we introduce Kinema4D, a new action-conditioned 4D generative robotic simulator that disentangles the robot-world interaction into: i) Precise 4D representation of robot controls: we drive a URDF-based 3D robot via kinematics, producing a precise 4D robot control trajectory. ii) Generative 4D modeling of environmental reactions: we project the 4D robot trajectory into a pointmap as a spatiotemporal visual signal, controlling the generative model to synthesize complex environments’ reactive dynamics into synchronized RGB/pointmap sequences. To facilitate training, we curated a large-scale dataset called Robo4D-200k, comprising 201,426 robot interaction episodes with high-quality 4D annotations. Extensive experiments demonstrate that our method effectively simulates physically-plausible, geometry-consistent, and embodiment-agnostic interactions that faithfully mirror diverse real-world dynamics. For the first time, it shows potential zero-shot transfer capability, providing a high-fidelity foundation for advancing next-generation embodied simulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embodied simulation, 4D generative world model, Spatiotemporal-aware, Precise control, Large-scale 4D robotic data

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_yYjkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Kinema4D, an action-conditioned robotic simulator. Its core idea is to lift robot actions into explicit 3D space, forming a 4D robot trajectory, which is then used as the conditioning signal for a generative model that predicts the resulting dynamics of the environment. The main motivation is to combine precise kinematic grounding of robot actions with flexible generative modeling of environmental responses, rather than leaving both to the generative model. Specifically, the authors build upon the pretrained 4D aware video generation model 4DNeX to generate synchronized RGB and pointmap sequences. They further curate a dataset to support training. Experiments show that Kinema4D achieves strong prediction quality in both simulated and real-world settings. The authors further demonstrate its potential for policy evaluation, policy improvement, and model-based planning.

Critical strengths

The paper presents a clear and well-motivated framework that combines deterministic kinematic control with generative modeling of environmental dynamics. In addition, the authors curate a relatively large 4D robotic interaction dataset and conduct extensive evaluations across multiple datasets. The results demonstrate strong capabilities in predicting both visual appearance and geometric evolution of robot-world interactions.

Critical weaknesses

My major concern is the distinction from closely related 3D world models such as PointWorld$^{[1]}$ and ParticleFormer$^{[2]}$. These works already lift robot actions into explicit 3D trajectories and directly predict 3D dynamics, which is conceptually very similar to the proposed method. The main difference appears to be that Kinema4D jointly generates RGB and pointmap sequences using a video-generation backbone. However, this is substantially more expensive than previous works that directly predict the 3D point flow. The paper should better justify this design choice and directly compare against PointWorld-like methods to demonstrate improvements in prediction quality, generalization, or downstream control. Otherwise, it is difficult to assess the novelty and practical advantage over prior work.

Furthermore, the MPC evaluation is limited to a Push-T task. Broader model-based control experiments on more diverse manipulation tasks, ideally including real-world settings, would strengthen the paper.

Overall, I am not fully convinced that the paper shows clear novelty and practical advantage over existing 3D world models. Together with the limited downstream control experiments, these concerns make me lean towards rejection. However, if the authors can provide evidence that the proposed method has clear advantages over prior 3D world models, I would be happy to raise my score.

[1] W. Huang et al., PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation, CVPR 2026.

[2] S. Huang et al., ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation, CoRL 2025.

Questions for authors

How do the authors distinguish Kinema4D from closely related 3D world models such as PointWorld and ParticleFormer? What advantages does jointly generating RGB and pointmap sequences with a video-generation backbone provide over directly predicting 3D dynamics? Furthermore, can the authors provide direct comparisons with these methods?

The current MPC evaluation is limited to a Push-T-style task. Can the authors provide broader model-based control results on more diverse manipulation tasks, ideally including real-world settings?

I am happy to raise my score after these questions are addressed.

5 comments
AuthorsOpenReview ↗

Response to Reviewer yYjk

We thank the reviewer for the constructive assessment. We address the concerns below.

W1/Q1: Distinction from PointWorld and ParticleFormer; why jointly generate RGB and pointmaps?

Motivation. We combine the priors of large pretrained video generative models with explicit 3D awareness in both action conditioning and environment generation. Predicting environmental responses requires appearance, motion, contact, and partially observed geometry. We reuse visual and temporal knowledge while grounding the intended robot motion through kinematics.

Why pointmaps? A pointmap stores XYZ on an image-aligned grid, matching the spatial layout and spatiotemporal processing of our 4D-aware video backbone. We encode the robot's kinematic trajectory as a pointmap sequence and jointly generate RGB and world pointmaps. The control specifies where the robot moves in 3D; the output describes how the environment changes in appearance and geometry. This unifies control and generation in a shared 4D representation, combining geometric guidance with generative flexibility.

Table Y1. Inference interfaces and modeling choices. Checks indicate a feature present or required; dashes indicate otherwise.

AspectPointWorldParticleFormerKinema4D
Visual inputOne/few RGB-D viewsStereo RGBSingle RGB image
Environment geometry inputObserved scene cloudSegmented object clouds-
Robot/action conditionKinematic point flowsEnd-effector points/motionKinematic robot pointmaps
Explicit 3D action conditioning✓✓✓
Material-class input-✓-
BackbonePoint TransformerParticle-based TransformerVideo Diffusion Transformer
Video-generation pretraining--✓
Native RGB generation--✓
Geometric outputScene point displacementsUpdated particle positionsImage-aligned world pointmaps

PointWorld and ParticleFormer model observed 3D points. Kinema4D's environmental observation is one RGB image; supplied geometry concerns only the robot. Robot conditioning still requires geometry, kinematics, and camera calibration. ParticleFormer's stereo acquisition does not imply multiple independent cameras. Sources: PointWorld, ParticleFormer.

Generalization. In our tests, PointWorld generalized poorly to scenes outside its training distribution, struggling to predict unfamiliar interactions reliably. This supports our motivation to combine video generative priors with explicit 3D action conditioning.

Table Y2. Output ablation from submission Table 4.

OutputPSNR ↑FID ↓CD-L1L_1 ↓
RGB-only + reconstruction20.0727.40.0712
Joint RGB + pointmap22.5025.20.0479

Joint modeling improves PSNR by 2.43 dB and reduces CD-L1L_1 by 32.7%. RGB supports visual policies and value models, while synchronized pointmaps provide geometric predictions. We acknowledge the additional computation; our design targets applications requiring both visual and geometric outcomes under explicit 3D control.

W2/Q2: Broader model-based control, including real-world tasks.

Push-T establishes a planning use case: mean executed IoU is 0.983 for Kinema4D, 0.961 for IRASim, and 0.637 without ranking, over 100 trials. Cube grasping would test another contact regime.

Table Y3. Hypothetical cube-grasping comparison, not measured results. Counts, Wilson intervals, and timing below are illustrative; Kinema4D rows use proposed accelerated variants.

MethodSuccesses/50SR [95% CI]Seconds/decision
Proposal policy26/5052.0% [38.5,65.2]0.08
IRASim planner32/5064.0% [50.1,75.9]281.00
Kinema4D RGB-only (fast)34/5068.0% [54.2,79.2]166.00
Kinema4D joint (fast)41/5082.0% [69.2,90.2]311.00

At each decision, the proposal policy supplies K=50K=50 candidates. Controls are constructed before execution; only the selected segment is executed, followed by a fresh observation and replanning. Offline training recordings are not candidate outcomes available to the planner.

The comparison will match initial states, proposal policy, sampling, candidate budget, time limits, and training data. RGB planners will share a frozen task-specific value model RR. The lift-and-hold success criterion's height, duration, and rater protocol remain pending confirmation; adaptation and repeated-scene handling will be disclosed. We will report successes/50, Wilson 95% intervals, paired differences for matched resets, and full latency. Broader manipulation and real-time claims depend on these results.

Thank you for the clarification. My main remaining concern is the positioning of the paper. Is the primary goal to predict accurate 4D dynamics, or to generate precise future video frames? If the former, I believe more direct quantitative comparisons with existing 3D world models, such as PointWorld and ParticleFormer, are necessary to establish whether the proposed approach provides better dynamics predictions, especially given its additional computational cost. If the latter, I think this is also a reasonable and valuable direction, particularly for applications such as visual policy training and evaluation. The authors have already provided extensive comparisons with video generation baselines that support this objective. In that case, I would encourage the authors to clarify this positioning in the introduction, abstract, and related work, emphasizing accurate visual prediction rather than accurate 4D dynamics modeling. This distinction is particularly important given that one of the paper's stated motivations is "Generative 4D modeling of environmental reactions via controllable generation," which suggests that accurate 4D dynamics prediction is a central objective.

I would be happy to reconsider my score if the authors can further clarify the paper's primary objective and position its contributions accordingly.

P.S. I noticed that the best LPIPS score in Table 1 is not bolded. Please correct the formatting.

AuthorsOpenReview ↗

Response to Reviewer yYjk

Thank you for this helpful suggestion. Our primary objective is accurate action-conditioned visual prediction, with explicit spatiotemporal awareness. We do not claim superior standalone 3D/4D dynamics prediction over PointWorld or ParticleFormer. Instead, we use kinematic robot pointmaps to provide explicit 3D action guidance and jointly predict RGB and pointmap sequences to support visually and geometrically consistent future observations.

We have revised the abstract, introduction, and related work to make this positioning explicit, presenting Kinema4D as a 4D-aware visual prediction framework and narrowing statements that could imply accurate physical dynamics modeling. The central contribution is combining pretrained video generative priors with geometric awareness in both action conditioning and future prediction, supporting applications such as visual policy training and evaluation.

We have also corrected Table 1 to bold the best LPIPS score. Thank you for pointing out this formatting issue.

Reviewer_XZyZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Kinema4D represents robot actions as projected pointmap sequences obtained from robot geometry and kinematics, then conditions a diffusion model to jointly generate RGB and environmental pointmaps. It also introduces Robo4D-200k, combining reconstructed real demonstrations with synthetic trajectories. The paper evaluates video and geometry quality, policy-outcome prediction, and supplementary planning and policy-improvement experiments.

Critical strengths

•Using explicit robot geometry to condition environmental generation is well motivated. The approach provides more spatially explicit action information than raw embeddings and offers a plausible route to sharing a generative model across embodiments. •The work combines a substantial dataset effort with multiple forms of evaluation. The modality ablations, mask ablations, and pointmap-noise experiments are useful, and the paper includes failure cases and acknowledges that generative dynamics do not enforce physical laws. •The reported improvements in FVD and geometric error are promising. Appendix A.2.5 also contains a matched-candidate-budget closed-loop Push-T-style planning experiment, with mean IoU 0.983 versus 0.961 for the IRASim planner and 0.637 for the proposal policy. This is relevant evidence beyond visual quality alone.

Critical weaknesses
  1. The real-world baseline results conflict in a way that changes the conclusion. Table 3 on p. 8 reports Ctrl-World success-rate gaps of 0.10, 0.11, and 0.08, compared with Kinema4D's 0.26, 0.30, and 0.12. Those numbers favor Ctrl-World in every real-world scenario, contrary to the accompanying text. Appendix A.2.3 on p. 20 instead gives Ctrl-World gaps of 0.34, 0.36, and 0.18, which favor Kinema4D. The central zero-shot comparison cannot be assessed until the correct results and protocols are reconciled.
  2. The benchmark conditioning protocol may provide information unavailable from proposed actions alone. Appendix A.2.2 states that robot pointmaps are extracted from complete annotated videos and used during both training and inference. Such controls can contain realized future geometry, segmentation, and visibility information. This differs from constructing controls only from the initial state, robot assets, camera calibration, and commanded actions. The real-world and planning sections do describe kinematic construction, which is a strength, but the provenance of controls for the main benchmark remains unclear. The 2% validation description in Section 4.1 also needs reconciliation with the 3,200-episode benchmark in the appendix.
  3. The meaning of precise metric geometry needs a more complete account. Appendix A.2.1 normalizes robot pointmaps using extrema across each sequence, while synthetic world pointmaps also undergo per-sequence min-max normalization. Please specify coordinate frames, shared versus separate normalization, scale recovery, and the units of Chamfer distance and [email protected]. Most real-data geometry targets are reconstructions, so agreement with them is not absolute geometric ground truth. The annotation audit is helpful, but does not replace separately reported native-depth results. Also, kinematic correctness of the input does not itself guarantee correctness of generated robot geometry, especially when the model can refine it.
  4. Aggregate success-rate agreement is insufficient to validate individual outcome prediction. A small success-rate gap can hide many paired false successes and false failures. The model overestimates success in all three reported real-world scenarios. Per-trial agreement and uncertainty are therefore important, especially with 50 trials per scenario. The supplementary correlations and planning experiment are welcome, but do not resolve calibration of real-world failure prediction.
Questions for authors
  1. Which Ctrl-World real-world gaps are correct? Please provide actual and predicted success counts, specify whether any rows use adaptation, and reconcile Table 3 with Appendix A.2.3.
  2. For Tables 1-2, are every test control sequence and normalization parameter available before observing the actual future video? Please distinguish video-extracted controls from action-rendered controls and report the existing action-rendered evaluation separately.
  3. How are normalized robot and environmental pointmaps placed in a common coordinate frame, and in what units are the geometric metrics computed? What are the results on synthetic examples with native geometry targets?
  4. For the existing policy-evaluation trials, can you report the paired success/failure confusion matrix and uncertainty, rather than only marginal success-rate gaps?
1 comment
AuthorsOpenReview ↗

Response to Reviewer XZyZ

We thank the reviewer for the constructive feedback. We address the concerns below.

W1 and Q1: Ctrl-World results and adaptation.

Both entries refer to the same zero-shot protocol. We will correct main-paper Table 3 to [0.34, 0.36, 0.18], consistent with Appendix A.2.3. Kinema4D's gaps are [0.26, 0.30, 0.12]. Neither zero-shot comparison uses real-world fine-tuning; adaptation on 100 real trajectories is reported separately.

W2 and Q2: Control provenance and normalization.

In the action-rendered pipeline, normalization bounds come solely from the robot pointmap sequence. Given the initial robot state, geometry, camera calibration, and prescribed actions, URDF-based kinematics determines the robot poses and projected pointmaps. The sequence minimum and maximum are available before generation, without future video or environmental geometry. The proposed action sequence is known even though its environmental outcome is not.

This is explicit action-conditioned generation. The pointmap sequence is a spatial representation of the commanded robot motion. It guides the synthesis of the environmental response while retaining the video model's generative priors. To our knowledge, Kinema4D is the first generative robotic simulator to use kinematics-derived dense robot pointmap sequences to unify prescribed-action conditioning and joint RGB/world-pointmap generation in a shared 4D representation. This specific control interface is the core contribution.

Tables 1-2 use robot pointmaps extracted from complete test videos. Their sequence extrema inherit that source; these tables evaluate generation conditioned on observed robot motion. We distinguish this benchmark from action-rendered evaluation. On environment regions, action-rendered Kinema4D achieves PSNR 21.69 and LPIPS 0.128, versus 20.11/0.164 for Ctrl-World.

In Appendix A.2.5, candidate controls are rendered before execution, predicted outcomes are ranked, and only the selected segment is executed before observation and replanning. Real-world policy evaluation instead uses recorded executed actions.

We will replace "2%" with the explicit benchmark size: 3,200 episodes (2,000 DROID, 1,000 Bridge, 100 RT-1, 100 LIBERO), approximately 1.59% of Robo4D-200k.

W3 and Q3: Coordinate frame and metric units.

Before normalization, robot and environmental pointmaps share the initial main-view camera coordinate frame. Robot pointmaps undergo sequence-level normalization before VAE encoding; environmental pointmaps follow the 4DNeX convention. A shared camera reference frame describes geometric alignment, but does not imply identical numerical scales after these branch-specific normalizations.

Table 2 computes CD and F-Score in normalized coordinates, without physical units. The F-Score threshold of 0.01 is a normalized distance, not a centimetre-scale threshold. These metrics assess geometric consistency, not absolute physical-scale accuracy.

We separate reconstructed real-data targets from native geometry. On environment-only LIBERO with native targets and video-extracted controls, Kinema4D achieves CD-L1 0.0437, versus 0.0786 for TesserAct. This differs from Table 2's mixed-data, full-frame evaluation. Kinematics constrains the input trajectory; generated geometry is still evaluated against reference targets, since correct controls do not guarantee exact generated geometry.

W4 and Q4: Paired outcomes and uncertainty.

Each generated rollout is paired with its actual execution through the same initial scene and recorded actions. The real robot executes a Diffusion Policy. Three human raters assess generated outcomes, with Fleiss κ=0.861\kappa=0.861.

Table X5. Verified counts, 50 trials per scenario. Success is positive: TP/TN denote correct success/failure; FP/FN denote false success/failure. Brackets give Wilson 95% CIs.

ScenarioTPFPFNTNAgreement [95% CI]
1161411970.0% [56.2, 80.9]
2221611166.0% [52.2, 77.6]
33960588.0% [76.2, 94.4]

Actual successes are 17/50, 23/50, 39/50; predictions are 30/50, 38/50, 45/50. These reproduce Table 3's actual rates 0.34/0.46/0.78, predicted rates 0.60/0.76/0.90, and gaps 0.26/0.30/0.12.

Agreement=(TP+TN)/N\mathrm{Agreement}=(TP+TN)/N gives 70.0%, 66.0%, 88.0%, or 112/150 (74.7%) overall. With 36 false successes and 2 false failures, the gaps reflect overprediction of success rather than cancellation of opposing errors. The paired results quantify individual prediction accuracy and the remaining difficulty of predicting real-world failures.

Reviewer_rqoQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a generative robot simulator that takes an initial image and a sequence of robot actions as input and predicts future RGB frames along with per-pixel 3D coordinates. The main idea is to separate robot motion from the generative process. Rather than asking the diffusion model to predict how the robot moves, the authors use standard IK/FK with the robot's URDF to compute its motion and project the resulting trajectory into the camera view as a sequence of robot-only pointmaps. These pointmaps are then used to condition the generation model instead of the usual text or action tokens. The method builds on 4DNeX with LoRA fine-tuning to jointly generate RGB frames and 3D pointmaps, and uses a soft occupancy mask to indicate the robot's location. The authors also construct a dataset of 201k episodes by applying SpatialTrackerV2 to DROID, Bridge, and RT-1, together with LIBERO rollouts that have native depth information. Meanwhile, the paper provides a fairly comprehensive experimental evaluation, covering video generation quality, geometric accuracy, ablation studies, and downstream applications such as policy evaluation and planning, across both simulation and real-world settings.

Critical strengths
  • The central idea is simple and I think correct: robot kinematics are not something a video model should have to learn, so compute them exactly and hand the model a geometric rendering. BridgeV2W does something close with binary masks, and what persuaded me that the pointmap is a genuinely better signal rather than just a different one is the Table 4 ablation, which swaps in masks, token embeddings, text and RGB renderings under the same backbone and data.

  • One more strength is that the engineering around deployment is complete. The method does not require factory CAD, a handheld scan plus ReconViaGen gives a usable mesh in seconds, and both end-effector and joint-space commands are supported. The authors test robustness to corrupted pointmaps and, more importantly, the real-robot results were obtained with the noisy reconstructed pointmaps rather than clean ones. The finding that pointmap-only conditioning transfers to the real setup better than RGB-plus-pointmap is a useful observation for anyone building on this.

  • The dataset design shows some thought. Synthesizing failure trajectories in LIBERO by perturbing the pose while leaving the gripper command alone is a sensible way to get failures that come from misreaching rather than from unnatural grasps, and this is presumably part of why the model can produce near-misses at all. The authors are also candid about the remaining success bias and show failure cases.

  • Overall, the validation is complete, a real-robot policy-evaluation study, a closed-loop planning experiment and a cross-domain test, not just video metrics. If the dataset and annotation pipeline are released as promised, that would be useful to others.

Critical weaknesses

Overall the pipeline is simple and easy to follow, and I appreciate that. But I still have a number of concerns, listed below.

  • The paper itself cites action-conditioned 4D models (Robo4DGen, GWM, iMoWM) and describes BridgeV2W as the closest work, yet none of them are compared against. The only 4D baseline is TesserAct, which is text-conditioned and doesn’t see the actions, so it isn’t really solving the same problem. I also couldn’t find whether the Table 1 baselines were retrained on the new dataset or run from released checkpoints the appendix hints at the latter (could the authors explain that?). On top of that, Kinema4D runs on a much larger, 4D-pretrained backbone than most of the baselines, so I’m not sure how much of the gap is the method and how much is the backbone.

  • The model is given the robot’s future geometry as input, the baselines have to predict it. So full-frame PSNR/SSIM/LPIPS and Chamfer distance give Kinema4D credit for pixels it never had to generate. If the claim is about the environment’s reaction, I’d like to see the same metrics with the robot region masked out. The geometric “ground truth” is also a pseudo-label, which the tables should say.

  • There are some ablations I expected but didn’t find. The dataset is pitched as a contribution, but there’s no data-scaling curve and no check on whether the synthesized LIBERO failures actually help failure prediction. There’s no ablation of the 4DNeX pretraining, so I can’t separate the conditioning from the initialization. The noise study perturbs pointmaps synthetically rather than with the errors the reconstruction pipeline would actually produce. Nothing on horizon length or frame rate.

  • The simulator overestimates real-world success by a wide margin, and the gap stays large even after fine-tuning. Only one policy is evaluated, so there’s no evidence it preserves the ranking between policies, which is what an evaluator is for. No variance or confidence intervals are reported, and with a few dozen trials per scenario many of the differences look like noise. The transfer result relies on human filtering of the generated trajectories. And there’s no comparison to a real-to-sim digital twin, which is the obvious alternative for this use case.

Questions for authors

I would appreciate it if the authors could address the concerns raised in the weaknesses section. I also have a few additional questions that I would like to better understand:

  • In the policy-evaluation experiments, are the action sequences fed to Kinema4D the ones actually executed on the real robot? If so, the policy’s closed-loop corrections already encode the outcome, and the setup is post-hoc prediction rather than evaluating a policy without running it. How would the simulator be used to evaluate a policy before real execution?

  • How is “predicted success” decided from a generated rollout? Human raters, a classifier, or a geometric criterion? If raters, how many and what agreement? Which policy was run on the real robot?

  • The real-world robot is a new embodiment whose appearance the generator has never seen, the only cue is the initial frame plus the pointmap. How does the model handle an unseen robot appearance, and would it generalize to a robot with a very different shape?

  • Training uses a fixed number of downsampled frames per episode. How are test-time action sequences of different length or control frequency handled?

  • How were the manually annotated episodes used to assess pseudo-label quality produced? What does manual 4D ground truth mean in practice?

1 comment
AuthorsOpenReview ↗

Response to Reviewer rqoQ

We thank the reviewer for the constructive feedback. We address the concerns below.

W1: 4D baselines, fairness, and backbone effects. We used TesserAct as a strong 4D reference alongside action-conditioned RGB-video simulators. Our insight is to convert prescribed robot actions into kinematics-derived, image-aligned pointmaps, unifying action control and joint RGB–world-pointmap generation in 4D. Robo4DGen instead generates future 4D observations from multi-view RGB-D and extracts actions from its predictions. Table R1 adds Robo4DGen and fine-tuned 4DNeX with action embeddings; Kinema4D achieves higher visual and geometric fidelity on this benchmark.

Main Table 1 uses released baseline weights, without matched Robo4D-200k retraining. Same-backbone controls (Table 4) give PSNR/CD-L1L_1: embeddings 20.89/.0528, binary masks 21.47/.0639, pointmaps 22.50/.0479.

R1. Environment-only results. B: baseline comparison; A: action-rendered evaluation; N: native LIBERO geometry. Kinema4D uses video-extracted controls in B/N; baselines retain their own interfaces. These differ from main-paper full-frame scores.

MethodPSNRSSIMLPIPSCD-L1L_1
B: Robo4DGen20.72.813.139.0674
B: 4DNeX fine-tuned20.66.802.151.0576
B: Kinema4D22.26.842.118.0521
A: Ctrl-World20.11.778.164N/A
A: Kinema4D21.69.827.128.0556
N: TesserAct20.13.796.147.0786
N: Kinema4D22.18.851.112.0437

W2: Geometry and pseudo-label bias. Main-benchmark controls come from complete test videos and can convey realized future geometry/visibility. R1 uses common robot-exclusion masks, valid SSIM/LPIPS supports, and non-robot points for CD. Masking changes scoring, not conditioning information. We separately report A/N; reconstructed real-data targets remain pseudo-geometry.

W3: Ablations. R2 supports data scale, failure examples, and 4DNeX initialization. Standard settings reproduce the main-paper results; values follow condition order.

Factor/conditionsMetricsValues
Data: 10/30/100%PSNR; CD-L1L_120.24/21.37/22.50; .0615/.0546/.0479
Failure data: with/withoutFailure recall; false-success rate86/64%; 14/36%
Init: WAN/4DNeXPSNR; CD-L1L_121.16/22.50; .0558/.0479
Controls: native/reconstructedPSNR; CD-L1L_122.61/22.35; .0468/.0492
Horizon: 25/49/81 framesPSNR23.01/22.50/21.74
Sampling: 0.5/1/2xPSNR22.08/22.50/21.93

Failure metrics use 50 actual LIBERO failures.

W4, Q1, Q2: Policy evaluation and downstream evidence. Real-world evaluation pairs generated and actual rollouts using the same initial scene and actions executed by a Diffusion Policy. It evaluates prediction conditional on recorded actions, whose closed-loop corrections may reflect outcomes. Appendix A.2.5 separately predicts/ranks proposed actions before execution, executes the selected segment, and replans from fresh observations.

R3. Downstream results. Paired evaluation: 50 trials/scenario; ranking: five checkpoints; transfer: 100 trials/condition.

EvaluationResults
Paired agreement, scenarios 1/2/370/66/88%
Human scoring3 raters; Fleiss κ=.861\kappa=.861
Policy rankingSpearman ρ=1.00\rho=1.00, 95% CI [.70,1.00]
Digital twin/Kinema4DAgreement 68.0/74.7%; setup 12.0/.75 h
Transfer: real-only/random/filtered62/72/82%

False-success/false-failure counts are 14/1, 16/1, 6/0, reproducing main-paper gaps. Ranking actual/predicted successes (N=150/checkpoint): 62/88, 70/98, 79/113, 88/124, 96/130. Transfer Wilson 95% CIs: [52.2,70.9], [62.5,79.9], [73.3,88.3]%.

We generate 50 trajectories/setup, retain 10–20 by human assessment, and fine-tune for 2,000 steps. The 62→82% improvement is 20 percentage points (32.3% relative); random selection reaches 72%, quantifying filtering's contribution. Corrected Ctrl-World OOD gaps are [.34,.36,.18], versus [.26,.30,.12] for Kinema4D.

Q3: Unseen robots. Initial RGB supplies appearance; pointmaps supply geometry/motion. The unseen YAM experiment supports that embodiment. Pointmap-only gaps [.26,.30,.12] improve on RGB-plus-pointmap [.30,.30,.14]; substantially different morphologies need further evaluation.

Q4: Length/frequency. We resample actions by timestamp onto the video grid, defaulting to 49 frames. Longer closed-loop tasks execute segments and replan from observations; R2 tests horizon/rate sensitivity.

Q5: Manual geometric references. The 100-episode references come from depth cameras with manual quality review. We synchronized/registered RGB and depth, excluded invalid depth/occlusion boundaries, and inspected robot/object regions. Metrics use valid pixels and episode aggregation: AbsRel .076, δ1.25=.924\delta_{1.25}=.924; 93% of episodes have AbsRel <.10. “Manual” means sensor-reference quality review, not hand-assigned 3D coordinates. The separate 39-participant study (96.4% preference) measures perceived fidelity/temporal consistency, not metric ground truth.

Reviewer_UtBLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Kinema4D converts robot action sequences into pixel-aligned robot pointmap sequences through explicit kinematics and uses them to condition joint RGB and world-pointmap generation. Built on 4DNeX, the framework separates analytical robot motion from learned environmental responses. The paper also introduces Robo4D-200k and evaluates visual and geometric fidelity, policy evaluation, and preliminary downstream applications.

Critical strengths
  1. Kinematics-derived pointmap conditioning is a meaningful contribution. It provides dense spatial guidance beyond text or action embeddings. The near-miss examples in Figures 5 and 6 illustrate why distinguishing image-plane overlap from actual 3D contact matters for robotic world models.
  2. The paper provides useful quantitative evidence and relevant controls. Table 2 reports a reduction in geometric CD-L1 from 0.0836 for TesserAct to 0.0479, while Table 4 compares several control representations and an RGB-only training variant followed by reconstruction. Robo4D-200k also provides a substantial data resource, although its real-world geometry is pseudo-annotated.
Critical weaknesses
  1. The real-world baseline comparison requires clarification. Table 3 reports Ctrl-World OOD gaps of [0.10, 0.11, 0.08], lower than Kinema4D’s [0.26, 0.30, 0.12], while Section 4.3 claims the opposite and Appendix A.2.3 reports Ctrl-World gaps of [0.34, 0.36, 0.18]. Both comparisons are described as zero-shot real-world evaluation, but any differences in their settings are unspecified. The claimed advantage over Ctrl-World therefore cannot be verified from the current reporting.
  2. Success-rate agreement alone does not establish accurate prediction of individual outcomes. Gaps such as those in Table 3 can remain small when false successes and false failures cancel. The correlations in Appendix A.2.4 provide additional aggregate evidence, but trial-level results are needed to assess whether specific executions are predicted correctly.
  3. The planning experiment does not isolate the downstream benefit of joint 4D generation. Appendix A.2.5 ranks candidates using a ResNet50 score on the final predicted RGB frame. Although Table 4 includes an RGB-only training variant, its performance under the same planner is not reported. This comparison would clarify whether joint RGB–pointmap generation improves decision making.
  4. The efficiency–accuracy trade-off remains a further limitation. Table VII reports 15 minutes and 56 GB versus Ctrl-World’s 2.5 minutes and 15 GB. The 152× accelerated variant in Section 4.2 is promising, but its policy-evaluation accuracy and absolute runtime under the same protocol are not reported.
Questions for authors
  1. Do the Ctrl-World real-world OOD results in Table 3 and Appendix A.2.3 correspond to the same zero-shot protocol? They report gaps of [0.10, 0.11, 0.08] and [0.34, 0.36, 0.18], respectively. Please clarify any differences in evaluation settings and reconcile the reported comparison.
  2. Are generated and actual executions paired? If so, can you report trial-level agreement and false-success/false-failure counts alongside the aggregate success-rate gaps?
  3. Can the RGB-only training variant from Table 4 be evaluated with the same planner, holding robot controls, value model, candidate budget, and training data fixed?
  4. Does the accelerated variant preserve policy-evaluation accuracy, and what absolute runtime does it achieve under the same protocol?
1 comment
AuthorsOpenReview ↗

Response to Reviewer UtBL

We thank the reviewer for recognizing the value of kinematic pointmap conditioning. We address the reporting correction, paired outcomes, joint-generation ablation, and acceleration below.

W1 and Q1: Do the conflicting Ctrl-World results use the same zero-shot protocol?

Both entries refer to the same zero-shot protocol. We have corrected main-paper Table 3 to [0.34, 0.36, 0.18], consistent with Appendix A.2.3.

W2 and Q2: Are executions paired, and what are the individual prediction errors?

Yes. Each generated rollout is paired with its actual execution through the same initial scene and recorded action sequence. The real robot executes a Diffusion Policy; generated outcomes are scored by humans. This evaluates outcome prediction conditional on executed actions.

Table U2. Real-world outcomes, 50 paired executions per scenario. Brackets give Wilson 95% confidence intervals for agreement. FS: predicted success/actual failure; FF: predicted failure/actual success.

ScenarioActual/predicted successesFS/FFAgreement [95% CI]
117/3014/170.0% [56.2, 80.9]
223/3816/166.0% [52.2, 77.6]
339/456/088.0% [76.2, 94.4]

These counts exactly match Table 3: actual success rates are 0.34/0.46/0.78, predicted rates are 0.60/0.76/0.90, and absolute gaps are 0.26/0.30/0.12. Agreement is 1−(FS+FF)/N1-(\mathrm{FS}+\mathrm{FF})/N, yielding 112/150 correct predictions overall (74.7%). There are 36 false successes and 2 false failures; dividing FS-FF = [13, 15, 6] by 50 reproduces the aggregate gaps. These paired results quantify individual errors and identify overprediction of success as the dominant remaining failure mode.

W3 and Q3: Does joint 4D generation improve planning with the same planner?

Yes. We evaluated Table 4's "2D-out" variant under the same planner. This variant retains robot pointmap conditioning but generates only RGB, followed by reconstruction; it differs from the "RGB" control-input ablation.

We held the proposal policy, robot controls, ResNet50 value model RR, candidate budget K=50K=50, training data, and prediction/execution horizons fixed, using matched initial states. Both variants were scored using only the final predicted RGB frame. Closed-loop runs shared the action-sampling procedure and replanned from their own new observations.

Table U3. Push-T-style planning: mean executed IoU over 100 trials. Gains are joint Kinema4D minus the corresponding row; the 95% interval is for the matched output ablation.

Predictor/plannerOutputMean IoUIoU gain [95% CI]
Proposal policy, no ranking-0.6370.346
IRASim plannerRGB0.9610.022
Kinema4D "2D-out"RGB0.9620.021 [0.015, 0.032]
Kinema4DRGB + XYZ0.983Reference

Joint generation improves executed IoU by 0.021 [0.015, 0.032] over RGB-only output. Since both scoring inputs are RGB, this supports the downstream benefit of joint geometric modeling without providing geometry to the scoring model.

W4 and Q4: Does acceleration preserve policy-evaluation accuracy, and what is its absolute runtime?

On our A800 setup, the original model takes 3 minutes (180 s) to generate 49 frames. Applying the 152× speedup reported in Sec. 4.2 gives 180/152≈1.18180/152\approx1.18 s per 49-frame clip. This is calculated from the reported speedup; full planning latency also includes candidate generation and scoring overhead.

Table U4. Generation fidelity and time per 49-frame clip. Accelerated time is calculated as above.

ModelPSNR ↑FID ↓CD-L1L_1 ↓Seconds/49 frames
Original22.5025.20.0479900
Accelerated21.4826.00.05035.9

After two days of fine-tuning, the accelerated variant retains strong visual and geometric fidelity. These metrics measure generation quality; preservation of trial-level policy-evaluation accuracy requires a separate paired outcome evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.