acceptodds
Under review as a conference paper at ICLR 2027

KineWorld: Action-Induced Transport Fields for Embodied World Modeling

Abstract

Embodied world models predict the visual consequences of candidate actions before execution. However, existing action-conditioned world models often adopt uniformly weighted visual generation objectives that can be misaligned with embodied prediction needs. Even with explicit motion conditioning, these objectives can underemphasize spatially sparse changes that are critical to interaction. We propose KineWorld, a transport-aware world-modeling framework that extends robot kinematics from motion conditioning to the spatial allocation of generative supervision. Kinematic Transport Lifting (KTL) constructs renderer-derived, camera-aligned transport fields from commanded robot motion. Transport-Aware World Diffusion (TAWD) calibrates their motion support on the video-latent grid and reweights future-RGB flow matching through a normalized mixture of uniform and transport-focused distributions. We train KineWorld using ALOHA-AgileX bimanual manipulation data from RoboTwin 2.0. KineWorld achieves an EWMScore-P of 68.95 in single-view evaluation and a TWB-Score of 54.82 in multi-view evaluation. These results support a shift from appearance fitting toward action-consequence modeling for embodied decision-making.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_opjZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

KineWorld proposes an action-conditioned world model for robotic manipulation, which improves action-conditioned video prediction through two key components. Kinematic Transport Lifting (KTL) converts robot kinematics into a camera-aligned visual transport field, which serves as a video generation condition to guide the prediction of action-relevant regions. Transport-Aware World Diffusion (TAWD) leverages the transport field to spatially reweight the flow matching loss, encouraging the model to focus more on robot motion regions. Experimental results demonstrate that KineWorld significantly improves action responsiveness and video prediction performance compared with the baseline.

Critical strengths
  1. The experimental results are reasonably comprehensive, both the ablation studies and comparison experiments validate the effectiveness of the proposed method. The ablation studies separately evaluate the contributions of KTL and TAWD, demonstrating that introducing the robot-motion transport field and the corresponding loss reweighting mechanism can both improve action responsiveness and future video prediction performance. Meanwhile, compared with baselines such as Wan2.2, KineWorld achieves significant improvements across multiple evaluation metrics, demonstrating the effectiveness of the proposed kinematics-guided mechanism for action-conditioned world models.
Critical weaknesses

1.Lack of validation in real-world robotic scenarios. The current method is trained and evaluated entirely on limited simulation data. The proposed framework heavily relies on accurate robot kinematic models, camera calibration information, and the robot foreground transport fields obtained through rendering. However, in real-world robotic environments, factors such as camera vibration, inaccurate intrinsic/extrinsic calibration, and discrepancies between simulated and real robot appearances make it difficult to obtain such precise foreground optical flow information. Therefore, the effectiveness of the proposed method in real-world settings remains unverified, and its generalization ability for practical robot deployment is unclear.

2.Insufficient fairness in experimental comparisons. The comparisons with existing world model approaches may not be fully fair, making it difficult to draw definitive conclusions about the superiority of KineWorld. Different methods are often trained under different robot embodiments and data distributions. For example, Ctrl-World is primarily trained on the Franka robot, while KineWorld is evaluated on the ALOHA–AgileX platform, and there is no unified training and evaluation protocol across methods. Therefore, directly comparing performance across different embodiments may not accurately reflect the relative capabilities of the models, limiting the persuasiveness of the comparison results.

Questions for authors

1.Generalization to Real-World Robotic Scenarios The current experiments are entirely conducted on the RoboTwin 2.0 ALOHA–AgileX simulation dataset. However, KineWorld heavily relies on accurate robot kinematics, camera calibration, and the rendered robot transport field. In real-world settings, factors such as camera motion, calibration errors, and the gap between simulated and real robot appearance may make it difficult to obtain reliable foreground optical flow and transport fields. It would be important to evaluate whether KineWorld can generalize to real robot scenarios. For example, experiments on real robot platforms or robustness analysis under different levels of calibration and perception errors would help demonstrate whether the proposed transport field remains reliable beyond simulation. 2.Fairness of Experimental Comparisons The comparison with existing world models is not entirely convincing because different methods are evaluated under different robot embodiments and data distributions. For example, Ctrl-World is mainly developed for the Franka robot, while KineWorld is evaluated on ALOHA–AgileX. Therefore, the performance differences may partly come from the choice of embodiment and dataset rather than the proposed method itself. A more meaningful comparison would be against action-conditioned world models trained on the same embodiment. Without such comparisons, it is difficult to isolate the actual contribution of KineWorld from the influence of different robot platforms and datasets. 3.Limited Improvement in Interaction Modeling Both KTL and TAWD derive their supervision signals from the robot-induced transport field. Therefore, the proposed method mainly emphasizes regions corresponding to robot motion. However, in manipulation tasks, successful execution depends more fundamentally on the interaction between the robot and the environment, including object motion, contact dynamics, and physical constraints. It remains unclear whether the robot transport field is a sufficient proxy for interaction-relevant regions. For tasks such as pushing, pulling, or grasping, object motion may be more important than robot motion itself. Additional analysis with object-motion-aware or contact-aware signals would help verify whether KineWorld truly improves interaction modeling rather than only improving robot motion prediction. 4.Definition of World Action Model The paper refers to KineWorld as a World Action Model (WAM), but the proposed method mainly focuses on action-conditioned future video prediction. It does not explicitly address action generation, planning, policy optimization, or closed-loop decision making, which are commonly associated with WAMs. The authors should clarify the distinction between KineWorld, action-conditioned world models, and World Action Models.

1 comment

Dear Reviewer opjZ,

Please respond to the author soon and engage in the discussion.

Best,

AC

Reviewer_eXwdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

KineWorld addresses the spatial allocation of supervision in action-conditioned video prediction. Kinematic Transport Lifting (KTL) rolls out commanded robot configurations, renders the robot, and estimates source-masked RAFT flow to condition a Wan2.2-based generator. Transport-Aware World Diffusion (TAWD) pools the flow magnitude onto the video-latent grid and reweights future-RGB flow matching using a mixture of uniform and transport-focused weights. The weights have unit mean, revert to uniform weighting when motion support is absent, and introduce no additional inference steps.

The paper reports EWMScore-P 68.95 and TWB-Score 54.82, while explicitly noting that these system profiles are not comparisons under common training conditions. The most direct evidence for TAWD is Table 6: two 50-update continuations from the same checkpoint, differing only in the weighting parameter. On 1,000 paired episodes, TAWD improves EWMScore-P by 1.75 (95% paired bootstrap CI [1.10, 2.37]), trajectory accuracy by 5.60, and interaction quality by 4.20. The visual-quality interval includes zero.

Critical strengths
  1. The distinction between providing an action condition and deciding where to allocate supervision is useful. The objective in Eqs. (15)–(17) is straightforward, retains supervision over the full scene, and has an explicit zero-motion fallback. Constructing the weights from the initial robot state and commanded actions also avoids using future RGB to determine where to focus the loss. TAWD can therefore be added to the fixed KTL architecture without increasing inference cost.

  2. Table 6 provides a meaningful controlled comparison. The two branches share their parent checkpoint, update budget, data order, seeds, conditioning, and inference settings. The paired results support a benefit from reweighting in this continuation setting. I also appreciate the supplement's explicit distinction between episode-sampling uncertainty and training-run variability, and its discussion of inconclusive visual-quality gains and object-interaction failures.

Critical weaknesses
  1. The evidence does not yet establish what the proposed weighting adds beyond simpler spatially focused losses. KTL follows a rendering-and-flow conditioning route close to approaches discussed in the paper; the main additional mechanism is a normalized mixture of uniform and motion-weighted RGB losses. Table 6 compares this only with uniform weighting. Without a matched robot-occupancy or binary motion-support weighting baseline, it is difficult to assess the value of the magnitude-based support and its calibration. This matters because the support follows projected robot motion, rather than the full consequences of interaction. Figure A.9 illustrates the gap: the robot moves, but the bottle remains on the table instead of being lifted as in the reference. The reported gains are encouraging, but stronger evidence is needed to connect them to improved object-interaction prediction.

  2. The scope of the controlled evidence is substantially narrower than the broader system claims. Tables 1 and 3 collect heterogeneous system profiles. Table 4 also changes NFE from 50 to 25, so its performance and speed differences do not isolate the effect of the modules. These caveats are disclosed, yet Section 5.3 still draws strong conclusions about complementarity and efficiency. Table 6 isolates TAWD, but only for a single checkpoint pair after 50 updates; its bootstrap intervals cannot establish robustness across training seeds or at convergence. Its separate 1,000-episode evaluation collection is also not the specified 500-episode Clean-50 held-out split. Appendix A.3 describes exact-image/action duplicate screening, but the collection's task composition and relationship to the training data need a clearer, auditable account. Finally, the 3.02 FPS figure excludes transport preparation and counts temporally expanded output frames, so it does not establish end-to-end efficiency for evaluating new candidate actions.

Questions for authors
  1. Please clarify the checkpoint, training-data scope, evaluation collection, and evaluator version behind the entries in Tables 1, 3, 4, and 6. In particular, what is the source and task composition of the 1,000-episode collection, and how does it relate to Clean-50? An anonymized manifest and checkpoint identifiers would help. Is there a comparison underlying Table 4 that holds training and sampling settings fixed apart from the proposed modules?

  2. Do the Table 6 gains persist across independent training seeds and longer continuations? If such results are unavailable, please state this explicitly. Also, when bootstrapping EWMScore-P, is the collection-level JEPA/JEDi component recomputed for each resampled collection, rather than reused as a constant from the original collection? This affects the interpretation of the aggregate-score interval.

  3. Under the same KTL condition, unit-mean weights, and training budget, how does TAWD compare with robot-occupancy or binary motion-support weighting? Are there object-track or contact-phase results showing that the improvement extends beyond reproducing robot motion? Existing evidence on either point would materially affect my assessment.

Reviewer_efmoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KineWorld, an action-conditioned video world model built on Wan2.2. Kinematic Transport Lifting (KTL) rolls out commanded robot actions, renders the robot at successive states, and uses RAFT to estimate camera-aligned optical flow. The resulting transport fields are encoded by the video VAE and provided to the generator through a second token stream with joint attention. Transport-Aware World Diffusion (TAWD) uses transport magnitude to reweight the future-RGB flow-matching loss, placing more emphasis on regions of projected robot motion. The authors evaluate the model on simulated RoboTwin 2.0 bimanual manipulation data, reporting single- and multi-view video metrics and module ablations.

Critical strengths

The motivation is sensible. Changes relevant to robot actions can occupy only a small part of an image, so it is reasonable to question whether uniform RGB supervision gives them enough weight. The proposed loss weighting is straightforward, and the normalization avoids changing the total loss weight. The matched TAWD comparison in Table 6 is useful. Starting from the same checkpoint and keeping the other settings fixed, TAWD improves several action-related metrics. This is positive evidence for the weighting idea, even though the size and generality of the improvement still need to be established.

Critical weaknesses
  1. The experiments do not establish the value of the specific transport construction. KTL uses a robot rollout, rendering, and RAFT to obtain an optical-flow field. Since the simulator already has robot geometry and calibrated cameras, a natural baseline would be motion vectors computed directly from geometry or renderer output. Simpler conditions such as robot masks or projected end-effector trajectories would also be informative. The reported KTL comparison with wrist-flow placeholders does not answer whether the full RAFT-based transport field is needed. This matters because KTL accounts for most of the gain in the main ablation.
  2. The evidence for physical action-consequence prediction is limited. Evaluation is performed on simulated demonstrations, without real-robot or downstream policy tests. More importantly, the transport field describes projected robot motion, not object motion or contact dynamics. Video and trajectory metrics alone do not show whether the model correctly predicts outcomes such as picking up or moving an object. The cross-system scores are also reference profiles rather than matched evaluations. In Table 4, the base Wan2.2 and KTL variants use different denoising evaluation counts; in Table 6, the well-controlled TAWD result comes from a 50-update continuation. These choices make it harder to judge the size and robustness of the improvement.
Questions for authors
  1. Can the authors compare KTL with geometry-derived motion fields, robot masks, or projected end-effector trajectories under otherwise matched conditions? Which part of the proposed representation drives the improvement?
  2. Can Table 4 be repeated with matched denoising evaluation counts? Do the TAWD gains persist after longer training and across independent training seeds?
  3. Is there a quantitative evaluation of object-level outcomes, such as object displacement, grasp success in predicted rollouts, or contact consistency? Such results would help support the claimed improvement in action-consequence modeling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.