acceptodds
Withdrawn from ICLR 2027

ContactFlow: A video action conditioning that transfers across embodiments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models offer a promising route toward more general embodied intelligence by enabling robotic agents to anticipate the consequences of actions before execution. Video Generation based world models are particularly attractive because they can learn rich spatiotemporal interaction dynamics from large-scale video data. By capturing aspects of real-world physics, these dynamics could allow the models to simulate how scenes evolve under different actions. Realizing this potential requires models to learn from interaction data at sufficient scale and diversity, most of which depict humans, and transfer the acquired knowledge to different robotic embodiments. However, such a transfer requires an action representation that provides a common interface across humans and robots. Existing approaches instead condition on embodiment-specific actions, poses, or kinematics, tying the learned dynamics to a particular actor or control space. This coupling limits both the use of heterogeneous interaction data and transfer across embodiments. We propose Contact Flow, an embodiment-agnostic action representation that encodes manipulation through the spatiotemporal trajectories of 3D contact points between an actor and a target object. Contact Flow captures the object-side interaction geometry that determines how actions affect the environment, providing a common interface across embodiments. We condition a video world model on Contact Flow to predict future scene evolution from an initial observation. Experiments across human and robotic manipulation benchmarks show that Contact Flow enables accurate video prediction and effective cross-embodiment transfer.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
contact flow, video generation, robotics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rbUmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes contact flow, a representation based on 3D contact-point trajectories. The main motivation is to disentangle passive dynamics from active dynamics, enabling embodiment-agnostic action conditioning. The authors develop a video generation model conditioned on contact flow and further design a pipeline to extract contact flow from human and robot interaction data, resulting in a dataset. In their experiments, the authors demonstrate that contact flow can effectively control the video generation model and transfers more reliably across human and robot embodiments than embodiment-specific conditioning representations.

Critical strengths

The idea of using contact flow to bridge the action-space gap across different embodiments is interesting and well motivated. By representing interactions through object-side contact trajectories rather than embodiment-specific actions, data from multiple sources and embodiments can be more effectively utilized for training a single shared world model.

Critical weaknesses

My main concern is how contact flow can be obtained at inference time. In most experiments, the authors extract contact flow from the ground-truth interaction and use it as conditioning. However, such information would not be available before executing an action in a real robotic setting. On the other hand, in the real-robot experiment, the authors instead convert simulated trajectories into contact flow, but it is unclear how this conversion is performed and, in particular, whether it requires access to the simulated object response. This ambiguity makes it difficult to assess whether contact flow is a practically useful action representation for real-world deployment.

Furthermore, although the paper includes a real-world trajectory-verification experiment, it does not demonstrate the proposed model being used as a world model for robot control, for example to rank candidate actions, plan actions, or improve real-world task success. The executed robot trajectories are predetermined rather than selected based on the model’s predictions.

Overall, these issues make it difficult for me to assess the practical applicability of the proposed method and lead me to lean toward rejection.

Questions for authors

Following the weakness discussed above, how can contact flow be obtained at inference time in a real robotic setting? In particular, can contact flow be computed from only the initial observation and a candidate robot action or trajectory, without access to the future object response?

Additionally, can the proposed world model be used for actual robot control or planning? For example, can the world model be used to rank or optimize candidate actions, and if so, how are executable robot actions mapped to the corresponding contact flows?

Overall, it remains unclear how executable robot actions are mapped to contact flows, which is the conditioning signal input to the world model.

1 comment

Q1.  Following the weakness discussed above, how can contact flow be obtained at inference time in a real robotic setting? In particular, can contact flow be computed from only the initial observation and a candidate robot action or trajectory, without access to the future object response?

Yes, Contact Flow can be derived from candidate robot actions without requiring access to future object responses. One practical approach is to approximate Contact Flow using gripper-pad trajectories computed from end-effector commands. To evaluate this approach, we have added an experiment to Table 2, in which we replace extracted Contact Flow with these trajectories for conditioning. We also expanded our evaluation to 1,200 windows across eight datasets, including unseen robot embodiments and out-of-distribution data. This approach shows only a small decrease in prediction quality and still outperforms the actor-based baselines across all  metrics. In our real-world experiment, we used a simulator of the manipulated object to generate contact flow of candidate robot actions. However, as the experiment with the gripper-pad approximation shows,our world model does not depend on this simulator and can also evaluate trajectories proposed by a VLA or robot policy. We will clarify this procedure in the revised paper. In future work, we plan to also study Robot Foundation Models that generate contact flow encoded robot actions from language instructions.

Control inputIoU ↑end IoU ↑PSNRₘ ↑
Full Contact Flow Tokens0.4880.38224.63
Contact Flow Tokens (Finger Tips only)0.4670.32324.44
Actor pointmap (Baseline)0.3670.23023.06
Actor skeleton (Baseline)0.3790.22221.19
Reviewer_UeQJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Contact Flow, an embodiment-agnostic action conditioning representation for video world models. The paper proposes to instead represent interactions through 3D contact-point trajectories between an actor and manipulated objects. Contact Flow removes actor-specific morphology and preserves the object-side interaction geometry that determines environment response. The authors train a Contact Flow-conditioned video diffusion world model based on Wan, and construct ContactFlow-212K, a dataset containing human, simulated robot, and real robot interaction sequences with extracted contact trajectories.

Critical strengths
  1. Strong motivation and a meaningful abstraction for cross-embodiment transfer
  2. Strong experimental design for embodiment transfer
  3. Valuable multi-embodiment dataset contribution
Critical weaknesses

Contact Flow extraction pipeline introduces significant dependence on privileged information.

Although Contact Flow is embodiment-agnostic, its extraction relies on a multi-stage pipeline of external components including depth estimation, object reconstruction, segmentation, gripper pose estimation, camera calibration, and contact estimation. While acceptable for training dataset construction, this approach cannot directly support deployment, where Contact Flow must be predicted or planned from task goals. The paper does not clarify how a robot would derive a target Contact Flow at test time: current real-world implementations convert simulated trajectories into Contact Flow rather than generating it directly from task objectives, which undermines Contact Flow’s validity as a standalone action representation.

The evaluation focuses on short-horizon interaction prediction

A further critical limitation is that evaluation is confined to verifying whether predicted videos conform to Contact Flow. While informative, embodied world models require capabilities beyond controllable short-term prediction. The work lacks assessments of long-horizon rollout stability and downstream applications such as policy improvement via imagined rollouts. Accordingly, it only demonstrates a controllable video predictor, with an incompletely established connection to a full embodied world model.

Questions for authors

see weakness

2 comments

We thank the reviewer for the constructive feedback and for recognizing the strong motivation behind Contact Flow, the experimental design for cross-embodiment transfer, and the value of ContactFlow-212K. Below, we provide an initial response to the reviewer’s concerns.

W1.Contact Flow extraction pipeline introduces significant dependence on privileged information.

The multi-stage Contact Flow extraction pipeline is designed for offline dataset construction and was also used to obtain conditioning trajectories for our original evaluation. Importantly, this pipeline is not required at inference time. We currently demonstrate two different ways to produce a contact-flow conditioning signal at inference time: (a) through a coarse simulator; (b) approximated with the robot’s finger tip surface.

Regarding (a), the coarse simulator only requires the object geometry and no model of the environment.

In addition, we now demonstrate (b) as an even simpler and effective inference input: We approximate the Contact Flow from gripper-pad trajectories computed directly from candidate robot actions. This approximation relies exclusively on robot action commands and requires no privileged information. To evaluate this approach, we added a gripper-only experiment to Table 2 in which we replace extracted Contact Flow with trajectories derived from end-effector commands. We also expanded our evaluation to 1,200 windows across eight datasets, including out-of-distribution data and unseen robot embodiments. The gripper-only approach shows only a small decrease in prediction quality compared with full Contact Flow, while still outperforming all actor-based baselines across every reported metric. 

Overall, while the training extracts contact flow from privileged access to future video frames and we provide pipelines for heterogeneous data sources, these experiments demonstrate that one trained, our video world model can infer correct action outcomes from differently derived contact flow signals with high robustness.

Control inputIoU ↑end IoU ↑PSNRₘ ↑
Full Contact Flow Tokens0.4880.38224.63
Contact Flow Tokens (Finger Tips only)0.4670.32324.44
Actor pointmap (Baseline)0.3670.23023.06
Actor skeleton (Baseline)0.3790.22221.19

W2. The evaluation focuses on short-horizon interaction prediction.

We acknowledge that long-horizon prediction remains a challenge for video world models due to error accumulation and limited temporal memory. However, our evaluation covers relatively long manipulation sequences through temporal subsampling. While our prediction windows are fixed to 49 frames, we sample these frames from full Contact Flow trajectories spanning complete interactions. While our direct baselines Kinema4D and OSCAR do not report their prediction window duration, a comparison to other relevant baselines shows that our prediction window length is longer than any other reported duration.

Our evaluation also goes beyond adherence to Contact Flow. We assess the quality of predicted videos using standard metrics such as PSNR or DreamSim. We also measure how closely the predicted object motion and final states match ground truth using object IoU, final-state IoU, and position error. This allows us to evaluate whether a world model correctly predicts how the environment responds to a given action, decoupled from appearance-based evaluation

Method / DomainFrames per WindowDuration per Window (s)
Contact Flow (all)494.6
↳ Human493.9
↳ Real robot495.5
↳ Simulation495.0
Kinema4D [1]49—
OSCAR [2]81—
IRASim [3]164.0
DreamDojo [4]131.2
Ctrl-World [5]51.0
Cosmos 3 [6]161.07

[1] M. Xu, T. Zhang, T. Liu, Z. Chen, X. Han, and Z. Liu, "Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), 2026

[2] Z. Wu and J. Gao, "OSCAR: Omni-Embodiment Skeleton-Conditioned World Action Model for Robotics," arXiv preprint arXiv:2606.04463, 2026

[3] F. Zhu, H. Wu, S. Guo, Y. Liu, C. Cheang, and T. Kong, "IRASim: A Fine-Grained World Model for Robot Manipulation," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2025

[4] S. Gao et al., "DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos," in Proc. 43rd Int. Conf. Mach. Learn. (ICML), Proc. Mach. Learn. Res., vol. 306, pp. 33340–33366, 2026

[5] Y. Guo, L. X. Shi, J. Chen, and C. Finn, "Ctrl-World: A Controllable Generative World Model for Robot Manipulation," in Proc. Int. Conf. Learn. Represent. (ICLR), 2026.

[6] NVIDIA, "Cosmos 3: Omnimodal World Models for Physical AI," arXiv preprint arXiv:2606.02800, 2026.

Reviewer_NEJDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Contact Flow, an object-centric action representation that encodes manipulation through 3D contact-point trajectories. The authors use it to condition a video world model and train on human and robot interaction data, aiming to improve cross-embodiment generalization. Experiments show promising results in video prediction, with a real-world application to action outcome prediction.

Critical strengths
  1. Large-scale and diverse data curation. The authors construct ContactFlow-212K from diverse human, simulated robot, and real-robot datasets. This dataset is a useful contribution for studying cross-embodiment interaction modeling.
  2. Simulation experiments are extensive. Contact Flow is evaluated across multiple datasets and embodiments, with comparisons against several baselines and ablations of different conditioning mechanisms.
Critical weaknesses
  1. The novelty and scope of cross-embodiment transfer are not sufficiently convincing. Flow-based representations are not new, and the main contribution is applying contact flow to video world models. However, the experiments mainly demonstrate that human interaction data can help predict robot videos when the conditioning trajectory is extracted from robot interactions. This does not convincingly establish cross-embodiment action transfer, where a human interaction trajectory is used to generate the outcome of a robot action from the robot's initial observation. The current evaluation setting is therefore relatively narrow, and the evidence for the broader cross-embodiment transfer claim is limited.
  2. The real-world evaluation is too limited to validate the practical effectiveness of the method. The real-world data and experiments are limited, and the action-verification pipeline appears to require a closely matched simulator to generate and evaluate candidate trajectories. This is a restrictive assumption for real-world deployment. Moreover, the paper does not provide a convincing real-robot comparison against alternative point-flow-based methods (e.g., [1] and [2]), making it difficult to determine whether Contact Flow itself improves action outcome prediction in practice.
  3. The reliability of the contact annotations and confidence scores is unclear. The data-processing pipeline combines several estimated quantities, including depth, object segmentation, 3D reconstruction, contact estimation, and temporal correspondence tracking. Errors at any of these stages could affect the resulting Contact Flow. The paper does not sufficiently establish how robust the final representation is to these errors. In particular, the role of the confidence score remains unclear: does it genuinely help the model ignore unreliable contact points, or is the model still sensitive to annotation noise?

[1] Li, Hongyu, et al. "Novaflow: Zero-shot manipulation via actionable flow from generated videos." 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.

[2] Lee, Seungjae, et al. "Tracegen: World modeling in 3d trace space enables learning from cross-embodiment videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Questions for authors

Please refer to the weaknesses above for further discussion on the novelty of cross-embodiment transfer, the limited real-world evaluation, and the robustness of Contact Flow to errors in contact estimation and tracking.

1 comment

We thank the reviewer for the constructive feedback and for recognizing the value of ContactFlow-212K and our extensive experiments across datasets and embodiments. Below, we provide an initial response to the reviewer’s comments.

W1. Novelty and cross-embodiment transfer

Given the provided references, we believe that there is a critical misunderstanding: We study action representations for input conditioning, not as part of the generated output. What the reviewer describes are transfer settings such as in Novaflow, where a model generates an encoded action from a video (hereafter referred to as action models). Instead, video world models as studied in our paper predict the outcome of actions that are given as input conditioning (hereafter referred to as video world models).

Consequently, cross-embodiment transfer differs between these two model families. In action models, the transfer refers to the output space, in video world model, the transfer refers to the input space: The reviewer is correct that in action models, the goal of cross-embodiment transfer is e.g. learning from human actions but generating robot actions. In video world models, however, the goal of cross-embodiment transfer is learning from human actions but generating the consequences of robot actions.

Our chosen baselines are therefore works that predict interaction outcomes conditioned on given actions and claim to achieve cross embodiment generalization. We demonstrate that Contact Flow consistently outperforms the actor-dependent representations used in these prior works, particularly when transferring learned interaction dynamics to unseen robot embodiments. Our key contribution is demonstrating that Contact Flow, as an embodiment-independent action representation, enables more effective cross-embodiment transfer in video world models.

W2 & W3: Practicality and Robustness to Noise in Conditioning

As discussed in Point 1, our objective is to predict interaction outcomes independently of the acting embodiment. To strengthen this claim, we expanded our evaluation to 1,200 windows across eight datasets, including out-of-distribution data and unseen robot embodiments. 

As an alternative to simulator-derived contact flow, we now add a study that uses the robot’s gripper tips as an approximation of the contact surface, a practical approximation at inference time. This describes the  conditioning directly from robot commands without requiring object-specific information or privileged annotations. Prediction quality remains strong and exceeds all baselines across every reported metric. This demonstrates that our video world model does not require an object simulator and can evaluate candidate actions from any robot policy or VLA. It also demonstrates that the learned model is very robust to differently derived contact flow inputs. 

We are currently conducting additional tests with respect to the confidence score and will report these before the end of the discussion phase.

Control inputIoU ↑end IoU ↑PSNRₘ ↑
Contact Flow Tokens (Full Contact Flow)0.4880.38224.63
Contact Flow Tokens (Finger Tips only)0.4670.32324.44
Actor pointmap (Baseline)0.3670.23023.06
Actor skeleton (Baseline)0.3790.22221.19

Comments on the paper

1 comment

Dear Reviewers,

Please take time to review the responses from the authors and engage with them for further questions and comments if any.

Time left is not much and please do shortly.

Thanks

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.