- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes contact flow, a representation based on 3D contact-point trajectories. The main motivation is to disentangle passive dynamics from active dynamics, enabling embodiment-agnostic action conditioning. The authors develop a video generation model conditioned on contact flow and further design a pipeline to extract contact flow from human and robot interaction data, resulting in a dataset. In their experiments, the authors demonstrate that contact flow can effectively control the video generation model and transfers more reliably across human and robot embodiments than embodiment-specific conditioning representations.
Critical strengths
The idea of using contact flow to bridge the action-space gap across different embodiments is interesting and well motivated. By representing interactions through object-side contact trajectories rather than embodiment-specific actions, data from multiple sources and embodiments can be more effectively utilized for training a single shared world model.
Critical weaknesses
My main concern is how contact flow can be obtained at inference time. In most experiments, the authors extract contact flow from the ground-truth interaction and use it as conditioning. However, such information would not be available before executing an action in a real robotic setting. On the other hand, in the real-robot experiment, the authors instead convert simulated trajectories into contact flow, but it is unclear how this conversion is performed and, in particular, whether it requires access to the simulated object response. This ambiguity makes it difficult to assess whether contact flow is a practically useful action representation for real-world deployment.
Furthermore, although the paper includes a real-world trajectory-verification experiment, it does not demonstrate the proposed model being used as a world model for robot control, for example to rank candidate actions, plan actions, or improve real-world task success. The executed robot trajectories are predetermined rather than selected based on the model’s predictions.
Overall, these issues make it difficult for me to assess the practical applicability of the proposed method and lead me to lean toward rejection.
Questions for authors
Following the weakness discussed above, how can contact flow be obtained at inference time in a real robotic setting? In particular, can contact flow be computed from only the initial observation and a candidate robot action or trajectory, without access to the future object response?
Additionally, can the proposed world model be used for actual robot control or planning? For example, can the world model be used to rank or optimize candidate actions, and if so, how are executable robot actions mapped to the corresponding contact flows?
Overall, it remains unclear how executable robot actions are mapped to contact flows, which is the conditioning signal input to the world model.
Q1. Following the weakness discussed above, how can contact flow be obtained at inference time in a real robotic setting? In particular, can contact flow be computed from only the initial observation and a candidate robot action or trajectory, without access to the future object response?
Yes, Contact Flow can be derived from candidate robot actions without requiring access to future object responses. One practical approach is to approximate Contact Flow using gripper-pad trajectories computed from end-effector commands. To evaluate this approach, we have added an experiment to Table 2, in which we replace extracted Contact Flow with these trajectories for conditioning. We also expanded our evaluation to 1,200 windows across eight datasets, including unseen robot embodiments and out-of-distribution data. This approach shows only a small decrease in prediction quality and still outperforms the actor-based baselines across all metrics. In our real-world experiment, we used a simulator of the manipulated object to generate contact flow of candidate robot actions. However, as the experiment with the gripper-pad approximation shows,our world model does not depend on this simulator and can also evaluate trajectories proposed by a VLA or robot policy. We will clarify this procedure in the revised paper. In future work, we plan to also study Robot Foundation Models that generate contact flow encoded robot actions from language instructions.