acceptodds
Withdrawn from ICLR 2027

Cross-Embodiment Causal Transfer from Human Videos to Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling robot manipulation models requires large amounts of interaction data, yet collecting diverse real-robot experience remains costly. Human manipulation videos, in contrast, provide abundant and diverse task experience that could potentially benefit robot learning. However, differences in morphology, action spaces, and control mechanisms make human motions unsuitable as direct supervision for robot actions, raising a scientific question: what knowledge in human videos should be transferred across embodiments to robot manipulation? To address this question, we analyze subtask transfer through the lens of causal beams, focusing on which contextual conditions and interaction dependencies should be retained to support local task effects. Based on this perspective, we propose Causal-Beam-Guided Subtask Transfer (CBST). CBST learns manipulation-related representations from human videos and grounds them in task-level semantic variables using robot demonstrations. Within a predefined manipulation model, causal-beam constraints guide contextual information selection by requiring retained assignments to sustain local mechanism outputs while preserving nontrivial responsiveness to retained inputs. The selected semantic information conditions a world-action model, allowing the robot to generate actions from its own observations without requiring human–robot motion correspondence. Experiments on robot manipulation benchmarks show improvements in task success over the corresponding world-action model baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Causal Representation, World Action Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_oaeyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Causal-Beam-Guided Subtask Transfer (CBST) for robot manipulation. A visual–temporal encoder is pretrained on human-video action classification and adapted using robot demonstrations to predict task-level semantic variables. Within a predefined structural causal model, causal-beam constraints guide the selection of context-relevant information, which is rendered as language to condition a frozen world-action model. Experiments with two backbones across simulation benchmarks and real-world manipulation tasks report improvements over the underlying policies.

Critical strengths
  1. The motivation of transferring task-relevant knowledge from human videos to robot manipulation is intuitive.

  2. The use of causal-beam constraints offers an interesting approach to selecting context-relevant semantic information for robot control.

  3. Experiments across simulation benchmarks and real-world tasks demonstrate modest performance improvements over the underlying world-action models.

Critical weaknesses
  1. Although the intuition is clear, the solution does not seem to fully address the motivating question of what knowledge should transfer across embodiments. The paper does not provide an interpretable analysis of which semantic conditions and dependencies the causal-beam mechanism retains in practice, why they are transferable, or how they contribute to successful robot execution.

  2. The performance improvements are modest, and the lack of a direct comparison with explicit rule-based semantic selection under the same prediction and conditioning setup makes it difficult to establish the added value of the causal-beam mechanism.

Questions for authors

See weaknesses.

Reviewer_D9NVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Causal-Beam-Guided Subtask Transfer (CBST), a framework for conditioning frozen world-action models on task-level semantic information. A visual–temporal encoder is first pretrained through action classification on human videos and then adapted using robot demonstrations to predict task conditions, manipulation attempts, object-state changes, and local effects. Within a predefined manipulation SCM, a learned selector chooses parent assignments subject to causal-beam constraints. The selected information is rendered as text and supplied to the WAM alongside robot observations. Experiments with LingBot-VA and Cosmos Policy on simulation benchmarks and real-world manipulation tasks report improvements over the corresponding backbones.

Critical strengths

The paper separates task conditions, manipulation attempts, object-state changes, and realized effects, providing a useful way to describe cross embodiment execution without requiring human–robot motion correspondence. The causal-beam formulation makes the information-selection criteria explicit within the assumed reference model.

Critical weaknesses
  1. The causal component is not directly applied to learning from human videos. Human videos are used only for action-classification pretraining of the visual–temporal encoder. The central components—semantic-variable grounding and causal-beam-constrained selection—are learned or applied during robot adaptation using robot demonstrations and predefined manipulation templates. Thus, the proposed causal framework does not directly determine which conditions or dependencies are extracted from human demonstrations and transferred to robots. This leaves a gap between the motivation of selectively preserving causal information from human execution and the implemented pipeline, which combines human-video representation pretraining with robot-side causal conditioning.

  2. Insufficient specification of the supervision and task-model construction. The method relies on task-specific semantic variables, interval annotations, prerequisite rules, and fixed reference mechanisms, but their construction is not described at a reproducible level. It is unclear how manipulation attempts, effective grasps, state changes, and local effects are labeled from demonstrations; how the relevant temporal intervals are defined; and how variables are associated with particular object instances. The actual task templates, reference mechanisms, and selector architecture are also insufficiently specified.

Questions for authors
  1. Mismatch between motivation and implementation. The paper motivates selecting causal information from human demonstrations, but causal-beam selection is applied only on the robot side; human videos serve solely as action-classification pretraining data. How does the implemented method address the stated human-to-robot causal-transfer problem, rather than robot-side information selection?
  2. Clarification — Supervision construction. How are semantic-state labels, operation intervals, subgoals, and prerequisite rules constructed, particularly for real-world demonstrations?
  3. Justification — Learned selector. Why learn a selector whose objectives reward validity and compactness, rather than directly computing a minimum-size compatible valid selection? How do these alternatives compare under identical conditions?
  4. Limited gains and missing scaling evidence. Human-video pretraining yields a modest 1.20-percentage-point improvement on RoboTwin Easy. Is this gain robust across independent training runs? Given the absence of data-scaling experiments, what evidence shows that increasing human-video data improves performance or reduces the need for robot demonstrations?
Reviewer_UueXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of transferring abstract task information from human videos to robots. To achieve this, the authors propose Causal Beam Guided Subtask Transfer (CBST), which represents manipulation processes using abstract concepts for task conditions, manipulation attempts, object-state changes, and local effects. CBST uses causal-beam constraints to retain context relevant to the current interaction and supplies this information to a frozen world action model. Experiments with LingBot-VA and Cosmos Policy report improvements across RoboTwin, LIBERO, and RoboCasa. Simulation gains are generally modest, while real-world success increases from 68.5% to 76.5% across ten placement tasks.

Critical strengths

The paper addresses a relevant problem: identifying useful information from human manipulation experience despite differences between human and robot execution.

Critical weaknesses

Weakness

  1. The paper is difficult to follow

  2. The baselines do not sufficiently isolate the contribution of causal-beam-based context retention. The state-only, full-structure, and size-matched random-selection comparisons are useful controls. However, they do not establish whether causal-beam selection is better than other competitive approaches to retaining relevant context. Random selection is a relatively weak alternative, while including the full structure may introduce unnecessary information as in Big picture policy [R1] which uses an oracle VLM to select these or use a RAG style system for retrieving relevant abstractions.

  3. The predicates, candidate dependencies, and reference mechanisms are insufficiently specified. These components determine which selections are considered valid and are therefore central to the method. However, stating that they are instantiated from task specifications and shared manipulation templates does not provide enough detail to reproduce the process or assess its task-specific engineering requirements.

  4. The source and construction of robot-adaptation ground-truth labels are unclear. Appendix B.3 states that demonstration replay provides semantic-state, interval-variable, and effect labels.

  5. The proposed semantic decomposition is not directly ablated. The decomposition into conditions, attempts, state changes, and effects is central to the paper’s argument. However, the existing experiments mainly evaluate pretraining and information selection within this predefined representation. They do not establish which aspects of the representation are responsible for the reported gains.

  6. The human-data contribution and several performance gains require stronger evidence. Human pretraining is evaluated, but the reported downstream differences are modest. On RoboTwin Easy, success increases from 92.80% without human pretraining to 94.00%, compared with 92.94% using shuffled pretraining. Without uncertainty estimates, it is difficult to assess the robustness of these differences.

  7. The real-world evaluation only partially supports the long-horizon motivation, and its protocol needs clarification. The simulation experiments include LIBERO-Long and horizon-grouped RoboTwin results. However, the real-world evaluation primarily consists of individual object-placement tasks, sometimes involving handover. These demonstrate useful coordination but provide limited evidence for retaining context across longer sequences of dependent subtasks.

Questions for authors

1, How sensitive is CBST to errors in its semantic predictions 2. How does prediction of beam search correlate with downstream success rate. 3. how is the data annotated for training

Reviewer_MJjyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Causal-Beam-Guided Subtask Transfer (CBST), a framework designed to leverage experience from human videos for robotic manipulation. The method begins by pre-training a visual-temporal encoder on the SSv2 dataset, followed by learning task-level semantic variables from robot demonstrations. Within a predefined reference framework, a selector identifies a subset of parent variables that satisfy "causal beam" constraints and are mutually compatible. These variable assignments are then converted into text, which serves as conditional input for a frozen World-Action Model (WAM). The approach is evaluated across two backbone models, three simulation benchmarks, and ten real-world robotic placement tasks.

Critical strengths
  1. The paper proposes an enlightening formal framework for the contextual information selection problem in robot operation. Sections 3.2-3.3 clearly distinguish between retaining the observed local task effects and retaining the conditions and dependencies that support these effects. It is worth noting that the authors explicitly limit the effectiveness of causal beams to specific mechanisms and execution contexts, neither assuming that humans and robots have the same causal mechanisms, nor believing that causal beams can be unconditionally transferred between different embodied entities. These rigorous theoretical definitions enhance the formal basis of the method and provide valuable design ideas for the construction of semantic conditions at the task level.

  2. This framework features a robust modular design and has been evaluated across various experimental settings. CBST can be used as an additional module to enhance the frozen LingBot-VA and Cosmos Policy backbone models. The experiments cover multiple simulation benchmarks as well as real dual-arm robot platforms. Ablation studies clearly distinguish between variants that require retraining of the robot-side modules and those that reuse the same semantic predictor while adjusting only the conditional inputs, thereby making the experimental comparisons easier to interpret.

Critical weaknesses
  1. The core contributions have not been fully verified. The human videos are only used for pre-training of action classification, and the causal structure comes from predefined templates. Therefore, it is unclear what causal knowledge has been transferred. The existing ablation studies have also failed to isolate the role of causal beam selection: State-only simultaneously removes interaction information, and random selection is not sufficient to represent an effective non-causal method. Additionally, the main results directly compare the published baseline scores, with small improvements and lack of uncertainty analysis. For example, LingBot-VA only improved from 98.5% to 98.60% on LIBERO-Long, while the reported fluctuations in the baseline were ±0.5. These pieces of evidence are not sufficient to confirm the reliability of the gains and their sources.
  2. The implementation details and experimental reports are incomplete. The training configuration lacks key parameters such as learning rate and training duration; the appendix describes adaptation curves, sensitivity experiments, and end-to-end timing, but does not provide corresponding results. These omissions affect the reproducibility of the method and also make the related analysis lack evidence support.
Questions for authors
  1. What specific interaction variables have been improved in the human video pre-training in terms of prediction or structure selection? Can direct evidence be provided to show that the benefits are related to transferable causal information?
  2. Please re-run the backbone model and CBST under the same evaluation settings and initial conditions, and report the results from multiple random seeds along with their uncertainties. Are the existing minor improvements still reliable?
  3. Under the condition of retaining the same semantic information and controlling the prompt length, is CBST superior to simple relevance selection or deterministic selection of the minimum effective parent sub-set? Can such a comparison isolate the contribution of causal beam constraints?
  4. Please add key configurations such as learning rate and training duration, as well as the adaptation curves, sensitivity analyses, and end-to-end timing results mentioned in the appendix. Have these analyses been completed?
Reviewer_prbQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Causal-Beam Guided Subtask Transfer(CBST), which tackles on human-to-robot learning using structural causal models. Specifically, CBST first learns manipulation-related temporal representations with human-video pretraining. Then, it learns a model to predict contextual information selection from robot demonstrations. The predicted information is used to condition a world-action model. The authors compared CBST against baselines in three simulation benchmarks (RoboTwin, LIBERO, and RoboCasa) and in one real world setup. While it proposes an interesting formulation, the results are not strong (i.e., baseline with and without CBST have very similar performance).

Critical strengths
  • What information is transferrable from human video to target robot embodiment is an important problem.
  • Plug-in design for existing policy / world action model architecture.
  • Extensive evaluations across multiple simulation benchmarks and in real-world.
Critical weaknesses
  • Weak results. Human videos are only used for pretraining the encoder, and the results are not much stronger compared to baselines with same architecture but without CBST, or to CBST without individual components.
  • No qualitative analysis of what transfers. While the authors motivate the research question of "what type of information is transferrable from human videos to target robot embodiment?", there isn't any analysis on what actually transfers.
  • The causal structure and states are predefined, not learned.
  • The size of possible subset of parents is exponential to the number of parents. It is unclear whether CBST would scale.
  • It is unclear why WAM architecture is required, if CBST only extends the text prompt.
Questions for authors

Check weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.