acceptodds
Withdrawn from ICLR 2027

EgoTACO: Egocentric Task Abstraction for CrossEmbodiment Object Manipulation in Editable Worlds

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Egocentric video records the goal, contact roles, event order, and object-state changes of everyday manipulation. Direct video-to-robot retargeting binds this evidence to a human execution that can become invalid when robot morphology or object physics changes. We introduce EgoTACO, a task-conditioned video generation framework that separates a source into completed background, human-operation, and object layers, infers an editable task latent from the latter two, and regenerates the full scene under target-object, physics, and robot-embodiment conditions. The latent specifies what must change in the world without prescribing a human wrist path. A simulation harness recompiles edited tasks and verifies kinematics, contact roles, and terminal state. Joint geometry and inertia adaptation eliminates the corresponding violations in 120 constructed interventions, while task-contact replanning completes all 60 cross-embodiment simulator goals. A conditional video predictor gains 3.79 dB motion-region PSNR over an equal-capacity unconditioned model. Missing end-to-end and policy results are explicitly marked in the tables.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Egocentric Video, Robot Manipulation, Cross-Embodiment Transfer, Task-Conditioned Video Generation, Object-Centric Representation, Contact-Aware Manipulation, World Models, Embodiment Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_5dPgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

EgoTACO proposes a task-state abstraction for transferring egocentric human demonstrations across objects and robot embodiments. Instead of replaying human trajectories, it represents goals, event order, object-state transitions, affordances, and contact roles, then replans robot-specific contacts and generates robot videos.

Critical strengths
  1. The problem is important and well motivated.
  2. Separating task semantics from embodiment-specific trajectories is conceptually sound.
Critical weaknesses
  1. My main concern is the reliability of the experimental evidence. Table 1 reports complete downstream results, while Appendix Figure 6 states that these scores remain TBD. The appendix also acknowledges inconsistent sample counts of 60, 100, and 120, and the reproducibility statement notes that code, checkpoints, and complete per-run results are unavailable.

  2. The method is also underspecified. The paper does not explain how task graphs, affordances, contacts, or terminal predicates are extracted; which planning, IK, collision, and dynamics algorithms are used; or how states and contacts are recovered from generated RGB videos.

  3. This paper seems to have been directly generated by an AI. Figures 1 and 2 are quite rough and look like they were created by an AI. The rationality of the experiments also seems to have been coded by an AI. The visualization of Figure 3 is a complete mess. I have no idea what it intends to convey.

Questions for authors

see weaknesses

Reviewer_yhWwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EgoTACO, a framework for cross-embodied task transfer from egocentric human manipulation videos to diverse robot morphologies. The core idea is to avoid directly imitating human frame-by-frame motion trajectories, and instead extract task state abstractions from videos comprising task goals, ordered events, object state changes, affordances, and contact roles, while treating object geometric/physical properties and robot embodiment conditions as independently editable factors. Building upon this, the method replans execution strategies suited to the target robot through task contact constraints, and generates robot execution videos via conditional video generation, physical and geometric feasibility verification, and local failure recompilation mechanisms. Experiments primarily validate the framework through object property editing, cross-robot morphology transfer, ablation studies, and downstream training data evaluation. Results demonstrate that compared to direct frame-by-frame inverse kinematics mapping of human trajectories, replanning based on task states and contact constraints better preserves task completion objectives and adapts to variations in robot morphology and object physical properties.

Critical strengths

The task abstraction perspective demonstrates strong validity and insight. Rather than directly mapping human manipulation trajectories frame-by-frame to the robot, the paper represents tasks in terms of goals, event sequences, object state changes, affordances, and contact roles. This design of "preserving task semantics rather than motion trajectories" aligns well with the essence of cross-embodiment transfer.

The experimental design directly supports the paper's core claims. The paper specifically compares human-path similarity with task completion, showing that while framewise IK achieves closer alignment with human trajectories, its task success rate is 0%, whereas task-contact replanning achieves higher feasibility and task completion rates despite permitting larger trajectory deviations.

Critical weaknesses

Insufficient description of key modules and limited verifiability of the method. The core of the paper relies on extracting task goals, event graphs, object state changes, affordances, and contact roles from egocentric videos, and validating by recovering robot and object states from generated videos. However, the specific implementations, supervision mechanisms, and prediction errors of these key modules are not adequately explained. Meanwhile, some experiments use pre-provided physical properties and task goals, making it unclear how much of the actual performance comes from egocentric task understanding versus external prior information. This weakens the assessment of the core method's effectiveness.

Significant inconsistencies in experimental protocols and results undermine the credibility of the main conclusions. The main text reports complete downstream ReKap comparison results, while the appendix explicitly states that downstream scores remain TBD/unreported; additionally, object editing sample sizes vary across experiments with settings of 60, 100, and 120, and the paper acknowledges that the relationships between these protocols have not yet been clarified. These internal inconsistencies make the current experimental results difficult to reliably reproduce and validate, and therefore constitute a major issue affecting the acceptance decision.

Questions for authors

Please clarify the implementation and training of the task encoder EτE_\tau. It is unclear how the task goal, event graph, object-state changes, affordances, and contact roles are extracted from egocentric videos, and whether they rely on manual annotations or predefined rules.

Please clarify which information is predicted from the input video and which is externally provided in the experiments.

Please explain the state recovery and verification process after video generation. In particular, what models or algorithms are used to recover robot and object states from the generated RGB videos?

Please clarify the consistency between the results reported in the main text and those in the appendix.

Could the authors provide stronger task-level or cross-embodiment baselines beyond framewise IK? The current comparisons are insufficient to establish the advantage of EgoTACO over other task-level planning or constraint-based transfer methods.

Reviewer_Det5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

this work propose a new data synthesis method which involving transfer egocentric data to robot-centric data with diverse dynamics. the solution sounds rational but presented content is low-quality, make human being hard to judge.

Critical strengths

N/A

Critical weaknesses
  1. The teaser is unclear and includes a low-quality figure. The task being performed is unkind, but one thing I can be sure of is that they do not all serve the same goal. The completed figure is very noisy and needs improvement.
  2. The paper has three figures, two of which are of low quality, making it hard for readers to judge.
  3. 100% SC versus 0% SC? Could the task setting be biased? More public benchmarks are needed.
Questions for authors
  1. The teaser is unclear and includes a low-quality figure. The task being performed is unkind, but one thing I can be sure of is that they do not all serve the same goal. The completed figure is very noisy and needs improvement.
  2. The paper has three figures, two of which are of low quality, making it hard for readers to judge.
  3. 100% SC versus 0% SC? Could the task setting be biased? More public benchmarks are needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.