acceptodds
Under review as a conference paper at ICLR 2027

DexWeave: Learning Dexterous Humanoid Loco-Manipulation from Human Demonstrations

17 views

Abstract

Learning dexterous humanoid loco-manipulation from human demonstrations requires transferring not only human motion, but also the coordinated interaction structure underlying the demonstrated behavior. This is challenging because embodiment differences distort the coupling among body motion, wrist placement, finger articulation, and object interaction, while kinematically accurate references may still be difficult to realize under robot dynamics. We present DexWeave, a unified framework that connects interaction-consistent motion retargeting with anatomy-aware whole-body policy learning. DexWeave first employs a two-stage retargeting procedure that initializes body and hand motions with specialized solvers and subsequently performs coupled refinement over the upper-body interaction chain while preserving lower-body support. The resulting references are tracked by an anatomy-aware Transformer policy that represents anatomical regions as structured tokens and uses directed masked attention to model their dependencies, with object information selectively conditioning the upper-body pathway for dexterous interaction. The policy jointly outputs body and dexterous-hand actions and is trained directly with reinforcement learning, without pretrained tracking policies, teacher–student distillation, or subsequent residual refinement. DexWeave improves retargeting fidelity and interaction consistency while achieving higher manipulation performance and faster policy convergence than MLP baselines. We further deploy the learned policies on a physical Unitree G1 humanoid equipped with Inspire dexterous hands, demonstrating dexterous whole-body loco-manipulation in the real world.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid Loco-Manipulation, Dexterous Manipulation, Motion Retargeting, Reinforcement Learning, Humanoid

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_csSNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DexWeave learns dexterous humanoid loco-manipulation skills from human demonstrations. It first retargets human motion to the robot, then jointly refines the arms, wrists, and fingers to improve hand–object alignment. An Anatomy-aware Transformer uses body-region tokens and masked attention to coordinate the body and hands, with object information provided to the upper-body branches. The authors train a separate policy for each reference motion and evaluate the method through retargeting benchmarks, simulation, and experiments on a Unitree G1 with Inspire hands.

Critical strengths
  1. Joint upper-limb refinement helps preserve hand–object alignment. Optimizing the arms, wrists, and fingers together addresses a practical problem in transferring human demonstrations: a locally reasonable hand pose can still miss the object because the wrist is misplaced. The reported results show lower thumb/index fingertip errors and fewer penetration violations on GRAB and HUMOTO. The refinement comes with some loss in palm-orientation accuracy, but the gains in fingertip placement make this a useful part of the method.

  2. The framework connects retargeting with whole-body execution. The authors carry the retargeted motions through policy learning and deployment on a physical robot. The policy jointly controls the body and hands without separate tracking-policy pretraining or residual refinement. It achieves higher episode completion than a similarly sized object-conditioned MLP on the evaluated motions. The hardware demonstrations add value by showing how the approach works on the target platform.

Critical weaknesses
  1. The hardware sensing and initialization setup is unclear. The policy requires the current object pose, but I could not find how this is measured on the robot. The paper also leaves unclear how the reference trajectory is aligned with the real scene and how precisely the object must be placed before execution. The simulated observation noise and delays do not explain the actual sensing setup. These details are needed to understand the deployment conditions and reproduce the demonstrations.

  2. The hardware results provide limited detail about finger coordination. The hardware images make it difficult to assess how the fingers participate in the tasks and whether the intended hand–object contacts are maintained. Richer examples, including tool use, appear in the retargeting results, but the paper does not clearly show whether these interactions are reproduced on hardware. Hand close-ups or an analysis of the existing robot experiments would help explain the role of finger articulation in the demonstrated tasks.

  3. The reported success rate does not directly measure complete-task success. The success metric counts episodes that reach the reference end or time limit without early termination. Since some episodes start partway through a motion, this can mean completing only a segment. Results for complete motions starting from the first frame, with success defined by the manipulation outcome as well as tracking errors, would make the reported performance easier to interpret.

  4. The architecture ablations leave some design choices insufficiently evaluated. The attention-mask ablations cover only two motions. The proposed masks give higher completion and better body tracking, while dense attention gives slightly better object-pose tracking. More motions would help determine whether this trade-off is consistent. The choice to exclude direct object information from the leg-control branches is also not tested separately, since the object-attention ablation retains the lower-body bypass. A comparison with direct object conditioning of the legs would help assess this choice, particularly for motions involving changing loads or lower-body adjustments during manipulation.

  5. The related-work discussion does not fully position the contribution. The paper does not discuss Beyond Mimicry: Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations or HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos. Both connect interaction-aware data generation or retargeting with humanoid skill learning. Although they address different settings, discussing their relationship to DexWeave would help explain what the proposed arm–wrist–finger refinement adds.

Questions for authors
  1. Could the authors clarify how the current object pose is measured during hardware execution and how the reference trajectory is aligned with the physical scene? What range of initial object-placement errors can the system tolerate? Please specify the sensing frequency, measurement latency, and treatment of occlusion.

2.Could the authors provide close-up views of the hands during the existing hardware experiments? For the simulation evaluation, could completion rates from first-frame initialization be reported separately, along with task-specific success criteria? For the hardware experiments, please report the number of trials and successful executions.

  1. Could the authors extend the attention-mask evaluation to additional reference motions and clarify whether direct object conditioning of the leg-control branches has been evaluated?
5 comments
AuthorsOpenReview ↗

Response to Reviewer csSN (Part 1/2)

We thank the reviewer for recognizing the effectiveness of our joint upper-limb refinement in preserving hand–object interactions, as well as the value of our unified framework for translating human demonstrations into physically executable humanoid loco-manipulation skills.

W1/Q1: How are object pose sensing and reference trajectory alignment implemented in real-world execution, and how robust is the system to initialization errors, sensing latency, and occlusion?

A1: Thank you for pointing out this omission. Our hardware deployment implementation estimates the object’s 6-DoF pose using an onboard wide-angle camera (640×480, configured at 30 FPS) and object-mounted AprilTags. Calibrated camera intrinsics, tag-to-object transforms, and camera-to-torso extrinsics convert detections into torso-frame object observations, delivered to the policy through DDS. Robot state estimation combines FAST-LIO odometry from the onboard MID-360 LiDAR–IMU for position estimation, and the robot’s IMU for orientation.

For real deployment, it contains two stages, i.e. a standby stage and a task policy executation stage. The standby stage is based on a seperate locomotion controller. We pad 2-3 seconds transition frames between the standby pose and the task first frame pose. The padding is also applied to the end of the task frame, such that the standby->task and task->standby switchings are smooth. At task initialization, a fixed horizontal translation and yaw rotation align the reference robot’s initial anchor pose with the robot’s current pose. This transform is applied consistently to the robot and object reference trajectories and remains fixed throughout execution; reference joint angles and velocities are unchanged.

The operator places the object near its reference starting pose relative to the robot. Since we applied initial pose DR during training, the policy is able to tolerate the initialization errors to some extent (i.e. the position error tolerance is around 5 cm and the orientation error is around 10 degrees). We did not test the effect of sensing latency in specific, the whole system is evaluated only under the nominal operation condition. We do observe occlusion issues with the AprilTag for some tasks, especially after the object is grasped by the dexterous hands. For those tasks, we only used the AprilTag pose for precise grasping, after the object is grasped (i.e. the AprilTag is being occluded), we switch the measured object pose to reference object pose to continue the rest of the motions.

We will include those deployment details in the final version.

 

W2/Q2-1: Can the authors provide close-up views of the hands during real-world experiments to demonstrate finger coordination and hand–object contact preservation? Can the authors demonstrate whether the diverse interactions shown in retargeting results, particularly tool use, can be successfully reproduced on real hardware?

A2: We thank the reviewer for this constructive suggestion. To better illustrate the role of dexterous finger articulation during manipulation, we provide additional close-up videos of our real-world experiments. The video links are Lifting a basket, Transferring a basket , Pouring water from a cup and Lifting a trash can. We further conduct real experiment with one more stirring food using a spatula motion sequence, as a tool use example. The real-world video is provided at Stirring food using a spatula.

AuthorsOpenReview ↗

Response to Reviewer csSN (Part 2/2)

W3/Q2-2: Can the authors report complete-task success rates using task-specific criteria and first-frame initialization in simulation, along with trial counts and successful executions on hardware?

A3: Apologize for the confusion. Table 3's 97.5% success rate is computed by rolling out the reference motion from first frame and scored for final object goals. To ensure reliability, we choose ten checkpoints for each sequence; For each checkpoint, we repeat ten rollouts. The success criteria is that the robot has to finish the whole motion sequence, and the object position/orientation error (relative to the reference object pose) should be smaller than 0.3 m and 40 degrees respectively, during manipulation. The settings are the same for other baselines. We further tighten the success criteria to 0.1m and 20 degrees for object position/orientation error. The success rates for DexWeave, ObjectMLP, InterMimic are 94.6%, 48.4% and 45.4% respectively. For real hardware, we deploy the policies in 10 consecutive trials. The average success rate is around 82.5%.

 

W4/Q3: Can the authors extend the attention-mask ablations to more reference motions and evaluate whether direct object conditioning of the leg-control branches improves performance?

A4: We thank the reviewer for these valuable suggestions. We have extended our attention-mask ablations from two to five representative motions and introduced an additional variant, Whole-body MOM_O, which allows the leg-control branches to directly access object information.

Attention settingSuccess rate (%) ↑Body pos. (cm) ↓Object pos. (cm) ↓Object rot. (deg) ↓
DexWeave (w/ MRM_R and MOM_O)94.64.673.638.83
w/o MRM_R72.87.868.7214.48
w/o MOM_O76.45.395.5511.21
Whole-body MOM_O81.25.004.548.92

As shown in prior table, DexWeave consistently achieves the best average performance across all four metrics, with a 94.6% success rate, compared with 81.2% Whole-body MOM_O. It demonstrates that directly conditioning leg control on object observations does not necessarily improve performance. Instead, our selective object conditioning achieves a 13.4 percentage-point higher success rate, supporting our design choice to coordinate lower-body control through shared body-state information rather than direct object observations. The reason might be that even the leg cannot observe the object information directly, it is able to know how to keep balance with its propriceptive measurements indirectly.

These additional results provide empirical evidence for the effectiveness of our structured attention and selective object conditioning in whole-body dexterous interaction.

 

W5: How does DexWeave relate to Beyond Mimicry[2] and HumanX[3], and can the authors clarify its contributions relative to these works?

A5: We thank the reviewer for highlighting these relevant works. We agree that both share a similar motivation of transferring human interaction demonstrations into executable humanoid skills. Unlike these approaches, DexWeave specifically addresses fine-grained dexterous hand–object interactions through coupled arm–wrist–finger refinement, jointly optimizing wrist placement and finger articulation while preserving lower-body support. We will include these works and clarify the distinctions in our revised related-work discussions.

 

[2] Wei-Jin Huang, Yue-Yi Zhang, Yi-Lin Wei, Zhi-Wei Xia, Juantao Tan, Yuan-Ming Li, Zhilin Zhao, and Wei-Shi Zheng. Beyond mimicry: Learning whole-body human-humanoid interaction from human-human demonstrations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 30740–30749, June 2026.

[3] Yinhuai Wang, Qihan Zhao, Yuen Fui Lau, Runyi Yu, Hok Wai Tsui, Qifeng Chen, Jingbo Wang, Jiangmiao Pang, and Ping Tan. HumanX: Toward agile and generalizable humanoid interaction skills from human videos. arXiv preprint arXiv:2602.02473, 2026.

Dear reviewer,

As the discussion deadline is approaching, we would be grateful if you could take a look at our responses and additional experimental results. Please let us know if any concerns remain or if there is anything we could further clarify during the discussion period.

Thank you again for your time and thoughtful feedback.

Reviewer_mTdxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents two separate methods: retargeting and whole-body motion tracking policy. For retargeting, they first applies a decoupled initialization where body motion is retargeted with a wrist target that interpolates between a robot-scaled position and a world-aligned position. Each hand’s finger pose is fitted by retrieving and blending the nearest entries from a precomputed codebook of feasible finger configurations. The second stage is a coupled refinement, which re-optimizes only the upper-body chain together, with priority-weighted thumb and index fingertip alignment and a softened wrist anchor. The result is a robot reference in which wrist placement and finger articulation are consistent with each other and with the object.

For tracking policy, they use PPO to train a per-sequence policy that tracks retargeted kinematic motion. The key design is an anatomy-aware Transformer where the robot observation is split into 8 regional tokens. Directed masked attention lets the hands see the body. Object information is injected only into the waist, arms, and hands, so leg actions are structurally independent of the object. Action heads are grouped by region, producing a 41-dimensional joint-position command for a Unitree G1 with Inspire hands.

Critical strengths
  1. This paper addresses an important problem of humanoid whole-body manipulation using human demonstration as reference motions. Human-to-humanoid transfer for dexterous tasks needs the body, wrist, and fingers to be consistent with each other and the object, and most prior pipelines treat body retargeting and hand retargeting as separate problems.
  2. The anatomically aware policy was well thought out. For example, splitting the observation into anatomical regions with a directed mask in which hands see the body but not the reverse, and injecting object information only into the upper-body part. Training is also simple with a single PPO stage and without teacher–student distillation or residual refinement.
Critical weaknesses
  1. No experiment shows that better retargeting leads to better policy outcomes. There's really no point for a retargeting method to exist if it does not make the policy better in the real world (not just in sim). I found it critically missing the real-world comparison of the same policy training recipe with DexWeave references versus a baseline retargeter’s references. If this evidence cannot be established, the entire retarget evaluation section is not useful.
  2. The real-world experiment is shown as execution snapshots on a G1 with Inspire hands, but they never provide no success rates, trial counts, or failure analysis.
  3. The paper emphasizes whole-body manipulation, but the success rate for object manipulation is very forgiving. The policy can only achieve 0.30m position error and 2.0 rad orientation error, which is very loose for dexterous manipulation.
Questions for authors

Please provide a real-world experiment that compares between your retargeting method and a baseline retargeting method. Please provide an ablation that demonstrates the contribution of anatomy conditioning, removing other confounders such as transformer vs MLP.

1 comment
AuthorsOpenReview ↗

Response to Reviewer mTdx (Part 2/2)

Q2: Can the authors provide a controlled ablation that isolates the contribution of anatomy conditioning?

A4: We conducted additional controlled ablations to isolate the contribution of anatomy conditioning. All ablation variants use the same Transformer architecture, eight robot-region tokens, grouped action heads, and training and evaluation protocols. We vary only the attention mask for different conditioning settings. The results are shown in following Table:

Attention settingSuccess rate (%) ↑Body pos. (cm) ↓Object pos. (cm) ↓Object rot. (deg) ↓
DexWeave (w/ MRM_R and MOM_O)94.64.673.638.83
w/o MRM_R72.87.868.7214.48
w/o MOM_O76.45.395.5511.21
Whole-body MOM_O81.25.004.548.92

It can show that DexWeave achieves 94.6% task success rate (with reviewer suggested stricter SR criteria) and the lowest mean body-position, object-position, and object-orientation tracking errors, which demonstrate the effectiveness of our network design. The detailed ablation settings are as follows:

  • Dense MR\boldsymbol{M_R} (w/o MR\boldsymbol{M_R}). We replace the directed mask with dense attention among all eight robot tokens, introducing hand-specific information into body-control representations, potentially weakening the separation between whole-body coordination and local grasp adjustments.
  • Bi-direction MO\boldsymbol{M_O} (w/o MO\boldsymbol{M_O}). We enable dense attention among the object token and five upper-body tokens, enabling bidirectional information exchange between the object and these upper-body regions. This introduces robot posture and hand features into object representation, potentially weakening its role as a dedicated conditioning signal for object state and tracking error.
  • Whole-body MO\boldsymbol{M_O}. We extend direct object conditioning to the base and both legs. This introduces object information into lower-body control representations, potentially weakening their focus on stable support and locomotion.
Reviewer_JgsHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DexWeave studies the transfer of human body–hand–object demonstrations to a G1 humanoid with Inspire hands. It initializes body and hand motion separately, then jointly refines arm, wrist, and finger configurations while retaining the initialized lower-body motion. A regional-token Transformer actor uses directed attention and selective object conditioning, and a separate policy is trained for each reference. Evaluation includes geometric retargeting metrics, body-only tracking, five manipulation references, attention-mask ablations, and qualitative hardware demonstrations. Table 3 reports 97.5% episode completion compared with 85.0% for an object-conditioned MLP of comparable parameter count, alongside lower body and object position errors. The authors present the combined retargeting and policy pipeline as a method for producing executable humanoid loco-manipulation skills.

Critical strengths
  1. DexWeave improves fingertip alignment over the tested modular retargeting baselines. Compared with the best arm-wrist IK baseline that holds fingers fixed, DexWeave reduces primary-fingertip error from 14.9 to 4.3 mm on GRAB and 30.0 to 7.2 mm on HUMOTO, with fewer colliding frames (Table 2). Table 12 supports the full refinement stage while reporting palm-orientation and penetration-depth tradeoffs. These are gains from the complete design; individual optimization choices remain coupled.

  2. The structured actor improves reference tracking against a matched MLP baseline. Table 3 uses the same embodiment and comparable parameter counts, with object input available to both policies. DexWeave achieves 97.5% completion versus 85.0% and lower position errors. Table 13 further tests attention masks and reports tradeoffs between completion and object-tracking accuracy.

Critical weaknesses
  1. The evaluation does not yet establish how reliably the full pipeline completes manipulation tasks from the prescribed initial state. Geometric alignment and tracking gains are supported, but Table 3's 97.5% completion counts reaching a reference end or time limit without early failure, and Appendix C.4 includes sampled-phase starts. These measures do not establish success at the final task goal. Please evaluate complete tasks from each reference's specified initial state and apply the same task-success criteria to the matched MLP baseline, to establish whether the tracking gains improve manipulation performance.
Questions for authors
  1. Relative to Body Transformer, which changes in regional grouping, directed body-hand routing, and object conditioning address the requirements of coordinated whole-body dexterous interaction? Which benefits do the existing ablations isolate?

  2. Can the existing first-frame rollouts be scored for final object goals and any required release or support conditions, separately from timeouts?

2 comments
AuthorsOpenReview ↗

Response to Reviewer JgsH

We thank the reviewer for the acknowledgement of the effectiveness of our method and the support of our work. We make replies to the questions from the reviewer as follows:

W1 & Q2: Can the tracking improvements translate into higher task success rates under explicit manipulation success criteria, compared with the matched MLP baseline?

A1: Apologize for the confusion. Table 3's 97.5% success rate is computed by rolling out the reference motion from first frame and scored for final object goals. To ensure reliability, we choose ten checkpoints for each sequence; For each checkpoint, we repeat ten rollouts. The success criteria is that the robot has to finish the whole motion sequence, and the object position/orientation error (relative to the reference object pose) should be smaller than 0.3 m and 40 degrees respectively, during manipulation. The settings are the same for other baselines. To evaluate in a more rigorous way, we tighten the success criteria to 0.1m and 20 degrees and report the corresponding metrics in following Table.

The "sampled-phase starts" mentioned in Appendix C.4 is used for better policy training, i.e. on how to initialize a sampled training episode and when to terminate the training episode, they are not used for task-success evaluations in Table 3.

We will revise the manuscript to make it clearer for the final version.

ActorSuccess rate (%) ↑Body pos. (cm) ↓Object pos. (cm) ↓Object rot. (deg) ↓
InterMimic45.416.7025.2326.31
Object MLP48.47.868.8217.97
DexWeave (Ours)94.64.673.638.83

 

Q1: How does DexWeave extend Body Transformer to support coordinated whole-body dexterous manipulation, and which architectural benefits are validated by the existing ablations?

A2: We thank the reviewer for this question. DexWeave shares a similar motivation with Body Transformer[1] (BoT) in exploiting robot embodiment as an inductive bias for policy learning, but differs in three key architectural aspects tailored to whole-body dexterous interaction:

  1. Regional grouping: Unlike BoT's joint-level graph representation, DexWeave organizes the humanoid into eight functional regions (base, legs, waist, arms, and hands), enabling structured coordination between locomotion and fine-grained manipulation.
  2. Directed body–hand routing: Unlike BoT's symmetric local attention, DexWeave allows hand tokens to access upstream body information without feeding hand-specific features back into body tokens. This supports dexterous control while maintaining a structured body-control representation.
  3. Selective object conditioning: DexWeave explicitly encodes object-state discrepancies and conditions only the waist, arms, and hands, keeping leg-action decoding independent of direct object observations while preserving whole-body coordination.

Ablation evidence (Appendix F, Table 13): Replacing structured robot attention with dense attention reduces completion from 100% to 91.30%, while allowing bidirectional object attention reduces it to 99.71%. The full design also achieves the lowest body-position error (4.58 cm). Additionally, DexWeave improves loco-manipulation success from 85.0% to 97.5% over a parameter-matched, object-conditioned MLP (Table 3). These results support the effectiveness of directed information flow and the overall structured architecture.

 

[1] Sferrazza C, Huang D M, Liu F, et al. Body transformer: Leveraging robot embodiment for policy learning[J]. arXiv preprint arXiv:2408.06316, 2024.

Dear reviewer,

As the discussion deadline is approaching, we would be grateful if you could take a look at our responses and additional experimental results. Please let us know if any concerns remain or if there is anything we could further clarify during the discussion period.

Thank you again for your time and thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.