- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DexWeave learns dexterous humanoid loco-manipulation skills from human demonstrations. It first retargets human motion to the robot, then jointly refines the arms, wrists, and fingers to improve hand–object alignment. An Anatomy-aware Transformer uses body-region tokens and masked attention to coordinate the body and hands, with object information provided to the upper-body branches. The authors train a separate policy for each reference motion and evaluate the method through retargeting benchmarks, simulation, and experiments on a Unitree G1 with Inspire hands.
Critical strengths
-
Joint upper-limb refinement helps preserve hand–object alignment. Optimizing the arms, wrists, and fingers together addresses a practical problem in transferring human demonstrations: a locally reasonable hand pose can still miss the object because the wrist is misplaced. The reported results show lower thumb/index fingertip errors and fewer penetration violations on GRAB and HUMOTO. The refinement comes with some loss in palm-orientation accuracy, but the gains in fingertip placement make this a useful part of the method.
-
The framework connects retargeting with whole-body execution. The authors carry the retargeted motions through policy learning and deployment on a physical robot. The policy jointly controls the body and hands without separate tracking-policy pretraining or residual refinement. It achieves higher episode completion than a similarly sized object-conditioned MLP on the evaluated motions. The hardware demonstrations add value by showing how the approach works on the target platform.
Critical weaknesses
-
The hardware sensing and initialization setup is unclear. The policy requires the current object pose, but I could not find how this is measured on the robot. The paper also leaves unclear how the reference trajectory is aligned with the real scene and how precisely the object must be placed before execution. The simulated observation noise and delays do not explain the actual sensing setup. These details are needed to understand the deployment conditions and reproduce the demonstrations.
-
The hardware results provide limited detail about finger coordination. The hardware images make it difficult to assess how the fingers participate in the tasks and whether the intended hand–object contacts are maintained. Richer examples, including tool use, appear in the retargeting results, but the paper does not clearly show whether these interactions are reproduced on hardware. Hand close-ups or an analysis of the existing robot experiments would help explain the role of finger articulation in the demonstrated tasks.
-
The reported success rate does not directly measure complete-task success. The success metric counts episodes that reach the reference end or time limit without early termination. Since some episodes start partway through a motion, this can mean completing only a segment. Results for complete motions starting from the first frame, with success defined by the manipulation outcome as well as tracking errors, would make the reported performance easier to interpret.
-
The architecture ablations leave some design choices insufficiently evaluated. The attention-mask ablations cover only two motions. The proposed masks give higher completion and better body tracking, while dense attention gives slightly better object-pose tracking. More motions would help determine whether this trade-off is consistent. The choice to exclude direct object information from the leg-control branches is also not tested separately, since the object-attention ablation retains the lower-body bypass. A comparison with direct object conditioning of the legs would help assess this choice, particularly for motions involving changing loads or lower-body adjustments during manipulation.
-
The related-work discussion does not fully position the contribution. The paper does not discuss Beyond Mimicry: Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations or HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos. Both connect interaction-aware data generation or retargeting with humanoid skill learning. Although they address different settings, discussing their relationship to DexWeave would help explain what the proposed arm–wrist–finger refinement adds.
Questions for authors
- Could the authors clarify how the current object pose is measured during hardware execution and how the reference trajectory is aligned with the physical scene? What range of initial object-placement errors can the system tolerate? Please specify the sensing frequency, measurement latency, and treatment of occlusion.
2.Could the authors provide close-up views of the hands during the existing hardware experiments? For the simulation evaluation, could completion rates from first-frame initialization be reported separately, along with task-specific success criteria? For the hardware experiments, please report the number of trials and successful executions.
- Could the authors extend the attention-mask evaluation to additional reference motions and clarify whether direct object conditioning of the leg-control branches has been evaluated?
5 comments
Response to Reviewer csSN (Part 2/2)
W3/Q2-2: Can the authors report complete-task success rates using task-specific criteria and first-frame initialization in simulation, along with trial counts and successful executions on hardware?
A3: Apologize for the confusion. Table 3's 97.5% success rate is computed by rolling out the reference motion from first frame and scored for final object goals. To ensure reliability, we choose ten checkpoints for each sequence; For each checkpoint, we repeat ten rollouts. The success criteria is that the robot has to finish the whole motion sequence, and the object position/orientation error (relative to the reference object pose) should be smaller than 0.3 m and 40 degrees respectively, during manipulation. The settings are the same for other baselines. We further tighten the success criteria to 0.1m and 20 degrees for object position/orientation error. The success rates for DexWeave, ObjectMLP, InterMimic are 94.6%, 48.4% and 45.4% respectively. For real hardware, we deploy the policies in 10 consecutive trials. The average success rate is around 82.5%.
W4/Q3: Can the authors extend the attention-mask ablations to more reference motions and evaluate whether direct object conditioning of the leg-control branches improves performance?
A4: We thank the reviewer for these valuable suggestions. We have extended our attention-mask ablations from two to five representative motions and introduced an additional variant, Whole-body , which allows the leg-control branches to directly access object information.
| Attention setting | Success rate (%) ↑ | Body pos. (cm) ↓ | Object pos. (cm) ↓ | Object rot. (deg) ↓ |
|---|---|---|---|---|
| DexWeave (w/ and ) | 94.6 | 4.67 | 3.63 | 8.83 |
| w/o | 72.8 | 7.86 | 8.72 | 14.48 |
| w/o | 76.4 | 5.39 | 5.55 | 11.21 |
| Whole-body | 81.2 | 5.00 | 4.54 | 8.92 |
As shown in prior table, DexWeave consistently achieves the best average performance across all four metrics, with a 94.6% success rate, compared with 81.2% Whole-body . It demonstrates that directly conditioning leg control on object observations does not necessarily improve performance. Instead, our selective object conditioning achieves a 13.4 percentage-point higher success rate, supporting our design choice to coordinate lower-body control through shared body-state information rather than direct object observations. The reason might be that even the leg cannot observe the object information directly, it is able to know how to keep balance with its propriceptive measurements indirectly.
These additional results provide empirical evidence for the effectiveness of our structured attention and selective object conditioning in whole-body dexterous interaction.
W5: How does DexWeave relate to Beyond Mimicry[2] and HumanX[3], and can the authors clarify its contributions relative to these works?
A5: We thank the reviewer for highlighting these relevant works. We agree that both share a similar motivation of transferring human interaction demonstrations into executable humanoid skills. Unlike these approaches, DexWeave specifically addresses fine-grained dexterous hand–object interactions through coupled arm–wrist–finger refinement, jointly optimizing wrist placement and finger articulation while preserving lower-body support. We will include these works and clarify the distinctions in our revised related-work discussions.
[2] Wei-Jin Huang, Yue-Yi Zhang, Yi-Lin Wei, Zhi-Wei Xia, Juantao Tan, Yuan-Ming Li, Zhilin Zhao, and Wei-Shi Zheng. Beyond mimicry: Learning whole-body human-humanoid interaction from human-human demonstrations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 30740–30749, June 2026.
[3] Yinhuai Wang, Qihan Zhao, Yuen Fui Lau, Runyi Yu, Hok Wai Tsui, Qifeng Chen, Jingbo Wang, Jiangmiao Pang, and Ping Tan. HumanX: Toward agile and generalizable humanoid interaction skills from human videos. arXiv preprint arXiv:2602.02473, 2026.
Dear reviewer,
As the discussion deadline is approaching, we would be grateful if you could take a look at our responses and additional experimental results. Please let us know if any concerns remain or if there is anything we could further clarify during the discussion period.
Thank you again for your time and thoughtful feedback.
Response to Reviewer csSN (Part 1/2)
We thank the reviewer for recognizing the effectiveness of our joint upper-limb refinement in preserving hand–object interactions, as well as the value of our unified framework for translating human demonstrations into physically executable humanoid loco-manipulation skills.
W1/Q1: How are object pose sensing and reference trajectory alignment implemented in real-world execution, and how robust is the system to initialization errors, sensing latency, and occlusion?
A1: Thank you for pointing out this omission. Our hardware deployment implementation estimates the object’s 6-DoF pose using an onboard wide-angle camera (640×480, configured at 30 FPS) and object-mounted AprilTags. Calibrated camera intrinsics, tag-to-object transforms, and camera-to-torso extrinsics convert detections into torso-frame object observations, delivered to the policy through DDS. Robot state estimation combines FAST-LIO odometry from the onboard MID-360 LiDAR–IMU for position estimation, and the robot’s IMU for orientation.
For real deployment, it contains two stages, i.e. a standby stage and a task policy executation stage. The standby stage is based on a seperate locomotion controller. We pad 2-3 seconds transition frames between the standby pose and the task first frame pose. The padding is also applied to the end of the task frame, such that the standby->task and task->standby switchings are smooth. At task initialization, a fixed horizontal translation and yaw rotation align the reference robot’s initial anchor pose with the robot’s current pose. This transform is applied consistently to the robot and object reference trajectories and remains fixed throughout execution; reference joint angles and velocities are unchanged.
The operator places the object near its reference starting pose relative to the robot. Since we applied initial pose DR during training, the policy is able to tolerate the initialization errors to some extent (i.e. the position error tolerance is around 5 cm and the orientation error is around 10 degrees). We did not test the effect of sensing latency in specific, the whole system is evaluated only under the nominal operation condition. We do observe occlusion issues with the AprilTag for some tasks, especially after the object is grasped by the dexterous hands. For those tasks, we only used the AprilTag pose for precise grasping, after the object is grasped (i.e. the AprilTag is being occluded), we switch the measured object pose to reference object pose to continue the rest of the motions.
We will include those deployment details in the final version.
W2/Q2-1: Can the authors provide close-up views of the hands during real-world experiments to demonstrate finger coordination and hand–object contact preservation? Can the authors demonstrate whether the diverse interactions shown in retargeting results, particularly tool use, can be successfully reproduced on real hardware?
A2: We thank the reviewer for this constructive suggestion. To better illustrate the role of dexterous finger articulation during manipulation, we provide additional close-up videos of our real-world experiments. The video links are Lifting a basket, Transferring a basket , Pouring water from a cup and Lifting a trash can. We further conduct real experiment with one more stirring food using a spatula motion sequence, as a tool use example. The real-world video is provided at Stirring food using a spatula.