- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EgoHumanoid-V2 transfers coordinated whole-body manipulation skills from egocentric human demonstrations to a humanoid robot. Its coarse-to-fine action alignment first corrects motion references using offline inverse kinematics, then refines motion tokens through a learned execution-response model and simulation feedback. Robot-arm rendering and image augmentation address visual differences between human demonstrations and robot deployment. The aligned data are used to fine-tune a separate pi-0.5 policy for each task, executed through a fixed SONIC whole-body controller. Experiments on a Unitree G1 cover closing a laptop, opening a refrigerator, pulling a curtain, and closing a drawer with the foot, without target-task robot demonstrations.
Critical strengths
-
The paper evaluates human-to-robot transfer on four real-world loco-manipulation tasks. I appreciate that the evaluation goes beyond motion tracking and tests whether the transferred demonstrations can actually be used to train manipulation policies.
-
The alignment ablations are fairly comprehensive. The endpoint position error decreases from 11.21 cm to 3.41 cm after the full pipeline. The additional experiments on 112 trajectories across 50 tasks also provide useful supporting evidence.
Critical weaknesses
-
I am concerned about the consistency of the generated image–action pairs. The robot arms are rendered using aligned joint states, but the background still comes from the human recording, even though refinement may change the robot's body motion. This could introduce inconsistencies in viewpoint or interaction location. Although the paper uses rendering and augmentation to reduce the visual gap, the remaining mismatch and its effect on policy learning are not evaluated.
-
The additional hardware benefit of dynamics-aware refinement and candidate selection is not very convincing. The mean task score increases from 47.09% after kinematic correction to 51.67% with the full pipeline. With only 20 trials per condition and no uncertainty estimates, it is difficult to judge how reliable this improvement is. Moreover, the metric gives partial credit for completing subtasks, so the results do not clearly establish whether the full pipeline improves complete-task success.
Additional Minor Concerns:
-
It is unclear how much the learned policies rely on online visual feedback. Since each task uses a separate policy and follows a relatively fixed procedure, a fixed-image ablation would help clarify this.
-
The practical cost advantage over robot demonstrations is not fully established. The reported processing time excludes failed attempts and subsequent selection, making the overall cost comparison difficult to assess.
Questions for authors
-
How much image–action mismatch remains after refinement? Is there any evidence that it affects policy learning?
-
The hardware score improves from 47.09% to 51.67%, but how reliable is this gain across trials? Uncertainty estimates and full-task success rates for each task would help clarify this.
-
Would the policies still perform similarly if the visual input were held fixed during execution? This could help determine whether they actually rely on online visual feedback.
-
How does the overall cost compare with collecting robot demonstrations when offline processing, failed attempts, and selection are included?
3 comments
Follow-up on the rebuttal
Thank you for the detailed responses and for the work that went into the additional experiments and revisions. The newly reported full-task success rates and standard errors, the fixed-image ablation, and the end-to-end processing cost breakdown address most of my empirical concerns. In particular, the full-task results clarify how dynamics-aware refinement improves task completion.
I would still like to see a direct quantitative measure of the remaining image–action mismatch. The overall labor advantage also depends on how many demonstrations are needed to reach comparable full-task success. On balance, I find the real-world human-to-humanoid transfer results solid and will raise my rating from 2 to 3.
Thank you for your constructive follow-up and for recognizing our additional experiments and real-world transfer results. We now quantify the remaining visual mismatch and compare total collection labor at similar full-task success rates.
1. Residual visual mismatch.
For Close Laptop, we compare human ego videos before and after visual alignment with robot ego recordings of the corresponding aligned motions. Using OpenPI SigLIP features, we match each frame to the most similar frame in the other video. We average cosine similarities over all frames in each direction, then take the mean of the two results. All results are averaged over 25 video pairs. Human–Human and Robot–Robot references pair videos 1–25 with 26–50.
| Ego pair | Mean cosine distance ↓ |
|---|---|
| Unaligned Human–Robot | 0.077 |
| Aligned Human–Robot | 0.049 |
| Human–Human | 0.033 |
| Robot–Robot | 0.030 |
\color{red}{\text{36.4\\%,}}
2. Total collection labor.
With 200 human vs. 50 robot demonstrations, full-task success reaches 50% vs. 50% on Close Laptop and 30% vs. 35% on Pull Curtain (Tables 6–7).
Across the two tasks, collection takes 6,660 vs. 7,042.5 person-seconds per task on average (Fig. 5). Human collection therefore requires 5.43% less labor, even with four times as many demonstrations.
Thank you for your detailed review. We have updated the submission PDF and highlighted revisions in blue.
We recommend opening our responsewebpage to follow the responses more easily and efficiently.
1. Interaction locations. The unchanged glove region preserves interaction-location alignment between images and actions. Human images and motions are recorded together. We use matching blue gloves on human and robot hands, preserve the glove region during rendering, and render robot arms from aligned joint states. This keeps the robot hands at the recorded interaction locations and consistent with the aligned actions (Sec. 3.4; Fig. 3).
2. Viewpoint alignment through hardware. Our chest-level human camera approximately matches the robot camera height, reducing the viewpoint gap at capture (Sec. 3.2; Fig. 2). Thus, the camera setup already limits the human–robot viewpoint difference before augmentation.
3. Residual viewpoint variation. Changes in robot body motion can still perturb the viewpoint. However, viewpoint variation need not be detrimental: we want the policy to learn spatial relationships and object interaction rather than overfit to a fixed viewpoint. Our augmentation ablation shows that stronger image augmentation, including viewpoint perturbations, improves policy performance, supporting this rationale. Across the three arm tasks, adding image augmentation improves the average task score from 27.78% to 42.78% without robot-arm rendering, and from 38.89% to 48.33% with it (Table 4). Action labels are unchanged across these visual conditions, so this comparison isolates the contribution of image augmentation to policy learning.
1. Error bars. D (MLP) uses a learned model for gradient-based optimization, similar to SuperMimic. We add SEs to Figs. 4–5 and Tables 1/4, shown as error bars and mean ± SE. From H+K to D (Unit) to D (MLP), mean task score rises 47.09→47.92→51.67% (Table 1). These gains support the contribution of D and the learned response model, with evaluation uncertainty now explicit. D (MLP) improves over D (Unit) on Close Laptop, Open Fridge and Close Drawer, suggesting benefits for precise interaction and stable whole-body coordination.
2. Full-task success. We add S1/S2/S3 success rates in Table 2. From H+K to D (Unit) to D (MLP), mean full-task success rises 23.75→30.00→38.75%. Full-task success increases from 30% to 50% on Close Laptop, 10% to 35% on Open Fridge, 25% to 30% on Pull Curtain and 30% to 40% on Close Drawer. The full pipeline therefore improves over H+K on all four tasks. Thus, D improves complete-task execution as well as intermediate progress. Table 11 also shows that D is selected for 97.1% of inputs (task mean), supporting its contribution to offline action alignment for most demonstrations.
We add a fixed-image ablation in simulation, replacing recorded video with a fixed first frame, black, fixed-noise or white images while retaining proprioceptive feedback. Fixed inputs yield mostly small posture changes, with little progression through the intended motion. These results show that visual feedback guides policy execution.
1. Complete processing cost. Table 10 reports per-worker K/D processing time; the added Table 11 reports 0.50 h/task on average using four workstations with 12 worker processes. This includes task-specific preparation, action and visual alignment, validation and candidate selection, including processing of rejected inputs (1.84%, task mean). Thus, the reported time covers the complete pipeline, with modest elapsed processing cost under this parallel setup.
2. Collection cost advantage. Human demonstrations require 33.30 vs. 140.85 person-seconds per demonstration: 76.36% less collection labor (Fig. 5). They also enable in-the-wild collection without a robot, avoiding robot procurement and maintenance for collection. The labor comparison includes capture, setup, calibration, troubleshooting and resets. The 76.36% saving concerns collection labor per demonstration. These advantages reduce collection labor and hardware requirements; both should be considered alongside processing costs.