acceptodds
Under review as a conference paper at ICLR 2027

EgoHumanoid-V2: Human-to-Humanoid Transfer of Coordinated Whole-Body Skills for Loco-Manipulation

12 views

Abstract

Human demonstrations capture diverse scenes and rich whole-body skills without requiring robot teleoperation. Prior work on egocentric transfer has emphasized scene generalization in loco-manipulation under decoupled control, leaving direct transfer of coordinated whole-body skills less explored. We present **EgoHumanoid-V2**, a human-to-humanoid framework to transfer coordinated whole-body loco-manipulation skills from egocentric demonstrations. At its core, coarse-to-fine action alignment combines kinematic reference correction with dynamics-aware refinement. It improves end-effector pose accuracy while preserving whole-body coordination. We also use robot-arm rendering and training-time image augmentation to reduce the visual embodiment gap and improve viewpoint robustness. On four real-world tasks, vision-language-action (VLA) policies trained on aligned human data show zero-shot skill transfer without target-task robot demonstrations. Task scores are comparable to those of policies trained on teleoperation data at a lower collection cost. These results support human data as direct skill supervision. Demos are available at https://egohumanoid-v2-iclr2027.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human-to-Humanoid Transfer, Coordinated Whole-Body Skills, Loco-Manipulation, Coordinated Whole-Body Loco-Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fQQ5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoHumanoid-V2 transfers coordinated whole-body manipulation skills from egocentric human demonstrations to a humanoid robot. Its coarse-to-fine action alignment first corrects motion references using offline inverse kinematics, then refines motion tokens through a learned execution-response model and simulation feedback. Robot-arm rendering and image augmentation address visual differences between human demonstrations and robot deployment. The aligned data are used to fine-tune a separate pi-0.5 policy for each task, executed through a fixed SONIC whole-body controller. Experiments on a Unitree G1 cover closing a laptop, opening a refrigerator, pulling a curtain, and closing a drawer with the foot, without target-task robot demonstrations.

Critical strengths
  1. The paper evaluates human-to-robot transfer on four real-world loco-manipulation tasks. I appreciate that the evaluation goes beyond motion tracking and tests whether the transferred demonstrations can actually be used to train manipulation policies.

  2. The alignment ablations are fairly comprehensive. The endpoint position error decreases from 11.21 cm to 3.41 cm after the full pipeline. The additional experiments on 112 trajectories across 50 tasks also provide useful supporting evidence.

Critical weaknesses
  1. I am concerned about the consistency of the generated image–action pairs. The robot arms are rendered using aligned joint states, but the background still comes from the human recording, even though refinement may change the robot's body motion. This could introduce inconsistencies in viewpoint or interaction location. Although the paper uses rendering and augmentation to reduce the visual gap, the remaining mismatch and its effect on policy learning are not evaluated.

  2. The additional hardware benefit of dynamics-aware refinement and candidate selection is not very convincing. The mean task score increases from 47.09% after kinematic correction to 51.67% with the full pipeline. With only 20 trials per condition and no uncertainty estimates, it is difficult to judge how reliable this improvement is. Moreover, the metric gives partial credit for completing subtasks, so the results do not clearly establish whether the full pipeline improves complete-task success.

Additional Minor Concerns:

  1. It is unclear how much the learned policies rely on online visual feedback. Since each task uses a separate policy and follows a relatively fixed procedure, a fixed-image ablation would help clarify this.

  2. The practical cost advantage over robot demonstrations is not fully established. The reported processing time excludes failed attempts and subsequent selection, making the overall cost comparison difficult to assess.

Questions for authors
  1. How much image–action mismatch remains after refinement? Is there any evidence that it affects policy learning?

  2. The hardware score improves from 47.09% to 51.67%, but how reliable is this gain across trials? Uncertainty estimates and full-task success rates for each task would help clarify this.

  3. Would the policies still perform similarly if the visual input were held fixed during execution? This could help determine whether they actually rely on online visual feedback.

  4. How does the overall cost compare with collecting robot demonstrations when offline processing, failed attempts, and selection are included?

3 comments

Thank you for your detailed review. We have updated the submission PDF and highlighted revisions in blue.

We recommend opening our responsewebpage\color{red}{\mathbf{response\\ webpage}} to follow the responses more easily and efficiently.

\color{#B6321C}{\text{Weakness 1 \\& Question 1:}} Image–action consistency.

1. Interaction locations. The unchanged glove region preserves interaction-location alignment between images and actions. Human images and motions are recorded together. We use matching blue gloves on human and robot hands, preserve the glove region during rendering, and render robot arms from aligned joint states. This keeps the robot hands at the recorded interaction locations and consistent with the aligned actions (Sec. 3.4; Fig. 3).

2. Viewpoint alignment through hardware. Our chest-level human camera approximately matches the robot camera height, reducing the viewpoint gap at capture (Sec. 3.2; Fig. 2). Thus, the camera setup already limits the human–robot viewpoint difference before augmentation.

3. Residual viewpoint variation. Changes in robot body motion can still perturb the viewpoint. However, viewpoint variation need not be detrimental: we want the policy to learn spatial relationships and object interaction rather than overfit to a fixed viewpoint. Our augmentation ablation shows that stronger image augmentation, including viewpoint perturbations, improves policy performance, supporting this rationale. Across the three arm tasks, adding image augmentation improves the average task score from 27.78% to 42.78% without robot-arm rendering, and from 38.89% to 48.33% with it (Table 4). Action labels are unchanged across these visual conditions, so this comparison isolates the contribution of image augmentation to policy learning.

\color{#B6321C}{\text{Weakness 2 \\& Question 2:}} Full-task success and standard errors.

1. Error bars. D (MLP) uses a learned model for gradient-based optimization, similar to SuperMimic. We add SEs to Figs. 4–5 and Tables 1/4, shown as error bars and mean ± SE. From H+K to D (Unit) to D (MLP), mean task score rises 47.09→47.92→51.67% (Table 1). These gains support the contribution of D and the learned response model, with evaluation uncertainty now explicit. D (MLP) improves over D (Unit) on Close Laptop, Open Fridge and Close Drawer, suggesting benefits for precise interaction and stable whole-body coordination.

2. Full-task success. We add S1/S2/S3 success rates in Table 2. From H+K to D (Unit) to D (MLP), mean full-task success rises 23.75→30.00→38.75%. Full-task success increases from 30% to 50% on Close Laptop, 10% to 35% on Open Fridge, 25% to 30% on Pull Curtain and 30% to 40% on Close Drawer. The full pipeline therefore improves over H+K on all four tasks. Thus, D improves complete-task execution as well as intermediate progress. Table 11 also shows that D is selected for 97.1% of inputs (task mean), supporting its contribution to offline action alignment for most demonstrations.

\color{#B6321C}{\text{Minor Concern 1 \\& Question 3:}} Visual feedback.

We add a fixed-image ablation in simulation, replacing recorded video with a fixed first frame, black, fixed-noise or white images while retaining proprioceptive feedback. Fixed inputs yield mostly small posture changes, with little progression through the intended motion. These results show that visual feedback guides policy execution.

\color{#B6321C}{\text{Minor Concern 2 \\& Question 4:}} Processing and collection costs.

1. Complete processing cost. Table 10 reports per-worker K/D processing time; the added Table 11 reports 0.50 h/task on average using four workstations with 12 worker processes. This includes task-specific preparation, action and visual alignment, validation and candidate selection, including processing of rejected inputs (1.84%, task mean). Thus, the reported time covers the complete pipeline, with modest elapsed processing cost under this parallel setup.

2. Collection cost advantage. Human demonstrations require 33.30 vs. 140.85 person-seconds per demonstration: 76.36% less collection labor (Fig. 5). They also enable in-the-wild collection without a robot, avoiding robot procurement and maintenance for collection. The labor comparison includes capture, setup, calibration, troubleshooting and resets. The 76.36% saving concerns collection labor per demonstration. These advantages reduce collection labor and hardware requirements; both should be considered alongside processing costs.

Reviewer_fQQ5OpenReview ↗

Follow-up on the rebuttal

Thank you for the detailed responses and for the work that went into the additional experiments and revisions. The newly reported full-task success rates and standard errors, the fixed-image ablation, and the end-to-end processing cost breakdown address most of my empirical concerns. In particular, the full-task results clarify how dynamics-aware refinement improves task completion.

I would still like to see a direct quantitative measure of the remaining image–action mismatch. The overall labor advantage also depends on how many demonstrations are needed to reach comparable full-task success. On balance, I find the real-world human-to-humanoid transfer results solid and will raise my rating from 2 to 3.

Thank you for your constructive follow-up and for recognizing our additional experiments and real-world transfer results. We now quantify the remaining visual mismatch and compare total collection labor at similar full-task success rates.

1. Residual visual mismatch.

For Close Laptop, we compare human ego videos before and after visual alignment with robot ego recordings of the corresponding aligned motions. Using OpenPI SigLIP features, we match each frame to the most similar frame in the other video. We average cosine similarities over all frames in each direction, then take the mean of the two results. All results are averaged over 25 video pairs. Human–Human and Robot–Robot references pair videos 1–25 with 26–50.

Ego pairMean cosine distance ↓
Unaligned Human–Robot0.077
Aligned Human–Robot0.049
Human–Human0.033
Robot–Robot0.030

Visual\color{red}{\text{Visual}} alignment\color{red}{\text{alignment}} reduces\color{red}{\text{reduces}} the\color{red}{\text{the}} human–robot\color{red}{\text{human–robot}} feature\color{red}{\text{feature}} distance\color{red}{\text{distance}} by\color{red}{\text{by}} approximately\color{red}{\text{approximately}} \color{red}{\text{36.4\\%,}} bringing\color{red}{\text{bringing}} it\color{red}{\text{it}} closer\color{red}{\text{closer}} to\color{red}{\text{to}} the\color{red}{\text{the}} variation\color{red}{\text{variation}} between\color{red}{\text{between}} robot\color{red}{\text{robot}} recordings\color{red}{\text{recordings}} (0.049\color{red}{\text{(0.049}} vs.\color{red}{\text{vs.}} 0.030).\color{red}{\text{0.030).}}

2. Total collection labor.

With 200 human vs. 50 robot demonstrations, full-task success reaches 50% vs. 50% on Close Laptop and 30% vs. 35% on Pull Curtain (Tables 6–7).

Across the two tasks, collection takes 6,660 vs. 7,042.5 person-seconds per task on average (Fig. 5). Human collection therefore requires 5.43% less labor, even with four times as many demonstrations.

Reviewer_vftaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoHumanoid-V2, a framework for transferring coordinated whole-body loco-manipulation skills from egocentric human demonstrations to a humanoid robot (Unitree G1 with BrainCo Revo2 hands) without requiring target-task robot teleoperation data. To resolve the conflict between end-effector tracking accuracy and whole-body balance under a fixed low-level whole-body tracking controller (SONIC), the authors propose an offline coarse-to-fine action alignment pipeline.

Specifically, the coarse stage (Kinematic Reference Correction, K) uses constrained offline inverse kinematics (IK) to reduce geometric mismatch before encoding motions into SONIC tokens, while the fine stage (Dynamics-Aware Refinement, D) utilizes simulation rollout feedback and a learned MLP execution response model to iteratively optimize the motion tokens. Additionally, the framework employs SAM2 segmentation, DiffuEraser video inpainting, robot-arm rendering, and training-time image augmentation to bridge the visual embodiment and viewpoint gaps.

Using the aligned data to post-train task-specific pi0.5 VLA policies, the method achieves a mean task score of 51.67% across four real-world whole-body manipulation tasks (Close Laptop, Open Fridge, Pull Curtain, and Close Drawer) with 200 human demonstrations per task, reducing per-episode human collection labor by 76.36% compared to robot teleoperation.

Critical strengths

The paper offers an insightful formulation of offline token-space action alignment that preserves whole-body coordination. A critical bottleneck in humanoid loco-manipulation is that unified whole-body tracking controllers (SONIC) suffer from large end-effector errors due to embodiment mismatch, whereas overriding joint commands via online IK disrupts closed-loop balance and causes falls. By shifting the correction offline and optimizing the input motion tokens rather than overriding the controller's outputs, the paper provides a practical and principled way to achieve high endpoint accuracy while retaining the low-level tracker's coordinated balance.

Secondly, the paper describes comprehensive real-world system integration and validation across diverse whole-body modalitiess. Comprehensive real-world system integration and validation across diverse whole-body modalities: The paper demonstrates a complete, working zero-shot human-to-humanoid pipeline across four distinct real-world tasks under a single shared SONIC checkpoint. The ablation studies across 4 real-world tasks and 50 simulation tasks clearly verify the complementarity between kinematic reference correction (K) and iterative execution feedback (D).

Critical weaknesses

First of all, the paper is lacking in terms of its free-space alignment assumption and potentially misleading partial-completion evaluation metrics. The entire offline action alignment pipeline replays motions in empty simulation without task objects or contact forces. This free-space assumption breaks down in contact-rich tasks requiring sustained force against resistance. Moreover, all main-text results report "Task score (%)" averaged over three sequential binary subtasks without error bars. For instance, a policy that merely approaches and touches the fridge door handle in every trial without ever opening the door still receives a 33.33% to 66.67% score, obscuring the true end-to-end task completion rate.

Secondly, data-efficiency bottleneck and hidden backend engineering costs associated with the proposed method undermine the core scalability premise. While raw front-end recording is faster for human demonstrations than robot teleoperation, human data suffers from severe sample inefficiency—policies trained on 200 human episodes (4x the data volume) still fall short of those trained on just 50 robot demonstrations on both Close Laptop and Pull Curtain. Furthermore, this cost accounting overlooks substantial backend overhead, including episode rejection under strict quality thresholds, heavy offline optimization compute, and hand-crafted task-specific rules, which collectively limit the framework's true scalability in practice.

Questions for authors

Q1: Since the reported "Task score (%)" averages over three sequential binary subtasks and measures partial completion rather than full task completion (Appendix B.3, B.4), it is difficult to gauge the true end-to-end reliability of the deployed policies. Could the authors provide the exact completion rates for each individual subtask—specifically Subtask 1, Subtask 2, and Subtask 3 (the full-task success rate)—along with standard errors across the 20 trials for the main experiments (Figures 4 & 5, Tables 1 & 4)?

Q2: Because the entire offline alignment pipeline (H, K, and D) replays trajectories in empty simulation without task objects or contact forces (Figures 8-10), the controller does not account for physical resistance during refinement. In contact-rich tasks such as Open Fridge and Pull Curtain, what proportion of real-world failures occur during the physical interaction phase due to insufficient contact force, hand slippage, or unexpected balance disturbances once external object resistance is encountered?

Q3: Appendix A.4 notes that different tasks require custom profiles, including "refrigerator trajectory conditioning and curtain hand-command and sequence-end processing". Could the authors clarify the exact hand-crafted rules involved in these profiles and how much policy performance degrades if they are removed? Additionally, given that policies trained on 200 human demonstrations still underperform those trained on 50 robot demonstrations on Close Laptop and Pull Curtain (Figure 5), what is the primary bottleneck causing this performance ceiling, and would collecting more human data close this gap or has performance already plateaued?

A suggestion:

The authors should consider incorporating virtual contact force or impedance proxies into the simulation rollout during Stage D to help bridge the gap between free-space kinematic tracking and force-intensive real-world manipulation.

1 comment

Thank you for your detailed review. We have updated the submission PDF and highlighted revisions in blue.

We recommend responsewebpage\color{red}{\mathbf{response\\ webpage}} for easier, faster reading.

\color{#B6321C}{\text{Weakness 1 \\& Question 1:}} Object forces, success rates and SEs.

1. Object interaction forces. Action alignment need not explicitly simulate object forces since SONIC’s closed-loop feedback maintains tracking under resistance. We add a 0–20 N opposing-force experiment with fixed motions and controller. Fridge/Curtain position errors change 1.80→2.09 / 2.78→3.72 cm, and orientation errors 3.45→4.32 / 7.91→8.66°. These small error increases demonstrate robust tracking under the tested resistance without object-force simulation in action alignment.

2. Subtask and full-task success. We now report subtask definitions (B.3) and S1/S2/S3 success rates for action/visual ablations (Tables 2/5) and all demonstration budgets (Tables 6–7), covering Tables 1/4 and Figs. 4–5. Our full pipeline achieves 50%/35%/30%/40% full-task success on Laptop/Fridge/Curtain/Drawer, averaging 38.75%. These rates specify full-task completion.

3. Standard errors. We add SEs to Figs. 4–5 and Tables 1/4, shown as error bars and mean ± SE, to quantify uncertainty in mean task scores.

Weakness 2:\color{#B6321C}{\text{Weakness 2:}} Sample efficiency and scalability.

1. Performance. Human demonstrations achieve similar full-task success with lower sample efficiency. With 200 human vs. 50 robot demonstrations, Laptop scores 53.33% vs. 63.33%, but full-task success is 50% for both. Curtain scores 53.33% vs. 56.67%, with full-task success 30% vs. 35% (Tables 6–7).

2. Processing and collection costs. Beyond Table 10’s per-worker K/D timings, Table 11 adds complete pipeline time, averaging 0.50 h/task using four workstations with 12 worker processes. This covers task-specific preparation, action and visual alignment, validation and candidate selection, including rejected inputs. Only 1.84% of inputs are rejected (task mean), so selection loses little data. In addition, human data require 76.36% less collection labor per demo and allow in-the-wild collection without a robot, avoiding robot procurement and maintenance for collection. Fair comparisons must consider these factors.

\color{#B6321C}{\text{Question 2 \\& Suggestion:}} Interaction failures and external forces.

1. Failure analysis. Balance loss is not a major failure source. Fig. 7 and B.4 analyze 40 trials per task: 20 each for D (Unit) and D (MLP). Of 28 failures each on Fridge and Curtain, balance loss causes 1/28 (3.6%) and 0/28, respectively. No failures due to insufficient contact force or hand slippage were observed in either task (0/28 each).

2. External forces and proxies. At 0→20 N, Fridge/Curtain position errors rise only 1.80→2.09 / 2.78→3.72 cm in our added opposing-force experiment, supporting tracking robustness through SONIC’s closed-loop feedback. We thank the reviewer for suggesting contact-force or impedance proxies in D, a promising future direction.

Question 3:\color{#B6321C}{\text{Question 3:}} Data preparation and scaling.

1. Task-specific preparation. A.4 details hand-command binarization for Fridge/Curtain, trajectory filtering to reduce noise, and tail trimming to remove incidental Curtain movements. These reduce noise and simplify learning for whole-body coordinated skill transfer from human data. We consider them minor steps unlikely to alter our conclusions.

2. Bottlenecks and scaling. With 200 human vs. 50 robot demos, Laptop scores 53.33% vs. 63.33%, with 50% full-task success for both. Curtain scores 53.33% vs. 56.67%, with 30% vs. 35% success. Thus, human data achieve similar full-task success despite lower sample efficiency. Additionally, fair comparisons must consider costs and in-the-wild collection without a robot.

To explain the remaining performance gap, our failure analysis identifies policy stalls in 30% of Laptop, 32.5% of Fridge and 12.5% of Curtain trials (Fig. 7). These stalls and premature stopping point to residual visual gaps with ego-only input, even after visual alignment. Motion-capture noise and controller tracking errors also limit action-data quality. Wrist cameras and improved capture hardware directly target these bottlenecks. Increasing human data also improves performance: with 25/50/100/200 demos, Laptop scores are 35.00/31.67/48.33/53.33%, and Curtain scores are 40.00/38.33/50.00/53.33% (Fig. 4). Both tasks continue to improve from 100 to 200 demos, motivating our expectation of further gains with more human data.

Reviewer_mSNHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EgoHumanoid-V2 transfers human demonstrations into humanoid loco-manipulation policies through coarse-to-fine action alignment. Offline inverse kinematics corrects motion references, followed by iterative motion-token refinement using a learned execution-response model under a fixed SONIC controller. Robot-arm rendering and image augmentation reduce visual differences between embodiments. Experiments demonstrate transfer to four real-world tasks without target-task robot demonstrations, supplemented by broader simulation-based alignment studies.

Critical strengths
  1. Relevant and practical problem. Learning coordinated humanoid skills from human demonstrations addresses an important bottleneck in embodied visual learning. Including both hand manipulation and foot-operated drawer closing provides useful diversity.

  2. Well-motivated refinement strategy. Correcting references offline while retaining a fixed whole-body controller is a sensible way to improve interaction accuracy without directly overriding coordinated control commands. The stage-wise experiments support the complementary roles of kinematic correction and iterative refinement.

  3. Substantial experimental effort and clear organization. The paper includes hardware evaluation, action and visual ablations, alternative motion initializations, collection-labor comparisons, and an expanded simulation study. The pipeline figures are informative, and the appendix explicitly distinguishes partial-completion scores from full-task success.

Critical weaknesses
  1. The incremental contribution needs stronger comparative evidence. EgoHumanoid [1] already combines action and visual alignment for egocentric human-to-humanoid transfer, while HuMI [2] and BifrostUMI [3] demonstrate whole-body learning from robot-free demonstrations. The distinctive contribution here is therefore primarily offline execution-aware refinement under a frozen controller. Although these predecessors are acknowledged, the experiments do not compare their complete pipelines; the HuMI-style initialization comparison explicitly evaluates only one component. Online Proceedings

  2. The learned response model’s practical contribution is not fully established. Table 1 indicates that kinematic correction provides much of the improvement. Tables 3 and 9 show relatively small additional benefits from the learned response for hand tasks and the expanded benchmark, with a more substantial benefit for the foot task. Appendix A.3 also blends the learned and unit responses for palm tasks. A real-world policy comparison against unit-response refinement is needed to establish whether learning the response materially improves autonomous execution.

  3. Performance and generalization evidence remain limited. The primary metric rewards intermediate progress, so favorable scores need not imply reliable task completion. Appendix B.4 reports no error bars, and repeated training seeds are not documented, making modest ablation differences difficult to assess. Separate policies are trained for each task; the expanded simulation benchmark evaluates motion alignment rather than autonomous policy generalization. Held-out environments, objects, and demonstrators are not evaluated through a sufficiently explicit protocol.

  4. Important implementation and scalability details remain incomplete. Equation 6 leaves the optimization neighborhood and several regularization settings underspecified. Appendix A.4 mentions task-specific trajectory and hand-command processing without describing it sufficiently for reproduction. Collection labor is informative, but rejection rates and the full processing burden are needed to assess practical scalability, particularly because the timing study excludes failed attempts and subsequent selection and validation.

[1] Shi et al. Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration. RSS, 2026.

[2] Nai et al. Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations. arXiv:2602.06643, 2026.

[3] Wang et al. BifrostUMI: Bridging Robot-Free Demonstrations and Humanoid Whole-Body Manipulation. arXiv:2605.03452, 2026.

Questions for authors
  1. Report full-task success alongside partial scores, including per-subtask outcomes, trial-level confidence intervals, multiple training seeds, and failure categories.

  2. Strengthen the refinement ablation with real-world comparisons of learned, unit, and linear response models. Report how often refinement improves upon kinematic correction, how often earlier candidates are retained, and how many demonstrations are rejected.

  3. Validate interaction accuracy beyond local tracking. The pelvis-centered formulation should be complemented by world- or object-relative errors and contact-phase stability measurements. Local endpoint accuracy alone does not establish correct placement relative to the manipulated object. Clarify whether refinement rollouts model task-object contacts.

  4. Make the zero-shot scope explicit throughout: transfer to the robot without target-task robot demonstrations, following task-specific training on human demonstrations.

3 comments

Thank you for your detailed review. We have updated the submission PDF and highlighted revisions in blue.

We recommend our responsewebpage\color{red}{\mathbf{response\\ webpage}} to ease reading.

Weakness 1:\color{#B6321C}{\text{Weakness 1:}} Prior-work comparisons.

1. Algorithm vs. EgoHumanoid. To transfer coordinated whole-body motion, we replace its decoupled controller with SONIC. However, applying direct online IK control from endpoint pose deltas disrupts SONIC’s balance feedback in our tests (Appendix D; Table 14). We therefore use offline action alignment to preserve SONIC’s feedback control (Sec. 3.3). For visual alignment, EgoHumanoid uses reprojection/inpainting; we use simpler camera-height matching, augmentation and robot-arm rendering (Figs. 2–3).

2. Hardware vs. HuMI/BifrostUMI. Our VR capture requires no handheld or matched robot grippers, enabling human-to-dexterous-hand transfer with lower gripper costs and scalable capture. Egocentric images, rather than wrist views, provide scene context for loco-manipulation (Pull Curtain). These differences create geometry/viewpoint gaps, addressed by two-stage offline action alignment and visual alignment.

3. Controller vs. HuMI/BifrostUMI. Endpoint tracking errors are a major source of the action gap. HuMI trains its controller on downstream demos; BifrostUMI also trains its own tracker. Task-specific controller tuning may be needed on new tasks. We use open-source SONIC without retraining.

Fair comparisons must reflect these differences.

\color{#B6321C}{\text{Weakness 2 \\& Question 2:}} D and response model benefit.

1. Real-world results. D (MLP) uses a learned model for gradient-based optimization, similar to SuperMimic. H+K→D (Unit)→D (MLP) raises mean scores 47.09→47.92→51.67% (Table 1), supporting D and its learned response model. D (MLP) improves over D (Unit) on Laptop, Fridge and Drawer, suggesting benefits for precise interaction and stable whole-body coordination.

2. Selection (Table 11). D is selected for 97.1% of inputs (task mean): most demonstrations benefit from D in action alignment.

\color{#B6321C}{\text{Weakness 3, Questions 1 \\& 4:}} Success and transfer scope.

1. Full-task success. We report subtask definitions, stage-wise success and failure analysis (Table 2; B.3–B.4). D (MLP) raises mean full-task success from 23.75% with H+K to 38.75%, improving all four tasks.

2. SEs and seed. We add SEs to Figs. 4–5 and Tables 1/4. Models share a training seed per task (B.5); SEs quantify uncertainty in mean task scores.

3. Research scope. We study whole-body loco-manipulation skill transfer from egocentric human data, with one policy per task. Unseen objects/scenes/tasks and instruction following are outside scope. Two collectors alternate. Tests randomly perturb object/robot poses in the same scenes. With 25→200 demos, mean scores rise 38.34→51.67% (Fig. 4), supporting within-task visual generalization. Fixed images yield little motion progress in our added ablation, also supporting visual feedback.

4. Zero-shot scope. Accordingly, zero-shot means transfer without target-task robot demonstrations, after task-specific training on human data, not unseen-task generalization (abstract; Secs. 3.1/4.1).

Weakness 4:\color{#B6321C}{\text{Weakness 4:}} Implementation and cost.

1. Optimization. A.2 details Eq. 6’s optimization bounds, regularization and schedule.

2. Data preparation. A.4 details hand-command binarization, trajectory filtering and tail trimming. We regard these as minor steps unlikely to affect our conclusions.

3. Selection and cost. Table 11 reports mean 1.84% rejection and 0.50 h/task on four workstations for the full pipeline: preparation, action/visual alignment, validation and selection.

Question 3:\color{#B6321C}{\text{Question 3:}} Interaction accuracy and stability.

1. Local coordinates. Local coordinates suffice for action alignment in these tasks because visual feedback handles pose adaptation. The policy learns when to turn, move and stop (e.g. Pull Curtain) and adapts arm actions to pose perturbations (e.g. Open Fridge).

2. Contact stability. Balance loss causes 4 failures in 160 trials (2.5%) (Fig. 7; B.4), so it is not a major failure source.

3. Object forces. SONIC’s closed-loop feedback limits tracking degradation without simulated object forces. Our added 0–20 N opposing-force experiment with fixed motions shows Fridge/Curtain position errors of 1.80→2.09 / 2.78→3.72 cm. This supports the design within the tested range.

I appreciate the authors’ careful responses and the effort in the additional experiments. After rereading the paper, particularly the method section, and reviewing the response to the first weakness I identified, I still believe that the methodological novelty lies in Section 3.3.2, “Dynamics-Aware Refinement.” However, I cannot persuade myself to consider this sufficient to warrant acceptance at a top-tier venue. I therefore have to maintain my original rating.

Thank you for rereading our paper and recognizing the contribution of D. We understand that you still find the methodological novelty insufficient, though the specific remaining concern is not yet clear to us.

Our\color{red}{\text{Our}} main\color{red}{\text{main}} contribution\color{red}{\text{contribution}} is\color{red}{\text{is}} the\color{red}{\text{the}} offline\color{red}{\text{offline}} action\color{red}{\text{action}} alignment\color{red}{\text{alignment}} framework,\color{red}{\text{framework,}} comprising\color{red}{\text{comprising}} K\color{red}{\text{K}} and\color{red}{\text{and}} D.\color{red}{\text{D.}} We therefore infer that the remaining concern may be the novelty of Kinematic Reference Correction (K), and focus below on its differences from HuMI’s Human-in-the-loop kinematic adaptation and BifrostUMI’s Spatial Keypoint Retargeting (SKR).

1. HuMI: collection guidance and controller training.

HuMI states in Sec. 3.1, final paragraph:

“letting operators adapt demonstrations to satisfy feasibility and task constraints”

Its IK preview checks feasibility, not execution accuracy. Sec. 3.2, first paragraph, notes:

“Yet, state-of-the-art controllers [16, 27] often incur tracking deviations of 4–6 cm, which is insufficient for fine manipulation.”

Reference [16] is SONIC. HuMI addresses this by training a new controller:

“we train an RL controller to track whole-body reference motions.”

Its training references are the IK results (Sec. 3.1):

“the resulting IK solutions provide full-body joint supervision for low-level controller training” HuMI, Secs. 3.1–3.2

Conclusion:\color{red}{\text{Conclusion:}} Our K corrects motion offline for policy training under fixed SONIC; HuMI’s IK guides collection and trains a new controller.

2. BifrostUMI: online construction of robot references.

Sec. III-C states:

“SKR maps sparse human keypoints to robot-native whole-body references”

SKR receives policy-predicted keypoints online during deployment (Sec. III-B, “Inference”); its references are:

“passed to the learned whole-body controller for closed-loop execution” BifrostUMI, Secs. III-B–C

Conclusion:\color{red}{\text{Conclusion:}} Our K corrects motion offline for policy training under fixed SONIC; SKR converts predicted keypoints into robot references online.

Technical distinction:\color{red}{\text{Technical distinction:}} K improves alignment while constraining changes to coordinated motion. H obtains robot motion through SONIC execution. K aligns its end-effectors with human targets while directly constraining IK to remain close to H’s motion (Sec. 3.3.1; Appendix A.2). This is designed to preserve whole-body coordination while improving endpoint accuracy under a fixed, general WBC (Table 3).

Please specify any remaining concerns; we would be happy to address them.

Comments on the paper

1 comment

Dear AC,

Thank you for reviewing our work. EgoHumanoid-V2 transfers coordinated whole-body skills from egocentric human data through offline action alignment and visual alignment, without target-task robot demonstrations or controller retraining.

Reviewers recognized the problem’s practical relevance (mSNH), offline alignment that preserves coordination (mSNH, vfta), complementary K/D stages (mSNH, vfta), complete system integration (vfta), diverse hand/foot tasks (mSNH, vfta), real-world policy transfer (all three), extensive ablations and simulation evidence (all three), and clear presentation (mSNH).

Across all reviews, we note a common thread: our approach was recognized as well-motivated and practically valuable, and the majority of concerns focused on requests for stronger empirical evidence and clearer distinctions from prior work.

Addressing Key Concerns. We have added experiments and detailed comparisons, with manuscript revisions highlighted in blue. We summarize the main responses below; supporting results and videos are available on our response webpage.

  • Methodological Contributions: mSNH(W1)

We provided detailed comparisons with EgoHumanoid, HuMI and BifrostUMI across hardware settings, whole-body controllers and alignment methods. These comparisons substantiate the methodological novelty of K, D and the overall offline action alignment framework.

  • Full-task Success, Uncertainty and Failure Analysis: mSNH(W2–3,Q1–2), vfta(W1,Q1), fQQ5(W2,Q2)

We added D (Unit) results, stage-wise success rates, standard errors and failure analysis, clarifying the contributions of K, D and the learned response model and identifying remaining failure modes (Tables 1–2; Figs. 4–5, 7).

  • Alignment without Simulated Object Forces: mSNH(Q3), vfta(W1,Q2,S1)

We added opposing-force experiments and real-world failure analysis, demonstrating that SONIC’s closed-loop feedback maintains tracking accuracy and stability under the tested external resistance. These results support our action alignment design without explicit simulation of task-object interaction forces (Fig. 7; Appendix B.4).

  • Data Requirements and Costs: mSNH(W4), vfta(W2,Q3), fQQ5(M2,Q4)

We added comparisons accounting for demonstration requirements and total collection labor, alongside complete processing costs. These results demonstrate comparable full-task success with lower collection labor, fast processing, and the ability to collect data in the wild without a robot (Fig. 5; Tables 6–7, 11).

Rebuttal Timeline and Status

  • Nov. 15: We submitted responses and added results to all three reviewers.
  • Nov. 18: mSNH remained concerned about W1: methodological novelty, without revisiting other weaknesses or questions.
  • Nov. 19: We clarified K’s differences from HuMI/BifrostUMI.
  • Nov. 19: fQQ5 confirmed most empirical concerns were addressed and raised the rating from 2 to 3.
  • Nov. 19: We added visual-mismatch and total collection-labor analyses.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.