acceptodds
Under review as a conference paper at ICLR 2027

VioLA: Learning Generalist Humanoid Control Policies from Human Data

77 views

Abstract

Teaching a humanoid to follow instructions with its whole body runs into two obstacles. Its action space is large and tightly coupled: legs, arms, and fingers must move together while the robot keeps its balance, which makes joint-level actions hard to learn. And humanoid demonstrations are scarce, so current humanoid generalist policies do not follow new instructions out of the box and are fine-tuned on teleoperated demonstrations of each task before deployment. Human demonstrations exist in far larger numbers, but a person's motion is not a robot command. We remove both obstacles by changing what the generalist policy predicts. We introduce VioLA, a generalist humanoid policy that predicts body and hand motion latents instead of joint commands. A pretrained body controller and hand decoder execute these latents on the robot. Their corresponding motion encoders map human and robot motion into the same latent spaces. A human recording is therefore labeled in the policy's action space, and the converted demonstration pool contains 140.6 million frames, 93.2% of them human. As a result, VioLA follows locomotion instructions on the real robot zero-shot, without task-specific fine-tuning, reaching 100% success where GR00T N1.7 and reach 16.7% and 0%, respectively. It also reaches 88.6% manipulation success without task-specific fine-tuning. The same approach works across two VLA and one world-action model backbones. A generalist policy trained on human demonstrations alone performs locomotion tasks on the real robot zero-shot. Code, data pipelines, and checkpoints will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Humanoid Robots, Vision-Language-Action Models, World Action Models, Learning from Human Video

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_E2k3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VioLA trains a generalist humanoid policy to predict body and hand motion latents, which frozen decoders execute on the robot. Shared motion encoders convert annotated human and robot demonstrations into common action targets, enabling large-scale human supervision.

Critical strengths

The framework provides a coherent connection between human motion supervision and executable robot actions.

Real-robot experiments cover locomotion, posture changes, and manipulation, with implementations across three policy backbones.

Human-only transfer and the matched-step comparison against robot-only training provide useful evidence that human demonstrations improve locomotion performance.

Critical weaknesses

The contribution of the action representation is insufficiently isolated. Comparisons with released checkpoints conflate representation, additional training, and control interfaces. Matched-data and matched-budget comparisons against reference-motion prediction and direct hand commands would better establish the benefits of the proposed representation, particularly given related approaches such as MotionWAM and UniT.

Evidence for human-to-robot skill transfer remains limited. The human-data ablation covers locomotion, with most improvements concentrated in bending. Corresponding manipulation ablations and experiments withholding robot demonstrations of selected task categories are needed to demonstrate broader transfer.

The evaluation scope limits the generalization claims. Deployment in a new laboratory with different object instances is valuable, but cross-task transfer remains unclear. Each task has only five trials. Additionally, bottle carrying achieves 0/5 and is reported separately; including it reduces manipulation success from 88.6% to 77.5%. The task-selection rationale should be explained.

Questions for authors

Could you provide controlled action-representation comparisons and a complete robot-only manipulation evaluation?

How was the shared hand module trained, and what objectives establish alignment between human, Dex3, and Inspire motions?

How much do evaluation task categories overlap with training demonstrations, and why is bottle carrying excluded from the primary aggregate?

2 comments
AuthorsOpenReview ↗

Response to Reviewer E2k3

We thank the reviewer for the constructive feedback and helpful suggestions. We revised the paper with additional experiments and explanations to address your comments.

W1-Q1: To isolate the effect of the action representation, we trained VioLA to predict continuous joint positions using the same data, backbone, training budget and reference window. We evaluated this policy in two ways:

  1. Executing the predicted joint positions directly on the robot (VioLA-C).
  2. Passing the predicted joint positions through the SONIC body tracker and hand codec before execution (VioLA-CT).

Section 4.2 (Figure 3 and Table 5) shows that predicting latent actions performs much better than predicting joint positions. VioLA achieves 77.5% manipulation and 100% locomotion success, while VioLA-CT reaches 20.0% and 83.3%, respectively. VioLA-C achieves 0% on both task groups.

The complete robot-only manipulation evaluation is now reported in Table 6, alongside the locomotion comparison (Section 4.3 and Appendix A.6).

W2: The human-only experiment already demonstrates useful human-to-robot transfer. Figure 5 shows the human-only policy following “walk to the yellow couch” and “squat down” on the robot, without robot demonstrations during high-level policy training. It also succeeds on the related walking and bending tasks that both released baselines fail in Figure 6.

Q2: Regarding the hand module, we now present the training procedure and direct cross-decoding validation in Appendix A.2 (Figure 11 and Table 3). We pair human MANO trajectories with their Dex3 and Inspire retargetings and supplement these with recorded Dex3 motion. The hand-specific encoders and decoders learn through self reconstruction, alignment before quantization, and cross reconstruction: a human code must reconstruct the paired robot trajectory through the robot decoder. The modules remain frozen during VioLA training. Table 3 reports cross-reconstruction and self-reconstruction errors, including human-to-Inspire and Dex3-to-Inspire joint MAEs of 0.0445 and 0.0548 rad.

W3-Q3: We originally presented bottle carrying as a failure analysis in the appendix to save space. We now include the task in the main aggregate (Section 4.5, Figure 6 and Table 5). VioLA takes the bottle and carries it to the table, but releases it while tilted and fails the upright-placement criterion in all five trials (Appendix A.7, Figure 14).

Regarding training overlap, the task families appear in training, while the evaluated episodes, deployment environment, and object instances differ. Our zero-shot setting (Section 4.1) is deployment without task-specific fine-tuning, where the released baselines largely fail. We added scene-change tests on three manipulation tasks (Section 4.5, Figure 7). For Place Dumpling and Close Kettle, we added a blue mat and distractor objects while keeping the instructions fixed; VioLA succeeds in 5/5 and 4/5 trials, respectively. For Close Laptop, we moved the laptop from the table to a chair and changed “table” to “chair” in the instruction; VioLA succeeds in 5/5 trials.

Thank you for the additional experiments, which address several concerns about the action representation and evaluation reporting. However, evidence for broader human-to-robot skill transfer remains limited: the locomotion gains are concentrated in bending, and the manipulation ablation does not demonstrate an aggregate benefit from human data. I therefore maintain my original rating.

Reviewer_JYtzOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VioLA, a generalist humanoid policy that predicts chunks of body and hand motion latents from visual observations, language instructions, and proprioception. Frozen pretrained controllers and decoders convert these predictions into robot commands, while motion encoders provide latent targets for both human and robot demonstrations. The authors train on 140.6 million frames, most of which come from human data, and report zero-shot performance on a real Unitree G1.

Critical strengths
  1. A clear and practically motivated action representation. VioLA predicts body and hand motion latents rather than joint commands, using frozen low-level modules to turn those predictions into robot commands. This cleanly separates high-level instruction following from whole-body coordination and execution.

  2. A meaningful use of human demonstrations. The shared action-target format lets human and robot demonstrations supervise the same policy output.

  3. Promising real-robot results. On a Unitree G1, VioLA completes 30/30 locomotion trials and 31/35 manipulation trials without task-specific fine-tuning.

Critical weaknesses
  1. The evaluation is narrow. The main study covers 13 tasks on one Unitree G1, with five trials per task. Although episodes, environment, and object instances are held out, the task families also appear in the training data; “zero-shot” therefore means no task-specific fine-tuning, not generalization to entirely unseen task families.

  2. The shared human–robot hand latent space is insufficiently documented. The paper says a paired hand encoder–decoder maps human and robot hand motion into a shared space, but does not provide the pretraining data, objective, or direct validation of that cross-domain alignment. This makes a central assumption difficult to assess or reproduce.

  3. Human-data benefits are task-dependent, not uniformly demonstrated. Compared with robot-only training, the human-then-mixed policy improves locomotion from 24/30 to 30/30 trials, but has fewer manipulation successes: 31/35 versus 33/35. The results support a locomotion benefit in this setup, but not a general claim that adding human data improves overall performance.

  4. The baseline comparison does not isolate the source of the gains. The main comparison is against released GR00T N1.7 and Ψ₀ models, whereas VioLA is a trained system with a modified action representation and extensive demonstration data. Thus, the results show that the full VioLA recipe outperforms those released baselines in this evaluation, but do not cleanly attribute the improvement to latent actions, human data, or additional training.

Questions for authors
  1. How is the human–robot hand latent space trained and validated? The paper says paired encoders map human and robot hand motion into a shared 64-dimensional space, but does not specify the pretraining data, pairing procedure, objective, or quantitative alignment results. Can you provide these details and show that equivalent human and robot motions decode to comparable robot hand trajectories?

  2. How sensitive is the method to errors in human motion annotations and simulated state tracking? Human demonstrations require SMPL/MANO motion estimates, while body targets and proprioceptive inputs are constructed by tracking those demonstrations in simulation. Have you measured how annotation or simulation-tracking errors affect latent targets and real-robot performance?

  3. How can the baseline comparison isolate the contribution of VioLA’s latent action space? The main comparison uses released GR00T N1.7 and Ψ₀ models, while VioLA is trained with a modified action interface and a large demonstration pool. Can you compare against baselines trained with the same data, training budget, and task-specific adaptation protocol, or otherwise separate the effects of representation, data, and additional training?

  4. Can you explain why does GR00T N1.7 and Ψ₀ yields 0 success in most of the tasks in Figure 5?

2 comments
AuthorsOpenReview ↗

Response to Reviewer JYtz

We thank the reviewer for the constructive feedback and suggestions. We have added the requested controlled comparison, hand-module details, and further evaluation to address the weaknesses and questions below.

W2-Q1: Regarding the hand module, we have added its training procedure and direct cross-decoding validation in Appendix A.2 (Figure 11 and Table 3). Briefly, we pair human MANO trajectories with their Dex3 and Inspire retargetings and supplement these with recorded Dex3 motion. The hand-specific encoders and decoders learn through self reconstruction, alignment before quantization, and cross reconstruction: a human code must reconstruct the paired robot trajectory through the robot decoder. The modules remain frozen during VioLA training. Table 3 reports self-reconstruction and cross-decoding errors, including human-to-Inspire and Dex3-to-Inspire joint MAEs of 0.0445 and 0.0548 rad.

Q2: We have not tested how annotation or tracking errors affect performance in a controlled experiment. EgoSuite already labels problems such as floating feet or intersecting body parts. We repair flagged lower-body segments lasting up to 0.5 seconds by interpolating between good poses on either side, and exclude training examples containing errors that cannot be repaired. We also discard episodes where the robot falls during simulated SONIC tracking. Across conversion and filtering, about 22% of EgoSuite episodes are removed (Appendix A.1).

W4-Q3: To isolate the effect of the action representation, we trained VioLA to predict continuous joint positions using the same data, backbone, training budget and reference window. We evaluated this policy in two ways:

  1. Executing the predicted joint positions directly on the robot (VioLA-C).
  2. Passing the predicted joint positions through the SONIC body tracker and hand codec before execution (VioLA-CT).

Section 4.2 (Figure 3 and Table 5) shows that predicting latent actions performs much better than predicting joint positions. VioLA achieves 77.5% manipulation and 100% locomotion success, while VioLA-CT reaches 20.0% and 83.3%, respectively. VioLA-C achieves 0% on both task groups.

Q4: Although Ψ₀ was trained on Humanoid Everyday, which contains our manipulation task families, both baselines fail to work without task-specific fine-tuning. For instance, Ψ₀ moves around a lot even when the task is Stand Still, and neither baseline raises an arm during Raise Arm. We also trained both baselines on LeVERB and Humanoid Everyday, which contain the evaluated task families, to examine whether further training closes the gap. After this additional training, GR00T reaches 7.5% manipulation and 20% locomotion success rate. Similarly, Ψ₀ reaches 38.8% manipulation and 0% locomotion success rate. Both baselines remain below VioLA’s 100% locomotion and 77.5% manipulation success. We report these results in Sections 4.4–4.5, Figures 12–13 and Table 5, with deployment details in Appendix A.6.

W1: By zero-shot we mean deployment without task-specific fine-tuning (Section 4.1). The task families appear in training, while the evaluated episodes, environment, and object instances differ. We added scene-change tests on three manipulation tasks (Section 4.5, Figure 7). For Place Dumpling and Close Kettle, we added a blue mat and distractor objects while keeping the instructions fixed; VioLA succeeds in 5/5 and 4/5 trials, respectively. For Close Laptop, we moved the laptop from the table to a chair and changed “table” to “chair” in the instruction; VioLA succeeds in 5/5 trials.

We think the difficulty of the deployment setting is also important. The released baselines struggle even with basic locomotion instructions without target-task fine-tuning. Training one policy across human and robot data and deploying it across locomotion and manipulation addresses this gap. We see this as an important step toward broader task generalization.

W3: Regarding human supervision, the human-only experiment already demonstrates useful transfer. Figure 5 shows the human-only policy following “walk to the yellow couch” and “squat down” on the robot. These behaviors are learned without robot demonstrations during high-level policy training, and neither released baseline completes them. It also succeeds on the related walking and bending tasks that both released baselines fail in Figure 6.

Area_Chair_G9AwOpenReview ↗

Reminder to respond to the authors’ rebuttal

With one day left in the rebuttal period, please read the authors’ response and leave a comment. It would help the authors to know which of your concerns they have addressed and which questions remain open, so they can make the most of the time left.

Thanks for your time. Your constructive feedback helps the authors improve their work and strengthens the ICLR community.

Best regards,

Area Chair

Reviewer_fj71OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VioLA develops a two-level system that high level policy(VLA/WAM) predicts the body and hand motion latent and low level trackers(Decoders) translate these latent into dynamic actions.Human and robot motion encoders provide targets in shared latent spaces. For human recordings, the original images and instructions are retained, while closed-loop simulation supplies robot-state inputs. This allows human recordings (cheap data) with motion annotations to be used alongside robot demonstrations (expensive data). On a real G1, the paper reports 30/30 locomotion/posture successes and 31/35 manipulation successes without target-task fine-tuning. Human-only high-level policy training achieves 12/30 locomotion successes, while human-then-mixed training improves over robot-only training from 24/30 to 30/30.

Critical strengths

The paper proposes a useful trainging recipe that using corresponding motion encoders to map human motoin and humanoid motion into the same latent spaces, so that for high level policy, expensive humanoid data can cotrain with cheaper human data. This provides a possible processing method for expanding the training data source of the high level policy.

The human-only results and the comparison with robot-only training provide evidence that human motion supervision contributes useful behaviors. Implementations across three backbones also suggest that the interface is reusable.

Critical weaknesses
  • I am not convinced why predicting latents is better than predicting joint references. SONIC already supports joint-reference tracking. We could retarget SMPL motions to robot joints, let the high-level policy predict these references, and still use SONIC to handle coordination and balance. The paper presents “Predicting motions, not joints” as a benefit, but does not show why predicting latents is better than predicting joint references for the same tracker. I am also unsure how much precision is lost through FSQ, especially for the hands.

  • I find the baseline results difficult to understand. These locomotion and posture tasks look quite simple, yet Ψ0 gets 0/30, including 0/5 on standing still, and GR00T N1.7 only succeeds on standing still. This makes me question whether the baseline settings are correct. The appendix does not give enough detail about their configurations or failure behaviors to resolve this concern, so I am not sure how much of the reported gap comes from the method itself.

Questions for authors
  • Can you compare latent prediction with predicting robot joint-reference trajectories for the same SONIC tracker, using the same data, backbone, training budget, and reference time window? Or can you explain why "Predicting motions, not joints" is better?

  • Which exact checkpoints and controller versions are used for Ψ0 and GR00T N1.7, and what are their deployment settings? What actually happens when Ψ0 fails Stand Still, and when both models fail Raise Arm? The paper reports the success rates but does not explain these failures, which seems somewhat abnormal to me.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fj71

We thank the reviewer for the constructive feedback and experimental suggestions. We added the requested action-representation study and clarified the baseline configurations and failure descriptions in the revised paper.

W1-Q1: To isolate the effect of the action representation, we trained VioLA to predict continuous joint positions using the same data, backbone, training budget and reference window. We evaluated this policy in two ways:

  1. Executing the predicted joint positions directly on the robot (VioLA-C).
  2. Passing the predicted joint positions through the SONIC body tracker and hand codec before execution (VioLA-CT).

Section 4.2 (Figure 3 and Table 5) shows that predicting latent actions performs much better than predicting joint positions. VioLA achieves 77.5% manipulation and 100% locomotion success, while VioLA-CT reaches 20.0% and 83.3%, respectively. VioLA-C achieves 0% on both task groups.

W1: Regarding FSQ, in the new study in Appendix A.2 (Table 3), we compare the quantized and continuous hand codecs on reconstruction error of human and robot hand motion, including human-to-robot and robot-to-human decoding. For human-to-Inspire decoding, FSQ and continuous-code joint MAEs are 0.0445 and 0.0433 rad; for human-to-Dex3 decoding, they are 0.1093 and 0.1088 rad. These human-to-robot cross-decoding errors are close. We use FSQ to obtain compact, discrete hand-motion targets.

W2-Q2: We follow the deployment guides presented under official GitHub repositories of baselines and use their released checkpoints. Appendix A.6 now describes the deployed variants, controller interfaces and inference settings.

Regarding the failures, we agree that they should be explained rather than shown only as success rates. Ψ₀ moves around a lot even when the task is Stand Still and neither baseline raises an arm during Raise Arm. As an additional study, we trained both baselines on LeVERB and Humanoid Everyday, which contain the task families used in our evaluation. GR00T reaches 20% locomotion and 7.5% manipulation success after this training; Ψ₀ reaches 0% and 38.8%, respectively. We report these results in Sections 4.4–4.5, Figures 12–13 and Table 5. Even after this training, both baselines fail to reach the performance of VioLA.

Thanks for the additional experiments. VioLA-CT addresses my concern. Joint references work for most locomotion trials with SONIC tracking, but succeed in only 8/40 manipulation trials compared with 31/40 for VioLA. The new hand codec results also address my question about reconstruction precision. I still have some reservations about the baseline comparison, especially the unsuccessful Ψ0-SONIC deployment.

I found the demos less convincing for tasks that combine walking and manipulation. Bottle carrying still fails in all five trials, the task families are covered in training, and adding human data does not improve overall manipulation success. Even so, the transfer from human demonstrations to real-robot locomotion and the new joint-reference comparison are enough for me to raise my score to 3 (weak acceptance).

Reviewer_HACHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether an executable motion representation can connect human demonstrations to generalist humanoid policies. VioLA predicts body and hand motion latents, while pretrained, frozen controllers translate these predictions into robot commands. Annotated human motion supplies supervision in the same action space as robot demonstrations. The central contribution is a practical interface between human supervision and deployed humanoid control.

Critical strengths
  1. The representation addresses an important practical bottleneck. A shared action interface could make human recordings substantially more useful for humanoid learning. Its value extends beyond representation alignment: the policy’s predictions are executable through the paired controllers. This is a compelling design principle for combining high-level instruction following with existing motor capabilities.

  2. The hardware results establish meaningful feasibility. The human-only policy-training experiment is particularly informative: it demonstrates a route by which human supervision can produce useful deployed behavior. This should be interpreted within the retained pretrained backbone and motion modules, rather than as learning the entire robot system exclusively from human demonstrations.

  3. The contribution can be valuable without a new learning objective. I consider a well-supported systems and empirical contribution appropriate for ICLR. Reusing strong components is a reasonable research strategy; the relevant question is whether their integration establishes sufficiently clear and generalizable new knowledge.

Critical weaknesses

W1. The experiments do not adequately isolate the contribution of the proposed action representation.

The released-checkpoint comparisons combine differences in action interface, additional training, and execution machinery. They establish a useful deployment comparison, but cannot determine how much improvement comes specifically from the representation.

A more diagnostic experiment would hold the backbone, robot demonstrations, training budget, and low-level execution machinery fixed as far as possible, while changing the predicted action interface. This could initially be conducted on a representative subset of tasks.

This matters because the paper motivates motion prediction as a way to simplify policy learning. Demonstrating successful behavior supports feasibility; establishing that the representation improves learning requires a controlled comparison.

The novelty boundary also deserves precision. SONIC already demonstrates VLA prediction of executable motion tokens. I therefore regard VioLA’s stronger potential contribution as the use of human recordings to supervise the deployed interface, together with the body–hand integration. The empirical argument should concentrate on that incremental contribution.

W2. The human-data study supports a task-dependent benefit, but does not yet characterize its reliability.

Human-then-mixed training obtains 30/30 locomotion successes versus 24/30 for robot-only training; manipulation obtains 31/35 versus 33/35. The locomotion difference is concentrated primarily in bending.

These results motivate further investigation, but leave several explanations open. Human supervision may expand motion coverage, improve sample efficiency, or interact favorably with the training curriculum. Those possibilities have different implications for the proposed method.

Conversely, the two-success manipulation difference is too small to establish harmful negative transfer without additional evidence. My concern is uncertainty about when and why human data help. Targeted repetitions on the tasks driving the difference would be more informative than simply expanding the benchmark.

W3. The generalization evidence supports a narrower conclusion than broad generalist capability.

The evaluation holds out episodes, environment, and object instances, while task families overlap training. Each task receives five trials on one robot. The paper explicitly defines “zero-shot” as deployment without task-specific fine-tuning.

This is a legitimate evaluation setting, and task-family overlap does not itself establish data leakage. However, the evidence primarily addresses transfer within familiar task families. It provides less information about unseen task compositions or robustness across deployment conditions.

I would not require additional robot embodiments for acceptance. A clearer separation of generalization claims, accompanied by a modest test targeting the strongest claim, would be sufficient to improve this aspect.

W4. The primary aggregate needs a clearer task-inclusion rationale.

Bottle carrying scores 0/5 and is reported separately; including it changes manipulation success from 88.6% to 77.5%. The failure is disclosed, which is a positive aspect of the reporting.

Nevertheless, transporting an object while walking is relevant to coordinated humanoid behavior. Readers need to understand why this task belongs outside the primary evaluation set. I recommend presenting all evaluated tasks together and clearly defining any separate subsets. This is a question about interpretation and reporting, rather than evidence of misconduct.

Questions for authors
  1. Representation: Can a controlled action-interface comparison substantiate the proposed learning advantage? Which part of the gain remains after matching training exposure and execution machinery?
  2. Human supervision: Can dataset coverage analysis and targeted repeated runs explain the bending improvement and assess the reliability of the manipulation results?
  3. Evaluation: How was the primary task set selected, and what criterion determined that bottle carrying should be analyzed separately?
3 comments
AuthorsOpenReview ↗

Response to Reviewer HACH

We thank the reviewer for the constructive feedback. We ran the suggested controlled comparison, repeated trials and scene-change tests, and revised the paper.

W1-Q1: To isolate the effect of the action representation, we trained VioLA to predict continuous joint positions using the same data, backbone, training budget and reference window. We evaluated this policy in two ways:

  1. Executing the predicted joint positions directly on the robot (VioLA-C).
  2. Passing the predicted joint positions through the SONIC body tracker and hand codec before execution (VioLA-CT).

Section 4.2 (Figure 3 and Table 5) shows that predicting latent actions performs much better than predicting joint positions. VioLA achieves 77.5% manipulation and 100% locomotion success, while VioLA-CT reaches 20.0% and 83.3%, respectively. VioLA-C achieves 0% on both task groups.

W1: Regarding SONIC, we think the distinction between VioLA and SONIC is more substantial than demonstrating another VLA connection. SONIC’s VLA experiments train on robot demonstrations collected for the evaluated tasks, which resembles imitation learning rather than generalist policy. VioLA instead trains one policy across heterogeneous human and robot datasets and deploys it across locomotion and manipulation without target-task fine-tuning. Therefore, the task-adapted VLA execution in SONIC establishes an important precedent, but does not establish this generalist training and deployment result.

W2-Q2: The bending result suggests that human recordings supply useful motion coverage that robot-only training misses. Following the reviewer's suggestion, we ran five additional trials per policy on Bend Down and Store Shampoo. Across all ten trials, VioLA succeeds in 10/10 bending trials and 4/10 shampoo trials, compared with 0/10 and 8/10 for robot-only training. The bending difference persists, while the shampoo difference narrows. Table 6 in Appendix A.6 reports the original and combined results separately.

Regarding manipulation, we agree that the small difference does not establish negative transfer and should not obscure the usefulness of human data. Figure 5 already shows the human-only policy following “walk to the yellow couch” and “squat down” on the robot, without robot demonstrations during high-level policy training. It also succeeds on the related walking and bending tasks that both released baselines fail in Figure 6.

W3: Following the reviewer's suggestion, we added scene-change tests on three manipulation tasks (Section 4.5, Figure 7). For Place Dumpling and Close Kettle, we added a blue mat and distractor objects while keeping the instructions fixed; VioLA succeeds in 5/5 and 4/5 trials, respectively. For Close Laptop, we moved the laptop from the table to a chair and changed “table” to “chair” in the instruction; VioLA succeeds in 5/5 trials.

Regarding generalization to unseen tasks, we think the difficulty of the setting is important. The released baselines struggle even with basic locomotion instructions without target task-specific fine-tuning. VioLA addresses this gap by training one policy across human and robot data and deploying it across locomotion and manipulation without additional task-specific demonstrations. We see this as an important step toward broader unseen task generalization, even though the task families appear in training for now.

W4-Q3: We originally moved the bottle-carrying failure analysis to the appendix for space. We now include the task in the main aggregate (Section 4.5, Figure 6 and Table 5). VioLA takes the bottle and carries it to the table, but releases it while tilted and fails the upright-placement criterion in all five trials (Appendix A.7, Figure 14).

Area_Chair_G9AwOpenReview ↗

Reminder to respond to the authors’ rebuttal

With one day left in the rebuttal period, please read the authors’ response and leave a comment. It would help the authors to know which of your concerns they have addressed and which questions remain open, so they can make the most of the time left.

Thanks for your time. Your constructive feedback helps the authors improve their work and strengthens the ICLR community.

Best regards,

Area Chair

Thank you for the substantial additional experiments and clarification. The rebuttal addresses my main concerns, and I am updating my recommendation to weak acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank the Area Chair and the reviewers.

This comment summarizes (1) what the reviewers found strong and (2) their shared concerns and what the revision adds. In short: all four reviewers see value in the idea, and the controlled experiment all four requested is now in the paper (real-robot manipulation success: 77.5% with motion latents vs at most 20.0% with joint positions).

The paper. VioLA predicts body and hand motion latents that frozen pretrained controllers execute. Paired encoders map human motion into the same latents, so human recordings directly supervise training of the generalist policy.


1. What the reviewers found strong

  • Problem and idea (HACH, JYtz, E2k3). HACH calls the central contribution "a practical interface between human supervision and deployed humanoid control". The representation "addresses an important practical bottleneck" and is "a compelling design principle for combining high-level instruction following with existing motor capabilities" (HACH). It "cleanly separates high-level instruction following from whole-body coordination and execution" (JYtz) and provides "a coherent connection between human motion supervision and executable robot actions" (E2k3).
  • Human data as supervision (HACH, fj71, JYtz). "A shared action interface could make human recordings substantially more useful for humanoid learning" (HACH), since "expensive humanoid data can cotrain with cheaper human data" (fj71). JYtz calls this "A meaningful use of human demonstrations".
  • Human-only transfer (HACH, E2k3, fj71). The experiment is "particularly informative: it demonstrates a route by which human supervision can produce useful deployed behavior" (HACH). With the robot-only comparison, it gives "useful evidence that human demonstrations improve locomotion performance" (E2k3) and "evidence that human motion supervision contributes useful behaviors" (fj71).
  • Real-robot evidence and generality (HACH, JYtz, fj71). "The hardware results establish meaningful feasibility" (HACH); "Promising real-robot results" (JYtz). Implementations across three backbones "suggest that the interface is reusable" (fj71).

2. Shared concerns and what the revision adds

(a) Action representation (all four reviewers). We ran the requested controlled comparison: same data, backbone, training budget and reference window, changing only what the policy predicts (Sec. 4.2, Table 5).

Policy predictsExecuted throughLocomotionManipulation
Motion latents (VioLA)SONIC controller + hand decoder100%77.5%
Joint positions (VioLA-CT)SONIC body tracker + hand codec83.3%20.0%
Joint positions (VioLA-C)directly0%0%

(b) Baselines (fj71, JYtz, HACH, E2k3). We use the released checkpoints with their official deployment guides (App. A.6). We evaluate them without task-specific fine-tuning, like VioLA. After further training on LeVERB and Humanoid Everyday, which contain the evaluated task families, GR00T N1.7 reaches 20% locomotion / 7.5% manipulation and Ψ₀ 0% / 38.8%, versus 100% / 77.5% for VioLA (Table 5).

(c) Evaluation scope and reporting (HACH, JYtz, E2k3). All success rates are zero-shot on a real Unitree G1: no task-specific fine-tuning. Bottle carrying (0/5) is now in the main aggregate, so headline manipulation is 77.5% (31/40) instead of 88.6% (31/35). We include new scene-change tests (added mat and distractor objects; laptop moved to a chair with the instruction changed), where VioLA succeeds in 14/15 trials (Sec. 4.5, Fig. 7).

(d) Human data (HACH, JYtz, E2k3). Trained on human data alone (no robot demonstrations in policy training), VioLA bends, walks, sits and stands up on the robot (12/30 locomotion successes); neither released baseline completes any of these. Table 6 reports the full robot-only manipulation comparison and five additional trials per policy on Bend Down and Store Shampoo. Over ten trials, human-then-mixed training achieves 10/10 on Bend Down versus 0/10 for robot-only training. In the main evaluation, manipulation totals are 31/40 versus 33/40. VioLA reaches this with human recordings supplying 80% of its training examples.

(e) Hand module (JYtz, E2k3, fj71). App. A.2 now documents training details and cross-decoding validation (Table 3): joint MAE is 0.0445 rad human-to-Inspire and 0.0548 rad Dex3-to-Inspire.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.