- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VioLA trains a generalist humanoid policy to predict body and hand motion latents, which frozen decoders execute on the robot. Shared motion encoders convert annotated human and robot demonstrations into common action targets, enabling large-scale human supervision.
Critical strengths
The framework provides a coherent connection between human motion supervision and executable robot actions.
Real-robot experiments cover locomotion, posture changes, and manipulation, with implementations across three policy backbones.
Human-only transfer and the matched-step comparison against robot-only training provide useful evidence that human demonstrations improve locomotion performance.
Critical weaknesses
The contribution of the action representation is insufficiently isolated. Comparisons with released checkpoints conflate representation, additional training, and control interfaces. Matched-data and matched-budget comparisons against reference-motion prediction and direct hand commands would better establish the benefits of the proposed representation, particularly given related approaches such as MotionWAM and UniT.
Evidence for human-to-robot skill transfer remains limited. The human-data ablation covers locomotion, with most improvements concentrated in bending. Corresponding manipulation ablations and experiments withholding robot demonstrations of selected task categories are needed to demonstrate broader transfer.
The evaluation scope limits the generalization claims. Deployment in a new laboratory with different object instances is valuable, but cross-task transfer remains unclear. Each task has only five trials. Additionally, bottle carrying achieves 0/5 and is reported separately; including it reduces manipulation success from 88.6% to 77.5%. The task-selection rationale should be explained.
Questions for authors
Could you provide controlled action-representation comparisons and a complete robot-only manipulation evaluation?
How was the shared hand module trained, and what objectives establish alignment between human, Dex3, and Inspire motions?
How much do evaluation task categories overlap with training demonstrations, and why is bottle carrying excluded from the primary aggregate?
2 comments
Thank you for the additional experiments, which address several concerns about the action representation and evaluation reporting. However, evidence for broader human-to-robot skill transfer remains limited: the locomotion gains are concentrated in bending, and the manipulation ablation does not demonstrate an aggregate benefit from human data. I therefore maintain my original rating.
Response to Reviewer E2k3
We thank the reviewer for the constructive feedback and helpful suggestions. We revised the paper with additional experiments and explanations to address your comments.
W1-Q1: To isolate the effect of the action representation, we trained VioLA to predict continuous joint positions using the same data, backbone, training budget and reference window. We evaluated this policy in two ways:
Section 4.2 (Figure 3 and Table 5) shows that predicting latent actions performs much better than predicting joint positions. VioLA achieves 77.5% manipulation and 100% locomotion success, while VioLA-CT reaches 20.0% and 83.3%, respectively. VioLA-C achieves 0% on both task groups.
The complete robot-only manipulation evaluation is now reported in Table 6, alongside the locomotion comparison (Section 4.3 and Appendix A.6).
W2: The human-only experiment already demonstrates useful human-to-robot transfer. Figure 5 shows the human-only policy following “walk to the yellow couch” and “squat down” on the robot, without robot demonstrations during high-level policy training. It also succeeds on the related walking and bending tasks that both released baselines fail in Figure 6.
Q2: Regarding the hand module, we now present the training procedure and direct cross-decoding validation in Appendix A.2 (Figure 11 and Table 3). We pair human MANO trajectories with their Dex3 and Inspire retargetings and supplement these with recorded Dex3 motion. The hand-specific encoders and decoders learn through self reconstruction, alignment before quantization, and cross reconstruction: a human code must reconstruct the paired robot trajectory through the robot decoder. The modules remain frozen during VioLA training. Table 3 reports cross-reconstruction and self-reconstruction errors, including human-to-Inspire and Dex3-to-Inspire joint MAEs of 0.0445 and 0.0548 rad.
W3-Q3: We originally presented bottle carrying as a failure analysis in the appendix to save space. We now include the task in the main aggregate (Section 4.5, Figure 6 and Table 5). VioLA takes the bottle and carries it to the table, but releases it while tilted and fails the upright-placement criterion in all five trials (Appendix A.7, Figure 14).
Regarding training overlap, the task families appear in training, while the evaluated episodes, deployment environment, and object instances differ. Our zero-shot setting (Section 4.1) is deployment without task-specific fine-tuning, where the released baselines largely fail. We added scene-change tests on three manipulation tasks (Section 4.5, Figure 7). For Place Dumpling and Close Kettle, we added a blue mat and distractor objects while keeping the instructions fixed; VioLA succeeds in 5/5 and 4/5 trials, respectively. For Close Laptop, we moved the laptop from the table to a chair and changed “table” to “chair” in the instruction; VioLA succeeds in 5/5 trials.