acceptodds
Under review as a conference paper at ICLR 2027

WAVE-VLA: A Whole-Body Humanoid VLA with Executable Vision–Motion Codes Learned from Heterogeneous Human and Robot Data

Abstract

Human video–motion offers behaviors beyond scarce humanoid demonstrations, but embodiment and supervision differ across sources. We present WAVE-VLA (Whole-body Action Vocabulary for Execution), which learns from task-unpaired G1 demonstrations and egocentric human video–motion. Both sources enter a common G1 action space and shared vision–motion codebook to extend one policy's repertoire while retaining robot skills. Current RGB, state, and language select body and hand codes that decode into synchronized references. A scene-conditioned Action Expert predicts a residual from the body reference, which zero residual preserves exactly. On HE and Nymeria development cohorts, refinement reduces mean forward-kinematics error relative to decoded references by 21.1% and 15.2%, respectively. The same frozen body–hand policy completes 39 of 60 real-G1 trials across four HE-covered and two Nymeria-covered tasks without task-specific parameter updates or retrieved trajectories. The HE-only baseline scores 27/40 on HE-covered and 0/20 on Nymeria-covered tasks, versus 25/40 and 14/20 for the joint policy in the reported robot trials.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Whole-Body Humanoid Control, Human-to-Robot Transfer, Heterogeneous Robot Learning, Vision–Motion Codebook, Action Tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1RDWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WAVE-VLA combines native G1 demonstrations from Humanoid Everyday with retargeted Nymeria human video-motion in a common 93D body/root representation. A shared vision-motion RVQ vocabulary and a separate body-conditioned hand stream produce synchronized 16-frame references from current RGB, state, and language. A scene-conditioned residual expert refines the decoded body reference before SONIC tracking. Human data supplies body supervision, while hand supervision comes from robot data. The paper evaluates reconstruction, current-only code selection, offline refinement, and six real-G1 tasks, reporting 39/60 successes for joint training versus 27/60 for HE-only training.

Critical strengths
  1. The system addresses a useful and concrete transfer setting: combining independently collected human and humanoid data without matched task pairs. The common robot-space interface, explicit handling of missing hand labels, and body-conditioned hand decoding form a coherent executable pipeline. The real-robot comparison provides encouraging evidence of added repertoire: the jointly trained policy succeeds in 14/20 trials on the two Nymeria-covered tasks, versus 0/20 for HE-only, while obtaining 25/40 versus 27/40 on HE-covered tasks (Table 1). This demonstrates a practical capability beyond motion reconstruction alone.

  2. The offline controls help separate several stages of the pipeline. On matched examples, the residual expert reduces decoder-reference FK error by 21.1%/15.2% on HE/Nymeria (Table 2). The separately trained full direct-action comparator in Table 3 is also useful evidence that decoding the trajectory improves motion prediction in this implementation. The paper carefully distinguishes current-only deployment from oracle future inputs, and explicitly avoids treating zero-residual identities as safety guarantees or the small HE success difference as an equivalence result.

Critical weaknesses
  1. The central control-interface novelty is insufficiently distinguished from the closest prior art. VQ-BeT already decodes predicted RVQ codes into an action chunk and adds an observation-conditioned continuous offset (Lee et al., 2024, Section 3.3, Eqs. 5-6). Zero offset also recovers its decoded action. WAVE-VLA adds meaningful implementation differences, including a physical-space objective and a reference-conditioned expert, but the broad contrast between optional code conditioning and decoded-plan residual control does not establish these additions' contribution. Joint visual-action cross-reconstruction is also already present in UniT, as the paper acknowledges. A matched action-only RVQ-plus-offset comparison is missing, so it remains unclear how much the particular vision-motion vocabulary and refinement design add beyond established components.

  2. The evidence for the proposed architecture's deployment advantage remains limited. Table 1 changes the training data, whereas the direct-action and decoder-only architecture controls are evaluated only offline. Thus, the robot results do not establish that the new architecture improves task success over a simpler jointly trained policy. Transfer is tested on only two human-covered tasks: 10 of the 14 successes are dance trials judged by qualitative resemblance, and kicking achieves 4/10 under a foot-contact criterion. The authors acknowledge this scope, but it still limits the demonstrated significance. The main offline comparisons use different development cohorts, with unclear separation from model selection; the q0-only three-seed diagnostic does not quantify variability of the final body-hand policy. Missing details such as the frozen Action Expert's initialization, loss/source weights, and executed prefix length further hinder assessment of the claimed matched comparison.

Overall, I lean toward weak rejection. The integration and initial results are promising, but the specific advance over existing decoded-action-plus-correction methods and its closed-loop benefit are not yet convincingly established.

Questions for authors
  1. What is the precise technical advance of the residual control interface over VQ-BeT's decoded RVQ action plus continuous offset? Is there a comparison using the same converted HE/Nymeria data, policy capacity, and training budget, or an ablation separating visual cross-reconstruction from the physical refinement objective? This would materially affect my assessment of the method's contribution.

  2. Do the Table 3 direct-action policy and the decoder-only policy have real-G1 results under the Table 1 protocol? Such evidence would connect the offline improvements to the claimed deployment benefit. Please also specify the scene/initial-state variation and the reproducible judging rubric for the dance trials; the reported qualitative criterion makes that substantial part of the transfer result difficult to assess.

  3. Which evaluation cohorts were kept untouched during architecture/checkpoint selection, and how many independently trained final policies were evaluated? Please identify the frozen Action Expert checkpoint and provide the optimization schedule, loss/source weights, and deployed K_exec. These details are needed to assess the matching and reproducibility of the main comparisons.

Reviewer_EwPtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WAVE-VLA maps native G1 demonstrations and retargeted human egocentric motion into a common robot action space. Five sequential training stages learn body and hand codecs, current-observation code selectors, and a residual action expert. Predicted codes decode into synchronized trajectories, with body references refined before execution through SONIC. Evaluation includes offline motion reconstruction and prediction, plus 39/60 real-G1 task completions across six tasks.

Critical strengths
  1. A single policy executes behaviors represented in both robot and human training sources, while accommodating missing human hand labels. The real-G1 demonstrations provide evidence that the combined representation can support deployment.
  2. The matched offline comparison in Table 3 supports decoding a reference before refinement: body FK error decreases from 33.81 to 14.07 mm on HE and from 68.27 to 47.63 mm on Nymeria relative to code-conditioned direct prediction.
Critical weaknesses
  1. No established external policy baseline is evaluated: examples include adapted GR00T, π0.5, and Ψ0, as well as the closely related UniT. The current comparisons therefore do not establish an advantage over existing approaches using the same downstream data and controller.
  2. Tables 2, 3, and 7 provide useful offline controls, but do not establish that the full five-stage pipeline improves robot task success over simpler alternatives. In particular, the benefits of visual co-reconstruction over action-only tokenization, separate body/hand tokenization over unified decoding, and residual refinement during real execution remain untested. Stage-wise training costs and the initialization of the frozen Action Expert backbone are also insufficiently specified, making the complexity–benefit tradeoff difficult to assess.
  3. The robot evaluation provides limited evidence of visually grounded whole-body task competence. The six tasks comprise isolated interactions and motion demonstrations, with ten trials each. The joint policy achieves only 25/40 on the four HE manipulation tasks, compared with 27/40 for HE-only. Kicking succeeds in 4/10 trials even though success requires only foot–ball contact (Appendix, p. 13). Dance improves from 0/10 to 10/10, but involves no object interaction and is scored by subjective resemblance to Nymeria motions. The HE-only failures may reflect absent behavior coverage; this contrast does not establish that visual grounding or the proposed architecture is necessary. A language/state-only policy trained on the same mixed data would be an informative control.
  4. The supplementary video also lacks synchronized G1 egocentric observations, and object-placement variation is not described, preventing assessment of object visibility, occlusion, and the need for visual corrections. These limitations leave the practical advantage uncertain.
Questions for authors
  1. How does the method compare with a strong external VLA baseline adapted to the same G1 action interface, training data, and SONIC controller?
  2. Which benefits of visual co-reconstruction, separate hand tokenization, and residual refinement persist in real-robot success? Please include comparisons with simpler alternatives and report stage-wise training costs.
  3. What behaviors caused the ten HE-only dance failures, and how was resemblance scored? Does removing visual input while retaining language and state change performance under the same mixed-data training?
Reviewer_jKKTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

WAVE-VLA learns a humanoid policy from G1 demonstrations in Humanoid Everyday and human egocentric video–motion data in Nymeria, without requiring task-level pairing between the sources. Human motion is retargeted into a common G1 action space, and a shared RVQ codebook jointly represents body motion and visual transitions, with a separate body-conditioned hand stream. Given current images, robot state, and language, the policy predicts codes that decode into synchronized body and hand trajectories. A scene-conditioned residual expert refines the body trajectory before execution through the SONIC controller. Offline experiments evaluate reconstruction, code prediction, and trajectory refinement. Real-G1 experiments cover four HE tasks and two Nymeria tasks, reporting 39/60 successful trials for joint training versus 27/60 for HE-only training, with the gains coming from the two additional human-covered behaviors.

Critical strengths
  1. A practical framework for combining heterogeneous human and robot demonstrations. The method maps both sources into a common G1 action space without requiring matched human–robot tasks, while handling missing hand supervision explicitly. One policy executes skills covered by both sources on real hardware. Although the evaluation is limited, it provides concrete evidence that human video–motion data can extend a robot policy’s behavior repertoire.
  2. An explicit connection between discrete plans and continuous action generation. Decoding the selected codes into a reference trajectory before predicting a residual gives the action representation a direct role in the output. The matched offline comparison in Table 3 supports this design: it reduces body FK error by 58.4% on HE and 30.2% on Nymeria relative to the code-conditioned direct-action policy. This is a useful architectural result, even though its effect on real-robot task success remains to be established.
Critical weaknesses
  1. The real-robot evaluation provides limited evidence of scene-dependent interaction. The two additional human-covered tasks are dancing and kicking. Dancing involves no object interaction, while kicking succeeds when the foot contacts the ball and achieves only 4/10 successes. The four manipulation tasks are already covered by HE and primarily assess skill retention. With ten trials per task and no systematic evaluation across changing object configurations, the results demonstrate limited behavior expansion rather than robust interactive control. Moreover, the scene-conditioned residual expert is evaluated through offline trajectory errors, without a real-robot decoder-only comparison establishing whether refinement improves task success.
  2. The policy comparisons do not isolate the benefit of the proposed learning framework. HE-only versus joint training primarily evaluates the effect of adding data containing new behaviors. A strong VLA baseline trained on the same HE–Nymeria mixture, with a comparable G1 action interface and the same SONIC controller, is missing. Table 3 provides a useful internal comparison, but only through offline prediction errors. Consequently, the experiments do not sufficiently establish the advantage of the proposed architecture and training scheme over a conventional policy trained with the same data.
  3. Key representation choices lack targeted ablations. The reconstruction results establish that the RVQ representation can encode both sources, but do not isolate the benefit of joint visual supervision over an action-only tokenizer. Likewise, the contribution of conditioning hand predictions on the body plan is not directly tested. These omissions leave the value of the proposed vision–motion and body–hand design insufficiently supported.
  4. The figures and experimental presentation need substantial improvement. The architecture figures do not communicate the staged training, frozen modules, and body–hand information flow clearly enough, requiring substantial cross-referencing with the text. The offline results also span different evaluation cohorts and model variants, making it difficult to connect individual improvements to the final deployed policy. More legible, clearly annotated diagrams and a consolidated description of the evaluation settings would improve readability and reproducibility.
Questions for authors
  1. How were object positions and robot initial states varied across the real-robot trials, particularly for kicking and manipulation? Could you clarify whether the policy was evaluated under changes that require adapting the motion to the current scene?
  2. Do you have real-robot results comparing decoder-only execution with residual refinement under matched conditions? This would help determine whether the offline FK improvements translate into higher task success or more precise interaction.
  3. Have you evaluated a continuous-action VLA baseline using the same HE–Nymeria training data, a comparable G1 action interface, and the same SONIC controller? Such a comparison would help separate the benefit of additional data from that of the proposed learning framework.
  4. What evidence isolates the contribution of visual supervision in the body codebook and body-plan conditioning in the hand stream? Could you provide ablations against an action-only tokenizer and a hand predictor without body-plan conditioning, respectively?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.