acceptodds
Under review as a conference paper at ICLR 2027

Commanding Humanoid by Free-form Language: A Large Language-Action Model with Unified Motion Vocabulary

Abstract

Enabling humanoid robots to follow free-form natural language commands is a critical step toward seamless human-robot interaction and general-purpose embodied AI. However, existing methods remain limited, often constrained to simple instructions or forced to sacrifice motion diversity for physical plausibility. To address this gap, we present Humanoid-LLA, a Large Language-Action model that translates abstract and complex instructions to executable whole-body humanoid behaviors. Our approach tackles two core challenges: paired language-humanoid motion data scarcity and physical instability. First, we bridge high-level language semantics with physically grounded control by learning a unified human-humanoid motion vocabulary. Second, we introduce a novel two-stage fine-tuning framework that begins with supervised motion CoT learning, followed by reinforcement learning refined with physical feedback to ensure robustness and stability. Extensive evaluation in simulation and real-world experiments demonstrates that Humanoid-LLA achieves superior generalization to novel language commands and diverse motion generation while maintaining high physical fidelity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Language-guided motion generation, Physic-based humanoid control

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3LHNOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Humanoid-LLA, which aims to turn free-form language instructions into executable humanoid motions. The authors build a unified human–humanoid motion vocabulary, train a vocabulary-directed controller, and fine-tune the language model using motion CoT and RL with physical feedback.

Critical strengths
  1. The paper provides fairly comprehensive comparisons, including existing methods, ablation studies and real-world experiments.
  2. The authors learn a shared human–humanoid motion vocabulary and train a controller to execute motion tokens directly, avoiding online retargeting. They also use physical-feedback RL to improve motion generation and execution.
Critical weaknesses
  1. Some vector symbols in the equations appear to be in bold upright rather than bold italic. I guess the authors may have used \mathbf for these symbols. They might consider using \bm or other appropriate LaTeX commands instead, following the mathematical notation conventions in [1] and ISO 80000-2 [2].
  2. The experimental analysis of the loss functions is somewhat limited. Additional figures or tables would help clarify the contributions of individual loss terms.

References: [1] https://www.nist.gov/pml/special-publication-811/nist-guide-si-chapter-10-more-printing-and-using-symbols-and-numbers [2] https://www.iso.org/standard/64973.html (not open access)

Questions for authors
  1. The real-world experiments include only 10 trials per task(table10). Could the authors increase the number of trials and include more quantitative Sim2Real results in the main text?
  2. In Table 2, removing rtrackr_{track} has only a small impact on the first four values (table 2) but leads to a noticeable drop in success rate. Could the authors further analyze the reasons for this difference, particularly the failure cases?
3 comments

We thank the reviewer for recognizing the comprehensiveness of our evaluation and the merits of our token-conditioned control design. We also appreciate the constructive suggestions for further strengthening our experimental analysis. We address these points and the notation concern below.

W1: Inconsistent mathematical vector notation.

A: We thank the reviewer for pointing out this inconsistency. We have corrected the affected vector symbols in the main text and appendix from bold upright to bold italic notation, following the suggested conventions. The corrections are highlighted in blue in our revised PDF.

W2: Limited experimental analysis of individual loss terms.

A: We agree that the roles of individual objectives should be presented more clearly. Existing experiments provide direct objective-level evidence for Lcross\mathcal L_{\mathrm{cross}} (Appendix, Tables 12–14) and rdistr_{\mathrm{dist}} and rtrackr_{\mathrm{track}} (Table 2). Appendix Table 16 evaluates the controller architecture, prior, and goal masking.

To strengthen analysis of different loss terms, we have added a dedicated stage-wise summary in Appendix Table 17, covering: (1) Unified Tokenizer: Lintra\mathcal L_{\mathrm{intra}}, Lcross\mathcal L_{\mathrm{cross}}, Lcommit\mathcal L_{\mathrm{commit}}, and Lreg\mathcal L_{\mathrm{reg}}; (2) Controller Distillation: action imitation and KL regularization; and (3) LLM Fine-tuning: SFT/GRPO objectives, rformatr_{\mathrm{format}}, rdistr_{\mathrm{dist}}, and rtrack=rpos+raccr_{\mathrm{track}} = r_{\mathrm{pos}} + r_{\mathrm{acc}}. The table clarifies each term's purpose. We have also added references to this summary in all three corresponding main-text sections (revised PDF, lines 238–239, 298, and 376-377, highlighted in blue).

Q1: Limited real-world trials and insufficient quantitative Sim2Real results in the main text.

A: We have expanded the real-world evaluation from 10 to 20 trials per task across seven tasks, while keeping the evaluation protocol unchanged. We have also moved the quantitative Sim2Real evaluation from the original Appendix Table 10 into Section 4.5 and Table 4 of the revised main text (revised PDF, lines 486-505, highlighted in blue). The updated results are presented below.

Table R1. Expanded quantitative Sim2Real evaluation.

TaskSim-OnlySim2SimSim2Real (Updated)
Walk100%100%100%
Turn100%97.3%95%
Run98.8%95.3%90%
Jump92.3%89.4%65%
Kick85.9%82.0%70%
Dance78.3%68.3%60%
Kung fu81.0%74.2%65%

Q2: Why does removing rtrackr_{\mathrm{track}} significantly reduce success while having limited effects on the first four metrics? What are the underlying failure cases?

A: The key distinction lies in distribution-level motion quality versus sequence-level physical robustness. The first four metrics primarily measure overall motion distribution and semantic alignment, whereas Success requires the tracking deviation in Eq. 26 to remain below 0.5,m0.5\\,\mathrm{m} throughout execution. Thus, even brief tracking failures can cause an entire sequence to fail without substantially affecting aggregate generation metrics. Importantly, removing rtrackr_{\mathrm{track}} retains rollout-based distributional feedback through rdistr_{\mathrm{dist}}, but eliminates explicit optimization for position and acceleration tracking accuracy. Consequently, Success drops from 87.6% to 76.72%, while MPJPE increases from 56.43 to 66.42 mm, velocity error from 8.92 to 10.89, and acceleration error from 8.74 to 9.77.

Representative failures involve balance- and contact-critical transitions, such as prolonged single-leg support, repeated high kicks, or rapid rotations, where localized tracking deviations can cause falls or exceed the sequence-level success threshold. Such failures can substantially reduce Success while leaving aggregate distributional and semantic metrics largely unchanged.

I have no further questions. It would be nice to include the Q2 explanation in the main text, the current response is OK.

I will raise my rating from 3 to 4.

Thank you for your constructive feedback. We will incorporate the analysis of Q2 into the ablation study discussion in the main text of our next revised PDF version.

We sincerely appreciate your positive assessment of our work and the clarifications provided in our rebuttal, as well as the decision to increase your rating from 3 to 4. Thank you again for your time and consideration.

Reviewer_UKWNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper connects language-conditioned motion generation to humanoid control through a shared human–robot motion vocabulary, a token-conditioned controller, and physical-feedback RL. It evaluates the complete system in simulation and on real humanoids.

Critical strengths

The complete language-to-execution system provides empirical contribution despite limited component novelty. The evaluation is comprehensive and demonstrate the effectiveness of the method design. I consider this is sufficient for a weak acceptance.

Critical weaknesses

The method contribution is incremental. Language-driven humanoid motion systems have close precedents, including components like physical-feedback optimization and unified motion interfaces [1–3].

The LLM's SFT targets remain tokens encoded from kinematic motions. Physical execution informs the generator only through the final RL stage. A kinematic-motion generator with a tracking policy can also receive such feedback. It's not clear what exactly is the benefit of direct token-conditioned control over decoding the same tokens into robot motions for a general tracker.

The related-work discussion omits relevant language-conditioned humanoid systems [4–6]. Although their tasks and interfaces differ, discussing how they connect language understanding to physical control would better establish the scope and contribution of this system.

[1] FRoM-W1 — Li et al.

[2] RLPF — Yue et al.

[3] UniAct — Jiang et al.

[4] BiBo: Endowing GPT-4 with a Humanoid Body — Jian et al.

[5] Helix: A Vision-Language-Action Model for Generalist Humanoid Control — Figure AI

[6] WholeBodyVLA — Jiang et al.

Questions for authors

The proposed method follows [1, 2], which encodes kinematic motions into VAE tokens and executes them through a physical decoder branch. However, its benefit over kinematic motion generation + tracking remains unclear:

  1. The motion tokens are learned by encoding and reconstructing kinematic motion data. A physics-based decoder branch does not make them physical action tokens. The real difference is that physical decoder tracks kinematic motion latent, and PHC-like tracker tracks kinematic motion itself. The gap between motion mimic and physical interaction remains. This problem has also stated in prior works [2, 3].
  2. Physical-feedback RL is supported by both token-conditioned control and explicit motion tracking, rather than being exclusive to the former.
  3. Compared with explicit motion trajectories, latent tokens are harder to understand and edit. It also does not reduce the transferring cost to another robot, as it still requires training a robot-specific decoder branch.

Could the authors share their insights on choosing token-conditioned decoding instead of motion-tracking frameworks such as [4, 5]?

[1] PULSE: Universal Humanoid Motion Representations for Physics-Based Control — Luo et al.

[2] MaskedMimic — Tessler et al.

[3] ResMimic — Zhao et al.

[5] PHC: Perpetual Humanoid Control — Luo et al.

[6] SONIC — Luo et al.

3 comments

We appreciate the reviewer's positive assessment of our evaluation and the thoughtful questions regarding token-conditioned control. We address the remaining concerns below.

W1: Limited methodological contribution relative to existing language-driven humanoid systems (FRoM-W1, RLPF, and UniAct).

A: We agree that physical-feedback RL and discrete motion tokens have close precedents. However, our contribution lies in coupling language generation and physical execution through a shared motion token space across three stages: SFT learns language-to-token prediction, controller distillation enables direct token-conditioned execution, and physical-feedback RLFT directly refines token generation. The controlled comparison in W2 further evaluates the benefit of this unified token design.

W2 & Q1–Q2: The benefits of token-conditioned control over explicit motion decoding and tracking (W2), given that both rely on kinematic motion representations (Q1) and support physical-feedback RL (Q2).

A: Both paradigms support physical-feedback RL but differ in how generated motions constrain execution. In decoded tracking, the controller follows explicit frame-wise references reconstructed from generated tokens, where imperfections such as foot sliding, interpenetration, or inconsistent contacts can challenge stable execution. In contrast, our token-conditioned controller treats tokens as high-level motion intent without requiring rigid pose matching. Further, during privileged-teacher distillation, reference masking progressively removes dense goals, teaching the student to map ambiguous kinematic-token observations and proprioception to physically grounded actions. This improves robustness to imperfect tokens without converting the tokens themselves into physical action tokens.

To further validate, we have conducted a 2×2 comparison of decoded tracking and token-conditioned control, each with frozen SFT or physical-feedback RLFT. Frozen SFT provides identical generated tokens to both controllers. Under RLFT, both use the same SFT initialization, reward design, and evaluation protocol. Decoded tracking uses our pretrained teacher tracker.

Table R1. 2×2 comparison of execution interfaces and physical-feedback RL fine-tuning.

LLA TrainingExecution MethodSucc.↑MPJPE↓EvelE_{vel}↓EaccE_{acc}↓
Frozen SFTDecoded Tracking64.8%80.1712.8610.36
Frozen SFTToken-Conditioned68.6%78.3112.1210.01
RLFTDecoded Tracking80.5%65.899.109.03
RLFTToken-Conditioned (Ours)87.6%56.438.928.74

With independently optimized physical feedback, token-conditioned control improves success by 7.1 percentage points and reduces MPJPE by 9.46 mm compared with decoded tracking. These results support the system-level advantage of directly executing motion tokens, while distinguishing the contribution of the execution interface from that of physical-feedback fine-tuning.

W3: Insufficient discussion of related language-conditioned humanoid systems (BiBo, Helix, and WholeBodyVLA).

A: We have expanded the related-work discussion to explain how these systems connect language understanding to physical control (revised PDF, lines 158–161 and 191–198). BiBo uses an off-the-shelf VLM to compile instructions into parameterized motion primitives for a diffusion-based executor; Helix connects vision–language understanding to a fast visuomotor policy through latent semantic representations; and WholeBodyVLA combines a VLA model with an RL locomotion policy for visually conditioned loco-manipulation. We have also clarified their connection to our design: these systems motivate an effective language–control interface, while our focus is text-to-motion generation and execution through a unified motion vocabulary that supports both semantic grounding and controller-aware physical grounding.

Q3: Limitations of latent motion tokens in interpretability, editability, and cross-robot transfer.

A: We acknowledge that latent tokens are less directly interpretable and harder to edit than explicit motion trajectories. We therefore view token-conditioned execution as a practical design trade-off rather than a universally preferable interface. We have explicitly clarified these limitations in the revised PDF (lines 517-521).

Regarding cross-robot transfer, our advantage lies in semantic reuse rather than zero-shot physical transfer. We reuse the SFT-pretrained LLA on Booster T1 without T1-specific language–motion fine-tuning, while incorporating T1 trajectories into vocabulary construction and training a T1-specific controller. The results in Table 11 support the reuse of high-level language–motion grounding across embodiments, while acknowledging that robot-specific motion data and physical adaptation remain necessary.

Thank you for the responses. These points have partially addressed my concerns. In particular, I appreciate the additional 2×2 comparison between token-conditioned control and decoded tracking, as well as the expanded related work discussion.

Taking into account the questions and responses from the other reviewers as well, I will maintain my original judgment that this work should be accepted.

Thank you for taking the time to consider our responses and the discussion with the other reviewers. We appreciate your positive feedback on the additional experiments and the expanded related work discussion presented in our rebuttal. Thank you also for maintaining your recommendation to accept our work.

Reviewer_8g31OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Humanoid-LLA, a framework for translating free-form language instructions into humanoid behaviors. It learns a shared human–humanoid motion vocabulary, distills a token-conditioned controller from a privileged motion-tracking policy, and fine-tunes a language model using motion chain-of-thought supervision followed by simulation-based physical feedback.

Critical strengths
  • Physical-feedback fine-tuning produces substantial execution gains. Table 2 shows success increasing from 68.64% for SFT-only to 87.6% after RL fine-tuning, with MPJPE decreasing from 78.31 to 56.43 mm.
  • The evaluation addresses several plausible sources of bias. These experiments strengthen the evidence beyond improvement under the model’s own training reward.
Critical weaknesses
  • The CoT ablation does not isolate structured reasoning from richer supervision. The improvement could reflect more informative intermediate supervision, structured reasoning, or both.
  • The claimed benefit of exploiting otherwise unusable human data is not isolated in the end-to-end experiments. They do not quantify how much the additional human-only data improves language-conditioned robot execution.
Questions for authors

How does full-human-corpus SFT compare with executable-subset SFT when the tokenizer, controller, and physical-feedback fine-tuning are held fixed?

3 comments

We sincerely appreciate the reviewer's insightful suggestions for strengthening our claims through more rigorous and carefully controlled ablation studies, particularly regarding the contributions of structured reasoning and the benefits of leveraging the full human motion corpus. We address these points in detail below.

W1: The CoT ablation does not isolate structured reasoning from richer supervision.

A: We thank the reviewer for this important observation. We agree that the original CoT ablation (Table 2) demonstrates the overall effectiveness of Motion CoT but does not distinguish the contribution of structured reasoning from that of richer intermediate supervision.

To isolate structured reasoning from richer supervision, we have introduced a new Richer Supervision baseline using the same motion–text pairs, rendered-motion inputs, VLM annotation pipeline, and GPT-4o verification as Full Motion CoT, with comparable annotation lengths and semantic detail. All variants undergo the same downstream RLFT and evaluation procedure, with the tokenizer, controller, and reward design held fixed.

Crucially, both Richer Supervision and Full Motion CoT follow the same output template, including <think>...</think> and <motion>...</motion> tags and cyclic token-order requirements. Thus, the format reward does not favor either variant.

The key distinction is therefore how the intermediate motion information is represented: Richer Supervision provides detailed motion descriptions without reasoning or subgoal decomposition, whereas Full Motion CoT organizes comparable information into explicit reasoning steps and ordered motion subgoals.

Table R1. Effect of structured Motion CoT under comparable motion-information supervision.

SFT SupervisionFID↓R-Prec.↑MM-Dist↓Succ.↑
No CoT10.4230.2706.22264.9%
Richer Supervision6.2670.3125.69573.9%
Full Motion CoT2.6260.4474.91187.6%

Under this controlled setting, Full Motion CoT improves execution success by 13.7 percentage points over Richer Supervision (73.9% → 87.6%), with consistent gains in FID and R-Precision. These results support an additional benefit from structured motion decomposition beyond richer descriptive supervision.

W2 & Q: Contribution of Full Human Motion Corpus. How much does full-corpus SFT improve language-conditioned robot execution compared with executable-subset SFT, with the tokenizer, controller, and physical-feedback RL held fixed?

A: We agree that the contribution of additional human-only motion data should be explicitly quantified. We have additionally conducted a controlled comparison between Full-Corpus SFT and Executable-Subset SFT. Both settings share the same pretrained tokenizer, vocabulary-directed controller, SFT configuration, and physical-feedback RL procedure. The only difference is the SFT training data.

Table R2. Contribution of full human motion corpus to language-conditioned robot execution.

SFT DataFID↓R-Prec.↑MM-Dist↓Succ.↑
Executable Subset3.1210.4105.18381.2%
Full Human Corpus2.6260.4474.91187.6%

As shown in Table R2, Full-corpus SFT improves R-Precision by 0.037 and execution success by 6.4 percentage points, demonstrating the benefit of additional human-only motion data for both semantic alignment and physical execution.

Our shared motion vocabulary enables broader language–motion grounding using the full human corpus, while subsequent physical-feedback RL on the executable subset ensures robot feasibility. This full-corpus semantic alignment followed by executable-subset physical refinement enables more effective use of the additional human motion data while maintaining strong physical executability.

Thanks for your rebuttal which address my concerns, so I'll revise my rating from 2 to 3.

Thank you for your valuable suggestions on improving the rigor of our experiments. We especially appreciate your positive feedback on our rebuttal and your willingness to raise your score from 2 to 3.

We are truly grateful for your time and thoughtful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.