acceptodds
Withdrawn from ICLR 2027

Language-Grounded Controller Latents for Humanoid Whole-Body Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-conditioned humanoid motion is commonly built as a cascade in which a text-to-motion model generates a kinematic trajectory that is subsequently tracked by a separate controller. We instead generate the latent commands of a frozen whole-body controller directly from text, removing runtime retargeting and motion tracking from the primary generation path. We also incorporate differentiable kinematics into diffusion sampling to impose hand and foot target constraints directly in the controller latent space, without retraining the generator or controller. Experiments demonstrate strong language alignment, high completion rates, and effective spatial-constraint preservation after physical execution. We further demonstrate diverse behaviors in simulation and on the real Unitree G1. Beyond generation and execution, we characterize the semantic and physical structure of the frozen controller latent representation and examine whether this structure is preserved in latents generated directly from language. Our analysis shows that the 64-dimensional latent representation encodes joint configuration, locomotion direction and intensity, and gait phase in a distributed manner, with much of this structure preserved after finite scalar quantization and also present in the generated latents. We further show that text-conditioned motion characteristics can be composed in latent space to generate motions combining multiple attributes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid Control, Language-Conditioned Motion Generation, Controller Latent Space, Whole-Body Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kw1hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper generates language-conditioned motion commands directly in the latent space of a frozen SONIC controller, avoiding explicit kinematic motion generation. It also explores spatial guidance and semantic composition. The idea is practical, and the experiments show good execution stability and competitive motion quality. Overall, I find the results encouraging, though a few aspects deserve further investigation.

Critical strengths
  1. This paper generates language-conditioned motion commands directly in the latent space of a frozen SONIC controller, avoiding explicit kinematic motion generation.

  2. The idea is practical, and the experiments show good execution stability and competitive motion quality.

Critical weaknesses
  1. Why generate continuous latents rather than quantized commands directly? Since SONIC eventually applies FSQ, I wonder whether modeling discrete commands could work equally well, or even better.

  2. The model is trained on successful SONIC rollouts, which may already favor stable motions. How much of the improvement comes from this filtering rather than the latent representation itself? A comparison using the same backbone and training data would help.

  3. I am not fully convinced that the method reduces the semantic gap. Even ground-truth motions lose R@3 after SONIC execution, and the proposed method does not outperform the strongest baselines in executed R@3. Perhaps the main advantage is stability rather than semantic preservation?

  4. The spatial guidance results are somewhat mixed. It improves foot constraints but is worse than Kimodo on hand constraints without IK. Also, the large error increase after IK and physical execution makes me wonder how accurately the kinematic decoder reflects actual controller behavior.

  5. The composition results seem weaker than the qualitative examples suggest. Many action pairs largely ignore one attribute. Is score composition really better than simply using a compound text prompt? A direct comparison would be useful.

  6. The latent sequence is generated before execution, without high-level feedback. How would the method handle disturbances or unexpected changes in motion timing? Even a simple perturbation test would help clarify its robustness.

Questions for authors

Please see the Weaknesses.

Reviewer_AMmhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method that takes language as input and directly outputs the latent codes of a humanoid robotics controllers. The usual pipeline goes from text to motion and then uses that motion to control a humanoid. This paper instead proposes to directly go from text to the latents of a humanoid controller.

Critical strengths

The paper presents an incrementally different pipeline compared to existing baselines. I am not sure if the idea is significantly novel compared to existing methods - the SONIC policy remains frozen and the paper basis itself around it. The architectures used also seem to be relatively same as used in existing methods.

The most novel part of the paper is perhaps the problem formulation itself - 'can one go directly from text to the laten codes of a robotics controller?' - in hindsight the answer seems obvious; if one creates a dataset of such latents using a fixed policy/decoder and then learns the mapping from language to the latents the problem becomes tractable. I am not sure if I have learned anything significant from this though. As I write in my other sections I will give a reject rating for now and will discuss with other reviewers.

Critical weaknesses

I understand what the paper is trying to do - directly generate latents that can then be executed on a robot - thus foregoing the intermediate text to motion network.

But I am not convinced that this is well motivated in the paper.

There are many questions in my mind. For example: is text to motion really useful in the context of robotics. One can imagine a robot dancing or waving in an empty room, but in what really matters in robotics is spatial control -- i.e can we make the robot interact with the 3D world in specific ways. The paper addresses that towards the end, but its main concern is with language.

Further, one can imagine that the gain from this setup is in terms of speed, but the supplementary video doesn't make this very clear. How much faster is this model on real hardware compared to the text-to-motion + SONIC baseline.

In terms of numbers, table 1 seems to indicate that there is no significant improvement if one uses this idea compared to other baselines such as KIMODO. This also raises questions about the utility of this paper.

Questions for authors

For now I will give a reject rating. I would like to see what other reviewers have to say about this paper and discuss with the ACs as well. Could the authors address the concerns I have raised in the weakness section.

Reviewer_BVvNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a text-conditioned diffusion model to generate the latent commands of a frozen SONIC humanoid controller directly. It also introduces sampling-time spatial guidance through a frozen kinematic decoder, supports semantic composition through conditional denoising directions, and analyzes the controller’s latent representation. Evaluation includes comparisons with kinematic generation pipelines, spatial-constraint experiments, and qualitative demonstrations on a Unitree G1.

Critical strengths
  1. A practical interface between language generation and an existing controller. Keeping SONIC frozen makes the approach modular and avoids training a new control policy. Evaluating all methods through the same downstream controller is a useful design choice, and the physical demonstrations support feasibility.

  2. Evaluation measures execution rather than only generated motion. The paper reports language alignment, falls, and constraint errors after physical execution. The stage-wise constraint analysis is particularly informative: error increases from 0.3 cm after IK to 3.3 cm after re-encoding and 7.2 cm after execution, clearly identifying limitations of the controller interface.

Critical weaknesses
  1. The benefit of the controller-latent representation is not sufficiently isolated. Compared with MoMask, completion improves only from 96.8% to 97.0% and execution FID from 7.3 to 6.9, while executed R@3 decreases from 0.892 to 0.879. Table 1 provides no uncertainty estimates to assess these differences. The architectures and training configurations also differ substantially; for example, the proposed denoiser uses 16 layers at width 1024, whereas MDM uses eight layers at width 512. A matched diffusion model predicting kinematics instead of controller latents would more directly test the central contribution. Additionally, the evaluation clips are explicitly excluded from TMR training, but their exclusion from generator training is not clearly documented.

  2. The spatial-control results provide mixed support for the proposed guidance mechanism. Guidance alone achieves 53% overall success within 10 cm, and its hand success is lower than Kimodo-G1’s native constraint interface: 42% versus 46%. The strongest result, 75%, uses the optional kinematic IK and re-encoding path; free generation with the same projection already achieves 69%. The paper transparently separates these variants, but stronger evidence is needed to establish the specific advantage of latent-space guidance. Evaluation is also restricted to six fixed pelvis-relative targets, with foot constraints tested on stationary prompts, limiting conclusions about broader spatial control.

Questions for authors
  1. Are the evaluation motion clips, all associated captions, and overlapping source segments excluded from training every locally trained generator? Please specify the split and whether all baselines use the same successful-rollout filtering.

  2. Does the advantage persist when the same diffusion architecture, training data, and optimization budget are used to predict kinematics rather than SONIC latents? Can uncertainty estimates be provided for the Table 1 differences?

  3. How does spatial guidance affect language alignment, particularly when the target conflicts with the requested action? Does its benefit persist for unseen target locations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.