acceptodds
Withdrawn from ICLR 2027

StaCo: Streaming Text-Driven Humanoid Control with Shared State–Command Latents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

15 views

Abstract

Existing latent-driven methods for text-driven humanoid control streamline the generation-to-control pipeline, but typically learn motion representations independently of control policies and rely on command-centric latents. This separation yields representations that are weakly aligned with robot execution and prevents the generator from incorporating the robot's actual execution state into future predictions. We propose StaCo, a shared state–command latent framework that couples motion representation learning with humanoid control and closes the feedback loop between streaming generation and physical execution. StaCo trains a Control-Native Motion VAE through teacher-student policy distillation, yielding a robot-native latent space that serves both as an execution-state representation and as an executable control target. Built on this shared latent space, a streaming DiT predicts future command latents from language and historical execution latents, while proprioceptive feedback is re-encoded after execution to inform subsequent generation. Experiments demonstrate that the learned latent space supports robust humanoid control while retaining high-fidelity motion structure, and that the streaming generator produces high-quality motions consistent with language instructions. End-to-end evaluation achieves a 97.35% strict control success rate in MuJoCo, while text-conditioned generation attains an FID of 19.14 on HumanML3D-derived robot motions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid locomotion, Reinforcement Learning, Motion Generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Uc49OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper proposes to jointly learn a shared latent space between motion controller and text specification of required behavior for streaming text-driven humanoid control. Specifically, paper introduces StaCo, which learns Control-Native Motion VAE through teacher–student policy distillation, enabling compositional latent space for downstream policy control. Learned DiT predicts future latents based on language and previous history of execution in order for low-level controller to output required behaviour. Empirical results on several established environments and datasets (HumanML3D, BABEL) showcase the performance boost compared to existing baselines.

Critical strengths

The method is intuitive and simple to understand. The main strength of the StaCo is in the idea of feeding encoded execution states back into the generator, which provides a practical mechanism for adapting future commands to the humanoid actual behavior. Ablations show improved tracking precision.

Critical weaknesses

The major weakness I see is lack of relevant comparisons with previous works. Although several recent methods are evaluated, MoMask [1] is absent from the generation comparison and the closely related RoboGhost [2] is not quantitatively evaluated.

The novelty also needs clearer and explicit positioning. Unified language motion latent representations and latent-conditioned control have already been in several works (e.g TEMOS [3], MotionCLIP [4]). StaCo specific contribution is its control-supervised state command representation and execution feedback. However, the experiments do not isolate its advantage over conditioning the generator on raw states or a separately learned state representation, as acknowledged in Appendix A.6

Moreover, the paper lacks individual loss-term ablations and loss-weight sensitivity analysis, making it hard to understand which component contributes most to the overall performance, while also making hard to tune. Also, in Section 3.3 the horizon LL for DiT future latent prediction is introduced, while no ablations conducted on this parameter. I think LL parameter is important and performance of StaCo could depend and vary greatly based on it from task to task.

[1] C. Guo "MoMask: Generative Masked Modeling of 3D Human Motions", CVPR 2024

[2] Li et al., "From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance", ICLR 2026

[3] Petrovich et al., "TEMOS: Generating Diverse Human Motions from Textual Descriptions", ECCV 2022

[4] G. Tevet et al. "MotionCLIP: Exposing Human Motion Generation to CLIP Space.", ECCV, 2022

Questions for authors

I suggest to redraw the main diagram of the method (Figure 2) since, despite being polished AI generated image, it is of low quality when scaled. Questions:

  1. Is it correct that the main purpose of Stage 1 is to distill the VAE from PPO in order to provide a prior over the actions for controller? I do not fully understand what is the main intuition behind such choice?

  2. How architectural choices of PPO parametrization from Stage 1, which acts as a teacher, impacts the overall quality? Does increasing the number of MOE experts results in predictable scaling in performance?

  3. It is not clear how well does the VAE decoder approximates multi-modal distribution over actions from PPO teacher.

  4. Am I correct that the main novelty and contribution lies in streaming component, driven by training DiT with perturbed noisy history and noisy future latents?

  5. Is it possible during inference for StaCo to drive its behaviour to OOD states? I expect this due to execution state feedback loop, which could compound small errors and diverge. Is this indeed the possible case and how to mitigate it?

Reviewer_1GJ9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a humanoid control framework to generate robot motion commands given streaming text. The framework consists of 2 parts: 1) the Control-Native Motion VAE, which encodes robot state and control into latent space, and 2) the Streaming Text-conditioned Diffusion Transformer, which autoregressively predicts latent code conditioned on text input. Evaluation on text-conditioned robot-motion generation and humanoid control demonstrate SOTA performance on text-motion alignment and motion control performance.

Critical strengths

Clear writing: The writing is clear and easy to follow. The figure presents the framework well and the paragraph explains details enough to support reproducibility.

Critical weaknesses

Insufficient comparison experiments and ablations: The diffusion timestep settings are not aligned across methods. In text-to-motion and long-term motion generation, StaCo uses 20 diffusion steps, while other methods (e.g., FloodDiffusion) use about 10. The paper should also report StaCo with 10 steps and the baselines with 20 steps, so that the comparison is fair.

Most comparisons are against streaming models. The paper should also compare with text-to-motion models (MDM, MLD, ReMoDiffuse) and long-term motion generation models (PCMDM, PriorMDM, TransPhase), as FloodDiffusion also does with MDM and MLD. Diffusion steps can be adjusted so that the baselines have a similar inference time to the proposed method. Diffusion steps can be adjusted so that the baselines have a similar inference time to the proposed method. Please also report both generation quality and inference time in more detail for a fair comparison.

Insufficient novelty: The model is a VAE + autoregressive diffusion trained on robot states, which is a common setup in real-time motion generation. Some novelty is using a privileged teacher to supervise VAE training, but this contribution is marginal. The “unified generation–control latent interface” is also not clearly distinguished from the other two, making this somewhat repetitive.

[1]: Tevet, Guy, et al. "Human motion diffusion model." International Conference on Learning Representations (ICLR). 2023. [2]: Chen, Xin, et al. "Executing your commands via motion diffusion in latent space." Computer Vision and Pattern Recognition (CVPR). 2023. [3]: Zhang, Mingyuan, et al. "Remodiffuse: Retrieval-augmented motion diffusion model." International Conference on Computer Vision (ICCV). 2023. [4]: Yang, Zhao, Bing Su, and Ji-Rong Wen. "Synthesizing long-term human motions with diffusion models via coherent sampling." ACM International Conference on Multimedia (ACMMM). 2023. [5]: Shafir, Yonatan, et al. "Human motion diffusion as a generative prior." International Conference on Learning Representations (ICLR). 2024. [6]: Au, Ho Yin, et al. "Deep compositional phase diffusion for long motion sequence generation." Neural Information Processing Systems (NeurIPS), 2025.

Questions for authors

How are the diffusion timesteps chosen for StaCo (20) and the compared methods (10)? Is there a runtime trade-off that motivates this setting?

The T2M metric values are much larger than those reported in prior work. Please clearly explain this discrepancy in the main paper.

Reviewer_QN31OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Paper proposes using a VAE to create a control latent that captures both command as well as state / proprioception (called the control-native motion VAE in the paper). This is done by using an encoder to encode the reference motion into this latent while the decoder is condition on proprioception to decode to control actions. The encoder and decoder are learned using a teacher-student architecture where the teacher is a mixture of experts. A Diffusion transformer is then designed to take the encoded proprioception and text commands to output command latents that the decoder than converts to actions. Simulation results are presented in Mujoco and Isaac Sim to showcase better tracking and higher success rate compared to baselines.

Critical strengths
  • The use of an Encoder + DiT as a motion generator in latent space and a Decoder as a controller is interesting.

  • Figures 2 clearly demonstrates the method.

  • The proposed method demonstrates better success rate and lower tracking error compared to baselines.

Critical weaknesses
  • The authors mention that they create a state-command (StaCo) latent. However the encoder encodes only the reference motion while the decoder is conditioned on the state / proprioception. I wouldn't call the latent space as state-command latent. It's just encoding the reference command.

  • This is a Humanoid paper but has no humanoid experiments. Experiments will add more value to the paper and also determine robustness of the policy. It might be possible that the motion GPT controller is sacrificing success rate in order to be robust and work in real, where as the proposed controller here has higher success rate but is not robust and can't be demonstrated on hardware.

  • Baselines like MotionStreamer (MS) / FloodDiffusion (FD) have separate motion generator and motion tracker. This may not be a completely fair comparison as a coupled generator + controller could potentially do better. Is the better performance in the paper coming from the coupled generator and controller or something specific to the method here?

  • Novelty of the paper is unclear.

Questions for authors
  • Why is the encoder not conditioned on the state / proprioception history?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.