- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper proposes to jointly learn a shared latent space between motion controller and text specification of required behavior for streaming text-driven humanoid control. Specifically, paper introduces StaCo, which learns Control-Native Motion VAE through teacher–student policy distillation, enabling compositional latent space for downstream policy control. Learned DiT predicts future latents based on language and previous history of execution in order for low-level controller to output required behaviour. Empirical results on several established environments and datasets (HumanML3D, BABEL) showcase the performance boost compared to existing baselines.
Critical strengths
The method is intuitive and simple to understand. The main strength of the StaCo is in the idea of feeding encoded execution states back into the generator, which provides a practical mechanism for adapting future commands to the humanoid actual behavior. Ablations show improved tracking precision.
Critical weaknesses
The major weakness I see is lack of relevant comparisons with previous works. Although several recent methods are evaluated, MoMask [1] is absent from the generation comparison and the closely related RoboGhost [2] is not quantitatively evaluated.
The novelty also needs clearer and explicit positioning. Unified language motion latent representations and latent-conditioned control have already been in several works (e.g TEMOS [3], MotionCLIP [4]). StaCo specific contribution is its control-supervised state command representation and execution feedback. However, the experiments do not isolate its advantage over conditioning the generator on raw states or a separately learned state representation, as acknowledged in Appendix A.6
Moreover, the paper lacks individual loss-term ablations and loss-weight sensitivity analysis, making it hard to understand which component contributes most to the overall performance, while also making hard to tune. Also, in Section 3.3 the horizon for DiT future latent prediction is introduced, while no ablations conducted on this parameter. I think parameter is important and performance of StaCo could depend and vary greatly based on it from task to task.
[1] C. Guo "MoMask: Generative Masked Modeling of 3D Human Motions", CVPR 2024
[2] Li et al., "From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance", ICLR 2026
[3] Petrovich et al., "TEMOS: Generating Diverse Human Motions from Textual Descriptions", ECCV 2022
[4] G. Tevet et al. "MotionCLIP: Exposing Human Motion Generation to CLIP Space.", ECCV, 2022
Questions for authors
I suggest to redraw the main diagram of the method (Figure 2) since, despite being polished AI generated image, it is of low quality when scaled. Questions:
-
Is it correct that the main purpose of Stage 1 is to distill the VAE from PPO in order to provide a prior over the actions for controller? I do not fully understand what is the main intuition behind such choice?
-
How architectural choices of PPO parametrization from Stage 1, which acts as a teacher, impacts the overall quality? Does increasing the number of MOE experts results in predictable scaling in performance?
-
It is not clear how well does the VAE decoder approximates multi-modal distribution over actions from PPO teacher.
-
Am I correct that the main novelty and contribution lies in streaming component, driven by training DiT with perturbed noisy history and noisy future latents?
-
Is it possible during inference for StaCo to drive its behaviour to OOD states? I expect this due to execution state feedback loop, which could compound small errors and diverge. Is this indeed the possible case and how to mitigate it?