- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces spectral skills, a framework for learning a latent space, which serves as interface between motion generation and motion tracking. The method includes training a predictive motion encoder trained offline with future motion prediction as the objective. Then the frozen motion encoder conditions a tracking policy. Experiments results include improved motion tracking performance, skill chaining and steering capabilities, latent space analysis and language-conditioned planning with the latent interface.
Critical strengths
- The direction of this research is worth exploring and has broad applications. Current motion generation plus motion tracking frameworks usually fall into two categories. One way is to use SMPL-format generative models such as Kimodo/GENMO to generate the motion offline, then followed by retargeting, then followed by motion tracking that takes in explicit humanoid motion as condition. Works like OMG and TextOp take this one step further, generating directly in humanoid motion space, as well as being able to run inference at real time, therefore being interactive. From my perspective, connecting motion generation with motion tracking with a latent interface is quite essential, and a promising direction with broad applications.
- The tracking results are substantial, and supported by useful ablations.
Critical weaknesses
-
The claim of "producing combinations that are unseen from the training data" in abstract is under-justified. Section 4.2 supports composition of rare combinations but not unseen combinations. It would be better to provide additional evidence or narrow the claim.
-
Lacking empirical evidence supporting latent generation-tracking interface being better than explicit motion generation-tracking interface. See Questions 1.
Questions for authors
-
Since a core argument of this work is using a latent interface is more performant than explicit motion interface. I would like to see comparison against two baselines from language-conditioned planning: (a.) GENMO motion generation->retarget->SONIC motion tracking; (b.) OMG motion generation->Holomotion motion tracking.
-
Can the authors provide gains of this specific affine/spectral construction against a baseline of predictive training itself? I am rather concerned on to what extend is the source of performance gain attributed to this spectral construction.
3 comments
As the rebuttal deadline approaches, we hope the reviewer has had a chance looking at our new experiments and that the requested experiments addressed their concerns.
I acknowledge the authors' efforts and additional evidence provided. However, I still do have concerns against the "latent generation-tracking interface being better", especially compared against additional OMG+holomotion performance, seems that the gain of this spectral skill decomposition is mostly gains in efficiency rather than in performance. I will maintain my rating.
We thank the reviewer for the recognition of our work. We first wish to highlight that this work focuses on skill composition, not merely motion generation and tracking performance for low-level policy. We wish to answer the questions of, how can we compose or steer learned atomic motion skills in a principled way? What kind of representation are we looking for to achieve this property? As it turns out, using an affine structure energy based model with diffusion, we can learn a JEPA style motion encoder through modeling the dynamics of human retargeted motion segments, without any action labels.
Here are responses to the concerns and questions.
W1. "Claiming unseen combination is under-justified" We test unseen composition by re-training a subset of the data that does not have arm-raise + locomotion and perform composition post-hoc to see if we can reproduce the motion (retraining both encoder and policy). We remove every clip, and its mirror through the dataset with walking speed ≥0.3m/s with minimum shoulder pitch ≤−0.4rad, which consists of around 12k clips.
Here A is locomotion, B an arm raise and C standing. This shows we can indeed produce unseen motion through steering. Additionally, using the full-corpus controller, the executed motions are also not copies of corpus motions. The joint-space distance to the nearest corpus motion is strictly positive, meaning they are separable from the motion dataset itself. To our knowledge this is the first skill space for whole-body control with this property. We will add the held-out experiment to the appendix and make this more clear in the paper.
W2 & Q1. "Lacking empirical evidence supporting latent generation-tracking interface being better" We also evaluate OMG and GENMO pipeline the reviewer has mentioned. We retrain OMG and GENMO on the same dataset. Here is the result under the same inference pipeline (realtime, async evaluation in sim); all trials share initialization, reference timing and success definition.
We note that larger size motion generation might perform better on motion generation itself but for online policy execution on a real robot, larger models have a computation tradeoff between accuracy and throughput. A compact latent space seems to be a better choice in this scenario. Additionally, notice that we used HoloMotion instead of SONIC as SONIC itself is a latent controller with FSQ.
Q2. "gains of this specific affine/spectral construction against a baseline of predictive training" We argue that predictive training improves tracking, and the affine construction supplies an exact, additive prediction response which allows to steer, mix or chain motion skills in a smoothed, "linear" space. We provide substantial and extensive ablation studies on each design choice of our method in the appendix.
We compare affine and concatenation predictors trained with the same predictive objective, each paired with a tracker trained for 9.5B frames.
We can see that concat is a more flexible representation choice for motion tracking but produces worse skill composition capability, shown in lower composition completion rate and higher unintended motion effects. While the affine construction has a similarly strong tracking performance and much better compositionally. Notice that due to the EBM form, concat itself has some compositionality due to the energy function being ⟨ϕ(X,z),μ(Y)⟩ but ϕ is nonlinear in z so this factorization does not give skill composition by construction, which the affine version provides. We will append this result in the appendix.