acceptodds
Under review as a conference paper at ICLR 2027

Learning Expressive and Compositional Motion Representation via Spectral Skills

Abstract

Robotic foundation models offer a promising path toward general-purpose humanoid robot control often with hierarchical architectures. However, their effectiveness depends on a command interface between the planner and the controller, which must support accurate execution while remaining easy to predict, and ideally compose new behaviors from prior ones. In this work, we introduce a latent-representation of such interface we term spectral skills, which meets these requirements through predictive representation learning. By design, the spectral skills compactly encodes short motion segments and are learned by predicting subsequent motion rather than reconstructing the encoder’s input. On a 29-DoF humanoid, a controller conditioned on spectral skills reduces global tracking error by relative to the state-of-the-art. The same frozen controller chains independently encoded skills without a separate transition policy. It also composes new behaviors by adding orthogonal directions to any compatible base skill, producing combinations that are unseen from the training data. We demonstrate tracking, chaining and composition, as well as through a languaged conditioned planner on Unitree G1 hardware. Project page: https://spectral-skill.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement Learning, Representation Learning, Humanoid, Whole-body Control

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_A47NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces spectral skills, a framework for learning a latent space, which serves as interface between motion generation and motion tracking. The method includes training a predictive motion encoder trained offline with future motion prediction as the objective. Then the frozen motion encoder conditions a tracking policy. Experiments results include improved motion tracking performance, skill chaining and steering capabilities, latent space analysis and language-conditioned planning with the latent interface.

Critical strengths
  1. The direction of this research is worth exploring and has broad applications. Current motion generation plus motion tracking frameworks usually fall into two categories. One way is to use SMPL-format generative models such as Kimodo/GENMO to generate the motion offline, then followed by retargeting, then followed by motion tracking that takes in explicit humanoid motion as condition. Works like OMG and TextOp take this one step further, generating directly in humanoid motion space, as well as being able to run inference at real time, therefore being interactive. From my perspective, connecting motion generation with motion tracking with a latent interface is quite essential, and a promising direction with broad applications.
  2. The tracking results are substantial, and supported by useful ablations.
Critical weaknesses
  1. The claim of "producing combinations that are unseen from the training data" in abstract is under-justified. Section 4.2 supports composition of rare combinations but not unseen combinations. It would be better to provide additional evidence or narrow the claim.

  2. Lacking empirical evidence supporting latent generation-tracking interface being better than explicit motion generation-tracking interface. See Questions 1.

Questions for authors
  1. Since a core argument of this work is using a latent interface is more performant than explicit motion interface. I would like to see comparison against two baselines from language-conditioned planning: (a.) GENMO motion generation->retarget->SONIC motion tracking; (b.) OMG motion generation->Holomotion motion tracking.

  2. Can the authors provide gains of this specific affine/spectral construction against a baseline of predictive training itself? I am rather concerned on to what extend is the source of performance gain attributed to this spectral construction.

3 comments

We thank the reviewer for the recognition of our work. We first wish to highlight that this work focuses on skill composition, not merely motion generation and tracking performance for low-level policy. We wish to answer the questions of, how can we compose or steer learned atomic motion skills in a principled way? What kind of representation are we looking for to achieve this property? As it turns out, using an affine structure energy based model with diffusion, we can learn a JEPA style motion encoder through modeling the dynamics of human retargeted motion segments, without any action labels.

Here are responses to the concerns and questions.

W1. "Claiming unseen combination is under-justified" We test unseen composition by re-training a subset of the data that does not have arm-raise + locomotion and perform composition post-hoc to see if we can reproduce the motion (retraining both encoder and policy). We remove every clip, and its mirror through the dataset with walking speed ≥0.3m/s\geq 0.3 m/s with minimum shoulder pitch ≤−0.4rad\leq -0.4 rad, which consists of around 12k clips.

Executed compositionHeld-out datasetFull corpus
Arm-direction steering, 3 SD: success ↑18/2118/21
Addition, zA+0.5(zB−zC)z_A+0.5(z_B-z_C): success ↑60/6360/63
Interpolation, 0.5zA+0.5zB0.5z_A+0.5z_B: success ↑54/6355/63
Steering: arm-raise magnitude (rad)0.600.55
Steering: forward speed, base → steered (m/s)0.76 → 0.770.75 → 0.73
Steering: falls ↓0/210/21

Here AA is locomotion, BB an arm raise and CC standing. This shows we can indeed produce unseen motion through steering. Additionally, using the full-corpus controller, the executed motions are also not copies of corpus motions. The joint-space distance to the nearest corpus motion is strictly positive, meaning they are separable from the motion dataset itself. To our knowledge this is the first skill space for whole-body control with this property. We will add the held-out experiment to the appendix and make this more clear in the paper.

W2 & Q1. "Lacking empirical evidence supporting latent generation-tracking interface being better" We also evaluate OMG and GENMO pipeline the reviewer has mentioned. We retrain OMG and GENMO on the same dataset. Here is the result under the same inference pipeline (realtime, async evaluation in sim); all trials share initialization, reference timing and success definition.

Online executionOursOMG + HoloMotionGENMO + GMR + HoloMotion
MPJPE-L, 74 trials completed by all (mm) ↓25.236.8104.5
Completed trials ↑84/8484/8474/84
Planner response time (ms) ↓604801,737

We note that larger size motion generation might perform better on motion generation itself but for online policy execution on a real robot, larger models have a computation tradeoff between accuracy and throughput. A compact latent space seems to be a better choice in this scenario. Additionally, notice that we used HoloMotion instead of SONIC as SONIC itself is a latent controller with FSQ.

Q2. "gains of this specific affine/spectral construction against a baseline of predictive training" We argue that predictive training improves tracking, and the affine construction supplies an exact, additive prediction response which allows to steer, mix or chain motion skills in a smoothed, "linear" space. We provide substantial and extensive ablation studies on each design choice of our method in the appendix.

We compare affine and concatenation predictors trained with the same predictive objective, each paired with a tracker trained for 9.5B frames.

MeasureAffineConcatenation
Tracking success ↑91.72%93.04%
MPJPE-G (mm) ↓100.7101.7
Arm raise + turn: upright completion ↑97.0%75.2%
Both effects ≥\geq 80% retained and upright ↑64.8%61.7%
Unintended left-arm motion (RMS, rad) ↓0.0520.075
Unintended speed change (m/s) ↓0.0730.129

We can see that concat is a more flexible representation choice for motion tracking but produces worse skill composition capability, shown in lower composition completion rate and higher unintended motion effects. While the affine construction has a similarly strong tracking performance and much better compositionally. Notice that due to the EBM form, concat itself has some compositionality due to the energy function being ⟨ϕ(X,z),μ(Y)⟩\langle \phi(X,z), \mu(Y)\rangle but ϕ\phi is nonlinear in zz so this factorization does not give skill composition by construction, which the affine version provides. We will append this result in the appendix.

As the rebuttal deadline approaches, we hope the reviewer has had a chance looking at our new experiments and that the requested experiments addressed their concerns.

I acknowledge the authors' efforts and additional evidence provided. However, I still do have concerns against the "latent generation-tracking interface being better", especially compared against additional OMG+holomotion performance, seems that the gain of this spectral skill decomposition is mostly gains in efficiency rather than in performance. I will maintain my rating.

Reviewer_4X9JOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Spectral Skills, a latent motion representation learned by predicting future motion rather than reconstructing the input motion. By extracting spectral directions from the predictor's Jacobian, the method enables motion editing and composition without retraining the controller. Experiments on the Unitree G1 demonstrate motion tracking, skill chaining, composition, and language-conditioned control. Compared with SONIC, the method reduces global tracking error by approximately 62% and demonstrates the ability to combine different motion attributes.

Critical strengths
  1. Meaningful research problem. Learning predictive skill representations provides a reasonable approach to designing the interface between high-level planning and low-level control.

  2. Strong tracking performance. Table 1 shows that MPJPE-G decreases from 187.86 mm to 70.51 mm, while MPJPE-L decreases from 26.74 mm to 20.54 mm, using approximately 12× fewer RL environment steps.

  3. Experimental support for the main design choice. Table 7 shows that predictive representations substantially reduce global tracking error compared with reconstruction-based representations under matched training budgets (99.51 vs. 278.98 mm).

  4. Practical demonstrations. The method demonstrates skill chaining and motion composition in simulation and on a real Unitree G1, with comparisons against SONIC, BFM-Zero, and joint-offset baselines.

Critical weaknesses
  1. Limited methodological originality and insufficient evidence for compositionality. The predictive representation learning largely builds on DiffSR, while Jacobian-based latent direction analysis is also well established. The main contribution is therefore an integration of existing techniques into humanoid control rather than a fundamentally new representation learning approach. More importantly, Proposition 3.1 establishes linearity only for a single denoising prediction under a fixed context, not for executed robot motions. In Table 13, spectral directions alone achieve success rates of only 3/5, 0/5, 1/5, and 2/5 for arm raising, arm spreading, speed adjustment, and turning, respectively. Some results are even worse than SONIC's PCA directions. These findings limit the evidence for general compositionality.

  2. Insufficient attribution of the main performance improvement. The reported 62% reduction in global tracking error primarily comes from reduced root-position drift. Appendix E acknowledges an important difference: SONIC does not provide reference root-position information to its actor, whereas the proposed encoder can convey this information to the controller. Although Table 7 supports the benefits of predictive learning, the current comparison does not isolate the contributions of the representation learning objective and the additional information available to the controller. Consequently, the main performance improvement cannot be fully attributed to Spectral Skills.

  3. Limited evidence for a unified control interface and generalization. The language-conditioned planning experiments in Table 3 use an earlier 256-D controller rather than the final 64-D controller used for the main tracking and composition experiments. Therefore, these results do not directly establish the planning advantages of the final Spectral Skills representation. Furthermore, the advantage of latent interfaces depends on the replanning frequency. With replanning at every step, explicit prediction outperforms latent prediction (94.5% vs. 84.3% success). The 4096 tracking motions are also drawn entirely from the training corpus, leaving generalization to unseen motions insufficiently evaluated.

Questions for authors

Q1. Compared with DiffSR and existing Jacobian-based latent steering methods, what is the key methodological contribution of Spectral Skills beyond integrating these techniques into humanoid control?

Q2. Appendix E identifies a difference in reference root-position information available to the controllers. Would the reported 62% reduction in global tracking error remain under a comparison with matched input information?

Q3. Why do spectral directions alone underperform SONIC's PCA directions on certain tasks in Table 13? Under what conditions does the linear response in latent space translate into reliable composition of executed robot motions?

Q4. Why are the language-conditioned planning experiments conducted with an earlier 256-D controller rather than the final 64-D Spectral Skills model? Does the proposed representation retain its advantages on unseen motions and across different replanning frequencies?

The paper demonstrates strong practical results, but its methodological contribution largely builds on existing techniques. I currently lean toward a borderline rejection, but I would be open to reconsidering my assessment if the authors can provide a more convincing justification of the methodological originality and stronger experimental evidence supporting their core claims.

4 comments

We thank the reviewer for spending time reviewing the paper and the recognition that this is a meaningful research project with strong performance. We wish to highlight that this work focuses on skill composition and steering, in hope to answer the questions of, how can we compose or steer learned atomic motion skills in a principled way? What kind of representation are we looking for to achieve this property? As it turns out, using an affine structure energy based model with diffusion, we can learn a JEPA-style motion encoder through modeling the dynamics of human retargeted motion segments, without any action labels.

Here we address the reviewer's concerns and questions.

W1

Limited methodological originality and insufficient evidence for compositionality

The prior work of diffsr cannot be applied in this problem as the factorization target does not have action labels. In DiffSR, the objective factorizes $ p(s_{t+1}\mid s_t,a_t) \propto \exp{\phi(s_{t}, a_t)^T \mu(s_{t+1})} $ using the policy replay buffer where we have $ (s,a) $ pairs. We are working on a different problem: we factorize $ p(x_{t+T:t+L+T}\mid x_{t-L:t-1}, z) \propto \exp{ (F(x_{t-L:t-1})^T (Az+b)) ^T \mu(x_{t+T:t+L+T})} $, where $ z = Enc(x_{t:t+T}) $, and $ x$ are retargeted human motion (not from policy replay buffer) and we pre-train the encoder with offline data. Neither the factorization target of the retargeted human motion, nor designing the additional affine structure inside the energy function has been widely explored before. We then use the $z $ to condition the policy for training goal-conditioned imitation policy $ \pi(a|s,z) $, which has no connection to how diffsr trains the policy whatsoever.

Additionally, for an arbitrary learned representation, such as SONIC's FSQ encoder, Jacobian analysis is a first-order approximation of a highly nonlinear decoder:

f(z+δz)=f(z)+J(z) δz+O(∥δz∥2).f(z+\delta z)=f(z)+J(z)\,\delta z+O(\|\delta z\|^2).

Such approximation does not hold meaningful outcomes in general, as shown in Table 13 on SONIC encoder for speed, turn, or arm spread. Even if we can find special directions, those obtained this way hold near one sample and for small steps. While due to our construction, our predictor needs no approximation, with no remainder. This is a property of the representation structure, and it is what the earlier Jacobian analyses lack.

Additionally, Prop 3.1 concerns the predicted motion, which is what the skill commands the controller to produce. Our factorization target is this commanded motion, not DiffSR's one-step transition.

Furthermore, in Table 13, only on simple right arm raise (change in a single joint), SONIC with PCA has small advantage, with merely 5% success rate with only 55 trials, while all other perturbation almost yield no success at all. Similarly, SONIC with ridge directions (approximation with the mentioned Jacobian analysis), SONIC only yields meaningful effect on turning, and completely fail on all other directions. On the other side, our construction outperforms by a large margin on average, and such performance is general for all test perturbation directions. For convenience, this is table 13. Note that the second row shows that even if we only compute the spectral directions only based on the walking motion, the directions themselves are meaningful, albeit weakly.

Table 13 passes / trialsArm raiseArm spreadSpeedTurn
Ours, target-directed9/129/104/126/11
Ours, spectral-only3/50/51/52/5
SONIC, token PCA4/52/50/50/5
SONIC, ridge0/50/50/54/5

W2.

Insufficient attribution of the main performance improvement

The reference reaches our controller only through the skill, so what the controller knows about the reference is what the skill carries. Table 7 compares predictive and reconstruction encoders, which receive the same reference window, including robot-relative root position, and their trackers train for 2B frames.

Matched encoder inputTracking success ↑Success-only MPJPE-G (mm) ↓Root-offset recovery ↑
Predictive92.85%99.583.3%
Reconstruction, continuous93.53%279.012.8%
Reconstruction, FSQ93.68%238.1—

This shows predictive loss indeed helps lowering global tracking error.

W3.

Limited evidence for a unified control interface and generalization.

We repeated planning on the final 64-D controller with the same data, planner architecture and training budget for both interfaces.

Final 64-D controller, 5 HzSkillsExplicit motion + encoding
MPJPE-L (mm) ↓35.9 ± 0.659.8 ± 1.5
MPJPE-G (mm) ↓175.9 ± 10.5432.3 ± 18.7
Root-threshold success ↑90.9 ± 1.6%73.2 ± 1.0%

We will replace the table with the new results to fairly demonstrate latent encoding indeed outperforms explicit interfaces.

For generalization, we also test forty LAFAN1 clips, absent from BONES training, without retraining.

LAFAN1OursSONIC
MPJPE-L (mm) ↓30.428.9
MPJPE-G (mm) ↓172.7425.6

This shows a clear edge on the global tracking performance of our method, while local tracking is slightly worse than SONIC by a very small margin <2mm<2mm.

Q1.

Compared with DiffSR and existing Jacobian-based latent steering methods, what is the key methodological contribution of Spectral Skills beyond integrating these techniques into humanoid control?

The contribution is learning the conditioning skill presentation of a window of retargeted human motion together with a prediction model whose steering response is exact at a fixed context. Spectral directions are the right singular vectors of stacked Jacobians. Latent PCA takes the eigenvectors of $ \operatorname{Cov}(z) $ instead. That matrix contains no $ J_\xi $: it ranks directions by how much the recorded skills vary along them and says nothing about what a direction does to the motion. PCA directions would be the wrong criterion for steering. For the same leading $ 16 $ spectral directions which Jacobian analysis carries $ 71 $% of the motion response, PCA only carries $ 16 $%, a drastic degradation.

Q2.

Appendix E identifies a difference in reference root-position information available to the controllers. Would the reported 62% reduction in global tracking error remain under a comparison with matched input information?

With matched input the predictive objective lowers MPJPE-G by 58–64% (W2), the same size as the 62% in Table 1.

Q3.

Why do spectral directions alone underperform SONIC's PCA directions on certain tasks in Table 13? Under what conditions does the linear response in latent space translate into reliable composition of executed robot motions?

Table 13 serves as an open-loop ablation, not a close-loop control feedback on the steer-ability of the encoder. In other words, Table 13 only shows qualitatively if any other approaches can have semantically meaningful steering across perturbation direction. SONIC's token PCA is above our method only on the arm raise, 4/5 against 9/12 (80% and 75%), a difference of one run, and it never passes speed or turn. Its ridge directions pass turn (4/5) and 0 of 15 trials elsewhere.

Q4.

Why are the language-conditioned planning experiments conducted with an earlier 256-D controller rather than the final 64-D Spectral Skills model? Does the proposed representation retain its advantages on unseen motions and across different replanning frequencies?

W3 now supplies the final 64-D results, and LAFAN1 evaluation. We will replace Table 3 and add these evaluations.

I appreciate the authors' additional experiments. I still have concerns about the methodological novelty beyond DiffSR and whether the tracking improvement holds against SONIC with matched root-position information. However, I remain uncertain about these concerns and would like to hear other reviewers' perspectives. I therefore maintain my score while lowering my confidence.

We thank the reviewer for the response. The requested root-heading comparison can be found in rebuttal W.2 and table &, which has the same setting for all, the root-heading input. FSQ (how SONIC learns the encoder) has much worse global tracking (238.1mm versus 99.5mm, ours) at matched training budget.

Additionally, we wish to highlight the novelty of this paper. For long, robot learning communities wish to learn how to combine prior human motions but lack an exact characterization of what composition means. Direct joint interpolation between two motions would simply fail. So composition of motion must happen in the latent space. But what property should the latent space possess? How to learn this latent space?

For the first time we provide a sharp and meaningful characterization of what skill composition is - we generate an algebra of the motion skills that allows us to steer, mix, and chain with linear functionals. Skill composition thus becomes predictable and interpretable as operations are linear.

Regarding novelty compared to DiffSR - they are simply solving different problems and we have a meaningful extension in our problem setting, for which otherwise in-applicable. In DiffSR, the transition is the MDP transition kernel, naturally existent for any RL problem. Here we need to come up a formulation completely out of thin air for the offline dataset which does not even have action labels, just sequences of human motion. Even with action label, simply applying DiffSR would not work, as the learned representation is entangled with action and state, and cannot be used for policy conditioning. So the part on how to learn a structured latent representation using diffusion and how to apply that to robot control is completely new.

Finally, in our early efforts, we have extensively experimented with DiffSR and other spectral representations for policy learning. They do not yield performance benefits for low-level policy learning at convergence, neither can they tell how to learn the skill encoder z.

Reviewer_8WitOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Spectral Skills, a latent motion representation for humanoid whole-body control. Instead of reconstructing past motions, the authors learn skill representations by predicting future motions. They also carefully design the predictor to be affine in the skill vector, allowing meaningful motion directions to be extracted through Jacobian-based spectral decomposition. These directions can then be added to existing skills to compose new behaviors without retraining the controller. The authors validate their method on a 29-DoF Unitree G1 humanoid robot in both simulation and real-world experiments. Compared with SONIC, their method reduces global tracking error by approximately 62%. They also demonstrate smooth skill chaining, zero-shot composition of different motion attributes, and improved performance in language-conditioned planning compared with predicting explicit motion trajectories.

Critical strengths

1. Clear and well-motivated method design. I really appreciate the design logic of this work. The authors first identify the properties they want the skill space to have, and then carefully design the network architecture to encourage these properties. In particular, the affine predictor provides a principled way to extract meaningful skill directions for motion composition. I find this connection between the desired representation properties and the actual network design to be the most interesting contribution of the paper.

2. Impressive experimental results and skill composition. The experimental results are convincing, especially the ability to add new motions to an existing skill through Spectral Skill Composition without retraining the controller. The authors also demonstrate substantial improvements in motion tracking compared with SONIC, along with real-world experiments on Unitree G1. These results make the proposed method more compelling and demonstrate its potential for flexible humanoid control.

Critical weaknesses

1. Limited understanding of conflicts between skill directions. The paper shows that combining multiple skill directions can sometimes lead to interference, but does not provide a clear way to identify which directions can be safely combined or how to resolve such conflicts. Since compositionality is one of the main contributions of this work, I think this limitation raises concerns about how reliably the method can be applied to more complex behaviors.

2. Insufficient ablation studies on the key composition mechanism. Although the paper includes extensive experiments, I feel that the contribution of the spectral decomposition itself could be better isolated. For example, comparing spectral directions with PCA directions extracted from the same skill space would help clarify whether the proposed method is necessary for effective skill composition. This would make the central contribution more convincing.

Questions for authors
  1. How much does spectral decomposition actually contribute to skill composition? The paper already includes some comparisons with skill differences and SONIC's PCA directions, but I think a more controlled comparison with PCA directions extracted from the same skill space would be helpful. Could the authors further clarify what makes spectral directions more effective than simpler alternatives? This would help me better assess the novelty and necessity of the proposed design.

  2. How can we determine which skill directions can be combined without interference? The paper shows that combining multiple directions can sometimes significantly weaken their intended effects. I wonder whether the authors have observed any patterns in these failures, or whether there are ways to predict which combinations are likely to work. Clarifying this would help me better understand the practical limitations of the method.

2 comments

We thank the reviewer for the insightful comments and here are the responses to the concerns and questions:

W1 & Q2

Limited understanding of conflicts between skill directions

Each direction's effect on a base motion predicts whether a pair will work. We run each direction alone on the base motion, and accept a pair if the two single effects are large enough. We found that directions that act on different body parts combine almost always, and interference concentrates in pairs that both load the legs, for example raised foot clearance with a lowered stance.

W2 & Q1

Insufficient ablation studies on the key composition mechanism

Spectral directions are chosen by what they do to the predicted motion. Let J be the Jacobian of the predicted motion with respect to the skill at one context. The spectral directions are the eigenvectors of the response Gram matrix

C=E[J⊤J],v⊤C v=E[∥Jv∥2],C=\mathbb{E}\left[J^\top J\right],\qquad v^\top C\, v=\mathbb{E}\left[\|Jv\|^2\right],

so the leading direction is the unit change of the skill that moves the predicted motion the most, averaged over contexts. Equivalently, they are the right singular vectors of the stacked Jacobians, an uncentered PCA of their rows.

PCA in the same skill space instead takes the eigenvectors of Cov(z). That matrix contains no Jacobian: it ranks directions by how much the recorded skills vary, not by what a direction does to the motion. The two bases differ in practice. The 16 leading PCA directions carry 16% of the predicted motion response (the trace of C), against 71% for the 16 leading spectral directions, and the two subspaces overlap less than two random subspaces.

We compared both bases in the same skill space, with the same controller, calibration and feedback, on a hand-target task during locomotion:

Rank-16 subspaceSpectral directionsSame-space PCA
Combined hand-target error (cm) ↓7.3411.17

What makes these directions usable at all is the affine predictor. Its response to a change of the skill, J times the change, is exact for steps of any size and the same for every base skill (Prop. 3.1), so one set of directions serves the whole skill space.

As the rebuttal deadline approaches, we hope the reviewer has had a chance looking at our new experiments and that the requested experiments addressed their concerns.

Reviewer_uimkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces spectral skills, a predictive latent interface for hierarchical humanoid whole‑body control. Instead of reconstructing observed motion clips (standard auto‑encoder style), the encoder‑decoder is trained to predict future subsequent motion chunks via a diffusion‑based spectral factorization of high‑level transition kernel. The latent skill enters the predictor via an affine mapping, yielding analytically derivable spectral directions: adding these directions to a base skill zero‑shot composes novel whole‑body behaviours without retraining the low‑level controller. The authors validate on Unitree G1 (29‑DoF), showing substantial reduction of global tracking drift compared to SONIC, skill chaining, latent‑space motion composition, language‑conditioned high‑level planning, plus real‑robot hardware deployment. Extensive ablations, baseline comparisons (SONIC, BFM‑Zero, joint offset baselines) and supplementary analysis are provided in the appendix.

Critical strengths
  1. Well‑motivated problem for humanoid foundation‑model control. Hierarchical interface between high‑level planner and low‑level whole‑body controller is a critical open problem; existing latent skill representations mostly rely on reconstruction loss and lack guaranteed compositionality property. This work highlights the limitation of reconstruction‑based latent skill and advocates predictive representation learning for skill interface, which is a valuable conceptual contribution.

  2. Nice theoretical property for composition. Proposition 3.1 shows that under the model’s affine formulation the Jacobian w.r.t skill is independent of base skill z, enabling additive steering via eigen‑derived spectral directions. This gives a principled way to modify one attribute (arm raise, turning, foot clearance) while preserving base locomotion behaviour, rather than heuristic latent interpolation or joint‑space offset.

  3. Strong empirical results on tracking performance. Compared against SONIC (state‑of‑the‑art humanoid tracker), the proposed spectral‑skill controller cuts global MPJPE‑G by ~62%, largely mitigating root drift. Notably it achieves this with ~12× fewer simulation environment training steps. Ablations convincingly demonstrate that predictive future‑chunk objective is key for suppressing global drift, whereas reconstruction‑based latent suffers large world‑frame error even with good local pose matching.

Critical weaknesses
  1. The exact affine response only holds for single‑step denoising estimate, not for full rollout / real‑robot execution. The authors clearly state this caveat, but readers may over‑interpret Proposition 3.1 as global linearity for real robot trajectories. I suggest adding a prominent warning in the main text (not only appendix) emphasizing: the additivity property is a model‑space property; physical rollout exhibits deviation, direction effectiveness varies across base skills, some directions cause falling. Many spectral directions do not work bidirectionally (Fig 10). It would help to quantify degree of additivity/superposition across more base behaviours in main text, not only appendix tables.

  2. Composition evaluation lacks standardized quantitative task metrics. Composition results are mostly qualitative / low‑level kinematic measurements (shoulder angle, heading change). There is no set of downstream task‑level metrics (e.g. reach success, walking while waving navigation task). The paper demonstrates “can raise arm while walking”, but not evaluate whether composed skills reliably solve concrete downstream tasks. It is hard to judge how useful spectral steering is for end‑user task solving beyond motion stylization.

Questions for authors

In Table 7: reconstruction‑based latent obtains competitive local MPJPE‑L but very poor global MPJPE‑G. Could authors elaborate: why reconstruction objective leads to severe root drift even if local joint pose fits well? Is it because reconstruction only sees short segment and does not optimize for long‑term transition consistency?

When computing spectral directions from response Gram matrix C=1N∑Jξ⊤JξC = \frac1N\sum J_\xi^\top J_\xi: is there any normalization / regularisation applied on Jacobian entries before eigen‑decomposition?

Skill chaining shows smoother transition than SONIC, but how much performance gain comes from latent representation versus better training budget / reward design? Are there ablations isolating chaining smoothness factor?

3 comments

We thank the reviewer for the recognition of the contribution of this work and the insightful comments. Here are our responses to the concerns and questions.

W1

The exact affine response only holds for single‑step denoising estimate, not for full rollout / real‑robot execution. The authors clearly state this caveat, but readers may over‑interpret Proposition 3.1 as global linearity for real robot trajectories. I suggest adding a prominent warning in the main text (not only appendix) emphasizing: the additivity property is a model‑space property; physical rollout exhibits deviation, direction effectiveness varies across base skills, some directions cause falling. Many spectral directions do not work bidirectionally (Fig 10). It would help to quantify degree of additivity/superposition across more base behaviours in main text, not only appendix tables.

We agree that the distinction matters. We will make this a separate remark and move the measured additivity into the main text. On bidirectionality, Fig. 10 executes all 64 eigenvectors in both signs on a walk at one amplitude: 30 work in both signs and 26 in one.

W2

Composition evaluation lacks standardized quantitative task metrics. Composition results are mostly qualitative / low‑level kinematic measurements (shoulder angle, heading change). There is no set of downstream task‑level metrics (e.g. reach success, walking while waving navigation task). The paper demonstrates “can raise arm while walking”, but not evaluate whether composed skills reliably solve concrete downstream tasks. It is hard to judge how useful spectral steering is for end‑user task solving beyond motion stylization.

We wish to highlight that this work studies the control interface, not building an autonomous task-solving loco-manipulation full-stack control pipeline. Thus, measuring task completion requires a separate, higher level planner either through VLA, or Agentic framework to command the robot through the designed latent interface. We believe obtaining an interface with nice properties such as compositionality, and compactness presents ample opportunities for task-centric control pipelines, which are ongoing extensions of this work.

Q1

In Table 7: reconstruction‑based latent obtains competitive local MPJPE‑L but very poor global MPJPE‑G. Could authors elaborate: why reconstruction objective leads to severe root drift even if local joint pose fits well? Is it because reconstruction only sees short segment and does not optimize for long‑term transition consistency?

We thank the reviewer for the insightful question. The reviewer's hypothesis matches what we see. The controller learns about the reference only through the skill. A reconstruction skill must preserve its window well enough to reproduce it, and the window's offset from the robot is a small part of that content, so it is weakly encoded. A predictive skill must predict where the motion goes next from where the robot is, so the offset is exactly what it has to carry. We tested this directly by displacing the robot by 10–20 cm from the reference with the same encoder input and training budget:

Recovery from a root offset (800 trials per controller)PredictiveReconstruction
Returns within 5 cm for 0.5 s ↑83.3%12.8%

The predictive controller corrects an offset; the reconstruction controller mostly does not, and its error accumulates as drift.

Q2

When computing spectral directions from response Gram matrix (C = \frac1N\sum J_\xi^\top J_\xi): is there any normalization / regularisation applied on Jacobian entries before eigen‑decomposition?

We convert predictions to motion unit before computing $ J_\xi $, average $ J_\xi^\top J_\xi $ with equal weight over contexts, and take the eigenvectors. We apply no whitening or regularization and do not subtract the mean Jacobian, so a response shared across contexts counts toward the score.

Q3

Skill chaining shows smoother transition than SONIC, but how much performance gain comes from latent representation versus better training budget / reward design? Are there ablations isolating chaining smoothness factor?

We use the highly similar training recipe as SONIC. Additionally, we isolated the representation with more ablations. Two trackers with the same reward, architecture and 2B-frame budget, one with a predictive and one with a reconstruction skill, ran all 42 chaining cases of the paper with three noise seeds:

Matched reward, architecture and budgetPredictiveReconstruction
Joint acceleration at the switch, median RMS (rad/s²) ↓72.4125.0

This shows that switches are 42% smoother with the predictive skill, so the representation itself accounts for the smoothness. We note that this is the desired affect as any linear interpolation of two skills still yields a valid product of experts. The skill enters the model only through ϕ(X,z)=F(X)⊤(Az+b)\phi(X,z)=F(X)^\top(Az+b), which is affine in zz. For zλ=(1−λ)zA+λzBz_\lambda=(1-\lambda)z_A+\lambda z_B the weights sum to one, so the offset bb is preserved:

ϕ(X,zλ)=(1−λ)ϕ(X,zA)+λϕ(X,zB).\phi(X,z_\lambda)=(1-\lambda) \phi(X,z_A)+\lambda \phi(X,z_B).

Substituting into Eq. 3 gives

P(Y∣X,zλ)∝P(Y∣X,zA)1−λP(Y∣X,zB)λ,P(Y\mid X,z_\lambda) \propto P(Y\mid X,z_A)^{1-\lambda} P(Y\mid X,z_B)^{\lambda},

a normalized product of the two skills' predictive distributions, each tempered by its weight.

As the rebuttal deadline approaches, we hope the reviewer has had a chance looking at our new experiments and that the requested experiments addressed their concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment to all reviewers: why the affine construction is the right one for composition

We thank the reviewers for their feedback. Several questions concern the affine form and its novelty and importance in which the skill enters our model. We want to explain why this construction is natural for composing skills, beyond the exact Jacobian of Prop. 3.1.

Our model is an energy-based model,

P(Y∣X,z)  ∝  exp⁡(⟨ϕ(X,z), μ(Y)⟩) ν(dY),ϕ(X,z)=F(X)⊤(Az+b).P(Y\mid X,z)\;\propto\;\exp\big(\langle \phi(X,z),\,\mu(Y)\rangle\big)\,\nu(dY),\qquad \phi(X,z)=F(X)^\top(Az+b).

The skill zz is an affine function of the parameter of this distribution. Combining skills linearly therefore combines their predicted motion distributions as a product of experts. Mixing two skills, zλ=(1−λ)zA+λzBz_\lambda=(1-\lambda)z_A+\lambda z_B, gives

P(Y∣X,zλ)∝P(Y∣X,zA)1−λP(Y∣X,zB)λ,P(Y\mid X,z_\lambda) \propto P(Y\mid X,z_A)^{1-\lambda} P(Y\mid X,z_B)^{\lambda},

and steering a base skill by a difference of skills gives

P(Y∣X,zA+λ(zB−zC))∝P(Y∣X,zA)(P(Y∣X,zB)P(Y∣X,zC))λ.P(Y\mid X,z_A+\lambda(z_B-z_C)) \propto P(Y\mid X,z_A)\left(\frac{P(Y\mid X,z_B)}{P(Y\mid X,z_C)}\right)^{\lambda}.

So every point of the skill space between or beyond the training skills is a valid, interpretable motion distribution. Such property does not exist without using the EBM model or the affine form. Composition by multiplying experts is a long-standing idea (Hinton, 2002), and energy-based and diffusion models use it by summing the scores of separate models during sampling (Du et al., 2020; Liu et al., 2022) but these methods cannot control the summed score having specific meanings. Here it happens inside one learned skill space, by adding vectors, which is exactly the operation a planner can output. In a skill space with a nonlinear decoder, such as a reconstruction latent or SONIC's tokens, the average of two skills has no such meaning.

In conclusion, the product-of-experts identity is a property of the predictive model, as characterized sharply by Prop. 3.1. We will add this interpretation to Sec. 3.4.

References:

  1. G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771–1800, 2002
  2. Y. Du, S. Li, I. Mordatch. Compositional visual generation with energy based models. NeurIPS 33:6637–6647, 2020.
  3. N. Liu, S. Li, Y. Du, A. Torralba, J. B. Tenenbaum. Compositional visual generation with composable diffusion models. ECCV, pp. 423–439, 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.