acceptodds
Under review as a conference paper at ICLR 2027

HetSkills: Reusable Heterogeneous Skill Learning for Physics-Based Control

Abstract

We propose HetSkills, a versatile and extensible framework for physics-based character control that supports modular acquisition and composition of heterogeneous skills without retraining learned skills or the low-level controller. The key idea is to preserve learned skills as executable capabilities and further reuse them as control foundations, expert teachers, or behavioral priors for acquiring new skills. HetSkills begins by learning a tracking skill that establishes a strong foundation in motion control and creates a shared motion decoder, which can be reused across tasks without the need for retraining or separate controllers. To prevent the text-to-motion skill from exploiting shortcut pathways instead of learning language semantics, we introduce motion intuition distillation to ground text-to-motion generation in language semantics and a task-guidance module that dynamically adjusts actions based on high-level language instructions. This enables HetSkills to preserve natural motion while continuously expanding its skill repertoire, making it highly adaptable for long-horizon tasks. Experimental results demonstrate the effectiveness in motion tracking, text-to-motion generation, motion completion, and downstream task adaptation, achieving impressive success rates even under challenging conditions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Physics-based Character Control, Heterogeneous Skill Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gMTQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors focus on the problem of synthesizing long-horizon behaviors for physically simulated characters. To minimize the need for training controllers from scratch, they introduce HetSkills, a mixture-of-experts framework that treats skills obtained from heterogeneous sources, including expert demonstrations, task-oriented controllers, and language-conditioned behaviors, as modular building blocks that can be combined to execute sequential tasks. The framework is evaluated on motion tracking, motion in-betweening, skill composition, and text-to-motion generation tasks.

(+) Modular composition of heterogeneous skills from disparate sources (+) Array of results (-) Issues with motivation and positioning with respect to prior work (-) Questionable algorithmic novelty (-) Insufficient experimental validation

Critical strengths

The paper is well written and easy to follow. I believe there is significant potential in combining and reusing skills obtained from heterogeneous sources. I also appreciate the breadth of experimental results and the authors' commitment to reproducibility through code release. However, the curremt exposition and experimental validation of the proposed work make it difficult for me to assess the novelty and significance of the contribution more highly (see next).

Critical weaknesses

I found the motivation and positioning of HetSkills to be insufficiently developed, the algorithmic novelty somewhat unclear, and the experimental validation requiring additional analysis and experiments:

  • After reading the paper, HetSkills comes across as a general framework for motion tracking, motion in-betweening, and text-to-motion synthesis, with a particular emphasis on long-horizon skill composition. However, these problems have already received substantial attention in the literature. To me, the more distinctive aspect of the work is the ability to compose and reuse skills acquired from heterogeneous sources that extend beyond traditional expert data. As such, I would have like the paper to motivate and present HetSkills primarily as a framework for heterogeneous skill integration where skills rather than as another general skill-composition architecture.
  • Regarding novelty, the core architecture of HetSkills appears to combine elements from PMP and PLT (Bae et al.) with a mixture-of-experts formulation and RSI-based training. Consequently, it is not entirely clear what the primary algorithmic contribution of HetSkills is beyond integrating existing components into a unified framework.
  • More broadly, skill composition through MoE architectures has been extensively studied. Examples include ScaDiver, Multiplicative Compositional Policies, and related work in robotics on modular neural networks, meta-learning, and skill composition. Similarly, approaches such as UniCon have explored unified interfaces for composing physics-based motion skills. As such, its particularly important to clearly articulate why heterogeneous skill sources introduce new challenges and why existing composition frameworks are insufficient for addressing them
  • The experimental evaluation likewise emphasizes motion tracking, motion in-betweening, and text-to-motion synthesis, all of which are well-established benchmarks. In contrast, experiments specifically demonstrating the benefits of heterogeneous skill composition are relatively limited (Figure 3 and Appendix A.3). Since heterogeneous skill integration appears to be the paper's key distinguishing feature, I would have expected more experiments and relate analysis. For example
    • Can skills derived from VR tracking be combined with text-to-motion?
    • What happens when the terminal state of the current skill is incompatible with the initial state distribution of the next skill?
    • Under what conditions can two skills be composed successfully? Are there theoretical guarantees of successful skill stitching?
    • What is the typical length of the composed skill sequences in long-horizon experiments?

The significance of the contribution would have been strengthened if the authors had focused on these types of experiments and analysis.

  • Finally, the authors mention that HetSkills allows adaptation to novel tasks, but this is not demonstrated. Evaluations involving genuinely OOD behaviors, such as interacting with unseen objects or performing novel combinations of skills (e.g., sitting on a table or kicking a chair), would strengthen these claims.
Questions for authors

In Table 1, what body-parts are used for the 5-part decomposition?

What occurs when the terminal state of one skill is incompatible with the initial state distribution of the subsequent skill?

What is the average number of skills in the composed sequences reported in the long-horizon experiments?

Could provide examples that go beyond the skill distributions seen during training and discuss the corresponding success rates?

Reviewer_JcWfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a modular framework for physics-based character control that acquires, reuses, and composes heterogeneous skills. The system first trains an arm and body partitioned tracking skills with reinforcement learning, and freezes the state encoder and part-wise decoders that serve as a shared execution foundation. Then, text-to-motion generation and motion completion tasks are obtained through behavior cloning from the tracker. Downstream tasks are solved without task-specific demonstration by freezing text-to-motion module and traning MoE gating networks and residual policies over a set of fixed candidate language prompts. Finally, learned skills are given a standardized interface, such as initialization, condition, and terminate, and LLM-based planner plans the sequence of the skills for long-horizon behaviors. Motion Intuition Distillation with Randomized Momory Initialization was proposed to train the policy to follow text commands more closely.

Critical strengths
  • The proposed RMI mechanism effectively decouples the initialization context from the target semantic instruction, so it prevents the language-conditioned physics-based controller from overfitting to preceding motion history. The experiments show a significant gain in text-to-motion generation from neutral-pose initialization and more reliable switching across conflicting action categories.
Critical weaknesses
  • Algorithmic novelty is rather incremental. Important elements, such as tracking-distilled teacher-student paradigm and part-wise partitioning, have been introduced in prior works. Text-conditioned adaptation follows a standard MoE mechanism coupled with residual policy. I do appreciate that the resulting pipeline is functioning well, but could not identify specific novely that has been well validated.

  • The standardized task description is an intuitive abstraction for sequence planning, but it is somewhat rudimentary from the standpoint of action planning. The paper does not sufficiently explore whether this deterministic, rule-based interface is optimal. Additionally, it is not clear whether it can deal with failure recovery.

  • In summary, among the three primary contributions highlighted by the authors (lines 083–091), the standardized interface and task adaptation mechanisms are somewhat weak, whereas Motion Intuition Distillation offers significant technical value.

Questions for authors

This may sound like a future work, but it would help if the paper provided systematic validation regarding the operational boundaries of the prior reuse scheme. I wonder how far this prior reusing can scale when applied to more challenging tasks that require coordination between body parts, such as contact-rich locomotion and parkour.

Reviewer_xFV4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

HetSkills learns a shared physics-based motion controller and then trains separate modules for text, sparse motion constraints, scene interactions, and downstream tasks. Randomized Memory Initialization (RMI) trains text control from physical states and motion histories taken from another clip. Learned text-conditioned motions also provide candidates for task-specific reinforcement learning. Evaluation covers tracking, text control, downstream learning, and scripted skill composition.

Critical strengths

The paper identifies a practical weakness in text control: a matching starting pose and history can let the model continue the motion without relying much on the instruction. RMI directly addresses this. In the controlled history test, accuracy with conflicting histories improves from 82.33% without RMI to 92.50%, while results with matching histories remain similar. Neutral-pose success also improves from 70.0% to 81.7%. The agreement between the motivation and these ablations is the main reason I favor acceptance.

Critical weaknesses

Major

  1. RMI changes both the initial physical state and the history. The ablation therefore does not separate better recovery from unfamiliar states from stronger use of language. Comparisons with state-only randomization, history-only randomization, and dropping history would clarify the contribution. Adding the same initialization to a close baseline such as MaskedMimic would also help establish its general value.

  2. The evidence for broad skill reuse is narrower than the framing suggests. Each downstream task still trains its own guidance module, and the six-skill demonstration follows an explicitly specified sequence. Its 99.5% success rate supports reliable execution of that recipe, but provides limited evidence for unfamiliar skill orders or larger-scale skill accumulation.

Minor

Text conditioning and semantic evaluation both use TMR, so an independent semantic assessment would strengthen the results. Reference-based success thresholds should also be distinguished from semantic correctness and motion quality.

Questions for authors
  1. Clarification: How much of the RMI gain comes from changing the physical starting state, and how much comes from changing the history? Are history removal or simpler random-initialization controls available?
  2. Evidence affecting the assessment: Does RMI improve a close text-control baseline such as MaskedMimic under the same training conditions?
  3. Clarification: Which composition settings were held out during development, and how does performance change with unfamiliar skill orders and transition states?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the Area Chair and all reviewers for their time and constructive feedback. We appreciate their recognition of the potential to reuse skills learned from heterogeneous sources, the breadth of the evaluation, and the evidence that Randomized Memory Initialization (RMI) improves instruction following when motion history conflicts with the text.

  • Heterogeneous skill reuse. Sections 4.2–4.3 explain how the tracker supervises text-to-motion learning and how the frozen text-to-motion skill supplies motion priors for downstream controllers trained with task rewards alone. Figure 4 shows that the policy with language guidance reaches a higher pick-up reward, and Section 5.3 and Figure 6 show how the arm branch shifts from lifting to carrying while the body branch draws on walking and standing commands.
  • RMI and evaluation. Section 5.2 distinguishes semantic retrieval from adherence to a paired reference motion. Table 3 reports that RMI raises neutral-pose reference-tracking success from 70.0% to 81.7% at the 0.5 m threshold, and Table 5 reports that it raises six-class accuracy with conflicting histories from 82.33% to 92.50% when both variants start from identical physical states and histories.
  • Skill switching and evaluation scope. Section 5.3 and Figure 5 show how a large language model organizes skill calls and how the program executes them using explicit switching conditions while passing the simulated state between skills. The six-skill program completes in 99.5% of 1,000 runs, and the two-box program completes in 99.1% of 1,000 runs, with interference between the boxes accounting for most failures after the first placement (Appendix A.3). Section 6 identifies larger skill repertoires and more complex long-horizon compositions as directions for further evaluation.

For details of all the changes, please refer to the latest PDF file.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.