- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors focus on the problem of synthesizing long-horizon behaviors for physically simulated characters. To minimize the need for training controllers from scratch, they introduce HetSkills, a mixture-of-experts framework that treats skills obtained from heterogeneous sources, including expert demonstrations, task-oriented controllers, and language-conditioned behaviors, as modular building blocks that can be combined to execute sequential tasks. The framework is evaluated on motion tracking, motion in-betweening, skill composition, and text-to-motion generation tasks.
(+) Modular composition of heterogeneous skills from disparate sources (+) Array of results (-) Issues with motivation and positioning with respect to prior work (-) Questionable algorithmic novelty (-) Insufficient experimental validation
Critical strengths
The paper is well written and easy to follow. I believe there is significant potential in combining and reusing skills obtained from heterogeneous sources. I also appreciate the breadth of experimental results and the authors' commitment to reproducibility through code release. However, the curremt exposition and experimental validation of the proposed work make it difficult for me to assess the novelty and significance of the contribution more highly (see next).
Critical weaknesses
I found the motivation and positioning of HetSkills to be insufficiently developed, the algorithmic novelty somewhat unclear, and the experimental validation requiring additional analysis and experiments:
- After reading the paper, HetSkills comes across as a general framework for motion tracking, motion in-betweening, and text-to-motion synthesis, with a particular emphasis on long-horizon skill composition. However, these problems have already received substantial attention in the literature. To me, the more distinctive aspect of the work is the ability to compose and reuse skills acquired from heterogeneous sources that extend beyond traditional expert data. As such, I would have like the paper to motivate and present HetSkills primarily as a framework for heterogeneous skill integration where skills rather than as another general skill-composition architecture.
- Regarding novelty, the core architecture of HetSkills appears to combine elements from PMP and PLT (Bae et al.) with a mixture-of-experts formulation and RSI-based training. Consequently, it is not entirely clear what the primary algorithmic contribution of HetSkills is beyond integrating existing components into a unified framework.
- More broadly, skill composition through MoE architectures has been extensively studied. Examples include ScaDiver, Multiplicative Compositional Policies, and related work in robotics on modular neural networks, meta-learning, and skill composition. Similarly, approaches such as UniCon have explored unified interfaces for composing physics-based motion skills. As such, its particularly important to clearly articulate why heterogeneous skill sources introduce new challenges and why existing composition frameworks are insufficient for addressing them
- The experimental evaluation likewise emphasizes motion tracking, motion in-betweening, and text-to-motion synthesis, all of which are well-established benchmarks. In contrast, experiments specifically demonstrating the benefits of heterogeneous skill composition are relatively limited (Figure 3 and Appendix A.3). Since heterogeneous skill integration appears to be the paper's key distinguishing feature, I would have expected more experiments and relate analysis. For example
- Can skills derived from VR tracking be combined with text-to-motion?
- What happens when the terminal state of the current skill is incompatible with the initial state distribution of the next skill?
- Under what conditions can two skills be composed successfully? Are there theoretical guarantees of successful skill stitching?
- What is the typical length of the composed skill sequences in long-horizon experiments?
The significance of the contribution would have been strengthened if the authors had focused on these types of experiments and analysis.
- Finally, the authors mention that HetSkills allows adaptation to novel tasks, but this is not demonstrated. Evaluations involving genuinely OOD behaviors, such as interacting with unseen objects or performing novel combinations of skills (e.g., sitting on a table or kicking a chair), would strengthen these claims.
Questions for authors
In Table 1, what body-parts are used for the 5-part decomposition?
What occurs when the terminal state of one skill is incompatible with the initial state distribution of the subsequent skill?
What is the average number of skills in the composed sequences reported in the long-horizon experiments?
Could provide examples that go beyond the skill distributions seen during training and discuss the corresponding success rates?
General Comment
We thank the Area Chair and all reviewers for their time and constructive feedback. We appreciate their recognition of the potential to reuse skills learned from heterogeneous sources, the breadth of the evaluation, and the evidence that Randomized Memory Initialization (RMI) improves instruction following when motion history conflicts with the text.
For details of all the changes, please refer to the latest PDF file.