- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Game-Guided Skill Discovery (GGSD), which uses competitive self-play to learn a small set of discrete skills for human control. A PPO-trained hierarchical policy combines game rewards with a MI objective, while opponents are sampled from past checkpoints. After training, humans replace the high-level policy with keyboard input. Experiments on AntSumo, AntFencing, FrankaAirHockey, and G1Boxing show interpretable skills, transition-based combos, higher VLM-measured semantic diversity, and human-controlled transfer to unseen maze and cube-pushing tasks.
Critical strengths
- Competitive self-play is a novel source of guidance for skill discovery. Game outcomes offer an alternative to language or motion demonstrations, and the opponent pool could provide an curriculum for discovering useful behaviors.
Critical weaknesses
-
Unclear practical value. The paper does not clearly establish what problem human-playable skills solve or why they are useful. For example, such skills could simplify teleoperation of high-DoF robots, but the paper neither identifies a concrete use case nor evaluates the benefits in that setting. Showing that humans can compose the skills to navigate a maze or push a cube demonstrates feasibility, but does not explain what practical advantage this provides. A clearer problem statement and an evaluation tied to that problem are needed to establish the method's utility.
-
Game guidance relies on substantial reward engineering. The claim that game rules require only a few lines of code understates the reward design in the appendix. AntSumo uses approach and push rewards, FrankaAirHockey uses dense puch-progress rewards, and G1Boxing includes eleven terms covering punching, facing, approach, and regularization. Since the low-level policy receives the full game reward (Eq. 11), walking and punching may largely reflect explicit reward shaping. This obscures the distinction between gamed-guided discovery and task-reward training with and additional MI objective.
-
Comparisons do not isolate the proposed mechanism. There is no indication that DIAYN, DADS, or METRA receive task-related rewards comparable to GGSD’s dense game rewards, making Figure 6 a guided approach vs. unguided approach comparison. The gains could therefore come from task-reward guidance rather than competitive self-play. No ablations are provided.
-
Limited novelty of the learned skills and their composition. Prior methods [1, 2] already learn interpretable, composable discrete skills, although they learn from demonstrations. While GGSD uses a different source of guidance, the reported behaviors (turning, locomotion, pushing, punching etc) appear fairly basic and do not seem particularly interesting. Prior work [1] has also shown that humans can compose learned skills to solve downstream tasks. It is therefore unclear how the proposed "combos" differ from ordinary sequential skill execution. Concrete examples or qualitative analysis are needed to establish whether they provide capabilities beyond simply chaining skills. [1] Garg et al., LISA: Learning Interpretable Skill Abstractions from Language (NeurIPS 2022) [2] Jiang et al., Discrete Latent Plans via Semantic Skill Abstractions (ICLR 2025)
-
The approach may further limit fine-grained skill discovery. Discovering fine-grained skills is an important challenge in unsupervised skill discovery. GGSD appears to move away from this goal by restricting the skill set to only five or six discrete behaviors, favoring coarse primitives over skills with subtle variations in speed, direction, or motion. A study varying the number of skills would help clarify whether the approach can discover finer-grained behaviors while maintaining interpretability.
Questions for authors
Please see the weaknesses above.
6 comments
[W3] Isolating Task-Reward Guidance and Competitive Self-Play
Yes, the comparison in Figure 6 was intentionally designed to evaluate the overall benefit of game guidance over conventional unsupervised skill discovery. We agree that it does not isolate the contributions of reward guidance and competitive self-play. We added self-play ablations in Section 5.6 (Ablation Studies), which evaluate the contribution of self-play while preserving the original game-derived rewards, as detailed in [W2].
We also want to clarify why naively incorporating game-derived rewards into existing skill discovery methods such as DIAYN, DADS, or METRA is not straightforward, and how GGSD addresses these challenges.
First, a hierarchical policy design is important for learning transferable skills. Simply adding game-derived rewards to a flat skill-conditioned policy may require game-specific observations, such as the opponent's position, creating potential out-of-distribution issues when transferring to tasks where the opponent is absent. GGSD addresses this by separating game-specific observations, handled by the high-level policy, from those used by the low-level skill policy. This enables transfer to tasks such as maze navigation and cube pushing without requiring the original game or opponent.
Second, competitive self-play is important for discovering diverse skills. Adding game-derived rewards alone does not necessarily encourage diverse behaviors if the agent encounters limited challenges. As shown in our ablations discussed in [W2], removing self-play while retaining game-derived rewards leads to substantially less diverse trajectories in AntSumo and FrankaAirHockey. In contrast, self-play continuously generates adaptive opponents that encourage diverse strategies.
Together, these considerations highlight that GGSD's framework design is not arbitrary, but is motivated by the challenges of integrating game guidance into skill discovery.
[W4] Novelty of Discovered Skills and Their Composition
We thank the reviewer for this comment, and we kindly ask the reviewer to watch the supplementary video. From 0:03 to 0:18 it shows the skills learned by pure METRA without guidance. They are mostly unstructured motions that move every joint in arbitrary directions. GGSD’s skills keep only the motions that help win the game. Turning, locomotion, pushing and punching may look basic in hindsight, but they emerge without any demonstrations or hand-designed skill definitions, and the contrast with pure METRA shows that getting them is not trivial.
Our work is fundamentally about discovery. Before training, we do not know which skills will emerge, and different seeds produce different skills. In AntSumo, for example, an agent can push its opponent with its torso or with its front legs. Pushing with the front legs looks natural once you see it, but nothing in our setup requires it. In some seeds that skill does not appear and the agent pushes with its torso instead. Data-driven methods solve a different problem. Methods like LISA [1] and [2], which learn from demonstrations, will obviously produce higher-quality motion, and plain imitation alone would already do so. However, they can only reproduce or recombine what is in the data. Recent works such as DSD (Kim et al., 2026) and RGSD (Rho et al., 2026) first imitate reference data and then move on to discovering new motions. That is a different setting from ours, which assumes no data.
Our approach is also orthogonal to data-driven methods. Following ASE (Peng et al.), one could add a GAIL-based imitation loss to GGSD to improve motion quality. That combination is out of scope for this paper.
LISA [1] in particular has two limitations that GGSD addresses. First, LISA is trained to imitate expert trajectories, so the skills it can discover are bounded by the demonstrations. It can recombine learned primitives, but it has no exploration mechanism for finding new strategies or behaviors that are absent from the data. Second, LISA does not explicitly ensure that each skill code means one consistent thing. Its VQ codebook encourages discrete skills, but nothing forces each code to match a single clear subtask. One code may cover several related behaviors, or one subtask may be split across several codes. This makes LISA’s skills hard to be used as “human playable” controls, which is what lets a human compose them.
Finally, combos are not simple sequential chaining. We kindly refer the reviewer to 2:42–3:00 of the video. It first shows the six individual G1 skills and then a combo. The combo produces a hard punch that uses the whole body, a motion that does not appear in any of the six individual skills. So composing skills in GGSD produces new behavior beyond running the learned skills one after another. We will add this qualitative analysis to the revised paper.
[W5] Fine-Grained Skill Discovery and the Effect of Skill Set Size
First, we clarify that having only five discrete skills does not necessarily restrict the agent to five distinct behaviors. For example, in FrankaAirHockey, the five learned skills move the end-effector toward different regions of the table. Nevertheless, as shown in our supplementary video (1:25–1:40), the agent can effectively play across the entire table because transitions between skills generate additional behaviors that cover intermediate regions.
To further investigate the reviewer's concern, we conducted additional experiments with 10 and 15 skills on both AntSumo and FrankaAirHockey, and incorporated the results into Section 5.6 (Ablation Studies). As shown in Figure 8, GGSD continues to discover distinct skills with larger skill sets. In FrankaAirHockey, the learned skills cover increasingly fine-grained regions of the table, while AntSumo also exhibits meaningful behavioral diversity.
These results support GGSD's ability to discover finer-grained skills while maintaining behavioral diversity. Our default choice of five skills was primarily motivated by human playability, as controlling an agent with more than ten discrete inputs can become cumbersome, rather than by an inherent limitation of our framework.
We thank the reviewer for their valuable comments. We address each point individually below.
[W1] Practical Value and Potential Applications of GGSD
We argue that practicality is precisely where GGSD makes its strongest contribution to the field of skill discovery.
Seminal works such as DIAYN, DADS, METRA, and CIC [1] aim to discover diverse, reusable skills without external task rewards. While these methods have made substantial contributions, discovering semantically meaningful and practically useful skills remains challenging, particularly for high-DoF agents. For example, 0:04–0:20 of our supplementary video shows that METRA, when applied to a humanoid agent without additional guidance, produces largely unstructured motions that do not even maintain stable standing. This illustrates a fundamental limitation: distinguishable skills are not necessarily meaningful or useful.
This challenge has motivated recent research on guided skill discovery, such as LGSD [2] and DoDon't [3], which leverage LLMs or VLMs to encourage meaningful behaviors. GGSD takes a distinct approach by using competitive self-play as a source of guidance, without relying on external foundation models or demonstrations. Through this mechanism, GGSD discovers interpretable skills even for high-DoF humanoid agents, while making them directly controllable by human players. To the best of our knowledge, GGSD is the first skill discovery framework to explicitly demonstrate such human-playable skills, even working on humanoid.
Importantly, these skills also demonstrate practical utility beyond the original games. For example, skills discovered in G1Boxing exhibit stable locomotion and can be reused for downstream tasks such as maze navigation and cube pushing, without task-specific skill training. In this sense, we believe GGSD represents a meaningful step toward practical skill discovery by producing skills that are interpretable, human-controllable, and transferable to downstream tasks.
[1] Laskin et al., CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery, 2022
[2] Rho et al., Language Guided Skill Discovery, 2025
[3] Kim et al., Do's and Don'ts: Learning Desirable Skills with Instruction Videos, 2025
[W2] Game Guidance, Reward Engineering, and the Role of Competitive Self-Play
Yes, we agree that game-derived rewards play an important role in guiding skill learning, and acknowledge that our original submission did not sufficiently disentangle the contributions of reward guidance and competitive self-play. To address this concern, we conducted additional ablation experiments on both AntSumo and FrankaAirHockey, removing self-play while preserving the original reward functions. These results have been incorporated into Section 5.6 (Ablation Studies).
Our central argument is that competitive self-play creates diverse and increasingly challenging situations under a fixed set of game rules, thereby encouraging the discovery of diverse skills. While reward shaping provides the underlying learning signal, self-play continuously exposes the agent to adaptive opponents that demand different strategies. Without such opponents, the same rewards may be satisfied through a narrow set of behaviors.
Our new ablation results support this distinction. Figure 8 in the revised manuscript visualizes 30 trajectories per skill from identical initial states in both environments.
These results provide evidence that game-derived rewards alone are insufficient to explain the diversity of skills discovered by GGSD. Competitive self-play contributes by creating diverse challenges that cannot be addressed through a single reward-maximizing behavior, distinguishing GGSD from conventional reward-guided skill learning with an additional MI objective.