acceptodds
Under review as a conference paper at ICLR 2027

Game-Guided Skill Discovery through Self-Play for Playable Agent Control

Abstract

We present Game-Guided Skill Discovery (GGSD), a framework that uses self- play in games to discover motor skills that are directly playable by humans. Playable skills provide a compact abstraction for controlling embodied agents through a small set of learned behaviors rather than low-level actions. To be effective, these skills should be semantically distinct, interpretable, and expres- sive—properties that existing unsupervised skill-discovery methods often fail to achieve simultaneously. GGSD achieves these desiderata by grounding skill dis- covery in competitive gameplay. A hierarchical agent competes against its past selves, with a high-level policy selecting from a small discrete skill set and a skill- conditioned low-level policy learning the corresponding behaviors. After training, a human can replace the high-level policy and directly control the agent through the same discrete skills. Despite the small number of high-level actions, skill transitions give rise to emergent combo behaviors, expanding expressivity beyond individual primitives. Across Ant, Franka-arm, and Unitree G1 environments, we show that GGSD produces human-playable skills that humans can compose to solve unseen tasks, such as Maze and CubePush, without additional training. An interactive demo is available at https://ggsd-demo.github.io

Primary area:
reinforcement learning
Keywords:
Skill Discovery, Self-Play, Learning in Games, Reinforcement Learning, Unsupervised Reinforcement Learning, Robot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eLYgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Game-Guided Skill Discovery (GGSD), which uses competitive self-play to learn a small set of discrete skills for human control. A PPO-trained hierarchical policy combines game rewards with a MI objective, while opponents are sampled from past checkpoints. After training, humans replace the high-level policy with keyboard input. Experiments on AntSumo, AntFencing, FrankaAirHockey, and G1Boxing show interpretable skills, transition-based combos, higher VLM-measured semantic diversity, and human-controlled transfer to unseen maze and cube-pushing tasks.

Critical strengths
  • Competitive self-play is a novel source of guidance for skill discovery. Game outcomes offer an alternative to language or motion demonstrations, and the opponent pool could provide an curriculum for discovering useful behaviors.
Critical weaknesses
  • Unclear practical value. The paper does not clearly establish what problem human-playable skills solve or why they are useful. For example, such skills could simplify teleoperation of high-DoF robots, but the paper neither identifies a concrete use case nor evaluates the benefits in that setting. Showing that humans can compose the skills to navigate a maze or push a cube demonstrates feasibility, but does not explain what practical advantage this provides. A clearer problem statement and an evaluation tied to that problem are needed to establish the method's utility.

  • Game guidance relies on substantial reward engineering. The claim that game rules require only a few lines of code understates the reward design in the appendix. AntSumo uses approach and push rewards, FrankaAirHockey uses dense puch-progress rewards, and G1Boxing includes eleven terms covering punching, facing, approach, and regularization. Since the low-level policy receives the full game reward (Eq. 11), walking and punching may largely reflect explicit reward shaping. This obscures the distinction between gamed-guided discovery and task-reward training with and additional MI objective.

  • Comparisons do not isolate the proposed mechanism. There is no indication that DIAYN, DADS, or METRA receive task-related rewards comparable to GGSD’s dense game rewards, making Figure 6 a guided approach vs. unguided approach comparison. The gains could therefore come from task-reward guidance rather than competitive self-play. No ablations are provided.

  • Limited novelty of the learned skills and their composition. Prior methods [1, 2] already learn interpretable, composable discrete skills, although they learn from demonstrations. While GGSD uses a different source of guidance, the reported behaviors (turning, locomotion, pushing, punching etc) appear fairly basic and do not seem particularly interesting. Prior work [1] has also shown that humans can compose learned skills to solve downstream tasks. It is therefore unclear how the proposed "combos" differ from ordinary sequential skill execution. Concrete examples or qualitative analysis are needed to establish whether they provide capabilities beyond simply chaining skills. [1] Garg et al., LISA: Learning Interpretable Skill Abstractions from Language (NeurIPS 2022) [2] Jiang et al., Discrete Latent Plans via Semantic Skill Abstractions (ICLR 2025)

  • The approach may further limit fine-grained skill discovery. Discovering fine-grained skills is an important challenge in unsupervised skill discovery. GGSD appears to move away from this goal by restricting the skill set to only five or six discrete behaviors, favoring coarse primitives over skills with subtle variations in speed, direction, or motion. A study varying the number of skills would help clarify whether the approach can discover finer-grained behaviors while maintaining interpretability.

Questions for authors

Please see the weaknesses above.

6 comments

We thank the reviewer for their valuable comments. We address each point individually below.

[W1] Practical Value and Potential Applications of GGSD

We argue that practicality is precisely where GGSD makes its strongest contribution to the field of skill discovery.

Seminal works such as DIAYN, DADS, METRA, and CIC [1] aim to discover diverse, reusable skills without external task rewards. While these methods have made substantial contributions, discovering semantically meaningful and practically useful skills remains challenging, particularly for high-DoF agents. For example, 0:04–0:20 of our supplementary video shows that METRA, when applied to a humanoid agent without additional guidance, produces largely unstructured motions that do not even maintain stable standing. This illustrates a fundamental limitation: distinguishable skills are not necessarily meaningful or useful.

This challenge has motivated recent research on guided skill discovery, such as LGSD [2] and DoDon't [3], which leverage LLMs or VLMs to encourage meaningful behaviors. GGSD takes a distinct approach by using competitive self-play as a source of guidance, without relying on external foundation models or demonstrations. Through this mechanism, GGSD discovers interpretable skills even for high-DoF humanoid agents, while making them directly controllable by human players. To the best of our knowledge, GGSD is the first skill discovery framework to explicitly demonstrate such human-playable skills, even working on humanoid.

Importantly, these skills also demonstrate practical utility beyond the original games. For example, skills discovered in G1Boxing exhibit stable locomotion and can be reused for downstream tasks such as maze navigation and cube pushing, without task-specific skill training. In this sense, we believe GGSD represents a meaningful step toward practical skill discovery by producing skills that are interpretable, human-controllable, and transferable to downstream tasks.

[1] Laskin et al., CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery, 2022

[2] Rho et al., Language Guided Skill Discovery, 2025

[3] Kim et al., Do's and Don'ts: Learning Desirable Skills with Instruction Videos, 2025


[W2] Game Guidance, Reward Engineering, and the Role of Competitive Self-Play

Yes, we agree that game-derived rewards play an important role in guiding skill learning, and acknowledge that our original submission did not sufficiently disentangle the contributions of reward guidance and competitive self-play. To address this concern, we conducted additional ablation experiments on both AntSumo and FrankaAirHockey, removing self-play while preserving the original reward functions. These results have been incorporated into Section 5.6 (Ablation Studies).

Our central argument is that competitive self-play creates diverse and increasingly challenging situations under a fixed set of game rules, thereby encouraging the discovery of diverse skills. While reward shaping provides the underlying learning signal, self-play continuously exposes the agent to adaptive opponents that demand different strategies. Without such opponents, the same rewards may be satisfied through a narrow set of behaviors.

Our new ablation results support this distinction. Figure 8 in the revised manuscript visualizes 30 trajectories per skill from identical initial states in both environments.

  • AntSumo: Without self-play, pushing a stationary cube produces largely overlapping, straight-line trajectories. In contrast, GGSD discovers diverse turning and maneuvering strategies against adaptive opponents.
  • FrankaAirHockey: Without self-play, the agent primarily learns to shoot toward an undefended goal. In contrast, GGSD discovers five distinct skills covering different regions of the table.

These results provide evidence that game-derived rewards alone are insufficient to explain the diversity of skills discovered by GGSD. Competitive self-play contributes by creating diverse challenges that cannot be addressed through a single reward-maximizing behavior, distinguishing GGSD from conventional reward-guided skill learning with an additional MI objective.

[W3] Isolating Task-Reward Guidance and Competitive Self-Play

Yes, the comparison in Figure 6 was intentionally designed to evaluate the overall benefit of game guidance over conventional unsupervised skill discovery. We agree that it does not isolate the contributions of reward guidance and competitive self-play. We added self-play ablations in Section 5.6 (Ablation Studies), which evaluate the contribution of self-play while preserving the original game-derived rewards, as detailed in [W2].

We also want to clarify why naively incorporating game-derived rewards into existing skill discovery methods such as DIAYN, DADS, or METRA is not straightforward, and how GGSD addresses these challenges.

First, a hierarchical policy design is important for learning transferable skills. Simply adding game-derived rewards to a flat skill-conditioned policy may require game-specific observations, such as the opponent's position, creating potential out-of-distribution issues when transferring to tasks where the opponent is absent. GGSD addresses this by separating game-specific observations, handled by the high-level policy, from those used by the low-level skill policy. This enables transfer to tasks such as maze navigation and cube pushing without requiring the original game or opponent.

Second, competitive self-play is important for discovering diverse skills. Adding game-derived rewards alone does not necessarily encourage diverse behaviors if the agent encounters limited challenges. As shown in our ablations discussed in [W2], removing self-play while retaining game-derived rewards leads to substantially less diverse trajectories in AntSumo and FrankaAirHockey. In contrast, self-play continuously generates adaptive opponents that encourage diverse strategies.

Together, these considerations highlight that GGSD's framework design is not arbitrary, but is motivated by the challenges of integrating game guidance into skill discovery.


[W4] Novelty of Discovered Skills and Their Composition

We thank the reviewer for this comment, and we kindly ask the reviewer to watch the supplementary video. From 0:03 to 0:18 it shows the skills learned by pure METRA without guidance. They are mostly unstructured motions that move every joint in arbitrary directions. GGSD’s skills keep only the motions that help win the game. Turning, locomotion, pushing and punching may look basic in hindsight, but they emerge without any demonstrations or hand-designed skill definitions, and the contrast with pure METRA shows that getting them is not trivial.

Our work is fundamentally about discovery. Before training, we do not know which skills will emerge, and different seeds produce different skills. In AntSumo, for example, an agent can push its opponent with its torso or with its front legs. Pushing with the front legs looks natural once you see it, but nothing in our setup requires it. In some seeds that skill does not appear and the agent pushes with its torso instead. Data-driven methods solve a different problem. Methods like LISA [1] and [2], which learn from demonstrations, will obviously produce higher-quality motion, and plain imitation alone would already do so. However, they can only reproduce or recombine what is in the data. Recent works such as DSD (Kim et al., 2026) and RGSD (Rho et al., 2026) first imitate reference data and then move on to discovering new motions. That is a different setting from ours, which assumes no data.

Our approach is also orthogonal to data-driven methods. Following ASE (Peng et al.), one could add a GAIL-based imitation loss to GGSD to improve motion quality. That combination is out of scope for this paper.

LISA [1] in particular has two limitations that GGSD addresses. First, LISA is trained to imitate expert trajectories, so the skills it can discover are bounded by the demonstrations. It can recombine learned primitives, but it has no exploration mechanism for finding new strategies or behaviors that are absent from the data. Second, LISA does not explicitly ensure that each skill code means one consistent thing. Its VQ codebook encourages discrete skills, but nothing forces each code to match a single clear subtask. One code may cover several related behaviors, or one subtask may be split across several codes. This makes LISA’s skills hard to be used as “human playable” controls, which is what lets a human compose them.

Finally, combos are not simple sequential chaining. We kindly refer the reviewer to 2:42–3:00 of the video. It first shows the six individual G1 skills and then a combo. The combo produces a hard punch that uses the whole body, a motion that does not appear in any of the six individual skills. So composing skills in GGSD produces new behavior beyond running the learned skills one after another. We will add this qualitative analysis to the revised paper.

[W5] Fine-Grained Skill Discovery and the Effect of Skill Set Size

First, we clarify that having only five discrete skills does not necessarily restrict the agent to five distinct behaviors. For example, in FrankaAirHockey, the five learned skills move the end-effector toward different regions of the table. Nevertheless, as shown in our supplementary video (1:25–1:40), the agent can effectively play across the entire table because transitions between skills generate additional behaviors that cover intermediate regions.

To further investigate the reviewer's concern, we conducted additional experiments with 10 and 15 skills on both AntSumo and FrankaAirHockey, and incorporated the results into Section 5.6 (Ablation Studies). As shown in Figure 8, GGSD continues to discover distinct skills with larger skill sets. In FrankaAirHockey, the learned skills cover increasingly fine-grained regions of the table, while AntSumo also exhibits meaningful behavioral diversity.

These results support GGSD's ability to discover finer-grained skills while maintaining behavioral diversity. Our default choice of five skills was primarily motivated by human playability, as controlling an agent with more than ten discrete inputs can become cumbersome, rather than by an inherent limitation of our framework.

Reviewer_s8U9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Ground unsupervised skill discovery agents in competitive gameplay against versions of self, using a high-level policy to select from a skill set. Keeps a pool of self-playing agents and then samples them, biased towards the most recent checkpoint, to play against. Optimize with bilevel PPO objectives

Critical strengths

The work suggests an interesting line of work where the environment fitness offers a path for training unsupervised skills. The experiments are meaningful and informative.

Critical weaknesses

Calling the skill discovery "unsupervised" is a bit of a misnomer, since the dynamics of a foe induce an implicit reward, so the argument that unsupervised skill discovery methods are not sufficiently "meaningful" becomes a bit circular: USD methods are not meaningful because they do not have engineered meaning (as these game scenarios do).

When using the skills that can be used by humans in the interface, they turned out to be quite clunky, especially for the ant pushing task, where the go straight skill pushed the block into the wall and got stuck constantly.

At the end of the day, the novelty appears to be limited: the only real change for this high- and low-level system is to incorporate the game reward into the low-level policy, and to some extent the choice of tasks. Everything else has already been used in prior works.

Using past checkpoints as a performance metric is somewhat dubious, and the limited number of samples for an unintuitive human interface seems likely to skew human results. It seems like a more objective measure of performance would be useful.

The human playability of the skills in purely maze environments seems somewhat dubious, since it seems like these environments would also be amenable to normal skill learning, and not specifically the ones from this work. It also is not clear how self-play is supposed to work in maze environments.

Questions for authors

See Weaknesses

2 comments

We thank the reviewer for the thoughtful feedback and for recognizing the potential of using game fitness as a path for skill discovery. During the discussion period we ran several additional experiments, some prompted by other reviewers, that bear directly on the reviewer's concerns. We summarize them below.

[W1] "Unsupervised" as a misnomer and the circularity concern

We agree that GGSD is not purely unsupervised, and we do not position it that way. As discussed in Section 2.2, GGSD belongs to the line of guided skill discovery, alongside methods such as LGSD (Rho et al., 2025) and DoDont (Kim et al., 2025), which use LLMs, VLMs, or instruction videos to steer discovery toward meaningful behaviors. GGSD uses competitive gameplay as its source of guidance instead.

On the circularity concern, the game reward specifies what it means to win, not which behaviors to perform. To test this, we retrained G1Boxing with only 4 of the original 11 reward terms (Damage, Facing, Facing velocity, Approach), removing all regularization and outcome terms (video 5:40–5:50). Diverse skills still emerge, including a two-handed punch that never appeared under the full reward, and semantic distance increases from 0.121 to 0.130. The specific behaviors, such as how to move, punch, or turn, are not encoded in the reward. They are discovered by GGSD.

[W2] Clunkiness of the human interface

We thank the reviewer for trying the interface. The situation the reviewer describes, where the cube gets stuck against the wall under the go-straight skill, is one that a human player can resolve with the discovered skills. Our supplementary video ([X:XX–X:XX]) shows how AntCubePush is completed. When the cube gets stuck against the wall, the player switches to the push skill, which frees the cube and lets the agent continue toward the goal. Getting through AntCubePush therefore depends on choosing the right skill for each situation, much like choosing the right button in a game. Holding a single skill throughout does not work. Once players learn which skill fits which situation, the task becomes manageable.

We also note that these downstream tasks were never seen during training. The skills were learned only in the game environments and applied zero-shot, with a human selecting among five or six discrete skills, so some coarseness in control is expected. To measure how useful the skills are independently of the human interface, we ran the standard downstream RL evaluation (Appendix F). We froze each method's skill-conditioned low-level policy and trained a PPO high-level controller on top of it. On G1CubePush, GGSD reaches a return of 43.89, compared with 6.63 for the strongest baseline, and the baselines fail to solve the task.

[W3] Limited novelty

GGSD is more than adding the game reward to the low-level policy. To make this precise, we decomposed GGSD into four components and ablated them (Section 5.6):

VariantComponents
GGSD(1) game reward + (2) self-play opponent + (3) hierarchical policy with separated observations + (4) MI reward
GGSD w/o self-play(1) + (3) + (4)
GGSD w/o self-play, METRA objective(1) + (3) + (4′)

When self-play is removed and the game reward is kept, the skills collapse into overlapping trajectories in both AntSumo and FrankaAirHockey: the Ant never learns to turn, and the Franka arm fails to cover the table. Replacing MI with the METRA objective does not prevent this collapse (video 5:50–5:55). The diversity of GGSD's skills therefore comes from competitive self-play, which continually presents an evolving opponent and turns the problem into one that a single reward-maximizing behavior cannot solve. Component (3) is also essential, because it keeps opponent-specific observations in the high-level policy and lets the low-level skills transfer to tasks with no opponent at all.

[W4] Need for a more objective measure of performance

We added the downstream RL evaluation described in [W2], which is the standard protocol in skill discovery (e.g., DADS, METRA). Because every method uses the same high-level RL algorithm, performance differences reflect the quality of the discovered skills alone. GGSD achieves more than 6× the return of the strongest baseline on both tasks (6.39 vs. 0.99 on G1GoalReach; 43.89 vs. 6.63 on G1CubePush). This complements the VLM-based semantic diversity metric (Figure 6), where GGSD outperforms DIAYN, DADS, and METRA in all environments. It also complements the human evaluation in Section 5.2, where four human players, whose policies are unseen by construction, won only 2 of 20 games against GGSD agents.

[W5] Human playability in maze environments and the role of self-play

We would like to clarify a possible misunderstanding: no self-play takes place in the maze environments. All skills are learned once in the game environment (e.g., G1Boxing) and transferred to G1-Maze, AntMaze, and AntCubePush without any additional training.

On whether conventional skill learning would suffice, our downstream results suggest that it does not, at least for high-DoF agents. On the same G1 humanoid, the skills learned by DIAYN, DADS, and METRA fail to solve G1GoalReach, a goal-reaching navigation task, even with a trained high-level controller. METRA's G1 skills are essentially unstructured joint motions that do not maintain stable standing (video 0:04–0:20). Competitive gameplay is what produces the stable, coherent locomotion that makes maze navigation possible.

We hope these additional results address the reviewer's concerns, and we would be happy to answer any further questions.

Reviewer_aghAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a method to discover distinct behaviors for a given environment through a self play objective. The agent consists of a high level policy that chooses a discrete skill index, and a low level policy that conditions on it. The high level policy is trained to maximize the game reward, and the low level policy is trained to maximize a game reward and diversity reward. The agent is trained to maximize the game reward against past versions of itself. The method is evaluated in simulated continuous control tasks, such as ants pushing each other (ant sumo) and humanoids hitting each other (boxing). They find their method can discover sensible skills for the tasks.

Critical strengths
  • Idea of using self play in the context of skill discovery for a task is interesting and novel.
  • the tasks are interesting and a step beyond what previous skill discovery papers do, such as humanoid boxing, air hockey, etc. the website is nicely made
Critical weaknesses

This paper is missing a rigorous comparison against baselines that also discover skills. There is no quantitative comparison against baselines with a quantitative metric (e.g. task reward, state diversity, etc.).

  • Typically, skill discovery papers evaluate the quality of their learned skills by training a high level controller to maximize reward in a downstream task, see METRA, DADS, etc.
  • The individual skills are further compared in terms of exploration performance, such as state coverage, etc.
  • The only comparison this paper has against skill discovery baselines is their "semantic diversity" comparison, which uses a VLM as a judge to grade videos of their skills against other methods. While a valid comparison, this comparison's metric is not as concrete as task reward or state diversity.

No discussion of the training costs and tradeoffs of this method against others. Typically, self play methods are notoriously sample inefficient, and prone to collapse. It would be good to discuss this.

Another weakness of self play methods is that they can overfit to their own play-style. It would be nice to have experiments that analyze how well the agent generalizes to other opponents.

Questions for authors

See weaknesses above, but the main concern is lack of quantitative comparisons with other skill discovery baselines.

4 comments

We thank the reviewer for the thoughtful feedback. We address each point below.

[W1] Downstream task evaluation with a high-level controller

Following the reviewer's suggestion, we ran the standard downstream evaluation and added the results in Appendix F. We froze the skill-conditioned low-level policy learned by each method (GGSD in G1Boxing, DIAYN, DADS, and METRA) and trained a high-level policy with PPO that selects a skill every 10 environment steps. We evaluated on two downstream tasks for the G1 humanoid, neither of which was seen during skill discovery:

  • G1GoalReach: reach five consecutive goals within 20 s, with each new goal sampled on a 5 m-radius circle around the previous one.
  • G1CubePush: push a cube to the end of a 15 m L-shaped corridor within 60 s.

As shown in Figure 10, GGSD achieves the highest return on both tasks, with a final return more than 6× that of the strongest baseline (6.39 vs. 0.99 on G1GoalReach, and 43.89 vs. 6.63 on G1CubePush). The baselines do not produce usable skills on the high-DoF G1 agent and fail to solve either task. Figure 11 further shows a single GGSD high-level policy reaching goals placed along the letters I, C, L, and R. We attribute this generalization to competitive gameplay: during training, a GGSD agent constantly receives pushes and contact forces from its opponent, so its skills transfer to unseen tasks that involve object interaction (G1CubePush) or sharp turns between goals (G1GoalReach).

We would also like to explain why we originally left out this evaluation. The protocol exists to show that a learned skill-conditioned policy is a useful low-level policy, which is why a high-level controller has to be trained on top of it. GGSD makes the high-level controller unnecessary, because every discovered skill can be used directly by a human. We see this as a key strength of our method.

Please see the supplementary video (3:00–4:30). Every clip in this segment shows a human controlling the agent through the discovered skills on downstream tasks that differ completely from the game environments the agent was trained in. The human solves all of these tasks, including G1-Maze, AntMaze, and AntCubePush. Section 5.5 presents this result.

This works because each skill GGSD discovers is highly coherent and human-interpretable, which purely unsupervised skill discovery methods rarely achieve. For comparison, the video (0:04–0:20) shows skills discovered by METRA on the same agent. They are essentially random movements, with each joint moving independently, and they are neither interpretable nor coherent.


[W2, W3] State coverage and other exploration metrics

As discussed in Section 2.2, a central motivation of our work is a known limitation of unsupervised skill discovery: optimizing only for distinctiveness or state coverage can yield behaviors that are diverse but semantically meaningless. The problem is especially severe for high-DoF agents, whichever way coverage is defined:

  • If coverage is defined over joint configurations, an agent can maximize it by flailing its limbs across many joint angles without ever learning a single walking gait.
  • If coverage is defined over root positions (the most common choice, e.g., x–y coverage in METRA), an agent can maximize it by rolling or tumbling across the arena instead of walking.

In both cases the skill set scores high on coverage but has little practical use. For this reason we did not use state coverage as a primary metric of skill quality.

Our skills still achieve broad coverage where coverage matters for the task. Please see the video (0:56–2:00). In Franka Air Hockey, the Franka arm covers its entire side of the table, which it must do to return pucks arriving from any direction. In Ant Sumo, the agent fights well from every region of the arena. So although we maintain that root-position coverage is not a meaningful measure of skill quality, GGSD also does well by that criterion.


[W4] Training cost and tradeoffs

GGSD uses exactly the same training budget as all baselines. As described in Section 5.1 (Implementation Details), every method is trained with the same 4,096 parallel agents. Training runs on a single A40 or L40 GPU and takes 10 hours– 1.5 days to reach 50k iterations. Self-play adds no computational resources beyond what the baselines use.

We’d like to kindly remind you that the rebuttal period ends in 2 days, and we would greatly appreciate any response or comments.

Reviewer_aghAOpenReview ↗

Acknowledgement of rebuttal

Dear authors, Thank you for the rebuttal.

First, thank you for running the downstream RL experiment. From my perspective, this is essential to quantify how useful the discovered skills are. Skills that are useful, should lead to high downstream performance. Since you use the same high-level RL algorithm across different skill discovery method, this isolates the performance gains to the efficacy of the skill discovery method.

To me, this result is much more important than showing that it is useful for humans, because humans are diverse and have their own idiosyncracies. To convince me of the human point, you would need (1) a human study that compares different skill discovery methods, and (2) a much bigger participant pool of 5 humans (2 including the authors). You could also use LLMs or other agents and use the low level skills as "tools" in an agentic workflow.

So in the future, I think you should present the quantitative results in the following order: Downstream RL (most controlled), Agentic Workflow (LLMs are more stochastic, but still controllable), and then Human Study (you would need more humans and compare different methods).

My concerns on exploration and resource usage are addressed.

Still unaddressed is my question on how well these skills generalize to other policies not seen during self play.

PS: I like the task where it spells out ICLR, that's fun :)

Response to Reviewer aghA: Generalization to Unseen Opponents

We thank the reviewer for the follow-up and are glad that the downstream RL results and our responses on exploration and resource usage were helpful. We would like to address the remaining question on generalization to policies not seen during self-play. We are especially confident about this aspect of GGSD.

Human players as unseen opponents. In Section 5.2, four human players played against our trained GGSD agents for a total of 20 games, and the humans won only 2 of them. A human opponent is, by construction, a policy the agent never encountered during self-play. Human strategies are also far more varied and less predictable than any checkpoint in the opponent pool. As the reviewer noted in the previous comment, "humans are diverse and have their own idiosyncrasies." That diversity is exactly what makes human play a demanding test of generalization: the agent cannot rely on having overfit to its own play style. We therefore regard this evaluation as one of the strongest possible tests of robustness to unseen opponents.

Try it yourself. As stated in the paper, our trained agents are publicly available at https://ggsd-demo.github.io/. We warmly invite the reviewer to play against them directly in AntSumo or AntFencing and see whether they can win even a single game. We believe this firsthand experience is the most direct way to confirm how well GGSD generalizes beyond its self-play opponents.

We hope this addresses the reviewer's remaining concern, and we would be happy to answer any further questions.

Comments on the paper

1 comment

Summary of Rebuttal for the AC

Dear Area Chair,

We thank all reviewers for their constructive feedback. Below we summarize the contribution of our work and how each concern raised during the discussion has been addressed.

Contribution

GGSD proposes competitive self-play as a new source of guidance for skill discovery. It requires no demonstrations, language models, or VLMs. With this guidance, GGSD discovers skills that are semantically meaningful and consistent, and it does so even for high-DoF agents such as the G1 humanoid. Existing unsupervised methods (DIAYN, DADS, METRA) have not achieved this; on the same humanoid they produce largely unstructured motions (video 0:04–0:20). Because each GGSD skill is coherent and interpretable, the skills are directly human-playable: a human can replace the high-level policy with keyboard input and solve unseen downstream tasks (G1-Maze, AntMaze, AntCubePush) with no additional training. To our knowledge, GGSD is the first skill discovery framework to demonstrate human-playable skills on a humanoid.

Concerns and how they were addressed

  1. Lack of quantitative comparison with skill discovery baselines (aghA, s8U9). We added the standard downstream RL evaluation (Appendix F). We froze each method's low-level policy and trained a PPO high-level controller on two unseen G1 tasks. GGSD achieves more than 6× the return of the strongest baseline (6.39 vs. 0.99 on G1GoalReach; 43.89 vs. 6.63 on G1CubePush), while the baselines fail on both tasks. Reviewer aghA confirmed that this experiment is "essential to quantify how useful the discovered skills are."

  2. Generalization to opponents unseen during self-play (aghA). In Section 5.2, four human players, whose policies are unseen by construction and far more diverse than any self-play checkpoint, won only 2 of 20 games against GGSD agents. Our agents are also publicly playable at https://ggsd-demo.github.io/, so anyone can verify this robustness directly.

  3. Exploration metrics and training cost (aghA). We explained why state coverage is a misleading measure for high-DoF agents and showed that GGSD still achieves broad task-relevant coverage. All methods use an identical training budget (4,096 parallel agents on a single GPU). Reviewer aghA stated that these concerns are addressed.

  4. Whether gains come from reward guidance or from self-play, and novelty (eLYg, s8U9). We decomposed GGSD into four components and added ablations (Section 5.6). Without self-play, skills collapse into overlapping trajectories in AntSumo and FrankaAirHockey. They collapse even when the MI objective is replaced with METRA. The diversity therefore comes from competitive self-play, not from the reward or the choice of skill objective. The hierarchical policy with separated observations is what enables zero-shot transfer to tasks with no opponent.

  5. Reliance on reward engineering and the "unsupervised" framing (eLYg, s8U9). We position GGSD as guided skill discovery, alongside LGSD and DoDont, rather than as purely unsupervised. We retrained G1Boxing with only 4 of its 11 reward terms. Diverse skills still emerge, including a new two-handed punch, and semantic distance increases from 0.121 to 0.130. The game reward specifies what winning means, not which behaviors to perform.

  6. Clunkiness of human control and maze transfer (s8U9). The video shows how AntCubePush is solved: when the cube gets stuck against the wall, the player switches to the push skill to free it, so the task depends on choosing the right skill for each situation. We also clarified that no self-play occurs in the maze tasks; skills learned once in the game are transferred zero-shot. On the same G1 humanoid, the baseline skills fail even at goal-reaching navigation.

  7. Scalability to more skills (eLYg). We trained G1Boxing with 10 and 15 skills and reported semantic diversity for all settings. Nearly all skills remain semantically meaningful (1/10 and 0.67/15 judged non-meaningful, averaged over 3 seeds), and new behaviors such as strong, weak, and fast punch variants emerge. GGSD stays above all baselines in every setting except Ant with 10 skills.

  8. Combos, comparison with demonstration-based methods, and hyperparameters (eLYg). GGSD addresses a different setting from data-driven methods such as LISA: it discovers skills without any data, so it is not bounded by demonstrations. A combo means the same skill "button" produces a new motion depending on the preceding skill, which is different from sequential chaining. As our hypothesis predicts, combos weaken when more skills are available. Only the MI scale β was coarsely adjusted per game, and no per-task hyperparameter search was performed.

We believe the additional experiments have substantially strengthened the paper, and we hope the AC will take these updates into account. Thank you for your time and consideration.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.