acceptodds
Under review as a conference paper at ICLR 2027

Put Your Skills To Work: Exploration via Random Walks in Skill Space

Abstract

Self-supervised pre-training in reinforcement learning aims to discover a diverse set of behaviours without extrinsic reward, which can then accelerate a wide range of downstream tasks. Recent skill discovery methods such as METRA and CSF learn skills as directions in a latent space and produce impressive behaviours in locomotion domains. However, the exploration of existing skill learning methods degrades in complex environments, and existing remedies add machinery on top of skill learning, such as an explicit skill tree or a learned guide policy. Leveraging the latent space that METRA/CSF already learn, we instead change only how skills are executed during training. Rather than committing to one skill per episode, we chain skills of a fixed horizon, resampling the latent direction at each boundary, so that a rollout performs a random walk whose steps are skill executions rather than primitive actions. Applying this to CSF gives Chained-CSF (CCSF), which leaves the base algorithm, its losses and its networks untouched and adds only the skill horizon as a hyperparameter. On a bottleneck maze, skill chaining increases state coverage from to . On Craftax-Classic, a Minecraft-like domain with a deep crafting tree, CCSF unlocks of the achievements per episode without task rewards, the highest we are aware of for an unsupervised RL method. Removing the survival pressures that end episodes early lets it climb further, obtaining an iron pickaxe (depth 6; of episodes) and diamonds (depth 7; ), while a baseline without skill switching does not reliably collect stone (depth 3). Without any reward or additional learning, the skills learned during pre-training can be used to solve downstream tasks: goals specified as a change in the raw observation are converted into latent directions and commanded from the frozen policy, reaching every stage up to iron in over of episodes and doing so several times faster than random skill selection.

Primary area:
reinforcement learning
Keywords:
Unsupervised Reinforcement Learning, Skill Discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2RNSOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses exploration in complex tasks by building on unsupervised skill-discovery methods like METRA and CSF and introducing a new approach called Chained CSF (CCSF). Unlike traditional methods that stick with one skill for an entire episode, CCSF resamples the skill every HH steps during training. This transforms each rollout into a random walk through different skills.

Critical strengths
  • This proposed change is straightforward and easy to add to existing methods. Notably, if H=TH = T the method reduces to standard CSF, making it simple to directly compare CSF and CCSF.
  • It's particularly interesting that CCSF enables progress deep into the Craftax technology tree without relying on extrinsic rewards.
Critical weaknesses

1. The paper's main contribution and positioning are unclear. While the introduction claims that existing skill-discovery methods struggle with exploring complex tasks and introduces random-walk skill chaining as a solution, the actual change amounts to a different skill-resampling schedule—without a new learning objective or supporting analysis. As a result, this feels more like an empirical observation than a principled learning advance. The paper often compares CCSF to METRA and CSF, which focus on learning a skill space, while CCSF builds on these for more challenging exploration. In this context, methods like EDL, UPSIDE, and DISCO-DANCE seem like more relevant baselines. Simply outperforming DISCO-DANCE in a single maze doesn't sufficiently establish the value of CCSF, especially without explaining why random-walk skill chaining works (see Weakness 2).

2. There's no clear analysis of why random walk skill resampling is helpful. The choice to uniformly resample skills every HH steps is mainly based on intuition. Since a random walk over skills is still undirected, it's not obvious why this should work better than more directed skill compositions. The evidence for the approach is limited to qualitative results from a single maze environment. A theoretical explanation of when and why random-walk skill chaining is effective would really strengthen the argument for its simplicity. The paper would also benefit from stronger experiments, including comparisons with other skill-selection strategies. For example, a more principled approach could use a high-level policy to select skills during pre-training, but the current method only trains such a policy for downstream tasks.

3. The experiments don't clearly back up the paper's claims, and some of the comparisons don't seem entirely fair. In particular, the experiments don't consistently test a clear hypothesis about why chaining skills should improve exploration.

  • Direct baselines are missing—EDL and UPSIDE aren't included, and DISCO-DANCE is only tested on the maze. Also, only CCSF is tested with skill chaining in the main comparison. When chaining is added to DISCO-DANCE using the same protocol, the performance difference becomes much smaller.
  • For Craftax, the experiments don't isolate the benefit of chaining from a credible baseline.
    • Table 1 relies on baselines from prior work, but none of these are skill-discovery methods (see lines 352–358).
    • The Peaceful variant removes death and adds a BREAK action, which raises questions about whether results from this modified environment are truly comparable. It would be better to compare CCSF against skill-discovery and exploration baselines on standard Craftax-Classic, and use Peaceful only for supplementary evidence.
  • For locomotion tasks, CCSF actually achieves lower coverage and underperforms CSF on HalfCheetahHurdle. This phenomenon needs to be addressed and explained.
  • The choice of benchmarks is also questionable. Figures 1–2 highlight exploration bottlenecks in continuous navigation, but only a small point maze is used to actually test this scenario. More challenging benchmarks like AntMaze, HumanoidMaze, or continuous manipulation tasks would provide much stronger evidence for the method. It's also not clear how the discrete Craftax environment supports the main argument.

4. There's limited ablation and no sensitivity analysis for HH. Although HH is the only new hyperparameter, each domain just uses a single value without exploring its impact. Running a sweep over HH is important to determine whether the observed improvement comes from chaining itself or from a specific resampling frequency.

Minor Weakness

  • The abstract should make it clear that the Peaceful results come from a modified environment.
Questions for authors

Please see the weaknesses above.

9 comments

Thank you! Your feedback was very helpful in strengthening the paper.

Due to the number of questions and comments, we will group and answer them together if possible. We will address more specific questions separately.

Issue 1: The contribution is unclear

W1 The paper's main contribution and positioning are unclear.

We have revised the framing with edits in the Abstract and Introduction. The key idea in short:

  • Random walks have guarantees in finite state spaces, but they scale poorly to high-dimensional and complex environments, e.g. a random walk on joint angles for an Ant in a maze will mostly jitter in place and take an impractical amount of time
  • METRA and CSF learn far-reaching skills that traverse the state space; however, they have limited exploration capability in complex environments, as shown by our expanded experiments. (2D bottleneck point maze, Ant maze Large and Giant, Craftax-Classic and Peaceful modification of Craftax-Classic.
  • Exploration in complex environments is a known problem in skill learning. Existing methods alleviate this with additional complex machinery (EDL, UPSIDE and DISCO-DANCE).
  • Exploration is improved if we chain CSF's skills, as these skills traverse the environment, "abstracting" the complex environment into a latent space where a random walk might scale better. E.g. random walk of an Ant in a maze with learnt movement skills instead of primitive joint action.

Our contributions are:

  • A new approach to learning skills that is simple but effective in increasing exploration by connecting random walks and skill learning.
  • Additional theoretical analysis that connects how chaining changes the mutual information optimisation to favour more controllable skills that lead to improved downstream performance.
  • Experimental results that show both improved coverage (point maze, Ant maze and Craftax) and improved downstream control (Ant, HalfCheetah and Craftax).

Issue 2: Theory and analysis

W2 There's no clear analysis of why random walk skill resampling is helpful.

W1 change ... skill-resampling schedule—without a new learning objective or supporting analysis ... feels more like an empirical observation than a principled learning advance

W2 A theoretical explanation of when and why random-walk skill chaining is effective would really strengthen the argument for its simplicity.

We strengthen our theoretical grounding in a few ways.

  • In the Introduction, Figure 2 and Section 4.2, we overhauled our analysis and connected chaining skills with theory on random walks to explain the improved exploration.
    • Added Section 3.1, a preliminary on random walks.
  • Moved the old analysis at Section 4.2 to the appendix as they serve as empirical observations of the consequences of random walks.
  • Connected our results in Section 5.2 on lower state coverage on Ant and HalfCheetah to the diffusive T\sqrt{T} scaling of a random walk.
  • Added theoretical connections to Mutual Information Skill Learning and Empowerment in Section 4.3, relating chaining to a change in how the MI objective is optimised.
    • In short, the CSF objective, I(S′,S;Z)I(S', S; Z), can be rearranged to I(S′;Z∣S)=I(S′,S;Z)−I(S;Z)I(S'; Z | S) = I(S', S; Z) - I(S; Z).
    • We argue, and verify empirically, that chaining suppresses I(S;Z)I(S; Z), which is how discriminable the skill is from the state alone.
    • Hence, optimising the CSF objective while chaining focuses on the conditional term I(S′;Z∣S)I(S'; Z | S). This term is a lower bound on empowerment [4], which measures the control an agent has over its future.
    • Added experimental results in Section 5.1 to test this by estimating MI terms with lower-bound classifiers.
    • Appendix C for additional proofs.

Issue 3: Baselines and Benchmarks

Issue 3.1: Need more direct baselines

W1 CCSF to METRA and CSF, which focus on learning a skill space, while CCSF builds on these for more challenging exploration. In this context, methods like EDL, UPSIDE, and DISCO-DANCE seem like more relevant baselines.

W3.1 Direct baselines are missing—EDL and UPSIDE aren't included, and DISCO-DANCE is only tested on the maze

  • EDL does not work by itself on the bottleneck maze. They "were unable to explore this type of maze effectively with SMM ... rely on an oracle to simulate perfect exploration and evaluate the skill discovery/learning stages" (Section 4, page 7 of [1]). Hence, we did not include EDL as a baseline.
  • "UPSIDE relies on the ability to reset to establish a root node for its growing tree" (Section 6, page 9 of [2]). Hence, UPSIDE requires episodic environments and will not work on Craftax.
  • Furthermore, there does not seem to be open-source code for UPSIDE, and it is not trivial to faithfully implement and tune it due to the multi-stage tree-expansion procedure with many interdependent hyperparameters.
  • However, we have implemented DISCO-DANCE and tuned it to the best of our ability for the 2D maze, Ant maze, Craftax and Craftax Peaceful.
  • Lastly, DISCO-DANCE reports significant gains over UPSIDE in maze environments and DMC hence we were confident comparing directly to DISCO-DANCE

Issue 3.2: Need more difficult benchmarks

W1 simply outperforming DISCO-DANCE in a single maze doesn't sufficiently establish the value of CCSF, especially without explaining why random-walk skill chaining works

W3.4 The choice of benchmarks is also questionable.

W3.4 only a small point maze is used to actually test this scenario.

W3.4 challenging benchmarks like AntMaze, HumanoidMaze, or continuous manipulation tasks would provide much stronger evidence for the method.

  • We agree and added the Large and Giant Ant Maze benchmarks from OGBench in Section 5.1 as challenging benchmarks, for stronger evidence of our method's improved exploration.
  • As mentioned in Issue 3.1, the bottleneck benchmark is not a trivial benchmark for skill learning algorithms, even for the state-of-the-art methods CSF and METRA, as shown in Figure 1.
  • We also believe that Craftax-Classic is an incredibly difficult and suitable benchmark, and we explain why below.

Issue 3.3: Why Craftax?

W3.4 not clear how discrete Craftax environment supports the main argument.

  • Fair point, the connection was not clear in the submission. We now make it explicit in Section 4.2, line 273.

  • Figure 5 also explains how the crafting tree in Craftax is strictly ordered with heavy dependencies. Exploring in such a bottlenecked enivronment is truly difficult. With many unsupervised RL methods with exploration bonuses tested in Section 5.3 making little progress (RND, ICM, E3B).

  • Craftax is a highly complex environment with CSF (Section 7, page 10 of [3]) claiming that it is not clear how to "scale the performance to increasingly complex environments such as Craftax".

  • To our knowledge, no skill learning algorithm has yet achieved significant results in Craftax without task rewards, so we see it as a clear gap and a good test of CCSF's exploration capacity.

  • The discrete action space does not change the method, as chaining only changes when skills are resampled.

Issue 3.4: Need more credible baselines in Craftax

W3.2 For Craftax, the experiments don't isolate the benefit of chaining from a credible baseline.

W3.2 CRAFTAX Table 1 relies on baselines from prior work, but none of these are skill-discovery methods (see lines 352–358)

W3.2 The Peaceful variant ... results from this modified environment are truly comparable. It would be better to compare CCSF against skill-discovery and exploration baselines on standard Craftax-Classic ...

  • As mentioned, no skill learning algorithm has managed to have any form of success with Craftax. However, we do agree that more baselines would concretely show this.
  • We implemented DIAYN and DISCO-DANCE on both Craftax-Classic and Craftax-Peaceful, though these skill learning methods are inherently not suited for such difficult environments with random starts, a stochastic environment with moving enemies, terminating states due to death and a constant flux of the state space (energy, hunger, thirst).
  • We added a quick note in Section 5.3.1 to provide some intuition as to why these methods do not work in Craftax.
  • C-TeC and the other exploration-based planners we compared against in Section 5.3 directly test the exploration capacity of CCSF.
  • C-TeC is also the strongest unsupervised baseline for Craftax-Classic that we know of at the time of writing.
  • Testing CSF and CCSF on Craftax and Craftax Peaceful explicitly isolates the benefit of chaining against CSF.

Issue 3.5: Peaceful variant is not comparable

W3.2 The Peaceful variant removes death and adds a BREAK action, which raises questions about whether results from this modified environment are truly comparable. It would be better to compare CCSF against skill-discovery and exploration baselines on standard Craftax-Classic, and use Peaceful only for supplementary evidence

  • We agree that standard Craftax-Classic should be the main comparison. It is our main result in Section 5.3, and we have now added skill-discovery baselines there (CSF, DIAYN and DISCO-DANCE) alongside the exploration baselines.
  • We keep Peaceful as supporting evidence, as you suggest. Its purpose is to isolate the exploration difficulty of the crafting tree from survival.
  • The modifications make the environment easier for every method, as there is no survival pressure or termination due to death. We expected this to help the other skill learning methods, but CSF and DISCO-DANCE still did not progress far, even with extensive tuning.
  • The BREAK action is available to every method. It only removes a placed table, furnace or plant, so that an agent cannot wall itself in over a long episode.

Issue 3.6: Performance

W3.1 CCSF is tested with skill chaining in the main comparison. When chaining is added to DISCO-DANCE using the same protocol, the performance difference becomes much smaller.

  • Yes, the scores of DISCO-DANCE and all the other methods did improve after chaining. However, they are still appreciably lower than the performance of CCSF on the 2D bottleneck maze.
  • The additional Ant Maze in Section 5.1 and the implementation of DISCO-DANCE for Craftax in Section 5.3 further demonstrate the gain in CCSF's exploration capability

W3.3 For locomotion tasks, CCSF actually achieves lower coverage and underperforms CSF on HalfCheetahHurdle. This phenomenon needs to be addressed and explained.

  • This result is a property of random walk's slower diffusion in a freely traversable state space vs a ballistic trajectory (CSF). We add this explanation in Section 5.2
  • Because of this, we observe that CSF learns skills that move as fast as possible by spinning forward in one direction, which is particularly suited to the Hurdle task and not to the zero-shot goal task.
  • We show the performance gain in zero-shot goal task for HalfCheetah in Section 5.2 where "staying time fraction" of CCSF is 28.3% which is significantly higher than the 10.0% of CSF.
Reviewer_3NyAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies that latent-directional skill discovery (METRA/CSF) explores poorly in environments with bottlenecks, and proposes CCSF, which resamples the skill direction every H steps during data collection, truncating each segment as a time limit and bootstrapping under the segment's own skill. No changes to losses, networks, or objectives.

Critical strengths
  • Simple, minimally invasive change with a large effect; H is the only added hyperparameter, and H=T recovers CSF exactly. Low barrier to adoption.
  • Strong empirical isolation: the CSF/CCSF pair shares every network, hyperparameter, and update rule, so the maze comparison cleanly attributes gains to chaining.
  • Downstream usability is tested three ways (zero-shot chained goals, single goal, short PPO high-level), showing the skills are not merely exploratory artifacts.
  • Craftax-Classic with survival pressures intact is a genuinely demanding benchmark.
Critical weaknesses
  • The framing repeatedly invokes METRA and claims the change "applies to any method whose skills are latent directions," yet only CSF is evaluated.
  • The natural comparison class is methods that compose or chain skills (UPSIDE's tree growth, DISCO-DANCE's guide skill, and the older skill-chaining line), not only single-skill discovery methods.
  • Ant and HalfCheetah are the easiest settings in this literature. METRA's own evaluation includes Humanoid, pixel-based Cheetah and Quadruped; the paper's locomotion section tests only whether CSF is not hurt, not whether chaining helps where skills are higher-dimensional.
  • H is set to different values across environments with no sweep and no analysis of how H trades off segment length against exploration. Since H is the method's only hyperparameter, it should be analyzed more extensively.
Questions for authors
  • In Fig.7, CSF obtains better state coverage, but why it performs worse in the downstream AntMultiGoal?
  • Achievements per episode depends on episode length, and CCSF's own episodes last only ~250 steps at the end of training (Table 4), while the baseline numbers are taken from previous work rather than rerun. Were baseline episode lengths and reset protocols matched?
5 comments

We appreciate your appreciation for our results on the Craftax-Classic benchmark. It is indeed a demanding benchmark.

More importantly, thank you for your feedback, which has substantially improved the manuscript.

Weakness 1

The framing repeatedly invokes METRA and claims the change "applies to any method whose skills are latent directions," yet only CSF is evaluated.

  • Agreed, this is important to the framing of our method.
  • We trained METRA with chaining (METRA Chain) on the 2D bottleneck maze and report the results in Table 1.
  • Chaining METRA during training gives a coverage of 94.6%, a large improvement over METRA's 41.2%. We added this in Section 5.1, line 359.

Weakness 2

The natural comparison class is methods that compose or chain skills (UPSIDE's tree growth, DISCO-DANCE's guide skill, and the older skill-chaining line), not only single-skill discovery methods.

  • We agree. DISCO-DANCE was already one of the strongest baselines on the 2D bottleneck maze, and we have now added it to the Ant mazes, Craftax-Classic and its Peaceful variant.
  • Its performance on Craftax remains weak despite our best efforts (Table 2). DISCO-DANCE treats a skill as converged once the discriminator accuracy q(z|s) exceeds a threshold, and only then selects a guide skill and adds new skills. With randomised starts, stochastic dynamics, deaths and continually changing vitals, the state alone rarely identifies the skill, so q(z|s) stays low and the guidance step never triggers. DISCO-DANCE then effectively reduces to DIAYN, consistent with their similar scores (1.31 vs. 1.59 achievements).
  • We did not implement UPSIDE. It "relies on the ability to reset to establish a root node for its growing tree" (Section 6 of [1]), which Craftax's procedurally generated starts do not provide. It also has no open-source code, and its multi-stage tree expansion has many interdependent hyperparameters, so a faithful reimplementation is not trivial.
  • The older skill-chaining line (Konidaris & Barto, 2009) [2] composes options backward from a goal, so it needs a task reward and does not apply to our unsupervised setting.
  • More generally, even state-of-the-art methods such as CSF are unable to "scale the performance to increasingly complex environments such as Craftax" [3], and the older method we did add, DIAYN, also fares poorly (Table 2).

Weakness 3

Ant and HalfCheetah are the easiest settings in this literature. METRA's own evaluation includes Humanoid, pixel-based Cheetah and Quadruped; the paper's locomotion section tests only whether CSF is not hurt, not whether chaining helps where skills are higher-dimensional.

  • We agree that these are the two easiest settings in this literature. Our goal is to improve the exploration of skill learning in complex environments such as mazes and Craftax, so we did not prioritise pixel-based locomotion. We intend to possibly add some of these experiments to the appendix of the final version.
  • Your point also made us realise that our framing was unclear, as Reviewer 2RNS noted too. We have substantially revised the Introduction and Section 4.2.
  • To test chaining on harder, higher-dimensional tasks, we added AntMaze-Large and AntMaze-Giant from OGBench, where CCSF raises coverage over CSF from 40.6% to 75.4% and from 12.8% to 39.5% (Table 1).
  • The locomotion section primarily tests downstream performance, not coverage. We expected CCSF to cover less there, because a random walk (CCSF) spreads as T\sqrt{T}, while a ballistic trajectory that travels in one direction (CSF) spreads as TT.
  • Hence, in the mostly fully traversable settings of this literature, CCSF will likely have lower overall state coverage. Our goal, as we reiterate in the paper, is to boost exploration in complex environments with bottlenecks and obstacles, such as Craftax.
  • The specific purpose of the locomotion experiments is to show the stronger downstream performance of CCSF, which we posited in the previous version and now support with a theoretical analysis in Section 4.3 and measurements in Section 5.1. The downstream performance is indeed better, validating our hypothesis.

Weakness 4

H is set to different values across environments with no sweep and no analysis of how H trades off segment length against exploration. Since H is the method's only hyperparameter, it should be analyzed more extensively.

  • We agree and have added a sweep over HH on four 2D mazes (Figure 3 in Section 5.1; Figure 11 in Appendix D.2).
  • We found that resampling every step (H=1H=1) and holding one skill for the whole episode (H=TH=T, which is CSF) both cover less than intermediate horizons.
  • The best HH differs between mazes, so a single fixed value does not transfer.
  • A Lévy schedule, which draws each horizon from p(h)∝h−2p(h) \propto h^{-2} on [1,T][1, T], is close to the best fixed horizon on all four mazes with nothing to tune. This fits the theory in Section 3.1 that Lévy walks search efficiently when the scale of the problem is unknown. We use it for the Ant mazes and locomotion tasks.
  • The sweep also validates our new framing around mutual information (Section 4.3). By the chain rule, the CSF objective splits into two terms, I(S,S′;Z)=I(S;Z)+I(S′;Z∣S)I(S,S';Z) = I(S;Z) + I(S';Z \mid S): a discriminability term, which measures how well the state alone identifies the skill, and a control term, which measures how much the skill determines the next state and is a lower bound on empowerment. We argue that chaining suppresses the first term, so training concentrates on the second. The sweep confirms this: as HH grows towards TT (CSF), I(S;Z)I(S;Z) grows and the estimated I(S′;Z∣S)I(S';Z \mid S) falls (Figure 3, left). Shorter horizons therefore give more controllable skills.
  • We did not run this sweep on the other environments due to compute cost, and instead empirically chose horizons that gave good performance.

Question 1

In Fig.7, CSF obtains better state coverage, but why it performs worse in the downstream AntMultiGoal?

  • Lower coverage in open space is expected for CCSF, as a random walk is diffusive and spreads as T\sqrt{T}, while a held skill moves ballistically and spreads as TT. We now explain this in Section 3.1 and Section 5.2. A random walk excels when obstacles or bottlenecks block a direct path, as in the Ant mazes and Craftax-Classic.
  • Coverage and goal reaching measure different things. Coverage rewards skills that travel far, while goal reaching requires skills that can be steered to a goal from any state and held there.
  • CSF's downstream performance is weaker for the reason given in our answer to Weakness 4. CSF can raise its objective through the discriminability term I(S;Z)I(S;Z), by reaching states that identify the skill, without making the next state more controllable. Chaining suppresses that term, so CCSF concentrates on the control term I(S′;Z∣S)I(S';Z \mid S), a lower bound on empowerment, and each skill is trained from many start states instead of only from reset.
  • As a result, CCSF's skills are easier to command zero-shot: the staying time fraction, our metric for the zero-shot multi-goal task, rises from 17.2% to 27.7% on Ant and from 10.0% to 28.3% on HalfCheetah (now Figure 4, Section 5.2).

Question 2

Achievements per episode depends on episode length, and CCSF's own episodes last only ~250 steps at the end of training (Table 4), while the baseline numbers are taken from previous work rather than rerun. Were baseline episode lengths and reset protocols matched?

  • You are right that achievements per episode depend on episode length, and that the baseline numbers were taken from previous work, so their episode lengths and reset protocols were not fully matched.
  • This question led us to revamp how we evaluate on the Craftax environments.
    • We retrained every method from scratch and were able to replicate the results of ICM, RND and C-TeC (the strongest unsupervised RL baseline on Craftax-Classic at the time of writing).
    • We also use a new evaluation protocol, described in Section 5.3 and Appendix G. We evaluate the final weights of every training seed of each method on the same 500 Worlds with fixed-seed.

[1] Kamienny et al. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. ICLR 2022. https://arxiv.org/pdf/2110.14457

[2] Konidaris and Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. NeurIPS 2009.

[3] Zheng et al. Can a MISL fly? Analysis and ingredients for mutual information skill learning. ICLR 2025. https://arxiv.org/pdf/2412.08021

Sincerely, thank you again for reviewing our manuscript thoroughly and spotting some discrepancies that we didn't, like the matching of achievements from Craftax-Classic due to episode lengths.

Reviewer_vg5FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses limited exploration in latent-direction skill discovery methods such as METRA and CSF, which hold one skill per episode and therefore stall at walls and bottlenecks. The proposed CCSF keeps the CSF objective, loss, and networks and resamples the skill every HH steps during training, turning each rollout into a random walk over skills. Segment boundaries are treated as time limits, and setting HH to the episode length recovers CSF. CCSF raises bottleneck-maze coverage from 51.6% to 97.8%, reaches 8.9 achievements per episode on Craftax-Classic, and obtains an iron pickaxe in 97.0% of episodes on a variant without survival pressure. The learned skills are also applied to zero-shot and hierarchical downstream tasks.

Critical strengths
  • The motivation and the method align closely. Skills held in one direction stop at walls, and periodic resampling turns them into steps of a random walk. Figures 1 and 2 illustrate this mechanism.

  • The proposed method is simple to implement and perform significant performance gains. It adds only the skill horizon HH and recovers CSF when HH equals the episode length. Because CSF and CCSF share all networks and hyperparameters in the maze and Peaceful experiments, these comparisons isolate the effect of chaining.

  • The method yields large gains in hard-exploration settings, including near-complete maze coverage and deep progress through the Craftax-Classic tech tree.

Critical weaknesses
  • Whether the proposed "Chaining" method would benefit across different skill discovery methods is not shown. Section 2 states that the approach applies to any method with latent-direction skills, but only CSF is trained with chaining. METRA appears only with evaluation-time chaining (Table 12). Training METRA with chaining would support the claim of algorithm independence.

  • The Craftax-Classic comparisons are split across environments. Exploration baselines are compared only in the original environment (Table 1), and CSF only in the Peaceful variant (Figure 5, Table 7). The Peaceful variant also adds a BREAK action and changes lava dynamics, which limits comparison with prior work. Reporting CSF in the original environment would isolate the contribution of chaining under survival pressure. In addition, no skill discovery baseline is evaluated on Craftax-Classic.

  • The downstream results support a narrower conclusion. In the Craftax zero-shot setting, random zz already reaches iron in 82.8% of episodes, so goal commands mainly help at the iron pickaxe stage (40.6% versus 8.2%). Ant multi-goal switches skills within each episode, as in CCSF training, so its gains may reflect skill switching rather than broader generalization. On HalfCheetah, CCSF also trails CSF in coverage and Hurdle return.

Questions for authors
  • Section 5.2 and Table 1 appear to use the original Craftax-Classic, whereas the Limitations section states that the original environment was not attempted. Could the authors clarify the environment for each result and report CSF on the original environment under the same budget?

  • Could the authors report results for METRA trained with chaining on the maze and Craftax Peaceful to show the generality of the proposed method?

  • Could the authors report a sweep over HH? This would show how sensitive exploration and downstream performance are to the only added hyperparameter.

  • Out of curiosity, the critic learns the value of continuing a skill beyond each segment, while γ=0.99\gamma = 0.99 implies an effective horizon of about 100 steps, longer than HH in every experiment. Does this gap affect the learned skills, for example the lower coverage on Ant or the weaker HalfCheetah results? Results that vary γ\gamma and HH jointly would be informative.

5 comments

Weakness 1 and Question 1

Whether the proposed "Chaining" method would benefit across different skill discovery methods is not shown. [...] Training METRA with chaining would support the claim of algorithm independence.

Could the authors report results for METRA trained with chaining on the maze and Craftax Peaceful to show the generality of the proposed method?

  • Agreed, this is important to the framing of our method.
  • We trained METRA with chaining (METRA Chain) on the 2D bottleneck maze (Table 1, Section 5.1). Coverage rises from 41.2% to 94.6%, close to CCSF's 97.3%. Chaining METRA only at evaluation reaches 73.2% (Table 11), so training under chaining matters.
  • We did not run METRA Chain on Craftax-Classic Peaceful due to compute cost.
  • We do not claim that chaining benefits every skill discovery method. Our claim, supported by the analysis in Section 4.3, is for methods whose skills are latent directions, such as METRA and CSF.

Weakness 2

The Craftax-Classic comparisons are split across environments. Exploration baselines are compared only in the original environment (Table 1), and CSF only in the Peaceful variant (Figure 5, Table 7) ... Reporting CSF in the original environment would isolate the contribution of chaining under survival pressure. In addition, no skill discovery baseline is evaluated on Craftax-Classic.

  • We agree that exploration-bonus and skill learning methods should be tested together. We revamped the Craftax experiments, retrained all baselines from scratch, and Table 2 now includes DIAYN, DISCO-DANCE and CSF on the original Craftax-Classic.
  • CSF unlocks 1.86 achievements per episode against 9.62 for CCSF, which isolates the contribution of chaining under survival pressure.

The Peaceful variant also adds a BREAK action and changes lava dynamics, which limits comparison with prior work.

  • To clarify, we do not compare across Craftax-Classic and Peaceful; they are treated as separate benchmarks. Peaceful results are only compared with methods trained on Peaceful.
  • You are right that there is no prior work on Peaceful. It is our own variant, inspired by Minecraft's Peaceful mode, to see how far skill learning methods climb the technology tree without survival pressure. We therefore compare only skill learning methods and a random policy there (Figure 7), not the exploration baselines.
  • Even without survival pressure, CSF, DIAYN and DISCO-DANCE fail, in line with CSF's own remark [1] that it is unclear how to scale such methods to Craftax. We added our intuition for why DIAYN and DISCO-DANCE fail in Section 5.3.1, line 495.
  • All methods are retrained on Peaceful with the same hyperparameters, so CCSF against CSF isolates chaining. CCSF reaches diamonds in 10.4% of episodes, while CSF barely obtains wood pickaxes and does worse than a random walk over primitive actions.

Weakness 3.1

The downstream results support a narrower conclusion ... random already reaches iron in 82.8% of episodes, so goal commands mainly help at the iron pickaxe stage (40.6% versus 8.2%).

  • We agree that random skill commands on the trained CCSF policy already reach iron in most episodes (76.0% in the updated Table 10; downstream results are now in Appendix H.1, Figure 16). However, they reach it much later, at a median of 534.5 steps, against 156, 108 and 193 steps for the hierarchical, chained-goal and single-goal arms.
  • That random commands reach iron at all is a strength of CCSF: a random walk over its skills goes deep into the tech tree, while a random walk over primitive actions obtains a stone pickaxe in only 8% of episodes (Table 9). It is slower because a random walk's coverage scales as T\sqrt{T} (Section 3.1).
  • In contrast, the skills learnt by CSF, DISCO-DANCE and DIAYN do not reliably collect stone during training, so chaining their skills would not lead to deeper achievements.
  • Goal commands matter most at the iron pickaxe, which needs wood, coal and iron in the inventory with a crafting table and a furnace nearby. There the goal-directed arms reach 57.4% to 64.2% against 6.0% for random commands.

Weakness 3.2

Ant multi-goal switches skills within each episode, as in CCSF training, so its gains may reflect skill switching rather than broader generalization. On HalfCheetah, CCSF also trails CSF in coverage and Hurdle return.

Skill switching in the multi-goal task

  • We clarify that the multi-goal task is run after training with the policy frozen. At every step the skill is recomputed as the direction to the goal in the latent space (Section 3.3), so CSF and CCSF both switch skills during this task.
  • We do agree that switching skills during training is why CCSF's skills are easier to control, and this is now our claim. The CSF objective splits into I(S,S′;Z)=I(S;Z)+I(S′;Z∣S)I(S,S';Z) = I(S;Z) + I(S';Z \mid S). Switching suppresses the first term, so training concentrates on the second, a lower bound on empowerment (Section 4.3), which gives skills that are controllable from any state.

Coverage and Hurdle on HalfCheetah

  • CCSF trails in coverage because a random walk spreads as T\sqrt{T} in open space, while a held skill spreads as TT (Section 3.1 and Section 5.2).
  • The Hurdle result has the same cause. CSF rewards moving as far as possible in the latent space, and from our observations the CSF cheetah rolls forward as fast as it can and cannot stop. This suits Hurdle but not goal reaching, where CCSF's staying time fraction is 28.3% against 10.0% for CSF.

Question 2

Could the authors report a sweep over HH? This would show how sensitive exploration and downstream performance are to the only added hyperparameter.

  • We agree and have added a sweep over HH on four 2D mazes (Figure 3 in Section 5.1; Figure 11 in Appendix D.2).
  • Resampling every step (H=1H=1) and holding one skill for the whole episode (H=TH=T, which is CSF) both cover less than intermediate horizons, and the best HH differs between mazes.
  • A Lévy schedule, p(h)∝h−2p(h) \propto h^{-2} on [1,T][1, T], is close to the best fixed horizon on all four mazes with nothing to tune, which fits the theory in Section 3.1. We use it for the Ant mazes and locomotion tasks.
  • The sweep measures exploration and the mutual-information terms, not downstream performance directly. As HH grows towards TT (CSF), I(S;Z)I(S;Z) grows and the estimated control term I(S′;Z∣S)I(S';Z \mid S) falls (Figure 3, left), so shorter horizons give more controllable skills.
  • We did not run this sweep on the other environments due to compute cost.

Question 3

Out of curiosity, the critic learns the value of continuing a skill beyond each segment, while γ\gamma implies an effective horizon of about 100 steps, longer than HH in every experiment. Does this gap affect the learned skills, for example the lower coverage on Ant or the weaker HalfCheetah results? Results that vary γ\gamma and HH jointly would be informative.

  • Good question. You are right about the gap, but we do not think it is a problem, for two reasons.
  • First, it is not specific to chaining. The environment time limit is also a truncation, so CSF and the other baselines bootstrap past the end of the episode in the same way.
  • Second, our ablation suggests the bootstrap helps. Terminating at the boundary, which removes the gap, lowers coverage from 97.3% to 92.9% when chained and from 68.1% to 53.2% when held (Section 5.4, Table 11).
  • The lower coverage on Ant and HalfCheetah has a different cause, the T\sqrt{T} scaling of a random walk (Section 3.1 and Section 5.2).
  • We did not run a joint sweep over γ\gamma and HH, and agree it would be informative.

[1] Zheng et al. Can a MISL fly? Analysis and ingredients for mutual information skill learning. ICLR 2025. https://arxiv.org/pdf/2412.08021

Thank you for seeing value in our work and for the positive review!

Comments on the paper

2 comments

Summary of changes (additions are marked in green in the revised PDF)

A sincere thank you to all reviewers who took their time to provide us with valuable feedback to improve our manuscript. Here are a summary of changes, that have likely been mentioned in my individual comments.

General

  • Tightened the framing of the paper (Introduction, Section 4.2).
  • Added a preliminary on random walks including coverage guarantees, diffusive vs. ballistic motion, and Lévy walks (Section 3.1).
  • Added Figure 2, an Ant example showing why a walk over skills explores where primitive actions and a single held skill do not.
  • Added an analysis showing that chaining suppresses I(S;Z)I(S;Z) in the CSF objective, so training concentrates on I(S′;Z∣S)I(S';Z \mid S), a lower bound on empowerment (Section 4.3, Appendix C).

Experiments

  • AntMaze-Large and AntMaze-Giant: CCSF raises coverage over CSF from 40.6% to 75.4% and from 12.8% to 39.5% (Table 1, Section 5.1).
  • METRA trained with chaining: coverage rises from 41.2% to 94.6% on the bottleneck maze (Table 1).
  • Sweep over the skill horizon HH on four mazes, with a Levy schedule that needs no tuning (Figure 3, Appendix D.2).
  • Mutual-information estimates over HH, which confirm the analysis: shorter horizons give more controllable skills (Figure 3, left).
  • Locomotion: explained the lower coverage in open space, since a random walk spreads as T\sqrt{T} and a held skill as TT. In contrast to complex environments where chaining shines(Section 5.2).
  • Locomotion: connected the higher zero-shot goal-reaching performance to the more controllable skills predicted by Section 4.3 (Figure 4).
  • Craftax: retrained every method from scratch and added DIAYN, DISCO-DANCE and CSF as baselines (Table 2).
  • Craftax: new evaluation procedure, with all methods evaluated on the same 500 worlds (Section 5.3, Appendix G).
AuthorsOpenReview ↗

Thank you Reviewers vg5F, 3NyA and 2RNS

Closing summary

We thank the reviewers for their thorough reviews and prompt feedback during the rebuttal. We believe the manuscript is stronger thanks to the reviewing process, after addressing most of the concerns raised and with several added ideas of our own along the way.

We hope the reviewers see the value of the contribution:

  • A simple yet effective change that improves the exploration of skill learning methods, motivated by random walks.
  • Chaining also shifts the training objective towards a lower bound on empowerment, an effect we validate empirically.
  • Strong results against state-of-the-art skill learning methods: 9.62 of 22 achievements on Craftax-Classic with no task reward, the highest we are aware of for an unsupervised method, large coverage gains on the Ant mazes, and stronger zero-shot control on state-based locomotion.

New results (reported in our reply to Reviewer 3NyA, not yet in the PDF)

AntMaze-Large, state coverage (3 seeds):

MethodCoverage (%)
DISCO-DANCE18.1 ±\pm 1.3
CSF40.6 ±\pm 22.2
CCSF75.4 ±\pm 20.2
METRA Chain (new)81.2 ±\pm 5.4

Pixel-based Quadruped (3 seeds, 5M steps):

MethodState coverageZero-shot goal reaching (staying time fraction)
METRA*≈ 300≈ 0.25
METRA Chain (new)252.7 ±\pm 94.30.275 ±\pm 0.087
CSF*≈ 240≈ 0.18
CCSF (new)223.5 ±\pm 27.60.16 ±\pm 0.064

*Read from the training curves in the CSF paper , we use the same repository and hyperparameters.

Chaining trains stably from pixels for both methods, with lower state coverage, as expected in open space (Section 5.2), and zero-shot goal reaching comparable to the unchained methods.


Clarifications

  • We scaled back the generality claim: chaining is applicable to methods whose skills are latent directions, and we validate it on CSF and METRA.
  • We clarified the nature of the contribution in our reply to Reviewer 2RNS: a simple change to how skills are sampled, with an explanation that makes falsifiable predictions which we verified empirically, and we discussed its relation to the concurrent work Scaling Empowered Agents.

Commitments for the final version

  • Add the results above to the paper.
  • Add unchained METRA on AntMaze-Large and AntMaze-Giant, including downstream performance.
  • Run the METRA and CSF baselines on pixel-based Quadruped ourselves.
  • Run CCSF and METRA Chain on other pixel-based environments from METRA and CSF, such as Humanoid.
  • Scale back the generality claim from all latent-direction methods to being validated on CSF and METRA.
  • State the limited comparison to skill-composition methods as a limitation.

Thank you again for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.