- Rating
- 2(+1)
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses exploration in complex tasks by building on unsupervised skill-discovery methods like METRA and CSF and introducing a new approach called Chained CSF (CCSF). Unlike traditional methods that stick with one skill for an entire episode, CCSF resamples the skill every steps during training. This transforms each rollout into a random walk through different skills.
Critical strengths
- This proposed change is straightforward and easy to add to existing methods. Notably, if the method reduces to standard CSF, making it simple to directly compare CSF and CCSF.
- It's particularly interesting that CCSF enables progress deep into the Craftax technology tree without relying on extrinsic rewards.
Critical weaknesses
1. The paper's main contribution and positioning are unclear. While the introduction claims that existing skill-discovery methods struggle with exploring complex tasks and introduces random-walk skill chaining as a solution, the actual change amounts to a different skill-resampling schedule—without a new learning objective or supporting analysis. As a result, this feels more like an empirical observation than a principled learning advance. The paper often compares CCSF to METRA and CSF, which focus on learning a skill space, while CCSF builds on these for more challenging exploration. In this context, methods like EDL, UPSIDE, and DISCO-DANCE seem like more relevant baselines. Simply outperforming DISCO-DANCE in a single maze doesn't sufficiently establish the value of CCSF, especially without explaining why random-walk skill chaining works (see Weakness 2).
2. There's no clear analysis of why random walk skill resampling is helpful. The choice to uniformly resample skills every steps is mainly based on intuition. Since a random walk over skills is still undirected, it's not obvious why this should work better than more directed skill compositions. The evidence for the approach is limited to qualitative results from a single maze environment. A theoretical explanation of when and why random-walk skill chaining is effective would really strengthen the argument for its simplicity. The paper would also benefit from stronger experiments, including comparisons with other skill-selection strategies. For example, a more principled approach could use a high-level policy to select skills during pre-training, but the current method only trains such a policy for downstream tasks.
3. The experiments don't clearly back up the paper's claims, and some of the comparisons don't seem entirely fair. In particular, the experiments don't consistently test a clear hypothesis about why chaining skills should improve exploration.
- Direct baselines are missing—EDL and UPSIDE aren't included, and DISCO-DANCE is only tested on the maze. Also, only CCSF is tested with skill chaining in the main comparison. When chaining is added to DISCO-DANCE using the same protocol, the performance difference becomes much smaller.
- For Craftax, the experiments don't isolate the benefit of chaining from a credible baseline.
- Table 1 relies on baselines from prior work, but none of these are skill-discovery methods (see lines 352–358).
- The Peaceful variant removes death and adds a BREAK action, which raises questions about whether results from this modified environment are truly comparable. It would be better to compare CCSF against skill-discovery and exploration baselines on standard Craftax-Classic, and use Peaceful only for supplementary evidence.
- For locomotion tasks, CCSF actually achieves lower coverage and underperforms CSF on HalfCheetahHurdle. This phenomenon needs to be addressed and explained.
- The choice of benchmarks is also questionable. Figures 1–2 highlight exploration bottlenecks in continuous navigation, but only a small point maze is used to actually test this scenario. More challenging benchmarks like AntMaze, HumanoidMaze, or continuous manipulation tasks would provide much stronger evidence for the method. It's also not clear how the discrete Craftax environment supports the main argument.
4. There's limited ablation and no sensitivity analysis for . Although is the only new hyperparameter, each domain just uses a single value without exploring its impact. Running a sweep over is important to determine whether the observed improvement comes from chaining itself or from a specific resampling frequency.
Minor Weakness
- The abstract should make it clear that the Peaceful results come from a modified environment.
Questions for authors
Please see the weaknesses above.
9 comments
Issue 3: Baselines and Benchmarks
Issue 3.1: Need more direct baselines
W1 CCSF to METRA and CSF, which focus on learning a skill space, while CCSF builds on these for more challenging exploration. In this context, methods like EDL, UPSIDE, and DISCO-DANCE seem like more relevant baselines.
W3.1 Direct baselines are missing—EDL and UPSIDE aren't included, and DISCO-DANCE is only tested on the maze
- EDL does not work by itself on the bottleneck maze. They "were unable to explore this type of maze effectively with SMM ... rely on an oracle to simulate perfect exploration and evaluate the skill discovery/learning stages" (Section 4, page 7 of [1]). Hence, we did not include EDL as a baseline.
- "UPSIDE relies on the ability to reset to establish a root node for its growing tree" (Section 6, page 9 of [2]). Hence, UPSIDE requires episodic environments and will not work on Craftax.
- Furthermore, there does not seem to be open-source code for UPSIDE, and it is not trivial to faithfully implement and tune it due to the multi-stage tree-expansion procedure with many interdependent hyperparameters.
- However, we have implemented DISCO-DANCE and tuned it to the best of our ability for the 2D maze, Ant maze, Craftax and Craftax Peaceful.
- Lastly, DISCO-DANCE reports significant gains over UPSIDE in maze environments and DMC hence we were confident comparing directly to DISCO-DANCE
Issue 3.2: Need more difficult benchmarks
W1 simply outperforming DISCO-DANCE in a single maze doesn't sufficiently establish the value of CCSF, especially without explaining why random-walk skill chaining works
W3.4 The choice of benchmarks is also questionable.
W3.4 only a small point maze is used to actually test this scenario.
W3.4 challenging benchmarks like AntMaze, HumanoidMaze, or continuous manipulation tasks would provide much stronger evidence for the method.
- We agree and added the Large and Giant Ant Maze benchmarks from OGBench in Section 5.1 as challenging benchmarks, for stronger evidence of our method's improved exploration.
- As mentioned in Issue 3.1, the bottleneck benchmark is not a trivial benchmark for skill learning algorithms, even for the state-of-the-art methods CSF and METRA, as shown in Figure 1.
- We also believe that Craftax-Classic is an incredibly difficult and suitable benchmark, and we explain why below.
Issue 3.3: Why Craftax?
W3.4 not clear how discrete Craftax environment supports the main argument.
-
Fair point, the connection was not clear in the submission. We now make it explicit in Section 4.2, line 273.
-
Figure 5 also explains how the crafting tree in Craftax is strictly ordered with heavy dependencies. Exploring in such a bottlenecked enivronment is truly difficult. With many unsupervised RL methods with exploration bonuses tested in Section 5.3 making little progress (RND, ICM, E3B).
-
Craftax is a highly complex environment with CSF (Section 7, page 10 of [3]) claiming that it is not clear how to "scale the performance to increasingly complex environments such as Craftax".
-
To our knowledge, no skill learning algorithm has yet achieved significant results in Craftax without task rewards, so we see it as a clear gap and a good test of CCSF's exploration capacity.
-
The discrete action space does not change the method, as chaining only changes when skills are resampled.
Issue 3.4: Need more credible baselines in Craftax
W3.2 For Craftax, the experiments don't isolate the benefit of chaining from a credible baseline.
W3.2 CRAFTAX Table 1 relies on baselines from prior work, but none of these are skill-discovery methods (see lines 352–358)
W3.2 The Peaceful variant ... results from this modified environment are truly comparable. It would be better to compare CCSF against skill-discovery and exploration baselines on standard Craftax-Classic ...
- As mentioned, no skill learning algorithm has managed to have any form of success with Craftax. However, we do agree that more baselines would concretely show this.
- We implemented DIAYN and DISCO-DANCE on both Craftax-Classic and Craftax-Peaceful, though these skill learning methods are inherently not suited for such difficult environments with random starts, a stochastic environment with moving enemies, terminating states due to death and a constant flux of the state space (energy, hunger, thirst).
- We added a quick note in Section 5.3.1 to provide some intuition as to why these methods do not work in Craftax.
- C-TeC and the other exploration-based planners we compared against in Section 5.3 directly test the exploration capacity of CCSF.
- C-TeC is also the strongest unsupervised baseline for Craftax-Classic that we know of at the time of writing.
- Testing CSF and CCSF on Craftax and Craftax Peaceful explicitly isolates the benefit of chaining against CSF.
Issue 3.5: Peaceful variant is not comparable
W3.2 The Peaceful variant removes death and adds a BREAK action, which raises questions about whether results from this modified environment are truly comparable. It would be better to compare CCSF against skill-discovery and exploration baselines on standard Craftax-Classic, and use Peaceful only for supplementary evidence
- We agree that standard Craftax-Classic should be the main comparison. It is our main result in Section 5.3, and we have now added skill-discovery baselines there (CSF, DIAYN and DISCO-DANCE) alongside the exploration baselines.
- We keep Peaceful as supporting evidence, as you suggest. Its purpose is to isolate the exploration difficulty of the crafting tree from survival.
- The modifications make the environment easier for every method, as there is no survival pressure or termination due to death. We expected this to help the other skill learning methods, but CSF and DISCO-DANCE still did not progress far, even with extensive tuning.
- The BREAK action is available to every method. It only removes a placed table, furnace or plant, so that an agent cannot wall itself in over a long episode.
Issue 3.6: Performance
W3.1 CCSF is tested with skill chaining in the main comparison. When chaining is added to DISCO-DANCE using the same protocol, the performance difference becomes much smaller.
- Yes, the scores of DISCO-DANCE and all the other methods did improve after chaining. However, they are still appreciably lower than the performance of CCSF on the 2D bottleneck maze.
- The additional Ant Maze in Section 5.1 and the implementation of DISCO-DANCE for Craftax in Section 5.3 further demonstrate the gain in CCSF's exploration capability
W3.3 For locomotion tasks, CCSF actually achieves lower coverage and underperforms CSF on HalfCheetahHurdle. This phenomenon needs to be addressed and explained.
- This result is a property of random walk's slower diffusion in a freely traversable state space vs a ballistic trajectory (CSF). We add this explanation in Section 5.2
- Because of this, we observe that CSF learns skills that move as fast as possible by spinning forward in one direction, which is particularly suited to the Hurdle task and not to the zero-shot goal task.
- We show the performance gain in zero-shot goal task for HalfCheetah in Section 5.2 where "staying time fraction" of CCSF is 28.3% which is significantly higher than the 10.0% of CSF.
Thank you! Your feedback was very helpful in strengthening the paper.
Due to the number of questions and comments, we will group and answer them together if possible. We will address more specific questions separately.
Issue 1: The contribution is unclear
We have revised the framing with edits in the Abstract and Introduction. The key idea in short:
Our contributions are:
Issue 2: Theory and analysis
We strengthen our theoretical grounding in a few ways.