acceptodds
Withdrawn from ICLR 2027

EXPAND: Extensible Policy Capacity for Unsupervised Skill Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unsupervised reinforcement learning pretrains agents without task rewards for downstream learning and adaptation. Active exploration broadens state coverage but often lacks a reusable skill interface; skill discovery learns controllable behaviors whose coverage can plateau. Fixed policy capacity may limit exploration even as intrinsic objectives and representations improve. We introduce **EX**tensible **P**olicy c**A**pacity for u**N**supervised skill **D**iscovery (EXPAND), a framework that adds skill-conditioned cascade stages during pretraining. Stages train jointly under the original objective; gates conditioned on state and skill recursively combine action distribution parameters. EXPAND increases METRA's final historical coverage by **71.1%** on average across seven environments and generally outperforms parameter-matched wider, deeper, and parallel gated policies. EXPAND also improves coverage with LSD and DIAYN, with gains varying by environment and metric. EXPAND also improves METRA's overall zero-shot goal reaching and hierarchical control with frozen skills. Applying the same architecture to LSD improves coverage in three environments. The skills learned with EXPAND applied to METRA also improve overall performance in zero-shot goal reaching and hierarchical control with frozen low-level skills. These results show that organizing policy capacity as a cascade improves exploration coverage and supports downstream skill reuse in the evaluated settings.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, unsupervised learning, skill discorvery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hpjLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper applies an ensemble reinforcement learning method built on Cascade to the setting of unsupervised RL as a plug and play mechanism to improve existing unsupervised RL algorithms. They mainly focus on using it on top of METRA. They also introduce and prove some results related to the method and motivations behind it. Their experiments test the method in environments from OGBench and DeepMind Control Suite. They test mainly for coverage, but also for downstream use of learned skills in zero-shot goal reaching settings and for hierarchical control. They find benefits of using their proposed approach. They also present auxiliary experiments investigating the impact of some choices in the method and on its learned behaviors.

Critical strengths
  • I think evaluating the method for coverage, zero-shot capability after pretraining, and hierarchical use of learned skills is a good choice.
  • The paper reports improvements from the proposed approach across several environments.
  • I appreciate the inclusion of auxiliary results for ablations and sensitivity studies.
Critical weaknesses

I see several weaknesses with the paper that lead to my recommendation to reject.

Limited novelty.

The paper’s main focus is on applying an existing ensemble method (which previous work already tested in other RL problems) to a specific RL setting (unsupervised RL) with small adaptations. I think this is a limited novelty. I also don’t see the paper presenting enough arguments for this setting being special in how much it can benefit from such an approach compared to other RL settings. With respect to the mathematical propositions the paper introduces, I don’t find them to be sufficiently interesting in and of themselves. I don’t find them particularly valuable nor do I think the paper convinces of such value.

Presentation and clarity.

I think the paper’s presentation could be significantly improved.

Experiments.

  • I think the results could be organized by the research question they address. The way the results are presented makes the links to the 4 research questions the paper introduces harder to determine.
  • Since the contribution is EXPAND, I would like to see more comparisons between methods and their + expand version. Several methods appear without their + expand extensions.
  • In terms of organization, figure 4 and Table 1 seem to address RQ1. I think it would be better if their results were discussed jointly. Why separate them?
  • While it is good that there are parameter-matched controls for METRA, these are limited. For many results, it is not clear what benefits are from the ensemble approach the paper focuses on vs simply increasing the capacity of the policy.
  • I think some of the ablations could be moved to the appendix to better discuss the main results. The experiment sections seem very compressed.
  • I did not find reporting on the hyperparameter tuning/selection procedure used for the method and baselines.
  • With respect to the comparison in Figure 8. Since all the components keep on being trained as more stages are added, the question it addresses is different from asking how many stages should be used or how varying the number of stages would affect coverage. How would you compare what the experiment addresses versus testing running the whole pre-training with different numbers of total stages?
  • Minor: The meaning of History, Replay, and Policy coverage should be explained in the main text in my opinion.
Questions for authors

None

Reviewer_QNKgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper "EXPAND: Extensible Policy Capacity for Unsupervised Skill Discovery" grounds in the Cascade architecture to define a ensemble of skill-conditoinned policies trained to maximize state space coverage.

Critical strengths
  • Interesting results. Using a cascade architecture looks helpful for exploration.
Critical weaknesses
  • Presentation: I spent a very hard time on this paper, as it stays very vague in many places. It presents very few components of the method. First, it strongly relies on the Cascade paper, reusing what they call a cascade, in fact a neural network architecture where networks are sequentially combined with gating mechanism, without never really defining it (only implicitly in (5) and (6)). In introduction and section 3, authors talk about a Cascade without specifying what it is at all (only a circular definition in section 3 : " a cascade stage is a policy component in Cascade" !!). Next, I understand that paper wants to stay general enough to be combined with any skill discovery algorithm (this was not clear to me before the xp section though), it would have been very usefull to give a general algorithm (more detailed than the one in appendix C3 which is particularly unclear) that shows sampling and use of z dureing training, and at least an exemple of concrete implementation (whith METRA or DYAN for nistances). Section 5.2 is not useful in the current form (see questions below). Even evaluation metrics are not defined. Coverage is not formally defined. And also its Policy, Replay and History implementation. The reader must infer a very large amount of core components of the analysis. I feel the paper needs a very important rewritting effort during rebuttal in order that we could accept it for publication (I set a very low score for that reason mainly, I can raise it if the paper is reworked with more pedagogy - changed parts of the text could be given in color in the revision).

  • Innovation : I cannot really see the innovative contribution in that paper. The architecture is the one from the Cascade paper, the skill discovery process is not defined, only borrowed from any existing discovery method. Is the innovation relying in the application of such kind of progressive architecture in the skill discovery field ? The paper should emphasize further that this is not easy.

  • Missing litterature and baselines: I feel the paper could consider more closely related work from the sequential construction of explorers. Hierarchical RL methods should be at least discussed. Chaining approaches or architectures like Progressive neural networks look to fall in the same spirit than the proposal. The sequential approach Direct then Diffuse is another candidate approach the related work should discuss (and maybe the experiments could consider). The experiments only considers the proposed aggregation approach with different skill discovery algorithms, but many other attempts proposed ensembles or sequences of policies.

  • Finally, starting with all components of the cascade look to obtain similar results than a constructive approach that intoduce each agent the one after the other.

Questions for authors
  • What is the purpose of section 5.2 ? Please be more pedagogical and give the reader some explanations of the implications of these theoretical findings.

    • I did not well understand prop 2 : the setting n^Tu>b looks very specific. Authors present it as general while it is an assumption for the study. It only holds when the effect of an action if affine right ? Also, what meant the probability of crossing before the expansion ? If it crosses the unvisite region before the expansion, thus it is not an unvisited region. More importantly, what do we learn concretely from this proposition ?
    • For me prop 3 does not show what it claims it does. In fact conditionning to an auxiliary variable cannot be detrimental to the optimum, whatever the approach... what should be studied is the sign change of D under two different skills z.
  • When you compare e.g. Metra vs Metra+EXPAND, do you consider an architecture for METRA that has a similar numbe of parameters ?

  • I cannot find the results of the announced Flat Gaussian Aggregation test.

Reviewer_43z8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EXPAND, a plug-in framework that increases policy capacity during unsupervised skill discovery by progressively adding gated cascade stages to a skill-conditioned policy while keeping the underlying intrinsic objective and skill interface unchanged. Extensive experiments across seven environments show consistent coverage gains over METRA, as well as paired gains for LSD and DIAYN on most settings.

Critical strengths
  1. Introduces a principled, capacity-expansion mechanism for skill-conditioned policies via a sequentially gated cascade that preserves the base algorithm’s intrinsic objective and skill interface.
  2. The cascade design interpolates action distribution parameters rather than mixing densities, offering a distinct alternative to parallel mixtures or naive widening/deepening.
  3. The paper explores a factor that has not been fully studied yet but is crucial in unsupervised skill discovery. That is, how to enhance the strategic ability without changing the core objective.
Critical weaknesses
  1. The theoretical results, while helpful, are mostly local or standard (TV coupling bound) and do not provide global guarantees that expansion improves long-horizon coverage beyond the local one-step model.
  2. The parameter composition interpolates Gaussian parameters rather than forming a mixture distribution. This precludes explicitly multimodal action distributions and could bound expressivity relative to MoE.
Questions for authors
  1. What happens to skill identifiability/semantics across expansion points? Can you quantify drift in p(s|z) (e.g., MI, Wasserstein distance, or skill-conditioned occupancy overlap) to show stability or controlled evolution of skills?
  2. Can EXPAND be combined with mixture-of-density gating (i.e., a true mixture model) to allow multi-modality, and if tested, how did it behave relative to parameter interpolation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.