acceptodds
Withdrawn from ICLR 2027

Returning Skill Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unsupervised skill discovery (USD) learns a repertoire of distinct behaviors without external reward, and across every major approach the learned skills are one-way: a skill leaves its starting state and stops wherever it ends, because nothing in the objective rewards coming back. We first show this is structural and then introduce Returning Skill Discovery (RSD), a method-agnostic add-on that makes skills return without changing the base objective. RSD adds three pieces: a phase input, without which a skill that retraces its path is provably inexpressible; a reward weight that flips sign at mid-episode, under which the optimal skill travels out for half the episode and home for the other half; and a dimensionless return constraint that catches everything the weighted reward cannot see. The weight is the same for every method, and a checkable property of the reward decides only what it multiplies. Paired with five USD methods (DIAYN, DADS, LSD, CSD, METRA) on locomotion and manipulation, with state and pixel inputs, RSD produces skills that return to their starting position and posture, chain zero-shot, and outperform one-way libraries as long-horizon options. Videos of returning skills are at https://sites.google.com/view/rsd-usd.

Primary area:
reinforcement learning
Keywords:
RL, unsupervised skill discovery

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EtAoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an additional module that enables skill discovery methods to learn skills that return to the initial state at the end of an episode. The proposed method applies to a wide range of SD methods. The methodology mainly involves adding phase information to the policy's input to ensure that the policy is aware of the current phase, and flipping the reward sign after the midpoint of the trajectory to reward returning behavior. Other techniques like return constraints are used to ensure the learning of returning skills. Experiments are conducted on a wide range of domains and base algorithms, demonstrating that RSD can learn returning skills and faithfully showing the advantages and weaknesses of the learned skills.

Critical strengths
  1. The problem is new and interesting.
  2. The proposed method is easy to understand and applies to both DSD methods and MI-based SD methods.
Critical weaknesses
  1. The proposed method introduces many hyperparameters and seems to be sensitive to hyperparameter tuning, which makes it hard to use in practice. As shown in Table 3, every environment and base algorithm requires different RSD parameters.
  2. Page 3 lines 147-150 say that "RSD leaves the base objective untouched, and what it changes is determined by whether that objective’s reward admits a potential in the sense of Equation (3)". However, on page 5 line 268-269, the authors states that they additionally phase-conditions the dynamics model of DADS. This suggests that RSD also requires individual changes for certain algorithms, which makes it not a method-agnostic add-on as claimed in the abstract.
Questions for authors
  1. How will RSD perform if using the same hyperparameters across all tasks and base algorithms? How will the performance change if these hyperparameters change?
  2. How will RSD perform if we flip the reward at other timesteps, like 1/4 or 3/4?
  3. The proposed algorithm does not seem to be applicable in scenarios that contain non-reversible parts. The forward and return dynamics may be asymmetric in practice. For example, it is easy to throw away an object held in the arm, but picking it up is more challenging. Such scenarios are revertable but hard to revert. How will RSD perform in such scenarios?
  4. Page 5, line 217, How is the O(1/T) derived?
  5. Why do we need returning skills if combining several non-returning skills can achieve the same result? A larger K, as stated in 5.2, is one reason. But a too-large K will degrade the hierarchical control into a process of first selecting then finetuning a certain skill, which is no longer hierarchical. How will RSD perform if using the same K as is used in the hierarchical control experiment in the original paper of the base SD algorithms?
  6. In Appendix A.2, prop.3, the authors make some assumptions on |Δft| and Δft*. However, these assumptions are not used in the proof. What are these assumptions for?
  7. (minor) Notation issues: e(tau) is defined in (5) but is not used. Kw in (7) is not defined. ξ in (9) is not defined (I see ξ∼π, but please explicitly define ξ with states and actions).
Reviewer_2wqaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Returning Skill Discovery (RSD), which aims to address a limitation of existing unsupervised skill discovery (USD) methods. The authors argue that existing USD methods tend to discover skills that continuously move away from the origin, which they identify as an undesirable property. To address this issue, the authors introduce three modifications: (1) Phase Lift, (2) Reweighting, and (3) Return Constraint. These modifications can be applied on top of existing USD algorithms. The authors integrate RSD with five different base USD algorithms and demonstrate that RSD enables agents to learn returning skills across both locomotion and manipulation environments.

Critical strengths
  1. I agree with the underlying problem identified in this paper. The objectives of distance-maximization-based skill discovery algorithms inherently conflict with learning periodic skills. The "returning skills" proposed in this paper can be viewed as a particular instance of periodic skills.

  2. The experiments across different embodiments provide empirical support for the proposed method. The results demonstrate that RSD can successfully discover returning skills in both locomotion and manipulation environments.

Critical weaknesses

I have several serious concerns about this paper, particularly regarding its problem formulation, novelty, and experimental design.

1. Why focus exclusively on spatially outward-moving skills?

The paper assumes that distance-maximization-based USD methods predominantly learn skills that move spatially away from the origin. However, this is not necessarily a general property of these algorithms; it largely depends on the state representation.

For example, when METRA and LSD are trained with vector-based state representations, the observations typically include global x- and y-coordinates. This is a major reason why these methods discover spatially moving skills. Most other state dimensions are effectively bounded, whereas global x- and y-coordinates are unbounded. Consequently, distance-maximization objectives tend to favor changes along these unbounded dimensions. If global positions were removed from the observations, these algorithms would not necessarily discover spatially outward-moving skills.

Therefore, the paper appears to address a specific instantiation of USD methods in which global positions are included in the state representation, rather than a general limitation of distance-maximization-based skill discovery.

More broadly, I do not see why the concept of "returning" should be restricted to spatial movements. Returning could also be defined in a learned latent space or other state dimensions. Restricting the discussion to spatial returning skills unnecessarily limits the scope and generality of the proposed framework.

I encourage the authors to conduct experiments without global coordinates and investigate what types of returning skills can emerge in such settings. This would help establish whether RSD addresses a fundamental limitation of existing USD algorithms or primarily a consequence of a particular state representation.

2. The problem statement is not novel.

The limitation identified in this paper has already been discussed in several prior works.

For example, Section 6 of Reference Grounded Skill Discovery (Rho et al., 2025) explicitly discusses how distance-maximization-based skill discovery methods struggle to learn repetitive motions. Similarly, Periodic Skill Discovery (Park et al., 2025) addresses a closely related problem and proposes a solution.

Therefore, the identification of this limitation should not itself be presented as a novel contribution of this paper. While the proposed solution may still be novel, the underlying motivation and problem formulation have already been explored in the literature.

At a minimum, the authors should discuss these prior works and clearly distinguish their contributions from those of RSD. More importantly, I believe that experimental comparisons against these methods are necessary to establish the advantages of the proposed approach.

3. The task design is heavily biased in favor of the proposed method.

I would like to see stronger evidence that the discovered returning skills provide advantages when transferred to downstream tasks that do not explicitly require returning behavior.

In particular, the second experiment in Section 5.2 is essentially another form of a returning task. The target alternates between a distant goal and the origin at every skill-selection interval. This experimental setup inherently favors skills that return to their starting positions. As a result, the experiment does not convincingly demonstrate that returning skills provide general advantages for downstream task performance.

Furthermore, the performance of baseline methods such as METRA and LSD could potentially be improved substantially through a simple modification: recentering the global x- and y-coordinates at the beginning of each skill execution. Specifically, the agent could observe its position relative to the position at which the current skill was initiated, rather than its absolute global position. This would prevent the policy from receiving increasingly out-of-distribution positional observations as the agent moves farther from the origin.

Without evaluating such a baseline, it is difficult to determine whether the reported improvements arise from the proposed returning-skill formulation or from the baselines' sensitivity to global positional coordinates.

I strongly encourage the authors to (1) evaluate downstream tasks that do not inherently favor returning skills and (2) include recentered versions of METRA and LSD as additional baselines. These experiments would provide more convincing evidence for the practical benefits of RSD.

Questions for authors

I would appreciate it if the authors could address the concerns raised in the Weaknesses section.

Reviewer_qFR4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper observes that skills learned by unsupervised skill discovery (USD) end far from their starting state and do not return. The authors identify two causes. Under distance-based objectives, a trajectory that returns to its start receives a total reward of zero (Prop. 1). In addition, a policy without time information cannot represent a trajectory that goes out and comes back (Prop. 2). The proposed RSD adds three components to an existing USD method, namely a time input to the policy, a reward weight that rewards moving back in the second half of the episode, and a constraint that penalizes skills that end far from their start. With five USD methods, RSD raises the fraction of returning skills from near zero to between 0.13 and 0.88.

Critical strengths
  • The same recipe applies to five base methods from both USD families. The only method-specific choice is which quantity the weight multiplies, and this is determined by whether the base return depends only on the start and end states. The experiments cover arms, locomotion, and manipulation with both state and pixel inputs.

  • The ablation in Fig. 5 removes every subset of the three components on two tasks and reports travel distance alongside the return rate. Every ablated variant travels about as far as the full method or farther and still fails to return. This rules out the explanation that RSD returns simply by moving less.

Critical weaknesses
  • RSD substantially reduces state coverage, a central goal of USD. On the six locomotion tasks, RSD keeps only 15 to 43% of the base coverage (Table 1), and its reach drops to roughly a quarter of the base (23 to 30% on the four state-based tasks, Table 8). The paper does not show a setting in which the benefit of returning outweighs this loss.

  • In the only downstream task with learning (Fig. 4, bottom), half of the goals are the origin, so half of the reward requires returning. Even in this setting, RSD exceeds the 0.5 bound of a one-way library only on HalfCheetah at K = 200 and 400, and it underperforms the base on pixel Cheetah. The chaining experiment (Fig. 4, top) mostly re-measures the return rate, since RSD skills end near their start by design. It therefore remains unclear whether the skills help on standard goal reaching or manipulation tasks, neither of which is evaluated downstream.

  • The paper is motivated by settings where resets are costly, but all experiments run in simulation with a reset at every episode, and no real-world results are reported. Returning is also limited in manipulation, where resets matter most. The return rate on FetchPush is 0.17, and in Kitchen the policy itself reverts only the light switch and the microwave door (Table 10).

Questions for authors
  • Beyond composability, what value does returning add that justifies the loss in reach and coverage? Is there a downstream task without origin goals in which RSD skills outperform base skills? Also, would doubling the training horizon, as in the kettle experiment (Appendix C.6), recover the base reach while keeping the return?

  • The paper attributes the failure of one-way skills after the first hand-off to training only from the initial state distribution. If so, the later base links in Fig. 4 (top) should travel close to zero, but on HalfCheetah with LSD they still travel about 0.1 to 0.4 of the first link. What evidence supports this explanation?

  • Why is State Cov. reported as n/a for PlanarArm, 6-link Arm, and FetchPush in Table 1? Could a comparable measure, such as end-effector or object-position coverage, be reported for these tasks and for the ablation in Fig. 5?

  • To reduce resets, a one-way skill could be followed by a retrace of its own path or by the skill in the opposite direction. Why must the return be learned within each skill? A comparison on locomotion, where retracing is possible, would clarify the benefit.

  • Has RSD been evaluated with reset-free training or on a real robot? If the start state of an episode is the end state of the previous one, how is R defined?

  • Could the authors define the metrics in Fig. 4? Is per-link travel the peak distance, the end-point displacement, or the path length? Appendix C.7 describes the goal schedule, but is the success rate the fraction of goals reached per episode, and what distance counts as reaching a goal? Reporting success separately for far and origin goals would also help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.