acceptodds
Under review as a conference paper at ICLR 2027

Controllable Skill Discovery with a Skill-Linear Critic

Abstract

Unsupervised skill discovery (USD) learns diverse behaviors without task rewards, but those behaviors are not necessarily controllable. Existing methods can obtain diversity through small state changes or varied locomotion directions, yet varying their skills can move state properties together or switch them among a few extremes. We ask whether the skill vector can provide a controllable interface for bounded state descriptors such as joint angles. We use controllable to mean that one descriptor can be varied gradually (graded control), other descriptors can be held approximately fixed (isolated control), and multiple descriptor targets can be achieved together (composable control). We propose the Skill-Linear Critic (SLC), which retains a mutual-information-based intrinsic reward but restricts skill dependence to a linear combination of skill-independent basis critics. This turns the skill vector from an arbitrary conditioning input into a coefficient vector for a skill-independent family of critics. We add a pairwise orthogonality penalty between the basis embeddings to mitigate observed critic-training instability. In static-pose evaluations of four simulated robots, SLC yields graded, isolated, and composable joint-angle behavior on HalfCheetah, Ant, and Humanoid. It is also the only evaluated learned pipeline with a reliable affine fit in a majority of evaluations on every platform, although this all-platform conclusion is threshold-sensitive for Franka.

Primary area:
reinforcement learning
Keywords:
Unsupervisded skill discovery, skill composition, reinforcement learning, disentangled control, linear critic, mutual information, robot control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o5GAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an unsupervised skill discovery method that learns skills for controlling specific state components by representing skill critics as linear combinations of basis critics. It also introduces an orthogonality loss to stabilize critic training. To evaluate the method, the authors introduce metrics that characterize the relationship between skills and state components. The results provide evidence of improved controllability on benchmark tasks compared with baselines. The authors also evaluate the method on a simulated Franka robot, demonstrating partial controllability.

Critical strengths
  • Building on the observation from prior work that distinguishable skills are not necessarily controllable, the paper proposes a method to address this gap and empirically evaluates its effectiveness through analysis of the learned policies.
  • The metrics for evaluating the proposed algorithm (Section 4) are well designed to clearly capture what they aim to assess and are reasonable.
Critical weaknesses
  • Unclear motivation

    • As a reviewer and researcher in this field, I understand unsupervised skill discovery as the discovery of meaningful skills through autonomous exploration without external rewards. Here, “meaningful” may refer to skills that are difficult to achieve in terms of temporal distance (METRA [1]), skills that are diverse in a linguistic context (LGSD [2]), or skills that maximize mutual information (DIAYN [3]). However, it is difficult to see the significance of discovering skills that independently control individual joint angles, as proposed in this paper. If independently controlling individual joints is itself the objective, what additional benefit does learning skills provide over controlling the joints directly using a PD controller or direct action commands?
  • Difficulty following the paper

    • Terms and expressions that admit multiple interpretations are introduced without sufficient context or concrete examples. Examples include “direct control interface,” “non-target descriptors,” “platform-selected,” and “geometric descriptor.” This makes it difficult for readers to readily understand their intended meaning.
    • Figure 2 is particularly difficult to interpret. The legend overlaps with the schematic, and the annotations are densely packed. It is unclear what the arrows and lines refer to, making the underlying geometric intuition difficult to grasp.
  • Concerns regarding the experiments and interpretation of the results

    • The fact that the Q-function is linear in the skill variable does not, by itself, imply that joint angles will vary smoothly, that other joints will remain fixed, or that multiple joint targets can be achieved simultaneously. Orthogonality regularization does not guarantee these behaviors either. The paper lacks a theoretical or intuitive explanation of why these effects arise and under what conditions they can be expected.
    • The experimental settings are overly simple. Evaluating HalfCheetah and Humanoid in fixed poses and discovering skills that merely correspond to independent joint control are substantially different from the complex, temporally extended behaviors generally studied in the field of unsupervised skill discovery. These results therefore provide only limited evidence that controllable skills can be learned under more general robot dynamics.

At this stage, I lean toward rejection for the reasons outlined in the weaknesses above and the questions in the following section.

[1] Park, Seohong, et al. "METRA: Scalable unsupervised rl with metric-aware abstraction." 2024.

[2] Rho, Seungeun, et al. "Language guided skill discovery." 2025.

[3] Eysenbach, Benjamin, et al. "Diversity is all you need: Learning skills without a reward function." 2018.

Questions for authors
  • What additional benefit do the learned skills provide over a PD controller or direct action commands for independent joint control?
  • Why should a critic that is linear in the skill variable, together with orthogonality regularization, lead to smooth joint control, limited changes in non-target joints, and simultaneous achievement of multiple joint targets?
  • Do the learned skills remain controllable during dynamic, temporally extended behaviors beyond the static-pose settings evaluated in the paper?
Reviewer_5EMsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to address the question of how to discover controllable skills. To this end, the authors propose Skill-Linear Critic (SLC), which constrains the critic to be a linear combination of skill-independent basis critics, weighted by the skill vector. The authors demonstrate that SLC helps learn controllable skills in terms of graded, isolated, and composable control of joint angles. Through experiments with simulated robots, they show that the proposed method improves controllability over baseline unsupervised skill discovery algorithms.

Critical strengths

I think the issue raised by the authors is timely and important. Controllability is an essential property for making learned skills fully reusable, yet it has received relatively little attention in recent works. I believe this paper addresses an important problem.

Critical weaknesses

I have several serious concerns about this paper.

1. Simple PD control could achieve the same outcome.

The final outcome of the proposed method is controlling individual joints through a skill interface. However, I cannot fully understand why these learned skills are needed. Couldn't the same outcome be achieved with a simple PD controller (or even direct torque control)?

Since the learned skills are very low-level, a user could simply provide a desired target angle for each joint and use a PD controller to track it. PD target control is a standard action interface in robotics, and I believe the proposed metrics, such as Valid, Iso, Gap, Err_in, and Success, could already be satisfied through this simple interface. In that sense, PD control is already controllable.

I believe "controllability" in skill discovery should mean something more. Learned skills should provide a high-level interface so that users do not need to care about individual joint angles. Users should be able to solve new tasks using the learned skills without knowing such low-level details. This is the level of controllability I would expect from this line of research. I find the scope of controllability in this paper too narrow, making it difficult to see the practical advantage over a conventional PD controller.

The authors may find it useful to consider a recent work that directly addresses human controllability, enabling users to control a humanoid with only six discrete buttons [1].

[1] Rho et al., Game-Guided Skill Discovery through Self-Play for Playable Agent Control, 2027. https://arxiv.org/abs/2609.40137

2. Unclear connection between the skill-linear critic and controllability.

Furthermore, I do not think the paper adequately explains the logical connection between SLC and controllability. Why should making the critic linear with respect to the skill vector lead to more controllable skills? What is the underlying mechanism that encourages graded, isolated, and composable control?

I believe the authors should address this logical gap, as it is central to the paper's main claim. At the very least, an intuitive explanation in Section 3.2 would be helpful.

3. Overly simplified experiments.

An essential aspect of skill discovery research is demonstrating the usefulness of the learned skills. Are these skills reusable? What downstream or novel tasks can be solved using them?

If the authors claim that the learned skills are controllable, they should demonstrate their usefulness by allowing users to directly control the agent through the learned skill interface. Alternatively, the authors could train a high-level controller using the discovered skills and demonstrate improved sample efficiency or task performance.

Without such experiments, I find it difficult to assess the practical value of the proposed method beyond controlling individual joint angles.

Questions for authors

I would appreciate it if the authors could address the concerns raised in the Weaknesses section, particularly regarding the practical advantage over conventional PD control, the connection between the skill-linear critic and controllability, and the usefulness of the learned skills for downstream tasks.

Reviewer_rXHSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the skill vector of an unsupervised skill-discovery policy can act as a control interface for joint angles: changing one angle smoothly, holding the others fixed, and reaching several targets at once. SLC keeps the DADS reward and SAC but makes the critic exactly linear in the skill, Qα(s,a)=∑kαk⟨ψk(s,a),w⟩Q_\alpha(s,a) = \sum_k \alpha_k \langle \psi_k(s,a), w \rangle, with a pairwise orthogonality penalty for stability. The evaluation fits an affine skill-to-descriptor map and keeps only cells with R 2 ≥ 0.75 R 2 ≥0.75. It then uses that map to test isolation, smoothness, and composition. On four static-pose robots, SLC with a per-platform penalty weight is the only learned method that passes this gate on every platform, with 3.5–7.4% composition error everywhere except Franka.

Critical strengths
  1. A clear, testable definition of controllability. USD papers rarely check whether an intermediate skill gives intermediate behavior. The three criteria and their protocol could be reused to evaluate other methods.

  2. Careful, honest experiments. The paper includes reference controllers, gate-threshold sweeps, and many ablations. It also reports its failures openly, and the intercept ablation (11 of 12 cells fail) is especially informative.

Critical weaknesses
  1. The testbed and the evaluation are both built around linearity, which is also the method's bias. Joint stiffness was chosen to make pose linear in torque (App. A.3), and both the gate and the composition test assume an affine map. A training-free controller a = M α a=Mα beats SLC by far (iso 381 vs. 4.1 on HalfCheetah, Table 10), and without the ankle spring SLC also fails the gate (Table 5). The protocol is also bespoke: the 0.75 gate is calibrated on synthetic sine and sawtooth responses, gap is normalized to a sampled sine, and composition depends on a first-SVD null-space fill. Since the protocol changed during the project (App. C.1 mentions pilots under an earlier protocol), these choices raise a forking-paths concern. The results therefore show affine behavior where the physics makes it easy, not controllability in general. This drives my rating.

  2. The penalty lacks a justification, and the comparison lacks controls. The paper's original Fisher-orthogonality motivation for the penalty does not hold (App. C). The penalty also does not fix the scale non-identifiability that Sec. 3.3 cites, and the weight selected on HalfCheetah is 0. It is therefore a heuristic stabilizer and should be compared with standard ones, such as a fixed ∣w∣|w| or LayerNorm. Its weight is tuned per platform (0, 1, 1, 100) on seed-0 iso, the evaluation metric itself, while the baselines are not tuned. With the value of 1 used on Ant and Humanoid, Franka fails the gate (Table 13). The SLC critic also has 3–9× the parameters of a baseline critic, yet no parameter-matched DADS critic is tested. Most ablations use a single seed, often from older batches. Without these controls, it is unclear how much of the gain comes from the skill-linear structure.

Questions for authors

Do the results hold with a parameter-matched DADS critic, a single shared λ ⊥ λ ⊥ ​ , and a standard stabilizer in place of the penalty?

Does SLC still win with default joint stiffness, or with a nonlinear inverse for composition?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.