acceptodds
Under review as a conference paper at ICLR 2027

Object-Centric Unsupervised Skill Discovery with Action-Influence Weighting

Abstract

Unsupervised skill discovery learns reusable behaviors without extrinsic task rewards. In multi-object environments, however, conventional skill representations typically encode behavior in a global state space, leaving unspecified which object a skill is intended to affect. This ambiguity can cause skill learning to favor easily induced changes in the agent or other state factors rather than meaningful changes in a target object. We address this problem with an object-centric skill representation , where the discrete variable selects a target object and specifies its continuous skill vector. Furthermore, we weight the object-level intrinsic reward by a causal action influence, which quantifies the conditional influence of actions on the target object's next-state distribution given the full current state. The resulting intrinsic objective focuses distance-maximizing skill learning on target-object state changes that are more controllable by the agent. Experiments across multiple object-interaction environments show that the proposed method improves object-state-space coverage, target-object adherence, and the utility of learned skills for downstream control compared with existing unsupervised skill discovery methods.

Primary area:
reinforcement learning
Keywords:
unsupervised skill discovery, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MKo3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new method for object-centric unsupervised skill discovery. Their main idea is to apply temporal distance-maximizing skill discovery (METRA) to each object. Specifically, they first assume that the state space consists of N factors, and apply the METRA formulation to each of the N components, weighted by a causal action influence estimate defined by I(A;Si′∣S)I(A; S_i' \mid S). They show that the resulting method, called OCUSD, outperforms previous object-centric and general unsupervised skill discovery algorithms on Elden Kitchen, Playground, and Fetch.

Critical strengths
  • The paper is clearly written and easy to understand.
  • The experiments are reasonably thorough. The authors show that their proposed method outperforms previous baselines across 10 environments. The ablation studies seem reasonable too.
Critical weaknesses
  • To my understanding, the main novelty of this paper is action influence weighting (because the remaining part is a rather straightforward application of METRA to individual objects). However, I found this formulation to be too heuristic and not convincing enough. For a fixed i∈{1,2,…,N}i \in \{1, 2, \ldots, N\}, suppose that we want to learn a skill zz that pushes the object to the right. Then this skill should be able to (1) approach the target object, and (2) push the object to the right. However, CAI will only upweight the second part because the mutual information is zero before the arm has contact with the object. Please correct me if I've misunderstood anything. Although it might still work well empirically despite this sort of pathology, I think the formulation is too heuristic to be widely applied to diverse scenarios.
  • Moreover, even if this weighting scheme hasn't had this pathology, I would have still been concerned about the significance of the proposed method. The core methodology is identical to METRA (except for its per-object application) and the method is based on a rather strong assumption that the state space provides ground-truth object segmentations. This assumption does not seem to hold in practice in general, as we don't typically have ground-truth information about the objects in robotics.
Questions for authors

I don't have any other questions than the one asked in the weaknesses section.

Reviewer_9mk7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a skill discovery method, OCUSD, for multi-object environments. The target is to learn object-specific skills. OCUSD explicitly specifies the target object with an additional discrete skill variable, and uses the proposed causal action influence measure to weight the intrinsic reward among objects. Experiments are conducted on ELDEN Kitchen, Playground, and Gym-Fetch, showing that the skills learned by OCUSD have higher downstream performance and that the learned behaviors follow the target object specified by the object latent.

Critical strengths

The paper is well-written. The methodology is easy to understand.

Critical weaknesses
  1. The proposed method is very similar to SUSD. Both use DSD as a backbone, learn a transition model for each factor/object, and design a weight function using the learned factor/object-specific transition model. However, this similarity is not thoroughly discussed and evaluated.
  2. OCUSD only ensures that the learned skills can manipulate the specific object i when the object latent g is ei. However, there is no mechanism to control the behavior of other objects when manipulating object i. Therefore, it is not guaranteed that the object latent g can specify which object a skill is intended to affect, as other objects can also be affected.
Questions for authors
  1. OCUSD selects a one-hot latent g to specify an object and then selects a continuous latent z to specify the skill. I am confused why this cannot be directly achieved in SUSD and DUSDi. SUSD and DUSDi factorize the continuous skill latent z into (z1, z2, ..., zn), each corresponding to a state factor (object). If we mask the skill latent factors for nontargeted objects, e.g., use z=(z1, 0, ..., 0) to specify skill z1 for object 1, it seems we can directly achieve the same effort as OCUSD.
  2. What is the functional difference between CAI weighting and the transition-likelihood-based weighting used in SUSD? Though formulated differently, I find both of them are determined by the state transition. Can you specify some scenarios where CAI weighting and the transition-likelihood-based weighting perform differently?
  3. How will OCUSD perform if replacing the CAI weighting with the transition-likelihood-based weighting used in SUSD?
  4. How will OCUSD perform if using the same factorized skill space (z1, z2, ..., zn) as in SUSD instead of using the proposed (g, z)?
  5. In Algorithm 1, when training OCUSD, the object latent g is fixed in each episode. That means OCUSD is only trained with trajectories where a single object is allowed to move. Can the skills learned by OCUSD move multiple objects at the same time? For example, if we set g=(1,1,0,...), can the skill move both the first two objects? If not, this can be a weakness compared with DUSDi and SUSD.
  6. In Fig.2, why do DUSDi and SUSD perform worse than other baselines that are not designed for factorized MDPs? This is quite different from the result reported in the SUSD paper (Fig. 3 in the SUSD paper).
  7. Can you compare the performance of OCUSD, DUSDi, and SUSD in Multi-Particle and 2D Gunner? Multi-Particle and 2D Gunner are used in both DUSDi and SUSD, but are not discussed in this paper. Adding these experiments can help better evaluate OCUSD.
  8. Can you provide code to reproduce the experiment?
  9. (minor) Page 5, lines 241-242, the nonnegativity of CAI is obvious from the nonnegativity of KL divergence. There is no need to establish that with a proposition.
Reviewer_QremOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work uses a representation that has an object selector g, which is a discrete variable, combined with a continuous z variable to represent the skill. Then uses an intrinsic objective based on causal action influence (which is a mutual information term). The skill objective follows from METRA but is factorized according to the particular target object.

Critical strengths

The method modernizes the current methods for factored hierarchical unsupervised skill discovery

Critical weaknesses

The work appears to take ideas from SUSD, DUSDi, and SkiLD and modernize them by using meta-style skill learning instead of their discrete skill-learning strategies, while removing other components, such as interactions. While this is not a particularly novel advancement, it does not appear to already exist.

Experimental results should compare against SkiLD, since it has a very similar formulation, except for this method replacing the interaction metric with an action influence mutual information term, which is generally a coarser term. Ideally, comparison on the SkiLD environments would also make sense. A comparison against some simple variants incorporating ELDEN would also make sense.

The performance on the five Elden kitchen tasks seems somewhat obvious, since these methods can be optimized for these tasks and perform better than zero performance, at least for the factored versions of these algorithms. Considering that object-centric skill learning methods usually need optimization, I am wary of perhaps an unfair comparison being made.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.