acceptodds
Under review as a conference paper at ICLR 2027

Data-dependent Kernel-Aware Unsupervised Skill Discovery

Abstract

Unsupervised reinforcement learning (URL) aims to train general agents by learning transferable knowledge through interaction without external rewards, then adapting to downstream tasks with fewer interactions. Skill discovery learns distinguishable, controllable skills with broad coverage through intra-skill consistency and inter-skill diversity. Temporal distance, however, does not explicitly adapt local similarity scales to the empirical state distribution. We propose **I**solation **K**ernel-aware **S**kill **D**iscovery (IKSD), which constructs data-adaptive similarity and geometric scales in a latent space. Online IKSD constrains representation changes with IK dissimilarity; offline IKSD learns representation distances from cumulative IK-weighted transition costs. In DMC Walker and Cheetah, online IKSD achieves the highest final mean valid-posture path length on challenging terrain and competitive position coverage on flat ground. Experiments on fixed MetaWorld and BiGym datasets show that offline IKSD improves downstream performance and solves some tasks outside the data collection set.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, unsupervised learning, skill discovery, isolation kernel

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4pwsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a distance-maximizing skill discovery method named IKSD that uses the estimated empirical state density as the distance measure. The distance metric here is the dissimilarity induced by the isolation kernel. Experiments are conducted in both online and offline settings, showing competitive performance in the online setting, superior performance in the online setting with challenging terrain, and in the offline setting.

Critical strengths

The proposed method applies to both online and offline settings.

Critical weaknesses
  1. The objective function of DSD (eq.1) is incorrect. In DSD, the constraint should be applied to an arbitrary state pair (x,y) instead of the adjacent states (s,s'). The authors should refer to the CSD paper for the correct DSD formulation. As a result, I am concerned that the implementation of IKSD may also be incorrect. As shown in Appendix C.5, the proposed IK dissimilarity only constrains the adjacent states (s,s').
  2. Many necessary details and explanations are missing in the current manuscript. Section 2.2 should be expanded. As the usage of Isolation Kernel dissimilarity is the core methodology of this method, the author should further explain what the Isolation Kernel is, how the partition is conducted, and why this is a "normalized Hamming pseudometric in [0, 1] between cell-index sequences". The computational overhead induced by the Isolation Kernel dissimilarity calculation should also be discussed.
Questions for authors
  1. In the experiment in Section 3.2, it seems that the reference set for IOU calculation is used in the IK calculation, while other baseline methods do not use that data. If a suboptimal reference set is used for IK calculation, will the performance of IK be largely affected?
  2. What is the functional difference between the proposed empirical state density and CSD’s conditional transition density? Why is the learned transition model in CSD not a good empirical state density? Can you specify a scenario where the two state densities behave differently, so that the readers can better understand your contribution?
  3. In Fig. 2, what do the dashed lines and the squares mean?
  4. The experiments in both sections 3.2 and 3.3 compare the top-k states measured by the empirical arrival probabilities and those measured by distance measures. Why are the experiments in sections 3.2 and 3.3 conducted on different environments?
  5. Can you explain in detail the relationship between the baseline measures in Section 3 and the distance measures used in LSD, CSD, and METRA?
  6. How is the "valid-posture path length per 10-second trajectory" calculated? The Appendix mentions that it accumulates distance traveled when the posture satisfies the validity criterion. However, I could not find the definition of the validity criterion.
  7. In Table 2, window open (one claimed seen task) and window close (one claimed unseen task) are almost identical. This might be the main reason for the superior performance on window close. If the window-open task is removed from the seen tasks, how will IKSD and the baselines perform?
  8. The number of partitions m and the size of the reference set seem to be important hyperparameters of IKSD. Is IKSD sensitive to these hyperparameters? The authors should conduct an ablation study for it, reporting the performance when changing these hyperparameters.
  9. (minor) Fig. 1 is hard to read. Could you please use more distinct colors to denote the three categories instead of three blues?
13 comments

Thank you for the detailed review and the suggestions on our formulation and experiments.

W1: DSD objective

The general constraint applies to every state pair:

∥ϕ(s′)−ϕ(s)∥2≤d(s,s′),∀s,s′∈S.\|\phi(s')-\phi(s)\|_2\leq d(s,s'),\qquad\forall s,s'\in\mathcal S.

Eq. (1) misstates its domain. Like METRA, CSD, and LSD, online IKSD learns from adjacent state transitions. Our implementation evaluates the constraint on transitions sampled from replay. Correcting the general definition leaves the implemented updates and reported results unchanged.

W2: Method details

New paper's Sec. 2.2 and Algorithm 1 describe reference encoding, partitions, shared-cell frequencies, and online/offline updates. Each representation update fixes the target network and partitions, stops kernel-cost gradients, then updates the target by EMA. For input dimension pp, center comparisons cost O(mψp)O(m\psi p) per state pair; computing similarity from cell indices costs O(m)O(m).

Q1: Reference data

We construct B\mathcal B from random walks starting uniformly over valid grid cells and evolving under the environment's transition matrix. Repeated visits and self-transitions remain in the pool. Six-step arrival probabilities define the Top-22 evaluation set and do not enter reference sampling or partition construction.

Mahalanobis and Adaptive-GK use the same B\mathcal B; Dyn-1 uses adjacent transitions from these trajectories, and Replay-temp uses their temporal order. New paper's Sec. 3.2 specifies these sources.

Q2: State density versus CSD

IK uses the empirical state marginal pD(s)p_{\mathcal D}(s) represented by replay: local sample concentration sets partition resolution. CSD learns the conditional transition density q(s′∣s)q(s'\mid s) under current skills. Its likelihood-derived cost gives larger constraint budgets to transitions that are harder to realize from a given state.

Q3: Fig. 2 symbols

Dashed circles show candidates' Euclidean distance ranges from the anchor; Large's inner and outer circles bound the candidate annulus. Boxes and connecting lines identify enlarged insets showing anchor and candidate positions.

Q4: Different environments

Both sections use arrival probabilities as the reference but test different properties. Sec. 3.2 uses a constructed two-room random walk and IoU to compare Top-kk set overlap. Its doorway geometry and stay probabilities are easy to specify, making the retrieved neighborhoods easy to visualize.

Sec. 3.3 tests candidate ordering in offline PointMaze-Umaze and PointMaze-Large trajectories. IoU ignores order within a set; NDCG evaluates whether states with higher empirical arrival probabilities rank earlier. The two settings test neighborhood membership in a constructed example and ranking in benchmark data.

Q5: LSD, CSD, and METRA

LSD uses Euclidean distance, METRA uses temporal distance, and CSD uses transition probability.

Q6: Valid-posture path length

Appendix B.3 defines a valid posture: the dot product of the torso's local vertical axis and the world vertical axis exceeds 0.7, and torso height above local terrain exceeds 0.8 m for Walker or 0.25 m for Cheetah. For a ten-second trajectory, let vtv_t indicate validity and xtx_t be horizontal torso position:

Lvalid=∑t=0T−11{vt∧vt+1}∥xt+1−xt∥2.L_{\mathrm{valid}}=\sum_{t=0}^{T-1}\mathbf1\{v_t\land v_{t+1}\}\|x_{t+1}-x_t\|_2.

Here TT is the number of transitions. We sum horizontal movement with both endpoints valid, including revisits, then average over evaluation trajectories. Units are meters per ten seconds.

Q7: Removing Window Open

We removed Window Open from the training data, retrained all four methods, and evaluated Window Close and the unseen Faucet Open and Drawer Close tasks. Success rates (%):

MethodWindow CloseFaucet OpenDrawer Close
HILP0.000.0055.21
IKSD41.1516.6776.04
GKSD0.000.0066.67
AGKSD20.570.0063.54

IKSD leads on all three tasks after removing Window Open. Its learned manipulation skills transfer across tasks and objects, reusing opening and closing behaviors across window, faucet, and drawer geometries.

Q8: Partition and reference sizes

The trial count mm controls averaging across partitions; centers per trial ψ\psi control local resolution; ∣B∣|\mathcal B| is the total pool size. With m=100m=100, we varied ψ\psi on challenging terrain. Pretraining XY-bin coverage:

Environmentψ=2\psi=2ψ=4\psi=4ψ=8\psi=8ψ=16\psi=16ψ=32\psi=32ψ=64\psi=64
Cheetah111044576759
Walker252130426142

Both environments achieve their highest coverage at ψ=32\psi=32 among these settings.

Q9: Fig. 1 colors

We changed reference-only states to orange, retrieved-only states to blue, and shared states to green, and updated the legend.

Thanks for the detailed response. However, some of my concerns remain unsolved.

W1:

I notice that Eq. 1 is not updated in the manuscript. Please ensure that your response is consistent with the manuscript.

I want to point out that LSD and CSD learn from arbitrary state pairs, not only from adjacent state transitions. METRA learns from adjacent state transitions because of Theorem B.3 in the METRA paper, which absorbs the temporal distance. If you want to retain the current implementation, you should prove a similar proposition. Additionally, you need to reconsider what the actual distance measure you employ is. It should be different from kiso.

Q1:

Practical trajectories are not from a random walk. Can you further justify the reasonableness of the current construction method of B?

If a suboptimal reference set is used for IK calculation, will the performance of IK be largely affected?

Q2:

My understanding is that IK encourages visiting regions with larger state density, where the state density is measured with states in the replay buffer; while the measure in CSD encourages visiting regions that are hard to reach with the current policy, and thus tend to have smaller state density. I find this understanding counter-intuitive. The target of IK seems to be somehow opposed to that of the CSD measure. Therefore, I do not understand why IK helps improve the state coverage.

Is my understanding correct?

Can you specify a scenario where the two state densities behave differently, so that the readers can better understand why IK is better?

Q4:

I am a bit worried the experimental results in Sections 3.2 and 3.3 are sensitive to the environment selection. I suggest adding the results with the same environment to address this concern.

Q5:

Thanks for the explanation.

In the current manuscript, the methods in fig 1(c) are IK, Mahalanobis, Adaptive-GK, Dyn-1, Replay-temp, Euclidean, GK, Oracle-temp, where either Euclidean distance, temporal distance, or transition probability is among them. This can cause confusion. Please explicitly state the relation between these baseline measures in Section 3 and the distance measures used in LSD, CSD, and METRA.

Q7:

It seems that the Window Open significantly affects the performance of IKSD (100.00->41.15) and AGKSD (84.38 -> 20.57). Please update Table 2 in the manuscript with the new results.

Q8:

The authors do not provide ablation results for the partition number m. It remains unclear whether IKSD is sensitive to m.

Thank you for clarifying.

W1: Global distance

Revised Eq. (1) states

∥ϕ(x)−ϕ(y)∥2≤d(x,y),∀x,y∈S.\|\phi(x)-\phi(y)\|_2 \le d(x,y),\quad \forall x,y \in\mathcal{S}.

Rewards use adjacent transitions; constraints cover arbitrary pairs, as in LSD/CSD. In Sec. 4.1, κiso\kappa_{\mathrm{iso}} is similarity; Eq. (4) defines the local cost:

cIK(s,s′)=β+α(1−κiso(s,s′)),α>0, β≥0.c_{\mathrm{IK}}(s,s')=\beta+\alpha(1-\kappa_{\mathrm{iso}}(s,s')),\quad\alpha>0,\ \beta\ge0.

For fixed kernel geometry and an undirected graph GG, Theorem A.6 equates all edge constraints with this bound for connected pairs:

∥ϕ(x)−ϕ(y)∥2≤DIK,G(x,y):=inf⁡P∈PG(x,y)∑e∈PcIK(e).\lVert\phi(x)-\phi(y)\rVert_{2}\le D_{\mathrm{IK},G}(x,y):=\inf_{P\in\mathcal{P}_{G}(x,y)}\sum_{e\in P}c_{\mathrm{IK}}(e).

The proof sums triangle inequalities along paths; conversely, each edge is a path. This extends METRA's Theorem B.3 to data-dependent budgets. DIK,GD_{\mathrm{IK},G} is the global distance. C.5 gives sampled replay dual updates; A.6 bounds pathwise errors from local violations, within observed-graph components.

Q1, Q8: Reference sampling and mm

Two-room random walks provide a fixed generation rule and exact finite-step arrival probabilities. From uniform valid initial states, we follow the transition matrix, exclude anchor and Top-22 records, and sample centers randomly, without optimizing IK retrieval. PointMaze uses offline trajectories.

Pool coverage affects IK geometry. For a fixed pool and regions, Proposition A.4 gives unbiased frequencies and finite-sample bounds under uniform sampling without replacement. A.5 fixes the encoder and finite query family: with fixed ψ\psi and retained fraction >0>0, the subpool MSE vanishes as the pool grows. Kernel MSE has subpool, shared-batch, and finite-partition terms. Given retained pool SS and batch RR, independent partitions give

Var⁡(κ^m∣S,R)=κR(1−κR)/m≤1/(4m).\operatorname{Var}(\widehat\kappa_m\mid S,R)=\kappa_R(1-\kappa_R)/m\le1/(4m).

With ψ=32\psi=32, pretraining XY-bin coverage is:

mm1102050100200
Cheetah576260656769
Walker515555576162

Coverage generally improves, with small gains beyond 100. We use m=100m=100 to balance accuracy and cost.

Q2: IK versus CSD

IK adapts local resolution to replay density, which adds no reward. Budgets bound feasible rewards. Most constraints stay inactive in Fig. S4, alongside higher XY-bin coverage and valid-posture movement.

A frequently visited Walker landing region is easy to enter with the right foot descending near the ground, but hard just after lift-off. Marginal visitation is unchanged; CSD's conditional transition probability differs. Rare transitions can end in frequently visited regions.

LLM-assisted labels of 37,309 valid ten-step MetaWorld segments show small hand adjustments at 37.19%, candidate outward pushes at 4.22%, and inward pulls at 1.97% (Table S3). These motions support contact adjustment, directed motion, and amplitude control. Without Window Open training data, success rates (%) are:

MethodWindow CloseFaucet OpenDrawer Close
HILP0.000.0055.21
IKSD41.1516.6776.04
GKSD0.000.0066.67
AGKSD20.570.0063.54

From the same initial state, IKSD contacts and pushes the window; HILP fails to contact it. Results support transferable contact alignment and directed motion.

Q4: Environment choice

We retain complementary diagnostics: two-room neighborhoods and PointMaze rankings. Within each task, all eight measures share anchors, candidates, and evaluation references. IK achieves IoU 1.00 and NDCG 0.9864/1.0000 in Umaze/Large. Control tests cover four DMC settings with eight seeds each and MetaWorld transfer; kernel controls fix other settings.

Q5: Baseline correspondence

The revised text maps Euclidean to LSD, Oracle-temp's graph step count to METRA, and Replay-temp to empirical temporal distance. Dyn-1 compares one-step successor distributions; CSD derives costs from conditional transition prediction. Mahalanobis, GK, and Adaptive-GK test covariance, fixed-bandwidth, and local scaling; IK tests reference-induced scaling.

Q7: Updated Table 2

Table 2 now reports results after removing Window Open. Success rates (%) for these six tasks:

MethodDoor OpenDoor CloseDial TurnWindow CloseFaucet OpenDrawer Close
HILP10010068.490055.21
Offline CRL87.5054.1719.01012.5031.25
GCIVL10093.2377.6007.2928.91
Dual-GCIVL10099.7446.884.695.2175
TRL91.4110082.038.5915.6332.55
GKSD10096.8878.390066.67
AGKSD10010087.5020.57063.54
IKSD10010090.1041.1516.6776.04
Reviewer_t7mTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a novel distance metric, Isolation Kernel (IK) distance, for distance-maximization-based unsupervised skill discovery. IK distance is data-dependent, assigning larger distances to densely sampled regions and smaller distances to sparsely sampled regions. The authors incorporate IK distance into two existing skill discovery frameworks: METRA for online skill discovery and HILP for offline skill discovery. In both settings, the authors demonstrate that IK-based skill discovery (IKSD) improves exploration performance and downstream task success over baseline algorithms on DMC locomotion and MetaWorld manipulation tasks.

Critical strengths
  1. I think the main idea of incorporating a data-dependent distance metric, derived from online or offline data distributions, into skill discovery frameworks is novel.
  2. The experiments are comprehensive, covering both locomotion and manipulation tasks, as well as online and offline settings.
Critical weaknesses

I have several concerns about the proposed method and its experimental evaluation.

1. Could IKSD potentially discourage exploration?

The proposed IK distance tends to be smaller in sparsely sampled regions and larger in densely sampled regions. Consequently, IKSD imposes tighter representation constraints in underexplored regions, potentially limiting intrinsic rewards for exploring novel states. This seems counterintuitive from an exploration perspective, particularly compared to count-based exploration methods, which assign larger exploration bonuses to less frequently visited states. Could the authors clarify why this design is expected to encourage exploration rather than reinforce visitation of already densely sampled regions?

I believe an intuitive explanation of this mechanism should be included in the paper.

2. More analysis of the experimental results and limitations is needed.

Although the paper includes several diagnostic and qualitative analyses, I find that they do not sufficiently explain why IKSD outperforms the baselines in certain environments but not others. What are the common characteristics of the tasks where IKSD outperforms the baselines? Why does IKSD perform particularly well on these tasks? The current experimental section largely presents a sequence of results without sufficiently connecting the observations to the proposed method and its underlying mechanisms. I would appreciate a deeper analysis of when and why IKSD is effective, as well as a discussion of its limitations.

3. A more intuitive explanation of the expected learned behaviors is needed.

What types of behaviors or skills is IKSD expected to discover that existing skill discovery algorithms struggle to learn? It would be helpful if the authors could establish a clearer logical connection between the expected behaviors, the advantages of IK distance, and the mechanism through which IKSD enables these behaviors.

4. Could IKSD introduce training instability in the online setting?

In the online setting, my understanding is that the distance metric continuously changes as the data distribution evolves with policy updates. One potential concern is training instability, as these changes effectively introduce a non-stationary learning objective. Have the authors encountered any such instability during training? If so, how was it addressed?

Questions for authors

I would appreciate it if the authors could address each of the concerns raised in the Weaknesses section.

Additionally, one potential advantage I would expect from IKSD is fine-grained control. Since IK distance assigns larger distances within densely sampled regions, it may provide more room for learning fine-grained distinctions between behaviors in these regions. In other words, different skills might produce more distinguishable behaviors even when their resulting states are relatively close to each other.

Do the authors think this interpretation is reasonable? If so, do the experimental results provide any evidence that IKSD learns more fine-grained or distinguishable skills than the baseline methods?

6 comments

We thank you for recognizing the strengths of our work and for the key insight in Q1. We strongly agree that density-adaptive resolution can support fine-grained, distinguishable behaviors even when their resulting states are close.

W1: Could IKSD discourage exploration?

The distance constraint sets an upper bound on intrinsic reward; a smaller upper bound does not imply a smaller realized reward. METRA uses a unit bound on adjacent transitions; LSD uses Euclidean distance. METRA's benchmarks report higher coverage than LSD.

In Fig. S4, few sampled transitions activate the constraint in Cheetah and Walker.

Starting from the same replay buffer, IKSD expands Cheetah position coverage during continued training, while METRA remains at a low level (Fig. S2).

W2: Analysis and applicability

We expect IKSD to benefit tasks where the data contain abundant, repeatable local motion experience and progress depends on distinguishing different action outcomes at nearby states. The benefit depends on whether finer resolution of these data-supported variations addresses the task's control bottleneck.

IK assigns local metric scales from this distribution. Denser regions tend to have finer partitions, allowing nearby states distinct codes and room for representation differences. Observed transitions connect these distinctions to actual motion, and the skill objective encourages the policy to learn corresponding actions.

Online position coverage measures the visited range; valid-posture path length measures accumulated motion under valid postures. Together with the continued-training results in W1, they show the policy expanding its range and maintaining valid motion. We interpret these gains as effective use of local motion experience.

Offline MetaWorld results show reuse across tasks. Our interpretation is that the policy learns contact and motion relationships from the data that remain useful across different object geometries.

W3 and Q1: Fine-grained behaviors and evidence

Based on the analysis in W2, we expect IKSD to better support learning hand adjustments before contact, directed motion after contact, and changes in movement amplitude.

LLM-assisted annotations of 37,309 valid ten-step MetaWorld segments describe hand adjustments and candidate co-motion and pushing/pulling motions. The complete behavior table (Table S3) reports their counts and proportions.

After removing Window Open data and retraining, success rates (%) are:

MethodWindow CloseFaucet OpenDrawer Close
HILP0.000.0055.21
IKSD41.1516.6776.04
GKSD0.000.0066.67
AGKSD20.570.0063.54

IKSD achieves the highest success rate on all three tasks. These tasks require appropriate contact followed by motion along the object's permitted direction. In the paper's paired window-closing example, both methods start from the same state: IKSD establishes hand-object contact and pushes the window, while HILP does not establish contact.

The terrain experiments provide another example. Walker steps are 0.5 m wide and 0, 4, or 8 cm high, requiring changes in foot lift, landing position, and posture. Cheetah hills are 0.96 m wide with peak heights of 8 to 16 cm, requiring coordinated body pitch and forelimb/hindlimb support. Similar postures can require different actions on an upslope, a downslope, or during a support change.

Replay records time steps: slowing down, repeated attempts, and getting stuck accumulate nearby states, while rapidly traversed regions contribute fewer records. Fig. S1 shows this nonuniform visitation.

W4: Online training stability

IK enters as a stopped-gradient constraint budget. For fixed samples, representation parameters, and dual multiplier, changing only this budget leaves the representation-constraint gradient unchanged while each sample stays on the same smooth branch of the clipped penalty. The IK distance does not enter the policy reward directly.

Refreshed partitions track the replay distribution as exploration changes the states visited. The target network uses EMA; its parameters and the partitions remain fixed within each representation update.

Fig. S5 compares actor loss, representation loss, intrinsic reward, and both gradient norms for IKSD and METRA over the same interaction budget. IKSD's Cheetah reward increases after roughly 3.3M steps, alongside changes in gradient norms; Walker evolves more gradually. The displayed runs show no sustained divergence or repeated collapse.

If the authors have addressed your concerns or provided clarifications, please join the discussion to indicate whether your concerns have been resolved or if you have any further questions.

We thank you again for recognizing the strengths of our work and for your insight in Q1. We agree that finer resolution in densely sampled regions can support more distinguishable behaviors, even when their resulting states are close. We also have quantitative online evidence relevant to this interpretation, from challenging Cheetah terrain (Fig. S6).

We evaluated three trained models on the same 45 fixed clips, with 15 clips each for forward motion, backward motion, and posture adjustment. Each model encodes endpoint representation differences in its original 8D space. Panel B measures behavioral distinction using one overall ratio RR: the RMS distance between class centers divided by within-class RMS variation. Shared clips fix the evaluation inputs, and RR is invariant to uniform scaling.

The IK-Balanced control examines the role of visitation-dependent resolution. IKSD draws partition centers uniformly from reference records, retaining repeated visits. A cluster containing njn_j of NN records supplies a center with probability nj/Nn_j/N. IK-Balanced selects each of KK nonempty clusters with probability 1/K1/K, then samples uniformly within it. For a given reference pool, the two schemes retain the same state support and IK construction rule. We initially calibrated IK-Balanced's mean budget to IKSD's. CSD provides a comparison using one-step dynamics to construct state-change costs.

We also measured physical movement. Valid path length sums horizontal displacement when both transition endpoints satisfy the posture criterion, then averages across ten-second trajectories. Valid coverage counts distinct 1 m position bins visited with valid posture, counting each bin once. Panels C and D report these measures.

MethodBehavior separation RRValid path length (m/10 s)Valid coverage (1 m bins)
IKSD0.80436.5967
IK-Balanced0.38828.6741
CSD0.45223.8635

IKSD has the highest separation ratio for the evaluated behaviors. Relative to IK-Balanced, it also achieves 27.6% longer valid paths and 63.4% more valid position bins, with both physical measures exceeding CSD's. At the training endpoint, IKSD's reconstructed adjacent-transition budget median is lower than IK-Balanced's in all three behavior classes (Panel A).

On difficult terrain, repeated visits include local posture and motion changes needed to continue moving. Finer resolution can help represent these recurring control differences. The higher behavioral separation in this online comparison accompanies longer valid paths and broader coverage, providing evidence for the control value of the distinctions you identified.

Reviewer_qBuyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose IKSD, a method that incorporates an Isolation Kernel (IK)-based dissimilarity into online and offline distance maximization-based skill discovery. For equally separated state pairs, IK tends to assign larger distances in dense regions of the reference data and smaller distances in sparse regions. The authors report that this data-adaptive geometry improves skill discovery and downstream task performance.

Critical strengths
  • The paper proposes using Isolation Kernel-based dissimilarity as a data-adaptive distance in skill discovery.
  • The method is evaluated in both online and offline settings.
  • The controlled kernel comparison in Section 5.3 helps assess the effect of kernel choice within the same offline learning framework.
Critical weaknesses
  • The PointMaze experiments in Section 3.3 are insufficiently specified. In particular, the construction and sampling of the IK reference states and the estimation of empirical arrival probabilities are unclear. These details are important for reproducing the results and interpreting why IK distances agree with the reported arrival-probability rankings.
  • While Sections 5.3–5.4 provide kernel comparisons and qualitative behavior analysis, the specific contribution of density-dependent scaling remains unclear. These results do not isolate when and why adapting to the reference-state distribution improves skill learning. A controlled sensitivity analysis of this distribution would strengthen the paper’s central motivation.
Questions for authors
  • Could you specify how the IK reference states are sampled and how empirical arrival probabilities are estimated in the PointMaze experiments?
  • Could you clarify how density-dependent scaling contributes to useful skill learning, beyond the kernel comparisons and qualitative analyses?
8 comments

Thank you for the careful review and the questions about reproducibility and skill learning.

W1&Q1: Reference states and empirical arrival probabilities

We construct the IK reference pool B\mathcal B by sampling state records uniformly without replacement from offline trajectories. Repeated visits at different time steps remain separate records. We then sample partition centers uniformly with replacement from B\mathcal B, using m=100m=100 trials and ψ=32\psi=32 centers per trial, or 3,200 center draws. The temperature is 0.004 in both Umaze and Large. Candidate arrival probabilities do not enter either sampling step.

For anchor aa, let I(a)\mathcal I(a) index state records whose positions lie near aa. From each index, we check whether the same episode visits a radius-rr neighborhood of candidate yy within the next HH steps:

p^H(y∣a)=1∣I(a)∣∑i∈I(a)1[∃τ∈{1,…,Hi}:∥xi+τ−y∥2≤r].\widehat p_H(y\mid a)=\frac{1}{|\mathcal I(a)|}\sum_{i\in\mathcal I(a)}\mathbf 1\left[\exists\tau\in\{1,\ldots,H_i\}:\|x_{i+\tau}-y\|_2\leq r\right].

Here HiH_i is the smaller of HH and the remaining episode length, and xx denotes position. Each starting index contributes at most one hit per candidate. These probabilities define the reference ranking.

W2 & Q2: How density scaling helps skill learning

Density scaling adapts skill representation resolution to the empirical distribution, allowing small, recurring changes relevant to control to have distinct representations. Walker steps and Cheetah hills require adjustments in foot placement, body posture, and propulsion timing. Slowing down, obstruction, and repeated attempts concentrate experience in parts of the state space.

Fig. S1 shows the distribution of positions recorded in replay. We loaded the same replay buffer and continued training with IKSD and METRA. In Cheetah, IKSD's XY-bin coverage continues to grow, while METRA stays at a low level (Fig. S2). IKSD thus uses the existing experience to continue exploring.

For offline MetaWorld, we used LLM-assisted annotations of 37,309 valid ten-step data segments to describe the available motion experience. The Table S3 gives the full counts and percentages, including hand adjustments, candidate hand-object co-motion, and directional motion.

After removing Window Open from the training data and retraining, success rates (%) are:

MethodWindow CloseFaucet OpenDrawer Close
HILP0.000.0055.21
IKSD41.1516.6776.04
GKSD0.000.0066.67
AGKSD20.570.0063.54

IKSD has the highest success rate on all three unseen tasks. Our interpretation is that the learned adjustment, co-motion, and directed movement relationships can be reused across windows, faucets, and drawers.

If the authors have addressed your concerns or provided clarifications, please join the discussion to indicate whether your concerns have been resolved or if you have any further questions.

Thank you for your question about W1 & Q1 how density-dependent scaling contributes to useful skills. We also report quantitative online evidence relevant to fine-grained skill discovery on challenging Cheetah terrain. IKSD achieves higher behavior separation and broader valid position coverage than IK-Balanced and CSD (Fig. S6).

To examine the role of visitation weights, we compare IKSD with IK-Balanced. IKSD samples partition centers uniformly from reference records, retaining repeated visits. A cluster containing njn_j of the pool's NN records therefore supplies a center with probability nj/Nn_j/N. IK-Balanced first selects one of KK nonempty clusters uniformly, then samples a record uniformly within that cluster:

PIKSD(j)=njN,PBalanced(j)=1K.P_{\mathrm{IKSD}}(j)=\frac{n_j}{N},\qquad P_{\mathrm{Balanced}}(j)=\frac{1}{K}.

For a given reference pool, both schemes retain the same state support and IK construction rule. We calibrated IK-Balanced's initial mean budget to IKSD's. This comparison examines whether reducing high-frequency clusters' sampling weight improves behavioral distinction and effective coverage. CSD provides a comparison using one-step dynamics to construct state-change costs.

We analyzed the three trained models using the same 45 fixed clips: 15 each for forward motion, backward motion, and posture adjustment. Each model encodes the clips' endpoint representation differences in its original 8D space. Panel B reports the overall separation ratio RR, defined as the RMS distance between class centers divided by within-class RMS variation. This ratio is invariant to uniform scaling, so the comparison measures behavioral separation relative to within-class variation.

Panels C and D measure physical behavior. Valid path length sums horizontal displacement when both transition endpoints satisfy the posture criterion, then averages across ten-second trajectories. Valid coverage counts distinct 1 m position bins visited with valid posture; each bin counts once.

MethodBehavior separation RRValid path length (m/10 s)Valid coverage (1 m bins)
IKSD0.80436.5967
IK-Balanced0.38828.6741
CSD0.45223.8635

Compared with IK-Balanced, IKSD achieves higher behavior separation, 27.6% longer valid paths, and 63.4% more valid position bins. Its physical measures also exceed CSD's. Panel A reconstructs adjacent-transition budgets at the training endpoint: IKSD's median is lower than IK-Balanced's in all three behavior classes.

Our interpretation is that resolving recurring posture and motion differences in frequently visited regions helps skills continue moving and

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.