- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a distance-maximizing skill discovery method named IKSD that uses the estimated empirical state density as the distance measure. The distance metric here is the dissimilarity induced by the isolation kernel. Experiments are conducted in both online and offline settings, showing competitive performance in the online setting, superior performance in the online setting with challenging terrain, and in the offline setting.
Critical strengths
The proposed method applies to both online and offline settings.
Critical weaknesses
- The objective function of DSD (eq.1) is incorrect. In DSD, the constraint should be applied to an arbitrary state pair (x,y) instead of the adjacent states (s,s'). The authors should refer to the CSD paper for the correct DSD formulation. As a result, I am concerned that the implementation of IKSD may also be incorrect. As shown in Appendix C.5, the proposed IK dissimilarity only constrains the adjacent states (s,s').
- Many necessary details and explanations are missing in the current manuscript. Section 2.2 should be expanded. As the usage of Isolation Kernel dissimilarity is the core methodology of this method, the author should further explain what the Isolation Kernel is, how the partition is conducted, and why this is a "normalized Hamming pseudometric in [0, 1] between cell-index sequences". The computational overhead induced by the Isolation Kernel dissimilarity calculation should also be discussed.
Questions for authors
- In the experiment in Section 3.2, it seems that the reference set for IOU calculation is used in the IK calculation, while other baseline methods do not use that data. If a suboptimal reference set is used for IK calculation, will the performance of IK be largely affected?
- What is the functional difference between the proposed empirical state density and CSD’s conditional transition density? Why is the learned transition model in CSD not a good empirical state density? Can you specify a scenario where the two state densities behave differently, so that the readers can better understand your contribution?
- In Fig. 2, what do the dashed lines and the squares mean?
- The experiments in both sections 3.2 and 3.3 compare the top-k states measured by the empirical arrival probabilities and those measured by distance measures. Why are the experiments in sections 3.2 and 3.3 conducted on different environments?
- Can you explain in detail the relationship between the baseline measures in Section 3 and the distance measures used in LSD, CSD, and METRA?
- How is the "valid-posture path length per 10-second trajectory" calculated? The Appendix mentions that it accumulates distance traveled when the posture satisfies the validity criterion. However, I could not find the definition of the validity criterion.
- In Table 2, window open (one claimed seen task) and window close (one claimed unseen task) are almost identical. This might be the main reason for the superior performance on window close. If the window-open task is removed from the seen tasks, how will IKSD and the baselines perform?
- The number of partitions m and the size of the reference set seem to be important hyperparameters of IKSD. Is IKSD sensitive to these hyperparameters? The authors should conduct an ablation study for it, reporting the performance when changing these hyperparameters.
- (minor) Fig. 1 is hard to read. Could you please use more distinct colors to denote the three categories instead of three blues?
13 comments
Thanks for the detailed response. However, some of my concerns remain unsolved.
W1:
I notice that Eq. 1 is not updated in the manuscript. Please ensure that your response is consistent with the manuscript.
I want to point out that LSD and CSD learn from arbitrary state pairs, not only from adjacent state transitions. METRA learns from adjacent state transitions because of Theorem B.3 in the METRA paper, which absorbs the temporal distance. If you want to retain the current implementation, you should prove a similar proposition. Additionally, you need to reconsider what the actual distance measure you employ is. It should be different from kiso.
Q1:
Practical trajectories are not from a random walk. Can you further justify the reasonableness of the current construction method of B?
If a suboptimal reference set is used for IK calculation, will the performance of IK be largely affected?
Q2:
My understanding is that IK encourages visiting regions with larger state density, where the state density is measured with states in the replay buffer; while the measure in CSD encourages visiting regions that are hard to reach with the current policy, and thus tend to have smaller state density. I find this understanding counter-intuitive. The target of IK seems to be somehow opposed to that of the CSD measure. Therefore, I do not understand why IK helps improve the state coverage.
Is my understanding correct?
Can you specify a scenario where the two state densities behave differently, so that the readers can better understand why IK is better?
Q4:
I am a bit worried the experimental results in Sections 3.2 and 3.3 are sensitive to the environment selection. I suggest adding the results with the same environment to address this concern.
Q5:
Thanks for the explanation.
In the current manuscript, the methods in fig 1(c) are IK, Mahalanobis, Adaptive-GK, Dyn-1, Replay-temp, Euclidean, GK, Oracle-temp, where either Euclidean distance, temporal distance, or transition probability is among them. This can cause confusion. Please explicitly state the relation between these baseline measures in Section 3 and the distance measures used in LSD, CSD, and METRA.
Q7:
It seems that the Window Open significantly affects the performance of IKSD (100.00->41.15) and AGKSD (84.38 -> 20.57). Please update Table 2 in the manuscript with the new results.
Q8:
The authors do not provide ablation results for the partition number m. It remains unclear whether IKSD is sensitive to m.
Thank you for clarifying.
W1: Global distance
Revised Eq. (1) states
Rewards use adjacent transitions; constraints cover arbitrary pairs, as in LSD/CSD. In Sec. 4.1, is similarity; Eq. (4) defines the local cost:
For fixed kernel geometry and an undirected graph , Theorem A.6 equates all edge constraints with this bound for connected pairs:
The proof sums triangle inequalities along paths; conversely, each edge is a path. This extends METRA's Theorem B.3 to data-dependent budgets. is the global distance. C.5 gives sampled replay dual updates; A.6 bounds pathwise errors from local violations, within observed-graph components.
Q1, Q8: Reference sampling and
Two-room random walks provide a fixed generation rule and exact finite-step arrival probabilities. From uniform valid initial states, we follow the transition matrix, exclude anchor and Top-22 records, and sample centers randomly, without optimizing IK retrieval. PointMaze uses offline trajectories.
Pool coverage affects IK geometry. For a fixed pool and regions, Proposition A.4 gives unbiased frequencies and finite-sample bounds under uniform sampling without replacement. A.5 fixes the encoder and finite query family: with fixed and retained fraction , the subpool MSE vanishes as the pool grows. Kernel MSE has subpool, shared-batch, and finite-partition terms. Given retained pool and batch , independent partitions give
With , pretraining XY-bin coverage is:
| 1 | 10 | 20 | 50 | 100 | 200 | |
|---|---|---|---|---|---|---|
| Cheetah | 57 | 62 | 60 | 65 | 67 | 69 |
| Walker | 51 | 55 | 55 | 57 | 61 | 62 |
Coverage generally improves, with small gains beyond 100. We use to balance accuracy and cost.
Q2: IK versus CSD
IK adapts local resolution to replay density, which adds no reward. Budgets bound feasible rewards. Most constraints stay inactive in Fig. S4, alongside higher XY-bin coverage and valid-posture movement.
A frequently visited Walker landing region is easy to enter with the right foot descending near the ground, but hard just after lift-off. Marginal visitation is unchanged; CSD's conditional transition probability differs. Rare transitions can end in frequently visited regions.
LLM-assisted labels of 37,309 valid ten-step MetaWorld segments show small hand adjustments at 37.19%, candidate outward pushes at 4.22%, and inward pulls at 1.97% (Table S3). These motions support contact adjustment, directed motion, and amplitude control. Without Window Open training data, success rates (%) are:
| Method | Window Close | Faucet Open | Drawer Close |
|---|---|---|---|
| HILP | 0.00 | 0.00 | 55.21 |
| IKSD | 41.15 | 16.67 | 76.04 |
| GKSD | 0.00 | 0.00 | 66.67 |
| AGKSD | 20.57 | 0.00 | 63.54 |
From the same initial state, IKSD contacts and pushes the window; HILP fails to contact it. Results support transferable contact alignment and directed motion.
Q4: Environment choice
We retain complementary diagnostics: two-room neighborhoods and PointMaze rankings. Within each task, all eight measures share anchors, candidates, and evaluation references. IK achieves IoU 1.00 and NDCG 0.9864/1.0000 in Umaze/Large. Control tests cover four DMC settings with eight seeds each and MetaWorld transfer; kernel controls fix other settings.
Q5: Baseline correspondence
The revised text maps Euclidean to LSD, Oracle-temp's graph step count to METRA, and Replay-temp to empirical temporal distance. Dyn-1 compares one-step successor distributions; CSD derives costs from conditional transition prediction. Mahalanobis, GK, and Adaptive-GK test covariance, fixed-bandwidth, and local scaling; IK tests reference-induced scaling.
Q7: Updated Table 2
Table 2 now reports results after removing Window Open. Success rates (%) for these six tasks:
| Method | Door Open | Door Close | Dial Turn | Window Close | Faucet Open | Drawer Close |
|---|---|---|---|---|---|---|
| HILP | 100 | 100 | 68.49 | 0 | 0 | 55.21 |
| Offline CRL | 87.50 | 54.17 | 19.01 | 0 | 12.50 | 31.25 |
| GCIVL | 100 | 93.23 | 77.60 | 0 | 7.29 | 28.91 |
| Dual-GCIVL | 100 | 99.74 | 46.88 | 4.69 | 5.21 | 75 |
| TRL | 91.41 | 100 | 82.03 | 8.59 | 15.63 | 32.55 |
| GKSD | 100 | 96.88 | 78.39 | 0 | 0 | 66.67 |
| AGKSD | 100 | 100 | 87.50 | 20.57 | 0 | 63.54 |
| IKSD | 100 | 100 | 90.10 | 41.15 | 16.67 | 76.04 |
Thank you for the detailed review and the suggestions on our formulation and experiments.
W1: DSD objective
The general constraint applies to every state pair:
∥ϕ(s′)−ϕ(s)∥2≤d(s,s′),∀s,s′∈S.
Eq. (1) misstates its domain. Like METRA, CSD, and LSD, online IKSD learns from adjacent state transitions. Our implementation evaluates the constraint on transitions sampled from replay. Correcting the general definition leaves the implemented updates and reported results unchanged.
W2: Method details
New paper's Sec. 2.2 and Algorithm 1 describe reference encoding, partitions, shared-cell frequencies, and online/offline updates. Each representation update fixes the target network and partitions, stops kernel-cost gradients, then updates the target by EMA. For input dimension p, center comparisons cost O(mψp) per state pair; computing similarity from cell indices costs O(m).
Q1: Reference data
We construct B from random walks starting uniformly over valid grid cells and evolving under the environment's transition matrix. Repeated visits and self-transitions remain in the pool. Six-step arrival probabilities define the Top-22 evaluation set and do not enter reference sampling or partition construction.
Mahalanobis and Adaptive-GK use the same B; Dyn-1 uses adjacent transitions from these trajectories, and Replay-temp uses their temporal order. New paper's Sec. 3.2 specifies these sources.
Q2: State density versus CSD
IK uses the empirical state marginal pD(s) represented by replay: local sample concentration sets partition resolution. CSD learns the conditional transition density q(s′∣s) under current skills. Its likelihood-derived cost gives larger constraint budgets to transitions that are harder to realize from a given state.
Q3: Fig. 2 symbols
Dashed circles show candidates' Euclidean distance ranges from the anchor; Large's inner and outer circles bound the candidate annulus. Boxes and connecting lines identify enlarged insets showing anchor and candidate positions.
Q4: Different environments
Both sections use arrival probabilities as the reference but test different properties. Sec. 3.2 uses a constructed two-room random walk and IoU to compare Top-k set overlap. Its doorway geometry and stay probabilities are easy to specify, making the retrieved neighborhoods easy to visualize.
Sec. 3.3 tests candidate ordering in offline PointMaze-Umaze and PointMaze-Large trajectories. IoU ignores order within a set; NDCG evaluates whether states with higher empirical arrival probabilities rank earlier. The two settings test neighborhood membership in a constructed example and ranking in benchmark data.
Q5: LSD, CSD, and METRA
LSD uses Euclidean distance, METRA uses temporal distance, and CSD uses transition probability.
Q6: Valid-posture path length
Appendix B.3 defines a valid posture: the dot product of the torso's local vertical axis and the world vertical axis exceeds 0.7, and torso height above local terrain exceeds 0.8 m for Walker or 0.25 m for Cheetah. For a ten-second trajectory, let vt indicate validity and xt be horizontal torso position:
Lvalid=∑t=0T−11{vt∧vt+1}∥xt+1−xt∥2.
Here T is the number of transitions. We sum horizontal movement with both endpoints valid, including revisits, then average over evaluation trajectories. Units are meters per ten seconds.
Q7: Removing Window Open
We removed Window Open from the training data, retrained all four methods, and evaluated Window Close and the unseen Faucet Open and Drawer Close tasks. Success rates (%):
IKSD leads on all three tasks after removing Window Open. Its learned manipulation skills transfer across tasks and objects, reusing opening and closing behaviors across window, faucet, and drawer geometries.
Q8: Partition and reference sizes
The trial count m controls averaging across partitions; centers per trial ψ control local resolution; ∣B∣ is the total pool size. With m=100, we varied ψ on challenging terrain. Pretraining XY-bin coverage:
Both environments achieve their highest coverage at ψ=32 among these settings.
Q9: Fig. 1 colors
We changed reference-only states to orange, retrieved-only states to blue, and shared states to green, and updated the legend.