- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a skill discovery strategy that adapts to the human preferences while preserving diversity. For the approach, they use foundation models to inject preferences, and validate their approach with strong experimental results compared with diverse baselines. Nevertheless, the proposed problem significance is limited and the algorithmic contribution is incremental.
Critical strengths
- The experimental results validate their approach. FoG shows strong results compared with baselines including previous privileged skill discovery approaches.
- Reasonable approach for preference-based skill discovery, using foundation model which is one of the way that can directly reflect the user preference.
Critical weaknesses
- The trajectory-level formulation is motivated by preferences being properties of whole behaviors (Sec. 3.1). However, the proposed gate is a state-wise, so reflecting the trajectory-level preference is limited. Although the derivation establishes a lowerbound on trajectory-level WDM, it does not guarantee whether the method can satisfy general trajectory-level preference. Incorporating "trajectory-level" preference in skill discovery is a significant problem, but as stated in conclusion, the approach addresses a substantially narrower class of preferences. This leaves gap between the trajectory-level motivation and the actual approach.
- The practical contribution of trajectory-level derivation is unclear. The final objective (Eq.9) is METRA with multiplicative gate, which the author agrees with the structural resemblance to the DoDont reward. The main difference is that the gate is obtained from a foundation model. While there was a practical improvement, it is unclear whether it sufficiently introduced a substantial methodological contribution beyond existing approaches.
- The experimental results do not sufficiently evaluate the general preference specification. The results mostly cover avoidance-type preferences. Preferences for new behaviors (twisted, running) are evaluated only by a small human study.
Questions for authors
- Can the authors provide quantitative results for non-avoidance preferences or downstream performance with those skills?
3 comments
W2. Practical contribution of the trajectory-level derivation, and the relation to DoDont
We agree that DoDont is the closest work, and we credit it for introducing the multiplicative gate, which FoG shares. The table summarizes what FoG adds. We will move this comparison from Appendix C into the main text.
| DoDont | FoG | What FoG adds | |
|---|---|---|---|
| Reward | Gated METRA reward (introduced by DoDont) | Same gated reward | — |
| What the gated reward optimizes | Argued to maximize dependency between the skill and the final state under a classifier-based distance. The derivation relies on replacing φ(s) with p̂(s, s′)·φ̃(s) (App. B), which holds for a constant gate but not for a general one | Proven to maximize a lower bound on dependency between the skill and the whole trajectory (Thm. 1) | To our knowledge, the first rigorous account of what the gated reward optimizes; the proof extends to general bounded gates, including DoDont's |
| What the user provides | "Do" and "don't" demonstration videos for each preference | One pair of sentences | No data collection |
| Before skill discovery | Train a classifier on the videos | Nothing | No extra training stage |
| Preferences without demonstrations (e.g. "twisted") | Need videos of the behaviour first | Stated directly in language | Covers preferences that are hard to demonstrate |
| Cheetah flips, pixel-only protocol | 35% | 12% | Fewer undesirable skills at the same coverage |
| Same CLIP labels: classifier supervision vs direct gate | 52% (DoDont+) | 12% | With the same CLIP labels, direct gating is far more robust than classifier training |
| Downstream return (AUC) | 0.69 | 0.79 | More useful skill library |
What the derivation contributes. DoDont arrives at the gated reward in two steps. It first states a METRA-style objective in which the classifier bounds the representation increments, ‖φ(s) − φ(s′)‖ ≤ p̂(s, s′) (Eq. 5). It then obtains the gated reward (Eq. 7) by replacing φ(s) with p̂(s, s′)·φ̃(s) (Appendix B, Eqs. 8–10). The left-hand side, φ(s), is a function of the state s alone, whereas the right-hand side depends on the transition (s, s′). A state s_t appears in two consecutive transitions, so it would be assigned p̂(s_{t−1}, s_t)·φ̃(s_t) in one term and p̂(s_t, s_{t+1})·φ̃(s_t) in the next. So φ(s) cannot be replaced in this way in general, which would be mathematically flawed. This does not affect DoDont's empirical results. However, it means the gated reward, which we share with DoDont, needs a different justification. The reason is structural: a gated sum does not telescope, so it is not a function of the final state and cannot come from the final-state dependency that METRA starts from. We therefore measure dependency between the skill and the whole trajectory, and prove that the gated reward maximizes a lower bound on it (Theorem 1). To our knowledge, this is the first rigorous account of what the gated reward optimizes. The same proof extends to any bounded gate on transitions, with the gate entering the trajectory metric, so it covers DoDont's reward as well; we will add this as a remark.
A small example makes the difference concrete. Consider trajectories that return to their starting state. The objective in DoDont's Eq. 5 is then zero for every policy and representation, because the sum telescopes to the difference between the final and the initial state. The gated objective in Eq. 7 can still be positive when the gate takes different values on the way out and on the way back. The two objectives therefore differ, even with strictly positive gates.
The practical side. CLIP's judgment fails when it is used as a reward (FR-SAC and METRA+FR barely move). The same CLIP labels also fail when they are used to train DoDont's classifier (DoDont+ flips in 52% of episodes and falls below METRA downstream); this is the controlled comparison in the paper. They work when applied directly as a binary gate. Together with the fact that no demonstrations or classifier training are needed, this is what the language interface adds in practice.
W3 and Q1. Quantitative results for non-avoidance preferences, and downstream performance with those skills
Thank you for this valuable suggestion. We fully agree that simulator-based measurements and downstream evaluation of the "twisted" and "running" skills would make this part of the paper stronger. In this paper we focused on showing that a preference stated in language can steer skill discovery zero-shot, and on giving the gated objective a rigorous foundation. Within that focus, these two tasks were included to show that FoG can elicit behaviours that have no hand-coded definition. Human judgment is the most direct test of that. In our study, which we agree is small (ten participants comparing ten randomly paired skills per task), participants chose FoG as more "twisted" in 90% of comparisons and as more "running" in 70%. The closest simulator-measured example we have is in the Kitchen appendix, although it can also be read as avoidance: with "cabinet is closed", cabinet opening drops from 78% (METRA) to 22% (FoG). The fuller quantitative and downstream study you suggest is a natural and important next step. We will state this clearly as a limitation and as future work in the main text.
Thank you for the review, and for recognizing the experimental results and the use of a foundation model as a direct way to reflect the user's preference. We reply to each point below.
W1. Trajectory-level motivation versus the state-wise gate
We thank the reviewer for this observation, and we agree: the opening sentence of Section 3.1 is misleading as it suggests that our method handles preferences over whole trajectories, which it does not; we will rewrite it. What is trajectory-level in our formulation is the dependency measure, not the preference.