acceptodds
Under review as a conference paper at ICLR 2027

Guiding Skill Discovery with Foundation Model

Abstract

Unsupervised skill discovery removes the need for hand-designed rewards, but maximizing behavioral diversity alone readily produces undesirable or dangerous skills: a cheetah robot learns to roll in every direction when we would prefer it to run without flipping or entering hazardous areas. We propose Foundation model Guided (FoG) skill discovery, which incorporates such preferences without reward functions or demonstrations. A preference is stated once, in language; a foundation model turns it into a binary label on states zero-shot, by writing the labeler as a short program from a description of the state vector or by scoring raw pixels against the sentence. Starting from a trajectory-level Wasserstein-dependency formulation, we show how this label enters the skill-discovery reward as a multiplicative gate through the critic of the Kantorovich-Rubinstein dual, and prove that the resulting objective lower-bounds the trajectory-level dependency. Across state- and pixel-based benchmarks, FoG eliminates flipping and rolling, avoids hazardous areas, discovers behaviors that are hard to specify explicitly, and yields skill libraries that adapt quickly and safely to downstream goals. Interactive visualizations are available from https://sites.google.com/view/submission-fog.

Primary area:
reinforcement learning
Keywords:
Unsupervised RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jKVDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a skill discovery strategy that adapts to the human preferences while preserving diversity. For the approach, they use foundation models to inject preferences, and validate their approach with strong experimental results compared with diverse baselines. Nevertheless, the proposed problem significance is limited and the algorithmic contribution is incremental.

Critical strengths
  • The experimental results validate their approach. FoG shows strong results compared with baselines including previous privileged skill discovery approaches.
  • Reasonable approach for preference-based skill discovery, using foundation model which is one of the way that can directly reflect the user preference.
Critical weaknesses
  • The trajectory-level formulation is motivated by preferences being properties of whole behaviors (Sec. 3.1). However, the proposed gate is a state-wise, so reflecting the trajectory-level preference is limited. Although the derivation establishes a lowerbound on trajectory-level WDM, it does not guarantee whether the method can satisfy general trajectory-level preference. Incorporating "trajectory-level" preference in skill discovery is a significant problem, but as stated in conclusion, the approach addresses a substantially narrower class of preferences. This leaves gap between the trajectory-level motivation and the actual approach.
  • The practical contribution of trajectory-level derivation is unclear. The final objective (Eq.9) is METRA with multiplicative gate, which the author agrees with the structural resemblance to the DoDont reward. The main difference is that the gate is obtained from a foundation model. While there was a practical improvement, it is unclear whether it sufficiently introduced a substantial methodological contribution beyond existing approaches.
  • The experimental results do not sufficiently evaluate the general preference specification. The results mostly cover avoidance-type preferences. Preferences for new behaviors (twisted, running) are evaluated only by a small human study.
Questions for authors
  • Can the authors provide quantitative results for non-avoidance preferences or downstream performance with those skills?
3 comments

Thank you for the review, and for recognizing the experimental results and the use of a foundation model as a direct way to reflect the user's preference. We reply to each point below.

W1. Trajectory-level motivation versus the state-wise gate

We thank the reviewer for this observation, and we agree: the opening sentence of Section 3.1 is misleading as it suggests that our method handles preferences over whole trajectories, which it does not; we will rewrite it. What is trajectory-level in our formulation is the dependency measure, not the preference.

  • What the term refers to. METRA measures the dependency between the skill and the final state. We measure the dependency between the skill and the whole trajectory. The preference is defined on states by design, as the abstract says ("a binary label on states").
  • Relation to METRA. Our trajectory-level formulation generalizes METRA's final-state formulation. Without gating (a gate identically equal to one), it recovers METRA's objective exactly. When the gate varies, it still yields a valid objective, which the final-state formulation cannot: a gated reward depends on the path taken, not only on where the trajectory ends.
  • What it provides. Measuring dependency on trajectories gives preference-gated skill discovery a sound foundation. This is the role of the trajectory level in the paper.
  • Preferences over sequences. We agree these are a valuable direction. They need an evaluator that looks at a short window of states, which we note in the conclusion and will discuss as future work.

W2. Practical contribution of the trajectory-level derivation, and the relation to DoDont

We agree that DoDont is the closest work, and we credit it for introducing the multiplicative gate, which FoG shares. The table summarizes what FoG adds. We will move this comparison from Appendix C into the main text.

DoDontFoGWhat FoG adds
RewardGated METRA reward (introduced by DoDont)Same gated reward—
What the gated reward optimizesArgued to maximize dependency between the skill and the final state under a classifier-based distance. The derivation relies on replacing φ(s) with p̂(s, s′)·φ̃(s) (App. B), which holds for a constant gate but not for a general oneProven to maximize a lower bound on dependency between the skill and the whole trajectory (Thm. 1)To our knowledge, the first rigorous account of what the gated reward optimizes; the proof extends to general bounded gates, including DoDont's
What the user provides"Do" and "don't" demonstration videos for each preferenceOne pair of sentencesNo data collection
Before skill discoveryTrain a classifier on the videosNothingNo extra training stage
Preferences without demonstrations (e.g. "twisted")Need videos of the behaviour firstStated directly in languageCovers preferences that are hard to demonstrate
Cheetah flips, pixel-only protocol35%12%Fewer undesirable skills at the same coverage
Same CLIP labels: classifier supervision vs direct gate52% (DoDont+)12%With the same CLIP labels, direct gating is far more robust than classifier training
Downstream return (AUC)0.690.79More useful skill library

What the derivation contributes. DoDont arrives at the gated reward in two steps. It first states a METRA-style objective in which the classifier bounds the representation increments, ‖φ(s) − φ(s′)‖ ≤ p̂(s, s′) (Eq. 5). It then obtains the gated reward (Eq. 7) by replacing φ(s) with p̂(s, s′)·φ̃(s) (Appendix B, Eqs. 8–10). The left-hand side, φ(s), is a function of the state s alone, whereas the right-hand side depends on the transition (s, s′). A state s_t appears in two consecutive transitions, so it would be assigned p̂(s_{t−1}, s_t)·φ̃(s_t) in one term and p̂(s_t, s_{t+1})·φ̃(s_t) in the next. So φ(s) cannot be replaced in this way in general, which would be mathematically flawed. This does not affect DoDont's empirical results. However, it means the gated reward, which we share with DoDont, needs a different justification. The reason is structural: a gated sum does not telescope, so it is not a function of the final state and cannot come from the final-state dependency that METRA starts from. We therefore measure dependency between the skill and the whole trajectory, and prove that the gated reward maximizes a lower bound on it (Theorem 1). To our knowledge, this is the first rigorous account of what the gated reward optimizes. The same proof extends to any bounded gate on transitions, with the gate entering the trajectory metric, so it covers DoDont's reward as well; we will add this as a remark.

A small example makes the difference concrete. Consider trajectories that return to their starting state. The objective in DoDont's Eq. 5 is then zero for every policy and representation, because the sum telescopes to the difference between the final and the initial state. The gated objective in Eq. 7 can still be positive when the gate takes different values on the way out and on the way back. The two objectives therefore differ, even with strictly positive gates.

The practical side. CLIP's judgment fails when it is used as a reward (FR-SAC and METRA+FR barely move). The same CLIP labels also fail when they are used to train DoDont's classifier (DoDont+ flips in 52% of episodes and falls below METRA downstream); this is the controlled comparison in the paper. They work when applied directly as a binary gate. Together with the fact that no demonstrations or classifier training are needed, this is what the language interface adds in practice.

W3 and Q1. Quantitative results for non-avoidance preferences, and downstream performance with those skills

Thank you for this valuable suggestion. We fully agree that simulator-based measurements and downstream evaluation of the "twisted" and "running" skills would make this part of the paper stronger. In this paper we focused on showing that a preference stated in language can steer skill discovery zero-shot, and on giving the gated objective a rigorous foundation. Within that focus, these two tasks were included to show that FoG can elicit behaviours that have no hand-coded definition. Human judgment is the most direct test of that. In our study, which we agree is small (ten participants comparing ten randomly paired skills per task), participants chose FoG as more "twisted" in 90% of comparisons and as more "running" in 70%. The closest simulator-measured example we have is in the Kitchen appendix, although it can also be read as avoidance: with "cabinet is closed", cabinet opening drops from 78% (METRA) to 22% (FoG). The fuller quantitative and downstream study you suggest is a natural and important next step. We will state this clearly as a limitation and as future work in the main text.

Reviewer_NomcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

the paper "Guiding Skill Discovery with Foundation Model" proposes a way to direct the unsupervised skill discovery in RL environnements, to focus on skills spread on desirable state spaces. Doing so, the skill discovery is not fully unsupervised anymore, as for instance in METRA the paper grounds in, but leverage desiderata from experts that are expressed through LLM (or VLMs). one central claim is that expressing constraints with natural language is easier than hand-crafting the reward function that includes it. The constraint is introduced in the learning process by the means of a gating mechanism in the METRA loss.

Critical strengths
  • The paper is rather easy to follow
  • The problem is interesting and usefull
  • The approach looks theoretically well sounded
  • The results look quite convincing
Critical weaknesses
  • One of my main concerns is about the framing of the approach, in which the use of foundation models is presented as central, while they could fully be replaced by hand-coded functions. This only a simple choice for introducing expert knowledge in the policy. Are foundation models really central in the approach ?

  • Experiments are only conducted on rather simple constraints, while real settings could imply many competting factors.

  • The innovation looks quite straightforward compared to METRA (only a gate that multiplies state terms).

Questions for authors
  • I understand that METRA+ is an ablation proposed by authors to compare the use of a foundation model with hand crafted constraints. Ok, but for simple settings I cannot see the difference as FOG already uses a simple function that could fully be used as the constraint function of METRA+. Where come the result differences from ?

  • The main innovation is the introduction of a gating mechanism in METRA. While the derivation of the state level terms from a trajectory level perspective is interesting, the constraint remains defined on states, not on (sub-) trajectories. Wouldn't is be possible to include constraints based on sequences of states (e.g. rotating on the left) rather than on the reaching of single states?

  • Why experiments do not include some alternative formulations such as successor-feature based ones. MetaMotivo, which includes demonstrations, in forward backward representation learning looks a good candidate to be considered as a baseline.

  • Evaluation metrics are not clearly stated. In figure 2 what means the % of rollouts ? Also, state coverage is not clearly defined. State-coverage in term of x,y positions ? Since if it was state coverage that include flipped positions, I couldn't see how Fog could be better. That's it ? Please clarify in the paper.

  • Figure 4 is particularly unclear. What does the left part show ? Where are hazardous states ? I cannot see what we can see on that figure.

  • d_temp should be defined in the paper to be self-contained. The full METRA loss also.

5 comments

We thank the reviewer for the constructive review, and for finding the problem interesting, the approach theoretically sound and the results convincing. We reply to each point below.

W1. Are foundation models really central?

Not to the mechanism, yes to its usability. The gated objective and its theory accept a gate from any source, and you are right that when a preference is easy to write as a function of the state, a hand-coded gate can replace the foundation model. The foundation model is what supplies a gate without state access or reward engineering, and this matters in three cases covered by the paper:

  • From pixels there is no state to hand-code on (all experiments in Section 5.2).
  • Some preferences have no obvious formula, such as "twisted" or "running".
  • Even a simple preference is easy to hand-code badly: the hand-coded score r_run − r_flip used by METRA+ suppresses flips only by suppressing motion.

We take your point about the framing. We will present the contribution as a preference-gated skill-discovery framework with its theory, and the foundation model as the instantiation that makes it zero-shot. We also agree with the remark in your summary: the preference is prior knowledge, so we will describe the setting as preference-guided skill discovery without task rewards or demonstrations, and not as fully unsupervised.

W2. Simple constraints versus competing factors

Thank you for this point. We fully agree that real settings involve several factors at once. In this paper we focused on the single-preference case and on its objective and theory. Each preference is simple to state by design, in one sentence, but they are of different kinds: posture (not flipping, "twisted", "running"), regions of the floor, and the state of an object (cabinet closed), from both state and pixel inputs. For combinations, the appendix reports a first experiment that asks Cheetah both not to flip and to avoid the right area: the agent stops flipping but does not avoid the area, because requiring both at once restricts exploration too much. A conjunction of binary labels is again a binary label, so the objective and Theorem 1 apply unchanged; what remains open is exploration under a stricter gate. Making combined preferences work in practice is a natural and important next step, which we could not complete within the rebuttal period. We will discuss it as a limitation and as future work in the main text.

W3. The innovation looks straightforward compared to METRA

We understand this impression, since the final reward differs from METRA's by one factor. We would like to point to two contributions behind that factor.

A rigorous account of what the gated reward optimizes. This turned out not to be straightforward. METRA starts from the dependency between the skill and the final state, and relies on the sum of representation increments telescoping to the final state. With a gate the sum no longer telescopes, so it is not a function of the final state and cannot come from METRA's derivation. The closest prior work, DoDont, which introduced the multiplicative gate, obtains it by replacing φ(s) with p̂(s, s′)·φ̃(s) (its Appendix B). The left-hand side depends on the state alone and the right-hand side on the transition, so this replacement is mathematically flawed in general. We therefore measure dependency between the skill and the whole trajectory, and prove that the gated reward maximizes a lower bound on it (Theorem 1). Our trajectory-level formulation generalizes METRA's: without gating (a gate identically equal to one) it is exactly METRA's objective, and it remains valid when the gate varies. To our knowledge, this is the first rigorous account of what the gated reward optimizes.

How a foundation-model signal should be used. CLIP's judgment fails when used as a reward (FR-SAC and METRA+FR barely move). The same CLIP labels also fail when used to train a classifier (DoDont+ flips in 52% of episodes), and they work when applied directly as a binary gate (FoG, 12%). In these comparisons, it is the direct gate that makes a noisy zero-shot signal usable.

Q1. Where do the differences between METRA+ and FoG come from?

From the gate, not from the algorithm. That is, if METRA+ used the same function as FoG, the two would be the same algorithm and would give the same results. In every experiment where they are compared, however, they use different functions.

  • METRA+ is compared with FoG on the pixel tasks (Figures 3 and 4). There FoG has no simple function: its gate is CLIP's judgment of an image. METRA+ uses privileged simulator state. The simple function you have in mind is the one written by the language model in the state-based tasks (Figure 2), where METRA+ is not a baseline.
  • METRA+ is the hand-coded baseline from the DoDont paper, not an ablation of ours. On the flip task it uses the score r_run − r_flip, a continuous score and not a binary gate; it avoids flips only by suppressing motion (coverage of 4 cells or fewer).
  • On the area tasks METRA+ uses a 0/1 indicator of the safe region, which is FoG's gate with ground-truth labels. There it does better than FoG, and FoG comes next using only pixels and a sentence.

A new experiment that tests your point directly. For the rebuttal we trained FoG on pixel Cheetah with the label source replaced by the simulator's binary flip indicator (torso rotated beyond 90°), keeping every other setting unchanged, for three seeds. This matches the "simple function as the constraint" as described in the question. Both rows below use the same automated evaluation at the end of training (48 episodes per seed), so they are directly comparable with each other.

GateSeedsEpisodes with a flip (%)Position cells visited
CLIP on pixels (FoG)312.5 ± 6.427.3 ± 1.5
Ground-truth binary indicator329.2 ± 9.132.0 ± 0.6

Values are means ± standard errors over seeds. With the same algorithm, the two gates give comparable behaviour: CLIP's gate is not worse than the ground-truth one, with flips at least as rare and somewhat lower coverage, and the differences are within the standard errors for three seeds. One likely reason is that the two gates do not mark the same states. In a separate check on 100,000 frames, CLIP marked 22–39% of upright frames as undesirable, so it acts as a stricter gate than the 90° rule. This supports the answer above: the differences between methods come from which states the gate marks, and even a hand-coded rule involves a choice, here the threshold.

We will state both METRA+ definitions in the main text and add this experiment to the paper.

Q2. Constraints on sequences of states

We thank the reviewer for this observation, and we agree: the opening sentence of Section 3.1 suggests that our method handles preferences over whole trajectories, which it does not. That sentence was misleading and we will rewrite it. What is trajectory-level in our formulation is the dependency we measure, between the skill and the whole trajectory instead of the final state; the preference is defined on states by design.

Regarding the question: yes, we believe constraints on sequences, such as rotating to the left, are possible within this approach. The gate is the only component that looks at states, and replacing the single-state evaluator by one that scores a short window is the natural way to do it, as we note in the conclusion. We have not evaluated this, and we will discuss it as future work.

Q3. Successor-feature formulations and Meta Motivo

We thank the reviewer for pointing us to this work, which we should have discussed. Meta Motivo (Tirinzoni et al., ICLR 2025) is close in spirit to steering skill discovery toward desirable behaviours while keeping zero-shot tracking, goal reaching and reward optimization for a humanoid.

In this paper we focused on how a language preference should enter a skill-discovery objective, so the preference-guided baselines (DoDont, DoDont+, LGSD, METRA+ and METRA+FR) are all built on METRA and differ in how the preference is supplied. This lets us attribute the differences mainly to that factor. Meta Motivo also relies on a dataset of behaviours, whereas FoG assumes only a sentence; for a preference like "twisted" we have no such dataset. We will add successor-feature and forward–backward methods, including Meta Motivo, to the related work, and discuss as future work whether a language gate can complement demonstration-based regularization in such methods, with empirical comparisons.

Q4. Evaluation metrics

These definitions were missing and we will add them to the experimental setup.

  • "% Rolls" in Figure 2. The axis label reads "% Rolls". It is the percentage of evaluation episodes in which the agent rolls over at least once; it does not refer to rollouts. We will rename the axis and define the criterion in the text.
  • State coverage. You comment Is correct that it is coverage of positions only. It is the number of distinct unit cells of position visited by the evaluation trajectories of randomly sampled skills: the x-coordinate for HalfCheetah and Cheetah, and the (x, y) position for Ant and Quadruped. This is the metric used by METRA. As stated in the paper, all reported values are means over three seeds with standard errors.
  • Why FoG can match METRA. Body orientation is not part of the metric, so flipping adds no coverage. FoG reaches the same range of positions while rarely flipping.

Q5. Figure 4

We agree the figure needs explanation.

  • Left part. The trajectories of the learned skills, one curve per skill: for Cheetah (top), x-position over time; for Quadruped (bottom), paths in the x–y plane.
  • Hazardous regions. For Cheetah, the right half (x > 0). For Quadruped, the bottom-left half of the plane (x + y < 0). The agent is told about them only through the colour of the floor.
  • Right part. Safe-state coverage: the number of distinct unit cells visited in the safe region minus the number of distinct unit cells visited in the hazardous region, over the evaluation trajectories. A method scores high only if it explores the safe region widely and stays out of the hazardous one. We will state this definition in the caption.

We will shade and label the hazardous region, mark its boundary, label the axes and explain each panel in the caption.

Q6. Definition of d_temp and the METRA loss

We agree both belong in the main text. d_temp is the temporal distance: the minimum number of environment steps needed to go from one state to another, so that consecutive states are at distance one. It is currently defined only in the appendix, and we will define it where it first appears. For reference, METRA maximizes, over the policy and the representation φ, the expected sum over the episode of (φ(s_{t+1}) − φ(s_t))ᵀz, subject to ‖φ(s) − φ(s′)‖ ≤ 1 for consecutive states; FoG multiplies each term of that sum by the gate w(s_{t+1}). We will state this objective and constraint in the main text, together with the practical losses (the dual update for the constraint and the SAC updates with the intrinsic reward), so that the paper is self-contained.

Reviewer_Tn5uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of unsupervised skill discovery, where conventional methods may discover diverse but undesirable skills. Motivated by the ability of foundation models to leverage pretrained knowledge to distinguish desirable from undesirable behaviors based on human preferences, the paper proposes Foundation Model Guided (FoG) skill discovery. FoG uses off-the-shelf foundation models as preference evaluators to guide exploration toward discovering desirable skills. In addition to introducing preference-aligned skill discovery through foundation model-generated preference gates, the paper extends the Wasserstein Dependency Measure (WDM) to the trajectory level, based on the observation that preferences are often related to properties of entire trajectories rather than individual states. Experiments in both state-based and pixel-based environments demonstrate that language-driven foundation models can provide useful preference signals to guide an existing unsupervised skill discovery algorithm (METRA [1]). The resulting skills reduce undesirable or potentially dangerous behaviors while maintaining behavioral diversity and improving performance on downstream tasks.

[1] Park et al. "METRA: Scalable Unsupervised RL with Metric-Aware Abstraction." 2024.

Critical strengths
  • Clear motivation: Conventional unsupervised skill discovery algorithms may learn undesirable or dangerous skills, as their objectives primarily focus on maximizing behavioral diversity. Incorporating human preferences through foundation model-based preference scoring is a reasonable and well-motivated direction.

  • Simple and intuitive approach: The proposed method is straightforward and easy to understand, with clearly defined roles for the foundation models and the underlying skill discovery algorithm (METRA). The authors further support their approach with theoretical justification and empirical results across both state- and pixel-based environments.

Critical weaknesses
  • Limited Methodological Novelty: Although the differences are discussed in Appendix C, the proposed FoG algorithm appears to be closely related to DoDont [1], with the main modification being the replacement of the learned preference gate with a zero-shot foundation model-based preference gate. Despite some differences in the theoretical formulation and implementation, the substantial similarity to DoDont raises concerns about the methodological novelty of FoG.

  • Insufficient Evaluation of Zero-shot Preference Reliability: The paper does not sufficiently evaluate how accurately foundation models can assess preferences in simulation environments such as MuJoCo and DMC. The reported differences between CLIP and SigLIP, along with the sensitivity to prompt wording, suggest that preference predictions can vary considerably depending on the foundation model and textual descriptions. Although the paper includes several ablation studies, more systematic evaluations of preference classification accuracy and robustness would strengthen the reliability of the proposed approach, particularly considering the potential domain gap between real-world image-text pretraining data and simulated robotic environments.

  • Trade-off Between Preference Alignment and Exploration Diversity: FoG regulates the diversity of skills discovered by METRA through preference-based gating. Although the experimental results demonstrate that this approach can effectively suppress undesirable behaviors, it may also restrict exploration that is necessary for discovering novel and potentially useful skills. Similar limitations are observed in the additional experiments presented in Appendix F. In particular, discovering new skills may require temporarily transitioning through states classified as undesirable. In such cases, preference gating could unintentionally discourage exploration toward valuable behaviors. The paper lacks a thorough analysis of this trade-off and its potential impact on the diversity and utility of the discovered skills.

  • Limited Evaluation on Complex Manipulation Tasks: The experimental evaluation is primarily focused on locomotion tasks, with only limited results presented for manipulation in the Franka Kitchen environment. In particular, it remains unclear how the discovered manipulation skills can be effectively utilized for downstream tasks. Although Appendix F discusses this limitation, more comprehensive experiments in manipulation domains, including evaluations of downstream task performance, would help demonstrate the generalizability of the proposed approach to more complex robotic tasks.

For these reasons, I currently lean toward rejection.

[1] Kim et al. "Do's and Don'ts: Learning Desirable Skills with Instruction Videos." 2024.

Questions for authors
  • Although the paper proposes a trajectory-level WDM formulation, the foundation model still evaluates preferences based on observations at individual time steps. While this limitation is mentioned in the Conclusion, could incorporating temporal information into the foundation model provide more informative preference signals, particularly for behaviors that depend on the evolution of states over time? Further discussion or experiments comparing single-state and temporal preference evaluation would be valuable.

  • The preferences considered in the experiments are primarily associated with visual characteristics of individual states or relatively simple physical conditions, such as avoiding flips, maintaining twisted postures, avoiding particular regions, or moving in preferred directions. However, more complex desirable skills may involve temporal dependencies, coordinated movements (e.g., periodic behavior), or interactions with objects. Can the proposed foundation model-based preference mechanism effectively guide the discovery of such more complex skills? Additional discussion or experiments involving more complex behaviors would help clarify the applicability of the proposed method.

5 comments

We thank the reviewer for the careful review, and for recognizing the motivation and the simplicity of the approach. We reply to each point below.

W1. Methodological novelty relative to DoDont

We agree that DoDont is the closest work, and we credit it for introducing the multiplicative gate, which FoG shares. The table summarizes what FoG adds. We will move this comparison from Appendix C into the main text.

DoDontFoGWhat FoG adds
RewardGated METRA reward (introduced by DoDont)Same gated reward—
What the gated reward optimizesArgued to maximize dependency between the skill and the final state under a classifier-based distance. The derivation relies on replacing φ(s) with p̂(s, s′)·φ̃(s) (App. B), which holds for a constant gate but not for a general oneProven to maximize a lower bound on dependency between the skill and the whole trajectory (Thm. 1)To our knowledge, the first rigorous account of what the gated reward optimizes; the proof extends to general bounded gates, including DoDont's
What the user provides"Do" and "don't" demonstration videos for each preferenceOne pair of sentencesNo data collection
Before skill discoveryTrain a classifier on the videosNothingNo extra training stage
Preferences without demonstrations (e.g. "twisted")Need videos of the behaviour firstStated directly in languageCovers preferences that are hard to demonstrate
Cheetah flips, pixel-only protocol35%12%Fewer undesirable skills at the same coverage
Same CLIP labels: classifier supervision vs direct gate52% (DoDont+)12%With the same CLIP labels, direct gating is far more robust than classifier training
Downstream return (AUC)0.690.79More useful skill library

How a foundation-model signal should be used. CLIP's judgment fails when it is used as a reward (FR-SAC and METRA+FR barely move). The same CLIP labels also fail when they are used to train DoDont's classifier (DoDont+ flips in 52% of episodes and falls below METRA downstream); this is the controlled comparison in the paper. They work when applied directly as a binary gate. We see this as a finding about how to use zero-shot preferences in skill discovery, beyond changing where the preference comes from.

What the gated reward optimizes. DoDont arrives at the gated reward in two steps. It first states a METRA-style objective in which the classifier bounds the representation increments, ‖φ(s) − φ(s′)‖ ≤ p̂(s, s′) (Eq. 5). It then obtains the gated reward (Eq. 7) by replacing φ(s) with p̂(s, s′)·φ̃(s) (Appendix B, Eqs. 8–10). The left-hand side, φ(s), is a function of the state s alone, whereas the right-hand side depends on the transition (s, s′). A state s_t appears in two consecutive transitions, so it would be assigned p̂(s_{t−1}, s_t)·φ̃(s_t) in one term and p̂(s_t, s_{t+1})·φ̃(s_t) in the next. These agree when p̂ is constant, which reduces to METRA up to a constant factor, but not for a general gate, so φ(s) cannot be replaced in this way. This does not affect DoDont's empirical results. It means the gated reward, which we share with DoDont, needs a different justification. Specifically, a gated sum does not telescope, so it is not a function of the final state and cannot come from the final-state dependency that METRA starts from. We therefore measure dependency between the skill and the whole trajectory, and prove that the gated reward maximizes a lower bound on it (Theorem 1). To our knowledge, this is the first rigorous account of what the gated reward optimizes. The same proof extends to any bounded gate on transitions, with the gate entering the trajectory metric, so it covers DoDont's reward as well; we will add this as a remark.

W2. Reliability of the zero-shot preference labels

We thank the reviewer for raising this. The paper showed the resulting behaviour but did not measure label accuracy directly, and we have now measured it.

Accuracy on Cheetah (new audit). We rolled out trained policies for 100 episodes of 200 steps each, 100,000 frames in total, and compared CLIP's label on every frame with the simulator (torso pitched beyond 90°). We used CLIP ViT-L/14 with the sentences from the paper, "The simulated two-legs robot flips over" and "The simulated two-legs robot stands normally". Values are means ± standard errors over policies.

Frames fromPoliciesAccuracy (%)Flipped frames detected (%)Upright frames wrongly flagged (%)Share of frames that are flipped (%)
METRA policies263.7 ± 4.071.4 ± 6.339.1 ± 2.424.6 ± 6.5
FoG policies377.0 ± 1.139.3 ± 12.122.5 ± 1.11.9 ± 0.9

The METRA rows show CLIP's accuracy on states that were not shaped by the gate. FoG's policies were trained to avoid what CLIP flags, so the few flips that remain are likely the ones CLIP tends to miss, which is consistent with the lower detection rate in the second row. CLIP's errors go both ways: it misses a share of the flipped frames and it flags a substantial share of upright ones (22–39%), so most of the frames it flags are in fact upright.

Across models and tasks. On a separate set of 3,200 frames with ground-truth labels per task (an offline check, independent of the audit above), balanced accuracy (the mean of the accuracy on desirable and on undesirable frames) is:

Model (sentences from the paper)Cheetah flipCheetah areaQuadruped area
CLIP ViT-L/1472.279.479.8
SigLIP so400m69.958.551.4

Across wordings (CLIP).

TaskWordings triedPaper's wordingRange over wordings
Cheetah flip572.246.3 – 72.2
Cheetah area679.450.1 – 82.5
Quadruped area379.875.6 – 82.5

These measurements show that:

  • The labels are noisy, and FoG works with them. CLIP is right on roughly 65–80% of frames, in line with the "about 70%" mentioned in the paper. With labels of this quality, flipped frames fall from 24.6% under METRA to 1.9% under FoG. The label only scales the diversity reward and is never optimized directly, which is what makes an imperfect zero-shot signal usable. The same labels fail when used to train a classifier (DoDont+).
  • The model and the wording matter. SigLIP is close to chance on the area tasks, and some wordings fall to chance. This agrees with the behaviour-level ablations in the paper, and we name phrasing as the main practical cost of the zero-shot interface. On the constructive side, this check is cheap though, as labelling a few thousand frames takes seconds (about 4 ms per frame for CLIP), so a wording can be validated before any training.
  • Domain gap. The audit measures accuracy in simulation directly, which is the quantity a domain gap would affect.
  • State-based tasks. There the labeler is a short program written by the language model and printed verbatim in the paper (for example, front-tip angle above 1.57 rad), so it can be checked by reading it.

We will add these tables to the paper.

W3. Preference alignment versus exploration diversity

We agree that this trade-off exists, and that the paper spreads the evidence over the main text and two appendices without presenting it as one analysis. We will bring it together. In short, the trade-off is controlled by one parameter, the gate floor α, and in our experiments the cost in diversity is small while the utility of the skills goes up.

WhatResult
Diversity under the gate (Cheetah)Coverage of about 29 cells for both FoG and METRA; flips in 12% vs 75% of episodes
Utility (downstream goal reaching with frozen skills)Return AUC 0.79 vs 0.74; final return 0.88 vs 0.85; safety 0.94 vs 0.81 (FoG vs METRA)
Gate floor on CheetahFlips in 10% of episodes at α = 0, 19% at α = 0.5, 42% at α = 0.8
Gate floor on Quadrupedα = 0 over-restricts (low coverage); α = 0.1 removes flips and restores coverage

On your specific point that reaching new skills may require passing through undesirable states: the gate removes the credit for entering such a state, but it is not a penalty. A skill that crosses an undesirable state and continues into preferred states keeps the credit it earns there, and with α > 0 it also keeps part of the credit for the crossing itself. This is the difference from a constraint or a negative reward, and α lets the user choose how strict the gate is, from the hard gate (α = 0) to plain METRA (α = 1). This does not guarantee that every useful path through such states is found, but it does not exclude them in the way a constraint would.

The limit we see is with several preferences combined, where the gate can become too restrictive (Appendix, multiple intentions). We will state this in the limitations.

W4. Manipulation and downstream utility

We thank the reviewer for this suggestion. We fully agree that a broader manipulation evaluation, including downstream tasks, would strengthen the paper. In this paper we focused on showing that a preference stated in language can steer skill discovery zero-shot, and on giving the gated objective a rigorous foundation. The Kitchen experiments were included to show that this interface carries over to manipulation without any change to the method: three different sentences move the arm's explored region accordingly, and with "cabinet is closed", cabinet opening drops from 78% of the time under METRA to 22% under FoG. These results show that language steers which configurations the arm explores; they do not show task completion. Downstream utility is so far shown on the locomotion skill libraries only. Downstream evaluation on object interaction and contact-rich tasks is a natural and important next step. We would very much have liked to include it, and we regret that we could not complete it within the rebuttal period with the compute available to us. We will discuss it clearly as a limitation and as future work in the main text; it is currently only in the appendix.

Q1. Would temporal information give more informative preference signals?

Yes. A short sequence of observations can separate behaviours that look alike in a single frame, for example the direction of a rotation or a periodic gait. In FoG the gate is the only component that looks at observations, so replacing the single-frame evaluator by one that scores a short window is the natural way to do this, as we note in the conclusion. We have not run a single-frame versus temporal comparison, and we will discuss it as future work. We would also like to clarify one term, because the opening sentence of Section 3.1 was misleading: "trajectory-level" in our formulation refers to the dependency we measure, between the skill and the whole trajectory, and not to the preference, which is defined on states. We will rewrite that sentence.

Q2. Temporal dependencies, coordinated movement and object interaction

We agree that such preferences matter, and that the experiments mostly use preferences that can be judged from a single state. In this paper we focused on showing that a preference stated in language can steer skill discovery zero-shot with a rigorous objective, and we chose preferences that a frozen image–text model can judge from one frame. Within that focus, the evidence closest to your question is the Kitchen result, where a preference about an object's state ("cabinet is closed") reduces cabinet opening from 78% to 22%. Preferences that depend on the order of states, such as periodic movement, would need an evaluator over short windows. This is a natural and important next step, which we could not complete within the rebuttal period, and we will discuss it as future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.