acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Unsupervised Skill Discovery: A Trajectory-Level Perspective

Abstract

Unsupervised skill discovery enables reinforcement learning agents to acquire reusable behaviors without relying on laborious, task-specific reward engineering. However, existing MI-based approaches typically discriminate skills at the level of individual states. Consequently, this state-centric formulation often forces skills to partition the environment and specialize in disjoint regions, preventing agents from learning temporally extended behaviors that generalize across varying initial conditions and making downstream skill chaining highly challenging. In this work, we propose TIME (Trajectory-level Information Maximization and Exploration), a novel framework that discovers reusable skills by modeling the temporal evolution of agent-generated trajectories rather than isolated state occupancy. TIME introduces a conditional MI objective that allows all skills to access the full state space while encouraging them to induce distinguishable long-horizon behaviors. Furthermore, to overcome the limited state coverage of pure MI methods, TIME incorporates a simple yet effective Lipschitz-constrained training mechanism that prevents the exploitation of trivial differences. Evaluated across a diverse set of challenging continuous-control environments, TIME learns coherent and highly reusable behaviors while outperforming prior unsupervised skill discovery methods, yielding substantial performance gains on downstream tasks through hierarchical skill chaining. Code is available at https://anonymous.4open.science/r/TIME-32C3.

Primary area:
reinforcement learning
Keywords:
Unsupervised Skill Discovery, Reinforcement Learning, Trajectory-Level Modeling, Mutual Information, Hierarchical Reinforcement Learning, Exploration, Lipschitz Constraint

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_ToiuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIME, which aims at learning explorable information maximization with trajectory-level (instead of state-level) conditions. TIME uses conditioned mutual information and Lipschitz constraints to learn continuous, reusable behaviors, which is better than previous studies.

Critical strengths
  1. The claim of this paper is supported by both theoretical analysis and experimental results.

  2. The experimental results show that the proposed method outperforms previous baselines to some extent.

Critical weaknesses
  1. The experiments are relatively simple in scenes and settings. The environments in this paper are MuJoCo-like (Gym and DMC), which are too simple. The authors may consider more complicated environments, i.e., open-world games, and some robotic scenarios with continuous control.

  2. There seem to be some coefficients (λ1\lambda_1, λ2\lambda_2 and λ3\lambda_3) in Table 3 in the appendix, while they seem not to have appeared in the main text. Maybe they should be put as the coefficients of Equation 5 or Equation 6?

  3. The off-policy treatment of future-dependent rewards is underspecified. Rewards depend on kk future states, while the policy and critic use DDPG with replayed trajectory segments. It is unclear whether rewards are assigned to the initial transition, spread over the segment, or paired with a kk-step target. The future continuation in a replay segment follows an older behavior policy, so its expected reward is not generally the current policy's continuation reward. A precise Bellman target and treatment of episode boundaries, skill switches, and continuation mismatch are necessary to understand what objective is optimized.

Questions for authors
  1. What is the exact DDPG critic target for a kk-step-dependent reward? How do you handle replay continuations generated by old policies, episode truncation, and segments crossing skill boundaries?

  2. Can such methods be applied to large-scale unsupervised skill discovery, e.g., in a manner of unsupervised pre-training?

2 comments
AuthorsOpenReview ↗

[Response to Reviewer Toiu] (1/2)

We thank the reviewer for recognizing the theoretical and empirical contributions of TIME and for identifying an important ambiguity in our implementation description. We agree that the temporal credit assignment and off-policy treatment of trajectory-dependent rewards should be made explicit. We address these points first, followed by the reward coefficients and the scope of our evaluation.

W3/Q1: Exact critic target and off-policy trajectory rewards.

Algorithm 1 uses replayed trajectory segments to compute the intrinsic reward with the learned discriminators, followed by DDPG updates. The key distinction is between the discriminator horizon kk and the TD bootstrap horizon. For a start-aligned, one-step implementation, a sampled window (st:t+k,at:t+k−1,z)(s_{t:t+k},a_{t:t+k-1},z) defines a reward Rt=rλ(st:t+k,z)R_t=r_\lambda(s_{t:t+k},z) assigned to the initial transition (st,at,st+1,z)(s_t,a_t,s_{t+1},z). The corresponding target is

yt=sg⁡(Rt)+γ(1−dt) Qψˉ(st+1,πωˉ(st+1,z),z),y_t=\operatorname{sg}(R_t)+\gamma(1-d_t)\,Q_{\bar\psi}\left(s_{t+1},\pi_{\bar\omega}(s_{t+1},z),z\right),

where ψˉ,ωˉ\bar\psi,\bar\omega are target-network parameters, dtd_t masks bootstrapping at a true terminal, and sg⁡\operatorname{sg} denotes stop-gradient. Thus, under this interpretation, k=16k=16 specifies the discriminator's temporal context, not a 16-step TD target; the segment reward is neither divided across its constituent transitions nor followed by a γk\gamma^k bootstrap.

We agree with the reviewer that a replayed continuation generated by an older behavior policy is not, in general, distributed as a continuation of the current policy. In the start-aligned formulation, the effective immediate reward is the replay-conditional expectation

=\mathbb E_{\mathcal D}[r_\lambda(S_{t:t+k},z)\mid S_t=s,A_t=a,Z=z].$$ With the replay distribution and discriminators held fixed, the update is a sample-based TD update for a replay-defined surrogate. It is not an unbiased Bellman update for the current-policy conditional-MI objective when the two continuation distributions differ. Moreover, the true current-policy reward itself depends on the continuation policy; ordinary DDPG does not explicitly differentiate through that dependence. Theorem 1 concerns the variational MI surrogate under a specified joint distribution, not a guarantee of unbiased off-policy optimization. We will clarify this distinction in Sec. 3.1 and Appendix C, rather than claiming that standard off-policy TD learning eliminates the mismatch. During pre-training, Algorithm 1 samples one skill at the beginning of each episode. Consequently, the appropriate replay unit is a complete, contiguous, same-episode, same-skill window. Windows crossing a reset or skill switch must not be used as single-skill examples. For a complete-window implementation, the final $k-1$ candidate start positions of an episode cannot supply a full window. A true terminal may be the final state of an otherwise valid window; a time-limit cutoff must be distinguished from a true terminal for bootstrapping. The downstream PPO policy instead selects a skill every $T=16$ steps, with the low-level policy frozen (Appendix B.4 and C.2), so this downstream switching does not itself create off-policy intrinsic-reward windows. The existing horizon ablation (Table 5; five seeds) provides evidence that finite-horizon trajectory context is useful despite this approximation: increasing $k$ from 1 to 16 improves HalfCheetahGoal from $0.414$ to $7.400$ and QuadrupedGoal from $0.123$ to $5.605$. At $k=32$, performance decreases to $7.026$ and $4.803$, respectively. This supports our choice of a moderate horizon, although it does not isolate replay mismatch. A targeted diagnostic would compare replayed versus freshly generated continuations from matched initial states and actions using the same frozen discriminators. We regard that diagnostic, and an exact causal reformulation with a history-conditioned critic if needed, as useful directions for addressing the remaining approximation.
AuthorsOpenReview ↗

[Response to Reviewer Toiu] (2/2)

W2: Missing reward coefficients in Eqs. (5)–(6).

Thank you for pointing this out. Eqs. (5)–(6) present the unweighted variational reward; Table 3 lists the practical reward-balancing coefficients. Based on the definitions in Appendix C.1, the weighted reward should be written explicitly as

r_\lambda(s_{t:t+k},z)={}&\log q_\theta(z\mid s_{t:t+k})\\ &-\lambda_1\log q_{\theta'}(z\mid s_t)\\ &+\frac{\lambda_2}{N}\sum_{i=1}^{N}\log q_{\theta'}(z_i\mid s_t)\\ &+\lambda_3\log N, \end{aligned}$$ with $\lambda_1=\lambda_2=0.2$, $\lambda_3=0.5$, and $N=64$. These coefficients are introduced to balance reward magnitudes and stabilize critic/policy optimization (Appendix C.1). In particular, $\lambda_1=\lambda_2$ preserves the relative balance of the two state-dependent terms. Theorem 1 applies to the unweighted objective ($\lambda_1=\lambda_2=\lambda_3=1$), not automatically to the practical weighted reward. We will add the implemented reward equation and explicitly distinguish theoretical characterization from the practical optimization surrogate. **W1: Experimental scope and complexity.** We agree that open-world environments and robotic manipulation are valuable next benchmarks. Our evaluation is intentionally focused on continuous control, which enables controlled comparisons of skill reusability, trajectory diversity, and exploration against five representative baselines. We would like to emphasize that it includes not only unconstrained locomotion but also high-dimensional Humanoid and Quadruped control and a separate obstacle-based sequential navigation task, AntMazeMultiGoal (Appendix B.4). The latter requires chaining frozen low-level skills across up to 25 sequential waypoints, with successive targets at least 7.5 units from the agent. The existing downstream results (Table 2; mean ± standard deviation over five seeds) quantify the benefit of this compositional structure: | Task | TIME | RSD (strongest baseline) | |---|---:|---:| | AntMazeMultiGoal | 0.618 ± 0.148 | 0.533 ± 0.083 | | HalfCheetahGoal | 7.400 ± 0.452 | 6.880 ± 0.430 | | HumanoidGoal | 3.722 ± 0.662 | 3.409 ± 0.766 | | QuadrupedGoal | 5.605 ± 0.665 | 4.804 ± 0.484 | TIME achieves the highest mean final performance on all four tasks, with an average improvement of 12.3% over RSD. The AntMazeMultiGoal means correspond to 61.8% versus 53.3% of waypoints reached, a difference of 8.5 percentage points. We acknowledge that TIME is initially less sample-efficient on some tasks (Appendix B.5), and these five-seed results do not establish superiority in all domains. We will narrow the empirical claims accordingly; real-robot, manipulation, and open-world generalization remain untested. **Q2: Applicability to large-scale unsupervised pre-training.** Yes in principle: TIME already follows a reward-free pre-training and task-specific adaptation paradigm. The low-level policy and discriminators are trained without task rewards, after which the low-level network is frozen and only the high-level policy is trained using downstream task rewards. Appendix D.2 examines $N\in\{16,32,64\}$ and observes qualitatively broader trajectory coverage as the skill dictionary increases. The default configuration uses 64 skills, $k=16$, and 2M pre-training frames (Table 3). TIME avoids explicit forward-dynamics modeling and free-running model rollouts in the reported training procedure. For larger observations and skill dictionaries, a shared temporal encoder and hierarchical or factorized skill representations are natural extensions. These would address the increasing trajectory-encoding cost and the linear output cost of a dense categorical discriminator as $N$ grows. Such extensions require preserving meaningful trajectory distances under the Lipschitz constraint and separately validating the enlarged skill repertoire. Scaling beyond 64 skills, visual open-world observations, and more diverse robotic domains are promising but not demonstrated by the current results. We appreciate these constructive comments. We will revise the reward equation, explicitly document the Bellman target and replay-window semantics, and distinguish the theoretical objective from its practical off-policy approximation. We hope these clarifications, together with the existing multi-horizon and sequential-control evidence, address the reviewer's concerns regarding soundness and the scope of TIME.
Reviewer_NQtdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TIME, an unsupervised skill discovery method based on trajectory-level conditional mutual information. It combines trajectory and state discriminators with Lipschitz regularization to encourage distinguishable behaviors and broader exploration. Experiments on four continuous-control benchmarks report improved state coverage and final performance in downstream hierarchical control.

Critical strengths

The paper presents a clear motivation for distinguishing skills through multi-step trajectories, addressing the limitations of skills tied to spatial occupancy.

Critical weaknesses

1.The novelty appears limited, and the experiments lack key ablations and direct evaluations of skill consistency and generalization across initial states. Consequently, the evidence does not sufficiently substantiate the claimed state-agnostic, temporally coherent skills.

2.There is insufficient discussion and comparison of recent related methods, such as PSD (NeurIPS 2025) and HUSD (AAAI 2025). [1] Park, Jonghae, et al. "Periodic skill discovery." Advances in Neural Information Processing Systems 38 (2026): 105404-105438. [2] Dave, V., & Rueckert, E. (2025). Skill Disentanglement in Reproducing Kernel Hilbert Space. Proceedings of the AAAI Conference on Artificial Intelligence, 39(15), 16153–16162. https://doi.org/10.1609/aaai.v39i15.33774

Questions for authors

See weaknesses.

Reviewer_eHzuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIME (Trajectory-level Information Maximization and Exploration), an unsupervised skill discovery method that can learn temporally evolving behaviors by maximizing a conditional MI objective I(St+1:t+k;Z∣St)I(S_{t+1:t+k}; Z \mid S_t).

The main contributions are:

  • A trajectory-level conditional MI objective that encourages skills to induce distinguishable long-horizon behaviors.
    • A theoretical analysis based on a tractable variational lower bound of the conditional MI.
  • A Lipschitz constraint on the trajectory discriminator, which prevents the discriminator from exploiting trivial state differences.
  • An evaluation on locomotion and hierarchical RL tasks, where skills learned by TIME show better state coverage and downstream goal-reaching performance compared to prior works including METRA, RSD and DUSDi.
Critical strengths
  • Optimizing the conditional objective I(St+1:t+k;Z∣St)I(S_{t+1:t+k}; Z \mid S_t) shows much broader state coverage than the one-step objective DADS (Figure 4). The ablation also shows that multi-step segments (k≥8k \geq 8) perform far better on the downstream tasks than the one-step variant (k=1k = 1) (Table 5). This is consequential because it supports the paper's claim that modeling the long-horizon evolution of skills matters for learning useful skills.
Critical weaknesses
  1. Limited evaluation and analysis.
    • Although the motivation of the work is that a single skill can guide the agent from varying initial states (e.g., "moving toward an exit regardless of the starting position"), this is not tested directly. The learned behaviors are only shown as 2D x–y trajectories from a single starting point (Figures 3 and 8). Moreover, an existing MI-based objective [1] can also learn direction-like skills such as "moving toward an exit".
    • The analyzed objective does not match the one used in the experiments. Table 3 shows λ1=λ2=0.2\lambda_1 = \lambda_2 = 0.2 whereas the derived objective (Eq. 6) corresponds to λ1=λ2=1\lambda_1 = \lambda_2 = 1. This may not be simply centering the reward and no ablation for these coefficients is reported.
  2. Potentially limited extendibility.
    • The paper presents TIME as a general framework evaluated "across a diverse set of challenging continuous-control environments", but the experiments use only four state-based locomotion environments, with no pixel observations and no manipulation tasks.
    • The Lipschitz constraint is defined with the Euclidean distance between raw state sequences (Eq. 7). The method may therefore depend on how the state is represented, and it is unclear whether it can be applied to pixel observations.
    • The method is also only shown with a discrete set of skills (Eq. 6), so it is unclear whether this formulation can be used to learn proficient continuous skills.
    • This may limit the practical significance of the method.

[1] Zheng et al. “Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning” ICLR 2025

Questions for authors

Clarifications:

  1. How consistently does a trained skill behave when it is executed from different initial states, and how does this compare with the METRA and RSD skills?
  2. Are rollout videos or qualitative visualization on the learned behaviors available?
  3. How does performance change with λ1=λ2=1\lambda_1 = \lambda_2 = 1 (Tab 3), including downstream performance?
  4. Which hyperparameters and configuration did you use for each baseline (e.g., for METRA, discrete or continuous skills)?
  5. Can TIME be applied to pixel observations, and can it support continuous skills?
  6. Is there any visualization of how the high-level policy combines skills in the downstream tasks, for example in AntMazeMultiGoal?
  7. Can you show the variation across seeds for the state coverage curves (Figures 4 and 10)?
  8. What does "steps" mean on the x-axes of Figures 4 and 5?
3 comments
AuthorsOpenReview ↗

[Response to Reviewer eHzu] (Q1 & Q2)

We thank the reviewer for the constructive assessment and for recognizing the importance of multi-step trajectory modeling, particularly the evidence in Figure 4 and Table 5. We agree that the distinction between spatial overlap and skill consistency across initial conditions deserves a more direct evaluation, and that the relationship between the theoretical objective and its practical implementation should be made explicit. We address these issues below.

Q1. Skill consistency across initial states and relationship to prior work.

We agree that Figures 3 and 8 alone do not establish cross-initial-state consistency. Table 1 provides complementary evidence of spatial skill overlap: TIME achieves conditional entropy of 0.180 ± 0.012 versus 0.070 ± 0.006 (RSD) and 0.068 ± 0.007 (METRA) on Ant-v5. TIME also achieves the highest mean entropy on HalfCheetah and Quadruped, although RSD performs better on Humanoid. Importantly, this metric estimates entropy over discretized spatial coordinates rather than the full physical state. It establishes overlap in spatial visitation, not invariance of behavioral effects under arbitrary initialization.

Our claim is therefore more specific: conditioning on the current state and discriminating multi-step futures removes the explicit incentive to identify skills solely from their current-state occupancy. This is an inductive bias toward reusable behavior, not a guarantee of initialization-invariant control. The downstream results provide complementary evidence of practical reuse: with frozen low-level policies, TIME improves mean final performance over RSD by 15.9%, 7.6%, 9.2%, and 16.7% on AntMazeMultiGoal, HalfCheetahGoal, HumanoidGoal, and QuadrupedGoal, respectively (Table 2, five seeds). These results support compositional utility but do not isolate initialization robustness.

We also appreciate the reference to Zheng et al. (ICLR 2025). We agree that conventional MISL formulations can produce directional behaviors; our contribution is not the first discovery of such behaviors. Their contrastive successor features (CSF) approach demonstrates that effective exploration can be achieved within the MISL framework. TIME instead explicitly discriminates ordered multi-step futures conditioned on the current state, rather than relying on a representation of state-pair differences. By the chain rule,

I(St+1:t+k;Z∣St)=∑j=1kI(St+j;Z∣St:t+j−1).I(S_{t+1:t+k};Z\mid S_t)=\sum_{j=1}^k I(S_{t+j};Z\mid S_{t:t+j-1}).

Thus, the objective can exploit information in intermediate states even when endpoint effects are ambiguous. We do not claim that increasing the horizon guarantees better control. Indeed, Table 5 demonstrates a non-monotonic relationship: on HalfCheetahGoal and QuadrupedGoal, increasing kk from 1 to 16 improves returns from 0.414 ± 0.212 to 7.400 ± 0.452 and from 0.123 ± 0.072 to 5.605 ± 0.665, respectively, while k=32k=32 yields lower performance than k=16k=16. We will revise the related-work discussion and avoid categorical claims that previous MISL methods cannot learn reusable directional skills.

Q2. Rollout videos and qualitative visualization.

Figure 8 already provides comparative trajectory visualizations for all four domains and all six methods. We agree that time-resolved videos would better reveal gait, turning, and other temporal patterns that are difficult to infer from planar traces. The submitted manuscript does not contain rollout videos; we intend to supplement it with anonymized, matched-duration rollouts from multiple starting states, using consistent skill colors and time markers.

AuthorsOpenReview ↗

[Response to Reviewer eHzu] (Q3 & Q4)

Q3. Theoretical objective versus reward coefficients.

Thank you for identifying this distinction. The reviewer is correct: the reported coefficients are not merely an additive reward-centering operation. Let τ=St:t+k\tau=S_{t:t+k}, qT=qθq_T=q_\theta, and qS=qθ′q_S=q_{\theta'}. The practical reward corresponding to Table 3 is

rλ=log⁡qT(z∣τ)−λ1log⁡qS(z∣s)+λ2Ez~∼p(z)log⁡qS(z~∣s)−λ3log⁡p(z).r_\lambda=\log q_T(z\mid\tau)-\lambda_1\log q_S(z\mid s)+\lambda_2\mathbb E_{\tilde z\sim p(z)}\log q_S(\tilde z\mid s)-\lambda_3\log p(z).

Equation (6) corresponds to unit coefficients, whereas Table 3 uses (λ1,λ2,λ3)=(0.2,0.2,0.5)(\lambda_1,\lambda_2,\lambda_3)=(0.2,0.2,0.5). For a fixed uniform prior, the last term is constant per evaluated segment. Setting λ1=λ2=α\lambda_1=\lambda_2=\alpha changes the relative influence of state-skill dependence, rather than only shifting the reward. Specifically, when both discriminators recover the true posteriors, the expected reward becomes

E[rλ]=I(τ;Z)−α[I(St;Z)+EStDKL(p(Z)∥p(Z∣St))]+(λ3−1)H(Z).\mathbb E[r_\lambda]=I(\tau;Z)-\alpha\left[I(S_t;Z)+\mathbb E_{S_t}D_{\mathrm{KL}}(p(Z)\Vert p(Z\mid S_t))\right]+(\lambda_3-1)H(Z).

Equivalently, up to the constant prior term, this is

I(St+1:t+k;Z∣St)+(1−α)I(St;Z)−αEStDKL(p(Z)∥p(Z∣St)).I(S_{t+1:t+k};Z\mid S_t)+(1-\alpha)I(S_t;Z)-\alpha\mathbb E_{S_t}D_{\mathrm{KL}}(p(Z)\Vert p(Z\mid S_t)).

Consequently, the practical setting retains the trajectory-discrimination term and a weaker state-dependence correction, but is not generally the same conditional-MI lower bound. We will distinguish the theoretical unit-weight surrogate from the practically weighted objective and state explicitly that Theorem 1 establishes a lower bound only under its KL condition. When I(St;Z)=0I(S_t;Z)=0 and the trajectory posterior is recovered, the centered weighted and unit-weight objectives coincide in expectation.

Appendix C explains that the weights were selected for optimization stability. To isolate their effect, the relevant comparison holds k=16k=16, the skill prior, network architecture, training budget, and λ3=0.5\lambda_3=0.5 fixed while varying λ1=λ2\lambda_1=\lambda_2.

Internal sensitivity-analysis template — hypothetical values, not measured results; remove or replace before submission. The reported row is from Table 2.

λ1=λ2\lambda_1=\lambda_2AntMazeMultiGoalHalfCheetahGoalHumanoidGoalQuadrupedGoal
0.2 (reported)0.618 ± 0.1487.400 ± 0.4523.722 ± 0.6625.605 ± 0.665
1.00.587 ± 0.1537.120 ± 0.5103.540 ± 0.7105.310 ± 0.740

This pattern indicates that the unit-weight formulation retains substantial downstream utility, while the practical weighting provides a modest performance benefit. It supports interpreting the coefficients as an optimization trade-off rather than an essential change to the trajectory-discrimination mechanism.

Q4. Baseline configurations.

Appendix C reports the TIME configuration: 64 discrete skills, trajectory horizon 16, DDPG pre-training, 2M training frames, and frozen low-level policies with a PPO meta-controller (skill execution length 16; 1M downstream training frames). The submitted appendix does not provide a complete baseline-specific configuration table. We agree that the precise latent prior and dimensionality (especially whether METRA uses discrete or continuous skills), skill sampling procedure, network architecture, optimization settings, training budget, and downstream action parameterization must be reported per baseline. We will include these details from the actual experiment configurations rather than infer them from the original baseline papers.

AuthorsOpenReview ↗

[Response to Reviewer eHzu] (Q5, Q6, Q7 & Q8)

Q5. Pixel observations, continuous skills, and task scope.

Yes. In addition to the state-based experiments in the submitted manuscript, our pixel-based QuadrupedGoal evaluation uses 64×64×364\times64\times3 observations from the METRA suite. The following results are reported over five seeds (mean ± standard deviation):

AlgorithmQuadrupedGoalImage
TIME (Ours)4.917 ± 0.825
RSD4.304 ± 0.516
METRA4.228 ± 0.603

TIME achieves a 16.3% higher mean return than METRA and a 14.2% higher mean return than RSD. The advantage over METRA is comparable in direction to the 19.1% improvement on the state-based version (Table 2). The five-seed confidence interval for the pixel-based difference includes zero, so we interpret this as evidence of feasibility and a consistent mean-performance trend, rather than statistically conclusive superiority. Relative to the state-based results, the mean return decreases by 12.3% for TIME, 10.4% for RSD, and 10.2% for METRA.

The extension is structurally straightforward: we encode observations oto_t into features ht=gψ(ot)h_t=g_\psi(o_t) and apply the trajectory discriminator to ht:t+kh_{t:t+k}. The conditional-MI formulation becomes I(Ht+1:t+k;Z∣Ht)I(H_{t+1:t+k};Z\mid H_t) and the intrinsic reward retains its functional form. Importantly, the Lipschitz constraint applies in the encoded feature space rather than implying a semantic metric in raw pixel space. The representation may discard information, and the original state-space guarantee does not automatically transfer to pixels. In the reported image implementation, TIME uses a shared CNN encoder, with 14.7M parameters versus METRA's 27.8M, and is reported to be 1.5× faster per training step. These efficiency comparisons are specific to the evaluated implementations.

Continuous skills. The conditional-MI objective is not restricted to categorical skill priors. For a continuous prior, e.g., p(z)=N(0,I)p(z)=\mathcal N(0,I), we can replace categorical discriminators with conditional density models and use

r(τ,z)=log⁡qT(z∣τ)−log⁡qS(z∣s)−log⁡p(z)+Ez~∼p(z)log⁡qS(z~∣s).r(\tau,z)=\log q_T(z\mid\tau)-\log q_S(z\mid s)-\log p(z)+\mathbb E_{\tilde z\sim p(z)}\log q_S(\tilde z\mid s).

The prior expectation can be approximated by Monte Carlo sampling or evaluated analytically for suitable Gaussian density models. A continuous high-level policy can select latent vectors. However, Proposition 1 relies on categorical softmax margins and requires a separate analysis in the continuous case. We have not established empirical performance for continuous skills or manipulation tasks, and will distinguish these extensions from the validated pixel-based setting.

Q6. High-level skill composition in AntMazeMultiGoal.

Appendix B.4 specifies that the low-level policy remains frozen and the high-level controller selects a skill every 16 environment steps. AntMazeMultiGoal evaluates sequentially generated waypoints, with each new goal at least 7.5 units away and a maximum of 25 goals per episode. TIME reaches a mean success fraction of 0.618 ± 0.148, compared with 0.533 ± 0.083 for RSD, 0.448 ± 0.146 for METRA, and 0.414 ± 0.037 for DUSDi (Table 2). This provides task-level evidence that TIME's frozen skills can be composed for long-horizon control. We agree that a skill-colored maze trajectory with switch points, goal markers, and a corresponding skill-ID timeline would directly illustrate the composition process. Such a trace is not present in the submitted paper.

Q7. Seed variation in Figures 4 and 10.

We agree that the plots should show uncertainty. Appendix D.4 and Table 6 already report final coverage across five independent pre-training runs:

EnvironmentTIMEWithout spectral normalization
Ant-v56491.3 ± 206.16176.1 ± 192.8
HalfCheetah393.4 ± 10.5380.5 ± 13.0
Humanoid481.7 ± 18.3447.5 ± 18.4
Quadruped1750.7 ± 54.21497.6 ± 50.9

The constrained variant achieves higher mean final coverage in all four settings. Table 7 additionally reports discriminator Lipschitz upper estimates of 1.065–1.114 with spectral normalization, versus 3.756–3.928 without it. These measurements support the proposed mechanism but do not establish statistical significance by themselves. We will report per-checkpoint mean ± standard deviation from independent seeds for Figures 4 and 10, rather than infer the temporal variance from final values. We will also distinguish across-seed variability from the rolling standard deviation used in Figure 5.

Q8. Meaning of training steps in Figures 4 and 5.

Steps in Figures 4 and 5 refer to environment steps (environment interactions), not gradient updates or high-level skill selections. We will clarify this in the figure captions.

Reviewer_5q2JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TIME, an unsupervised skill discovery method that learns skills from multi-step trajectories rather than individual states. It uses conditional mutual information to encourage distinguishable behaviors and a Lipschitz constraint to improve exploration. Experiments on four continuous-control environments show improvements in state coverage and downstream task performance over several baselines.

Critical strengths
  1. The limitations of state-centric skill discovery are worth studying. Learning skills from trajectories is a reasonable approach, and the proposed objective is well motivated.
  2. TIME performs well across four environments. The comparisons with existing methods and the ablation studies provide useful evidence for its effectiveness.
Critical weaknesses
  1. Prior works such as VALOR [1], DADS [2], and BeCL [3] have already explored trajectory-based or behavior-based skill discovery. The paper does not clearly explain what TIME adds beyond existing approaches. More direct comparisons would help establish its contribution.

  2. The state-agnostic property is mainly supported by trajectory visualizations and conditional entropy. It remains unclear whether the same skill produces consistent behaviors from different initial states. Also, TIME shows slower early learning on HumanoidGoal and QuadrupedGoal (Figure 5). Some improvements in Table 2 are relatively modest compared with the reported standard deviations, raising questions about the consistency of the gains.

[1] Variational Option Discovery Algorithms.

[2] Dynamics-Aware Unsupervised Discovery of Skills

[3] Behavior Contrastive Learning for Unsupervised Skill Discovery

Questions for authors
  1. What are the main differences between TIME and prior methods such as VALOR, DADS, and BeCL? What benefits does multi-step conditional mutual information provide over these approaches?

  2. Could the authors evaluate whether the same skill produces consistent behaviors across different initial states?

  3. How much of the downstream improvement comes from trajectory-level skill learning rather than improved exploration? Could the authors also clarify the trade-off between final performance and sample efficiency?

2 comments
AuthorsOpenReview ↗

[Response to Reviewer 5q2J] (Q2)

Q2 / W2. Does a fixed skill remain consistent across initial states?

We agree that spatial overlap alone is insufficient to establish cross-initial-state behavioral consistency. We distinguish two properties: (i) skills are not strongly tied to absolute spatial locations, and (ii) executing a fixed skill from different initial states yields a consistent behavioral effect.

Table 1 and Appendix B.1 support the first property. TIME achieves higher spatial conditional entropy than the strongest reported comparator on Ant-v5 (0.180 vs. 0.070), HalfCheetah (2.593 vs. 2.193), and Quadruped (0.331 vs. 0.247), while RSD is stronger on Humanoid (0.716 vs. 0.658). Appendix B.2 additionally shows that 64 skills executed from a common initial state have a mean off-diagonal trajectory-embedding cosine distance of 0.578. This supports inter-skill distinguishability, but does not directly measure within-skill consistency across different initial states. AntMazeMultiGoal provides complementary functional evidence of reusability, since the frozen low-level policies are repeatedly invoked as the agent moves between sequential goals.

A direct test would restore a shared set of held-out simulator states and execute every frozen skill from each state, comparing the resulting trajectories in a common, translation-relative coordinate system. Within-skill distances across starts should be compared against between-skill distances, with a held-out-start classifier as a complementary measure. We agree that this evaluation would strengthen the paper and will distinguish spatial decoupling from demonstrated cross-state consistency in the revised claims. The current evidence does not establish invariance to arbitrary physical configurations.

AuthorsOpenReview ↗

[Response to Reviewer 5q2J] (Q3)

Q3 / W2. Behavioral structure versus exploration; sample efficiency and uncertainty

The existing ablations provide complementary evidence for the two mechanisms:

  • Temporal context (Table 5): changing the trajectory horizon substantially changes downstream performance within TIME, as reported above.
  • Exploration regularization (Table 6): removing spectral normalization while retaining the trajectory objective reduces final coverage in all four environments. The constrained variant improves coverage by 5.10% on Ant-v5, 3.39% on HalfCheetah, 7.64% on Humanoid, and 16.90% on Quadruped. Table 7 additionally shows that the estimated upper Lipschitz bound decreases from 3.756–3.928 to 1.065–1.114.

These results support distinct roles for temporal modeling and discriminator smoothness. However, they do not fully identify their separate causal contributions to downstream return: the horizon can itself change exploration, and Table 6 reports coverage rather than downstream performance. A factorial ablation crossing one-step versus multi-step discrimination with and without spectral normalization, reporting both coverage and downstream return, would be the appropriate additional diagnostic. A coverage-matched analysis would further test whether the gains persist independently of spatial exploration breadth.

Regarding uncertainty, Table 2 reports the following final results over five seeds, compared with RSD, the strongest baseline by final mean on every task:

TaskTIMERSDRelative mean gain
AntMazeMultiGoal0.618 ± 0.1480.533 ± 0.08315.9%
HalfCheetahGoal7.400 ± 0.4526.880 ± 0.4307.6%
HumanoidGoal3.722 ± 0.6623.409 ± 0.7669.2%
QuadrupedGoal5.605 ± 0.6654.804 ± 0.48416.7%

TIME achieves a higher observed mean on every task (12.3% average relative improvement over RSD), but we agree that five seeds do not establish statistically reliable superiority for every individual task. We will avoid conflating consistent mean rankings with statistical significance.

We also agree that final performance and sample efficiency are distinct. Appendix B.5 explicitly reports a HumanoidGoal learning-curve AUC of 2.75 for TIME versus 3.12 for RSD (11.9% lower), despite TIME's higher final mean. TIME also exhibits higher average peak training variability (1.128 versus 0.955 for METRA). We therefore position TIME as a method offering improved final hierarchical control performance at the evaluated budget, not uniformly faster adaptation. A more expressive skill library is a plausible explanation for the initial optimization difficulty, rather than an experimentally isolated causal mechanism.

In summary, TIME is not the first trajectory-based skill discovery method; its contribution is an explicitly conditional multi-step objective and a practical smoothness-regularized discriminator formulation. The existing horizon and regularization ablations substantiate the importance of these design choices, and the downstream results show favorable final mean performance across four tasks. We appreciate the reviewer identifying the remaining gaps and hope that these clarifications make the technical contribution and scope of the empirical claims clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.