- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TIME, which aims at learning explorable information maximization with trajectory-level (instead of state-level) conditions. TIME uses conditioned mutual information and Lipschitz constraints to learn continuous, reusable behaviors, which is better than previous studies.
Critical strengths
-
The claim of this paper is supported by both theoretical analysis and experimental results.
-
The experimental results show that the proposed method outperforms previous baselines to some extent.
Critical weaknesses
-
The experiments are relatively simple in scenes and settings. The environments in this paper are MuJoCo-like (Gym and DMC), which are too simple. The authors may consider more complicated environments, i.e., open-world games, and some robotic scenarios with continuous control.
-
There seem to be some coefficients (, and ) in Table 3 in the appendix, while they seem not to have appeared in the main text. Maybe they should be put as the coefficients of Equation 5 or Equation 6?
-
The off-policy treatment of future-dependent rewards is underspecified. Rewards depend on future states, while the policy and critic use DDPG with replayed trajectory segments. It is unclear whether rewards are assigned to the initial transition, spread over the segment, or paired with a -step target. The future continuation in a replay segment follows an older behavior policy, so its expected reward is not generally the current policy's continuation reward. A precise Bellman target and treatment of episode boundaries, skill switches, and continuation mismatch are necessary to understand what objective is optimized.
Questions for authors
-
What is the exact DDPG critic target for a -step-dependent reward? How do you handle replay continuations generated by old policies, episode truncation, and segments crossing skill boundaries?
-
Can such methods be applied to large-scale unsupervised skill discovery, e.g., in a manner of unsupervised pre-training?
2 comments
[Response to Reviewer Toiu] (2/2)
W2: Missing reward coefficients in Eqs. (5)–(6).
Thank you for pointing this out. Eqs. (5)–(6) present the unweighted variational reward; Table 3 lists the practical reward-balancing coefficients. Based on the definitions in Appendix C.1, the weighted reward should be written explicitly as
r_\lambda(s_{t:t+k},z)={}&\log q_\theta(z\mid s_{t:t+k})\\ &-\lambda_1\log q_{\theta'}(z\mid s_t)\\ &+\frac{\lambda_2}{N}\sum_{i=1}^{N}\log q_{\theta'}(z_i\mid s_t)\\ &+\lambda_3\log N, \end{aligned}$$ with $\lambda_1=\lambda_2=0.2$, $\lambda_3=0.5$, and $N=64$. These coefficients are introduced to balance reward magnitudes and stabilize critic/policy optimization (Appendix C.1). In particular, $\lambda_1=\lambda_2$ preserves the relative balance of the two state-dependent terms. Theorem 1 applies to the unweighted objective ($\lambda_1=\lambda_2=\lambda_3=1$), not automatically to the practical weighted reward. We will add the implemented reward equation and explicitly distinguish theoretical characterization from the practical optimization surrogate. **W1: Experimental scope and complexity.** We agree that open-world environments and robotic manipulation are valuable next benchmarks. Our evaluation is intentionally focused on continuous control, which enables controlled comparisons of skill reusability, trajectory diversity, and exploration against five representative baselines. We would like to emphasize that it includes not only unconstrained locomotion but also high-dimensional Humanoid and Quadruped control and a separate obstacle-based sequential navigation task, AntMazeMultiGoal (Appendix B.4). The latter requires chaining frozen low-level skills across up to 25 sequential waypoints, with successive targets at least 7.5 units from the agent. The existing downstream results (Table 2; mean ± standard deviation over five seeds) quantify the benefit of this compositional structure: | Task | TIME | RSD (strongest baseline) | |---|---:|---:| | AntMazeMultiGoal | 0.618 ± 0.148 | 0.533 ± 0.083 | | HalfCheetahGoal | 7.400 ± 0.452 | 6.880 ± 0.430 | | HumanoidGoal | 3.722 ± 0.662 | 3.409 ± 0.766 | | QuadrupedGoal | 5.605 ± 0.665 | 4.804 ± 0.484 | TIME achieves the highest mean final performance on all four tasks, with an average improvement of 12.3% over RSD. The AntMazeMultiGoal means correspond to 61.8% versus 53.3% of waypoints reached, a difference of 8.5 percentage points. We acknowledge that TIME is initially less sample-efficient on some tasks (Appendix B.5), and these five-seed results do not establish superiority in all domains. We will narrow the empirical claims accordingly; real-robot, manipulation, and open-world generalization remain untested. **Q2: Applicability to large-scale unsupervised pre-training.** Yes in principle: TIME already follows a reward-free pre-training and task-specific adaptation paradigm. The low-level policy and discriminators are trained without task rewards, after which the low-level network is frozen and only the high-level policy is trained using downstream task rewards. Appendix D.2 examines $N\in\{16,32,64\}$ and observes qualitatively broader trajectory coverage as the skill dictionary increases. The default configuration uses 64 skills, $k=16$, and 2M pre-training frames (Table 3). TIME avoids explicit forward-dynamics modeling and free-running model rollouts in the reported training procedure. For larger observations and skill dictionaries, a shared temporal encoder and hierarchical or factorized skill representations are natural extensions. These would address the increasing trajectory-encoding cost and the linear output cost of a dense categorical discriminator as $N$ grows. Such extensions require preserving meaningful trajectory distances under the Lipschitz constraint and separately validating the enlarged skill repertoire. Scaling beyond 64 skills, visual open-world observations, and more diverse robotic domains are promising but not demonstrated by the current results. We appreciate these constructive comments. We will revise the reward equation, explicitly document the Bellman target and replay-window semantics, and distinguish the theoretical objective from its practical off-policy approximation. We hope these clarifications, together with the existing multi-horizon and sequential-control evidence, address the reviewer's concerns regarding soundness and the scope of TIME.
[Response to Reviewer Toiu] (1/2)
We thank the reviewer for recognizing the theoretical and empirical contributions of TIME and for identifying an important ambiguity in our implementation description. We agree that the temporal credit assignment and off-policy treatment of trajectory-dependent rewards should be made explicit. We address these points first, followed by the reward coefficients and the scope of our evaluation.
W3/Q1: Exact critic target and off-policy trajectory rewards.
Algorithm 1 uses replayed trajectory segments to compute the intrinsic reward with the learned discriminators, followed by DDPG updates. The key distinction is between the discriminator horizon k and the TD bootstrap horizon. For a start-aligned, one-step implementation, a sampled window (st:t+k,at:t+k−1,z) defines a reward Rt=rλ(st:t+k,z) assigned to the initial transition (st,at,st+1,z). The corresponding target is
yt=sg(Rt)+γ(1−dt)Qψˉ(st+1,πωˉ(st+1,z),z),
where ψˉ,ωˉ are target-network parameters, dt masks bootstrapping at a true terminal, and sg denotes stop-gradient. Thus, under this interpretation, k=16 specifies the discriminator's temporal context, not a 16-step TD target; the segment reward is neither divided across its constituent transitions nor followed by a γk bootstrap.
We agree with the reviewer that a replayed continuation generated by an older behavior policy is not, in general, distributed as a continuation of the current policy. In the start-aligned formulation, the effective immediate reward is the replay-conditional expectation
=\mathbb E_{\mathcal D}[r_\lambda(S_{t:t+k},z)\mid S_t=s,A_t=a,Z=z].$$ With the replay distribution and discriminators held fixed, the update is a sample-based TD update for a replay-defined surrogate. It is not an unbiased Bellman update for the current-policy conditional-MI objective when the two continuation distributions differ. Moreover, the true current-policy reward itself depends on the continuation policy; ordinary DDPG does not explicitly differentiate through that dependence. Theorem 1 concerns the variational MI surrogate under a specified joint distribution, not a guarantee of unbiased off-policy optimization. We will clarify this distinction in Sec. 3.1 and Appendix C, rather than claiming that standard off-policy TD learning eliminates the mismatch. During pre-training, Algorithm 1 samples one skill at the beginning of each episode. Consequently, the appropriate replay unit is a complete, contiguous, same-episode, same-skill window. Windows crossing a reset or skill switch must not be used as single-skill examples. For a complete-window implementation, the final $k-1$ candidate start positions of an episode cannot supply a full window. A true terminal may be the final state of an otherwise valid window; a time-limit cutoff must be distinguished from a true terminal for bootstrapping. The downstream PPO policy instead selects a skill every $T=16$ steps, with the low-level policy frozen (Appendix B.4 and C.2), so this downstream switching does not itself create off-policy intrinsic-reward windows. The existing horizon ablation (Table 5; five seeds) provides evidence that finite-horizon trajectory context is useful despite this approximation: increasing $k$ from 1 to 16 improves HalfCheetahGoal from $0.414$ to $7.400$ and QuadrupedGoal from $0.123$ to $5.605$. At $k=32$, performance decreases to $7.026$ and $4.803$, respectively. This supports our choice of a moderate horizon, although it does not isolate replay mismatch. A targeted diagnostic would compare replayed versus freshly generated continuations from matched initial states and actions using the same frozen discriminators. We regard that diagnostic, and an exact causal reformulation with a history-conditioned critic if needed, as useful directions for addressing the remaining approximation.