acceptodds
Under review as a conference paper at ICLR 2027

Hamiltonian Temporal Contrastive Curiosity for Reinforcement Learning Exploration

Abstract

Efficient exploration is essential for reinforcement learning in physical control, where useful behavior depends on temporally extended dynamics. Existing curiosity methods typically reward prediction error, state novelty, uncertainty, representation change, or instantaneous physical variation. Hamiltonian-based exploration further introduces physical structure by rewarding changes in a learned Hamiltonian across individual transitions, but such signals remain local in time and do not directly reveal whether an action reaches an informative future over longer horizons. This raises a key question: how can exploration identify action-conditioned futures that remain poorly captured over multiple temporal scales while reflecting structured variation in the underlying dynamics? To address this challenge, we propose Hamiltonian Temporal Contrastive Curiosity (HTC), which formulates exploration as temporal future discovery in a learned Hamiltonian phase space. HTC trains a horizon-conditioned contrastive critic on same-episode future pairs and uses the positive-pair representation distance to prioritize action-conditioned futures that remain insufficiently organized by the current temporal representation. Multi-step Hamiltonian variation further provides a bounded structural modulation of this temporal curiosity signal, allowing exploration to account for structured changes in the learned phase dynamics without treating energy variation itself as the exploration objective. We evaluate HTC on six DeepMind Control Suite tasks with SAC and PPO against representative exploration methods including ICM, RIDE, E3B, and Plan2Explore, together with a single-step Hamiltonian-change curiosity variant, with additional coverage, ablation, and Hamiltonian diagnostics characterizing its exploration behavior and learned phase-space structure. A quick visual overview of this work is available at https://iclr27-htc2.github.io/.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, World models, Intrinsic motivation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RP23OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an exploration objective that encourages informative temporal changes over the future at different time scales modulated by a change in a learned Hamiltonian, which represents a change in the total energy of the system. The change in energy incorporates the physical structure of the system. This modulated objective addresses the limitation of one-step Hamiltonian curiosity exploration, which focuses only on local changes of the Hamiltonian and does not capture long-term temporal variations. The new objective aims to capture both physical structure via the Hamiltonian and temporal structure via contrastive learning. The paper evaluates the proposed method on the DMC environments, and it shows strong performance. The paper also analyzes the method by conducting an ablation study and shows that both physical structure and temporal structure affect the method's performance.

Critical strengths
  • The paper is clear and easy to follow. I understand the paper's claims and contributions.
  • The various evaluation metrics in the experiment section help in understanding the method's performance not just with respect to the final return but also with respect to state coverage and robustness to physical changes.
  • The method shows robust performance across physical changes in the environment.
Critical weaknesses
  • From my perspective, the method's performance is not much stronger than the baselines; many of the baselines perform as well as the proposed method, and some results have high variance.
  • The RND warm start is practically justified but undermines the main idea of the paper, and it seems to affect performance significantly, as shown in the ablation study.
  • The figures (2, 4, and 5) are a bit hard to read; the font is small. Can you make it larger?
  • The clipping in equation 13 seems somewhat arbitrary; I understand that you want to bound the modulation via tanh, but can you justify the clipping term? Especially since you are dividing ΔHt,k\Delta H_{t,k} by a running scale σH\sigma_{H}, which in principle should also bound ΔHt,k\Delta H_{t,k}.
Questions for authors
  • Can the authors run the method also on the humanoid environment? That helps in understanding the limitations of such a method.
5 comments
AuthorsOpenReview ↗

Response to Reviewer RP23 (1/3)

Q1: The performance improvements of HTC² over existing baselines appear limited, and some results exhibit high variance.

A1: We thank the reviewer for this valuable comment. We further evaluate HTC² from two complementary perspectives: final control performance and sample efficiency. Beyond final Episode Return, we provide additional Learning AUC and Steps-to-Threshold analyses based on the complete learning curves in Appendix Figures 6–7, allowing us to quantify training efficiency and environmental interaction costs.

(1). Final control performance. As reported in Table 4, HTC² achieves higher mean returns across multiple DMC tasks under both SAC and PPO. For example:

  • SAC / Walker Walk: HTC² achieves 827.0 ± 115.2, outperforming the strongest baseline RIDE (717.9 ± 67.5) by 15.2%.
  • PPO / Walker Run: HTC² achieves 307.9 ± 22.8, improving over E3B (277.5 ± 16.4) by 11.0%.
  • PPO / Hopper Hop: HTC² achieves 100.7 ± 58.8, improving over ICM (89.5 ± 61.2) by 12.5%.

HTC² also achieves the highest mean returns on SAC Walker Run, Hopper Hop, Hopper Stand, and Cheetah Run, as well as PPO Walker Walk, Walker Stand, Hopper Stand, and Cheetah Run. These results demonstrate consistent competitiveness across different continuous-control tasks and both SAC/PPO optimizers, rather than improvements concentrated on a single environment.

(2). Sample efficiency: fewer interactions to reach the same target return.

An important benefit of exploration is not only achieving higher final returns, but also reaching a given control performance with fewer environment interactions. Appendix Figure 6 (SAC) and Figure 7 (PPO) present the complete learning curves. HTC² achieves higher Episode Returns throughout substantial portions of training on several tasks, with particularly favorable learning trajectories on PPO Walker Run and Hopper Stand. To quantify these observations, we introduce two additional metrics using the existing five-seed training curves:

  • Steps-to-Threshold: We set a common target at 80% of the strongest core baseline's final mean return. A run reaches the target after three consecutive evaluation windows satisfy this criterion. We compare the interactions required by HTC² against the fastest core baseline.
  • Learning AUC: We integrate Episode Return over the same interaction budget and normalize by training duration, measuring average performance throughout training. We compare against the strongest core baseline in AUC.

Table R1. Sample-efficiency gains over strong baselines (5 seeds).

SettingSteps SavedAUC Gain
SAC / Hopper Stand+11.9%+4.9%
SAC / Cheetah Run+16.8%+4.9%
PPO / Hopper Stand+10.5%+13.6%
PPO / Cheetah Run+20.5%+1.5%

Positive values indicate fewer interactions or higher Learning AUC for HTC². Each metric uses its strongest corresponding core baseline. Values are point estimates summarized over five seeds. The additional experiments show that HTC² achieves lower estimated interaction costs in all four SAC/PPO comparisons.

Specifically, under SAC, HTC² reduces the interactions required to reach the target return by 11.9% on hopper_stand and 16.8% on cheetah_run. Under PPO, the corresponding reductions are 10.5% and 20.5%, respectively. Furthermore, HTC² improves Learning AUC by 4.9%, 4.9%, 13.6%, and 1.5% across these four comparisons, indicating favorable cumulative control performance throughout training. These findings complement the learning curves in Appendix Figures 6–7, demonstrating that the benefits of HTC² extend beyond final returns to the interaction efficiency required to achieve comparable control performance.

Overall, HTC² achieves higher final mean returns across multiple DMC tasks, while our additional sample-efficiency analysis shows lower estimated interaction costs and improved Learning AUC in the evaluated settings.

AuthorsOpenReview ↗

Response to Reviewer RP23 (2/3)

Q2: Does RND warm-start dominate the performance gains of HTC²?

A2: We sincerely thank the reviewer for raising this important question. RND warm-start primarily serves as an initialization mechanism for early exploration, whereas the core exploration mechanism of HTC² is multi-horizon temporal contrastive learning with Hamiltonian physical guidance. To distinguish their contributions, we introduce additional RND-only baselines under SAC/PPO and examine the existing w/o RND ablation.

(1). Motivation for RND warm-start. Following the exploration initialization strategy in DreamSAC [1], HTC² incorporates RND during early training to support the initial learning of the Hamiltonian World Model and Temporal Contrastive Critic. Specifically, all experiments use a total training budget of 5M environment interactions. RND initializes exploration during the first 50K interactions, followed by a gradual transition from RND to HTC² temporal curiosity over the next 200K interactions. After 250K interactions, the intrinsic reward is entirely provided by HTC² for the remaining 4.75M interactions (95% of the total training budget), without any RND contribution. Therefore, RND serves only as an early exploration initialization mechanism, while HTC²'s multi-horizon temporal contrastive curiosity drives exploration throughout the vast majority of training, rather than relying on RND as the primary exploration reward.

(2). Additional RND-only comparisons. We introduce RND + SAC/PPO baselines on walker_walk, hopper_stand, and cheetah_run, using the same interaction budgets and evaluation protocols to directly compare the final control performance of RND-only and HTC². Table R3. RND-only comparison under SAC (mean ± std).

MethodWalker WalkHopper StandCheetah Run
RND + SAC821.9 ± 88.3871.4 ± 31.1573.2 ± 118.7
HTC² + SAC827.0 ± 115.2883.8 ± 12.5656.5 ± 45.0

Table R4. RND-only comparison under PPO (mean ± std).

MethodWalker WalkHopper StandCheetah Run
RND + PPO705.3 ± 94.0650.0 ± 83.5447.5 ± 8.1
HTC² + PPO733.6 ± 160.7670.2 ± 25.1458.0 ± 27.4

The additional experiments show that HTC² achieves higher mean returns than RND-only in all six SAC/PPO comparisons. In particular, the improvement reaches 14.5% on cheetah_run under SAC, while the remaining tasks also show additional gains of varying magnitudes. These results indicate that the performance improvements of HTC² are not solely attributable to standalone RND exploration rewards.

(3). Evidence from the w/o RND ablation. Furthermore, when RND warm-start is completely removed, HTC² still achieves mean returns of 785.1, 227.8, and 578.8 on walker_walk, hopper_hop, and cheetah_run, respectively. These results demonstrate that HTC² can learn effective control policies through its own multi-horizon temporal contrastive exploration mechanism even without RND initialization. RND warm-start improves early exploration conditions but is not a prerequisite for HTC² to acquire effective exploration capabilities. Overall, RND warm-start and the core HTC² mechanism play distinct roles: the former initializes early exploration, while the latter enables long-horizon future-state discovery through multi-horizon temporal contrastive learning and Hamiltonian physical guidance.


Q3: The fonts and numerical annotations in Figures 2, 4, and 5 are relatively small. Could their readability be improved?

A3: We sincerely thank the reviewer for the careful examination of our figures and this valuable suggestion. Figures 2, 4, and 5 are provided in vector format, preserving the clarity of graphical elements and text when enlarged. To further improve readability at the normal viewing size, we have increased the font sizes of axis labels, legends, subplot titles, and key numerical annotations while preserving all original experimental data and visualizations. These improvements will be reflected in the revised manuscript PDF.

References

[1] Tang et al. DreamSAC: Learning Hamiltonian World Models via Symmetry Exploration. CVPR, 2026.

AuthorsOpenReview ↗

Response to Reviewer RP23 (3/3)

Q4: Why is clipping needed in Eq.13 when tanh already bounds the modulation? Does the running scale not sufficiently constrain the normalized input?

A4: We sincerely thank the reviewer for this insightful mathematical question. The tanh function already guarantees bounded Hamiltonian modulation, while clipping serves as a conservative safeguard against extreme normalized energy variations rather than imposing an additional reward-magnitude constraint.

(1). Mathematical justification. Let x=ΔHt,k/(σH+ϵ)≥0x=\Delta H_{t,k}/(\sigma_H+\epsilon)\geq0. The Hamiltonian gates with and without clipping are

GHclip=1+αHtanh⁡(min⁡(x,cH))G_H^{clip}=1+\alpha_H\tanh(\min(x,c_H)),

GHno−clip=1+αHtanh⁡(x)G_H^{no-clip}=1+\alpha_H\tanh(x).

Since tanh is monotonically increasing and bounded,

0≤GHno−clip−GHclip≤αH[1−tanh⁡(cH)]0\leq G_H^{no-clip}-G_H^{clip}\leq\alpha_H[1-\tanh(c_H)].

The two gates are identical when x≤cHx\leq c_H; clipping only affects inputs exceeding this threshold.

With our settings αH=0.1\alpha_H=0.1 and cH=5c_H=5,

GHno−clip−GHclip<9.1×10−6G_H^{no-clip}-G_H^{clip}<9.1\times10^{-6}. Therefore, tanh provides the essential boundedness, while clipping introduces a negligible difference in the resulting Hamiltonian reward modulation.

(2). Design rationale. The running scale σH\sigma_H adapts to the evolving magnitude of Hamiltonian energy variations, but normalization does not impose a strict upper bound on the normalized input, particularly during early training or under unusually large energy changes. We therefore adopt a conservative threshold cH=5c_H=5, restricting extreme normalized inputs only after tanh has nearly saturated. This is an auxiliary numerical safeguard rather than a separate exploration mechanism or a source of performance gains. We will clarify the distinct roles of running-scale normalization, tanh modulation, and clipping, together with the above mathematical justification, in the revised manuscript.


Q5: Can HTC² be evaluated on more challenging Humanoid environments to further assess its applicability?

A5: We sincerely thank the reviewer for this valuable experimental suggestion. We extend our evaluation to two challenging high-dimensional continuous-control tasks, humanoid_stand and humanoid_walk, comparing HTC² against ICM, RIDE, E3B, and C-TeC under SAC to assess its performance in complex multi-joint coordination and balance control.

(1). Additional Humanoid experiments. All methods use the same environment interaction budget and evaluation protocol. We report final Episode Return as mean ± standard deviation over four random seeds. Table R5. Final performance on DMC Humanoid tasks (SAC, 4 seeds).

MethodHumanoid StandHumanoid Walk
ICM + SAC147.6 ± 279.42.5 ± 0.4
RIDE + SAC218.5 ± 422.22.3 ± 0.3
E3B + SAC613.0 ± 121.4399.3 ± 264.8
C-TeC + SAC6.4 ± 0.31.6 ± 0.1
HTC² + SAC677.4 ± 74.0649.8 ± 129.5

HTC² achieves the highest mean returns on both Humanoid tasks. Compared with the strongest baseline, E3B:

  • Humanoid Stand: HTC² improves from 613.0 to 677.4, a 10.5% gain.
  • Humanoid Walk: HTC² improves from 399.3 to 649.8, a 62.7% gain.

Moreover, HTC² exhibits lower standard deviations than E3B on both tasks: 74.0 vs. 121.4 on Humanoid Stand and 129.5 vs. 264.8 on Humanoid Walk. These results provide additional evidence that multi-horizon temporal contrastive exploration with Hamiltonian physical guidance can extend to more challenging high-dimensional continuous-control settings, achieving stronger mean control performance than the evaluated baselines on both Humanoid tasks.

(2). Training visualizations. Beyond final returns, we will include complete SAC learning curves for humanoid_stand and humanoid_walk, showing mean Episode Return and standard deviation across four seeds to further illustrate training dynamics.These additional learning curves and experimental analyses will be included in the appendix of the revised manuscript PDF.

Reviewer_TCz2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Hamiltonian Temporal Contrastive Curiosity (HTC²), a method designed to enhance intrinsic exploration in reinforcement learning for physical control tasks. Existing curiosity-driven methods encourage exploration through prediction error, novelty, episodic uncertainty, or representation changes, but these approaches may fail to capture informative future interactions over extended temporal horizons. To address this limitation, HTC² combines temporal contrastive learning with a learned Hamiltonian latent world model. Specifically, the method employs a horizon-conditioned temporal contrastive critic to capture relationships between state-action pairs and their multi-step future states, using positive-pair representation distance as the primary curiosity signal and bounded multi-step Hamiltonian variation as a modulation factor. The authors evaluate HTC² on six DeepMind Control Suite tasks using both SAC and PPO, comparing it against representative exploration methods, including ICM, RIDE, E3B, and Plan2Explore. Additionally, the paper evaluates robustness under parametric dynamics shifts and conducts ablation studies to investigate the contributions of individual components. The experimental results suggest that HTC² generally outperforms the evaluated baselines in terms of control performance.

Critical strengths
  1. A reasonable and effective exploration method. The paper does not simply rely on state novelty or one-step prediction errors to encourage exploration. Instead, it introduces multi-horizon temporal contrastive learning to measure the representation distance between current state-action information and future states, allowing the intrinsic reward to reflect relationships over multiple time scales. The method also incorporates a Hamiltonian-based latent dynamics model and uses Hamiltonian variation to modulate the intrinsic reward. This combination provides a reasonable way to incorporate temporal information and physical dynamics into exploration, and the experimental results suggest that the proposed design can improve exploration and control performance.
  2. A relatively comprehensive experimental evaluation. The authors evaluate HTC² on six DeepMind Control Suite tasks and integrate it with both SAC and PPO, showing that the method can be applied to different reinforcement learning algorithms rather than relying on a specific policy optimization framework. The paper also includes comparisons with several existing exploration methods, state coverage analysis, robustness evaluation under changes in physical parameters, and ablation studies. In particular, the ablation results provide useful evidence for the contributions of individual components. Overall, these experiments offer relatively broad empirical support for the effectiveness of the proposed approach.
Critical weaknesses
  1. The relationship between temporal contrastive distance and exploration novelty needs more explanation.The paper uses positive-pair representation distance as the main intrinsic reward signal. However, a larger distance does not always mean that a future state is more novel, as the distance may also be affected by transition uncertainty or the learned representations. In addition, the temporal contrastive critic takes the current state, action, and prediction horizon as inputs, but does not explicitly consider intermediate actions. This may affect how the model handles different possible future states. The paper also does not directly analyze how the contrastive distance relates to exploration behavior. It would be useful to examine how this distance changes with state visitation frequency, training progress, or different prediction horizons.
  2. The contribution of the Hamiltonian structure is not fully isolated in the experiments.The paper includes ablation studies that remove the physics-consistency loss and Hamiltonian modulation. However, these experiments do not directly show how much the Hamiltonian world model contributes to the performance gains. A comparison with a standard latent dynamics model using the same temporal contrastive objective would help clarify whether the improvements come from the Hamiltonian structure or mainly from temporal contrastive learning. This would provide stronger evidence for the benefit of using Hamiltonian dynamics in the proposed method.
Questions for authors
  1. Could the authors further explain why the temporal contrastive distance in Eq. (11) is considered a suitable measure of exploration novelty? In particular, how can the method distinguish unfamiliar future transitions from uncertainty caused by stochastic dynamics or imperfect representation learning?
  2. The temporal contrastive critic uses the current state, action, and prediction horizon, but does not explicitly consider the intermediate action sequence. Could the authors clarify how different action sequences leading to different future states may affect the contrastive distance and the resulting intrinsic reward?
  3. Could the authors provide further analysis of the relationship between temporal contrastive distance and actual exploration behavior? For example, it would be useful to examine how the distance changes with state visitation frequency, training progress, or different prediction horizons.
  4. Could the authors clarify the specific benefits of introducing Hamiltonian structure into the world model? In addition to the existing ablation studies, would it be possible to compare HTC² with a standard latent dynamics model using the same temporal contrastive objective and a comparable model capacity?
  5. Since the temporal contrastive reward depends on a future state at time t+kt+k, could the authors clarify how this reward is computed and assigned to transitions during SAC and PPO training? In particular, how are trajectory boundaries and transitions without sufficient future observations handled?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TCz2 (1/3)

We sincerely thank the reviewer for the careful evaluation and constructive feedback, which help us further clarify our methodology and strengthen the experimental analysis. We address each question below.

Q1: Why is the temporal contrastive distance in Eq.11 suitable for measuring exploration novelty, and how are stochastic dynamics and representation quality addressed?

A1: We thank the reviewer. HTC² uses multi-horizon temporal contrastive learning to measure how well an actually reached future matches the current state–action–horizon condition, while incorporating Hamiltonian physical structure to guide long-horizon future-state exploration.

(1). Theoretical motivation. Under the ideal InfoNCE setting, where positives follow the conditional future distribution and negatives follow the marginal distribution, the optimal critic satisfies

C∗(x,zf)/τ=log⁡[p(zf∣x)/p(zf)]+b(x)C^*(x,z_f)/\tau=\log[p(z_f\mid x)/p(z_f)]+b(x),

where x=(zt,at,k)x=(z_t,a_t,k), zf=zt+kz_f=z_{t+k}, and b(x)b(x) is independent of the candidate future. HTC² uses negative Euclidean distance as its contrastive score, Cθ(x,zf)=−Dθ(t,k)C_\theta(x,z_f)=-D_\theta(t,k) (Eqs.9–11).

A larger positive-pair distance therefore indicates weaker temporal-conditional matching of the reached future, providing a signal for future-state relationships that remain insufficiently learned. Unlike visitation counts or single-step prediction errors, the horizon-conditioned critic captures future-state relationships across multiple temporal scales.

(2). Stochastic dynamics, representation learning, and empirical evidence. HTC² incorporates three complementary mechanisms:

  • Temporal Contrastive Learning: InfoNCE uses actually reached, same-episode posterior future states as positives, learning temporally discriminative representations without directly relying on accumulated multi-step imagination errors.
  • Physics-Structured Representation: Controlled Hamiltonian dynamics, inverse-dynamics supervision, and physical consistency encourage action-relevant, physically structured latent representations for temporal contrastive learning.
  • Empirical Validation: Our new walker_walk diagnostics (Q3) show that temporal contrastive distance decreases by 10.5%, 10.4%, 16.2%, and 24.5% for k=1,2,4,8k=1,2,4,8 as physical-state visitation increases. Across training from 50K to 3M steps, distances decrease by 80.0%, 82.3%, and 81.5% for k=1,2,4k=1,2,4, respectively.

These findings show that temporal contrastive distance is systematically related to physical-state familiarity and progressively decreases as future-state relationships are learned. Together with the physics-structured representation, they support Eq.11 as an adaptive multi-horizon exploration signal rather than a measure of arbitrary local state changes.


Q2: Why are intermediate actions not explicitly encoded, and how do different action sequences affect the distance and intrinsic reward?

A2: HTC² aims to learn future-state relationships induced by the current state–action pair and subsequent policy execution at different horizons, rather than predicting deterministic trajectories conditioned on a complete future action sequence. Intermediate actions are therefore modeled implicitly through the policy-induced future-state distribution.

(1). Implicit modeling of intermediate actions.

For policy π\pi, the future-state distribution at horizon kk can be expressed as

pπ(zt+k∣zt,at,k)=∫pπ(zt+k,at+1:t+k−1∣zt,at,k) dat+1:t+k−1p_\pi(z_{t+k}\mid z_t,a_t,k)=\int p_\pi(z_{t+k},a_{t+1:t+k-1}\mid z_t,a_t,k)\,da_{t+1:t+k-1}.

Thus, intermediate actions are marginalized through the future-state distribution jointly induced by policy execution and environment dynamics.

During training, HTC² constructs (zt,at,k,zt+k)(z_t,a_t,k,z_{t+k}) positive pairs from actual same-episode trajectories. Different intermediate action sequences may lead to different reached futures zt+kz_{t+k}, thereby changing the temporal contrastive distance in Eq.11 and the multi-step Hamiltonian variation in Eq.12:

ΔHt,k=∣Hϕ(zt+k)−Hϕ(zt)∣\Delta H_{t,k}=|H_\phi(z_{t+k})-H_\phi(z_t)|.

These quantities jointly determine the intrinsic exploration reward assigned to the anchor transition through Eqs.13–14.

(2). Design rationale. Horizon embedding eke_k explicitly distinguishes different temporal scales, allowing HTC² to learn multi-step outcomes under subsequent policy execution without encoding variable-length intermediate action sequences. Hamiltonian phase-space dynamics and multi-step energy variation further provide physics-structured guidance. Importantly, HTC² focuses on the future states reached by different action sequences rather than intermediate-path diversity itself.

In summary, HTC² implicitly captures the effects of intermediate actions through the policy-induced future distribution, while horizon conditioning and Hamiltonian structure enable multi-horizon future-state exploration without explicitly encoding complete action sequences.

AuthorsOpenReview ↗

Response to Reviewer TCz2 (2/3)

Q3: How does temporal contrastive distance relate to state visitation frequency, training progress, and prediction horizon?

A3: We thank the reviewer. We add two diagnostics on walker_walk to examine how the distance in Eq.11 varies with physical-state visitation, training progress, and prediction horizon. Distances are computed from posterior latent states of real trajectories and reported as mean ± std over 3 seeds.

(1). State visitation frequency and prediction horizon.

We discretize torso height and forward velocity into a 2D physical-state grid, group target states by their previous visits in diagnostic rollouts, and measure positive-pair distances for k=1,2,4,8k=1,2,4,8.

Table R7. Temporal contrastive distance by state visitation frequency (mean ± std, 3 seeds).

Previous visitsk=1k=2k=4k=8
0 (first visit)0.2692 ± 0.01710.2763 ± 0.01440.3269 ± 0.02890.4651 ± 0.0611
1–40.2659 ± 0.01900.2714 ± 0.01320.3193 ± 0.02410.4493 ± 0.0525
5–190.2596 ± 0.01860.2648 ± 0.01130.3102 ± 0.02220.4265 ± 0.0502
20–990.2516 ± 0.02220.2555 ± 0.01420.2924 ± 0.02170.3983 ± 0.0515
100–4990.2495 ± 0.02680.2521 ± 0.02120.2821 ± 0.03020.3776 ± 0.0632
≥5000.2408 ± 0.04620.2476 ± 0.04000.2739 ± 0.05720.3513 ± 0.0993

Mean distance decreases consistently across visitation bins at all four horizons. From first visits to ≥500 previous visits, decreases are 10.5%, 10.4%, 16.2%, and 24.5% for k=1,2,4,8k=1,2,4,8, respectively. Thus, less-visited physical-state regions tend to exhibit larger temporal mismatch.

Longer horizons also yield larger distances: for first visits, distance increases from 0.2692 (k=1k=1) to 0.4651 (k=8k=8), consistent with the greater difficulty of matching more distant futures.

(2). Training progress.

Using HTC² with maximum horizon K=4K=4, we save checkpoints at 50K, 250K, 1M, and 3M interactions. For each checkpoint, we re-infer posterior latents on the same fixed diagnostic trajectories and measure distances for k=1,2,4k=1,2,4.

Table R8. Temporal contrastive distance over training (mean ± std, 3 seeds).

Training checkpointk=1k=2k=4
50K1.115 ± 0.1641.270 ± 0.0251.328 ± 0.125
250K1.145 ± 0.1301.130 ± 0.1301.170 ± 0.111
1M0.316 ± 0.0270.322 ± 0.0160.340 ± 0.009
3M0.223 ± 0.0240.225 ± 0.0250.246 ± 0.037

From 50K to 3M, mean distances decrease by 80.0%, 82.3%, and 81.5% for k=1,2,4k=1,2,4. This pattern is consistent with the temporal critic progressively learning reached future-state relationships, reducing positive-pair mismatch on fixed trajectories. Even at k=4k=4, distance decreases from 1.328 to 0.246.

Overall, our results show that the temporal contrastive distance in Eq.11 is closely related to physical-state visitation frequency, training progress, and prediction horizon. Larger distances generally correspond to less-visited, insufficiently learned, or longer-horizon future-state relationships, while distances decrease as states are revisited and temporal representations improve. These findings clarify the underlying meaning of temporal contrastive distance: it measures how well the learned temporal-conditional representation captures actually reached future-state relationships, rather than simply reflecting state differences or visitation counts.

AuthorsOpenReview ↗

Response to Reviewer TCz2 (3/3)

Q4: What are the benefits of Hamiltonian structure, and how does HTC² compare with standard latent dynamics?

A4: We thank the reviewer. We add w/o H_phi, replacing the Hamiltonian World Model with a standard RSSM [1,2] and removing its energy-dependent reward gate, while retaining the same multi-horizon contrastive objective, horizons, SAC protocol, and interaction budget. HTC² instead organizes latents into phase space (q,p)(q,p) and uses controlled Hamiltonian dynamics, leapfrog integration, and physical consistency to introduce action-conditioned physical inductive biases.

We also examine w/o G_H (Hamiltonian dynamics retained; reward gate removed) and w/o L_phys (physical consistency removed), testing reward modulation and dynamics regularization separately.

Table R9. Final return (mean ± std, four seeds).

VariantWalker WalkHopper HopCheetah Run
HTC² full814.342 ± 162.857250.865 ± 9.598659.810 ± 33.525
w/o L_phys693.166 ± 203.967245.416 ± 14.651566.209 ± 159.871
w/o G_H718.941 ± 16.122235.203 ± 1.896609.108 ± 16.185
one-step ΔH761.051 ± 195.3241.843 ± 1.183510.093 ± 128.266
w/o RND785.069 ± 93.809227.774 ± 0.719578.816 ± 9.910
w/o repr. norm757.758 ± 114.117144.466 ± 85.169594.697 ± 16.876
w/o H_phi764.373 ± 157.24880.701 ± 110.434614.235 ± 33.350

HTC² full achieves the highest mean returns on all three tasks. Relative to w/o H_phi, gains are 6.5%, 210.9%, and 7.4% on Walker, Hopper, and Cheetah. Removing G_H or L_phys also reduces mean returns on all three tasks. These results support the complementary benefits of multi-horizon contrastive exploration, Hamiltonian dynamics, physical consistency, and bounded reward modulation.


Q5: How is future-dependent intrinsic reward computed and assigned in SAC/PPO, and how are trajectory boundaries handled?

A5: HTC² computes curiosity from an actually reached, same-episode posterior future state and assigns it to the anchor transition at time t, rather than to time t+k.

(1). Reward calculation. For each valid anchor, we sample horizon k and compute temporal distance, Hamiltonian gate, and HTC² reward (Eqs.11–14). After normalization and RND warm-start mixing, the training reward (Eq.17) is

r~t=rtext+β[(1−wn)rˉtRND+wnrˉtHTC].\tilde r_t = r_t^{\mathrm{ext}} + \beta[(1-w_n)\bar r_t^{\mathrm{RND}} + w_n\bar r_t^{\mathrm{HTC}}].

(2). SAC/PPO pipelines. SAC first stores environment transitions in replay; contiguous segments are sampled to infer posterior states and compute anchor rewards before updating Q-functions and policy. PPO collects a rollout, computes anchor rewards before policy optimization, then uses the augmented rewards for GAE, return estimation, and PPO updates. Both follow the same reward-assignment rule (Appendix B.6, Algorithms 1–2).

(3). Boundaries and short futures. Anchor–future pairs remain within the same episode and available contiguous segment; none crosses termination, truncation, or rollout/sequence boundaries. We sample only horizons with an available future state, renormalizing probabilities over valid horizons. Near boundaries, shorter horizons are used. If no valid future exists, HTC² temporal reward is omitted, but extrinsic reward, any available RND reward, and the transition itself are retained.

Thus, HTC² retrospectively attributes exploration feedback from observed futures to the initiating action, without assigning rewards from imagined futures.

References

[1] Hafner et al. Mastering Atari with Discrete World Models. ICLR, 2021.

[2] Hafner et al. Mastering Diverse Control Tasks through World Models. Nature, 2025.

Reviewer_joaNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hamiltonian Temporal Contrastive Curiosity (HTC²) for exploration in physical-control reinforcement learning. A recurrent Hamiltonian world model produces latent phase states, while a horizon-conditioned contrastive critic learns relationships between current state–action pairs and actually reached, same-episode futures. Positive-pair representation distance supplies the main intrinsic reward, modulated by bounded Hamiltonian variation and combined with an action-smoothness penalty and an RND warm-start schedule. Experiments evaluate six DMC control tasks with SAC and PPO, alongside separate coverage, dynamics-shift, ablation, and latent-structure studies, and HTC² achieves the highest mean final return.

Critical strengths
  1. HTC² combines multi-step temporal contrastive curiosity with bounded Hamiltonian modulation, giving temporal novelty and physical structure clearly defined roles.

  2. Experiments across SAC and PPO, supplemented by coverage, robustness, and ablation studies, evaluate the method from complementary perspectives.

  3. Detailed loss formulations and phase-space diagnostics in the Appendix help readers understand the model’s training objectives and learned dynamics.

Critical weaknesses
  1. The comparisons rely primarily on baselines before 2022 and would benefit from broader coverage of recent exploration methods, such as BYOL-Explore [1], DEIR [2], and ETD [3] (Table 4). Comparisons with closely related C-TeC [4], alongside RND-only and plain SAC/PPO controls, would further clarify the incremental benefits of HTC².

  2. The experimental setup would benefit from clearer reporting of the total environment interaction budget and the evaluation protocol used to obtain the reported returns. Clarifying how the baselines, particularly Plan2Explore, are adapted to the shared training framework would also help readers assess reproducibility and the fairness of the comparisons.

  3. The experiments could more clearly distinguish exploration-specific gains from representation learning and reward-shaping effects, particularly because coverage is evaluated on different tasks from the main control benchmark.

  4. Additional component ablations and clarification of hyperparameter selection would strengthen the justification for the world-model losses and the contribution of Hamiltonian structure.

[1] Guo et al. BYOL-Explore: Exploration by Bootstrapped Prediction. NeurIPS, 2022. [2] Wan et al. DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards. IJCAI, 2023. [3] Jiang et al. Episodic Novelty Through Temporal Distance. ICLR, 2025. [4] Mohamed et al. Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards. ICLR, 2026.

Questions for authors
  1. Could the authors clarify which components in Section 3.1 are inherited from prior work and which are introduced in this paper, and explain the rationale for including each term in the world-model loss?

  2. Could the authors clarify what supplementary material the project website provides beyond the manuscript and whether the code will be publicly released? https://iclr27-htc2.github.io/

6 comments
AuthorsOpenReview ↗

Response to Reviewer joaN (1/4)

We thank the reviewer for the constructive comments on comparisons, reproducibility, and Hamiltonian exploration. We respond to Q1 below.

Q1: Comparisons with recent exploration methods and RND + SAC/PPO baselines.

A1: HTC² combines multi-horizon temporal contrastive exploration with Hamiltonian physical inductive biases. Controlled Hamiltonian dynamics learn a structured latent phase space, while bounded multi-step Hamiltonian variation modulates temporal contrastive curiosity. We add C-TeC (ICLR 2026) and RND-only comparisons under SAC/PPO.

(1). Recent exploration methods. BYOL-Explore [1] uses latent prediction errors; DEIR [2] uses discriminative episodic rewards; and ETD [3] uses learned temporal distance. A complete public official training implementation of BYOL-Explore has not been confirmed; released DEIR/ETD implementations primarily support PPO and do not directly provide SAC configurations. Their original settings also differ from our DMC protocol, requiring additional adaptation. We therefore implement C-TeC [4] under our shared SAC/PPO framework. Unlike C-TeC, HTC² further incorporates Hamiltonian phase-space dynamics, physical consistency, and bounded energy-based reward modulation.

Table R1. SAC control returns (mean ± std).

MethodWalker WalkWalker RunWalker Stand
ICM677.8 ± 230.2356.7 ± 52.9936.9 ± 31.7
RIDE717.9 ± 67.5364.9 ± 117.7967.1 ± 3.3
E3B679.2 ± 199.4401.6 ± 92.1961.3 ± 8.1
C-TeC732.3 ± 87.7362.4 ± 113.6971.0 ± 5.2
HTC²827.0 ± 115.2415.8 ± 111.4966.5 ± 3.8

Table R2. PPO control returns (mean ± std).

MethodWalker WalkWalker RunWalker Stand
ICM601.9 ± 179.6269.1 ± 47.1921.0 ± 25.4
RIDE641.1 ± 153.5254.2 ± 23.1877.4 ± 82.0
E3B698.5 ± 129.7277.5 ± 16.4865.7 ± 100.1
Plan2Explore647.8 ± 47.3239.2 ± 38.4929.2 ± 6.2
C-TeC472.1 ± 127.064.7 ± 11.1648.5 ± 35.6
HTC²733.6 ± 160.7307.9 ± 22.8930.6 ± 20.1

HTC² achieves higher mean returns than C-TeC in 5/6 settings: under SAC, +12.9% (Walk), +14.7% (Run), and −0.5% (Stand); under PPO, +55.4%, +375.9%, and +43.5%, respectively.

(2).Comparison with RND + SAC/PPO. To assess gains beyond RND warm-start, we evaluate RND-only and HTC² using identical interaction budgets and return-reporting protocols.

Table R3. RND-only vs. HTC² under SAC (mean ± std).

MethodWalker WalkHopper StandCheetah Run
RND821.9 ± 88.3871.4 ± 31.1573.2 ± 118.7
HTC²827.0 ± 115.2883.8 ± 12.5656.5 ± 45.0

Table R4. RND-only vs. HTC² under PPO (mean ± std).

MethodWalker WalkHopper StandCheetah Run
RND705.3 ± 94.0650.0 ± 83.5447.5 ± 8.1
HTC²733.6 ± 160.7670.2 ± 25.1458.0 ± 27.4

HTC² exceeds RND-only in all six settings: +0.6%, +1.4%, +14.5% under SAC and +4.0%, +3.1%, +2.3% under PPO (Walk/Hopper Stand/Cheetah Run). Together with the existing w/o RND warm-start ablation, these results support the additional benefits of HTC²'s multi-horizon temporal contrastive exploration.

References

[1] Guo et al. BYOL-Explore: Exploration by Bootstrapped Prediction. NeurIPS, 2022.

[2] Wan et al. DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards. IJCAI, 2023.

[3] Jiang et al. Episodic Novelty Through Temporal Distance. ICLR, 2025.

[4] Mohamed et al. Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards. ICLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer joaN (2/4)

Q2: Clarification of the environment interaction budget, evaluation protocol, and the fairness and reproducibility of Plan2Explore.

A2: We thank the reviewer for this valuable suggestion. In the revised Appendix C.1 (Detailed DMC Control Results), we will clarify the environment interaction budget, Episode Return reporting protocol, and shared training settings, and provide additional Plan2Explore hyperparameters and adaptation details to improve experimental transparency, fairness, and reproducibility.

(1). Unified environment interaction budget and return reporting protocol.

All methods follow the same training and reporting settings on each DMC task:

  • Environment Interaction Budget: Each method and random seed uses 5×1065\times10^6 environment interactions, counted as cumulative agent decision steps. Action repeat is 2, corresponding to approximately 10710^7 underlying DMC control steps. HTC²'s RND warm-start and reward annealing are included in this budget without additional interactions.
  • Episode Length: Each episode contains at most 500 agent decision steps, corresponding to approximately 1,000 underlying control steps.
  • Return Definition: Episode Return is the undiscounted sum of extrinsic environment rewards within an episode, including rewards accumulated during action repetition and excluding intrinsic rewards. Thus, all methods use the same task-performance metric.
  • Reporting Protocol: Each learning-curve point averages returns over the 10 most recently completed training episodes, recorded approximately every 3,072 cumulative environment interactions. These curves reflect online training performance rather than returns from a separate evaluation phase requiring additional interactions.
  • Random Seeds: Each method uses 5 random seeds, with final results reported as mean ± standard deviation.

These settings ensure consistent interaction budgets, return definitions, and reporting protocols across exploration methods.

(2). Adaptation and fair comparison of Plan2Explore.

Both Plan2Explore and HTC² construct exploration signals using recurrent latent world models, but differ in dynamics modeling and intrinsic reward design. Plan2Explore employs a standard Recurrent State-Space Model (RSSM) [1,2] to learn latent dynamics and uses prediction-model ensemble disagreement to estimate future-state uncertainty. In contrast, HTC² incorporates Hamiltonian dynamics and physical consistency constraints, constructing intrinsic rewards from multi-horizon temporal contrastive distance and bounded Hamiltonian variation.

We adapt Plan2Explore's exploration module to our unified PPO training framework, ensuring consistency in:

  • DMC environments, observation spaces, and action spaces, without additional privileged environment information;
  • PPO optimization settings, including learning rate, rollout length, batch size, optimization epochs, and other policy update parameters;
  • Environment interaction budget, episode length, and number of random seeds;
  • Extrinsic reward definition and return reporting protocol, with intrinsic rewards used only as auxiliary exploration signals during training.

Thus, the PPO optimizer and environment interaction protocol remain identical, while the main differences lie in the world-model dynamics structure and intrinsic reward construction. We will also provide Plan2Explore's world-model architecture, ensemble configuration, optimization parameters, intrinsic reward normalization, and PPO adaptation details in Appendix C.1 to facilitate reproducibility.

Table R5. Comparison with Plan2Explore under the shared PPO training framework (mean ± std, 5 seeds).

MethodWalker WalkHopper StandCheetah Run
Plan2Explore + PPO647.8 ± 47.3620.7 ± 51.8435.2 ± 37.6
HTC² + PPO733.6 ± 160.7670.2 ± 25.1458.0 ± 27.4
Absolute Improvement+85.8+49.5+22.8
Relative Improvement+13.2%+8.0%+5.2%

Under the unified PPO framework, HTC² achieves higher mean returns than Plan2Explore on all three tasks, with relative improvements of 13.2%, 8.0%, and 5.2%, respectively. These results demonstrate that HTC², combining multi-horizon temporal contrastive objectives with Hamiltonian physical structure, achieves higher mean control returns on the evaluated tasks than the ensemble-disagreement-based exploration mechanism.

References

[1] Hafner et al. Mastering Atari with Discrete World Models. ICLR, 2021.

[2] Hafner et al. Mastering Diverse Control Tasks through World Models. Nature, 2025.

AuthorsOpenReview ↗

Response to Reviewer joaN (3/4)

Q3: Distinguishing exploration gains from representation learning/reward shaping and explaining the coverage tasks.

A3: We thank the reviewer. We add a standard RSSM ablation alongside existing physical-consistency and Hamiltonian-gate ablations to examine physics-structured dynamics and temporal curiosity. We also clarify the selection of additional DMC coverage tasks.

(1). Representation learning vs. physics-structured exploration.

We introduce w/o HϕH_\phi, replacing the Hamiltonian World Model with a standard RSSM [1,2] while retaining multi-horizon temporal contrastive learning and removing Hamiltonian energy modulation. All variants share the SAC protocol and interaction budget. HTC² adds controlled Hamiltonian dynamics and physical consistency to structure the latent phase space (q,p)(q,p).

We also consider w/o GHG_H (retains the Hamiltonian model but removes its reward gate) and w/o Lphys\mathcal{L}_{phys} (removes physical consistency). These ablations examine dynamics structure, physical consistency, and reward modulation.

Table R6. HTC² ablations: final return (mean ± std, four seeds).

VariantWalkerHopperCheetah
w/o Lphys\mathcal{L}_{phys}693.166 ± 203.967245.416 ± 14.651566.209 ± 159.871
w/o GHG_H718.941 ± 16.122235.203 ± 1.896609.108 ± 16.185
one-step ΔH\Delta H761.051 ± 195.3241.843 ± 1.183510.093 ± 128.266
w/o RND785.069 ± 93.809227.774 ± 0.719578.816 ± 9.910
w/o repr. norm757.758 ± 114.117144.466 ± 85.169594.697 ± 16.876
w/o HϕH_\phi764.373 ± 157.24880.701 ± 110.434614.235 ± 33.350
HTC² full814.342 ± 162.857250.865 ± 9.598659.810 ± 33.525

HTC² full achieves the highest mean returns on all three tasks. Compared with w/o HϕH_\phi, returns increase from 764.373 to 814.342 (walker_walk), 80.701 to 250.865 (hopper_hop), and 614.235 to 659.810 (cheetah_run): +6.5%, +210.9%, and +7.4%, respectively. Removing GHG_H or Lphys\mathcal{L}_{phys} also lowers mean returns across all three tasks. Together with the one-step ΔH\Delta H ablation, these results support the benefits of combining multi-horizon temporal exploration with Hamiltonian structure and reward modulation beyond generic latent representation learning.

(2). Why evaluate State Coverage on additional DMC tasks?

Walker, Hopper, and Cheetah have high-dimensional physical states involving body orientation, joint positions/velocities, and contacts. Grid-based coverage scales as BdB^d for dd dimensions and BB bins per dimension, making direct counting impractical and sensitive to discretization [3,4]. Conversely, low-dimensional projections may omit gait and joint coordination. In DMC Walker, rewards depend on torso height, uprightness, and forward velocity [5]; coverage over torso height × forward velocity may therefore correlate with task reward rather than independently reflect physical-state diversity.

We therefore supplement the six main control tasks with three tasks having interpretable low-dimensional physical variables:

  • Cartpole Swingup: cart position and pole angle (xcart,θpole)(x_{\mathrm{cart}},\theta_{\mathrm{pole}}), capturing swing-up configurations.
  • Ball in Cup Catch: relative ball–cup position (xball−xcup,zball−zcup)(x_{\mathrm{ball}}-x_{\mathrm{cup}},z_{\mathrm{ball}}-z_{\mathrm{cup}}), capturing catching trajectories.
  • Finger Spin: object angle and angular velocity (θobj,θ˙obj)(\theta_{\mathrm{obj}},\dot{\theta}_{\mathrm{obj}}), capturing rotational states.

Fixed two-dimensional grids on these variables provide interpretable coverage diagnostics without compressing complex multi-joint locomotion into reward-related coordinates.

Figure 3 and Appendix C.2 report both State Coverage and Episode Return. HTC² achieves the highest mean final coverage and return on cartpole_swingup and finger_spin, and competitive results on ball_in_cup_catch. These experiments complement the main benchmarks. Alongside the HϕH_\phi, GHG_H, and Lphys\mathcal{L}_{phys} ablations, they inform our analysis of world-model structure, intrinsic rewards, state visitation, and control performance.

References

[1] Hafner et al. Mastering Atari with Discrete World Models. ICLR, 2021.

[2] Hafner et al. Mastering Diverse Control Tasks through World Models. Nature, 2025.

[3] Tang et al. #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning. NeurIPS, 2017.

[4] Mazzaglia et al. Curiosity-Driven Exploration via Latent Bayesian Surprise. AAAI, 2022.

[5] Tunyasuvunakool et al. dm_control: Software and Tasks for Continuous Control. Software Impacts, 2020.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.