Hamiltonian Temporal Contrastive Curiosity for Reinforcement Learning Exploration

Efficient exploration is essential for reinforcement learning in physical control, where useful behavior depends on temporally extended dynamics. Existing curiosity methods typically reward prediction error, state novelty, uncertainty, representation change, or instantaneous physical variation. Hamiltonian-based exploration further introduces physical structure by rewarding changes in a learned Hamiltonian across individual transitions, but such signals remain local in time and do not directly reveal whether an action reaches an informative future over longer horizons. This raises a key question: how can exploration identify action-conditioned futures that remain poorly captured over multiple temporal scales while reflecting structured variation in the underlying dynamics? To address this challenge, we propose Hamiltonian Temporal Contrastive Curiosity (HTC${}^2$), which formulates exploration as temporal future discovery in a learned Hamiltonian phase space. HTC${}^2$ trains a horizon-conditioned contrastive critic on same-episode future pairs and uses the positive-pair representation distance to prioritize action-conditioned futures that remain insufficiently organized by the current temporal representation. Multi-step Hamiltonian variation further provides a bounded structural modulation of this temporal curiosity signal, allowing exploration to account for structured changes in the learned phase dynamics without treating energy variation itself as the exploration objective. We evaluate HTC${}^2$ on six DeepMind Control Suite tasks with SAC and PPO against representative exploration methods including ICM, RIDE, E3B, and Plan2Explore, together with a single-step Hamiltonian-change curiosity variant, with additional coverage, ablation, and Hamiltonian diagnostics characterizing its exploration behavior and learned phase-space structure. A quick visual overview of this work is available at https://iclr27-htc2.github.io/.