- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces an exploration objective that encourages informative temporal changes over the future at different time scales modulated by a change in a learned Hamiltonian, which represents a change in the total energy of the system. The change in energy incorporates the physical structure of the system. This modulated objective addresses the limitation of one-step Hamiltonian curiosity exploration, which focuses only on local changes of the Hamiltonian and does not capture long-term temporal variations. The new objective aims to capture both physical structure via the Hamiltonian and temporal structure via contrastive learning. The paper evaluates the proposed method on the DMC environments, and it shows strong performance. The paper also analyzes the method by conducting an ablation study and shows that both physical structure and temporal structure affect the method's performance.
Critical strengths
- The paper is clear and easy to follow. I understand the paper's claims and contributions.
- The various evaluation metrics in the experiment section help in understanding the method's performance not just with respect to the final return but also with respect to state coverage and robustness to physical changes.
- The method shows robust performance across physical changes in the environment.
Critical weaknesses
- From my perspective, the method's performance is not much stronger than the baselines; many of the baselines perform as well as the proposed method, and some results have high variance.
- The RND warm start is practically justified but undermines the main idea of the paper, and it seems to affect performance significantly, as shown in the ablation study.
- The figures (2, 4, and 5) are a bit hard to read; the font is small. Can you make it larger?
- The clipping in equation 13 seems somewhat arbitrary; I understand that you want to bound the modulation via tanh, but can you justify the clipping term? Especially since you are dividing by a running scale , which in principle should also bound .
Questions for authors
- Can the authors run the method also on the humanoid environment? That helps in understanding the limitations of such a method.
5 comments
Response to Reviewer RP23 (2/3)
Q2: Does RND warm-start dominate the performance gains of HTC²?
A2: We sincerely thank the reviewer for raising this important question. RND warm-start primarily serves as an initialization mechanism for early exploration, whereas the core exploration mechanism of HTC² is multi-horizon temporal contrastive learning with Hamiltonian physical guidance. To distinguish their contributions, we introduce additional RND-only baselines under SAC/PPO and examine the existing w/o RND ablation.
(1). Motivation for RND warm-start. Following the exploration initialization strategy in DreamSAC [1], HTC² incorporates RND during early training to support the initial learning of the Hamiltonian World Model and Temporal Contrastive Critic. Specifically, all experiments use a total training budget of 5M environment interactions. RND initializes exploration during the first 50K interactions, followed by a gradual transition from RND to HTC² temporal curiosity over the next 200K interactions. After 250K interactions, the intrinsic reward is entirely provided by HTC² for the remaining 4.75M interactions (95% of the total training budget), without any RND contribution. Therefore, RND serves only as an early exploration initialization mechanism, while HTC²'s multi-horizon temporal contrastive curiosity drives exploration throughout the vast majority of training, rather than relying on RND as the primary exploration reward.
(2). Additional RND-only comparisons.
We introduce RND + SAC/PPO baselines on walker_walk, hopper_stand, and cheetah_run, using the same interaction budgets and evaluation protocols to directly compare the final control performance of RND-only and HTC².
Table R3. RND-only comparison under SAC (mean ± std).
| Method | Walker Walk | Hopper Stand | Cheetah Run |
|---|---|---|---|
| RND + SAC | 821.9 ± 88.3 | 871.4 ± 31.1 | 573.2 ± 118.7 |
| HTC² + SAC | 827.0 ± 115.2 | 883.8 ± 12.5 | 656.5 ± 45.0 |
Table R4. RND-only comparison under PPO (mean ± std).
| Method | Walker Walk | Hopper Stand | Cheetah Run |
|---|---|---|---|
| RND + PPO | 705.3 ± 94.0 | 650.0 ± 83.5 | 447.5 ± 8.1 |
| HTC² + PPO | 733.6 ± 160.7 | 670.2 ± 25.1 | 458.0 ± 27.4 |
The additional experiments show that HTC² achieves higher mean returns than RND-only in all six SAC/PPO comparisons. In particular, the improvement reaches 14.5% on cheetah_run under SAC, while the remaining tasks also show additional gains of varying magnitudes. These results indicate that the performance improvements of HTC² are not solely attributable to standalone RND exploration rewards.
(3). Evidence from the w/o RND ablation.
Furthermore, when RND warm-start is completely removed, HTC² still achieves mean returns of 785.1, 227.8, and 578.8 on walker_walk, hopper_hop, and cheetah_run, respectively.
These results demonstrate that HTC² can learn effective control policies through its own multi-horizon temporal contrastive exploration mechanism even without RND initialization. RND warm-start improves early exploration conditions but is not a prerequisite for HTC² to acquire effective exploration capabilities.
Overall, RND warm-start and the core HTC² mechanism play distinct roles: the former initializes early exploration, while the latter enables long-horizon future-state discovery through multi-horizon temporal contrastive learning and Hamiltonian physical guidance.
Q3: The fonts and numerical annotations in Figures 2, 4, and 5 are relatively small. Could their readability be improved?
A3: We sincerely thank the reviewer for the careful examination of our figures and this valuable suggestion. Figures 2, 4, and 5 are provided in vector format, preserving the clarity of graphical elements and text when enlarged. To further improve readability at the normal viewing size, we have increased the font sizes of axis labels, legends, subplot titles, and key numerical annotations while preserving all original experimental data and visualizations. These improvements will be reflected in the revised manuscript PDF.
References
[1] Tang et al. DreamSAC: Learning Hamiltonian World Models via Symmetry Exploration. CVPR, 2026.
Response to Reviewer RP23 (3/3)
Q4: Why is clipping needed in Eq.13 when tanh already bounds the modulation? Does the running scale not sufficiently constrain the normalized input?
A4: We sincerely thank the reviewer for this insightful mathematical question. The tanh function already guarantees bounded Hamiltonian modulation, while clipping serves as a conservative safeguard against extreme normalized energy variations rather than imposing an additional reward-magnitude constraint.
(1). Mathematical justification. Let . The Hamiltonian gates with and without clipping are
,
.
Since tanh is monotonically increasing and bounded,
.
The two gates are identical when ; clipping only affects inputs exceeding this threshold.
With our settings and ,
. Therefore, tanh provides the essential boundedness, while clipping introduces a negligible difference in the resulting Hamiltonian reward modulation.
(2). Design rationale. The running scale adapts to the evolving magnitude of Hamiltonian energy variations, but normalization does not impose a strict upper bound on the normalized input, particularly during early training or under unusually large energy changes. We therefore adopt a conservative threshold , restricting extreme normalized inputs only after tanh has nearly saturated. This is an auxiliary numerical safeguard rather than a separate exploration mechanism or a source of performance gains. We will clarify the distinct roles of running-scale normalization, tanh modulation, and clipping, together with the above mathematical justification, in the revised manuscript.
Q5: Can HTC² be evaluated on more challenging Humanoid environments to further assess its applicability?
A5: We sincerely thank the reviewer for this valuable experimental suggestion. We extend our evaluation to two challenging high-dimensional continuous-control tasks, humanoid_stand and humanoid_walk, comparing HTC² against ICM, RIDE, E3B, and C-TeC under SAC to assess its performance in complex multi-joint coordination and balance control.
(1). Additional Humanoid experiments. All methods use the same environment interaction budget and evaluation protocol. We report final Episode Return as mean ± standard deviation over four random seeds. Table R5. Final performance on DMC Humanoid tasks (SAC, 4 seeds).
| Method | Humanoid Stand | Humanoid Walk |
|---|---|---|
| ICM + SAC | 147.6 ± 279.4 | 2.5 ± 0.4 |
| RIDE + SAC | 218.5 ± 422.2 | 2.3 ± 0.3 |
| E3B + SAC | 613.0 ± 121.4 | 399.3 ± 264.8 |
| C-TeC + SAC | 6.4 ± 0.3 | 1.6 ± 0.1 |
| HTC² + SAC | 677.4 ± 74.0 | 649.8 ± 129.5 |
HTC² achieves the highest mean returns on both Humanoid tasks. Compared with the strongest baseline, E3B:
- Humanoid Stand: HTC² improves from 613.0 to 677.4, a 10.5% gain.
- Humanoid Walk: HTC² improves from 399.3 to 649.8, a 62.7% gain.
Moreover, HTC² exhibits lower standard deviations than E3B on both tasks: 74.0 vs. 121.4 on Humanoid Stand and 129.5 vs. 264.8 on Humanoid Walk. These results provide additional evidence that multi-horizon temporal contrastive exploration with Hamiltonian physical guidance can extend to more challenging high-dimensional continuous-control settings, achieving stronger mean control performance than the evaluated baselines on both Humanoid tasks.
(2). Training visualizations.
Beyond final returns, we will include complete SAC learning curves for humanoid_stand and humanoid_walk, showing mean Episode Return and standard deviation across four seeds to further illustrate training dynamics.These additional learning curves and experimental analyses will be included in the appendix of the revised manuscript PDF.
Response to Reviewer RP23 (1/3)
A1: We thank the reviewer for this valuable comment. We further evaluate HTC² from two complementary perspectives: final control performance and sample efficiency. Beyond final Episode Return, we provide additional Learning AUC and Steps-to-Threshold analyses based on the complete learning curves in Appendix Figures 6–7, allowing us to quantify training efficiency and environmental interaction costs.
(1). Final control performance. As reported in Table 4, HTC² achieves higher mean returns across multiple DMC tasks under both SAC and PPO. For example:
HTC² also achieves the highest mean returns on SAC Walker Run, Hopper Hop, Hopper Stand, and Cheetah Run, as well as PPO Walker Walk, Walker Stand, Hopper Stand, and Cheetah Run. These results demonstrate consistent competitiveness across different continuous-control tasks and both SAC/PPO optimizers, rather than improvements concentrated on a single environment.
(2). Sample efficiency: fewer interactions to reach the same target return.
An important benefit of exploration is not only achieving higher final returns, but also reaching a given control performance with fewer environment interactions. Appendix Figure 6 (SAC) and Figure 7 (PPO) present the complete learning curves. HTC² achieves higher Episode Returns throughout substantial portions of training on several tasks, with particularly favorable learning trajectories on PPO Walker Run and Hopper Stand. To quantify these observations, we introduce two additional metrics using the existing five-seed training curves:
Table R1. Sample-efficiency gains over strong baselines (5 seeds).
Positive values indicate fewer interactions or higher Learning AUC for HTC². Each metric uses its strongest corresponding core baseline. Values are point estimates summarized over five seeds. The additional experiments show that HTC² achieves lower estimated interaction costs in all four SAC/PPO comparisons.
Specifically, under SAC, HTC² reduces the interactions required to reach the target return by 11.9% on
hopper_standand 16.8% oncheetah_run. Under PPO, the corresponding reductions are 10.5% and 20.5%, respectively. Furthermore, HTC² improves Learning AUC by 4.9%, 4.9%, 13.6%, and 1.5% across these four comparisons, indicating favorable cumulative control performance throughout training. These findings complement the learning curves in Appendix Figures 6–7, demonstrating that the benefits of HTC² extend beyond final returns to the interaction efficiency required to achieve comparable control performance.Overall, HTC² achieves higher final mean returns across multiple DMC tasks, while our additional sample-efficiency analysis shows lower estimated interaction costs and improved Learning AUC in the evaluated settings.