acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Contrastive Control: Learning Subgoals via Future-State Density Estimation

Abstract

Hierarchical reinforcement learning (HRL) provides a principled framework for solving long-horizon decision-making tasks by jointly learning a high-level policy that predicts subgoals and a low-level policy that executes primitive actions. However, HRL suffers from training instability because the high-level transition dynamics and rewards change as the low-level policy improves, leading to unstable higher-level temporal-difference learning. We investigate an alternative formulation of hierarchical goal-conditioned RL based on future-state density estimation via contrastive learning, where the high-level policy predicts subgoals that maximize the induced future-state density assigned to the final goal, instead of relying on higher-level temporal-difference bootstrapping. However, directly applying this formulation to hierarchical control is non-trivial, as the high-level policy may predict infeasible subgoals for the current low-level policy, thus hindering final-goal reachability. To address this, we propose Hierarchical Contrastive Control (HCC), a constrained optimization approach that regularizes the future-state density objective with a lower-level optimality constraint that guides the high-level policy toward feasible subgoal predictions for the current low-level policy. We perform experiments on sparse-reward robotic tasks, showing that HCC achieves over 55% success rates, outperforms state-of-the-art baselines and yields feasible subgoals, establishing hierarchical future-state density estimation as an effective learning signal for high-level policy optimization in jointly trained hierarchical control.

Primary area:
reinforcement learning
Keywords:
Hierarchical Reinforcement Learning, Contrastive Learning, Future-State Density Estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ALhTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Contrastive Control (HCC) for jointly trained, goal-conditioned hierarchical reinforcement learning. The high-level policy scores subgoals using a classifier trained on future states, rather than high-level temporal-difference targets. Its objective also includes a term based on the estimated low-level optimality gap, with twin target classifiers used for stabilization. The method is evaluated on Pick and Place, Hollow, and Franka Kitchen, with comparisons and component ablations.

Critical strengths
  1. As the low-level policy changes during joint training, the effective high-level transition and reward distributions change with it. Using future-state classification in place of high-level temporal-difference targets is a well-motivated alternative.
  2. The reported performance gains are substantial on all three tasks. Appendix A.4.1, Table 1 gives final success rates of 0.81, 0.67, and 0.82 for HCC with one demonstration; Figure 2 also shows a clear drop when the low-level value term is removed.
  3. The analysis includes more than final task success. The paper reports subgoal-to-achieved-state distances, low-level Q-values, classifier-stabilization ablations, and sensitivity to several hyperparameters (Section 5 and Appendix A.4).
  4. The authors report using the same single-demonstration condition for the baselines. They also test HCC without this supervision: success rates remain 0.55, 0.41, and 0.51, respectively, whereas HIER scores zero in that comparison (Appendix A.4.1, Table 1).
Critical weaknesses
  1. Appendix A.3 establishes that, for a fixed future-state distribution and an ideal classifier, classifier probability and future-state density rank candidate subgoals in the same order. This does not make expected log classifier probability equivalent to expected log density (or log classifier odds) when optimizing a stochastic high-level policy with entropy regularization. Thus, even with an ideal classifier, Equation (12) is a surrogate for Equation (6), not an equivalent objective. The effect of this substitution is not examined.
  2. The low-level optimality gap in Equation (4) does not itself measure subgoal reachability. A subgoal may be unattainable even for the optimal low-level policy, in which case the gap can be zero despite failure to reach it. The subgoal-distance and Q-value results provide useful empirical evidence for the proposed term, but not a general reachability guarantee. Equation (7) further approximates the optimal value after m = 5 low-level updates, without examining sensitivity to this choice.
  3. The future-state classifier is trained from replay data collected as both policies evolve. Positive examples in Equation (8) and Algorithm 1 therefore reflect earlier behavior policies, whereas the density in Equation (1) is motivated by future states under the current policies. Polyak-averaged target classifiers may smooth learning but do not, by themselves, correct this sampling mismatch. Figure 4 compares against C-Learning's off-policy correction, yet the distribution actually estimated by the replay-based classifier remains insufficiently characterized.
  4. Figure 3 removes the second classifier and target-network stabilization together. Appendix Figure 7 separately studies Polyak averaging, which is useful, but neither comparison isolates the effect of taking the minimum of two classifiers while keeping the target-update scheme fixed.
  5. Appendix Figures 9, 11, and 12 show sizable changes in success rate with the optimality weight, subgoal interval, and target-update rate. The paper reports these sensitivity experiments and common implementation settings, but Appendix A.5 does not consolidate the final, task-specific HCC settings or explain how they were selected. The robustness claim should be confined to the settings actually tested.
Questions for authors
  1. Appendix A.3 establishes a ranking equivalence for an ideal classifier at a fixed future-state distribution. What justifies using log classifier probability instead of log density or log classifier odds in the entropy-regularized actor update? A direct comparison would be informative.
  2. Can the authors relate low-level optimality gaps to achieved-subgoal success or distance, and show how the results change when the number of low-level updates in Equation (7) varies from m = 5?
  3. What future-state distribution does the replay-based classifier estimate during joint policy training? Is replay age controlled or is any off-policy correction applied when the two policies change?
  4. Could the authors compare one and two target classifiers at the same Polyak rate, ideally also separating the minimum operator from simply averaging their scores?
  5. What final values of the optimality weight, discount factor, subgoal interval, Polyak rate, and imitation weight were used for each task, and how were they selected?
Reviewer_Nfa1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Contrastive Control (HCC) to address the instability of high-level policy learning caused by evolving low-level policies. This paper proposes an alternative formulation of hierarchical goal-conditioned RL based on future-state density estimation via contrastive learning instead of conventional temporal-difference (TD) bootstrapping. This kind of learning paradigm works by encouraging the selection of subgoals that increase the likelihood of reaching the final task goal. To address potential subgoal infeasibility, the authors introduce a lower-level optimality constraint that regularizes the high-level policy toward subgoals with smaller low-level optimality gaps. They further stabilize the practical future-state density estimator using twin Polyak-averaged target classifiers. Experiments on three sparse-reward robotic manipulation tasks demonstrate substantial improvements over several hierarchical and non-hierarchical baselines.

Critical strengths
  1. Important and well-motivated problem. The non-stationarity of high-level transition dynamics induced by evolving low-level policies is a meaningful challenge in jointly trained HRL. Exploring alternatives to conventional high-level TD learning is a worthwhile research direction.

  2. An interesting alternative for high-level policy optimization. The use of contrastive future-state density estimation provides an intuitive learning signal for selecting subgoals that contribute to long-term goal achievement. Combining this objective with information about the current low-level policy is a potentially useful approach.

  3. Promising empirical performance. HCC achieves substantial improvements over the evaluated baselines across three robotic manipulation environments. The paper also provides multiple ablation studies investigating the lower-level optimality term, classifier stabilization, and relevant hyperparameters.

Critical weaknesses
  1. The connection between lower-level optimality and subgoal feasibility is insufficiently justified.

A central contribution of the paper is the lower-level optimality constraint in Equation (4), which is claimed to guide the high-level policy toward feasible subgoals. However, a small optimality gap does not necessarily imply subgoal reachability. The theoretical connection between these concepts requires further clarification.

  1. The proposed approach does not eliminate non-stationarity, and its stability advantage needs stronger justification.

For example, the density estimator is trained using historical trajectories from the replay buffer, which may not accurately reflect the future-state distribution induced by the current low-level policy. Although twin classifiers and Polyak averaging can stabilize estimation, they do not fundamentally eliminate this distribution mismatch. The experimental results are promising, but they do not fully explain why future-state density estimation should be intrinsically more robust to non-stationarity than conventional high-level TD learning.

Questions for authors
  1. The authors argue that future-state density estimation alone may lead to infeasible subgoals. However, if a subgoal is infeasible for the current low-level policy and consequently hinders reaching the final task goal, shouldn't this already be reflected in a lower future-state density assigned to the final goal? It is unclear why the future-state density objective alone cannot account for subgoal infeasibility and why an additional constraint is necessary.

  2. The proposed constraint encourages the current low-level policy to achieve the optimal value for a given subgoal. However, lower-level optimality does not necessarily imply subgoal reachability. Even if a subgoal is unreachable, the current low-level policy may already be optimal, resulting in a zero optimality gap. Could the authors clarify the theoretical connection between the proposed optimality constraint and subgoal feasibility?

  3. Could the authors provide a direct comparison between HCC and HCC-No-V using the Subgoal Distance Metric? This would help establish whether the proposed optimality constraint actually improves subgoal reachability, rather than simply improving overall task performance.

Thank you for addressing these questions. I would reconsider my assessment and potentially raise my score if my concerns are addressed.

Reviewer_Qmq1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the classic training instability in hierarchical reinforcement learning caused by low-level policy changes during training. This work mainly proposes two components for efficient and effective Hierarchical GCRL:

  • It proposes using contrastive reinforcement learning to estimate the final-goal state density ρ(g⋆∣s,g)\rho(g^\star \mid s, g) for the high-level policy, which considers the combined transition of both environment dynamics and the low-level policy.
  • It also proposes a learning-progress regularization to regularize high-level subgoal generation, ensuring the low-level policy can achieve the generated subgoal.
  • Experiments on goal-conditioned manipulation tasks show significant improvements over baselines.
Critical strengths
  • This work has a natural and simple high-level objective. It evaluates the high-level policy’s subgoal based on the final goal's future-state density, which naturally evaluates the combined impact of the current low-level policy and the environment dynamics. Using MC-style CRL also avoids the high-level need for a reward-based TD target. It is a simple way to bring contrastive RL into hierarchical goal-conditioned control, and it fits into existing HGCRL pipelines with little change.


  • This work’s HCC method improves success rates over all tested baselines on three robotic tasks. The ablation study also provided some analysis of the components and provided information about how practically this algorithm can be configured to achieve good performances

Critical weaknesses

Despite this paper having good high-level insights into the problem, it has several weaknesses making it imperfect: 1) the discussion of high-level future state/goal density estimation, 2) still missing ablation studies, making the contribution/effectiveness of each component unclear.

W1. The learned density lags behind the current low-level policy.

  • Like contrastive RL (CRL), HCC estimates the discounted future-state density at the goal. The difference is that the subgoal plays the role of the action, and time is counted in high-level steps.
  • Because the low-level policy executes the subgoal, the true density of the current policy already reflects what the low level can reach.
  • The learned density, however, can lag behind: Its positive samples are future states from replay trajectories collected by past low- and high-level policies. More complex than vanilla CRL methods.
  • So the classifier fits a mixture over past policies, which may not match the current low-level policy. The paper neither discusses nor measures this lag.

W2. Missing ablations on the high-level objective.

  • Figure 4 experiments is not very fair:
    • The experiment is comparing C-learning (MC version + twin-classifier stabilization) with C-learning (MC version)
      • If the aim is to address the effectiveness of twin-classifier stabilization, the experiment should be:
        C-learning (MC version) vs C-learning (MC version + twin-classifier stabilization)
  • MC vs TD for future state/goal density estimation:
    • Correspondingly, this is another question that needs to be answered: how MC and TD version of C-learning can affect performance:
      • So should have another experiment:
        C-learning (MC version + twin-classifier stabilization)
        vs
        C-learning (TD version + twin-classifier stabilization)
  • Missing ablations on the high-level density estimator.
    • The contrastive RL family offers several ways to estimate the future-state density, and this work did not provide analysis or proof for why choosing C-learning only as the high-level  density estimator.
      • Should have the following comparisons:
        • C-learning (MC/TD): use twin-critic stabilization.
        • CRL(MC-NCE/TD- NCE) : use twin-critic stabilization.
        • Contrastive Difference Predictive Coding (MC-infoNCE/TD- infoNCE) : use twin-critic stabilization.
Questions for authors

Beside the points listed in the weakness section, here are additional questions

Q1. Where does the gain come from?

  • From Fig. 1, we can tell most of the gain comes from adding the regularizer (20% to 73%), not from replacing TD with density estimation (2.7% to 20%). Without the missing cell, it is unclear whether η(VL−VmL)\eta(V^L - V^L_m) is a generic improvement for HRL or works specifically with the density objective. Could the authors add HIER with the same regularizer, with η\eta tuned separately?

Q2. Why use V^L - V^L_m rather than V^L regularization?

  • I think this motivation could be elaborated in the paper.

  • In practice, VL−VmLV^L - V^L_m reflects short-term learning progress rather than current reachability. And VLV^L directly tells the low-level reachability. Could the authors compare the VL−VmLV^L - V^L_m regularizer with a variant that uses VLV^L only,  and report both the performance of two different regularizers?

  • Another sensitivity analysis is also needed for mm, which is fixed to 5 in all experiments. Since the mm updates in Alg. 1 are also the actual low-level training steps, VmLV^L_m should ideally be estimated on a copy of the critic, so that the sweep does not change low-level training.

Q3. How well is the high-level future goal/state density estimated?

  • The target density \rho(g^\star \mid s, g) depends on both levels: the low-level policy executes g, and the high-level policy chooses the later subgoals. The classifier estimates a mixture over all past versions of both policies rather than the density under the current \pi^H, \pi^L.
  • Could the authors check how well the learned density matches this ground truth?

Q4. The motivation of the value regularization term is not well justified.

  • The connection between the value-gap regularizer and subgoal feasibility is insufficiently justified.
  • The term (\eta(V^L-V^{L,})) favors subgoals with smaller optimality gaps, but a small gap does not necessarily imply reachability: the current policy may already be optimal for a subgoal that remains unreachable within (k) steps.
  • It would be nice to see the discussion of regularization term \eta (V^{L,}) vs  (\eta(V^L-V^{L*})), why not using  \eta (V^{L,}) only?
  • V^L − V^L_m is itself built from learned critics, so it is not an unbiased measure of current reachability either.
  • Although Figure 2 shows that the regularizer improves task performance, the feasibility analyses in Figures 5–6 omit HCC-No-V and therefore do not isolate its effect on subgoal reachability. A direct feasibility comparison and a baseline using (V^L(s,g)) alone would help justify the proposed design.
2 comments
Reviewer_Qmq1OpenReview ↗

Response to answers

I really appreciate the effort the authors have put into the additional experiments. I think my main concerns have been addressed. To have a proper submission. I think the clarifications and the ablation study in the rebuttal could have been presented better in the main content rather than in the appendix. For example, the design motivation would be better in the main content rather than in Appendix Figures 5 and 6, etc.

Regarding the oracle future density estimation (e.g., w1, Q1, Q3), I think it would be helpful to include tabular toy examples to analyze the algorithm more thoroughly in a controlled setting, rather than relying only on empirical results from high-dimensional continuous tasks.

Sorry for the late reply. From the rebuttal round, the results addressed my main concerns. Thus, I consider raising my score.

AuthorsOpenReview ↗

Author response to the reviewer

We sincerely thank the reviewer for their continued engagement, constructive feedback, and recognition of our additional experimental results. We are glad that our responses have addressed the reviewer's main concerns.

Following the reviewer's suggestion, we have revised the manuscript to incorporate the key theoretical clarifications and design motivations directly into Section 4. We have also expanded the discussion in Section 5 to highlight the additional ablation results, including the effect of lower-level value regularization on task performance and subgoal feasibility. We also appreciate the suggestion regarding controlled tabular experiments. We will include an additional analysis using a tabular setting in the final manuscript.

We appreciate the reviewer's thoughtful feedback, which has helped us improve the clarity and presentation of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.