- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Hierarchical Contrastive Control (HCC) for jointly trained, goal-conditioned hierarchical reinforcement learning. The high-level policy scores subgoals using a classifier trained on future states, rather than high-level temporal-difference targets. Its objective also includes a term based on the estimated low-level optimality gap, with twin target classifiers used for stabilization. The method is evaluated on Pick and Place, Hollow, and Franka Kitchen, with comparisons and component ablations.
Critical strengths
- As the low-level policy changes during joint training, the effective high-level transition and reward distributions change with it. Using future-state classification in place of high-level temporal-difference targets is a well-motivated alternative.
- The reported performance gains are substantial on all three tasks. Appendix A.4.1, Table 1 gives final success rates of 0.81, 0.67, and 0.82 for HCC with one demonstration; Figure 2 also shows a clear drop when the low-level value term is removed.
- The analysis includes more than final task success. The paper reports subgoal-to-achieved-state distances, low-level Q-values, classifier-stabilization ablations, and sensitivity to several hyperparameters (Section 5 and Appendix A.4).
- The authors report using the same single-demonstration condition for the baselines. They also test HCC without this supervision: success rates remain 0.55, 0.41, and 0.51, respectively, whereas HIER scores zero in that comparison (Appendix A.4.1, Table 1).
Critical weaknesses
- Appendix A.3 establishes that, for a fixed future-state distribution and an ideal classifier, classifier probability and future-state density rank candidate subgoals in the same order. This does not make expected log classifier probability equivalent to expected log density (or log classifier odds) when optimizing a stochastic high-level policy with entropy regularization. Thus, even with an ideal classifier, Equation (12) is a surrogate for Equation (6), not an equivalent objective. The effect of this substitution is not examined.
- The low-level optimality gap in Equation (4) does not itself measure subgoal reachability. A subgoal may be unattainable even for the optimal low-level policy, in which case the gap can be zero despite failure to reach it. The subgoal-distance and Q-value results provide useful empirical evidence for the proposed term, but not a general reachability guarantee. Equation (7) further approximates the optimal value after m = 5 low-level updates, without examining sensitivity to this choice.
- The future-state classifier is trained from replay data collected as both policies evolve. Positive examples in Equation (8) and Algorithm 1 therefore reflect earlier behavior policies, whereas the density in Equation (1) is motivated by future states under the current policies. Polyak-averaged target classifiers may smooth learning but do not, by themselves, correct this sampling mismatch. Figure 4 compares against C-Learning's off-policy correction, yet the distribution actually estimated by the replay-based classifier remains insufficiently characterized.
- Figure 3 removes the second classifier and target-network stabilization together. Appendix Figure 7 separately studies Polyak averaging, which is useful, but neither comparison isolates the effect of taking the minimum of two classifiers while keeping the target-update scheme fixed.
- Appendix Figures 9, 11, and 12 show sizable changes in success rate with the optimality weight, subgoal interval, and target-update rate. The paper reports these sensitivity experiments and common implementation settings, but Appendix A.5 does not consolidate the final, task-specific HCC settings or explain how they were selected. The robustness claim should be confined to the settings actually tested.
Questions for authors
- Appendix A.3 establishes a ranking equivalence for an ideal classifier at a fixed future-state distribution. What justifies using log classifier probability instead of log density or log classifier odds in the entropy-regularized actor update? A direct comparison would be informative.
- Can the authors relate low-level optimality gaps to achieved-subgoal success or distance, and show how the results change when the number of low-level updates in Equation (7) varies from m = 5?
- What future-state distribution does the replay-based classifier estimate during joint policy training? Is replay age controlled or is any off-policy correction applied when the two policies change?
- Could the authors compare one and two target classifiers at the same Polyak rate, ideally also separating the minimum operator from simply averaging their scores?
- What final values of the optimality weight, discount factor, subgoal interval, Polyak rate, and imitation weight were used for each task, and how were they selected?
Response to answers
I really appreciate the effort the authors have put into the additional experiments. I think my main concerns have been addressed. To have a proper submission. I think the clarifications and the ablation study in the rebuttal could have been presented better in the main content rather than in the appendix. For example, the design motivation would be better in the main content rather than in Appendix Figures 5 and 6, etc.
Regarding the oracle future density estimation (e.g., w1, Q1, Q3), I think it would be helpful to include tabular toy examples to analyze the algorithm more thoroughly in a controlled setting, rather than relying only on empirical results from high-dimensional continuous tasks.
Sorry for the late reply. From the rebuttal round, the results addressed my main concerns. Thus, I consider raising my score.