- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the problem of stability of predictions from hierarchical models in the context of offline goal-conditioned reinforcement learning. They propose HiQFlow that, as stated by the authors, empirically justify the theoretical insight on using undiscounted quasimetric distance representations for subgoals consistency over long-horizon tasks. Such a formulation enables to keep the margin from the further subgoals in the original state space with a conditional normalizing flow for both low- and high-level policies. On the mazes from OGBench they report improved performance over multi-seed evaluation with additional ablations compared to SOTA baselines.
Critical strengths
- the results of the method are above or comparable with the state-of-the-art baselines as in their original form, so as when normalizing flows are integrated into their pipelines. There is also the empirical justification of the success rates on stitch datasets, what the theory is about, where there is a dominating gap in performance between the proposed method and related work, the ablation studies are fulfilled to explain important architectural choices such as flow- or gaussian-based policy and the ones mentioned above
Critical weaknesses
- there is no enough evidence in terms of motivation and model stability that this quasi metric formulation inside the method provides the opportunity for trajectory stitching of the datasets - maybe it is just about the expressivity combined with the normalizing flow pipeline
Questions for authors
- are and processed somehow before , and which beta was used for OTA-NF and HQRL-NF? at the moment it is hard to distinguish how much of Table 2 is the distance or scale weighting
- can you provide also the RLiable metrics on the already existing checkpoints to address the robust performance improvements of your method upon the baselines?
- there is a different random of seeds for eik-hiqrl and hiqflow - can there be different evaluation scenarios with your own method and the related work? can you justify the unified evaluation protocol, including same amount of random seeds, the rule of picking out the checkpoint whose results would be reflected in the tables and re-evaluating all the baselines on a single machine?
9 comments
[2/2] To W1: The quasimetric evaluator’s contribution to trajectory stitching
To empirically distinguish the evaluator's contribution from flow expressiveness, we conduct two complementary groups of ablations.
-
First, we retain HiQFlow's policy architectures and AWR objectives and change the evaluator. HiQFlow-TD (MLP) replaces it with MLP value networks trained by discounted expectile TD learning. HiQFlow-TD (IQE) retains the quasimetric parameterization but uses the same TD objective, testing whether retaining the triangle-inequality structure is sufficient to match HiQFlow's performance. Both TD variants use two value networks and their target networks. For the additional HumanoidMaze results, we evaluate both TD variants at exactly million updates across four seeds and retain the HiQFlow results reported in Table 1 in the manuscript.
Table R-2 reports the results. Averaged equally across the five stitching tasks, HiQFlow achieves success, compared with for HiQFlow-TD (MLP) and for HiQFlow-TD (IQE).
Table R-2. Stitching success rates for evaluator ablations (%, mean std over 8 seeds).
Task HiQFlow (ours) HiQFlow-TD (MLP) HiQFlow-TD (IQE) antmaze-medium-stitch antmaze-large-stitch antmaze-giant-stitch humanoidmaze-medium-stitch humanoidmaze-large-stitch Average 84.6 30.0 29.9 -
As a complementary comparison, we test whether adding the flow policy to other methods is sufficient to match HiQFlow's stitching performance. We retain the value-learning backbones of HIQL, OTA, and HQRL and replace their high-level policies with conditional normalizing flows using HiQFlow's flow implementation. HIQL-NF retains discounted expectile TD learning and HIQL's learned subgoal representation, while OTA-NF and HQRL-NF retain option-aware TD learning and latent-space quasimetric learning, respectively. All variants are evaluated on the same five predefined OGBench goals per environment, with episodes per goal.
As shown in Table R-3, HiQFlow achieves an average success rate of on the five stitching tasks, compared with for HIQL-NF, for OTA-NF, and for HQRL-NF. On antmaze-giant-stitch, HiQFlow achieves , compared with , , and , respectively. These results show that the flow-based variants retain substantial differences in average stitching performance.
Table R-3. Flow-policy comparisons on stitching tasks (%, mean std over 8 seeds).
Task HiQFlow (ours) HIQL-NF OTA-NF HQRL-NF antmaze-medium-stitch antmaze-large-stitch antmaze-giant-stitch humanoidmaze-medium-stitch humanoidmaze-large-stitch Average 84.6 53.0 77.0 36.0
To Q1: Advantage processing and AWR settings in OTA-NF and HQRL-NF
Thank you for highlighting the importance of the AWR settings in interpreting the comparison. All three methods use raw advantages and a common exponential-weight cap. We detail this processing and the task-dependent AWR coefficients below.
-
HiQFlow, OTA-NF, and our HQRL-NF implementation use raw advantages at both policy levels, without clipping, centering, or standardization before exponentiation. Following the HIQL and OTA implementations for OGBench, we cap the exponential weights at , using , where denotes the high-level or low-level policy. We do not normalize the weights across the minibatch.
-
For OTA-NF, we follow the original OTA task-specific hyperparameter settings, including its released tuned configurations. For the 16 tasks in Table 2 in the manuscript, we use on the large-stitch tasks in PointMaze, AntMaze, and HumanoidMaze, on the PointMaze giant-stitch task, and on the remaining 12 tasks.
For HQRL-NF, we construct the implementation from the network definitions and training objectives described in the HQRL paper, since an official implementation was unavailable for our reproduction and the paper does not provide task-specific AWR coefficients. We set its AWR coefficients to those used by HiQFlow in Table 6 in the manuscript: for PointMaze, for AntMaze, and for HumanoidMaze.
[1/2] To Q3: Comparison under a common evaluation protocol
Thank you for highlighting the importance of a consistent evaluation protocol. We clarify the original evaluation differences and retrain and evaluate the baselines using a common protocol with matched random seeds and training budgets, final-checkpoint evaluation, and the same predefined tasks and evaluation hardware.
-
The Eik-HiQRL results in Table 1 in the manuscript were taken from Table 7 of its paper, which reports the best evaluation results during 1 million training steps across ten seeds. Evaluations are conducted every 100,000 steps on five predefined tasks, with 50 episodes per task. For HiQFlow, we use eight random seeds per environment and report final-checkpoint results, as described in the Training and evaluation setup paragraph of Appendix E in the manuscript. Training lasts 1 million gradient steps for PointMaze and AntMaze, and 2 million steps for HumanoidMaze. Each final checkpoint is evaluated on the five predefined OGBench tasks with 50 episodes per task. We average success rates over these five tasks for each seed and report the mean and standard deviation across the eight seeds.
-
To provide a comparison under a common protocol, we retrain the baselines using the same eight random seeds as HiQFlow and the training budgets described above. For each environment, we evaluate the final saved checkpoint on the five predefined OGBench tasks, with episodes per task. All checkpoint evaluations use the same server equipped with NVIDIA RTX 4090 GPUs.
For GCIVL, CRL, QRL, HIQL, OTA, and Eik-HiQRL, we use the released implementations and configurations for the corresponding environments. CGCIVL provides an official implementation but does not include environment-specific configuration files for the method. We use its released code with the hyperparameters reported in Appendix C.3 and Table 2 of the CGCIVL paper, and follow the corresponding OGBench environment settings for hyperparameters not specified there. Since an official implementation of HQRL was unavailable for our reproduction, we implement its training objectives and architecture as described in the paper and follow its reported hyperparameter settings. For hyperparameters not specified in the HQRL paper, we use the corresponding HiQFlow settings.
We sincerely thank Reviewer u7mb for the detailed comments on our algorithm design and experimental scope. Below, we provide detailed responses to these points.
[1/2] To W1: The quasimetric evaluator’s contribution to trajectory stitching
We thank the reviewer for this helpful comment. We would like to clarify that HiQFlow uses a conditional normalizing flow only for the high-level policy, while the low-level policy is a goal-conditioned Gaussian MLP. To explain the quasimetric evaluator’s role in stitching, we present the theoretical motivation and learned ranking measurements below. Part 2 provides two complementary groups of ablation experiments.
Theoretically, HiQFlow's IQE parameterization enforces the triangle inequality on the learned distances. The learning objective encourages distances on observed one-step transitions to remain within the unit transition cost, while separating general state–goal pairs. For connected trajectory fragments forming a feasible path (s0,…,sT=g) of T transitions, the triangle inequality gives dθ(s0,g)≤∑i=0T−1dθ(si,si+1). These fragments can come from different recorded trajectories, so local transition constraints also restrict the distance along a composed path that need not appear in full in the dataset.
HiQFlow uses these distances in the high-level advantage Ath=dθ(st,g)−dθ(w,g), where w=st+k is a future state from the same training trajectory as st. AWR therefore favors subgoals with greater estimated progress toward g, including those whose continuation to g relies on connected fragments from other trajectories.
To assess whether the evaluator reliably ranks progress toward the final goal, we examine its long-horizon ranking margins and measure ordering on shared oracle trajectories.
Theoretically, Section 4 in the manuscript shows that the ideal undiscounted distance margin for a k-step successor on an optimal trajectory is mQRL⋆(H,k)=k, independent of the remaining horizon H. The corresponding error difference needed to reverse the ranking therefore does not shrink with H, whereas discounted TD values such as those used by HIQL have shrinking ideal margins. Learning undiscounted distances avoids this discount-induced margin compression.
Empirically, we measure the order-consistency ratio rc, the fraction of state pairs separated by k steps along an oracle trajectory whose estimated distances have the correct ordering. All methods use the same five oracle trajectories per environment, with k=25 for AntMaze and k=100 for HumanoidMaze. Table R-1 reports the results, also included in Appendix C.1 in the manuscript. HiQFlow has the highest ratio in all nine settings, averaging 0.974, compared with 0.890 for OTA and 0.806 for HIQL, indicating more reliable ordering on the evaluated trajectories.
Table R-1. Order-consistency ratios (mean over 5 oracle trajectories per environment).