acceptodds
Under review as a conference paper at ICLR 2027

Quasimetric Distance with Flow-Based Planning for Offline Hierarchical Goal-Conditioned RL

Abstract

Offline hierarchical goal-conditioned reinforcement learning (GCRL) decomposes long-horizon control into high-level subgoal selection and low-level execution. However, existing temporal-difference (TD) value-based methods often yield unstable high-level subgoal rankings, because they derive advantages from discounted value differences where long-range progress signals are heavily compressed and easily overwhelmed by bootstrapped approximation errors. By contrast, we revisit hierarchical GCRL through the lens of the value-quasimetric distance equivalence, and show theoretically and empirically that an undiscounted quasimetric distance induces a long-horizon-consistent ordering over subgoals, while its triangle-inequality geometry naturally supports stitching trajectory fragments. Nevertheless, preserving such equivalence requires high-level planning to choose subgoals directly in the original state space, thereby aligning policy optimization with value evaluation. This reformulates high-level policy learning into multimodal density modeling over feasible intermediate states induced by different routes and stitching choices. Motivated by these insights, we propose Hierarchical offline goal-conditioned RL with shared Quasimetric distance and conditional normalizing Flows (HiQFlow), which uses a ranking order-stable quasimetric distance across both hierarchy levels and a conditional normalizing flow to model multimodal subgoals with tractable likelihoods and efficient sampling. Evaluations on OGBench show that HiQFlow outperforms strong offline GCRL baselines, especially on long-horizon, stitching, and exploration-heavy tasks.

Primary area:
reinforcement learning
Keywords:
Goal-Conditioned RL, Hierarchical Planning, Quasimetric, Normalizing Flows, Trajectory Stitching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u7mbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of stability of predictions from hierarchical models in the context of offline goal-conditioned reinforcement learning. They propose HiQFlow that, as stated by the authors, empirically justify the theoretical insight on using undiscounted quasimetric distance representations for subgoals consistency over long-horizon tasks. Such a formulation enables to keep the margin from the further subgoals in the original state space with a conditional normalizing flow for both low- and high-level policies. On the mazes from OGBench they report improved performance over multi-seed evaluation with additional ablations compared to SOTA baselines.

Critical strengths
  • the results of the method are above or comparable with the state-of-the-art baselines as in their original form, so as when normalizing flows are integrated into their pipelines. There is also the empirical justification of the success rates on stitch datasets, what the theory is about, where there is a dominating gap in performance between the proposed method and related work, the ablation studies are fulfilled to explain important architectural choices such as flow- or gaussian-based policy and the ones mentioned above
Critical weaknesses
  • there is no enough evidence in terms of motivation and model stability that this quasi metric formulation inside the method provides the opportunity for trajectory stitching of the datasets - maybe it is just about the expressivity combined with the normalizing flow pipeline
Questions for authors
  • are AthA_t^h and AtℓA_t^\ell processed somehow before exp⁡(βA)\exp(\beta A), and which beta was used for OTA-NF and HQRL-NF? at the moment it is hard to distinguish how much of Table 2 is the distance or scale weighting
  • can you provide also the RLiable metrics on the already existing checkpoints to address the robust performance improvements of your method upon the baselines?
  • there is a different random of seeds for eik-hiqrl and hiqflow - can there be different evaluation scenarios with your own method and the related work? can you justify the unified evaluation protocol, including same amount of random seeds, the rule of picking out the checkpoint whose results would be reflected in the tables and re-evaluating all the baselines on a single machine?
9 comments

We sincerely thank Reviewer u7mb for the detailed comments on our algorithm design and experimental scope. Below, we provide detailed responses to these points.

[1/2] To W1: The quasimetric evaluator’s contribution to trajectory stitching

We thank the reviewer for this helpful comment. We would like to clarify that HiQFlow uses a conditional normalizing flow only for the high-level policy, while the low-level policy is a goal-conditioned Gaussian MLP. To explain the quasimetric evaluator’s role in stitching, we present the theoretical motivation and learned ranking measurements below. Part 2 provides two complementary groups of ablation experiments.

  • Theoretically, HiQFlow's IQE parameterization enforces the triangle inequality on the learned distances. The learning objective encourages distances on observed one-step transitions to remain within the unit transition cost, while separating general state–goal pairs. For connected trajectory fragments forming a feasible path (s0,…,sT=g)(s_0,\ldots,s_T=g) of TT transitions, the triangle inequality gives dθ(s0,g)≤∑i=0T−1dθ(si,si+1)d_\theta(s_0,g)\le\sum_{i=0}^{T-1}d_\theta(s_i,s_{i+1}). These fragments can come from different recorded trajectories, so local transition constraints also restrict the distance along a composed path that need not appear in full in the dataset.

    HiQFlow uses these distances in the high-level advantage Ath=dθ(st,g)−dθ(w,g)A_t^h=d_\theta(s_t,g)-d_\theta(w,g), where w=st+kw=s_{t+k} is a future state from the same training trajectory as sts_t. AWR therefore favors subgoals with greater estimated progress toward gg, including those whose continuation to gg relies on connected fragments from other trajectories.

  • To assess whether the evaluator reliably ranks progress toward the final goal, we examine its long-horizon ranking margins and measure ordering on shared oracle trajectories.

    Theoretically, Section 4 in the manuscript shows that the ideal undiscounted distance margin for a kk-step successor on an optimal trajectory is mQRL⋆(H,k)=km^\star_{\mathrm{QRL}}(H,k)=k, independent of the remaining horizon HH. The corresponding error difference needed to reverse the ranking therefore does not shrink with HH, whereas discounted TD values such as those used by HIQL have shrinking ideal margins. Learning undiscounted distances avoids this discount-induced margin compression.

    Empirically, we measure the order-consistency ratio rcr^c, the fraction of state pairs separated by kk steps along an oracle trajectory whose estimated distances have the correct ordering. All methods use the same five oracle trajectories per environment, with k=25k=25 for AntMaze and k=100k=100 for HumanoidMaze. Table R-1 reports the results, also included in Appendix C.1 in the manuscript. HiQFlow has the highest ratio in all nine settings, averaging 0.9740.974, compared with 0.8900.890 for OTA and 0.8060.806 for HIQL, indicating more reliable ordering on the evaluated trajectories.

    Table R-1. Order-consistency ratios (mean over 5 oracle trajectories per environment).

    TaskHIQLOTAHiQFlow (ours)
    AntMaze
    antmaze-large-navigate0.9120.8400.980
    antmaze-large-stitch0.8800.8100.970
    antmaze-large-explore0.7560.9660.982
    antmaze-giant-navigate0.9220.8581.000
    antmaze-giant-stitch0.8540.8620.984
    HumanoidMaze
    humanoidmaze-large-navigate0.7660.9000.972
    humanoidmaze-large-stitch0.7480.8760.960
    humanoidmaze-giant-navigate0.7180.9460.954
    humanoidmaze-giant-stitch0.7000.9500.966
    Average0.8060.8900.974

[2/2] To W1: The quasimetric evaluator’s contribution to trajectory stitching

To empirically distinguish the evaluator's contribution from flow expressiveness, we conduct two complementary groups of ablations.

  • First, we retain HiQFlow's policy architectures and AWR objectives and change the evaluator. HiQFlow-TD (MLP) replaces it with MLP value networks trained by discounted expectile TD learning. HiQFlow-TD (IQE) retains the quasimetric parameterization but uses the same TD objective, testing whether retaining the triangle-inequality structure is sufficient to match HiQFlow's performance. Both TD variants use two value networks and their target networks. For the additional HumanoidMaze results, we evaluate both TD variants at exactly 11 million updates across four seeds and retain the HiQFlow results reported in Table 1 in the manuscript.

    Table R-2 reports the results. Averaged equally across the five stitching tasks, HiQFlow achieves 84.684.6\\% success, compared with 30.030.0\\% for HiQFlow-TD (MLP) and 29.929.9\\% for HiQFlow-TD (IQE).

    Table R-2. Stitching success rates for evaluator ablations (%, mean ±\pm std over 8 seeds).

    TaskHiQFlow (ours)HiQFlow-TD (MLP)HiQFlow-TD (IQE)
    antmaze-medium-stitch97±197\pm 189±689\pm 688±588\pm 5
    antmaze-large-stitch92±292\pm 229±929\pm 935±635\pm 6
    antmaze-giant-stitch76±676\pm 61±11\pm 11±11\pm 1
    humanoidmaze-medium-stitch91±291\pm 221±921\pm 922±522\pm 5
    humanoidmaze-large-stitch67±267\pm 210±210\pm 24±24\pm 2
    Average84.630.029.9
  • As a complementary comparison, we test whether adding the flow policy to other methods is sufficient to match HiQFlow's stitching performance. We retain the value-learning backbones of HIQL, OTA, and HQRL and replace their high-level policies with conditional normalizing flows using HiQFlow's flow implementation. HIQL-NF retains discounted expectile TD learning and HIQL's learned subgoal representation, while OTA-NF and HQRL-NF retain option-aware TD learning and latent-space quasimetric learning, respectively. All variants are evaluated on the same five predefined OGBench goals per environment, with 5050 episodes per goal.

    As shown in Table R-3, HiQFlow achieves an average success rate of 84.684.6\\% on the five stitching tasks, compared with 53.053.0\\% for HIQL-NF, 77.077.0\\% for OTA-NF, and 36.036.0\\% for HQRL-NF. On antmaze-giant-stitch, HiQFlow achieves 7676\\%, compared with 00\\%, 5050\\%, and 00\\%, respectively. These results show that the flow-based variants retain substantial differences in average stitching performance.

    Table R-3. Flow-policy comparisons on stitching tasks (%, mean ±\pm std over 8 seeds).

    TaskHiQFlow (ours)HIQL-NFOTA-NFHQRL-NF
    antmaze-medium-stitch97±197\pm 195±195\pm 193±293\pm 285±685\pm 6
    antmaze-large-stitch92±292\pm 253±653\pm 689±389\pm 364±1064\pm 10
    antmaze-giant-stitch76±676\pm 60±00\pm 050±350\pm 30±00\pm 0
    humanoidmaze-medium-stitch91±291\pm 292±292\pm 291±391\pm 327±227\pm 2
    humanoidmaze-large-stitch67±267\pm 225±625\pm 662±362\pm 34±14\pm 1
    Average84.653.077.036.0

To Q1: Advantage processing and AWR settings in OTA-NF and HQRL-NF

Thank you for highlighting the importance of the AWR settings in interpreting the comparison. All three methods use raw advantages and a common exponential-weight cap. We detail this processing and the task-dependent AWR coefficients below.

  • HiQFlow, OTA-NF, and our HQRL-NF implementation use raw advantages at both policy levels, without clipping, centering, or standardization before exponentiation. Following the HIQL and OTA implementations for OGBench, we cap the exponential weights at 100100, using wtj=min⁡exp⁡(βjAtj),100w_t^j=\min\\{\exp(\beta_j A_t^j),100\\}, where j∈h,ℓj\in\\{h,\ell\\} denotes the high-level or low-level policy. We do not normalize the weights across the minibatch.

  • For OTA-NF, we follow the original OTA task-specific hyperparameter settings, including its released tuned configurations. For the 16 tasks in Table 2 in the manuscript, we use (βh,βℓ)=(1.0,3.0)(\beta_h,\beta_\ell)=(1.0,3.0) on the large-stitch tasks in PointMaze, AntMaze, and HumanoidMaze, (5.0,3.0)(5.0,3.0) on the PointMaze giant-stitch task, and (3.0,3.0)(3.0,3.0) on the remaining 12 tasks.

    For HQRL-NF, we construct the implementation from the network definitions and training objectives described in the HQRL paper, since an official implementation was unavailable for our reproduction and the paper does not provide task-specific AWR coefficients. We set its AWR coefficients to those used by HiQFlow in Table 6 in the manuscript: (βh,βℓ)=(0.2,2.0)(\beta_h,\beta_\ell)=(0.2,2.0) for PointMaze, (0.3,3.0)(0.3,3.0) for AntMaze, and (0.1,1.0)(0.1,1.0) for HumanoidMaze.

[1/2] To Q3: Comparison under a common evaluation protocol

Thank you for highlighting the importance of a consistent evaluation protocol. We clarify the original evaluation differences and retrain and evaluate the baselines using a common protocol with matched random seeds and training budgets, final-checkpoint evaluation, and the same predefined tasks and evaluation hardware.

  • The Eik-HiQRL results in Table 1 in the manuscript were taken from Table 7 of its paper, which reports the best evaluation results during 1 million training steps across ten seeds. Evaluations are conducted every 100,000 steps on five predefined tasks, with 50 episodes per task. For HiQFlow, we use eight random seeds per environment and report final-checkpoint results, as described in the Training and evaluation setup paragraph of Appendix E in the manuscript. Training lasts 1 million gradient steps for PointMaze and AntMaze, and 2 million steps for HumanoidMaze. Each final checkpoint is evaluated on the five predefined OGBench tasks with 50 episodes per task. We average success rates over these five tasks for each seed and report the mean and standard deviation across the eight seeds.

  • To provide a comparison under a common protocol, we retrain the baselines using the same eight random seeds as HiQFlow and the training budgets described above. For each environment, we evaluate the final saved checkpoint on the five predefined OGBench tasks, with 5050 episodes per task. All checkpoint evaluations use the same server equipped with NVIDIA RTX 4090 GPUs.

    For GCIVL, CRL, QRL, HIQL, OTA, and Eik-HiQRL, we use the released implementations and configurations for the corresponding environments. CGCIVL provides an official implementation but does not include environment-specific configuration files for the method. We use its released code with the hyperparameters reported in Appendix C.3 and Table 2 of the CGCIVL paper, and follow the corresponding OGBench environment settings for hyperparameters not specified there. Since an official implementation of HQRL was unavailable for our reproduction, we implement its training objectives and architecture as described in the paper and follow its reported hyperparameter settings. For hyperparameters not specified in the HQRL paper, we use the corresponding HiQFlow settings.

Reviewer_K3D4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces HiQFlow for offline hierarchical GCRL on OGBench. It replaces the discounted TD critic used in HIQL/OTA with a shared, undiscounted IQE quasimetric distance learned from states and uses a conditional normalizing flow to generate a high-level subgoal policy.

Critical strengths
  • The motivation seems good. The paper clearly explains the limitations of discounted values in long-horizon tasks. The quasimetric margin d(st,g)−d(s,g)=kd^(s_t,g)-d^(s_,g)=k remains constant, making it useful for measuring progress toward distant goals.
  • The use of conditional normalizing flows is a reasonable choice for learning multiple possible subgoals.
Critical weaknesses
  • The author claims the high-level policy can learn different routes, but Algorithm 1 samples a new subgoal at every step. This could cause the robot to switch routes repeatedly instead of following a consistent path. The authors should explain how they avoid this issue and whether subgoals are kept for multiple steps or selected using a fixed rule.

  • The experiments only cover maze locomotion tasks, although OGBench also includes manipulation tasks such as cube, scene, and puzzle. Testing on these tasks would help show whether the proposed method works beyond maze navigation, especially for long-horizon planning and subgoal selection.

  • Figure 2 includes an “InvertiblePLU” block, but its role and implementation are not explained. The same applies to the “Conditional Affine Coupling” blocks. Please define these transformations and explain how the current state and goal condition the coupling layers.

Questions for authors
  • Could you clarify whether the ranking analysis is valid only for the ideal margin or also applies to the learned evaluator?
  • How do you keep the exponential weights exp⁡(βhAth)\exp(\beta_h A_t^h) in Eq. (14) stable during training, especially when k=100k=100?
  • Do you clip or normalize AthA_t^h or the resulting weights to prevent a few transitions with large estimated advantages from dominating the updates?
  • Could you evaluate HiQFlow on one or more of OGBench’s manipulation tasks, such as cube, scene, or puzzle, to see whether the benefits of flow-based subgoal generation extend beyond maze locomotion?
6 comments

We sincerely thank Reviewer K3D4 for the detailed comments on subgoal execution, evaluator reliability, and evaluation beyond navigation. Please find our clarifications on implementation details and performance comparison below.

To W1: Subgoal updates and temporal consistency

We thank the reviewer for raising the question of temporal consistency when the high-level policy represents multiple feasible subgoals. We explain how per-step subgoal generation operates during evaluation and compare it with holding subgoals for multiple steps.

  • During execution, we recompute the subgoal at every environment step from the current state and final goal, then execute one low-level action. This follows the per-step execution pattern of HIQL and OTA. The parameter kk specifies the temporal offset used to construct training subgoals. It is not a duration for holding a subgoal during execution.

  • Multimodality here refers to the high-level subgoal distribution conditioned on the current state and final goal. Different offline trajectories can provide alternative future subgoals for the same or nearby states. During execution, regenerating the subgoal at each step allows the policy to adapt its guidance to the state actually reached, without requiring the previous subgoal to be reached first. This also allows successive decisions to use subgoal predictions learned from different trajectory segments.

    During evaluation, we generate each subgoal deterministically by passing the conditional Gaussian base mean through the flow at zero sampling temperature. This removes sampling randomness as a source of switching between alternative subgoals.

  • To test whether holding subgoals for multiple steps improves performance, we evaluate HiQFlow with different subgoal holding intervals on the medium and large AntMaze stitching and exploration tasks. These tasks involve combining trajectory segments or learning from exploratory trajectories, making them relevant to the concern about successive subgoal choices. We use the trained policies from Table 1 in the manuscript across eight seeds and vary only the holding interval during evaluation. Each subgoal remains fixed for mm environment steps, while the low-level policy recomputes its action from the latest state at every step. For each seed and interval, we evaluate five predefined goals per environment with 5050 episodes per goal. We compare m∈5,10,25m\in\\{5,10,25\\} with m=1m=1, the per-step baseline reported in Table 1 in the manuscript. Table R-1 shows that per-step updates achieve the highest mean success on all four tasks, with the largest reductions under multi-step holding at m=25m=25, particularly on large-stitch and large-explore.

Table R-1. Success rates under different subgoal holding intervals (%, mean ±\pm std over 8 seeds).

Taskm=1m=1m=5m=5m=10m=10m=25m=25
antmaze-medium-stitch97±197\pm 195±195\pm 196±196\pm 192±292\pm 2
antmaze-large-stitch92±292\pm 290±190\pm 191±291\pm 258±658\pm 6
antmaze-medium-explore94±294\pm 291±391\pm 388±588\pm 576±576\pm 5
antmaze-large-explore89±389\pm 388±488\pm 483±483\pm 438±638\pm 6

To W2 and Q4: Manipulation performance

We thank the reviewer for suggesting an evaluation beyond maze locomotion. We have added experiments on Cube and Scene to examine HiQFlow’s performance on manipulation tasks.

  • To assess performance beyond maze navigation, we evaluate HiQFlow on Cube and Scene manipulation tasks. Our evaluation protocol uses 11 million training updates across eight seeds and evaluates the final checkpoint on five goals with 5050 episodes per goal. GCIQL, QRL, and HIQL results are taken from the OGBench paper and are reported across eight seeds. OTA does not report results for these two state-based manipulation tasks. Eik-HiQRL reports the best evaluation during 11 million updates across ten seeds, which differs from our final-checkpoint, eight-seed protocol. We retrain both methods using their released implementations and reported architectures and hyperparameters, supplemented by the corresponding OGBench task settings, following the same training budget and evaluation protocol as HiQFlow.

    As shown in Table R-2, HiQFlow achieves 1818\\% success on Cube and 2929\\% on Scene, with higher mean success than QRL, OTA, and Eik-HiQRL on both tasks. Compared with HIQL, HiQFlow has higher mean success on Cube but lower mean success on Scene, and it remains below GCIQL on both tasks. These mixed results show that HiQFlow's advantages on maze navigation do not consistently extend to manipulation.

    Table R-2. Manipulation success rates (%, mean ±\pm std over 8 seeds).

    TaskGCIQLQRLHIQLOTAEik-HiQRLHiQFlow (ours)
    cube-single-play68±668\pm 65±15\pm 115±315\pm 310±310\pm 30±00\pm 018±118\pm 1
    scene-play51±451\pm 45±15\pm 138±338\pm 323±323\pm 38±38\pm 329±429\pm 4
  • To investigate why HiQFlow's gains on navigation do not consistently extend to manipulation, we conduct two groups of ablations to examine the roles of evaluator learning and high-level policy modeling. In the evaluator ablation, we retain the flow policy and replace the distance-learning objective with discounted expectile TD learning. HiQFlow-TD (IQE) keeps the IQE parameterization to examine the change in learning objective, while HiQFlow-TD (MLP) also replaces IQE with unconstrained MLP value networks to test an alternative evaluator without quasimetric structure. In the policy ablation, we retain HiQFlow's evaluator and replace the high-level flow with a Gaussian (HiQ-Gaussian) or a five-component Gaussian mixture (HiQ-MoG), testing the effect of the policy parameterization. All variants follow the same evaluation protocol, and the results are shown in Table R-3.

    Table R-3. Success rates for manipulation ablations (%, mean ±\pm std over 8 seeds).

    TaskHiQFlow (ours)HiQFlow-TD (IQE)HiQFlow-TD (MLP)HiQ-GaussianHiQ-MoG
    cube-single-play18±118\pm 134±534\pm 532±532\pm 512±412\pm 412±312\pm 3
    scene-play29±429\pm 465±465\pm 463±763\pm 721±621\pm 620±520\pm 5

    Both TD variants improve mean success. HiQFlow-TD (IQE) increases success from 1818\\% to 3434\\% on Cube and from 2929\\% to 6565\\% on Scene, with similar results for the MLP variant (3232\\% and 6363\\%). In contrast, replacing the flow with a Gaussian or mixture policy reduces mean success on both tasks. These comparisons identify evaluator learning as a promising direction for improving manipulation performance, while the flow remains beneficial among the tested policy parameterizations. In particular, the improvement with IQE retained shows that quasimetric architecture alone does not explain the performance gap.

    One possible explanation is that estimating the remaining goal-reaching cost aligns closely with assessing route progress in navigation, whereas manipulation additionally requires evaluating the prerequisites and local feasibility of successive object interactions. HiQFlow's shared full-state distance evaluator must capture these relations at both policy levels. Kang et al. [1] also report that QRL's strong maze performance does not extend to manipulation and highlight a potential limitation of enforcing quasimetric constraints throughout learning. They argue that enforcing quasimetric structure on intermediate value estimates can restrict optimization, since these estimates need not be quasimetric.

    We acknowledge that HiQFlow's performance gains on maze navigation do not consistently extend to manipulation tasks and have added a discussion of this limitation in Appendix F (Limitations and Future Work) in the manuscript.

References

[1] H. Kang, B. Kim, and M. Oh, “Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning,” ICML 2026.

To W3: Flow transformations and state–goal conditioning

Thank you for the suggestion. We have clarified the roles of the flow blocks in Section 5 and their transformations and conditioning mechanism in Appendix E of the manuscript.

Given the current state sts_t and final goal gg, we concatenate them into the conditioning vector ct=[st,g]c_t=[s_t,g]. A base-distribution MLP takes ctc_t as input and predicts the mean and log-standard deviation of a conditional diagonal Gaussian.

We sample an initial latent vector z0z_0, with the same dimension as the subgoal, from this distribution and pass it through the flow to obtain w=fη(z0;ct)w=f_\eta(z_0;c_t), where fηf_\eta denotes the flow with parameters η\eta. During evaluation, zero sampling temperature sets z0z_0 to the predicted Gaussian mean. Each block applies PLU followed by conditional affine coupling, with ctc_t fixed throughout the flow. Below, xx and yy denote a transformation's input and output vectors, both with the same dimension as ww.

  • For coordinate mixing, the InvertiblePLU block applies y=xWy=xW, where W=PLUW=PLU. Here, PP is a fixed permutation matrix, LL is a learned unit-diagonal lower-triangular matrix, and UU is a learned upper-triangular matrix. The triangular entries, including the diagonal of UU, are directly learned. The map is invertible when all diagonal entries of UU are nonzero, with inversion computed by triangular solves and log-Jacobian determinant ∑ilog⁡∣Uii∣\sum_i\log|U_{ii}|. Each block uses the same learned matrix across state–goal conditions to mix coordinates before the coupling transformation.

  • To condition the coupling transformation on the state and goal, a binary mask divides xx into retained coordinates xAx_A and coordinates xBx_B to be transformed. We retain xAx_A, set the positions corresponding to xBx_B to zero, and concatenate this masked vector with ctc_t. Each block's MLP takes this concatenated input and outputs shift and raw log-scale vectors, each with the subgoal dimension.

    After bounding the log-scales to [−2,2][-2,2] with a scaled tanh, we apply the predicted transformation only to xBx_B. Writing the corresponding log-scales and shifts as aBa_B and bBb_B, the transformation is yA=xAy_A=x_A and yB=xB⊙exp⁡(aB)+bBy_B=x_B\odot\exp(a_B)+b_B, where yA,yBy_A,y_B are the output coordinate groups and ⊙\odot denotes element-wise multiplication. Thus, the state and goal enter every coupling MLP together with the retained coordinates to determine the scale and shift.

    The unchanged coordinates allow the same scale and shift to be recomputed during inversion, and the log-Jacobian determinant is ∑j(aB)j\sum_j(a_B)_j. Masks alternate across blocks so that both coordinate groups are transformed.

Likelihood evaluation reverses the transformations and accumulates their log-Jacobian determinants to compute the conditional log-likelihood used by high-level AWR. Further implementation details are given in Appendix E, with network widths and block counts in Table 6 in the manuscript.

To Q1: Ranking consistency of learned evaluators

Thank you for this question. The ranking advantage is established theoretically for the ideal margins and is also observed empirically in the learned evaluators across the tested settings.

Section 4 in the manuscript analyzes the long-horizon ranking advantage of undiscounted quasimetric distances over discounted TD values. In addition, we directly test the trained evaluators using the order-consistency ratio rcr^c, the fraction of state pairs separated by kk steps whose estimated distances preserve the correct ordering along an oracle trajectory. All methods use the same five oracle trajectories per environment, one for each predefined OGBench evaluation task, with k=25k=25 for AntMaze and k=100k=100 for HumanoidMaze. The results are shown in Table R-4 and are also reported in Appendix C.1 and Figure 7 in the manuscript.

The results show that HiQFlow achieves the highest rcr^c in all nine settings. Its mean ordering-violation rate, 1−rc1-r^c, is 2.62.6\\%, compared with 11.011.0\\% for OTA and 19.419.4\\% for HIQL. This more reliable trajectory ordering in the learned quasimetric evaluator provides empirical support for the margin analysis in the evaluated settings.

Table R-4. Order-consistency ratios (mean over 5 oracle trajectories per environment).

TaskHIQLOTAHiQFlow (ours)
AntMaze
antmaze-large-navigate0.9120.8400.980
antmaze-large-stitch0.8800.8100.970
antmaze-large-explore0.7560.9660.982
antmaze-giant-navigate0.9220.8581.000
antmaze-giant-stitch0.8540.8620.984
HumanoidMaze
humanoidmaze-large-navigate0.7660.9000.972
humanoidmaze-large-stitch0.7480.8760.960
humanoidmaze-giant-navigate0.7180.9460.954
humanoidmaze-giant-stitch0.7000.9500.966
Average0.8060.8900.974
Reviewer_ti5jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a HiQFlow method for offline goal-conditioned reinforcement learning, which is based on the quasimetric distance for scoring sub-goals.

Critical strengths

The author conducted extensive experiments to verify the effectiveness of HiQFlow.

Critical weaknesses
  1. The writing is not very clear. For example, in section 4, the author listed and analyzed several previous works, including HIQL, OTA, and QRL. However, when comparing the experimental results, only HIQL and QTA were provided, while QRL was omitted. The consistency here is slightly weak. Additionally, HIQL, OTA, and QRL should be cited. (2) At line 148, "QMet(S)" can be further explained.
  2. If I have not misunderstood, HiQFlow is based on the idea of QRL to learn a function for the score of high-level subgoals, which is equivalent to applying the QRL mechanism for scoring states to hierarchical goal-conditioned RL. However, the author only described the advantages of QRL's idea and did not highlight the differences between HiQFlow and QRL.
  3. The author has been emphasizing the problems of other methods in subgoal scoring before the section 5, but in the section 5, they introduced "flow" as the high-level policy. The sudden introduction of the "flow-based strategy" here is slightly disconnected from the scoring described earlier.
Questions for authors
  1. What is the difference between HiQFlow and QRL? Where does HiQFlow's advantage lie compared to QRL?
  2. Where are the points of alignment between flow matching and the scoring mechanism adopted in this paper?
5 comments

We sincerely thank Reviewer ti5j for the careful reading and constructive comments. Below, we provide point-by-point responses to the weaknesses and questions raised.

To W1: QRL comparison, citations, and notations

Thank you for these suggestions on presentation. We clarify the comparison with QRL and have supplemented the citations and the definition of QMet(S)\mathrm{QMet}(\mathcal S) in the manuscript.

  • We would like to clarify that Table 1 in the original manuscript has already included QRL, HIQL, and OTA on all 16 tasks, covering the three methods analyzed in Section 4.

  • For the citations, we have provided the citations of HIQL, OTA, and QRL when first introducing them and in the first paragraph of Section 4. Following the reviewer's suggestion, we have reviewed and supplemented the references throughout the paper and appendix, citing each method at its first mention within the relevant paragraphs.

  • For the notation, following Section 2.2 of QRL, QMet(S)\mathrm{QMet}(\mathcal S) denotes the set of quasimetric functions defined on the state space S\mathcal S. Each such function satisfies three conditions for all s,g,z∈Ss,g,z\in\mathcal S: nonnegativity d(s,g)≥0d(s,g)\geq0, zero self-distance d(s,s)=0d(s,s)=0, and the triangle inequality d(s,g)≤d(s,z)+d(z,g)d(s,g)\leq d(s,z)+d(z,g). Thus, −V⋆∈QMet(S)-V^\star\in\mathrm{QMet}(\mathcal S) means that the negative optimal value has these quasimetric properties.

    We have revised the Quasimetric RL paragraph in Section 3 (Preliminaries), Page 3 of the manuscript, to define QMet(S)\mathrm{QMet}(\mathcal S) and its defining properties at first occurrence.

To W2 and Q1: Differences and advantages over QRL

Thank you for highlighting the need to distinguish HiQFlow from QRL. The differences concern how quasimetric distances guide hierarchical policy learning and our theoretical analysis of long-horizon subgoal ranking, as detailed below.

  • HiQFlow differs from QRL in how the learned distance guides policy learning. Compared with QRL's flat policy, HiQFlow separates long-horizon goal reaching into high-level subgoal selection and low-level control. The low-level policy focuses on nearby subgoals, while the high-level policy guides progress toward the final goal. HiQFlow uses a shared quasimetric evaluator to train both policy levels through advantage-weighted regression (AWR). The high-level update weights data subgoals by their estimated progress toward the final goal, while the low-level update weights actions by their estimated progress toward the assigned subgoal.

    For offline continuous control, both the original QRL and its OGBench implementation learn a latent dynamics model to estimate Q(s,a,g)Q(s,a,g) for DDPG+BC policy updates. HiQFlow computes AWR weights directly from distance differences on observed transitions and future-state subgoals, without requiring a learned dynamics model for its policy updates.

  • Our theoretical analysis investigates quasimetric distance learning from the perspective of subgoal ranking in hierarchical GCRL. We compare the ideal high-level ranking margins of undiscounted distances and discounted TD values, and characterize how estimation errors can reverse the rankings underlying AWR weights. The ideal undiscounted distance margin does not shrink with the remaining horizon, providing a rationale for using this evaluator to guide long-range subgoal selection.

    Empirically, Table 1 in the manuscript reports an average success rate of 89.889.8\\% for HiQFlow across all 1616 environments, compared with 45.445.4\\% for QRL.

To W3 and Q2: Connecting quasimetric scoring to the flow policy

We thank the reviewer for highlighting the need to clarify how subgoal scoring informs the policy design. To preserve the long-horizon ranking advantage of quasimetric distances for hierarchical subgoal selection, HiQFlow predicts subgoals in the original state space, where future subgoals from different trajectories can form a multimodal conditional distribution. To model complex multimodal subgoals, we therefore use a conditional normalizing flow, enabling exact conditional log-likelihood computation for AWR training and efficient subgoal sampling through a single pass through its invertible layers.

  • For the subgoal representation, the quasimetric distance underlying our ranking analysis is defined over pairs of original states, with the value–distance correspondence −V⋆∈QMet(S)-V^\star\in\mathrm{QMet}(\mathcal S). To preserve its long-horizon ranking advantage during hierarchical policy extraction, the subgoal representation should retain the goal-reaching value distinctions underlying these rankings. The existing HQRL method [1] applies quasimetric distances to hierarchical GCRL. However, it learns these distances in a latent goal space obtained through contrastive state compression. As analyzed in Appendix B of our manuscript, the contrastive objective alone does not ensure that the compressed representation preserves the value-relevant information needed for subgoal ranking.

    HiQFlow therefore evaluates quasimetric distances and predicts subgoals in the original state space, avoiding a separate contrastive compression stage that may discard value-relevant information. With subgoals represented in this space, the high-level policy needs to model distinct future states provided by trajectories passing through the same or nearby states. For a given current state and final goal, these alternatives can form a multimodal conditional distribution that a single Gaussian may not adequately capture.

  • To represent these original-state subgoals while supporting the evaluator-guided AWR objective, we use a conditional normalizing flow rather than flow matching. The quasimetric distance difference determines the AWR weight for each data subgoal, and the flow is trained to maximize its weighted conditional log-likelihood. Its invertible transformations allow a flexible multimodal density, while the change-of-variables formula gives the exact conditional log-likelihood required by this objective. For sampling, each subgoal is generated by passing a base Gaussian sample through a fixed stack of invertible transformations once. This avoids the repeated denoising evaluations used by standard multistep diffusion policies and the numerical ODE integration used by ODE-based flow-matching policies. These properties make the conditional flow well suited to quasimetric-guided AWR, combining multimodal subgoal modeling with exact likelihood computation and direct subgoal generation.

References

[1] K. Ke, Z. Ruan, S. Tan, and W. Wu, “Hierarchical Quasimetric Reinforcement Learning,” MLNN 2025.

AuthorsOpenReview ↗

A gentle reminder for feedback

Dear Reviewer ti5j,

We would appreciate your feedback on our responses. Could you please let us know whether our responses have addressed your concerns? We would be happy to discuss any remaining issues.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.