acceptodds
Under review as a conference paper at ICLR 2027

Doubly Dual-Weighted Flow Matching for Hierarchical Offline Goal-Conditioned RL

Abstract

Offline goal-conditioned reinforcement learning (GCRL) learns goal-reaching policies from static datasets without further environment interaction. We propose *Doubly Dual-Weighted Flow Matching* (DFM), a hierarchical goal-conditioned flow policy that mitigates distant goals by decomposing long-horizon goal reaching into a high-level module that proposes nearer subgoals and a low-level flow module that generates actions conditioned on those subgoals. To mitigate sparse rewards, we introduce a doubly dual-weight mechanism that derives two optimal correction ratios from a single reward-tilted target distribution: a state–action weight ratio, obtained from a primal–dual Bellman-flow program for low-level policy extraction; and a state weight ratio, recovered through an auxiliary off-policy evaluation objective, that reweights high-level subgoal selection. On OGBench, a benchmark covering navigation, manipulation, and combinatorial-puzzle tasks, DFM achieves higher average success rates than the evaluated GCRL baselines on long-horizon goal-reaching tasks.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, goal-conditioned rl, flow policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WAdwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a flow matching policy formulation for offline goal-conditioned reinforcement learning as a construction upon the existing hierarchical methods. they derive a state-action ratio that weights low-level conditional flow matching and a state ratio that reweights expectile regression of the value, these are both integrated in low- and high-actor trainings. Authors report improved success rates on OGBench compared with the baselines.

Critical strengths
  • the theory looks correct and evidential for all the parts of the method to be applied, including low-level flow objective which is related to occupancy ratio of the dual formulation acquired by the available reward. The state-action dual itself is the standard form of DICE, and there is no need to have a separate density estimator which provides potential intuition for better scaling
  • the authors provide a reasonable amount of baselines to compare on a justified OGBench benchmark with multiple random seeds, additional ablation studies also demonstrate the necessity in each part of the main method to be integrated and justify the needed complexity of the idea
Critical weaknesses
  • the empirical implementation of the method seems to be misaligned with the theoretical background (or at least it is not so straightforward to understand how exactly training of the method is implemented): wSw_S is the key term to resolve corrections as on low-, so with high-level flows, but high-actor is being unweighted by this term. Here comes the question - is this fact addresses the ambiguity of the exact method pipeline or theoretical support is insufficient for this concrete hierarchical flow agent formulation?
  • the main text attributes to weighting as the support for improved occupancy, thereby this fact should provide improved performance on stitch tasks on OGBench, but there is the gap with the baselines such as HIQL and OTA (e.g. 91±2 on antmaze-medium-stitch, against 94±1 for HIQL and 93±1 for OTA)
Questions for authors
  • can you clarify the necessity of wSw_S for high-level flow and provide a more motivational and intuitive aspects of how exactly the theoretical propositions reflect the empirical results?
  • is the distillation really necessary for better success rate in tables 2-4? what is the performance without it and without Q-guidance?
  • have you tried to isolate the effect of wSAw_{SA} and wSw_S on HIQL and OTA? what is more important - flow integration or these additional occupancy weights?
Reviewer_2SJcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper weights a hierarchical flow policy for offline GCRL with DICE-derived ratios, a state-action ratio for the low-level policy and a state ratio for value learning that guides subgoal selection, and adds Q-guided one-step distillation. It reports the best average success on OGBench navigation, manipulation, stitching and noisy-data tasks.

Critical strengths

Giving both levels of the hierarchy a principled occupancy weight is a reasonable idea, the derivations are clear, and the evaluation covers a broad range of OGBench datasets with useful ablations.

Critical weaknesses
  1. The second weight adds less than claimed. The state ratio is simply the action marginal of the state-action ratio and could be obtained by regression, and weighting the expectile value loss by a state-only weight leaves its target unchanged, so it only redistributes approximation error. The low-level objective is the standard regularized DICE formulation.
  2. The gains may come from Q-guided distillation rather than from the weights. This stage gives a large boost on its own, the hierarchical baselines do not use it, and in the stitching comparison the closest ablation appears to be reported without it.
  3. The evaluation is selective. On the longest-horizon maze domain, reported only in the appendix, the method falls far behind OTA unless OTA's value learning is added; the larger stitching datasets are missing, the strongest baseline is absent from some tables, and GoFAR and SMORe are not compared.
  4. The protocol favors the method. Baseline numbers are copied from OGBench while the method is tuned per domain, and the robustness claim rests on a single, nearly saturated task.
Questions for authors
  1. Why estimate w_S with the OPE dual instead of regressing w_SA on (s, g), given L236-238? Could you compare the two estimators and report results when w_S is replaced by uniform weights or by a shuffled weight with the same distribution?
  2. Please confirm which HFIQL configuration is reported in Table 3, and report HIQL, OTA and HFIQL with the same Q-guided one-step distillation on the tasks of Tables 2-4. Which variants in Fig. 2 include distillation and Q-guidance?
  3. Can you report D²FM without OTA on the full set of OGBench state-based navigate, stitch and noisy datasets, including large/giant stitch and HumanoidMaze, and add CGCIVL to Tables 3 and 4?
  4. How were β_hi and λ_q selected per domain (selection criterion, seeds, number of configurations), and how does D²FM perform with a single shared configuration? How many seeds do the copied baseline numbers use?
  5. How does D²FM compare with GoFAR and SMORe under the same protocol, and with a hierarchical flow policy whose low level is weighted by GoFAR's ratio?
Reviewer_ZEQ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a hierarchical offliee goal-conditional rl method. The author conducted a large number of experiments, and the experimental results proved the effectiveness of this method.

Critical strengths
  1. The experimental results are very good
  2. The experiment was conducted quite thoroughly
Critical weaknesses

To be honest, I didn't really understand the hierarchical details of this paper, I can only comment the paper from a high-level perspective:

  1. the introduction of this paper was not well motivated. The author introduced various types of methods in the second paragraph of the introduction, but only mentioned the shortcomings of the Dice, without summarizing the shortcomings of other works.
  2. Based on the introduction, which was not well motivated, I did not find any theoretical advantages of this work compared to existing works. Table 1 only shows the differences between this work and existing works, but does not indicate where the advantages lie.
  3. There are also some minor writing issues, such as FQL should add a citation (line 70); In addition, it would be helpful to clarify the meanings of (v_{\theta_{hi}}) and (v_{\theta_{lo}}) in Equations 13 and 15, respectively.
Questions for authors
  1. can you explain the motivation of the work?
  2. what the advantages of this work compared with the previous works?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.