acceptodds
Under review as a conference paper at ICLR 2027

Skill-Flow: Learning Skill-based Hierarchical Policies via Flow Models for Offline Goal-Conditioned Reinforcement Learning

Abstract

We address the problem of learning policies for long-horizon goal-conditioned tasks with sparse rewards from offline datasets. Such problems are difficult for two primary reasons: sparse rewards in a long-horizon setting hinder value function learning, and limited data induces out-of-distribution (OOD) errors during policy learning. We tackle these challenges using a skill-based hierarchical framework with flow policies at two levels. The high-level policy selects temporally extended skills to guide long-horizon planning, while the low-level policy executes actions conditioned on the selected skill. Temporally extended skills alleviate sparse rewards by enabling the learning of goal-conditioned value functions, while hierarchical decomposition reduces error accumulation over long horizons. To mitigate OOD errors, we learn behavior-cloning flow policies over skills and actions, then steer their bounded input noise using learned value functions towards high value regions. This construction keeps the resulting policies within the support of the learned behavior-cloning policies. We evaluate our approach on the OGBench benchmark and demonstrate superior performance over state-of-the-art hierarchical and value learning based baselines.

Primary area:
reinforcement learning
Keywords:
Offline RL, Goal-Conditioned RL, Hierarchical RL

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_JFMsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses long-horizon offline goal-conditioned reinforcement learning through hierarchical flow policies, bounded-noise guidance, and a delayed-reward high-level IQL update. Experiments on OGBench demonstrate strong performance across navigation and manipulation tasks.

Critical strengths
  1. The method demonstrates strong performance across challenging long-horizon navigation and manipulation tasks.

  2. The ablations and sensitivity analyses provide useful evidence for the main design choices and the method’s robustness.

Critical weaknesses
  1. The high-level update changes both reward aggregation and effective discounting, making it difficult to attribute the improvement specifically to delayed rewards.

  2. The ablation replaces the entire multi-step update with one-step IQL. A comparison with standard multi-step learning under the same hierarchy is needed to isolate the proposed contribution.

  3. Algorithm 1 differs from the main text in both the skill encoding and high-level reward, leaving unclear which implementation produced the reported results.

  4. Theorem 1 claims containment in the original behavior-policy support, whereas the construction only establishes containment in the learned BC support. I suggest clarifying the assumptions needed to connect these supports and justify the claimed guarantee.

Questions for authors
  1. Which skill encoding and high-level reward were used in the experiments?

  2. Is Theorem 1 intended to assume exact behavior cloning or support consistency? If not, what justifies transferring support containment from the learned BC policy to the original behavior policy?

2 comments

Response to reviewer ECwA:

We thank the reviewer for recognizing the strong performance of our method across challenging long-horizon navigation and manipulation tasks. We also appreciate the reviewer acknowledging the usefulness of our ablation studies and sensitivity analyses in validating the key design choices and demonstrating the robustness of our method.

Summary

In brief, we address the following:

  • Clarification of the delayed-reward formulation and skill-level discounting.
  • Clarification of the existing ablation against standard N-step IQL under the same hierarchy.
  • Correction of typographical errors in Algorithm 1 regarding skill encoding and high-level rewards.
  • Clarification of the theoretical conditions underlying behavior-policy support containment in Theorem 1.

Weaknesses

W1: The high-level update changes both reward aggregation and effective discounting.

In our proposed delayed-reward formulation, rewards are assigned upon the completion of each skill rather than at every primitive time step. Since no intermediate rewards are assigned during skill execution, we perform discounting at the skill level. The change in effective discounting is therefore a direct consequence of defining the reward and return at the skill level, rather than an independently introduced modification.


W2: A comparison with standard multi-step learning under the same hierarchy is needed.

We thank the reviewer for raising this concern. We have already conducted the requested comparison, with the results reported in Section 5.3 (Q4, Figure 6) and Table 8 (column 3) of the supplementary material, under the ablation No N-step IQL-D. We have revised the description of this ablation in Section 5.3 to eliminate any ambiguity. Specifically, we replace the proposed high-level N-step IQL-D update with a standard N-step Bellman backup that accumulates per-step rewards and uses γN\gamma^N for discounting, while keeping the hierarchical architecture, skill encoder, and both policy extractors unchanged. The results show that removing N-step IQL-D substantially degrades performance, with success rates dropping to zero on all three giant-stitch tasks, compared with 68–91% for Skill-Flow. This controlled comparison demonstrates the effectiveness of our proposed high-level value learning formulation over standard N-step value learning.


W3: Algorithm 1 differs from the main text in skill encoding and high-level reward.

We thank the reviewer for pointing out the typographical errors in Algorithm 1. Throughout the paper, the skill encoding is defined as zk=ϕ(sk,sk+N)z_k=\phi(s_k,s_{k+N}), and the high-level reward is defined as the endpoint reward r(sk+N,g)r(s_{k+N},g) in Equation (4). We have corrected the corresponding typographical errors in lines 6 and 10 of Algorithm 1 in the revised supplementary material.


W4: Theorem 1 claims behavior-policy support containment, whereas the construction only establishes BC support containment.

We thank the reviewer for raising this concern. The flow-matching objectives in Equations (6) and (12) are formulated to recover the underlying behavior-policy distributions at the high and low levels, respectively. In the theoretical setting of exact distribution recovery, the BC policies and their corresponding behavior policies have identical supports. We have clarified this relationship immediately after Equations (6) and (12) in the revised manuscript. Furthermore, following Theorems 1 and 2 of ReFORM (Zhang et al., 2026), Remarks 2 and 3 establish containment within the respective BC policy supports. We have revised both remarks to explicitly show how this containment extends to the corresponding behavior-policy supports. Theorem 1 follows directly from these results, and we have revised its statement to explicitly clarify the theoretical setting under consideration.

Questions

Q1: Which skill encoding and high-level reward were used in the experiments?

As clarified in our response to W3, we use the skill encoding zk=ϕ(sk,sk+N)z_k=\phi(s_k,s_{k+N}) and the high-level endpoint reward r(sk+N,g)r(s_{k+N},g) in all our experiments. We have corrected the corresponding typographical errors in Algorithm 1.

Q2: Does Theorem 1 assume exact behavior cloning or support consistency?

As clarified in our response to W4, Theorem 1 considers the theoretical setting of exact behavior distribution recovery, in which the supports of the BC policies coincide with those of the corresponding behavior policies. We have revised Remarks 2 and 3 and Theorem 1 to make this relationship and the underlying theoretical setting explicit.

AuthorsOpenReview ↗

Response follow up request

Dear Reviewer,

As the rebuttal period is coming to an end, we kindly ask you to review our response. If you have any remaining concerns, we would be happy to provide further clarification. If our response has addressed your concerns, please consider raising your score.

Best, The Authors

Reviewer_ECwAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Skill-Flow, a skill-based hierarchical method for offline goal-conditioned RL that learns high-level values with N-step IQL with delayed rewards and extracts flow policies at both levels by steering the bounded input noise of behavior-cloning flows toward high-value regions.

Critical strengths

The paper addresses two important challenges in long-horizon offline GCRL, value learning and OOD errors in policy extraction, and supports each design choice with a didactic example, which I found helpful. The evaluation also covers a broad range of OGBench tasks across horizon lengths and dataset qualities, and is complemented by ablations on each component (hierarchy, N-step IQL-D, and support constraints at each level) as well as sensitivity analyses.

Critical weaknesses

1. Clarity of writing: Several key components are defined inconsistently or not at all. For example, the skill is defined as zk=ϕ(sk,sk+N)z_k = \phi(s_k, s_{k+N}) (Section 4) but formed as ϕ(sk,g)\phi(s_k, g) in Algorithm 1, and how ϕ\phi is learned is not explained. Given that multiple components are introduced in the paper, such as the skill encoder, the high- and low-level policies and critics, and the noise generators, it is understandable that the writing is not completely clean. However, I believe that the current version of the paper requires significant effort to improve the writing and clarity of Skill-Flow.

2. Clarity of experimental settings: Only Skill-Flow's hyperparameters are reported, and baselines are said to use "reported values in the paper," although SHARSA, DSHARSA, and HFQL are evaluated on environments not covered in their original papers (e.g., pointmaze), with some near-zero results. Furthermore, the noise-ball radii (hh and ll), key hyperparameters of Skill-Flow, are not reported. Overall, it is hard to fairly assess the claimed empirical gain in the current version of the paper.

3. Computational cost: The method uses two flow networks plus separate Q and V networks at each level, totaling 8 learnable networks, which makes the proposed method computationally heavy. Moreover, Appendix F.5 shows that Skill-Flow is slower at inference than DSHARSA, contradicting the claim that rejection-sampling methods are prohibitive in high dimensions. Providing a training- and inference-cost comparison with the baselines would be beneficial for further understanding the advantage of Skill-Flow over the baselines.

Questions for authors

See Weaknesses please

3 comments
AuthorsOpenReview ↗

Response to questions

Response to reviewer ECwA:

We thank the reviewer for recognizing the importance of the challenges addressed by our work and acknowledging the usefulness of our motivating examples, comprehensive evaluations, ablations, and sensitivity analyses.

Summary

In brief, we address the following:

  • Skill encoder notation and joint learning.
  • Baseline hyperparameters and noise-ball radii.
  • Inference cost and rejection-sampling trade-off.
  • Training cost and evaluation frequency.

Weaknesses

W1: Skill encoder notation and joint learning.

Thank you for pointing out the typographical error. We have corrected the notation in line 6 of Algorithm 1 in the revised supplementary material. Specifically, the latent skill is defined as zk=ϕ(sk,sk+N)z_k=\phi(s_k,s_{k+N}), consistent with the rest of the paper. Regarding the learning of ϕ\phi, Section 4.2.1 (immediately after Equation (4)) already states that the skill encoder is learned jointly with the high-level value functions through the N-step IQL-D objectives. To further improve clarity, we have expanded this explanation in the revised manuscript.

W2: Missing baseline hyperparameters and noise-ball radii.

We thank the reviewer for pointing this out. In the revised manuscript, we have included two additional tables (Table 10 and Table 11) in the supplementary material reporting the network architecture, algorithm-specific hyperparameters, and goal-sampling configurations for all eight baselines. The hyperparameter values were adopted from their respective original papers and official implementations. For HFQL, we adopted the hyperparameter values reported for FQL. Furthermore, we have updated the hyperparameter table for Skill-Flow to explicitly report the high- and low-level noise-ball radii as lh=10l_h=\sqrt{10} and lℓ=dim⁡(A)l_\ell=\sqrt{\dim{(\mathcal{A})}}, respectively, where dim⁡(A)\dim{(\mathcal{A})} denotes the action dimensionality of the corresponding environment.

W3: Computational overhead of eight networks and slower inference than DSHARSA.

Although Skill-Flow learns eight networks across the two hierarchical levels, the Q and V networks are only required during training and are not used during inference. As reported in Table 6, Skill-Flow achieves inference times (within 1.18×) comparable to those of DSHARSA. However, rejection-sampling-based policy extraction introduces an additional hyperparameter, namely the number of candidate samples, which governs the trade-off between policy performance and inference time. As the dimensionality of the search space increases, a small sampling budget may compromise policy performance, necessitating a substantially larger number of samples to identify high-value candidates, thereby increasing the computational cost. DSHARSA uses 32 candidate samples, which results in slightly lower inference times but comes at the expense of substantially lower policy performance across several challenging environments, as demonstrated in our experimental results. In contrast, Skill-Flow eliminates the need for this additional hyperparameter thereby eliminating this trade-off.

It is also worth noting that we consider the offline RL setting, where policies are learned from fixed datasets without requiring additional interactions with the environment as in online RL, thereby eliminating the substantial training overhead associated with traditional online RL. Although learning separate networks at each hierarchical level introduces additional computational costs during training, this computation is incurred offline and does not impose additional computational requirements during policy deployment. We believe that this additional training cost is justified by the substantial performance improvements achieved by Skill-Flow across challenging long-horizon tasks.

Additionally, we provide the training times in Table 6. Skill-Flow requires at most 1.58× the training time of DSHARSA and 1.25× the training time of HFQL across all considered environments. We argue that this additional offline training cost is well justified by the substantial performance improvements achieved by Skill-Flow while maintaining comparable inference times. Finally, the overall training time also depends on the evaluation frequency and can be further reduced by performing evaluations less frequently during training. In the table below, we compare the training times of DSHARSA and Skill-Flow under different evaluation intervals (measured in training iterations). All training times are reported in hours. As the evaluation frequency decreases (from 10,000 to 100,000 iterations), the training time decreases, with Skill-Flow achieving lower training time than DSHARSA when the evaluation interval is increased to 100,000 training iterations.

EnvironmentDSHARSA (10,000)Skill-Flow (10,000)Skill-Flow (50,000)Skill-Flow (100,000)
pointmaze-medium-navigate3.233.913.433.02

Thanks for the responses and clarifications. I highly recommend further revising the paper in future iterations to improve the clarity of the proposed methods. Regarding the computational overhead, I appreciate the additional explanations and specific measurements of training and inference times. However, I do not believe that the offline setting alone sufficiently justifies the additional computational cost or addresses concerns about fair comparisons with the baselines. Therefore, it would be useful to provide readers with some indication of how Skill-Flow performs under a matched training compute budget. Furthermore, while reading through the revised version of the paper, I noticed that several important experimental hyperparameters, particularly the discount factors and goal-sampling distributions, differ substantially across some methods. I believe that these choices can materially affect performance and make it difficult to isolate the contributions of Skill-Flow. I hope the authors can clarify the rationale behind these differences in the experimental settings.

Thank you for your comments. Please find our response below.

Training compute. We thank the reviewer for this observation. Our primary objective is to improve policy performance on challenging long-horizon offline GCRL tasks rather than minimize training computation. While Skill-Flow incurs moderately higher training costs than HFQL and DSHARSA, these costs are accompanied by substantial improvements in task success, particularly on challenging long-horizon environments where existing methods often achieve near-zero success rates. Importantly, all methods are trained for the same number of training iterations, ensuring a consistent optimization-step budget and enabling a fair comparison of algorithmic effectiveness under a common training protocol.

Hyperparameter configurations. We thank the reviewer for raising this concern. The discount factors and goal-sampling configurations for the baselines were adopted from their respective original papers and official implementations, rather than independently tuned or selected to favor Skill-Flow. These configurations reflect the algorithm-specific design choices recommended by the original authors. To ensure transparency and reproducibility, we have explicitly reported these settings for all eight baselines in Tables 10 and 11 of the revised supplementary material.

Importantly, we also conduct component-wise ablations in Table 9, where the goal-sampling configuration, discount factor, hierarchical architecture, and other hyperparameters of Skill-Flow are held fixed while individual algorithmic components are modified. These controlled comparisons directly assess the contributions of N-step IQL-D and support-constrained policy extraction at both hierarchical levels, independently of differences in hyperparameters across baselines.

Furthermore, to investigate the influence of goal sampling, we conducted additional experiments with DSHARSA using the same policy goal-sampling configuration as Skill-Flow on two challenging giant-stitch environments, while retaining its original value goal-sampling configuration and all other hyperparameters.

EnvironmentDSHARSA (Original)DSHARSA (Modified)Skill-Flow
antmaze-giant-stitch0 ± 07 ± 368 ± 4
humanoidmaze-giant-stitch0 ± 00 ± 091 ± 7

These results provide further evidence that differences in goal sampling alone cannot explain the substantial performance improvements of our method.

Please let us know your further concerns and we will address them. Alternatively, we will greatly appreciate if you could revise your score if your concerns have been addressed.

Reviewer_d9gPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Skill-Flow, a hierarchical method for OGCRL. In it, a high-level policy selects latent skills, while a low-level policy produces skill-conditioned actions. The policies use behavior-cloning flow models with reflected noise generators so that high-value outputs are favored while remaining within the learned models' support. The authors test their approach using 31 OGBench environments with different horizons and dataset qualities. The paper reports improvements over its reproduced baselines and includes supporting evidence.

Critical strengths
  • The method combines complementary mechanisms in a useful/intuitive way. Specifically, the temporal abstraction reduces the decision horizon, while constrained policy extraction operates at both the skill and action levels.

  • The analyses provide are quite useful, especially the sensitivity to skill horizon and integration steps and inference-time measurements which help characterize the method's robustness and computational cost.

Critical weaknesses
  • Something seems off with several of the main empirical results. Table 7 reports HIQL success rates of 22% on antmaze-large-navigate and 25% on antmaze-medium-stitch, compared with 91% and 94% in OGBench [1] (notably these values exceed Skill-Flow's 85% and 92%). But the authors' QRL results in Table 7 closely match the QRL values reported in OGBench so it's hard to reconcile exactly what's going on.

  • The benefit of delayed rewards isn't differentiated from standard horizon reduction. More specifically, the rationale that a skill can only be evaluated after execution applies equally to a standard NN-step return. On fixed-length nonterminal segments with constant −1 rewards, replacing accumulated rewards with an endpoint reward changes the reward scale, while using γ\gamma rather than γN\gamma^N lengthens the effective discount horizon. There is an ablation but it compares against one-step IQL, and the didactic example assumes fixed relative noise on optimal values in a deterministic chain. So neither really isolates delaying rewards as the source of improvement.

[1] Park et al. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR 2025.

Questions for authors
  1. What explains the large discrepancies between the reproduced HIQL results and those in the original OGBench paper on antmaze-large-navigate and antmaze-medium-stitch?

  2. Did you compare NN-step backup with accumulated rewards and γN\gamma^N, keeping everything else (i.e., the hierarchy, skill representation, and policy extractors) unchanged? Also can you clarify whether the experiments use the endpoint reward in Equation 4 or the current-state reward in Algorithm 1.

  3. Does Theorem 1 assume that the learned behavior-cloning models recover the behavior policies' support? Remarks 2 and 3 establish containment within learned-model support, while the theorem states containment within behavior-policy support.

  4. Most components appear in prior work. For example, latent subgoals learned through the value function in HIQL [1], hierarchical flow policies with multi-step values in SHARSA [2], and noise-steered extraction in ReFORM [3]. Can you clarify what is new beyond combining these components, and what the flow-based design contributes compared with a unimodal extractor using the same critics?

[1] Park et al. HIQL: Offline Goal-Conditioned RL with Latent States as Actions. NeurIPS 2023.

[2] Park et al. Horizon Reduction Makes RL Scalable. NeurIPS 2025.

[3] Zhang et al. ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation. ICLR 2026.

5 comments
AuthorsOpenReview ↗

Response to Questions

Response to reviewer d9gP:

We thank the reviewer for acknowledging the intuitive design of our method, which combines temporal abstraction with constrained policy extraction at both levels, and for recognizing the usefulness of our analyses of skill horizon, integration steps, and inference time in characterizing the method's robustness and computational cost.

We start by responding to the posed questions.

Details

Q1: Comparison against standard NN-step backup with accumulated rewards and γN\gamma^N? Endpoint reward in Equation (4) or current-state reward in Algorithm 1?

We thank the reviewer for this question. We have already conducted the requested comparison in Table 8 (column 3) of the supplementary material, under the ablation No N-step IQL-D. Specifically, we replace the proposed high-level N-step IQL-D update with a standard N-step Bellman backup that accumulates per-step rewards and uses (\gamma^N) for discounting, while keeping the hierarchical architecture, skill encoder, and both policy extractors unchanged. The results show that removing N-step IQL-D substantially degrades performance, with success rates dropping to zero on all three giant-stitch tasks, compared with 68–91% for Skill-Flow. This controlled comparison demonstrates the benefit of delayed-reward value learning beyond standard N-step temporal abstraction. We have also clarified this distinction in Section 5.3 (Q4, Ablation Results) of the revised manuscript. We use the endpoint reward as in (4); we have fixed the notational inconsistency in algorithm 1 in the revised manuscript.


Q2: Does Theorem 1 assume exact recovery of behavior-policy support by the BC models?

We thank the reviewer for raising this question. The flow-matching objectives in Equations (6) and (12) are formulated to recover the underlying behavior-policy distributions at the high and low levels, respectively. In the theoretical setting of exact distribution recovery, the BC policies and the corresponding behavior policies have identical supports. We have explicitly clarified this relationship immediately after Equations (6) and (12) in the revised manuscript. Furthermore, following Theorems 1 and 2 of ReFORM (Zhang et al., 2026), Remarks 2 and 3 establish that reflected-flow policy extraction preserves support containment within the respective BC policies at both hierarchical levels. We have revised both remarks to explicitly establish the connection between BC support containment and behavior-policy support containment. Theorem 1 follows directly from these results and has been revised to explicitly specify the theoretical setting under consideration.

Q3: Novelty beyond combining HIQL, SHARSA, and ReFORM? Benefits of flow-based over unimodal policy extraction?

We thank the reviewer for this question. We build on HIQL, hierarchical flow policies, and ReFORM, and do not claim these individual components as new. Our contribution is to identify and address two coupled challenges in hierarchical OGCRL: high-level value learning over long horizons and policy-extraction errors at both hierarchical levels. Specifically, we introduce N-step IQL-D for high-level value learning. As illustrated by our first didactic example, delayed high-level rewards reduce the impact of value-estimation errors on skill selection, improving decision making over long horizons. We further develop a hierarchy that constrains both skill selection and action selection to their respective BC supports. The motivating example in Figure 2 illustrates why support constraints are required at both levels to avoid errors caused by OOD skills at the high level and OOD actions during skill execution, while the ablations in Table 8 quantify the contributions of the high-level update and support constraints at each level.

References

[1] Park et al. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR, 2025.

[2] Park et al. HIQL: Offline Goal-Conditioned RL with Latent States as Actions. NeurIPS, 2023.

[3] Park et al. Horizon Reduction Makes RL Scalable. NeurIPS, 2025.

[4] Zhang et al. ReFORM: Reflected Flows for On-Support Offline RL via Noise Manipulation. ICLR, 2026.

AuthorsOpenReview ↗

Response to remaining Questions and Weaknesses

Here we add response to the remaining questions and the weaknesses.

Details:

Q4: What explains the discrepancies between the reproduced HIQL and OGBench results?

We thank the reviewer for bringing this discrepancy to our attention. The original HIQL paper (Park et al., 2023) predates OGBench (Park et al., 2025) and does not provide environment-specific hyperparameter configurations for the environments considered in our work. Hence, in our initial experiments, we adopted the HIQL implementation from the more recent paper Horizon Reduction Makes RL Scalable (Park et al., 2025), by the same lead authors as HIQL and OGBench.

Upon further investigation, we identified several differences between the HIQL implementations provided in the Horizon Reduction and OGBench repositories. Specifically, the default hyperparameters in the Horizon Reduction repository are different from the OGBench repository for the considered environments. We summarize the key differences below.

HyperparameterHorizon ReductionOGBench
Actor and value hidden dimensions(1024, 1024, 1024, 1024)(512, 512, 512)
Discount factor0.9990.99 / 0.995
Representation dimension12810
Actor trajectory goal probability0.51.0 (navigate), 0.5 (stitch), 0.0 (explore)
Actor random goal probability0.50.0 (navigate), 0.5 (stitch), 1.0 (explore)
Actor geometric samplingTrueFalse
Value geometric samplingFalseTrue

We believe that these implementation and hyperparameter differences contribute to the discrepancies observed in our initial experiments. Hence, we have rerun the HIQL experiments using the OGBench implementation and its corresponding hyperparameter configurations. Notably, of the 31 environments considered, changes were only observed in only six: pointmaze-medium-stitch and antmaze-{large-navigate, giant-navigate, medium-stitch, large-stitch, medium-explore}. The updated results, now consistent with those reported in OGBench, are highlighted in red in Table 8 of the revised supplementary material. We have also updated Figures 4 and 5 accordingly. Importantly, these revisions do not alter our overall findings or conclusions. Skill-Flow continues to achieve the best overall performance, with its performance advantage over the baselines increasing with the task horizon and dataset complexity

Weaknesses

W1: Discrepancies in reproduced HIQL results despite consistent QRL results with OGBench.

As clarified in our response to Q1, the discrepancies arise from differences in the HIQL implementations and hyperparameter configurations provided in the Horizon Reduction and OGBench repositories. We have addressed this concern by rerunning the experiments using the OGBench implementation and updating Table 8 and Figures 4 and 5 accordingly. Importantly, the revised results do not alter the overall findings or conclusions of our work.

W2: Insufficient isolation of delayed rewards from standard horizon reduction, reward scaling, and discounting.

As clarified in our response to Q1, the existing ablation compares N-step IQL-D against standard N-step IQL, not one-step IQL, while keeping all other components unchanged. We have clarified this distinction in Section 5.3 of the revised manuscript.

References

[1] Park et al. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR, 2025.

[2] Park et al. HIQL: Offline Goal-Conditioned RL with Latent States as Actions. NeurIPS, 2023.

[3] Park et al. Horizon Reduction Makes RL Scalable. NeurIPS, 2025.

[4] Zhang et al. ReFORM: Reflected Flows for On-Support Offline RL via Noise Manipulation. ICLR, 2026.

Thanks for the response and for rerunning HIQL. For algorithm 1 the reward has been clarified as has the assumption for Theorem 1.

I do, however, have remaining concerns. Specifically, HIQL went from 22% to OGBench-level results just based on a config/implementation change. This is particularly concerning because the experimental settings don't seem very consistent. In Table 10, HFQL, SHARSA, and DSHARSA sample policy goals only from the same trajectory on all maze datasets, while Skill-Flow, HIQL, GCIQL, and QRL use random goals on stitch and explore. SHARSA and DSHARSA match or beat Skill-Flow on antmaze-navigate but score near zero on the pointmaze and antmaze stitch and explore tasks, so it seems quite possible that the robustness to dataset quality may come from goal sampling. Also, Table 11 lists N=50 for these baselines on pointmaze and antmaze, but several of the HFQL and DSHARSA results at N=25 in Table 4 exactly match the corresponding results in Table 8.

Finally, the ablation still changes both reward aggregation and the bootstrap discount together. Specifically, your ablation uses γN\gamma^N per skill step, an effective horizon ~N times shorter, which alone could explain the collapse on giant-stitch.

Reviewer_BwUWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Skill-Flow, which addresses offline goal-conditioned reinforcement learning with hierarchical policies.

Critical strengths
  1. This paper provides both theoretical analysis and experimental results, which are used to strengthen the claim of the paper.

  2. The proposed method can outperform the baselines, indicating the effectiveness of the method.

Critical weaknesses
  1. Lack of novelty. Offline goal-conditioned RL has been widely studied, and this paper just modeled GCRL with a hierarchical framework, which does not show enough new gains or essential progress in this field.

  2. The hypothesis is too strong. In the motivating example, the authors use a 1-D chain example, which is not general in common MDP settings.

  3. The experiments are mainly conducted on OGBench, which is simple. The authors may consider adding more experiments to support their claims.

Questions for authors
  1. In more general MDP cases, does the motivating example still hold?

  2. In N-step IQL-D, how is NN decided? As a hyper-parameter here, do the authors make any ablations on this factor?

3 comments
AuthorsOpenReview ↗

Response to Weaknesses

Response to reviewer BwUW:

We thank the reviewer for recognizing the importance of our theoretical analysis and experimental evaluations in supporting the key claims of our work. We also appreciate the reviewer acknowledging the effectiveness of our proposed method in outperforming existing baselines. These results highlight the benefits of our hierarchical learning framework in addressing the challenges of long-horizon offline goal-conditioned reinforcement learning across tasks with varying horizon lengths and dataset complexities.

Weaknesses:

W1: The hypothesis is too strong. In the motivating example, the authors use a 1-D chain example, which is not general in common MDP settings.

We thank the reviewer for raising this concern. The 1-D chain example in Section 4.1.1 is intended as a didactic example to motivate our proposed high-level value learning formulation, rather than to establish a general theoretical result for arbitrary MDPs. The simplified setting enables us to analytically characterize the impact of value-estimation errors on policy selection under different reward formulations and temporal abstraction lengths. While extending this theoretical analysis to general MDPs is an interesting direction for future work, we empirically validate the underlying motivation through controlled ablations across OGBench environments with substantially more complex dynamics, higher-dimensional state and action spaces, and varying task horizons highlighting the merit and validity of our proposed value learning approach.

W2: The experiments are mainly conducted on OGBench, which is simple. The authors may consider adding more experiments to support their claims.

We respectfully disagree with the characterization of OGBench as a simple benchmark. OGBench [1] was specifically designed to evaluate offline GCRL algorithms across challenging tasks involving long-horizon reasoning, trajectory stitching, high-dimensional continuous control, and combinatorial generalization. In our work, we consider 31 environment–dataset configurations spanning six environment families, covering several dimensions of complexity:

  • Robotic systems and dynamics: Point-mass agents, quadrupedal robots, humanoid robots, and robotic manipulators, with substantially different dynamics and state and action dimensionalities.
  • Task horizons: Medium, large, and giant mazes, with episode horizons extending up to 4,000 time steps.
  • Dataset complexity: Navigate, stitch, explore, play, and noisy datasets, requiring policies to learn from expert trajectories, fragmented trajectories, random exploration, and noisy demonstrations.
  • Task diversity: Navigation, legged locomotion, multi-object manipulation, articulated object manipulation, and combinatorial puzzle solving. For instance, the puzzle-4x6 environment alone contains over 16 million possible button configurations.

Furthermore, OGBench has become a standard benchmark for evaluating offline GCRL algorithms, as evidenced by several recent works, including SHARSA, DSHARSA [2], CGCIVL [3], and TMD [5]. Our evaluation also includes established hierarchical OGCRL methods such as HIQL [4]. Hence, we believe that our extensive evaluation across 31 environment–dataset configurations against eight baselines provides comprehensive empirical evidence supporting the effectiveness of our method across diverse and challenging offline GCRL tasks.

References

[1] Park et al. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR, 2025.

[2] Park et al. Horizon Reduction Makes RL Scalable. NeurIPS, 2025.

[3] Ke et al. Conservative Offline Goal-Conditioned Implicit V-Learning. ICML, 2025.

[4] Park et al. HIQL: Offline Goal-Conditioned RL with Latent States as Actions. NeurIPS, 2023.

[5] Myers et al. Offline Goal-Conditioned Reinforcement Learning with Quasimetric Representations. 2025.

AuthorsOpenReview ↗

Response to Weaknesses 2 and Questions

Here, we respond to the remaining weaknesses and questions identified by the reviewer.

Weaknesses:

W3: Lack of novelty. Offline goal-conditioned RL has been widely studied, and this paper just modeled GCRL with a hierarchical framework, which does not show enough new gains or essential progress in this field.

We respectfully disagree that our contribution is limited to modeling offline GCRL within a hierarchical framework. While hierarchical RL has been extensively studied, our work identifies and addresses two fundamental challenges that remain unresolved in existing hierarchical OGCRL methods: (i) erroneous high-level value learning in long-horizon sparse-reward settings and (ii) OOD errors during policy extraction at both hierarchical levels.

First, unlike existing hierarchical approaches that primarily rely on temporal abstraction to mitigate the long-horizon problem, we introduce N-step IQL-D, a novel high-level value learning formulation with delayed rewards. Our theoretical analysis in Appendix B.1 motivates this formulation by demonstrating how delayed rewards can reduce the impact of value-estimation errors on high-level decision-making. Furthermore, our controlled ablation against standard N-step IQL demonstrates the benefit of our formulation beyond conventional horizon reduction.

Second, we identify how OOD errors can originate and propagate across hierarchical levels, a challenge that has received limited attention in existing hierarchical OGCRL literature. To address this, we develop a hierarchical policy learning framework that incorporates expressive flow-based policies with support-constrained policy extraction at both the skill and action levels. Our motivating example in Figure 2 and the corresponding ablation studies demonstrate the importance of enforcing support constraints at both levels.

Finally, the empirical gains are substantial. Our method outperforms eight competitive baselines in overall performance across 31 OGBench environments, with its advantage becoming particularly pronounced as task horizons increase. For instance, on the three giant-stitch environments, Skill-Flow achieves success rates of 68–91%, whereas most baselines achieve near-zero success rates.

Therefore, our contribution goes beyond introducing hierarchy into offline GCRL. It advances both high-level value learning and hierarchical policy extraction, supported by theoretical analysis, controlled ablations, and extensive empirical evaluations.

Details:

Q1: In more general MDP cases, does the motivating example still hold?

As discussed in our response to W1, the underlying intuition extends beyond the 1-D chain setting under the same assumptions on value-estimation noise. While the simplified setting enables analytical characterization, our controlled ablations across OGBench environments empirically validate the underlying motivation.

Q2: In N-step IQL-D, how is NN decided? As a hyperparameter, do the authors perform any ablations on this factor?

We thank the reviewer for this question. The skill length NN is a design parameter that introduces a trade-off between the effective high-level planning horizon and the difficulty of learning and executing longer skills at the low level. Increasing NN reduces the effective high-level planning horizon but can increase value-estimation and policy-selection errors at the low level, as illustrated by our motivating example in Figure 1. Hence, NN is chosen to balance this trade-off.

To empirically evaluate the sensitivity to this hyperparameter, we have already conducted a comprehensive analysis in Appendix F.3 (Table 4) of the supplementary material. Specifically, we compare Skill-Flow, HFQL, and DSHARSA for five different values of NN (15, 20, 25, 30, and 35) across three environments: pointmaze-giant-navigate, antmaze-giant-stitch, and antmaze-medium-navigate. These environments capture three important dimensions of complexity:

  • Agent dynamics and dimensionality: AntMaze has a substantially higher-dimensional observation space than PointMaze (29 dimensions versus 2), making skill representation and low-level execution more challenging.
  • Task horizon: Giant mazes have considerably longer planning horizons than medium mazes.
  • Dataset complexity: Navigate and stitch datasets differ in the availability of complete goal-reaching trajectories, with the latter requiring trajectory composition.

Across the considered values of NN, Skill-Flow consistently achieves success rates of 94–95%, 66–72%, and 92–96%, respectively. Additionally, the skill completion rates remain between 94% and 99% across all three environments and all five abstraction lengths, demonstrating that the low-level policy can reliably execute the learned skills. These results establish the robustness of Skill-Flow to the choice of NN within the considered range.

AuthorsOpenReview ↗

Response Follow up Request

Dear Reviewer,

As the rebuttal period is coming to an end, we kindly ask you to review our response. If you have any remaining concerns, we would be happy to provide further clarification. If our response has addressed your concerns, please consider raising your score.

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.