- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses long-horizon offline goal-conditioned reinforcement learning through hierarchical flow policies, bounded-noise guidance, and a delayed-reward high-level IQL update. Experiments on OGBench demonstrate strong performance across navigation and manipulation tasks.
Critical strengths
-
The method demonstrates strong performance across challenging long-horizon navigation and manipulation tasks.
-
The ablations and sensitivity analyses provide useful evidence for the main design choices and the method’s robustness.
Critical weaknesses
-
The high-level update changes both reward aggregation and effective discounting, making it difficult to attribute the improvement specifically to delayed rewards.
-
The ablation replaces the entire multi-step update with one-step IQL. A comparison with standard multi-step learning under the same hierarchy is needed to isolate the proposed contribution.
-
Algorithm 1 differs from the main text in both the skill encoding and high-level reward, leaving unclear which implementation produced the reported results.
-
Theorem 1 claims containment in the original behavior-policy support, whereas the construction only establishes containment in the learned BC support. I suggest clarifying the assumptions needed to connect these supports and justify the claimed guarantee.
Questions for authors
-
Which skill encoding and high-level reward were used in the experiments?
-
Is Theorem 1 intended to assume exact behavior cloning or support consistency? If not, what justifies transferring support containment from the learned BC policy to the original behavior policy?
2 comments
Response follow up request
Dear Reviewer,
As the rebuttal period is coming to an end, we kindly ask you to review our response. If you have any remaining concerns, we would be happy to provide further clarification. If our response has addressed your concerns, please consider raising your score.
Best, The Authors
Response
Response to reviewer ECwA:
We thank the reviewer for recognizing the strong performance of our method across challenging long-horizon navigation and manipulation tasks. We also appreciate the reviewer acknowledging the usefulness of our ablation studies and sensitivity analyses in validating the key design choices and demonstrating the robustness of our method.
Summary
In brief, we address the following:
Weaknesses
In our proposed delayed-reward formulation, rewards are assigned upon the completion of each skill rather than at every primitive time step. Since no intermediate rewards are assigned during skill execution, we perform discounting at the skill level. The change in effective discounting is therefore a direct consequence of defining the reward and return at the skill level, rather than an independently introduced modification.
We thank the reviewer for raising this concern. We have already conducted the requested comparison, with the results reported in Section 5.3 (Q4, Figure 6) and Table 8 (column 3) of the supplementary material, under the ablation No N-step IQL-D. We have revised the description of this ablation in Section 5.3 to eliminate any ambiguity. Specifically, we replace the proposed high-level N-step IQL-D update with a standard N-step Bellman backup that accumulates per-step rewards and uses γN for discounting, while keeping the hierarchical architecture, skill encoder, and both policy extractors unchanged. The results show that removing N-step IQL-D substantially degrades performance, with success rates dropping to zero on all three giant-stitch tasks, compared with 68–91% for Skill-Flow. This controlled comparison demonstrates the effectiveness of our proposed high-level value learning formulation over standard N-step value learning.
We thank the reviewer for pointing out the typographical errors in Algorithm 1. Throughout the paper, the skill encoding is defined as zk=ϕ(sk,sk+N), and the high-level reward is defined as the endpoint reward r(sk+N,g) in Equation (4). We have corrected the corresponding typographical errors in lines 6 and 10 of Algorithm 1 in the revised supplementary material.
We thank the reviewer for raising this concern. The flow-matching objectives in Equations (6) and (12) are formulated to recover the underlying behavior-policy distributions at the high and low levels, respectively. In the theoretical setting of exact distribution recovery, the BC policies and their corresponding behavior policies have identical supports. We have clarified this relationship immediately after Equations (6) and (12) in the revised manuscript. Furthermore, following Theorems 1 and 2 of ReFORM (Zhang et al., 2026), Remarks 2 and 3 establish containment within the respective BC policy supports. We have revised both remarks to explicitly show how this containment extends to the corresponding behavior-policy supports. Theorem 1 follows directly from these results, and we have revised its statement to explicitly clarify the theoretical setting under consideration.
Questions
As clarified in our response to W3, we use the skill encoding zk=ϕ(sk,sk+N) and the high-level endpoint reward r(sk+N,g) in all our experiments. We have corrected the corresponding typographical errors in Algorithm 1.
As clarified in our response to W4, Theorem 1 considers the theoretical setting of exact behavior distribution recovery, in which the supports of the BC policies coincide with those of the corresponding behavior policies. We have revised Remarks 2 and 3 and Theorem 1 to make this relationship and the underlying theoretical setting explicit.