acceptodds
Under review as a conference paper at ICLR 2027

HorizonFlow: Variable-Length Planning for Offline Goal-Conditioned RL

Abstract

Recent advances in generative planning have made trajectory inpainting a promising approach to offline goal-conditioned reinforcement learning. However, these methods typically specify the planning horizon before generating plan content, even though the appropriate horizon depends on the route itself. A horizon that is too short can force infeasible transitions, whereas one that is too long can introduce redundant motion. We introduce HorizonFlow, a hierarchical planner that treats plan length as an output of generation rather than a prescribed input. Its subgoal route planner guides its action-prefix controller through a sequence of latent subgoals. Both components combine insertion-based generation with flow matching to jointly generate continuous plan content and length, using the partially generated plan to guide token insertion. HorizonFlow reuses the resulting length information to select candidates and steer generation toward shorter plans without a separate learned value model. Across Maze2D, Multi2D, and OGBench navigation and visual manipulation benchmarks, HorizonFlow achieves the highest average performance among the compared methods.

Primary area:
reinforcement learning
Keywords:
Offline Goal-Conditioned Reinforcement Learning, Generative Trajectory Planning, Variable-Length Generation, Flow Matching, Hierarchical Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CAgsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HorizonFlow, a hierarchical generative planner for offline goal-conditioned RL. The key idea is to generate the plan length together with the plan content, instead of fixing a horizon before generation. It combines insertion-based generation (per-gap count heads, Bernoulli + zero-truncated Poisson) with flow matching on continuous tokens, and each token carries a continuous order coordinate. A route planner generates a variable number of latent subgoals, and a prefix controller generates a variable-length action prefix toward the first subgoal. At test time the method samples KK candidates (16 routes, 4 prefixes), picks the one with the fewest tokens, and optionally uses FK resampling with the expected remaining count to push generation toward shorter plans. Only the first action is executed, then it replans. Experiments cover Maze2D/Multi2D, 9 OGBench navigation tasks and 4 OGBench visual manipulation tasks. The method reports the best average in all three groups, with very large gains on large/giant mazes. A horizon diagnostic on Maze2D-Large and ablations on candidate count, FK, replan period, executed prefix and inference cost are given.

Critical strengths
  1. The idea is simple and well motivated. Fig. 4(a-c) shows clearly that no single fixed horizon works for both near and far goals, and the joint model keeps $\sim$100% reach rate at every distance even with one candidate. The selection signal (token count) is a free by-product of generation, so no value function is needed. This turns the horizon from a hyperparameter into a model output, which is a real pain point for inpainting planners. The diagnostic design (binning start-goal pairs by nominal steps, same backbone, data and controller for both models) is clean and convincing on this point.

  2. The OGBench navigation results are strong and carefully done: 94.2% average vs 75.4% for the best prior method, 8 seeds, the standard OGBench protocol (5 tasks ×\times 50 episodes), small std, and humanoidmaze-giant at 95.5% vs 35%. The analysis section is also honest: it reports the K=1K=1 number, the inference cost per step, and the selection risk. If the numbers hold, a 20-60 point jump on giant mazes is a significant result for this field.

Critical weaknesses
  1. The claim "joint generation fixes the two-sided horizon mismatch" is not what the evidence shows. What the evidence shows is "sample many candidates and pick the shortest" fixes it. In Fig. 5(a), KRP=1K_{\mathrm{RP}}=1 gives 56.2%, which is below HIQL (65.0), SAW (75.4) and CTA (64.3); the jump to 94.3% comes from best-of-16 min-count selection. In Fig. 4, the joint model at K=1K=1 scores lower than the oracle fixed horizon in every bin (242/207/150/103 vs 274/248/199/148) and takes 1.3-1.6$\times$ more steps, and even one global H=384H=384 beats joint K=1K=1 at N∗=256N^*=256 and 384384. Appendix B.1 says the same on OGBench: a single joint RP sample gets 41.9%/54.4% on antmaze-giant/humanoidmaze-giant vs 86.9%/94.7% with selection. So joint generation alone is biased long, and the "too long" side is solved by search with a count prior, not by the model. This matter because the fixed-length baselines never get the same candidate budget: the fixed-HH model in Fig. 4 is one sample, the OGBench planning baselines are mostly one-shot, and the fixed-length control in Appendix B.1 runs only at K=1K=1 on two environments. A fair control is a fixed-length planner with the same KK and a cheap ranking signal (XY path length in Fig. 4; HIQL value or the VHD predictor on OGBench). Appendix A.4 does something like this for Diffuser on Maze2D only. Without this control I cannot tell how much of the gain is the modeling idea and how much is test-time search. I think the paper should re-frame the contribution around count-based selection/steering and add the matched-budget control. The method can still be good, but the story needs to match the data.

  2. On OGBench, where the biggest claims are made, the closest competitors are missing and the task subset is narrow. (a) VH-Diffuser is the only other method that adapts the horizon per start-goal pair, and the intro is written against it, but it appears only on Maze2D/Multi2D. VHD itself reports OGBench antmaze and cube results, so the comparison is possible. (b) The only planning baselines in Table 1(b) are Diffuser/HD/DF reproduced by the authors (planning in full observation space with a shared controller), and they collapse to 0-7% on antmaze-giant and humanoidmaze-giant. CompDiffuser and HDFlow are cited as the representative fixed-horizon / hierarchical flow planners but not compared; CompDiffuser reports high success on OGBench maze stitch datasets. (c) The main text never says Table 1(b) uses the navigate datasets; the stitch datasets (the most relevant setting for a planner that composes routes) are absent; visual manipulation reports 4 of 9 tasks (no cube-quadruple, no puzzle) and no state-based manipulation. Among the 4, the method loses cube-triple (14.4 vs 21) and cube-double is 46.8±11.346.8 \pm 11.3 over 4 seeds. So "highest average in all three groups" is on a selected subset. I do not think this is on purpose, but the claim should be scoped or the missing comparisons added.

Questions for authors
  1. Humanoidmaze: large is 80.2 but giant is 95.5. Every baseline is monotone medium > large > giant. Is this caused by the recipe (stride 32 / Kdiv=2K_{\mathrm{div}}=2 for medium and large, 64 / 4 for giant)? Did you try the giant setting on large? If success depends this much on the stride rule, please say so in the main text.
  2. Matched-budget control for weakness 1: can you run (a) the Fig. 4 fixed-HH model with K=64K=64 candidates selected by XY path length, and (b) on OGBench the fixed-length RP from Appendix B.1 with K=16K=16 candidates ranked by a HIQL value or by the length predictor? This is the experiment that would decide weakness 1 for me.
  3. For a fixed (s,g)(s,g), what is the spread of generated counts across the 16 route candidates? Appendix B says the count tracks distance with ρ=0.95\rho=0.95, so why does K=1K=1 fail so badly (41.9% on antmaze-giant)? Are the K=1K=1 failures mostly too-long routes, or infeasible routes?
  4. Order coordinate: when a new token is born in gap gg, is its rkr_k initialized from Gaussian noise, or placed between the two neighbors? If from noise, the gap structure is only nominal and only the total count carries information. Fig. 3 at σ=0.6\sigma=0.6-0.90.9 shows tangled orderings. Please clarify, and report how often the final sorted order differs from the birth gap order.
  5. PC level: only the first action is executed and KPCK_{\mathrm{PC}} adds about 2 points (92.3 →\to 94.6). Appendix B.2 compares against a "full-horizon" fixed-length PC that always generates the full capacity, which looks like a weak control. What about a short fixed-length PC (e.g. 4 or 8 actions) with the same RP? If it matches, the variable-length PC is not needed.
  6. The encoder collapse claim in Sec. 4.2 has no number behind it. Please add α=0\alpha=0 and FM-gradient-unblocked ablations.
  7. Baseline provenance should be in the main text: which numbers are copied and which are your reproductions. Also Table 1(a) mixes planning-seed SE×n\mathrm{SE}\times\sqrt{n} (HD, HDMI, SSD) with training-seed std (DF, VHD, Ours) under one "mean ±\pm std" label.
5 comments

We thank the reviewer for the thorough review and for proposing concrete experiments. The matched-budget control in particular sharpened what the paper claims, and we ran it as requested.

W1 & Q2. Do the gains come from joint generation or from test-time search?

Selection can raise success only because the candidates are jointly generated. A fixed-length planner does not know the right horizon, and trying many horizons does not help, because it cannot tell at test time which plans are feasible ((a), (b)). HorizonFlow keeps inserting subgoals until the plan is complete, so most of its candidates reach the goal (99–100% in the Maze2D diagnostic), and this is what lets the shortest one be chosen. The revised introduction now states this division explicitly. Joint generation removes the too-short side, and selection over the generated candidates trims the too-long side. The same preference could also be built into sampling. With a learned value over plans, plans of higher value, that is shorter ones, could be made more likely to be drawn, which would raise single-sample success with no test-time search. That gain, too, would come from joint generation, because the length is decided during generation.

(a) Fig. 4 (Maze2D-Large XY, K=64K = 64). HorizonFlow is highest in every bin and every seed, even against the fixed-length planner given N∗N^{\ast}. N∗N^{\ast} is a nominal horizon computed from the maze distance, not the shortest executable one, so the shortest generated plan can be shorter than N∗N^{\ast} and still reach the goal. Candidates are ranked by the HIQL value (Appendix A.5) or, as the reviewer suggested, by XY path length. With spread horizons, ranking by path length fails, because the shortest candidate is always the straight line from start to goal, which passes through the walls.

Normalized score, mean ± SD over 3 seedsN∗=64N^{\ast} = 64128192256320384
HorizonFlow, K=64K = 64 (min count, FK)275.0 ± 0.1254.4 ± 0.3228.5 ± 0.7204.5 ± 0.4181.4 ± 1.5156.7 ± 0.2
Fixed H=N∗H = N^{\ast} (privileged), K=64K = 64 + HIQL273.1 ± 0.1249.1 ± 1.6225.0 ± 0.1200.4 ± 1.3177.8 ± 0.0155.1 ± 0.1
Fixed H=N∗H = N^{\ast} (privileged), K=64K = 64 + XY path length273.1 ± 0.0247.4 ± 3.0225.0 ± 0.1198.4 ± 1.2173.6 ± 2.1152.0 ± 2.6
Best fixed HH per bin (in hindsight, H∈{64,128,256,384}H \in \{64, 128, 256, 384\}), K=64K = 64 + HIQL274.3 ± 0.1246.6 ± 1.4206.4 ± 0.2200.1 ± 1.1156.1 ± 0.1149.8 ± 4.1
Fixed HH spread over the candidates (log-spaced, 13 to 819 steps), K=64K = 64 + HIQL268.6 ± 0.3225.9 ± 1.0190.9 ± 4.1159.3 ± 1.4136.0 ± 1.1100.6 ± 4.1
Fixed HH spread over the candidates, K=64K = 64 + XY path length264.8 ± 0.0172.5 ± 0.081.2 ± 0.034.7 ± 0.028.9 ± 0.07.4 ± 0.0

(b) OGBench (K=16K = 16). With 16 candidates ranked by the length predictor or by the HIQL value, as requested, the fixed-length RP at the VH-Diffuser length stays far below HorizonFlow. The fixed-length RP is the same model trained without insertion. It shares HorizonFlow’s architecture, data, training budget, subgoal encoder and prefix controller, and differs only in that its length is set externally. The VH-Diffuser predictor overestimates the length on the giant mazes, and all 16 candidates share its length, so ranking cannot correct it. Rows other than the default run with RP FK off, since FK cannot act on candidates of equal length.

Pool (K = 16), ranking; success (%), mean ± SD over 3 seedsAntMaze-LargeAntMaze-GiantHumanoidMaze-LargeHumanoidMaze-Giant
HorizonFlow, count (default, for reference)93.1 ± 1.286.9 ± 2.679.2 ± 6.294.7 ± 0.5
HorizonFlow, HIQL value94.4 ± 2.881.1 ± 9.277.1 ± 4.993.9 ± 3.2
Fixed length at the VH-Diffuser length, predictor47.5 ± 7.215.2 ± 3.516.3 ± 5.25.3 ± 1.2
Fixed length at the VH-Diffuser length, HIQL value32.3 ± 9.312.0 ± 2.914.1 ± 2.42.1 ± 1.2

W2. Are the closest OGBench competitors and tasks covered?

(a) VH-Diffuser's OGBench results use a non-standard protocol. They cover only AntMaze-Medium with random start–goal pairs, and a cube environment in which the arm only has to reach a target position, without moving any cube, so they cannot be added to Table 1; Appendix B.1 is our matched comparison (41.9% / 54.4% vs 15.7% / 2.7%).

(b) The reproduced planners are verified JAX ports of the official code, and we tuned them carefully. For each environment we searched the training horizon, the network size, the replanning period, the low-level controller and the other inference settings (appendix, "Implementations and Reproduction Protocols"). Their drop grows with the route length. On AntMaze, where the observation is the same at every size, Diffuser falls from 77% (Medium) to 61% (Large) and 5% (Giant), and HD and DF fall in the same way, because one fixed-length plan has to cover ever longer routes (on two-dimensional PointMaze, Diffuser and HD stay above 90%). With a fixed length, our own system shows the same pattern (47.5% on AntMaze-Large, 15.2% on AntMaze-Giant; W1 (b)), whereas HorizonFlow, which generates the route length, stays at 86.5% on AntMaze-Giant. CompDiffuser addresses a different problem, the stitching of short trajectories, and is evaluated only on the stitch and explore datasets; setting the plan length is complementary (Section 2). HDFlow requires additional data that includes failure trajectories, as well as a pretrained DINOv2 encoder with a world model, so its numbers cannot be compared directly with ours; the appendix lists them for reference only. Even so, HorizonFlow, which uses neither, has a higher average success on the four visual tasks (55.0% vs 41.8%).

(c) The revision states that Table 1(b) uses the navigate datasets and names the evaluated tasks wherever we claim the highest average. For navigation and visual manipulation, we use the tasks of SAW [1] (Table 1), our most recent baseline, using its nine state navigation tasks and its four visual manipulation tasks. For manipulation we chose the image-based tasks because they are closest to a real robot setting, where the agent observes camera images rather than object states. The set was not chosen in our favor. It includes visual cube-triple, where HorizonFlow (14.4%) is below HIQL (21%) and SAW (20%), and we report it as is. Section 5 now states this choice and scopes the claim to the evaluated tasks.

Q1. Why is HumanoidMaze-Large below Giant?

Mostly the episode limit. In OGBench, maze size alone does not set task difficulty. HumanoidMaze-medium and -large allow 2,000 steps per episode but -giant allows 4,000 (every PointMaze and AntMaze size allows 1,000), so a giant episode has twice the time budget of a large one. For HorizonFlow, with the limit doubled to 4,000 steps (Giant's budget), the same checkpoints reach 98.4%, and 92% of the failures at 2,000 steps reach the goal before 4,000. Training Large with the Giant recipe (stride 64, KdivK_{\mathrm{div}} 4) gives the same success as the default recipe, so the stride rule is not the cause (table below). Section 5 now states this budget dependence, and Appendix F reports both runs.

HumanoidMaze-Large, mean ± SD over 3 seedsSuccess (%)
Default recipe, 2,000-step limit80.1 ± 2.6
Default recipe, 4,000-step limit98.4 ± 0.4
Giant recipe (stride 64, KdivK_{\mathrm{div}} 4), 2,000-step limit79.7 ± 3.6

Q3. How spread are counts within a pair, and why does K=1K = 1 fail?

Mostly too-long routes; infeasible ones are a small minority. The 16 candidates for a pair differ substantially in count (first table below), because the data contain trajectories of different routes and durations between the same endpoints, and the model generates that variety. The ρ=0.954\rho = 0.954 of Appendix A.5 does not describe a single draw. It is the rank correlation between the nominal steps N∗N^{\ast} and the minimum count among 64 candidates per pair, that is, the count after selection. A single draw, in contrast, is longer than this minimum in almost every problem, so most K=1K = 1 failures are timeouts while the agent is still approaching the goal (second table below); minimum-count selection largely removes them. Details are in new Appendix B.4.

Generated count of a pair's 16 candidates, mean ± SD over 3 seedsAntMaze-GiantHumanoidMaze-Giant
Mean candidate (tokens)47.4 ± 0.345.4 ± 0.7
Longest minus shortest (tokens)15.0 ± 0.115.4 ± 0.4
Longest minus shortest (environment steps)240 ± 2983 ± 28
Share of K=1K = 1 failures (%), pooled over 3 seedsAntMaze-GiantHumanoidMaze-Giant
Timeout while still approaching the goal7571
Stuck at a wall29
Fall (no motion)136
Moving away from the goal1015

Q4. Are newborn order coordinates drawn from noise, making the gap structure nominal?

From Gaussian noise (Appendix D.3). Each gap only proposes how many tokens to add; a newborn is not assigned to the gap that proposed it, and its position is the order coordinate that the flow generates. There is therefore no birth-gap order to compare the final order with. The orderings in Fig. 3 at σ=0.6\sigma = 0.6–0.90.9 are intermediate states that the flow is still refining.

The gap structure is still not nominal, because it is the supervision. The per-gap head predicts how many tokens are missing between each pair of adjacent tokens, which tells the encoder where the plan is incomplete. With a single head that predicts only the total count, the method fails in every environment (table below; new Appendix B.6). The predicted length becomes nearly the same for every start–goal pair, because the latent loses the temporal ordering that per-gap counts impose.

Success (%), mean ± SD over 3 seedsAntMaze-LargeAntMaze-GiantHumanoidMaze-LargeHumanoidMaze-Giant
HorizonFlow, per-gap insertion (Table 1, 8 seeds)93.3 ± 1.186.5 ± 2.980.2 ± 1.595.5 ± 1.3
Global remaining-count head0.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.0

Q5. Would a short fixed-length controller suffice?

A short fixed-length controller does not suffice (table below; Appendix B.2). We trained controllers that generate only the first H=8H = 8 or 4 actions toward the subgoal and ran them under the same RP with KPC=1K_{\mathrm{PC}} = 1. Although only the first action is executed, in our controller it belongs to a plan that reaches the subgoal, and the length of that plan has to vary. The RP is trained on subgoals one stride apart, but with model error its generated subgoals are not exactly one stride away, and the agent keeps approaching the same subgoal between route updates, so the number of actions still needed can be shorter or longer than any fixed length. Because the PC generates the length together with the actions, it can absorb this mismatch, which a fixed-length controller cannot.

Success (%), KPC=1K_{\mathrm{PC}} = 1, mean ± SD over 3 seedsAntMaze-LargeAntMaze-GiantHumanoidMaze-LargeHumanoidMaze-Giant
Fixed-length PC, H=8H = 869.3 ± 6.136.0 ± 6.231.7 ± 5.683.2 ± 3.2
Fixed-length PC, H=4H = 449.1 ± 8.512.0 ± 6.032.5 ± 5.173.1 ± 9.2
HorizonFlow PC (variable length)90.1 ± 2.683.2 ± 3.779.5 ± 2.493.1 ± 0.9

Q6. What supports the encoder-collapse claim?

Both ablations confirm the claim (table below; Section 4.2, new Appendix B.5). If the flow-matching gradient reaches the encoder, one way to lower that loss is to shrink the scale of the latents. The noise has a fixed scale, so as the latents shrink the regression target becomes easier to predict, and the loss falls without the flow fitting the data any better. The latents then contract, and success collapses in every environment; InfoNCE compares normalized vectors and cannot stop this. Without InfoNCE (α=0\alpha = 0), state latents keep their spread, but pixel latents collapse. We attribute this to the input. Neighboring frames differ in few pixels, so early in training the image encoder maps them to nearly the same latent, and the insertion loss alone does not pull them apart; once collapsed, a latent does not recover. InfoNCE provides the signal that separates them.

Success (%), mean ± SD over 3 seedsAntMaze-LargeAntMaze-GiantHumanoidMaze-LargeHumanoidMaze-Giantvisual cube-singlevisual cube-double
HorizonFlow (Table 1; 8 state / 4 visual seeds)93.3 ± 1.186.5 ± 2.980.2 ± 1.595.5 ± 1.393.2 ± 1.146.8 ± 11.3
FM gradient unblocked36.5 ± 12.60.7 ± 0.824.7 ± 3.014.4 ± 9.1––
α=0\alpha = 0 (no InfoNCE)95.5 ± 0.682.0 ± 1.882.1 ± 4.695.1 ± 0.60.1 ± 0.20.3 ± 0.5

Q7. Where do the baseline numbers come from, and which uncertainty is reported?

We moved baseline provenance to the main text. Table 1 now marks our reproductions (‡), and the other numbers are copied from the original papers. We also mark the uncertainty type per method in Table 1(a), with planning-seed SD for HD and HDMI and training-seed SD for SSD, DF, VH-Diffuser and ours. The appendix now states the seed type of each published source.

References

[1] John L. Zhou and Jonathan C. Kao. Flattening hierarchies with policy bootstrapping. NeurIPS 2025.

Reviewer_DgFEOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

HorizonFlow proposes a hierarchical planner for offline goal-conditioned RL that treats plan length as an output of generation rather than a prescribed input. The method combines insertion-based generation with flow matching to jointly generate continuous plan content and length. A subgoal route planner generates variable-length latent subgoal routes, and an action-prefix controller generates variable-length action prefixes toward the next subgoal. The generated token count is reused for candidate selection (favoring shorter plans) and for steering generation via Feynman-Kac resampling, avoiding the need for a separate learned value model. Evaluated on Maze2D, Multi2D, and OGBench navigation and visual manipulation benchmarks, HorizonFlow achieves the highest average performance among compared methods.

Critical strengths

a. Treating plan length as an output of generation rather than a prescribed input is a meaningful conceptual contribution to the generative planning space. The insertion + flow matching combination for continuous-valued plans is technically novel and well-executed. The XY diagnostic (Figure 4) clearly demonstrates the horizon mismatch problem and shows that joint content-length generation addresses it.

b. The paper provides extensive ablations, sensitivity analyses, and diagnostic experiments. The within-pair selection analysis (Table 2), the length-signal comparison (Figure 9), and the inference cost analysis (Figure 5e-h) are all well-designed and informative. The matched full-system controls in Appendix B (Figures 10-11) are particularly rigorous for isolating the contribution of variable-length generation.

Critical weaknesses

a. The novelty is incremental and the paper does not clearly position itself against key predecessors. The core technical approach, insertion + flow matching for continuous plans, is a reasonable extension of OneFlow (Nguyen et al., 2025) and Insertion Process (Zhang et al., 2026), but the paper does not clearly articulate what is fundamentally new vs. what is a straightforward adaptation. Similarly, VH-Diffuser (Liu et al., 2025) already addresses variable-horizon planning with a learned length predictor; HorizonFlow's improvement (determining length during generation rather than before) is meaningful but incremental. The paper would benefit from a direct comparison or at least a clearer discussion of how it differs from these specific works.

b. The paper does not cite Diffusion Policy (Chi et al., RSS 2023), Decision Transformer (Chen et al., NeurIPS 2021), Trajectory Transformer (Janner et al., NeurIPS 2021), or classical MPC literature. These are foundational works in diffusion-based control, sequence modeling for RL, and receding-horizon execution; all directly relevant to HorizonFlow's approach.

c. HorizonFlow combines insertion-based generation, flow matching, FK steering, hierarchical planning, count-based selection, and receding-horizon execution. The ablations show that the hierarchy and the generative PC are important, but the paper does not clearly demonstrate that all components are necessary. For example, FK steering provides modest gains mainly at small K (Figure 5a), and the paper could provide a clearer recommendation about when it is needed. A simpler variant (e.g., insertion + flow matching without FK, or with a simpler selection heuristic) might achieve similar performance with less complexity.

d. The assumption that shorter plans are better works in the tested domains (maze navigation, simple manipulation), but it is not theoretically justified and may not generalize. In domains where the optimal plan requires a longer trajectory (e.g., multi-step manipulation with intermediate goals, navigation with complex obstacle layouts), count-based selection could fail. The paper acknowledges this as "selection risk" but does not provide a principled analysis of the failure modes or a fallback mechanism.

e. The average success rate on visual manipulation is 55.0%, with only 14.4% on visual-cube-triple. The paper does not analyze why performance drops on harder tasks or whether the variable-length generation provides any advantage in these domains. A deeper analysis of failure cases or a comparison with simpler baselines on these tasks would strengthen the paper.

Questions for authors
  1. Can you provide a theoretical analysis or at least a more detailed empirical analysis of when count-based selection fails? For example, are there domains or task configurations where shorter plans are systematically worse?

  2. How sensitive is the method to the choice of KK (candidate count)? The paper shows that K≥64K \geq 64 is needed for best performance (Figure 5a), but this increases inference cost. Is there a principled way to choose KK, or is it a hyperparameter that must be tuned per domain?

  3. Can the method be extended to stochastic dynamics? If so, how would the count-based selection heuristic need to be modified?

  4. How does the method compare against a simpler variant without FK steering? The gains from FK are modest (Figure 5a), and the paper could clarify when FK is necessary.

5 comments

We thank the reviewer for the detailed and critical reading. The questions on novelty, on the necessity of each component and on the failure cases of count-based selection led to clearer positioning and to new analyses, which we describe below.

W1 (a). What is new relative to OneFlow, the Insertion Process and VH-Diffuser?

HorizonFlow differs from OneFlow and the Insertion Process in two ways. First, it is, to our knowledge, the first insertion-based planner that jointly generates the content and length of continuous-valued plans. OneFlow inserts discrete text tokens (plus image slots denoised by flow matching), and the Insertion Process maze and graph nodes from a fixed vocabulary. A control plan has no vocabulary to insert from, so HorizonFlow inserts continuous subgoals and actions and generates each token's position in the plan as a continuous order coordinate. Second, the generated count is itself a planning signal. A plan's token count is the time that plan takes to reach the goal, so HorizonFlow picks the shortest of its candidate plans and steers partial plans toward shorter ones during generation, with no separately trained value function. In OneFlow and the Insertion Process the output length is only an outcome of generation; neither uses it to choose among samples or to steer generation.

VH-Diffuser's performance depends on the accuracy of a separately trained length predictor and inherits its error. The predictor maps a start–goal pair to a length in one shot. HorizonFlow instead settles the length gradually, in the same way that flow matching reaches a sample through many small denoising steps rather than one direct map from noise. Each step only asks how many subgoals are still missing between adjacent subgoals of the partial plan. This has two benefits. The count tracks goal distance better. The predictor ranks goal distances worse than our generated count (ρ=0.653\rho = 0.653 vs 0.954, Appendix A.5). The count is available during generation, so it can steer partial plans (FK steering), which a length fixed in advance cannot. Appendix B.1 places VH-Diffuser's predictor inside our system with all else shared. Given the same budget of 16 candidates, chosen by the predictor's own route estimate or by a learned value function, it still falls far short of HorizonFlow (table below; Appendix B.3). The revised Section 5 reports this comparison, now extended to AntMaze-Large and HumanoidMaze-Large.

Success (%), mean ± SD over 3 seedsAntMaze-LargeAntMaze-GiantHumanoidMaze-LargeHumanoidMaze-Giant
VH-Diffuser length predictor, 1 sample39.7 ± 5.415.7 ± 3.214.4 ± 2.92.7 ± 0.9
VH-Diffuser length predictor, 16 samples chosen by the length predictor47.5 ± 7.215.2 ± 3.516.3 ± 5.25.3 ± 1.2
VH-Diffuser length predictor, 16 samples chosen by a learned value32.3 ± 9.312.0 ± 2.914.1 ± 2.42.1 ± 1.2
HorizonFlow, 1 sample81.1 ± 2.641.9 ± 6.059.2 ± 2.154.4 ± 4.8
HorizonFlow, count-based selection (K=16K = 16)93.1 ± 1.286.9 ± 2.679.2 ± 6.294.7 ± 0.5

W2 (b). Missing related work on Diffusion Policy, Decision Transformer, Trajectory Transformer and MPC.

We now cite and discuss these works in Section 2. Diffusion Policy generates fixed-length action chunks, and Decision Transformer and Trajectory Transformer are sequence models with a fixed context length; HorizonFlow instead generates the plan length as part of its output. MPC is the receding-horizon principle that our execution follows.

W3 (c) & Q4. Are all components necessary, and when is FK steering needed?

FK steering looks modest only because success is already high at KRP=16K_{\mathrm{RP}} = 16; measured in candidates, its gain is large. Unsteered planning needs two to four times more candidates to reach the same success. Without FK, KRP=64K_{\mathrm{RP}} = 64 is needed to match what FK reaches at KRP=16K_{\mathrm{RP}} = 16 (94.2% vs 94.3% on navigation; Figure 5(a)). FK therefore gives the same success at lower latency. It adds a small overhead at a fixed KRPK_{\mathrm{RP}}, but a much smaller KRPK_{\mathrm{RP}} suffices (Figure 5(a, e)). The gain is larger where success is not saturated. It is 1.7 points on tasks at or above 90% and 4.1 points on tasks below 90% (second table below). We recommend FK whenever higher success is wanted within a latency budget, and the revised Section 5 says so. The other components are ablated in the first table below (Appendix B). Minimum-count selection is already the simplest selection rule, an argmin over a count that the generator produces anyway. Without selection the agent follows longer routes and reaches the goal later, so fewer episodes succeed within the time limit (Appendix B.4).

VariantNavigation average (%)
Full HorizonFlow (KRP=16K_{\mathrm{RP}} = 16, KPC=4K_{\mathrm{PC}} = 4, FK on)94.3
RP FK off (KRP=16K_{\mathrm{RP}} = 16, KPC=4K_{\mathrm{PC}} = 4)92.1
KPC=1K_{\mathrm{PC}} = 1 (KRP=16K_{\mathrm{RP}} = 16, FK on)92.3
No selection (KRP=1K_{\mathrm{RP}} = 1, KPC=4K_{\mathrm{PC}} = 4, FK on)56.2
Behavior-cloning controller instead of the generative PC (KRP=16K_{\mathrm{RP}} = 16, FK on)67.8
Flat planner (no hierarchy, KRP=16K_{\mathrm{RP}} = 16, FK on)42.9
FK at KRP=16K_{\mathrm{RP}} = 16 (13 OGBench tasks)TasksFK off (%)FK on (%)Gain
Success at least 90%895.296.9+1.7
Success below 90%554.558.6+4.1

W4 (d), Q1 & Q3. Why is the shortest plan preferred, and when can count-based selection fail?

Under deterministic dynamics the shortest feasible plan is optimal. The standard goal-conditioned objective maximizes E[γT]\mathbb{E}[\gamma^T] for the first time TT at which the goal is reached (Section 3), and γT\gamma^T strictly decreases in TT. In the reviewer's examples (intermediate goals, complex obstacles) the long route is still the shortest plan that reaches the goal, so it is not a failure case. Selection can fail in two ways.

(i) An infeasible short candidate. As in other sampling-based planners, the best of many samples can be an outlier [1], here a route that is short because it is infeasible. The risk grows with KK. At the default K=16K = 16 every OGBench task is within 1.7 points of its best success over K≤16K \leq 16, and clear drops appear only at much larger KK (cube-single 93.2% at K=16K = 16, 88.3% at K=64K = 64). Existing methods that detect infeasible plans of a generative planner [2] could also be applied to HorizonFlow’s candidates.

(ii) Stochastic dynamics (Q3). A short route can then be riskier than a longer one. Our count is learned from segments that reached their goals, so from offline data alone it cannot tell a reliably short route from a lucky one, the same problem that return-conditioned policies face in stochastic environments [3], and it would be optimistic. HorizonFlow does not handle this case, and we state it as a limitation. Changing the selection rule alone would not fix this, because the count itself is biased. We believe the count has to be corrected with what the agent observes during execution. The revised Section 5 ("Selection risk") and the Limitations section state both cases, with per-environment results in Appendix F.

Q2. How sensitive is the method to KK, and how should KK be chosen?

K≥64K \geq 64 is not needed, since every benchmark result uses KRP=16K_{\mathrm{RP}} = 16 and KPC=4K_{\mathrm{PC}} = 4, the same for all 16 datasets, with no per-environment tuning. In Fig. 5(a) the navigation average at KRP=16K_{\mathrm{RP}} = 16 (94.3%) already equals the one at KRP=64K_{\mathrm{RP}} = 64. The paper asks for K≥64K \geq 64 only in the Maze2D diagnostic (Fig. 4), where the comparison is against a fixed-length planner that is given a privileged per-pair horizon. Sensitivity is low. Navigation success varies by about one point for KK between 8 and 64 (Fig. 5(a)), and on all 13 OGBench tasks it rises steeply up to KRP=4K_{\mathrm{RP}} = 4–88 and then levels off, with a drop on a few tasks only at the largest KK (see (i) above; Appendix F). To choose KK, start from the default and double KRPK_{\mathrm{RP}} within the runtime budget until success stops improving.

W5 (e). Why is performance lower on visual manipulation, especially cube-triple?

Visual manipulation is harder than navigation for every method, but HorizonFlow is not weak there relative to the baselines. Its 55.0% average is the highest among the methods in Table 1(c), and it is the best of them on three of the four tasks. Success also rises with the number of route candidates KRPK_{\mathrm{RP}}, the mechanism our method relies on (per-task curves in Appendix F). Cube-triple is the one exception, and our analysis points to the subgoal latent. We measured how much cube-position information the subgoal latent retains. For each evaluation frame, we take the training frame whose latent is nearest and compare the cube positions of the two frames. The median position error is 2.2 cm on cube-double but 9.0 cm on cube-triple. This suggests that on cube-triple, frames with different cube arrangements lie close together in the latent, so a generated subgoal does not specify where the three cubes should be, which can explain the drop. A likely reason is that cube-triple has far more possible configurations than the other tasks, which makes the latent harder to learn. The revision states this as a limitation with the analysis in Appendix F.

References

[1] Donghyeon Ki et al. Prior-guided diffusion planning for offline reinforcement learning. NeurIPS 2025.

[2] Kyowoon Lee, Seongun Kim, and Jaesik Choi. Refining diffusion planner for reliable behavior synthesis by automatic detection of infeasible plans. NeurIPS 2023.

[3] Keiran Paster, Sheila McIlraith, and Jimmy Ba. You can't count on luck: Why decision transformers and RvS fail in stochastic environments. NeurIPS 2022.

Reviewer_3KdgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers the goal-conditioned RL setting where an agent learns goal-conditioned policies based on trajectories consisting of state-action pairs. The key idea proposed by this work is to treat the length of a generated plan not as an input parameter to the learned policy but as an output parameter, resulting in stochastic policies generating plans of different lengths with the work subsequently choosing the shortest one. This allows steering towards the shortest path without the learning of a separate value model

Critical strengths

Overall, I found the paper very interesting, and I applaud the authors' attention to detail.

The writing is also very rigorous and comprehensive, and the work is largely reproducible. I ran a reproducibility check by asking an LLM to look for any missing parameters/details needed to reproduce the paper's results, and there were still a few that should be easily fixable for the final version.

Critical weaknesses

The main weakness is that the paper is only accessible to someone familiar with robot learning and some of the underlying terminology. This is not a reason to reject the work, but making it more broadly accessible for the ICLR audience is what could take the work from good to great. For example, the abstract speaks of "subgoal route planner guides its action-prefix controller", but it is not really made clear beforehand what an action-prefix controller is.

Questions for authors
  • The authors argue that their work does not involve a learned value model. While technically true, I wonder whether the prediction of plan-length isn't implicitly something that can be considered a learned value model.
3 comments

We thank the reviewer for the encouraging assessment and for taking the time to run a reproducibility check. The comments on accessibility helped us make the paper readable beyond the robot-learning audience.

W1. Can the paper be made accessible beyond robot learning?

We revised the abstract and introduction to define the subgoal route planner and the action-prefix controller at first use, and added a glossary at the start of the appendix.

Q1. Is the predicted plan length an implicit value model?

The count plays a role similar to a value function, but it differs in two ways. It scores a candidate plan directly, even a partial one during generation, rather than a state pair, which is what enables FK steering in addition to selection, and it is produced by the planner itself rather than by a separately trained model. The revision therefore says "no separately trained value function".

S1. Are any reproduction details still missing?

Thank you for the reproducibility check. We went through the appendix for missing parameters and added them in the revision. These are the layout of the Transformer blocks, the input normalization, the training seeds, and the pinned software versions of the released code.

I appreciate the authors' responses and maintain my positive view of the paper.

We sincerely thank the reviewer for their positive feedback and continued support of our work!

Comments on the paper

2 comments

We thank all reviewers for their careful reading and constructive comments. The revised PDF marks all changes in blue. Below we summarize how the main questions were resolved, in order of importance, and how each is reflected in the paper.

Contribution. HorizonFlow generates the length of a plan together with its content, by inserting continuous subgoals and actions with flow matching. To our knowledge it is the first insertion-based planner to do so. Because a plan’s token count is the time that plan takes to reach the goal, the planner itself supplies the signal for choosing among candidate plans and for steering generation, with no separately trained value function.

1. Does the gain come from joint generation or from test-time search? (CAgs, DgFE) Selection helps only because the candidates are jointly generated. Generating the length with the content avoids plans that are too short, and selection over the generated candidates avoids plans that are too long. In the requested matched-budget controls, the fixed-length planner is the same model trained without insertion; given the same number of candidates and an external ranking signal, it stays behind HorizonFlow on Maze2D and on OGBench, whether its length is set by a length predictor or spread over the candidates. In the paper, the abstract and Section 1 state this division of roles, and Section 5 and Appendix B (Table 5) report the controls.

2. What is new relative to OneFlow, the Insertion Process and VH-Diffuser? (DgFE) Unlike OneFlow and the Insertion Process, HorizonFlow inserts continuous tokens and uses the generated length as a planning signal. Unlike VH-Diffuser, it settles the length during generation instead of predicting it beforehand. In a matched comparison where only the way the length is set differs, generating the length outperforms VH-Diffuser’s length predictor by a wide margin. In the paper, Section 2 states these differences and discusses the related work the reviewers named.

3. Is every component needed? (DgFE, CAgs) The ablations show that per-gap counts, the variable-length controller, the hierarchy and the encoder training choices are each needed, and FK steering reaches the same success with fewer candidates. In the paper, Appendix B collects the ablations and adds the requested controls, and Section 5 recommends when to use FK steering.

4. Why is the shortest plan preferred, and is the count a value function? (DgFE, 3Kdg) Under deterministic dynamics the shortest feasible plan is optimal for the goal-conditioned objective. The count plays a role similar to a value function, but it scores a whole plan, even a partial one during generation, and it comes from the planner itself. In the paper, Sections 3 and 5 give this argument and the two cases in which count-based selection can fail, and the Limitations paragraph states the deterministic-dynamics assumption.

5. Benchmark coverage and weaker tasks (CAgs, DgFE) On OGBench we evaluate the navigation and visual manipulation tasks that our most recent baseline uses, with one setting for all datasets. For the one weak task, visual cube-triple, our analysis points to the subgoal latent. In the paper, Section 5 names the evaluated tasks and scopes the claims to them, Table 1 marks our reproductions, and the Limitations paragraph states the weakness on multi-object manipulation, with the analysis in Appendix F.

6. Accessibility and reproducibility (3Kdg) In the paper, the abstract and introduction define the route planner and the action-prefix controller at first use, and the appendix adds a glossary and the missing implementation details.

Dear Reviewers,

Less than two days remain before the discussion period ends on November 18 (AoE). If you have not yet replied to the author rebuttal, please do so as soon as possible, indicating which of your concerns are resolved and which remain.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.