- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes HorizonFlow, a hierarchical generative planner for offline goal-conditioned RL. The key idea is to generate the plan length together with the plan content, instead of fixing a horizon before generation. It combines insertion-based generation (per-gap count heads, Bernoulli + zero-truncated Poisson) with flow matching on continuous tokens, and each token carries a continuous order coordinate. A route planner generates a variable number of latent subgoals, and a prefix controller generates a variable-length action prefix toward the first subgoal. At test time the method samples candidates (16 routes, 4 prefixes), picks the one with the fewest tokens, and optionally uses FK resampling with the expected remaining count to push generation toward shorter plans. Only the first action is executed, then it replans. Experiments cover Maze2D/Multi2D, 9 OGBench navigation tasks and 4 OGBench visual manipulation tasks. The method reports the best average in all three groups, with very large gains on large/giant mazes. A horizon diagnostic on Maze2D-Large and ablations on candidate count, FK, replan period, executed prefix and inference cost are given.
Critical strengths
-
The idea is simple and well motivated. Fig. 4(a-c) shows clearly that no single fixed horizon works for both near and far goals, and the joint model keeps $\sim$100% reach rate at every distance even with one candidate. The selection signal (token count) is a free by-product of generation, so no value function is needed. This turns the horizon from a hyperparameter into a model output, which is a real pain point for inpainting planners. The diagnostic design (binning start-goal pairs by nominal steps, same backbone, data and controller for both models) is clean and convincing on this point.
-
The OGBench navigation results are strong and carefully done: 94.2% average vs 75.4% for the best prior method, 8 seeds, the standard OGBench protocol (5 tasks 50 episodes), small std, and humanoidmaze-giant at 95.5% vs 35%. The analysis section is also honest: it reports the number, the inference cost per step, and the selection risk. If the numbers hold, a 20-60 point jump on giant mazes is a significant result for this field.
Critical weaknesses
-
The claim "joint generation fixes the two-sided horizon mismatch" is not what the evidence shows. What the evidence shows is "sample many candidates and pick the shortest" fixes it. In Fig. 5(a), gives 56.2%, which is below HIQL (65.0), SAW (75.4) and CTA (64.3); the jump to 94.3% comes from best-of-16 min-count selection. In Fig. 4, the joint model at scores lower than the oracle fixed horizon in every bin (242/207/150/103 vs 274/248/199/148) and takes 1.3-1.6$\times$ more steps, and even one global beats joint at and . Appendix B.1 says the same on OGBench: a single joint RP sample gets 41.9%/54.4% on antmaze-giant/humanoidmaze-giant vs 86.9%/94.7% with selection. So joint generation alone is biased long, and the "too long" side is solved by search with a count prior, not by the model. This matter because the fixed-length baselines never get the same candidate budget: the fixed- model in Fig. 4 is one sample, the OGBench planning baselines are mostly one-shot, and the fixed-length control in Appendix B.1 runs only at on two environments. A fair control is a fixed-length planner with the same and a cheap ranking signal (XY path length in Fig. 4; HIQL value or the VHD predictor on OGBench). Appendix A.4 does something like this for Diffuser on Maze2D only. Without this control I cannot tell how much of the gain is the modeling idea and how much is test-time search. I think the paper should re-frame the contribution around count-based selection/steering and add the matched-budget control. The method can still be good, but the story needs to match the data.
-
On OGBench, where the biggest claims are made, the closest competitors are missing and the task subset is narrow. (a) VH-Diffuser is the only other method that adapts the horizon per start-goal pair, and the intro is written against it, but it appears only on Maze2D/Multi2D. VHD itself reports OGBench antmaze and cube results, so the comparison is possible. (b) The only planning baselines in Table 1(b) are Diffuser/HD/DF reproduced by the authors (planning in full observation space with a shared controller), and they collapse to 0-7% on antmaze-giant and humanoidmaze-giant. CompDiffuser and HDFlow are cited as the representative fixed-horizon / hierarchical flow planners but not compared; CompDiffuser reports high success on OGBench maze stitch datasets. (c) The main text never says Table 1(b) uses the navigate datasets; the stitch datasets (the most relevant setting for a planner that composes routes) are absent; visual manipulation reports 4 of 9 tasks (no cube-quadruple, no puzzle) and no state-based manipulation. Among the 4, the method loses cube-triple (14.4 vs 21) and cube-double is over 4 seeds. So "highest average in all three groups" is on a selected subset. I do not think this is on purpose, but the claim should be scoped or the missing comparisons added.
Questions for authors
- Humanoidmaze: large is 80.2 but giant is 95.5. Every baseline is monotone medium > large > giant. Is this caused by the recipe (stride 32 / for medium and large, 64 / 4 for giant)? Did you try the giant setting on large? If success depends this much on the stride rule, please say so in the main text.
- Matched-budget control for weakness 1: can you run (a) the Fig. 4 fixed- model with candidates selected by XY path length, and (b) on OGBench the fixed-length RP from Appendix B.1 with candidates ranked by a HIQL value or by the length predictor? This is the experiment that would decide weakness 1 for me.
- For a fixed , what is the spread of generated counts across the 16 route candidates? Appendix B says the count tracks distance with , so why does fail so badly (41.9% on antmaze-giant)? Are the failures mostly too-long routes, or infeasible routes?
- Order coordinate: when a new token is born in gap , is its initialized from Gaussian noise, or placed between the two neighbors? If from noise, the gap structure is only nominal and only the total count carries information. Fig. 3 at - shows tangled orderings. Please clarify, and report how often the final sorted order differs from the birth gap order.
- PC level: only the first action is executed and adds about 2 points (92.3 94.6). Appendix B.2 compares against a "full-horizon" fixed-length PC that always generates the full capacity, which looks like a weak control. What about a short fixed-length PC (e.g. 4 or 8 actions) with the same RP? If it matches, the variable-length PC is not needed.
- The encoder collapse claim in Sec. 4.2 has no number behind it. Please add and FM-gradient-unblocked ablations.
- Baseline provenance should be in the main text: which numbers are copied and which are your reproductions. Also Table 1(a) mixes planning-seed (HD, HDMI, SSD) with training-seed std (DF, VHD, Ours) under one "mean std" label.
5 comments
W2. Are the closest OGBench competitors and tasks covered?
(a) VH-Diffuser's OGBench results use a non-standard protocol. They cover only AntMaze-Medium with random start–goal pairs, and a cube environment in which the arm only has to reach a target position, without moving any cube, so they cannot be added to Table 1; Appendix B.1 is our matched comparison (41.9% / 54.4% vs 15.7% / 2.7%).
(b) The reproduced planners are verified JAX ports of the official code, and we tuned them carefully. For each environment we searched the training horizon, the network size, the replanning period, the low-level controller and the other inference settings (appendix, "Implementations and Reproduction Protocols"). Their drop grows with the route length. On AntMaze, where the observation is the same at every size, Diffuser falls from 77% (Medium) to 61% (Large) and 5% (Giant), and HD and DF fall in the same way, because one fixed-length plan has to cover ever longer routes (on two-dimensional PointMaze, Diffuser and HD stay above 90%). With a fixed length, our own system shows the same pattern (47.5% on AntMaze-Large, 15.2% on AntMaze-Giant; W1 (b)), whereas HorizonFlow, which generates the route length, stays at 86.5% on AntMaze-Giant. CompDiffuser addresses a different problem, the stitching of short trajectories, and is evaluated only on the stitch and explore datasets; setting the plan length is complementary (Section 2). HDFlow requires additional data that includes failure trajectories, as well as a pretrained DINOv2 encoder with a world model, so its numbers cannot be compared directly with ours; the appendix lists them for reference only. Even so, HorizonFlow, which uses neither, has a higher average success on the four visual tasks (55.0% vs 41.8%).
(c) The revision states that Table 1(b) uses the navigate datasets and names the evaluated tasks wherever we claim the highest average. For navigation and visual manipulation, we use the tasks of SAW [1] (Table 1), our most recent baseline, using its nine state navigation tasks and its four visual manipulation tasks. For manipulation we chose the image-based tasks because they are closest to a real robot setting, where the agent observes camera images rather than object states. The set was not chosen in our favor. It includes visual cube-triple, where HorizonFlow (14.4%) is below HIQL (21%) and SAW (20%), and we report it as is. Section 5 now states this choice and scopes the claim to the evaluated tasks.
Q1. Why is HumanoidMaze-Large below Giant?
Mostly the episode limit. In OGBench, maze size alone does not set task difficulty. HumanoidMaze-medium and -large allow 2,000 steps per episode but -giant allows 4,000 (every PointMaze and AntMaze size allows 1,000), so a giant episode has twice the time budget of a large one. For HorizonFlow, with the limit doubled to 4,000 steps (Giant's budget), the same checkpoints reach 98.4%, and 92% of the failures at 2,000 steps reach the goal before 4,000. Training Large with the Giant recipe (stride 64, 4) gives the same success as the default recipe, so the stride rule is not the cause (table below). Section 5 now states this budget dependence, and Appendix F reports both runs.
| HumanoidMaze-Large, mean ± SD over 3 seeds | Success (%) |
|---|---|
| Default recipe, 2,000-step limit | 80.1 ± 2.6 |
| Default recipe, 4,000-step limit | 98.4 ± 0.4 |
| Giant recipe (stride 64, 4), 2,000-step limit | 79.7 ± 3.6 |
Q3. How spread are counts within a pair, and why does fail?
Mostly too-long routes; infeasible ones are a small minority. The 16 candidates for a pair differ substantially in count (first table below), because the data contain trajectories of different routes and durations between the same endpoints, and the model generates that variety. The of Appendix A.5 does not describe a single draw. It is the rank correlation between the nominal steps and the minimum count among 64 candidates per pair, that is, the count after selection. A single draw, in contrast, is longer than this minimum in almost every problem, so most failures are timeouts while the agent is still approaching the goal (second table below); minimum-count selection largely removes them. Details are in new Appendix B.4.
| Generated count of a pair's 16 candidates, mean ± SD over 3 seeds | AntMaze-Giant | HumanoidMaze-Giant |
|---|---|---|
| Mean candidate (tokens) | 47.4 ± 0.3 | 45.4 ± 0.7 |
| Longest minus shortest (tokens) | 15.0 ± 0.1 | 15.4 ± 0.4 |
| Longest minus shortest (environment steps) | 240 ± 2 | 983 ± 28 |
| Share of failures (%), pooled over 3 seeds | AntMaze-Giant | HumanoidMaze-Giant |
|---|---|---|
| Timeout while still approaching the goal | 75 | 71 |
| Stuck at a wall | 2 | 9 |
| Fall (no motion) | 13 | 6 |
| Moving away from the goal | 10 | 15 |
Q4. Are newborn order coordinates drawn from noise, making the gap structure nominal?
From Gaussian noise (Appendix D.3). Each gap only proposes how many tokens to add; a newborn is not assigned to the gap that proposed it, and its position is the order coordinate that the flow generates. There is therefore no birth-gap order to compare the final order with. The orderings in Fig. 3 at – are intermediate states that the flow is still refining.
The gap structure is still not nominal, because it is the supervision. The per-gap head predicts how many tokens are missing between each pair of adjacent tokens, which tells the encoder where the plan is incomplete. With a single head that predicts only the total count, the method fails in every environment (table below; new Appendix B.6). The predicted length becomes nearly the same for every start–goal pair, because the latent loses the temporal ordering that per-gap counts impose.
| Success (%), mean ± SD over 3 seeds | AntMaze-Large | AntMaze-Giant | HumanoidMaze-Large | HumanoidMaze-Giant |
|---|---|---|---|---|
| HorizonFlow, per-gap insertion (Table 1, 8 seeds) | 93.3 ± 1.1 | 86.5 ± 2.9 | 80.2 ± 1.5 | 95.5 ± 1.3 |
| Global remaining-count head | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 |
Q5. Would a short fixed-length controller suffice?
A short fixed-length controller does not suffice (table below; Appendix B.2). We trained controllers that generate only the first or 4 actions toward the subgoal and ran them under the same RP with . Although only the first action is executed, in our controller it belongs to a plan that reaches the subgoal, and the length of that plan has to vary. The RP is trained on subgoals one stride apart, but with model error its generated subgoals are not exactly one stride away, and the agent keeps approaching the same subgoal between route updates, so the number of actions still needed can be shorter or longer than any fixed length. Because the PC generates the length together with the actions, it can absorb this mismatch, which a fixed-length controller cannot.
| Success (%), , mean ± SD over 3 seeds | AntMaze-Large | AntMaze-Giant | HumanoidMaze-Large | HumanoidMaze-Giant |
|---|---|---|---|---|
| Fixed-length PC, | 69.3 ± 6.1 | 36.0 ± 6.2 | 31.7 ± 5.6 | 83.2 ± 3.2 |
| Fixed-length PC, | 49.1 ± 8.5 | 12.0 ± 6.0 | 32.5 ± 5.1 | 73.1 ± 9.2 |
| HorizonFlow PC (variable length) | 90.1 ± 2.6 | 83.2 ± 3.7 | 79.5 ± 2.4 | 93.1 ± 0.9 |
Q6. What supports the encoder-collapse claim?
Both ablations confirm the claim (table below; Section 4.2, new Appendix B.5). If the flow-matching gradient reaches the encoder, one way to lower that loss is to shrink the scale of the latents. The noise has a fixed scale, so as the latents shrink the regression target becomes easier to predict, and the loss falls without the flow fitting the data any better. The latents then contract, and success collapses in every environment; InfoNCE compares normalized vectors and cannot stop this. Without InfoNCE (), state latents keep their spread, but pixel latents collapse. We attribute this to the input. Neighboring frames differ in few pixels, so early in training the image encoder maps them to nearly the same latent, and the insertion loss alone does not pull them apart; once collapsed, a latent does not recover. InfoNCE provides the signal that separates them.
| Success (%), mean ± SD over 3 seeds | AntMaze-Large | AntMaze-Giant | HumanoidMaze-Large | HumanoidMaze-Giant | visual cube-single | visual cube-double |
|---|---|---|---|---|---|---|
| HorizonFlow (Table 1; 8 state / 4 visual seeds) | 93.3 ± 1.1 | 86.5 ± 2.9 | 80.2 ± 1.5 | 95.5 ± 1.3 | 93.2 ± 1.1 | 46.8 ± 11.3 |
| FM gradient unblocked | 36.5 ± 12.6 | 0.7 ± 0.8 | 24.7 ± 3.0 | 14.4 ± 9.1 | – | – |
| (no InfoNCE) | 95.5 ± 0.6 | 82.0 ± 1.8 | 82.1 ± 4.6 | 95.1 ± 0.6 | 0.1 ± 0.2 | 0.3 ± 0.5 |
Q7. Where do the baseline numbers come from, and which uncertainty is reported?
We moved baseline provenance to the main text. Table 1 now marks our reproductions (‡), and the other numbers are copied from the original papers. We also mark the uncertainty type per method in Table 1(a), with planning-seed SD for HD and HDMI and training-seed SD for SSD, DF, VH-Diffuser and ours. The appendix now states the seed type of each published source.
References
[1] John L. Zhou and Jonathan C. Kao. Flattening hierarchies with policy bootstrapping. NeurIPS 2025.
We thank the reviewer for the thorough review and for proposing concrete experiments. The matched-budget control in particular sharpened what the paper claims, and we ran it as requested.
W1 & Q2. Do the gains come from joint generation or from test-time search?
Selection can raise success only because the candidates are jointly generated. A fixed-length planner does not know the right horizon, and trying many horizons does not help, because it cannot tell at test time which plans are feasible ((a), (b)). HorizonFlow keeps inserting subgoals until the plan is complete, so most of its candidates reach the goal (99–100% in the Maze2D diagnostic), and this is what lets the shortest one be chosen. The revised introduction now states this division explicitly. Joint generation removes the too-short side, and selection over the generated candidates trims the too-long side. The same preference could also be built into sampling. With a learned value over plans, plans of higher value, that is shorter ones, could be made more likely to be drawn, which would raise single-sample success with no test-time search. That gain, too, would come from joint generation, because the length is decided during generation.
(a) Fig. 4 (Maze2D-Large XY, K=64). HorizonFlow is highest in every bin and every seed, even against the fixed-length planner given N∗. N∗ is a nominal horizon computed from the maze distance, not the shortest executable one, so the shortest generated plan can be shorter than N∗ and still reach the goal. Candidates are ranked by the HIQL value (Appendix A.5) or, as the reviewer suggested, by XY path length. With spread horizons, ranking by path length fails, because the shortest candidate is always the straight line from start to goal, which passes through the walls.
(b) OGBench (K=16). With 16 candidates ranked by the length predictor or by the HIQL value, as requested, the fixed-length RP at the VH-Diffuser length stays far below HorizonFlow. The fixed-length RP is the same model trained without insertion. It shares HorizonFlow’s architecture, data, training budget, subgoal encoder and prefix controller, and differs only in that its length is set externally. The VH-Diffuser predictor overestimates the length on the giant mazes, and all 16 candidates share its length, so ranking cannot correct it. Rows other than the default run with RP FK off, since FK cannot act on candidates of equal length.