acceptodds
Under review as a conference paper at ICLR 2027

It's the Model, Not the Macro-Actions: What Hierarchy Buys in Latent-Space Planning

Abstract

A hierarchical latent world model raises two abstraction levels at once: it rolls plans through a model that predicts ten steps at a time, and it searches over learned temporally extended macro-actions. The pair beats flat planners on long-horizon tasks, and the standard explanation credits the action abstraction. Using a released model pair whose flat and hierarchical checkpoints have bitwise-identical level-1 weights — so arms differ only in the planner and in which of the two predictors rolls the plan — we find that, on this pair, the credit belongs elsewhere. We first characterize what the learned macro-action space contains: a -dimensional code whose variance the two lowest temporal modes of a ten-step action chunk reach to and plain constant-hold chunking to . We then run the same hand-designed action classes through each of the two models. Over the fine model, every restriction scores below the unrestricted planner in the same harness — , , , across classes spanning to coverage, against (three of the four gaps significant). Over the coarse model, the crudest class — two numbers per macro-action — reaches , matching the full released hierarchy, with 14 discordant trials to 0 against the identical class over the fine model (); searching the entire learned -D space does no better (). Model-side measurements taken with no planner in the loop suggest why: at long lookahead the fine model's cost stops ordering plans the way the environment does (latent cost, from to steps; position cost, at ), while the coarse model's does not, and the gap is significant at steps (, , 17/20 layouts, plans). At steps an earlier estimate of that gap (, ) did not survive a larger plan sample (, ); we report it as within noise. We also report what failed: truncating the flat planner's lookahead to where its model is still trustworthy does not help, monotonically, across a range (, , , ), because the task's own objective is uninformative at short horizons ( at 50 steps). Of this paper's preregistered predictions three were confirmed, three refuted and two null: the confirmations are two model-side measurements and the prediction that constant hold over the coarse model reaches ; every refutation is a prediction about the flat planner at its own timescale.

Primary area:
applications to robotics, autonomy, planning
Keywords:
hierarchical planning, world models, latent-space planning, temporal abstraction, macro-actions

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kanqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether the hierarchical latent world model is benefitted from 1) macro action space or 2) coarse level planner.

Critical strengths
  • The paper provides a deep analysis between two published models: PLDM and HWM to show how exactly hierarchical model performs better than flat planning model.
Critical weaknesses
  • I am not sure how I should feel about this paper. Abstract is not abstract, introduction uses unintroduced terms, and figures are barely figurable. I try not to weigh too much on the paper’s presentation when reviewing, but this paper is on un-ignorable level.
  • The paper has only 13 references and misses rich literature context behind the latent world model. I do not even see JEPA-based world models.
  • This work verify its hypothesis using two pretrained checkpoints under maze setups. The scope is too narrow to draw meaningful and generalizable lesson over these scenarios.
  • No clear explanations provided for how two models (PLDM, HWM) differ in their mechanisms; at least provide how the inputs are given, how these model process the inputs, and how their planning differs, in a thorough way.
  • The section names are not organized and inconsistent.
Questions for authors
  • Could you restate the question the whole paper is dealing with in a plain language? Planning on the macro-action space & using coarse-level planning model do sound very similar for me, and I could not clearly understand what this paper tries to discover. If a model is applied on a coarse timestep in the micro-action representation space, it actually will have to return the most similar micro-action that the model has to head to, which essentially is a sort of macro-action (especially in this maze setup, where the action is configured only by 2-dimentional directions).
  • Could you expand the experiments over the maze dataset? The maze environment makes sense in that micro actions often lead to pitfalls, but essentially this setup could be the only beneficial setup where having coarse-level decision timesteps often leads model closer to the goal. Again, the generalizability problem emerges.
  • Similarly, could you generalize the idea over PLDM and HWM?
  • What’s the lesson the research community can take away from this paper? How will this be used to improve, motivate, and inspire other hierarchical world model attempts in the future? Should they stop creating a macro-action representation space (specifically for maze scenario)? Is this really a lesson that is not constrained to solving maze in a virtual environment?
2 comments
AuthorsOpenReview ↗

The question in plain words, how the two planners differ, and what the results support

Thank you for the review. You are right about the presentation, the references and the scope. The main limits (one environment, a missing comparison described below, the organisation of the paper) need more than the discussion period allows, so we have not uploaded a revised PDF.

The question in plain words (your first question). A hierarchical planner differs from a flat one in two ways at once. It rolls each candidate plan forward with a second predictor that jumps ten environment steps per call (the "coarse" model of Table 2), where the flat planner chains 250 one-step predictions (the "fine" model). And it searches over learned macro-actions: an 8-number code that a learned action encoder computes from a ten-step chunk of actions. A common reading credits the second. The paper asks which of the two the gain comes from.

You are right that the two overlap: any ten-step predictor must be told what happens during its ten steps, and that input is a macro-action of some kind. So the test cannot remove macro-actions; it can only make the search over them crude. The crudest search we ran is over one held 2-D action per ten steps (two numbers), which the released action encoder then turns into the code the ten-step predictor takes. On the same 20 hard trials, a planner that searches only these two numbers solves 5 when its plans are rolled through the one-step predictor and 19 through the ten-step predictor; 14 trials are solved only with the ten-step predictor and none only with the one-step one (Table 2). Searching the full learned 8-number space solves 17, which 20 trials cannot tell apart from 19. So the planner does not need to search its learned macro-action space.

Three qualifications:

  • The action encoder is still used, so this is a result about what the planner searches. It does not show that a learned macro-action space can be dropped when such a model is built.
  • The macro-action matters. When the two-number planner's macro-actions are shuffled among its candidate plans, success falls to 2/20 (Section 7). The title's "not the macro-actions" should read "not the search over the learned macro-action space".
  • With the ten-step predictor the plan is not executed directly. As in the released hierarchy, its predicted states are subgoals that the one-step planner tracks. So "the ten-step predictor" above means that predictor together with its subgoal loop; no experiment in the paper separates the two (Section 9).

How the two planners differ. Both encode the image observation with the same encoder into one latent space; the encoder and the one-step predictor are bit-for-bit identical in the two released checkpoints (Section 2). The flat planner (PLDM) samples 250-step action sequences, rolls each through the one-step predictor, scores it by the squared distance between the last predicted latent and the encoded goal, moves its plan toward the low-cost samples, executes the first actions and replans. The hierarchy (HWM) has no second observation encoder (Section 2); it adds an action encoder, which turns a ten-step chunk into the 8-number macro-action, and a predictor that advances the same latent ten steps per macro-action. Its top level samples sequences of 47 macro-actions in the release (470 steps; 25 in Table 2) and rolls them through that predictor; the predicted latents become subgoals that the one-step planner tracks.

Presentation and references. Agreed. The abstract is a list of results with their test statistics, and the abstract and introduction use terms before they are defined ("level-1 weights", "harness", "discordant", "coverage", κ\kappa). The section titles mix questions, claims and labels and follow the order of the experiments, so the decisive comparison arrives only in Section 7. Figure 1's label line runs outside its box and an axis label of Figure 4 is cut off. The paper has thirteen references; it never says that the two models it studies are themselves JEPA world models, and it cites little of the latent world-model literature.

Scope (your second and third questions). Agreed. Every result is on one released model pair and one maze family (Section 9); the release ships the hierarchy only for this maze model. We have no result outside mazes, so we cannot say whether they are a favourable case. Within them, coarse decisions alone show no sign of helping: the held actions solve 5 of 20 through the one-step predictor, against 9 to 12 for the unrestricted planner.

The lesson (your last question). On this model pair, searching the learned macro-action space is not what makes the hierarchy work: a search over two numbers per macro-action could not be told apart from the full search. The practical use is a control: run the crude search before crediting a learned action space. The results do not say that builders should stop learning a macro-action space, since the action encoder is still used here, nor that any of this holds outside this pair.

Reviewer_kanqOpenReview ↗

Post-rebuttal comments

Thanks for the honest answers. Although questions are answered, most weaknesses remain. I maintain my rating as it is.

Reviewer_joTxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that the standard explanation for why hierarchical planners beat flat planners is that learned macro-action space is responsible, which is incorrect; instead, the coarser rollout model should take the credit. It uses the released PLDM and HWM checkpoint pair and swaps the planners and rollout models without retraining. Table 2 presents the key result: regardless of how the action class is decided, using a coarser model is the main reason for the higher performance.

Critical strengths
  1. The paper clearly preregisters all its predictions and is honest about all its findings.
  2. The paper presents a useful negative result by showing that truncating the lookahead steps hurts the planner, and richer fine-model classes do not help.
  3. The paper also clearly points out its limitations and describes the future work adequately.
Critical weaknesses
  1. The paper is difficult to follow. The abstract is dense with numbers and statistical tests, making it hard to understand without the full context. The decisive experiment comes very late in Section 7; putting it up front can make the reading a bit easier.
  2. The claim is validated with a very narrow scope. All conclusions are drawn from a single model pair and a single environment family. Moreover, navigation can sometimes work by simply holding the same action for a few time steps, since it doesn't require fine control. Validating the same observation in manipulation tasks, which require fine control, would help support the central hypothesis of the paper.
  3. The underlying intuition of using a coarse model to accumulate less compounding error is well established in prior works such as jumpy models, clockwork VAEs, and short-rollout MBPO, which the paper cites. The related horizon-reduction literature, e.g., SHARSA [1], is not discussed as such in the paper. Further, even the reduced horizon can be used when a high-level planner provides the directions, as shown in PcLast [2] and ALPS [3], which is closely related to the paper's finding that it struggles on its own but can be useful when paired within a hierarchy.
  4. The paper uses the fixed stride length of 10 from the HWM release; how to choose this stride still remains open.
  5. The rank-fidelity measurements use only one seed (Appendix E), and the statistical tests consider only trials as a source of randomness; they don't capture the run-to-run variability. Running with more seeds can help in making the statistical claims stronger.

[1] Park et al., "Horizon reduction makes RL scalable", NeurIPS 2025.

[2] Koul et al., "PcLast: Discovering Plannable Continuous Latent States", ICML 2024.

[3] Shehmar et al., "Laplacian Representations for Decision Time Planning", ICML 2026.

Questions for authors
  1. What does the dimension 20 --- 8 mean in Table 2?
  2. What happens with different strides like 5 or 20? It will be helpful to show that the effect scales with stride, if that's the case.
  3. Why does DCT, which reaches 98% of the learned space, hurt the fine-model planner more compared to the unrestricted one?
  4. What is the latent-cost rank fidelity at K=256 for all lookaheads?

Minor Suggestions:

  1. Add a line in the abstract indicating that the released models are for a maze-navigation pair. Cut down the abstract and keep only the important things there. Present the decisive experiment (Section 7) before the failed accounts and negative results so it's easy to focus on what matters the most.
  2. Figure 1 content is overflowing from the box. Figure 4 is getting cut at the moment.
2 comments
AuthorsOpenReview ↗

Replies to Q1-Q4, two corrections of our own, and where we agree

Thank you for the review; we agree with the weaknesses. The main limits (one environment, a missing comparison described below, the organisation of the paper) need more than the discussion period allows, so we have not uploaded a revised PDF.

Q1. "20 — 8" in Table 2. The number of values the planner searches per ten-step chunk, fine-model planner then coarse-model planner: 10 steps × 2 action dimensions = 20 for the unrestricted fine planner, and the 8-number learned code for the coarse one. The dash is a typesetting fault; a vertical bar was meant.

Q2. Other strides. We have no result. The release has one coarse predictor, trained at stride ten, and nothing in the paper uses retrained weights. Whether the effect grows with the stride is untested.

Q3. Why DCT hurts the fine-model planner. We do not know. The three DCT classes solve 2, 0 and 2 of 20, below each of three runs of the unrestricted planner (9, 11 and 12 of 20; p≤0.04p \le 0.04 for each pair). The pattern is not monotone in smoothness (constant hold solves 5), and Table 1 was measured on unrestricted plans only, so we have no measurement of how the fine model ranks smooth plans.

Q4. Latent-cost rank fidelity at K = 256. "Latent" and "position" name the true cost the model's cost is compared with: distance to the goal in latent space, or in a straight line. At K = 256 the latent version exists only for the extended run, at four lookaheads; the main-run rows of Table 1 (10 to 250 steps), and the abstract's 0.20 to 0.05, are K = 64 only. The per-model means, from the per-layout measurements in the supplementary material, at 50, 100, 250 and 500 steps:

  • fine model: 0.182, 0.114, 0.042, 0.010
  • coarse model: 0.100, 0.119, 0.091, 0.063
  • coarse − fine, from unrounded means (Table 3, last row): −0.082, +0.004, +0.049, +0.053

At 500 steps the latent-cost gap is about a third of the position-cost gap (+0.053 against +0.154), with p=0.03p = 0.03 by the test of Appendix A (0.09 after the Holm correction over the four lookaheads). At 50 steps the fine model is ahead in 17 of 20 layouts (−0.082, p=0.003p = 0.003; 0.013 after the correction).

Two corrections. (1) Appendix A and Tables 1 and 3 say the 64 plans are the first 64 of the 256 ("nested draw"). That holds in the first layout only: one random stream runs on through all layouts, so from the second layout on the two runs drew different plans, and the two estimates are two plan samples. The 500-step position gap is significant in both (64-plan value first: +0.193, +0.154); the 250-step position gap (+0.140, +0.054) and the 500-step latent gap (+0.007, +0.053) do not. (2) In the sample-budget sweep of Section 3, the 500- and 2000-sample runs are identical on every trial, including the step at which each ends, so the sweep has four distinct points, not five; the records do not show which label is wrong.

W5. One seed, and randomness over trials only. Right on both. The rank fidelity has one seed and, as corrected above, two plan samples; every success count is one planner run, tested over trials. Some run-to-run information comes from repeated runs: the unrestricted flat planner solves 11 (published run), 12 and 9 of 20 (our runs), yet the published and the 12/20 run differ on 11 of the 20 trials (Table 5); constant hold solves 4 and 5 through the fine model (Appendix E) and 19/20 twice through the coarse model (the second in the first 20 trials of Section 7's 40-trial re-run; records in the supplement). Totals move by at most 3 of 20, against the difference of 14 between 5/20 and 19/20 in Table 2; single trials are not stable.

W2. Scope, and holding an action. Agreed: one model pair, one maze family, and no manipulation task. On holding: through the fine model, held actions solve 4/20 and 5/20 against 9, 11 and 12 of 20 for the unrestricted planner, so there is no sign that holding helps by itself. With the coarse model the held action is not executed directly: it only defines the coarse plan, whose subgoals the fine planner tracks (Section 7).

W3. Prior work. Agreed. That a coarser model accumulates less error is established. What the paper adds is narrower: on one pair with identical low-level weights, the action class searched is held fixed and the predictor is changed, together with the subgoal loop it drives; no comparison in the paper separates the two (Section 9). SHARSA, PcLast and ALPS should have been discussed: Park et al. show that reducing the value horizon alone already improves scaling, and PcLast and ALPS keep each planning subproblem within the short range where the model is reliable. As you say, that is close to our truncation result (Table 5), where a short lookahead fails without a higher-level plan.

W1, W4, minor points. Agreed on all: how to choose the stride is open (Section 9); the abstract carries too many numbers and does not name the maze-navigation pair; the decisive comparison comes too late; both figure faults are real.

Thank you for the detailed and candid response. The paper still needs significant revisions, including experiments across different environments, so I will keep my score as it is.

Reviewer_He48OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates if the benefits of hierarchical world models in long-horizon planning come from learned macro-actions or coarse-grained world models. The author conduct experiments on maze navigation, and find that a coarse world model with simple, hand-designed actions performs as well as the full hierarchical planner, while the same actions perform poorly with a fine-grained model. They further show that coarse models better preserve the relative ranking of candidate plans over long horizons. The results suggest that the main advantage of hierarchical planning comes from temporal abstraction in world modeling rather than learned action representations.

Critical strengths

The author makes a good point on an important question: whether the benefits of hierarchical planning come from action abstraction or temporal abstraction in world modeling. And the paper disentangle the effects of action representations and prediction resolution using models with identical low-level weights, allowing a relatively clean comparison.

The experiment results show that simple, hand-designed macro-actions combined with the coarse model match the performance of the full hierarchical planner.

Critical weaknesses

My major concern is that the experiments are mostly on deterministic, static maze environments. It is unclear whether the conclusions extend to stochastic environments with complex dynamics, where coarse-grained predictions may be less reliable.

Also, the coarse model relies on a hierarchical subgoal-tracking mechanism with a low-level planner. Therefore, the performance gains may come from the global-local planning decomposition rather than the coarse model itself. Although the coarse model achieves better long-horizon rank fidelity, the absolute correlation remains very low, I'm wondering if the weak ranking signal sufficiently explains the substantial improvement in planning success.

Predicting multiple steps ahead reduces the number of model rollouts but does not necessarily reduce prediction errors. The paper does not fully explain when or why coarse-grained prediction should outperform fine-grained prediction.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

You are right about the subgoal loop; what the ranking measurement shows and does not show

Thank you. We agree with your concerns; details below, in your order. The main limits (one environment, a missing comparison described below, the organisation of the paper) need more than the discussion period allows, so we have not uploaded a revised PDF.

Deterministic, static mazes. Agreed. Every result is on one maze family and one released model pair. Whether the coarse predictor stays reliable under stochastic or more complex dynamics is untested; Table 7 is a measurement on these mazes only.

The subgoal loop. You are right. With the coarse model the plan is not executed directly: its predicted states are subgoals that the low-level (fine) planner tracks, as in the released hierarchy. With the fine model there is no such loop. So the two columns of Table 2 differ in two things together, the predictor and the global-local decomposition, and no experiment in the paper separates them. Section 9 says this; the title and abstract do not. What the results support:

  • The planner does not need to search the learned macro-action space. With the loop and the coarse predictor in place, searching two numbers per macro-action (a held action, passed through the released action encoder) solves 19/20 and searching the full learned space 17/20, not distinguishable at 20 trials.
  • The low-level planner is not doing the work by itself. With the same loop and the same low-level planner, shuffling the two-number planner's macro-actions among its candidate plans gives 2/20 (Section 7). The content of the coarse plan matters.

What they do not show is whether a global plan made in another way, for example from states of the fine model's own long rollout, would serve the low-level planner as well. That comparison would separate "the coarse model" from "the decomposition", and it was not run. The supported statement is "not the search over the learned macro-action space; the coarse predictor together with its subgoal loop", which is less than the title says.

Does the ranking signal explain the gain? The paper does not show that it does. At 500 steps the coarse model's rank correlation with the true cost is 0.089 and the fine model's is −0.065 (Table 1). The difference is consistent across layouts (17 of 20, p=0.0012p = 0.0012), but "orders plans the way the environment does", the wording of our third contribution, goes beyond a correlation of 0.089. Four further limits (numbers not printed in the paper come from its per-layout measurements in the supplementary material):

  • The correlation is measured on random action plans, drawn as the flat planner draws its first samples, and not on the plans either planner converges to (Section 9).
  • Both columns of Table 2 plan 250 steps ahead. At 250 steps the two models are not distinguishable (+0.054, p=0.26p = 0.26, Table 1); the clear difference is at 500 steps only.
  • The numbers above compare the model's cost with the true straight-line distance to the goal. Compared with the true distance in latent space, the gap at 500 steps is +0.053 (Table 3, last row; p=0.03p = 0.03, 0.09 after the Holm correction over the four lookaheads). Also, the 64-plan and 256-plan estimates are two separate plan samples in 19 of the 20 layouts, not one nested sample as Appendix A says; in the 64-plan sample this gap is +0.007 (and the straight-line gap +0.193).
  • The plan that each model's cost ranks first is not measurably better with the coarse model. Measured as the share of the 256 sampled plans that are nearer the goal (in a straight line, at the same step) than that plan, lower being better: 50% (coarse) and 52% (fine) at 500 steps, which is what a plan drawn at random gives on average; 51% (coarse) and 34% (fine) at 250 steps.

So the ranking measurement does not account for the planning result.

When or why should coarse prediction be better? Agreed that predicting ten steps at once does not have to reduce error. On this pair it was measured directly: the two predictors are within 0.03 of each other up to 50 steps (cosine to the true latent state 0.675 and 0.679 at 50; the fine one slightly higher at 10 and 20), and at 250 steps the fine model is at 0.340 and the coarse one at 0.613 (Table 7, Appendix E). That shows it happens here. It does not say when it should happen in general, and the paper has no account of that; how to choose the prediction step is left open (Section 9).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Corrections found after our replies: the 2x2 of Table 2 was not matched in search budget or lookahead

On re-reading the run scripts and logs behind Table 2, we found that its 2x2 was not matched the way the paper says. Figure 1(a), Table 2, Section 7 ("Same horizon, same trials, same sample budget") and Section 8 ("depending on the model alone") describe the two columns as differing only in the model that rolls the plan (and, for the coarse model, the subgoal loop it drives). They also differed in two other ways:

  • Search budget: the fine-model cells searched with 250 samples per planning call, the released flat planner's budget. The coarse-model cells searched with 1000 samples at the upper level, and their subgoals were tracked by the released lower-level planner with 1000 samples of its own.
  • Lookahead: the fine-model cells planned 250 steps ahead. The coarse-model cells planned up to 47 ten-step macro-actions (470 steps), not 25 as Table 2 and Figure 1 state.

Each column also used its own level's planner settings and cost, and the scramble control and the 40-trial re-run used the same coarse-model settings. So every comparison across the two columns, including 5/20 against 19/20, 2/20 against 17/20, 9/20 against 17/20 and 7/40 against 37/40, changes budget and lookahead as well as the model and its subgoal loop. The paper's own results bear on these factors without settling them: the flat planner given twenty times its samples went from 11/20 to 13/20, and the held-action class through the fine model from 4/20 to 0/20 with ten times the samples (Section 3); the hierarchy capped at 25 macro-actions instead of 47 scored 16/20 against 17/20 (Appendix E). A comparison matched in both was not run.

The bottom line: Table 2 does not separate the model from search budget and lookahead, so crediting the model with the gap between 5/20 and 19/20 (abstract, Contribution 1, Figure 1, Conclusion) is not established. Comparisons within the coarse column are not affected: held actions against the full learned space (19/20 against 17/20) and the scramble control against its intact cell (2/20 against 19/20).

Our replies repeated the paper's description in several places, and each needs the same addition, "a larger search budget and a longer lookahead". To Reviewer kanq, Table 2's coarse cells do not plan 25 macro-actions, and "the ten-step predictor together with its subgoal loop" also had the larger budget and lookahead. To Reviewer He48, the two columns differ not only in the predictor and the decomposition but also in search budget, lookahead and planner settings, and they do not both plan 250 steps ahead; at the nearest lookaheads in Table 1 (250 steps for the fine cells, 500 for the coarse cells' 470) the rank correlation is low for both models (0.072 and 0.089, position cost, 256 plans), so our conclusion that the ranking measurement does not explain the planning result stands. To Reviewer joTx, the comparison held the action class fixed but changed the predictor, its subgoal loop, the search budget and the lookahead together.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.