acceptodds
Under review as a conference paper at ICLR 2027

HRP: Hierarchical Residual Prediction for Latent World Models

Abstract

A world model is only useful for long-horizon planning if the agent can search it, and search is only affordable if most of the search can be skipped. One that predicts every step the same way gives no indication of which steps matter, so a hierarchy built on it runs its coarse level regardless of need. We introduce Hierarchical Residual Prediction, a latent world model whose upper level predicts the lower level's error instead of the future. The correction it produces is near zero wherever the lower level is already accurate, and because it is a vector rather than a score it carries how the lower level failed and not only that it failed. Dynamics the agent does not control produce corrections as large as a decision point does, so the size of the correction identifies decision points no better than chance, while the way it varies with the action identifies them reliably. Decision points therefore emerge from offline, reward-free transitions with no labels, rewards, or tuned sparsity, and restricting coarse search to them improves long-horizon success while reaching the best success of prior hierarchies at 2.4x fewer model rollout steps. The construction recurses, and a third level commits at timescales the second cannot reach.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, hierarchical planning, temporal abstraction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bWA1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Hierarchical Residual Prediction (HRP), a latent world model in which the upper level predicts not the future, but the prediction error (residual) of the lower level's k-step rollout. The residual is sparse without any sparsity penalty — it vanishes wherever the lower level is already correct — and, being a vector, it records how the lower level failed rather than merely that it failed. Building on this, the paper proposes two readings of the residual: its magnitude locates the states where the lower level fails, while its dependence on the action (clusterability) separates genuine decision points, where the agent's action settles the outcome, from uncontrolled branching driven by dynamics the agent cannot control. The planner escalates to coarse search only where the residual is both large and clusterable. Experiments span navigation, manipulation, and open-ended survival tasks of varying difficulty; restricting coarse search to the detected decision points improves long-horizon success while reaching the best success of prior hierarchies at 2.4× fewer model rollout steps.

Critical strengths
  1. The overall concept of Hierarchical Residual Prediction for latent world models is novel.
  2. The key contributions are each supported by fairly detailed experimental validation.
Critical weaknesses
  1. The paper is difficult to read, and several key concepts are not explained clearly or rigorously — "decision points" being a prominent example.
  2. The entire motivation rests on the premise that a level 1 trained with squared-error regression necessarily outputs the mean of possible successors (the "averaged state"). If level 1 were instead a stochastic or distributional dynamics model, it could sample from the branches directly, potentially rendering the residual signal unnecessary. The paper only compares against deterministic hierarchical methods and lacks this class of closely related baselines, which leaves the claim that "the residual is a necessary signal" incompletely supported.
  3. The method's premise is that prediction error is concentrated at a small set of states and that the agent's action resolves them. The authors themselves note: "On Push-T, contact makes the dynamics branch continuously, mean clusterability is 0.04 against 0.29 at GraphMaze junctions, and HWM stays ahead at every horizon." In other words, the method is currently effective only for environments with discrete branching — precisely the class of problems where hierarchical planning already works reasonably well — while offering no benefit for the genuinely difficult case of continuous contact-rich manipulation.
Questions for authors
  1. How does the set of detected decision points change with the training level / capacity of level 1, and with the environment?
  2. Does the failure on Push-T stem from an architectural limitation of the regression-based level 1, or from a limitation of the estimator? Could distributional or generative predictions (e.g., quantiles, flow matching) for Δ or for level 1 extend residual-based detection to continuously branching environments?
  3. The Crafter evaluation provides goals in a topological order of the achievement dependency graph. What is HRP's score without this guidance? How much of the 9.2% comes from the hierarchical model itself, and how much from the goal ordering?
  4. Regarding the claim of "separating real decision points from random events that the agent cannot control": a comparative analysis against methods such as ICM (intrinsic curiosity model), which are known to be attracted to uncontrollable stochasticity, would make the design choices of HRP considerably clearer.
Reviewer_nyvuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Hierarchical Residual Prediction (HRP), a hierarchical latent world model. As in prior hierarchical world models, the lowest level predicts the next latent state of the underlying decision process, whereas a higher level predicts only every kk-th state. The key innovation is that the higher level predicts vector-based residual errors of the latent states obtained from kk-step rollouts of the lower level. According to the authors, this makes it possible to detect decision points as states with elevated, action-dependent errors. The authors further propose a planning strategy in which a coarse search focuses on the detected decision points. Across several simulated control environments, the authors analyze the high-level predictions and how they can be clustered, and report improved planning performance over two hierarchical world-model baselines. Additionally, they show that the approach is not limited to two levels and can be extended to a three-level hierarchy.

Critical strengths
  • The authors identify a gap in hierarchical world-modeling research and propose a novel idea that partially addresses it
    The authors survey the landscape of hierarchical world models and identify two general approaches, each with key limitations. Either the higher-level predictions are tied to a fixed stride, which makes them inherently dependent on the choice of this window, or adaptive temporal abstractions rely on heuristics such as sparsity regularization to detect abstraction boundaries, and these heuristics cannot distinguish stochasticity from action-dependent decision points. Their approach of predicting vector-valued residual errors at the higher level is interesting and partially addresses this gap. Although the higher level still operates at a fixed stride, its predictions become trivial (zero vectors) wherever the lower level suffices. Thus, the higher-level predictions are non-zero only when the lower level actually fails. That said, the gap is only partially addressed, since the higher level still relies on a fixed temporal-stride hyperparameter, and it is unclear how this stride affects performance.
Critical weaknesses
  • Key parts of the method cannot be understood from the main paper alone
    While the first part of the method section (Sec. 3.1–3.4) is sufficiently explained, I could not fully follow the latter part (Sec. 3.5–3.6). Importantly, this part describes a key contribution: how the hierarchical predictions are used to determine decision points and guide control, i.e., how they are used in practice. For example, the acronym EVF is never defined, and Eq. 8 is defined in terms of "Y for a reached level-2 state", whereas the higher level was previously defined in terms of residual-error predictions. This gap is only closed in App. D.

  • Detailed control results are limited to 2D navigation tasks
    Planning results are reported in detail only for GraphMaze and DiverseMaze-Slip. For Crafter, the paper reports a single score (9.2% vs. "2.1% for the best prior agent"), but neither the overall setup nor how the baseline results were obtained is adequately described in the main paper. Push-T is not used for control. This makes it unclear whether the method is restricted to 2D navigation tasks.

  • Qualitative results would aid understanding of the method
    Visual examples of identified decision points or high-level predictions would make the method much easier to understand.

Questions for authors
  1. Temporal stride kk: The authors argue that a fixed temporal stride for higher-level predictions is insufficient. However, their approach also relies on a fixed stride for the high-level decisions. What happens if a decision point occurs within the stride window? In the worst case, would it not take k−1k-1 steps for the higher level to detect it, and could this be problematic? Have you ablated the choice of kk and its effect on the model?

  2. Use of the method for control: Key aspects of how the method is used for planning are unclear. "Level 1 plans alone by sampling primitive action sequences, rolling them forward with f1f_1, and scoring the endpoints against the goal." How is this scoring implemented? "We escalate only when predicted residuals are large and clusterable[...]" What does escalation mean in practice, and how is it decided, e.g., by a fixed threshold? How does level 2 then inform low-level control? The paper states that "the selected macro-action fixes an implicit waypoint that level 1 then plans towards until the next escalation." It is not clear to me why this higher-level waypoint should help, or how the macro-action is determined.

  3. Representation collapse: How is representation collapse prevented in this architecture? The model can be viewed as a JEPA-style world model, and such models are often prone to representation collapse in practice. For level 2, the paper addresses this with SIGReg regularization. How is it addressed for level 1? Is this achieved through the TD-JEPA head? If so, why did you choose different regularization methods for the two levels? If not, how is collapse avoided at level 1?

  4. Decision points: The authors argue that a key advantage of this architecture is that it identifies decision points. However, "decision points" are never properly defined. What counts as a decision point? If it is something like a fork in a maze, why would the errors of f1f_1 accumulate specifically there? Since the lower-level model can distinguish the paths via the action sequences, it could in principle predict the different routes.

  5. Data source: In all experiments, the world models are trained on offline data. Collecting data with two different policies could introduce artificial decision points where the two policies disagree. How was the data collected, and does this affect how the hierarchy develops?

  6. CEM for discrete actions: GraphMaze has discrete actions, whereas the planner appears to use CEM, which typically assumes continuous actions. Why was CEM chosen over a discrete search method such as MCTS?

  7. Warm-up and ramp hyperparameters: Have these hyperparameters been ablated? How critical is the choice of λres\lambda_{\text{res}} and NwarmN_{\text{warm}}?

Reviewer_3u1qOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Hierarchical World Models (HWM) predict the next states at a fixed stride for high-level planning. The paper proposed an algorithm, Hierarchical Residual Prediction (HRP), which changes this high-level prediction step by predicting the residual error from low-level predictions. HRP invokes the higher level only when the residual error is large and the clusterability factor exceeds a threshold. HRP matches HWM at shorter horizons and outperforms it at longer ones with less computation, with the exception on Push-T.

Critical strengths
  1. The notion of clusterability is well-motivated and is insightful. Its connection to the law of total variance indicates what it measures.
  2. Figure 1 provides a nice description of the overall architecture.
  3. Tables 2-3 and Appendix K ablates the key components to isolate their impact on HRP.
  4. The paper also clearly points out its limitations and describes the future work adequately.
Critical weaknesses
  1. The paper mentions that when there are several possible futures, level 1 averages out the effect because of the MSE minimization. But level 1 is conditioned on the actions, and in deterministic environments, each state-action pair will have a single successor, and MSE will predict the correct successor for each action. Thus, the averaging comes from level 1 failing to reach that optimum.
  2. Even for the stochastic variants, the level-2 predictor remains a deterministic squared-error predictor, so it will learn the conditional mean residual per action. As a result, the noise in the slip cells should average out, and the predicted residual should remain small. Still in Table 3, the size-only gate, which uses the predicted residual, waste escalations at the slip cells.
  3. HRP's sparsity claim is a bit overstated. Similar to THICK's coefficient, the training-state magnitude (above the 80th percentile) and the clusterability threshold (0.15) are hyperparameters that directly affect the escalation frequency. Also, the top-decile residual mass evidence is more about concentration rather than sparsity.
  4. Most of the evidence relies on custom environments, and the gains become modest on established ones like DiverseMaze and negative on Push-T. OGBench is considered in the appendix, but it only shows that the level 1's representation is preserved. It doesn't show whether it aids in planning.
  5. Figure 7 compares the RND, Plan2Explore methods, and shows that the three levels of HRP are better than them. However, HRP is given goal info in topological order (Appendix O.3), which is indeed a task structure. RND and other no-reward baselines don't capture this task structure, so the comparison isn't apples-to-apples.
Questions for authors
  1. Figure 1 mentions "L3 and upward," but the maximum level tested in the paper is only L3. Does HRP get similar gains from higher levels, or does it saturate after some level?
  2. What is "expected" in the Flat (expected) label in Figure 3?
  3. In Line 37-38, "the upper level predictions on a fixed stride, which assumes uncertainty accumulates with elapsed time. However, that's not the case for most environments." -- Is it true? Errors will always compound along a rollout due to auto-regressive prediction; they will just be uneven. Also, this claim conflicts with the Push-T environment.
  4. Why is the TD-JEPA loss being used in the paper? How does it affect the representation? Is it impactful for planning? It will be helpful to add the loss term in the main text as well.
  5. How are the "80th percentile" and "0.15" values for the threshold decided?

Minor Suggestions:

  1. The encoder is ϕi\phi_i in the main text while it is iii_i in the appendix. It would be helpful if it were consistent throughout the paper.
  2. In the main text, diverse maze environments are cited from Wang et al. (2026), while in the appendix, they are cited from Neitz et al. (2018). Neitz et al. (2018) do not have a diverse maze environment in their paper. I believe it's a typo.
  3. Typo in Line 200: recieves -> receives

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.