acceptodds
Under review as a conference paper at ICLR 2027

LeWM++: Repairing the Planning Interface of Latent World Models

Abstract

Latent world models enable visual control without reconstructing pixels, yet remain unreliable over long horizons. We identify three planning mismatches that worsen with goal offset: target mismatch between a distant goal and a short reliable rollout; search mismatch between sparse effective action sequences and uninformed sampling; and cost mismatch between goal-reaching semantics and restrictive terminal-only cost. Addressing these failures requires a planning interface that effectively translates latent representations into actions. We introduce LeWorldModel++ (LeWM++), a planning-interface framework built on a frozen latent world model. LeWM++ comprises three complementary components guided by a central principle: preserving global intent through the final goal while using a local target to drive short-horizon planning. LatentPathFlow generates a temporally aligned latent path and uses its endpoint as the local target; a final-goal-conditioned Action Chunk Prior guides CEM search with temporally coherent action-chunk proposals; and Min-over-Horizon scores each rollout by its closest approach to the local target. On the LeWM Control Suite, LeWM++ achieves the highest average success at all four evaluated goal offsets; as goal distance increases, its lead over LeWM grows from 12.0 to 32.3 percentage points. Across eight visual OGBench datasets, LeWM++ achieves 50.62% average success, exceeding the model-based planning method Latent Diffusion Planning (LDP) by 34.2 percentage points. On two real-robot tasks, LeWM++ outperforms goal-conditioned Diffusion Policy by 18.9 percentage points, demonstrating its control effectiveness under the hardware protocol. These results show that effective latent planning depends not only on predictive dynamics, but also on aligning the planner's target, proposal, and cost.

Primary area:
applications to robotics, autonomy, planning
Keywords:
JEPA, Latent World Models, Long-Horizon Planning, Planning Interface, Hierarchical Planning, Goal-Conditioned Planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EuEHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long-horizon, goal-conditioned planning with a frozen latent world model (LeWM). The authors argue that planning failures worsen as the goal offset grows, and attribute this to three "mismatches": a target mismatch, a search mismatch, and a cost mismatch. They then propose LeWM++, an external planning layer with three components. LatentPathFlow uses flow matching to generate a short latent path and takes its endpoint as the local target. The Action Chunk Prior is a goal-conditioned action-chunk policy trained with IQL/AWR, used to initialize CEM or to inject candidates into it. Min-over-Horizon scores each rollout by its closest distance to the local target. Experiments cover the LeWM Control Suite at multiple goal offsets, visual OGBench, the DINO-WM deformable-object tasks, and real-robot tasks, together with component ablations and sensitivity analyses.

Critical strengths
  1. The problem is meaningful, and the framework itself has value. How planning can be improved once the world model is frozen is a question worth studying. Designing planning as a layer outside the world model has a certain degree of generality, and the paper applies the same interface to two world models, LeWM and DINO-WM. The goal-offset analysis on the Control Suite clearly supports one part of the argument: the gap between the full method and the variant without LatentPathFlow grows steadily with goal offset, which shows that local targets are genuinely useful for distant goals.

  2. Substantial experimental effort. The evaluation spans several simulated benchmarks and real-robot tasks, with fairly complete ablations and runtime measurements.

Critical weaknesses
  1. Much of the experimental work demonstrates that the framework is effective, rather than directly supporting the core claim. The abstract asserts that all three mismatches worsen with goal offset, but the paper does not measure any of them directly; the main evidence is how much performance drops when the corresponding component is removed. In Table 10, only the gap from removing LatentPathFlow grows consistently with goal offset; the gaps from removing the Action Prior or MoH stop growing after H = 50 and even shrink. For the search and cost mismatches, the evidence only shows that the corresponding components help. It does not show that these mismatches exist as described, or that they worsen with distance.

  2. The framework design is overly complex, which reduces its contribution. The framework contains several models that must be trained: each environment or dataset requires separately training LatentPathFlow, a value function, two chunk critics, and a policy, and moving to DINO-WM requires retraining all of them in the new latent space. How the Action Prior is used also differs considerably across benchmarks. All of this adds complexity and weakens the generality that an "external planning layer" should provide.

Questions for authors
  1. Naming of the method. The name "LeWM++" suggests an improved version of LeWM, yet the LeWM++ planning interface does not modify the world model (which is kept frozen), and the same interface is also applied to DINO-WM (as DINO-WM++). Could the authors explain what "++" refers to, and how the framework relates to LeWM? If the intent is a planning layer independent of any specific world model, the current name may lead readers to take it as a new version of LeWM.

  2. Source of baseline numbers. Some baseline results appear to be taken directly from prior papers without attribution: the DINO-WM results in Table 1 match Figure 6 of the LeWM paper, and the DreamerV3 and TD-MPC2 results in Table 3 match the DINO-WM paper. Other baseline results differ noticeably from those reported in the original papers: the LeWM result on PushT in Tables 1 and 10 is substantially lower than in the LeWM paper, and the DINO-WM result on Rope in Table 3 is clearly worse than in the DINO-WM paper. Please clarify which results are cited and which were re-run, explain these discrepancies, and provide comparisons against the originally published results.

  3. Implementation of LeWM. Appendix A.2 states that the LeWM used in this paper is a JAX re-implementation with an IMPALA-small encoder, whereas the original uses a ViT encoder. Is the LeWM baseline this same version, and is it representative of the original LeWM? This matters especially on OGBench, where LeWM fails almost completely.

  4. Direct execution of the Action Prior. This baseline is reported only on OGBench and DINO-WM, not on the Control Suite or the real robot. Without it, one cannot tell how much of the improvement comes from the planning interface and how much from the offline-learned policy, particularly in settings where most CEM candidates are sampled from the prior.

  5. Choice of K_π. The number of prior-sampled candidates differs considerably across benchmarks. How was it chosen, and how sensitive are the OGBench and real-robot results to it?

Minor issues:

  • The averages of the full method at H = 50 and H = 75 are inconsistent between Tables 10 and 15 on the one hand and Table 13 on the other.
  • The "w/o Action Prior" result in Table 1 differs from the "zero initialization" result in Figure 6(a). Are these the same setting?
  • The same ablation is referred to as "w/o LatentPathFlow", "w/o Subgoal", and "w/o Subgoal Path"; the caption of Table 5 is unclear.
  • Section 4.6 states that Table 1 and Figure 3 establish the contributions of the three components, but Table 1 has no result without LatentPathFlow and Figure 3 lacks the other two ablations; these results appear only in Table 10.
Reviewer_7mQmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LeWM++, a framework for improving planning with frozen latent world models. It combines LatentPathFlow for local subgoal generation, an Action Chunk Prior for guiding CEM search, and Min-over-Horizon scoring for trajectory evaluation. Experiments cover several simulation benchmarks and two tasks on real robots.

Critical strengths

1.Practical Design: The method addresses how to use a fixed world model more effectively for control, without retraining its representation or dynamics. 2.Promising Results: At comparable planning latency, LeWM++ achieves 97.0% average success on the Control Suite at H = 25, compared with 77.7% for LeWM with five CEM iterations. This provides useful evidence of improved planning performance. 3.Substantial Experimental Work: The paper includes comparisons between target generators with matched parameter counts, component ablations, evaluation with DINO-WM, and quantitative results on real robots.

Critical weaknesses

1.Incomplete Evaluation of Path Generation: Table 5 supports LatentPathFlow over EndpointFlow at H = 25, but this comparison is missing at larger goal offsets. The longer-offset ablations remove the subgoal generator entirely, so they do not show whether predicting a path remains better than predicting only its endpoint. 2.Unclear Baseline Protocols: The paper gives general evaluation settings but does not clearly identify which baseline results were reproduced and which were taken from previous papers. The configurations used for SAGE and GC-IDM also need clarification before their performance differences can be properly assessed. 3.Limited Gains on More Complex Tasks: On both triple-cube datasets, LeWM++ performs almost identically to directly executing the Action Prior. The overall OGBench improvement is useful, but the paper does not sufficiently explain why planning provides little benefit on these more complex compositional tasks.

Questions for authors

1.Path Generation Ablation: Could the authors compare EndpointFlow and LatentPathFlow on PushT and Cube at H = 100? Please keep the Action Prior, MoH objective, local planning horizon, and CEM settings unchanged. 2.Baseline Settings: Please identify which baseline results were reproduced and report the backbone and evaluation protocol used for SAGE and GC-IDM. For SAGE, please also specify the replanning interval and search budget. If the protocols differ materially, a comparison on representative tasks under a common evaluation protocol and comparable computational budget would be helpful. 3.Failure Analysis: Could the authors analyze a sample of failed triple-cube episodes and report the main failure types and their frequencies? Examples showing the generated subgoal, predicted outcome, and actual outcome would help explain why planning adds little over the Action Prior.

Reviewer_t4TTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long-horizon planning with frozen latent world models and identifies three types of mismatch: target, search, and cost. It proposes LeWM++, which uses LatentPathFlow to generate local latent paths and subgoals, an Action Chunk Prior conditioned on the final goal to guide CEM, and Min-over-Horizon (MoH) to score candidate trajectories. The method is evaluated in both simulation and real-world robotic tasks.

Critical strengths
  1. Well-motivated approach. The analysis of failure modes in latent world model planning is convincing and provides a reasonable motivation for MoH, LatentPathFlow, and the Action Chunk Prior. The design of MoH is particularly interesting.

  2. Extensive experimental evaluation. The authors report extensive simulation experiments across multiple tasks, along with ablation studies, real-world robotic evaluations, and additional latency measurements. These results demonstrate the method’s strong performance.

Critical weaknesses
  1. Limited novelty. Compared with existing approaches based on action priors and subgoal planning, particularly SAGE as discussed in the paper, the proposed method primarily represents a systematic integration of existing ideas. This integration clarifies the roles of the individual components and enables better performance, but the methodological novelty appears limited.

  2. The role of MoH requires further clarification. Although Table 1 includes an ablation without MoH, the paper lacks a comparison with an alternative that omits MoH and instead makes the predicted trajectory track the entire generated latent path point by point. Such a comparison would help determine whether an endpoint target combined with MoH is more suitable for planning than directly tracking the generated path.

  3. The role of the action prior needs a clearer explanation. In particular, the authors use substantially different strengths of prior guidance across tasks. For the four tasks in Table 1, the prior mean initializes the first action chunk. In contrast, at each CEM iteration, the first action chunks of 200 out of 300 candidates on OGBench and 256 out of 280 candidates in the real-world experiments are sampled from the prior. The rationale for these different guidance strengths deserves further explanation.

Questions for authors
  1. Add a comparison that removes MoH and instead makes the predicted trajectory track the entire generated latent path point by point. This would help establish whether the endpoint-target-plus-MoH formulation is preferable to directly tracking the path.

  2. Further explain the role of the action prior across different tasks. Given that the prior alone already provides substantial capability, on which tasks does additional planning offer meaningful improvements, as shown in Table 2, and why? An analysis of these task-dependent benefits would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.