acceptodds
Under review as a conference paper at ICLR 2027

Learning Latent Waypoints for World-Model Planning

Abstract

Latent world models enable efficient planning by evaluating candidate actions in compact representation spaces, but terminal goal-matching costs provide little information about useful intermediate states. We propose Latent Residual Waypoints, a lightweight method for generating intermediate waypoints between start and goal latents. Given a frozen, pretrained world-model encoder, our method predicts encoded intermediate frames by learning a residual correction to latent linear interpolation. During planning, we generate a sequence of latent waypoints recursively. These are incorporated through a minimum-over-time waypoint cost that encourages trajectories to pass near future waypoints while still optimizing for the final goal. This enables effective planning in difficult settings, such as planning with budget constraints. A planner solving episodes of the Two-Rooms and Cube tasks under a constrained CEM budget can outperform full-budget final-goal planning. At matched budgets, the performance deltas are even more pronounced. We also find that our method improves Le-WorldModel's CEM planning performance across navigation and manipulation tasks and outperforms simpler waypoint methods like linear interpolation and nearest-neighbor snapping. We extend this to other world models and show improvements on gradient-based closed-loop MPC planning. Though our waypoints generally improve mean success rate compared to final-goal planning, they yield fewer gains on some easier tasks. Qualitative analysis of visualized latent waypoints and nearest-neighbor analyses further show that Latent Residual Waypoints remain closer to realistic trajectory latents than those produced by naive interpolation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, planning, linear interpolation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ph31OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This study proposes latent residual waypoints to address the problem of existing world model planning, which relies solely on goal-matching costs and fails to provide information about intermediate states. Given an initial image and a goal image, a lightweight model is trained to predict the residuals between the linear interpolation in latent space and the actual intermediate latents obtained from the data. During planning, this model constructs an intermediate trajectory based on the initial and goal images and guides the model to follow it. It demonstrated a higher success rate compared to goal-matching-only methods and simple waypoint methods in tasks such as Cube, PushT, and Two-Rooms.

Critical strengths
  • The fact that goal-matching-only costs cannot provide useful intermediate information seems to be a valid motivation, and there is actually a performance improvement.
Critical weaknesses
  • The scope of comparison and verification is too narrow, making it difficult to assess the advantages and generality compared to existing methods. I understand this work to be about creating subgoals, and to my knowledge there are many related baselines as mentioned in the paper, I wonder if this can be compared to them such as hierarchical planning using subgoals. The experiment is also too simple, leaving doubts about its applicability in more complex simulation environments and real-world scenarios. It is difficult to tell when it is helpful and when it is not.

  • I wonder whether the proposed method involves a trade-off where it improves in-distribution performance at the expense of planning flexibility. Although the paper separates training and evaluation episodes, the start-goal pairs used for evaluation seem to be sampled from the existing dataset trajectory, so it is unclear whether the proposed method is effective in situations requiring paths significantly different from the training data. Since the world model learns local transitions such as (state, action) -> next state, it has the potential to handle unfamiliar start-goal combinations. However, since the start-goal is just input for the waypoint prediction model, it is unclear if it will work well if there are no similar start-goals in the data or if the environment changes even slightly. In short, wouldn't the generalization capabilities of world models and latent waypoint predictor be too different? World model planning works demonstrate a certain level of performance even in unseen environment configurations [1-2], so I wonder if the proposed method can operate in such situations. In particular, unreachable intermediate states may arise in such situations, so it seems beneficial to address this aspect.

[1] Zhou et al., DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning, ICML 2025

[2] Zhang et al., Hierarchical Planning with Latent World Models, CoRL 2026

Questions for authors
  • There is insufficient justification and ablation regarding the methodology design. For example, I do not understand why linear interpolation and residual prediction are necessary; why can't we just predict waypoints directly? Also, I do not understand why both MSE loss and cosine loss are required; what happens if one is missing?

  • Can you compare the proposed method to existing subgoal generation and hierarchical planning methods (e.g., [2])?

  • Can you demonstrate applicability of the proposed method in more complex environments and real-world applications?

  • Can you show performance improvement of the proposed method in unseen environment configurations (e.g., a wall changing in two rooms)? Are there any issues arising from the difference in generalization capabilities between the world model and the latent waypoint predictor? Can you analyze whether the subgoals created in such situations are still reachable?

4 comments

We thank the reviewer for their comments and experiment suggestions, and address each individually below.

The scope of comparison and verification is too narrow, making it difficult to assess the advantages and generality compared to existing methods. I understand this work to be about creating subgoals, and to my knowledge there are many related baselines as mentioned in the paper, I wonder if this can be compared to them such as hierarchical planning using subgoals. The experiment is also too simple, leaving doubts about its applicability in more complex simulation environments and real-world scenarios. It is difficult to tell when it is helpful and when it is not.

Can you compare the proposed method to existing subgoal generation and hierarchical planning methods (e.g., [2])?

We thank the reviewer for this suggestion for additional comparisons and agree that they would be useful. To compare with HWM, we train a version adapted to Le-WorldModel, following [1], labelled Hi-LeWM and additionally compare with the stronger method they introduce, Hi-C, which uses retrieval of action sequences seen during training. For a fair comparison, our trained Hi-LeWM models are models parameter-matched to the Latent Residual Waypoints mode and trained using data with the same horizon as the latter’s training data. Results are shown below:

EnvironmentHorizonLatent ResidualHi-LeWMHi-C
Cube2583.80 ± 4.4769.00 ± 5.3285.67 ± 4.59
5060.40 ± 7.2353.73 ± 6.8060.20 ± 6.84
7568.60 ± 4.5365.60 ± 4.7672.47 ± 3.73
10070.40 ± 6.1067.13 ± 7.0871.93 ± 5.47
PushT2589.40 ± 4.2288.80 ± 4.4984.47 ± 4.06
5049.60 ± 6.1752.20 ± 4.1047.47 ± 4.93
7524.60 ± 9.2927.00 ± 7.9725.73 ± 5.90
10012.40 ± 4.3017.80 ± 4.6016.00 ± 4.00
TwoRoom2589.80 ± 4.4782.80 ± 4.2198.60 ± 1.24
5066.60 ± 6.3351.40 ± 6.4489.73 ± 2.31
7535.80 ± 5.2028.67 ± 5.2463.47 ± 7.06
10018.60 ± 5.5013.47 ± 4.6839.67 ± 4.91

We find that our method outperforms Hi-Le-WM for all four horizons considered on both Cube and Two-Rooms environments. Additionally, we find that our method even sometimes exceeds the performance of Hi-C, without using retrieved sequences.

[1] Caselli, Niccolò, Francesco Massafra, Samuele Punzo, Salvatore Lo Sardo, Ippokratis Pantelidis, and Sathya Kamesh Bhethanabhotla. “Mind the Gap: Promises and Pitfalls of Hierarchical Planning in LeWorldModel.” World Modeling at Booth 2026 Workshop. https://arxiv.org/abs/2607.12547.

I wonder whether the proposed method involves a trade-off where it improves in-distribution performance at the expense of planning flexibility. Although the paper separates training and evaluation episodes, the start-goal pairs used for evaluation seem to be sampled from the existing dataset trajectory, so it is unclear whether the proposed method is effective in situations requiring paths significantly different from the training data. Since the world model learns local transitions such as (state, action) -> next state, it has the potential to handle unfamiliar start-goal combinations. However, since the start-goal is just input for the waypoint prediction model, it is unclear if it will work well if there are no similar start-goals in the data or if the environment changes even slightly. In short, wouldn't the generalization capabilities of world models and latent waypoint predictor be too different? World model planning works demonstrate a certain level of performance even in unseen environment configurations [1-2], so I wonder if the proposed method can operate in such situations. In particular, unreachable intermediate states may arise in such situations, so it seems beneficial to address this aspect.

Can you show performance improvement of the proposed method in unseen environment configurations (e.g., a wall changing in two rooms)? Are there any issues arising from the difference in generalization capabilities between the world model and the latent waypoint predictor? Can you analyze whether the subgoals created in such situations are still reachable?

We thank the reviewer for this suggestion. To assess how well our method generalizes, we evaluate on Two-Rooms tasks where the door is moved as well as Cube tasks where the cube is made smaller. We find that the underlying Le-WorldModel has some difficulty generalizing to these scenarios, as they are outside of training distribution, but that Latent Residual Waypoints improve planning performance.

EnvironmentConditionDirect planning (%)Latent Residual (%)
CubeCross-episode goals15.6 ± 5.519.4 ± 4.6
2 cm cube58.2 ± 6.262.2 ± 5.2
3 cm cube60.6 ± 4.979.2 ± 6.9
4 cm cube — matched control67.8 ± 5.882.4 ± 5.0
Two-RoomOriginal — control82.8 ± 6.489.2 ± 4.3
Cross-episode goals58.2 ± 9.160.0 ± 8.7
Moved door54.6 ± 9.366.0 ± 5.5

There is insufficient justification and ablation regarding the methodology design. For example, I do not understand why linear interpolation and residual prediction are necessary; why can't we just predict waypoints directly? Also, I do not understand why both MSE loss and cosine loss are required; what happens if one is missing?

We compare against direct intermediate-latent prediction below:

EnvironmentGoal offsetLatent ResidualDirect intermediate prediction
Cube5060.40 ± 7.2360.00 ± 6.98
7568.60 ± 4.5369.40 ± 3.93
10070.40 ± 6.1071.00 ± 5.26
PushT5049.60 ± 6.1751.20 ± 4.57
7524.60 ± 9.2925.87 ± 6.54
10012.40 ± 4.3013.20 ± 5.86
TwoRoom5066.60 ± 6.3365.73 ± 5.71
7535.80 ± 5.2036.93 ± 5.38
10018.60 ± 5.5019.07 ± 4.97

We find direct intermediate prediction performs comparably to the more complex latent residual prediction. We will update the final version of our paper to reflect the simpler method. Regarding ablations for the individual loss terms, we agree with the reviewer that these additional ablations would be helpful, but due to computational constraints it is unlikely that they will finish before the end of the rebuttal period, and will be reflected in the camera-ready version of our paper if accepted.

Can you demonstrate applicability of the proposed method in more complex environments and real-world applications?

Our evaluations cover a mix of manipulation and navigation tasks, and are fairly standard in the world modeling literature. Our Cube task is more realistic insofar as it involves utilizing a simulated robotic arm. That being said, we agree with the reviewer that more complex settings would be interesting to include and may explore this in follow up work. The goal of our current work is to propose and validate a simple approach to generating waypoints for latent world model planning.

I appreciate the authors' effort in the rebuttal, but I have some additional concerns after reading it.

  1. Hi-LeWM seems to be a weak baseline, as it underperforms flat LeWM on PushT in the original paper. Also, Latent Residual does not consistently outperform it (e.g., on PushT).
  2. If direct prediction performs comparably and the authors will update the manuscript with this simpler version, then what is the novelty of this work?

Thank you for acknowledging our additional experiments.

Hi-LeWM seems to be a weak baseline, as it underperforms flat LeWM on PushT in the original paper. Also, Latent Residual does not consistently outperform it (e.g., on PushT).

We note that in our experiments, Hi-LeWM does outperform flat Le-WM on PushT (the Final Goal planning baseline mean success rate is 87.60 in our experiments). That being said, we do believe that this tests a natural hierarchical method of generating waypoints, and the results show that a much simpler method often performs better. Regarding PushT, we recognize (and acknowledge in the paper) that PushT is the task where our method is weakest.

If direct prediction performs comparably and the authors will update the manuscript with this simpler version, then what is the novelty of this work?

Our contributions do not depend solely on the residual parameterization. First, we provide a novel method of integrating waypoints with the planner, using a minimum-over-time cost and a projection-based progress measure. Additionally, we show that a simple method can produce waypoints which offer significant benefits in constrained planning settings, most strongly on Cube and Two-Rooms, and that match oracle-trajectory waypoints on Cube. Finally, we show that the produced waypoints are more visually coherent when decoded than those produced by linear interpolation and also lie closer to the data manifold. The ablation on the linear interpolation input shows that no prior on the geometry of the latent path is required and simplifies the resulting method.

Let us know if this addresses your concerns or if there are more concerns we can address before the end of the rebuttal period.

Reviewer_QxDSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a method to learn intermediate waypoints for planning with a frozen world model. A small network corrects linear interpolation between start and goal embeddings, using intermediate observations from training trajectories. At test time, it generates waypoints recursively and adds a cost that encourages the planner to pass near them. Experiments cover navigation and manipulation with three world models, CEM, and gradient-based planning. The largest gains occur on Cube, and improvements persist under limited planning budgets.

Critical strengths
  • The improvement in difficult planning settings is meaningful. On Cube, CEM success increases from 66.6% to 86.0% (Table 1). With only 75 samples and 10 optimization steps, the method also exceeds final-goal planning with 300 samples and 30 steps on Cube and Two-Rooms. This is useful evidence that intermediate guidance can help when search is limited.
  1. The method works with frozen encoders and is tested beyond one model or planner. The interpolation baselines use the same planning cost and tuning space. I also appreciate the reported cases with little benefit, including several open-loop settings. These comparisons help establish where the method is useful.
Critical weaknesses
  • I am unsure how much the residual formulation contributes beyond learning intermediate states. The waypoint controls are untrained interpolation and nearest-neighbor snapping. The MPC integration is useful, but what does correcting interpolation add over direct intermediate-latent prediction? Around the interpolation construction at lines 145–155, a brief comparison with optimal-transport and diffusion-based path construction would also help motivate this choice.

  • I am confused about the harder-task results with Figure 3b. Lines 322–323 report gains of 6.4–17 percentage points, while the CEM top30 curves appear near 72% and 41%, a gap of about 31 points. Table 10 gives bucket statistics but no success rates. Both accounts support an improvement, yet the discrepancy leaves its size uncertain.

The motivation should explain earlier why waypoints help when a dynamics model already supports planning. At lines 62–75, a simple example of limited-budget MPC would connect the problem to the later results and explain the trade-off introduced by the extra cost. Before Eqs. (7)–(12), summarize how progress is located and earlier waypoints are discarded. Also, lines 93–96 introduce decoded images soon after motivating reconstruction-free models. Appendix A, lines 674–680, explains the auxiliary decoder; state there in the main text that it is used only for visualization and does not train the planning encoder. These later answers should be available when the corresponding choices are introduced.

Questions for authors
  1. Is there a comparison with direct intermediate-latent prediction using the same capacity, data, planning cost, and tuning? This would clarify whether the residual structure contributes to the gains beyond the benefit of learned waypoints.

  2. What are the success rates underlying Figure 3b, and how do they match lines 322–323? Does the difference come from the figure, the prose, or different evaluation settings?

  3. Do any observed failures arise because the waypoint term in Eq. (12) favors a rollout with worse final-goal error? Eq. (11) also allows each waypoint to choose its closest rollout time independently. Could approaching waypoints out of order contribute to such failures? This would help clarify when the added guidance can be counterproductive.

2 comments

We thank the reviewer for their comments regarding our work, and address each individually below.

I am unsure how much the residual formulation contributes beyond learning intermediate states. The waypoint controls are untrained interpolation and nearest-neighbor snapping. The MPC integration is useful, but what does correcting interpolation add over direct intermediate-latent prediction? Around the interpolation construction at lines 145–155, a brief comparison with optimal-transport and diffusion-based path construction would also help motivate this choice.

Is there a comparison with direct intermediate-latent prediction using the same capacity, data, planning cost, and tuning? This would clarify whether the residual structure contributes to the gains beyond the benefit of learned waypoints.

We thank the reviewer for suggesting this comparison. We compare against direct intermediate-latent prediction below:

EnvironmentGoal offsetLatent ResidualDirect intermediate prediction
Cube5060.40 ± 7.2360.00 ± 6.98
7568.60 ± 4.5369.40 ± 3.93
10070.40 ± 6.1071.00 ± 5.26
PushT5049.60 ± 6.1751.20 ± 4.57
7524.60 ± 9.2925.87 ± 6.54
10012.40 ± 4.3013.20 ± 5.86
TwoRoom5066.60 ± 6.3365.73 ± 5.71
7535.80 ± 5.2036.93 ± 5.38
10018.60 ± 5.5019.07 ± 4.97

We find direct intermediate prediction performs comparably to the more complex latent residual prediction. We will update the final version of our paper to reflect the simpler method.

I am confused about the harder-task results with Figure 3b. Lines 322–323 report gains of 6.4–17 percentage points, while the CEM top30 curves appear near 72% and 41%, a gap of about 31 points. Table 10 gives bucket statistics but no success rates. Both accounts support an improvement, yet the discrepancy leaves its size uncertain.

What are the success rates underlying Figure 3b, and how do they match lines 322–323? Does the difference come from the figure, the prose, or different evaluation settings?

We thank the reviewer for pointing this out. The text in lines 322-323 contains a typo, and the larger gains shown in Figure 3b are correct. This will be corrected in the updated PDF.

The raw numbers are shown below:

Displacement SubsetCEM Final GoalCEM Latent ResidualAdam Final GoalAdam Latent Residual
Top 50%51.40 ± 8.3377.00 ± 5.9149.00 ± 6.0668.60 ± 7.60
Top 40%44.20 ± 7.6978.20 ± 4.2640.80 ± 6.5567.00 ± 5.52
Top 30%41.00 ± 5.2771.60 ± 5.4833.80 ± 6.2158.20 ± 7.63
Top 20%28.20 ± 6.0757.20 ± 8.0128.40 ± 5.7244.00 ± 7.54
Top 10%19.00 ± 3.5644.60 ± 5.9713.80 ± 6.3629.20 ± 8.18

The motivation should explain earlier why waypoints help when a dynamics model already supports planning. At lines 62–75, a simple example of limited-budget MPC would connect the problem to the later results and explain the trade-off introduced by the extra cost. Before Eqs. (7)–(12), summarize how progress is located and earlier waypoints are discarded. Also, lines 93–96 introduce decoded images soon after motivating reconstruction-free models. Appendix A, lines 674–680, explains the auxiliary decoder; state there in the main text that it is used only for visualization and does not train the planning encoder. These later answers should be available when the corresponding choices are introduced.

We thank the reviewer for these suggestions. We will update the PDF to make it clear in the main text that the decoder is only used for visualization and provide better motivation for why waypoints would help.

Do any observed failures arise because the waypoint term in Eq. (12) favors a rollout with worse final-goal error? Eq. (11) also allows each waypoint to choose its closest rollout time independently. Could approaching waypoints out of order contribute to such failures? This would help clarify when the added guidance can be counterproductive.

In our experiments, we do not find much evidence of either of these issues being a significant cause of failures.

As the discussion period is coming to an end, we would like to kindly follow up on our response above. Has our response sufficiently addressed the reviewer's concerns, or are there additional concerns we can address before the end of the discussion period?

Reviewer_gzWGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Latent Residual Waypoints, which is a lightweight method for generating intermediate waypoints between start and goal latens for goal-conditioned planning. With a 4 layer MLP, it predicts an intermediate latent as a residual on top of linear interpolation between current and the endpoint latent. The experiments use LeWM DINO-WM and ResNet world model on several benchmark. It show that the proposed mothed can enable planning in difficult scenarios and more effective planning.

Critical strengths
  1. A simple, plug-in and cheap method with large performance improvement. with small MLP is can boost leWM by 19.4, and by 6-17 on harder displacement buckets and all CEM budget in Table 9.

  2. The result is reported in a transparent manner. The hyperparameters are fairly tuned in same search space as baseline. Moreover the author presentation their mixed results.

Critical weaknesses
  1. The baselines are not fair. The proposed method was trained on the offline trajectories, so for a short 25-step goal, its just drawing a dense reference path. However on the other hand, the baselines like linear interpolation never got to learn from the trajectory data.
  2. the experiment does not match the claim of "long-horizon", and the gain is noisy. the paper is motivated around "long-horizon planning", but we only test 25 step goals, which is exactly our training length. Also some of the performance gains are so small that they are with in std.
  3. the claim is "more effective" is not fully supported, the author show that the proposed method need fewer samples/steps, but did not show the actual time it used.
Questions for authors

Some experiments the could change my assessment.

  1. Result with start-goal beyond the 25-step training segments like 50, 100 steps.
  2. Can you compared with HWM (arXiv 2604.03208) and a GCBC baseline on the same trajectories.

What is the planning-time of waypoint generation and how sensitive are the results to K, λ and p?

4 comments

We thank the reviewer for their helpful comments, and address each individually below.

The baselines are not fair. The proposed method was trained on the offline trajectories, so for a short 25-step goal, its just drawing a dense reference path. However on the other hand, the baselines like linear interpolation never got to learn from the trajectory data.

Our baselines are designed to test simpler methods of waypoint generation, to determine whether trained residual waypoints are really necessary for improved planning performance. We agree with the reviewer, however, that additional comparisons against methods which learn from trajectory data are necessary. We show results comparing our approach with additional trained methods for waypoint generation below.

the experiment does not match the claim of "long-horizon", and the gain is noisy. the paper is motivated around "long-horizon planning", but we only test 25 step goals, which is exactly our training length. Also some of the performance gains are so small that they are with in std.

Result with start-goal beyond the 25-step training segments like 50, 100 steps.

We agree with the reviewer that long horizon comparisons would be helpful, and we provide the requested comparisons below:

TaskGoal offsetFinal goalLinearLinear-snapLatent Residual
Cube5051.20 ± 5.1852.00 ± 6.5353.00 ± 7.3860.40 ± 7.23
7563.40 ± 5.5065.00 ± 4.8367.60 ± 4.7068.60 ± 4.53
10062.80 ± 7.3261.20 ± 7.5563.40 ± 6.1970.40 ± 6.10
PushT5040.20 ± 4.4741.00 ± 4.0342.20 ± 3.5849.60 ± 6.17
7523.40 ± 6.6721.40 ± 6.7423.80 ± 7.8624.60 ± 9.29
10013.80 ± 4.4712.60 ± 5.2511.20 ± 4.4412.40 ± 4.30
TwoRoom5055.60 ± 8.1054.80 ± 7.4455.00 ± 8.0766.60 ± 6.33
7530.60 ± 4.5329.00 ± 4.6427.80 ± 5.0335.80 ± 5.20
10014.60 ± 2.6713.80 ± 2.7414.20 ± 4.6618.60 ± 5.50

We find that Latent Residual Waypoints improve planning performance compared to Final Goal, Linear, and Linear-Snap Baselines in all cases aside from PushT with a horizon of 100. In addition, our method was only trained on horizon 25 sequences, demonstrating that our method can generalize effectively beyond its training data.

the claim is "more effective" is not fully supported, the author show that the proposed method need fewer samples/steps, but did not show the actual time it used.

To support our claim of more effective planning, we show that a world model equipped with our Latent Residual Waypoints can show strong performance under a restricted planning budget (in terms of CEM samples and optimization steps). In particular, for both Cube and Two-Rooms, we show that planning with Latent Residual Waypoints under the most restricted budget outperforms full-budget final goal planning. For the PushT task at each planning budget, planning with Latent Residual Waypoints performs better than Final-Goal planning. However, it appears that the reviewer is questioning whether our method enables faster planning in terms of wallclock time, which is a separate claim from its effectiveness. To address this issue, we compare the wallclock time of planning under the most restricted budget (75 CEM samples and 10 optimizer steps) using Latent Residual Waypoints with the wallclock time of full-budget Final Goal planning on a batch of 50 parallel episodes:

TaskRestricted-Budget Latent ResidualFull-Budget Final GoalSpeedup
Cube45.1 ± 5.9 s447.2 ± 43.8 s9.92×
PushT48.4 ± 6.4 s409.2 ± 48.0 s8.46×
TwoRoom47.8 ± 8.4 s427.3 ± 50.1 s8.94×

In the above table, for Latent Residual Waypoints we compute the wallclock time of the CEM solver, waypoint setup, and waypoint preparation (initializing the waypoint path, encoding the current observation, calculating progress, and constructing the ahead-waypoint masks and weights). For Final Goal planning, we compute the wallclock time of the CEM solver. Measurements were taken on L40S GPUs. Latent Residual Waypoints enable strong performance at a strong speedup in wallclock time.

Can you compared with HWM (arXiv 2604.03208) and a GCBC baseline on the same trajectories.

We agree with the reviewer that comparing with hierarchical methods and a trained policy could be helpful. For the comparison with GCBC, we emphasize that our method does not involve training a policy. As a result, we view the degree to which we can approach the performance of a trained policy as the measure of success for those experiments, since that describes how well our waypoints capture information relevant for planning.

To compare with HWM, we train a version adapted to Le-WorldModel, following [1], labelled Hi-LeWM and additionally compare with the stronger method they introduce, Hi-C, which uses retrieval of action sequences seen during training. For a fair comparison, our trained Hi-LeWM models are models parameter-matched to the Latent Residual Waypoints mode and trained using data with the same horizon as the latter’s training data. For our GCBC comparisons, we train policies on top of the pretrained Le-WorldModel checkpoints.

When comparing our method with hierarchical approaches, we find that our method outperforms Hi-Le-WM for all four horizons considered on both Cube and Two-Rooms environments. Additionally, we find that our method even sometimes exceeds the performance of Hi-C, without using retrieved sequences. Our method also outperforms GCBC on the PushT environment and 25-horizon Two-Rooms, but otherwise performs worse (Hi-LeWM often performs worse than GCBC as well). However, we emphasize that our method does not involve training a policy and is not a 1-1 comparison with GCBC.

EnvironmentHorizonGCBCLatent ResidualHi-LeWMHi-C
Cube2594.20 ± 2.4483.80 ± 4.4769.00 ± 5.3285.67 ± 4.59
5063.93 ± 6.0260.40 ± 7.2353.73 ± 6.8060.20 ± 6.84
7571.27 ± 3.9768.60 ± 4.5365.60 ± 4.7672.47 ± 3.73
10080.33 ± 3.6870.40 ± 6.1067.13 ± 7.0871.93 ± 5.47
PushT2587.33 ± 5.1789.40 ± 4.2288.80 ± 4.4984.47 ± 4.06
5043.20 ± 3.3849.60 ± 6.1752.20 ± 4.1047.47 ± 4.93
7520.93 ± 3.7524.60 ± 9.2927.00 ± 7.9725.73 ± 5.90
10011.93 ± 3.3512.40 ± 4.3017.80 ± 4.6016.00 ± 4.00
Two-Rooms2589.67 ± 1.9489.80 ± 4.4782.80 ± 4.2198.60 ± 1.24
5075.73 ± 4.4566.60 ± 6.3351.40 ± 6.4489.73 ± 2.31
7557.27 ± 4.1535.80 ± 5.2028.67 ± 5.2463.47 ± 7.06
10039.33 ± 3.9018.60 ± 5.5013.47 ± 4.6839.67 ± 4.91

[1] Caselli, Niccolò, Francesco Massafra, Samuele Punzo, Salvatore Lo Sardo, Ippokratis Pantelidis, and Sathya Kamesh Bhethanabhotla. “Mind the Gap: Promises and Pitfalls of Hierarchical Planning in LeWorldModel.” World Modeling at Booth 2026 Workshop. https://arxiv.org/abs/2607.12547.

What is the planning-time of waypoint generation and how sensitive are the results to K, λ and p?

Excluding start and goal embedding, we find that it takes 3.512 ± 0.121 ms per trajectory to perform inference for our Latent Residual Waypoints method. Measurements were taken on L40S GPUs

In terms of sensitivity, for Latent Residual Waypoints applied on top of Le-WorldModel, we found that performance is most sensitive to changes in the number of waypoints K and the weighting term λ\lambda, with larger swings on the Cube task than other tasks.

Thanks authors for the detailed extra experiments, most of my concerns have been addressed. I still have some concerns about the comparison with GCBC/Hi-LeWM/Hi-C since the new results are mixed. For now, I accept the authors’ explanation that this is not a one-to-one comparison.

For my last concern, could you provide some numbers?

Thank you for acknowledging our new experiments.

Below, we provide results for varying each parameter while holding the others fixed (using the tuned hyperparameters for each setting).

Varying K with λ\lambda and p fixed:

TaskPlannerFixed (λ,p)(\lambda, p)K = 3K = 4K = 5K = 8
CubeCEM(1, 1)77.6 ± 9.486.0 ± 4.288.4 ± 6.488.4 ± 5.6
Adam(1, 0.5)76.4 ± 6.779.0 ± 3.780.4 ± 4.581.2 ± 7.0
PushTCEM(0.5, 2)91.0 ± 2.589.8 ± 2.492.8 ± 2.390.4 ± 3.4
Adam(0.5, 0)85.8 ± 5.186.4 ± 4.487.2 ± 2.788.8 ± 4.8
TwoRoomCEM(1, 0)88.2 ± 3.290.0 ± 4.190.8 ± 2.590.6 ± 4.0
Adam(0.1, 0.5)93.0 ± 4.194.6 ± 4.094.4 ± 3.095.4 ± 2.8

Varying λ\lambda with K and p fixed:

TaskPlannerFixed (K,p)(K, p)λ = 0.1λ = 0.3λ = 0.5λ = 0.7λ = 1
CubeCEM(8, 1)73.0 ± 7.079.6 ± 8.281.6 ± 7.885.6 ± 5.188.4 ± 5.6
Adam(8, 0.5)71.2 ± 8.773.4 ± 7.077.2 ± 5.879.2 ± 5.881.2 ± 7.0
PushTCEM(5, 2)90.2 ± 3.690.0 ± 5.092.8 ± 2.390.0 ± 3.789.0 ± 3.9
Adam(8, 0)85.2 ± 5.885.6 ± 5.388.8 ± 4.887.2 ± 4.686.6 ± 4.3
TwoRoomCEM(5, 0)85.4 ± 3.986.8 ± 4.589.2 ± 3.089.4 ± 4.190.8 ± 2.5
Adam(8, 0.5)95.4 ± 2.890.2 ± 4.492.0 ± 4.289.8 ± 4.691.6 ± 3.6

Varying p with λ\lambda and K fixed:

TaskPlannerFixed (K,λ)(K, \lambda)p = 0p = 0.5p = 1p = 2
CubeCEM(8, 1)86.2 ± 4.987.4 ± 6.388.4 ± 5.687.8 ± 5.7
Adam(8, 1)80.8 ± 6.481.2 ± 7.080.2 ± 6.479.6 ± 7.0
PushTCEM(5, 0.5)92.2 ± 2.791.4 ± 3.890.6 ± 3.792.8 ± 2.3
Adam(8, 0.5)88.8 ± 4.887.2 ± 3.486.2 ± 3.888.4 ± 4.0
TwoRoomCEM(5, 1)90.8 ± 2.590.6 ± 4.289.6 ± 3.690.4 ± 3.7
Adam(8, 0.1)94.0 ± 2.895.4 ± 2.895.2 ± 3.494.6 ± 2.3

For comparison, the baseline Final Goal planning results for each setting are below:

TaskCEMAdam
Cube66.60 ± 4.4367.00 ± 4.83
PushT87.60 ± 6.5283.20 ± 4.34
TwoRoom83.20 ± 4.9293.20 ± 1.69

For Cube. PushT, and Two-Rooms CEM, every combination shown outperforms the Final Goal baseline. For Two-Rooms Adam, all p values outperform the Final Goal baseline as well as the configurations at K = 4, 5, and 8 and λ\lambda = 0.1.

Let us know if you have any other concerns we can address before the deadline.

Comments on the paper

1 comment

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.