acceptodds
Under review as a conference paper at ICLR 2027

Graph-Guided Long-Horizon MPC with LeWorldModel

17 views

Abstract

Terminal latent distance can poorly reflect progress in long-horizon world-model planning. We add a temporal distance representation, a graph of offline experience, and a budget-conditioned reachability critic to a frozen LeWorldModel (LeWM). Graph search supplies an intermediate target. The critic scores predicted endpoints by the expected time to reach the goal, and the controller switches to latent L2 distance near the goal. We keep the encoder, predictor, and cross-entropy method (CEM) search settings fixed. We bound the number of replanning calls needed to reach the goal under explicit execution, model, search, and critic error conditions. We also bound critic error across horizons and identify when the critic's score equals a truncated expected time to reach the goal. On fixed held-out tasks, success on goals from a different episode increases from 14.4 to 42.4% on Push-T, 29.2 to 65.6% on Reacher, and 16.4 to 27.6% on Cube. Our method shows greatest improvement on long-horizon tasks, while remaining competitive with LeWM on goals 25 steps ahead in Push-T and Reacher. Ablations show that graph guidance and the critic are complementary on Push-T, whereas Cube benefits primarily from the learned temporal representation.

Primary area:
reinforcement learning
Keywords:
world models, long-horizon planning, graph-based planning, temporal distance representation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_y1ULOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes using the temporal distance representations, a graph of offline experiences, and a critic in combination with LeWorldModel to improve planning performance. The authors also propose switching the cost from temporal distance to LeWM’s L2 distance near the goal. The paper also investigates why LeWM’s latent L2 distance fails at longer horizons and compares the results to LeWM and other baselines in three environments and provides ablation studies showing the effects are dependent on the environment. It provides progress and convergence bounds under model, search and critic error conditions.

Critical strengths

The analysis of using L2 latent distance for long-horizon planning showing that at 50 steps the distance between two states is close to distance between two random points is interesting. The theoretical explanation of why the distance between two samples from isotropic gaussian concentrates so L2 cannot rank them and why the straight path leaves distribution is useful. The same issue likely applies to other models with gaussian-regularized latents.

Critical weaknesses

The method combines components from GAS (temporal distance representations, offline graph) with using MPC in LeWM and standard reachability critic as a planning cost. ALPS (Shehmar et al. 2026) and PcLast (Koul et al. 2024) already combine cluster-based planning and CEM. The contribution therefore is mainly showing that this specific combination adds value, i.e. outperforms other methods and all of its components synergize.

  • The comparison to the prior works is very weak. The LeWM baseline was introduced in the paper whose main contribution was a world model. It evaluated on 25 steps and it never claimed long-horizon planning. The learned GCIQL baseline scores below random policy on Reacher task, which suggests that the baseline is misconfigured. At very least the paper should compare to training GAS on frozen LeWM latents (to show that world model contributes beyond supplying latents) and to training-free planning methods using L2 latent distance similar to SoRB (Eysenbach et al. 2019) or TTGS (Opryshko et al. 2026) to show that L2 distance, shown to be reliable on short horizons, is not sufficient for planning on its own. Comparisons to other similar works (ALPS, PcLast) would strengthen the contribution.

  • The ablation studies do not show that this combination of components is particularly useful. Out of three tested environments, on one (Cube) the temporal distance outperforms the full method, on another one (Push-T) critic provides the majority of the cross-episode gain, and Reacher has no component ablations.

Questions for authors
  • Can you compare your method to the related works?
  • Can you show the necessity of all components?
  • Does your method work with different world models, did you test other design choices?
4 comments
AuthorsOpenReview ↗

Response to Reviewer y1UL [1/2]

The reviewer finds the analysis of latent L2 concentration interesting and useful, but raises three concerns: (1) the comparison with prior work is weak, including the LeWM and GCIQL baselines and the absence of GAS, SoRB/TTGS, ALPS and PCLaSt; (2) the ablations do not show that every component is needed, given the Cube result and the absence of Reacher ablations; (3) whether the method works with other world models and design choices.


We thank the reviewer for the careful reading. Shared experiments are in the General Response.


Scope of the contribution

We agree that the components build on prior work (Sec. 2). Alongside the combination, the paper gives a diagnosis of why terminal L2 fails at long range (Sec. 5.3, App. E–F), which the reviewer notes may extend to other Gaussian-regularised latents, and an analysis of the resulting objective under model, search and critic error (Thms 1–3). The evaluation also includes cross-episode goals, which the compared planners do not report.


W1 / Q1: Comparison to related work

We added SAGE, Hi-LeWM, PCLaSt and ALPS on Push-T with the same tasks, budgets and success test (General Response 1). Our method is best on cross-episode goals (42.4 vs at most 31.6 for SAGE and 21.6 for Hi-LeWM). SAGE is best on same-episode goals, with or without the true state.

Training-free planning with L2 distances (SoRB, TTGS). We did not run these. We measured the distance they would rely on. On held-out pairs, latent L2 ranks time-to-goal worse than our graph distance (Spearman 0.81 vs 0.92). On cross-episode pairs checked in the simulator, L2 predicts reachability only modestly above chance (AUROC 0.56–0.62, against 0.75–0.78 for graph distance). This supports the reviewer's point that L2 is reliable only at short range.


W1: The LeWM baseline is weak

We agree that LeWM was not designed for long-horizon planning. We therefore added four long-horizon planners (General Response 1); Ours is best on cross-episode goals, 10.8 points above the best SAGE variant and 20.8 above Hi-LeWM. LeWM remains in the table as the planner our method modifies, and our analysis of its latent space (Sec. 5.3, App. E) shows why L2 alone fails at long range.


W1: GAS on frozen LeWM latents

In a preliminary experiment on an earlier task set, we trained GAS (TDR, graph and low-level policy) on the same frozen LeWM latents and offline data. Its learned policy came close to the random floor on cross-episode goals and was well below the L2 baseline at longer same-episode offsets. This suggests that planning through the LeWM predictor contributes beyond supplying the latent space.


W1: GCIQL is below random on Reacher

The LeWM paper (Maes et al. 2026) reports no GCIQL result on Reacher (Table 1(B), "--"), so there is no published reference for this setting, and we do not use the Reacher GCIQL row as evidence for our method. On Push-T and Cube, GCIQL uses the configuration of LeWM's baselines (Sec. 5.1). Our task set is harder, since it rejects pairs already solved at the start, so absolute numbers differ from the published ones. GCIQL's performance relative to the random policy is consistent across papers for Push-T and Cube.

Part 2 covers the ablations and other world models.

AuthorsOpenReview ↗

Response to Reviewer y1UL [2/2]

W2 / Q2: Necessity of all components

General Response 2 removes each component from the full method in turn on Push-T. Every removal lowers cross-episode success, by 2.8 to 25.2 points. The critic matters most (−25.2 on cross-episode goals, −15.6 at same-ep 100). The graph, the local switch, TDR subgoal scoring and the ET form each cost 6 to 8 points on cross-episode goals. The graph and the switch help mainly together: each adds about 1 point alone and 8.4 together.


W2: The critic provides most of the gain

The reviewer is correct that the critic gives the largest single gain on Push-T cross-episode goals (14.4 → 31.6 when added to L2). Reachability critics are not new. TDM and RC-aux condition reachability on a step budget. Our contribution is in how the critic is shaped and combined. We score each candidate by its expected hitting time, capped at the remaining budget, and add this to the graph subgoal cost inside CEM. Because CEM keeps candidates by rank, the shape of the critic only matters when it is combined with another term. Our ablations show that scoring by −log V, the negative log-probability of reaching the goal in time, is 8.0 points lower on cross-episode goals, and querying a fixed budget instead of the remaining one changes success by −4.4 to +0.8 points. The remaining components (TDR cost, graph subgoals and L2 switch) add a further 10.8 points over L2 + critic (31.6 → 42.4, a 34% relative gain): 3.2 from the TDR cost and switch, and 7.6 from the graph. The full method is also 11.6 and 8.0 points higher than L2 + critic at same-ep 50 and 100.


W2: TDR alone is better on Cube

On Cube, TDR alone exceeds the full method because no planner grasps an unheld cube: all success is a short carry finished within 50 steps, and the graph's first subgoal delays it. On Push-T, where goals need long plans, our method keeps gaining to the end of the budget (General Response 4).


W2: No component ablations on Reacher

We have now run matching ablations on Reacher. The results are as below, with the rows matching the configurations in the Push-T table.

Configurationsame-ep 25same-ep 50same-ep 100cross-ep
L2 (LeWM)77.2 ± 5.490.8 ± 1.182.8 ± 1.129.2 ± 2.7
(a) TDR cost76.0 ± 9.394.4 ± 1.791.2 ± 0.048.4 ± 1.4
(b) graph subgoal71.6 ± 5.091.6 ± 3.090.8 ± 0.060.8 ± 0.9
(c) (b) + switch71.2 ± 4.691.6 ± 3.093.2 ± 1.166.0 ± 1.1
(d) L2 + critic74.0 ± 7.398.0 ± 2.495.6 ± 0.053.2 ± 3.8
(e) (b) + critic76.4 ± 3.796.4 ± 0.994.0 ± 0.964.0 ± 2.3
(f) (c) + −log V critic71.6 ± 4.394.8 ± 3.695.6 ± 0.063.6 ± 0.0
(h) (g), L2 subgoal score78.0 ± 5.897.2 ± 1.796.4 ± 1.164.4 ± 1.7
(i) TDR + critic78.4 ± 4.696.8 ± 0.095.2 ± 0.059.6 ± 2.6
(j) (i) + switch81.6 ± 3.897.6 ± 1.896.0 ± 0.060.4 ± 2.0
(k) TDR + switch75.6 ± 5.296.0 ± 2.095.6 ± 0.064.4 ± 1.4
(l) (g), fixed 50-step critic75.6 ± 5.096.8 ± 1.797.6 ± 0.962.4 ± 2.4
(g) Full method76.0 ± 5.198.0 ± 1.497.6 ± 0.965.6 ± 2.2
  • Every configuration exceeds L2 at same-ep 100 and on cross-episode goals. At same-ep 25 all rows are within seed noise of L2.
  • As on Push-T, removing the graph from (g) lowers cross-episode success (−5.2, (j)), and ET is above −log V (−2.0, (f)).
  • Unlike in Push-T, the role of the critic is lower. We attribute this to Reacher being an obstacle-free reach with mostly reversible actions. Thus any cost that still ranks states at long range suffices.

Q3: Other world models and design choices

We did not test other world models, which is a limitation of the current study. As the reviewer notes, the concentration argument applies to any latent regularised towards an isotropic Gaussian, so we expect the L2 failure to recur; we have not tested whether our fix transfers. Within LeWM we varied the critic form (−log V is 10.8 points lower at same-ep 25 and 8.0 lower on cross), budget-aware vs fixed-horizon critic, critic weight, and graph radius.


In summary, the method changes only the planning cost of a frozen LeWM, raises cross-episode success in all three environments, and remains best on cross-episode goals against the added planners. We believe these additions address the soundness concerns in W1 and W2, and we would welcome a reassessment of the score in light of them. We are glad to answer any further questions.

Reviewer_y1ULOpenReview ↗

Reviewer's reply

Thank you for engaging in the rebuttal process. I have read through your replies and I maintain my score.

My concerns were not sufficiently addressed.

  • The comparisons to prior work are not sufficient to justify your method. If I understand correctly, the baseline table in General Response 1 uses one environment (Push-T), and your method is outperformed by SAGE in all same-episode settings and leads only in the cross-episode setting, which the compared planners do not evaluate.
  • The Reacher ablations don’t clearly show that all components are useful. The cross-episode success rate (which you pose as the main metric) is higher for graph + switch without the critic, and TDR + switch alone is within 1.2 points of the full method.
  • The whole empirical part relies on 3 environments, of which one has success only on the tasks that “start with the cube held and are short carries”.
  • GAS remains a close method, and preliminary experiments, while a valuable source of intuition, might have used a different set of hyperparameters or have other confounding factors. They don’t replace a clean comparison.
  • The time-to-goal ranking and AUROC scores, computed over pairs at arbitrary distances, do not establish that L2 distances are insufficient to distinguish between close (reachable) and distant pairs of states, which might be sufficient for graph-based planning methods.
  • There were no changes to the paper to improve presentation or add the new experiments.

Thank you for the time you put into your responses.

We thank Reviewer y1UL for following up and reviewing our responses. We respect the reviewer’s perspective and maintain this brief clarification:

  1. Cross-Episode Stitching vs. Same-Episode Playback (SAGE Comparison): Same-episode evaluation tasks are sampled from within single demonstration trajectories, functioning largely as trajectory-following or imitation tasks where SAGE’s learned action prior excels. However, the core challenge of long-horizon planning in offline datasets is cross-episode composition (reaching unseen goals across disjoint trajectories). In this true long-horizon setting, which neither SAGE nor Hi-LeWM originally evaluated, our method achieves 42.4% success, outperforming the best SAGE variant (27.2% with its action prior, 31.6% with LeWM CEM) by +10.8 to +15.2 points, and Hi-LeWM by +20.8 points.

  2. Reacher Ablations & Inductive Biases Across Environments: On Reacher, the scores for the full method (65.6 ± 2.2), graph + switch (66.0 ± 1.1), and TDR + switch (64.4 ± 1.4) are all within mutual standard error. Reacher is an obstacle-free and fully reversible planar arm task. Distance to goal is monotonic and reachability failures are minimal. A->B can be achieved just as easily as B->A for states A,B. This is unlike Push-T where pushing the block can be achieved much quicker than 'pulling' the block. This necessitates a quasimetric distance function that is provided with TDR+critic. However, any well-calibrated symmetric distance function suffices for Reacher, reducing the necessity of the critic.

    Each component provides inductive biases tailored to distinct physical bottlenecks.

  3. Benchmark Scope (3 Environments): Push-T, Reacher, and Cube (OGBench) represent the evaluation suite introduced by the LeWorldModel paper that we build on. On Cube, zero-shot visual grasping without task-specific fine-tuning remains an unsolved challenge across all latent MPC planners (all baselines achieve 0% on ungrasped tasks). Such an action sequence has no support in the dataset, and these tasks are an expected quirk of random sampling for cross-episode tasks. On the remaining held-out manipulation tasks, our method improves over vanilla LeWM from 16.4% to 27.6%, and up to 48% using action retrieval.

  4. Distinction from GAS (Model-Based MPC vs. Model-Free Policy Learning): We agree with the reviewer that our preliminary policy experiment is not a substitute for a fully tuned benchmark sweep of GAS. However, we hope to clarify the paradigm difference: GAS (Baek et al., 2025) is a model-free hierarchical offline RL method that learns a parameterized goal-conditioned actor policy πθ(a∣s,g)\pi_\theta(a \mid s, g) to reach graph subgoals without a world model. In contrast, our work explicitly studies model-based visual MPC: we keep the visual world model completely frozen and optimize actions online through forward simulation (CEM) with zero policy training. Because our contribution is repairing test-time visual world-model planning, the directly matched baselines are hierarchical and subgoal planners evaluated on frozen world models (SAGE, Hi-LeWM, PCLaSt, and ALPS), which we evaluated on the same offline data, latent encoder, and tasks.

  5. L2 vs. TDR for Graph Construction (SoRB / TTGS Comparison): The reviewer asks whether latent L2 thresholding could suffice to identify locally reachable neighbors for graph construction (as in SoRB/TTGS), avoiding the need for a learned TDR. In visual domains with contact dynamics (e.g., Push-T), latent L2 reflects visual/pixel similarity rather than dynamical reachability: two states can share high visual overlap (small L2) while being separated by contact boundaries (e.g., the pusher on opposite sides of the block, requiring a 40-step circumnavigation). Thresholding L2 creates spurious edges across kinematic barriers.

    While AUROC or time-to-goal rankings only provide an indication of such restrictions of L2, the component-wise ablations (General Response 2), show that using L2 for graph construction vs TDR ((h) vs. (g)) decreases success rate across all same-episode and cross-episode goals. This confirms the importance of TDR distances.

  6. Manuscript Revisions: External baselines for all 3 environments, and all newly generated experiments including the 4 external baselines (SAGE, Hi-LeWM, PCLaSt, ALPS), the compute-matched controls, the Reacher ablation suite, and the graph reachability tests will be included in the final version of the paper.

We thank the reviewer once again for their time and constructive critiques throughout the review process.

Reviewer_kuxqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Here long-horizon planning is improved with a frozen version of LeWorldModel by changing the cost used to rank CEM candidates. This is achieved with a temporal distance representation, building a graph over offline data to help choose an intermediate, and then training a reachability critic. This critic has an “expected time” score that ranks the endpoints and switches back to L2 near the goal. On standard benchmarks such as Push-T, Reacher and Cube they improve episode success.

Critical strengths
  • The method to my knowledge is novel.
  • Gains on Push-T and Reacher are large and consistent.
  • Computation time is reported in the paper and will be helpful for future researchers.
Critical weaknesses
  • The paper doesn’t clearly isolate the improvements their graph method contributes. This is best shown by adding the critic to L2 giving 31.6% success vs 42.4% for their method.
  • On Cube the temporal distance cost achieves 36.8% success compared to 27.6% for full method. This is the opposite of what I would have expected?
  • Some controls are missing. The method takes 1.8s per decision vs 1.14s for L2 and if L2 was given the same time you might expect the performance gap to shrink.
Questions for authors
  • In Section 5, could you report two changes removing the graph? First, using TDR cost plus reachability critic (no graph). Second, the full method with no graph but keeping the local switch to L2 near the goal.
  • Would you expect (and does) a critic with a fixed budget perform any differently? If you give L2 matched wall clock time does it perform differently to the given method?
1 comment
AuthorsOpenReview ↗

Response to Reviewer kuxq

The reviewer finds the method novel, the Push-T and Reacher gains large and consistent, and the compute reporting useful, but raises three concerns: (1) the graph's contribution is not isolated; (2) TDR alone beats the full method on Cube; (3) L2 is not compared at matched compute, and the effect of a fixed-budget critic is unclear.


We thank the reviewer for the positive assessment and for specific, testable questions. We ran each of them; the results are below.


W1 / Q1: Isolating the graph

We ran both requested rows on Push-T; they are rows (i) and (j) of General Response 2. On cross-episode goals, TDR + critic without the graph gives 34.0, and adding the local switch (the full method without the graph) gives 34.8, against 42.4 for the full method. At same-episode offsets the graph changes success by 2 points or less.

Against L2 + critic (31.6 on cross), the full method gains 10.8 points on cross-episode goals: 3.2 from the TDR cost and switch, and 7.6 from the graph. This fits the graph's purpose. The TDR cost and the critic are trained with positive signals sampled only from same-episode pairs, and the graph is the only component that connects different episodes explicitly.


W2: TDR beats the full method on Cube

See General Response 4. The full method still improves on L2 in every Cube protocol (Table 1), but almost every Cube cross-episode success is a short-horizon task finished within 50 steps. In environments such as Push-T, our method keeps gaining to the end of the budget while TDR does not.


W3 / Q2b: L2 at matched wall-clock time

We enlarged L2's CEM to 1.31× and 2.47× our per-decision planning time (420×40 and 600×56 candidates × iterations). L2 stays within 2 points of its default at every protocol: 40.8 and 39.6 at same-ep 50, 12.8 at same-ep 100, and 14.8 and 16.0 on cross-episode goals, against 70.0, 46.4 and 42.4 for our method. L2's CEM does not benefit from a larger search space or longer planning time, which matches the paper's diagnosis that terminal L2 cannot rank distant candidates (Sec. 5.3).


Q2a: Fixed-budget critic

This is row (l) of General Response 2, a critic that always scores 50 steps. Our critic's sum is capped at 50 steps, so the two differ only when fewer than 75 steps remain. We therefore expected the largest change at short budgets. The fixed-budget critic is 3.2 points lower at same-ep 25, 4.4 at same-ep 50 and 2.8 on cross-episode goals, and 0.8 higher at same-ep 100. These gaps are within about one seed standard deviation, so budget awareness gives at most a small gain on Push-T. Most of the critic's contribution comes from scoring reachability at all.


General Response 2 extends Table 2 with these rows, and General Response 3 adds the matched-time controls. In each case the long-horizon gains remain, which supports the paper's central point that the planning objective, rather than the model or the search, limits long-horizon performance.

We hope these results are helpful for the final assessment, and we are glad to answer any further questions.

Reviewer_AwtHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper improves long-horizon planning with a frozen world model by combining graph-based subgoals, budget-aware reachability scoring, and a local distance switch. My concern lies in the novelty of this paper. There are many subgoal or hierarchical rl papers. The authors are expected to further clarify their novelty. Also, the experiments demonstrate improvements over the original LeWM planner, but lack matched comparisons with closely related graph-based or hierarchical subgoal planners.

Critical strengths
  1. The paper is well-organized, including both empirical evidence and theoretical findings.
Critical weaknesses
  1. Given the extensive literature on subgoal-based planning and hierarchical RL, the novelty of the proposed combination remains insufficiently established. Although the paper discusses related methods, the experiments mainly compare against the original LeWM planner and internal ablations. A clearer account of the methodological differences, together with matched comparisons against closely related graph-based or hierarchical planners, would help establish the contribution.
  2. The experiments demonstrate improvements over the original LeWM planner, but lack matched comparisons with closely related graph-based or hierarchical subgoal planners.
Questions for authors
  1. What distinguishes the proposed approach from existing graph-based subgoal planners and hierarchical RL methods? Which aspects of the combination are essential beyond applying established components to a frozen world model?
  2. Could you provide a matched comparison with a closely related planner, such as SAGE, Hi-LeWM, or ALPS, using the same offline data, task set, and execution budget?
  3. Since graph edges are based on proximity in TDR space, how often are the selected subgoals actually reachable within the 25-step planning horizon? How does the controller handle infeasible subgoals?
  4. Does the improvement over terminal-L2 planning persist under an equal wall-clock budget, allowing the baseline additional CEM samples or iterations?
  5. Would the proposed objective also improve performance under Best-of-N action-sequence sampling? This would help determine whether its benefits depend on CEM’s adaptive search.
  6. Why does TDR alone outperform the full method on Cube cross-episode tasks? Can the degradation be attributed to graph routing, critic errors, or the local distance switch?
1 comment
AuthorsOpenReview ↗

Response to Reviewer AwtH

The reviewer finds the paper well organised, with both empirical evidence and theory, but raises two concerns: (1) the novelty relative to subgoal-based and hierarchical planners; (2) the lack of matched comparisons with these planners. The reviewer also asks about subgoal reachability, compute-matched and Best-of-N baselines, and the Cube result.


We thank the reviewer for the specific questions. We address each below.


Q1 / W1: What distinguishes the method, and what is essential

Compared with graph-based and hierarchical planners such as SAGE, Hi-LeWM, PCLaSt and ALPS, our method keeps the world model and its CEM planner frozen and changes only the cost that CEM ranks; the methods above instead train a subgoal generator, a high-level model or a policy. The paper also explains why the unmodified planner fails: latent L2 between distant states concentrates near the value for two random states, so CEM cannot rank candidates beyond about 50 steps (Sec. 5.3, App. E), and states the conditions under which the new objective makes progress (Thms 1–2). Finally, none of these methods evaluate goals taken from a different episode. On such goals, our method is best among matched planners (42.4 vs at most 31.6 for SAGE and 21.6 for Hi-LeWM).

General Response 2 removes each component from the full method in turn on Push-T. Removing the reachability critic costs 25.2 points on cross-episode goals, and its expected-hitting-time form matters (−log V is 8.0 points lower). Removing the graph or the local switch costs 7.6 and 7.2 points on cross-episode goals, and the two help mainly together. The graph matters because the TDR cost and critic learn reachability only within episodes.


Q2 / W2: Matched comparison with SAGE, Hi-LeWM or ALPS

See General Response 1.


Q3: Are subgoals reachable, and what happens when they are not?

Most graph edges are reachable, and the planner re-selects a subgoal at every replan, so it never permanently commits to an infeasible one.

Reachability. We tested graph edges in the simulator, using a planner with access to the true state, so that a failure reflects the edge rather than the world model. The edge radius corresponds to about 12 environment steps, so we report success within 12 steps (one edge length) and within the 25-step planning horizon:

Pair typeWithin 12 stepsWithin 25 steps
Edges, shortest / middle / longest third89% / 77% / 60%89% / 81% / 61%
Non-edges, 1–2× edge length31%41%
Non-edges, 2–4× edge length9%17%

A subgoal can be several edges ahead of the current state, so we also executed full graph paths on 50 cross-episode tasks, one hop per 25-step plan, with each hop starting from the state actually reached. 82% of first hops succeed. After a successful hop the next one succeeds 59% of the time, but after a failed hop only 13%. Routes are therefore mostly valid, but errors compound when the planner keeps to a fixed path. These tests use graph edges and paths, not the subgoals chosen during evaluation runs.

Failure handling. This is why the planner does not commit to a path. At every 25-step replan it re-attaches the current state to the graph and picks a new subgoal, so an unreachable subgoal is abandoned after at most one plan.


Q4: Equal wall-clock budget

With 1.31× and 2.47× our planning time, L2 stays within 2 points of its default (General Response 3). Extra compute does not improve L2's performance, consistent with the finding that terminal L2 cannot rank distant candidates (Sec. 5.3, App. F).


Q5: Best-of-N

With CEM replaced by Best-of-N at the same 9,000 cost evaluations, our method still beats L2 at every protocol: 67.6 / 46.8 / 34.8 / 32.0 against 61.6 / 30.4 / 6.4 / 12.8. The objective, not CEM's adaptive search, carries most of the gain; CEM's refinement helps both.


Q6: Why TDR alone beats the full method on Cube

On Cube the full method still improves on L2 in every protocol (Table 1). We attribute the gap to TDR alone to the composition of the Cube cross-episode task set rather than to incorrect subgoals. The 30 solvable tasks are short carries that the direct TDR cost completes within one or two plans, so any intermediate subgoal increases the number of plans and the risk of dropping the cube. Details are in General Response 4.


We hope this makes the contribution clearer: a change to the planning cost alone, supported by a diagnosis of the L2 failure and a progress analysis, that improves long-horizon and cross-episode success in all three environments. We hope the added results are useful in the reviewer's final assessment, and we are glad to answer any further questions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers [1/2]

We thank the reviewers for detailed and constructive reviews, and for their comments on the analysis of latent L2 (y1UL), the size of the gains (kuxq) and the combination of theory and experiments (AwtH).

To frame the responses, we briefly restate the paper's central claim. Terminal latent L2 stops ranking CEM candidates beyond about one planning horizon; between states 50 steps apart it is already close to the value for two random states (Sec. 5.3, App. E–F). Changing only the planning cost of the frozen LeWM, with the encoder, predictor and CEM settings unchanged, raises cross-episode success in all three environments (Push-T 14.4 → 42.4, Reacher 29.2 → 65.6, Cube 16.4 → 27.6; Table 1), and Thms 1–3 state the conditions under which this progress is guaranteed. The experiments below test this claim against related planners, ablations and compute-matched baselines, on the same held-out tasks and CEM settings as the paper, including the cross-episode setting that the compared planners do not evaluate.

Three requests were common to several reviews: comparisons with related planners, an ablation that isolates the graph, and controls with matched planning compute. Part 2 covers compute controls and the Cube results.

We will revise the paper with the updated results.


1. External baselines

All methods use the same 50 held-out tasks and budgets (50/100/200 steps for same-episode offsets 25/50/100, 250 for cross-episode).

Methodsame-ep 25same-ep 50same-ep 100cross-ep
LeWM (L2)91.2 ± 1.040.8 ± 3.713.2 ± 3.214.4 ± 3.9
Ours88.4 ± 2.370.0 ± 2.246.4 ± 3.242.4 ± 3.2
SAGE (subgoal + action generator)96.8 ± 1.082.8 ± 3.068.0 ± 5.527.2 ± 4.3
SAGE (subgoal + LeWM CEM)91.2 ± 1.073.2 ± 2.764.8 ± 5.231.6 ± 2.3
Hi-LeWM-C (best config)82.4 ± 1.746.4 ± 3.322.4 ± 5.221.6 ± 4.3
PCLaSt (on LeWM's CEM)79.2 ± 3.033.2 ± 2.316.0 ± 3.510.8 ± 3.6
ALPS21.2 ± 3.610.0 ± 2.03.2 ± 1.10.0 ± 0.0

Our method is best on cross-episode goals: 10.8 points above the best SAGE variant and 20.8 above Hi-LeWM. SAGE is best on same-episode goals but reaches 27.2 on cross-episode goals. To separate SAGE's subgoal generator from its learned action prior, we also ran its generated subgoals through LeWM's plain CEM (the same CEM our method uses). This variant keeps most of SAGE's same-episode gain (73.2 at same-ep 50, 64.8 at same-ep 100) and reaches 31.6 on cross-episode goals. SAGE's subgoal generator is trained on same-episode samples only and thus underperforms on cross-episode tasks. ALPS's Laplacian representation did not converge on Push-T in three attempts, so its row is not a fair estimate of ALPS.


2. Ablations (extending Table 2)

Rows (i)–(l) are new and use the same tasks and seeds as Table 2 of the paper; (d) and (g) are from Table 2.

Configurationsame-ep 25same-ep 50same-ep 100cross-ep
(d) L2 + critic89.2 ± 3.058.4 ± 5.138.4 ± 7.131.6 ± 2.3
(i) TDR + critic84.0 ± 3.865.6 ± 1.747.6 ± 6.234.0 ± 3.3
(j) TDR + critic + switch (no graph)88.8 ± 2.368.8 ± 2.748.4 ± 5.034.8 ± 6.5
(k) TDR + switch (no critic)89.6 ± 3.851.6 ± 3.825.2 ± 4.416.0 ± 3.7
(l) Full method, fixed 50-step critic85.2 ± 3.365.6 ± 4.347.2 ± 3.039.6 ± 7.1
(g) Full method88.4 ± 2.370.0 ± 2.246.4 ± 3.242.4 ± 3.2

Row (l) always scores 50 steps instead of the remaining budget. Our critic's sum is capped at 50 steps, so (l) differs from (g) only when fewer than 75 steps remain: in both plans at same-ep 25, and in the last two plans of longer tasks.

Each row below changes one component of the full method (g) and keeps everything else fixed. Entries are differences from (g) in percentage points. Rows (c), (e), (f) and (h) are from Table 2.

Change to the full method (g)Rowsame-ep 25same-ep 50same-ep 100cross-ep
Remove the critic(c)−0.4−7.6−15.6−25.2
Remove the graph(j)+0.4−1.2+2.0−7.6
Remove the local switch(e)−2.4−5.2−2.8−7.2
Score the subgoal by L2 instead of TDR(h)−2.4−6.4−1.2−6.0
Replace ET by −log V(f)−10.8−0.8+0.8−8.0
Fixed 50-step critic instead of remaining budget(l)−3.2−4.4+0.8−2.8
  • Every component is needed for cross-episode goals: removing any one of them lowers success there.
  • The critic carries most of the long-horizon gain.
  • The graph and the local switch together help, primarily on cross-episode goals. The graph is the only component that links different episodes explicitly; without it, same-episode success changes by 2 points or less.
AuthorsOpenReview ↗

General response to all reviewers [2/2]

3. Compute controls

L2 given 1.31× and 2.47× our per-decision planning time stays within 2 points of its default (cross: 14.8 and 16.0 vs 42.4). With CEM replaced by Best-of-N at the same 9,000 cost evaluations, our method remains above L2 at every protocol (cross: 32.0 vs 12.8). The gain therefore comes mainly from the objective rather than from additional search.

PlannerComputesame-ep 25same-ep 50same-ep 100cross-ep
L2, CEM 300×30 (default)0.80× time91.2 ± 1.040.8 ± 3.713.2 ± 3.214.4 ± 3.9
L2, CEM 420×401.31× time90.4 ± 3.640.8 ± 4.412.8 ± 3.614.8 ± 3.0
L2, CEM 600×562.47× time90.8 ± 3.939.6 ± 3.012.8 ± 3.016.0 ± 7.3
Ours, CEM 300×301.00× time88.4 ± 2.370.0 ± 2.246.4 ± 3.242.4 ± 3.2
L2, Best-of-N9,000 evals61.6 ± 6.130.4 ± 5.06.4 ± 1.712.8 ± 3.3
Ours, Best-of-N9,000 evals67.6 ± 3.346.8 ± 7.834.8 ± 8.232.0 ± 4.2

CEM is candidates × iterations; default CEM also scores 9,000 candidates per decision. Time is per-decision planning time relative to ours. These timings come from a single task run, while the paper's 1.14 s vs 1.80 s is averaged over batches of 25 tasks, so the ratios differ.


4. Why TDR alone beats the full method on Cube cross-episode goals

These tasks do not test stitching. On the 20 tasks where the cube is not already held, all methods score 0%. The other 30 start with the cube held and are short carries. Success stays constant from step 100 onward for every method (from step 50 for TDR and ours), and extending the budget from the paper's 200 steps to 250 does not change it. TDR gets 35.2 of its 36.8 points in the first 25-step plan. On these, the path along the graph is always longer than the direct start-to-goal TDR distance, so it needs more plans, and each extra plan while holding the cube adds a chance of dropping it, which cannot be recovered.

On Push-T, where goals require long plans, L2 and TDR still stall near 14% after step 100, while ours keeps gaining (30.4 → 42.4). This shows that the critic and graph components help when multiple replans are necessary.

The gap is also not a limit of the graph-based cost itself. Initializing CEM from a logged action sequence, with the graph, critic and cost unchanged, raises our cross-episode success on Cube from 27.6 to 48.0 (Table 4), above TDR alone (36.8; Table 7). On Cube the bottleneck is finding good actions, not ranking them.

Success by environment step on cross-episode goals:

EnvMethodby step 25by step 50by step 100final (by step 250)
CubeL213.614.816.416.4
TDR35.236.836.836.8
Ours9.227.627.627.6
Push-TL24.86.012.014.4
TDR3.67.212.414.0
Ours2.014.830.442.4
Area_Chair_tsadOpenReview ↗

Official Comment by Area Chair tsad

Dear reviewers,

Thank you very much for your time and effort in reviewing this submission. The authors have now submitted their rebuttal, and the discussion period is nearing its end. We would greatly appreciate it if you could take some time to carefully review the authors’ responses and share your assessment of whether your concerns have been adequately addressed. Thank you again for your valuable time and contributions to the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.