acceptodds
Under review as a conference paper at ICLR 2027

SURVEYOR: Training-Free Decomposition for Long-Range Latent World-Model Planning

Abstract

Latent world-model planners succeed at short goal distances but fail at long ones, and hierarchical variants repair this failure by decomposing a distant goal into a sequence of latent subgoals. However, existing hierarchical planners make two decisions by fixed schedule rather than by evidence: they regenerate the subgoal sequence at every replanning step, even when the previous draft remains consistent with execution, and they decompose every goal, even when the executor could reach it directly. We introduce Surveyor, a training-free, plug-and-play decomposition layer that sits between an executor and a frozen world model and makes both decisions from the world model's own latents and predictions. A drafter proposes latent subgoals a fixed number of steps ahead; an accept rule reuses the remaining draft while the achieved latent state agrees with the subgoal just pursued; and an arbiter bypasses or retires drafting once the goal is predicted to be within the executor's reach. Both decisions use a tolerance calibrated offline from observation pairs that the benchmark's success criterion treats as equivalent. Extensive experiments over five executors, spanning sampling-based, gradient-based and amortized planners, and four environments show that Surveyor raises success at the longest PushT goal distance from below 10% to above 92% for every search-based executor while issuing up to two fifths fewer drafter calls than every-step drafting, and improves both the average and the worst-case success of all five executors over the seventeen combinations of environment and goal distance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, latent planning, model predictive control, hierarchical planning, subgoal decomposition, plan reuse, goal-conditioned control, joint-embedding predictive architectures

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4XqYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SURVEYOR, a training-free framework for long-horizon planning with latent world models. The method introduces two mechanisms into existing hierarchical planning frameworks: an execution–plan consistency check that determines whether previously generated subgoals can be reused, and a reachability-based arbiter that decides whether hierarchical decomposition is necessary. Experiments across four simulated environments and five executors demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper addresses a practically meaningful problem by targeting redundant subgoal generation and unnecessary task decomposition in hierarchical planning.
  2. The proposed method is simple, requires no additional training, and can be integrated with various existing planners, demonstrating practical applicability.
  3. The experimental evaluation is relatively comprehensive, covering different types of executors, multiple goal distances, and component ablations. The method achieves substantial improvements in success rates on certain long-horizon tasks.
Critical weaknesses
  1. The execution-discrepancy-based replanning mechanism is conceptually similar to existing event-triggered replanning methods, such as AdaReP. The reachability-based arbiter also relies primarily on a simple distance threshold in latent space. The authors should further clarify the technical contributions relative to prior work.
  2. Compared with flat planners, SURVEYOR substantially improves success rates and reduces execution time in certain long-horizon tasks. However, compared with the existing FF-JEPA hierarchical planning framework, the reduction in subgoal-generation calls does not translate into consistent end-to-end computational savings. Since the drafter accounts for only a small fraction of the total computation and the arbiter introduces additional planning overhead, the full method increases per-episode runtime in most settings.
  3. The arbiter uses a shared CEM planner to estimate goal reachability, although different executors may have substantially different capabilities, leading to performance degradation in some settings. Furthermore, the same threshold is used for execution consistency and goal reachability, which requires stronger justification. The authors should further analyze the accuracy of reachability estimation and the impact of incorrect decisions on task success rates.
Questions for authors

Please refer to the weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4XqY (1/3): Technical Contribution and Cost

Dear Reviewer 4XqY,

We sincerely appreciate your careful review and constructive comments. We provide point-by-point responses below.

Q1. Technical contribution relative to prior work

"The execution-discrepancy-based replanning mechanism is conceptually similar to existing event-triggered replanning methods, such as AdaReP."

A1(1): Thank you for pointing us to AdaReP. Section 2 of our submission already discusses AdaReP and event-triggered control and states how the accept rule differs, and we have expanded this discussion in the revision. While both mechanisms monitor a discrepancy during execution, they act on different objects. AdaReP caches the action plan of a single-level world-model MPC and re-solves it when the deviation between the predicted and observed states exceeds a tolerance that is adapted online. It does not involve subgoals. In contrast, our accept rule checks a drafted subgoal, i.e., whether the achieved latent agrees with the subgoal just pursued, so that the remaining subgoals can be reused. Moreover, its tolerance is not adapted online. It is measured offline from observation pairs that the task's success criterion treats as equivalent, so it requires no tuning.

More importantly, Surveyor makes a second decision that is absent from event-triggered replanning, namely whether the goal should be decomposed at all. This decision accounts for most of the gain of Surveyor over every-step drafting: over the search executors, the worst PushT or Reacher setting rises from 54.2%–66.8% to 81.0%–93.3%. We therefore view AdaReP as complementary to our work, since it could run inside the executor underneath Surveyor. The new Table 29 extends this comparison to FF-JEPA, SAGE and RDM.

"The reachability-based arbiter also relies primarily on a simple distance threshold in latent space."

A1(2): We would like to clarify that the arbiter does not threshold the distance between the current state and the goal. Instead, it rolls out a flat plan through the frozen world model and compares the predicted outcome of that plan with the goal. This distinction is essential. The appendix of our submission evaluates the simple distance threshold as a control, and we have moved this result into the main text (Section 4.4). We list it below.

Table: A distance threshold versus the arbiter. Success (%) at Reacher t=25.

DecisionCEMiCEM
Always decompose (every-step drafting, FF-JEPA)65.261.5
Route on the latent distance to the goal63.052.5
Route on the predicted outcome of a flat plan (Surveyor)98.298.8

As shown in the table, a distance threshold almost never routes an episode, while the arbiter's test recovers the short-range success. The contribution of the arbiter thus lies in the test itself, i.e., in using the predicted outcome of a plan together with a tolerance that is measured from the task.

Q2. End-to-end cost

"... compared with the existing FF-JEPA hierarchical planning framework, the reduction in subgoal-generation calls does not translate into consistent end-to-end computational savings ... the full method increases per-episode runtime in most settings."

A2: Thank you for this comment. The additional runtime comes from the arbiter's reachability probe, which is what recovers the short-range success lost by FF-JEPA's every-step drafting. To examine how much of this time is necessary, we ran Surveyor with a lighter probe (100 samples and 10 iterations instead of 300 and 30; the budget was fixed before the run). The results are reported below and in Table 24.

Table: Time and success with a lighter reachability probe. CEM on one A100. Seconds per episode are measured at t=150 and exclude model loading. Success is the mean / worst over the nine PushT and Reacher settings.

Every-step drafting (FF-JEPA)Surveyor, default probeSurveyor, lighter probe
Seconds per episode, PushT4.156.164.07
Seconds per episode, Reacher5.085.323.89
Success (%), mean / worst91.2 / 65.295.4 / 90.996.4 / 92.1

The results show that, with the lighter probe, Surveyor runs as fast as FF-JEPA's setting while keeping its higher success. The additional runtime is therefore determined by the probe budget. As for the accept rule, its saving is in subgoal-generation calls (0.58 to 0.73 of the calls of every-step drafting at PushT t=150). We have clarified in the introduction that this saving is most relevant when subgoals come from large generative models or from models served remotely, where every call has a computational or monetary price.

AuthorsOpenReview ↗

Response to Reviewer 4XqY (2/3): Accuracy of the Reachability Estimate

Q3. The reachability estimate

"The authors should further analyze the accuracy of reachability estimation and the impact of incorrect decisions on task success rates."

A3(1): Thank you for this valuable suggestion. Following it, we conducted a per-episode analysis (new Appendix H). For every episode, we log the decision of the arbiter and additionally run both the flat planner and every-step drafting on the same episode with the same seed. This allows us to measure the accuracy of each routing decision and to compare Surveyor with an oracle that selects, for each episode, whichever of the two planners solves it.

Table: Accuracy of the arbiter's routing (new Table 19). CEM, all values in %, eight repeated runs.

SettingRouted to the flat plannerFlat planner on routed episodesEvery-step on routed episodesSurveyorPer-episode oracle
PushT t=2562.599.799.199.499.9
PushT t=500.875.095.893.396.2
Reacher t=2591.299.566.098.297.7
Reacher t=5077.099.690.497.596.6
Reacher t=10049.799.099.095.096.1
Reacher t=15039.498.3100.090.893.6
Cube, mean over t48.372.862.345.949.2
Two-Room, mean over t52.482.976.769.676.4

The results show that the estimate is accurate. Pooled over PushT and Reacher, the flat planner solves 99% of the episodes that the arbiter routes to it. On Cube and Two-Room, where all planners are weaker, the routed episodes are still the ones that the flat planner solves more often than every-step drafting (72.8% versus 62.3%, and 82.9% versus 76.7%). Correct routing brings a large gain: at Reacher t=25 the arbiter routes 91.2% of the episodes, of which the flat planner solves 99.5%, whereas every-step drafting would solve only 66.0%.

Regarding the impact of incorrect decisions, Surveyor stays within 2.9 points of the oracle at every PushT and Reacher distance, which bounds how much a perfect choice between the two planners could add. Note that Surveyor can slightly exceed this oracle, because it may also retire drafting in the middle of an episode.

"Furthermore, the same threshold is used for execution consistency and goal reachability, which requires stronger justification."

A3(2): Thank you for the question. The two tests share a tolerance because they ask the same question, namely whether two latents describe states that the task's success criterion treats as the same. The accept rule asks this of the achieved latent and the subgoal just pursued, while the arbiter asks it of the predicted final latent of a flat plan and the goal. The tolerance is the measured answer to this question, so a single calibration rule serves both tests and no second parameter needs to be tuned. We have added this explanation to Section 3.5. To be precise, the rule yields one value per latent space: on Two-Room, the accept rule reads DINOv2 embeddings (0.127) while the probe reads LeWM latents (1.323), and both values are measured by the same rule on the same pairs.

We also verified this choice empirically. Since the probe of every episode is logged, we can vary the routing threshold offline.

Table: Success (%) when routing at a multiple of the calibrated tolerance τ (Table 20). CEM, eight repeated runs. Episodes whose probe lies below the threshold go to the flat planner and the rest to every-step drafting, so the values isolate the routing decision and differ slightly from those of the full system above.

Setting0.5τ0.75ττ (calibrated)1.5τ2τ3τ
PushT t=2599.399.299.499.098.698.7
Reacher t=2564.579.394.996.996.996.9
Reacher t=15093.193.092.492.192.091.8
Cube t=5043.943.643.140.740.038.4
Two-Room t=7557.458.257.653.149.043.0

As shown in the table, a threshold that is too strict stops routing and loses the short-range recovery (64.5% at 0.5τ on Reacher t=25), while a threshold that is too loose routes episodes that the flat planner cannot finish (43.0% at 3τ on Two-Room t=75). The calibrated value avoids both cases without any tuning. It is within 2.0 points of the best multiple in all seventeen settings of the four environments, which are all listed in Table 20. In addition, moving the tolerance across its bootstrap interval changes no result by more than 1.8 points (Table 12).

AuthorsOpenReview ↗

Response to Reviewer 4XqY (3/3): The Shared Probe

Q3 (continued). The shared probe

"The arbiter uses a shared CEM planner to estimate goal reachability, although different executors may have substantially different capabilities, leading to performance degradation in some settings."

A3(3): Thank you for raising this concern. We use a shared probe by design, and our submission already evaluates the alternative that you describe, in which each executor serves as its own probe (Appendix J of the revised PDF). We summarize the comparison below.

Table: Shared CEM probe versus each executor as its own probe. Mean / worst success (%) of Surveyor over the nine PushT and Reacher settings, computed from the per-setting values in Appendix A. CEM is omitted because the two probes coincide for it.

ExecutorShared CEM probe (used in the paper)Executor as its own probe
iCEM94.0 / 89.795.6 / 92.8
MPPI92.7 / 81.093.0 / 82.6
Gradient96.3 / 93.396.3 / 94.1
GC-IDM (amortized)95.5 / 93.591.3 / 64.3

For the search executors, sharing the probe has little effect: the two probes are within 1.6 points of each other in mean success for iCEM, MPPI and the gradient solver, which suggests that a CEM plan is a good estimate of what these executors can reach. For the amortized executor GC-IDM, which does not search the world model, the shared probe is necessary. With GC-IDM as its own probe, the worst setting drops from 93.5% to 64.3%.

Furthermore, the shared probe keeps the layer plug-and-play. Since it only reads the frozen world model, the same arbiter is applied unchanged to every executor, and it routes the same fraction of episodes for every search executor to within one point (Appendix G).

Since no probe is the best for every executor, we use one uniform setting instead of selecting a probe per executor on the evaluation results. With this setting, Surveyor improves on the vanilla version of all five executors in both the mean and the worst-case success over the seventeen settings. The settings in which the arbiter costs success are reported in the main text (Section 4.4, "What the arbiter costs"), and Appendix J reports the variants that recover them.

In summary, we made the following revisions in response to your review:

  • Appendix H and Section 4.4 report the per-episode accuracy of the reachability estimate and the effect of incorrect decisions.
  • Section 3.5 and Table 20, which covers all four environments, justify the single calibration rule.
  • Section 4.4 now reports the distance-threshold control in the main text.
  • Table 24 removes the time gap to FF-JEPA's every-step drafting.
  • Section 2 and Table 29 position Surveyor against AdaReP and the other related planners.

We appreciate your detailed comments and hope we have addressed your concerns. Please let us know if you have any additional questions.

Sincerely, Authors

Reviewer_jPhBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper improves world model planning by using a pretrained diffusion model to propose latent waypoints and adding numerical rules for reusing subgoals and switching between hierarchical and direct planning. In this way, it uses a simple integration to build robust work model planning across different goal distances.

Critical strengths
  1. The experiment section and appendix provide extensive ablation results.
  2. The work addresses different modes of failure together. The accept rule addresses unnecessary regeneration of subgoals, the arbiter addresses unnecessary decomposition, and the median latent discrepancy addresses an arbitrary threshold for calibration.
Critical weaknesses
  1. The idea combines execution-based subgoals reuse, predicted reachability for routing or retirement, and task based tolerance calibration. Neither of them is new, nor is the broad idea of latent subgoals and adaptive planning.
  • For example, "FF-JEPA: Long-Horizon Planning in World Models with Latent Planners" predicts latent subgoals and uses a shorter horizon controller to reach them.
  • For example, "AdaReP: Adaptive Re-Planning under Model Mismatch for Neural World-Model Predictive Control" is a training-free wrapper to reuse cached plans while monitoring executation mismatch.
  • For example, "Adaptive Online Replanning with Diffusion Models" switch between generation or reuse of diffusion-gnereated plans.
  1. The major issue is that the paper overclaims the improvement.
  • Most of the claimed improvements in the abstract and intro come from the existing decomposition mechanism. For example, adding the drafter has already significantly improved the success rate, and the rest of the improvement is very marginal: about 1.3$ diffrence between every-step drafting vs the full system is hard to argue for the contribution of the whole system
  • Full system takes much longer per episode over every-step drafting. It is a trade-off between cost/time vs incremental improvement.
  • The blind two-waypoint reuse basically achieve the same success while having fewer drafter call and smaller runtime.
Questions for authors

See the weakness part. Needs more evidence to show the importance of the full system vs. the drafter.

Also needs to compare with some approaches that reports even stronger long-range planning success

  • SAGE: SUBGOAL-CONDITIONED ACTION GENERATION FOR LATENT WORLD MODEL PLANNING
  • FF-JEPA: Long-Horizon Planning in World Models with Latent Planners
  • AdaReP: Adaptive Re-Planning under Model Mismatch for Neural World-Model Predictive Control
  • Hierarchical Planning with Latent World Models
9 comments
AuthorsOpenReview ↗

Response to Reviewer jPhB (1/4): Novelty

Dear Reviewer jPhB,

We sincerely appreciate your detailed and constructive comments. We provide point-by-point responses below.

Q1. Novelty

"The idea combines execution-based subgoals reuse, predicted reachability for routing or retirement, and task based tolerance calibration. Neither of them is new, nor is the broad idea of latent subgoals and adaptive planning."

A1: Thank you for pointing us to these works. We respectfully disagree that they cover our contribution. We note that Section 2 of our submission already discusses all three works and states how the accept rule differs from AdaReP and RDM. In the revision we have expanded this discussion and added Table 29. The main difference is that AdaReP and RDM decide when to recompute a single plan and FF-JEPA follows a fixed schedule, whereas Surveyor decides whether a drafted subgoal is still valid and whether a goal needs subgoals at all. None of the cited methods makes the latter decision. We elaborate on each method below.

  • FF-JEPA contributes the subgoal model and a short-horizon controller. Our work builds directly on it: its subgoal model is our drafter, and its setting (every-step drafting) is a baseline that we evaluate for every executor at every goal distance (Appendix A). However, FF-JEPA regenerates the subgoals at every replanning step and decomposes every goal, so it makes neither of our two decisions. Always decomposing is costly at short range. For example, it lowers the success of CEM at Reacher t=25 from 95.7% to 65.2%.
  • AdaReP caches the action plan of a single-level world-model MPC and re-solves it when the deviation exceeds an online-adapted tolerance. Since it has no subgoals, it cannot decide whether a drafted subgoal is still valid or whether a goal should be decomposed. We view it as complementary, as it could run inside the executor underneath Surveyor.
  • RDM (Adaptive Online Replanning with Diffusion Models) regenerates a single-level diffusion plan when the likelihood of the plan falls. It requires a trained diffusion planner and likewise has no decomposition decision, whereas Surveyor requires no training and only reads the latents of the frozen world model.

Table: How each method makes the two decisions (new Table 29).

MethodWhat triggers re-planningWhether to decomposeTraining
FF-JEPAevery step (fixed)always (fixed)subgoal model
SAGE (concurrent)every stage (fixed)from the known remaining horizonsubgoal model, action prior
RDMlikelihood of a single-level plannot applicablediffusion planner
AdaRePdeviation of a single-level action plannot applicablenone
Surveyorachieved latent versus the pursued subgoalper episode, from the predicted outcome of a flat plannone

Regarding the three components mentioned in the review:

  • Execution-based reuse. AdaReP and RDM reuse a single plan and decide when to recompute it. Our accept rule instead compares a drafted subgoal with the latent that was actually achieved, in order to decide whether the remaining subgoals can still be used.
  • Predicted reachability for routing or retirement. None of the three works performs routing or retirement, which is the role of our arbiter. This test is not trivial: a threshold on the latent distance to the goal fails (63.0% versus 98.2% at Reacher t=25 over CEM), while the arbiter's decisions are accurate (on PushT and Reacher, the flat planner solves 99% of the episodes routed to it; see the new Appendix H).
  • Task-based tolerance calibration. None of the three works derives its trigger from the task's success criterion. AdaReP adapts its tolerance online, RDM relies on a likelihood, and FF-JEPA has no trigger. Our tolerance is the typical latent distance between observations that the success criterion treats as equivalent. It involves no tuning and is within 2.0 points of the best threshold in all four environments (Table 20).

To the best of our knowledge, Surveyor is therefore the first training-free layer that decides both when a drafted subgoal plan can be reused and whether a goal needs subgoals at all. The new findings that follow from it are listed in our general response.

AuthorsOpenReview ↗

Response to Reviewer jPhB (2/4): Attribution and Runtime

Q2. Attribution of the improvement

"The major issue is that the paper overclaims the improvement. Most of the claimed improvements in the abstract and intro come from the existing decomposition mechanism ... about 1.3 difference between every-step drafting vs the full system is hard to argue for the contribution of the whole system."

A2: Thank you for this comment. We would like to clarify that the paper attributes the long-range improvement to decomposition and not to our two decisions. The submission states this in several places:

  • Section 4 opens with: "gains over the vanilla executor at long range come mostly from decomposition, which the drafter provides, whereas gains over every-step drafting, in drafter calls and in short-range success, come from the accept rule and the arbiter."
  • Section 4.2 states: "Most of this lift comes from decomposition itself. Every-step drafting, with neither of our decisions, already solves 92.1% to 96.9% at PushT t=150 ..."
  • The first contribution reads "We show that decomposition lifts every executor", and the third contribution reports the gain of the arbiter relative to the accept rule alone.

To make this attribution equally clear on the first page, we have revised the abstract, which now states that every-step drafting shares the long-range lift and reports the gain of Surveyor over every-step drafting. We have also added Table 3, which adds one component at a time.

Regarding the size of the improvement, the 1.3-point difference corresponds to GC-IDM only, which is the one executor whose vanilla planner barely degrades with distance. For the four search executors, the full system improves over every-step drafting by 4.2 to 5.9 points in mean success and by 26 to 28 points in the worst setting. The results for all five executors are listed below.

Table: Every-step drafting (FF-JEPA's setting) versus the full system. Mean / worst success (%) over the nine PushT and Reacher settings with eight repeated runs (Table 17). Drafter calls are per replanning step at PushT t=150, where every-step drafting makes 1.00.

ExecutorEvery-step draftingSurveyorGain (mean / worst)Drafter calls of Surveyor
CEM91.2 / 65.295.4 / 90.9+4.2 / +25.70.58
iCEM88.1 / 61.594.0 / 89.7+5.9 / +28.20.64
MPPI87.6 / 54.292.7 / 81.0+5.1 / +26.80.67
Gradient91.6 / 66.896.3 / 93.3+4.7 / +26.50.60
GC-IDM94.2 / 80.695.5 / 93.5+1.3 / +12.90.73

As shown in the table, the full system outperforms every-step drafting for all five executors in both the mean and the worst setting, while making 27% to 42% fewer drafter calls. The improvement is concentrated where every-step drafting fails: decomposition alone lowers success at short range (CEM at Reacher t=25 falls from 95.7% to 65.2%), and the arbiter restores it (98.2%).

Q3. Time per episode

"Full system takes much longer per episode over every-step drafting. It is a trade-off between cost/time vs incremental improvement."

A3: Thank you for raising this concern. The additional time is spent on the arbiter's reachability probe, and it depends on the probe budget. To examine whether this time is necessary, we ran Surveyor with a lighter probe (100 samples and 10 iterations instead of 300 and 30; the budget was fixed before the run). Over CEM, Surveyor then runs as fast as every-step drafting (Table 24): 4.07 s versus 4.15 s per episode at PushT t=150, and 3.89 s versus 5.08 s at Reacher t=150. Meanwhile, its mean / worst success over the nine PushT and Reacher settings is 96.4% / 92.1%, compared with 91.2% / 65.2% for every-step drafting. Therefore, the improvement over every-step drafting does not have to be traded against time.

Note that these times exclude model loading, while the 7.8 s and 12.1 s in Table 3 include it. All other results in the paper keep the default probe, so that no setting is selected on the evaluation results.

AuthorsOpenReview ↗

Response to Reviewer jPhB (3/4): Blind Reuse

Q4. Blind reuse and the full system

"The blind two-waypoint reuse basically achieve the same success while having fewer drafter call and smaller runtime."

"Needs more evidence to show the importance of the full system vs. the drafter."

A4: Thank you for this observation. Blind reuse indeed matches the full system at PushT t=150, which is the setting of the cost table. However, this does not hold in general. Section 4.3 of our submission reports that "on Reacher the check itself matters, not only the reuse", where the blind chain trails the accept rule by 3 to 18 points for every search executor. To make the comparison explicit, we list the blind two-waypoint chain together with the accept rule and the full system below.

Table: Blind two-waypoint reuse / accept rule / Surveyor. Success (%) with eight repeated runs (Tables 4 and 5).

SettingCEMiCEMMPPIGradient
PushT t=15097.2 / 97.7 / 97.195.0 / 93.9 / 93.193.6 / 94.9 / 94.498.0 / 96.8 / 96.9
Reacher t=10091.1 / 94.5 / 94.089.9 / 90.3 / 95.989.4 / 92.7 / 96.293.2 / 95.5 / 96.4
Reacher t=5078.7 / 82.6 / 97.673.4 / 79.7 / 98.268.7 / 78.5 / 93.880.0 / 86.6 / 98.8
Reacher t=2557.7 / 62.8 / 98.249.8 / 53.0 / 98.848.1 / 52.4 / 81.059.7 / 63.6 / 96.5

Three observations can be made from the table. First, blind reuse matches the full system only at PushT t=150, and it is 19 to 49 points below Surveyor at Reacher t=25 and t=50 for every executor. Second, the consistency check improves over blind reuse: on Reacher, the accept rule outperforms the blind two-waypoint chain for every executor at every distance, by 0.4 to 9.8 points. Third, the arbiter contributes the short-range recovery, as the full system is a further 12 to 46 points above the accept rule at Reacher t=25 and t=50.

Regarding drafter calls and runtime, the blind two-waypoint chain and the accept rule make a similar number of drafter calls at PushT t=150 (0.51 versus 0.58 to 0.67 per replanning step, compared with 1.00 for every-step drafting). The remaining difference in runtime comes from the arbiter's probe, which the lighter probe removes (see A3).

Regarding the importance of the full system relative to the drafter, the table in A2 provides this comparison for all five executors. The drafter alone (every-step drafting) is 1.3 to 5.9 points below the full system in mean success and 13 to 28 points below it in the worst setting.

In summary, blind reuse works well only when the execution follows the draft closely. The consistency check makes reuse reliable for every executor, and the arbiter recovers the short-range success lost by decomposition. Appendix J reports reuse with and without each component.

Reviewer_oa7EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces Surveyor, a decomposition layer which breaks up the typical hierarchical planner pattern by allowing the planner to draft goals with multiple steps ahead, an accept rule which automatically continues the draft when the agent is on-track, and an arbiter, which re-plans when the goal is out of bounds. The authors run extensive experiments across multiple executors and environments to show that their technique has the longest success across the board while having fewer drafter calls overall. Notably, this technique is training-free and only requires some calibration on an offline observation pair data.

Critical strengths

The motivation and presentation of the paper is very clean and clear. It is very easy to see what the problem the paper is trying to tackle, the different components and their contributions, and how they are orchestrated to resolve the state issue with over-drafting and unnecessary de-compositions into sub-agents. Experimental setup is thorough and the presentation of the results make it clear the difference between the method and the baselines it is augmenting.

Experiments are thorough, with 5 different executors and 4 environments which Surveyor is evaluated on, and it looks like overall across the board, Surveyor improves performance throughout.

Critical weaknesses

As noted in 4.3, it does seem like Surveyor seems to be slower on half of the tasks, where the drafting time is very low (1.3%) of the time, so it seems like in terms of efficiency, this will depend on the cost of the drafter, whether we are saving potential compute costs or time. The motivation of the paper are to reduce over-drafting and over-decomposition, which I would associate these with time constraints, but if Surveyor is slower, then the motivation might need to be reformulated to also include things like compute / API costs.

This work might seem a bit incremental, since this kind of control over the drafting frequency seems to be a common knob in order hierarchical planning work. Episode-level routing seems to be a fairly standard adaptive router Arbiter.

Questions for authors

How much is the cost to configure and calibrate the Surveyor components in comparison to actually running the method on the datasets? How much data is required for this component, compared to the actual runtime environment?

4 comments
AuthorsOpenReview ↗

Response to Reviewer oa7E (1/2)

Dear Reviewer oa7E,

We sincerely appreciate your positive assessment and your constructive comments. We provide point-by-point responses below.

Q1. Motivation and cost

"As noted in 4.3, it does seem like Surveyor seems to be slower on half of the tasks, where the drafting time is very low (1.3%) of the time ... the motivation might need to be reformulated to also include things like compute / API costs."

A1: Thank you for this constructive suggestion. We have reformulated the motivation accordingly. The introduction now states that the cost of re-drafting grows with the subgoal model: it is small for the lightweight diffusion drafter used in our experiments, and it becomes the main expense when subgoals come from large generative models or from models served remotely, where every call has a computational or monetary price. Accordingly, we report the saving of Surveyor in terms of drafter calls. At the longest PushT distance, Surveyor makes 0.58 to 0.73 of the calls of every-step drafting, while its success is higher than or within 0.7 points of every-step drafting for every executor.

Regarding the runtime, we would like to clarify that the additional time comes from the arbiter's reachability probe rather than from drafting. The probe is what recovers the success lost by over-decomposition. For example, at Reacher t=25 over CEM, decomposing every goal lowers success from 95.7% to 65.2%, and the arbiter restores it to 98.2%.

To further address your concern, we examined how much of this additional time is necessary by running Surveyor with a lighter probe (100 samples and 10 iterations instead of 300 and 30). The results are presented below and in Table 24 of the revision.

Table: Time and success with a lighter reachability probe. CEM on one A100. Seconds per episode are measured at t=150 and exclude model loading. Success is the mean / worst over the nine PushT and Reacher settings.

Every-step drafting (FF-JEPA)Surveyor, default probeSurveyor, lighter probe
Seconds per episode, PushT4.156.164.07
Seconds per episode, Reacher5.085.323.89
Success (%), mean / worst91.2 / 65.295.4 / 90.996.4 / 92.1

As shown in the table, with the lighter probe Surveyor runs as fast as every-step drafting while keeping its higher success. This indicates that the additional time is determined by the probe budget and can be removed. We fixed the budget of the lighter probe before running it, and all other results in the paper keep the default probe, so that no setting is selected on the evaluation results.

Finally, compared with the vanilla executor, which is the comparison made in Section 4.3, Surveyor with the default probe is already cheaper per solved episode at the longest distance in 13 of the 20 executor–environment pairs (Figure 3 and Table 25).

AuthorsOpenReview ↗

Response to Reviewer oa7E (2/2)

Q2. The two decisions and prior work

"This work might seem a bit incremental, since this kind of control over the drafting frequency seems to be a common knob in [other] hierarchical planning work. Episode-level routing seems to be a fairly standard adaptive router Arbiter."

A2: Thank you for raising this point. While the drafting frequency can indeed be exposed as a knob, we would like to emphasize that Surveyor does not set a knob in advance. Instead, it makes both decisions online for each episode. In the hierarchical latent planners closest to ours, both choices are fixed beforehand: FF-JEPA re-drafts subgoals at every replanning step and decomposes every goal, and SAGE switches to the final goal based on the known remaining horizon. Adaptive triggers have been proposed for single-level plans (AdaReP and RDM), but they decide when to recompute one plan and do not address subgoals or decomposition.

To our knowledge, a fixed setting cannot serve every goal; never decomposing fails at long range (below 10% at PushT t=150 for every search executor), while always decomposing fails at short range (65.2% at Reacher t=25 over CEM). In contrast, Surveyor decides from the frozen world model's own predictions, without being told the goal distance and without any training. Its only tolerance is measured offline from the benchmark's success criterion.

Regarding episode-level routing, we also evaluated the most direct adaptive router, which thresholds the latent distance between the current state and the goal with the same tolerance. The comparison is given below.

Table: Four ways to decide whether to decompose. Success (%) at Reacher t=25.

DecisionCEMiCEM
Never decompose (vanilla executor)95.797.4
Always decompose (every-step drafting, FF-JEPA)65.261.5
Route on the latent distance to the goal63.052.5
Route on the predicted outcome of a flat plan (Surveyor)98.298.8

As shown in the table, the distance-based router almost never routes, whereas the arbiter's test on the predicted outcome of a flat plan recovers the short-range success. To the best of our knowledge, Surveyor is the first training-free layer that decides both when a drafted subgoal plan can be reused and whether a goal needs subgoals at all. A detailed comparison with FF-JEPA, SAGE, AdaReP and RDM is given in our general response and in Section 2 and Table 29 of the revision.

Q3. Cost of calibration

"How much is the cost to configure and calibrate the Surveyor components in comparison to actually running the method on the datasets? How much data is required for this component, compared to the actual runtime environment?"

A3: Thank you for the question. Calibration requires neither environment interaction nor training. It only encodes observation pairs from the offline dataset of each environment, which takes a few minutes. We have added these numbers to Section 4.1 and list them below.

Table: Offline calibration per environment. One A100. Evaluation refers to the full system over CEM at every goal distance with eight repeated runs.

EnvironmentObservation pairsCalibration (minutes)Evaluating one configuration (GPU-hours)
PushT4582.517.4
Reacher5023.62.1
Cube5123.07.2
Two-Room6,000at most 7.9 (two encoders)2.7

As shown in the table, calibration takes at most 5% of the GPU time needed to evaluate a single configuration. Moreover, it is performed once per environment and reused for all executors and goal distances.

We appreciate your thoughtful feedback and hope we have addressed your concerns. Please let us know if you have any additional questions.

Sincerely, Authors

Reviewer_oa7EOpenReview ↗

Response to Rebuttal

I thank the authors for their thoughtful responses.

I will keep my rating and raise my soundness score.

We are deeply grateful for your increased soundness score. Your valuable feedback has greatly contributed to the improvement of our work. We appreciate your time and consideration in reviewing our submission and maintaining your acceptance score.

Warm regards,

The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2): Novelty

We sincerely thank all reviewers and the area chair for their time and insightful comments. We are glad that the reviewers found that the paper "addresses a practically meaningful problem" (Reviewer 4XqY), that its motivation and presentation are "very clean and clear" (Reviewer oa7E), and that it "addresses different modes of failure together" (Reviewer jPhB). We have uploaded a revised manuscript with all modifications highlighted in blue. Section, table and appendix numbers below refer to the revised PDF.

Since all three reviews raise the question of novelty, we address it first. The second part of this response summarizes the new experiments.

Novelty

Our study is novel. Prior hierarchical planners for latent world models, such as FF-JEPA, focus on how to generate good subgoals. Once a subgoal model is available, they use it on a fixed schedule: the subgoals are regenerated at every replanning step, and every goal is decomposed. In this paper we study the question that comes next, namely when the output of a subgoal model can be reused and when it should not be used at all. To the best of our knowledge, this question has not been studied for latent world-model planning. We believe it is an important one, because always decomposing lowers the success of every executor at Reacher t=25 by 19 to 36 points (for example, CEM falls from 95.7% to 65.2%).

Our method differs from the methods cited by the reviewers.

  • FF-JEPA provides the subgoal model, which we use as our drafter. It re-drafts at every replanning step and always decomposes. This setting is the baseline called "every-step drafting" in our paper. We evaluate it for every executor at every goal distance (Appendix A), and it is now a row of the new Table 3 in the main text.
  • AdaReP and RDM decide when to recompute a single-level plan. Since they have no subgoals, they neither test whether a drafted subgoal is still valid nor decide whether a goal should be decomposed. Section 2 of our submission already discusses both methods and states that the accept rule instead governs a drafted subgoal sequence.
  • SAGE (concurrent work) switches to the final goal based on the known remaining horizon, whereas Surveyor is not told the goal distance.

To the best of our knowledge, the arbiter is the first test that uses the world model's own prediction to decide whether a goal needs subgoals at all. A simple threshold on the latent distance to the goal does not have the same effect: it solves 63.0% of the episodes at Reacher t=25 over CEM, whereas the arbiter solves 98.2%. In addition, both decisions use a tolerance obtained by a single calibration rule from the benchmark's success criterion, which involves no tuning. Surveyor also requires no training and is applied unchanged to five executors (sampling-based, gradient-based and amortized) in four environments.

Our findings are novel. To the best of our knowledge, the following observations are not reported in the cited works.

  1. Decomposition comes at a cost at short range even with FF-JEPA's diffusion subgoal model, which shows no such cost in FF-JEPA's own PushT evaluation. On Reacher, the cost appears for all five executors (Section 4.2).
  2. The predicted outcome of a flat plan indicates whether a goal needs subgoals, while the latent distance to the goal does not. On PushT and Reacher, the flat planner solves 99% of the episodes that the arbiter hands to it (new Appendix H).
  3. Routing does more than fall back on the flat planner. At Reacher t=25, Surveyor solves more episodes than the flat planner alone (98.2% versus 95.7% over CEM, and 98.8% versus 97.4% over iCEM).
  4. The long-range gain of decomposition is bounded by the short-range success of the executor (Appendix D), which explains where decomposition helps.
  5. A tolerance measured from the success criterion is within 2.0 points of the best routing threshold in all seventeen settings (Table 20).

As a result, a single planner stays reliable at every goal distance without being told the distance: the worst PushT or Reacher setting of every executor is 81.0%–93.5% under Surveyor, compared with 1.8%–62.7% for flat planning and 54.2%–80.6% for FF-JEPA's every-step drafting.

AuthorsOpenReview ↗

General Response to All Reviewers (2/2): New Experiments

Following the reviewers' suggestions, we conducted the additional experiments below. All of them are included in the revised manuscript.

  1. What each component adds (Reviewers jPhB and oa7E). The new Table 3 adds one component at a time: the vanilla executor, every-step drafting with FF-JEPA's subgoal model, the accept rule, and the arbiter. Over the four search executors, the worst PushT or Reacher setting rises from 54.2%–66.8% under every-step drafting to 81.0%–93.3% under Surveyor, while the long-range success is maintained.

  2. Accuracy of the reachability estimate (Reviewer 4XqY). The new Appendix H logs the decision of the arbiter for every episode and runs both planners on the same episodes. On PushT and Reacher, the flat planner solves 99% of the episodes that the arbiter routes to it, and Surveyor stays within 2.9 points of an oracle that selects the better planner for each episode.

  3. A single tolerance for both decisions (Reviewer 4XqY). Table 20 varies the routing threshold from 0.5 to 3 times the calibrated tolerance in all four environments. The calibrated value is within 2.0 points of the best threshold in all seventeen settings.

  4. End-to-end time (all reviewers). Table 24 evaluates Surveyor with a lighter reachability probe. Over CEM, it then runs as fast as every-step drafting (4.07 s versus 4.15 s per episode at PushT t=150), with a higher mean success over the nine PushT and Reacher settings (96.4% versus 91.2%).

  5. Calibration cost (Reviewer oa7E). Section 4.1 now reports the cost of calibration: 458 to 6,000 observation pairs from the offline dataset and 2.5 to 3.6 minutes on one GPU per environment (at most 8 minutes on Two-Room), without any environment interaction.

  6. Additional controls (Reviewers jPhB and oa7E). Appendix I increases the cost of each draft (Table 23), and Appendix J adds a control that keeps the arbiter and reuses drafts without the consistency check.

  7. Positioning (all reviewers). Section 2 and the new Table 29 compare Surveyor with FF-JEPA, SAGE, AdaReP and RDM.

We kept the changes to the main text small. They consist of two clauses in the abstract, two additions to the list of contributions, Table 3 with one paragraph, and a few sentences in Sections 1, 2, 3.5, 4.1, 4.3, 4.4 and 5. The remaining additions are in the appendix.

Should any aspect require further clarification, we welcome additional questions. If the reviewers find that their concerns have been addressed, we would appreciate their consideration in updating the score to reflect these improvements.

Area_Chair_HmZYOpenReview ↗

Please engage in the discussion

Dear Reviewers,

Thank you for your thoughtful reviews. The authors have posted their rebuttal, please read it and share any updated assessments.

As you know, we have only 3 reviews in total. Each piece of your opinions is valuable and may have crucial impact on the final decison. So please take it seriously.

Thanks again for your contribution!

Best Regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.