acceptodds
Under review as a conference paper at ICLR 2027

Planning Is Test-Time Scaling: A World Planning Model

Abstract

Effective decision-making requires more than predicting the future. An agent must also know how to use predicted outcomes to improve its actions. We introduce the World Planning Model (WPM), a learned iterative decision process that couples future prediction with action refinement in a single network. Starting from an initial action proposal, WPM predicts its consequences and uses the resulting future representation to update the action, repeatedly applying the same operator to produce progressively refined decisions. By training prediction and refinement jointly, WPM learns not only task-relevant dynamics, but also how predictive information should be translated into action updates. This replaces explicit candidate sampling, scoring, and selection with implicit refinement, while retaining the ability to allocate additional computation at test time through more refinement steps. In offline goal-conditioned control from pixels, WPM matches or exceeds explicit planners built on the same model on most environments. Furthermore, WPM's success rises with refinement depth on most environments without retraining. Controlled ablations further indicate that predictive supervision improves the learned decision process. These results suggest a general alternative to both one-shot policy execution and explicit search: learning a reusable planning operator whose computation can be scaled at test time.

Primary area:
applications to robotics, autonomy, planning
Keywords:
planning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WLzeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses offline goal-conditioned RL. The idea is to learn a "forcaster" which rolls out future (embeddings of) observations, a "refiner" that iteratives on imagined actions, and a policy which adjudicates the next action on the basis of these model's outputs, but which interestingly is not constrained to pick the utility-maxing imagined rollout.

They evaluate on OGBench, offline goal conditioned simulated robot manipulation tasks. They compare against published OGBench results, a JEPA baseline and a TD-MPC baseline (which admittedly I'm not familiar with), and various ablations/versions of their approach, finding their approach generally >= alternatives, and that it improves with extra refinement steps.

Critical strengths

The paper blends a bunch of nice research areas -- offline RL, test time compute, world models -- packaging them in one system. It's nice that the system does not degrade and in fact can somewhat improve with extra test time compute.

Critical weaknesses
  1. The idea that extra test time compute can aid model based control is an old one. It's great your method benefits from extra inference time compute, but so does a planner. How does your method's test time compute scale vs. a planner? No analysis of runtime, which is odd for a work whose signature contribute is advertised in the title as "test time compute". Can you make graphs showing how your scaling fares vs. conventional MPC / planning? Can you compare to MuZero / other systems that integrate sophisticated search (not CEM) with learning?
  2. Complex method -- I get that you're combining many cool ideas at once, but the payoff is not commensurate with the complexity. There's 5 different losses, two different training stages, and arbitrary hyperparameters with no corresponding sensitivity analysis.
  3. Little discussion of nondeterminism and partial observability -- these are central concerns in model based control and deeply influence the nature of the planning problem, and indeed what we expect the forcaster to represent. Is OGBench stochastic? If so we expect the embeddings to encode belief states, not future observations. If your method effectively crumbles under nondeterminism, that would be good to be upfront about.
Questions for authors

How does your method's use of test time compute scale differently/the same/wall-clock-wise vs planning approaches? Can you handle stochastic environments, not just in practice on-average on a benchmark, but theoretically in-principle?

Reviewer_pGbWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes WPM, a three-part architecture (encoder, forecaster, actor) that iteratively refines 8-step action chunks over N test-time steps using shared weights. At each step, the actor proposes a chunk, the forecaster writes predicted consequences into three latent "future slots," and the actor reads those slots to output a revised chunk. Training proceeds in two stages: Stage 1 substitutes observed futures into the slots alongside contrastive grounding (L_ground) and action-contrast (L_ac) losses, while Stage 2 switches to predicted futures with a prior–posterior alignment loss (L_pp). The authors claim that joint prediction and refinement improves policy performance, matches or beats explicit candidate selection built on the same model, and enables test-time scaling with refinement depth N without retraining.

Critical strengths
  1. Well-Motivated Formulation, Gradient Design, and Controlled Baselines: Asking whether a policy can learn to use forecasts internally rather than handing candidates to an external scorer is timely and well-motivated. The stop-gradient argument for Lpp\mathcal{L}_{\text{pp}} (§3.4) is carefully reasoned to prevent the actor from exploiting forecaster errors, and the explicit planning baselines (Q-argmax, CEM, and imagined-futures) all draw candidates from the exact same N=0N=0 proposal, cleanly isolating the selection rule.

  2. Informative Mechanism Diagnostics & Candid Reporting: The wrong-goal substitution experiment (Table 2) convincingly shows that the actor actively depends on its latent slots, while the attention and cosine-similarity analyses (Fig. 2) offer a useful window into how the slot representations behave across refinement steps. The detailed reproducibility statement, promised paired-bootstrap protocol, and candor about non-monotonic depth gains are also clear strengths.

Critical weaknesses
  1. Missing BC Initialization Baseline, Hidden Value Supervision, and Overstated Depth Gains: WPM initializes from a "converged goal-conditioned behavior-cloning policy" (§3.5) with action chunking and 256-bin discretization, yet never reports that baseline's performance. Because WPM at N=0N=0 already vastly exceeds published GCBC (e.g., 50.9 vs. 1 on cube-double; 61.2 vs. 12 on scene) and matches HIQL, most of the gain over published rows likely comes from the backbone plus a hidden offline-RL ingredient: Lground\mathcal{L}_{\text{ground}} is advantage-weighted by a frozen GCIVL value function (Eq. 5), and two of the three future slots (offsets 16 and 24) sit beyond the 8-step chunk, predicting value-preferred continuations (like HIQL subgoals) rather than pure consequences of the proposal. Meanwhile, actual refinement changes from N=0N=0 to N=4N=4 (Table 2: +1.2,+1.4,−0.4,+0.2,+2.6,+0.8,+8.4,+4.8+1.2, +1.4, -0.4, +0.2, +2.6, +0.8, +8.4, +4.8) exceed the 1–3 point seed noise on only 2 of 8 tasks (puzzle-4x4 and noisy cube-double). Table 3’s Δ\Delta inflates scaling by picking the best depth on evaluation episodes, only 3 of 8 gains are positive across all seeds, and gains saturate after 1–2 steps while N=4N=4 costs 3.4–5.3×3.4\text{--}5.3\times the compute—making the "Planning Is Test-Time Scaling" title an overstatement.
  2. Diagnostics Cut Against the "Planning" Story & Baseline Comparisons Have Confounds: Writing the true future into the slots at step 1 collapses puzzle-4x4 from 72.5 to 22.4 and hurts scene (Table 2), and slot cosine similarity to the realized future drops from 0.73–0.940.73\text{--}0.94 at step 1 to 0.12–0.690.12\text{--}0.69 at step 4 (§4.2). Thus, beyond step 1, the slots act like unconstrained recurrent state (cf. Tur et al., 2026) rather than dynamics forecasts—yet the crucial "all-slot twin" ablation (recurrent depth without forecasting supervision) covers just one task in the main text (Table 1) and adds only +2.2 points in aggregate (Appendix C.2). Furthermore, the explicit-planner comparisons are confounded: WPM executes the distribution mean on cube tasks while planners execute samples at temperature 0.5 (§4.1); Q-argmax and CEM rely on a weak GCIQL critic (1% published on cube-double); and the fairest explicit baseline—the imagined-futures planner—actually beats WPM on cube-double, cube-triple, and puzzle-3x3, while tying within noise on cube-single and cube-quadruple.
Questions for authors
  • Questions for Authors:
    1. BC Baseline & Value Weighting: What is the success rate of the initializing BC policy under the exact same protocol, and how does it compare to WPM at N=0? How do results change with unweighted grounding (w_{b,j} = 1), and do the offset-16 and offset-24 slots still predict the realized future when conditioned on a non-recorded chunk?
    2. True-Future Diagnostic & Recurrent Twin: How is the "true future at step 1" obtained at evaluation time, and why does it hurt puzzle-4x4 by 50 points? Can you report the all-slot twin across all eight environments in the main text, with paired-bootstrap CIs for both N=4 vs. N=0 and WPM vs. imagined-futures? Also, without L_ground in the Table 1 ablation, what trains the forecaster during Stage 1?
    3. Sampling Confounds & Training Depth: How do the explicit planners perform when executing the distribution mean, or WPM when sampling at temperature 0.5? What range of N_b is sampled during training, and is N=8 extrapolation?
    4. Reporting Discrepancies: Were Table 2 checkpoints selected on evaluation episodes? Please reconcile §4.1 ("fixed checkpoints, on which nothing is tuned") with Table 2’s caption ("each at its best checkpoint"), reconcile cube-double numbers between Table 1 (49.3 / 54.5) and Table 2 (50.9 / 52.3), and fill in the missing seed counts/standard deviations in Tables 2 and 3.
  • Additional Feedback & Minor Cleanup:
    • Consider retitling the paper and recasting "test-time scaling" as "variable refinement depth with early saturation." Avoid using the ≤ 2% re-implementations of TD-MPC2 and LeWorldModel to claim WPM is a "more reliable alternative" (§4.3), as those numbers suggest a setting mismatch.
    • Move the definitions of L_ac, L_cons, the depth schedule, and the twin setups from the appendix into §3.4, and replace the confusing term "implicit selection" (since no candidate set is selected from).
    • Discuss iterative amortized inference (Marino et al., 2018) and diffusion-policy denoising as closely related forms of iterative action refinement, and bring key related-work distinctions from Appendix F into the main text.
Reviewer_XDKyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper introduces a method to review action selection based on a sampled one-step lookahead obtained from an iteratively refined action selection. A vector summarizing what happens at the next step is refined based on a selected action and action selection is refined based on the updated vector.

This can be viewed as a kind of iterative in-context learning of action selection where the context is the results of next actions.

Critical strengths

This appears to be a novel combination of in-context learning and planning.

Critical weaknesses

The paper lacks clarity in many places and, more importantly, key comparisons and analyses.

Instead of doing the iterative refinement of futures and actions, what if N actions were sampled, their next state encodings were obtained, and this was then aggregated to sample the next action? This test will determine if any of the learning of the futures and actions is even necessary.

How does the distribution over actions change as the number of futures increases? Is there any redundancy?

Not much is said about how one can expect this method to scale as more actions are sampled. This method can, at best, look at every possible next state. However, it does not look beyond this.

In terms of formalisms, the action proposer in Equations 3 and 4 return \pi, which is the typical symbol for a policy. However, \pi is then defined as a distribution over actions for that state. A policy is a mapping from states to probabilities over actions, not probabilities for a single action.

Questions for authors

What parameters do the losses optimize? What is the recorded action chunk? The action to imitate? Are there any visualizations on how the action selection changes across time? “The true future at step 1” What is meant by the “true future”?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.