acceptodds
Under review as a conference paper at ICLR 2027

Extracting Behavior from Frozen Latent World Models: Experience-Anchored Proposals with Offline Verifier Alignment

Abstract

Planning in frozen latent world models is usually performed with the cross-entropy method (CEM). CEM is memoryless: every replan starts a new Gaussian search that uses none of the experience the model was trained on. Planning is therefore slow, 2.1s per replan in our setup, and uneven, from 67% to 91% success across four suites. We replace this search with a proposer that learns from the model's experience. A generative proposer is behavior-cloned from the corpus. At plan time it generates candidates and the model selects among them by imagined latent distance to the goal. We then improve the proposer by offline reinforcement learning with this imagined distance as the reward, and analytic relative-position constraints can be added at plan time without retraining. We recommend a rectified-flow proposer, the LeWM Action Proposer (LeAP), improved by World-model Advantage-weighted Verifier Regression (WAVR), which needs no trust-region term. With 256 candidates, a replan takes 24ms and of CEM's model evaluations, about 90 times faster per planning call. On Cube, TwoRoom and PushT, whose corpora contain directed or expert behavior, the behavior-cloned proposer alone reaches 99.9%, 98.0% and 96.2%, against 67.0%, 88.8% and 91.4% for CEM. On Reacher, whose corpus is random behavior, WAVR raises success from 76.1% to 97.3%, against 84.3% for CEM. With a keep-out region added on Cube at plan time, the planner reaches the goal without entering it in 91.6% of episodes, against 55.9% for CEM.

Primary area:
reinforcement learning
Keywords:
latent world models, model-based planning, offline reinforcement learning, JEPA, flow matching, cross-entropy method, goal-conditioned control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jrUbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a framework for goal-conditioned control using a frozen world model. The framework first trains a policy and a world model using offline data, and then improves the policy using a GRPO-like loss that assigns greater weight to action chunks with higher imagined rewards. At inference, the world model can select among multiple sampled action chunks, while an optional guidance mechanism incorporates additional constraints without retraining. Experiments on four environments report success rates and lower latency than CEM.

Critical strengths

S1. Thorough experimental design. The paper reports every combination of BC vs. BC+WAVR and N=1 vs. N=256, compares against a broad set of baselines, and replicates results over multiple pretraining draws.

S2. Practical constraint adaptation without retraining. Sec. 4.3. Compose adds a keep-out constraint at test time without retraining any model, reaching 91.6% joint success on Cube versus 55.9% for constrained CEM. This flexibility is useful when deployment constraints change after training.

Critical weaknesses

W1. Limited novelty. Learning a policy inside a world model and planning with learned proposals are both established approaches. The former was done in Dreamer 4 [1], which trains a policy inside an offline world model. For the latter, TD-MPC [2] accelerates model-based planning with samples from a learned policy. The GRPO-style update in WAVR is also very close to Pfrommer et al. [3], which combines group-relative advantages with weighted flow matching and scores sampled action chunks with a reward model. The authors should state clearly how their approach differs from these works and what advantages it offers.

W2. Efficiency gains over learned baselines are not clearly established. Although Figure 1 plots success against planning time for each method, many markers and whiskers overlap, so latency values for individual methods are difficult to read. The main text lacks a numerical latency comparison against other learned baselines (e.g., Diffusion Policy, GC-IDM), which makes it hard to assess the efficiency differences precisely. Since efficiency is central to the paper's claims, a detailed latency analysis would strengthen the evidence.

W3. Limited evaluation breadth. The manuscript reports results on only four environments and a single world-model backbone (LeWM), which limits the evidence for generality. Evaluation on more OGBench tasks and on other world-model backbones is needed.

W4. Presentation needs improvement. Figures 1 and 4 have heavily overlapping markers and lines, which makes individual methods hard to distinguish. Table 1 includes many variants of the proposed method (LeAP, LeAP-Flash, LeAP + refine, WAVR, WAVR-Flash, WAVR-zero, each at N=1 and/or N=256), which obscures the main comparison. Moving the less essential variants to the appendix would make the key results easier to follow.

[1] D. Hafner, W. Yan, and T. Lillicrap. Training Agents Inside of Scalable World Models. arXiv:2509.24527, 2025.

[2] N. Hansen, X. Wang, and H. Su. Temporal Difference Learning for Model Predictive Control. ICML, 2022.

[3] S. Pfrommer, Y. Huang, and S. Sojoudi. Reinforcement Learning for Flow-Matching Policies. arXiv:2507.15073, 2025.

Questions for authors

See weaknesses

1 comment

Thank you for the detailed review.

W1, novelty. We should have cited Pfrommer et al. Its GRPO variant uses the same construction as WAVR's update. The difference is where the scores come from. Pfrommer et al. fit a reward model to true returns and collect new rollouts between rounds. WAVR scores chunks with a frozen pixel world model that was trained without reward labels, and it takes no environment step. Dreamer 4 and TD-MPC rely on reward and value predictors trained with the model. Here the model is given and frozen, and what we train afterwards is an independent policy network of 2.6M parameters. This is post-training for a latent world model, and it is where the work differs from both.

W2, latency against learned baselines. The numbers are in Table S6. Planner time per 50-step episode is 4,240 ms for CEM, 146 ms for GC-IDM, 53 ms for Diffusion Policy, 55 ms for LeAP and 56 ms for WAVR, and the Flash variants reduce it to 46 ms, against a floor of 41 ms for encoding the start and goal frames alone. WAVR-Flash reaches a four-suite mean success of 95.1 at that cost, against 96.0 for WAVR, with a 0.31M-parameter network. The arithmetic cost is smaller still: proposing a chunk with a Flash network takes about 6 ms and 2.5 MFLOP by PyTorch's flop counter, roughly 50 times less than the 130 MFLOP of the default proposer and 0.04% of the 6.8 GFLOP spent on the two encoder passes (Section 1; per-call times in Table S21). Our efficiency claim is against search, and among learned chunk policies the differences are small because the encoder dominates every row. We agree that these numbers belong in the main text.

W3, breadth. We agree, and we ran a second world model during the discussion: PLDM, with its objective trained on the LeWM backbone rather than on PLDM's original one and without its inverse-dynamics term, using three pretraining draws per suite and no changed hyperparameter. Success (%), mean over three draws and three evaluation seeds, with the LeWM value of Table 1 in parentheses:

TwoRoomCubePushTReacher
CEM97.7 (88.8)59.3 (67.0)75.8 (91.4)81.8 (84.3)
LeAP, N=193.4 (94.2)97.9 (99.9)81.9 (87.7)53.4 (55.4)
LeAP, N=25698.4 (98.0)97.6 (99.9)93.6 (96.2)74.3 (76.1)
WAVR, N=1100.0 (99.8)98.5 (99.9)83.6 (90.2)90.2 (94.0)
WAVR, N=25699.9 (99.7)98.4 (100.0)85.6 (92.6)95.1 (97.3)

The single-sample WAVR policy is above CEM on all four suites, and its gain over the prior has the size Table 1 shows. This is a second training objective on the same architecture and suites, not a second architecture. PushT is lower on it in every row, and the PushT loss at N=256 is larger.

W4, presentation. We agree. A future version will keep the main comparison in Table 1, move the variants to the appendix, and redraw Figures 1 and 4.

Reviewer_2zJMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper focuses on planning with frozen latent world models, which commonly use CEM, requiring extensive test-time search and resulting in slow planning. To address this issue, the paper introduces three core modules:

  1. LeAP, a lightweight diffusion/flow-based action proposer trained through behavior cloning.
  2. WAVR, an advantage-weighted regression training recipe for sharpening the distribution and improving best-of-1 performance.
  3. Composition, a trivial approach to guiding a diffusion policy through either cost-guided selection or energy-mode-based diffusion guidance.
Critical strengths
  1. The paper targets the relatively novel area of efficient planning with frozen latent world models. LeAP provides an intuitive approach to modeling the initialization distribution to accelerate or even eliminate search. To improve N=1 performance and reduce inference-time ranking, WAVR sharpens the distribution and distills best-of-N results into N=1 performance, forming a sound methodological path.
  2. The evaluation is in-depth, exploring variants that highlight opportunities for further speed improvements. Means and standard deviations are also reported across complete, independently trained runs, indicating the method’s robustness.
Critical weaknesses
  1. Insufficient baselines. PRISM [1] is a very similar work: both methods train a prior distribution to initialize CEM/MPPI search. A comparison against PRISM would therefore be appropriate. Although LeAP + refine is close in spirit to PRISM, the work is neither cited nor compared against.
  2. The overall presentation is poor, despite the method being simple and generally understandable. The ablations are crowded into the main table, with no bolded results to highlight performance. The analysis in Section 6 is sound but poorly structured and appears disconnected from the experimental settings.
  3. Section 6 attributes WAVR’s support preservation to its update rule. However, Appendix C.4 shows that weighted regression also becomes unstable under low-staleness on-policy sampling (FlowGRPO). This can indicate that WAVR is unstable and can cause severe policy drift, thus requiring a fixed buffer to constrain updates to a safe region. Such severe drift can itself contradict the goal of pass@k distillation, rendering the algorithm’s usability questionable.

[1] Wang et al., PRISM: PRior-guided Imagination Sampling in world Models

Questions for authors
  1. Why use Rectified Flow rather than DDPM? In Table S30, LeAP (RF) underperforms Diffusion Policy (DDPM) in terms of denoising steps, Pass@1, and Pass@256. According to [2], VP-SDEs can also produce more diverse outputs than SDEs derived from RF, which may provide a good starting point for RL. The authors need to explain why they selected RF despite these advantages of DDPM.
  2. Abnormal performance of flow models. In Table S24, LeAP performs better with fewer denoising steps and worse with more denoising steps, despite using a standard CFM training objective. The common expectation is that diffusion/flow-matching performance scales with test-time compute, which contradicts the paper’s results. I would expect further explanation or case studies addressing this behavior.

[2] Kim et al., Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcing

1 comment

Thank you for reading the supplement closely.

W1, PRISM. We were not aware of this paper and thank you for pointing to it. PRISM learns an action prior on the frozen encoder and fuses it into the sampling planner, which is the same premise as our propose-then-refine configuration. It uses the same frozen LeWM, the same four suites and the same 25-step goals, so its numbers can be read beside Table 1. The PRISM rows are taken from their paper, not from our own replication. Success (%):

TwoRoomCubePushTReacher
PRISM, MLP prior + MPPI, 32 samples81.882.880.660.0
PRISM, diffusion prior + MPPI, 32 samplesnot reported99.893.6not reported
PRISM, MLP prior + CEM, 128 samplesnot reported82.792.7not reported
LeAP, N=256 (Table 1)98.099.996.276.1
LeAP + refine (Table 1)98.399.995.576.0
WAVR, N=1 (Table 1)99.899.990.294.0

With a diffusion prior, PRISM is close to our rows on Cube and PushT, and the differences are on TwoRoom and Reacher, where PRISM reports the MLP prior only. PRISM also keeps a 30-iteration planner at test time, whereas WAVR at N=1 does not run the predictor at all. A future version of the paper will cite and discuss PRISM.

W2, presentation. We agree. The rows of Table 1 are baseline families and our variants rather than ablations, and the fact that they read as ablations shows that the table is overloaded.

W3, Section 6 and Appendix C.4. You read C.4 correctly: weighted regression at the gradient cadence drifts without an anchor (Table S32), and that control is ours. The fixed buffer, however, is part of WAVR's definition in Section 4.2, where each round samples one buffer from the current policy and fits it. With it, WAVR leaves 0.0% of its chunks outside the action support, against 17 to 18% for the gradient baseline (Table S41). Section 6 should therefore say "reweighting a fixed buffer sampled from the current policy" where it says "update rule".

Q1, rectified flow or DDPM. We trained a DDPM proposer in our network and ran WAVR from it, with the same advantage weights on the epsilon-prediction loss, both on LeWM and on PLDM (PLDM's objective on the LeWM backbone, see the general response). Only the row marked Table 1 is in the submission. Success (%) at N=1 / N=256:

World modelPolicyTwoRoomCubePushTReacher
LeWMDDPM proposer95.2 / 98.8100.0 / 100.087.0 / 94.850.5 / 83.2
LeWMWAVR from the DDPM proposer99.3 / 99.499.8 / 99.889.0 / 92.694.2 / 96.2
LeWMWAVR from the flow (Table 1)99.8 / 99.799.9 / 100.090.2 / 92.694.0 / 97.3
PLDMDDPM proposer95.1 / 99.393.7 / 96.280.1 / 92.741.9 / 79.4
PLDMWAVR from the DDPM proposer99.2 / 99.396.5 / 96.882.2 / 86.491.5 / 94.9
PLDMWAVR from the flow100.0 / 99.998.5 / 98.483.6 / 85.690.2 / 95.1

WAVR from the DDPM proposer ends within 1.2 points of the flow rows in every cell on LeWM and within 2.0 on PLDM, so WAVR does not depend on the flow. We keep the flow as the default because one network can be sampled at fewer Euler steps with no retraining (Table S22), although few-step samplers for diffusion models exist and we did not run them. Weighting a diffusion loss by reward or advantage is itself known (the DRWR and DAWR baselines of the DPPO paper, arXiv:2409.00588, and QVPO, arXiv:2405.16173), and what differs here is the source of the weights.

Q2, fewer steps doing better. With a straight-path flow, taking fewer Euler steps shrinks the samples toward the mean chunk, so the step count sets the width of the sample pool as well as the integration accuracy. This matters most on Reacher, whose corpus comes from a random policy: a narrower pool gives selection better candidates, which is why N=256 success rises from 76.8 at 25 steps to 89.2 at one step in Table S24. A direct test that we ran during the discussion supports this reading, since halving the initial noise while keeping all 25 steps gives 88.7, almost the same lift.

Reviewer_ZryPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to learn a policy from a pretrained, frozen world model, making planning faster. First, it uses behavior cloning to train a policy that generates actions toward a goal. Then, it improves the policy using offline reinforcement learning, where the world model predicts the outcomes of different actions and gives higher scores to those that get closer to the goal. During execution, the policy can directly generate actions or use the world model to choose the best among several options. Experiments on four environments show that this approach is much faster than traditional planning methods while achieving similar or better performance.

Critical strengths

The method uses a frozen world model to guide policy learning without requiring additional environment interactions. And the learned policy significantly reduces planning time compared to traditional search methods like CEM, while maintaining strong performance. Also, the policy can generate actions directly for fast execution or use the world model to select among multiple candidates for better performance.

Critical weaknesses

The overall framework resembles existing model-based RL approaches, while the proposed optimization is closely related to advantage-weighted regression and best-of-N distillation. Its advantages over more established approaches, such as Dreamer, are not fully demonstrated.

Also the proposed offline RL method mainly performs advantage-weighted behavior cloning, which relies heavily on existing action proposals and may struggle to discover better behaviors beyond the initial policy distribution.

I'm worried that the policy is optimized using goal distance over a fixed action chunk, without long-horizon value estimation or credit assignment. This may limit performance on tasks requiring complex planning or temporarily moving away from the goal.

About the experiments, offline RL improves performance significantly on Reacher but reduces success rates on PushT under best-of-N selection, suggesting that the optimization may not consistently improve the policy.

Questions for authors

N/A

1 comment

Thank you for the careful review. We answer the four concerns in order.

AWR and Dreamer. We agree that WAVR belongs to the advantage-weighted regression family, and Section 2 says so. Dreamer trains reward and value predictors with its model. Here the world model is frozen and was trained without reward labels, so the only training signal is the latent distance to a goal frame.

Behavior beyond the initial distribution. Table 1 includes WAVR started from random weights. On Reacher it reaches 91.9 / 96.2 (N=1 / N=256), above CEM at 84.3 and above the behavior-cloned prior at 55.4 / 76.1. The concern does hold on Cube and PushT, where the random start reaches only 67.4 and 60.3 at N=1.

Long horizons. We agree. The cost is a goal distance over one chunk, and TwoRoom success falls to 77.0 to 87.3 for goals 100 steps away.

PushT under best-of-N. You are right. WAVR lowers PushT at N=256 from 96.2 to 92.6, and on the second world model from 93.6 to 85.6 (general response). At N=1 no suite decreases. We inspected the PushT corpus to find out why. LeWM takes it from DINO-WM, whose authors describe it as expert trajectories replayed with various levels of noise. In the file we find 185 base trajectories, each followed by 100 open-loop replays with Gaussian action noise at ten levels. Only 64% of the replays end with the block at the target, within 20 pixels and 20 degrees. The spread of the behavior-cloned policy's samples follows that replay noise. So the update mostly removes spread, and spread is what selection among 256 samples uses. On Reacher, whose corpus comes from a random policy, the same update has much to learn. The loss is not specific to WAVR: Flow-GRPO with a KL anchor ends at 94.8 from the same 96.2 prior. The accurate claim is narrower than the one we made. WAVR improves the single-sample policy, and on expert data selection should stay at test time.

Comments on the paper

1 comment

We thank the three reviewers. All three reviews point to the presentation, and we agree that it is the main weakness of the paper. Table 1 and Figures 1 and 4 are overloaded, and Section 6 is hard to follow. The paper argues too many points at once, so its main claim is hard to find. It also reads as if the method required a rectified flow, and it does not.

Two points in the reviews could be settled by a measurement, so we ran both during the discussion period. Neither result is in the submission.

  • A second world model, PLDM (table in the reply to Reviewer jrUb). We train PLDM's objective on the LeWM backbone, with three pretraining draws per suite and every later stage unchanged. The single-sample WAVR policy reads 100.0 / 98.5 / 83.6 / 90.2 on TwoRoom / Cube / PushT / Reacher, against 97.7 / 59.3 / 75.8 / 81.8 for CEM.
  • WAVR from a DDPM proposer (table in the reply to Reviewer 2zJM), in the same network and on the same training windows. It ends within 1.2 points of the flow version in every cell on LeWM, and within 2.0 on PLDM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.