- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework for goal-conditioned control using a frozen world model. The framework first trains a policy and a world model using offline data, and then improves the policy using a GRPO-like loss that assigns greater weight to action chunks with higher imagined rewards. At inference, the world model can select among multiple sampled action chunks, while an optional guidance mechanism incorporates additional constraints without retraining. Experiments on four environments report success rates and lower latency than CEM.
Critical strengths
S1. Thorough experimental design. The paper reports every combination of BC vs. BC+WAVR and N=1 vs. N=256, compares against a broad set of baselines, and replicates results over multiple pretraining draws.
S2. Practical constraint adaptation without retraining. Sec. 4.3. Compose adds a keep-out constraint at test time without retraining any model, reaching 91.6% joint success on Cube versus 55.9% for constrained CEM. This flexibility is useful when deployment constraints change after training.
Critical weaknesses
W1. Limited novelty. Learning a policy inside a world model and planning with learned proposals are both established approaches. The former was done in Dreamer 4 [1], which trains a policy inside an offline world model. For the latter, TD-MPC [2] accelerates model-based planning with samples from a learned policy. The GRPO-style update in WAVR is also very close to Pfrommer et al. [3], which combines group-relative advantages with weighted flow matching and scores sampled action chunks with a reward model. The authors should state clearly how their approach differs from these works and what advantages it offers.
W2. Efficiency gains over learned baselines are not clearly established. Although Figure 1 plots success against planning time for each method, many markers and whiskers overlap, so latency values for individual methods are difficult to read. The main text lacks a numerical latency comparison against other learned baselines (e.g., Diffusion Policy, GC-IDM), which makes it hard to assess the efficiency differences precisely. Since efficiency is central to the paper's claims, a detailed latency analysis would strengthen the evidence.
W3. Limited evaluation breadth. The manuscript reports results on only four environments and a single world-model backbone (LeWM), which limits the evidence for generality. Evaluation on more OGBench tasks and on other world-model backbones is needed.
W4. Presentation needs improvement. Figures 1 and 4 have heavily overlapping markers and lines, which makes individual methods hard to distinguish. Table 1 includes many variants of the proposed method (LeAP, LeAP-Flash, LeAP + refine, WAVR, WAVR-Flash, WAVR-zero, each at N=1 and/or N=256), which obscures the main comparison. Moving the less essential variants to the appendix would make the key results easier to follow.
[1] D. Hafner, W. Yan, and T. Lillicrap. Training Agents Inside of Scalable World Models. arXiv:2509.24527, 2025.
[2] N. Hansen, X. Wang, and H. Su. Temporal Difference Learning for Model Predictive Control. ICML, 2022.
[3] S. Pfrommer, Y. Huang, and S. Sojoudi. Reinforcement Learning for Flow-Matching Policies. arXiv:2507.15073, 2025.
Questions for authors
See weaknesses
Thank you for the detailed review.
W1, novelty. We should have cited Pfrommer et al. Its GRPO variant uses the same construction as WAVR's update. The difference is where the scores come from. Pfrommer et al. fit a reward model to true returns and collect new rollouts between rounds. WAVR scores chunks with a frozen pixel world model that was trained without reward labels, and it takes no environment step. Dreamer 4 and TD-MPC rely on reward and value predictors trained with the model. Here the model is given and frozen, and what we train afterwards is an independent policy network of 2.6M parameters. This is post-training for a latent world model, and it is where the work differs from both.
W2, latency against learned baselines. The numbers are in Table S6. Planner time per 50-step episode is 4,240 ms for CEM, 146 ms for GC-IDM, 53 ms for Diffusion Policy, 55 ms for LeAP and 56 ms for WAVR, and the Flash variants reduce it to 46 ms, against a floor of 41 ms for encoding the start and goal frames alone. WAVR-Flash reaches a four-suite mean success of 95.1 at that cost, against 96.0 for WAVR, with a 0.31M-parameter network. The arithmetic cost is smaller still: proposing a chunk with a Flash network takes about 6 ms and 2.5 MFLOP by PyTorch's flop counter, roughly 50 times less than the 130 MFLOP of the default proposer and 0.04% of the 6.8 GFLOP spent on the two encoder passes (Section 1; per-call times in Table S21). Our efficiency claim is against search, and among learned chunk policies the differences are small because the encoder dominates every row. We agree that these numbers belong in the main text.
W3, breadth. We agree, and we ran a second world model during the discussion: PLDM, with its objective trained on the LeWM backbone rather than on PLDM's original one and without its inverse-dynamics term, using three pretraining draws per suite and no changed hyperparameter. Success (%), mean over three draws and three evaluation seeds, with the LeWM value of Table 1 in parentheses:
The single-sample WAVR policy is above CEM on all four suites, and its gain over the prior has the size Table 1 shows. This is a second training objective on the same architecture and suites, not a second architecture. PushT is lower on it in every row, and the PushT loss at N=256 is larger.
W4, presentation. We agree. A future version will keep the main comparison in Table 1, move the variants to the appendix, and redraw Figures 1 and 4.