acceptodds
Under review as a conference paper at ICLR 2027

Learning multi-step policy for planning via world-model-guided flow matching

12 views

Abstract

Planning with learned world models has demonstrated strong performance in online reinforcement learning for continuous control, but prominent TD-MPC2-style methods generate action sequences by repeatedly sampling from a one-step Gaussian policy conditioned on model-predicted states. This construction limits the policy's ability to capture diverse planner-generated actions, reduces temporal coherence across the full sequence, and makes sequence generation sensitive to model prediction errors. We propose FLOWP, which replaces the one-step Gaussian policy with a multi-step flow policy. We train the policy with world-model-guided flow matching, which leverages model-predicted returns to improve the policy while regularizing it toward the planner's action distribution. By jointly sampling entire action sequences, our method can represent diverse planner-induced distributions and temporal dependencies while reducing sensitivity to model prediction errors. We evaluate FLOWP on 7 high-dimensional DMControl tasks and 14 HumanoidBench tasks using the same hyperparameters across all tasks. FLOWP demonstrates strong sample efficiency, achieving the highest normalized task-average performance throughout most of training on both benchmarks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, World Models, Flow Policies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NzoFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors present a method where they use flow matching to sample action sequences to optimise rewards in a sampling-based MPC style of algorithm. They argue this is more powerful than traditional approaches that overly rely on gaussian sampling, and present a range of theoretical evaluations to show their method improves over baselines.

Critical strengths
  • The paper's method is well motivated.
Critical weaknesses
  • The contribution is extremely thin. Using flow matching is a particular algorithmic choice that allows the authors to amortise the action improvement steps, but it does not really represent a novel contribution. Also, there is no reason why one needs to use gaussian distributions with existing methods, it is simply the common approach, purely for convenience. Essentially, what the authors propose is to learn a map that amortises the steps that otherwise a planner would do when run in receding horizon, but other existing works already do this amortised-style of planning, eg [1] do this amortisation by improving a stored prior policy that is then used to sample around at inference time.

  • The empirical results are equally thin. The method seems to improve, slightly, over existing variants, but not uniformly, and at the cost of learning this amortised transport map.

[1] Serra-Gomez, A., Ornia, D. J., Tirumala, D., & Moerland, T. (2025). A KL-regularization framework for learning to plan with adaptive priors. arXiv preprint arXiv:2510.04280.

Questions for authors

I do not really have any questions. Nothing is fundamentally wrong with the paper, it is simply very incremental, both conceptually and empirically, and I'm not convinced the extra complexity added by learning the transport map compensates or is worth the marginal improvement in the benchmarks.

Reviewer_m8jzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FLOWP (Flow Policy for World-model-based Planning), which replaces the one-step Gaussian policy used to guide planning in TD-MPC2-style methods with a multi-step flow policy that generates full action sequences jointly. To train the flow policy, the paper introduces world-model-guided flow matching, with an objective function derived from a KL-regularized policy objective. The method weights planner-generated action sequences by returns predicted by the learned world model, improving the policy while keeping it close to the planner's action distribution. The method is evaluated on 7 high-dimensional DMControl tasks and 14 HumanoidBench tasks, and achieves competitive performance on several continuous-control tasks.

Critical strengths
  1. Replacing the Gaussian policy prior in TD-MPC2-style planners with an expressive flow policy is a promising and timely direction. Flow and diffusion policies have been very effective elsewhere in RL and robotics, and applying them to online model-based planning is a reasonable step. The method is also implemented through a principled objective which is derived from a KL-regularized policy objective, giving a weighted flow-matching loss that both improves the policy using model-predicted returns and keeps it close to the agent's behavior.
  2. FLOWP is compared against a strong and current set of baselines, with results reported both as learning curves and tables of AUC and final return. On DMControl's high-dimensional dog and humanoid tasks, FLOWP achieves the highest average AUC, and the highest AUC on six of seven tasks, while matching the strongest baselines in final performance. The evaluation uses demanding high-dimensional tasks, with action spaces up to 61 dimensions, and one set of hyperparameters across all 21 tasks.
Critical weaknesses
  1. The paper motivates FLOWP by stating several problems with Gaussian policies: limited expressiveness for multimodal planner actions, lack of temporal coherence, and sensitivity to model errors. However, none of these claims seem to be directly supported empirically. The only reported ablation is on the flow matching weights, and is only performed on two tasks. Two separate properties are changed at once, the shift from Gaussian to flow policy, and autoregressive to joint generation. Testing a one-step flow policy used autoregressively might help to isolate expressiveness, and a multi-step Gaussian policy that outputs H actions jointly would isolate joint generation.
  2. The HumanoidBench aggregate advantage depends heavily on a single task. Returns are normalized by TD-MPC2’s final return, which is very small on several HumanoidBench tasks. On run, TD-MPC2 scores 28.5 (Table 5), so FLOWP’s 653 normalizes to 22.9 and contributes about 1.64 of its 5.12 average by itself. The claim made in the abstract of the highest normalized average on HumanoidBench depends largely on one task amplified by a small denominator. More robust aggregates would make this comparison more convincing, such as normalization by a fixed reference or per-task rank statistics.
Questions for authors
  1. Equation 16 samples st:t+H+1s_{t:t+H+1} and rt:t+Hr_{t:t+H}, which are states st,…,st+Hs_t, …, s_{t+H} and rewards rt,…,rt+H−1r_t, …, r_{t+H−1}. However, the target in Equation 15 at step h needs rewards up to rt+h+N−1r_{t+h+N−1} and the state st+h+Ns_{t+h+N}. At the last step (h=H−1)(h = H − 1), this will be st+H−1+Ns_{t+H−1+N}, which seems to be beyond the sampled segment whenever N>1N > 1. What TD horizon N is used for experiments, and what segment length is actually sampled for Eq. 16?
  2. TD-M(PC)² is reported to have very wide intervals on stand (645 ± 566) and walk (657 ± 526), which may suggest failed seeds in these experiments. Were these results validated against the original paper?
  3. Generating each policy candidate requires integrating the flow ODE over 10 steps (Table 1), which happens at every planning call during both training and inference. What are the planning time per environment step and total wall-clock training time for FLOWP relative to TD-MPC2 and BMPC, and how does performance vary with the number of flow steps?
  4. Maximizing expected return minus a KL penalty toward μ (Eq. 11) has the well-known closed-form solution π∗(a∣z)∝μ(a∣z)exp⁡(Jθ(z,a)/α)\pi^*(a | z) \propto \mu(a | z) \exp(J_\theta(z, a)/\alpha). Since π∗\pi^* is then defined as a tilted version of \mu, the improved policy can only reweight sequences from the current support of \mu (sequences that have already been executed). This limitation is acknowledged in the conclusion, but its practical consequences are not analyzed. Could the authors add some discussion or analysis of how this compares with gradient-based improvements, and whether this method can be successful when the starting policy is low-performing?
Reviewer_cpPqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets TD-MPC2-style planning with learned world models. These methods build H-step candidate sequences for MPPI by repeatedly sampling a one-step Gaussian policy along model-predicted states. The authors argue that this limits expressiveness, prevents modeling temporal dependencies across the sequence, and propagates model error. FLOWP replaces this prior with a flow-matching policy that generates an entire H-step action sequence from the current latent state. The policy is trained with “world-model-guided flow matching”: The objective maximizes the model-based bootstrapped return J minus a reverse-KL penalty toward the planner’s action distribution. Its closed-form optimum is the exponentially tilted distribution π∗∝μexp⁡(J/α)\pi^*\propto\mu\exp(J/\alpha). This is implemented as advantage-weighted flow matching on replay-buffer action sequences.

Critical strengths
  1. The key idea is simple: drop-in design. Only the policy prior and its training objective change, while world-model learning, value learning, and planning are untouched. This makes the method easy to reproduce and to combine with other improvements.

  2. It has a principled objective. The KL-regularized formulation unifies pure planner imitation (α → ∞) and return maximization in one framework, and its population-level equivalence to weighted flow matching is derived correctly.

  3. Broad evaluation under a single configuration. The paper covers 21 high-dimensional tasks, including a 61-dimensional action space, with one hyperparameter set. It reports both AUC and final return, along with full per-task tables.

Critical weaknesses

My main concerns are summarized as follows:

  1. The objective is not new. Eq. 11, which maximizes J minus a reverse-KL penalty toward the planner distribution, is the objective the paper itself attributes to TD-M(PC)² (Lin et al., 2026) in Section 4. The differences are only that J replaces Q and an H-step sequence replaces a single action.

  2. The derivation is standard. The closed-form solution and its reduction to weighted regression (Appendix A) are the classical reward- and advantage-weighted regression results (Peters & Schaal, 2007; Peng et al., 2019; Nair et al., 2020). Combining them with flow matching yields the weighted flow-matching objectives already used for RL policies (Zhang et al., 2025).

  3. The practical update is value-weighted planner imitation. BOOM (Zhan et al., 2025) already regularizes a Gaussian policy with a value-weighted forward KL toward the planner, i.e., a value-weighted likelihood on planner actions. Eq. 13 is essentially the flow-matching analogue of this term.

  4. The only distinguishing element is unvalidated. Joint H-step generation is the one choice not covered above, and flow policies over action chunks already exist (Li et al., 2025; Park et al., 2026). The paper provides no evidence that this element is responsible for the gains (see W5–W6).

  5. No ablation isolates the claimed benefits. The paper attributes its gains to expressiveness, temporal coherence, and robustness to model error, but there is no one-step flow policy or multi-step Gaussian policy as a control, and no experiment tests sensitivity to model error. The only ablation, removing the return weights, covers two tasks.

  6. H = 3 weakens the multi-step argument. With a three-step horizon, both “joint modeling of temporal dependencies” and “avoiding error propagation along the sequence” operate over very few steps. No results are given for other values of H.

  7. The derivation and the implementation train different distributions. The derivation assumes µ is the planner’s H-step plan distribution at sts_t. Per Algorithm 1, however, the buffer stores executed transitions, so at:t+Ha_{t:t+H} is a sequence of first actions from successive replans under past policies — a closed-loop behavior distribution. Approximating ZZ by exp⁡(V/α)\exp(V/\alpha) is also only a Jensen-type approximation, and weight clipping plus adaptive α make the trained objective a heuristic version of Eq. 12.

[1]Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., & Bellemare, M. G. (2021). Deep reinforcement learning at the edge of the statistical precipice. NeurIPS.

[2]Hansen, N., Su, H., & Wang, X. (2024). TD-MPC2: Scalable, robust world models for continuous control. ICLR.

[3]Li, Q., Zhou, Z., & Levine, S. (2025). Reinforcement learning with action chunking. NeurIPS.

[5]Lin, H., Wang, P., Schneider, J., & Shi, G. (2026). TD-M(PC)²: Improving temporal difference MPC through policy constraint. L4DC.

[5]Nair, A., Gupta, A., Dalal, M., & Levine, S. (2020). AWAC: Accelerating online reinforcement learning with offline datasets. arXiv:2006.09359.

[6]Park, K., Park, S., Lee, Y., & Levine, S. (2026). Scalable offline model-based RL with action chunks. ICLR.

[7]Peng, X. B., Kumar, A., Zhang, G., & Levine, S. (2019). Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv:1910.00177.

[8]Peters, J., & Schaal, S. (2007). Reinforcement learning by reward-weighted regression for operational space control. ICML.

[9]Zhan, G., Wang, L., Zhang, X., Gao, J., Tomizuka, M., & Li, S. E. (2025). Bootstrap off-policy with world model. NeurIPS.

[10]Zhang, S., Zhang, W., & Gu, Q. (2025). Energy-weighted flow matching for offline reinforcement learning. ICLR.

Questions for authors

1 . What is the technical novelty relative to the combination of TD-M(PC)², BOOM, and weighted flow matching?

  1. Can you add a one-step flow policy and a multi-step Gaussian policy under otherwise identical settings, and an experiment that directly tests robustness to model error?

3 . How do the results change for H ∈ {1, 3, 5, 8}?

4 . Does the buffer sequence at:t+Ha_{t:t+H} consist of executed actions or of the planner’s stored H-step plan? If the former, how does the theory apply?

5 . Can you report median, IQM, and results excluding run, with more seeds and t-based (or bootstrap) intervals?

6 . Are the baseline results matched to your environment versions and seed counts? What is the wall-clock overhead per environment step?

Comments on the paper

1 comment

Dear Reviewers,

We are now halfway through the discussion phase. The authors have provided their responses to your reviews. If their responses have not adequately addressed your concerns, please take this opportunity to raise any remaining questions or concerns.

I encourage you to engage with the authors as soon as possible to ensure a productive discussion.

Many thanks,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.