acceptodds
Under review as a conference paper at ICLR 2027

Model-based Flow Policy Optimization

Abstract

Generative control policies (GCPs), such as diffusion and flow matching policies can represent expressive, multi-modal action distributions, but training them with reinforcement learning is difficult: their likelihoods are expensive to compute, and backpropagating through the sampling chain is costly and unstable. Existing methods sidestep this with model-free, on-policy RL, which is prohibitively sample-inefficient, especially from pixels. We observe that a learned world model resolves this tension. Likelihood-free objectives such as advantage-weighted flow matching require abundant on-policy data, which a latent world model can generate cheaply in imagination. We propose MB-FPO, which trains flow policies by regressing onto imagined actions weighted by their advantages, with no likelihoods and no backpropagation through sampling. With temporally correlated exploration noise, MB-FPO is substantially more sample efficient than prior flow policy optimization methods on the DeepMind Control and Meta-World benchmarks, outperforms model-free baselines on most pixel-based tasks, and matches or exceeds a model-based Gaussian policy trained on the same world.

Primary area:
reinforcement learning
Keywords:
Flow Policy, Model-based Reinforcement Learning, World Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_w5aKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper MB-FPO, a model-based RL algorithm for flow matching (FM) policy, to improve sampling-efficiency. The paper employs a Dreamer style RSSM world model to generate rollouts, and update actors with advantage weighting regression to sidestep log likelihook calculation. The experiments are conducted on two widely adopted benchmarks MetaWorld and DMC where MB-FPO outperforms conventional FM RL algorithms.

Critical strengths

Devleoping Model-based RL for flow matching policy is a timely and important research topic. I also like the insights that on-policy nature of MBRL turns importance sampling into AWR. One of the strengths is the simplicity of the method which sidesteps log likelihood without further complications (a major challenge in RL for FM), and removes clipping thresholds, making it easier to deploy the algorithm.

Critical weaknesses

The current Gaussian vs FM experiment has several confounders. Ideally, only the actor itself should be ablated. But Gaussian seems to use advantage weighted REINFORCE while FM uses exponentially weighted regression. So the comparison does not isolate the effect of FM expressiveness or regression objective. A Gaussian baseline with the same objective could improve the ablation. Additionally, although the author suggests FM is more expressive, the gains of FM over Gaussian seems marginal, and the expressiveness assumption conflicts with the fact that some of FM baselines are outperformed by Gaussian. I would suggest making the assumptions and claims more accurate.

There are several diffusion or flow matching MBRL works such as MBDPO, DiWA, FlowMPC, RFO (https://arxiv.org/pdf/2602.03501). Some works were also cited by the paper. However, they were not used as baselines. The paper could benefit from more comparisons with FM/DP based MBRL baselines.

Questions for authors

See weaknesses above.

3 comments

We thank the reviewers for their time and positive feedback, we believe the feedback helped us improve our experimental section

The current Gaussian vs FM experiment has several confounders. Ideally, only the actor itself should be ablated. But Gaussian seems to use advantage weighted REINFORCE while FM uses exponentially weighted regression. So the comparison does not isolate the effect of FM expressiveness or regression objective. A Gaussian baseline with the same objective could improve the ablation. Additionally, although the author suggests FM is more expressive, the gains of FM over Gaussian seems marginal, and the expressiveness assumption conflicts with the fact that some of FM baselines are outperformed by Gaussian. I would suggest making the assumptions and claims more accurate.

We applied an experiment addressing the confounders in the comparison to the Gaussian policy (Sec. 4.5, Fig. 8). we tightened some of the claims and sentences. The expressiveness would mostly help in the exploration this is shown in the sample efficiency in the sparse reward tasks, we analyze that for the pendulum tasks (App. E, Fig. 11) and see that the multi modality leads to a more multimodal behavior leading to better exploration leading to better performance especially in sparse reward tasks. The results of combining pink noise and using the same advantage weighting with the Gaussian policy show that those changes don't close the gap to MB-FPO. This shows that the multimodality of the flow policy seems to have a strong exploration behavior, combined with the iterative nature of flow leads to better performance compared to Gaussian policies as seen in the aggregate results in figure 1.

Gaussian actors on the same world model (Fig. 8). Mean ± std over 3 seeds, at 100k / 500k environment steps; return for DMC, success rate for Meta-World.

Methodfinger_spinpendulum_swingupreacher_hardhammerhand-insert
MB-FPO (flow, pink)576 ± 109 / 827 ± 8815 ± 2 / 833 ± 1358 ± 53 / 923 ± 450.57 ± 0.35 / 1.00 ± 0.000.65 ± 0.39 / 1.00 ± 0.00
R2-Dreamer (Gaussian, REINFORCE, white)561 ± 31 / 619 ± 33280 ± 379 / 816 ± 36223 ± 7 / 952 ± 100.32 ± 0.38 / 1.00 ± 0.000.78 ± 0.08 / 0.98 ± 0.02
Gaussian, REINFORCE, pink582 ± 5 / 644 ± 15702 ± 122 / 832 ± 5127 ± 98 / 962 ± 80.00 ± 0.00 / 1.00 ± 0.000.68 ± 0.13 / 0.98 ± 0.02
Gaussian, MB-FPO objective, pink551 ± 21 / 648 ± 8349 ± 333 / 814 ± 17155 ± 53 / 771 ± 1080.15 ± 0.21 / 0.67 ± 0.470.58 ± 0.02 / 1.00 ± 0.00

Behaviour policy on pendulum_swingup, 20 episodes started hanging at rest (App. E, Fig. 11): side at which each episode first reaches upright (+π / −π).

Policy75k steps100k steps
R2-Dreamer (Gaussian)0 / 200 / 20
MB-FPO (flow)11 / 910 / 10

There are several diffusion or flow matching MBRL works such as MBDPO, DiWA, FlowMPC, RFO (https://arxiv.org/pdf/2602.03501). Some works were also cited by the paper. However, they were not used as baselines. The paper could benefit from more comparisons with FM/DP based MBRL baselines.

We added a comparison to DiWA [1] in App. D, Fig. 10. DiWA uses DPPO [2] which increases the horizon of the problem and is designed to finetune pretrained policies rather than training policies from scratch. We don't add it to the main results due to the different settings. MBDPO [3] and RFO [4] are preprints so we don't compare them. FlowMPC [5] refines an imitation policy at inference which we find to be an orthogonal setting to ours.

DiWA run online from scratch, actor gradient steps per environment step matched (App. D, Fig. 10). Mean ± std over 3 seeds, at 100k / 500k environment steps; return for finger_spin, success rate for Meta-World.

Methodfinger_spinhammerhand-insert
MB-FPO576 ± 109 / 827 ± 80.57 ± 0.35 / 1.00 ± 0.000.65 ± 0.39 / 1.00 ± 0.00
DiWA486 ± 74 / 599 ± 1340.00 ± 0.00 / 0.00 ± 0.000.00 ± 0.00 / 0.02 ± 0.02

[1] Chandra, Akshay L., et al. "DiWA: Diffusion policy adaptation with world models." arXiv preprint arXiv:2508.03645. 2025.

[2] Ren, Allen, et al. "Diffusion policy policy optimization." International Conference on Learning Representations. 2025.

[3] Cheng, Xiaoyuan, et al. "Scaling world-model reinforcement learning through diffusion policy optimization." arXiv preprint arXiv:2605.26282. 2026.

[4] Zhong, Hai, et al. "Reparameterization flow policy optimization." arXiv preprint arXiv:2602.03501. 2026.

[5] Hamel, Chandon. "FlowMPC: Improving flow matching policies with world models." arXiv preprint arXiv:2606.16286. 2026.

Reviewer_w5aKOpenReview ↗

comments by the reviwer

We thank the reviewer for the new clarifications and experiments. All my concerns have been addressed. I keep my rating for acceptance.

Thank you! We updated all our comments to include the figures from the updated manuscript to make it easier for reviewers/AC to have an overview. In case you have any further feedback or questions, please let us know!

Reviewer_wmuYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents Model-Based Flow Policy Optimization (MB-FPO), a method for training flow-matching policies using imagined trajectories from a learned world model. The method builds on R2-Dreamer's latent dynamics model and replaces its Gaussian actor with a flow policy trained using advantage-weighted conditional flow matching. This avoids computing action likelihoods or backpropagating through the flow sampling process. The method also uses temporally correlated exploration noise. Experiments on DeepMind Control and Meta-World show improved sample efficiency over several model-free baselines and competitive performance against a model-based Gaussian policy.

Critical strengths
  1. The method is relatively simple and builds on established components. The use of imagined trajectories to train flow policies is a potentially useful direction.
  2. The experimental results are promising, particularly in comparison with model-free flow policy optimization methods.
  3. The analysis of exploration noise and multimodal action distributions provides useful insight into the behavior of the learned policies.
Critical weaknesses
  1. The main idea of improving sample efficiency through imagined trajectories is well established in methods such as Dreamer, TD-MPC2, and PDDM. MB-FPO uses an existing world model and trains the policy through advantage-weighted regression, an objective closely related to prior work. The paper also acknowledges that FMER derives the same flow-matching objective. While combining these components is useful, it is not clear which part represents the main methodological advance beyond existing model-based policy optimization. A more explicit comparison of the optimization objectives and design choices would help establish the contribution.
  2. The comparison with model-based planning is limited. The paper compares MB-FPO against MPPI using the same world model, which is a useful experiment. However, the comparison uses horizon-one planning with a critic trained for a different policy. It therefore does not fully evaluate the advantages of amortized flow policy optimization against established multi-step model-based planning methods such as TD-MPC2.
  3. MB-FPO matches the Gaussian R2-Dreamer baseline on several tasks and improves performance on others. However, flow policies require multiple network evaluations to generate each action. The paper does not provide a detailed comparison of inference cost, training time, or performance relative to this additional computation. The multimodality analysis is interesting, but it is not clear how much multimodal action representation contributes to the reported control improvements.
Questions for authors
  1. What is the key methodological distinction between MB-FPO and existing approaches such as Dreamer, TD-MPC2 and FMER? Is the main contribution the use of imagined trajectories for flow matching, the specific advantage-weighting objective, or the integration of these components?

  2. How might the findings of the experiments change with a longer MPPI planning horizon or a stronger planning baseline? In particular, what advantages does the learned flow policy provide over repeatedly optimizing actions in the same world model?

  3. What is the computational overhead of using a flow actor instead of a Gaussian actor? How do training cost and action inference latency compare, and on which tasks does the added expressiveness provide a clear benefit?

  4. How important is the multimodal nature of the learned policy for control performance? Can the authors identify tasks or states where maintaining multiple action modes leads to better returns, rather than only showing that the policy can represent these modes?

4 comments

We thank the reviewers for their time and feedback, below we answer their questions:

Q1: The key contribution is a model-based, on-policy optimization method for flow policies that avoids two problems in existing methods. The first is the PPO-style ratio clipping used by FPO [9], which is shown to be very sensitive to hyperparameter tuning [10]. The second is the horizon lengthening used by DPPO [5] and, therefore, by DiWA [4], which treats each denoising step as an MDP step and makes the RL problem harder to solve. We evaluate DiWA in our setting and find that applying it to learning from scratch leads to worse results than our approach (App. D, Fig. 10) and shown in the table below. The result is a maodel-based flow policy optimization method that matches or improves on the Gaussian actor [7] and especially learns faster on sparse-reward tasks.

DiWA run online from scratch, actor gradient steps per environment step matched (App. D, Fig. 10). Mean ± std over 3 seeds, at 100k / 500k environment steps; return for finger_spin, success rate for Meta-World.

Methodfinger_spinhammerhand-insert
MB-FPO576 ± 109 / 827 ± 80.57 ± 0.35 / 1.00 ± 0.000.65 ± 0.39 / 1.00 ± 0.00
DiWA486 ± 74 / 599 ± 1340.00 ± 0.00 / 0.00 ± 0.000.00 ± 0.00 / 0.02 ± 0.02

MB-FPO achieves this through four design choices. Fresh imagined rollouts for every update remove the need for clipping. Clipping exists in PPO-style methods because they reuse each batch of real data for several gradient steps, so the policy drifts away from the one that collected the data and the update must be constrained. Because a learned world model makes on-policy data cheap, MB-FPO generates a new batch of imagined rollouts for every gradient step. The data is therefore always on-policy, and no trust region or clipping is needed. An advantage-weighted CFM objective keeps the original MDP horizon. DPPO and DiWA need a likelihood for each denoising step, so they fold the denoising chain into the MDP. MB-FPO instead weights the standard conditional flow matching loss by the advantage of the sampled action. The denoising chain stays inside the policy, the MDP horizon is unchanged, and no likelihoods or backpropagation through sampling are required. Dropping non-positive advantages (the indicator in Eq. 4) keeps the objective bounded: applied naively, negative advantages push the flow away from sampled actions without bound, and training diverges (App. B, Fig. 9). Time-correlated (pink) noise [6] provides exploration in place of an entropy term, which would be hard to compute for flow policies.

  • Dreamer [1] uses the same imagination loop but a unimodal Gaussian actor. MB-FPO replaces it with a multimodal flow policy, which requires the update rule above.

  • TD-MPC2 [2] improves actions by planning online at every step. MB-FPO also refines actions iteratively, through denoising, but amortizes that cost into the policy, so it needs no test-time planning, and matches or beats TD-MPC2's planner on the same world model on all tasks except walker_walk (Sec. 4.4, Fig. 6).

  • FMER [3] also uses advantage-weighted flow matching, but it is off-policy and model-free and relies on entropy regularization. MB-FPO is on-policy in imagination and explores through correlated noise.

Q2: Increasing the MPPI horizon to 3, as in TD-MPC2, improves performance in some environments (Sec. 4.4, Fig. 6), but the flow policy beats the planners on all tasks except walker_walk. The cost in test-time planning is higher than that of the flow policy (Table 1). This shows that flow policies combine the benefits of the iterative action prediction of planning with the test-time benefits of amortized policy optimization. We think that analyzing the differences and similarities between flow policies and test-time planning is an interesting direction for future work. We follow TD-MPC2 in optimizing the critic, where the critic evaluates the prior policy, not the planner. The mismatch between the critic and the planner is an open question for planning-based approaches and is outside the scope of our paper. This problem is actually one of the advantages of flow-based policies, in that we can use the flow policy to optimize the critic in a wall-time-efficient manner.

Flow policy vs planning on the same world model (Sec. 4.4, Fig. 6). Return, mean ± std over 3 seeds, at 100k / 300k environment steps.

Methodwalker_walkcheetah_runcartpole_swingupreacher_hard
MB-FPO314 ± 81 / 694 ± 87399 ± 13 / 598 ± 6640 ± 30 / 857 ± 058 ± 53 / 665 ± 47
MPPI, H=1362 ± 33 / 740 ± 102268 ± 19 / 490 ± 110347 ± 60 / 819 ± 1081 ± 53 / 471 ± 209
TD-MPC2 planner, H=3, no prior473 ± 34 / 701 ± 28237 ± 31 / 510 ± 49543 ± 82 / 827 ± 1686 ± 87 / 514 ± 20
TD-MPC2 planner, H=3, policy prior403 ± 47 / 816 ± 13295 ± 15 / 417 ± 116541 ± 35 / 806 ± 42101 ± 87 / 663 ± 90

Q3: We show the computational overhead in Table 1 (MBFPO 107 vs R2Dreamer [7] 167 fps during training, and 2.37 vs 1.21 ms per step for the walker_walk environment). The expressiveness benefits mostly appear in sparse-reward and harder exploration problems; we show this in the pendulum task (App. E, Fig. 11), where the flow policy shows better exploration behavior. Indeed, there is no free lunch; the compute cost of multiple inference steps is the cost paid by using generative control policies. This was shown to be beneficial in robot manipulation [8]; we add a runtime comparison (Table 1) to make it clear to the reader. Generally, we find that the multi-iterativeness of generative policies is a cost that must be paid to refine the policy's predictions and should be assessed according to the domain. We note, however, that this is still much more compute- and wall-time efficient than test-time planning, as shown in Table 1.

Compute on one RTX 5090 (Table 1). Latency is per control step on walker_walk (pixels).

MethodTraining (frames/s)Latency, 1 env (ms)Latency, 16 envs (ms)
R2-Dreamer (Gaussian)1671.211.32
MB-FPO1072.372.55
MPPI, H=1–8.128.9
TD-MPC2 planner, H=3, policy prior–21.182.4

Q4: We investigate the effect of multimodality on the pendulum environment, where it is relatively easier to assess. We evaluate the behavior on different checkpoints throughout learning. In App. E, Fig. 11, it is clear that the flow policy can discover different ways to solve the task, leading to better exploration and coverage of the state-action space and better performance.

Behaviour policy on pendulum_swingup, 20 episodes started hanging at rest (App. E, Fig. 11): side at which each episode first reaches upright (+π / −π).

Policy75k steps100k steps
R2-Dreamer (Gaussian)0 / 200 / 20
MB-FPO (flow)11 / 910 / 10

[1] Hafner, Danijar, et al. "Mastering diverse control tasks through world models." Nature. 2025.

[2] Hansen, Nick, Hao Su, and Xiaolong Wang. "TD-MPC2: Scalable, robust world models for continuous control." International Conference on Learning Representations. 2024.

[3] Gao, Ting, et al. "Flow matching policy optimization with mirror descent and entropy constraints." arXiv preprint arXiv:2603.17685. 2026.

[4] Chandra, Akshay L., et al. "DiWA: Diffusion policy adaptation with world models." arXiv preprint arXiv:2508.03645. 2025.

[5] Ren, Allen, et al. "Diffusion policy policy optimization." International Conference on Learning Representations. 2025.

[6] Eberhard, Onno, et al. "Pink noise is all you need: Colored noise exploration in deep reinforcement learning." International Conference on Learning Representations. 2023.

[7] Morihira, Naoki, et al. "R2-Dreamer: Redundancy-reduced world models without decoders or augmentation." arXiv preprint arXiv:2603.18202. 2026.

[8] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." The International Journal of Robotics Research. 2025.

[9] McAllister, David, et al. "Flow matching policy gradients." International Conference on Learning Representations. 2026.

[10] Yi, Brent, et al. "Flow policy gradients for robot control." arXiv preprint arXiv:2602.02481. 2026.

Dear reviewer, as the discussion period comes to an end, please let us know if there are any further questions or if anything in our rebuttal is unclear.

Thanks for the detailed rebuttal. Most of my questions have been addressed. However, the novelty concern regarding the similarity to existing model-based planning approaches (e.g., Dreamer, as acknowledged by the authors in their rebuttal) remains. The proposed approach can still be viewed as applying an advantage-weighted flow objective within a conventional Dreamer-style model-based RL framework, and the distinction from prior work could be further clarified.

The responses to my other questions are convincing. In particular, the response to Q4 clarifies how the flow-based policy parameterization preserves multimodality in the pendulum task and outperforms a Dreamer variant. I find this result compelling, as it provides empirical evidence of the benefits of the proposed approach and helps alleviate some of my concerns above.

Overall, given the satisfactory responses to my review, I lean toward acceptance and raise my score to Weak Accept.

Reviewer_AmWaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MB-FPO trains a flow-matching policy from scratch inside an R2-Dreamer world model that is learned online. For each actor update it generates fresh imagined rollouts and fits the policy to their actions with conditional flow matching, weighting each action by the exponential of its advantage and giving no weight to actions with non-positive advantage. This avoids computing action likelihoods and backpropagating through the sampler. In the real environment the policy explores with pink (temporally correlated) noise; in imagination it uses white noise. The experiments cover ten pixel-based DMC tasks and six state-based Meta-World tasks, with three seeds each. The baselines include model-free methods and R2-Dreamer's own Gaussian actor on the same world model. There is also an ablation that crosses the two actors with the two noise types, and a limited comparison with one-step MPPI planning.

Critical strengths

A simple method, compared against a baseline on the same world model. The paper combines existing components into a flow actor that learns online from scratch, without backpropagating through the sampling chain and without likelihood ratios. The most informative comparison is with R2-Dreamer's Gaussian actor, which uses the same world-model architecture and hyperparameters (though a different actor objective and exploration noise; see Weakness 1). The large gap to the model-free FPO++ is less informative, because the paper notes that FPO++ is not effective at this sample budget. Against the Gaussian actor, Figs. 3–4 show competitive overall performance, with gains on finger spin, pendulum swingup and hammer and a smaller gain on hand insert. The Gaussian does end higher on walker run and acrobot swingup, so the abstract's claim that MB-FPO "matches or exceeds" it is too strong. My positive score rests mainly on this comparison.

Critical weaknesses
  1. It is unclear whether the flow's expressiveness is responsible for the gains. Sec. 4.5 argues that the flow helps because it is more expressive than a Gaussian. But MB-FPO and the Gaussian baseline differ in three ways, not one: the policy class, the exploration noise, and the actor objective, since MB-FPO uses the positive-advantage regression of Eq. 4 while the baseline uses R2-Dreamer's REINFORCE update (Sec. 4, lines 262–265). The paper is also inconsistent on this point. The captions of Figs. 3–4 say the two actors "differ only in the actor objective", while Sec. 4.5 says they differ in the policy class and the noise.

    Fig. 7 controls for the noise but not for the objective, and on two of its three tasks it does not support the expressiveness argument. On pendulum swingup, the Gaussian actor with pink noise almost matches MB-FPO, while the flow actor with white noise lags behind both pink-noise runs. On reacher hard, the four curves overlap. That leaves finger spin, and even there DrQ-v2, whose actor is also unimodal (a deterministic policy with Gaussian exploration noise), ends above MB-FPO (about 860 vs 820 in Fig. 3). The Gaussian actor's plateau near 620 is therefore not clearly a limitation of Gaussian policies. The bimodal action distribution in Fig. 5 is shown on pendulum, where pink noise alone closes the gap, so the figure does not show that multimodality improves returns either. A Gaussian actor trained on the same advantage-weighted targets, with the same noise and similar capacity, would settle the question. If it did as well as MB-FPO, I would read the paper as a recipe for advantage-weighted regression in imagination rather than as evidence for flow policies.

  2. The compute cost is not reported. Each action takes ten Euler steps, and the loss uses sixteen CFM samples per action (Appendix C), yet the paper reports neither training time nor inference latency. The case for MB-FPO is sample efficiency against a Gaussian actor that needs a single forward pass per action, so readers need to know what the gain costs in compute.

Questions for authors
  1. Requested experiment: Do the gains over R2-Dreamer's Gaussian actor remain if the Gaussian is trained on the same positive-advantage regression targets, with the same exploration noise and similar capacity? Finger spin, hand insert and hammer would be the most informative tasks, since the pendulum gain is already explained by the noise.
  2. Requested experiment: What are the total training time and the per-action inference latency of MB-FPO compared with the Gaussian baseline, at the reported model size and update ratio?
3 comments

We thank the reviewer for their time and helpful feedback, we think the experiments and added evaluations strengthened the manuscript. We reply to the questions point by point below.

Q1: Yes. We train a Gaussian actor using MB-FPO's objective noise and capacity (Sec. 4.5, Fig. 8, App. C). We run it on the three suggested tasks, plus pendulum_swingup and reacher_hard. It does not close the gap, and it does worse than R2-Dreamer's [1] REINFORCE Gaussian. Trained only on positive advantages removes the signal from negative advantage actions and the Gaussian's standard deviation collapses onto its lower bound and it stops exploring. For a Gaussian, negative advantages are harmless because its likelihood is normalized: lowering the density of a bad action moves mass elsewhere. For a flow they are not. With signed weights the CFM loss has no minimum wherever negative-advantage actions dominate, so ||v|| grows without bound (App. B). We confirm this on walker_walk. With signed weights, the flow collapses while ||v|| explodes; with MB-FPO's weights it remains stable (App. B, Fig. 9).Further, giving the REINFORCE Gaussian pink noise also leaves it below MB-FPO on finger_spin and hammer (Fig. 7, Fig. 8).

Gaussian actors on the same world model (Fig. 8). Mean ± std over 3 seeds, at 100k / 500k environment steps; return for DMC, success rate for Meta-World.

Methodfinger_spinpendulum_swingupreacher_hardhammerhand-insert
MB-FPO (flow, pink)576 ± 109 / 827 ± 8815 ± 2 / 833 ± 1358 ± 53 / 923 ± 450.57 ± 0.35 / 1.00 ± 0.000.65 ± 0.39 / 1.00 ± 0.00
R2-Dreamer (Gaussian, REINFORCE, white)561 ± 31 / 619 ± 33280 ± 379 / 816 ± 36223 ± 7 / 952 ± 100.32 ± 0.38 / 1.00 ± 0.000.78 ± 0.08 / 0.98 ± 0.02
Gaussian, REINFORCE, pink582 ± 5 / 644 ± 15702 ± 122 / 832 ± 5127 ± 98 / 962 ± 80.00 ± 0.00 / 1.00 ± 0.000.68 ± 0.13 / 0.98 ± 0.02
Gaussian, MB-FPO objective, pink551 ± 21 / 648 ± 8349 ± 333 / 814 ± 17155 ± 53 / 771 ± 1080.15 ± 0.21 / 0.67 ± 0.470.58 ± 0.02 / 1.00 ± 0.00

Signed advantage weights on walker_walk (App. B, Fig. 9). Mean ± std over seeds (MB-FPO: 3 seeds for return, 2 reruns for ||v||; signed: 2 seeds).

WeightsReturn, 100kReturn, 200kMean ‖v‖, 200k
MB-FPO (positive advantages only)314 ± 81557 ± 473.5 ± 0.0
Signed, w = Â450 ± 3636 ± 12130 ± 46

Q2: We add these in Table 1. Training runs at 107 frames/s for MB-FPO and 167 for the Gaussian. Per-action inference latency is 2.37 vs 1.21 ms. The overhead comes from the 10 Euler steps per action and the 16 CFM samples per action in the loss. For comparison, planning on the same world model costs 8.1 ms per action for MPPI at horizon 1 and 21.1 ms for TD-MPC2's [2] planner at horizon 3 with a policy prior. We report these figures in the manuscript.

Compute on one RTX 5090 (Table 1). Latency is per control step on walker_walk (pixels).

MethodTraining (frames/s)Latency, 1 env (ms)Latency, 16 envs (ms)
R2-Dreamer (Gaussian)1671.211.32
MB-FPO1072.372.55
MPPI, H=1–8.128.9
TD-MPC2 planner, H=3, policy prior–21.182.4

It is unclear whether the flow's expressiveness is responsible for the gains

To further compare and assess the behavior of flow policies, we also evaluate them in the pendulum environment and find that, in the pendulum task (App. E, Fig. 11), the flow policy exhibits better exploration behavior.

Behaviour policy on pendulum_swingup, 20 episodes started hanging at rest (App. E, Fig. 11): side at which each episode first reaches upright (+π / −π).

Policy75k steps100k steps
R2-Dreamer (Gaussian)0 / 200 / 20
MB-FPO (flow)11 / 910 / 10

The most informative comparison is with R2-Dreamer's Gaussian actor, which uses the same world-model architecture and hyperparameters (though a different actor objective and exploration noise; see Weakness 1). The large gap to the model-free FPO++ is less informative, because the paper notes that FPO++ is not effective at this sample budget.

We find that our method also outperforms the off-policy flow methods such as DACER [3]. We also investigate DiWA [4] which uses model rollouts to finetune diffusion policy, and we find that their method has worse sample efficiency in comparison to ours due to it being designed to finetune trained policies instead of training policies from scratch (App. D, Fig. 10), we don't add it to the main results due to the different setting. FPO and FPO++ [5, 6] are the closest methods to our paper in the use of on-policy advantage weighted conditional flow loss. The 2 methods require the use of trust regions which brings its own complications due to being off policy after the first gradient step. The use of model imaginations which don't require sampling from the environment, removes the need for trust regions as we can sample fresh data for each gradient step. Finally looking at the results in figure 3 and 4 and the aggregate performance in figure 1 it seems that matches or exceeds is a reasonable assessment.

DiWA run online from scratch, actor gradient steps per environment step matched (App. D, Fig. 10). Mean ± std over 3 seeds, at 100k / 500k environment steps; return for finger_spin, success rate for Meta-World.

Methodfinger_spinhammerhand-insert
MB-FPO576 ± 109 / 827 ± 80.57 ± 0.35 / 1.00 ± 0.000.65 ± 0.39 / 1.00 ± 0.00
DiWA486 ± 74 / 599 ± 1340.00 ± 0.00 / 0.00 ± 0.000.00 ± 0.00 / 0.02 ± 0.02

[1] Morihira, Naoki, et al. "R2-Dreamer: Redundancy-reduced world models without decoders or augmentation." arXiv preprint arXiv:2603.18202. 2026.

[2] Hansen, Nick, Hao Su, and Xiaolong Wang. "TD-MPC2: Scalable, robust world models for continuous control." International Conference on Learning Representations. 2024.

[3] Wang, Yinuo, et al. "Diffusion actor-critic with entropy regulator." Advances in Neural Information Processing Systems. 2024.

[4] Chandra, Akshay L., et al. "DiWA: Diffusion policy adaptation with world models." arXiv preprint arXiv:2508.03645. 2025.

[5] McAllister, David, et al. "Flow matching policy gradients." International Conference on Learning Representations. 2026.

[6] Yi, Brent, et al. "Flow policy gradients for robot control." arXiv preprint arXiv:2602.02481. 2026.

Dear reviewer, as the discussion period comes to an end, please let us know if there are any further questions or if anything in our rebuttal is unclear.

Comments on the paper

2 comments

We thank the reviewers for their time and feedback. Below, we summarize our contribution and the results added for the rebuttal. Changes in the manuscript are marked in blue. We respond to the reviewers' questions in more detail in our individual replies.

To clarify our contribution: MBFPO trains flow policies on-policy inside a learned world model with advantage-weighted conditional flow matching. Every actor update uses fresh imagined rollouts. Unlike FPO/FPO++ [1, 2], the method needs no trust region or ratio clipping. It also needs no action likelihoods and no backpropagation through the sampling chain as in DPPO [3]. Keeping only positive advantages stabilizes this regression for flows, and exploration uses time-correlated noise in place of entropy regularization. On pixel-based DMC and state-based Meta-World, MBFPO matches or improves on R2-Dreamer's [4] Gaussian actor on the same world model, learns fastest on the sparse-reward tasks, and is substantially more sample-efficient than model-free flow and diffusion baselines.

We performed more experiments to answer the reviewer questions, we list them below:

  • We show that a Gaussian trained on the same positive-advantage targets with pink noise and the same network sizes does not close the gap to MBFPO (Sec. 4.5, Fig. 8, App. C).

  • We explain why signed advantage weights leave the CFM loss without a minimum. We confirm it empirically: with signed weights the flow's velocity norm explodes and performance collapses, while MB-FPO stays stable (App. B, Fig. 9).

  • We report training wall-time and per-action latency for MB-FPO, the Gaussian actor and planners on the same world model. MB-FPO costs about twice the Gaussian per action and is much cheaper than test-time planning.

  • We add TD-MPC2's [5] planner at horizon 3 as a stronger planning baseline (Sec. 4.4, Fig. 6), with and without a policy prior, on the same world model. MB-FPO is competitive with it at a fraction of the per-step compute.

  • We run DiWA [6] online from scratch, matching actor gradient steps per environment step. It trails MB-FPO on finger_spin and solves neither hammer nor hand-insert within the budget.

  • We explain the exploration behavior for both flow and gaussian policies to analyze whether the expressivity of the policy end up causing multi modal behavior(App. E, Fig. 11). On pendulum swing-up from rest, MB-FPO's behaviour policy reaches upright from both sides, while the Gaussian actor always commits to one side.

[1] McAllister, David, et al. "Flow matching policy gradients." International Conference on Learning Representations. 2026.

[2] Yi, Brent, et al. "Flow policy gradients for robot control." arXiv preprint arXiv:2602.02481. 2026.

[3] Ren, Allen, et al. "Diffusion policy policy optimization." International Conference on Learning Representations. 2025.

[4] Morihira, Naoki, et al. "R2-Dreamer: Redundancy-reduced world models without decoders or augmentation." arXiv preprint arXiv:2603.18202. 2026.

[5] Hansen, Nick, Hao Su, and Xiaolong Wang. "TD-MPC2: Scalable, robust world models for continuous control." International Conference on Learning Representations. 2024.

[6] Chandra, Akshay L., et al. "DiWA: Diffusion policy adaptation with world models." arXiv preprint arXiv:2508.03645. 2025.

As the review phase concludes, we have added the results of the additional experiments directly in the comments to make them easier for the reviewers. We look forward to the reviewers' feedback and are open to any suggestions to further strengthen our paper and are happy to answer any further questions or run additional experiments if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.