- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper MB-FPO, a model-based RL algorithm for flow matching (FM) policy, to improve sampling-efficiency. The paper employs a Dreamer style RSSM world model to generate rollouts, and update actors with advantage weighting regression to sidestep log likelihook calculation. The experiments are conducted on two widely adopted benchmarks MetaWorld and DMC where MB-FPO outperforms conventional FM RL algorithms.
Critical strengths
Devleoping Model-based RL for flow matching policy is a timely and important research topic. I also like the insights that on-policy nature of MBRL turns importance sampling into AWR. One of the strengths is the simplicity of the method which sidesteps log likelihood without further complications (a major challenge in RL for FM), and removes clipping thresholds, making it easier to deploy the algorithm.
Critical weaknesses
The current Gaussian vs FM experiment has several confounders. Ideally, only the actor itself should be ablated. But Gaussian seems to use advantage weighted REINFORCE while FM uses exponentially weighted regression. So the comparison does not isolate the effect of FM expressiveness or regression objective. A Gaussian baseline with the same objective could improve the ablation. Additionally, although the author suggests FM is more expressive, the gains of FM over Gaussian seems marginal, and the expressiveness assumption conflicts with the fact that some of FM baselines are outperformed by Gaussian. I would suggest making the assumptions and claims more accurate.
There are several diffusion or flow matching MBRL works such as MBDPO, DiWA, FlowMPC, RFO (https://arxiv.org/pdf/2602.03501). Some works were also cited by the paper. However, they were not used as baselines. The paper could benefit from more comparisons with FM/DP based MBRL baselines.
Questions for authors
See weaknesses above.
3 comments
comments by the reviwer
We thank the reviewer for the new clarifications and experiments. All my concerns have been addressed. I keep my rating for acceptance.
Thank you! We updated all our comments to include the figures from the updated manuscript to make it easier for reviewers/AC to have an overview. In case you have any further feedback or questions, please let us know!
We thank the reviewers for their time and positive feedback, we believe the feedback helped us improve our experimental section
We applied an experiment addressing the confounders in the comparison to the Gaussian policy (Sec. 4.5, Fig. 8). we tightened some of the claims and sentences. The expressiveness would mostly help in the exploration this is shown in the sample efficiency in the sparse reward tasks, we analyze that for the pendulum tasks (App. E, Fig. 11) and see that the multi modality leads to a more multimodal behavior leading to better exploration leading to better performance especially in sparse reward tasks. The results of combining pink noise and using the same advantage weighting with the Gaussian policy show that those changes don't close the gap to MB-FPO. This shows that the multimodality of the flow policy seems to have a strong exploration behavior, combined with the iterative nature of flow leads to better performance compared to Gaussian policies as seen in the aggregate results in figure 1.
Gaussian actors on the same world model (Fig. 8). Mean ± std over 3 seeds, at 100k / 500k environment steps; return for DMC, success rate for Meta-World.
Behaviour policy on pendulum_swingup, 20 episodes started hanging at rest (App. E, Fig. 11): side at which each episode first reaches upright (+π / −π).
We added a comparison to DiWA [1] in App. D, Fig. 10. DiWA uses DPPO [2] which increases the horizon of the problem and is designed to finetune pretrained policies rather than training policies from scratch. We don't add it to the main results due to the different settings. MBDPO [3] and RFO [4] are preprints so we don't compare them. FlowMPC [5] refines an imitation policy at inference which we find to be an orthogonal setting to ours.
DiWA run online from scratch, actor gradient steps per environment step matched (App. D, Fig. 10). Mean ± std over 3 seeds, at 100k / 500k environment steps; return for finger_spin, success rate for Meta-World.
[1] Chandra, Akshay L., et al. "DiWA: Diffusion policy adaptation with world models." arXiv preprint arXiv:2508.03645. 2025.
[2] Ren, Allen, et al. "Diffusion policy policy optimization." International Conference on Learning Representations. 2025.
[3] Cheng, Xiaoyuan, et al. "Scaling world-model reinforcement learning through diffusion policy optimization." arXiv preprint arXiv:2605.26282. 2026.
[4] Zhong, Hai, et al. "Reparameterization flow policy optimization." arXiv preprint arXiv:2602.03501. 2026.
[5] Hamel, Chandon. "FlowMPC: Improving flow matching policies with world models." arXiv preprint arXiv:2606.16286. 2026.