- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper works on how to do RL on a deterministic MeanFlow. The resultant MRFT achieves better or on-par results on multiple simulations and real robot tasks.
I hope to kindly report to the AC that my expertise in RL theory is limited, which I hope the AC can take into considerations.
Critical strengths
- A full mathematical proof is offered to support the proposed method.
- Both real robot and simulation results are offered.
Critical weaknesses
- The proposed method seems to outperform the baseline very little in Figure 1. It would be great if more analysis can be provided to justify this, and to offer insights on where MRFT is suitable or not.
Questions for authors
Please see the weaknesses.
4 comments
Response to Reviewer 3Jor (Part 2)
[W1.2] It would be great if offering insights on where MRFT is suitable or not.
Reply: We thank the reviewer for encouraging some insights on where MRFT is suitable or not. MRFT is well suited to online fine-tuning of pretrained MeanFlow policies for continuous control, enabling stochastic exploration while retaining few-step action generation. Its broader applicability comes from MeanFlow-SDE, which constructs stochastic finite-interval transitions directly from the existing MeanFlow field, requiring one model evaluation per interval without an auxiliary learnable noise network or anchor-based resampling. Within the MeanFlow formulation studied here, this construction is independent of the downstream task and policy backbone. Its tractable sampling-path likelihoods also support both on-policy and off-policy optimization (Section 4).
Our experiments support this flexibility across locomotion, state-based and visual manipulation, a -style VLA, and online fine-tuning on a physical robot (Sections 5.1–5.4). In practice, MRFT combines controllable stochastic exploration with efficient few-step MeanFlow sampling. The noise level provides a simple way to adjust exploration strength, while our ablations show that two model evaluations per sample achieve a favorable balance between task performance and training cost (Section 5.3). These properties make MRFT particularly attractive for online reinforcement fine-tuning, where action generation is repeated throughout training.
Beyond robotics, related SDE-based sampling constructions have enabled RL fine-tuning in image generation (Flow-GRPO$^{[1]}^{[2]}$). MeanFlow-SDE offers a corresponding route to reward-based fine-tuning of conditional MeanFlow generators in these domains, although these extensions remain to be evaluated.
[1] Flow-GRPO: Training Flow Matching Models via Online RL, NIPS 2025.
[2] DanceGRPO: Unleashing GRPO on Visual Generation, Arxiv 2025.
I sincerely thank the authors for the explanations and justifications. I suggest adding them to the main text for broader readers to better understand the method's significance. I remain weak accept.
Dear Reviewer 3Jor,
Thank you very much for taking the time to consider our explanations and for your helpful suggestion!
We have moved the final performance table into the main text and incorporated the key insights from into the conclusion. We agree that these changes help readers better understand the performance and significance of our method.
If our responses have helped resolve your concerns, might you also feel comfortable revisiting your confidence level? We would, of course, fully respect your judgment.
Best regards, The Authors
Response to Reviewer 3Jor (Part 1)
We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one by one in what follows.
[W1]: The proposed method seems to outperform the baseline very little in Figure 1. It would be great if more analysis can be provided to justify this, and to offer insights on where MRFT is suitable or not.
[W1.1]: The proposed method seems to outperform the baseline very little in Figure 1. It would be great if more analysis can be provided to justify this.
Reply: Thank you for raising this concern. To clarify the performance differences in Figure 1, we first report the detailed results from Appendix F below, making both the magnitude of the improvements and the variability across runs explicit. Results are reported as mean ± standard deviation over three random seeds.
Gym Locomotion Tasks (Average Episode Reward)
Franka Kitchen Tasks (Task Completion Rate, %)
Robomimic Tasks (Success Rate, %)
Bold and underlined values indicate the best and second-best results in each task, respectively.
The observed gains on Kitchen-Partial and Square are substantial rather than merely marginal: MRFT exceeds the strongest competing baseline by 6.9 and 3.8 percentage points, respectively. Overall, MRFT achieves the highest mean performance on eight of nine tasks. Starting from the same pretrained MeanFlow policy, it improves mean performance over DMPO on all nine tasks, including gains of 21.8 percentage points on Kitchen-Partial and 99–132 reward points across the three Gym tasks. We provide a more detailed analysis in Appendix F for your reference.
For context, DMPO reports a 3-percentage-point gain in peak success rate over ReinFlow-R on Square (Section 4.2.1), while D²PPO$^{[1]}$ reports a 6-percentage-point gain over DPPO$^{[2]}$ on Square after fine-tuning (Table 2). These reported gains provide context for the magnitude of our improvements.
The gains also extend to training efficiency. Under the same interaction budget and hardware, MRFT achieves the highest mean success rate across all three Robomimic tasks, with iteration times 9.1–18.3% shorter than those of DMPO and 3.3–21.0% shorter than those of ReinFlow-S, while remaining comparable to those of ReinFlow-R (Table 1 of the manuscript).
Additionally, to make the performance differences easier to discern, we have revised Figure 1 by adding zoomed-in insets to all three Gym tasks, highlighting the key regions where the differences between methods are more visible.
[1] Diffusion Policy Policy Optimization with Dispersive Loss, AAAI 2026.
[2] Diffusion policy policy optimization, ICLR 2025.