- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Refinement-Based Flow Policy Optimization (RFPO) that trains flow-based policies by alternating between Q-guided action refinement and self-target flow matching. Specifically, RFPO uses the Metropolis-adjusted Langevin algorithm (MALA) to refine actions sampled from the current policy toward a Q-induced Boltzmann distribution. The authors theoretically show that, under idealized assumptions and a fixed critic, each refinement–learning cycle corresponds to applying MALA transitions to the policy distribution, guaranteeing monotonic improvement of the entropy-regularized objective. Experiments on synthetic multimodal distributions and six MuJoCo tasks demonstrate effective distribution modeling and competitive performance against existing online RL baselines, although the contributions of individual design choices remain insufficiently validated through ablations.
Critical strengths
- The authors establish a connection between iterative policy refinement and MALA transitions, proving monotonic improvement of the entropy-regularized objective under idealized assumptions. The method captures complex multimodal distributions without mode collapse and achieves competitive performance against several online RL baselines across six MuJoCo environments.
Critical weaknesses
1.The refinement-and-distillation framework resembles existing amortized MCMC approaches and prior methods such as DIPO and DACER-F. 2. Insufficient ablation studies. The experiments do not isolate the contributions of MALA versus ULA, Metropolis-Hastings correction, current-policy versus replay-buffer initialization, or the number of refinement steps. Consequently, it is unclear which components drive the observed improvements.
Questions for authors
-
Could the authors clarify the main technical challenges and novel theoretical insights in establishing this result beyond existing properties of MCMC and flow matching?
-
Missing ablations: Could the authors provide controlled ablations comparing MALA with ULA, with and without the Metropolis-Hastings correction, and current-policy versus replay-buffer initialization? These comparisons would help isolate the contributions of the key design choices and determine whether MALA provides practical benefits beyond its theoretical guarantees.
5 comments
Regarding Weakness 2 and Question 2
We conducted the requested ablations within RFPO using a common flow-matching objective. To isolate the effect of the MH correction, we compare MALA with ULA using the same Langevin proposal rule and the step-size sequence recorded for each RFPO seed. Accepting every proposal lowers the mean final return on all six tasks at these matched step sizes. With independent source-noise pairing in both initialization variants, current-policy initialization yields higher mean returns on four tasks, including 4,754 versus 326 on Ant. We also compare . On Ant, increasing from 1 to 5 raises the mean final return from 2,396 to 5,854 and lowers its across-seed standard deviation from 2,118 to 501.
We chose MALA because its MH correction leaves the potentially multimodal -induced target distribution invariant, including the relative probability mass across modes. The choice of controls how many stochastic refinement transitions are used to construct each flow-matching target. Current-policy initialization closes RFPO's refinement–learning cycle by letting the policy fitted in one cycle generate the starting actions for the next. Appendix G of the revised manuscript provides the experimental setups, full results, and detailed analyses.
| Refinement rule | HalfCheetah | Walker2d | Hopper | Ant | Swimmer | HumanoidStandup |
|---|---|---|---|---|---|---|
| MALA | 13,269 ± 516 | 5,456 ± 262 | 2,110 ± 122 | 5,854 ± 501 | 343 ± 5 | 158.4 ± 5.1 |
| ULA, adaptive step | 8,933 ± 1,095 | 4,503 ± 248 | 1,741 ± 408 | 30 ± 26 | 329 ± 4 | 152.3 ± 3.8 |
| ULA, RFPO steps | 11,661 ± 1,010 | 4,793 ± 416 | 1,644 ± 232 | -12 ± 17 | 340 ± 7 | 154.3 ± 3.5 |
| Gradient ascent | 12,218 ± 2,543 | 5,126 ± 566 | 2,048 ± 260 | 5,448 ± 2,005 | 341 ± 2 | 131.4 ± 10.8 |
| Initialization | HalfCheetah | Walker2d | Hopper | Ant | Swimmer | HumanoidStandup |
|---|---|---|---|---|---|---|
| Current policy | 12,013 ± 784 | 5,376 ± 204 | 2,165 ± 280 | 4,754 ± 1,338 | 343 ± 9 | 146.8 ± 6.8 |
| Replay buffer | 9,481 ± 834 | 4,745 ± 436 | 1,927 ± 165 | 326 ± 72 | 344 ± 3 | 148.3 ± 4.9 |
| Source noise | HalfCheetah | Walker2d | Hopper | Ant | Swimmer | HumanoidStandup |
|---|---|---|---|---|---|---|
| Original source | 13,269 ± 516 | 5,456 ± 262 | 2,110 ± 122 | 5,854 ± 501 | 343 ± 5 | 158.4 ± 5.1 |
| Independent | 12,013 ± 784 | 5,376 ± 204 | 2,165 ± 280 | 4,754 ± 1,338 | 343 ± 9 | 146.8 ± 6.8 |
| HalfCheetah | Walker2d | Hopper | Ant | Swimmer | HumanoidStandup | |
|---|---|---|---|---|---|---|
| 1 | 11,334 ± 1,143 | 5,055 ± 507 | 2,167 ± 257 | 2,396 ± 2,118 | 317 ± 57 | 154.3 ± 2.4 |
| 3 | 11,101 ± 1,474 | 5,191 ± 435 | 2,091 ± 403 | 4,961 ± 833 | 349 ± 4 | 152.9 ± 10.6 |
| 5 (default) | 13,269 ± 516 | 5,456 ± 262 | 2,110 ± 122 | 5,854 ± 501 | 343 ± 5 | 158.4 ± 5.1 |
| 10 | 13,267 ± 1,202 | 5,370 ± 231 | 1,781 ± 119 | 5,834 ± 324 | 337 ± 5 | 149.1 ± 5.0 |
Thank you again for your time and constructive feedback. We understand that this is a busy period and would be grateful if, when your schedule permits, you could let us know whether our additional experiments and manuscript revisions address your concerns. If any concerns remain, we would appreciate the opportunity to provide further clarification.
Regarding Weakness 1 and Question 1
[Weakness 1]
Section 2.2 discusses the refinement–learning structure shared with amortized MCMC and MCMC teaching. RFPO adapts this structure to online actor–critic learning to approximate a potentially multimodal Q-induced action distribution (Sections 4.1 and 4.2). RFPO refines actions from the current flow policy with MALA and fits the same policy through self-target flow matching, pairing each refined action with the source noise used to generate its starting action.
The key distinction from the cited online RL baselines is how refinement enters the actor update. In the released implementation of DIPO [1] and in DACER-F [2], buffered actions are refined to construct training targets. In RFPO, the policy fitted in one cycle generates the starting actions for the next, making refinement part of a recursive policy-improvement update. Under the assumptions of Section 5, the policy's KL divergence to the fixed Q-induced target monotonically decreases across these cycles (Theorem 5.3).
[1] Long Yang et al. Policy Representation via Diffusion Probability Model for Reinforcement Learning. arXiv:2305.13122, 2023.
[2] Tianze Zhu et al. Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving. arXiv:2603.02613, 2026.
[Question 1]
RFPO refines actions sampled from the current flow policy with MALA and uses the refined actions as targets to train that same policy through self-target flow matching. The fitted policy then generates the starting actions for the next round of refinement, forming a refinement–learning cycle.
The main technical challenge in Section 5 is to show how repeating this cycle moves the policy's action distribution toward the Q-induced target. Using established flow-matching results, we show that under exact optimization and exact ODE integration, the fitted policy reproduces the distribution of the refined actions. One cycle therefore has the same effect on the policy distribution as K MALA steps (Lemma 5.2). With the critic and refinement kernel held fixed, n cycles have the same effect as nK MALA steps starting from the initial policy distribution. The invariance of the target under MALA and the KL data-processing inequality then imply that the policy's KL divergence to the target monotonically decreases across cycles, establishing a policy-improvement property of RFPO's iterative update (Theorem 5.3).