acceptodds
Under review as a conference paper at ICLR 2027

Refinement-Based Flow Policy Optimization

12 views

Abstract

Flow-based policies offer an expressive representation for online reinforcement learning, but conventional flow matching requires samples drawn from the distribution to be modeled. This poses a challenge when the desired action distribution is defined only implicitly by a Q-function, since directly sampling actions from the resulting distribution is generally intractable. We propose Refinement-Based Flow Policy Optimization (RFPO), a novel framework for training a flow policy in online reinforcement learning by alternating between Q-guided sample refinement and self-target flow matching. RFPO first generates actions from Gaussian noise using the current flow policy and then uses a finite-step stochastic refinement procedure to move them toward an energy-based distribution induced by the Q-function. Each refined action is then paired with its corresponding initial noise sample and used as a fixed target for flow-matching training. By repeatedly refining its own outputs and learning from the resulting targets, RFPO incorporates Q-guidance into the policy without requiring direct samples from the target distribution, while retaining the capacity to represent multiple action modes. We further provide a theoretical analysis of the distributional dynamics induced by RFPO. Experiments on six synthetic two-dimensional target distributions with diverse geometries demonstrate that alternating Q-guided sample refinement and self-target flow matching captures complex multimodal structure without mode collapse. Across six continuous-control environments in MuJoCo, RFPO shows promising performance against representative online RL baselines in almost every environment.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Flow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kpG3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Refinement-Based Flow Policy Optimization (RFPO) that trains flow-based policies by alternating between Q-guided action refinement and self-target flow matching. Specifically, RFPO uses the Metropolis-adjusted Langevin algorithm (MALA) to refine actions sampled from the current policy toward a Q-induced Boltzmann distribution. The authors theoretically show that, under idealized assumptions and a fixed critic, each refinement–learning cycle corresponds to applying MALA transitions to the policy distribution, guaranteeing monotonic improvement of the entropy-regularized objective. Experiments on synthetic multimodal distributions and six MuJoCo tasks demonstrate effective distribution modeling and competitive performance against existing online RL baselines, although the contributions of individual design choices remain insufficiently validated through ablations.

Critical strengths
  • The authors establish a connection between iterative policy refinement and MALA transitions, proving monotonic improvement of the entropy-regularized objective under idealized assumptions. The method captures complex multimodal distributions without mode collapse and achieves competitive performance against several online RL baselines across six MuJoCo environments.
Critical weaknesses

1.The refinement-and-distillation framework resembles existing amortized MCMC approaches and prior methods such as DIPO and DACER-F. 2. Insufficient ablation studies. The experiments do not isolate the contributions of MALA versus ULA, Metropolis-Hastings correction, current-policy versus replay-buffer initialization, or the number of refinement steps. Consequently, it is unclear which components drive the observed improvements.

Questions for authors
  1. Could the authors clarify the main technical challenges and novel theoretical insights in establishing this result beyond existing properties of MCMC and flow matching?

  2. Missing ablations: Could the authors provide controlled ablations comparing MALA with ULA, with and without the Metropolis-Hastings correction, and current-policy versus replay-buffer initialization? These comparisons would help isolate the contributions of the key design choices and determine whether MALA provides practical benefits beyond its theoretical guarantees.

5 comments
AuthorsOpenReview ↗

Regarding Weakness 1 and Question 1

[Weakness 1]

Section 2.2 discusses the refinement–learning structure shared with amortized MCMC and MCMC teaching. RFPO adapts this structure to online actor–critic learning to approximate a potentially multimodal QQ-induced action distribution (Sections 4.1 and 4.2). RFPO refines actions from the current flow policy with MALA and fits the same policy through self-target flow matching, pairing each refined action with the source noise used to generate its starting action.

The key distinction from the cited online RL baselines is how refinement enters the actor update. In the released implementation of DIPO [1] and in DACER-F [2], buffered actions are refined to construct training targets. In RFPO, the policy fitted in one cycle generates the starting actions for the next, making refinement part of a recursive policy-improvement update. Under the assumptions of Section 5, the policy's KL divergence to the fixed QQ-induced target monotonically decreases across these cycles (Theorem 5.3).

[1] Long Yang et al. Policy Representation via Diffusion Probability Model for Reinforcement Learning. arXiv:2305.13122, 2023.

[2] Tianze Zhu et al. Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving. arXiv:2603.02613, 2026.

[Question 1]

RFPO refines actions sampled from the current flow policy with MALA and uses the refined actions as targets to train that same policy through self-target flow matching. The fitted policy then generates the starting actions for the next round of refinement, forming a refinement–learning cycle.

The main technical challenge in Section 5 is to show how repeating this cycle moves the policy's action distribution toward the QQ-induced target. Using established flow-matching results, we show that under exact optimization and exact ODE integration, the fitted policy reproduces the distribution of the refined actions. One cycle therefore has the same effect on the policy distribution as KK MALA steps (Lemma 5.2). With the critic and refinement kernel held fixed, nn cycles have the same effect as nKnK MALA steps starting from the initial policy distribution. The invariance of the target under MALA and the KL data-processing inequality then imply that the policy's KL divergence to the target monotonically decreases across cycles, establishing a policy-improvement property of RFPO's iterative update (Theorem 5.3).

AuthorsOpenReview ↗

Regarding Weakness 2 and Question 2

We conducted the requested ablations within RFPO using a common flow-matching objective. To isolate the effect of the MH correction, we compare MALA with ULA using the same Langevin proposal rule and the step-size sequence recorded for each RFPO seed. Accepting every proposal lowers the mean final return on all six tasks at these matched step sizes. With independent source-noise pairing in both initialization variants, current-policy initialization yields higher mean returns on four tasks, including 4,754 versus 326 on Ant. We also compare K∈1,3,5,10K \in \\{1,3,5,10\\}. On Ant, increasing KK from 1 to 5 raises the mean final return from 2,396 to 5,854 and lowers its across-seed standard deviation from 2,118 to 501.

We chose MALA because its MH correction leaves the potentially multimodal QQ-induced target distribution invariant, including the relative probability mass across modes. The choice of KK controls how many stochastic refinement transitions are used to construct each flow-matching target. Current-policy initialization closes RFPO's refinement–learning cycle by letting the policy fitted in one cycle generate the starting actions for the next. Appendix G of the revised manuscript provides the experimental setups, full results, and detailed analyses.

Refinement ruleHalfCheetahWalker2dHopperAntSwimmerHumanoidStandup
MALA13,269 ± 5165,456 ± 2622,110 ± 1225,854 ± 501343 ± 5158.4 ± 5.1
ULA, adaptive step8,933 ± 1,0954,503 ± 2481,741 ± 40830 ± 26329 ± 4152.3 ± 3.8
ULA, RFPO steps11,661 ± 1,0104,793 ± 4161,644 ± 232-12 ± 17340 ± 7154.3 ± 3.5
Gradient ascent12,218 ± 2,5435,126 ± 5662,048 ± 2605,448 ± 2,005341 ± 2131.4 ± 10.8
InitializationHalfCheetahWalker2dHopperAntSwimmerHumanoidStandup
Current policy12,013 ± 7845,376 ± 2042,165 ± 2804,754 ± 1,338343 ± 9146.8 ± 6.8
Replay buffer9,481 ± 8344,745 ± 4361,927 ± 165326 ± 72344 ± 3148.3 ± 4.9
Source noiseHalfCheetahWalker2dHopperAntSwimmerHumanoidStandup
Original source13,269 ± 5165,456 ± 2622,110 ± 1225,854 ± 501343 ± 5158.4 ± 5.1
Independent12,013 ± 7845,376 ± 2042,165 ± 2804,754 ± 1,338343 ± 9146.8 ± 6.8
KKHalfCheetahWalker2dHopperAntSwimmerHumanoidStandup
111,334 ± 1,1435,055 ± 5072,167 ± 2572,396 ± 2,118317 ± 57154.3 ± 2.4
311,101 ± 1,4745,191 ± 4352,091 ± 4034,961 ± 833349 ± 4152.9 ± 10.6
5 (default)13,269 ± 5165,456 ± 2622,110 ± 1225,854 ± 501343 ± 5158.4 ± 5.1
1013,267 ± 1,2025,370 ± 2311,781 ± 1195,834 ± 324337 ± 5149.1 ± 5.0

Thank you again for your time and constructive feedback. We understand that this is a busy period and would be grateful if, when your schedule permits, you could let us know whether our additional experiments and manuscript revisions address your concerns. If any concerns remain, we would appreciate the opportunity to provide further clarification.

Reviewer_py8FOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Refinement-Based Flow Policy Optimization (RFPO), which alternates between refining current-policy action samples using MALA and fitting a flow policy to the refined samples. Each refined action is paired with the source noise that generated its initial action. The paper analyzes the resulting distributional dynamics under exact flow matching and exact ODE integration, and evaluates the method on synthetic distributions and six MuJoCo tasks.

The approach is coherent and the empirical results are encouraging. However, my primary concern is that the paper does not sufficiently establish a substantive methodological advance over existing amortized MCMC and action-refinement-based policy learning.

Critical strengths
  1. Clear formulation
  2. Useful explanatory analysis
  3. Good empirical results and transparent reporting
Critical weaknesses
  1. The key refinement–learning framework substantially overlaps with established methods.
  2. The theoretical contribution is largely a combination of standard properties.
Questions for authors
  1. What concrete benefit does original-source-noise pairing provide over shuffled or independent pairing, given that the idealized endpoint-distribution argument does not require this particular coupling?
  2. How much of the empirical improvement depends on the Metropolis–Hastings correction, rather than stochastic refinement itself?
2 comments
AuthorsOpenReview ↗

Regarding Weakness 1,2 and Question 1,2

[Weakness 1]

Section 2.2 discusses the refinement–learning structure shared with amortized MCMC and MCMC teaching. RFPO uses this structure to update an online flow policy toward a potentially multimodal QQ-induced action distribution. Refinement-based online RL baselines such as DIPO [1] and DACER-F [2] start refinement from buffered actions. By contrast, RFPO draws actions from the current flow policy, refines them with MALA, and trains the same policy on the refined actions via self-target flow matching. Section 5 shows that, under its assumptions, repeating this refinement–learning cycle monotonically decreases the policy's KL divergence to the QQ-induced target (Theorem 5.3). Furthermore, the same design lets RFPO use critic gradients during refinement while holding the resulting targets fixed during actor fitting. The actor update thus bypasses backpropagation through the iterative flow sampler, avoiding the attenuation or amplification of the QQ gradient that can arise from products of state Jacobians when directly maximizing QQ through the sampler (Appendix A).

In summary, the key distinction from these online RL baselines is how refinement enters the actor update. The baselines refine buffered actions to construct training targets. In RFPO, each fitted policy generates the starting actions for the next refinement cycle, making refinement part of a recursive policy-improvement update. Furthermore, RFPO's actor update bypasses backpropagation through the iterative flow sampler, avoiding solver-dependent attenuation or amplification of the QQ gradient.

[1] Long Yang et al. Policy Representation via Diffusion Probability Model for Reinforcement Learning. arXiv:2305.13122, 2023.

[2] Tianze Zhu et al. Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving. arXiv:2603.02613, 2026.

[Weakness 2]

Our main contribution is the refinement–learning cycle introduced in Sections 4.1 and 4.2. It combines QQ-guided refinement with self-target flow matching to train a flow policy to approximate a potentially multimodal QQ-induced action distribution. The role of Section 5 is to formalize the policy-improvement intuition behind this design using established results from flow matching and MCMC. Our theoretical analysis shows that, under the assumptions of Section 5, the policy's KL divergence to the fixed QQ-induced target distribution monotonically decreases across refinement–learning cycles, establishing a policy-improvement property of the update (Theorem 5.3).

[Question 1 and 2]

Under the idealized assumptions of Section 5, original-source and independent pairing recover the same refined action distribution for a given policy. In our ablation, both variants refine current-policy actions with MALA and use the same flow-matching loss. Original-source pairing yields mean final returns that are 10.5%, 23.1%, and 7.9% higher than those of independent pairing on HalfCheetah, Ant, and HumanoidStandup, respectively. The differences are smaller on the other tasks, with independent pairing yielding a slightly higher mean on Hopper. Independent pairing changes the source draw, which can substantially change the velocity target even for a small refinement displacement. We hypothesize that this makes the refined distribution harder to fit within the same optimization budget. Appendix G.3 provides the experimental setup and full results.

Source noiseHalfCheetahWalker2dHopperAntSwimmerHumanoidStandup
Original source (RFPO)13,269 ± 5165,456 ± 2622,110 ± 1225,854 ± 501343 ± 5158.4 ± 5.1
Independent12,013 ± 7845,376 ± 2042,165 ± 2804,754 ± 1,338343 ± 9146.8 ± 6.8

To isolate the effect of the Metropolis–Hastings correction, we use the same stochastic Langevin proposal rule in both variants and replay the step-size sequence from the MALA run with the same seed. The ULA variant accepts every proposal. At these matched step sizes, removing the correction lowers the mean final return on all six tasks. The decreases are about 12% on HalfCheetah and Walker2d and 22% on Hopper. On Ant, the mean return drops from 5,854 to -12, and the decreases on Swimmer and HumanoidStandup are below 3%. Appendix G.1 provides the full refinement-rule comparison, including adaptive ULA and deterministic gradient ascent.

Refinement ruleHalfCheetahWalker2dHopperAntSwimmerHumanoidStandup
MALA13,269 ± 5165,456 ± 2622,110 ± 1225,854 ± 501343 ± 5158.4 ± 5.1
ULA, adaptive step8,933 ± 1,0954,503 ± 2481,741 ± 40830 ± 26329 ± 4152.3 ± 3.8
ULA, RFPO steps11,661 ± 1,0104,793 ± 4161,644 ± 232-12 ± 17340 ± 7154.3 ± 3.5
Gradient ascent12,218 ± 2,5435,126 ± 5662,048 ± 2605,448 ± 2,005341 ± 2131.4 ± 10.8

Thank you again for your time and constructive feedback. We understand that this is a busy period and would be grateful if, when your schedule permits, you could let us know whether our additional experiments and manuscript revisions address your concerns. If any concerns remain, we would appreciate the opportunity to provide further clarification.

Reviewer_GX9iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Refinement-Based Flow Policy Optimization (RFPO), an online reinforcement learning framework that combines Q-guided action refinement with self-target flow matching. Specifically, the method generates actions from the current flow policy, refines them using the Metropolis-adjusted Langevin algorithm (MALA) toward a Q-induced Boltzmann distribution, and subsequently trains the flow policy using the refined actions as fixed regression targets.

The paper also provides a theoretical analysis of the resulting distributional dynamics, establishing monotonic improvement of an entropy-regularized policy objective under idealized assumptions. Experiments on synthetic multimodal distributions and six MuJoCo continuous-control tasks demonstrate the feasibility and competitive performance of the proposed approach.

Critical strengths
  • Clear motivation and a reasonable algorithmic framework. The paper addresses an important challenge in flow-based reinforcement learning: optimizing expressive policies when the target action distribution is implicitly defined by a Q-function. The separation between Q-guided sample refinement and flow-matching optimization is conceptually straightforward and avoids backpropagating critic gradients through the iterative flow sampler during policy updates.

  • Well-structured theoretical analysis. The paper provides a mathematically organized characterization of the refinement-learning procedure. In particular, the connection between iterative MALA refinement, self-target flow matching, and the induced policy distribution helps explain the behavior of the algorithm under idealized conditions.

  • Reasonable experimental evaluation and clear presentation. The experiments include synthetic multimodal distributions and several standard continuous-control benchmarks, with comparisons against representative diffusion- and flow-based reinforcement learning methods. The paper is generally well written, and the implementation details are sufficiently documented.

Critical weaknesses
  • Limited theoretical novelty and lack of new insights into policy optimization. My primary concern is that the theoretical analysis largely characterizes an already-defined algorithm rather than providing new insights that meaningfully inform its design. Specifically, the main theoretical result establishes that, under exact flow-matching optimization and a fixed critic, each refinement-learning cycle is equivalent to applying a finite number of MALA transitions. The resulting monotonic improvement follows from the invariant-distribution property of MALA and the data-processing inequality for KL divergence. While mathematically valid and useful for understanding the proposed procedure, these results are largely consequences of established properties of MCMC and flow matching. They do not appear to reveal fundamentally new principles of policy optimization or provide actionable guidance for improving the algorithm. Furthermore, the analysis assumes a fixed critic, exact flow-matching optimization, and exact ODE integration, whereas practical online reinforcement learning involves continuously updated critics and approximate optimization. Consequently, the theoretical guarantees offer limited insight into the behavior and stability of the actual learning algorithm.

  • Limited and inconsistent empirical improvements. Although RFPO achieves competitive performance on several MuJoCo tasks, its improvements over existing methods are not consistently substantial. For example, RFPO performs well on HalfCheetah and Walker2d but is noticeably outperformed by SAC and DPMD on Hopper, while DIPO achieves slightly better performance on Ant. On other tasks, the improvements over the strongest baselines are relatively small. These results demonstrate that RFPO is a viable alternative to existing methods, but they do not provide sufficiently strong evidence of a clear and consistent advantage.

  • Insufficient evaluation on more challenging environments. The empirical evaluation is restricted to six standard MuJoCo continuous-control tasks and synthetic two-dimensional distributions. While these benchmarks are useful for establishing feasibility, they provide limited evidence of the broader effectiveness and scalability of the proposed framework. In particular, the paper does not investigate more challenging settings involving sparse rewards, difficult exploration, or complex contact-rich control. Given the additional complexity introduced by MALA-based refinement and iterative flow matching, stronger empirical evidence is needed to demonstrate meaningful advantages beyond conventional continuous-control benchmarks.

  • Insufficient empirical analysis of the key algorithmic components. The paper does not adequately isolate the contributions of MALA refinement, current-policy initialization, and self-target flow matching. In particular, it remains unclear whether the Metropolis adjustment provides meaningful empirical benefits over simpler alternatives such as unadjusted Langevin refinement or Q-gradient ascent. Such comparisons are important for establishing whether the specific design choices of RFPO are necessary, especially given the close relationship between the proposed approach and existing refinement-based generative policy learning methods.

Questions for authors
  • What new insights does the theoretical analysis provide beyond existing MCMC and flow-matching results? The main theoretical guarantee appears to follow from the invariant-distribution property of MALA and the KL data-processing inequality, under exact distribution matching. Could the authors clarify what genuinely new understanding of policy optimization is obtained from this analysis? In particular, does the theory suggest any concrete algorithmic design principles or explain the practical advantages of RFPO over existing methods?

  • Can the theoretical analysis be extended to practical online reinforcement learning? The current results assume a fixed Q-function and exact flow-matching optimization. How would critic nonstationarity, approximation errors, and finite optimization steps affect the claimed monotonic improvement? Even a preliminary analysis or empirical investigation of these effects would help bridge the gap between theory and practice.

  • Can the authors provide results on more challenging environments? The current evaluation is limited to standard MuJoCo benchmarks. It would be valuable to evaluate RFPO on more challenging continuous-control tasks, particularly those involving complex dynamics, sparse rewards, or difficult exploration, to better establish its scalability and practical advantages.

  • How essential is MALA compared with simpler refinement methods? Could the authors provide controlled comparisons with unadjusted Langevin refinement and deterministic Q-gradient ascent under the same flow-matching framework? Such ablations would help establish whether the Metropolis correction is responsible for meaningful performance improvements.

  • What explains the relatively weak performance on certain tasks? RFPO performs substantially worse than several baselines on Hopper and does not consistently outperform the strongest methods on other environments. Could the authors provide an analysis of these cases and discuss whether the limitations arise from critic estimation, refinement dynamics, or policy optimization?

10 comments
AuthorsOpenReview ↗

Regarding Weakness 1

Thank you for raising this point. Our novelty claim concerns the iterative policy update introduced in Sections 4.1 and 4.2, while the role of the theoretical analysis in Section 5 is to establish a policy-improvement property of that update. RFPO refines actions generated by the current policy and trains that same policy on the refined actions through self-target flow matching.

Using established results from flow matching and MCMC, we show that under exact flow-matching optimization and exact ODE integration, each refinement–learning cycle is equivalent to applying a finite number of MALA transitions to the current policy distribution (Lemma 5.2). With the critic and refinement kernel held fixed, the Q-induced target distribution is a fixed point, and the KL divergence from the policy to that target monotonically decreases across successive cycles (Theorem 5.3). Equivalently, each cycle yields a policy whose state-wise entropy-regularized objective for the fixed critic is at least as high as that of the preceding policy.

Some refinement-based online RL methods (e.g., DIPO’s released implementation [1] and DACER-F [2]) start each round of refinement from buffered actions. In RFPO, the policy fitted to the refined action distribution in one cycle directly generates fresh starting actions for the next, closing the refinement–learning loop through the policy itself. Under the stated assumptions, current-policy initialization allows finite-step refinement to accumulate across cycles, yielding πn=π0MsnK\pi_n = \pi_0 M_s^{nK}. The Metropolis--Hastings (MH) correction ensures that MsM_s preserves the QQ-induced target. Theorem 5.3 uses this recursion and target invariance to establish the policy-improvement guarantee.

Appendix G in the revised version tests these two design choices separately in online RL. At matched step sizes, removing the MH correction lowers the mean final return on all six tasks (Table 5). With independently sampled source noise in both initialization variants, starting refinement from the current policy yields higher mean final returns than starting from replay-buffer actions on four of the six tasks (Table 6).

In Section 5, we treat the critic as given when analyzing policy improvement. This isolates the effect of the refinement–learning cycle on the policy distribution under exact fitting and integration. Section 6.2 complements this analysis by reporting online returns for the full actor–critic algorithm, where the critic evolves, flow matching uses a finite number of optimization steps, and the ODE is integrated numerically.

We thank the reviewer again for this feedback. We have revised the opening paragraph of Section 5 to make the motivation and scope of the analysis explicit and explain how it establishes a policy-improvement property of the refinement–learning cycle introduced in Sections 4.1 and 4.2.

[1] Long Yang et al. Policy Representation via Diffusion Probability Model for Reinforcement Learning. arXiv:2305.13122, 2023.

[2] Tianze Zhu et al. Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving. arXiv:2603.02613, 2026.

AuthorsOpenReview ↗

Regarding Weakness 2

In Section 6.2, we already included Table 2 to make final performance easier to compare across all evaluated methods, given the closely spaced learning curves in Figure 3. The table reports the final smoothed mean returns and highlights the two highest means for each task. RFPO ranks first on four of the six tasks and second on Ant, placing in the top two on five tasks, the most of any evaluated method. The margins vary across tasks. RFPO's final mean is 10.5% higher than that of the strongest baseline on Walker2d and 0.9% higher on Swimmer. On Hopper, RFPO ranks fourth, behind DPMD, SAC, and DIPO.

To compare the same final means across tasks with different return scales, we use task-wise linear normalization with two reference points [3]. For each task, we map the lowest and highest final mean returns among the evaluated methods to 0 and 1, respectively. The last column reports each method's mean normalized score across the six tasks, computed before rounding.

MethodHalfCheetahWalker2dHopperAntSwimmerHumanoidStandupMean
RFPO (Ours)1.0001.0000.3620.9711.0001.0000.889
DIPO0.9040.7310.6071.0000.6820.6630.764
SAC0.1470.6120.9530.6750.9800.7310.683
DPMD0.4400.4381.0000.9530.0000.0000.472
SDAC0.8300.0000.3040.0000.6610.8580.442
RFM0.0000.1080.0000.0060.5650.6790.226

RFPO achieves the highest mean normalized score (0.889), followed by DIPO (0.764) and SAC (0.683). We add this aggregate summary alongside the per-task results in the revised manuscript.

[3] R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021.

AuthorsOpenReview ↗

Regarding Weakness 3

Our evaluation asks whether RFPO can train an expressive flow policy when a Q-function implicitly defines the target action distribution. Assessing how this actor update interacts with reward discovery in sparse-reward tasks and with task-specific design choices in contact-rich manipulation goes beyond the empirical scope of this study.

In sparse-reward settings, obtaining experience that provides an informative learning signal for the critic can itself be a major bottleneck. Goal relabeling [4], intrinsic exploration rewards [5, 6], and explicit exploration strategies such as Go-Explore [7] address this part of the learning problem.

Contact-rich manipulation introduces additional choices about object observations, control interfaces, and reward design. These choices vary across benchmarks such as Fetch and Shadow Hand [8], Meta-World [9], robosuite [10], Adroit [11], and ManiSkill [12].

Our experiments directly test the contribution studied here. Section 6.1 uses specified Q-functions to assess whether the refinement–learning cycle captures multimodal target structure. Section 6.2 tests the same cycle in online RL with a continually updated critic across six MuJoCo tasks, including HumanoidStandup-v4 with a 17-dimensional action space. These tasks also provide a common benchmark for comparison with representative online generative-policy methods (DIPO [1], DPMD and SDAC [13], and RFM [14]). Appendix I estimates RFPO's network computation at 24.7 GFLOPs per environment step, including MALA refinement and flow matching. The corresponding estimates are 0.6 for SAC, 14.6 for DIPO, 51.6 for DPMD, 85.3 for RFM, and 124.4 for SDAC. RFPO's estimated cost on Hopper-v4 is therefore less than half that of three of the four generative-policy baselines. Across the six MuJoCo tasks, RFPO achieves the highest mean final return on four tasks (Figure 3 and Table 2). The controlled ablation in Appendix G.1 further shows that removing the Metropolis–Hastings correction while retaining the same Langevin proposal rule and step-size schedule lowers the mean final return on all six tasks, with the largest drop on Ant, from 5,854 to -12.

[4] M. Andrychowicz et al. Hindsight Experience Replay. NeurIPS, 2017.

[5] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven Exploration by Self-supervised Prediction. ICML, 2017.

[6] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by Random Network Distillation. ICLR, 2019.

[7] A. Ecoffet et al. First return, then explore. Nature, 2021.

[8] M. Plappert et al. Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research. arXiv:1802.09464, 2018.

[9] T. Yu et al. Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. CoRL, 2020.

[10] Y. Zhu et al. robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. arXiv:2009.12293, 2020.

[11] A. Rajeswaran et al. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. RSS, 2018.

[12] T. Mu et al. ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations. NeurIPS Datasets and Benchmarks, 2021.

[13] H. Ma, T. Chen, K. Wang, N. Li, and B. Dai. Efficient Online Reinforcement Learning for Diffusion Policy. ICML, 2025.

[14] Z. Li, S. Tang, and N. Azizan. Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies. ICML, 2026.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions in Response to Reviewer Feedback

We thank the Area Chair and all reviewers for the time and care they devoted to evaluating our work. The reviewers' insightful and constructive feedback helped us substantially improve the manuscript by clarifying RFPO's contribution, explaining the scope of our theoretical analysis, and strengthening the empirical evaluation.

Summary of revisions

Contribution and the scope of the theoretical analysis (Reviewer GX9i: W1, Q1, Q2; Reviewer py8F: W2; Reviewer kpG3: Q1)

  • (Section 5, second sentence of the opening paragraph) We have revised the opening paragraph to clarify that the analysis aims to establish a policy-improvement interpretation of RFPO's refinement–learning cycle (Sections 4.1 and 4.2) for a fixed critic.

Aggregate normalized performance (Reviewer GX9i: W2)

  • (Table 2) We have added a column reporting each method's mean normalized score across the six MuJoCo tasks alongside the per-task returns.

Design rationale, differences from refinement-based baselines, and component ablations (Reviewer GX9i: W4, Q4; Reviewer py8F: W1, Q1, Q2; Reviewer kpG3: W1, W2, Q2)

  • (Appendix G) We have added three tables reporting controlled ablations of the refinement rule, refinement initialization, and source-noise pairing, along with a figure comparing final returns across different numbers of MALA refinement steps per update. The appendix explains the rationale for these design choices and details the experimental setups and results.

Hopper performance (Reviewer GX9i: Q5)

  • (Appendix K and Section 6.2) We have added rollout diagnostics and a retraining experiment with γ=0.998\gamma=0.998 on Hopper. Section 6.2 summarizes how the training discount contributes to the gap in undiscounted return.

We have also updated the Limitations discussion in Section 7 (Conclusion) to remove the statement about the limited evaluation of individual design choices, since Appendix G now reports controlled ablations that isolate their effects.

AuthorsOpenReview ↗

Further Clarifications on RFPO’s Novelty and Baseline Selection

We thank the reviewers for the time and care they devoted to evaluating our work and for their insightful feedback. The discussion during the rebuttal period helped us clarify our contribution and strengthen the manuscript.

We have added two clarifications to the revised manuscript.

  • At the start of Section 4, we have added a paragraph clarifying RFPO's novelty and its key distinction from prior refinement-based methods such as DIPO and DACER-F, which initialize refinement with stored actions. The paragraph explains how RFPO adapts the refinement–learning structure of amortized MCMC to online policy optimization, with the flow policy fitted in one cycle directly generating fresh starting actions for the next round of refinement.

  • In the first paragraph of Section 6.2, we have made our baseline selection criteria more explicit. The baselines are selected to evaluate RFPO in online RL, with an emphasis on policy expressiveness and how each method optimizes its actor.

AuthorsOpenReview ↗

Additional manuscript updates

Additional manuscript updates

Since our previous update, we have made two further revisions.

  1. Accumulation of refinement across cycles (Sections 5 and 6.1, Figure 3). We added a controlled experiment to examine whether refinement gains accumulate when flow matching uses a finite number of updates and the flow ODE is integrated numerically. Both variants train the actor with the same refinement and learning budgets. RFPO starts each refinement from the current policy, while the comparison method always starts from the same pretrained policy. After 3,000 cycles, RFPO achieved a lower empirical total variation distance to the target for all five seeds on each of the six synthetic targets. This finding is consistent with the accumulation of refinement described in Lemma 5.2.

  2. Refinement initialization in online RL (Section 6.2). We now discuss the initialization ablation previously reported in Appendix G.2 (Table 6) in the main text. Using the updated policy to generate the starting actions for the next refinement cycle yielded higher mean returns than replay-buffer initialization on four of the six MuJoCo tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.