acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows

Abstract

Diffusion and flow policies can model complex behaviors in offline reinforcement learning (RL). However, penalizing their KL divergence from the behavior policy can discourage actions having high critic values with low behavior density. Directly refining behavior proposals may be an alternative, yet Gaussian or deterministic editors limit expressiveness to represent multiple separated modes for the same proposal. In this work, we introduce Proposal-Conditioned Refinement Flows (PReFlow), a policy extraction method combining critic-based proposal selection with a conditional refinement flow. To optimize proposal selection and refinement together, we formulate a KL-regularized objective whose optimum induces a Gibbs policy over final actions under a Gaussian-smoothed behavior prior. The refinement flow can represent multiple high value modes, while a proposal centered Gaussian reference regulates large action changes. This Gaussian reference further enables us to make use of simulation-free, closed form adjoint matching targets from sampled endpoints and critic gradients, yielding a single velocity regression loss without a backward adjoint solve. On 50 OGBench tasks, PReFlow achieves competitive offline performance and the highest aggregate score among the compared methods after online fine-tuning, reaching 91% after 500K environment steps.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Flow Matching, Policy Improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5kVCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PReFlow, which is a policy extraction method that applies additive action deltas using conditional flows. To my understanding, the method samples candidate actions from a pretrained behavior policy, and chooses the candidate with the highest critic value. Then, it uses a conditional flow to refine the action such that the refinement distribution is KL-regularized toward some proposal-centered Gaussian reference. The method is evaluated on a wide array of tasks in OGBench and additional analyses on the method such as computational efficiency.

Critical strengths
  • Training is efficient under the proposal conditioned refinement flow, and reduced training time supported in results section (Figure 4).
  • Very extensive empirical evaluation (50 OGBench tasks).
Critical weaknesses
  • I am not convinced why proposal-conditioned refinement is preferable to directly optimizing an expressive flow policy. The motivation of selecting high-Q actions while avoiding out-of-distribution actions is closely related to existing work on flow-based offline RL, particularly ReFORM [1], which aims to improve a flow policy while preserving the learned behavior-policy support. However, I do not see a discussion or comparison with this work, nor sufficient justification for why action-space refinement with a Gaussian reference is preferable to support-constrained policy improvement. While the paper compares against QAM, it is unclear whether the improvements arise from the refinement architecture itself or from the different regularization and optimization procedures.
  • The motivating comparison between Gaussian and flow-based refinement largely demonstrates the known limitation of unimodal Gaussian distributions. Since existing flow policies are already capable of representing multimodal action distributions, I do not find this comparison sufficient to establish the necessity of a separate refinement flow. Furthermore, the theoretical proposal-selection objective depends on the post-refinement value, whereas the practical algorithm uses greedy selection based on the original critic value. It is unclear how much this approximation affects performance, especially given that sampling a finite number of behavior-policy actions does not guarantee coverage of different action modes.

[1] Zhang, S., So, O., Ahmad, H. M. S., Yu, E. Y., Cleaveland, M., Black, M., & Fan, C. (2026). ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation. The Fourteenth International Conference on Learning Representations.

Questions for authors
  • How does PReFlow relate to support-constrained policy improvement methods such as ReFORM [1]? In particular, what are the advantages of allowing local action-space refinements beyond the behavior-policy support, compared with improving the policy while preserving its learned support?
  • Why is proposal-conditioned action-space refinement necessary or preferable to directly learning an expressive flow policy? Could the authors provide more controlled comparisons that isolate the benefits of the refinement architecture from differences in regularization and optimization?
4 comments
AuthorsOpenReview ↗

Response to W1–W2 and Q1: ReFORM Comparison and Refinement Design (1/2)

W1/Q1: Comparison with ReFORM

We thank the reviewer for highlighting ReFORM and will add the citation and discussion. ReFORM$^{[1]}$ learns a reflected-flow noise generator that maximizes the critic value of the resulting actions while keeping the generated noise inside the bounded source domain of a learned behavior flow. This preserves the learned behavior support while allowing probability to concentrate on high-value actions within it.

PReFlow’s proposal-conditioned formulation provides direct control over action changes while retaining an expressive refinement distribution. Each sampled behavior action serves as the center of a Gaussian reference, and a conditional flow learns critic-guided refinements around that proposal. The Gaussian reference penalizes large action displacements, with the refinement scale δ\delta and temperature τ\tau controlling the trade-off between displacement and critic value. This gives PReFlow the flexibility to favor nearby actions with high critic values, including those excluded by a learned support constraint.

The same conditional Gaussian structure enables closed-form adjoint-matching targets, reducing refinement learning to a single velocity regression loss. These targets are constructed from sampled endpoints and endpoint critic gradients, eliminating the need to backpropagate through either the behavior or refinement rollout. ReFORM distills its behavior flow into a one-step model to reduce the backpropagation cost, but training its noise generator still requires backpropagation through the flow rollout. PReFlow therefore combines expressive action-space refinement with a training procedure that avoids this rollout backpropagation.

Furthermore, we evaluate ReFORM under the advanced QAM protocol. We port the official ReFORM implementation into the QAM codebase and evaluate it under exactly the same protocol, network architecture, and shared hyperparameters as PReFlow and all other baselines. Following the architecture of QAM, we apply action chunking with horizon 5 on manipulation tasks, which increases the radius of a uniform ball. ReFORM uses a single hyperparameter configuration across all domains, so we keep its default.

Table R.7. Method comparison on OGBench advanced setting (3 seeds).

Methodalaghmhlscenep33p44c2c3c4Average
ReFORM360300751000555230
QAM81186711971000643344
PReFlow8212652497100665971553

As shown in Table R.7, ReFORM degrades sharply on long-horizon locomotion tasks. ReFORM performs critic-guided policy improvement by optimizing the source-noise distribution, while its behavior flow and one-step approximation are trained through cloning and distillation, respectively. For a given learned behavior mapping, noise optimization is restricted to actions reachable from its bounded latent domain and cannot recover a high-value action outside this reachable set. PReFlow instead refines each proposal directly in action space, so the critic can correct an imperfect behavior proposal through a local displacement controlled by δ\delta and τ\tau, rather than inheriting its errors.

W2: Refinement Expressivity and Proposal Selection

Thank you for the comments. The comparison in Table 2 is intended to isolate the refinement family under the same conditional objective. The comparison with directly fine-tuning a flow policy toward the same target is given in our response to Q2.

For the proposal selection, we replaced the critic score Q(s,ak)Q(s, a_k) with a Monte-Carlo estimate of the post-refinement value Vδ(s,ak)V_\delta(s, a_k) using J=8J=8 samples per proposal and observed no consistent difference, while the estimate requires JJ times more critic evaluations per proposal. Please see Table R.1 in our response to Reviewer K3Gn.

As for mode coverage, we want to emphasize that a single proposal can generate multiple final-action modes. Each proposal conditions an entire refinement distribution, and our fixed state, fixed proposal analysis provides direct empirical evidence of this behavior. Refinement with K=1K=1 already substantially improves over selection among multiple unrefined proposals (Appendix E.4). Increasing KK to 2 provides further gains (Figure 5).

References

[1] ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation. ICLR, 2026.

AuthorsOpenReview ↗

Response to Q2: Controlled Comparisons of Proposal-Conditioned Refinement (2/2)

Q2: Controlled Comparisons of Proposal-Conditioned Refinement

We do not claim that directly learning an expressive flow policy is wrong. It is a valid alternative, and QAM is one instance of it. Our focus is the narrower problem of reaching high-value actions where the behavior density is low, which can be viewed as sampling from the tilted density πβeτQ\pi_\beta e^{\tau Q}. When the whole policy is fine-tuned directly, it has to move probability mass across the action space to reach these low-density regions. Conditioning on the proposal makes the problem local. For each proposal, the refinement only needs to tilt a Gaussian neighborhood N(a,δ2I)\mathcal{N}(a,\delta^2 I) toward higher QQ, and this conditional target admits a closed-form training objective.

We compare the two formulations through the toy experiment in Fig. 2 and an additional OGBench comparison with aligned regularization.

The toy experiment in Fig. 2 uses a fixed proposal, and both the refinement flow and the behavior flow fine-tuned with QAM are trained toward the same tilted target with the same network and training budget. The refinement recovers all three modes (KL 0.09), while the directly fine-tuned flow concentrates most of its mass on a single mode (KL 0.53).

For the OGBench comparison, we train QAM’s behavior flow on Gaussian-perturbed actions a+δϵa+\delta\epsilon and fine-tune it with the same τ\tau and δ\delta as PReFlow. Both methods then share the reference and the temperature, and the remaining difference is the proposal-conditioned formulation together with the closed-form training procedure it enables.

Table R.8. Method comparison with the same marginal distribution (3 seeds).

Methodal (τ=100\tau=100)hm (τ=30\tau=30)scene (τ=3\tau=3)c2 (τ=3\tau=3)Avg
PReFlow79.069.273.153.968.8
QAM, BC on a+δϵa+\delta\epsilon16.80.054.133.326.1
QAM, BC on aa16.40.356.842.829.1

As shown in Table R.8, PReFlow achieves higher mean success rates. We hypothesize that this separation facilitates optimization by allowing the refinement model to learn critic-guided improvements conditioned on each proposal. This interpretation is consistent with the more accurate mode recovery observed in our toy experiment under an exactly matched target distribution.

PReFlow also selects among KK proposals, so we further check whether selection alone explains its advantage. We add the same best-of-KK selection to QAM-E (K=4K=4, at acting time only) on the six domains where PReFlow uses K>1K>1.

Table R.9. QAM-E with best-of-KK selection. PReFlow results are from Table 1.

Methodhmhlscenep33c2c4Avg
QAM-E (K=1K=1)55 → 823 → 2696 → 100100 → 10061 → 1006 → 8053.5 → 81.3
QAM-E (K=4K=4)57.9 → 891 → 3498 → 99100 → 10061 → 1004 → 8053.65 → 83.7
PReFlow65 → 9524 → 6597 → 99100 → 10059 → 10015 → 8060.0 → 89.8

As shown in Table R.9, selection leaves QAM-E’s offline success unchanged and raises its online success only slightly. Both scores remain below those of PReFlow.

Thank you for the detailed response and new experimental results.

W1/Q1. Largely resolved. The additional ReFORM comparison is valuable and demonstrates promising advantages for PReFlow. Please mention how ReFORM was adapted for this action chunking version, and what hyperparameters were used.

W3. Resolved. Tables R.2 and R.3 and the dip test show the benefit of flow refinement persists with sampled proposals and against a full-covariance Gaussian editor.

W4. Resolved by Table R.1.

I share the other reviewers’ views that the aggregate gains over the strongest baselines are modest and concentrated in puzzle-4x4. Still, my main concerns were addressed. I raise to 3.

We sincerely thank the reviewer for their thoughtful feedback and for reconsidering the score. We are glad that our additional experiments and clarifications helped address the reviewer's main concerns. We will include further details on how ReFORM was adapted to the action chunking setting, along with the hyperparameters used, in the revised manuscript.

We greatly appreciate the reviewer's constructive suggestions and support of our work.

Reviewer_RyZqOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes PReFlow, a policy extraction method for offline and offline-to-online RL with flow policies. A behavior flow generates action proposals, the critic selects the best one, and a conditional flow adds a learned refinement to it. The refinement is regularized toward a proposal-centered Gaussian, and the authors show that the joint optimum of selection and refinement is a Gibbs policy under a Gaussian-smoothed behavior prior. This Gaussian reference also yields a closed-form adjoint matching target, so the refinement flow is trained with a single regression loss without a backward adjoint solve. On 50 OGBench tasks, PReFlow is competitive offline and achieves the highest aggregate success rate after online fine-tuning.

Critical strengths

S1. Principled joint objective.

The paper formulates proposal selection and refinement under a single KL-regularized objective, and shows that its optimum is a Gibbs policy under a Gaussian-smoothed behavior prior (Prop. 3.1). This gives a clean interpretation of selection-plus-refinement methods and explains how nearby proposals can lend density to high-value actions with low behavior density.

S2. Simulation-free training.

Using a proposal-centered Gaussian reference, the authors derive a closed-form adjoint matching target (Prop. 3.2). This removes the backward adjoint solve required by QAM and reduces training to a single velocity regression loss, which is simple to implement.

Critical weaknesses

W1. Why a multimodal refinement?

The motivation (Sec. 3.1, Fig. 2) is that Gaussian editors cannot capture multiple modes for a fixed proposal. However, the executed policy is the marginal π(a~∣s)=∫πbase(a∣s) q(a~−a∣s,a) da.\pi(\tilde{a} \mid s) = \int \pi_{\mathrm{base}}(a \mid s)\, q(\tilde{a} - a \mid s, a)\, \mathrm{d}a. Since πbase\pi_{\mathrm{base}} is already a multimodal flow, π(a~∣s)\pi(\tilde{a} \mid s) is multimodal even with a Gaussian editor. The toy in Fig.2 fixes a single proposal and thus does not test this.

W2. Marginal performance gains.

Offline, PReFlow (53) is below RQL (55) in aggregate (Table 1), and much of its offline score comes from p44, the only domain using a large refinement scale (δ=0.9\delta = 0.9). After online fine-tuning, its aggregate score (91, CI [90,92]) overlaps with QPILOTS-M (90, CI [87,91]). Moreover, hyperparameters (τ\tau, δ\delta, KK, critic expectile κ\kappa) are tuned per domain on the evaluation tasks themselves, which makes these small gaps hard to interpret.

W3. Missing inference and interaction cost.

Generating each action requires two sequential flow rollouts (behavior and refinement) plus KK critic evaluations, roughly doubling the sequential depth of single-flow methods such as QAM. Fig.4 and Table.5 report only per-update time.

Questions for authors

Q1. In the toy of Fig.2, if proposals are sampled from πβ\pi_\beta rather than fixed, does the marginal π(a~∣s)\pi(\tilde{a} \mid s) under a Gaussian editor still fail to match the target πδ⋆\pi^\star_\delta? Reporting the KL to the marginal target, in addition to the conditional KL in Fig.7, would clarify this.

Q2. Table 2 shows a clear gain over a diagonal Gaussian editor, which is a reasonable practical baseline. However, it does not isolate multimodality, since a diagonal Gaussian cannot capture correlations across action-chunk dimensions. Given that the toy in Fig. 2 already uses a full-covariance Gaussian, does the gap persist against a full-covariance Gaussian editor on OGBench? Is there evidence that the learned refinement is multimodal for a given (s,a)(s, a)?

Q3. The current performance gains appear marginal (W2). Could the authors provide additional evidence of improvement, for example, results on more challenging tasks where existing methods struggle?

Q4. Since computational efficiency is claimed as a contribution, could the authors report total wall-clock time for the full offline-to-online pipeline, including environment interaction, compared with QAM and RQL?

4 comments
AuthorsOpenReview ↗

Response to W1–W2 and Q1–Q2: Refinement Expressivity and Performance Gains (1/2)

W1/Q1: Marginal Policies with Sampled Proposals

We thank the reviewer for this helpful suggestion. We keep the toy experiment of Fig. 2 and change only the proposal. Instead of a fixed point cc, proposals are drawn as a∼N(c,σβ2I)a\sim\mathcal{N}(c,\sigma_\beta^2 I). We train the flow refinement and a full-covariance Gaussian editor for each σβ\sigma_\beta. We then compute the KL between each executed marginal and πδ\pi_\delta. We report the conditional KL (Figure 7), and report the KL between the executed marginal and the target marginal ∫πβ(a),q⋆(⋅∣a),da\int \pi_\beta(a)\\,q^\star(\cdot\mid a)\\,da.

Table R.2. Marginal KL of flow and full-covariance Gaussian editors (toy).

σβ\sigma_\betaMarginal KL (Flow)Marginal KL (Gaussian)Conditional KL (Flow)Conditional KL (Gaussian)
0.00.0760.5850.0750.585
0.10.1140.5960.1390.608
0.20.1480.6240.1960.674
0.30.1530.6620.2670.776
0.40.1230.2180.2920.617

As shown in Table R.2, flow refinement achieves lower marginal KL at every tested proposal variance. Up to σβ=0.3\sigma_\beta=0.3, the Gaussian editor sends every proposal to the single highest mode, while the flow refinement places mass on all three modes. At σβ=0.4\sigma_\beta=0.4, the proposals are spread widely enough that the Gaussian marginal reaches more than one mode, yet its marginal KL remains higher than that of flow refinement. Since the Gaussian conditional is still unimodal, its conditional KL is also about twice as large.

This same gap appears in OGBench, where proposals are drawn from a flow behavior policy as the reviewer notes. Both the diagonal and the full-covariance Gaussian editors remain below the flow refinement in this setting. Please see Table R.3 for the result.

Q2: Full-Covariance Gaussian Refinement and Multimodality

We thank the reviewer for this suggestion. We have added a full-covariance Gaussian editor by parametrizing the covariance with a learned Cholesky factor.

Table R.3. Full-Covariance Gaussian editor (OGBench).

DomainDiag GaussianFull-cov GaussianPReFlow
al76.376.179.0
hm57.160.869.2
scene68.170.773.1
c240.841.753.9
p4421.146.860.8

As shown in Table R.3, full covariance improves the Gaussian editor’s average performance across the four domains in Table 2. Furthermore, on puzzle-4×4, the improvement with flow refinement is substantial, compared with the diagonal and full-covariance Gaussian editors.

To directly show whether the learned refinement is multimodal for a fixed state and proposal, we take the p44 PReFlow checkpoints and sample 128 data states per task. For each state we fix a proposal a∼πβ(⋅∣s)a\sim\pi_\beta(\cdot |s) and draw 4096 refinements y∼qϕ(⋅∣s,a)y\sim q_\phi(\cdot |s,a) (d=25d=25). We use one half of the samples to estimate the first principal axis and apply Hartigan’s dip test to the projections of the other half onto this axis.

We find unimodality is rejected (p<0.01p<0.01) for 49% of the 640 (s,a)(s,a) pairs, compared with 0% for Gaussian controls with matched means and covariances. These results align with the result of Table R.3, which shows the remaining gap between the full-covariance Gaussian and PReFlow on p44.

W2: Performance Gains and Hyperparameter Settings

Thank you for the careful reading of Table 1. For the aggregate performance comparison in W2, please see the response to the reviewer K3Gn (W4), who raised a similar concern about the magnitude of the performance gains. We discuss the comparisons with RQL and QPILOTS across offline and online settings.

Regarding puzzle-4×4, a large refinement scale alone does not explain the gain. The released QAM configurations also use a large edit scale on this domain, 0.9 for QAM-E and 0.8 for FEdit, yet they achieve offline success rates of only 39% and 34%, respectively, compared with 66% for PReFlow. As shown in Table R.3, the gain on puzzle-4×4 comes from the expressive refinement, not from the scale alone.

Regarding hyperparameters, PReFlow uses a single configuration per domain shared across its five tasks, following the same protocol as QAM and the other baselines with domain-specific settings.

AuthorsOpenReview ↗

Response to Q3–Q4 and W3: Challenging Tasks and Computational Cost (2/2)

Q3: Challenging Tasks and Additional AntMaze Evaluation

Thank you for this suggestion. We would first like to point out that our experiments already follow the advanced setting of 50-task OGBench evaluation used by QAM, which includes long-horizon tasks with sparse rewards and complex behavior distributions. In particular, puzzle-4×4 remains challenging for strong existing methods: QAM-E and FEdit achieve offline success rates of 39% and 34%, respectively, compared with 66% for PReFlow. Our existing evaluation thus demonstrates substantial improvement on a difficult domain.

Nevertheless, we understand the reviewer’s concern and provide additional evidence of improvement. We extended our evaluation to the six D4RL AntMaze tasks, following the training and evaluation protocol used by FAV$^{[1]}$ for its comparison with QAM.

We report the normalized score averaged over the evaluations at 300K, 400K, and 500K steps across 8 seeds. As FAV selects QAM’s temperature separately for each maze size, we select PReFlow’s τ\tau, δ\delta, and KK per maze size.

The baseline results are taken from Table 11 of FAV.

Table R.4. D4RL AntMaze offline results.

TaskReBRACFQLQAMFAVPReFlow (ours)
umaze9896959496
umaze-diverse8489868687
medium-play9078838182
medium-diverse8471786777
large-play5284637580
large-diverse6483727881
Average Score78.783.579.580.283.8

As shown in Table R.4, PReFlow reaches 83.8, which is higher than QAM (79.5) and FAV (80.2) and comparable to FQL (83.5). On large-play and large-diverse, PReFlow improves over QAM by 17 and 9 percentage points, respectively, providing further evidence of its effectiveness on challenging tasks beyond OGBench.

W3/Q4: End-to-End Runtime and Refinement Steps

We thank the reviewer for raising this point. PReFlow generates each action through two sequential flow rollouts, first sampling a proposal from the behavior flow and refining it afterwards. The training update, however, is cheaper than QAM’s, since the refinement’s regression target is constructed in closed form without the backward adjoint simulation that QAM requires (Table 5). In practice, the cheaper update offsets the extra rollout, which allows PReFlow to use a second refinement conditioned on the state and proposal without increasing the overall training time relative to QAM.

To make this concrete, we measured the full wall-clock time of the full offline-to-online pipeline on cube-triple task 2 over 3 seeds. Each run uses a single RTX 3090 GPU.

Table R.5. Wall-clock time (hours) on Cube-triple-t2 (3 seeds).

MethodOffline timeOnline timeTotal timeFinal Success (%)
PReFlow2.051.533.696
QAM2.331.664.094
RQL1.921.583.5147

As seen in Table R.5, despite the additional rollout, PReFlow completes the pipeline approximately 10% faster than QAM, and in roughly the same time as RQL, while reaching a much higher final score than RQL.

Table R.6. Number of refinement flow steps. Success rates (%) averaged over the 50 OGBench tasks with 8 seeds.

Refinement stepsOfflineOnline
145.890.5
349.191.2
10 (PReFlow)52.590.7
QAM4470

We further want to emphasize that the extra sequential depth can also be reduced directly. With a single refinement step, PReFlow reaches 90.5 after online fine-tuning, matching the ten-step result, while its offline score still remains above QAM.

References

[1] Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inference. NeurIPS, 2026.

Thank you for the detailed response and additional experiments.

W1/Q1. Resolved. With proposals sampled from the behavior distribution, the flow refinement achieves lower marginal KL than the full-covariance Gaussian editor at every tested variance, which addresses my concern that the toy in Fig. 2 did not test the executed marginal.

Q2. Resolved. The full-covariance Gaussian editor on OGBench and the dip test on p44 provide direct evidence that the gain comes from multimodal refinement rather than from modeling correlations alone.

W3/Q4. Resolved. The end-to-end wall-clock comparison and the refinement-step ablation show that the additional rollout does not increase the overall cost relative to QAM.

W2/Q3. This remains my main concern. PReFlow clearly improves over QAM and its variants, but against the strongest baselines the gains are still small. On OGBench, it is below RQL offline (53 vs. 55), and its online score (91) overlaps with QPILOTS-M (90) within the confidence intervals. The new D4RL AntMaze results show the same pattern: PReFlow (83.8) is on par with FQL (83.5) and below ReBRAC on the medium tasks, with hyperparameters selected per maze size. The substantial improvement is largely confined to puzzle-4x4.

Overall, the concerns about the motivation and the cost are addressed, and the method is well supported technically. However, the empirical advantage over existing methods remains limited. I am raising my score to 2.

We sincerely thank the reviewer for their thoughtful feedback and for reconsidering the score. We appreciate the reviewer's constructive comments and are glad that our additional experiments and clarifications helped address several of their concerns.

Thank you again for your time and careful consideration of our work.

Reviewer_K3GnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies training diffusion, and flow, models in the offline-to-online reinforcement learning (RL) setting. The paper builds on previous methods that refine behavior proposals (action proposals generated from a behavior cloning policy) that either use a gaussian or deterministic refiners/editors. The main contribution of the paper is the introduction of proposal conditioned refinement flow (PReFlow) that works in two stages: First, it samples from a flow policy trained using behavior cloning. Then, it refines this action using a flow model that is regularized to follow a gaussian distribution centered around the behavior cloning action. The method typically samples multiple actions (proposals) and assess their return to determine the best action. Finally, to train the model, the paper derives a training target that avoids backward adjoint simulation.

The paper validates the proposed method in OGBench in offline-to-online RL training. The paper achieve competitive results in comparison to previous methods.

Critical strengths

The paper's strengths can be summarized as follows:

  1. The paper tackles an interesting and open problem of training flow models for RL.
  2. The paper presentation is clear.
  3. The paper does extensive evaluation and achieves a competitive performance in the online RL setting.
Critical weaknesses

The papers' weaknesses can be summarized as follows:

  1. It seems that the main contribution of the paper is using flow instead of, as in prior method, a gaussian distribution at the refinement stage. However, the paper also regularizes this flow to as isotropic gaussian distribution! Furthermore, the introduction of this gaussian adds a variance parameter to control the regularization that is an additional parameter to the temperature parameter in offline RL.

  2. The derived target for the flow model, although valid, doesn't necessarily converge to the optimal tilted distribution according to the papers' formulation. Moreover, the idealized Adjoint Sampling characterization requires both population regression stationarity and consistency of the learned dynamics with the reference bridges. The latter was mentioned in the appendix as well. It would be nice to give this more weight and justification. That should also be placed, or at least mentioned, in the main text due to its importance.

  3. In the theoratical formulation, the proposal value is evaluated to be the expected value after the regularized refinement (Equation 16.) However, in practice, it is selected to be only one action sampled from the behavior policy and evaluated using the critic.

  4. Experimentally, the papers aggregate gains seem modest despite the apparent parameter fine-tuning. In the offline setting, the paper doesn't achieve the best performance. In the online setting, the gains seems somewhat modest especially when compared to other baselines like QPILOTS.

Questions for authors

Please see the weaknesses part.

4 comments
AuthorsOpenReview ↗

Response to W1–W2: Gaussian Reference and Optimality Assumptions (1/2)

W1: Gaussian Reference and Refinement Expressivity

We would like to clarify that using a Gaussian reference for KL regularization and restricting the refinement distribution itself to a Gaussian family are two distinct modeling choices.

A Gaussian reference permits a multimodal optimal refinement distribution. In PReFlow, the conditional flow represents the refinement distribution, and the Gaussian reference only encourages the refinement to stay close to the proposal. Under our regularized objective, the optimal refinement distribution is πref,⋆(Δa∣s,a)∝exp⁡(τQψ(s,a+Δa)−∣Δa∣22δ2),\pi^{\mathrm{ref}, \star}(\Delta a\mid s,a)\propto \exp(\tau Q_\psi(s, a+\Delta a)-\frac{\\|\Delta a\\|^2}{2\delta^2}),

where the quadratic term comes from the Gaussian reference and the critic reweights it through τQψ\tau Q_\psi.

This target can be non-Gaussian and multimodal even though the reference is an isotropic Gaussian, which a conditional flow can approximate but a Gaussian refinement policy cannot. Table 2 compares the flow and Gaussian parameterizations under the same objective, and the flow achieves higher success on all four domains. For the additional multimodality discussion, please see the response to the Reviewer RyZq (Q2).

Regarding the additional parameter, prior methods that refine actions with a Gaussian edit policy, such as EXPO$^{[1]}$ and QAM-E$^{[2]}$, also have an edit scale that is tuned per task. In PReFlow, δ\delta plays this role, so it does not introduce an extra parameter relative to these methods.

W2: Assumptions for the Optimality Characterization

Thank you for highlighting this point. We agree that the assumptions discussed in Appendix A.2 should be stated more prominently in the main text. We will explicitly state that the idealized optimality characterization holds under both population regression stationarity and consistency of the learned dynamics with the reference bridges, and briefly explain the role of the bridge-consistency condition. As empirical support, in the toy problem of Appendix D.4, where the target q⋆q^\star is available in closed form, the learned flow refinement reaches a KL divergence to q⋆q^\star below 0.1 which is lower than both Gaussian refinement and QAM.

References

[1] EXPO: Stable Reinforcement Learning with Expressive Policies. ICLR, 2026.

[2] Q-learning with Adjoint Matching. ICLR, 2026.

AuthorsOpenReview ↗

Response to W3–W4: Proposal Selection and Offline/Online Performance (2/2)

W3: Proposal Scoring and Selection

Thank you for pointing this out. We would like to clarify how the proposal score relates to the theoretical proposal distribution.

The theoretical proposal distribution is proportional to πβ(a∣s)exp⁡(τVδ(s,a))\pi_\beta(a\mid s)\exp(\tau V_\delta(s,a)), where Vδ(s,a)V_\delta(s,a) is the regularized refinement value:

Vδ(s,a)=1τlog⁡Z(s,a),V_\delta(s,a)=\frac{1}{\tau}\log Z(s,a),

and

Z(s,a)=Ey∼N(0,I)[exp⁡(τQ(s,a+δy))]Z(s,a)=\mathbb{E}_{y\sim\mathcal N(0,I)}\left[\exp\left(\tau Q(s,a+\delta y)\right)\right]

As shown in Appendix A.3, under the smoothness assumption and for small δ\delta, Vδ(s,a)=Q(s,a)+O(δ2)V_\delta(s,a)=Q(s,a)+O(\delta^2) holds. This motivates us to use Q(s,a)Q(s,a) as a computationally efficient approximation to the refinement value.

We additionally tested whether this approximation matters by replacing Q(s,ak)Q(s,a_k) with a Monte-Carlo estimate of VδV_\delta using J=8J=8 samples per proposal:

V^δ(s,ak)=1τlog⁡[1J∑j=1Jexp⁡(τQ(s,ak+δyj))],yj∼i.i.d.N(0,I).\hat V_\delta(s,a_k)=\frac{1}{\tau}\log\left[\frac{1}{J}\sum_{j=1}^{J}\exp\left(\tau Q(s,a_k+\delta y_j)\right)\right],\qquad y_j\overset{\mathrm{i.i.d.}}{\sim}\mathcal N(0,I).

Table R.1. Proposal selection by Q(s,a)Q(s,a) versus MC estimate of Vδ(s,a)V_\delta(s,a).

DomainOffline QQOffline V^δ\widehat V_\deltaOnline QQOnline V^δ\widehat V_\delta
al68.063.699.599.3
hm64.066.591.998.8
scene96.596.599.598.8
c261.557.299.9100.0
Average72.571.097.799.2

Table R.1 shows that MC scoring does not consistently improve the performance, while it additionally requires J×J\times more critic evaluations per proposal. Given its lower scoring cost and competitive performance, we retain Q(s,a)Q(s,a) for proposal selection.

We would also like to note that greedy selection among behavior-policy samples has been used in several offline RL methods, including EMaQ$^{[3]}$ (Sec. 3.3), IDQL$^{[4]}$ (Eq. 13) and its flow-based counterpart IFQL$^{[5]}$ (Eq. 11), and RISE$^{[6]}$ (Eq. 6).

In particular, IDQL reports that selecting the highest-Q candidate tends to perform better at evaluation than weighted resampling.

Our selection-rule ablation in Table 6 further supports this choice: argmax achieves the highest mean success rate in four of six settings and remains competitive in the other two.

W4: Offline and Online Performance

We respectfully disagree with this assessment. PReFlow achieves strong performance in both offline learning and online fine-tuning, while RQL and QPILOTS each fall behind in one of the two settings. In the offline setting, PReFlow is second only to RQL with overlapping confidence intervals, while RQL falls clearly behind PReFlow after online fine-tuning (80 vs. 91).

In the online setting, QPILOTS-M and QPILOTS-U are comparable to PReFlow, but their offline scores are lower, and QPILOTS-M takes about seven times longer per training update.

We have additionally evaluated PReFlow on D4RL AntMaze under the setting in which FAV$^{[7]}$ was compared against QAM, where PReFlow performs better than QAM and FAV and is comparable with FQL. Please see our response to Reviewer RyZq (Q3).

References

[3] EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL. ICML, 2021.

[4] IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies. arXiv:2304.10573, 2023.

[5] Flow Q-Learning. ICML, 2025.

[6] Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning. ICRA, 2026.

[7] Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inference. NeurIPS, 2026.

AuthorsOpenReview ↗

Gentle Reminder

As the discussion period is coming to a close, we would like to kindly ask whether our responses have adequately addressed the reviewer's concerns. If so, we would sincerely appreciate it if the reviewer could consider revisiting the score in light of our clarifications.

If the reviewer has any additional comments or questions, please let us know, and we will do our utmost to address them before the deadline.

Thank you for your time and thoughtful consideration.

I thank the authors for their rebuttal.

W1: Resolved

W2: Resolved. The assumptions are clearly stated in the paper text.

W3: Not completely resolved. While the small-(\delta) approximation motivates replacing (V_\delta(s,a)) with (Q(s,a)), it does not justify replacing the theoretically optimal Gibbs proposal distribution (Eq. 16) with greedy best-of-(K) selection. This either requires large number of samples which can be computationally inefficient, or the resulting distribution isn't the same.

W4: In my opionion, this one still is unresolved. The aggregate performance isn't the best in Offline RL. Moreoverover, the gains in offline-to-online RL are modest, which also overlaps in its confidence interval. The fact that it's comparable to FQL[1] isn't a strong point. FQL itself is designed to be an a simple and easy implement algorithm and not the strongest baseline. The main reason in my opinion for keeping this unresolved is that the algorithm either be used in offline RL alone or offline-to-online. In offline, it's not the strongest performance. In offline-to-online setting, the gains are modest and other methods achieve comparable performance with interleaving confidence interval.

I thank the authors so much for their rebuttal, but I'm inclined to keeping my score.


[1]: Flow Q-Learning. ICML, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their thoughtful feedback. We appreciate their recognition of the principled joint objective (Reviewer RyZq), efficient simulation-free training (Reviewers RyZq and 5kVC), extensive evaluation on 50 OGBench tasks (Reviewers K3Gn and 5kVC), and competitive online performance (Reviewers K3Gn and RyZq).

PReFlow formulates proposal selection and refinement under a single KL-regularized objective and derives a closed-form adjoint-matching target that trains the refinement flow through a single regression loss, without backward adjoint propagation.

We address the shared concern about the performance gains (Reviewer K3Gn, W4; Reviewer RyZq, W2) by examining performance and providing additional comparisons with strong baselines.

1. Consistent performance across offline and online settings

Table 1 shows that PReFlow combines strong offline performance with effective online fine-tuning. We further examine consistency across domains by computing average ranks from the 10 OGBench domains in Table 1.

Table R.10. Average rank over the 10 OGBench domains in Table 1 (14 methods; 1 = best; ties receive the average rank). Overall averages the 20 domain phase ranks. Parentheses give the position within each column.

MethodOffline avg. rank ↓Online avg. rank ↓Overall avg. rank ↓
PReFlow (ours)4.95 (1)3.85 (1)4.40 (1)
QPILOTS-M5.40 (3)4.65 (2)5.03 (2)
QAM-E7.15 (6)5.40 (3)6.28 (3)
RQL5.15 (2)7.50 (8)6.33 (4)
QPILOTS-U7.35 (7)5.95 (4)6.65 (5)
Q-Flow5.60 (4)7.70 (10)6.65 (5)
QAM-F6.90 (5)6.50 (5)6.70 (7)
QAM7.60 (9)6.85 (6)7.22 (8)
TRQAM8.70 (11)7.65 (9)8.18 (9)
ReBRAC8.35 (10)8.25 (11)8.30 (10)
FEdit10.60 (14)6.85 (6)8.72 (11)
QSM7.40 (8)10.95 (13)9.18 (12)
DSRL9.95 (13)10.15 (12)10.05 (13)
IFQL9.90 (12)12.75 (14)11.32 (14)

PReFlow ranks first in both phases across all 14 methods, including QPILOTS-M, Q-Flow, RQL, and the QAM variants.

2. Comparison using common QAM baselines

To put these gains in context, we apply the same average-rank analysis to the OGBench tables of recent methods, using the QAM variants as common reference baselines. Within their respective comparison sets, QPILOTS-M leads QAM-E in average rank by 2.20 offline and 0.90 online, and Q-Flow leads QAM-E by 0.60 offline, and RQL leads QAM-F by 1.30 offline. In our Table 1, PReFlow leads QAM-E by 2.20 offline and 1.55 online, and QAM-F by 1.95 offline. Our comparison also directly includes these recent methods such as QPILOTS-M, Q-Flow, and RQL already, and achieves the best average rank in both offline and online evaluation among all 14 methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.