- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PReFlow, which is a policy extraction method that applies additive action deltas using conditional flows. To my understanding, the method samples candidate actions from a pretrained behavior policy, and chooses the candidate with the highest critic value. Then, it uses a conditional flow to refine the action such that the refinement distribution is KL-regularized toward some proposal-centered Gaussian reference. The method is evaluated on a wide array of tasks in OGBench and additional analyses on the method such as computational efficiency.
Critical strengths
- Training is efficient under the proposal conditioned refinement flow, and reduced training time supported in results section (Figure 4).
- Very extensive empirical evaluation (50 OGBench tasks).
Critical weaknesses
- I am not convinced why proposal-conditioned refinement is preferable to directly optimizing an expressive flow policy. The motivation of selecting high-Q actions while avoiding out-of-distribution actions is closely related to existing work on flow-based offline RL, particularly ReFORM [1], which aims to improve a flow policy while preserving the learned behavior-policy support. However, I do not see a discussion or comparison with this work, nor sufficient justification for why action-space refinement with a Gaussian reference is preferable to support-constrained policy improvement. While the paper compares against QAM, it is unclear whether the improvements arise from the refinement architecture itself or from the different regularization and optimization procedures.
- The motivating comparison between Gaussian and flow-based refinement largely demonstrates the known limitation of unimodal Gaussian distributions. Since existing flow policies are already capable of representing multimodal action distributions, I do not find this comparison sufficient to establish the necessity of a separate refinement flow. Furthermore, the theoretical proposal-selection objective depends on the post-refinement value, whereas the practical algorithm uses greedy selection based on the original critic value. It is unclear how much this approximation affects performance, especially given that sampling a finite number of behavior-policy actions does not guarantee coverage of different action modes.
[1] Zhang, S., So, O., Ahmad, H. M. S., Yu, E. Y., Cleaveland, M., Black, M., & Fan, C. (2026). ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation. The Fourteenth International Conference on Learning Representations.
Questions for authors
- How does PReFlow relate to support-constrained policy improvement methods such as ReFORM [1]? In particular, what are the advantages of allowing local action-space refinements beyond the behavior-policy support, compared with improving the policy while preserving its learned support?
- Why is proposal-conditioned action-space refinement necessary or preferable to directly learning an expressive flow policy? Could the authors provide more controlled comparisons that isolate the benefits of the refinement architecture from differences in regularization and optimization?
4 comments
Response to Q2: Controlled Comparisons of Proposal-Conditioned Refinement (2/2)
Q2: Controlled Comparisons of Proposal-Conditioned Refinement
We do not claim that directly learning an expressive flow policy is wrong. It is a valid alternative, and QAM is one instance of it. Our focus is the narrower problem of reaching high-value actions where the behavior density is low, which can be viewed as sampling from the tilted density . When the whole policy is fine-tuned directly, it has to move probability mass across the action space to reach these low-density regions. Conditioning on the proposal makes the problem local. For each proposal, the refinement only needs to tilt a Gaussian neighborhood toward higher , and this conditional target admits a closed-form training objective.
We compare the two formulations through the toy experiment in Fig. 2 and an additional OGBench comparison with aligned regularization.
The toy experiment in Fig. 2 uses a fixed proposal, and both the refinement flow and the behavior flow fine-tuned with QAM are trained toward the same tilted target with the same network and training budget. The refinement recovers all three modes (KL 0.09), while the directly fine-tuned flow concentrates most of its mass on a single mode (KL 0.53).
For the OGBench comparison, we train QAM’s behavior flow on Gaussian-perturbed actions and fine-tune it with the same and as PReFlow. Both methods then share the reference and the temperature, and the remaining difference is the proposal-conditioned formulation together with the closed-form training procedure it enables.
Table R.8. Method comparison with the same marginal distribution (3 seeds).
| Method | al () | hm () | scene () | c2 () | Avg |
|---|---|---|---|---|---|
| PReFlow | 79.0 | 69.2 | 73.1 | 53.9 | 68.8 |
| QAM, BC on | 16.8 | 0.0 | 54.1 | 33.3 | 26.1 |
| QAM, BC on | 16.4 | 0.3 | 56.8 | 42.8 | 29.1 |
As shown in Table R.8, PReFlow achieves higher mean success rates. We hypothesize that this separation facilitates optimization by allowing the refinement model to learn critic-guided improvements conditioned on each proposal. This interpretation is consistent with the more accurate mode recovery observed in our toy experiment under an exactly matched target distribution.
PReFlow also selects among proposals, so we further check whether selection alone explains its advantage. We add the same best-of- selection to QAM-E (, at acting time only) on the six domains where PReFlow uses .
Table R.9. QAM-E with best-of- selection. PReFlow results are from Table 1.
| Method | hm | hl | scene | p33 | c2 | c4 | Avg |
|---|---|---|---|---|---|---|---|
| QAM-E () | 55 → 82 | 3 → 26 | 96 → 100 | 100 → 100 | 61 → 100 | 6 → 80 | 53.5 → 81.3 |
| QAM-E () | 57.9 → 89 | 1 → 34 | 98 → 99 | 100 → 100 | 61 → 100 | 4 → 80 | 53.65 → 83.7 |
| PReFlow | 65 → 95 | 24 → 65 | 97 → 99 | 100 → 100 | 59 → 100 | 15 → 80 | 60.0 → 89.8 |
As shown in Table R.9, selection leaves QAM-E’s offline success unchanged and raises its online success only slightly. Both scores remain below those of PReFlow.
Thank you for the detailed response and new experimental results.
W1/Q1. Largely resolved. The additional ReFORM comparison is valuable and demonstrates promising advantages for PReFlow. Please mention how ReFORM was adapted for this action chunking version, and what hyperparameters were used.
W3. Resolved. Tables R.2 and R.3 and the dip test show the benefit of flow refinement persists with sampled proposals and against a full-covariance Gaussian editor.
W4. Resolved by Table R.1.
I share the other reviewers’ views that the aggregate gains over the strongest baselines are modest and concentrated in puzzle-4x4. Still, my main concerns were addressed. I raise to 3.
We sincerely thank the reviewer for their thoughtful feedback and for reconsidering the score. We are glad that our additional experiments and clarifications helped address the reviewer's main concerns. We will include further details on how ReFORM was adapted to the action chunking setting, along with the hyperparameters used, in the revised manuscript.
We greatly appreciate the reviewer's constructive suggestions and support of our work.
Response to W1–W2 and Q1: ReFORM Comparison and Refinement Design (1/2)
W1/Q1: Comparison with ReFORM
We thank the reviewer for highlighting ReFORM and will add the citation and discussion. ReFORM$^{[1]}$ learns a reflected-flow noise generator that maximizes the critic value of the resulting actions while keeping the generated noise inside the bounded source domain of a learned behavior flow. This preserves the learned behavior support while allowing probability to concentrate on high-value actions within it.
PReFlow’s proposal-conditioned formulation provides direct control over action changes while retaining an expressive refinement distribution. Each sampled behavior action serves as the center of a Gaussian reference, and a conditional flow learns critic-guided refinements around that proposal. The Gaussian reference penalizes large action displacements, with the refinement scale δ and temperature τ controlling the trade-off between displacement and critic value. This gives PReFlow the flexibility to favor nearby actions with high critic values, including those excluded by a learned support constraint.
The same conditional Gaussian structure enables closed-form adjoint-matching targets, reducing refinement learning to a single velocity regression loss. These targets are constructed from sampled endpoints and endpoint critic gradients, eliminating the need to backpropagate through either the behavior or refinement rollout. ReFORM distills its behavior flow into a one-step model to reduce the backpropagation cost, but training its noise generator still requires backpropagation through the flow rollout. PReFlow therefore combines expressive action-space refinement with a training procedure that avoids this rollout backpropagation.
Furthermore, we evaluate ReFORM under the advanced QAM protocol. We port the official ReFORM implementation into the QAM codebase and evaluate it under exactly the same protocol, network architecture, and shared hyperparameters as PReFlow and all other baselines. Following the architecture of QAM, we apply action chunking with horizon 5 on manipulation tasks, which increases the radius of a uniform ball. ReFORM uses a single hyperparameter configuration across all domains, so we keep its default.
Table R.7. Method comparison on OGBench advanced setting (3 seeds).
As shown in Table R.7, ReFORM degrades sharply on long-horizon locomotion tasks. ReFORM performs critic-guided policy improvement by optimizing the source-noise distribution, while its behavior flow and one-step approximation are trained through cloning and distillation, respectively. For a given learned behavior mapping, noise optimization is restricted to actions reachable from its bounded latent domain and cannot recover a high-value action outside this reachable set. PReFlow instead refines each proposal directly in action space, so the critic can correct an imperfect behavior proposal through a local displacement controlled by δ and τ, rather than inheriting its errors.
W2: Refinement Expressivity and Proposal Selection
Thank you for the comments. The comparison in Table 2 is intended to isolate the refinement family under the same conditional objective. The comparison with directly fine-tuning a flow policy toward the same target is given in our response to Q2.
For the proposal selection, we replaced the critic score Q(s,ak) with a Monte-Carlo estimate of the post-refinement value Vδ(s,ak) using J=8 samples per proposal and observed no consistent difference, while the estimate requires J times more critic evaluations per proposal. Please see Table R.1 in our response to Reviewer K3Gn.
As for mode coverage, we want to emphasize that a single proposal can generate multiple final-action modes. Each proposal conditions an entire refinement distribution, and our fixed state, fixed proposal analysis provides direct empirical evidence of this behavior. Refinement with K=1 already substantially improves over selection among multiple unrefined proposals (Appendix E.4). Increasing K to 2 provides further gains (Figure 5).
References
[1] ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation. ICLR, 2026.