acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning for Flow-Matching Policies with Density Transport

Abstract

We present an online reinforcement learning (RL) algorithm for fine-tuning flow-matching policies in continuous-control problems. Our key insight is to view RL-based policy improvement as a transport of action densities towards regions of high reward, which naturally aligns with the transport formulation of flow matching models. Prior methods either approximate the current or optimal policy distribution or resort to distillation, which introduces biased gradients or sacrifices multimodal modeling capacity. In contrast, our approach for RL with Density Transport, which we name RLDT, constructs a transport field from a maximum-entropy RL objective using Stein Variational Gradient Descent (SVGD). Then, it fine-tunes a pretrained flow matching policy to align with this field. Training with this alignment objective is nontrivial because flow-matching policies generate actions via a multi-step process, making direct gradient-based optimization challenging. To overcome this challenge and stabilize training, we approximate policy actions from intermediate denoising steps via expected-target estimation. This allows the transport-field update to propagate into the network parameters without unstable backpropagation through time. Experimental results demonstrate that RLDT outperforms competitive baselines in reward quality and convergence speed. This performance holds across diverse continuous-control tasks, encompassing both dense and sparse rewards, as well as state- and vision-based long-horizon robot manipulation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TwGXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RLDT, a new online reinforcement learning algorithm for fine-tuning flow-matching policies. Specifically, RLDT applies the SVGD in Soft Q-Learning to a multi-step flow strategy and provides a method for assigning SVGD update directions to each denoising step. The final training objective consists of three components: RL loss, consistency loss, and constraint loss. The experiment results indicate that RLDT achieves or outperforms the baseline method on three benchmarks OpenAI Gym, Furniture-Bench and Robomimic.

Critical strengths

The description of the methodology section in this paper is relatively detailed, the overall process is clear, and easy to understand.

Critical weaknesses
  1. In my opinion, the proposed RLDT mainly adapts the the SVGD-based policy update of Soft Q-Learning (Eq. (11), following Wang & Liu, 2016) to flow policies, so the overall novelty is quite limited.

  2. Some information in Section 4 are not clearly described. Please refer to the Questions for Authors for details.

Questions for authors
  1. In Figure 2, RLDT shows a clear performance drop at around 3M steps on Hopper and Walker2d tasks, and then recovers rapidly from the lowest reward. What is the reason for this phenomenon? Why does the similar issues not occurr on other tasks?

  2. According to Table 3, when λ2=0\lambda_2=0, the entire training crashes with 0 success rate. Could the authors explain the reason? In addition, the results on the task Lamp-Low are quite sensitive to λ1\lambda_1 and λ2\lambda_2 at the early training stage (45M). How did author choose these parameters? At the late training stage (215M), all setting achieve almost the same results. In particular, the λ1=0\lambda_1=0 setting (row 4) has the same final success rate as the default setting (row1). Does these results suggest that the consistency loss can be removed when training runs long enough? The authors are also encouraged to add more ablation results on other benchmarks to demonstrate the effectiveness of each component.

2 comments

In my opinion, the proposed RLDT mainly adapts the SVGD-based policy update of Soft Q-Learning (Eq. (11), following Wang & Liu, 2016) to flow policies, so the overall novelty is quite limited.

As the reviewer correctly points out, our work builds on SQL[Haarnoja et al. (2017)]. Similarly to the latter, our method amortizes the SVGD field into a network. However, the two methods differ in the way that they define the SVGD field. SQL uses a one-step sampler a=fϕ(ξ;s)a = f_\phi(\xi; s) whose output is the action, so the transport field can be applied to the output directly. This does not apply to a flow policy, where a network outputs a velocity at each of TT steps and the action is obtained only by integration. This creates two problems that do not arise for one-step samplers that predict actions:

(1) Supervising every denoising step through one-step prediction a†a^\dagger . We project each intermediate state onto the action manifold through the one-step prediction a†=aτ+(1−τ)vθa^\dagger = a_\tau + (1-\tau)v_\theta. This lets us evaluate the SVGD transport field at every step, avoiding both back-propagation through the sampler and log-likelihood estimation.

(2) Projecting an action update into a network update. The SVGD displacement ϵ ϕ(a†)\epsilon\,\phi(a^\dagger) lives in action space, but the network output vv is related to the approximate action after denoising a†a^\dagger through a map a†=g(v)a^\dagger = g(v), where g is the above-mentioned one-step prediction via Euler integration, with Jacobian J=∂a†/∂vJ = \partial a^\dagger/\partial v. Treating vv as a free variable, there are two ways to map the update back to the network output.

Gradient formulation (naive application of SVGD in SQL). Set ∂Jπ/∂a†∝ϕ\partial \mathcal{J}_\pi/\partial a^\dagger \propto \phi and apply the chain rule, we get Δv=ϵJ⊤ϕ\Delta v = \epsilon J^\top\phi. The realized displacement is then

a~†−a†≈JΔv=ϵJJ⊤ϕ(a†)\tilde a^\dagger - a^\dagger \approx J\Delta v = \epsilon J J^\top \phi(a^\dagger)

which is a distorted field not equivalent to the required SVGD field ϵϕ\epsilon \phi .

Transport formulation (ours). Differentiating a†=g(v)a^\dagger = g(v) gives Δa†=JΔv\Delta a^\dagger = J\Delta v to first order. We then set the transport displacement Δa†=ϵϕ(a†)\Delta a^\dagger = \epsilon\phi(a^\dagger), by definition of T=id+ϵϕT = \mathrm{id} + \epsilon\phi, to yield JΔv=ϵϕ(a†)J\Delta v = \epsilon\phi(a^\dagger). We can then solve for Δv\Delta v:

Δv=ϵJ+ϕ(a†)=ϵJ⊤(JJ⊤)−1ϕ(a†).\Delta v = \epsilon J^{+}\phi(a^\dagger) = \epsilon J^\top(JJ^\top)^{-1}\phi(a^\dagger).

The realized displacement is now consistent with the SVGD field by design, since a~†−a†≈JΔv=ϵϕ\tilde a^\dagger - a^\dagger \approx J\Delta v = \epsilon\phi.

The gradient formulation operates on a distorted field, which leads to noise in the update signals and therefore slower learning. We provide a more extensive analysis and empirical comparisons in Fig. 6, where we show that using the SVGD update empirically leads to faster learning. Specifically, given the same 225M environment steps in one-leg med and lamp med settings, Transport formulation achieves 63.7% (vs. Gradient 31.7%) and 38.5% success rate (vs. Gradient 6.2%). For comparable performance, the Transport formulation uses only 99M environment steps, less than half of the Gradient formulation.

Overall, our method is novel because of two main technical innovations: (1) training multiple-step flow-matching policies without restrictions on the policy design; and (2) ensuring the network’s updates are consistent with the SVGD field by design, removing the distortions introduced by a SQL-style application of SVGD to flow-matching policies. We have clarified the novelty of our technical contribution in the manuscript’s introduction (Line 77, 86) and related work (Line 134), and experiments (Line 499).

In Figure 2, RLDT shows a clear performance drop at around 3M steps on Hopper and Walker2d tasks, and then recovers rapidly from the lowest reward. What is the reason for this phenomenon? Why does the similar issues not occurr on other tasks?

We have run additional experiments to study the causes behind the dip observed in the Hopper and Walker2d. Our analysis indicates that the critic update is the primary underlying factor. Our Gym experiments used a one-step target (λ=0\lambda=0), whereas FurnitureBench and Robomimic used a λ\lambda-return with λ=0.5\lambda=0.5. With λ=0\lambda=0, reward information propagates one step per update, so during the initial training iterations the critic’s action gradient is poorly informed and the actor follows it into a lower-reward region before the critic catches up. Setting λ=0.5\lambda=0.5 on Gym, as in the other benchmarks, removes the dip (Figure 2). No other hyperparameter was changed. We have used λ=0.5\lambda=0.5 uniformly across all benchmarks in the revision.

According to Table 3, when λ2=0\lambda_2=0, the entire training crashes with 0 success rate. Could the authors explain the reason?

At the beginning of training, the critic cannot provide accurate information of Q gradient in sparse reward setting. If the constraint loss is removed, the policy will change drastically and exploit the wrong Q, leading to worse performance. Since we use sparse rewards, once the actor collapses to 0 success rate, it no longer provides useful information for the critic and it stays at 0.

How did author choose these parameters?

We tune the parameter λ1,λ2\lambda_1, \lambda_2 on the lamp-low task and choose the best for the remaining tasks. We use the same setting for all the tasks. This suggests the RLDT is robust to the parameter setting. We find that tuning parameter could lead to only minor performance improvement for certain tasks (Fig. 9).

Does these results suggest that the consistency loss can be removed when training runs long enough?

Consistency loss only helps with learning efficiency. This is because the SVGD field does not require the particles to be close to the sampling distribution. As is shown in Fig. 9, non-zero consistency loss weight λ1\lambda_1 only helps for faster learning for the Lamp-low task, when the initial success rate is low. For the Robomimic Square and Transport task , given a better initial success rate, removing consistency loss does not influence the learning. We have clarified this in the manuscript (LL 967).

Reviewer_By7cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reinforcement Learning with Density Transport (RLDT), a novel off-policy online RL algorithm for fine-tuning flow matching policies. Its key insight is to formulate policy improvement as transporting the original action distribution towards a maximum-entropy target distribution guided by high Q-values. Specifically, RLDT employs Stein Variational Gradient Descent (SVGD), combining kernel-based smoothing with critic gradient guidance, to construct a transport field on the action manifold. It then applies this field to intermediate ODE denoising steps using expected-posterior action estimates, yielding a local velocity-alignment objective that avoids costly and potentially unstable backpropagation through the entire denoising chain. Experiments on dense reward continuous control, sparse reward state-based manipulation, and sparse reward vision-based manipulation tasks show that RLDT achieves performance comparable to or better than competing baselines.

Critical strengths
  1. RLDT presents a well-motivated adaptation of an established value-based policy optimization framework to fine-tune flow matching policies. The maximum-entropy objective, soft Q-learning formulation, Boltzmann target distribution, and SVGD-based distribution matching are established techniques from prior work. The main contribution lies in adapting this framework to the multi-step ODE generation process of flow matching policies. Specifically, RLDT uses expected-posterior action estimates at intermediate denoising steps to evaluate the transport field and derives a local velocity-alignment objective, supplemented by consistency and constraint losses. This construction offers a potentially useful way to transfer value-guided transport updates to flow matching policies without explicitly evaluating the policy likelihood or backpropagating through the entire denoising chain.
Critical weaknesses
  1. Limited novelty and an unverified theoretical assumption underlying the proposed adaptation. Sections 3.1-3.2 largely build on established techniques, including standard flow matching, the maximum-entropy RL and soft Q-iteration framework of Haarnoja et al. (2017), and KL-based policy optimization via SVGD (Liu & Wang, 2016). The expected-posterior identity in Eq. (7) is also drawn from prior work. Thus, the main methodological contribution lies in Section 3.3, which adapts these existing techniques to intermediate steps of the flow matching policies. However, Eq. (7)'s interpretation of aτ†=aτ+(1−τ)vθ(aτ,τ∣s)a^\dagger_\tau=a_\tau+(1-\tau)v_\theta(a_\tau,\tau\mid s) as the conditional expectation of the terminal action relies on vθv_\theta being the marginal velocity field associated with the prescribed linear interpolation path. RLDT updates vθv_\theta through the combined objective in Eq. (15), but the paper does not establish that the updated velocity field continues to satisfy this correspondence after successive RL fine-tuning iterations. Nor does it empirically measure the discrepancy between the estimated action aτ†a^\dagger_\tau and the corresponding conditional expectation during training. The consistency loss in Eq. (13) encourages agreement between estimated and sampled terminal actions, but does not guarantee that the conditional-expectation identity in Eq. (7) remains valid. Consequently, the theoretical justification for using this estimate to evaluate the SVGD transport field at intermediate denoising steps remains incomplete.

  2. Unexplained decline-and-surge behavior and insufficient empirical evidence. Figure 2 shows pronounced, abrupt changes in RLDT reward on Hopper and Walker2d. On Hopper, the return drops from approximately 1.5k to below 1k around 2-3M environment steps, then rises to approximately 3.5k near 5M steps and around 4k near 7M steps. On Walker2d, it falls to approximately 2.7k around 3M steps before rapidly jumping to nearly 5.9k around 5M steps, overtaking the competing baselines. Other methods on the same tasks show no comparable collapse-and-surge pattern, while similar transitions recur for all three ensemble sizes (K=4,8,16K=4,8,16) in Figure 7. These learning dynamics remain unexplained. The averaged curves alone do not establish whether the pattern reflects training instability. In addition, the gains on manipulation tasks are mixed: RLDT is comparable to DPPO on RoboMimic Transport, FPO++ is competitive on Square, and FurnitureBench is evaluated only against DPPO. Representative manipulation rollouts, images, or videos would help assess the actual task execution behind the reported metrics.

Questions for authors
  1. Under what assumptions does the conditional expectation interpretation of Eq. (7) remain valid after the velocity field is updated through Eq. (15)? Can the authors theoretically justify that this property is preserved during RL fine-tuning?

  2. If this property cannot be guaranteed theoretically, can the authors directly measure the discrepancy between aτ†a^\dagger_\tau and the corresponding conditional mean during training? Does this discrepancy accumulate during fine-tuning, and how does it affect the SVGD updates?

  3. What explains the repeated reward declines and abrupt improvements on Hopper and Walker2d in Figure 2 and Figure 7? How can these phenomena be reconciled with the paper's claims of stability of policy optimization?

  4. Given the mixed results on RoboMimic and the absence of comparisons beyond DPPO on FurnitureBench, why are the other baselines omitted from FurnitureBench? And can the authors provide a clear comparability rationale alongside representative manipulation rollouts or videos to substantiate the claimed performance gains?

4 comments

Under what assumptions does the conditional expectation interpretation of Eq. (7) remain valid after the velocity field is updated through Eq. (15)? Can the authors theoretically justify that this property is preserved during RL fine-tuning?

If this property cannot be guaranteed theoretically, can the authors directly measure the discrepancy between aτ†a^\dagger_\tau and the corresponding conditional mean during training? Does this discrepancy accumulate during fine-tuning, and how does it affect the SVGD updates?

We thank the reviewer for pointing this out. We agree with the reviewer that after an RL update, vθv_\theta is no longer the linear-path field for the independent linear coupling used in pre-training. For RLDT, the coupling is constructed from the ODE trajectory as aτ=ψ0→τ(a0)a_\tau = \psi_{0 \rightarrow \tau}(a_0) instead of the linear coupling aτ=τa1+(1−τ)a0a_\tau = \tau a_1 + (1 - \tau) a_0 .

However, our derivation in Eq. (8)–(12) does not assume linearity because the SVGD field does not require the particles to be close to the sampling distribution, since it evaluates the gradient of the Q network on final samples. For simplicity, we use one-step Euler integration at,τ†:=at,τ+(1−τ) vθ(at,τ,τ)a^\dagger_{t,\tau}:= a_{t,\tau} + (1-\tau)\,v_\theta(a_{t,\tau},\tau) to convert network output to actions, following Back to Basics (Tianhong et al., 2026). Empirically, this leads to good performance and decreasing consistency loss as the training progresses. To measure it, as suggested by the reviewer, we measure ∥aτ†−ψτ→1(aτ)∥\Vert a_\tau^\dagger - \psi_{\tau \rightarrow 1} (a_\tau) \Vert directly throughout the training in Robomimic. The result in Fig. 10 shows an increased consistency as the policy improves. Therefore, for simplicity, we avoid using higher-order approximations for $a^\dagger_{t,\tau} $. We have added this clarification to the manuscript (LL 253).

What explains the repeated reward declines and abrupt improvements on Hopper and Walker2d in Figure 2 and Figure 7? How can these phenomena be reconciled with the paper's claims of stability of policy optimization?

We have run additional experiments to study the causes behind the dip observed in the Hopper and Walker2d. Our analysis indicates that critic update is the primary underlying factor. Our Gym experiments used a one-step target (λ=0\lambda=0), whereas FurnitureBench and Robomimic used a λ\lambda-return with λ=0.5\lambda=0.5. With λ=0\lambda=0, reward information propagates one step per update, so during the initial training iterations the critic’s action gradient is poorly informed and the actor follows it into a lower-reward region before the critic catches up. Setting λ=0.5\lambda=0.5 on Gym, as in the other benchmarks, removes the dip (Figure 2). No other hyper-parameter was changed. We have used λ=0.5\lambda=0.5 uniformly across all benchmarks in the revision.

Given the mixed results on RoboMimic and the absence of comparisons beyond DPPO on FurnitureBench, why are the other baselines omitted from FurnitureBench? And can the authors provide a clear comparability rationale alongside representative manipulation rollouts or videos to substantiate the claimed performance gains?

We include all baseline methods for FurnitureBench in the revision. We show videos of rollouts in the Lamp-Med task in the FurnitureBench tasks. Among the three stages of picking up, inserting the bulb, and placing the hood, we find RLDT shows better reliability and robustness given the setup randomness. We included rendered videos in the supplementary materials for this.

  • Pick up: Given the medium random initialization, the pre-trained policy, ReinFlow, FPO, and QAM fail to pick up the bulb in 42–45% of episodes. RLDT only fails to pick up in 6% of episodes.
  • Insertion: once the bulb reaches the socket, RLDT threads it 81% of the time, against 25% for the pre-trained policy. It also reaches the socket at about step 182, against 267–302 for the others.
  • Hood: once the bulb is threaded, RLDT places the hood 92% of the time, against 42% for DPPO.
AuthorsOpenReview ↗

Novelty Explanation

As the reviewer correctly points out, our work builds on SQL[Haarnoja et al. (2017)]. Similarly to the latter, our method amortizes the SVGD field into a network. However, the two methods differ in the way that they define the SVGD field. SQL uses a one-step sampler a=fϕ(ξ;s)a = f_\phi(\xi; s) whose output is the action, so the transport field can be applied to the output directly. This does not apply to a flow policy, where a network outputs a velocity at each of TT steps and the action is obtained only by integration. This creates two problems that do not arise for one-step samplers that predict actions:

(1) Supervising every denoising step through one-step prediction a†a^\dagger . We project each intermediate state onto the action manifold through the one-step prediction a†=aτ+(1−τ)vθa^\dagger = a_\tau + (1-\tau)v_\theta. This lets us evaluate the SVGD transport field at every step, avoiding both back-propagation through the sampler and log-likelihood estimation.

(2) Projecting an action update into a network update. The SVGD displacement ϵ ϕ(a†)\epsilon\,\phi(a^\dagger) lives in action space, but the network output vv is related to the approximate action after denoising a†a^\dagger through a map a†=g(v)a^\dagger = g(v), where g is the above-mentioned one-step prediction via Euler integration, with Jacobian J=∂a†/∂vJ = \partial a^\dagger/\partial v. Treating vv as a free variable, there are two ways to map the update back to the network output.

Gradient formulation (naive application of SVGD in SQL). Set ∂Jπ/∂a†∝ϕ\partial \mathcal{J}_\pi/\partial a^\dagger \propto \phi and apply the chain rule, we get Δv=ϵJ⊤ϕ\Delta v = \epsilon J^\top\phi. The realized displacement is then

a~†−a†≈JΔv=ϵJJ⊤ϕ(a†)\tilde a^\dagger - a^\dagger \approx J\Delta v = \epsilon J J^\top \phi(a^\dagger)

which is a distorted field not equivalent to the required SVGD field ϵϕ\epsilon \phi .

Transport formulation (ours). Differentiating a†=g(v)a^\dagger = g(v) gives Δa†=JΔv\Delta a^\dagger = J\Delta v to first order. We then set the transport displacement Δa†=ϵϕ(a†)\Delta a^\dagger = \epsilon\phi(a^\dagger), by definition of T=id+ϵϕT = \mathrm{id} + \epsilon\phi, to yield JΔv=ϵϕ(a†)J\Delta v = \epsilon\phi(a^\dagger). We can then solve for Δv\Delta v:

Δv=ϵJ+ϕ(a†)=ϵJ⊤(JJ⊤)−1ϕ(a†).\Delta v = \epsilon J^{+}\phi(a^\dagger) = \epsilon J^\top(JJ^\top)^{-1}\phi(a^\dagger).

The realized displacement is now consistent with the SVGD field by design, since a~†−a†≈JΔv=ϵϕ\tilde a^\dagger - a^\dagger \approx J\Delta v = \epsilon\phi.

The gradient formulation operates on a distorted field, which leads to noise in the update signals and therefore slower learning. We provide a more extensive analysis and empirical comparisons in Fig. 6, where we show that using the SVGD update empirically leads to faster learning. Specifically, given the same 225M environment steps in one-leg med and lamp med setting, Transport formulation achieves 63.7% (vs. Gradient 31.7%) and 38.5% success rate (vs. Gradient 6.2%). For comparable performance, the Transport formulation uses only 99M environment steps, less than half of the Gradient formulation.

Overall, our method is novel because of two main technical innovations: (1) training multiple-step flow-matching policies without restrictions on the policy design; and (2) ensuring the network’s updates are consistent with the SVGD field by design, removing the distortions introduced by a SQL-style application of SVGD to flow-matching policies. We have clarified the novelty of our technical contribution in the manuscript’s introduction (Line 77, 86) and related work (Line 134), and experiments (Line 499).

Thank you for the rebuttal and the additional experiments. Most of my main concerns have been addressed. Although the linear velocity field assumption cannot be strictly guaranteed in theory during RL fine-tuning, the new empirical tracking of the consistency distance in Fig. 10 shows that the approximation error remains well-bounded and benign in practice. The explanation regarding the critic λ\lambda-return setting (λ=0.5\lambda=0.5 vs. λ=0\lambda=0) is also convincing, and the updated learning curves on Hopper and Walker2d look much more stable (along with the added baselines on FurnitureBench).

One minor point I still find slightly overstated is the "pseudoinverse / distorted field" explanation in the novelty discussion (Eq. 16–17). Since the one-step Euler map a†=aτ+(1−τ)va^\dagger = a_\tau + (1-\tau)v simply has an isotropic Jacobian J=(1−τ)IJ = (1-\tau)I, the pseudoinverse J+J^+ is really just a scalar rescaling by 1/(1−τ)1/(1-\tau) across denoising steps rather than correcting a directional distortion of the vector field (and the step from vv to network parameters θ\theta in Eq. 11 still uses standard gradient backpropagation). That said, Fig. 6 clearly shows this timestep rescaling matters empirically. Overall, I am satisfied with the revisions and will raise my score from 2 to 3.

We thank the reviewer for pointing this out and for letting us better explain our method.

The pseudo-inverse is really just a scalar rescaling of the denoising steps rather than correcting a directional distortion of the vector field.

We agree with the reviewer that the Transport formulation corrects the scale instead of the direction. However, the scale correction is still nontrivial in this case since it enforces consistency between different intermediate steps. The gradient formulation shows increasing consistency distance without scale correction in Figure 6. Such inconsistency leads to slower improvement on the FurnitureBench tasks.

The step from v to network parameters theta in Eq. 11 still uses standard gradient back-propagation.

We agree with the reviewer that for a single network parameter update step, it will still lead to a distorted field since we use −∂θvθ⊤ϵϕ/(1−τ)-\partial_\theta v_\theta^\top \epsilon \phi/(1 - \tau) . However, in practice, we can do multiple gradient steps for the network parameter to regress to the same target such that the network output can align with the SVGD updates. It is easy to show that −∂θvθ⊤ϵϕ/(1−τ)-\partial_\theta v_\theta^\top \epsilon \phi/(1 - \tau) is the gradient for the following loss L(θ)\mathcal{L}(\theta) :

L(θ)=12∥v(at,τ,st;θ0+θ)−(v0+ϵϕ1−τ)∥2v(at,τ,st;θ0)=v0\mathcal{L}(\theta) = \frac{1}{2} \left \Vert v(a_{t, \tau}, s_t;\theta_0+\theta) - (v_0 + \frac{\epsilon \phi}{1 - \tau}) \right \Vert^2\quad v(a_{t,\tau}, s_t;\theta_0) = v_0

Reviewer_mPy3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Reinforcement Learning with Density Transport (RLDT), an off-policy RL algorithm for fine-tuning pretrained flow-matching policies in continuous control. Instead of estimating intractable flow log-likelihoods or backpropagating through the multi-step ODE denoising chain, RLDT constructs an action-space transport vector field using Stein Variational Gradient Descent (SVGD) under a maximum-entropy RL objective. To update intermediate denoising steps, RLDT maps intermediate noisy actions to the action manifold via a one-step Euler lookahead and aligns the velocity network with the transport direction, regularized by a trajectory consistency loss and a proximal velocity-constraint loss. The method is evaluated on Gym locomotion, state-based FurnitureBench, and vision-based Robomimic manipulation benchmarks.

Critical strengths
  • The paper is well motivated, and the challenge of bypassing exact log-likelihood estimation is clearly presented.
  • Framing flow-matching policy improvement as action-space density transport via SVGD is intuitive.
Critical weaknesses
  • Novelty is limited as constructing a transport field toward a soft Q-target via amortized SVGD closely follows Soft Q-Learning (Haarnoja et al., 2017).
  • While the paper claims that few-step distillation constrains policy expressivity, it lacks empirical evaluation against one-step distillation methods such as FQL (Park et al., 2025) or flow-map policies (Ziakas et al., 2026).
  • There is no analysis of the early-training instability indicated by the sharp performance drop early in training (Figure 2).
  • There is no analysis of critic initialization and warm-up.

References:

  • Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. "Reinforcement learning with deep energy-based policies." ICML 2017.
  • Seohong Park, Qiyang Li, and Sergey Levine. "Flow Q-learning." ICML 2025.
  • Christos Ziakas, Alessandra Russo, and Avishek Joey Bose. "Aligning flow map policies with optimal Q-guidance." NeurIPS 2026.
Questions for authors
  • Can the authors provide a clearer explanation of the core algorithmic differences between RLDT, Soft Q-Learning (Haarnoja et al., 2017), and flow-map Q-guidance (FMQ, Ziakas et al., 2026)?
  • Can the authors evaluate or compare their approach with one-step policies?
  • Can the authors explain the early-training performance drop on Hopper and Walker2d in Figure 2? Is it related to critic initialization?
3 comments

Can the authors provide a clearer explanation of the core algorithmic differences between RLDT, Soft Q-Learning (Haarnoja et al., 2017), and flow-map Q-guidance (FMQ, Ziakas et al., 2026)?

As the reviewer correctly points out, our work builds on SQL[Haarnoja et al. (2017)]. Similarly to the latter, our method amortizes the SVGD field into a network. However, the two methods differ in the way that they define the SVGD field. SQL uses a one-step sampler a=fϕ(ξ;s)a = f_\phi(\xi; s) whose output is the action, so the transport field can be applied to the output directly. This does not apply to a flow policy, where a network outputs a velocity at each of TT steps and the action is obtained only by integration. This creates two problems that do not arise for one-step samplers that predict actions:

(1) Supervising every denoising step through one-step prediction a†a^\dagger . We project each intermediate state onto the action manifold through the one-step prediction a†=aτ+(1−τ)vθa^\dagger = a_\tau + (1-\tau)v_\theta. This lets us evaluate the SVGD transport field at every step, avoiding both back-propagation through the sampler and log-likelihood estimation.

Similarly to us, FMQ supervises intermediate sampling steps. However, they do so via the average action velocity field uu. Extracting uu is computationally expensive for traditional flow-policies, and therefore FMQ is restricted to operate only with mean-flow policies, which produce uu by design. RLDT removes this restriction which, in practice, lead to comparable or higher performance (see quantitative results below).

(2) Projecting an action update into a network update. The SVGD displacement ϵ ϕ(a†)\epsilon\,\phi(a^\dagger) lives in action space, but the network output vv is related to the approximate action after denoising a†a^\dagger through a map a†=g(v)a^\dagger = g(v), where g is abovementioned one-step prediction via Euler integration, with Jacobian J=∂a†/∂vJ = \partial a^\dagger/\partial v. Treating vv as a free variable, there are two ways to map the update back to the network output.

Gradient formulation (naive application of SVGD in SQL). Set ∂Jπ/∂a†∝ϕ\partial \mathcal{J}_\pi/\partial a^\dagger \propto \phi and apply the chain rule, we get Δv=ϵJ⊤ϕ\Delta v = \epsilon J^\top\phi. The realized displacement is then

a~†−a†≈JΔv=ϵJJ⊤ϕ(a†)\tilde a^\dagger - a^\dagger \approx J\Delta v = \epsilon J J^\top \phi(a^\dagger)

which is a distorted field not equivalent to the required SVGD field ϵϕ\epsilon \phi .

Transport formulation (ours). Differentiating a†=g(v)a^\dagger = g(v) gives Δa†=JΔv\Delta a^\dagger = J\Delta v to first order. We then set the transport displacement Δa†=ϵϕ(a†)\Delta a^\dagger = \epsilon\phi(a^\dagger), by definition of T=id+ϵϕT = \mathrm{id} + \epsilon\phi, to yield JΔv=ϵϕ(a†)J\Delta v = \epsilon\phi(a^\dagger). We can then solve for Δv\Delta v:

Δv=ϵJ+ϕ(a†)=ϵJ⊤(JJ⊤)−1ϕ(a†).\Delta v = \epsilon J^{+}\phi(a^\dagger) = \epsilon J^\top(JJ^\top)^{-1}\phi(a^\dagger).

The realized displacement is now consistent with the SVGD field by design, since a~†−a†≈JΔv=ϵϕ\tilde a^\dagger - a^\dagger \approx J\Delta v = \epsilon\phi.

The gradient formulation operates on a distorted field, which leads to noise in the update signals and therefore slower learning. We provide a more extensive analysis and empirical comparisons in Fig. 6, where we show that using the SVGD update empirically leads to faster learning. Specifically, given the same 225M environment steps in one-leg med and lamp med setting, Transport formulation achieves 63.7% (vs. Gradient 31.7%) and 38.5% success rate (vs. Gradient 6.2%). For comparable performance, the Transport formulation uses only 99M environment steps, less than half of the Gradient formulation.

Overall, our method is novel because of two main technical innovations: (1) training multiple-step flow-matching policies without restrictions on the policy design (as in FMQ); and (2) ensuring the network’s updates are consistent with the SVGD field by design, removing the distortions introduced by a SQL-style application of SVGD to flow-matching policies. We have clarified the novelty of our technical contribution in the manuscript’s introduction (Line 77, 86) and related work (Line 134), and experiments (Line 499).

Can the authors evaluate or compare their approach with one-step policies?

We compare with the one-step policies (FQL, FMQ) as suggested by the reviewer on the Gym benchmark.

Since FMQ is restricted to mean-flow policies, we pretrain a model for our tasks following their paper’s recipe using the same dataset. We then fine-tune this model with the FMQ algorithm.

For FQL, we use the same base flow matching checkpoint as RLDT. The one-step actor has the same hidden state dimension as the flow policy. For parameter selection, we try three values of alpha, {1, 10, 100}, and find that 10 works the best.

The result is summarized below. We find that FQL could improve over the Flow Matching base policy but still underperforms RLDT. This is because the BC loss regularizes the one-step policy near the base policy, constraining further optimization. Similarly, FMQ could improve from its base mean flow policy. It matches RLDT’s performance on HalfCheetah but underperforms on Hopper and Walker.

HalfCheetahHopperWalker
Flow Matching (Base Policy)4579 ± 771744 ± 103175 ± 30
Flow Map (Base Policy)5043 ± 201008 ± 123014 ± 5
FQL5108 ± 153241 ± 114067 ± 90
FMQ (Ziakas et al., 2026)6287 ± 672896 ± 1894733 ± 221
RLDT6278 ± 803896 ± 1505869 ± 125

Can the authors explain the early-training performance drop on Hopper and Walker2d in Figure 2? Is it related to critic initialization?

We have run additional experiments to study the causes behind the dip observed in Hopper and Walker2d. Our analysis indicates that critic update is the primary underlying factor. Our Gym experiments used a one-step target (λ=0\lambda=0), whereas FurnitureBench and Robomimic used a λ\lambda-return with λ=0.5\lambda=0.5. With λ=0\lambda=0, reward information propagates one step per update, so during the initial training iterations the critic’s action gradient is poorly informed and the actor follows it into a lower-reward region before the critic catches up. Setting λ=0.5\lambda=0.5 on Gym, as in the other benchmarks, removes the dip (Figure 2). No other hyperparameter was changed. We have used λ=0.5\lambda=0.5 uniformly across all benchmarks in the revision.

Reviewer_mPy3OpenReview ↗

Response to Authors

Thanks for your detailed rebuttal. Most of my concerns have been resolved. I will keep my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Meta Response from authors

We thank all the reviewers for their constructive feedback. To address the concerns, we have made a series of major revisions to the paper summarized below (the section and figure numbering are consistent with the revised version):

[Experiment] We find that the cause for early training performance drop for Gym environemtn is in λ\lambda-Return setting. We change to λ=0.5\lambda=0.5 for Gym, which is the same for all the rest of tasks and update the results. We also include one-step policy baselines for the Gym benchmark.

[Experiment] We include all baseline methods for the FurnitureBench tasks and upload video to show qualitative results for the Lamp-Med task.

[Experiment] We compare our method and a naive application of SVGD on flow matching policy in Sec 4.2 Q4 to show the novelty in our method.

[Experiment] We add an additional ablation study of the consistency loss weight λ1\lambda_1 and constraint loss weight λ2\lambda_2. We find λ1\lambda_1 only helps with learning efficiency when the initial success rate is low.

[Presentation] In Sec. 1 and Sec 4.2 Q4, we explain the novelty of RLDT and the difference from SQL (Haarnoja et al., 2017)

Please see the attached revised paper with the changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.