acceptodds
Under review as a conference paper at ICLR 2027

Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning

Abstract

Generative policies offer expressive action distributions for online reinforcement learning (RL), but many diffusion- and flow-based policies require costly iterative sampling. We propose DriftSpread, a one-step generative policy based on Drifting for online maximum-entropy (MaxEnt) RL. To incorporate entropy without a tractable policy density, we build on an energy-based view of Drifting and derive critic-guided Drift and entropy-induced Spread from the negative MaxEnt objective. Our method learns these updates sequentially in the same actor, using action regression for value improvement and distribution matching to capture the effect of entropy. The actor updates require neither direct samples from the critic-defined Boltzmann target nor policy-density or score estimation. Under exact fitting with a fixed critic, we prove convergence to the Boltzmann target in the small-step limit followed by the long-time limit. On MuJoCo, DriftSpread achieves the highest aggregate normalized return among all evaluated methods, while outperforming the diffusion- and flow-based baselines in both training and inference efficiency.

Primary area:
reinforcement learning
Keywords:
Online reinforcement learning, Generative policies, Drifting models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_g2fcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DriftSpread, a one-step generative policy learning method for online maximum entropy reinforcement learning based on the Drifting framework. Building on an energy-based formulation of maximum-entropy policy improvement, the method decomposes policy updates into critic-guided Drift and entropy-induced Spread. Drift updates the actor through regression toward actions displaced by critic gradients, while Spread uses characteristic-function-based maximum mean discrepancy (MMD) to match the entropy-induced target distribution without requiring policy density estimation. The paper also provides a theoretical analysis showing convergence to the critic-defined Boltzmann distribution under exact fitting and fixed-critic assumptions.

Critical strengths

S1 (Methodological contribution). The paper presents a meaningful extension of the Drifting framework to online maximum-entropy RL. In particular, decomposing policy improvement into critic-guided Drift and entropy-induced Spread provides a novel approach to learning expressive one-step generative policies.

S2 (Density-free entropy regularization). The proposed Spread objective provides a way to incorporate entropy regularization without explicitly estimating policy densities. The use of characteristic-function-based MMD is particularly interesting, as it enables distribution matching using generated samples.

S3 (Theoretical analysis). The paper establishes a theoretical connection between the Drift–Spread updates and convergence to the critic-defined Boltzmann distribution. Although the analysis relies on fixed-critic and exact-fitting assumptions, it provides useful theoretical justification for the proposed learning procedure.

S4 (Empirical performance and computational efficiency). The proposed method achieves strong aggregate performance on 5 MuJoCo tasks while maintaining efficient one-step action generation.

Critical weaknesses

W1 (Missing ablation of the entropy-induced Spread component). The paper demonstrates the effect of Spread on the learned distribution through ring-shaped toy experiments. However, the main MuJoCo experiments do not provide a direct comparison between Drift-only and the full DriftSpread method. Since Spread is a central component of the proposed approach, evaluating its contribution to online performance, action diversity, and exploration would help establish its practical necessity. Furthermore, sensitivity analyses of the entropy coefficient λ\lambda and evolution step size η\eta, which directly control the Drift–Spread updates, are missing.

W2 (Limited experimental diversity). The evaluation is limited to 5 MuJoCo locomotion tasks. Additional experiments on more challenging benchmarks, such as OGBench or Robomimic, would help demonstrate the effectiveness and generalizability of the proposed method in diverse manipulation and (semi-) sparse reward settings.

W3 (Sensitivity of distribution matching to frequency sampling). The proposed Spread objective depends on the frequency distribution used for characteristic-function matching. Although the Gaussian frequency distribution satisfies the full-support assumption in Theorem 4.1, the practical implementation relies on finitely sampled frequencies and actor samples. Figure 11 further shows that the frequency scale can substantially affect performance. Additional sensitivity analyses across tasks, particularly regarding the number of sampled frequencies and actor samples, would help clarify the robustness of the proposed distribution-matching procedure.

Questions for authors

Could you provide experimental results addressing the concerns raised in the Weaknesses?

4 comments
AuthorsOpenReview ↗

Response to Reviewer g2fc — Part 1/2

Thank you for your supportive and constructive review. We appreciate your recognition of the methodological contribution of our sequential Drift–Spread formulation, its strong empirical performance, and its efficient one-step action generation. We address each of your questions below.

(W1-1) Contribution of Spread to Online Reinforcement Learning

"The main MuJoCo experiments do not provide a direct comparison between Drift-only and the full DriftSpread method."

"Evaluating its contribution to online performance, action diversity, and exploration would help establish its practical necessity."

Thank you for this suggestion. We have added direct comparisons between the full DriftSpread method and Drift-only to assess Spread’s contribution to online performance, action diversity, and exploration. Specifically, we evaluate online performance and action diversity on Humanoid-v3 and Walker2d-v3, and provide a complementary qualitative comparison of exploration behavior on AntMaze-v3.

Online performance. DriftSpread improves returns over Drift-only by 40.0% on Humanoid-v3 and 29.8% on Walker2d-v3, providing direct evidence of Spread’s practical benefit on both tasks.

EnvironmentDrift-onlyDriftSpread
Humanoid-v36158.0 ± 1001.98619.5 ± 503.5 (+40.0%)
Walker2d-v34700.5 ± 1121.06100.6 ± 462.3 (+29.8%)

Effect of Spread on Policy Performance

Action diversity. Every 10,000 environment steps, we sample 128 states from each run’s own replay buffer and generate 512 actions per state. We report two complementary metrics: action variance, computed by averaging the within-state sample variance across action dimensions and states; and action entropy, estimated using a kNN estimator with k=5k=5 and averaged over states.

Removing Spread results in highly concentrated action distributions: action variance is approximately 10−510^{-5} for Drift-only, compared with 10−210^{-2} for DriftSpread—a difference of roughly three orders of magnitude. DriftSpread also exhibits higher estimated action entropy, providing complementary evidence of greater conditional sampling diversity. Together with the return improvements, these results support Spread’s practical value in maintaining action diversity and improving online performance. The experimental details and results are included in Appendix D.5.

EnvironmentMetricDrift-onlyDriftSpread
Humanoid-v3Action variance(2.931 ± 0.164) × 10⁻⁵(2.947 ± 0.155) × 10⁻² (1,006×)
Humanoid-v3kNN action entropy (nats)−74.11 ± 0.79−13.47 ± 1.10 (+60.64 nats)
Walker2d-v3Action variance(3.568 ± 0.709) × 10⁻⁵(2.279 ± 0.284) × 10⁻² (639×)
Walker2d-v3kNN action entropy (nats)−28.60 ± 0.91−6.72 ± 0.80 (+21.89 nats)

Effect of Spread on Conditional Action Diversity

Exploration-related behavior. To provide complementary qualitative evidence, we additionally compare the two methods on AntMaze-v3, a sparse-reward navigation task with two goal regions, where rewards are provided only upon reaching a goal. Across the evaluation trajectories shown in Appendix D.7, Figure 15, DriftSpread reaches both goal regions, whereas Drift-only concentrates on a single goal. These trajectories suggest that Spread encourages broader exploration, complementing the quantitative evidence of its benefits for online performance and action diversity.

AuthorsOpenReview ↗

Response to Reviewer g2fc — Part 2/2

(W1-2) Sensitivity to the Entropy Coefficient and Evolution Step Size

"Sensitivity analyses of the entropy coefficient and evolution step size, which directly control the Drift–Spread updates, are missing."

We have added sensitivity analyses of the entropy coefficient λ\lambda and evolution step size η\eta on Humanoid and Walker2d in Appendix D.11 (Figure 19) of the revised manuscript.

For the entropy coefficient, performance remains relatively stable across the tested values in [0.01,0.1][0.01, 0.1]. Using an excessively large entropy regularization coefficient reduces performance. We use λ=0.04\lambda=0.04 as the default across all tasks.

For the evolution step size, performance remains relatively stable across the tested values below 11, but declines for excessively large step sizes, such as η=10\eta=10. We use η=0.1\eta=0.1 as the default.

MethodParameterValueHumanoidWalker2d
DriftSpreadλ\lambda0.028010 ± 9945467 ± 613
DriftSpreadλ\lambda0.048620 ± 5046101 ± 462
DriftSpreadλ\lambda0.088526 ± 14465644 ± 572
DriftSpreadη\eta0.058372 ± 4595631 ± 632
DriftSpreadη\eta0.18620 ± 5046101 ± 462
DriftSpreadη\eta0.28112 ± 5985715 ± 528

(W2) Generalization to More Challenging Reinforcement Learning Benchmarks

"Additional experiments on more challenging benchmarks, such as OGBench or Robomimic, would help demonstrate the effectiveness and generalizability of the proposed method."

Thank you for this suggestion. To further evaluate DriftSpread’s applicability beyond locomotion, we have added experiments on three MyoSuite manipulation tasks: KeyTurn, Reach, and ObjHold. The table below reports each method’s success rate averaged across the three tasks. DriftSpread achieves the highest average success rate among all evaluated methods, providing further evidence of its effectiveness on manipulation tasks.

MethodAverage success across three tasks (%) ↑
DriftSpread (ours)90.2
DIME85.7
CrossQ76.5
MFPO61.1
SAC-Flow55.3
SDAC24.7
DACER23
DFP18.3
SAC14.2
MaxEntDP13.8
QVPO8.9
TD35.9
DPMD5.3
FlowRL3.4
WPPG1.9

Average success rate across three MyoSuite manipulation tasks: KeyTurn, Reach, and ObjHold.

Section 5.3 and Figure 6 present the per-task success-rate learning curves and the comparison of average success rate versus average training time. The experimental settings are provided in Appendix E.3.

(W3) Sensitivity to Frequency and Actor Sampling in the Spread Objective

"The practical implementation relies on finitely sampled frequencies and actor samples."

"Additional sensitivity analyses across tasks, particularly regarding the number of sampled frequencies and actor samples, would help clarify the robustness of the proposed distribution-matching procedure."

We have added sensitivity analyses varying the number of actor samples (HH) and sampled frequencies (MM) on Humanoid and Walker2d. DriftSpread maintains relatively stable performance across the tested settings on both tasks, supporting the robustness of the Spread distribution-matching procedure to these sampling choices within the evaluated ranges.

The full learning curves are provided in Appendix D.11, with Figure 20 varying the number of actor samples (HH) and Figure 21 varying the number of sampled frequencies (MM).

MethodParameterValueHumanoidWalker2d
DriftSpreadHH168337 ± 10295855 ± 531
DriftSpreadHH328620 ± 5046101 ± 462
DriftSpreadHH647979 ± 15965582 ± 279
DriftSpreadMM327843 ± 9575730 ± 804
DriftSpreadMM648620 ± 5046101 ± 462
DriftSpreadMM1287979 ± 12615529 ± 675

Thank you for the additional experiments. I believe my concerns have been sufficiently addressed, and I will maintain my positive rating.

Thank you very much for your positive assessment and support.

Reviewer_LBnoOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes an approach to learn drifting policies in off-policy maximum entropy reinforcement learning by using the wasserstein gradient flow based on the reverse KL divergence between a latent conditioned policy the optimal target distribution given by the boltzmann distribution of the current estimate of the Q function. From this they propose two losses: First a drifting loss that regresses the next parameterized drift towards the previous one plus the a term that is proportional to the gradient of the Q function. Secondly they propose to additionally use a spreading loss that is based on the MMD between two policy iterates. However these parts were difficult to assess for me because the authors do not provide a rigorous derivation of these losses, neither in the main paper nor in the appendix. the authors evaluate their method on a simple toy example, showing that their model is able to learn a ring distribution, which shows that their method can be more expressive than a simple gaussian policy. However, the authors do not provide results on a multimodal toy example. Thus multi modality was not tested. Finally the authors evaluate their method on 8 mujoco control benchmarks, where they in some cases show slight improvement compared to competitor methods, while also reporting faster runtime compared to more expensive diffusion approaches.

Critical strengths

I think the direction of having expressive and ideally multimodal one step generative policies is an interesting direction and in principle the algorithm presented here might be interesting.

Critical weaknesses

The paper is really poorly written and not presented in a human readable form. The authors do not provide a rigorous step by step derivation of their method, neither in the main text nor in the appendix. Also the authors barely give any intuition of what each loss term does exactly. The notation also sometimes is unclear to be and also sometimes never properly defined: Examples:

  • Eq. 9 L_spread depends on parameter \theta at iterate k+1/2 on the left side but then policy \tilde \pi depends on k+1. This does not make any sense to me.
  • Eq. 10 Suddenly the authors use \omega but its meaning is never introduced.
  • The authors never properly introduce the latent z with its distribution p_\ref, they just write down formula without properly introducing it.

From my experience I would say that AI was largely used for writing and the authors have not made a lot of effort to present their method in a way such that humans can understand it.

Questions for authors
  1. please explain how you derive/ arrive at L_drift and L_spread
  2. please explain what L_spread is supposed to do
  3. why does equation depend on k+1/2 on the left side but on the right side on k+1?
  4. what is \omega in equation 10?
  5. can you method also model simple 2-d multimodal toy examples and also more complex ones in higher dimensions? Such as GMM-40 in 2-D and in larger dimensions such as 10-D? (see [1], where the benchmark is named MoG)
  6. what is actually plotted in Fig 5 and Fig 6. Do the authors plot IQM return?

[1] Blessing, Denis, et al. "Beyond ELBOs: A large-scale evaluation of variational methods for sampling." arXiv preprint arXiv:2406.07423 (2024).

7 comments
AuthorsOpenReview ↗

Response to Reviewer LBno — Part 1/3

Thank you for your detailed feedback. We have revised Section 3.2 and Sections 4.1–4.2 to introduce the generator and its inputs, explain how the MaxEnt objective leads to the Drift and Spread updates, and show how the two losses learn these updates. We have also clarified the iteration indices and frequency variables, with detailed derivations provided in Appendix A. We address your questions below.

(W1 & Q1) Step-by-Step Derivation of the Drift and Spread Objectives

"The authors do not provide a rigorous step by step derivation of their method."

"please explain how you derive/ arrive at LdriftL_{\mathrm{drift}} and LspreadL_{\mathrm{spread}}"

Our actor generates a=fθ(s,z)a=f_\theta(s,z) from a state ss and noise z∼prefz\sim p_{\mathrm{ref}}, where prefp_{\mathrm{ref}} is a fixed reference distribution such as a standard Gaussian. The two losses fit different target updates obtained from the same MaxEnt objective.

1. From the MaxEnt objective to Drift and Spread.

For a fixed state and critic, minimizing FQ,s(π)=−Eπ[Q(s,a)]−λH(π)\mathcal F_{Q,s}(\pi)=-\mathbb E_\pi[Q(s,a)]-\lambda\mathcal H(\pi) balances high critic values with action diversity.

To turn this distribution objective into training targets, we need a rule for moving the actor’s action samples. Wasserstein gradient flow supplies this rule: it specifies sample movements that decrease the objective. For the MaxEnt objective, the rule in Section 3.2, Eq. (4), gives Vπ=∇aQ−λ∇alog⁡πV_\pi=\nabla_a Q-\lambda\nabla_a\log\pi. The continuity equation (Eq. (5)) then describes how these movements change the policy distribution, yielding the decomposition in Section 4.1, Eq. (9):

∂τπ=−∇a⋅(π∇aQ)+λΔaπ.\partial_\tau\pi=-\nabla_a\cdot(\pi\nabla_aQ)+\lambda\Delta_a\pi.

The first term comes from the critic objective and moves actions toward higher Q-values (Drift). The second comes from the entropy objective and spreads the action distribution (Spread). Directly using the full velocity would require the policy score ∇alog⁡π\nabla_a\log\pi, which is not readily available from our actor. We therefore handle the two terms in sequence: Drift moves actions along the critic gradient, and Spread adds noise to form a broader target distribution. The two losses train the actor to match these updates. Appendix A.1 gives the detailed derivation.

2. Drift: learn the update from the first term.

The first term, −∇a⋅(π∇aQ)-\nabla_a\cdot(\pi\nabla_aQ), describes how the policy changes when each action moves along the critic gradient ∇aQ(s,a)\nabla_aQ(s,a). This gradient points toward nearby actions with higher Q-values. The connection follows from the continuity equation (Section 3.2, Eq. (5)): moving actions with velocity vv gives ∂τπ=−∇a⋅(πv)\partial_\tau\pi=-\nabla_a\cdot(\pi v), so setting v=∇aQv=\nabla_aQ gives exactly the first term.

We therefore move each action a small step from aa to a+η∇aQ(s,a)a+\eta\nabla_aQ(s,a), using a forward Euler step. For an action generated from (s,z)(s,z), the moved action becomes a fixed training target. We then train the actor to output this target from the same input (s,z)(s,z). This gives the target and squared regression loss below (Section 4.2, Eqs. (10)–(11)):

\widetilde a_{k+1/2}(s,z)&=\operatorname{sg}\\!\left[f_{\theta_k}(s,z)+\eta\nabla_aQ\bigl(s,f_{\theta_k}(s,z)\bigr)\right],\\\\ \mathcal L_{\mathrm{Drift}}(\theta;\theta_k)&=\tfrac12\mathbb E_{s,z}\left\\|f_\theta(s,z)-\widetilde a_{k+1/2}(s,z)\right\\|^2. \end{aligned}$$ Here, $\operatorname{sg}$ holds the target fixed. Fitting this loss gives the post-Drift actor $f_{\theta_{k+1/2}}$. #### 3. Spread: learn the update from the second term. The second term, $\lambda\Delta_a\pi$, describes diffusion: it spreads the action distribution by adding random changes to the actions. We construct the Spread target distribution as the distribution of actions from the post-Drift actor perturbed by independent, zero-mean noise $\epsilon_\eta$. The noise covariance is $2\lambda\eta I_d$, so its scale follows from the coefficient $\lambda$ in the second term and the step size $\eta$. Under the conditions in Appendix A.1, Eqs. (20)–(21), this noise update matches the change described by $\partial_\tau\pi=\lambda\Delta_a\pi$ to first order in $\eta$. Gaussian noise gives the exact diffusion update over time $\eta$ on $\mathbb R^d$. At each state $s$, the distribution of the noisy actions $f_{\theta_{k+1/2}}(s,z)+\epsilon_\eta$ is therefore used as the Spread target $\widetilde\pi_{k+1}(\cdot\mid s)$. The actor must then learn this broader distribution. Using the Drift regression loss would not achieve this: for the same input $(s,z)$, the noisy targets vary around the original action, so squared regression would learn their average and remove the added noise. We instead compare the actor’s distribution with the target distribution using MMD.
AuthorsOpenReview ↗

Response to Reviewer LBno — Part 2/3

Minimizing this difference gives the Spread loss (Section 4.2, Eq. (12)):

LSpread(θ;θk+1/2)=Es![MMD⁡h2(πθ(⋅∣s),π~k+1(⋅∣s))].\mathcal L_{\mathrm{Spread}}(\theta;\theta_{k+1/2})=\mathbb E_s\\!\left[\operatorname{MMD}_h^2\bigl(\pi_\theta(\cdot\mid s),\widetilde\pi_{k+1}(\cdot\mid s)\bigr)\right].

Here, hh is a characteristic kernel. MMD compares the two distributions using samples, without evaluating policy densities. The target distribution π~k+1\widetilde{\pi}_{k+1} is constructed by adding independent noise to samples generated by the post-Drift actor fθk+1/2f_{\theta_{k+1/2}}. Its subscript k+1k+1 indicates the distribution we aim for the actor fθk+1f_{\theta_{k+1}} to represent after the Spread update. This target stays fixed while the actor parameters are updated from θk+1/2\theta_{k+1/2} to θk+1\theta_{k+1}, yielding a policy πk+1\pi_{k+1} that approximates π~k+1\widetilde{\pi}_{k+1}. Equation (14) computes this MMD loss using characteristic functions estimated from samples; Appendix A.2.1 gives the derivation.

(W1 & Q2) Interpretation and Role of the Spread Objective

"The authors barely give any intuition of what each loss term does exactly."

"please explain what LspreadL_{\mathrm{spread}} is supposed to do"

The Spread loss encourages the policy to produce diverse actions. Drift moves actions toward higher Q-values, but it can also make them concentrate around a few actions. Spread accounts for the entropy term in maximum entropy RL, which favors a broader action distribution.

We first add independent noise to actions sampled from the actor after Drift. The distribution of these noisy actions is the target for Spread. The Spread loss uses MMD to compare the actor’s distribution with this target. Minimizing the loss makes the actor’s distribution closer to the broader target. The actor then generates actions from the learned distribution in one forward pass.

The entropy temperature and step size determine the noise scale. Section 4.2 explains how the Spread target is constructed and fitted, while Appendix A.1 derives the underlying diffusion update.

(W1 & Q3) Indexing and Notation in the Spread Objective

"LspreadL_{\mathrm{spread}} depends on parameter θ\theta at iterate k+1/2k+1/2 on the left side but then policy π~\tilde\pi depends on k+1k+1."

For a given noise distribution, the Spread target π~k+1\widetilde\pi_{k+1} depends on the actor parameters only through the post-Drift parameters θk+1/2\theta_{k+1/2}. At each state, we generate target actions of the form a~k+1(s,z,ϵλ,η)=fθk+1/2(s,z)+ϵλ,η\widetilde a_{k+1}(s,z,\epsilon_{\lambda,\eta})=f_{\theta_{k+1/2}}(s,z)+\epsilon_{\lambda,\eta}, where the actor input z∼prefz\sim p_{\mathrm{ref}} and ϵλ,η∼Kλ,η\epsilon_{\lambda,\eta}\sim K_{\lambda,\eta} are sampled independently. The resulting actions have the target distribution π~k+1(⋅∣s)=Kλ,η∗πk+1/2(⋅∣s)\widetilde\pi_{k+1}(\cdot\mid s)=K_{\lambda,\eta}*\pi_{k+1/2}(\cdot\mid s). Here Kλ,ηK_{\lambda,\eta} denotes the distribution of the additive noise ϵλ,η\epsilon_{\lambda,\eta}. The post-Drift policy and the noise distribution therefore determine the target before the Spread update begins.

To fit the actor to this target, we minimize the Spread loss LSpread(θ;θk+1/2)\mathcal L_{\mathrm{Spread}}(\theta;\theta_{k+1/2}) over its first argument, θ\theta, while holding the second argument fixed. The loss measures the squared MMD between the actor and target action distributions, averaged over states. We denote the updated actor parameters by θk+1\theta_{k+1}. The resulting policy πk+1\pi_{k+1} approximates the target π~k+1\widetilde\pi_{k+1}. We clarify these indices in Section 4.2, Eq. (12) of the revision.

(W1 & Q4) Definition of the Frequency Variable and Latent Prior

"Eq. 10 Suddenly the authors use ω\omega but its meaning is never introduced."

"The authors never properly introduce the latent zz with its distribution prefp_{\mathrm{ref}}."

In Eq. (10) of the submitted version, ω∈Rd\omega\in\mathbb R^d is the frequency vector used to compare the actor and Spread target through their characteristic functions (CFs), where dd is the action dimension. A CF represents a distribution qq as φq(ω)=Ea∼q[eiω⊤a]\varphi_q(\omega)=\mathbb E_{a\sim q}[e^{i\omega^\top a}]. Here, ω\omega specifies the function of actions being averaged, and the average can be estimated directly from samples.

We use CFs to compute the MMD objective for Spread. Specifically, we sample frequencies ω∼ρ\omega\sim\rho, estimate the actor and target CFs at the same frequencies, and minimize their squared absolute differences. The expectation of these differences under a Gaussian ρ\rho equals the squared MMD with a Gaussian kernel. This gives a way to match the two distributions without evaluating their densities. Section 4.2 now defines ω\omega and ρ\rho around Eqs. (13)–(14). Appendix A.2.1 shows how to write the MMD loss using CFs and estimate it from samples.

AuthorsOpenReview ↗

Response to Reviewer LBno — Part 3/3

The latent variable zz is the random input used to generate an action: z∼prefz\sim p_{\mathrm{ref}} and a=fθ(s,z)a=f_\theta(s,z). Here, prefp_{\mathrm{ref}} is a fixed reference distribution, such as a standard Gaussian. At a given state, sampling different inputs zz produces the actor’s action distribution. The revision introduces zz and prefp_{\mathrm{ref}} in Section 3.2 and the state-conditioned actor at the start of Section 4.

(Q5) Multimodal Distribution Modeling in Two and Higher Dimensions

"can you method also model simple 2-d multimodal toy examples and also more complex ones in higher dimensions?"

"Such as GMM-40 in 2-D and in larger dimensions such as 10-D?"

Thank you for suggesting this benchmark. We have added MoG-40 experiments from [6] in 2D and 10D, and extended the evaluation to 50D (Appendix D.6). We compare DriftSpread with SAC and Drift-only, which removes the Spread update.

2D visualization. Figure 14 shows the target and 2,000 generated samples from each method. DriftSpread produces samples across the separated target modes, while Drift-only and SAC concentrate on a small subset. This observation is supported by the statistics across five training seeds: DriftSpread covers all 40 modes in every seed, compared with 1.0 ± 0.0 for Drift-only and 2.4 ± 1.3 for SAC.

Quantitative results across dimensions. The table below reports results in 2D, 10D, and 50D as means ± sample standard deviations over five training seeds, evaluated separately. Entropic mode coverage (EMC) measures how evenly samples are distributed across the target modes; higher is better. Sinkhorn divergence measures the overall difference from the target distribution; lower is better.

DimensionMethodEntropic mode coverage ↑Sinkhorn divergence ↓
2DDriftSpread0.94 ± 0.02171 ± 85
2DDrift-only0.00 ± 0.00999 ± 16
2DSAC0.17 ± 0.152,067 ± 202
10DDriftSpread0.93 ± 0.011,129 ± 155
10DDrift-only0.48 ± 0.013,810 ± 42
10DSAC0.00 ± 0.007,011 ± 199
50DDriftSpread0.90 ± 0.0211,430 ± 1,199
50DDrift-only0.65 ± 0.0123,447 ± 281
50DSAC0.00 ± 0.0043,308 ± 1,013

DriftSpread maintains high EMC in 10D (0.93 ± 0.01) and 50D (0.90 ± 0.02), and achieves lower Sinkhorn divergence than both baselines in every tested dimension. These results support its ability to capture separated modes beyond 2D.

(Q6) Interpretation of Figures 5 and 6

"what is actually plotted in Fig 5 and Fig 6. Do the authors plot IQM return?"

Figures 5 and 6 of the submitted version, now Figures 5 and 17 in the revised manuscript, show arithmetic mean evaluation returns. At each evaluation checkpoint, each seed's score is the mean undiscounted episode return over 10 evaluation episodes. The solid lines show the mean of these scores across five training seeds, and the shaded bands show ±1 sample standard deviation across seeds. The horizontal axis is the number of training environment interactions. The evaluation protocol is described in Appendix E.4. This reporting convention is also used in prior work on diffusion- and flow-based policies for online RL [1–5].

We have additionally included interquartile mean (IQM) return curves in Appendix D.12 (Figure 22). The table below compares the mean and IQM summaries, with each of the five MuJoCo tasks weighted equally.

MethodMean returnIQM return
DriftSpread8145.78206.4
DIME7559.27574.4
MFPO6825.66832.1
SAC-Flow6594.26732.6
CrossQ6512.36614.6
SDAC6489.86532.2
DACER6316.56419.9
SAC61366157
MaxEntDP5950.46075.9
WPPG5464.95665.3
TD35476.55574.9
DPMD5310.75418.6
QVPO49415109.6
FlowRL4294.54372.2
DFP3587.23942.4

Mean and IQM evaluation returns averaged across five MuJoCo tasks, with equal weight assigned to each task.

DriftSpread achieves the highest average return across the five tasks under both summaries: 8,145.7 using the mean and 8,206.4 using IQM. Its advantage in this comparison is therefore consistent across the two aggregation choices.

Reference List

[1] Mean Flow Policy Optimization. ICML 2026.

[2] Maximum Entropy Reinforcement Learning with Diffusion Policy. ICML 2025.

[3] Efficient Online Reinforcement Learning for Diffusion Policy. ICML 2025.

[4] Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization. NeurIPS 2024.

[5] Flow-Based Policy for Online Reinforcement Learning. NeurIPS 2025.

[6] Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Sampling. ICML, 2024.

Reviewer_XKXYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DriftSpread, a one-step generative policy for online maximum-entropy reinforcement learning (RL). It decomposes policy improvement into two sequential updates: Drift, which moves actions towards higher critic values, and Spread, which uses distribution matching to capture the effect of entropy without estimating policy densities or scores. The authors prove convergence to the critic-defined Boltzmann policy under exact fitting, a fixed critic and a small-step, long-time limit. Experiments on five MuJoCo tasks compare the method with 13 baselines, reporting competitive or superior returns alongside lower training and inference costs than most diffusion- and flow-based alternatives.

Critical strengths
  1. A clear and technically motivated learning mechanism. The sequential Drift–Spread formulation addresses a genuine challenge in implicit-policy learning: balancing value improvement and distributional diversity without tractable policy densities. The use of characteristic functions for distribution matching is well motivated, and the analysis explains why sequential fitting can succeed where a jointly weighted objective may fail.

  2. Broad empirical comparisons and practical efficiency. Evaluation against 13 baselines across five MuJoCo tasks, together with controlled comparisons of spreading noise and sequential versus joint fitting, provides useful evidence for the method. One-step action generation is particularly attractive for settings where inference latency matters.

Critical weaknesses
  1. The theoretical guarantee does not directly establish convergence of the practical algorithm. The theorem assumes exact population fitting and a fixed critic, and takes the small-step limit before the long-time limit. In practice, the actor receives only one optimiser step per update and the critic changes throughout training. Moreover, the implemented critic target omits the entropy term, despite the theoretical analysis targeting the maximum-entropy Boltzmann distribution. The gap between the theoretical setting and the actual algorithm therefore remains insufficiently resolved.

  2. The evaluation protocol complicates the interpretation of performance gains. DriftSpread and MFPO select the highest-critic-valued action from ten candidates during evaluation, whereas other methods follow their reference action-selection rules. Although the reported inference time accounts for candidate generation and selection, this asymmetry makes it difficult to attribute the return improvements solely to the learned policy. Results using a single sampled action would help establish a fairer comparison. In addition, five random seeds provide limited evidence about the robustness of the reported differences.

  3. This paper miss some important baselines like CrossQ and SAC-Flow.

Questions for authors
  1. How do the MuJoCo results change when DriftSpread and MFPO are evaluated using a single sampled action, without selecting the best of ten candidates? Could the authors report this comparison to isolate the contribution of the learned policy from candidate selection?

  2. Why does the practical critic update omit the entropy term while the convergence analysis targets the Boltzmann distribution induced by the maximum-entropy objective? What justifies this choice, and is there empirical evidence that it does not materially affect the learned policy?

  3. Can the authors investigate whether the theoretical convergence result can be extended to finite step sizes, approximate actor fitting and a changing critic, or otherwise quantify the discrepancy between the idealised dynamics and practical training?

6 comments
AuthorsOpenReview ↗

Response to Reviewer XKXY — Part 1/3

Thank you for your constructive review. We provide the response to each question as below.

(W1 & Q3) Relationship Between Theoretical Convergence and Practical Training

"The theoretical guarantee does not directly establish convergence of the practical algorithm."

"Can the authors investigate whether the theoretical convergence result can be extended to finite step sizes, approximate actor fitting and a changing critic?"

To address this concern, we extend our theoretical analysis to establish convergence with a changing critic and quantify the effects of finite step sizes and approximate actor fitting. We summarize our two main results below and provide the full assumptions and proofs in Appendix B (Sections B.4–B.7).

1. Convergence to the MaxEnt-optimal policy with a changing critic.

We extend the fixed-critic analysis in Theorem 4.1 to a setting in which the critic is updated with the current policy. Under exact soft policy evaluation Qk=QπkQ_k=Q^{\pi_k} and exact actor fitting, we prove that DriftSpread converges to the MaxEnt-optimal policy with sufficiently small diminishing step sizes satisfying

\qquad \sum_{k=0}^{\infty}\eta_k^2<\infty.$$ Specifically, $$\max_s D_{\mathrm{KL}}\\!\left( \pi_k(\cdot\mid s)\\,\middle\\|\\,\pi^\star(\cdot\mid s) \right)\longrightarrow 0,$$ where $\pi^\star$ is the MaxEnt-optimal policy. The critic is recomputed after each DriftSpread cycle, so the actor need not converge to the current Boltzmann target before the next critic update. This establishes optimal-policy convergence when the critic evolves with the policy throughout training. #### 2. Error bounds for finite step sizes and approximate actor fitting. We next quantify how finite step sizes and approximate actor fitting affect the actor’s evolution under a changing critic. For a given critic sequence, we establish the following bound on the discrepancy from the corresponding idealized continuous-time dynamics: $$\sup_{k\ge 0} W_1\\!\left(\pi_k^{(\eta)},p_{k\eta}^{(\eta)}\right) \le C_1\eta+C_2\frac{\delta_\eta}{\eta}.$$ Here, $W_1$ denotes the first-order Wasserstein distance, and $\pi_k^{(\eta)}$ is the actor distribution after approximate fitting. The reference distribution $p_\tau^{(\eta)}$ corresponds to the idealized continuous-time actor dynamics driven by the same changing critic sequence, with $Q_k$ used over $[k\eta,(k+1)\eta)$. The quantity $\delta_\eta$ bounds the per-update fitting error, measured by the second-order Wasserstein distance $W_2$ between the fitted actor distribution and the exact DriftSpread target. The first term captures discretization error, while the second captures the accumulated effect of actor-fitting errors. This bound explicitly quantifies how the step size and fitting accuracy affect the discrepancy from the idealized continuous dynamics. Crucially, the bound holds for all training iterations, with constants independent of training duration, even as the critic changes throughout training. Together, these results extend the optimality guarantee to changing critics and quantify the effects of finite step sizes and approximate actor fitting. We would also highlight that extending theoretical guarantees to the full neural-network training process remains a shared challenge across deep actor–critic methods, including those with diffusion- and flow-based policies. For example, the policy-iteration guarantees presented in DIME [11], MFPO [8], and MaxEntDP [9] rely on exact policy evaluation and exact actor optimization. DPMD/SDAC [3], FlowRL [5], and DFP [6] primarily analyze actor objectives or approximation errors for a given fixed critic. Even for SAC [10], the original convergence theorem concerns exact soft policy iteration in a tabular setting, while its practical implementation alternates stochastic gradient updates of neural actors and critics. Extending these guarantees to the joint training of neural actors and critics, with stochastic gradient updates per iteration, remains a broader challenge in deep actor–critic convergence analysis.
AuthorsOpenReview ↗

Response to Reviewer XKXY — Part 2/3

(W1 & Q2) Entropy Term in the Practical Critic Update

"The implemented critic target omits the entropy term, despite the theoretical analysis targeting the maximum-entropy Boltzmann distribution."

"What justifies this choice, and is there empirical evidence that it does not materially affect the learned policy?"

Our choice is motivated by two considerations: empirical performance and consistency with prior work.

First, experiments conducted during the development of DriftSpread showed limited benefits from soft-Q updates and informed our choice of critic. In those experiments, we compared the standard critic with a variant incorporating an estimated entropy correction. Following the GMM-based density approximation used in DACER [7], we sampled multiple actions from DriftSpread at each next state and fitted a Gaussian mixture model. We then evaluated the sampled next action’s log-density under the fitted GMM and incorporated the corresponding entropy correction into the Bellman target.

As shown in the table, the soft-Q variant yielded a modest improvement on Humanoid-v3 and comparable returns on Walker2d-v3. These findings motivated our use of the standard critic in the original implementation.

Second, several related generative-policy methods adopt the same design choice. DQS [1] and RFM [2] train actors toward Boltzmann targets, while QVPO [4] introduces an entropy-promoting regularizer in the actor loss. All three use critic targets without an explicit policy-entropy correction. Notably, RFM [2] directly evaluates this choice and reports that adding a soft-Q correction slightly degrades performance in its ablation study.

We now report these results and provide a detailed discussion in Appendix D.4 of the revised manuscript.

EnvironmentDriftSpreadDriftSpread (soft-Q)
Humanoid-v38,620 ± 5049,242 ± 539
Walker2d-v36,101 ± 4626,092 ± 688

Effect of the critic entropy correction on DriftSpread.

(W2 & Q1) Single-Action Evaluation and Best-of-Ten Candidate Selection

"DriftSpread and MFPO select the highest-critic-valued action from ten candidates during evaluation, whereas other methods follow their reference action-selection rules."

"Results using a single sampled action would help establish a fairer comparison."

Critic-based selection among multiple action candidates is also used by MFPO [8], MaxEntDP [9], QVPO [4], SDAC and DPMD [3], and DFP [6]. To directly assess its impact and enable comparison under a single-candidate evaluation budget, we conducted ablations for every compared method whose default evaluation uses multiple candidates.

The table reports average returns averaged across the five MuJoCo tasks. Parentheses indicate the percentage decrease relative to each method’s default mean return. Methods that already use a single candidate retain their default results and are marked with a dash in the ablation column.

DriftSpread retains the highest average return under single-candidate evaluation, achieving 7,754 compared with 8,146 under its default setting. Its 4.8% decrease is the smallest among all methods included in this ablation, while the other ablated methods show decreases ranging from 7.1% to 90.3%. These results indicate that DriftSpread’s strong performance is less dependent on critic-based candidate selection than that of the other methods evaluated in this ablation.

We have added the full results and learning curves to Appendix D.10 of the revised manuscript.

MethodDefault1 candidate
DriftSpread8,1467,754 (↓4.8%)
MFPO6,8266,050 (↓11.4%)
DIME7,559—
SAC-Flow6,594—
DACER6,317—
FlowRL4,295—
QVPO4,9413,805 (↓23.0%)
MaxEntDP5,9505,529 (↓7.1%)
SAC6,136—
TD35,477—
DFP3,5871,961 (↓45.3%)
DPMD5,311517 (↓90.3%)
SDAC6,4903,330 (↓48.7%)
WPPG5,465—
CrossQ6,512—
AuthorsOpenReview ↗

Response to Reviewer XKXY — Part 3/3

(W2) Statistical Robustness Across Random Seeds

"Five random seeds provide limited evidence about the robustness of the reported differences."

To strengthen our evaluation, we expanded DriftSpread from five to ten random seeds across all five MuJoCo tasks, completing 25 additional training runs. The table below compares the original five-seed results (seeds 0–4) with the expanded ten-seed results (seeds 0–9).

The mean return differs by less than 2.7% on every task, showing that the estimated average performance remains consistent after doubling the number of seeds. The five-seed and ten-seed learning curves also exhibit similar overall learning trends across all five tasks. These results provide additional evidence that DriftSpread’s reported mean performance is stable across a larger set of random seeds. The corresponding learning curves are provided in Appendix D.8, “Robustness Across Ten Random Seeds.”

Our original five-seed protocol is consistent with evaluations reported in related work, including MFPO [8], SDAC and DPMD [3], DACER [7], and FlowRL [5]. Given the rebuttal timeline and computational cost, we prioritized extending DriftSpread’s evaluation. We will extend the baseline evaluations to ten seeds and update the main comparative results accordingly in the camera-ready version.

Task5 seeds (0–4)10 seeds (0–9)Relative change in mean
Humanoid-v38619.5 ± 503.58760.3 ± 890.5+1.63%
HalfCheetah-v315048.0 ± 817.814856.6 ± 692.9−1.27%
Ant-v37288.5 ± 182.57121.7 ± 517.7−2.29%
Walker2d-v36100.6 ± 462.35938.6 ± 542.9−2.65%
Hopper-v33672.0 ± 279.13741.0 ± 254.4+1.88%

DriftSpread performance with five and ten random seeds.

(W3) Additional Comparisons with CrossQ and SAC-Flow

"This paper miss some important baselines like CrossQ and SAC-Flow."

SAC-Flow was already included in the submitted version, both in the baseline list under “MuJoCo Benchmarks” and in Figures 1 and 5.

Following your suggestion, we have additionally evaluated CrossQ on all five MuJoCo tasks using five random seeds. The revised Figure 1 presents the aggregate performance and runtime comparison, while Figure 5 shows the per-task learning curves with mean ± standard deviation.

DriftSpread retains the highest average return across the five MuJoCo tasks after including CrossQ, further supporting its strong empirical performance against these additional baselines.

References

[1] Sampling from Energy-based Policies using Diffusion. RLC, 2025.

[2] Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies. ICML, 2026.

[3] Efficient Online Reinforcement Learning for Diffusion Policy. ICML, 2025.

[4] Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization. NeurIPS, 2024.

[5] Flow-Based Policy for Online Reinforcement Learning. NeurIPS, 2025.

[6] Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flow. arXiv:2605.07727, 2026.

[7] Diffusion Actor-Critic with Entropy Regulator. NeurIPS, 2024.

[8] Mean Flow Policy Optimization. ICML, 2026.

[9] Maximum Entropy Reinforcement Learning with Diffusion Policy. ICML, 2025.

[10] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML, 2018.

[11] DIME: Diffusion-Based Maximum Entropy Reinforcement Learning. ICML, 2025.

[12] SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling. ICLR, 2026.

[13] Wasserstein Proximal Policy Gradient. arXiv:2603.02576, 2026.

[14] Addressing Function Approximation Error in Actor-Critic Methods. ICML, 2018.

[15] CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity. ICLR, 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response and Summary of Revisions

We thank the reviewers for their careful reading and constructive feedback. We have revised the manuscript as follows.

Main-text revisions

  • Sections 4.1–4.2 — Derivation and training objectives. We revised the derivation to show how the value and entropy terms give rise to Drift and Spread. We also explain why Drift can be learned by regressing onto critic-guided actions, whereas Spread requires matching the distribution of noise-perturbed actions.

  • Section 4.3 — Changing critics. We added two results beyond the fixed-critic analysis. With exact soft policy evaluation, exact actor fitting, and suitably decreasing step sizes, the policy converges to the maximum-entropy optimum. For a given critic sequence, we also bound the effect of finite step sizes and actor-fitting error at every iteration.

  • Sections 5.2–5.3 — MuJoCo and MyoSuite experiments. We added CrossQ to the MuJoCo comparison and evaluated DriftSpread on three MyoSuite manipulation tasks: KeyTurn, Reach, and ObjHold. Across these tasks, DriftSpread achieves the highest mean success rate among the methods tested (90.2%, compared with 85.7% for DIME) while training faster than DIME on average.

Appendix additions

  • Derivations and proofs. Appendix A.1 gives the derivation of the Drift and Spread updates. Appendix B.4 states the assumptions and both results for changing critics, and Appendices B.5–B.7 give the supporting estimates and proofs.

  • Evaluation protocol and robustness. Appendix D.8 extends the evaluation of DriftSpread from five to ten seeds on all five MuJoCo tasks. The mean return over the final ten checkpoints changes by less than 2.7% on each task. Appendix D.10 compares single-candidate and default action selection, and Appendix D.12 reports interquartile mean (IQM) return curves.

  • Ablations and distribution learning. Appendices D.4–D.5 examine the entropy correction in the critic update and the effect of Spread on action diversity. Appendix D.6 tests coverage of 40-mode distributions in 2, 10, and 50 dimensions, and Appendix D.7 evaluates behavioral diversity on sparse-reward AntMaze tasks. Appendices D.9 and D.11 examine the choice of spreading noise and sensitivity to key hyperparameters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.