- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DriftSpread, a one-step generative policy learning method for online maximum entropy reinforcement learning based on the Drifting framework. Building on an energy-based formulation of maximum-entropy policy improvement, the method decomposes policy updates into critic-guided Drift and entropy-induced Spread. Drift updates the actor through regression toward actions displaced by critic gradients, while Spread uses characteristic-function-based maximum mean discrepancy (MMD) to match the entropy-induced target distribution without requiring policy density estimation. The paper also provides a theoretical analysis showing convergence to the critic-defined Boltzmann distribution under exact fitting and fixed-critic assumptions.
Critical strengths
S1 (Methodological contribution). The paper presents a meaningful extension of the Drifting framework to online maximum-entropy RL. In particular, decomposing policy improvement into critic-guided Drift and entropy-induced Spread provides a novel approach to learning expressive one-step generative policies.
S2 (Density-free entropy regularization). The proposed Spread objective provides a way to incorporate entropy regularization without explicitly estimating policy densities. The use of characteristic-function-based MMD is particularly interesting, as it enables distribution matching using generated samples.
S3 (Theoretical analysis). The paper establishes a theoretical connection between the Drift–Spread updates and convergence to the critic-defined Boltzmann distribution. Although the analysis relies on fixed-critic and exact-fitting assumptions, it provides useful theoretical justification for the proposed learning procedure.
S4 (Empirical performance and computational efficiency). The proposed method achieves strong aggregate performance on 5 MuJoCo tasks while maintaining efficient one-step action generation.
Critical weaknesses
W1 (Missing ablation of the entropy-induced Spread component). The paper demonstrates the effect of Spread on the learned distribution through ring-shaped toy experiments. However, the main MuJoCo experiments do not provide a direct comparison between Drift-only and the full DriftSpread method. Since Spread is a central component of the proposed approach, evaluating its contribution to online performance, action diversity, and exploration would help establish its practical necessity. Furthermore, sensitivity analyses of the entropy coefficient and evolution step size , which directly control the Drift–Spread updates, are missing.
W2 (Limited experimental diversity). The evaluation is limited to 5 MuJoCo locomotion tasks. Additional experiments on more challenging benchmarks, such as OGBench or Robomimic, would help demonstrate the effectiveness and generalizability of the proposed method in diverse manipulation and (semi-) sparse reward settings.
W3 (Sensitivity of distribution matching to frequency sampling). The proposed Spread objective depends on the frequency distribution used for characteristic-function matching. Although the Gaussian frequency distribution satisfies the full-support assumption in Theorem 4.1, the practical implementation relies on finitely sampled frequencies and actor samples. Figure 11 further shows that the frequency scale can substantially affect performance. Additional sensitivity analyses across tasks, particularly regarding the number of sampled frequencies and actor samples, would help clarify the robustness of the proposed distribution-matching procedure.
Questions for authors
Could you provide experimental results addressing the concerns raised in the Weaknesses?
4 comments
Response to Reviewer g2fc — Part 2/2
(W1-2) Sensitivity to the Entropy Coefficient and Evolution Step Size
"Sensitivity analyses of the entropy coefficient and evolution step size, which directly control the Drift–Spread updates, are missing."
We have added sensitivity analyses of the entropy coefficient and evolution step size on Humanoid and Walker2d in Appendix D.11 (Figure 19) of the revised manuscript.
For the entropy coefficient, performance remains relatively stable across the tested values in . Using an excessively large entropy regularization coefficient reduces performance. We use as the default across all tasks.
For the evolution step size, performance remains relatively stable across the tested values below , but declines for excessively large step sizes, such as . We use as the default.
| Method | Parameter | Value | Humanoid | Walker2d |
|---|---|---|---|---|
| DriftSpread | 0.02 | 8010 ± 994 | 5467 ± 613 | |
| DriftSpread | 0.04 | 8620 ± 504 | 6101 ± 462 | |
| DriftSpread | 0.08 | 8526 ± 1446 | 5644 ± 572 | |
| DriftSpread | 0.05 | 8372 ± 459 | 5631 ± 632 | |
| DriftSpread | 0.1 | 8620 ± 504 | 6101 ± 462 | |
| DriftSpread | 0.2 | 8112 ± 598 | 5715 ± 528 |
(W2) Generalization to More Challenging Reinforcement Learning Benchmarks
"Additional experiments on more challenging benchmarks, such as OGBench or Robomimic, would help demonstrate the effectiveness and generalizability of the proposed method."
Thank you for this suggestion. To further evaluate DriftSpread’s applicability beyond locomotion, we have added experiments on three MyoSuite manipulation tasks: KeyTurn, Reach, and ObjHold. The table below reports each method’s success rate averaged across the three tasks. DriftSpread achieves the highest average success rate among all evaluated methods, providing further evidence of its effectiveness on manipulation tasks.
| Method | Average success across three tasks (%) ↑ |
|---|---|
| DriftSpread (ours) | 90.2 |
| DIME | 85.7 |
| CrossQ | 76.5 |
| MFPO | 61.1 |
| SAC-Flow | 55.3 |
| SDAC | 24.7 |
| DACER | 23 |
| DFP | 18.3 |
| SAC | 14.2 |
| MaxEntDP | 13.8 |
| QVPO | 8.9 |
| TD3 | 5.9 |
| DPMD | 5.3 |
| FlowRL | 3.4 |
| WPPG | 1.9 |
Average success rate across three MyoSuite manipulation tasks: KeyTurn, Reach, and ObjHold.
Section 5.3 and Figure 6 present the per-task success-rate learning curves and the comparison of average success rate versus average training time. The experimental settings are provided in Appendix E.3.
(W3) Sensitivity to Frequency and Actor Sampling in the Spread Objective
"The practical implementation relies on finitely sampled frequencies and actor samples."
"Additional sensitivity analyses across tasks, particularly regarding the number of sampled frequencies and actor samples, would help clarify the robustness of the proposed distribution-matching procedure."
We have added sensitivity analyses varying the number of actor samples () and sampled frequencies () on Humanoid and Walker2d. DriftSpread maintains relatively stable performance across the tested settings on both tasks, supporting the robustness of the Spread distribution-matching procedure to these sampling choices within the evaluated ranges.
The full learning curves are provided in Appendix D.11, with Figure 20 varying the number of actor samples () and Figure 21 varying the number of sampled frequencies ().
| Method | Parameter | Value | Humanoid | Walker2d |
|---|---|---|---|---|
| DriftSpread | 16 | 8337 ± 1029 | 5855 ± 531 | |
| DriftSpread | 32 | 8620 ± 504 | 6101 ± 462 | |
| DriftSpread | 64 | 7979 ± 1596 | 5582 ± 279 | |
| DriftSpread | 32 | 7843 ± 957 | 5730 ± 804 | |
| DriftSpread | 64 | 8620 ± 504 | 6101 ± 462 | |
| DriftSpread | 128 | 7979 ± 1261 | 5529 ± 675 |
Thank you for the additional experiments. I believe my concerns have been sufficiently addressed, and I will maintain my positive rating.
Thank you very much for your positive assessment and support.
Response to Reviewer g2fc — Part 1/2
Thank you for your supportive and constructive review. We appreciate your recognition of the methodological contribution of our sequential Drift–Spread formulation, its strong empirical performance, and its efficient one-step action generation. We address each of your questions below.
(W1-1) Contribution of Spread to Online Reinforcement Learning
Thank you for this suggestion. We have added direct comparisons between the full DriftSpread method and Drift-only to assess Spread’s contribution to online performance, action diversity, and exploration. Specifically, we evaluate online performance and action diversity on Humanoid-v3 and Walker2d-v3, and provide a complementary qualitative comparison of exploration behavior on AntMaze-v3.
Online performance. DriftSpread improves returns over Drift-only by 40.0% on Humanoid-v3 and 29.8% on Walker2d-v3, providing direct evidence of Spread’s practical benefit on both tasks.
Effect of Spread on Policy Performance
Action diversity. Every 10,000 environment steps, we sample 128 states from each run’s own replay buffer and generate 512 actions per state. We report two complementary metrics: action variance, computed by averaging the within-state sample variance across action dimensions and states; and action entropy, estimated using a kNN estimator with k=5 and averaged over states.
Removing Spread results in highly concentrated action distributions: action variance is approximately 10−5 for Drift-only, compared with 10−2 for DriftSpread—a difference of roughly three orders of magnitude. DriftSpread also exhibits higher estimated action entropy, providing complementary evidence of greater conditional sampling diversity. Together with the return improvements, these results support Spread’s practical value in maintaining action diversity and improving online performance. The experimental details and results are included in Appendix D.5.
Effect of Spread on Conditional Action Diversity
Exploration-related behavior. To provide complementary qualitative evidence, we additionally compare the two methods on AntMaze-v3, a sparse-reward navigation task with two goal regions, where rewards are provided only upon reaching a goal. Across the evaluation trajectories shown in Appendix D.7, Figure 15, DriftSpread reaches both goal regions, whereas Drift-only concentrates on a single goal. These trajectories suggest that Spread encourages broader exploration, complementing the quantitative evidence of its benefits for online performance and action diversity.