acceptodds
Under review as a conference paper at ICLR 2027

Score-Based One-step MeanFlow Policy Optimization

Abstract

Diffusion and flow matching have emerged as expressive policy classes in reinforcement learning, but their reliance on multi-step denoising imposes substantial computational overhead at inference time, which is particularly problematic in online RL. MeanFlow offers a promising alternative by learning an average velocity field that maps noise to data in a single network evaluation. However, MeanFlow requires samples from the target distribution to construct its target velocity field, which are unavailable in online RL, and existing workarounds that use best-of- selections as training targets can only reinforce actions the current policy already produces. We propose Score-Based One-step MeanFlow Policy Optimization (SOM), an actor-critic algorithm that constructs the target velocity field directly from Q-function gradients via score estimation and the probability flow ODE, thereby transporting noise toward high-value modes. To stabilize score estimation under an evolving critic, we adapt the inverse temperature to control candidate-weight concentration via the effective sample size (ESS) across diffusion times, and anneal the target ESS over training. In fully online RL, SOM achieves the highest normalized return across five MuJoCo locomotion benchmarks with a single generation step, while substantially reducing inference latency compared to diffusion-based policies.

Primary area:
reinforcement learning
Keywords:
Flow Matching, Online Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oRFVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents one step MeanFlow policies for fully online RL. MeanFlow is normally trained by copying example actions from the distribution it should learn but in online RL no such examples exist and prior method like MVP instead imitates the best of N sampled actions. Authors proposed SOM which takes the Boltzmann policy as the target and builds the MeanFlow velocity from the scores of its noised levels estimated by importance weighting critic gradients over sampled candidates with inverse temperature constant adapted per noise level to hold a target effective sample size. Experiments on nine MuJoCo tasks show the highest return on the five main locomotion tasks with one step inference supported by a frozen critic comparison with MVP 2D bandit studies and extensive ablations in the appendix.

Critical strengths
  1. SOM solves a real gap for one step generative policies in online RL by building the training target from critic's gradients instead of copying best of N samples and the frozen critic and bandit experiments show this behaves differently and clearly from best of N imitation.

  2. SOM leads on the main tasks with one step inference and the appendix ablates nearly every design choice and openly reports where the defaults lose and how much the method relies on best of N selection.

Critical weaknesses
  1. My main concern is that the gains rely heavily on best of N selection which SOM uses both to pick actions and to compute critic targets. The appendix reports that with N = 1 the returns drop sharply (for example 96.5% on Hopper) and SOM falls below SAC on four of five tasks but the main text describes best of N only as a reranking step. Since MVP also uses best of N and is run with different settings (learning rates, warm-up, no exploration noise) so it is a bit unclear to me how much of the gain comes from the new update rule itself. The frozen critic experiment compares SOM and MVP under a shared critic but only when fine tuning from a frozen checkpoint and with best of N kept on.
  2. The baselines also look weaker than usual (for example SAC on Humanoid and Hopper) and the UTD ratio is given only for SOM but not for the baselines.

It would be useful to see MVP trained with SOM's settings and critic with and without best of N even on two tasks such as Hopper and Humanoid and to know the UTD ratio used for each baseline which would show whether the gains come from the update rule rather than from selection and setup.

Questions for authors
  1. Do the baselines use the same critic as SOM (distributional, RMS-normalized) and what UTD ratio does each use? How were MVP's settings chosen for the online setting?
  2. Table 16 shows runs with αmin\alpha_{min} = 1 so what returns do they reach given that alpha sits at the lower bound for much of training on HalfCheetah and Ant?
  3. As Appendix F says that the estimator is only checked at t = 0 where it reduces to the critic gradient. Could you report its error at intermediate tt on the 2D bandit where the exact score can be computed?
  4. What is the training wall clock time of SOM compared with SDAC and MVP?
9 comments
AuthorsOpenReview ↗

Reply to Reviewer oRFV (1/7)

We sincerely thank the reviewer for the favorable evaluation of our work, as well as for the time spent on our manuscript and the insightful comments and questions. All of the reviewer's points have been addressed in the revised manuscript, where new or modified content appears in blue; for each response below, we indicate where in the manuscript the change is located. Throughout this response, table, figure, and section numbers follow the revised manuscript unless otherwise noted, and tables report the mean of the last five evaluations ±\pm 95%95\% CI over five seeds, with bold indicating the highest mean.

Response to Question 1 & Weakness 2.

Do the baselines use the same critic as SOM (distributional, RMS-normalized)?

In Table 1, each baseline uses its original critic (a scalar Q-function, except DACER, which is already distributional), whereas SOM uses a Gaussian distributional critic with first-layer RMSNorm. To test whether SOM's gains come from the critic, we retrained all eight off-policy baselines with SOM's critic, keeping all other settings as in Table 1, and added the results to Appendix H.6 (Table 25).

Table R[1]. All methods with SOM's critic (Table 25; SOM row from Table 1). Parentheses: rank among the nine methods; bold: highest.

AlgorithmAnt-v4HalfCheetah-v4Hopper-v4Humanoid-v4Walker2d-v4Avg. rank
DACER7100.3±951.2\mathbf{7100.3 \pm 951.2} (1)14615.1±200.414615.1 \pm 200.4 (3)3281.5±584.8\mathbf{3281.5 \pm 584.8} (1)8178.7±423.08178.7 \pm 423.0 (2)5121.1±524.65121.1 \pm 524.6 (5)2.42.4
DIPO6986.9±235.56986.9 \pm 235.5 (2)12436.4±682.312436.4 \pm 682.3 (5)1489.5±120.41489.5 \pm 120.4 (9)4638.5±1002.24638.5 \pm 1002.2 (7)5540.8±307.85540.8 \pm 307.8 (4)5.45.4
DPMD4978.8±2755.44978.8 \pm 2755.4 (7)11291.3±972.611291.3 \pm 972.6 (7)3061.3±487.23061.3 \pm 487.2 (3)6270.8±616.86270.8 \pm 616.8 (4)4788.4±632.94788.4 \pm 632.9 (6)5.45.4
MVP5904.5±749.65904.5 \pm 749.6 (5)8090.9±1271.28090.9 \pm 1271.2 (8)2223.5±934.32223.5 \pm 934.3 (6)4950.5±284.14950.5 \pm 284.1 (5)3228.9±1255.63228.9 \pm 1255.6 (9)6.66.6
QSM2693.1±739.72693.1 \pm 739.7 (9)11736.0±161.111736.0 \pm 161.1 (6)2113.1±516.82113.1 \pm 516.8 (7)945.9±179.2945.9 \pm 179.2 (9)4376.5±415.14376.5 \pm 415.1 (8)7.87.8
QVPO5285.1±565.25285.1 \pm 565.2 (6)6268.6±999.66268.6 \pm 999.6 (9)2965.3±365.12965.3 \pm 365.1 (4)2176.3±983.32176.3 \pm 983.3 (8)4758.0±1010.24758.0 \pm 1010.2 (7)6.86.8
SAC6053.7±1530.86053.7 \pm 1530.8 (4)14119.1±1370.514119.1 \pm 1370.5 (4)2021.7±286.32021.7 \pm 286.3 (8)4765.7±1101.14765.7 \pm 1101.1 (6)6072.8±424.46072.8 \pm 424.4 (2)4.84.8
SDAC3783.8±753.83783.8 \pm 753.8 (8)16690.6±149.6\mathbf{16690.6 \pm 149.6} (1)2807.8±471.42807.8 \pm 471.4 (5)6325.3±676.16325.3 \pm 676.1 (3)5761.9±141.95761.9 \pm 141.9 (3)4.04.0
SOM (Ours)6947.1±552.56947.1 \pm 552.5 (3)16543.7±165.816543.7 \pm 165.8 (2)3122.2±445.43122.2 \pm 445.4 (2)8451.7±310.1\mathbf{8451.7 \pm 310.1} (1)6109.3±126.9\mathbf{6109.3 \pm 126.9} (1)1.8\mathbf{1.8}

SOM's critic substantially improves several baselines, but with the same critic SOM still has the best average rank. The critic design thus contributes to performance, yet it does not account for SOM's overall advantage.

What UTD ratio does each method use?

Following the reviewer's comment, we now report the update schedule of every method in Appendix H.7 (Table 26). All off-policy methods, including SOM, share one training loop: each iteration steps five parallel environments once and performs one gradient update with batch size 256256, i.e., 0.20.2 updates per environment step (about 11M updates over 55M steps).

Table R[2]. Update schedule of the off-policy methods in Table 1 (Table 26).

MethodParallel envsUpdates per env stepBatch sizeGradient updates (5M steps)
SOM550.20.22562560.980.98M
SDAC, DPMD, QSM, DIPO, DACER, QVPO, SAC, MVP550.20.22562560.990.99M

All methods therefore use the same UTD ratio; SOM performs slightly fewer updates because of its longer warm-up (10510^5 vs. 3×1043\times10^4 steps; Table 23). SOM's gains thus do not come from a higher UTD ratio or more gradient updates. We apologize for not stating these update schedules clearly in the submission.

Baseline returns (Weakness 2).

Standard SAC performs one update per environment step, whereas all methods here share 0.20.2 updates per step. This lower shared budget is the main difference from the standard SAC setup and a likely reason why some absolute baseline returns (e.g., SAC on Humanoid and Hopper) are below commonly reported values; since every method, including SOM, uses the same budget, the comparison remains fair.

AuthorsOpenReview ↗

Reply to Reviewer oRFV (2/7)

How were MVP's settings chosen for the online setting?

MVP was proposed for offline-to-online RL, so no configuration for purely online MuJoCo training existed. We followed the official MVP implementation (batch size 256256, best-of-3232, no additional exploration noise) within the training loop and critic shared by all baselines (twin critics with clipped double-Q, 0.20.2 updates per environment step, warm-up 3×1043\times10^4). The only remaining difference from the official code is the learning rate (10−410^{-4} instead of 3×10−43\times10^{-4}), and restoring it gives nearly the same results. We added this sensitivity study to Appendix H.8 (Table 27).

Table R[3]. MVP under different settings (Table 27; mean over five seeds).

MVP settingAnt-v4HalfCheetah-v4Hopper-v4Humanoid-v4Walker2d-v4Avg. % of SOM
Table 1 setting4156415671347134460460540854083256325647%47\%
learning rate 3×10−43\times10^{-4}4140414069116911852852537453742900290048%48\%
exploration noise60706070106731067325202520572057204345434574%74\%
warm-up 10510^5616861686138613810791079558355833663366357%57\%
N=16N=163804380454905490477477397639761907190736%36\%
N=64N=645060506078917891832832510651062886288651%51\%
lr + noise + warm-up56565656101191011917391739615361534525452569%69\%
matched to SOM7512\mathbf{7512}110231102326692669821882184128412885%85\%
SOM (Table 1)6947694716544\mathbf{16544}3122\mathbf{3122}8452\mathbf{8452}6109\mathbf{6109}100%100\%

Unless noted otherwise, each row changes one factor of the Table 1 setting (learning rate 10−410^{-4}, no exploration noise, warm-up 3×1043\times10^4, best-of-3232, MVP's own critic):

  • learning rate 3×10−43\times10^{-4}: the official MVP value for actor and critic.
  • exploration noise: SOM's Gaussian exploration noise (scaled by the learned entropy coefficient), added to the action after best-of-NN selection.
  • warm-up 10510^5: SOM's number of initial random-action steps.
  • N=16N=16 / N=64N=64: the number of best-of-NN candidates.
  • lr + noise + warm-up: the three changes above combined; MVP's critic and losses are unchanged.
  • matched to SOM: lr + noise + warm-up, plus SOM's critic (Gaussian distributional with first-layer RMSNorm) and TD target (next action from the target actor); only MVP's actor update rule remains.

MVP benefits considerably from exploration noise and from the SOM-matched configuration, yet SOM remains higher on four of the five tasks.

AuthorsOpenReview ↗

Reply to Reviewer oRFV (3/7)

Response to Weakness 1.

Response to Weakness 1: Dependence on best-of-NN selection.

We agree that best-of-NN selection contributes substantially: without it, SOM falls below SAC on four of the five tasks (Tables 1 and 21). This is common to selection-based methods rather than specific to SOM, as Table R[4] shows for every method that uses best-of-NN selection.

Table R[4]. Training N=1N=1 / evaluation N=1N=1 for all best-of-NN methods (Table 22), with SAC, which does not use selection, from Table 1 for reference. Bold: highest among the best-of-NN methods.

AlgorithmAnt-v4HalfCheetah-v4Hopper-v4Humanoid-v4Walker2d-v4
DPMD774.7±94.1774.7 \pm 94.1881.5±212.9881.5 \pm 212.9334.5±80.2334.5 \pm 80.2182.2±37.3182.2 \pm 37.3332.6±101.8332.6 \pm 101.8
MVP (Table 1 setting)965.7±14.1965.7 \pm 14.1−2.4±0.8-2.4 \pm 0.852.4±38.152.4 \pm 38.1150.4±28.4150.4 \pm 28.421.6±15.821.6 \pm 15.8
QSM−1295.6±30.9-1295.6 \pm 30.9−240.3±2.5-240.3 \pm 2.5263.5±31.5263.5 \pm 31.5107.5±16.5107.5 \pm 16.5−3.7±0.6-3.7 \pm 0.6
QVPO−484.5±463.4-484.5 \pm 463.4−81.6±76.0-81.6 \pm 76.017.7±13.017.7 \pm 13.0138.8±24.7138.8 \pm 24.7−6.4±4.8-6.4 \pm 4.8
SDAC46.1±42.846.1 \pm 42.89344.6±106.3\mathbf{9344.6 \pm 106.3}1014.5±817.8\mathbf{1014.5 \pm 817.8}5806.7±106.85806.7 \pm 106.83809.1±62.1\mathbf{3809.1 \pm 62.1}
SOM2307.7±1256.6\mathbf{2307.7 \pm 1256.6}3788.7±3066.73788.7 \pm 3066.7107.8±78.6107.8 \pm 78.67516.5±449.8\mathbf{7516.5 \pm 449.8}3464.0±2082.53464.0 \pm 2082.5
SAC (no selection)5545.1±1089.05545.1 \pm 1089.013625.4±301.213625.4 \pm 301.22297.4±171.42297.4 \pm 171.44532.3±1483.74532.3 \pm 1483.75716.2±259.35716.2 \pm 259.3

Without selection, every best-of-NN method falls below SAC on at least four tasks; SOM and SDAC remain the strongest among them, and SOM on Humanoid exceeds every Table 1 baseline even without selection.

Selection is used both for acting and for computing critic targets, as stated in Section 4.5, which now also points to the analysis of training- and evaluation-time selection in Appendix H.4. We isolate the contribution of the update rule from that of selection in the next response.

Reviewer_U52uOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work proposed a policy extraction method that trains a one-step policy by taking Q-function gradients from a set of samples around a noised action. This is used to get the target velocity field to train the one-step policy. This is in contrast to other policy-extraction approaches such as Best-of-N sampling or direct gradients.

Critical strengths
  • Reasonable level of analysis
  • Good writing
Critical weaknesses
  • DQS appears close to this work [albeit not one-step] and is not cited (https://arxiv.org/abs/2410.01312). This work seems to do a similar sampling around noised actions to get a Q-weighted average gradient.
  • Relatively narrow set of benchmarks
  • Quite high training compute cost in terms of Q-function evaluation it appears
Questions for authors
  • How does the method compare to DQS?
  • How does the method perform w/<32 samples?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer U52u (1/3)

We thank the reviewer for the positive assessment and helpful suggestions, which have improved the manuscript. We have addressed each point in the revised manuscript, with all modifications marked in blue, and each response below gives the section of the corresponding change. Table, figure, and section numbers refer to the revised manuscript unless stated otherwise; reported values are the mean of the last five evaluations ±\pm 95%95\% CI over five seeds, with the highest mean in bold.

Response to Question 1 & Weakness 1, Weakness 2.

We have added DQS to Section 2 (Related Work).
Comparison with DQS. Both methods build on the iDEM Monte-Carlo score estimator, which forms a Q-weighted average of action gradients over noised candidate actions. Our contribution is therefore not the estimator but how the estimated score is controlled and turned into a policy: SOM converts it into a probability-flow velocity target, so that a one-step MeanFlow policy is trained toward the Boltzmann policy within online actor-critic learning, without target-distribution samples or distillation from a multi-step teacher. DQS instead regresses a diffusion score network onto the estimated score and generates actions through 10001000-step SDE integration.

Table R[1]. Design comparison between DQS and SOM.

DQSSOM
Score estimatoriDEM Monte-Carlo estimator, K=1000K=1000iDEM Monte-Carlo estimator, K=100K=100
Temperaturefixed (T=0.05T=0.05 for all tasks)adapted online by ESS feedback with an annealed target
Policydiffusion score network; 10001000-step SDE at inferenceone-step MeanFlow policy trained on the estimated score
Action selectionsingle samplebest-of-3232 with critic reranking

(i) DeepMind Control Suite. This also addresses the concern about benchmark breadth (Weakness 2). In addition to the nine MuJoCo tasks in Section 5.1, which include swimming, manipulation, and balancing tasks (Swimmer, Pusher, Reacher, and InvertedDoublePendulum) as well as locomotion, we trained SOM on the eight DMC tasks used by DQS under its protocol (250250k environment steps, one update per step, two-layer 256256-unit networks, 1010 seeds, and 100100 evaluation episodes per seed), with the same reward scale as DQS. SOM uses a single setting for all eight tasks (Appendix I, Table 29).

Table R[2]. DeepMind Control Suite at 250250k steps (1010 seeds, 100100 evaluation episodes each). SOM entries are mean ±\pm 95%95\% CI over seeds. Baseline entries are the means reported in DQS [1]. Bold: highest mean per task.

TaskSOMDQSQSMDACERDIPOSACTD3
cheetah-run593.8±23.1593.8 \pm 23.1683.6\mathbf{683.6}607.2607.2599.6599.6651.0651.0588.9588.9592.1592.1
finger-spin898.3±134.5\mathbf{898.3 \pm 134.5}835.0835.0874.5874.5486.7486.7830.4830.4854.0854.0835.5835.5
finger-turn_hard616.3±77.9\mathbf{616.3 \pm 77.9}361.5361.5297.7297.7142.7142.7113.3113.3169.6169.6319.8319.8
fish-swim289.4±61.2\mathbf{289.4 \pm 61.2}221.7221.7204.2204.2158.8158.8129.8129.8111.1111.1106.1106.1
reacher-hard837.8±57.6837.8 \pm 57.6974.1\mathbf{974.1}887.3887.3821.9821.9894.9894.9666.1666.1883.6883.6
hopper-hop9.4±10.99.4 \pm 10.90.70.741.041.024.024.00.00.056.5\mathbf{56.5}21.921.9
quadruped-run488.1±135.0\mathbf{488.1 \pm 135.0}277.6277.6293.1293.1466.8466.8126.1126.1308.6308.6381.8381.8
walker-run724.1±20.8\mathbf{724.1 \pm 20.8}679.2679.2606.0606.0602.1602.1518.5518.5569.3569.3548.1548.1

SOM has a higher mean than DQS on six tasks, while DQS is better on cheetah-run and reacher-hard; among all seven methods, SOM has the highest mean on five tasks. Because of the limited rebuttal period, the baseline values are taken from [1]; the final version will include DQS and the remaining Table 1 baselines (SDAC, DPMD, QVPO, MVP, and PPO), run by us under the same protocol.

(ii) Computational cost. SOM is about 1515 times faster than DQS per training cycle and about 125125 times faster at inference (Table 2), because it replaces 10001000-step SDE integration with a single network evaluation (details in our response to Weakness 3).

(iii) MuJoCo. We are also running the official DQS implementation on all five MuJoCo tasks (55M environment steps, five seeds each), in the same training loop as the other baselines with its official hyperparameters. DQS has the slowest training cycle of the ten methods (Table R[7]), so each 55M-step run takes about two days. We will report the results during the discussion period once they are complete, and otherwise in the final version.

Update. We have posted the DQS results on MuJoCo obtained so far in a separate comment below, "Reply to Reviewer U52u: Follow-up to Question 1 (iii)," and we would be grateful if the reviewer could take a look.

[1] Jain, Vineet, Tara Akhound-Sadegh, and Siamak Ravanbakhsh. "Sampling from energy-based policies using diffusion." arXiv preprint arXiv:2410.01312 (2024).

AuthorsOpenReview ↗

Reply to Reviewer U52u (2/3)

Response to Question 2.

We thank the reviewer for this question.

Fewer candidates (Table 21, Appendix H.4). Retraining SOM with the same NN during training and evaluation, N=16N=16 stays close to N=32N=32 on all five tasks, N=8N=8 remains close on most tasks but drops sharply on Hopper, and N=1N=1 lowers the return substantially; N=64N=64 brings no consistent gain, so we use N=32N=32.

Training-time versus evaluation-time selection. To locate where selection matters, we added a 2×22\times2 ablation to Appendix H.4 (Table 22, referenced from Section 4.5). It removes selection at training time (data collection and critic-target computation), at evaluation time, or both, for SOM and every baseline that uses best-of-NN selection. Policies with training N=1N=1 are trained from scratch; SAC, DIPO, DACER, and PPO do not use selection and are unaffected. The train N=32N=32 / eval N=32N=32 condition is Table 1.

Table R[3]. Train N=32N=32 / Eval N=1N=1.

AlgorithmAnt-v4HalfCheetah-v4Hopper-v4Humanoid-v4Walker2d-v4
DPMD5767.8±183.2\mathbf{5767.8 \pm 183.2}9616.8±951.0\mathbf{9616.8 \pm 951.0}1988.9±372.11988.9 \pm 372.13506.5±742.5\mathbf{3506.5 \pm 742.5}4137.5±383.54137.5 \pm 383.5
MVP3890.6±944.83890.6 \pm 944.85277.7±717.85277.7 \pm 717.8398.1±42.6398.1 \pm 42.61010.3±154.01010.3 \pm 154.01144.2±406.41144.2 \pm 406.4
QSM−1154.4±75.6-1154.4 \pm 75.6−178.2±92.0-178.2 \pm 92.0154.8±71.7154.8 \pm 71.795.5±12.495.5 \pm 12.4−5.0±0.5-5.0 \pm 0.5
QVPO3578.3±675.53578.3 \pm 675.54469.9±1138.64469.9 \pm 1138.62500.5±397.0\mathbf{2500.5 \pm 397.0}613.3±142.8613.3 \pm 142.81632.2±608.51632.2 \pm 608.5
SDAC74.9±67.674.9 \pm 67.62811.3±559.82811.3 \pm 559.8796.4±84.3796.4 \pm 84.3427.2±43.2427.2 \pm 43.2196.0±43.7196.0 \pm 43.7
SOM2865.7±1387.52865.7 \pm 1387.51832.3±743.61832.3 \pm 743.62001.6±361.32001.6 \pm 361.31565.8±245.71565.8 \pm 245.74474.9±1319.9\mathbf{4474.9 \pm 1319.9}

Table R[4]. Train N=1N=1 / Eval N=32N=32.

AlgorithmAnt-v4HalfCheetah-v4Hopper-v4Humanoid-v4Walker2d-v4
DPMD826.9±108.0826.9 \pm 108.01452.7±209.01452.7 \pm 209.0401.8±149.5401.8 \pm 149.5202.0±34.7202.0 \pm 34.7351.9±80.8351.9 \pm 80.8
MVP981.5±10.9981.5 \pm 10.91.9±2.01.9 \pm 2.0323.0±286.2323.0 \pm 286.2286.1±48.4286.1 \pm 48.4128.6±66.1128.6 \pm 66.1
QSM5.3±1.25.3 \pm 1.25577.6±83.95577.6 \pm 83.9382.1±41.2382.1 \pm 41.24534.8±373.84534.8 \pm 373.8288.6±18.7288.6 \pm 18.7
QVPO−461.8±457.2-461.8 \pm 457.2−53.0±98.9-53.0 \pm 98.917.8±13.217.8 \pm 13.2142.4±24.3142.4 \pm 24.3−5.4±5.7-5.4 \pm 5.7
SDAC558.7±436.7558.7 \pm 436.711949.6±124.0\mathbf{11949.6 \pm 124.0}1574.0±1372.1\mathbf{1574.0 \pm 1372.1}6272.3±283.86272.3 \pm 283.84779.4±308.5\mathbf{4779.4 \pm 308.5}
SOM4693.6±323.3\mathbf{4693.6 \pm 323.3}10255.3±926.310255.3 \pm 926.3374.8±206.4374.8 \pm 206.47623.0±686.4\mathbf{7623.0 \pm 686.4}4429.8±2143.64429.8 \pm 2143.6

Table R[5]. Train N=1N=1 / Eval N=1N=1 (no selection).

AlgorithmAnt-v4HalfCheetah-v4Hopper-v4Humanoid-v4Walker2d-v4
DPMD774.7±94.1774.7 \pm 94.1881.5±212.9881.5 \pm 212.9334.5±80.2334.5 \pm 80.2182.2±37.3182.2 \pm 37.3332.6±101.8332.6 \pm 101.8
MVP965.7±14.1965.7 \pm 14.1−2.4±0.8-2.4 \pm 0.852.4±38.152.4 \pm 38.1150.4±28.4150.4 \pm 28.421.6±15.821.6 \pm 15.8
QSM−1295.6±30.9-1295.6 \pm 30.9−240.3±2.5-240.3 \pm 2.5263.5±31.5263.5 \pm 31.5107.5±16.5107.5 \pm 16.5−3.7±0.6-3.7 \pm 0.6
QVPO−484.5±463.4-484.5 \pm 463.4−81.6±76.0-81.6 \pm 76.017.7±13.017.7 \pm 13.0138.8±24.7138.8 \pm 24.7−6.4±4.8-6.4 \pm 4.8
SDAC46.1±42.846.1 \pm 42.89344.6±106.3\mathbf{9344.6 \pm 106.3}1014.5±817.8\mathbf{1014.5 \pm 817.8}5806.7±106.85806.7 \pm 106.83809.1±62.1\mathbf{3809.1 \pm 62.1}
SOM2307.7±1256.6\mathbf{2307.7 \pm 1256.6}3788.7±3066.73788.7 \pm 3066.7107.8±78.6107.8 \pm 78.67516.5±449.8\mathbf{7516.5 \pm 449.8}3464.0±2082.53464.0 \pm 2082.5

What the ablation shows. For SOM, most of the loss at N=1N=1 comes from removing selection during training: selection only at evaluation (Table R[4]) recovers much of the Table 1 return on four of the five tasks. Even without any selection (Table R[5]), SOM on Humanoid still outperforms every baseline in Table 1, including those that use best-of-3232 selection. Selection also matters for the other best-of-NN methods: removing it at evaluation sharply lowers the returns of SDAC and MVP (Table R[3]), and without selection during training (Tables R[4] and R[5]), SOM and SDAC remain the strongest, while the other methods largely fail.

AuthorsOpenReview ↗

Reply to Reviewer U52u (3/3)

Response to Weakness 3.

We thank the reviewer for raising this point and agree that the training cost should be reported explicitly. Each update evaluates Q^\widehat Q and its action gradient at K=100K=100 candidates per state. To measure what this costs, we timed SOM's training cycle while varying only KK, using the same harness as Table 2.

Table R[6]. SOM training-cycle time relative to K=1K=1 (Table 18, Appendix H.1). Bold: default.

KKRelative to K=1K=1
111.001.00
10101.191.19
50501.311.31
100\mathbf{100}1.52\mathbf{1.52}
2502502.772.77

Using K=100K=100 makes a training cycle about 1.521.52 times as long as with K=1K=1; this is SOM's main training cost beyond MVP. Inference does not use these candidates, and its latency is unchanged.

Setup. We re-measured ten methods on a single machine, including DQS, which the reviewer kindly brought to our attention, and CGPO, suggested by another reviewer, and report the result as Table 2 (Section 5.1). A training cycle comprises action selection, one environment step, replay operations, one batch-256256 update, and device synchronization; inference selects one action for one observation. Values are mean ±\pm approximate 95%95\% CI over 3030 timing blocks (protocol in Appendix H.1).

Table R[7]. Training-cycle and inference times in ms (Table 2). TdT_d: generation steps; NinfN_{\mathrm{inf}}: inference candidates. Parentheses give the rank by mean in each column (1 = fastest), with methods whose 95%95\% CIs overlap sharing a rank; bold marks the fastest.

MethodTdT_dNinfN_{\mathrm{inf}}Training cycle (ms)Inference (ms)
SOM1132324.080±0.0884.080 \pm 0.088 (3)0.138±0.002\mathbf{0.138 \pm 0.002} (1)
QSM202032323.534±0.0513.534 \pm 0.051 (2)0.465±0.0050.465 \pm 0.005 (4)
DPMD202032324.889±0.0864.889 \pm 0.086 (4)0.610±0.0060.610 \pm 0.006 (5)
DIPO100100118.219±0.0698.219 \pm 0.069 (6)1.690±0.0131.690 \pm 0.013 (8)
SDAC202032327.088±0.0147.088 \pm 0.014 (5)0.611±0.0070.611 \pm 0.007 (5)
DACER2020119.759±0.1189.759 \pm 0.118 (7)0.443±0.0040.443 \pm 0.004 (3)
QVPO2020323244.905±0.07444.905 \pm 0.074 (9)0.610±0.0050.610 \pm 0.005 (5)
MVP1132322.788±0.121\mathbf{2.788 \pm 0.121} (1)0.153±0.0010.153 \pm 0.001 (2)
CGPO2020323235.458±0.10635.458 \pm 0.106 (8)5.709±0.0175.709 \pm 0.017 (9)
DQS100010001159.559±0.16559.559 \pm 0.165 (10)17.256±0.04817.256 \pm 0.048 (10)

Training cost. SOM's training cycle is 42%42\% shorter than that of SDAC; it is slower than MVP and QSM but faster than DPMD and the remaining baselines.

In summary, the K=100K=100 Q-function evaluations are a real cost: they make SOM's training cycle about 1.51.5 times as long as MVP's (Table R[6]). Even so, SOM trains faster than every multi-step diffusion baseline except QSM, and its one-step policy gives the lowest inference latency.

Reviewer_WKhmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SOM, an off-policy actor-critic algorithm combining iDEM-style score estimation with one-step MeanFlow. The target velocity field is constructed from Q-function gradients, while an ESS-based controller adjusts inverse temperatures across diffusion timesteps. Experiments on MuJoCo benchmarks demonstrate strong performance and reduced inference latency.

Critical strengths
  1. The paper addresses the computational overhead of multi-step generative policies in online reinforcement learning.

  2. Combining critic-gradient-based score estimation with one-step MeanFlow is an interesting direction.

  3. The empirical results are promising, particularly on Humanoid, with extensive ablation studies.

Critical weaknesses
  1. Limited methodological novelty and excessive engineering components.

The method largely combines existing techniques, including iDEM-based score estimation [1], MeanFlow [2], distributional critics, and Best-of-N selection. The additional ESS controller and annealing schedule introduce further complexity, while their independent contributions and necessity remain insufficiently justified.

  1. Strong dependence on Best-of-N selection.

The method heavily relies on Best-of-N selection with (N=32). Appendix H.4 shows a 96.5% performance drop on Hopper when (N=1). Although each candidate requires only one network evaluation, the effectiveness of the standalone one-step policy remains questionable.

  1. Insufficient justification for ESS adaptation.

The time-dependent inverse temperatures break the probability-flow interpretation associated with a single Boltzmann target. Moreover, ESS measures weight concentration rather than critic reliability, making the proposed annealing schedule largely heuristic. Some simpler configurations also outperform the default setting on individual tasks.

  1. Incomplete computational efficiency evaluation.

Score estimation requires Q-value evaluations and action gradients for (K=100) candidates during training. However, the paper mainly emphasizes inference latency without adequately analyzing training overhead or overall computational efficiency.

  1. Missing recent related work.

The paper overlooks CGPO [3], FPO [4], FPO++ [5], GenPO [6], and GenPO++ [7]. In particular, CGPO also exploits critic guidance for generative policy improvement. A more comprehensive discussion of these methods would help clarify SOM's novelty and technical positioning.

  1. Limited evaluation scope. The experiments are restricted to simulated MuJoCo control tasks and synthetic bandit environments. No real-world robotic experiments are provided, leaving the practical applicability of SOM to physical robotic systems unverified.

[1] Akhound-Sadegh et al. Iterated Denoising Energy Matching for Sampling from Boltzmann Densities. ICML, 2024.

[2] Geng et al. Mean Flows for One-step Generative Modeling. NeurIPS, 2025.

[3] Ding et al. Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance. ICML, 2026.

[4] McAllister et al. Flow Matching Policy Gradients. ICLR, 2026.

[5] Yi et al. Flow Policy Gradients for Robot Control. arXiv:2602.02481, 2026.

[6] Ding et al. GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning. NeurIPS, 2025.

[7] Hu et al. GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios. arXiv:2606.06967, 2026.

Questions for authors
  1. Can the authors provide more controlled experiments isolating the contributions of iDEM-based score estimation, ESS adaptation, and Best-of-N selection?

  2. What is the actual training overhead introduced by the (K=100) candidate-based score estimation? Can the authors report wall-clock training time and computational costs compared with existing methods?

  3. What motivates the specific ESS annealing schedule (50 → 5)? Can the authors provide stronger justification beyond empirical hyperparameter tuning?

  4. Given the substantial performance degradation when reducing Best-of-N to (N=1), how much of SOM's improvement comes from the learned one-step policy itself rather than critic-based action selection?

9 comments
AuthorsOpenReview ↗

Reply to Reviewer WKhm (1/7)

We thank the reviewer for the careful reading of our manuscript and the thoughtful, constructive feedback. We have incorporated the reviewer's suggestions into the revised manuscript, where all changes are highlighted in blue, and below we indicate the section in which each change appears. Unless noted otherwise, table, figure, and section numbers refer to the revised manuscript, tables report the mean of the last five evaluations ±\pm 95%95\% CI over five seeds, and bold marks the highest mean.

Response to Question 1 & Weakness 1, Weakness 2

Novelty (Weakness 1). iDEM estimates Boltzmann scores to train a diffusion sampler, and MeanFlow learns a one-step generator from samples of the target distribution. In online RL, samples from the target policy π(a∣s)∝exp⁡(αQ(s,a))\pi(a\mid s)\propto\exp(\alpha Q(s,a)) are unavailable. SOM's contribution is to connect the two: it converts critic-based score estimates into probability-flow velocity targets, so that a one-step MeanFlow policy is trained toward the Boltzmann policy without target samples, a multi-step teacher, or distillation. The distributional critic and best-of-NN selection are not part of this contribution: best-of-NN is also used by SDAC, DPMD, QSM, QVPO, and MVP, and when all baselines use SOM's critic, SOM still has the best average rank (Table 25, Appendix H.6).

(i) Score estimator. Following the reviewer's suggestion, we added a comparison of score estimators within SOM (Appendix B.2, referenced from Section 4.2). To isolate the estimator, we disabled ESS control (α=15\alpha=15) and replaced iDEM with three alternatives, keeping all other components fixed: a DSI importance-sampling estimator [8], RDMC-MALA [9], and the reweighted score matching of SDAC (SDAC-RSM) [10].

Table R[1]. Score estimators within SOM (α=15\alpha=15, no ESS control; Table 4).

EstimatorHalfCheetah-v4Ant-v4Walker2d-v4Hopper-v4Humanoid-v4
iDEM12897±589512897 \pm 58957141±1677141 \pm 1675699±2335699 \pm 2332365±2442365 \pm 2448276±261\mathbf{8276 \pm 261}
DSI16197±21416197 \pm 2147011±1987011 \pm 1985667±4335667 \pm 4333143±618\mathbf{3143 \pm 618}8183±1928183 \pm 192
RDMC-MALA16242±165\mathbf{16242 \pm 165}7182±140\mathbf{7182 \pm 140}6027±69\mathbf{6027 \pm 69}2533±5132533 \pm 5137711±2997711 \pm 299
SDAC-RSM15927±31215927 \pm 3126889±3306889 \pm 3306011±1996011 \pm 1992548±3442548 \pm 3448068±1148068 \pm 114

No estimator is best on every task, so we do not claim that iDEM is uniformly superior. iDEM's weaker HalfCheetah result comes from a single collapsed seed, an instability that ESS control removes (Table R[2]). SOM's velocity supervision thus works with score estimators from prior work; the contribution lies in this construction and its ESS control rather than in a specific estimator.

[8] He et al. Training neural samplers with reverse diffusive KL divergence. arXiv:2410.12456, 2024.
[9] Huang et al. Reverse diffusion Monte Carlo. ICLR, 2024.
[10] Ma et al. Efficient online reinforcement learning for diffusion policy. arXiv:2502.00361, 2025.

AuthorsOpenReview ↗

Reply to Reviewer WKhm (2/7)

(ii) ESS adaptation. ESS feedback controls candidate-weight concentration under an evolving critic, and annealing sets how much concentration is allowed over training. In Appendix B.4, we added the fixed targets 11.211.2 and 15.815.8 and the best fixed target per task to Table 6.

Table R[2]. Fixed target ESS with temperature adaptation enabled (Table 6). The last row gives the best fixed target per task.

Target ESSHalfCheetah-v4Ant-v4Walker2d-v4Hopper-v4Humanoid-v4
Ours (50→550 \rightarrow 5)16544±166\mathbf{16544 \pm 166}6947±5536947 \pm 5536109±127\mathbf{6109 \pm 127}3122±445\mathbf{3122 \pm 445}8452±3108452 \pm 310
Keff⋆=1K_{\mathrm{eff}}^{\star} = 1−127±661-127 \pm 661−2923±90-2923 \pm 90−0±16-0 \pm 163±13 \pm 170±1070 \pm 10
Keff⋆=5K_{\mathrm{eff}}^{\star} = 516095±65916095 \pm 6597168±376\mathbf{7168 \pm 376}5856±1805856 \pm 1802248±11072248 \pm 11078660±377\mathbf{8660 \pm 377}
Keff⋆=11.2K_{\mathrm{eff}}^{\star} = 11.216460±35216460 \pm 3526585±6446585 \pm 6445994±1315994 \pm 1313071±7983071 \pm 7987773±5467773 \pm 546
Keff⋆=15.8K_{\mathrm{eff}}^{\star} = 15.814211±189914211 \pm 18996758±3066758 \pm 3066028±1496028 \pm 1492814±7322814 \pm 7327862±1027862 \pm 102
Keff⋆=30K_{\mathrm{eff}}^{\star} = 3016042±62516042 \pm 6256453±2066453 \pm 2065707±4955707 \pm 4952869±3282869 \pm 3287316±2817316 \pm 281
Keff⋆=60K_{\mathrm{eff}}^{\star} = 6015671±34915671 \pm 3496187±3096187 \pm 3095827±1055827 \pm 1052506±4932506 \pm 4937273±1787273 \pm 178
Keff⋆=100K_{\mathrm{eff}}^{\star} = 10015819±41015819 \pm 4106359±1386359 \pm 1385646±2205646 \pm 2202919±3052919 \pm 3057276±1917276 \pm 191
Best fixed ESS per env.11.211.25515.815.811.211.255

As Table 5 (Appendix B.3) shows, the best fixed α\alpha differs across tasks, and a single fixed ESS target comes close to the per-task best α\alpha on most tasks. Fixed ESS targets also have task-dependent optima (Table R[2]), and, as with α\alpha, an extreme setting fails: Keff⋆=1K_{\mathrm{eff}}^{\star}=1, which puts all weight on a single candidate, collapses on every task, just as an overly large α\alpha does. Away from this extreme, however, every target from 55 to 100100 remains usable (Table R[2]), whereas fixed α\alpha already breaks training on several tasks at α=60\alpha = 60 (Table 5).

Annealing the target from 5050 to 55 removes the need to choose the ESS target per task: a single schedule stays within a few percent of the best fixed target selected separately for each task (Table R[2]). The controller is a scalar update per diffusion-time bin that reuses the estimator's weights, so its cost is negligible; Question 3 discusses the motivation of the schedule.

(iii) Best-of-NN selection. We acknowledge that selection contributes substantially (Table 21). To separate its training-time and evaluation-time contributions, we added a comparison that disables selection during training, evaluation, or both (Table 22, Appendix H.4); our response to Question 4 discusses the results.

AuthorsOpenReview ↗

Reply to Reviewer WKhm (3/7)

Response to Question 2 and Weakness 4.

We thank the reviewer for pointing out that inference latency alone does not provide a complete assessment of computational efficiency. We agree that the training cost, including the Q-value evaluations and action gradients required by the K=100K=100 score estimator, should also be reported, and we now report both in Section 5.1 and Appendix H.1.

Q: What is the actual training overhead introduced by the K=100K=100 candidate-based score estimation?

We timed SOM's training cycle while varying only KK, using the same harness as Table 2.

Table R[3]. SOM training-cycle time relative to K=1K=1 (Table 18, Appendix H.1). Bold: default.

KKRelative to K=1K=1
111.001.00
10101.191.19
50501.311.31
100\mathbf{100}1.52\mathbf{1.52}
2502502.772.77

Using K=100K=100 makes a training cycle about 1.521.52 times as long as with K=1K=1; this is SOM's main training cost beyond MVP. Inference does not use these candidates, and its latency is unchanged.

Q: Can the authors report wall-clock training time and compare computational costs with existing methods?

Following the reviewer's suggestion, we re-measured ten methods on a single machine, including CGPO, the off-policy baseline you kindly brought to our attention, and DQS, suggested by another reviewer, and report the result as Table 2 (Section 5.1). A training cycle comprises action selection, one environment step, replay operations, one batch-256256 update, and device synchronization; inference selects one action for one observation. Values are mean ±\pm approximate 95%95\% CI over 3030 timing blocks (protocol in Appendix H.1).

Table R[4]. Training-cycle and inference times in ms (Table 2). TdT_d: generation steps; NinfN_{\mathrm{inf}}: inference candidates. Parentheses give the rank by mean in each column (1 = fastest), with methods whose 95%95\% CIs overlap sharing a rank; bold marks the fastest.

MethodTdT_dNinfN_{\mathrm{inf}}Training cycle (ms)Inference (ms)
SOM1132324.080±0.0884.080 \pm 0.088 (3)0.138±0.002\mathbf{0.138 \pm 0.002} (1)
QSM202032323.534±0.0513.534 \pm 0.051 (2)0.465±0.0050.465 \pm 0.005 (4)
DPMD202032324.889±0.0864.889 \pm 0.086 (4)0.610±0.0060.610 \pm 0.006 (5)
DIPO100100118.219±0.0698.219 \pm 0.069 (6)1.690±0.0131.690 \pm 0.013 (8)
SDAC202032327.088±0.0147.088 \pm 0.014 (5)0.611±0.0070.611 \pm 0.007 (5)
DACER2020119.759±0.1189.759 \pm 0.118 (7)0.443±0.0040.443 \pm 0.004 (3)
QVPO2020323244.905±0.07444.905 \pm 0.074 (9)0.610±0.0050.610 \pm 0.005 (5)
MVP1132322.788±0.121\mathbf{2.788 \pm 0.121} (1)0.153±0.0010.153 \pm 0.001 (2)
CGPO2020323235.458±0.10635.458 \pm 0.106 (8)5.709±0.0175.709 \pm 0.017 (9)
DQS100010001159.559±0.16559.559 \pm 0.165 (10)17.256±0.04817.256 \pm 0.048 (10)

Training. SOM is not the cheapest method to train: its training cycle is about 1.51.5 times that of MVP and slightly longer than that of QSM, but shorter than those of DPMD, SDAC, and the remaining baselines. The difference from MVP comes from the K=100K=100 candidates of the score estimator (Table R[3]).

Inference. SOM has the lowest inference latency, and the two one-step methods, SOM and MVP, are several times faster than all multi-step baselines. SOM's efficiency advantage therefore lies in inference rather than training.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses and Additional Results

Dear Reviewers, Area Chair,

We sincerely thank all reviewers for their time and thoughtful feedback. As the discussion period ends today, we would like to briefly summarize our responses and the additional results provided during the rebuttal, in case this is helpful. We fully understand how busy this period is; if time permits, we would be truly grateful if the reviewers could take a brief look at our responses. We would be happy to clarify anything further before the discussion closes.

Novelty and component contributions. SOM's contribution is to convert critic-based score estimates into velocity targets for a one-step MeanFlow policy, without target samples, a multi-step teacher, or distillation. The one-step policy generates each action with a single network evaluation, giving SOM the lowest inference latency among the ten methods (Table 2), and ESS feedback controls the score estimates so that a single setting works across tasks without per-task tuning.

Additional baselines. We compared SOM with CGPO and DQS. On the five MuJoCo tasks of Table 1, SOM attains a higher normalized score than CGPO at a fraction of its training and inference cost. On the MuJoCo tasks completed so far, SOM also outperforms DQS, and on the DeepMind Control Suite, under DQS's own protocol, SOM attains the highest mean return on 55 of the 88 tasks.

Fairness of the comparison. All off-policy methods share the same training loop and update-to-data ratio (0.20.2 updates per environment step). Retraining all eight baselines with SOM's critic improves several of them, but SOM still has the best average rank, so the critic alone does not account for SOM's advantage.

Role of best-of-NN selection. Selection matters for every best-of-NN method, and SOM's update rule contributes on its own. When MVP is matched to SOM's settings and critic and trained without selection, it fails to learn on every task, whereas SOM still learns useful policies on several tasks; without any selection, SOM on Humanoid outperforms every baseline in Table 1.

Computational cost. We re-measured training and inference times for ten methods, including CGPO and DQS, on a single machine (Table 2). Besides its lowest inference latency, SOM's training cycle is shorter than those of all multi-step diffusion baselines except QSM.

ESS control. Annealing the ESS target from 5050 to 55 removes the need to choose the target per task: a single schedule stays within a few percent of the best fixed target selected separately for each task.

Further analysis. We added the score estimator's error at intermediate noise levels on the 2D bandit.

Thank you again for your valuable comments, which have helped us substantially improve the paper. We truly appreciate your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.