acceptodds
Under review as a conference paper at ICLR 2027

Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent

Abstract

Online off-policy reinforcement learning (RL) is governed by two coupled design dimensions: policy representation (e.g. Gaussian vs generative policies) and optimisation methodology (e.g. soft actor-critic (SAC) vs mirror descent (MD)). Gaussian policies offer fast inference and tractable entropy estimation but have limited ability to model multimodal action distributions, whereas generative policies provide richer action distributions at the cost of iterative sampling or intractable entropy estimation. On the optimisation side, SAC-style entropic exploration and MD updates can be interpreted as minimising distinct Kullback-Leibler divergences: SAC-style exploration performs soft policy improvement towards a value-induced Boltzmann distribution, whereas MD constrains updates to remain close to the previous policy. Combining SAC-style entropy and MD regularisation supports exploration and conservative policy updates, but can yield multimodal targets beyond the representational capacity of unimodal Gaussian policies. This mismatch calls for an expressive policy with efficient sampling and explicit entropy control. To this end, we introduce Stochastic MeanFlow Policies (SMFP), a one-step generative policy class that combines MeanFlow-based noise-to-action mappings with Gaussian reparameterisation. The latent mapping represents diverse action modes, while the learnable conditional Gaussian scale yields an analytic entropy lower bound. We use this bound to regulate stochasticity alongside direct value guidance and advantage-weighted MeanFlow regression, making entropy-regularised mirror-descent learning practical with one-step generative policies. Empirically, on seven MuJoCo benchmarks, SMFP achieves strong performance against both Gaussian and generative baselines while maintaining single-step inference efficiency. Complementary experiments demonstrate multimodal action modelling and effective exploration in sparse-reward navigation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Generative Policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dDpJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes SMFP, a one-step MeanFlow policy with a learned Gaussian noise scale for entropy control. It trains the policy using Q-value gradients, an entropy-floor penalty, and advantage-weighted MeanFlow regression. Experiments on MuJoCo, sparse-reward AntMaze, and Push-T examine control performance, exploration, and multimodal action modeling.

Critical strengths
  1. I like the explicit control over the policy’s noise scale. It gives a one-step generative policy a simple way to maintain exploration, and the entropy ablation suggests that this helps learning.

  2. The experimental results are strong across several control tasks. Experiments support the method’s usefulness in sparse-reward settings.

Critical weaknesses
  1. The novelty seems somewhat limited. EXPO [1], GoldenStart [2], and QAM’s Gaussian-edit variant [3] have related ideas about stochastic generative policies and entropy control. A more comparison and discussion would help explain what SMFP adds.

  2. The connection to mirror descent could be clearer. The practical loss combines positive-advantage-weighted MeanFlow regression with a separate entropy-floor penalty. It is unclear how closely this follows the motivating entropy-regularized MD objective, which makes the theoretical contribution harder to assess.

[1] EXPO: Stable Reinforcement Learning with Expressive Policies. [2] GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies. [3] Q-learning with Adjoint Matching.

Questions for authors

Figure 8(a) shows that K = 1 already performs quite well. Could the authors discuss when sampling more candidates is useful and why? In particular, when do the gains justify the additional cost, and when would a smaller K be sufficient?

3 comments
AuthorsOpenReview ↗

Clarification of Methodological Contributions and Empirical Evidence

We thank the reviewer for the positive assessment of SMFP and the insightful questions. Below, we clarify our methodological contributions and provide additional experimental evidence.

W1: How does SMFP differ from related stochastic generative policies such as EXPO, GoldenStart, and QAM?

A1: SMFP directly trains a stochastic one-step MeanFlow policy for online off-policy RL. Its MeanFlow mapping captures diverse action modes, while a learnable conditional Gaussian scale provides explicit stochasticity control and a tractable entropy lower bound (Eqs. 10 and 12). Unlike EXPO and QAM-EDIT, which employ Gaussian action refinement, or GoldenStart, which distils a stochastic one-step policy using a Q-guided latent prior, SMFP incorporates the conditional Gaussian scale into both action generation and the MeanFlow regression target (Eqs. 10 and 16). This enables joint policy optimisation through direct value guidance, entropy regularisation, and advantage-weighted MeanFlow regression, without requiring an additional Gaussian policy or a teacher. Moreover, SMFP is trained directly from agent-collected online interactions, without external offline datasets or pretrained policies. We have clarified these distinctions in Section 2.

W2: How closely does the practical loss follow the motivating entropy-regularised mirror-descent objective?

A2: We use entropy-regularised mirror descent as the theoretical motivation for developing a tractable learning objective for implicit generative policies. For a fixed state, critic, and previous policy, the joint entropy-KL objective in Eq. (11) admits the optimal distribution:

\pi_{\mathrm{old}}(a\mid s)^{ \frac{\lambda}{\alpha+\lambda}} \exp\left(\frac{Q(s,a)}{\alpha+\lambda}\right),$$ where $\alpha\geq 0$, $\lambda>0$, and the normalising constant is finite. Setting $\alpha=0$ recovers the value-based MD target in Eq. (5). This target motivates sampling from the previous policy and weighting actions according to their values. Specifically, forward-KL projection yields exponential Q-weights, which can be centred by the state-value baseline to obtain advantage-based weights (Eq. 14 and Appendix B.3). Since the marginal likelihood of SMFP is intractable, we replace likelihood fitting with stochastic MeanFlow regression (Eqs. 15–18) and use positive-advantage weighting to mitigate sensitivity to noisy online value estimates. Meanwhile, the conditional-entropy lower bound (Eqs. 12–13) provides tractable stochasticity control. Thus, Eq. (19) combines MD-motivated policy fitting, direct value guidance, and entropy control into a practical surrogate, rather than an exact implementation of the original objective. We have clarified this distinction in Sections 4.2–4.3 and Appendix B.4. > **Q1:** Given that $K=1$ already performs well in Figure 8(a), when are additional candidates worthwhile, and when is a smaller $K$ sufficient? **A3:** Additional candidates improve value-guided action selection by increasing the chance of identifying high-value actions. However, excessively large behaviour pools may restrict exploration, while large TD-target pools can amplify critic overestimation, as also discussed in QVPO [4]. As shown in the left panel of Figure 7, $K=8$ achieves the highest mean return on both Ant and HumanoidStandup, whereas $K=32$ offers no further benefit. We therefore use $K_b=8$ and $K_t=4$ by default. Additional joint hyperparameter sensitivity results are provided in Figure 8. We further benchmarked inference latency on Ant-v4 using an NVIDIA RTX 4090 (48G) with FP32. **Table R1. SMFP inference latency on Ant-v4 (ms).** | Candidate count $K$ | Action generation | Full policy | |:---:|:---:|:---:| | 1 | 0.260 ± 0.00486 | 0.277 ± 0.00233 | | 2 | 0.251 ± 0.00133 | 0.305 ± 0.00406 | | 4 | 0.252 ± 0.00501 | 0.309 ± 0.00498 | | 8 | 0.247 ± 0.00321 | 0.305 ± 0.00414 | | 16 | 0.251 ± 0.00176 | 0.312 ± 0.00229 | | 32 | 0.261 ± 0.00153 | 0.322 ± 0.00608 | Thanks to vectorized evaluation, candidate-generation latency remains nearly constant. Increasing $K$ from 1 to 32 raises full-policy latency by only 16.2%, from 0.277 to 0.322 ms. The additional overhead mainly comes from critic-based selection, which is unnecessary at $K=1$. These results demonstrate that SMFP can efficiently exploit additional candidates with modest inference overhead. Thus, $K=1$ suits latency-sensitive settings, while $K=8$ provides a favourable performance-efficiency trade-off in our experiments. We thank the reviewer again for the constructive feedback and helpful suggestions. ### References [1] EXPO: Stable Reinforcement Learning with Expressive Policies. ICLR, 2026. [2] GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies. ICLR, 2026. [3] Q-learning with Adjoint Matching. ICLR, 2026. [4] Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization. NeurIPS, 2024.

Thank you for the detailed response. I appreciate your efforts. After considering other reviewers’ comments, I still view the methodological contribution as incremental and maintain my original decision.

Thank you for your feedback. We appreciate your recognition of our efforts and respect your remaining assessment. Thank you again for your time and valuable suggestions.

Reviewer_WiXGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Stochastic MeanFlow Policies (SMFP), a one-step generative policy framework that combines MeanFlow-based noise-to-action mappings with Gaussian reparameterization. The proposed approach aims to capture multimodal action distributions while maintaining explicit stochasticity control through an analytic entropy lower bound. The learning objective combines value guidance, advantage-weighted MeanFlow regression, and entropy-regularized mirror-descent updates. The method is evaluated on seven MuJoCo continuous-control benchmarks, with additional experiments investigating multimodal action modeling and exploration in sparse-reward navigation tasks.

Critical strengths
  1. Interesting methodological combination. The proposed integration of MeanFlow-based generative policies, Gaussian reparameterization, and entropy-regularized mirror-descent learning is an interesting direction for improving policy expressiveness while maintaining efficient action generation.
  2. Literature coverage and reproducibility. The paper provides good coverage of relevant literature and includes implementation details and hyperparameter configurations that support reproducibility.
Critical weaknesses
  1. Limited empirical evidence supporting the central claims. The performance of SMFP appears comparable to, and sometimes worse than, existing Gaussian and generative baselines, with limited separation relative to the reported variability. Although multimodal action representation is a central motivation, the experiments do not clearly demonstrate when this additional expressiveness leads to meaningful performance improvements. Several baselines also appear to eventually achieve similar performance. The discussion around lines 449–455 emphasizes multimodal expressiveness, but the empirical evidence supporting its practical benefits remains limited. More challenging settings where unimodal Gaussian policies are insufficient would help establish the significance of the proposed approach.
  2. Insufficient justification and analysis of the proposed learning framework. The method combines several learning objectives, including value guidance, advantage-weighted regression, and entropy and KL regularization. However, the stability of this combined optimization and its sensitivity to learning rates and other hyperparameters are not sufficiently clear. The theoretical justification for why this particular combination should improve multimodal policy learning is also limited. Furthermore, the claimed computational advantages of one-step inference are not sufficiently quantified against iterative generative alternatives.
Questions for authors
  1. Why are the empirical improvements relatively limited despite the emphasis on multimodal policy expressiveness, and are there more challenging settings where its benefits become more pronounced?
  2. How sensitive is the proposed method to learning rates and loss weighting, and how stable is the optimization across different hyperparameter configurations?
  3. What is the main conclusion from Figure 3, particularly the use of different versions of the same environment and the comparison with PPO?
  4. How does the computational efficiency and performance of one-step action generation compare with multistep generative policies, and what are the associated trade-offs?
3 comments
AuthorsOpenReview ↗

Empirical Performance and Optimisation Analysis (1/2)

We thank the reviewer for recognising the methodological direction and for raising important questions regarding empirical performance and optimisation. We provide further clarification and additional experimental evidence below.

W1/Q1: The return gains appear limited relative to variability. When does multimodal expressivity provide practical benefits, especially in more challenging tasks?

A1: Table 1 shows that SMFP achieves the highest reported mean return on six of seven MuJoCo tasks and ranks second on Humanoid. Notably, the strongest competing flow- and diffusion-based baselines, SAC Flow and DIME, require 4 and 16 generation steps, respectively, whereas SMFP generates each action in a single step. Thus, SMFP achieves competitive or superior control performance with substantially more efficient inference. Beyond returns, Figure 2(b) illustrates its ability to represent separated action modes on Push-T, while Figure 4 demonstrates sustained near-ceiling success across four sparse-reward AntMaze layouts under shared NovelD-style intrinsic rewards and matched interaction budgets. Together, these results highlight the practical advantages of combining multimodal expressivity, effective exploration, and one-step generation.

W2/Q2: What justifies the combined loss, and how sensitive and stable is optimisation with respect to learning rates and loss weights?

A2: The three terms in Eq. (19) serve complementary roles: direct Q-gradients improve predicted action values, the entropy-floor loss regulates conditional stochasticity, and advantage-weighted MeanFlow regression fits high-value samples from the previous policy, motivated by the MD update in Eq. (5). As detailed in Appendices B.2–B.3, positive-advantage weighting mitigates the amplification of noisy value estimates, while Huber regression limits sensitivity to large fitting errors.

Our ablations support these components: removing entropy regularisation slows learning and lowers final returns on Hopper and Walker2d (Fig. 5(a)), while removing advantage-weighted regression reduces performance across all seven tasks (Fig. 5(c)). Figure 6 further evaluates α\alpha and λ\lambda separately over five seeds. We additionally conduct an extensive joint sensitivity study on Swimmer-v4, covering 48 configurations of actor learning rates, MeanFlow regression weights, and candidate counts, with five training seeds per configuration. All runs follow the same training and evaluation protocols as the main experiments.

Table R1. Joint hyperparameter sensitivity on Swimmer-v4. Final return (mean ± sample SD over five training seeds).

KKActor LRλ=0.3\lambda=0.3λ=1\lambda=1λ=3\lambda=3
11.5×10−41.5\times10^{-4}130 ± 6.44139 ± 9.04133 ± 5.84
13×10−43\times10^{-4}137 ± 4.66125 ± 1.99135 ± 2.68
16×10−46\times10^{-4}136 ± 4.69130 ± 9.43125 ± 15.6
11.2×10−31.2\times10^{-3}136 ± 7.99134 ± 8.55135 ± 1.55
41.5×10−41.5\times10^{-4}127 ± 4.56131 ± 5.47118 ± 10.7
43×10−43\times10^{-4}133 ± 3.25123 ± 18.4133 ± 0.321
46×10−46\times10^{-4}136 ± 3.58127 ± 9.89136 ± 2.88
41.2×10−31.2\times10^{-3}139 ± 3.22131 ± 5.84133 ± 1.52
81.5×10−41.5\times10^{-4}122 ± 18.9126 ± 10.5127 ± 4.57
83×10−43\times10^{-4}130 ± 7.35126 ± 8.93130 ± 5.35
86×10−46\times10^{-4}134 ± 3.04136 ± 2.31126 ± 6.86
81.2×10−31.2\times10^{-3}130 ± 2.30126 ± 7.63134 ± 7.58
161.5×10−41.5\times10^{-4}123 ± 16.6137 ± 7.85125 ± 17.2
163×10−43\times10^{-4}129 ± 6.15125 ± 16.6131 ± 3.20
166×10−46\times10^{-4}142 ± 13.0125 ± 11.1134 ± 14.3
161.2×10−31.2\times10^{-3}135 ± 5.73129 ± 5.43128 ± 8.13

Across all 48 configurations, mean returns range from 118 to 142. Notably, these results demonstrate a broad range of effective hyperparameter settings. We have incorporated these sensitivity results into the revised manuscript, with the complete experiments presented in Figure 8 and the evaluation protocol detailed in Appendix C.2.

AuthorsOpenReview ↗

Exploration and Computational Efficiency (2/2)

Q3: What does Figure 3 establish, and why compare four AntMaze versions and PPO?

A3: Figure 4, revised from the original Figure 3, demonstrates SMFP's effective exploration and sustained goal-reaching performance across four distinct sparse-reward AntMaze layouts, including the challenging AntMaze-v3. We compare SMFP against 16 baseline algorithms covering Gaussian, diffusion-based, and flow-based policies, under a shared evaluation protocol with NovelD-style intrinsic rewards. PPO serves as an important reference, while the other methods provide complementary off-policy comparisons. SMFP achieves and maintains near-ceiling success across all four layouts, demonstrating its effectiveness in exploration-intensive tasks. We have clarified the task configurations and evaluation protocol in the revised figure caption and Appendix C.1.

Q4: How does one-step generation affect measured inference cost and control performance, and what trade-offs arise?

A4: SMFP requires only one generation step per candidate, compared with four for SAC Flow-T/G and sixteen for DIME. Table 2 reports the training and inference costs on Ant-v4 using an NVIDIA RTX 5090.

Table R2. Training and inference efficiency on Ant-v4.

MethodTraining time (h)Inference latency (ms)Inference stepsReturn
SMFP2.70.4217518.3
QVPO5.25.77204040.1
DIME3.54.68167103.6
DIPO9.65.20205665.9
TD30.40.2714583.8
SAC2.20.1315530.6
PPO0.30.2212781.9
SPO0.30.3212100.2
QSM0.85.15204206.4
MaxEntDP2.35.62205717.9

Notably, SMFP achieves approximately 11.1× faster inference than DIME, while obtaining a higher Ant-v4 mean return (7518.3 vs. 7103.6). Although Gaussian SAC offers lower inference latency, SMFP provides more expressive action modelling. Its training-time advantage is more modest due to the additional MeanFlow regression and previous-policy sampling.

We further evaluate candidate-selection overhead on an NVIDIA RTX 4090. Table R3 reports the inference latency across different candidate counts.

Table R3. SMFP inference latency with different candidate counts on Ant-v4 (ms).

KKCandidate generationFull policy
10.260 ± 0.004860.277 ± 0.00233
20.251 ± 0.001330.305 ± 0.00406
40.252 ± 0.005010.309 ± 0.00498
80.247 ± 0.003210.305 ± 0.00414
160.251 ± 0.001760.312 ± 0.00229
320.261 ± 0.001530.322 ± 0.00608

Thanks to vectorized evaluation, candidate-generation latency remains nearly constant. Increasing KK from 1 to 32 raises full-policy latency by only 16.2%, from 0.277 to 0.322 ms. This demonstrates that SMFP can efficiently exploit multiple candidates with modest additional inference overhead, while K=8K=8 offers a favourable performance-efficiency trade-off.

We thank the reviewer again for the constructive feedback and insightful questions. We hope that the additional experiments and clarifications address the concerns raised and provide a more complete picture of SMFP's methodological contributions and empirical advantages. We would appreciate the reviewer's reconsideration of the overall assessment in light of these revisions.

AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Response – Submission458

Dear Reviewer,

Thank you for your valuable feedback on our paper. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could review our responses.

If you have any remaining questions or concerns, please feel free to let us know. We would be happy to provide further clarifications or make additional revisions.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_7iuwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes stochastic mean flow policies, which aims to combine the benefits of generative policies and mirror-descent-style learning.

Critical strengths

I found this paper to be incredibly challenging to parse. See weaknesses.

Critical weaknesses

I found this paper to be dense and essentially unreadable. As a result, I cannot follow the argumentation. For example, the abstract requires substantial effort to motivate a simple premise: when doing RL, we choose both the policy and the algorithm that optimizes it. This problem is present throughout the paper. For this reason, I cannot provide a detailed review and recommend rejection because of poor intelligibility.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Clarification of the Motivation and Key Contributions of SMFP

Thank you for identifying the clarity issues in our motivation and method. We sincerely acknowledge that the original presentation, particularly the abstract, may have made the central motivation less intuitive than intended. We would like to briefly clarify our key idea.

W1: The paper is dense and difficult to follow, particularly the abstract’s motivation of the basic choice of policy and optimisation algorithm.

Our central goal is to develop an expressive policy for online off-policy RL that combines multimodal action modelling, efficient one-step generation, and explicit stochasticity control. Gaussian policies offer efficient sampling and tractable entropy regularisation but struggle to represent separated action modes, whereas generative policies are more expressive but often require iterative sampling or lack tractable entropy estimation. SMFP bridges this gap by combining a one-step MeanFlow mapping with Gaussian reparameterisation: latent-conditioned means capture diverse action modes, while a learnable conditional Gaussian scale controls dispersion and provides a tractable entropy lower bound. Building on this design, we develop an MD-motivated learning objective that combines advantage-weighted MeanFlow regression, direct Q-gradient guidance, and conditional-entropy regularisation for tractable policy optimisation. Our experiments demonstrate the effectiveness of this design, showing multimodal action modelling, improved learning through the proposed optimisation components, and competitive control performance with efficient inference.

We have revised the abstract, introduction, and method description to clarify the central motivation, highlight the technical contributions, and improve the overall logical flow. New experiments and substantive revisions are highlighted in blue. We sincerely hope these clarifications make our contributions more accessible, and would be very grateful if the reviewer could kindly revisit the revised manuscript and reconsider the assessment. We warmly welcome any further questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.