- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes SMFP, a one-step MeanFlow policy with a learned Gaussian noise scale for entropy control. It trains the policy using Q-value gradients, an entropy-floor penalty, and advantage-weighted MeanFlow regression. Experiments on MuJoCo, sparse-reward AntMaze, and Push-T examine control performance, exploration, and multimodal action modeling.
Critical strengths
-
I like the explicit control over the policy’s noise scale. It gives a one-step generative policy a simple way to maintain exploration, and the entropy ablation suggests that this helps learning.
-
The experimental results are strong across several control tasks. Experiments support the method’s usefulness in sparse-reward settings.
Critical weaknesses
-
The novelty seems somewhat limited. EXPO [1], GoldenStart [2], and QAM’s Gaussian-edit variant [3] have related ideas about stochastic generative policies and entropy control. A more comparison and discussion would help explain what SMFP adds.
-
The connection to mirror descent could be clearer. The practical loss combines positive-advantage-weighted MeanFlow regression with a separate entropy-floor penalty. It is unclear how closely this follows the motivating entropy-regularized MD objective, which makes the theoretical contribution harder to assess.
[1] EXPO: Stable Reinforcement Learning with Expressive Policies. [2] GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies. [3] Q-learning with Adjoint Matching.
Questions for authors
Figure 8(a) shows that K = 1 already performs quite well. Could the authors discuss when sampling more candidates is useful and why? In particular, when do the gains justify the additional cost, and when would a smaller K be sufficient?
3 comments
Thank you for the detailed response. I appreciate your efforts. After considering other reviewers’ comments, I still view the methodological contribution as incremental and maintain my original decision.
Thank you for your feedback. We appreciate your recognition of our efforts and respect your remaining assessment. Thank you again for your time and valuable suggestions.
Clarification of Methodological Contributions and Empirical Evidence
We thank the reviewer for the positive assessment of SMFP and the insightful questions. Below, we clarify our methodological contributions and provide additional experimental evidence.
A1: SMFP directly trains a stochastic one-step MeanFlow policy for online off-policy RL. Its MeanFlow mapping captures diverse action modes, while a learnable conditional Gaussian scale provides explicit stochasticity control and a tractable entropy lower bound (Eqs. 10 and 12). Unlike EXPO and QAM-EDIT, which employ Gaussian action refinement, or GoldenStart, which distils a stochastic one-step policy using a Q-guided latent prior, SMFP incorporates the conditional Gaussian scale into both action generation and the MeanFlow regression target (Eqs. 10 and 16). This enables joint policy optimisation through direct value guidance, entropy regularisation, and advantage-weighted MeanFlow regression, without requiring an additional Gaussian policy or a teacher. Moreover, SMFP is trained directly from agent-collected online interactions, without external offline datasets or pretrained policies. We have clarified these distinctions in Section 2.
A2: We use entropy-regularised mirror descent as the theoretical motivation for developing a tractable learning objective for implicit generative policies. For a fixed state, critic, and previous policy, the joint entropy-KL objective in Eq. (11) admits the optimal distribution:
\pi_{\mathrm{old}}(a\mid s)^{ \frac{\lambda}{\alpha+\lambda}} \exp\left(\frac{Q(s,a)}{\alpha+\lambda}\right),$$ where $\alpha\geq 0$, $\lambda>0$, and the normalising constant is finite. Setting $\alpha=0$ recovers the value-based MD target in Eq. (5). This target motivates sampling from the previous policy and weighting actions according to their values. Specifically, forward-KL projection yields exponential Q-weights, which can be centred by the state-value baseline to obtain advantage-based weights (Eq. 14 and Appendix B.3). Since the marginal likelihood of SMFP is intractable, we replace likelihood fitting with stochastic MeanFlow regression (Eqs. 15–18) and use positive-advantage weighting to mitigate sensitivity to noisy online value estimates. Meanwhile, the conditional-entropy lower bound (Eqs. 12–13) provides tractable stochasticity control. Thus, Eq. (19) combines MD-motivated policy fitting, direct value guidance, and entropy control into a practical surrogate, rather than an exact implementation of the original objective. We have clarified this distinction in Sections 4.2–4.3 and Appendix B.4. > **Q1:** Given that $K=1$ already performs well in Figure 8(a), when are additional candidates worthwhile, and when is a smaller $K$ sufficient? **A3:** Additional candidates improve value-guided action selection by increasing the chance of identifying high-value actions. However, excessively large behaviour pools may restrict exploration, while large TD-target pools can amplify critic overestimation, as also discussed in QVPO [4]. As shown in the left panel of Figure 7, $K=8$ achieves the highest mean return on both Ant and HumanoidStandup, whereas $K=32$ offers no further benefit. We therefore use $K_b=8$ and $K_t=4$ by default. Additional joint hyperparameter sensitivity results are provided in Figure 8. We further benchmarked inference latency on Ant-v4 using an NVIDIA RTX 4090 (48G) with FP32. **Table R1. SMFP inference latency on Ant-v4 (ms).** | Candidate count $K$ | Action generation | Full policy | |:---:|:---:|:---:| | 1 | 0.260 ± 0.00486 | 0.277 ± 0.00233 | | 2 | 0.251 ± 0.00133 | 0.305 ± 0.00406 | | 4 | 0.252 ± 0.00501 | 0.309 ± 0.00498 | | 8 | 0.247 ± 0.00321 | 0.305 ± 0.00414 | | 16 | 0.251 ± 0.00176 | 0.312 ± 0.00229 | | 32 | 0.261 ± 0.00153 | 0.322 ± 0.00608 | Thanks to vectorized evaluation, candidate-generation latency remains nearly constant. Increasing $K$ from 1 to 32 raises full-policy latency by only 16.2%, from 0.277 to 0.322 ms. The additional overhead mainly comes from critic-based selection, which is unnecessary at $K=1$. These results demonstrate that SMFP can efficiently exploit additional candidates with modest inference overhead. Thus, $K=1$ suits latency-sensitive settings, while $K=8$ provides a favourable performance-efficiency trade-off in our experiments. We thank the reviewer again for the constructive feedback and helpful suggestions. ### References [1] EXPO: Stable Reinforcement Learning with Expressive Policies. ICLR, 2026. [2] GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies. ICLR, 2026. [3] Q-learning with Adjoint Matching. ICLR, 2026. [4] Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization. NeurIPS, 2024.