- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents one step MeanFlow policies for fully online RL. MeanFlow is normally trained by copying example actions from the distribution it should learn but in online RL no such examples exist and prior method like MVP instead imitates the best of N sampled actions. Authors proposed SOM which takes the Boltzmann policy as the target and builds the MeanFlow velocity from the scores of its noised levels estimated by importance weighting critic gradients over sampled candidates with inverse temperature constant adapted per noise level to hold a target effective sample size. Experiments on nine MuJoCo tasks show the highest return on the five main locomotion tasks with one step inference supported by a frozen critic comparison with MVP 2D bandit studies and extensive ablations in the appendix.
Critical strengths
-
SOM solves a real gap for one step generative policies in online RL by building the training target from critic's gradients instead of copying best of N samples and the frozen critic and bandit experiments show this behaves differently and clearly from best of N imitation.
-
SOM leads on the main tasks with one step inference and the appendix ablates nearly every design choice and openly reports where the defaults lose and how much the method relies on best of N selection.
Critical weaknesses
- My main concern is that the gains rely heavily on best of N selection which SOM uses both to pick actions and to compute critic targets. The appendix reports that with N = 1 the returns drop sharply (for example 96.5% on Hopper) and SOM falls below SAC on four of five tasks but the main text describes best of N only as a reranking step. Since MVP also uses best of N and is run with different settings (learning rates, warm-up, no exploration noise) so it is a bit unclear to me how much of the gain comes from the new update rule itself. The frozen critic experiment compares SOM and MVP under a shared critic but only when fine tuning from a frozen checkpoint and with best of N kept on.
- The baselines also look weaker than usual (for example SAC on Humanoid and Hopper) and the UTD ratio is given only for SOM but not for the baselines.
It would be useful to see MVP trained with SOM's settings and critic with and without best of N even on two tasks such as Hopper and Humanoid and to know the UTD ratio used for each baseline which would show whether the gains come from the update rule rather than from selection and setup.
Questions for authors
- Do the baselines use the same critic as SOM (distributional, RMS-normalized) and what UTD ratio does each use? How were MVP's settings chosen for the online setting?
- Table 16 shows runs with = 1 so what returns do they reach given that alpha sits at the lower bound for much of training on HalfCheetah and Ant?
- As Appendix F says that the estimator is only checked at t = 0 where it reduces to the critic gradient. Could you report its error at intermediate on the 2D bandit where the exact score can be computed?
- What is the training wall clock time of SOM compared with SDAC and MVP?
9 comments
Reply to Reviewer oRFV (2/7)
How were MVP's settings chosen for the online setting?
MVP was proposed for offline-to-online RL, so no configuration for purely online MuJoCo training existed. We followed the official MVP implementation (batch size , best-of-, no additional exploration noise) within the training loop and critic shared by all baselines (twin critics with clipped double-Q, updates per environment step, warm-up ). The only remaining difference from the official code is the learning rate ( instead of ), and restoring it gives nearly the same results. We added this sensitivity study to Appendix H.8 (Table 27).
Table R[3]. MVP under different settings (Table 27; mean over five seeds).
| MVP setting | Ant-v4 | HalfCheetah-v4 | Hopper-v4 | Humanoid-v4 | Walker2d-v4 | Avg. % of SOM |
|---|---|---|---|---|---|---|
| Table 1 setting | ||||||
| learning rate | ||||||
| exploration noise | ||||||
| warm-up | ||||||
| lr + noise + warm-up | ||||||
| matched to SOM | ||||||
| SOM (Table 1) |
Unless noted otherwise, each row changes one factor of the Table 1 setting (learning rate , no exploration noise, warm-up , best-of-, MVP's own critic):
- learning rate : the official MVP value for actor and critic.
- exploration noise: SOM's Gaussian exploration noise (scaled by the learned entropy coefficient), added to the action after best-of- selection.
- warm-up : SOM's number of initial random-action steps.
- / : the number of best-of- candidates.
- lr + noise + warm-up: the three changes above combined; MVP's critic and losses are unchanged.
- matched to SOM: lr + noise + warm-up, plus SOM's critic (Gaussian distributional with first-layer RMSNorm) and TD target (next action from the target actor); only MVP's actor update rule remains.
MVP benefits considerably from exploration noise and from the SOM-matched configuration, yet SOM remains higher on four of the five tasks.
Reply to Reviewer oRFV (3/7)
Response to Weakness 1.
Response to Weakness 1: Dependence on best-of- selection.
We agree that best-of- selection contributes substantially: without it, SOM falls below SAC on four of the five tasks (Tables 1 and 21). This is common to selection-based methods rather than specific to SOM, as Table R[4] shows for every method that uses best-of- selection.
Table R[4]. Training / evaluation for all best-of- methods (Table 22), with SAC, which does not use selection, from Table 1 for reference. Bold: highest among the best-of- methods.
| Algorithm | Ant-v4 | HalfCheetah-v4 | Hopper-v4 | Humanoid-v4 | Walker2d-v4 |
|---|---|---|---|---|---|
| DPMD | |||||
| MVP (Table 1 setting) | |||||
| QSM | |||||
| QVPO | |||||
| SDAC | |||||
| SOM | |||||
| SAC (no selection) |
Without selection, every best-of- method falls below SAC on at least four tasks; SOM and SDAC remain the strongest among them, and SOM on Humanoid exceeds every Table 1 baseline even without selection.
Selection is used both for acting and for computing critic targets, as stated in Section 4.5, which now also points to the analysis of training- and evaluation-time selection in Appendix H.4. We isolate the contribution of the update rule from that of selection in the next response.
Reply to Reviewer oRFV (1/7)
We sincerely thank the reviewer for the favorable evaluation of our work, as well as for the time spent on our manuscript and the insightful comments and questions. All of the reviewer's points have been addressed in the revised manuscript, where new or modified content appears in blue; for each response below, we indicate where in the manuscript the change is located. Throughout this response, table, figure, and section numbers follow the revised manuscript unless otherwise noted, and tables report the mean of the last five evaluations ± 95% CI over five seeds, with bold indicating the highest mean.
Response to Question 1 & Weakness 2.
Do the baselines use the same critic as SOM (distributional, RMS-normalized)?
In Table 1, each baseline uses its original critic (a scalar Q-function, except DACER, which is already distributional), whereas SOM uses a Gaussian distributional critic with first-layer RMSNorm. To test whether SOM's gains come from the critic, we retrained all eight off-policy baselines with SOM's critic, keeping all other settings as in Table 1, and added the results to Appendix H.6 (Table 25).
Table R[1]. All methods with SOM's critic (Table 25; SOM row from Table 1). Parentheses: rank among the nine methods; bold: highest.
SOM's critic substantially improves several baselines, but with the same critic SOM still has the best average rank. The critic design thus contributes to performance, yet it does not account for SOM's overall advantage.
What UTD ratio does each method use?
Following the reviewer's comment, we now report the update schedule of every method in Appendix H.7 (Table 26). All off-policy methods, including SOM, share one training loop: each iteration steps five parallel environments once and performs one gradient update with batch size 256, i.e., 0.2 updates per environment step (about 1M updates over 5M steps).
Table R[2]. Update schedule of the off-policy methods in Table 1 (Table 26).
All methods therefore use the same UTD ratio; SOM performs slightly fewer updates because of its longer warm-up (105 vs. 3×104 steps; Table 23). SOM's gains thus do not come from a higher UTD ratio or more gradient updates. We apologize for not stating these update schedules clearly in the submission.
Baseline returns (Weakness 2).
Standard SAC performs one update per environment step, whereas all methods here share 0.2 updates per step. This lower shared budget is the main difference from the standard SAC setup and a likely reason why some absolute baseline returns (e.g., SAC on Humanoid and Hopper) are below commonly reported values; since every method, including SOM, uses the same budget, the comparison remains fair.