- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper examines what happens when RL-trained diffusion/flow policies are deployed on a real robot. Sampling a fresh latent at every control step causes jittery actions and inconsistent trajectories, while fixing the latent to zero removes the jitter but drifts and cannot adapt to the command. The proposed method freezes a GenPO-trained generator and trains a second PPO actor that picks the latent from the current observation. The latent is constrained to a sphere of radius sqrt(D) through an analytic angular map, and at deployment the actor's mean is used so the controller is deterministic. The method is evaluated on six Isaac Lab tasks (training reward vs. PPO and GenPO), on Unitree G1 locomotion in MuJoCo (random vs. zero vs. learned latent on tracking, dispersion and smoothness metrics), on path tracking against PPO and zero-latent GenPO, and with a sim-to-real demonstration on a physical G1.
Critical strengths
The paper identifies and measures a real problem. Per-step latent resampling in RL-trained diffusion/flow policies can cause jitter, body sway, and trajectory dispersion. The paper demonstrates this with concrete simulation results (Table 3: 8.57% tracking success and 111 mrad action-increment RMS for random latents; Figures 1 and 10). This establishes a practically relevant problem that motivates improving generative-policy deployment.
The method is relatively simple and demonstrated on a real robot. The generator is frozen, only the latent actor and value network are trained during post-training, and the sphere constraint is enforced analytically without a penalty term. The resulting controller transfers to a physical Unitree G1 without real-world fine-tuning (Figures 6 and 7).
Critical weaknesses
- The two-stage design is not justified and the experiments do not isolate its benefit.
The method trains a generative policy with RL, runs a second RL stage on top. Then removes the stochasticity at deployment by using the mean latent. The deployed controller is a deterministic observation-to-action map with a frozen observation-conditioned decoder in between. The closest prior method (DSRL) learns a stochastic, entropy-regularized latent policy and frames steering as reshaping the noise distribution. Deterministic selection may help repeatability. But the paper never shows what the pretrained generative representation adds over directly learning a deterministic policy with the same compute.
The comparison that could answer this is confounded. In all six Isaac Lab tasks GenPO is stopped at two-thirds of the total budget and SteerGenPO gets the remaining third. PPO gets the full budget, GenPO does not.
Table 2 reports stochastic training reward only. The switch to Stage 2 changes the stochastic policy this reward is measured under, and the paper does not separate that immediate change from learning. No deterministic evaluation is reported for the Isaac Lab tasks, although Appendix B says the two returns differ.
The fixed-radius sphere, the main new idea, is never ablated. No box-constrained latent actor as in DSRL, no normalized output (which the paper mentions), no fixed latent on the sphere such as Tr(0).
- The deployment claims are not supported by quantitative evidence.
The motivation is that diffusion RL beats PPO in simulation but is unreliable on robots. Yet there is no repeated-trial PPO comparison on the Table 3 metrics. Figures 5 and 7 are illustrative only.
Learned improves tracking over Zero, but Table 3 gives only mean ± sd. over five seeds and no analysis of the uncertainty of the differences. The reader cannot tell how reliable the gains are. Zero is as smooth or slightly smoother on both variation metrics (which raises the question of how much learned latent selection adds beyond removing stochastic sampling).
The headline 79.5% and 75.1% reductions are measured against random latent resampling. However, zero-latent inference (already employed by FPO++) achieves comparable or slightly better smoothness without additional RL training. Thus, these reductions primarily demonstrate the benefit of eliminating stochastic sampling rather than the benefit of the proposed learned latent selection.
Table 1 and Figure 2 are single runs. Figure 5 has no tracking-error numbers. The hardware results show one illustrated circuit per method, with approximate lap time as the only reported numerical measurement and no physical-robot comparison against random or zero-latent GenPO is provided.
Questions for authors
-
Please report GenPO trained for the full budget (e.g., 12,000 iterations on Allegro Hand, matching PPO and SteerGenPO) and evaluated deterministically. Does SteerGenPO still improve over it?
-
Table 2 reports stochastic training reward. Please report deterministic evaluation returns for PPO (mean action), GenPO (zero latent and a fixed on-sphere latent), and SteerGenPO (mean angular coordinate) on all six Isaac Lab tasks.
-
How much of the jump at the Stage 1/Stage 2 boundary in Figure 4 is already present at iteration 0 of Stage 2, before any latent-actor update? A single evaluation of the freshly initialized actor would answer this.
-
Please compare the fixed-radius angular actor against (a) an unconstrained or box-constrained latent actor as in DSRL (b) a normalized D-dimensional output (c) a fixed latent on the sphere such as Tr(0), under the same budget.
-
Since the deployed policy is a deterministic function of the observation, what advantage does GenPO pretraining plus latent selection have over a deterministic policy with the same total compute, for example PPO with an equivalent low-noise second stage? Any evidence here would be helpful.
-
Please add PPO (mean action) to Table 3 under the same commands and perturbed initial states.
-
Appendix F.3 says the same command-initialization pairs are used for every controller. Can you report paired differences with confidence intervals for the Table 3 metrics?
-
How many physical trials were run per method in Figures 6 and 7? Figure 7 shows one circuit each. Were any trials of GenPO with zero or random latent attempted on the robot?
7 comments
Response to Reviewer deWw — Part 2/5
3. W1(3) / Q3 — Immediate effect of switching to Stage 2
W1(3): The switch to Stage 2 changes the stochastic policy this reward is measured under, and the paper does not separate that immediate change from learning.
Q3: How much of the jump at the Stage 1/Stage 2 boundary in Figure 4 is already present at iteration 0 of Stage 2, before any latent-actor update? A single evaluation of the freshly initialized actor would answer this.
Response to W1(3) & Q3: We compared returns at the Stage 1 endpoint and iteration 0 of Stage 2 to quantify the immediate change before any latent-actor update. The absolute change in mean return is at most 3.01% on five tasks, while Franka Reach shows a 5.08% decrease. Both evaluations use stochastic sampling under matched conditions.
| Task | Stage 1 endpoint | Stage 2, iteration 0 | Return increase (%) |
|---|---|---|---|
| Ant | 150.53 ± 10.27 | 150.74 ± 10.02 | +0.14 |
| Franka Reach | 0.2164 ± 0.0082 | 0.2054 ± 0.0058 | −5.08 |
| ANYmal-D | 17.69 ± 0.26 | 17.66 ± 0.23 | −0.17 |
| Go2 | 22.20 ± 0.11 | 22.03 ± 0.47 | −0.77 |
| Allegro Hand | 2137.57 ± 203.23 | 2073.14 ± 206.99 | −3.01 |
| Shadow Hand | 6244.86 ± 126.06 | 6231.21 ± 146.30 | −0.22 |
Percentage changes are calculated using the displayed mean returns; negative values indicate decreases.
These results show that switching to the untrained latent actor does not automatically improve performance, and its immediate effect varies across tasks.
4. W1(4) / Q4 — Ablations of the spherical constraint and latent selection
W1(4): The fixed-radius sphere, the main new idea, is never ablated. No box-constrained latent actor as in DSRL, no normalized output (which the paper mentions), no fixed latent on the sphere such as Tr(0).
Q4: Please compare the fixed-radius angular actor against (a) an unconstrained or box-constrained latent actor as in DSRL (b) a normalized D-dimensional output (c) a fixed latent on the sphere such as Tr(0), under the same budget.
Response to W1(4) & Q4: We have added the requested ablations on Unitree G1, where SteerGenPO achieves the highest mean test return and tracking success among all compared methods. The comparisons include unconstrained and box-constrained learning, L2 normalization of a -dimensional output, and fixed spherical latents. The fixed-sphere controls use and five randomly selected spherical directions, each held constant during deployment.
| Method | Test return ↑ | Tracking success (%) ↑ |
|---|---|---|
| Unconstrained | 61.97 ± 2.37 | 41.14 ± 3.83 |
| Box-constrained | 62.21 ± 1.94 | 40.00 ± 15.65 |
| Fixed spherical latent | 35.13 ± 16.47 | 2.86 ± 6.39 |
| Fixed spherical latent (random directions) | 42.05 ± 6.50 | 6.63 ± 4.79 |
| Sphere–L2 (-dimensional normalization) | 62.66 ± 0.54 | 42.86 ± 0.00 |
| SteerGenPO (Sphere–Angular) | 67.24 ± 0.20 | 54.29 ± 11.95 |
Both learned spherical variants achieve higher mean test return and tracking success than the unconstrained and box-constrained variants, supporting the design of keeping latents at the training prior’s typical radial scale. At the same radius, SteerGenPO improves tracking success over Sphere–L2 from 42.86% to 54.29% and test return from 62.66 to 67.24, further supporting the angular parameterization.
The fixed spherical latents perform substantially worse, indicating that being on the sphere does not guarantee a latent that produces high-quality control behavior. The spherical constraint defines the search space, but learning is still needed. Complete ablation metrics for the learned variants are provided in Table 4 of the revised main text.
Response to Reviewer deWw — Part 3/5
5. Q5 — Value of generative pretraining and latent selection
Related concern in W1: But the paper never shows what the pretrained generative representation adds over directly learning a deterministic policy with the same compute.
Q5: Since the deployed policy is a deterministic function of the observation, what advantage does GenPO pretraining plus latent selection have over a deterministic policy with the same total compute, for example PPO with an equivalent low-noise second stage? Any evidence here would be helpful.
Response to Q5: Diffusion RL requires an additional mechanism for effective deterministic deployment, whereas PPO can directly deploy its Gaussian policy’s mean action. Generative policies such as GenPO produce actions through a nonlinear generation process without the same directly available mean-action output. They also offer greater flexibility in representing action distributions than single-Gaussian policies; the GenPO–PPO comparisons in Table 1 and Figure 4 demonstrate higher returns in our evaluated tasks.
SteerGenPO therefore freezes the generator and learns state-conditioned latent selection for deterministic deployment, aiming to bridge the sim-to-real deployment gap in diffusion RL while retaining the control capabilities acquired through generative pretraining.
We also tested PPO with the suggested low-noise second stage. In this setting, mean-action deployment still performed best; this training adjustment retains PPO’s Gaussian policy representation and its directly available deterministic mean-action output.
W2. Quantitative evidence for deployment performance
6. W2(1) / Q6 — Repeated deployment comparison with PPO
W2(1): The motivation is that diffusion RL beats PPO in simulation but is unreliable on robots. Yet there is no repeated-trial PPO comparison on the Table 3 metrics.
Q6: Please add PPO (mean action) to Table 3 under the same commands and perturbed initial states.
Response to W2(1) & Q6: We have added repeated evaluations of mean-action PPO to Table 3, using five training seeds and the same commands and perturbed initial states. SteerGenPO achieves higher mean tracking success and test return, as shown below:
| Method | Tracking success (%) ↑ | Test return ↑ |
|---|---|---|
| PPO | 41.71 ± 20.36 | 47.61 ± 13.13 |
| SteerGenPO | 54.29 ± 11.95 | 67.24 ± 0.20 |
SteerGenPO increases mean tracking success from 41.71% to 54.29% and test return from 47.61 to 67.24. Across the full evaluation, it achieves better mean performance than PPO on all eight metrics, including velocity-tracking errors, lateral deviation, trajectory dispersion, and action variation. Complete results are included in Table 3 of the revised manuscript.
Response to Reviewer deWw — Part 1/5
We thank you for your careful review and detailed suggestions. We appreciate your recognition of the deployment problem identified and quantified in our work, as well as the method’s simplicity and real-robot validation. Below, we address your concerns about the two-stage design and deployment evidence with additional experiments and explanations.
W1. Two-stage design and its benefits
1. W1(1) / Q1 — GenPO with the full training budget
Response to W1(1) & Q1: We have added full-budget GenPO comparisons on all six tasks, matching the total training iterations of PPO and SteerGenPO. SteerGenPO achieves higher mean training rewards on all six tasks, as shown below:
We also evaluated deterministic deployment, using zero latents for GenPO and mean angular coordinates for SteerGenPO. SteerGenPO likewise achieves higher mean evaluation returns on all six tasks:
Compared with full-budget GenPO, SteerGenPO improves mean training rewards by 4.55%–81.80% and deterministic evaluation returns by 0.45%–16.65%. This comparison contrasts full-budget GenPO training with GenPO pretraining followed by latent post-training with the generator frozen. SteerGenPO achieves higher mean training rewards and deterministic evaluation returns while keeping the generator frozen.
The learning curves and training rewards have been incorporated into Figure 4 and Table 2 of the revised manuscript, and the deterministic evaluation results and settings into Appendix C.1, Table 11.
2. W1(2) / Q2 — Deterministic evaluation on all six benchmarks
Response to W1(2) & Q2: We have added deterministic evaluations on all six Isaac Lab tasks: SteerGenPO achieves the highest mean return on every task, consistent with the main training-return trends. PPO uses its mean action, GenPO uses zero or fixed on-sphere latents, and SteerGenPO uses the latent actor’s mean output. Both GenPO variants use the same Stage 1 models. The results are:
The fixed spherical latent is z=TD(0), held constant throughout evaluation.
These results support state-conditioned latent selection as a way to better realize the base generator’s learned control capabilities during deterministic inference. The PPO, zero-latent GenPO, and SteerGenPO results are included in Appendix C.1; the fixed-sphere comparison is provided here to address your question.