acceptodds
Under review as a conference paper at ICLR 2027

Bridging the Deployment Gap in Diffusion Reinforcement Learning via State-Conditioned Latent Control

Abstract

Recent diffusion and flow policies provide expressive action distributions and achieve strong performance in reinforcement learning, yet their deployment on physical robots remains underexplored. We identify a key deployment gap: although diffusion policies outperform Gaussian policy baselines in simulation, independently resampling latent inputs at every control step introduces temporal inconsistency, leading to high-frequency action variation, increased body sway, and trajectory dispersion. Fixing the latent input reduces this variation but cannot adapt latent behaviors to changing robot states and task commands. To address this issue, we introduce SteerGenPO, a state-conditioned latent control framework that freezes the reinforcement-learning-trained generator and learns an actor to select latent inputs from the robot state and task command. An angular parameterization constrains the latent input to a sphere of radius , matching the typical radial scale of the Gaussian prior used during generator training. During deployment, the actor deterministically selects latent inputs using its mean angular coordinates. We evaluate SteerGenPO on robot control benchmarks, Unitree G1 locomotion and path tracking, and physical-robot experiments. Compared with random latent sampling, SteerGenPO reduces high-frequency action variation and trajectory dispersion while improving command tracking, enabling more reliable deployment of diffusion reinforcement learning on real robots.

Primary area:
reinforcement learning
Keywords:
Diffusion Reinforcement Learning, Generative Policies, Latent Control, Humanoid Locomotion, Sim-to-Real Deployment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_deWwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines what happens when RL-trained diffusion/flow policies are deployed on a real robot. Sampling a fresh latent at every control step causes jittery actions and inconsistent trajectories, while fixing the latent to zero removes the jitter but drifts and cannot adapt to the command. The proposed method freezes a GenPO-trained generator and trains a second PPO actor that picks the latent from the current observation. The latent is constrained to a sphere of radius sqrt(D) through an analytic angular map, and at deployment the actor's mean is used so the controller is deterministic. The method is evaluated on six Isaac Lab tasks (training reward vs. PPO and GenPO), on Unitree G1 locomotion in MuJoCo (random vs. zero vs. learned latent on tracking, dispersion and smoothness metrics), on path tracking against PPO and zero-latent GenPO, and with a sim-to-real demonstration on a physical G1.

Critical strengths

The paper identifies and measures a real problem. Per-step latent resampling in RL-trained diffusion/flow policies can cause jitter, body sway, and trajectory dispersion. The paper demonstrates this with concrete simulation results (Table 3: 8.57% tracking success and 111 mrad action-increment RMS for random latents; Figures 1 and 10). This establishes a practically relevant problem that motivates improving generative-policy deployment.

The method is relatively simple and demonstrated on a real robot. The generator is frozen, only the latent actor and value network are trained during post-training, and the sphere constraint is enforced analytically without a penalty term. The resulting controller transfers to a physical Unitree G1 without real-world fine-tuning (Figures 6 and 7).

Critical weaknesses
  1. The two-stage design is not justified and the experiments do not isolate its benefit.

The method trains a generative policy with RL, runs a second RL stage on top. Then removes the stochasticity at deployment by using the mean latent. The deployed controller is a deterministic observation-to-action map with a frozen observation-conditioned decoder in between. The closest prior method (DSRL) learns a stochastic, entropy-regularized latent policy and frames steering as reshaping the noise distribution. Deterministic selection may help repeatability. But the paper never shows what the pretrained generative representation adds over directly learning a deterministic policy with the same compute.

The comparison that could answer this is confounded. In all six Isaac Lab tasks GenPO is stopped at two-thirds of the total budget and SteerGenPO gets the remaining third. PPO gets the full budget, GenPO does not.

Table 2 reports stochastic training reward only. The switch to Stage 2 changes the stochastic policy this reward is measured under, and the paper does not separate that immediate change from learning. No deterministic evaluation is reported for the Isaac Lab tasks, although Appendix B says the two returns differ.

The fixed-radius sphere, the main new idea, is never ablated. No box-constrained latent actor as in DSRL, no normalized output (which the paper mentions), no fixed latent on the sphere such as Tr(0).

  1. The deployment claims are not supported by quantitative evidence.

The motivation is that diffusion RL beats PPO in simulation but is unreliable on robots. Yet there is no repeated-trial PPO comparison on the Table 3 metrics. Figures 5 and 7 are illustrative only.

Learned improves tracking over Zero, but Table 3 gives only mean ± sd. over five seeds and no analysis of the uncertainty of the differences. The reader cannot tell how reliable the gains are. Zero is as smooth or slightly smoother on both variation metrics (which raises the question of how much learned latent selection adds beyond removing stochastic sampling).

The headline 79.5% and 75.1% reductions are measured against random latent resampling. However, zero-latent inference (already employed by FPO++) achieves comparable or slightly better smoothness without additional RL training. Thus, these reductions primarily demonstrate the benefit of eliminating stochastic sampling rather than the benefit of the proposed learned latent selection.

Table 1 and Figure 2 are single runs. Figure 5 has no tracking-error numbers. The hardware results show one illustrated circuit per method, with approximate lap time as the only reported numerical measurement and no physical-robot comparison against random or zero-latent GenPO is provided.

Questions for authors
  1. Please report GenPO trained for the full budget (e.g., 12,000 iterations on Allegro Hand, matching PPO and SteerGenPO) and evaluated deterministically. Does SteerGenPO still improve over it?

  2. Table 2 reports stochastic training reward. Please report deterministic evaluation returns for PPO (mean action), GenPO (zero latent and a fixed on-sphere latent), and SteerGenPO (mean angular coordinate) on all six Isaac Lab tasks.

  3. How much of the jump at the Stage 1/Stage 2 boundary in Figure 4 is already present at iteration 0 of Stage 2, before any latent-actor update? A single evaluation of the freshly initialized actor would answer this.

  4. Please compare the fixed-radius angular actor against (a) an unconstrained or box-constrained latent actor as in DSRL (b) a normalized D-dimensional output (c) a fixed latent on the sphere such as Tr(0), under the same budget.

  5. Since the deployed policy is a deterministic function of the observation, what advantage does GenPO pretraining plus latent selection have over a deterministic policy with the same total compute, for example PPO with an equivalent low-noise second stage? Any evidence here would be helpful.

  6. Please add PPO (mean action) to Table 3 under the same commands and perturbed initial states.

  7. Appendix F.3 says the same command-initialization pairs are used for every controller. Can you report paired differences with confidence intervals for the Table 3 metrics?

  8. How many physical trials were run per method in Figures 6 and 7? Figure 7 shows one circuit each. Were any trials of GenPO with zero or random latent attempted on the robot?

7 comments
AuthorsOpenReview ↗

Response to Reviewer deWw — Part 1/5

We thank you for your careful review and detailed suggestions. We appreciate your recognition of the deployment problem identified and quantified in our work, as well as the method’s simplicity and real-robot validation. Below, we address your concerns about the two-stage design and deployment evidence with additional experiments and explanations.

W1. Two-stage design and its benefits

1. W1(1) / Q1 — GenPO with the full training budget

W1(1): The comparison that could answer this is confounded. In all six Isaac Lab tasks GenPO is stopped at two-thirds of the total budget and SteerGenPO gets the remaining third. PPO gets the full budget, GenPO does not.

Q1: Please report GenPO trained for the full budget (e.g., 12,000 iterations on Allegro Hand, matching PPO and SteerGenPO) and evaluated deterministically. Does SteerGenPO still improve over it?

Response to W1(1) & Q1: We have added full-budget GenPO comparisons on all six tasks, matching the total training iterations of PPO and SteerGenPO. SteerGenPO achieves higher mean training rewards on all six tasks, as shown below:

TaskGenPO Stage 1GenPO Full BudgetSteerGenPO
Ant149.37 ± 10.68159.93 ± 8.92194.51 ± 2.45
Allegro Hand2100.06 ± 203.152392.51 ± 228.422764.93 ± 78.71
ANYmal-D Rough17.09 ± 0.3118.69 ± 0.1519.54 ± 0.38
Franka Reach0.2034 ± 0.00570.2720 ± 0.00730.4945 ± 0.0054
Go2 Rough22.84 ± 0.0724.20 ± 0.1126.65 ± 0.89
Shadow Hand6111.49 ± 131.526580.81 ± 141.287046.48 ± 112.75

We also evaluated deterministic deployment, using zero latents for GenPO and mean angular coordinates for SteerGenPO. SteerGenPO likewise achieves higher mean evaluation returns on all six tasks:

TaskGenPO Stage 1GenPO Full BudgetSteerGenPO
Ant164.56 ± 6.39177.75 ± 1.61207.34 ± 12.84
Allegro Hand2657.43 ± 254.752965.24 ± 326.003052.26 ± 83.08
ANYmal-D Rough20.97 ± 0.4821.61 ± 0.3221.83 ± 0.59
Franka Reach0.3456 ± 0.02740.4925 ± 0.04290.5450 ± 0.0949
Go2 Rough25.43 ± 0.7926.46 ± 0.7727.76 ± 1.13
Shadow Hand7588.97 ± 191.927704.63 ± 294.587739.16 ± 58.19

Compared with full-budget GenPO, SteerGenPO improves mean training rewards by 4.55%–81.80% and deterministic evaluation returns by 0.45%–16.65%. This comparison contrasts full-budget GenPO training with GenPO pretraining followed by latent post-training with the generator frozen. SteerGenPO achieves higher mean training rewards and deterministic evaluation returns while keeping the generator frozen.

The learning curves and training rewards have been incorporated into Figure 4 and Table 2 of the revised manuscript, and the deterministic evaluation results and settings into Appendix C.1, Table 11.

2. W1(2) / Q2 — Deterministic evaluation on all six benchmarks

W1(2): Table 2 reports stochastic training reward only. No deterministic evaluation is reported for the Isaac Lab tasks, although Appendix B says the two returns differ.

Q2: Table 2 reports stochastic training reward. Please report deterministic evaluation returns for PPO (mean action), GenPO (zero latent and a fixed on-sphere latent), and SteerGenPO (mean angular coordinate) on all six Isaac Lab tasks.

Response to W1(2) & Q2: We have added deterministic evaluations on all six Isaac Lab tasks: SteerGenPO achieves the highest mean return on every task, consistent with the main training-return trends. PPO uses its mean action, GenPO uses zero or fixed on-sphere latents, and SteerGenPO uses the latent actor’s mean output. Both GenPO variants use the same Stage 1 models. The results are:

TaskPPO / Mean actionGenPO / ZeroGenPO / Fixed sphereSteerGenPO / Mean output
Ant117.29 ± 8.27164.56 ± 6.39109.62 ± 17.29207.34 ± 12.84
Franka Reach0.1337 ± 0.14680.3456 ± 0.0274−0.0594 ± 0.05260.5450 ± 0.0949
ANYmal-D Rough19.00 ± 0.1620.97 ± 0.4812.90 ± 1.4721.83 ± 0.59
Go2 Rough23.94 ± 0.4625.43 ± 0.7915.72 ± 2.7227.76 ± 1.13
Allegro Hand2412.61 ± 363.162657.43 ± 254.752184.50 ± 270.173052.26 ± 83.08
Shadow Hand6960.07 ± 124.607588.97 ± 191.925459.68 ± 871.397739.16 ± 58.19

The fixed spherical latent is z=TD(0)z=T_{\sqrt{D}}(0), held constant throughout evaluation.

These results support state-conditioned latent selection as a way to better realize the base generator’s learned control capabilities during deterministic inference. The PPO, zero-latent GenPO, and SteerGenPO results are included in Appendix C.1; the fixed-sphere comparison is provided here to address your question.

AuthorsOpenReview ↗

Response to Reviewer deWw — Part 2/5

3. W1(3) / Q3 — Immediate effect of switching to Stage 2

W1(3): The switch to Stage 2 changes the stochastic policy this reward is measured under, and the paper does not separate that immediate change from learning.

Q3: How much of the jump at the Stage 1/Stage 2 boundary in Figure 4 is already present at iteration 0 of Stage 2, before any latent-actor update? A single evaluation of the freshly initialized actor would answer this.

Response to W1(3) & Q3: We compared returns at the Stage 1 endpoint and iteration 0 of Stage 2 to quantify the immediate change before any latent-actor update. The absolute change in mean return is at most 3.01% on five tasks, while Franka Reach shows a 5.08% decrease. Both evaluations use stochastic sampling under matched conditions.

TaskStage 1 endpointStage 2, iteration 0Return increase (%)
Ant150.53 ± 10.27150.74 ± 10.02+0.14
Franka Reach0.2164 ± 0.00820.2054 ± 0.0058−5.08
ANYmal-D17.69 ± 0.2617.66 ± 0.23−0.17
Go222.20 ± 0.1122.03 ± 0.47−0.77
Allegro Hand2137.57 ± 203.232073.14 ± 206.99−3.01
Shadow Hand6244.86 ± 126.066231.21 ± 146.30−0.22

Percentage changes are calculated using the displayed mean returns; negative values indicate decreases.

These results show that switching to the untrained latent actor does not automatically improve performance, and its immediate effect varies across tasks.

4. W1(4) / Q4 — Ablations of the spherical constraint and latent selection

W1(4): The fixed-radius sphere, the main new idea, is never ablated. No box-constrained latent actor as in DSRL, no normalized output (which the paper mentions), no fixed latent on the sphere such as Tr(0).

Q4: Please compare the fixed-radius angular actor against (a) an unconstrained or box-constrained latent actor as in DSRL (b) a normalized D-dimensional output (c) a fixed latent on the sphere such as Tr(0), under the same budget.

Response to W1(4) & Q4: We have added the requested ablations on Unitree G1, where SteerGenPO achieves the highest mean test return and tracking success among all compared methods. The comparisons include unconstrained and box-constrained learning, L2 normalization of a DD-dimensional output, and fixed spherical latents. The fixed-sphere controls use Tr(0)T_r(0) and five randomly selected spherical directions, each held constant during deployment.

MethodTest return ↑Tracking success (%) ↑
Unconstrained61.97 ± 2.3741.14 ± 3.83
Box-constrained62.21 ± 1.9440.00 ± 15.65
Fixed spherical latent Tr(0)T_r(0)35.13 ± 16.472.86 ± 6.39
Fixed spherical latent (random directions)42.05 ± 6.506.63 ± 4.79
Sphere–L2 (DD-dimensional normalization)62.66 ± 0.5442.86 ± 0.00
SteerGenPO (Sphere–Angular)67.24 ± 0.2054.29 ± 11.95

Both learned spherical variants achieve higher mean test return and tracking success than the unconstrained and box-constrained variants, supporting the design of keeping latents at the training prior’s typical radial scale. At the same radius, SteerGenPO improves tracking success over Sphere–L2 from 42.86% to 54.29% and test return from 62.66 to 67.24, further supporting the angular parameterization.

The fixed spherical latents perform substantially worse, indicating that being on the sphere does not guarantee a latent that produces high-quality control behavior. The spherical constraint defines the search space, but learning is still needed. Complete ablation metrics for the learned variants are provided in Table 4 of the revised main text.

AuthorsOpenReview ↗

Response to Reviewer deWw — Part 3/5

5. Q5 — Value of generative pretraining and latent selection

Related concern in W1: But the paper never shows what the pretrained generative representation adds over directly learning a deterministic policy with the same compute.

Q5: Since the deployed policy is a deterministic function of the observation, what advantage does GenPO pretraining plus latent selection have over a deterministic policy with the same total compute, for example PPO with an equivalent low-noise second stage? Any evidence here would be helpful.

Response to Q5: Diffusion RL requires an additional mechanism for effective deterministic deployment, whereas PPO can directly deploy its Gaussian policy’s mean action. Generative policies such as GenPO produce actions through a nonlinear generation process without the same directly available mean-action output. They also offer greater flexibility in representing action distributions than single-Gaussian policies; the GenPO–PPO comparisons in Table 1 and Figure 4 demonstrate higher returns in our evaluated tasks.

SteerGenPO therefore freezes the generator and learns state-conditioned latent selection for deterministic deployment, aiming to bridge the sim-to-real deployment gap in diffusion RL while retaining the control capabilities acquired through generative pretraining.

We also tested PPO with the suggested low-noise second stage. In this setting, mean-action deployment still performed best; this training adjustment retains PPO’s Gaussian policy representation and its directly available deterministic mean-action output.

W2. Quantitative evidence for deployment performance

6. W2(1) / Q6 — Repeated deployment comparison with PPO

W2(1): The motivation is that diffusion RL beats PPO in simulation but is unreliable on robots. Yet there is no repeated-trial PPO comparison on the Table 3 metrics.

Q6: Please add PPO (mean action) to Table 3 under the same commands and perturbed initial states.

Response to W2(1) & Q6: We have added repeated evaluations of mean-action PPO to Table 3, using five training seeds and the same commands and perturbed initial states. SteerGenPO achieves higher mean tracking success and test return, as shown below:

MethodTracking success (%) ↑Test return ↑
PPO41.71 ± 20.3647.61 ± 13.13
SteerGenPO54.29 ± 11.9567.24 ± 0.20

SteerGenPO increases mean tracking success from 41.71% to 54.29% and test return from 47.61 to 67.24. Across the full evaluation, it achieves better mean performance than PPO on all eight metrics, including velocity-tracking errors, lateral deviation, trajectory dispersion, and action variation. Complete results are included in Table 3 of the revised manuscript.

Reviewer_CP74OpenReview ↗
Rating
3(−1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the deployment gap in RL-trained generative policies, like GenPO, by introducing a post-training steering framework that resolves the trade-off between action jitter from random latent sampling (in N(0,I)) and drift from fixed-zero latents — applied to high-frequency control in robotics. The authors propose freezing the pretrained GenPO generator and training a state-conditioned latent actor to deterministically select latent directions that steer GenPO.

Additionally, the actor output maps onto a hypersphere of radius r=Dr = \sqrt{D} to keep the generated latent in the region of max probability mass of N(0,ID)N(0,I_D).

Some results are shown for a G1 humanoid, mostly in sim and some in real. It achieves zero-shot physical hardware deployment without real-world fine-tuning.

Critical strengths

Elegant simplicity of method and easy comparison to baseline.

Critical weaknesses

Some missing ablations:

  1. Unconstrained Learned Latent (eg DSRL) vs. Spherical Latent. This could serve as an ablation for enforcing the spherical latent, or an exptl resutl with DSRL as a baseline.

  2. Analytic Angular Map vs. Euclidean Normalization: Section 4.2 notes that radius restriction could also be achieved by normalizing a D-dimensional actor output (z=Dv∥v∥z = \sqrt{D} \frac{v}{\|v\|}), but no empirical comparison is provided between norm-normalization and the angular coordinate map Tr(u)T_r(u).

  3. Stochastic vs. Deterministic Deployment of Actor: The authors claim that 'The stochastic training return and deterministic deployment return are different objectives in general and must be evaluated separately'. This makes sense, given the high nonlinearity of the downstream models that the latent goes into. It would be good to compare deploying the learned deterministic actor vs. stochastic actor, even if only in sim.

  4. Novelty I agree with reviewer 6iiF's view that SteerGenPO has "limited novelty compared to DSRL". It feels like applying DSRL to a different problem, with the added novelty of constraining the latents to a sphere.

Questions for authors

Real world deployment (Fig 7): For this expt, did you train PPO in real or sim? Do the Fig 7 results also have zero-shot sim2real, or is it just Fig 6?

4 comments
AuthorsOpenReview ↗

Response to Reviewer CP74 — Part 1/2

We thank you for your careful review and constructive suggestions, and for recognizing our method’s simplicity and clear comparison with the baseline. We address your suggestions below with additional experiments and clarifications.

W1: Unconstrained Learned Latent (eg DSRL) vs. Spherical Latent. This could serve as an ablation for enforcing the spherical latent, or an exptl resutl with DSRL as a baseline.

A1: Thank you for this suggestion. We have added the requested latent-constraint ablations on Unitree G1 to the main text, comparing unconstrained and box-constrained latents, along with the two spherical parameterizations raised in W2: L2 normalization and angular mapping. The unconstrained variant learns over the full latent space; the box-constrained variant bounds each component; both spherical variants restrict latents to a sphere with the same fixed radius. The table below reports command-tracking success in MuJoCo and test return in Isaac Lab.

MethodTracking success (%) ↑Isaac Lab test return ↑
Unconstrained41.14 ± 3.8361.97 ± 2.37
Box-constrained40.00 ± 15.6562.21 ± 1.94
Sphere–L242.86 ± 0.0062.66 ± 0.54
Sphere–Angular (Ours)54.29 ± 11.9567.24 ± 0.20

SteerGenPO achieves the highest mean tracking success and test return, as well as the best mean performance on all remaining reported deployment metrics. These results support our fixed-radius spherical design. Complete metrics are reported in Table 4 of the revised main text.

Regarding your suggestion to compare with DSRL, the two methods differ in their primary objectives: DSRL focuses on improving task performance through RL post-training, whereas our work addresses deterministic real-world deployment of diffusion RL policies. To address your suggestion, we have also adapted DSRL to a frozen GenPO generator and added a direct comparison on the same Unitree G1 task:

MetricDSRLSteerGenPO
Planar velocity RMSE (cm/s) ↓23.89 ± 7.9311.76 ± 0.32
Yaw-rate RMSE (rad/s) ↓0.25 ± 0.050.17 ± 0.01
Forward lateral RMSE (cm) ↓107.19 ± 69.0445.31 ± 26.68
Trajectory dispersion (cm) ↓14.39 ± 10.365.22 ± 0.88
Action-increment RMS (mrad) ↓27.38 ± 8.3822.80 ± 1.50
Joint-acceleration RMS (rad/s²) ↓14.38 ± 5.1911.05 ± 1.27
Tracking success (%) ↑14.29 ± 17.5054.29 ± 11.95
Isaac Lab test return ↑60.97 ± 2.2667.24 ± 0.20

Entries report means ± sample standard deviations across five random seeds.

SteerGenPO achieves better mean performance across all eight reported metrics in this humanoid-control setting.

W2: Analytic Angular Map vs. Euclidean Normalization: Section 4.2 notes that radius restriction could also be achieved by normalizing a D-dimensional actor output (z=D v∥v∥2z=\sqrt{D}\,\frac{v}{\|v\|_2}), but no empirical comparison is provided between norm-normalization and the angular coordinate map Tr(u)T_r(u).

A2: We choose angular coordinates because they directly represent spherical directions and satisfy the fixed-radius constraint by construction, without additional L2 normalization. For example, in two dimensions with radius 2\sqrt2, different output vectors produce the same normalized latent:

2(1,1)2=2(2,2)22=(1,1).\frac{\sqrt{2}(1,1)}{\sqrt{2}} = \frac{\sqrt{2}(2,2)}{2\sqrt{2}} = (1,1).

Angular coordinates eliminate this radial scaling redundancy, providing a more direct representation for latent-policy optimization.

To further evaluate this design, we have added a direct comparison between Sphere–L2 and Sphere–Angular on Unitree G1 at the same spherical radius:

MethodTracking success (%) ↑Forward lateral RMSE (cm) ↓Isaac Lab test return ↑
Sphere–L242.86 ± 0.0078.75 ± 40.3662.66 ± 0.54
Sphere–Angular (Ours)54.29 ± 11.9545.31 ± 26.6867.24 ± 0.20

The angular map improves tracking success by 11.43 percentage points, while reducing lateral deviation and increasing test return. These results provide empirical support for angular parameterization as a more suitable representation for latent-policy optimization. Thank you for suggesting this comparison; the results are now included in Table 4 of the revised main text.

AuthorsOpenReview ↗

Response to Reviewer CP74 — Part 2/2

W3: Stochastic vs. Deterministic Deployment of Actor: The authors claim that “The stochastic training return and deterministic deployment return are different objectives in general and must be evaluated separately”. This makes sense, given the high nonlinearity of the downstream models that the latent goes into. It would be good to compare deploying the learned deterministic actor vs. stochastic actor, even if only in sim.

A3: Following your suggestion, we have added a direct comparison of stochastic and deterministic deployment in Unitree G1 simulation, using the same trained SteerGenPO actor. Stochastic deployment samples angular coordinates at each control step; deterministic deployment uses their mean. Both share the spherical mapping and frozen generator, with no weight updates during evaluation. Results are means ± standard deviations across five paired sets of test conditions:

Deployment modeTrajectory dispersion (cm) ↓Action-increment RMS (mrad) ↓Joint-acceleration RMS (rad/s²) ↓
Sampled angular coordinates (stochastic)8.28 ± 0.8242.66 ± 0.0417.83 ± 0.03
Mean angular coordinates (deterministic)4.40 ± 0.5120.22 ± 0.018.97 ± 0.02

Deterministic deployment reduces trajectory dispersion, action-increment RMS, and joint-acceleration RMS by 46.9%, 52.6%, and 49.7%, respectively. These results support using mean angular coordinates to reduce execution variability introduced by stepwise sampling while retaining the actor’s ability to respond to changing observations and commands.

Q1: Real world deployment (Fig 7): For this expt, did you train PPO in real or sim? Do the Fig 7 results also have zero-shot sim2real, or is it just Fig 6?

A1: Both PPO and SteerGenPO in Figure 7 are trained entirely in simulation and deployed zero-shot on the physical robot. As in Figure 6, neither policy undergoes real-world fine-tuning or weight updates.

Figure 7 shows open-loop circular walking: both methods receive the same prescribed linear- and angular-velocity commands, without external position feedback for trajectory correction. All real-robot experiments in this paper use simulation-trained policies without real-world fine-tuning or weight updates.

We would be happy to clarify any remaining questions.

Reviewer_CP74OpenReview ↗

Reviewer CP74 response

Thank you for your responses.

Angular coordinates eliminate this radial scaling redundancy, providing a more direct representation for latent-policy optimization.

They remove scaling redundancy, but they have angular redundancy instead. Consider a 2-dim angular prediction in radians: a value (0,0)(0, 0) and (0,2π−ϵ)(0, 2 \pi - \epsilon) should give very similar final spherical vectors. Thus, different output vectors produce nearly the same latent. A way to work around this is to predict all-but-one column of an N×NN \times N rotation matrix for an n-dimensional vector, meaning you have to predict N(N−1)N(N-1) scalars [1]

The stochastic training return and deterministic deployment return are different objectives in general and must be evaluated separately

Thanks for adding expts to compare the stochastic and deterministic actors. It seems odd that all training was with stochastic actors and yet it is worse that the deterministic actor in every metric, even the std deviation 3σ3 \sigma bounds don't overlap. If you train with a stochastic actor in the loop (different objective that deterministic actor in highly nonlinear systems), then I would expect it to be better or at least comparable in some metric. Note that this would not be a concern in linear/quasi-linear systems. Please give more details about this evaluation, and your thoughts on these results.

W1(1): The main weakness is the limited novelty compared to DSRL. The core difference is that the proposed method constrains latents to a radius and steers them using angles.

I agree with reviewer 6iiF. It feels like applying DSRL to a different problem, with the added novelty of constraining the latents to a radius. However, application of one method to another domain could still be considered a contribution - this is subjective.

[1] "On the Continuity of Rotation Representations in Neural Networks"), Zhou et al. (2019)

AuthorsOpenReview ↗

Response to Reviewer CP74 — Follow-up

Q1: They remove scaling redundancy, but they have angular redundancy instead. Consider a 2-dim angular prediction in radians: a value (0,0)(0,0) and (0,2π−ϵ)(0,2\pi-\epsilon) should give very similar final spherical vectors. Thus, different output vectors produce nearly the same latent. A way to work around this is to predict all-but-one column of an N×NN\times N rotation matrix for an n-dimensional vector, meaning you have to predict N(N−1)N(N-1) scalars [1].

A1: Thank you for highlighting angular periodicity. Across 477,180 control steps from five models under multiple commands and initial conditions, 100% of the recorded angular outputs from deterministic and stochastic deployment lay within [−π,π][-\pi,\pi], with each coordinate’s observed span smaller than 2π2\pi. Our earlier statement referred specifically to radial scaling redundancy under L2 normalization.

The continuous rotation representation [1] introduces additional redundancy for our task: the latent direction has 57 degrees of freedom, whereas this construction predicts 3,306 values, and different full rotations can yield the same latent direction. This construction is therefore not well matched to our task.

[1] Zhou et al. On the Continuity of Rotation Representations in Neural Networks. CVPR, 2019.

Q2: It seems odd that all training was with stochastic actors and yet it is worse that the deterministic actor in every metric, even the std deviation 3σ3\sigma bounds don't overlap. If you train with a stochastic actor in the loop (different objective that deterministic actor in highly nonlinear systems), then I would expect it to be better or at least comparable in some metric. Note that this would not be a concern in linear/quasi-linear systems. Please give more details about this evaluation, and your thoughts on these results.

A2: Thank you for this helpful question. The original table reported standard deviations from repeated evaluations of a single model. We extended the comparison to five models and additional metrics, finding results consistent with your expectation: the two deployment modes have similar Planar velocity RMSE, Forward lateral RMSE, and Isaac Lab test return. Deterministic deployment nevertheless reduces mean Action-increment RMS by 36.1% and Joint-acceleration RMS by 31.3%, supporting our use of deterministic deployment to reduce action variation while retaining task performance.

MetricStochasticDeterministic
Planar velocity RMSE (cm/s) ↓11.581 ± 0.33411.758 ± 0.321
Yaw-rate RMSE (rad/s) ↓0.1802 ± 0.02070.1657 ± 0.0096
Forward lateral RMSE (cm) ↓47.86 ± 27.7246.55 ± 28.11
Trajectory dispersion (cm) ↓7.38 ± 2.535.38 ± 0.96
Action-increment RMS (mrad) ↓35.73 ± 13.8522.84 ± 1.49
Joint-acceleration RMS (rad/s²) ↓16.09 ± 4.9211.05 ± 1.27
Isaac Lab test return ↑66.892 ± 0.50567.251 ± 0.178

Q3: I agree with reviewer 6iiF. It feels like applying DSRL to a different problem, with the added novelty of constraining the latents to a radius. However, application of one method to another domain could still be considered a contribution - this is subjective.

A3: Thank you for your acknowledgement. Our core contribution is to address an underexplored deployment problem in diffusion RL: bridging the sim-to-real deployment gap through state-conditioned latent control to enable reliable and repeatable execution on a physical humanoid robot.

Reviewer_6iiFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This method proposes to train a noise selection module for diffusion policy based on input observation. The idea is that randomly sampled noise induces flickering during deployment, and learning a state to noise model makes sampling deterministic and can reduce flickering. The proposed approach is evaluated across simulated robot control benchmarks and physical Unitree G1 experiments, demonstrating improved command tracking and execution consistency over random sampling.

Critical strengths
  1. The idea of reducing flickering by state-to-noise deterministic sampling is sound. The method is useful and simple.
  2. Experimental results demonstrate improvements in both simulation and real hardware.
  3. The paper addresses a practically meaningful problem and demonstrates a useful way to recover adaptive behavior while preserving deterministic inference
Critical weaknesses
  1. The main weakness is the limited novelty compared to DSRL. The core difference is that the proposed method constrains latents to a radius and steers them using angles. Although the spherical constraint could be a useful improvement, the experiments does not include an ablation study to analyze the effectiveness of this component.
  2. Comparison to DSRL is missing. It is crucial for demonstrating the contribution of the proposed method, as it seems to be very similar to DSRL
  3. The GenPO stage is standalone and does not seem to be relevant to the proposed method. What's the purpose of including this stage?
  4. Deterministic inference does not by itself imply temporally smooth control. It only constrain the changes in action due to the noise, but not high-frequency changes due to the observations. For example, for image diffusion, when the text prompt is different, the same noise would produce very different images
Questions for authors
  1. Does the proposed latent-selection method depend on any particular property of a GenPO-trained generator? What motivates GenPO beyond its role as the base training algorithm?
  2. Proposition 1 establishes fixed-radius feasibility and sphere coverage. Beyond verifying the parameterization, does it provide an insight relevant to deployment performance or optimization? What motivates angular coordinates over direct vector normalization?
  3. Table 3 suggests that zero-latent inference already accounts for most of the reduction in action variation, while learned selection primarily improves tracking. Is this the intended interpretation? How does the method compare against a fixed nonzero latent on the same sphere, or a globally optimized constant latent?
  4. Relative to DSRL, what benefit comes from the fixed-radius spherical constraint?

This is a useful deployment paper but the method seems to be incremental. I am leaning toward borderline given its effectiveness and simplicity

7 comments
AuthorsOpenReview ↗

Response to Reviewer 6iiF — Part 1/4

We thank you for your time and thoughtful feedback. We appreciate your recognition of our method’s simplicity and practical value, as well as the improvements demonstrated in simulation and on real hardware. Below, we address your concerns with further explanations and additional experiments.

W1(1): The main weakness is the limited novelty compared to DSRL. The core difference is that the proposed method constrains latents to a radius and steers them using angles.

A1(1): We would like to clarify that DSRL and SteerGenPO differ in their research objectives and real-world applications. DSRL’s real-robot experiments focus on manipulation, whereas we study high-frequency humanoid locomotion and demonstrate zero-shot deployment on a physical humanoid robot. DSRL focuses on sample-efficient RL adaptation of pretrained diffusion and flow policies to improve task performance, while SteerGenPO addresses deterministic real-world deployment of diffusion RL policies.

Our work is motivated by the observation that diffusion RL policies can achieve strong simulation returns, yet latent-sampling randomness at deployment can compromise execution consistency and repeatability. Our state-conditioned latent post-training aims to bridge this sim-to-real deployment gap, enabling deterministic real-world command execution while retaining adaptation to the current state and command.

W1(2): Although the spherical constraint could be a useful improvement, the experiments does not include an ablation study to analyze the effectiveness of this component.

A1(2): To examine the effect of the spherical constraint, we have added latent-constraint ablations on Unitree G1, comparing unconstrained, box-constrained, and spherical latent learning (SteerGenPO). The table below reports command-tracking success in MuJoCo and test return in Isaac Lab. Our method achieves the highest mean tracking success and test return, as well as the best mean performance on all remaining reported deployment metrics, supporting our fixed-radius spherical constraint design. Complete results are reported in Table 4 of the revised main text.

The unconstrained variant learns without explicit bounds on the latent; the box-constrained variant bounds each component; our method restricts the latent to a sphere of radius D\sqrt D and learns its direction.

MetricUnconstrainedBox-constrainedSphere (Ours)
Tracking success (%) ↑41.14 ± 3.8340.00 ± 15.6554.29 ± 11.95
Isaac Lab test return ↑61.97 ± 2.3762.21 ± 1.9467.24 ± 0.20

Thank you for highlighting the need for this ablation; we have added the comparison to Table 4 of the revised main text.

W2: Comparison to DSRL is missing. It is crucial for demonstrating the contribution of the proposed method, as it seems to be very similar to DSRL

A2: As discussed in W1, the two methods differ in their primary objectives: DSRL focuses on improving task performance through RL post-training, while SteerGenPO addresses deterministic real-world deployment of diffusion RL policies. Nevertheless, to further address your concern, we have adapted DSRL to a frozen GenPO generator and added a direct comparison with SteerGenPO on Unitree G1:

MetricDSRLSteerGenPO
Planar velocity RMSE (cm/s) ↓23.89 ± 7.9311.76 ± 0.32
Yaw-rate RMSE (rad/s) ↓0.25 ± 0.050.17 ± 0.01
Forward lateral RMSE (cm) ↓107.19 ± 69.0445.31 ± 26.68
Trajectory dispersion (cm) ↓14.39 ± 10.365.22 ± 0.88
Action-increment RMS (mrad) ↓27.38 ± 8.3822.80 ± 1.50
Joint-acceleration RMS (rad/s²) ↓14.38 ± 5.1911.05 ± 1.27
Tracking success (%) ↑14.29 ± 17.5054.29 ± 11.95
Isaac Lab test return ↑60.97 ± 2.2667.24 ± 0.20

Entries report means ± sample standard deviations across five random seeds.

In this humanoid-control setting, SteerGenPO achieves higher tracking success and test return, and outperforms DSRL in mean performance across all reported metrics, including velocity-tracking errors, lateral deviation, trajectory dispersion, and action variation.

AuthorsOpenReview ↗

Response to Reviewer 6iiF — Part 2/4

W3: The GenPO stage is standalone and does not seem to be relevant to the proposed method. What's the purpose of including this stage?

Q1: Does the proposed latent-selection method depend on any particular property of a GenPO-trained generator? What motivates GenPO beyond its role as the base training algorithm?

Response to W3 and Q1: The GenPO stage trains the base generative policy, which SteerGenPO then freezes and steers through state-conditioned latent post-training for deterministic deployment. Our goal is to address the real-world deployment of diffusion RL policies, so we choose GenPO as a representative diffusion RL algorithm and comparison baseline.

SteerGenPO does not rely on GenPO-specific training objectives or algorithmic mechanisms. It operates through the latent inputs of a frozen generator and can therefore also be applied to other diffusion and flow-based RL policies with this interface.

W4: Deterministic inference does not by itself imply temporally smooth control. It only constrain the changes in action due to the noise, but not high-frequency changes due to the observations. For example, for image diffusion, when the text prompt is different, the same noise would produce very different images

A4: Some of our wording may have given this impression. Our method aims to reduce execution variability caused by latent-sampling randomness in diffusion RL and improve real-robot execution consistency, rather than guarantee temporally smooth control.

The importance of deterministic deployment is also evident in FPO++ [1], a related flow-based RL method: it fixes the initial noise to zero at test time and emphasizes the importance of zero-sampling for sim-to-real transfer. Zero-sampling can eliminate latent-sampling randomness, but cannot by itself ensure accurate command execution. Our experiments show that a fixed zero latent can still produce persistent trajectory drift. SteerGenPO therefore learns latents suited to the current state and command, retaining adaptation during deterministic deployment rather than suppressing action adjustments needed to respond to changing observations.

[1] Yi et al. Flow Policy Gradients for Robot Control. 2026.

AuthorsOpenReview ↗

Response to Reviewer 6iiF — Part 3/4

Q2: Proposition 1 establishes fixed-radius feasibility and sphere coverage. Beyond verifying the parameterization, does it provide an insight relevant to deployment performance or optimization? What motivates angular coordinates over direct vector normalization?

A2: The angular parameterization is better suited to latent-policy optimization because it directly represents spherical directions without redundant radial scaling or additional L2 normalization. Proposition 1 guarantees that the angular mapping exactly satisfies the fixed-radius constraint while covering the entire sphere. This connects directly to our prior-informed design: high-dimensional Gaussian probability mass concentrates in a shell near radius D\sqrt D. Choosing this representative radius keeps latent optimization at the training prior’s typical radial scale without excluding any on-sphere direction. The mapping also uses a fixed angular offset, θ=u+b\theta=u+b, so that zero actor output corresponds to a non-polar reference point, avoiding the coordinate degeneracy where changes in some angles leave the latent unchanged. Thus, the fixed-radius constraint keeps deployment latents at norm D\sqrt D, consistent with the training prior’s typical radial scale, while the offset provides a non-degenerate reference point for initializing angular optimization.

We choose angular coordinates because they directly represent spherical directions and satisfy the fixed-radius constraint by construction, without additional L2 normalization. For example, in two dimensions with r=2r=\sqrt2, different output vectors produce the same latent:

2(1,1)2=2(2,2)22=(1,1)\frac{\sqrt{2}(1,1)}{\sqrt{2}} = \frac{\sqrt{2}(2,2)}{2\sqrt{2}} = (1,1)

Indeed, any positive rescaling leaves the normalized latent unchanged. Angular coordinates directly represent the direction, eliminating this radial scaling redundancy and providing a more direct representation for latent-policy optimization. We have added a direct comparison on Unitree G1 at the same spherical radius:

MethodTracking success (%) ↑Planar velocity RMSE (cm/s) ↓Yaw-rate RMSE (rad/s) ↓Forward lateral RMSE (cm) ↓Trajectory dispersion (cm) ↓Action-increment RMS (mrad) ↓Joint-acceleration RMS (rad/s²) ↓Isaac Lab test return ↑
Sphere–L242.86 ± 0.0013.85 ± 0.590.22 ± 0.0278.75 ± 40.366.41 ± 1.0922.95 ± 1.0611.60 ± 0.6762.66 ± 0.54
Sphere–Angular (Ours)54.29 ± 11.9511.76 ± 0.320.17 ± 0.0145.31 ± 26.685.22 ± 0.8822.80 ± 1.5011.05 ± 1.2767.24 ± 0.20

At the same spherical radius, Sphere–Angular achieves better mean performance on all eight evaluation metrics, including higher tracking success and test return, as well as lower tracking errors, trajectory dispersion, and action-variation metrics. These results support the angular parameterization as a more suitable representation for latent-policy optimization in this humanoid-control setting. Complete results are included in Table 4 of the revised main text.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response by Authors

We sincerely thank all three reviewers for recognizing the strengths of our work. Reviewer 6iiF highlighted its simplicity and practical value, along with improvements in simulation and on real hardware. We thank 6iiF for the positive reassessment and increased score during the discussion. Reviewer CP74 appreciated the simple design and clear baseline comparison. Reviewer deWw recognized the deployment problem identified and quantified in our work, the method’s simplicity, and physical-robot validation. We also appreciate their constructive suggestions. During the rebuttal, we added experiments and clarifications in four areas.

1. Distinction from DSRL and direct comparison.

In response to 6iiF and CP74, we clarified the difference in research objectives: our work addresses the sim-to-real deployment gap of generative RL policies, while DSRL focuses on improving task performance through RL post-training. We added a direct comparison on Unitree G1, where SteerGenPO achieves better mean performance on all eight metrics. The complete results and experimental settings are included in Appendix J.

2. Spherical constraints and state-conditioned latent selection.

Following all three reviewers’ suggestions, we compared four latent-learning variants: unconstrained learning, component-wise box constraints, and fixed-radius spherical constraints implemented through either L2 normalization or angular parameterization. Both spherical variants achieve better mean performance on most deployment metrics, and Sphere–Angular achieves the best mean performance on all eight metrics (Table 4).

To examine whether fixed latents can replace state-conditioned selection, as questioned by 6iiF and deWw, we tested fixed spherical latents and globally optimized constant latents. Both perform worse than SteerGenPO in mean test return and tracking success. Repeated straight-line trials also reveal persistent drift under Zero inference: after 5 m of forward travel, absolute lateral deviation is 73.88 ± 10.49 cm for Zero and 6.80 ± 4.85 cm for SteerGenPO. These results support learning state-conditioned latents for accurate deployment.

3. Full-budget training and deterministic testing.

Following deWw’s requests, we added full-budget GenPO comparisons and deterministic tests of PPO, GenPO, and SteerGenPO on all six Isaac Lab tasks. SteerGenPO achieves higher mean training returns and deterministic test returns than full-budget GenPO on every task (Figure 4, Table 2, and Appendix C.1).

Isaac Lab benchmark returns alone cannot assess sim-to-real deployment performance. We therefore compared full-budget zero-latent GenPO and SteerGenPO using forward lateral RMSE in MuJoCo. Full-budget GenPO has a mean error of 131.04 cm, while SteerGenPO achieves 45.31 cm—a 65.42% reduction. SteerGenPO retains an advantage in deployment trajectory accuracy under the full-budget comparison.

4. Deployment comparisons and repeated evaluations.

Following CP74’s suggestion, we compared stochastic and deterministic deployment across five models and seven metrics. The two modes have similar mean planar velocity RMSE, forward lateral RMSE, and Isaac Lab test return. Deterministic deployment reduces mean action-increment RMS and joint-acceleration RMS by 36.1% and 31.3%, respectively.

To address deWw’s concerns about deployment baselines and quantitative evidence, we added PPO’s complete deployment metrics to Table 3 and conducted 30 trials per method across six reference paths. SteerGenPO achieves lower mean errors on all four path-tracking metrics (Appendix G.3).

We also added repeated-run statistics to Table 1 and Figure 2, provided command-wise returns and confidence intervals for paired differences, and clarified physical trial counts and zero-shot deployment settings.

Manuscript revisions are marked in red. Additional experiments and explanations are provided in the corresponding responses. We thank the reviewers again for helping us strengthen the experimental evidence and improve the presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.