acceptodds
Under review as a conference paper at ICLR 2027

Streaming Flow Policy Optimization

Abstract

Proximal policy optimization (PPO) with a diagonal Gaussian actor is the standard method for massively parallel robot learning, yet its exploration is memoryless: independent action noise is injected at every control step. Physical behaviors, by contrast, unfold as temporally coherent trajectories, so exploration should be coherent as well. We introduce Streaming Flow Policy Optimization (SFPO), which builds temporal persistence into the actor while preserving PPO’s simplicity and efficiency. Flow and diffusion policies generate structured action trajectories, but their flow time indexes the noise level: policy gradients must score a K-step denoising chain or a surrogate, not the executed action. Streaming flow policies, designed for imitation learning, instead identify flow time with execution time, advancing the flow one deterministic Euler step per control step. SFPO replaces this step with an Euler–Maruyama step, so each control-step transition is an exact diagonal Gaussian over the executed action. Augmenting the state with the previous action preserves PPO’s exact per-step likelihood, importance ratio, entropy, and KL divergence at one network evaluation per control step. A single retention parameter interpolates between memoryless PPO and persistent streaming integration, giving a prior for correlated exploration without restricting the learned policy. On 13 of 14 continuous-control, manipulation, and locomotion tasks, with policies trained from scratch, SFPO matches or exceeds PPO and recent diffusion and flow-policy methods at PPO’s cost, with PPO’s hyperparameters and one fixed retention setting. These results position SFPO as a drop-in replacement for PPO with temporally coherent exploration.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Streaming Flow Policy, PPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Nxh1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SFPO adds a previous-action carry term to a Gaussian policy’s mean to encourage temporally coherent exploration. The policy is trained with the usual PPO objective, retaining tractable conditional likelihoods and one network evaluation per action. Experiments report improved or competitive training returns across 14 tasks at approximately PPO’s computational cost.

Critical strengths
  • The method is simple, inexpensive, and compatible with existing PPO implementations.
  • The experiments cover a broad range of control tasks, with substantial gains on several robotics benchmarks using one fixed retention setting.
Critical weaknesses
  • The main results report shaped training returns, combining task progress with smoothness and other penalties. Higher return does not necessarily imply more reliable task completion. Manipulation success rates and locomotion tracking errors or failure rates would better establish the practical gains.
  • The real-world demonstrations establish transfer feasibility but provide no quantitative comparison with PPO or other baselines. It remains unclear whether SFPO’s simulation advantages translate into better performance on hardware.
  • The methodological advance over prior autoregressive policies is relatively narrow. The paper explains how the action carry induces temporal persistence, but this analysis largely concerns a fixed network output and does not establish when carrying the whole action is preferable to correlating the exploration residual. The empirical gains are promising, but the broader insight provided by this particular parameterization remains limited.
Questions for authors
  1. Do the improvements persist when evaluating without exploration noise using manipulation success rates and locomotion tracking errors or failure rates, rather than aggregate shaped return?
  2. Can you provide a quantitative hardware comparison with PPO under the same evaluation conditions?
  3. What evidence explains when carrying the previous action is preferable to correlating the exploration residual? Do the gains persist across different ARPPO smoothing settings?
2 comments

Thank you for the detailed review. We address your three questions.

Do the improvements persist without exploration noise, measured by task metrics? Yes. Without exploration noise, SFPO's mean task metric is better than PPO's on velocity tracking, skateboarding, motion tracking and reorientation, and its return is the highest on handstand; on dexterous lifting, pick-up and piano the three methods are within the seed spread. We evaluated the final checkpoints of all three seeds of PPO, ARPPO and SFPO with zero exploration noise and the carried action reset at each episode start; the full table is in our response to Reviewer L8ga. In order SFPO, PPO, ARPPO: velocity tracking ends in a fall on 0.7%, 47% and 0.5% of episodes; skateboarding fails on 8%, 39% and 11%; motion tracking succeeds on 98.5%, 96.3% and 93.0%, with body position error 0.026, 0.030 and 0.029; reorientation reaches 10.3, 3.7 (one PPO seed diverged) and 8.7 goals per episode; piano reaches key-press F1 0.942, 0.933 and 0.947. Dexterous lifting and pick-up have no success metric; their deterministic returns are 198, 117 and 204, and 145, 151 and 148. Per second of episode, SFPO's gain over PPO on velocity tracking, motion tracking, dexterous lifting and piano is in the task terms of the reward; its action-rate penalty is lower than PPO's on the first two and higher on the other two.

Quantitative hardware comparison with PPO. In place of robot trials, we ran the final PPO and SFPO checkpoints of all three seeds closed-loop in MuJoCo, an independent physics engine, on the two tasks we deployed on hardware: velocity tracking on the LimX Oli and motion tracking on the Unitree G1 (MuJoCo Menagerie model). Both methods ran the same 16 episodes per seed without exploration noise or randomization: velocity commands that change every 5 s on flat ground, and 10 s windows of the reference motion. We ran the same episodes in Genesis to measure each method's transfer gap. Mean over three seeds (Appendix E.7):

TaskMetricPPO, MuJoCoPPO, GenesisSFPO, MuJoCoSFPO, Genesis
Velocity trackingfalls0%0%0%0%
xy-velocity error (m/s)0.6140.6140.2570.251
yaw-rate error (rad/s)0.0520.0540.0770.067
action rate0.0360.0340.0840.081
mean torque (N·m)4.584.965.926.09
Motion trackingfalls0%0%0%0%
body position error (m)0.0250.0240.0210.020
body rotation error (rad)0.1000.0970.0780.071
joint position error (rad)0.480.490.330.32
action rate0.1240.1240.1220.120
jerk0.0730.0750.0680.068
mean torque (N·m)4.994.895.054.94

SFPO's advantage carries over to MuJoCo. On velocity tracking, its velocity error is less than half of PPO's in both engines. PPO's error equals the mean commanded speed (0.61 m/s), so its deterministic policy barely follows the command, which is also why its action rate and torque are lower; its yaw-rate error is lower than SFPO's. On motion tracking, SFPO's body, rotation and joint errors are 16–33% lower than PPO's in both engines, with a lower action rate and jerk and a torque about 1% higher than PPO's. Neither method falls, and the velocity, body position and joint position errors of both methods change by at most 8% between Genesis and MuJoCo.

When is carrying the whole action preferable to correlating the exploration residual, and do the gains persist across ARPPO settings? We ran two experiments.

First, we removed the carry from SFPO: kΔt = 1, so β = 0, with the same network inputs (including the previous action) and normalizer. Velocity tracking falls from 44.6 to 25.8, 24.6 and 27.7; skateboarding from 143.4 to 46.9, 63.5 and 43.6; and in-hand reorientation, in the runs of Fig. 3 at T = 1, from 1155 to 763 and 833 (the third seed diverged). PPO, which also observes the previous action, ends at 27.2 and 80.1 in the rerun and at 800 on reorientation (Fig. 2).

Second, we swept ARPPO. The paper's ARPPO uses AR(3) noise with smoothing α = 0.3. We added α = 0.5 and 0.8, and an AR(1) arm with α = 0.5. In AR(1) the noise coefficient is α, so this arm correlates only the residual, with the same coefficient that SFPO's carry applies to the whole action (β = 0.5). Final return per seed:

TaskAR(3), α = 0.3 (paper)AR(3), α = 0.5AR(3), α = 0.8AR(1), α = 0.5SFPO
Velocity tracking44.1 ± 0.319.6, 19.9, 19.76.2, 6.2, 6.243.9, 43.0, 42.944.6 ± 0.4
Humanoid skateboarding95.8 ± 23.6--145.7, 146.7, 111.5143.4 ± 2.9
Motion tracking38.2 ± 0.1--39.9, 39.9, 39.741.0 ± 0.1
In-hand reorientation874 ± 150272, 193, 309diverged (3 of 3)1231, 1005, 9751027 ± 243
Dexterous lifting183.6 ± 16.9168.2, 154.3, 39.833.6, 33.8, 33.2190.9, 162.1, 156.9185.4 ± 78.8

ARPPO is sensitive to its noise settings. Raising the AR(3) smoothing from 0.3 to 0.5 cuts its return to 19.7 on velocity tracking and to 258 on reorientation, and at α = 0.8 it learns none of the three tasks; the paper's setting is the best of the three. The AR(1) arm, which correlates only the residual with SFPO's coefficient, is the ARPPO variant closest to SFPO, and we ran it on five tasks. SFPO's mean return is higher on four of them: velocity tracking (44.6 against 43.3), skateboarding (143 against 135), motion tracking (41.0 against 39.8; on both tracking tasks every SFPO seed ends above every AR(1) seed) and dexterous lifting (185 against 170). On reorientation the two are within the seed spread (1027 against 1070).

On the three robotics tasks we ran, removing the carry while keeping the previous-action input loses the gains, so there they come from the carry, not from the input. The AR(1) arm, which correlates the noise with the same coefficient, comes close, so in our tasks carrying the whole action and correlating the residual reach similar returns. SFPO builds this correlation into the policy through the retention β, with one setting, kΔt = 0.5, on all 14 tasks and at 0.90–1.11× PPO's update cost. ARPPO comes close to SFPO only at some noise settings, the paper's AR(3) on velocity tracking and dexterous lifting and the AR(1) arm, set to SFPO's coefficient, on the five tasks above, and a wrong choice can cost most of the return. ARPPO also pays for the noise process in every update: the paper's AR(3) form costs a median 1.52× PPO's update (Table 2), and on reorientation, on L40S nodes of the same configuration, AR(1) and AR(3) cost 1.26× and 1.24× PPO's update against SFPO's 0.97× (on H200, AR(3) 1.22× against 0.99×).

The analysis concerns a fixed network output. We agree, and we measured the effective coefficient of trained policies (table in our response to Reviewer L8ga). The learned field keeps the carry near β on four robotics tasks, strengthens it on velocity tracking and dexterous lifting, and weakens or reverses it on piano and handstand. The stationary spread of the linearized process stays within 15% of the reference process on all eight.

Reviewer_L8gaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Streaming Flow Policy Optimization (SFPO), which incorporates temporally correlated exploration into PPO by adding a fixed carry term from the previous action to the mean of a Gaussian policy.

The authors interpret this construction as an Euler–Maruyama update of Streaming Flow Policy. By augmenting the state with the previous action and flow phase, the policy retains tractable conditional likelihoods, importance ratios, entropy, and KL divergence.

Experiments cover six DeepMind Control tasks, eight robotics tasks, and qualitative deployment on two physical robots, and SFPO achieves higher or comparable stochastic training returns on most evaluated tasks at approximately PPO’s computational cost.

Critical strengths
  1. The method is simple as well as elegant and preserves PPO’s main computational advantages. Equation (9) requires one network evaluation per action, while Equations (10), (14), and (15) provide directly computable likelihoods, entropy, and KL divergence. Table 2 reports update costs of 0.90–1.11 times PPO’s across eight robotics tasks, with similarly close rollout costs. This combination of performance and computational efficiency is practically valuable for large-scale robot learning.

  2. The reported benefits extend across several tasks rather than a single favorable example. Figures 1–2 cover different robots, action dimensions, and reward structures. Reported improvements over PPO are approximately 65% on velocity tracking, 47% on skateboarding, and 46% on in-hand reorientation. SFPO also exceeds the implemented ARPPO baseline on in-hand reorientation and dexterous lifting. Using a fixed main retention setting and PPO’s optimization hyperparameters further strengthens the practical contribution.

Critical weaknesses
  1. The ablations do not fully isolate the proposed temporal-exploration mechanism. The stationary variance and autocorrelation formulas in Equation (11) and Appendix A.12 assume a fixed network output. The actual network depends on the previous action and closed-loop state, and the paper explicitly allows it to cancel or amplify the carry. For example, setting vθ(a)=ca changes the effective autoregressive coefficient to β+Δt·c. The actual stationary variance is then no longer determined by β alone. Consequently, Figure 3 matches the scale of a fixed-drive reference process; this does not establish that the learned policies have matched exploration spread throughout training. In addition, Appendix B freezes SFPO’s actor-normalization statistics without clearly stating whether PPO and ARPPO receive identical treatment, and the DMC PPO baseline lacks the same previous-action input. The single-task ablation provides partial evidence, but these uncertainties weaken attribution of the gains specifically to the carry term and its temporal-exploration prior.

  2. The main evaluation measures stochastic training policies and does not establish the extent of the final deployment advantage. Section 5 reports training returns that include exploration noise, while seven of the eight robotics tasks penalize action differences. Correlated noise can directly change these penalties, so improved training return need not translate fully into improved control with exploration disabled. Appendix A.12 usefully discusses this issue and shows that smoothness penalties do not decrease uniformly across tasks. Nevertheless, matched-checkpoint deterministic evaluations and task-level reward decompositions are missing. Hardware results are qualitative, simulation results use three seeds per method, and the closest ARPPO comparator uses one fixed order and smoothing setting. This supports a promising method, but the “matches or exceeds on 13 of 14 tasks” result should not be interpreted as a broadly established or statistically significant advantage. This limits my recommendation to weak acceptance.

Questions for authors
  1. Implementation and attribution: Do PPO, ARPPO, and SFPO use identical actor-normalization update schedules? On DMC, do the main gains persist when PPO receives the same previous-action input?

  2. Performance validation: For the main gain-driving tasks—velocity tracking, skateboarding, and in-hand reorientation—do SFPO’s improvements over PPO and ARPPO persist when evaluating the existing final checkpoints from all three seeds without exploration noise? Please distinguish task rewards from action-difference penalties.

  3. The notation and terminology around temporal retention could be made more precise and consistent. Section 4.3 defines the retention coefficient as β=1−kΔt, whereas kΔt is also referred to as “retention,” including in Figure 3. Although their mathematical relationship is explicitly stated, the two quantities have opposite interpretations: increasing β strengthens the explicit carry term, while increasing kΔt weakens it.

3 comments

Thank you for the detailed review. We answer your questions in order.

Do PPO, ARPPO and SFPO use identical actor-normalization schedules? No. On the five tasks where their actors normalize observations, PPO and ARPPO update the normalizer at every rollout step; SFPO normalizes on all eight tasks and freezes the actor's statistics as described in Appendix B. To measure the effect, we reran PPO and ARPPO with SFPO's frozen schedule, with the same code and seeds. Final return, mean ± standard deviation over three seeds:

TaskPPOPPO, frozenARPPOARPPO, frozenSFPO
Velocity tracking27.2 ± 0.327.3 ± 0.544.1 ± 0.344.1 ± 0.344.6 ± 0.4
Skateboarding80.1 ± 31.264.9 ± 37.295.8 ± 23.686.9 ± 17.5143.4 ± 2.9
In-hand reorientation551 (2 seeds; 1 diverged)847 ± 341874 ± 150735 ± 1961027 ± 243
Pick-up110.9 ± 5.9117.2 ± 5.798.2 ± 4.187.0 ± 5.6114.7 ± 7.8
Bimanual piano65.8 ± 0.666.7 ± 0.566.7 ± 0.566.7 ± 0.365.9 ± 0.6
Quadruped handstand19.7 ± 0.0no normalizer19.7 ± 0.0no normalizer19.8 ± 0.0

On velocity tracking and skateboarding, freezing PPO's normalizer does not close the gap: frozen PPO stays as far behind on the first and falls further behind on the second. On reorientation, freezing raises PPO from 551, where the unfrozen PPO lost one seed to divergence, to 847, still below SFPO's 1027 and within the seed spread. On pick-up, frozen PPO ends slightly above SFPO (117.2 against 114.7, within one standard deviation). On piano all five arms end within 1 of each other. Freezing lowers ARPPO's return on skateboarding, reorientation and pick-up and leaves it unchanged on velocity tracking and piano. On motion tracking, dexterous lifting and handstand, PPO's and ARPPO's actors have no observation normalizer, so a frozen arm would be PPO itself; SFPO's actor normalizes there too.

On DMC, do the gains persist when PPO receives the previous action? Partly. We ran an arm that isolates the input: SFPO's actor at kΔt = 1, so β = 0 and the mean is the network output, with the previous action as an input. It is a Gaussian PPO policy that observes the previous action, with SFPO's parameterization of σ and its frozen normalizer. Final return, three seeds (Acrobot Swingup was not run with this arm):

TaskPPOPPO with previous action (kΔt = 1)SFPO (kΔt = 0.5)
Ball-in-cup Catch911 ± 38940 ± 21955 ± 8
Finger Spin738 ± 123884 ± 39908 ± 9
Cheetah Run462 ± 42575 ± 16558 ± 34
Hopper Hop86 ± 25175 ± 103124 ± 45
Walker Run333 ± 27433 ± 186549 ± 56

On DeepMind Control, on four of the five tasks most of SFPO's gain over PPO is already present in this arm; on Walker Run the carry adds more than the arm's other changes (549 against 433, from PPO's 333). SFPO ends above this arm on Ball-in-cup, Finger Spin and Walker Run and below it on Cheetah Run and Hopper Hop, each within the seed spread. On the robotics tasks PPO already observes the previous action, so a comparison there measures the carry alone, and on the three tasks we ran the same kΔt = 1 arm falls to PPO's level or below. Final return per seed:

TaskPPO (3 seeds)kΔt = 1, per seedSFPO (3 seeds)
Velocity tracking27.225.8, 24.6, 27.744.6
Skateboarding80.146.9, 63.5, 43.6143.4
In-hand reorientation (runs of Fig. 3, T = 1)800 (Fig. 2)763, 833, diverged1155

With the network inputs and normalizer held fixed, removing the carry removes the gain on these three tasks (on reorientation the initial σ also differs, set for the same stationary spread).

The variance analysis assumes a fixed network output. We agree. With a learned field the effective autoregressive coefficient is J = β·I + Δt·∂vθ/∂a(t−1), as your example shows, and the analysis behind Fig. 3 describes only the reference process. We measured J on trained policies, at initialization and at the final checkpoint, on the states each policy visits when it samples as in training (256 environments, every 25 steps, two episodes each; three seeds at the final checkpoint, one at initialization). We report the mean diagonal of J, its mean spectral radius, and the stationary action spread of the linearized process x(t) = J̄x(t−1) + ε(t) relative to the reference process with J = β·I:

TaskDiagonal of J, init → finalSpectral radius, finalStationary spread / reference, final
Velocity tracking0.50 → 0.630.821.14
Skateboarding0.50 → 0.520.661.03
Motion tracking0.50 → 0.510.691.02
In-hand reorientation0.50 → 0.500.601.00
Dexterous lifting0.50 → 0.550.861.14
Pick-up0.51 → 0.520.781.10
Bimanual piano0.50 → 0.410.510.96
Quadruped handstand0.50 → −0.180.820.99

At initialization J ≈ β·I on every task. After training, the diagonal stays at β on skateboarding, motion tracking, reorientation and pick-up, rises on velocity tracking and dexterous lifting, falls on piano, and is negative on handstand, where PPO, ARPPO and SFPO end at nearly the same return. The mean spectral radius is 0.51–0.86, below 1 on every task. The stationary spread of the linearized process stays within 15% of the reference process on all eight tasks, including those where the diagonal moves; we measured it only for the default kernel, not for the other values of kΔt in Fig. 3. We also measured the lag-ℓ autocorrelation of executed actions, which is normalized by each action's own variance. SFPO's is close to PPO's on velocity tracking (lag 1: 0.84 against 0.85) and motion tracking (0.91 against 0.91), and lower on skateboarding (0.85 against 0.89) and reorientation (0.89 against 0.92). In absolute terms, SFPO's action-rate penalty with exploration noise on is 11% below PPO's on velocity tracking and 13% below on motion tracking, per second of episode, close to the 12% and 14% that Section 5 reports from the training rollouts. The revised PDF adds this table as Appendix E.5 and states the fixed-field assumption next to Fig. 3.

Hardware results are qualitative. We could not run new hardware trials during the discussion period. For the new Appendix E.7, we ran the final PPO and SFPO checkpoints of all three seeds closed-loop in MuJoCo, an independent physics engine, on both hardware tasks (table in our response to Reviewer Nxh1). SFPO's velocity error is 0.26 m/s against PPO's 0.61 (PPO's yaw-rate error is lower, 0.052 rad/s against 0.077), and its joint position error in motion tracking 0.33 rad against 0.48, with no falls for either method.

Deterministic evaluation of the final checkpoints. We evaluated the final checkpoints of all three seeds of PPO, ARPPO and SFPO with the exploration noise set to zero and the carried action reset at the start of each episode, for two episodes on each of 1,024 copies of the training environment. Mean ± standard deviation over seeds:

TaskMeasurePPOARPPOSFPO
Velocity trackingepisodes ending in a fall47 ± 5%0.5 ± 0.1%0.7 ± 0.3%
Velocity trackingreturn59.7 ± 2.178.7 ± 0.179.1 ± 0.0
Skateboardingepisodes ending in a failure39 ± 49%11 ± 7%8 ± 2%
Skateboardingreturn121 ± 66161 ± 33191 ± 4
Motion trackingsuccess rate96.3 ± 0.8%93.0 ± 0.7%98.5 ± 0.0%
Motion trackingbody position error0.0300.0290.026
In-hand reorientationgoals reached per episode3.7 ± 6.4 (1 seed diverged)8.7 ± 2.610.3 ± 3.6
Dexterous liftingreturn117 ± 66204 ± 29198 ± 104
Pick-upreturn151 ± 16148 ± 7145 ± 7
Bimanual pianokey-press F10.933 ± 0.0110.947 ± 0.0060.942 ± 0.007
Quadruped handstandreturn19.7 ± 0.119.7 ± 0.019.8 ± 0.0

SFPO has the highest return or task metric on velocity tracking, skateboarding, motion tracking, reorientation and handstand. ARPPO is higher on dexterous lifting and piano, and PPO on pick-up (151 against 148 and 145), each within the seed spread. In the same evaluation with exploration noise on, SFPO's pick-up return is the highest of the three (117 against 112 and 100). We also split the deterministic return into task terms and action-rate penalties, using the per-term episode sums that the task code logs, divided by each episode's duration, so that an early fall does not shrink a penalty. Per second, SFPO's task terms exceed PPO's by 0.23 on velocity tracking, 0.08 on motion tracking, 13.5 on dexterous lifting and 0.01 on piano. Its action-rate penalty is lower than PPO's on velocity tracking (by 0.007) and motion tracking (0.006), and higher on dexterous lifting (0.09) and piano (0.0002). On these four tasks the gains are in the task terms. On skateboarding, SFPO's push, steer and transition terms sum to 7.9 per second against 6.5 for PPO and ARPPO; on reorientation its task terms are 29.5 against ARPPO's 25.8, with a smaller action-rate penalty (2.4 against 3.5). On pick-up with exploration noise on, SFPO's task terms are 1.0 below PPO's and its penalty is 1.9 lower, so its lead there comes from the penalty.

Three seeds, and the "13 of 14" claim. We added seeds 7103 and 7104 on velocity tracking, skateboarding and reorientation. Final return over five seeds, with the interquartile mean (IQM) and its 95% bootstrap confidence interval:

TaskPPOARPPOSFPO
Velocity tracking, mean ± s.d.26.8 ± 0.944.1 ± 0.244.5 ± 0.4
Velocity tracking, IQM [95% CI]27.1 [25.8, 27.5]44.2 [43.9, 44.4]44.4 [44.2, 45.0]
Skateboarding, mean ± s.d.89.1 ± 30.594.4 ± 26.6144.3 ± 4.0
Skateboarding, IQM [95% CI]98.9 [54.9, 109.9]97.7 [64.4, 115.5]144.0 [140.2, 148.7]
In-hand reorientation, per seed936, 700, 699, 166, diverged1071, 845, 760, 707, 4401225, 1171, 1098, 686, diverged
In-hand reorientation, IQM522771985

Final return is the mean of the last 100 iterations, as in the paper. On velocity tracking and skateboarding, every SFPO seed ends above every PPO seed, and SFPO's IQM interval lies above PPO's. On skateboarding it also lies above ARPPO's; on velocity tracking SFPO's IQM is 0.2 above ARPPO's, with overlapping intervals. On reorientation SFPO has the highest IQM, but five seeds are too few for a useful interval. SFPO and PPO each lost one seed to divergence, and a resample that draws the diverged seed twice pulls the IQM down, so the intervals for those two arms have no useful lower bound. The revised PDF replaces "matches or exceeds on 13 of 14 tasks" with "has the highest mean final return on 12 of the 14 tasks and ties PPO on a thirteenth", and reports these intervals in Appendix E.2.

ARPPO uses one order and one smoothing setting. We swept the AR(3) smoothing α ∈ {0.3, 0.5, 0.8} on velocity tracking, reorientation and dexterous lifting, and added an AR(1) arm with α = 0.5, whose coefficient equals SFPO's β, on those three tasks plus skateboarding and motion tracking. ARPPO's return falls as its AR(3) smoothing grows, and SFPO's mean return is above even the AR(1) arm's on four of the five tasks and within the seed spread on the fifth. The table is in our response to Reviewer Nxh1.

Retention terminology. Thank you. The revised PDF uses "retention" only for β = 1 − kΔt and refers to kΔt only by its symbol, including on the axis of Fig. 3.

Reviewer_LDTFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Streaming Flow Policy Optimization (SFPO). This method introduce temporal action persistence into PPO through an common Euler–Maruyama update. The method preserves exact Gaussian likelihoods and demonstrates promising results across continuous-control tasks. Despite encouraging empirical results, SFPO appears to be a relatively straightforward reparameterization of Gaussian PPO, with limited theoretical contribution and insufficient isolation of its empirical benefits.

Critical strengths

The method is computationally efficient and compatible with standard PPO optimization.

Lots of extensive experiments demonstrate encouraging improvements across diverse robotic tasks.

Critical weaknesses

SFPO essentially adds a previous-action residual connection to Gaussian PPO. Similar ideas appear in autoregressive policies, Streaming Flow Policy, and Action-to-Action Flow Matching (A2A).

Limited theoretical contribution. The exact likelihood follows from standard Gaussian identities, while the AR(1) analysis assumes a fixed velocity field and may not characterize learned policies.

Optimization stability. As in A2A, using previous actions changes the effective source distribution over training, unlike a fixed Gaussian prior. The implications of this evolving distribution for optimization stability deserve further analysis.

Insufficient experimental isolation. Most robotics tasks in appendixs penalize action variation, potentially favoring SFPO. Stronger baselines and evaluations are needed.

Questions for authors

refer my weakness

2 comments

Thank you for the review. We address each of your concerns.

SFPO adds a previous-action residual to Gaussian PPO, similar to autoregressive policies, Streaming Flow Policy and A2A. SFPO's policy is a Gaussian conditioned on the state augmented with the previous action, and Appendix A.5 states that a PPO actor that observes the previous action can represent every SFPO mean. Its contribution is the exploration prior it places on that class, at PPO's cost and with PPO's exact likelihood ratio. The new experiments test the prior. With the same actor, inputs and normalizer, removing the carry (kΔt = 1) lowers the mean over three seeds from 44.6 to 26.0 on velocity tracking and from 143.4 to 51.3 on skateboarding, and lowers reorientation, in the runs of Fig. 3, from 1155 to 763 and 833 (the third seed diverged). The previous-action input alone does not produce the gains on these tasks.

The three prior methods differ from SFPO as follows.

  • Autoregressive policies (ARP). ARP correlates the exploration noise around a mean recomputed at every step; SFPO's carry acts on the executed action. ARPPO is our closest baseline. In our rerun, SFPO's mean final return is above ARPPO's on seven of the eight robotics tasks (all but piano), by more than the seed spread on motion tracking, skateboarding, pick-up and handstand. SFPO's update costs 0.90–1.11× PPO's against ARPPO's median 1.52× (Table 2). Our response to Reviewer Nxh1 adds a sweep of ARPPO's smoothing and an AR(1) arm matched to SFPO's β. ARPPO's return drops at larger AR(3) smoothing, and SFPO's mean return is above even the AR(1) arm's on four of the five tasks we ran and within the seed spread on the fifth.
  • Streaming Flow Policy (SFP). SFPO builds on SFP, as Sec. 2 and Sec. 4.2 state. SFP is trained by imitation and has no action likelihood for RL. SFPO makes the likelihood exact through the Euler–Maruyama step, which lets PPO train a streaming flow policy.
  • Action-to-Action Flow Matching (A2A; Jia et al., 2026). Like A2A, SFPO flows from past actions: its source distribution is the distribution of the previous step's action. A2A learns from demonstrations, regressing a flow from an embedding of the action history to future actions, and does not evaluate an action likelihood. SFPO is trained with RL rather than with a supervised diffusion or flow-matching objective, and uses an exact per-step Gaussian likelihood for PPO's ratio. The revised PDF cites A2A in Related Work (Section 2) with this distinction.

Limited theory: the exact likelihood follows from Gaussian identities. We chose the Gaussian kernel so that the likelihood would be exact. It gives SFPO temporal structure at one network evaluation per action with an exact ratio, while DPPO needs K evaluations per action and FPO++ optimizes a surrogate (Table 1, Table 2). The paper's theoretical results are Proposition 1, which shows that any stateful policy with an exact density over the augmented state gets PPO's exact ratio; Proposition 2 (Appendix A.11), which shows that PPO is SFPO at β = 0 with a network that ignores the previous action; and the closed-form entropy and KL of Section 4.4 and Appendix A.6. We agree that the AR(1) analysis assumes a fixed velocity field. On trained policies (our response to Reviewer L8ga), the effective coefficient stays at or above β on six of the eight robotics tasks, and the stationary spread of the linearized process stays within 15% of the reference process on all eight.

Optimization stability under an evolving source distribution. In SFPO the source distribution is the distribution of the previous action, which is also part of the augmented state, and Proposition 1 shows that the augmented transition does not depend on θ (proof in Appendix A.9). The per-step ratio is therefore exact for any distribution of previous actions. A shift in that distribution during training is an on-policy shift in the state distribution, which every PPO run has. As in PPO, the clip and the KL-adaptive learning rate limit how far each update moves the policy, and SFPO computes that KL in closed form (Eq. 15). SFPO uses PPO's clip, KL target, learning rate and epochs unchanged on all 14 tasks (Sec. 5). In the training diagnostics of the new Appendix E.6, SFPO's KL on velocity tracking and skateboarding settles at 0.015 and 0.014, inside the band of 0.005 to 0.02 where the KL-adaptive rule leaves the learning rate unchanged. The rule cuts the rate on about a third of the iterations for both methods, and SFPO's rate settles about three times higher than PPO's. None of the 24 SFPO runs on the eight robotics tasks in our rerun diverged. Of the extra seeds, one SFPO seed diverged on in-hand reorientation, which trains every method at a fixed learning rate, and PPO lost one of its five seeds there too. In both runs the value loss and the gradient norm grew by orders of magnitude while σ stayed within the range of the other seeds.

Most robotics tasks penalize action variation, which may favor SFPO. Seven of the 14 tasks have no action penalty: the six DeepMind Control tasks use the suite's rewards, and quadruped handstand has none. SFPO's final return is above PPO's on all seven. Appendix A.12 addresses the confound on the penalized tasks. SFPO's action-rate penalty is similar to or higher than PPO's on dexterous lifting, reorientation and piano. In the reorientation sweep of Fig. 3, the integrator (k = 0) has the same per-step noise as kΔt = 0.5, and the noise alone predicts a 36% lower action-smoothness penalty for it; its measured penalty is instead slightly higher (6.8 against 6.4), which suggests that the learned mean, not the noise filter, sets the penalty at the end of training. We also split the return of a deterministic evaluation of the final checkpoints into task terms and action-rate penalties. Per second of episode, SFPO's task terms exceed PPO's by 0.23 on velocity tracking and 13.5 on dexterous lifting, while its action-rate penalty is 0.007 lower and 0.09 higher; on these tasks the gains come from the task terms. On pick-up, SFPO's lead under exploration noise comes from a lower action-rate penalty, with task terms 1.0 below PPO's, which is the confound you describe.

On stronger baselines, the paper compares against ARPPO, the closest prior method, and against DPPO and FPO++, and the new sweep covers four ARPPO noise settings (Appendix E.3).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading. Two questions recur across the three reviews: whether the carry causes SFPO's gains, rather than another difference between SFPO and PPO, and whether the gains hold under stricter evaluation. We list the experiments we ran for them below and give the details in the response to each reviewer.

  1. PPO and ARPPO with SFPO's frozen observation normalizer (L8ga). We kept the code and seeds and changed only when the normalizer updates. With the frozen normalizer, PPO stays below SFPO on velocity tracking (27.3 against 44.6) and skateboarding (64.9 against 143.4). On in-hand reorientation it rises from 551 to 847, within the seed spread of SFPO's 1027, and on pick-up it ends slightly above SFPO (117.2 against 114.7).
  2. Memoryless SFPO, kΔt = 1 (L8ga, Nxh1, LDTF). This arm keeps SFPO's network, its inputs (including the previous action) and its normalizer, and sets the retention β to 0. On the three robotics tasks we ran, where PPO already observes the previous action, removing the carry lowers the return to PPO's level or below. The arm ends at 26.0 on velocity tracking, against 44.6 for SFPO and 27.2 for PPO, and at 51.3 on skateboarding, against 143.4 and 80.1. In the reorientation runs behind Fig. 3 it ends at 763 and 833, against 1155 for SFPO and 800 for PPO in Fig. 2; the third memoryless seed diverged.
  3. Five seeds on velocity tracking, skateboarding and reorientation, with interquartile means (IQM) and bootstrap confidence intervals (L8ga). On velocity tracking and skateboarding, every SFPO seed ends above every PPO seed. On reorientation, SFPO and PPO each lost one of five seeds to divergence; the IQM is 985 for SFPO, 771 for ARPPO and 522 for PPO.
  4. ARPPO sweep: AR(3) smoothing α ∈ {0.3, 0.5, 0.8}, and an AR(1) arm whose coefficient equals SFPO's β = 0.5 (Nxh1, L8ga). ARPPO is sensitive to its noise settings: raising the AR(3) smoothing from the paper's 0.3 to 0.5 cuts its return from 44.1 to 19.7 on velocity tracking and from 874 to 258 on reorientation, and at 0.8 it learns none of the three tasks. The AR(1) arm, which copies SFPO's coefficient, comes closest, and SFPO's mean return is still higher on four of five tasks (skateboarding 143 against 135, motion tracking 41.0 against 39.8) and within the seed spread on reorientation. SFPO uses one setting, kΔt = 0.5, on all 14 tasks.
  5. Deterministic evaluation of the final checkpoints with task metrics and reward decompositions (L8ga, Nxh1). Without exploration noise, SFPO falls on 0.7% of velocity-tracking episodes against PPO's 47%, fails on 8% of skateboarding episodes against PPO's 39%, and reaches 10.3 goals per reorientation episode against ARPPO's 8.7. On velocity tracking, motion tracking, dexterous lifting and piano, SFPO's gain over PPO is in the task terms of the reward, not in the action-rate penalty; on pick-up, its lead under exploration noise comes from a lower penalty.
  6. Effective retention of trained policies, measured from the Jacobian of the kernel mean with respect to the previous action (L8ga, Nxh1, LDTF). The learned field keeps the carry at or above β on six of the eight robotics tasks: the mean diagonal of the effective coefficient stays near β = 0.5 on four, rises on velocity tracking (0.63) and dexterous lifting (0.55), falls on piano (0.41) and turns negative on handstand (−0.18). The stationary spread of the linearized process stays within 15% of the reference process on all eight.
  7. Sim-to-sim transfer to MuJoCo of the PPO and SFPO policies trained in Genesis, on the two tasks we deployed on hardware: velocity tracking on the LimX Oli and motion tracking on the Unitree G1 (Nxh1). SFPO's advantage carries over: its velocity error is 0.26 m/s against PPO's 0.61, and its joint position error in motion tracking 0.33 rad against 0.48, with no falls for either method.

We uploaded a revised PDF with every change in blue. It uses "retention" only for β = 1 − kΔt, including on the axis of Fig. 3; cites and discusses Action-to-Action Flow Matching (A2A); replaces "matches or exceeds on 13 of 14 tasks" with "has the highest mean final return on 12 of the 14 tasks and ties PPO on a thirteenth"; and adds Appendix E with the new experiments: the memoryless kernel (E.1), five seeds with interval estimates (E.2), ARPPO's noise settings (E.3), the deterministic evaluation (E.4), the effective retention of trained policies (E.5), optimization stability (E.6) and sim-to-sim transfer to MuJoCo (E.7).

Apart from values attributed to Fig. 2, Fig. 3 or Table 2, the robotics numbers in this response come from a rerun of the eight tasks with the paper's task definitions and settings; Fig. 2 keeps the original runs. In wall-clock time, SFPO reaches PPO's final return in 20–56% of PPO's training time on seven of the eight robotics tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.