- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SFPO adds a previous-action carry term to a Gaussian policy’s mean to encourage temporally coherent exploration. The policy is trained with the usual PPO objective, retaining tractable conditional likelihoods and one network evaluation per action. Experiments report improved or competitive training returns across 14 tasks at approximately PPO’s computational cost.
Critical strengths
- The method is simple, inexpensive, and compatible with existing PPO implementations.
- The experiments cover a broad range of control tasks, with substantial gains on several robotics benchmarks using one fixed retention setting.
Critical weaknesses
- The main results report shaped training returns, combining task progress with smoothness and other penalties. Higher return does not necessarily imply more reliable task completion. Manipulation success rates and locomotion tracking errors or failure rates would better establish the practical gains.
- The real-world demonstrations establish transfer feasibility but provide no quantitative comparison with PPO or other baselines. It remains unclear whether SFPO’s simulation advantages translate into better performance on hardware.
- The methodological advance over prior autoregressive policies is relatively narrow. The paper explains how the action carry induces temporal persistence, but this analysis largely concerns a fixed network output and does not establish when carrying the whole action is preferable to correlating the exploration residual. The empirical gains are promising, but the broader insight provided by this particular parameterization remains limited.
Questions for authors
- Do the improvements persist when evaluating without exploration noise using manipulation success rates and locomotion tracking errors or failure rates, rather than aggregate shaped return?
- Can you provide a quantitative hardware comparison with PPO under the same evaluation conditions?
- What evidence explains when carrying the previous action is preferable to correlating the exploration residual? Do the gains persist across different ARPPO smoothing settings?
2 comments
When is carrying the whole action preferable to correlating the exploration residual, and do the gains persist across ARPPO settings? We ran two experiments.
First, we removed the carry from SFPO: kΔt = 1, so β = 0, with the same network inputs (including the previous action) and normalizer. Velocity tracking falls from 44.6 to 25.8, 24.6 and 27.7; skateboarding from 143.4 to 46.9, 63.5 and 43.6; and in-hand reorientation, in the runs of Fig. 3 at T = 1, from 1155 to 763 and 833 (the third seed diverged). PPO, which also observes the previous action, ends at 27.2 and 80.1 in the rerun and at 800 on reorientation (Fig. 2).
Second, we swept ARPPO. The paper's ARPPO uses AR(3) noise with smoothing α = 0.3. We added α = 0.5 and 0.8, and an AR(1) arm with α = 0.5. In AR(1) the noise coefficient is α, so this arm correlates only the residual, with the same coefficient that SFPO's carry applies to the whole action (β = 0.5). Final return per seed:
| Task | AR(3), α = 0.3 (paper) | AR(3), α = 0.5 | AR(3), α = 0.8 | AR(1), α = 0.5 | SFPO |
|---|---|---|---|---|---|
| Velocity tracking | 44.1 ± 0.3 | 19.6, 19.9, 19.7 | 6.2, 6.2, 6.2 | 43.9, 43.0, 42.9 | 44.6 ± 0.4 |
| Humanoid skateboarding | 95.8 ± 23.6 | - | - | 145.7, 146.7, 111.5 | 143.4 ± 2.9 |
| Motion tracking | 38.2 ± 0.1 | - | - | 39.9, 39.9, 39.7 | 41.0 ± 0.1 |
| In-hand reorientation | 874 ± 150 | 272, 193, 309 | diverged (3 of 3) | 1231, 1005, 975 | 1027 ± 243 |
| Dexterous lifting | 183.6 ± 16.9 | 168.2, 154.3, 39.8 | 33.6, 33.8, 33.2 | 190.9, 162.1, 156.9 | 185.4 ± 78.8 |
ARPPO is sensitive to its noise settings. Raising the AR(3) smoothing from 0.3 to 0.5 cuts its return to 19.7 on velocity tracking and to 258 on reorientation, and at α = 0.8 it learns none of the three tasks; the paper's setting is the best of the three. The AR(1) arm, which correlates only the residual with SFPO's coefficient, is the ARPPO variant closest to SFPO, and we ran it on five tasks. SFPO's mean return is higher on four of them: velocity tracking (44.6 against 43.3), skateboarding (143 against 135), motion tracking (41.0 against 39.8; on both tracking tasks every SFPO seed ends above every AR(1) seed) and dexterous lifting (185 against 170). On reorientation the two are within the seed spread (1027 against 1070).
On the three robotics tasks we ran, removing the carry while keeping the previous-action input loses the gains, so there they come from the carry, not from the input. The AR(1) arm, which correlates the noise with the same coefficient, comes close, so in our tasks carrying the whole action and correlating the residual reach similar returns. SFPO builds this correlation into the policy through the retention β, with one setting, kΔt = 0.5, on all 14 tasks and at 0.90–1.11× PPO's update cost. ARPPO comes close to SFPO only at some noise settings, the paper's AR(3) on velocity tracking and dexterous lifting and the AR(1) arm, set to SFPO's coefficient, on the five tasks above, and a wrong choice can cost most of the return. ARPPO also pays for the noise process in every update: the paper's AR(3) form costs a median 1.52× PPO's update (Table 2), and on reorientation, on L40S nodes of the same configuration, AR(1) and AR(3) cost 1.26× and 1.24× PPO's update against SFPO's 0.97× (on H200, AR(3) 1.22× against 0.99×).
The analysis concerns a fixed network output. We agree, and we measured the effective coefficient of trained policies (table in our response to Reviewer L8ga). The learned field keeps the carry near β on four robotics tasks, strengthens it on velocity tracking and dexterous lifting, and weakens or reverses it on piano and handstand. The stationary spread of the linearized process stays within 15% of the reference process on all eight.
Thank you for the detailed review. We address your three questions.
Do the improvements persist without exploration noise, measured by task metrics? Yes. Without exploration noise, SFPO's mean task metric is better than PPO's on velocity tracking, skateboarding, motion tracking and reorientation, and its return is the highest on handstand; on dexterous lifting, pick-up and piano the three methods are within the seed spread. We evaluated the final checkpoints of all three seeds of PPO, ARPPO and SFPO with zero exploration noise and the carried action reset at each episode start; the full table is in our response to Reviewer L8ga. In order SFPO, PPO, ARPPO: velocity tracking ends in a fall on 0.7%, 47% and 0.5% of episodes; skateboarding fails on 8%, 39% and 11%; motion tracking succeeds on 98.5%, 96.3% and 93.0%, with body position error 0.026, 0.030 and 0.029; reorientation reaches 10.3, 3.7 (one PPO seed diverged) and 8.7 goals per episode; piano reaches key-press F1 0.942, 0.933 and 0.947. Dexterous lifting and pick-up have no success metric; their deterministic returns are 198, 117 and 204, and 145, 151 and 148. Per second of episode, SFPO's gain over PPO on velocity tracking, motion tracking, dexterous lifting and piano is in the task terms of the reward; its action-rate penalty is lower than PPO's on the first two and higher on the other two.
Quantitative hardware comparison with PPO. In place of robot trials, we ran the final PPO and SFPO checkpoints of all three seeds closed-loop in MuJoCo, an independent physics engine, on the two tasks we deployed on hardware: velocity tracking on the LimX Oli and motion tracking on the Unitree G1 (MuJoCo Menagerie model). Both methods ran the same 16 episodes per seed without exploration noise or randomization: velocity commands that change every 5 s on flat ground, and 10 s windows of the reference motion. We ran the same episodes in Genesis to measure each method's transfer gap. Mean over three seeds (Appendix E.7):
SFPO's advantage carries over to MuJoCo. On velocity tracking, its velocity error is less than half of PPO's in both engines. PPO's error equals the mean commanded speed (0.61 m/s), so its deterministic policy barely follows the command, which is also why its action rate and torque are lower; its yaw-rate error is lower than SFPO's. On motion tracking, SFPO's body, rotation and joint errors are 16–33% lower than PPO's in both engines, with a lower action rate and jerk and a torque about 1% higher than PPO's. Neither method falls, and the velocity, body position and joint position errors of both methods change by at most 8% between Genesis and MuJoCo.