- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that when one flow policy controls a whole humanoid, gradients from different body parts interfere. It proposes FFPO, which splits the policy into body-part agents updated sequentially as in HAPPO: the ratio of already updated parts uses a first-order tangent estimate, the current part uses the CFM surrogate, and the joint update is rescaled to a target KL. It reports a 14.4% higher reward AUC than the strongest baseline on 16 tasks and a Unitree G1 deployment.
Critical strengths
Among the compared methods, FFPO reports the best results on a broad benchmark of 16 tasks, with the highest mean reward AUC and final reward and the best AUC on 12 of the 16 tasks. If these results hold under fair comparisons, they make a strong case for the method.
Critical weaknesses
W1. The baselines do not receive FFPO's step control, and the source of the gain is not isolated.
In the component ablation of Appendix D.6, the largest gain comes from the step control, which rescales the joint update to a target KL and could also be applied to FPO++; the one-agent variant there may be such a comparison, but its configuration is not described. The cross-body noise tax that motivates the factorization is not checked either, since the signal and noise of the cross-body component are never measured on the tasks. Whether the tangent prefix improves learning over the CFM prefix is also unclear, as the three-task prefix ablation shows no significant aggregate difference. Without a Gaussian HAPPO baseline using the same partition, it is also unclear whether a flow policy is needed. It therefore remains unclear how much of the improvement comes from the factorization and the flow policy rather than from the step control.
W2. The baseline results are hard to reconcile with prior work and cannot be checked.
Humanoid motion tracking is routinely learned with a single Gaussian PPO policy, as in PHC by Luo et al. (2023), yet on this benchmark, where most tasks track a single reference motion, Gaussian PPO ends several tasks below the early-training level of the baselines. Yi et al. (2026) report that Gaussian PPO slightly outperforms FPO++ in humanoid motion tracking once entropy regularisation and a KL-adaptive learning rate are added, so the large gap here may reflect the baseline settings. The baselines' own actor settings and tuning budget are given neither in the paper nor in the supplementary code, so these results cannot be reproduced. Since the headline gain is measured against these baselines, its size is hard to trust.
Luo, Z., Cao, J., Winkler, A., Kitani, K., Xu, W. Perpetual Humanoid Control for Real-time Simulated Avatars. ICCV 2023.
Questions for authors
Q1 (W1). Could you compare FFPO, on the ablation tasks or ideally all 16, with Gaussian PPO, Gaussian HAPPO with the same partition and FPO++, each given the same step control, and with FFPO using the CFM prefix or a two-agent partition? Clear gains would confirm what the factorization, prefix estimator and flow policy contribute.
Q2 (W2). Could you report each baseline's actor settings, tuning budget and wall-clock time, release their configurations, and explain why Gaussian PPO degrades late in training on several tasks? If the baselines stay clearly behind when tuned as carefully as FFPO, my concern in W2 would be resolved.
14 comments
Official Comment by Authors (Part 2/2)
Q4: Gaussian PPO degradation in late training, and the effect of body factorization and step control.
| Method | Update | Normalized ESS (%) | Within range fraction (%) |
|---|---|---|---|
| Gaussian PPO | 1000 | 94.73 | 66.60 |
| Gaussian PPO | 3000 | 2.27 | 2.15 |
| FFPO | 1000 | 96.47 | 69.24 |
| FFPO | 3000 | 77.51 | 36.13 |
We conduct experiment on Crate Carry and measure the Normalized Effective Sample Size to show the effective sample fraction, and within range fraction to show the fraction of samples within an acceptable ratio range.
Gaussian PPO learns effective behavior by mid-training, but later updates can disrupt this behavior. Most sampled ratios remain within range early in training, while both measures fall sharply for late Gaussian PPO. Concretely, as lower-body tracking error rises from 0.06 to 0.09, growing gradient contributions from that body affect the whole policy through the joint ratio and shared actor parameters. This cross-body interference can accumulate over repeated updates, shifting previously learned behaviors and moving more ratios out of clip range, which weakens the corrective learning signal and contributes to the eventual performance collapse. FFPO's factorized gradients and independent body actors isolate these updates and substantially reduce cross-body interference. Step control then permits larger calibrated updates for faster learning, while FFPO's higher late-stage ESS and within range fraction in the table are consistent with its more stable learning and stronger task performance. In other tasks that Gaussian PPO behaves degradation, we also observe similar dynamics.
Q5: The baselines' wall-clock training and inference costs.
| Method | Time to 3,000 updates (h) | Time to 90% target (h) | Single-step inference (ms) |
|---|---|---|---|
| Gaussian PPO | 1.182 | Not reached | 0.128 |
| FPO | 2.982 | Not reached | 0.189 |
| FPO++ | 4.340 | 1.249 | 0.794 |
| Gaussian HAPPO | 2.669 | 0.516 | 0.463 |
| GenPO | 1.476 | 0.445 | 2.169 |
| PolicyFlow | 1.224 | 0.791 | 0.205 |
| FFPO | 5.703 | 1.116 | 0.640 |
We profile the baselines and FFPO on Normal Walk using an RTX 5090, with one process per GPU and three seeds. The table reports the wall-clock cost of 3,000 updates and the time to reach a common reward target, defined as 90% of the mean FFPO peak using a 200-update trailing average. Gaussian HAPPO, GenPO and PolicyFlow reach this target earlier on Normal Walk, while Gaussian PPO and FPO do not reach it within the training budget. FFPO reaches the target in 1.116 h, taking 10.7% less wall-clock time than FPO++, as its lower update count offsets part of its higher per-update cost.
I thank the authors for the additional comparisons, which show that step control helps FPO++ and that the flow policy matters on Crate Carry.
Q1, FPO++ with step control reaches a mean AUC of 0.881 against 0.883 for FFPO, over two seeds, and FFPO leads only on Crate Carry. Could you clarify what the factorization adds beyond the step control, with per-seed values, and why the one-agent variant in Appendix D.6 averages 0.704 when FPO++ with step control, also a single flow actor, reaches 0.881?
Q3, Tuning alone roughly doubles the AUC of Gaussian PPO, and GenPO, the reference for the 14.4% gain, runs with repository defaults while FFPO was tuned per task. Could you say whether Table 1 will be regenerated with tuned baselines and with FPO++ plus step control?
Official Comment by Authors (Part 1/2)
We thank the reviewer for the detailed feedback. Below are our responses.
Q1: Baselines with step control and the source of the gain.
The table reports normalized reward AUC over two training seeds; Mean AUC is the equal-weight average over the three tasks.
Overall, careful tuning improves the baselines, while FFPO retains stronger average performance across tasks, particularly on complex interaction tasks such as Crate Carry. To show the gain of step control, we observe that adding step control to FPO++ substantially improves its performance, confirming the controller’s contribution, while body factorization provides further gains when combined with body factorization on Crate Carry.
Q2: Measurement of the cross-body signal and noise.
The relevant quantities of the cross-body noise tax can be measured in the tasks. Please refer to our response to Reviewer xumL, Q1.
Q3: Actor settings and tuning budgets.
The tables summarize the baseline actor settings and selected task-specific configurations. Gaussian PPO uses KL-adaptive learning rates on all three tasks and entropy regularization on Crate Carry. We also update the code in the anonymous link (https://ffpo.pages.dev/) to provide the baseline settings. Please refer to our response to Reviewer xumL, Q3, for the search ranges and tuning budgets.