acceptodds
Under review as a conference paper at ICLR 2027

Coherence-Guided Factorized Flow Policy for Efficient Humanoid Whole-Body Control

Abstract

Flow-matching policies can represent complex, multimodal action distributions, providing an expressive policy model for humanoid whole-body control including locomotion, balance, and physical interaction. However, existing methods commonly optimize all joints within a single flow network, where gradients from heterogeneous body parts can interfere, obscuring reliable improvement directions and constraining effective policy updates. To address this problem, we propose Factorized Flow Policy Optimization (FFPO), which decomposes the whole-body flow policy into functional body agents and improves them sequentially while retaining parallel action generation during execution. Within this sequential process, likelihood-ratio prefixes account for preceding agents’ policy updates when evaluating each body update. And body-wise gradient coherence, estimated using a gradient-SNR proxy, guides the allocation of a fixed optimization-epoch budget. FFPO then synchronously rescales the resulting candidate parameter updates according to their aggregate log-ratio magnitude, allowing reliable local improvements to accumulate under a controlled joint policy update. To implement these updates efficiently, FFPO uses a first-order flow-ratio estimator for the prefixes and a differentiable flow-matching proxy for current-agent gradients. Our theoretical analysis characterizes the policy-update cost of uncertain cross-body interactions and establishes conditions under which body-wise factorization yields greater expected improvement. Across 16 simulated locomotion, balance, and loco-manipulation tasks, FFPO improves mean normalized reward AUC by 14.4% and mean normalized final reward by 14.1% over the strongest evaluated baseline for each metric. Deployment on the Unitree G1 further demonstrates coordinated whole-body motion on hardware.

Primary area:
reinforcement learning
Keywords:
Whole-Body Control, Reinforcement Learning, Embodied Intelligent, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WbBPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that when one flow policy controls a whole humanoid, gradients from different body parts interfere. It proposes FFPO, which splits the policy into body-part agents updated sequentially as in HAPPO: the ratio of already updated parts uses a first-order tangent estimate, the current part uses the CFM surrogate, and the joint update is rescaled to a target KL. It reports a 14.4% higher reward AUC than the strongest baseline on 16 tasks and a Unitree G1 deployment.

Critical strengths

Among the compared methods, FFPO reports the best results on a broad benchmark of 16 tasks, with the highest mean reward AUC and final reward and the best AUC on 12 of the 16 tasks. If these results hold under fair comparisons, they make a strong case for the method.

Critical weaknesses

W1. The baselines do not receive FFPO's step control, and the source of the gain is not isolated.

In the component ablation of Appendix D.6, the largest gain comes from the step control, which rescales the joint update to a target KL and could also be applied to FPO++; the one-agent variant there may be such a comparison, but its configuration is not described. The cross-body noise tax that motivates the factorization is not checked either, since the signal and noise of the cross-body component are never measured on the tasks. Whether the tangent prefix improves learning over the CFM prefix is also unclear, as the three-task prefix ablation shows no significant aggregate difference. Without a Gaussian HAPPO baseline using the same partition, it is also unclear whether a flow policy is needed. It therefore remains unclear how much of the improvement comes from the factorization and the flow policy rather than from the step control.

W2. The baseline results are hard to reconcile with prior work and cannot be checked.

Humanoid motion tracking is routinely learned with a single Gaussian PPO policy, as in PHC by Luo et al. (2023), yet on this benchmark, where most tasks track a single reference motion, Gaussian PPO ends several tasks below the early-training level of the baselines. Yi et al. (2026) report that Gaussian PPO slightly outperforms FPO++ in humanoid motion tracking once entropy regularisation and a KL-adaptive learning rate are added, so the large gap here may reflect the baseline settings. The baselines' own actor settings and tuning budget are given neither in the paper nor in the supplementary code, so these results cannot be reproduced. Since the headline gain is measured against these baselines, its size is hard to trust.

Luo, Z., Cao, J., Winkler, A., Kitani, K., Xu, W. Perpetual Humanoid Control for Real-time Simulated Avatars. ICCV 2023.

Questions for authors

Q1 (W1). Could you compare FFPO, on the ablation tasks or ideally all 16, with Gaussian PPO, Gaussian HAPPO with the same partition and FPO++, each given the same step control, and with FFPO using the CFM prefix or a two-agent partition? Clear gains would confirm what the factorization, prefix estimator and flow policy contribute.

Q2 (W2). Could you report each baseline's actor settings, tuning budget and wall-clock time, release their configurations, and explain why Gaussian PPO degrades late in training on several tasks? If the baselines stay clearly behind when tuned as carefully as FFPO, my concern in W2 would be resolved.

14 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We thank the reviewer for the detailed feedback. Below are our responses.

Q1: Baselines with step control and the source of the gain.

MethodStand AUCTurn-Jog AUCCrate Carry AUCMean AUC
Tuned Gaussian PPO0.9190.7920.3440.685
Gaussian PPO + step control0.9080.8380.3030.683
Tuned FPO++0.1610.5840.6700.472
FPO++ + step control0.8920.8950.8570.881
Gaussian HAPPO + step control0.8920.8560.2800.676
FFPO, two agents0.8520.8620.9030.872
FFPO, four agents, CFM prefix0.7370.8940.8010.811
FFPO, four agents, tangent prefix0.8060.8840.9590.883

The table reports normalized reward AUC over two training seeds; Mean AUC is the equal-weight average over the three tasks.

  • Step control. We apply the same task-specific step controller to tuned Gaussian PPO and FPO++; the one-agent FFPO ablation retains FFPO's flow construction with one actor covering all joints. Adding step control improves FPO++ on all three tasks, while its effect on Gaussian PPO is task-dependent.
  • Tangent prefix. With four agents, the tangent prefix improves Mean AUC, especially on Crate Carry, and diagnostics show lower gradient error than the CFM prefix. Late Turn-Jog variability is associated with prefix weights concentrating on a small subset of states, making updates more sensitive to sampling.
  • Gaussian HAPPO. This baseline uses the same four-way partition and step control, with approximately matched actor capacity. FFPO achieves higher Mean AUC, with its strongest advantage on Crate Carry, supporting the value of the flow-based scheme for coordinated support and manipulation.

Overall, careful tuning improves the baselines, while FFPO retains stronger average performance across tasks, particularly on complex interaction tasks such as Crate Carry. To show the gain of step control, we observe that adding step control to FPO++ substantially improves its performance, confirming the controller’s contribution, while body factorization provides further gains when combined with body factorization on Crate Carry.

Q2: Measurement of the cross-body signal and noise.

The relevant quantities of the cross-body noise tax can be measured in the tasks. Please refer to our response to Reviewer xumL, Q1.

Q3: Actor settings and tuning budgets.

MethodActor architecturePolicy-specific setting
Gaussian PPOThree 256-unit ELU hidden layersDiagonal Gaussian with tanh-squashed actions
FPOThree 256-unit ELU hidden layersConditional flow, CFM objective
FPO++Three 256-unit ELU hidden layersConditional flow, 64-step Euler sampling, per-sample asymmetric objective
GenPOThree 256-unit ELU hidden layersFive-step augmented coupling flow
PolicyFlowThree 256-unit ELU hidden layersConditional flow with learned Gaussian residual
Gaussian HAPPOTwo 156-unit ELU hidden layersFour independent Gaussian body actors, capacity matched
Selected method / taskLearning rateEntropy coefficientLearning-rate scheduleClipping parameterActor epochs
Gaussian PPO / Stand1e−40KL-adaptive, target 0.010.232
Gaussian PPO / Turn-Jog1e−40KL-adaptive, target 0.010.232
Gaussian PPO / Crate Carry1e−40.005KL-adaptive, target 0.010.232
FPO++ / Stand1e−40Fixed0.116
FPO++ / Turn-Jog1e−40Fixed0.132
FPO++ / Crate Carry1e−40Fixed0.132

The tables summarize the baseline actor settings and selected task-specific configurations. Gaussian PPO uses KL-adaptive learning rates on all three tasks and entropy regularization on Crate Carry. We also update the code in the anonymous link (https://ffpo.pages.dev/) to provide the baseline settings. Please refer to our response to Reviewer xumL, Q3, for the search ranges and tuning budgets.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

Q4: Gaussian PPO degradation in late training, and the effect of body factorization and step control.

MethodUpdateNormalized ESS (%)Within range fraction (%)
Gaussian PPO100094.7366.60
Gaussian PPO30002.272.15
FFPO100096.4769.24
FFPO300077.5136.13

We conduct experiment on Crate Carry and measure the Normalized Effective Sample Size (∑iri)2/(N∑iri2),ri=πnew(ai∣si)/πold(ai∣si)(\sum_i r_i)^2/(N\sum_i r_i^2),r_i=\pi_{\mathrm{new}}(a_i\mid s_i)/\pi_{\mathrm{old}}(a_i\mid s_i) to show the effective sample fraction, and within range fraction N−1∑i1[1−ϵ≤ri≤1+ϵ],ϵ=0.2N^{-1}\sum_i\mathbf{1}[1-\epsilon\le r_i\le1+\epsilon],\epsilon=0.2 to show the fraction of samples within an acceptable ratio range.

Gaussian PPO learns effective behavior by mid-training, but later updates can disrupt this behavior. Most sampled ratios remain within range early in training, while both measures fall sharply for late Gaussian PPO. Concretely, as lower-body tracking error rises from 0.06 to 0.09, growing gradient contributions from that body affect the whole policy through the joint ratio and shared actor parameters. This cross-body interference can accumulate over repeated updates, shifting previously learned behaviors and moving more ratios out of clip range, which weakens the corrective learning signal and contributes to the eventual performance collapse. FFPO's factorized gradients and independent body actors isolate these updates and substantially reduce cross-body interference. Step control then permits larger calibrated updates for faster learning, while FFPO's higher late-stage ESS and within range fraction in the table are consistent with its more stable learning and stronger task performance. In other tasks that Gaussian PPO behaves degradation, we also observe similar dynamics.

Q5: The baselines' wall-clock training and inference costs.

MethodTime to 3,000 updates (h)Time to 90% target (h)Single-step inference (ms)
Gaussian PPO1.182Not reached0.128
FPO2.982Not reached0.189
FPO++4.3401.2490.794
Gaussian HAPPO2.6690.5160.463
GenPO1.4760.4452.169
PolicyFlow1.2240.7910.205
FFPO5.7031.1160.640

We profile the baselines and FFPO on Normal Walk using an RTX 5090, with one process per GPU and three seeds. The table reports the wall-clock cost of 3,000 updates and the time to reach a common reward target, defined as 90% of the mean FFPO peak using a 200-update trailing average. Gaussian HAPPO, GenPO and PolicyFlow reach this target earlier on Normal Walk, while Gaussian PPO and FPO do not reach it within the training budget. FFPO reaches the target in 1.116 h, taking 10.7% less wall-clock time than FPO++, as its lower update count offsets part of its higher per-update cost.

I thank the authors for the additional comparisons, which show that step control helps FPO++ and that the flow policy matters on Crate Carry.

Q1, FPO++ with step control reaches a mean AUC of 0.881 against 0.883 for FFPO, over two seeds, and FFPO leads only on Crate Carry. Could you clarify what the factorization adds beyond the step control, with per-seed values, and why the one-agent variant in Appendix D.6 averages 0.704 when FPO++ with step control, also a single flow actor, reaches 0.881?

Q3, Tuning alone roughly doubles the AUC of Gaussian PPO, and GenPO, the reference for the 14.4% gain, runs with repository defaults while FFPO was tuned per task. Could you say whether Table 1 will be regenerated with tuned baselines and with FPO++ plus step control?

Reviewer_xumLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the efficiency and stability of flow-policy optimization for humanoid whole-body control, motivated by gradient interference between body parts with different control roles. It proposes Factorized Flow Policy Optimization (FFPO), which decomposes the policy into functional body agents and updates them sequentially while retaining parallel action generation during execution. Earlier body updates are accounted for through likelihood-ratio prefixes computed by a tangent flow estimator, while the current body is optimized using a differentiable flow-matching surrogate. A gradient-SNR proxy guides the allocation of a fixed optimization budget, and a common rescaling factor controls the combined policy change. A local theoretical analysis characterizes the cost of noisy cross-body updates and identifies conditions under which factorization yields greater expected improvement. Experiments on 16 simulated locomotion, balance, and loco-manipulation tasks report gains of 14.4% in mean normalized reward AUC and 14.1% in mean normalized final reward over the strongest evaluated baseline for each metric. Deployment on a Unitree G1 further demonstrates coordinated whole-body motion on hardware.

Critical strengths

The asymmetric ratio-estimation design is a useful technical contribution. Previously updated bodies require numerically accurate prefix weights, whereas the current body needs an inexpensive differentiable objective. FFPO addresses these requirements with a cached tangent estimator and a CFM surrogate, respectively, and combines the resulting sequential updates with joint step control. This offers a practical way to exploit functional body structure in flow-policy optimization while retaining parallel action generation at execution time.

The evaluation provides convincing evidence for the overall method. Across 16 tasks and five seeds under a common interaction budget, FFPO improves both mean normalized reward AUC and mean normalized final reward over the evaluated baselines (Section 5.2). The ablations in Section 5.4 and Appendix D.6 show clear benefits from body factorization and joint step control, while the G1 deployment demonstrates coordinated motion on physical hardware. The simulation results are complemented by G1 hardware demonstrations that clearly illustrate coordinated whole-body motion and successful completion of the reference motions.Together, these results support the method's learning efficiency and practical value.

Critical weaknesses

My main reservation is the empirical link to the cross-body noise-tax explanation in Section 4.4. Figure 3 illustrates the emergence of coordinated walking, and the gradient-SNR diagnostics in Appendix D.3–D.4 support improved gradient coherence. Their connection to the specific signal/noise condition in Theorem 1, however, remains indirect. Figure 9 also compares policy changes using different proxies for FPO++ and FFPO, with the latter directly regulated by the FFPO controller. The paper should clarify how these diagnostics support the proposed mechanism and account for the different proxies when interpreting the policy-update comparison.

A smaller reporting gap is that Appendix C.3 describes task-specific AUC-based tuning for FFPO but gives little detail on comparable tuning for the baselines, which would help interpret the main performance comparison.

Questions for authors

Question 1: How do the behavioral analysis in Section 5.3 and the gradient-SNR diagnostics relate to the cross-body signal/noise condition in Theorem 1? Please also clarify what comparison of policy changes Figure 9 supports given the different proxies used for the two methods.

Question 2: Appendix C.3 describes task-specific parameter tuning for FFPO. What task-specific tuning was performed for the main baselines, and how did its scope compare with the AUC-based search used for FFPO?

3 comments

We thank the reviewer for the thoughtful assessment. Below please find our response.

Q1: The connection of behavioral and gradient-SNR analysis to the theory.

TaskSeed / training updateGain conditionPredicted gainMeasured gain [MC95%]
Stand40 / 300098.25 > 80.01+0.35198+0.34477 [+0.34401, +0.34553]
Stand41 / 100064.97 < 75.47−0.03232−0.04128 [−0.04399, −0.03857]
Turn-Jog41 / 100063.49 > 47.61+0.01414+0.01373 [+0.01262, +0.01483]
Turn-Jog41 / 100063.49 < 73.30−0.00873−0.00768 [−0.00803, −0.00732]
Crate Carry41 / 296457.77 > 47.67+0.00340+0.00270 [+0.00182, +0.00358]
Crate Carry41 / 296457.77 < 73.45−0.00527−0.00532 [−0.00593, −0.00472]

In the behavioral analysis, FFPO allocates more optimization epochs to bodies with higher gradient SNR, concentrating effort on more informative and consistent update directions to accelerate learning. Gradient SNR provides a practical proxy for the signal–noise balance formalized in Theorem 1, where a higher SNR corresponds to a lower noise fraction τ\tau. To examine this relationship, we measured signal and noise in controlled updates and compared the predicted and observed gains under the same KL budget (6.25×10−56.25\times10^{-5}). The condition column expresses the theorem's gain condition ΔJB>ΔJM\Delta J_B > \Delta J_M in normalized form: the left side is the fraction of total information retained by body factorization IB/(IB+I×)I_B/(I_B+I_\times), while the right side is the square root of its relative update cost$\sqrt{C_B/(C_B+C_\times)}$. A larger left side predicts a higher factorized-update gain at the same KL budget. The table shows agreement between predicted and measured gains on both sides of the theorem's gain condition, including the reversal in their ordering when cross-body noise increases. These results support using gradient SNR as a reliability proxy to allocate optimization effort toward directions more likely to yield improvement.

Q2: The different policy-change proxies in Figure 9.

TaskSeed / UpdateMethodCFM changeAction RMS
Turn-Jog40 / 1000FPO++0.042 → 0.0420.015 → 0.015
Turn-Jog40 / 1000FFPO0.081 → 0.1610.031 → 0.062
Turn-Jog40 / 3000FPO++132.624 → 132.6240.129 → 0.129
Turn-Jog40 / 3000FFPO215.526 → 41.2790.113 → 0.021
Crate Carry40 / 3000FPO++15.820 → 15.8200.328 → 0.328
Crate Carry40 / 3000FFPO0.032 → 0.0770.012 → 0.029

Figure 9 reports each algorithm's own log-ratio proxy and shows that FFPO can take larger updates while maintaining more stable learning. For an aligned comparison, we measure CFM change as ΔCFM=E[(ℓθnew−ℓθold)2]\Delta_{\mathrm{CFM}}=\sqrt{\mathbb{E}[(\ell_{\theta_{\mathrm{new}}}-\ell_{\theta_{\mathrm{old}}})^2]}, where ℓθ(s,a,ϵ,t)=d−1/2∥vθ((1−t)a+tϵ,t,s)−(ϵ−a)∥22\ell_\theta(s,a,\epsilon,t)=d^{-1/2}\|v_\theta((1-t)a+t\epsilon,t,s)-(\epsilon-a)\|_2^2 is the whole-body CFM loss, aa is the action in flow coordinates, and dd is the action dimension. Both algorithms use the same reduction and noise/time probes. Each entry shows the candidate → committed change. Action RMS additionally measures the corresponding change in generated actions. The common measurements show the same pattern: FFPO takes larger early updates while controlling late update excursions.

Q3: Task-specific tuning for the baselines.

MethodSequential search columnsCandidate budgetConfigurations per task
Gaussian PPOLearning rate {3e−5, 1e−4, 3e-4}; entropy coefficient {0, 0.001, 0.005}; fixed/adaptive-KL schedule; actor epochs {8, 16, 32}; action-output scale {1, 2}1,500 updates × two seeds6: initial configuration plus one alternative per column
FPO++Learning rate {3e−5, 1e−4, 3e-4}; clipping parameter {0.05, 0.1}; actor epochs {8, 16, 32}1,500 updates × two seeds4: initial configuration plus one alternative per column
FFPOTarget radius {0.15, 0.20, 0.30, 0.40, 0.50}; maximum scale {1, 2, 3, 4, 5}1,500 updates × two seedsTwo-stage search, fixing the selected radius before searching scale

FFPO's task-specific tuning selects the step-control radius and then the maximum scale using reward AUC. We follow the same sequential-selection principle for Gaussian PPO and FPO++: search one parameter, retain the best configuration with the highset mean reward AUC, and proceed to the next parameter. The searches target each algorithm’s relevant parameters, with their ranges and evaluation budgets reported in the table. For FPO, GenPO and PolicyFlow we use the default settings from the corresponding repositories. This provides a transparent task-specific comparison of the tuned baselines and FFPO.

Thank you for the thorough responses and additional experiments. These additions address my concerns and strengthen my positive assessment of the method’s technical contribution and practical value. I maintain my weak-accept recommendation.

Thank you for your kindly follow-up and for strengthening your assessment.

We appreciate your careful reading of our responses. Your comments have helped us sharpen both the analysis and the theory of the paper.

Thank you again for your valuable time and engagement.

Reviewer_aifPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Factorized Flow Policy Optimization (FFPO) for efficient humanoid whole-body control. The main idea is to decompose the whole-body flow policy into multiple body-wise agents and optimize them sequentially to reduce gradient interference between different body parts. The method introduces likelihood-ratio prefixes to account for preceding policy updates, gradient coherence to allocate optimization epochs, and a rescaling mechanism to control the overall policy update magnitude. The authors also provide a theoretical analysis of cross-body gradient interference. Experiments on 16 simulated humanoid control tasks show improvements in learning efficiency and final performance over existing baselines. The method is further evaluated on a real Unitree G1 humanoid robot.

Critical strengths

The idea of dividing whole-body control into different body-wise policies is intuitive and well motivated. I appreciate that the authors also consider how to coordinate the optimization of these policies rather than simply training them independently. This makes the overall framework more convincing. The authors evaluate their method on 16 simulated tasks covering different types of humanoid control. The comparisons with several existing methods show encouraging improvements in learning efficiency and final performance. I also appreciate the inclusion of real-world experiments on Unitree G1, which provides additional evidence of the practical applicability of the proposed method.

Critical weaknesses

The experiments are mainly conducted on a single humanoid platform with a predefined body partition. It remains unclear whether the proposed method would perform equally well on robots with different structures or when the body partition is changed. This somewhat limits the generality of the conclusions. Although the real-world experiments are a valuable addition, they mainly focus on relatively simple locomotion and motion-tracking tasks. More challenging tasks involving physical interaction, such as object manipulation, are only evaluated in simulation. Therefore, the practical effectiveness of the method for more complex whole-body control remains uncertain.

Questions for authors
  1. The real-world experiments mainly focus on locomotion and motion tracking. What are the main challenges in applying FFPO to more complex real-world tasks involving object manipulation? Are there any existing results or observations that could help assess its effectiveness in these scenarios?
  2. Since FFPO uses multiple policy networks and sequential optimization, could the authors provide a comparison of actual training time or computational overhead against the baselines? This would help clarify whether the improvements in learning efficiency also translate into computational efficiency.
  3. Appendix D.6 shows that using two body agents achieves better average performance than four agents on the evaluated tasks. Could the authors explain why four agents were selected for the main experiments? How sensitive is the overall performance to this choice?
1 comment

We thank the reviewer for the constructive feedback and for appreciating the motivation and practical value of FFPO. Below are our responses.

Q1: The challenges and existing results for more complex real-world tasks.

Thank you for highlighting the challenges of real-world object interaction. The main challenge lies in the gap between simulated environments and real object dynamics. Real objects may deform, and their geometry and friction can vary across the object, whereas simulation often relies on rigid bodies and simplified contact properties. These tasks also require accurate object-state and pose estimation through motion capture or other sensing systems, making successful deployment dependent on suitable sim-to-real techniques. To demonstrate FFPO's applicability to real-world interaction, we additionally deployed it on Crate Carry and Bimanual Lift, with all 5 repeated hardware trials succeeding. Videos of these experiments are available on the anonymous project webpage https://ffpo.pages.dev/.

Q2: Actual training time and computational overhead of FFPO and baselines.

MethodTime to 3,000 updates (h)Time to 90% target (h)Single-step inference (ms)
Gaussian PPO1.182Not reached0.128
FPO2.982Not reached0.189
FPO++4.3401.2490.794
Gaussian HAPPO2.6690.5160.463
GenPO1.4760.4452.169
PolicyFlow1.2240.7910.205
FFPO5.7031.1160.640

We train all methods for 3,000 updates and profile wall-clock training and inference costs on Normal Walk using an RTX 5090 (one process per GPU, averaged over three seeds). The common reward target is set to 90% of the mean FFPO peak across the three benchmark seeds, using a 200-update trailing average to determine peak reward and first target crossing, with target times including initialization, compilation, warm-up, sampling, and optimization. The results show that both FFPO and FPO++ reach the target in all seeds, with FFPO taking 10.7% less wall-clock time than FPO++. Single-step inference is measured at batch size 1 after compilation, where FFPO achieves 0.640 ms latency, slightly faster than FPO++. Together, these measurements show that FFPO's sample-efficiency gains offset part of its sequential body-wise optimization cost while maintaining fast execution.

Q3: Four body agents choice and sensitivity to the partition.

We use four agents throughout the main experiments since this choice follows an intuitive functional partition of the body. Figure 13 in the appendix shows that two agents achieve slightly better average performance, while two and four agents perform comparably overall. More pronounced degradation occurs with a single whole-body agent or a finer seven-agent partition. We therefore treat the number of agents as a task-dependent hyperparameter and retain four agents as a consistent default across tasks.

Q4: Application to robots with different structures/platform.

FFPO's objective is defined over a robot's complete action space. Adapting FFPO to a different robot structure or platform requires specifying its action-space partition and low-level execution settings. To demonstrate this applicability, we deployed FFPO on the Unitree Go2 quadruped. All 5 repeated hardware trials succeeded, and videos are available on the anonymous project webpage: https://ffpo.pages.dev/. As discussed in Q3, the partition granularity can be treated as a task-dependent hyperparameter, allowing the same framework to accommodate different control requirements.

Comments on the paper

1 comment

Dear ACs,

Thank you for the time you have devoted to our submission and for overseeing the discussion. We would like to summarize the main additions addressing the three reviewers’ concerns, particularly the contribution of body factorization beyond step control. The additional results and discussion are marked in blue in the revised manuscript.

Training and inference costs. In response to Reviewer aifP, we added wall-clock comparisons covering the training budget, time to a common reward target, and single-step inference. On Normal Walk, FFPO reaches the common target in 1.116 hours versus 1.249 hours for FPO++, a 10.7% reduction, while maintaining sub-millisecond inference. These measurements complement the learning curves and clarify the computational cost of FFPO’s sequential optimization.

Theory and aligned policy-change measurements. In response to Reviewer xumL, we supplemented Theorem 1 with controlled measurements of body and cross-body signal and noise, comparing predicted and measured surrogate gains under a common KL budget. The measurements agree with the predicted ordering on both sides of the theorem’s gain condition, including changes induced by increasing cross-body noise. We also clarified that Figure 9 uses each algorithm’s own log-ratio proxy and added a common CFM-change measure to compare policy updates on an aligned scale.

Baseline comparisons and the contribution of factorization. In response to Reviewer WbBP, we added FPO++ with step control, Gaussian PPO with step control, and a Gaussian HAPPO baseline, alongside component ablations and per-seed results. We extended the FPO++ with step control comparison to all 16 tasks, disclosed the controller settings, and added longer-horizon checks of smaller targets and scales. These comparisons show that step control substantially improves the monolithic baseline and that its calibration affects long-term stability. The component ablations provide complementary evidence for factorization, with factorized actors improving three-task mean AUC from 0.348 to 0.567 before adding the remaining FFPO components. Together, the results support complementary roles for body factorization and step control, with task-dependent additional gains from separating body updates, particularly in coordinated interaction tasks such as Crate Carry.

We hope this summary helps you assess the revised paper and the evidence added during the discussion. We would be happy to clarify any remaining questions.

Best regards,

Authors of Paper24399

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.