acceptodds
Under review as a conference paper at ICLR 2027

PredActor: Predictive Action Diffusion for Steerable Onboard Humanoid Control

Abstract

Diffusion models offer flexible motion generation, but translating this flexibility into feedback-responsive humanoid control remains challenging. Hierarchical systems steer motion through references that may exceed a separate tracker's capabilities, leaving recovery and physical execution largely to the tracker. Action-only diffusion generates actions directly but lacks an explicit future-state trajectory for test-time motion objectives. Joint state–action diffusion provides this representation, yet representative controllers often depend on privileged full-body states, and support for learned behavior selection and test-time motion steering remains fragmented. We present PredActor, a predictive action diffusion policy that brings these complementary steering capabilities into one directly executed policy using proprioceptive observations. Conditioned on proprioceptive history and optional task context, PredActor jointly generates executable actions and an internal future-state trajectory. Classifier-free guidance strengthens text-conditioned behavior, while classifier guidance steers predicted states toward test-time objectives. Only actions are executed, without a separate motion-reference tracker or externally estimated full-body states as policy inputs. In simulation, PredActor reaches 44 of 45 destination targets and achieves a text retrieval score of 0.539, compared with 0.424 for conditional action diffusion, with similar observed disturbance survival. To make this guided policy practical onboard, rolling denoising and computation-preserving runtime optimizations reduce the complete callback to 16.790 ms median and 19.383 ms p95 on a Jetson Orin NX, both below the 20 ms control period. We deploy PredActor on a Unitree G1; evaluations across simulation and physical hardware demonstrate text-conditioned motion, disturbance response, joystick control, and semantic interpolation. Project page: https://anonymous.4open.science/w/predactor_anonymous/

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid control, diffusion policies, joint state-action diffusion, guided action generation, imitation learning, onboard deployment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uGJ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Diffusion-based humanoid control policies struggle to use future-state prediction to guide long-horizon motion objectives unless they rely on a separate low-level motion tracker and an external privileged state estimator. This paper proposes a predictive joint state–action diffusion policy that relies solely on onboard proprioceptive observations. Full-body physical states serve as supervision signals only during training. At inference time, they are reconstructed from noise purely as internal latent variables that serve goal guidance. This eliminates the need for a separate motion-reference tracker and an external privileged state estimator, enabling 50 Hz high-frequency control deployed on onboard edge computing hardware.

Critical strengths
  1. The proposed method has been deployed fully onboard the robot. A single policy accepts multiple command modalities, such as text instructions and joystick velocity commands, and runs closed-loop inference in real time.
  2. The work is highly reproducible. The paper gives a thorough account of the technical principles, implementation details and relevant hyperparameters, and the released inference source code should make it easy for the community to reproduce the results.
  3. The manuscript is well written and reads smoothly, with abundant figures and a high level of completeness.
Critical weaknesses
  1. The main baseline, Diffuse-CLoC, predicts future states from the full-body states observed over the past few frames. The paper's claimed novelty, "not relying on past full-body state observations," amounts to predicting the past and future states together. The paper neither ablates this predictive approximation nor offers any physical guarantee for it. Moreover, on a real robot most full-body state components can be measured or computed, such as the relative positions of body parts and the pelvis angular velocity. Only a few global quantities, such as pelvis linear velocity, global position/height, and yaw, are hard to obtain. Yet PredActor uses base linear velocity, which is exactly this kind of "privileged information" (it is part of PredActor's 96-dimensional observation, as shown in Table 6). I therefore believe the core assumption of the method does not hold.

  2. According to the paper, action-only diffusion produces no future-state trajectory, so it cannot apply classifier guidance and therefore cannot impose analytic objectives defined on future body states (e.g., a specified root velocity, a body height, or point-goal navigation). In practice, however, tasks such as joystick velocity control, body height control, and point-goal navigation can be handled simply by feeding the commands (e.g., velocity commands) to the network as an additional input. This is standard practice in imitation learning: the condition labels are simply fed to the diffusion network together with the observations.

  3. The real-time performance comes mainly from common engineering optimizations and does not justify a separate claim of novelty.

Questions for authors
  1. The authors should clarify where the velocity sensor readings come from on the real robot.
  2. What results would be obtained if Diffuse-CLoC used the full-body state computed from the same set of deployable sensors as PredActor?
5 comments
AuthorsOpenReview ↗

Response to Reviewer uGJ5 (1/2)

Response to Reviewer uGJ5

We thank the reviewer for the constructive review and for noting the fully onboard deployment of a single multi-command policy, its reproducibility and the clarity of the manuscript. All new results use 3 training seeds per model. The complete reply, with all figures, tables and videos, is at anonymous link.

W1-1 · Ablating joint state prediction; physical consistency

Q: The reviewer asked for an ablation of joint state prediction and questioned whether predicted states are consistent with executed actions.

A: Why predict states as well as actions? PredActor combines behaviour planning (S1) and execution (S0) in one policy: it jointly generates future states and actions, executes the first action and re-plans from new observations at every control step. The predicted states make the plan explicit and give test-time guidance (point goals, hand targets) variables to act on. We therefore test separately whether joint state prediction improves the policy and whether predicted states agree with execution.

Six-model ablation. We agree that PredActor predicts past and future states; our claim concerns the inputs: no full-body state input, and no base-velocity estimate in the deployed 93-D version. We compare PredActor, PredActor w/o state (same inputs, loss and size, no state stream) and Diffuse-CLoC + FK (observes FK-computed full-body states), each with the deployed 93-D input (no base velocity) and a 96-D input (with it), 3 seeds each, same data and protocol (Table R1).

Table R1. Six-model ablation, mean over 3 training seeds

InputModelPush survival ↑Point goals ↑Text-trial survival ↑Text retrieval, scored (/540)Text retrieval, all trials
93-Dw/o state0.42–0.670.38 (334)0.22
93-DPredActor0.4742/450.800.40 (431)0.28
93-DCLoC + FK0.2821/450.540.56 (244)0.23
96-Dw/o state0.52–0.630.51 (301)0.27
96-DPredActor0.5145/450.890.54 (468)0.42
96-DCLoC + FK0.3015/450.780.46 (395)0.29

Joint prediction helps. It raises text-trial survival under both inputs and push survival under 93-D, and only it allows point goals.

Consistency is learned, and we measure it. Training windows pair each observation history with the states a tracking teacher executed on the simulated G1 under noise and pushes, and iterative data aggregation adds teacher continuations from states the policy visits. With no theoretical guarantee, we now measure it. Without guidance, predicted and executed states agree to 1.9 cm (links) and 0.14 m/s (root velocity) 100 ms ahead, vs. 7.6 cm for a constant-state baseline. With vanilla CG, the original study gives a 0.42 m/s gap and 9/120 falls (Figure R1); in a separate paired comparison, keeping the guided target in the training range cuts the gap from 0.41 to 0.24 m/s and falls from 11/120 to 1/120 (Appendix Table A2, Figure A2). The gap and falls grow with guidance gain, and more denoising steps shrink the gap (Figure R2). Here and below, our improved CG is the paper's classifier guidance with rebuttal fixes: feasibility-limited guided targets (consistency), and hand-correction temporal smoothing plus a corrected few-step velocity sampler (command vs. guidance).

Figure R1. Predicted vs. executed state without guidance, with vanilla CG and with our...

Figure R2. Predicted-vs-executed gap and falls by guidance gain and number of denoising steps

W1-2 / Q2 · FK-computed instead of predicted states

Q: The reviewer asked whether predicting full-body states is necessary when most can be computed onboard, and how Diffuse-CLoC does with the same sensors.

A: Computing states by FK is worse. Diffuse-CLoC + FK tests this directly (under 93-D, retrained on the onboard-computable part only) and is worse than PredActor under both inputs (Table R1, Figure R3), so predicting the states is necessary.

A likely reason. We hypothesise that FK turns the measured pelvis angular velocity into every link velocity through the term ω × r, so gyro jitter is amplified by the lever arm and accumulates along the chain. The input would thus be least reliable at the feet and hands, exactly where the robot makes and breaks contact.

Figure R3. Six-model ablation per metric and training seed

AuthorsOpenReview ↗

Response to Reviewer uGJ5 (2/2)

W1-3 / Q1 · Base linear velocity

Q: The reviewer argued that PredActor relies on hard-to-obtain base velocity, so the novelty claim and core assumption fail, and asked where velocity readings come from.

A: Both versions are deployable. PredActor-93D and -96D both ran on the G1. The 96-D version uses the vendor's onboard velocity estimate with velocity domain randomisation; all the paper's hardware results use PredActor-93D, which takes no base velocity (joints, IMU, previous action and text only). Without base velocity PredActor drops only slightly (push 0.51 → 0.47, point goals 45/45 → 42/45, text survival 0.89 → 0.80), so the method does not depend on it (Appendix Table A3 and Table A4).

W2 · Command conditioning vs. guidance

Q: The reviewer doubted that test-time (classifier) guidance is needed when the goal can be a command input.

A: Complementary, not competing. PredActor supports both conditioning (a command input or CFG) and guidance (CG), so this is a choice, not a limitation. Conditioning needs labelled data and training per command; CG needs neither but may be less precise. Table R2 compares them with the same 96-D inputs.

Task: accuracy ↑ · falls ↓PredActor + our improved CGPredActor + command inputPredActor w/o state + commandDiffuseLoco + command
Static hand targets: reached within 8 cm · falls (of 486 runs)198 · 821 · 159 · 5759 · 36
Moving hand targets: mean tracking error in m · falls (of 27 runs)0.16 · 00.12 · 30.14 · 00.13 · 4
Forward speed 0.5 / 1.0 / 1.5 m/s: segments held (of 45) · runs with a fall (of 15)15 · 045 · 044 · 036 · 5
Lateral ±0.3 m/s and yaw ±0.6 rad/s steps: segments held (of 60) · runs with a fall (of 15)8 · 04 · 04 · 03 · 6

Table R2. Our improved CG vs. three command-input models, all reading the same 96-D...

Coverage decides. Without retraining, our improved CG is by far the most accurate on static hand targets (198/486 vs. ≤ 59/486 for every command-input model; bars in Figure R4). Fall differences between command-input models come from individual weak training seeds and are only descriptive. We have no matched command-input policy for body height or point goals, so we make no comparison there.

Figure R4. Table R2 as bars, one colour per model

Videos. Video V1, Video V2, Video V3 and Video V4 show the four models reaching 18 static two-hand targets (spheres; one training seed); full results in Appendix Table A5.

Video V1. PredActor + our improved CG reaching 18 static targets with both hands

Video V2. PredActor + command input reaching 18 static targets with both hands

Video V3. PredActor w/o state + command reaching 18 static targets with both hands

Video V4. DiffuseLoco + command reaching 18 static targets with both hands

W3 · Real-time performance

Q: The reviewer noted that the real-time performance comes from common engineering optimisations.

A: A system contribution. We agree the individual techniques are known and present them as a system and infrastructure contribution. Prior diffusion controllers met the latency barrier by giving something up: an add-on or off-board GPU, a lower rate (25 Hz for BeyondMimic), a tracker underneath, or no gradient guidance. PredActor is, to our knowledge, the first joint state–action diffusion policy run entirely on the G1's Jetson Orin NX at 50 Hz, with guidance in the loop, direct joint targets and no tracker. In the paper's onboard timing study, graph, structural and runtime optimisations cut the guided actor from 21.85 to 13.18 ms; the full callback takes 16.79 ms at p50 and 19.38 ms at p95 (7/600 cycles over budget). We release the C++ deployment library. → Appendix, W3

Paper changes

We have uploaded the revised paper; the changes are summarised in our general comment Paper Revision 1.

We hope these new experiments resolve the reviewer's concerns.

Thank you very much for your candor and detailed rebuttal; however, I would like to respectfully maintain my current score. The baseline comparison still seems slightly asymmetric. Also, it remains somewhat unclear where the performance ceiling of predicting states lies relative to observing ground-truth states directly.

Reviewer_L6YXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PredActor, a humanoid controller that jointly generates future states and actions using diffusion, enabling language-conditioned control and test-time motion guidance. The method combines classifier-free and classifier guidance, with training based on teacher demonstrations and DAgger-style data aggregation. Rolling diffusion inference enables onboard deployment at 50 Hz on a Unitree G1, with experiments demonstrating navigation, language-conditioned behaviors, and disturbance robustness.

Critical strengths

The paper presents a strong practical implementation, supported by extensive simulation and real-world experiments. The hardware optimizations are particularly impressive, enabling onboard diffusion-based control at high frequency.

Critical weaknesses
  1. Limited methodological novelty: The proposed method largely combines existing techniques, including joint state-action diffusion, DAgger-style training, classifier(-free) guidance. The paper does not establish a sufficiently novel methodological contribution beyond integrating these components.

  2. Limited inference novelty: The hardware optimizations primarily rely on established techniques, including partial denoising, rolling inference, and standard computational optimizations. While the implementation is interesting, the algorithmic novelty of these improvements is limited.

  3. Unclear execution feasibility: Classifier guidance steers predicted future states without explicitly ensuring that the corresponding actions realize these trajectories. The paper would benefit from further discussion of how dynamically feasible behaviors can be synthesized and how consistency between guided state predictions and actual execution can be ensured.

  4. Incomplete related work: The discussion overlooks relevant work such as [1], which also employs classifier guidance to steer diffusion-based control toward test-time objectives. A more comprehensive comparison would help clarify the novelty of the proposed approach.

[1] Huang, Runhan, et al. "Flexible locomotion learning with diffusion model predictive control."

Questions for authors
  1. Methodological contribution: Given that most components are adopted from existing methods, what do the authors consider the main methodological novelty of PredActor beyond their integration?

  2. Execution feasibility: How can the method ensure that guided state trajectories are dynamically feasible and consistent with the actions actually executed by the robot? Are there mechanisms to minimize the gap between synthesized and realized behaviors?

3 comments
AuthorsOpenReview ↗

Response to Reviewer L6YX (1/2)

Response to Reviewer L6YX

We thank the reviewer for the careful and constructive review, and for pointing out the strong practical implementation, the extensive simulation and real-world experiments, and the hardware optimizations that enable onboard diffusion-based control at high frequency. We address each point below. The complete reply, with all figures, tables and videos, is available at anonymous link.

W1 / Q1 · Methodological novelty

Q: The reviewer asked what PredActor contributes beyond combining joint state–action diffusion, training with iterative data aggregation and classifier(-free) guidance, and what we see as its main methodological novelty beyond this integration.

A: PredActor imagines the future and steers inside that imagination. We agree these parts exist, and restate the contribution of our introduction. PredActor jointly predicts future states and actions from proprioception. This is an internal "imagination" of what will happen next, much like a world model, whose states are never observed. PredActor steers behaviour inside this imagination: classifier guidance (CG) puts test-time objectives on the predicted states, classifier-free guidance applies text, and only the generated action is executed, with no tracker. What sets this apart is that the imagined states are never observed. Methods that also diffuse states and actions, such as Diffuse-CLoC and Diffusion-MPC (W4), predict states of the same kind they observe as input, so their state prediction extends an observation they already have. PredActor observes only proprioception, yet predicts and steers full-body link states it never sees at test time, so the imagination itself supplies the states that guidance acts on. Predicting the states beats observing them with the same sensors (push survival 0.47 vs. 0.28, Table R1 and Appendix Table A1).

Table R1. Six-model ablation, mean over 3 training seeds

InputModelPush survival ↑Point goals ↑Text-trial survival ↑Text retrieval, scored (/540)Text retrieval, all trials
93-Dw/o state0.42N/E0.670.38 (334)0.22
93-DPredActor0.4742/450.800.40 (431)0.28
93-DCLoC + FK0.2821/450.540.56 (244)0.23
96-Dw/o state0.52N/E0.630.51 (301)0.27
96-DPredActor0.5145/450.890.54 (468)0.42
96-DCLoC + FK0.3015/450.780.46 (395)0.29

Joint prediction helps with both inputs. The six models are PredActor, PredActor w/o state (same inputs, loss and size, no state stream) and Diffuse-CLoC + FK (observes FK-computed full-body states), each with the deployed 93-D input (no base velocity) and a 96-D input (with it), 3 training seeds per model and the same data and protocol. As Table R1 and Figure R1 show, joint prediction raises text-trial survival with both inputs and push survival with the 93-D input, and only joint prediction allows point goals.

Figure R1. Six-model ablation per metric and training seed

W2 · Inference speedups

Q: The reviewer noted that partial denoising, rolling inference and the optimisations are established techniques.

A: A system contribution: fully onboard at 50 Hz. We agree, and present them as a system and infrastructure contribution. Prior diffusion controllers met the latency barrier by giving something up: an add-on or off-board GPU, a lower rate (BeyondMimic: 25 Hz), a tracker underneath, or no gradient guidance. PredActor is, to our knowledge, the first joint state–action diffusion policy run entirely on the G1's Jetson Orin NX at 50 Hz, with guidance and direct joint targets. The paper's onboard timing study isolates each optimisation: graph, structural and runtime optimisations cut the guided actor from 21.85 ms to 13.18 ms, and the complete callback takes 16.79 ms at p50 and 19.38 ms at p95, with 7 of 600 cycles over budget. We release the C++ deployment library.

AuthorsOpenReview ↗

Response to Reviewer L6YX (2/2)

W3-1 / Q2 · How feasible behaviour is synthesised

Q: The reviewer asked how dynamically feasible behaviour arises when guidance acts on predicted states and nothing forces the actions to realise them, and how guided states stay feasible.

A: PredActor combines planning and execution in one policy. A reference tracker performs S0: it executes a supplied motion. PredActor combines S1 and S0 by jointly generating future states and actions from the command and observation history. It operates like learning-based MPC: execute the first action, receive new observations, and re-plan at every control step. The predicted states describe the planned behaviour; the actions produce its execution.

Feasibility is learned from executed trajectories. The training data pair states and actions from a tracking teacher executing motions on the simulated G1 under noise and pushes. Iterative data aggregation adds teacher continuations (teacher trajectory shooting) from states the policy visits. The training objective learns to jointly generate these state–action trajectories conditioned on commands and observation history.

This is not a hard dynamics constraint. Joint prediction and re-planning alone do not guarantee feasibility. We therefore measure prediction–execution consistency and test mechanisms that reduce the gap in W3-2.

W3-2 / Q2 · Consistency of predictions and execution

Q: The reviewer asked how guided predictions are kept consistent with execution, and which mechanisms reduce the gap.

A: We measure consistency directly. We compare predicted and executed states without and with CG in MuJoCo on the paper checkpoints (Appendix Table A3). Without CG, predictions match execution 100 ms ahead (root velocity 0.14 m/s, body links 1.9 cm, vs. 0.34 m/s and 7.6 cm for holding the current state). CG widens the gap when it asks for more than the robot can do (original study: 0.42 m/s at the paper gain, 9/120 falls; Figure R2). In a separate paired test, limiting guided targets to the training range reduces velocity RMSE from 0.41 to 0.24 m/s and falls from 11 to 1 of 120 (Appendix Figure A2). Figure R3 shows how the gap and the falls grow with the guidance gain and how more denoising steps narrow the gap. For strong guidance (2–10× the paper gain), the bounded sampler PA3 cuts falls from 304/360 to 24/360 (Appendix, W3-2). Here and below, our improved CG is the paper's classifier guidance with fixes made during the rebuttal: feasibility-limited guided targets (consistency results), and a temporal-smoothing term on the hand correction with a corrected few-step sampler for velocity (command vs. guidance results).

Figure R2. Predicted vs. executed state without guidance, with vanilla CG and with our...

Figure R3. Predicted-vs-executed gap and falls by guidance gain and number of denoising steps

W4 · Diffusion-MPC (Huang et al.)

Q: The reviewer pointed to Huang et al. as missing related work that also guides diffusion control, and asked for a fuller comparison.

A: Diffusion-MPC is included in the prepared revision. We thank the reviewer. We have added a citation and one explanatory sentence on Diffusion-MPC in Related Work, and added its rows to Table 1 and the appendix capability and deployment tables (Figure R4 below; Appendix Table A4 and Figure A4). The revision explicitly acknowledges its joint state-action diffusion, reward guidance and constraint projection for quadruped locomotion. Its codebase, built for the Go2 quadruped in Isaac Gym with a robot-specific PPO demonstrator, differs too much from ours, and we regret that there was no time for an experimental comparison.

Figure R4. Snapshot of revised paper Table 1, including Diffusion-MPC (Huang et al.) and...

Paper changes

We have uploaded the revised paper; the changes are summarised in our general comment Paper Revision 1.

We hope these new experiments resolve the reviewer's concerns.

I thank the authors for their efforts and appreciate the additional experiments. My concerns have been adequately addressed, although I still consider the novelty of the contribution to be moderate. Therefore, I am happy to maintain my original score.

Reviewer_SpDUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PredActor, a joint state-action diffusion policy for humanoid control that directly predicts executable actions along with future states for test-time guidance. The method combines classifier-free guidance for text-conditioned motion and classifier guidance for motion steering, without requiring a separate motion tracker. The authors also introduce rolling denoising and inference optimizations to achieve 50 Hz onboard control on a Unitree G1. Experiments in simulation and on hardware demonstrate text-conditioned motion, joystick steering, and disturbance recovery.

Critical strengths
  1. The onboard deployment of a diffusion-based controller at 50 Hz on a Unitree G1 is technically impressive, especially given the limited computational resources.
  2. The unified framework supporting both text-conditioned motion and test-time steering is interesting, with reasonably comprehensive simulation evaluations and hardware demonstrations.
Critical weaknesses
  1. Unconvincing motivation and limited comparison with existing BFMs. I am not fully convinced by the motivation for replacing the conventional planner-controller architecture. A well-trained tracking controller can already respond to external disturbances while staying as close to the reference motion as possible. Modern behavior foundation models (BFMs) also offer diverse motion capabilities with higher-frequency feedback. The paper does not sufficiently demonstrate why the proposed diffusion-based controller is preferable to these existing approaches. In particular, the lack of quantitative comparisons with strong BFM baselines makes it difficult to assess the practical advantages.
  2. Limited motion diversity and relatively simple demonstrations. The motion library relies partly on AMASS, which has limited coverage of complex humanoid behaviors. This is also reflected in the experiments, where most text commands are simple and generic, such as walking, running, and squatting. More fine-grained text-conditioned motion generation and robot-object interactions would make the contribution more compelling (there are many recent works on this topic). Furthermore, the current hardware videos do not show a clear qualitative advantage over existing BFM approaches, making it difficult to judge the practical significance of the proposed method.
Questions for authors
  1. Can the authors provide quantitative comparisons with recent BFM-based approaches, particularly regarding disturbance robustness, motion tracking quality, and control responsiveness?
  2. What are the practical advantages of directly generating actions through diffusion compared with using a diffusion motion planner and a high-frequency tracking controller?
  3. How well does PredActor generalize to more fine-grained text commands or motions outside the training distribution? Can the authors provide more challenging hardware demonstrations, including robot-object interactions?
  4. Could the authors provide additional side-by-side hardware videos against existing baselines? The current demonstrations make it difficult to identify clear improvements in motion quality or robustness.
4 comments
AuthorsOpenReview ↗

Response to Reviewer SpDU (1/3)

Response to Reviewer SpDU

We thank the reviewer for the constructive review and for noting the 50 Hz onboard deployment of a diffusion controller on the Unitree G1 and the unified framework for text-conditioned motion and test-time steering. The complete reply, with all figures, tables and videos, is at anonymous link.

W1 · Motivation: why one end-to-end policy instead of a planner + tracker

Q: The reviewer doubted the motivation for replacing a planner + tracker, since a well-trained tracker (the BFMs the reviewer mentions, following an explicit reference motion or a latent reference code) already handles disturbances while staying close to its reference.

A: We do not claim that PredActor is preferable to, or a replacement for, a planner + tracker. We want a humanoid policy that follows commands and adapts to physical feedback at the same time, which calls for re-planning from the current state, as in model predictive control. The paper therefore asks whether one policy can plan future motion in a receding-horizon manner and directly produce executable actions.

The two paradigms solve different problems. A tracker, including the BFM trackers the reviewer mentions, is an S0 controller: given a reference motion, it executes that motion robustly. PredActor is a unified S1+S0 policy: from a semantic or goal command it decides the whole-body behaviour (S1) and executes it (S0), with no reference. The two are therefore not directly comparable, and tracking error is undefined for PredActor. When a reference already exists, as in teleoperation, a reference plus a tracker is the better tool. PredActor aims to make a physical robot character come alive: it generates its behaviour in real time, switches motions seamlessly and adapts behaviourally to environment perturbations.

Why one policy: intuition.

  • S1 and S0 increasingly have to be designed together. As Weng argues, an S0 that only tracks references cannot respond to the environment until S1 sends a new reference, so a stronger S1 needs a co-designed S0 [1]. PredActor takes this coupling to its simplest form: S1 and S0 learned as one policy.
  • An explicit reference is not necessary. From first principles, for most commands, the event and the intention matter more than reproducing an exact trajectory. People act toward an intention and keep adjusting, without tracking a pre-computed trajectory. Mapping the command and the robot's state directly to actions is therefore the plainer formulation.
  • PredActor works like learning-based MPC. It predicts future states and actions, executes the first action and re-plans at every control step. With the prior learned offline and the plan made online, planning and control are one computation on one clock, a more elegant formulation than connecting two layers.

Why one policy: technical reasons.

  • The generator must stay inside what the tracker can execute. A generator usually produces kinematic motion only, so its reference may leave the tracker's executable set and turn into unwanted motion. A single policy has no separate reference to fall outside its support.
  • The generator reacts slowly or not at all. It runs slower than the tracker and often does not observe the robot, so after a disturbance the robot keeps tracking the old trajectory and the reaction is left to the tracker. A single policy re-plans from the robot's state at every step.
  • Two layers need careful systems engineering. Layers at different rates need re-planning schedules, asynchronous reference updates and hand-over logic, as ReactiveBFM introduces to bridge the planning–tracking latency gap [2]; a single policy on one clock avoids this.

The two levels are complementary. A tracker's abilities reach PredActor through its training data, so a stronger tracker makes a better teacher. W1 / Q1 compares trackers and their PredActor students on this basis.

[1] Haoyang Weng, "Beyond Motion Tracking: Steerable Physical Interaction Competence", research notes, 2026. https://haoyangweng.pages.dev/blog/2026-10-steerable-interaction-competence/en/

[2] X. Chen et al., "ReactiveBFM: Reactive Closed-Loop Motion Planning Towards Universal Humanoid Whole-Body Control", arXiv:2606.30362, 2026. https://arxiv.org/abs/2606.30362

AuthorsOpenReview ↗

Response to Reviewer SpDU (2/3)

W1 / Q1 · Quantitative comparison with BFM trackers

Q: The reviewer asked for quantitative comparisons with BFM trackers on disturbance robustness, tracking quality and responsiveness.

A: How we compare. As tracking error is undefined for PredActor (W1), we assess its state–action physical consistency, the agreement between predicted and executed states.

  • Tracking quality. Without pushes, the trackers follow a shared reference with joint RMSE 0.062–0.134 rad (Table R1). UFO's joint error understates its drift from the reference path: it covers 69 % of the reference distance and ends about 3 m away, vs. 0.17–0.31 m for the RL tracker and MimicLite.
  • PredActor physical consistency. Without guidance, predicted and executed states agree to 1.9 cm (body-link position) and 0.14 m/s (root velocity) RMSE at 100 ms, vs. 7.6 cm and 0.34 m/s for a constant-state baseline. Vanilla CG gives 0.42 m/s and 9/120 falls in the original study; in a separate paired 120-episode comparison, our improved CG with feasibility-limited targets cuts the gap from 0.41 to 0.24 m/s and falls from 11/120 to 1/120 (Figure R1; Appendix Table A5). This is measured agreement, not a physical guarantee.
  • Disturbance robustness. Each tracker's data trains a PredActor student, and each pair is tested under one protocol (Table R1, Figure R3; Appendix Table A2 and Table A3).
  • Responsiveness. To a new command, PredActor changes behaviour within a median 0.66–2.16 s (57/60 switches, Appendix Table A4) in one onboard loop, with no reference to regenerate. To a physical disturbance, it responds at the behaviour level: under a sustained 50 N pull on each hand, PredActor keeps standing while HoloMotion falls (Video V1), and with a 150 N push added, PredActor, TextOp and UFO stay upright while HoloMotion falls (Video V7 and Video V8).

Figure R1. Predicted vs. executed states across prediction horizons

Figure R2. Prediction-execution error and falls across guidance gains and denoising steps

Horizon, guidance and denoising. Errors grow with the horizon (Figure R1), and stronger guidance widens the gap and increases falls (Figure R2). At the paper's total guidance strength, 10–20 denoising steps improve consistency: 100 ms root-velocity RMSE drops from 0.41 to 0.15 m/s, body-link RMSE from 2.8 to 2.0–2.1 cm, and falls from 11/120 to 0/120. However, point-goal success drops, so better consistency alone does not mean better goal completion.

Table R1. Teacher / PredActor student per data-collection teacher, one protocol in MuJoCo

Data-collection teacherTeacher tracking error, no push (joint RMSE, rad) ↓Push survival ↑Resumed walk ↑Get-up after 700 NStudent models
RL tracker0.0800.60 / 0.550.45 / 0.130/20 / 0/603
HoloMotion0.0870.63 / 0.550.51 / 0.230/20 / 0/603
MimicLite0.0620.54 / 0.560.38 / 0.100/20 / 0/603
UFO, data with falls and recoveries0.1340.92 / 0.610.32 / 0.2418/18 / 96/1005
UFO, data without falls0.1340.92 / 0.600.32 / 0.1318/18 / 0/201
SONIC0.0850.53 / –0.38 / –0/20 / –none

A student inherits what its teacher's data contains. Trained on UFO data with falls and recoveries, PredActor gets up in 96/100 knock-downs, but in 0/20 when trained on the same teacher's data without falls, and the paper model in 0/60 (Table R1, Figure R3, Video V2). Students of HoloMotion and our RL tracker survive pushes alike (0.55 each), and students stay within 0.09 of their teacher's push survival, except for UFO, whose stiff, slow gait survives 0.92.

Figure R3. Push survival, resumed walk and get-up after a 700 N...

Figure R4. Pelvis height and distance from the walkin...

Teachers and students under increasing pushes. In Figure R4, each teacher and its student walk through five 0.5 s pelvis pushes of 100–500 N (100 N steps, 3 s apart, same direction) in one 19 s episode chosen in advance. All stay up through 200 N; the RL-tracker, HoloMotion and MimicLite pairs fall at 300 or 400 N and stay down. Only UFO and its fall-and-recovery student get up: the student falls at 400 and 500 N, gets up both times and walks on.

Video V1. PredActor and HoloMotion standing under a sustained 50 N upward pull on...

Video V2. A 700 N knock-down while walking

AuthorsOpenReview ↗

Response to Reviewer SpDU (3/3)

Q2 · What generating actions directly adds

Q: The reviewer asked what directly generating actions through diffusion adds over a diffusion motion planner plus a high-frequency tracker.

A: Generating actions directly adds four properties (measured in later answers).

  • Capabilities learned from the data source. Behaviour is learned from executed data, e.g., getting up when the teacher's data contains recoveries.
  • Seamless re-planning. The plan is renewed from the robot's state at every control step.
  • Behavioural adaptation. A disturbance changes the plan itself rather than being left to a tracker that returns to its reference.
  • Test-time objectives without references or labels. Classifier guidance on the predicted future states lets point and hand-target goals steer motion without an authored reference or labelled command; classifier-free guidance applies text in the same policy.

W2-1 / Q3 · Generalisation to fine-grained text and object interaction

Q: The reviewer noted the AMASS-based library covers few complex behaviours and our commands are simple, and asked how PredActor generalises to finer or out-of-distribution text and robot–object interaction, with harder hardware demos.

A: We agree. PredActor aims to validate end-to-end unified diffusion for humanoid locomotion on a real robot. Trained on a limited AMASS subset, it currently supports only a few kinds of motion and was not originally designed for object interaction. Following the reviewer's question, we are scaling up the dataset for a broader vocabulary and, since classifier guidance already supports hand end-point objectives, testing human–object interaction. Given the limited time, these partial results are in simulation; hardware demos are in progress.

  • Text. The paper model is reliable near its training vocabulary (known prompts 25/45; unseen paraphrases 6/45, part of the 36/180 unseen wordings in Table A6). Trained on human-written BABEL captions with a longer schedule and 10 denoising steps, it completes 65/180 unseen wordings vs. 36/180, with 4 vs. 71 falls out of 360, while known prompts go from 25/45 to 20/45. Video V3 shows 16 selected prompts, including boxing. It still misses many prompts, mainly due to the text encoder, and for commands outside its motion vocabulary it usually keeps standing. Besides scaling up the dataset, we are improving the text-encoder settings, and will report results when ready.
  • Interaction. By composing test-time objectives (point goal, two hand targets, text), PredActor walks to a box and grasps it with both hands in 23/36 runs (Table A7). In the latest contact-only test, with the place table about 5 m farther away, it grasps in 11/36 held-out runs, lifts above 10 cm in 10/36 and carries at least 5 m in 4/36; two runs complete pick, a 6.5 m carry and place (2/36, box resting on its side; 0/36 upright; Table A8, Video V6). Loaded walking averages 0.55 m/s vs. 0.32 m/s for the previous controller on the same layouts. Video V4 and Video V5 show earlier successes.

Video V3. Sixteen selected prompts, including boxing, judged by viewing

Video V4. Box carry0

Video V5. Box carry1

Video V6. Box carry2

W2-2 / Q4 · Side-by-side demos against baselines

Q: The reviewer found no clear advantage over BFMs in the hardware videos and asked for side-by-side hardware videos against baselines.

A: Synchronized simulation side-by-sides. We agree and provide simulation side-by-sides against BFM trackers and TextOp with the same seed, command and push, selected in advance (Video V1, Video V7 and Video V8; Appendix Video V15; TextOp details in Appendix, Q2-3). They show a behavioural difference, not higher robustness: after a push the trackers return to their reference clock, while PredActor continues its gait from a new phase.

Hardware side-by-sides are in Update1. See Update1 · Hardware side-by-side videos below (Video V19 and Video V20).

Video V7. PredActor, TextOp, UFO and HoloMotion standing under a 50 N pull on...

Video V8. PredActor, TextOp, UFO and HoloMotion walking under a 50 N pull on...

Paper changes

We have uploaded the revised paper; the changes are summarised in our general comment Paper Revision 1.

We hope these new experiments resolve the reviewer's concerns.

AuthorsOpenReview ↗

Response to Reviewer SpDU (Update1)

Update1 · Hardware side-by-side videos (W2-2 / Q4)

Q: The reviewer asked for side-by-side hardware videos against existing baselines (W2-2 / Q4); in our first reply these were still in progress.

A: We have now recorded them on the Unitree G1. PredActor and three BFM trackers, UFO, SONIC and MimicLite, each run on the same robot while a person lifts and pulls the hands (Video V19). We also deployed the PredActor student trained on UFO data with falls and recoveries (W1 / Q1, Table R1) and recorded its get-ups (Video V20).

  • Hand lift and pull. PredActor lets its arms follow the pull and keeps its balance, whereas the trackers keep pulling their arms back toward their reference pose.
  • Get-up on hardware. The student gets up on its own, with no human assistance, consistent with the simulation result (96/100 knock-downs, Table R1): recovery learned from the teacher's data carries over to the real robot.

All videos are real time. Forces are applied by hand and are not matched across controllers, so these are qualitative demonstrations.

Video V19. Hand lift and pull on hardware

Video V20. Get-up on hardware

We hope these new experiments resolve the reviewer's concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Paper Revisions 1

Summary of revisions to the paper

We thank all reviewers for their constructive feedback. We have uploaded a revised version. Revisions made in response to the reviews are highlighted in yellow. The main changes are:

1. Observation ablation without base linear velocity (uGJ5 W1, Q1, Q2)

  • Table 2 now has ten rows. Three new -93D rows (PredActor, PredActor w/o state, CLoC+FK w TextCFG) are retrained on the 93-D observation used by our hardware policy, which omits base linear velocity, and are evaluated under the same protocol (Section 4.1; Appendix C.2).
  • With this observation, the predicted-state stream still improves push and text survival. Removing base velocity costs PredActor point goals (45/45 → 42/45) and text retrieval (0.539 → 0.400).
  • Table 9 (Appendix C.2) adds a supplementary cross-check with 540 text trials per policy. Appendix B.1 states that all hardware results use the 93-D policy.

2. State–action consistency (L6YX W3 / Q2, SpDU Q1)

  • Section 4.1 and the new Appendix D.1 (Table 11, Figures 10–11) measure how closely predicted future states agree with the states the robot then executes in MuJoCo.
  • Without guidance, predictions 100 ms ahead match execution to 0.14 m/s (root velocity) and 1.9 cm (body links), against 0.34 m/s and 7.6 cm for holding the current state. With feasibility-limited guidance targets, guided predictions match execution to 0.24 m/s, with 1/120 falls.
  • Figure 11 shows how the guidance gain and the number of denoising steps trade consistency against point-goal completion.

3. BFM trackers as data-collection teachers (SpDU W1, Q1, Q2, Q4)

  • The new Appendix D.2 (Table 12) trains a PredActor student on data from each BFM tracker (RL tracker, HoloMotion, MimicLite, UFO, SONIC) and evaluates teachers and students under one protocol.
  • Students stay close to their teacher's push survival, and get up after a 700 N knock-down when the teacher's data contain recoveries (96/100 for the UFO falls-and-recoveries student; 0/60 for the paper model).
  • Figures 12–13 show frame sequences in simulation and on the Unitree G1 hardware: hand lift and pull for PredActor, UFO, SONIC and MimicLite, and autonomous get-up of the UFO-data student.

4. Related work (L6YX W4)

  • Section 2 discusses Diffusion-MPC, which is also added to Tables 1 and 5 (Appendix A.1). The classifier-free-guidance entry for ReactiveBFM is corrected in both tables.

We also made minor editorial revisions (wording of the abstract and the Figure 1 caption, presentation of the comparison tables and the experiment section, and the layout of Equations 2–3); these are not highlighted.

The complete replies, including all videos, remain available at the anonymous link given in each reviewer's response.

AuthorsOpenReview ↗

Gentle reminder to reviewers

Dear Reviewers uGJ5, L6YX and SpDU,

Thank you again for your time and thoughtful reviews. We have posted a detailed response to each review and uploaded a revised paper. The changes are summarised in our general comment Paper Revision 1.

If you have a moment, we would be grateful if you could read our responses and let us know whether they address your concerns, or whether anything is still unclear. We are happy to answer further questions or provide more detail during the discussion period.

Thank you very much for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.