acceptodds
Under review as a conference paper at ICLR 2027

: A Steerable Humanoid VLA Model for Long-Horizon Loco-Manipulation

41 views

Abstract

Vision-language-action models predict action chunks efficiently but tend to blindly follow training trajectories when facing out-of-domain tasks. While world models demonstrate more robust generalization, they require longer inference times, which critically hinder humanoid loco-manipulation, where real-time execution is demanded. Combining the best of both, we present , a hybrid humanoid foundation model with unprecedented steerability that enables up to five minutes of autonomous execution on long-horizon, whole-body loco-manipulation tasks. Specifically, we first train a predictor: a time-conditioned image world model that predicts a future goal image to steer task execution, conditioned on a time horizon and a language instruction. We then re-purpose an existing VLA model, which predicts actions from solely the current visual observation, to additionally condition on a goal image. It acts as a planner in action space, outputting actions that steer the humanoid from the current observation toward the goal image. Finally, we implement a real-time execution pipeline that streams actions to a whole-body controller, achieving whole-body control capable of more diverse motions, such as kicking and bending, that are challenging for a decoupled controller. To obtain steerability, we draw on priors from the existing image generation model and further condition it on the time horizon and language instruction. In addition, we pretrain the VLA model with variable-length action chunk prediction, allowing it to develop a better understanding the dynamics between the current observation and goal image. Compared with baseline methods, , achieves better generalization, more reliable language instruction following, and higher success rates on long-horizon tasks. Through exhaustive experiments across seven real-world tasks, we demonstrate that our model achieves the highest overall performance, outperforming the strongest baseline by 25% in success rate in novel scenes with unseen objects and layouts, and by 43% in task progress on cross-task transitions, supporting whole-body loco-manipulation tasks lasting up to five minutes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, humanoid, robot learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yHLyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Ψ0.1 is a humanoid control framework that explicitly guides a VLA policy with generated future images. A subtask predictor manages task progression, while a time-conditioned image model generates near-term visual goals. The planner conditions on the current observation and goal image to produce whole-body actions executed through SONIC with real-time action chunking. Its VLM is pretrained to predict variable-length action sequences connecting current and future frames, then frozen while a continuous action expert learns fixed-length action chunks. Real-G1 experiments compare the system with VLA and world-action-model baselines on pick-and-place tasks involving novel objects, containers, and layouts, and on a household routine lasting approximately five minutes. The paper also evaluates one-demonstration adaptation to a changed task order and provides real-robot ablations of goal-image conditioning, variable-length pretraining, and humanoid post-training.

Critical strengths
  1. A coherent design for explicit visual-goal guidance. The paper connects subtask prediction, time-conditioned future-image generation, and goal-conditioned action generation in a clear architecture. Generated goals give the policy a concrete visual target, while asynchronous execution allows the planner to keep acting as goals are refreshed. This is a practical approach to improving instruction following in unfamiliar scenes.
  2. An effective variable-length action pretraining objective. Predicting the actions connecting current and future frames is a well-motivated way to teach the VLM to use goal images. Table 1 provides direct downstream evidence: variable-length pretraining improves real-robot success from 4/10 to 7/10 on both tabletop and whole-body pick-and-place compared with fixed-length pretraining. This is a useful training contribution supported by more than offline prediction metrics.
  3. Convincing real-robot evaluation of generalization and whole-body execution. The experiments test novel objects, containers, and destination layouts, with comparisons against both VLA and world-action-model baselines. The whole-body tasks require coordinated behaviors such as walking, operating a bin pedal, and kneeling to place an object. These evaluations meaningfully test whether the policy follows the intended goal beyond familiar training configurations.
  4. A substantial demonstration of task composition. The system chains separately demonstrated household tasks into a routine lasting approximately five minutes and evaluates transitions between tasks. The additional one-demonstration adaptation experiment tests a changed task order rather than only replaying the original sequence. Together, these results show the practical relevance of the proposed guidance mechanism for extended humanoid operation.
Critical weaknesses
  1. Temporal conditioning — Suggested additional experiment. The predictor ablation shows that caption-plus-frame-rate conditioning improves image metrics over caption-only conditioning, but does not establish whether this improves closed-loop control. Please compare these predictor variants using the same downstream planner and task settings, and report whether the image-quality gains translate into better task performance.
  2. Explicit versus implicit guidance — Suggested discussion. Comparisons with Cosmos3 and DiT4DiT establish a system-level advantage, but architectural and pretraining differences prevent attributing the gains solely to explicit goal conditioning. Please discuss these confounding factors and distinguish the demonstrated performance gains from the proposed explanation that explicit guidance is more effective than implicit guidance. I view this primarily as a matter of interpretation rather than a request for an additional baseline.
  3. Evaluation reliability — Suggested additional results and clarification. The small trial counts leave uncertainty around the reported success rates. Please provide uncertainty estimates and, where feasible, additional trials. For the long-horizon routine, report fully autonomous completions out of all attempts and clarify the intervention protocol, distinguishing uninterrupted execution from task progress evaluated with assistance after failures.
Questions for authors
  1. Operating boundaries of explicit visual guidance — Additional analysis and discussion. This is the most important question for my assessment. Beyond the successful demonstrations, when does an explicit visual goal help the policy, when can it mislead the policy, and why? Appendix G provides useful subtask-prediction statistics, and the conclusion acknowledges limitations in goal-image quality, but I would appreciate a richer end-to-end analysis with representative successes and failures. In particular, how does the planner respond to inaccurate, ambiguous, or infeasible goals, and do subsequent goal updates correct or propagate errors? Where possible, please distinguish failures of subtask selection, image prediction, action planning, and execution. This would help establish how robust the central contribution is and under what conditions its benefits hold. A convincing analysis addressing these boundaries would materially strengthen my assessment and could lead me to raise my rating.
  2. Temporal conditioning — Additional experimental evidence. Does the improvement from caption-plus-frame-rate conditioning over caption-only conditioning translate into better closed-loop task performance when both predictors are paired with the same planner? A matched comparison would help establish the control benefit of temporal conditioning beyond improved image metrics.
  3. Generated versus ground-truth goals — Clarification and ablation. What motivated the 50/50 mixture of generated and ground-truth future images during training? Have you compared it with training exclusively on ground-truth future images while using generated goals at deployment for both variants? Please also provide the evidence supporting the statement that using only ground-truth future frames during final task fine-tuning causes no performance drop.
1 comment
AuthorsOpenReview ↗

Response to Reviewer yHLy: operating boundaries of explicit visual guidance and failure analysis

We thank the reviewer for the constructive review.

Q1: Operating boundaries of explicit visual guidance. In short, explicit visual guidance works because the planner does what the future image shows. We screened the logs of all 618 real-robot rollouts and added a failure analysis with representative cases (new App. I), attributing each failure to the component that failed first (Table 5):

Outcome (App. I)Share of rolloutsExample
Robot follows a correct future image and executes it~89%–
Robot follows an inaccurate, ambiguous or infeasible future image~4%Figs. 9, 10
Planner deviates from the future image: corrected / propagated~1% / ~3%Fig. 11
Low-level execution (e.g., the object slips)~4%–

The planner follows the motion, grasp pose and object shown in the future image. A correct future image therefore steers the robot through situations that the instruction does not describe, such as a novel layout that requires turning farther than in training (Fig. 8). For the same reason, an error in the future image becomes an error in the action: a blurred hand leads to a grasp from above instead of upright (Fig. 9a,b), and a wrong subtask leads to an infeasible future image that the robot still follows (Fig. 10). As future images are regenerated every 0.8 s, a small deviation of the robot is corrected by the next future image, whereas a large one propagates, because the world model is then conditioned on the deviated view (Fig. 11). The benefits thus hold when the future image is clear and correct, the subtask is correct, and the robot stays close to the predicted state (App. I, "Operating boundary"). A more accurate world model that is less affected by such deviations would directly remove the largest group of guidance-related failures, which is our main direction for future work.

W1/Q2: Temporal conditioning in closed loop. With the same planner on the 12 novel tabletop settings, the predictor with caption and frame rate reaches 9/12 and the caption-only predictor 8/12. The image-quality gain (App. G, Table 3) carries over in direction, but the effect is small and shows up in fine details such as the hand pose (App. I, Fig. 9a; see also our response to Reviewer f7S3, Weakness 2).

W2: Explicit versus implicit guidance. We agree that the system-level comparison alone cannot establish this; Table 1 gives more direct evidence. In variant (c), we take the Ψ0\Psi_0 VLM, which is pre-trained on single images only, and add the future image only during task fine-tuning, without any pre-training for explicit guidance. This alone raises success from 18% to 50% (a→c) and already matches the two WAMs (Fig. 3), although they are pre-trained at scale on video for their implicit guidance. Once the VLM is also pre-trained for explicit guidance, with variable-length action chunks that connect the current and future images, Ψ0.1\Psi_{0.1} performs far above both. We read this as evidence that explicit visual guidance is an effective way to steer the policy, while acknowledging the remaining architectural differences.

W3: Evaluation reliability. This overlaps with Weakness 3 of Reviewer f7S3, and our response there gives the full numbers. In short, we doubled the ablation to 20 trials per variant and task (Table 1, Sec. 4.4), and large gaps, such as adding the future image, have clearly separated 95% intervals. In the long-horizon routine (10 runs per method), Ψ0.1\Psi_{0.1} completes the entire routine without any assistance in 3 of 10 runs, and no baseline does. Task Progress is measured under a fixed intervention protocol in which the evaluator only restores the state at which a failed stage would have ended and assisted stages receive no credit (Sec. 4.3; new App. J, Table 6).

Q3: Generated versus ground-truth goals. The mixture (Sec. 3.3) exposes the VLM to the artifacts of generated images, which it sees at deployment, while keeping clean supervision; generated images alone would pair the demonstrated actions with lower-quality future images, which makes the supervision less precise. Comparing them in pre- and post-training would require repeating both stages (pre-training alone takes 20 days on 32 H100 GPUs, Sec. 4.1), which we leave to future work. For task fine-tuning, we ran the comparison (App. H, Fig. 7b; see our response to Reviewer f7S3, Weakness 4): with generated future images at deployment, fine-tuning on ground truth alone matches the 50/50 mixture on both tasks, since the planner has already seen generated images in the earlier stages, while generated images alone lower whole-body success.

We hope these additions address the reviewer's concerns and are happy to discuss further.

Reviewer_zT4yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors presents \psi_0.1 a world model predictor with a VLA planner that uses SONIC as a whole-body controller for a humanoid. The paper hopes to combine the generalization capability of a world model with the lower inference time of a VLA. The world model predictor uses language instruction and current observation to predict future images that serve as sub-goal images for the VLA planner. The VLA planner takes the output of the predictor along with subtask language command. The VLM features are then input to an action expert that is post-trained with proprioception input to predict action tokens that are compatible with SONIC. SONIC executes the whole-body actions on a humanoid. Experimental locomanipulation results are presented on a G1 humanoid to do various long-horizon household tasks.

Critical strengths
  • Paper breaks up the problem into a predictor (WM), a planner (VLA), and a controller (SONIC). This split enables combining the strengths of each of the models.

  • Proposed method has higher success rate for both in-domain and out-of domain tasks involving novel objects / novel destinations.

  • Experimental results are strong for the presented five tasks.

Critical weaknesses
  • The paper is incremental. There is no new method proposed, but rather a combination of existing methods.

  • The source of the Subtask prompts for the predictor and language prompts for the VLM are not clearly specified. What's the memory aspect of the subtask predictor H_\phi?

  • The world model W_\theta takes language instruction, subtask prompt, and \Delta t as input. It's unclear how a continuous \Delta t helps. If a large \Delta t is used (one that takes you out of the current subtask), will the world model predict an image from the next subtask?

  • An ablation of the subtask predictor would have been good but is missing. Will the proposed method work in the absence of H_\phi?

Questions for authors
  • Where is the text command for the VLM coming from? Is this the subtask prompt?
1 comment
AuthorsOpenReview ↗

Response to Reviewer zT4y: novelty, data flow, continuous horizon, and subtask predictor ablation

We thank the reviewer for the positive assessment and the helpful questions. We made the data flow explicit in the revised paper (see Sec. 3.2) and ran the requested ablation of Hϕ\mathcal H_\phi.

W1: Novelty. The central idea of Ψ0.1\Psi_{0.1} is to steer a VLA with an explicit near-future image rather than with language alone. This differs from typical VLAs, which map the instruction directly to actions and leave the policy to infer which object to handle, where to go and how far to proceed; in Ψ0.1\Psi_{0.1}, the future image shows this explicitly. Two components make this work, and to our knowledge neither has been used in VLAs before. First, variable-length action pre-training: rather than a fixed action chunk, the VLM learns to predict the actions that connect the current image to a future image, so it learns to read the future image and how far ahead it lies. Second, a time-conditioned image world model: rather than a full video, it renders a single future image at a chosen, continuous horizon, fast enough to refresh it while the robot acts. Both are validated in isolation (Table 1; App. H, Fig. 7a; see also our response to Reviewer f7S3, Weakness 1).

W2/Q1: Source of the prompts and memory of Hϕ\mathcal H_\phi. The text command of the planner's VLM is the user's instruction LL for the whole task, not the subtask prompt. Subtask prompts come from Hϕ\mathcal H_\phi and go only to the world model:

ModuleInputOutput
Hϕ\mathcal H_\phi (once per second)current image, LL, memory ℓ1:k\ell_{1:k}continue, switch with the next prompt, or done
Wθ\mathcal W_\theta (every 0.8 s)current image, active prompt ℓk\ell_k, Δt\Delta tfuture image
Plannercurrent image, future image, LLactions

The memory is the ordered list of subtask prompts that Hϕ\mathcal H_\phi has started in this episode, kept as text: when Hϕ\mathcal H_\phi switches, the new prompt is appended. It lets Hϕ\mathcal H_\phi tell apart moments that look alike but belong to different steps, such as the two grasps of a two-object task, and keeps it from repeating a finished step. In training, the subtask prompts are the annotations of our data (about 37k subtasks; Sec. 3.2, App. B); at deployment, they come from Hϕ\mathcal H_\phi, whose accuracy is evaluated in App. G (Table 4). The future image tells the planner which step to carry out.

W3: Continuous Δt\Delta t. Tasks and embodiments move at different paces, so the useful look-ahead differs. Training over a continuous range of Δt\Delta t lets one world model serve all of them, with Δt\Delta t chosen at deployment. It also gives the model a sense of time: from the same observation, a larger Δt\Delta t renders more progress (Fig. 1, bottom right), whereas general-purpose generators render about the same moment for every requested Δt\Delta t (App. G, Table 3). On the real robot, the best Δt\Delta t differs between tasks: tabletop PnP performs best at 2.4–3.2 s and drops at 4.0 s, whereas whole-body PnP, whose motions are longer, peaks at 3.2 s (App. H, Fig. 7a; full numbers in our response to Reviewer f7S3, Weakness 2).

Large Δt\Delta t: no, the world model does not jump to the next subtask. During training, a future target never crosses the end of the current subtask (App. C), so for a large Δt\Delta t the model shows the current subtask completed. Moving to the next subtask is decided by Hϕ\mathcal H_\phi: once it switches the prompt, the future image moves on. Future images are also regenerated from the current view every 0.8 s (Sec. 4.1), so an occasional frame that runs ahead is replaced quickly.

W4: Ablation of Hϕ\mathcal H_\phi. We removed Hϕ\mathcal H_\phi and prompted the world model with the whole instruction, keeping all other components unchanged:

Tabletop PnP, 12 novel settingsSuccess
Ψ0.1\Psi_{0.1} (with Hϕ\mathcal H_\phi)9/12
Without Hϕ\mathcal H_\phi7/12

On short tasks, the method still works without Hϕ\mathcal H_\phi, but the future images become less reliable. For example, when the instruction asks for only one of two objects, the world model without Hϕ\mathcal H_\phi sometimes renders a reach for the second object after the first one is placed, and the robot follows it; with Hϕ\mathcal H_\phi, the prompt switches to retracting the hands and finishing. Long-horizon tasks require Hϕ\mathcal H_\phi: in the five-task routine (Sec. 4.3), the same view can occur at different points, and the instruction for the whole routine does not tell the world model which step comes next, especially at transitions between tasks. Since every future image follows the active subtask (App. I, Fig. 10), Hϕ\mathcal H_\phi decides which step the robot carries out, which chains separately demonstrated tasks correctly.

We hope these clarifications address the reviewer's questions and are happy to discuss further.

Reviewer_f7S3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Ψ0.1, a hybrid humanoid vision-language-action (VLA) system designed for long-horizon whole-body locomotion and manipulation. The system contains three components:

  1. Predictor: Generates future goal images conditioned on language instructions and a specified time horizon Δt.
  2. Planner: Translates visual goals into action sequences.
  3. Controller: Executes whole-body motions using SONIC and real-time action chunking (RTC).

The main methodological contributions include 1) explicit time-conditioned future-image generation and 2) variable-length action-chunk pretraining, which aims to improve the planner's ability to infer actions connecting current observations to future visual states.

The authors evaluate Ψ0.1 on seven real-world tasks using a Unitree G1 humanoid robot, including pick-and-place tasks involving novel objects and layouts, as well as long-horizon task sequences lasting up to five minutes. Comparisons with existing VLA and world-action models demonstrate improved task success rates, generalization, and cross-task transitions. Ablation studies further investigate the contributions of future-image conditioning, variable-length pretraining, and humanoid post-training.

Critical strengths

This work demonstrates high system maturity and strong performance in real-world robotic experiments. A notable strength of this paper is that, beyond proposing a model, the authors deployed the entire system onto a physical humanoid robot. The robot successfully executed actions requiring full-body coordination (such as kneeling to place objects and using its feet to operate pedals) and demonstrated an autonomous operational sequence comprising five distinct tasks around 5 minutes. The authors also compared their model against recent alternatives Cosmos3, DiT4DiT, GR00T N1.7, and Ψ0. Experimental results indicate that Ψ0.1 performs superiorly when encountering novel objects, new environmental layouts, or transitions between tasks. These findings demonstrate that the system is not only technically feasible but also delivers robust practical performance on a real robot.

The design of pre-training with variable-length action sequences is practically innovative and supported by experimental evidence. The authors propose a pre-training method in which the model predicts an action sequence of an appropriate length based on the change between the current image and a future target image, rather than consistently predicting sequences of a fixed length. Furthermore, the authors observe that, given an actual target image representing the state 2.4 seconds into the future, the model’s prediction error for the action duration is approximately 0.3 seconds. These experiments demonstrate that pre-training with variable-length actions enables the model to effectively leverage information from future target images, rather than simply ignoring this input.

Critical weaknesses

Critical Weaknesses

1. Lack of comparisons with other visual-goal methods.

The paper compares Ψ0.1 with two VLAs (Ψ0 and GR00T N1.7) and two WAMs (Cosmos3 and DiT4DiT). However, previous work, such as SuSIE, has already used intermediate future images to guide robot actions. The authors do not compare their time-conditioned future images with other types of visual goals, such as subtask-completion images. A comparison using the same planner and controller would help show the benefits of the proposed method. Without this comparison, it is unclear how much improvement comes from the Δt-based design.

2. Limited evidence for the benefits of time-horizon conditioning.

The paper presents time-horizon conditioning as an important factor for stable steerability. Appendix G shows that different ways of encoding time affect future-image quality, but it does not show how they affect robot performance. Section 3.2 states that Δt can be chosen based on the task. However, Section 4.1 and Appendix E use a fixed value of Δt = 2.4 s for all real-world experiments. The authors do not explain why they chose this value or test how different values affect task success. Table 2 also shows that generating a short visual output is faster than generating a full video. However, this speed advantage comes mainly from generating fewer frames. It does not show that Δt-based future images perform better than other single-image goals.

3. Limited number of trials and incomplete long-horizon evaluation.

The paper uses only six trials for each novel setting and ten trials for each ablation variant. Some reported improvements are small, such as 7/10 versus 8/10. The authors do not provide confidence intervals or statistical tests for these real-robot results. This makes it difficult to judge how reliable the improvements are. The long-horizon experiments also report task progress rather than the success rate of the full five-task routine. The authors do not state how many trials they conducted or how often the robot completed all five tasks without human help. The paper also reports an average execution time of about five minutes, but it does not state how many trials were used to calculate this average.

4. Lack of experiments on the fine-tuning strategy.

The paper uses generated future images during earlier training stages, but task-specific fine-tuning uses only ground-truth future images. During deployment, the robot relies on generated images instead. The authors state that this difference does not reduce performance, but they do not provide experimental results to support this claim. A comparison between fine-tuning with ground-truth images, generated images, and a mixture of both would help show whether this difference affects robot performance.

Questions for authors
  1. What are the benefits of Δt-based future images over other visual-goal methods? Have the authors compared them with subtask-completion goals using the same planner and controller?

  2. How does Δt affect robot performance? Have other values than Δt = 2.4 s been tested on real-world tasks?

  3. How reliable are the experimental results? Could the authors provide confidence intervals for the real-world results? How many trials were conducted for the five-task routine, and how many completed all tasks without human help?

  4. Does the difference between fine-tuning and deployment affect performance? Have the authors compared fine-tuning with ground-truth future images, generated images, and a mixture of both?

1 comment
AuthorsOpenReview ↗

Response to Reviewer f7S3: goal-type and horizon comparisons, fine-tuning ablation, and reliability

We thank the reviewer for the constructive suggestions. We ran all four requested experiments on the real robot and added them to the revised paper: the goal-type and Δt\Delta t comparison and the fine-tuning comparison in the new App. H (Fig. 7), Table 1 with 20 trials per task (Sec. 4.4), and the long-horizon protocol in the new App. J (Table 6). The table collects the new results. All rows use the 12 novel settings of tabletop and whole-body PnP (Sec. 4.2), the same planner, predictor and controller, and generated future images at deployment.

ExperimentVariantTabletopWhole-body
Ψ0.1\Psi_{0.1} (paper setting)Δt=2.4\Delta t = 2.4 s, ground-truth fine-tuning9/1210/12
W1: goal type (Fig. 7a)Subtask-completion image7/126/12
W2: horizon (Fig. 7a)Δt=1.8\Delta t = 1.8 s6/128/12
Δt=3.2\Delta t = 3.2 s9/1211/12
Δt=4.0\Delta t = 4.0 s7/1210/12
W4: fine-tuning goals (Fig. 7b)50/50 mixture9/1210/12
Generated only9/127/12

W1/Q1: Other visual goals. To our knowledge, no other VLA guided by goal images is open-sourced for humanoid whole-body control, so we ablate the goal type within our own model. SuSIE generates a subgoal a fixed number of steps ahead, without a notion of time. Replacing our future image with a subtask-completion image lowers success on both tasks (App. H, Fig. 7a), for two reasons. (1) Denser supervision: a completion image exists once per annotated subtask, whereas a Δt\Delta t-future target exists for every frame without annotation, so the predictor also learns from 1.75k hours of human video. (2) A closer target: a completion image shows the end state of a step but not how to reach it. In novel whole-body layouts, it can show the robot already at a destination that is not yet in view, whereas the future image shows how the robot should move in the next few seconds.

W2/Q2: Effect of Δt\Delta t. As Fig. 7a shows, 2.4 s and 3.2 s perform best. At 1.8 s, the future image differs little from the current view and steers less. At 4.0 s, it lies too far ahead for the short tabletop task, while whole-body PnP, whose motions are longer, is barely affected. Shorter tasks thus favor a smaller Δt\Delta t and tasks with locomotion a larger one, and 2.4 s works well on both (see Sec. 3.2 revised). For the time encoding, the two predictor variants of App. G, paired with the same planner on the 12 novel tabletop settings, reach 9/12 (caption and frame rate) and 8/12 (caption only). The caption-only predictor has lower image quality (Table 3), but the closed-loop difference is small: in its one additional failure, the hand in the future image was blurred, and the robot grasped a water bottle from above instead of upright and missed (App. I, Fig. 9a). Table 2 only shows that a single latent frame is fast enough for closed-loop control (see W1 for other single-image goals).

W3/Q3: Reliability. We doubled the ablation to 20 trials per variant and task (Table 1 and Sec. 4.4 updated). Success pooled over both tasks (40 trials per variant), with 95% Wilson intervals:

Table 1(a)(b)(c)(d)(e)(f) Ψ0.1\Psi_{0.1}
Success (%)184050486878
95% CI[9, 32][26, 55][35, 65][33, 63][52, 80][62, 88]

Large gaps, such as (a) vs. (f) or fixed- vs. variable-length pre-training, have clearly separated intervals. Humanoid post-training (e→f) gives a smaller gain, since both variants are fine-tuned on the same task data afterwards; qualitatively, the robot moves more smoothly, which reduces precision errors such as off-center grasps.

For the long-horizon routine, each method runs 10 times, and the five-minute execution time is averaged over these runs. Ψ0.1\Psi_{0.1} completes the entire routine without any assistance in 3 of 10 runs, and no baseline does (Sec. 4.3, Table 6). Since any single error ends an autonomous run, we also report Task Progress under a fixed intervention protocol (App. J): after a failed stage, the evaluator restores only the state at which that stage would have ended, the assisted stage receives no credit, and the policy continues on its own. Ψ0.1\Psi_{0.1} needs assistance in 15.2% of the stages, compared with 27.8–44.3% for the baselines (Table 6).

W4/Q4: Fine-tuning strategy. All variants use the same data and schedule (App. H, Fig. 7b). Ground truth alone matches the 50/50 mixture, since the planner has already seen generated images in pre- and post-training, and it avoids running the predictor on every new demonstration. Generated images alone lower whole-body success: without ground-truth goals, the supervision for how far and in which direction to move becomes less precise, which matters most for tasks with locomotion. Sec. 3.3 now refers to this comparison.

We hope these results address the reviewer's concerns and are happy to discuss any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: New Experiments, Failure Analysis and Summary of Revisions

We sincerely thank Reviewers f7S3 (R1), zT4y (R2) and yHLy (R3) for their time and constructive feedback. We are glad that the reviewers recognize the following strengths of Ψ0.1\Psi_{0.1}:

  1. Strong real-robot results: "high system maturity and strong performance in real-world robotic experiments" (R1), "experimental results are strong" (R2), and a "convincing" evaluation with "a substantial demonstration of task composition" (R3).
  2. Variable-length action pre-training: "practically innovative and supported by experimental evidence" (R1) and "a useful training contribution" (R3).
  3. A coherent system design that connects subtask prediction, future-image generation and action generation (R2, R3).

We addressed the main concerns with new real-robot experiments and analyses:

  1. Does the time-conditioned future image itself bring the gains? (R1, R2, R3) With the same planner and controller, the future image outperforms a subtask-completion image (9/12 and 10/12 vs. 7/12 and 6/12 on novel tabletop and whole-body PnP), and a sweep of Δt\Delta t shows that 2.4–3.2 s works best (App. H, Fig. 7a). Our time conditioning also slightly improves closed-loop success over caption-only conditioning.
  2. When does explicit visual guidance help, and when does it mislead? (R3) We screened all 618 logged real-robot rollouts, attribute each failure to subtask selection, future-image prediction, action planning or execution, and show representative successes and failures (new App. I).
  3. Are the results reliable? (R1, R3) We doubled the ablation to 20 trials per variant and task (Table 1) and report 95% confidence intervals in our responses. For the long-horizon routine, we report the fully autonomous completions (3 of 10 runs for Ψ0.1\Psi_{0.1}, none for the baselines) and state the intervention protocol (new App. J).
  4. Ground-truth or generated future images in fine-tuning? (R1, R3) Fine-tuning on ground-truth future images matches a 50/50 mixture when the planner is deployed with generated ones (App. H, Fig. 7b).
  5. How do the modules interact? (R2) We state the inputs and outputs of each module (Sec. 3.2) and ablate the subtask predictor in our response to R2.

Summary of revisions. Changes to the existing text are marked in blue, and Apps. H, I and J are new.

  • Sec. 3.2: the choice of Δt\Delta t, and a new paragraph on the data flow between the modules.
  • Sec. 3.3: the statement on fine-tuning with ground-truth future images now refers to App. H.
  • Sec. 4.3: the number of runs and the fully autonomous completions of the long-horizon routine.
  • Sec. 4.4 and Table 1: 20 trials per task, and a pointer to the failure analysis.
  • App. H (Planner Ablations): goal type and Δt\Delta t (Fig. 7a), and future images in fine-tuning (Fig. 7b).
  • App. I (Failure Cases Analysis): failure attribution (Table 5) and representative cases (Figs. 8–11).
  • App. J (Intervention Protocol): the protocol of the long-horizon routine and the fully autonomous runs (Table 6).

We hope these additions resolve the reviewers' concerns and are glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.