- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Zero-WAM, a causal world-action model that uses human demonstration videos as in-context instructions for unseen robotic manipulation tasks. The authors introduce HumanGen, a dataset of 74.2K human-robot pairs across 8.6K tasks, and propose In-Context Future Chunk Prediction (IFP) to encourage long-term task understanding. Zero-WAM achieves 46.95% success on seven unseen RoboTwin tasks, outperforming LingBot-VA by 29.5 percentage points, with additional real-world validation.
Critical strengths
-
Scalable human-robot data generation: HumanGen provides a practical pipeline for generating human demonstration videos from existing robot trajectories, enabling task-diverse training without costly manual data collection.
-
Promising generalization results: Zero-WAM substantially outperforms the selected baselines on unseen RoboTwin tasks and shows encouraging results in real-world manipulation. The ablations also demonstrate the benefits of task-balanced pretraining and IFP.
Critical weaknesses
-
Limited methodological novelty: Human-video-conditioned robot policies have already been explored in Vid2Robot (Jain et al., 2024) and WAM-TTT (Feng et al., 2026), although the latter relies on test-time memory adaptation. Moreover, IFP shares substantial similarities with the multi-chunk prediction objective in Next Forcing (Xu et al., 2026). The paper primarily integrates existing ideas without sufficiently clarifying the unique methodological contribution of IFP.
-
Insufficient evidence for human-video in-context learning: The text-only variant already achieves 39.44% success, compared with 46.95% for the full model. This suggests that much of the improvement may come from task-balanced pretraining rather than human-video conditioning. The current experiments do not sufficiently isolate these contributions or demonstrate that the policy genuinely follows the video instructions.
[1] Vid2Robot: Jain, V., et al. (2024). Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers. Robotics: Science and Systems (RSS 2024).
[2] WAM-TTT: Feng, Y., et al. (2026). WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time. arXiv preprint.
[3] Next Forcing: Xu, G., et al. (2026). Next Forcing: Causal World Modeling with Multi-Chunk Prediction. arXiv preprint.
Questions for authors
-
What is the key methodological difference between IFP and the multi-chunk prediction objective in Next Forcing (Xu et al., 2026)? Could the authors provide a controlled comparison?
-
Given the strong performance of the text-only variant (39.44%), how can the authors isolate the contribution of human-video conditioning from task-balanced pretraining and IFP? Could they further validate genuine video-guided task following by testing different human demonstrations under the same initial robot observation?
3 comments
response to reviewer A25h - 2
Q2 — Contributions of task-balanced training, IFP, and human-video conditioning
We clarify that each component is effective: task-balanced pre-training, human-video ICL, and IFP each contribute to cross-task generalization, as the comparisons below show.
We address cross-task generalization at two complementary levels: the generalization capability of text-conditioned autoregressive video-action generation, and guidance from a human video supplied as an additional task condition. Task-balanced text-only training strengthens the first, yielding 39.44% average success compared with 17.45% for LingBot-VA. The following comparisons examine the additional contributions of human-video conditioning and IFP.
ICL training is effective. Without task-balanced pretraining, directly performing human-video ICL post-training with IFP on RoboTwin achieves 36.36% average success, compared with 17.45% for LingBot-VA (+18.91 percentage points). This demonstrates that ICL training with IFP can itself substantially improve cross-task performance without task-balanced pretraining.
Adding IFP to the text-only model produces only a modest gain. Without human-video conditioning, text-only + IFP achieves 41.14%, compared with 39.44% for text-only (+1.70 percentage points). This suggests that future prediction alone does not account for the full improvement.
Combining human-video ICL with IFP raises success from the task-balanced text-only baseline's 39.44% to 46.95% (+7.51 percentage points), also exceeding text-only + IFP by 5.81 points. On the challenging unseen task Stack three blocks, text-only + IFP still records 0%, while the full model reaches 9%. In real-world Two-table-leg insertion, the full model achieves 16.7%, compared with 0% for the task-balanced text-only baseline. These breakthroughs from zero success highlight the value of human-video guidance for tasks that remain unresolved by the text-only models.
Q3 — Different human video following
We clarify that our real-world evaluation includes cases across the three task families where the initial robot scene and object arrangement are held fixed, while the human video is changed to specify different desired manipulations.
For example, in Two-table-leg insertion (see the third panel of Figure 5 for an example inference rollout), the robot observes four table legs of different colors in fixed positions. Each human video shows a person picking up two differently colored legs in sequence and inserting them into two target holes at different positions. During testing, we keep the initial robot scene unchanged and provide five different human videos showing different choices of two table legs, different pickup and insertion orders, and different target holes. Each video is tested six times (30 trials in total).
Without the human video, the same Zero-WAM model given only the correct task text achieves 0/30 success in this setting. These failures occur because the model does not follow the intended manipulation (e.g., it fails to determine which colored leg should be inserted into which hole).
With the human video, we inspected semantic following in the 30 rollouts. Only two trials failed because the model did not correctly follow the task semantics demonstrated in the human video. Full-task success was 16.7%, with the remaining failures occurring during precision insertion in this long-horizon task. These real-world experiments demonstrate that Zero-WAM can follow the task semantics of different human videos under the same initial robot observation. Consistently, in simulation, replacing the correct human video with a mismatched one reduces average success from 46.95% to 21.00%.
Follow-Up to Your Review
Dear Reviewer A25h,
Thank you for your thoughtful review and for the constructive feedback on our submission.
Since only a very short time remains in the discussion period, we are kindly following up on our rebuttal. In our responses, we clarify our core novelty and how our work differs from Vid2Robot, WAM-TTT, and Next Forcing, and provide a controlled comparison for IFP. We further separate the contributions of task-balanced pre-training, IFP, and human-video conditioning, and describe how the model follows different human videos under the same initial robot scene.
We would greatly appreciate it if you could review our responses and tell us whether they address your concerns.
Thank you very much for your consideration.
Kind regards,
The Authors of Submission 4343
response to reviewer A25h - 1
Thank you for your detailed and constructive review. We believe the clarifications and new results below give a clearer picture of the contributions of our work and of the role of human-video conditioning.
Q1 — Clarification of Novelty
Our core novelty is learning human-video following as an in-context capability through large-scale pre-training on task-matched human–robot pairs, so that a new human video specifies a new task at inference without any parameter update. It does not lie in human-video conditioning or multi-chunk prediction by themselves, which, as you note, have precedents. Achieving this raises three challenges: how to curate and balance the language-conditioned pre-training data, how to construct human–robot pairs at scale, and how to make in-context human-video following effective with a simple training objective while keeping the architecture scalable. Our task-balanced data curation, HumanGen (74.2K pairs over 8.6K tasks), and IFP are designed specifically for these challenges, and they differ from existing work in purpose, as detailed below.
(1) Differences from Vid2Robot and WAM-TTT.
Neither work learns human-video following through pre-training across thousands of tasks. They also differ from Zero-WAM in how the human video is used:
Vid2Robot. Zero-WAM uses video generation as a bridge from human videos to robot actions: the human video guides future robot-video generation, and actions are then decoded through inverse dynamics. Vid2Robot instead conditions action prediction directly on the human video without predicting future robot videos, which can make action prediction more difficult.
WAM-TTT. Zero-WAM learns human-video following during training and uses a new human video as an in-context task condition at inference, without parameter updates or fast-weight optimization. WAM-TTT instead optimizes fast-weight memory from human videos at deployment while keeping its WAM backbone frozen.
(2) Difference from Next Forcing.
IFP is designed specifically for human-video ICL. It adopts the multi-chunk prediction form of Next Forcing, which we cite in Section 4.3, but it serves a different purpose, and its benefit is specific to human-video ICL.
Purpose. Next Forcing predicts future video chunks to accelerate model convergence. In human-video-conditioned training, the model can predict the immediate robot future from robot history while ignoring the human video; IFP supervises more distant robot futures so that the model has to extract task-relevant information from the human video.
Controlled comparison. To verify this, we train both the text-only model and the human-video-conditioned model with and without IFP, keeping everything else the same. The table reports the seven-task average success rate (%) on RoboTwin.
These results show that the benefit of IFP is specific to human-video ICL, as designed. Applied to the text-only model, IFP brings only a modest gain (+1.70 points). Applied to the human-video-conditioned model, it brings +18.40 points: without IFP, the model does not make effective use of the human video (28.55%), and with IFP it achieves the best result (46.95%). Compared with the text-only + IFP model, the full model also shows clear advantages on tasks that are difficult to describe with text; for example, it breaks the zero-success barrier on our real-world Two-table-leg insertion task.