acceptodds
Under review as a conference paper at ICLR 2027

Zero-WAM: In‑Context World‑Action Modeling from Human Videos for Open‑Ended Task Generalization

15 views

Abstract

Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic manipulation, in-context learning, cross-task generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A25hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Zero-WAM, a causal world-action model that uses human demonstration videos as in-context instructions for unseen robotic manipulation tasks. The authors introduce HumanGen, a dataset of 74.2K human-robot pairs across 8.6K tasks, and propose In-Context Future Chunk Prediction (IFP) to encourage long-term task understanding. Zero-WAM achieves 46.95% success on seven unseen RoboTwin tasks, outperforming LingBot-VA by 29.5 percentage points, with additional real-world validation.

Critical strengths
  1. Scalable human-robot data generation: HumanGen provides a practical pipeline for generating human demonstration videos from existing robot trajectories, enabling task-diverse training without costly manual data collection.

  2. Promising generalization results: Zero-WAM substantially outperforms the selected baselines on unseen RoboTwin tasks and shows encouraging results in real-world manipulation. The ablations also demonstrate the benefits of task-balanced pretraining and IFP.

Critical weaknesses
  1. Limited methodological novelty: Human-video-conditioned robot policies have already been explored in Vid2Robot (Jain et al., 2024) and WAM-TTT (Feng et al., 2026), although the latter relies on test-time memory adaptation. Moreover, IFP shares substantial similarities with the multi-chunk prediction objective in Next Forcing (Xu et al., 2026). The paper primarily integrates existing ideas without sufficiently clarifying the unique methodological contribution of IFP.

  2. Insufficient evidence for human-video in-context learning: The text-only variant already achieves 39.44% success, compared with 46.95% for the full model. This suggests that much of the improvement may come from task-balanced pretraining rather than human-video conditioning. The current experiments do not sufficiently isolate these contributions or demonstrate that the policy genuinely follows the video instructions.

[1] Vid2Robot: Jain, V., et al. (2024). Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers. Robotics: Science and Systems (RSS 2024).

[2] WAM-TTT: Feng, Y., et al. (2026). WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time. arXiv preprint.

[3] Next Forcing: Xu, G., et al. (2026). Next Forcing: Causal World Modeling with Multi-Chunk Prediction. arXiv preprint.

Questions for authors
  1. What is the key methodological difference between IFP and the multi-chunk prediction objective in Next Forcing (Xu et al., 2026)? Could the authors provide a controlled comparison?

  2. Given the strong performance of the text-only variant (39.44%), how can the authors isolate the contribution of human-video conditioning from task-balanced pretraining and IFP? Could they further validate genuine video-guided task following by testing different human demonstrations under the same initial robot observation?

3 comments
AuthorsOpenReview ↗

response to reviewer A25h - 1

Thank you for your detailed and constructive review. We believe the clarifications and new results below give a clearer picture of the contributions of our work and of the role of human-video conditioning.

 

Q1 — Clarification of Novelty

Our core novelty is learning human-video following as an in-context capability through large-scale pre-training on task-matched human–robot pairs, so that a new human video specifies a new task at inference without any parameter update. It does not lie in human-video conditioning or multi-chunk prediction by themselves, which, as you note, have precedents. Achieving this raises three challenges: how to curate and balance the language-conditioned pre-training data, how to construct human–robot pairs at scale, and how to make in-context human-video following effective with a simple training objective while keeping the architecture scalable. Our task-balanced data curation, HumanGen (74.2K pairs over 8.6K tasks), and IFP are designed specifically for these challenges, and they differ from existing work in purpose, as detailed below.

(1) Differences from Vid2Robot and WAM-TTT.

Neither work learns human-video following through pre-training across thousands of tasks. They also differ from Zero-WAM in how the human video is used:

  • Vid2Robot. Zero-WAM uses video generation as a bridge from human videos to robot actions: the human video guides future robot-video generation, and actions are then decoded through inverse dynamics. Vid2Robot instead conditions action prediction directly on the human video without predicting future robot videos, which can make action prediction more difficult.

  • WAM-TTT. Zero-WAM learns human-video following during training and uses a new human video as an in-context task condition at inference, without parameter updates or fast-weight optimization. WAM-TTT instead optimizes fast-weight memory from human videos at deployment while keeping its WAM backbone frozen.

(2) Difference from Next Forcing.

IFP is designed specifically for human-video ICL. It adopts the multi-chunk prediction form of Next Forcing, which we cite in Section 4.3, but it serves a different purpose, and its benefit is specific to human-video ICL.

Purpose. Next Forcing predicts future video chunks to accelerate model convergence. In human-video-conditioned training, the model can predict the immediate robot future from robot history while ignoring the human video; IFP supervises more distant robot futures so that the model has to extract task-relevant information from the human video.

Controlled comparison. To verify this, we train both the text-only model and the human-video-conditioned model with and without IFP, keeping everything else the same. The table reports the seven-task average success rate (%) on RoboTwin.

w/o IFPw/ IFPGain from IFP
Text-only39.4441.14+1.70
Human video28.5546.95+18.40

These results show that the benefit of IFP is specific to human-video ICL, as designed. Applied to the text-only model, IFP brings only a modest gain (+1.70 points). Applied to the human-video-conditioned model, it brings +18.40 points: without IFP, the model does not make effective use of the human video (28.55%), and with IFP it achieves the best result (46.95%). Compared with the text-only + IFP model, the full model also shows clear advantages on tasks that are difficult to describe with text; for example, it breaks the zero-success barrier on our real-world Two-table-leg insertion task.

AuthorsOpenReview ↗

response to reviewer A25h - 2

Q2 — Contributions of task-balanced training, IFP, and human-video conditioning

We clarify that each component is effective: task-balanced pre-training, human-video ICL, and IFP each contribute to cross-task generalization, as the comparisons below show.

We address cross-task generalization at two complementary levels: the generalization capability of text-conditioned autoregressive video-action generation, and guidance from a human video supplied as an additional task condition. Task-balanced text-only training strengthens the first, yielding 39.44% average success compared with 17.45% for LingBot-VA. The following comparisons examine the additional contributions of human-video conditioning and IFP.

ICL training is effective. Without task-balanced pretraining, directly performing human-video ICL post-training with IFP on RoboTwin achieves 36.36% average success, compared with 17.45% for LingBot-VA (+18.91 percentage points). This demonstrates that ICL training with IFP can itself substantially improve cross-task performance without task-balanced pretraining.

Adding IFP to the text-only model produces only a modest gain. Without human-video conditioning, text-only + IFP achieves 41.14%, compared with 39.44% for text-only (+1.70 percentage points). This suggests that future prediction alone does not account for the full improvement.

Combining human-video ICL with IFP raises success from the task-balanced text-only baseline's 39.44% to 46.95% (+7.51 percentage points), also exceeding text-only + IFP by 5.81 points. On the challenging unseen task Stack three blocks, text-only + IFP still records 0%, while the full model reaches 9%. In real-world Two-table-leg insertion, the full model achieves 16.7%, compared with 0% for the task-balanced text-only baseline. These breakthroughs from zero success highlight the value of human-video guidance for tasks that remain unresolved by the text-only models.

 

Q3 — Different human video following

We clarify that our real-world evaluation includes cases across the three task families where the initial robot scene and object arrangement are held fixed, while the human video is changed to specify different desired manipulations.

For example, in Two-table-leg insertion (see the third panel of Figure 5 for an example inference rollout), the robot observes four table legs of different colors in fixed positions. Each human video shows a person picking up two differently colored legs in sequence and inserting them into two target holes at different positions. During testing, we keep the initial robot scene unchanged and provide five different human videos showing different choices of two table legs, different pickup and insertion orders, and different target holes. Each video is tested six times (30 trials in total).

Without the human video, the same Zero-WAM model given only the correct task text achieves 0/30 success in this setting. These failures occur because the model does not follow the intended manipulation (e.g., it fails to determine which colored leg should be inserted into which hole).

With the human video, we inspected semantic following in the 30 rollouts. Only two trials failed because the model did not correctly follow the task semantics demonstrated in the human video. Full-task success was 16.7%, with the remaining failures occurring during precision insertion in this long-horizon task. These real-world experiments demonstrate that Zero-WAM can follow the task semantics of different human videos under the same initial robot observation. Consistently, in simulation, replacing the correct human video with a mismatched one reduces average success from 46.95% to 21.00%.

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer A25h,

Thank you for your thoughtful review and for the constructive feedback on our submission.

Since only a very short time remains in the discussion period, we are kindly following up on our rebuttal. In our responses, we clarify our core novelty and how our work differs from Vid2Robot, WAM-TTT, and Next Forcing, and provide a controlled comparison for IFP. We further separate the contributions of task-balanced pre-training, IFP, and human-video conditioning, and describe how the model follows different human videos under the same initial robot scene.

We would greatly appreciate it if you could review our responses and tell us whether they address your concerns.

Thank you very much for your consideration.

Kind regards,

The Authors of Submission 4343

Reviewer_LViMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Zero-WAM is a causal video-action model that is conditioned on language or on an in-context human video. Its contributions are HumanGen (74.2K synthetic human-video/robot pairs over 8.6K tasks, produced by generating human videos from robot trajectories) and IFP, an auxiliary loss on future video chunks. On 7 held-out RoboTwin tasks it reaches 46.95% vs. 17.45% for LingBot-VA, and it also beats LingBot-VA on three real-world Franka task families.

Critical strengths

S1. Scalable, action-grounded paired human-robot data: 8.6K tasks, far beyond manually collected datasets. S2. Large gains on held-out tasks. Task-balanced re-sampling alone gives +22 pp over LingBot-VA, which is a useful practical finding.

Critical weaknesses

W1. The gains are not clearly due to the human video. The text-only variant already reaches 39.44% vs. 46.95% for the full model, and adding the video without IFP (28.55%) is worse than text-only. There is no no-video or mismatched-video control. The test videos are generated from expert demonstrations of the same 'unseen' tasks, while baselines receive only language, so the setting is closer to one-shot imitation than to zero-shot. W2. Several Table 1 values (e.g. 69.14, 84.87±0.18) cannot arise from the stated 3 seeds x 100 rollouts. The ablations have no variance. The real-world evaluation is small (30 trials, and the baseline uses a different modality).

Questions for authors
  1. Please explain the Table 1 values that are not attainable under 3 x 100 rollouts.
  2. Please report results with no video, a mismatched video, and text-only + IFP, and explain how the 7 held-out tasks were chosen.
  3. Were successes such as open microwave verified manually, and can the results be reproduced from the released data?
5 comments
AuthorsOpenReview ↗

response to reviewer LViM - 1

Thank you for your thoughtful questions. We address each point below and hope our responses resolve your concerns.

 

Q1 — Gains from human video and the role of IFP

(1) Gains from human video.

Thank you for highlighting the gains from task-balanced training of the text-only model. We address cross-task generalization at two levels: first, improving the text-only model's ability to generalize to new tasks; second, using human videos as an additional in-context task condition to help this strong model perform tasks that remain challenging with text alone.

Task-balanced training improves text-only success from 17.45% to 39.44%. Building on this stronger baseline, our full human-video ICL method reaches 46.95% (+7.51 percentage points). The benefit is particularly evident on challenging unseen tasks: while the task-balanced text-only model records 0% success on both Stack three blocks in RoboTwin and real-world Two-table-leg insertion configurations, our full method achieves 9% and 16.7%, respectively. These breakthroughs from zero success highlight the value of human-video ICL beyond stronger text-only generalization.

The benefit of our ICL approach is also observed without the proposed pre-training stages. With only ICL post-training on RoboTwin, and no Task-diverse VA or HumanGen pre-training, Zero-WAM achieves 36.36% average success, compared with 10.98% for WAN-Action, which shares the same Wan-2.2 initialization and RoboTwin training tasks (+25.38 percentage points, Table 1). Against the same baseline, this gain is comparable in size to the one from task-balanced pre-training alone (10.98% to 39.44% for our text-only variant, +28.46 points).

(2) The role of IFP.

Without IFP, directly adding human videos as a task condition performs worse than the text-only model, a phenomenon we also observed during our experiments. One possible explanation is that, without an explicit learning objective encouraging the extraction of task-relevant information, the model may distribute attention to the human video without identifying the cues useful for generating the target robot video. This unfocused attention may interfere with prediction and reduce performance.

IFP introduces supervision for predicting robot-video chunks further into the future. This more demanding prediction objective encourages the model to focus on information in the human video that is useful for predicting future robot behavior. We believe this helps explain why IFP enables the model to benefit from human-video conditioning.

The table below reports the seven-task average success rate (%) with and without IFP, for both the text-only and the human-video-conditioned models.

w/o IFPw/ IFPGain from IFP
Text-only39.4441.14+1.70
Human video28.5546.95+18.40

The text-only model with IFP achieves an average success rate of 41.14% across the seven unseen RoboTwin tasks. This is a modest improvement of 1.70 percentage points over the reported text-only baseline (39.44%), whereas IFP improves the human-video-conditioned model by 18.40 percentage points. This result suggests that the gains from IFP alone do not fully account for the benefit of our approach, and that IFP and human-video ICL are complementary.

AuthorsOpenReview ↗

response to reviewer LViM - 2

Q2 — No-video and mismatched-video evaluations

(1) Mismatched-video evaluation.

We evaluated a mismatched-video setting using the full Zero-WAM model. For each of the seven unseen RoboTwin tasks, we randomly selected a human video from one of the other six tasks as the in-context input and provided it together with the correct text instruction for the target task. Success is measured against the original target task. The table reports per-task success rates (%); the additional setting uses 100 rollouts per task.

TaskCorrect videoMismatched video
Place object on scale24.679.00
Stamp seal47.0010.00
Open microwave59.0032.00
Move stapler to pad69.1443.00
Place bread in basket35.0024.00
Place empty cup84.8728.00
Stack three blocks9.001.00
Average46.9521.00

Mismatched videos reduce average success to 21.00%, compared with 46.95% using correct videos. These results support the model's dependence on task-relevant information in the human video.

(2) No-video evaluation.

In this additional evaluation, we ran 100 rollouts per RoboTwin task and 30 trials for real-world Two-table-leg insertion, using the full Zero-WAM model with the human video removed at inference and only the correct task text provided, without retraining the model. The table compares success rates (%) with the standard human-video-conditioned setting.

TaskCorrect video No text (full)No video Correct text
Place object on scale24.6721.00
Stamp seal47.0035.00
Open microwave59.0055.00
Move stapler to pad69.1438.00
Place bread in basket35.0045.00
Place empty cup84.8774.00
Stack three blocks9.002.00
Two-table-leg insertion (real world)16.70.0

Human-video conditioning improves the average success rate across the seven RoboTwin tasks from 38.57% to 46.95% (+8.38 percentage points). On Stack three blocks, success increases from 2% without video to 9% with video. In real-world Two-table-leg insertion, the same model without video achieves 0/30 successes, while human-video conditioning achieves 5/30 (16.7%). These results highlight the benefit of human-video ICL on challenging unseen tasks whose task dynamics can be difficult to convey through text alone.

 

Q3 — Zero-shot and one-shot terminology

The core of this work is to learn, through large-scale in-context pre-training, the ability to generalize to new robot tasks using a human video as an additional task condition, without requiring robot demonstrations of the new task for training or adaptation. Our use of “zero-shot” refers to the absence of such task-specific robot training data. We agree that, when the human video provided at inference is counted as a task demonstration, this setting can also be described as one-shot learning from a human video. We will clarify this distinction in the revised manuscript. Thank you for this suggestion.

In RoboTwin, the test-time human videos are generated with our pipeline from demonstrations of the held-out tasks; they serve only as test-time prompts, and no data from the held-out tasks is used for training. Our method does not depend on generated prompts: in the real-world experiments, the prompts are videos recorded directly by humans, and no robot expert demonstration of the tested task configuration is needed.

AuthorsOpenReview ↗

response to reviewer LViM - 3

Q4 — Table 1 statistics

We traced the discrepancy in the full Zero-WAM results to two tasks whose evaluations did not complete 100 rollouts because RoboTwin stalled on our GPU setup. We stopped an affected run after more than 90 completed rollouts and a stall lasting over 10 hours. We described these as approximately 100-rollout evaluations, but used the actual completed rollout counts to calculate success rates.

For the full Zero-WAM model, only two task–seed runs were affected: Move stapler to pad (Seed 50) and Place empty cup (Seed 100), each completing 91 rollouts. The other 19 runs of the full model completed 100 rollouts each. The table below provides the actual evaluation records for the full Zero-WAM model, including the number of successes and completed rollouts for each task and seed.

TaskSeed 0Seed 50Seed 100Mean ± SD (%)
Place obj. on scale25/100 (25.00%)22/100 (22.00%)27/100 (27.00%)24.67 ± 2.05
Stamp seal52/100 (52.00%)41/100 (41.00%)48/100 (48.00%)47.00 ± 4.55
Open microwave57/100 (57.00%)63/100 (63.00%)57/100 (57.00%)59.00 ± 2.83
Move stapler to pad71/100 (71.00%)65/91 (71.43%)65/100 (65.00%)69.14 ± 2.93
Place bread in basket39/100 (39.00%)36/100 (36.00%)30/100 (30.00%)35.00 ± 3.74
Place empty cup85/100 (85.00%)85/100 (85.00%)77/91 (84.62%)84.87 ± 0.18
Stack three blocks6/100 (6.00%)10/100 (10.00%)11/100 (11.00%)9.00 ± 2.16
Seven-task macro average47.86%46.92%46.09%46.95 ± 0.72

We apologize for the ambiguity in the original description and will clarify the actual rollout counts and stopping criterion in a footnote in the revised manuscript.

 

Q5 — Standard deviations of ablation results

Our ablation study includes three model variants. The table below reports their success rates (%, mean ± standard deviation across three seeds).

TaskZero-WAM w/o pre-trainingZero-WAM text-onlyZero-WAM w/o IFP
Place object on scale22.00 ± 1.6322.67 ± 6.8519.67 ± 6.02
Stamp seal53.33 ± 2.3617.33 ± 1.2521.33 ± 3.68
Open microwave25.90 ± 11.0341.43 ± 6.923.80 ± 3.02
Move stapler to pad74.33 ± 0.4765.00 ± 2.4556.33 ± 1.89
Place bread in basket32.63 ± 2.8364.00 ± 0.0039.70 ± 6.87
Place empty cup46.33 ± 0.4765.67 ± 14.7059.00 ± 4.24
Stack three blocks0.00 ± 0.000.00 ± 0.000.00 ± 0.00
Average36.36 ± 1.8139.44 ± 3.3028.55 ± 1.13

 

Q6 — Real-world evaluation scale and input modalities

(1) Real-world evaluation scale.

Real-world robot evaluation is time-consuming and labor-intensive, requiring physical execution and repeated scene resets. Representative works such as Diffusion Policy report 20 trials per bimanual task (Sections 7.3–7.5), while π₀ reports 10 trials per task in its cited real-world evaluations (Sections VI-A, VI-C–D). Our initial evaluation used 30 trials per task, exceeding these trial counts. To further assess the consistency of our results, we conducted an additional 30 trials per task. Zero-WAM succeeded in 18/30, 12/30, and 4/30 trials on object-to-container placement, three-object sequential manipulation, and two-table-leg insertion, respectively, close to the initial evaluation (16/30, 10/30, and 5/30).

(2) Input modalities and baseline choice.

To our knowledge, when this work was completed, no open-source WAM method pre-trained at scale with human videos as task conditions had been released for comparison. We therefore use LingBot-VA, a strong open pre-trained model with a closely related causal video-action modeling architecture. Our comparison evaluates the benefit of explicitly training a world-action model to use human videos as task conditions for cross-task generalization. LingBot-VA receives detailed textual task descriptions, while Zero-WAM receives human-video instructions. In addition, the text-only ablations in Q1 and the no-video and mismatched-video evaluations in Q2 provide comparisons between human-video and text-only conditions within our own model framework. We will also add these experiments to the revised manuscript, and we hope they give a clear picture of the value of our work.

Reviewer_rY4vOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work takes a robot trajectory, converts the scene to a human one, roles out, and converts to a robot trajectory. I think the idea is that there's more... support? training data? for the WM in the human space so if it has matched semantics with the original intent then it's a useful intermediary. It then uses this approach for ICL.

Critical strengths

The results appear to be promising and the scale of the data substantial

Critical weaknesses

I'm unclear on where, when, or why the approach works unfortunately. I'll outline a few questions here:

  1. The human videos are generated not real. The motivation had a deployment setup with humans providing them but that's not the case here which is why I'm inferring there's some benefit to the training objective. In short, why do I need human videos here?
  2. Most of the gains don't seem to come from the paper's key innovations(?) but from data balancing?
  3. There's no discussion of which videos, number of videos, with what properties, under what conditions, etc are the right/necessary ones. We could include experiments with different viewpoints? mismatched videos? etc to try and start to explain the procedure/setup more clearly?
  4. Related to (3), I'm unclear on if we've ever validated the "task semantics" filter? I really appreciate Fig 5 as it's what I'm having to base my intuitions on. Are all scenes this clean? Is the VLM filter robust?
Questions for authors

See above

3 comments
AuthorsOpenReview ↗

Response to Reviewer rY4v - 1

Thank you for these constructive questions. We first summarize the overall idea and then address each point. We believe the clarifications below give a clearer picture of the motivation and design of our work.

Our central goal is cross-task generalization: enabling a robot to execute a new task without updating its parameters for that task. Existing vision-language-action and video-action models predominantly use text instructions as task conditions. However, for an unfamiliar task, text alone may leave important execution details underspecified. For example, in our real-world Two-table-leg insertion task, the instruction must convey which two of the four colored legs to pick, in which order, and which holes to insert them into; this is hard to express precisely in text, whereas a short human video shows it directly. We therefore use a human demonstration video, provided at test time, to supply these task-semantic and visual cues as an in-context task specification. Learning to follow such a video requires human-video/robot-trajectory pairs with matching task semantics for training. To this end, we generate human videos that match robot trajectories, to train the model to understand the manipulation intent in a human video. At deployment, the human video can be either generated or recorded by a person, and we test both cases: generated human videos for simulated tasks, and human-recorded videos for real-world deployment.

 

1. W1 — Why in-context human video?

As summarized above, we generate human videos in order to train the model to follow human videos. Human–robot pairs with matching task semantics are scarce, so HumanGen generates, for each robot trajectory, a human video that performs the same task. Training on these pairs teaches the model to understand the task semantics of the human actions in a video and to follow them when predicting future robot videos and actions.

Since what the model learns is to follow the task semantics of human actions, the human video provided at deployment can be either generated or recorded by a person. Our evaluations cover both: the RoboTwin experiments (Table 1) use human videos generated with our proposed pipeline, and the real-world experiments (Table 2) use videos recorded by humans, which is the deployment setup described in our motivation. Zero-WAM outperforms the language-conditioned baselines in both settings. We will state the source of the test-time videos explicitly in Section 5.3.

The human video also matters at test time: removing it from the same model reduces average success from 46.95% to 38.57% in RoboTwin and from 5/30 to 0/30 on real-world Two-table-leg insertion (see W3, Part B).

 

2. W2 — Gains beyond task-balanced training

Although task-balanced training is an important component, we demonstrate that the ICL formulation and IFP are equally important. The table below compares model variants that differ in these components on the seven unseen RoboTwin tasks (success rate, %).

ModelTask-balanced pre-trainingHuman-video ICLIFPSeven-task averageStack three blocks
WAN-Action✗✗✗10.980.00
Zero-WAM w/o pre-train✗✓✓36.360.00
Zero-WAM text-only✓✗✗39.440.00
Zero-WAM w/o IFP✓✓✗28.550.00
Zero-WAM (full)✓✓✓46.959.00
  • ICL without task-balanced pre-training. With the same Wan-2.2 initialization and the same 43 RoboTwin training tasks, Zero-WAM w/o pre-train (with our ICL formulation) improves average success over WAN-Action from 10.98% to 36.36% (+25.38 percentage points). This gain is comparable in size to the one from task-balanced pre-training alone (Zero-WAM text-only: 10.98% to 39.44%, +28.46 points).
  • IFP. Within the human-video-conditioned setting, adding IFP to Zero-WAM w/o IFP gives the full Zero-WAM and improves average success from 28.55% to 46.95% (+18.40 percentage points).
  • Gains on top of task-balanced training. The full Zero-WAM further improves over Zero-WAM text-only from 39.44% to 46.95% (+7.51 percentage points). It is also the only variant that breaks the zero-success barrier on Stack three blocks (9%) and on unseen real-world Two-table-leg insertion configurations (16.7%, versus 0% for Zero-WAM text-only).

These results show that our ICL approach contributes to cross-task generalization beyond task-balanced training.

AuthorsOpenReview ↗

Response to Reviewer rY4v - 2

3. W3 — Video requirements and robustness

Following your suggestion, we add experiments on the number of videos, video removal, mismatched videos, and following different videos in the same scene. The requirement on the human video is simple: it should demonstrate the same task semantics as the target robot task; its background and object positions need not match the robot scene. Since the human videos used during post-training are egocentric, our evaluation also uses egocentric human videos. Our standard setting uses one human video as the in-context prompt. Each additional RoboTwin setting uses 100 rollouts per task with the same full model, without retraining. The table reports success rates (%).

Task1 correct video (standard)3 correct videosNo video Correct textMismatched video Correct text
Place object on scale24.6732.0021.009.00
Stamp seal47.0058.0035.0010.00
Open microwave59.0068.0055.0032.00
Move stapler to pad69.1483.0038.0043.00
Place bread in basket35.0029.0045.0024.00
Place empty cup84.8775.0074.0028.00
Stack three blocks9.007.002.001.00
Average46.9550.2938.5721.00

A. Number of human videos. Using three videos demonstrating the same target task achieves 50.29% average success, compared with 46.95% using one video. We consider the performance with a single video sufficient.

B. No human video. Removing the human video from the full model at inference and providing only the correct task text reduces the seven-task average from 46.95% to 38.57%. On Stack three blocks, success drops from 9% to 2%. In real-world Two-table-leg insertion, removing the human video reduces success from 5/30 (16.7%) to 0/30 (0%). These results support the benefit of human-video guidance for challenging unseen tasks whose task dynamics are difficult to convey through text alone.

C. Mismatched human videos. For each target task, we randomly selected a human video from one of the other six tasks, provided it together with the correct target-task text, and evaluated success against the original target task. Average success falls to 21.00%, compared with 46.95% using a correct video. These results support the model's use of task-relevant information in human videos.

D. Following different human videos. In real-world Two-table-leg insertion (Figure 5, third panel), we keep the initial robot scene unchanged and provide five different human videos, which specify different choices and insertion orders of colored legs and different target holes. Each video is tested six times (30 trials in total). Inspection of these 30 trials found only two failures in semantic following; the remaining failures occurred during precision insertion. These real-world experiments demonstrate that Zero-WAM can follow the task semantics of different human videos under the same initial robot observation.

 

4. W4 — Validation of the semantic filter

We clarify that the task-semantics filter has been validated, by a strict automatic check and by human experts. The VLM scores how well each generated human video matches its source robot video in task semantics on three levels, and we keep only videos with the top score; they must also pass a physical-plausibility check (Appendix A.1). About 75% of generated videos pass this filter. In addition, two experts in robotic manipulation manually checked 3,000 retained human–robot video pairs, comparing each generated human video against its source robot video in terms of the manipulated objects, task goal, and key object-state changes. Over 94% of the checked pairs passed, which supports the reliability of the VLM filter. We will add these details to the revised manuscript.

We clarify that not all scenes are clean. The real-world evaluation in Figure 5 also includes more complex scenes: three-object sequential manipulation uses scenes containing 6–7 objects and 3–4 containers (Appendix D.1). For data generation, most of the source robot teleoperation videos contain relatively clean scenes. We also segment particularly long or complex videos into shorter clips to facilitate conversion. Before generation, the pipeline explicitly parses the task and relevant objects, including their initial states, key state changes, and final states (Section 3.2). These steps help make the intended manipulation explicit during generation. Together, these design choices and the expert audit support the reliability of our pipeline.

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer rY4v,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a very busy period for you. As the discussion period is closing soon, we would like to briefly bring our point-by-point responses to your attention. In particular, we (1) clarified why human videos are generated for training while real human-recorded videos are used in our real-world experiments, (2) showed that the ICL formulation and IFP bring gains beyond task-balanced training, (3) added new experiments on the number of videos, removing the video, mismatched videos, and following different videos in the same scene, and (4) reported an expert validation of the task-semantics filter.

We would be grateful if you could take a look and let us know whether the responses address your concerns.

Thank you!

Best, The Authors of Submission 4343

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.