- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents WildWAM, a framework for learning robot manipulation from in-the-wild human videos. It introduces RIVER for retrieving relevant human demonstrations through multimodal search and event-aware verification, and SAIL for selectively aligning human video context with robot history. Using 59,589 robot episodes and approximately 100k annotated human–robot pairs, the method achieves 62.0% retrieval precision@5 and improves real-robot task success from 28.0% to 44% compared to target-only adaptation.
Critical strengths
-
RIVER effectively retrieves transferable manipulation procedures from unstructured human videos, achieving substantial improvements over existing retrieval baselines.
-
The method improves action prediction and achieves higher success rates across five real-robot manipulation tasks, demonstrating the potential of Internet human videos for robot learning.
Critical weaknesses
-
The contributions of SAIL's components remain unclear, and additional pre-adaptation updates may confound the reported improvements.
-
Real-robot evaluations rely on collected human references rather than directly retrieved Internet videos, leaving the complete pipeline insufficiently validated.
Questions for authors
-
Can the authors provide compute-matched comparisons and controls using randomly matched human references?
-
Can the authors provide component-level ablations of SAIL against standard cross-attention?
-
Can the authors evaluate real-robot performance using human videos directly retrieved by RIVER?
2 comments
Continued Response: Direct Internet References in Robot Execution
We evaluate the end-to-end connection from retrieval to control using retrieved Internet references during robot execution.
Retrieved Internet clips enter the same model interface as collected demonstrations. The reported 44.0% final success and 67.7% mean stage progress use collected references. Offline predictions with Internet references have also been evaluated: the human clip is the existing top RIVER result for another robot demonstration of the same task, selected independently of the evaluated robot's future. Table II reports the robot-execution comparison.
The robot test separates policy learning from reference source. WildWAM and Target-only receive the same Internet clip; a second comparison fixes WildWAM and switches between Internet and collected references. The design uses five tasks and fifteen trials per task and condition, randomizes execution order within matched initial states, and freezes the model and references before testing. Retrieval uses the deployment instruction and robot history available before execution.
Table II. Robot execution with Internet and collected references. Each condition has 75 trials. Success rates follow integer counts; stage progress averages the completed milestone fraction with equal weight per task.
| Policy and reference | Full-task success | Mean stage progress |
|---|---|---|
| Target-only + retrieved Internet reference | 22/75 (29.3%) | 51.3% |
| WildWAM + the same Internet reference | 32/75 (42.7%) | 65.1% |
| WildWAM + collected reference | 33/75 (44.0%) | 67.7% |
Table II records a 13.3-percentage-point increase in final success and a 13.8-point increase in stage progress for WildWAM relative to Target-only with the same Internet clip. Holding the reference fixed measures the trained-policy effect without differences in the demonstration supplied to each model.
At a fixed WildWAM model, replacing the collected reference with the retrieved Internet reference produces a 1.3-point difference in final success. This comparison measures the effect of reference source with the policy held fixed. Together, the same-reference policy comparison and the fixed-policy reference comparison evaluate the use of retrieved human procedures during robot execution.
Response: Separating Training Exposure, Correspondence, and SAIL Components
Thank you for recognizing RIVER's ability to retrieve transferable procedures and the gains across our five robot tasks. We agree that the two central questions are how to attribute the learning gains and whether directly retrieved references support execution. Here, P denotes Internet pre-adaptation and T denotes target-scene adaptation.
The controls separate extra training, correct pairing, and SAIL. P→T models share the robot data, task text, sampled segments, generation noise, and training updates. Shuffled videos preserve the human clips and their sampling frequency. The robot-only, shuffled-video, and matched-video controls disable the extra losses derived from pair annotations. SAIL ablations keep the full objective, encoders, model size, and training schedule fixed.
Table I. Training controls and SAIL ablations. Entries give relative MAE reductions against Target-only; higher is better. Matched-video P→T disables pair-specific losses to evaluate correspondence under matched supervision.
The full model reduces action error by 17.23% and RGB error by 2.57% relative to Target-only. Matched-video training without pair-specific losses reduces action MAE by 9.55% relative to robot-only P and by 10.07% relative to shuffled-video P. These comparisons quantify correct pairing at fixed robot exposure and supervision. Relative to doubled-T training, the full model reduces action error by 13.09%. SAIL improves on standard cross-attention by 2.77% and on selection without robot history by 6.91%, quantifying the contribution of the tested components.
The trained history ablation measures the role of reference selection. In the measured control study, history-conditioned selection reduces action MAE from 0.09896 to 0.09246, a 6.56% reduction. Both seeds improve, by 8.43% and 4.68%. Robot history remains available to the controller in both variants; the changed variable is its use to select human content. This comparison measures the contribution of robot-history-conditioned selection.
Reference-content interventions complement the trained controls. With observations, history, and generation noise fixed, a different-program reference increases action MAE by 6.22% and motion-weighted video-feature MSE by 4.08%, whereas another valid same-task reference changes both by less than 0.5%. This intervention demonstrates content-dependent prediction at inference. Table I separately evaluates training exposure, correspondence, and SAIL.
The next section evaluates whether a retrieved Internet procedure can serve as the reference during robot execution.