acceptodds
Withdrawn from ICLR 2027

Ego4WAM: What Matters When Scaling Egocentric Human Data for Robot Learning?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

25 views

Abstract

Egocentric human data provides a scalable source of experience for robot learning, but varies substantially in human-robot alignment, behavioral coverage, and available supervision. Existing work shows favorable scaling with increasing human data, but it remains unclear which data properties drive downstream robot gains and how to use such data throughout the training pipeline. We present a systematic study of egocentric human data with different alignment and supervision under a unified world-action model framework. With the model backbone fixed, we disentangle the effects of human-robot alignment, data duration and task diversity, action supervision, and data usage strategies. We find that aligned human demonstrations substantially improve out-of-distribution generalization and reduce target-task robot data requirements; data duration and task diversity affect downstream capabilities differently; and video-only supervision remains effective without action labels, providing a strong foundation for subsequent video-action training. We validate these findings through closed-loop policy evaluation on both real robots and RoboDojo. Rather than treating data duration as the sole scaling axis, Ego4WAM shows how alignment, task diversity, available supervision, and usage strategy jointly shape the value of egocentric human data for robot learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Egocentric Human Data, World-Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Zc1oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effect of gocentric human data for a fixed world-action model framework. Real-robot and RoboDojo results show useful transfer from aligned human demonstrations, benefits from video pretraining.

Critical strengths

The question that how to use human data is important. Duration/diversity scaling comparisons fix the backbone, initialization, and training budget, and include matched-volume composition controls at 2K and 4K ho

Critical weaknesses
  • The paper is more like an experimental report; many of its conclusions have already been discussed in prior work, and the model’s performance is not particularly strong.
  • The supervision comparisons use different data and compute budgets, which is confusing.
Questions for authors
  1. Do the differences between video-only, video-action, and staged training still exist on the same 12K-hour samples under matched compute?
  2. Please provide more details about real-robot trial counts, RoboDojo training/evaluation seeds and confidence intervals.
  3. The best model in Table 1 does not use mid-training, but the model in Table 2 use 0.1K hours of multi-task robot data. At which stage are these data used, and do they include downstream adaptation?
Reviewer_j2ydOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents Ego4WAM, a study that investigates how to leverage the unique properties of egocentric data for world-action model learning. Specifically, it examines which data properties benefit downstream policy learning and how to effectively utilize egocentric data for policy training. The paper validates its findings through closed-loop policy evaluations on both real robots and RoboDojo.

Critical strengths

Ego4WAM achieves a score of 20.29 and a success rate of 14.35% with pre-training on only 0.1K robot demonstrations and 15K egocentric human videos. These results demonstrate promising performance, particularly given the limited amount of robot data used.

Critical weaknesses
  1. The paper aims to study how to effectively leverage egocentric data for policy learning. It would be beneficial to compare against other training strategies, such as comparing the difference in performance between policies fine-tuned with robot + human data versus robot-only data,[1] to provide a more comprehensive understanding of how to utilize egocentric data.

  2. As a study paper, the presentation and experimental design are somewhat difficult to follow. For example, in Figure 5 and Section 4.2, "Scaling Beyond Task-Specific Egocentric Data," the left half of the figure keeps the number of tasks fixed while varying data duration, which is reasonable. However, in the right half, the experimental setups use 500 tasks / 1K hours, 1,000 tasks / 2K hours, 2,000 tasks / 4K hours, and 6,000 tasks / 12K hours. Both task diversity and data duration change simultaneously, making it difficult to determine whether the performance gains come from increased task diversity or longer data duration.

  3. The paper mainly studies two dimensions of egocentric data: duration and task diversity. These factors are relatively straightforward and intuitive. However, other potentially important dimensions, such as scene diversity and demonstrator diversity, remain unexplored.

  4. Learning Dynamics Priors without Action Supervision, followed by post-training with joint video-action prediction, achieves the strongest results. One key distinction between egocentric data and other internet video data is the possibility of annotating actions. Does this finding suggest that dynamics modeling, rather than action supervision, is the critical factor? It would be interesting to investigate whether replacing egocentric data with other types of internet video data could achieve similar results.

[1] Emergence of Human to Robot Transfer in VLAs. https://www.pi.website/research/human_to_robot

Questions for authors
  1. In Figure 5, both task diversity and data duration increase simultaneously. How can the authors disentangle their individual contributions to performance gains?

  2. Why does the paper only study data duration and task diversity? Could the authors also investigate other factors, such as scene diversity and demonstrator diversity?

  3. Does the strong performance of dynamics pre-training suggest that dynamics modeling matters more than action supervision? Could the authors replace egocentric videos with general internet videos during dynamics pre-training to investigate whether the performance gains come from learning general visual dynamics or from the unique properties of egocentric data?

Reviewer_Ea8oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how different properties of egocentric human data affect robot learning under a a world-action model framework. It investigates human–robot alignment, data duration versus task diversity, action supervision, and training strategies. Experiments on real robots and RoboDojo show improvements in OOD performance with aligned human demonstrations and from video-only pre-training. The authors address an important problem, but several experimental comparisons do not fully isolate the factors being investigated.

Critical strengths
  • Understanding what makes human data useful for robot learning, beyond simply increasing dataset size, is very important.
  • The results are promising, particularly the improvements in OOD manipulation performance with aligned human demonstrations and the gains from large-scale video pre-training. The finding that video-action mid-training is much less effective without prior video pre-training (10.28 vs. 15.04 average score).
Critical weaknesses
  • The role of human–robot alignment is not fully isolated. Section 4.1 shows that adding aligned human demonstrations improves OOD performance, but there is no comparison with an equivalent amount of less-aligned human data. Rows 3 and 4 of Figure 3 are matched in human-data volume, and they show that each OOD set mainly helps its own condition, which supports the role of coverage. However, all human data comes from EgoAlign. The experiments therefore show that human data covering the test condition helps, rather than that alignment itself matters. Figure 4 also lacks a multi-task mid-training baseline without aligned human demonstrations, so it is unclear how much the aligned data contributes after mid-training. Moreover, the ~90h of robot data used in mid-training includes towel folding, which is closely related to Fold Cloth. The recovery with 20 robot demonstrations may therefore partly come from this robot data rather than from human experience.
  • The data-duration and task-diversity comparisons are not fully controlled. Tasks are added in frequency order, so increasing task diversity also means adding rarer tasks. The increasingly long-tail data may also differ in source dataset, action-label quality, and relevance to the evaluation tasks. All scaling variants are trained for the same number of steps (40K, Table 3), so larger datasets see each sample far fewer times. In addition, the reported Precision trend appears to rely largely on a single task. The drop from Setup 1 to Setup 4 (7.0 → 2.8) is almost entirely explained by Build tower falling from 36% to 0%. Precision also falls from Setup 2 to Setup 3 (9.2 → 5.2) without any long-tail tasks being added. The conclusion that denser coverage provides “limited additional gains” also seems inconsistent with Section 4.2. There, duration scaling to 4K hours matches or exceeds task scaling to 12K hours on Generalization and Long-Horizon. These results are useful for studying data allocation, but do not fully establish the independent effects of duration and diversity.
  • The video pre-training conclusions need stronger ablations. The 15K-hour video pre-training set includes 12K hours of action-valid videos and only 3K additional hours of video-only data. Without comparing 12K versus 15K hours of video pre-training, it is unclear whether the additional data without reliable action labels provides a measurable benefit. This is central to the claim that video-only data extends usable data beyond the action subset. Furthermore, the pre-training and mid-training comparisons are not compute-matched: video pre-training uses 150K steps versus 80K for mid-training on the same number of nodes (Table 3).
Questions for authors
  • How does aligned human data compare against the same amount of unaligned human data? Does multi-task mid-training with 20 robot demonstrations achieve similar performance without EgoAlign? How much of the gain comes from the towel-folding robot data used in mid-training?
  • Does the additional 3K hours of video-only data improve performance over pre-training on the 12K-hour action-valid subset alone? Why does the best-performing model in Table 1 omit video-action mid-training? How does joint attention perform without video pre-training?
  • How many evaluation trials and training seeds are used? Are the scaling trends, particularly the precision decline, consistent across independent runs?
  • The paper states that the best model uses approximately 0.1K hours of multi-task robot data. However, this data is introduced during mid-training, which the best model skips. Which robot data does this model use?
Reviewer_Z7guOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies egocentric human data for world-action models through task-aligned demonstrations, large-scale mid-training, and video pretraining. It examines robot-data efficiency, generalization, and scaling with data duration and task diversity.

Critical strengths
  • The real-world experiments show where human demonstrations help and where they remain insufficient. In particular, aligned human data alone does not rescue the smallest robot-data setting.
  • Keeping the model architecture fixed makes the training comparisons useful. The equal-duration comparisons also provide evidence that collecting more tasks is not automatically better than collecting more examples of existing tasks.
Critical weaknesses
  • Several effects are not isolated. Mid-training adds approximately 90 hours of robot data as well as human data, and the pretraining table lacks joint video–action training without video pretraining. Could these controls be added? Given prior results from EgoMimic and LAPA, identifying the source of improvement is central to this paper.
  • Increasing task diversity also changes task relevance, frequency, and potentially annotation quality. The results support the particular data mixtures tested, but not a general conclusion about diversity. Could the authors control these factors and report training-seed variation and explicit hardware trial counts?
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.