acceptodds
Under review as a conference paper at ICLR 2027

WorldZoo: Compositional World Editing for Generalizable Robot Learning

Abstract

Imitation learning for robotic manipulation is limited by the high cost of collecting demonstrations and insufficient data diversity. By reusing existing demonstrations, data augmentation offers an effective way to broaden training data coverage under a limited collection budget. However, composing multiple edits without disrupting task-relevant interactions and selecting augmentations that benefit a target task remain challenging. We present **WorldZoo**, a unified world-editing framework that leverages robot demonstrations and human egocentric videos to generate augmented data through composable edits along three dimensions: embodiment, interaction, and scene. WorldZoo coordinates embodiment retargeting, visual editing, and trajectory adaptation to maintain consistency among observations, actions, and task-relevant contact relationships across diverse robotic arms and dexterous hands. Object editing adapts the demonstrated contacts to new object geometry within a range compatible with the original manipulation, expanding object diversity while preserving the demonstrated task. UniAug composes these edits within each sample as a reusable default recipe. We extend existing robotic simulation benchmarks to evaluate generalization across all three dimensions, systematically assess the effectiveness of augmentation along each dimension, and evaluate the training value of augmented data by fine-tuning generalist robot policies. In real-world experiments, UniAug matches or exceeds applying edits separately with one-sixth of the samples.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Intelligence, Demonstration Augmentation, World Editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_pfjiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WorldZoo to improve robot policy generalization by expanding the diversity of existing demonstrations without additional robot data collection. It converts robot demonstrations, UMI data, and egocentric human videos into aligned robot video–action pairs, then composes embodiment, interaction, and scene edits while updating observations and action labels together. UniAug provides a fixed sampling recipe for combining these edits under a limited augmentation budget. Experiments on RoboDojo, DexJoco, and three real-robot tasks show improved generalization across visual conditions, objects, and embodiments. On the real robot, UniAug achieves slightly higher task-average success than single-axis editing using 100 rather than 600 augmented samples per task. The framework also enables policies trained on converted human videos alone to perform a real-robot manipulation task.

Critical strengths
  • This is a practical framework with consistent improvements across two policies. WorldZoo combines embodiment, interaction, and scene editing while updating observations and action labels together. On RoboDojo, it improves average success from 39.3% to 55.4% for pi 0.5 and from 28.4% to 36.5% for LingBot-VLA. These comparisons use matched policy-training settings, supporting the usefulness of the generated data.

  • It showed strong retargeting results supported by real-robot experiments. With the remaining pipeline held fixed, the proposed retargeting achieves 45.3% success compared with 9.7% for fingertip retargeting on Rizon4 with ShadowHand. The real-robot experiments further demonstrate practical value: converted human videos alone support 54% and 48% success on Toy into box, without teleoperated demonstrations.

Critical weaknesses
  • The UniAug "allocation" contribution is weakly supported. The recipe is motivated by an allocation study on one task, while the real-robot comparison changes both edit composition and sample count with 100 UniAug samples versus 600 single-axis samples. Equal-budget comparisons and controls matching edit frequencies are missing. The task-average advantages are only 2-3% points, without reported training seed variation, so the generality of the sixfold sample reduction remains uncertain.
Questions for authors
  • UniAug reliability and composition prove could be explore deeper. Could you provide UniAug with single-axis editing at the same sample budget, and with a composition baseline using matched edit frequencies? Do the improvements remain consistent across training seeds?

  • How does performance change when WorldZoo’s object and trajectory edits retain the original action labels? This question could help better understanding how much of the improvement comes from maintaining observation–action consistency.

Reviewer_4A9COpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldZoo, a unified world-editing framework that leverages robot demonstrations and human egocentric videos to generate augmented data through composable edits - embodiment, interaction, and scene. The paper also introduces, UniAug, a sampling recipe that activates each edit with a fixed probability and composes the sampled edits in a fixed order. Experiments compare WorldZoo with MiniMax H3 on edit quality and fine-tune π0.5 and LingBot-VLA on the augmented data, evaluating on RoboDojo, DexJoco cross-embodiment transfer, and three real-world tasks.

Critical strengths
  • The proposed pipeline can be broadly used in diverse data sources (egocentric human videos, UMI, and teleoperated data), which is a substantial engineering contribution that would be useful for the community if released.
  • Strong results in both simulation benchmarks and real world experiments.
Critical weaknesses
  • The edit-quality metrics in Table 1 largely favor WorldZoo by construction. WorldZoo copies protected pixels from the source, so BG-LPIPS is near zero by design. Contact-PS rewards an unchanged contact region even for edits where that region should change.
  • Limited baselines. MiniMax H3 model is only generating video, and the corresponding actions are not recomputed. Robot-specific augmentation methods cited in the related work (e.g., RoboEngine, RoVi-Aug, DemoGen, Gaussian-splatting approaches) are not compared.
  • No analysis on per augmentation cost. It is unclear how much time / compute each augmentation takes.
  • There are many components that are not ablated. (e.g., depth calibration, contact correction, occlusion anchors, surface regrasp)
  • UniAug’s recipe may not transfer when the test distribution is unknown. The probabilities come from Table 4, whose eight test conditions are mostly viewpoint and scene shifts with none trajectory-related, and the ranking of edit types follows that mix. In real deployment the test distribution isn’t known in advance, so a recipe tuned to one evaluation, derived from a single task, may not serve as a reusable default.
Questions for authors

Check weaknesses.

Reviewer_i4nhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a data augmentation strategy for robotics manipulation to help preserve the diversity and robustness of policies. They point out that existing approaches using video models either just change the videos of one robot to another, or a human egovideo to a robot without changing the corresponding actions. Or the more 3D/robotics aware approaches require substantial setup/3D processing/simulation.

Their method first aligns different data sources, like egomotion data, UMI, teleop into a combined representation with both video and actions. Then they propose three types of edits

  1. Embodiement editing - You change the human hand or robot to another robot. The catch here is that they also account for the change in entire hand morphology and don't just align the fingertips. They also do some clever depth rendering so that the new robot hand is placed at the right depth in the edited video. This helps because the actions are now aligned with the edited videos.
  2. Interaction edting - Change the object which the robot is manipulating. They again take care of the object geometry to do surface regrasp and keep the actions aligned. They also edit the trajectory a bit in the demo, and induce some recovery behaviors along the way, while keeping the edited trajectory feasible.
  3. Scene editing - Mostly just visual scene/viewpoint/lighting/appearance editing which doesn't change the robot or the actions.

They also then propose UniAug, which is a way to determine with what probability should one sample all of these edits and do a search in simulation to propose a transferable recipe of augmentation allocations.

For evaluation, they show that training using their augmentation improves over using a SOTA video model for augmentation, because their video and actions are actually aligned. They analyze what perturbations break the models the most and how their method handles them better than the video model baseline. The pipeline enables cross-emodiement transfer, being able to generate videos for a new robot from a particular robot and transferring successfully to it, including generalising from human ego videos to a real world robot. They also show that UniAug beats vanilla augmentation strategies.

Overall I felt the paper is like a complex pipeline with many moving parts, and might be infeasible for people with limited budgets.

Critical strengths
  1. The cross embodiment transfer results seem interesting, and their approach takes care of the very important action-video alignment issue through multiple checks and balances.
  2. The analysis of which perturbations hurt the most and which don't really matter is insightful and helpful for the community for designing augmentations in general.
Critical weaknesses
  1. They only compare against one baseline which is also video model based. It would have been nice to see comparison with some of the more 3D aware/embodiment aware baselines, some of which they even cite.
  2. The pipeline has too many models being used and many moving parts. This raises the question of the practical utility of it from a compute perspective. Also an analysis of which parts are more expensive to run compared to others vs their utility in improving performance seems missing. This would be some sort of a cost aware UniAug.
  3. The approach looks for kinematic feasibility, but not dynamics. If the changed object has different friction or mass, then the approach might fail.
Questions for authors
  1. Where does the pipeline currently fail? The paper seem to be missing a limitations and future work section.
Reviewer_tJ9qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorldZoo augments robot demonstrations by editing the robot embodiment, object interaction, and scene, with corresponding updates to the action labels. The pipeline combines motion reconstruction, retargeting, object and trajectory editing, and rendering. UniAug applies several editing operators to each generated sample. The paper evaluates policy learning in simulation, transfer across robot embodiments, three real-robot tasks, and training from egocentric human videos. It reports improvements over the original demonstrations and a general video-generation baseline. On the real robot, 100 composed samples per task slightly outperform 600 samples produced with separate edits.

Critical strengths
  1. Keeping edited observations consistent with their action labels is the right problem to address. WorldZoo handles this across several types of edits, including changes to robot morphology and object geometry. The appendix gives concrete checks for contact, inverse-kinematics feasibility, joint limits, and collisions. These details make the approach useful beyond visual augmentation alone.
  2. Some of the controls give convincing evidence for individual components. In Table 3, full retargeting achieves 45.3% success on Rizon4-Shadow, compared with 9.7% for fingertip-only retargeting. Table 6 keeps the inferred action labels fixed and changes the human-hand observations to robot observations; success increases from 4% to 54% for pi0.5 and from 0% to 48% for LingBot-VLA. The latter comparison makes a good case for embodiment editing when learning from human video.
Critical weaknesses
  1. I am not yet convinced that the experiments establish the advantage of the UniAug recipe. Table 4 changes the allocation of an augmentation budget on one task; it does not directly test the probabilities used to combine operators within a sample. Table 5 then compares 100 composed samples with 600 single-axis samples. The average gains are only 2–3 percentage points, with no uncertainty across training runs. Each condition also includes 100 original demonstrations, so the training sampling ratio could affect the comparison. Equal-budget controls are needed to attribute the result to composition. The reported sample counts support the narrow claim of using one-sixth as many added samples, but leave the broader efficiency argument unresolved.
  2. The physical-feasibility claim goes beyond what the reported checks establish. Contact alignment and collision-free joint trajectories do not by themselves ensure a stable grasp after changing the hand or object geometry. Appendix A.3 describes useful geometric checks, but does not report how often edited trajectories succeed under independent physics replay. The policy results show that the data help; they do not establish that the edited interactions are physically valid by construction. Together with the missing composition controls, this leaves two central claims insufficiently supported for acceptance in the current form.
Questions for authors
  1. How are original and augmented demonstrations sampled during training? Does UniAug still outperform separate edits at the same augmentation budget and source-data sampling ratio?
  2. What evidence supports the chosen operator probabilities over uniform composition? Are the 2–3-point gains in Table 5 consistent across training seeds?
  3. Have the embodiment- and geometry-edited trajectories been re-executed in physics? If so, what are the success and rejection rates, and which failures remain after contact correction?
Reviewer_jC8uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a data augmentation framework for robot learning that generates diverse demonstrations by editing embodiment, interaction, and scene while maintaining consistency between observations and actions. The authors also propose UniAug, which composes different augmentation operators using fixed sampling probabilities. Experiments on simulation benchmarks and real robots show improved generalization to unseen objects, viewpoints, and environments, with UniAug achieving comparable or better performance than single-axis augmentation using fewer samples.

Critical strengths

The paper addresses an important problem in robot learning. Composing different augmentation types while preserving observation-action consistency is an interesting and useful direction.

I like that the framework supports different data sources, including robot demonstrations, UMI, and human egocentric videos. The unified formulation is practical and potentially useful for scaling robot learning.

The experiments are fairly comprehensive, covering simulation, cross-embodiment transfer, and real-world tasks. The results are promising, especially the improvements from UniAug and the ability to train robot policies using only human videos.

Critical weaknesses

The technical novelty is somewhat limited. Most components rely on existing methods for retargeting, pose estimation, and video editing. The main contribution seems to be integrating these components, and it is unclear which parts are technically new.

The computational cost is unclear. Since the pipeline combines several expensive models, it would be useful to understand whether generating compositional augmentations is actually more efficient than simpler alternatives.

Questions for authors

How sensitive is UniAug to the chosen sampling probabilities? Do the same probabilities work well across different tasks?

How do the authors evaluate the physical validity of generated demonstrations? What fraction of generated samples fail the feasibility checks?

Can the authors provide more ablations to show which components contribute most to downstream policy performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.