acceptodds
Under review as a conference paper at ICLR 2027

UMI-WM: Interactive World Modeling for Bimanual Manipulation from Wrist Cameras

Abstract

Universal Manipulation Interface (UMI) pairs wrist-camera videos with camera trajectories and gripper openings, providing scalable, robot-free data for learning how bimanual actions change a shared scene. Existing interactive world models, however, typically rely on text or single-camera control and struggle to follow each arm's actions while maintaining consistency between the two wrist views. We introduce UMI-WM, an action-conditioned bimanual world model that jointly generates both wrist views from observation history, per-arm relative camera poses, and gripper widths, with cross-view consistency, action following, and visual fidelity. UMI-WM injects each arm's actions into its own wrist stream, while sparse hub tokens exchange scene information between the two streams. Camera-aware geometry conditioning and camera-guided noise further support per-view motion control and cross-view geometric exchange. We adapt the model for block-causal generation and distill the 30-step Teacher into a Student with four denoising steps. We train UMI-WM on self-collected open-world and in-studio UMI recordings combined with public manipulation data. To evaluate action following, we use an inverse dynamics model (IDM) to recover relative camera poses and gripper widths from generated videos, and introduce Bimanual Action Fidelity (BAF) to measure how closely these recovered actions match the conditioning actions. UMI-WM achieves the best results on all five reference-video fidelity metrics and the best or competitive performance on VBench and WorldArena. The Teacher and Student outperform all compared baselines in action following, while the four-step Student generates paired wrist videos faster than the Teacher.

Primary area:
generative models
Keywords:
World Models, Action-Conditioned Video Generation, Bimanual Manipulation, Multi-View Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Kq7YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UMI-WM uses bimanual trajectories and gripper openings from UMI recordings as a unified action representation. It jointly predicts both wrist-camera videos through cross-view information exchange, camera geometry conditioning, and few-step distillation. The paper also introduces BAF, an inverse-dynamics-based metric for evaluating action following. Its main contribution lies in the system design and data integration for bimanual manipulation.

Critical strengths
  1. The problem addresses a practical need, and the data and control interfaces suit bimanual manipulation. Both wrist cameras move with the end effectors while observing a scene affected by both hands, making this more than two independent camera-control problems. The paper brings together public robot data, in-the-wild UMI recordings, and in-studio recordings registered in a shared coordinate frame, with approximately 20,000, 1,000, and 100 hours, respectively, as shown in Figure 2. This integration accommodates both independent local coordinate frames and calibrated shared frames.
  2. Video prediction and action recovery improve substantially over the strongest reported baseline. Compared with Gamma-World fine-tuned on UMI data, the Teacher reduces LPIPS from 0.397 to 0.332 and FVD from 331.25 to 232.53, while improving BAF from 52.93 to 76.36. Table 3 also reports position, rotation, and gripper errors for each wrist, allowing readers to assess the improvements beyond the aggregate score.
  3. The evaluation includes useful controls and an informative efficiency analysis. The authors use real videos and static frames as BAF references, explicitly separate IDM training and test episodes, and weight episodes equally when aggregating results. The Student reduces denoising from 30 steps to four, achieving a roughly 6.9× speedup while retaining good generation quality. Results with two, four, and eight steps also show the trade-off between generation speed and action error.
Critical weaknesses
  1. The claim of shared-scene modeling across both arms is not directly evaluated. The paper states that changes caused by either hand should appear consistently in both wrist views. However, the reference-video metrics in Table 1 largely assess each wrist separately, and Appendix C.1 explicitly states that FID and FVD are computed per wrist. BAF likewise measures each wrist’s camera pose and gripper opening. These metrics do not directly establish whether the two views depict a consistent object state or whether object motion during grasping, pushing, and handovers is correct.

  2. Evidence for interactive use and robot planning remains at the demonstration level. The Student achieves a clear 6.9× speedup, but its throughput is only 1.81 paired-wrist frames per second, whereas the videos represent motion at 30 Hz. At the reported throughput, generating a 189-frame clip covering approximately 6.3 seconds would take about 104 seconds. This estimate is not equivalent to the latency of the first feedback block, but it shows why relative speedup alone does not establish the responsiveness needed for continuous VR operation. The paper does not report the block length, first-block latency, waiting time after an action update, or end-to-end latency over successive interactions.

  3. Generalization to in-the-wild settings is insufficiently established. The paper emphasizes support for recordings in which the two wrist trajectories use independent local coordinate frames. Yet Appendix B.2 states that the geometry branches operate only on samples with valid calibration and shared-frame registration, and Stage 3 uses only in-studio data. The test set is described as 320 held-out UMI episodes, without specifying the proportions of in-studio and in-the-wild recordings, the availability of shared-frame registration, or whether scenes, objects, and tasks are held out from training.

Questions for authors
  1. Can the reported metrics assess whether the model captures a shared scene across both arms? How do the results support this claim?

  2. How are the contributions of the newly introduced components evaluated?

2 comments
AuthorsOpenReview ↗

Response to Kq7Y: Shared-scene Evaluation and Interaction Latency

We have uploaded a revised manuscript, with new and revised content highlighted in blue, and an updated supplementary video.

Thank you for distinguishing generation throughput from interaction latency and for asking which settings the model actually generalizes to. We have added direct cross-view tests, block-level and end-to-end timing, an explicit test-split description, and a full held-out in-the-wild evaluation.

  1. What supports the shared-scene claim?

Section 5.3 and Table 2 add measurements that compare the two generated views directly. The cross-view rotation loop connects both wrists at two consecutive timestamps. Consistent views should imply the same relative rotation along either path through that loop. On the same valid loops across the 320 episodes, Student and Teacher errors are 28.54∘28.54^\circ and 31.74∘31.74^\circ, compared with 39.94∘39.94^\circ for fine-tuned Gamma-World* and 3.52∘3.52^\circ for real video.

The object-state evaluation uses 54 episodes where the target appears in at least one initial wrist view. The Teacher obtains 0.38 on cross-view 3D localization and 0.35 on mask IoU; Student obtains 0.35 and 0.33; Gamma-World* obtains 0.31 and 0.32. At timestamps when both real views show the target, localization requires detections in both generated views and a distance of less than 0.25 camera baselines between the two estimated 3D centroids. Real video scores 0.57. We also test 16 episodes where an initially single-view object enters the other wrist's view; the Teacher reproduces its appearance in 0.43 of eligible cases. These measurements directly evaluate cross-view geometric agreement, object localization, and object persistence as the wrists move.

  1. Interaction is blockwise, with measured waiting time

Sections 5.2 and 5.6 now distinguish the video time represented by a block from the wall-clock time needed to generate and display it:

QuantityMeasurement
Block length12 frames per wrist; 0.4 s of video at 30 Hz
First block, including encoding and decoding6.56 s on one A100-80GB
Next block, generation and decoding6.54 s on one A100-80GB
Full 189-frame clip103.8 s
Action submission to headset display6.6--7 s over successive interactions

The end-to-end measurement starts when the operator submits an action at a block boundary and ends when the corresponding wrist views appear in the headset. The operator waits for these views before deciding the next action. Here, interactive refers to this action--generation--feedback cycle: the operator reviews each generated block before revising the next action.

Section 5.6 and Table 5 also add quantitative robotic utility: diffusion policies trained on 50 UMI-WM rollouts or 50 teleoperation demonstrations per task are each tested in 30 trials on three real-robot tasks. Success counts differ by at most two trials per task, with 2.25 versus 5.42 person-hours of demonstration preparation. These experiments evaluate the use of generated rollouts for training real-robot policies.

AuthorsOpenReview ↗

Response to Kq7Y: Held-out Wild Evaluation and Component Ablations

  1. Test composition and in-the-wild generalization

Section 5.1 now states that the original 320-episode benchmark is entirely in-studio, from three held-out device--date recording sessions. The scene is shared with training, and object instances and task families overlap. This benchmark evaluates new-session generalization within the studio.

Appendix C.1 and Table 6 add a separate wild evaluation. We identified 1,928 eligible episodes from 33 device--date sessions with paired wrist videos and per-arm action records. All 33 sessions were held out throughout Stages 1--3 and subsequent Teacher and Student training. We evaluate 320 sampled episodes, using one 189-frame window per episode and two generation seeds per model (640 rollouts each).

SourceBAF95% CI, episodes95% CI, sessions
Real wild video94.03[93.16, 94.81][92.29, 95.52]
UMI-WM Teacher92.92[91.92, 93.86][91.06, 94.47]
UMI-WM Student92.62[91.50, 93.67][90.66, 94.34]

The geometry branch is disabled because these wrist trajectories lack shared-frame registration. This differs from the in-studio configuration. Both models score within 1.5 BAF points of real wild video. Real video scores 1.11 BAF points above the Teacher and 1.41 above the Student. The 95% confidence intervals for both paired differences exclude zero when we resample episodes and include zero when we resample recording sessions. The Teacher scores 0.31 points above the Student on average. These measurements support action following on held-out wild inputs. Figure 10 in Appendix C.1 compares GT, Teacher, and Student wrist views from two held-out episodes: a kitchen countertop and a fabric-covered tabletop. Both wrists are shown at 1.47, 3.07, and 6.27 seconds, with the geometry branch disabled.

  1. Contributions of the components

Section 5.4 and Table 3 now include matched-budget hub, PRoPE, and CameraNoise removals at 150 updates. On 64 episodes, removing the hub lowers BAF by 2.94 points and removing PRoPE lowers it by 2.75. Removing CameraNoise raises BAF by 2.41 but worsens loop error by 15.45∘15.45^\circ, demonstrating why action recovery and cross-view geometry must both be evaluated. We also compare joint and separate-wrist generation on all 320 episodes with the same data and training schedule: joint generation improves BAF by 1.19 points (95% CI [0.76, 1.68]). Table 3B separately reports adapter-only geometry probes. Together, the fixed-budget ablations and the joint-versus-per-wrist comparison quantify the contributions of cross-view exchange and geometric conditioning.

Reviewer_wanyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UMI-WM, an action-conditioned bimanual world model. Its inputs are the observation history and, for each arm, the camera pose relative to the first frame (10D: translation + 6D rotation + gripper opening). From these, it generates both wrist-camera videos at once. The paper also trains an inverse dynamics model (IDM) that recovers actions from the generated videos, and uses it to define an action-following metric called BAF.

Critical strengths

Strength 1: The model clearly follows bimanual actions, and the gain comes from its design. UMI-WM reaches a BAF of 76.36. Gamma-World, fine-tuned on the same data with the same 20D action interface, reaches 52.93, and text-conditioned models score near the static-frame level (18.78). The 4-step Student keeps most of this (71.55) and runs 6.9× faster. Removing CamRay or PRoPE drops BAF to about 47, so the gain comes from the camera-aware design. This matters most because action-faithful dual-wrist generation is the paper's central claim.

Strength 2: The evaluation protocol is careful and could be reused by others. BAF is anchored by GT videos (97.72) and a static frame (18.78), so its scores are easy to interpret. The authors also excluded existing benchmarks that failed sanity checks, for example one that accepted a frozen-frame control. The same IDM detects deliberately corrupted recordings (AUROC 0.90–0.98). This makes the headline comparison credible and gives the community a useful tool. My confidence is somewhat limited because the IDM is trained in-domain.

Critical weaknesses

Weakness 1: The main evidence for action following uses an in-domain evaluator and does not show the model is usable for control. BAF depends on an IDM trained on the same UMI domain that UMI-WM is trained on, so it may favor UMI-WM over baselines whose controls had to be mapped into this format. The tolerance τ and decay rate λ are set by hand, with no sensitivity analysis. Even under this metric, the Teacher's position error (12–16 cm) is far from that of GT videos (about 4 cm). Physical robot execution is shown in only one qualitative demo, with no success rate. This matters most because action following is the paper's central claim. Beating weak baselines on a metric the authors designed themselves does not show that the model can support policy evaluation or control.

Weakness 2: The ablations cannot separate the paper's own contribution from the components it reuses. Most components come from prior work: Matrix-Game 3.5, Gamma-World's SRAE and Sparse Hub, CameraNoise, PRoPE, and CausVid-style distillation. The only ablation starts from a Stage-2 checkpoint with a frozen backbone, rather than a full retrain. There are no ablations for the Hub, CameraNoise, the individual data tiers, or joint versus separate generation. The main text also does not report how much training data was used. Without these, I cannot tell whether the gains over Gamma-World come from new design choices or from more and better-calibrated data, which weakens the paper's case for methodological novelty.

Questions for authors

In weakness

2 comments
AuthorsOpenReview ↗

Response to wany: BAF Reliability and Real-robot Results

We have uploaded a revised manuscript, with new and revised content highlighted in blue, and an updated supplementary video.

Thank you for raising these concerns. We added tests of BAF reliability, quantitative robot policy evaluations, and matched-budget ablations.

  1. BAF reliability, parameter sensitivity, and absolute errors

The updated main BAF comparison uses the shared 320-episode test set and the same frozen IDM (Section 5.5, Table 4). We map each baseline's available controls to the common per-arm action space before applying the same frozen IDM and BAF computation, as detailed in Appendix C.2. Selected rows of Table 4 are reproduced below. Position errors are in cm, rotation errors in degrees, and gripper errors in radians; L/R denote the wrists.

ModelPos. LRot. LGrip LPos. RRot. RGrip RBAF
GT video3.302.880.02564.523.260.022698.82
Static frame18.4454.040.049229.7156.730.074334.63
Gamma-World20.8357.260.083730.5656.310.100632.94
Gamma-World*12.2915.490.044416.0716.620.064372.20
UMI-WM Teacher8.435.690.018911.8710.530.024286.61
UMI-WM Student9.275.600.018212.6611.550.023385.15

Gamma-World uses public weights with mapped key/camera controls; Gamma-World* is fine-tuned on our UMI data with continuous per-arm controls.

Appendix C.2, Table 7 reports the full BAF sensitivity sweep:

ParameterTested values
Common multiplier on the three tolerances0.5, 0.75, 1, 1.5, 2
Decay parameter λ\lambda0.1, 0.25, 0.5, 1
Model rankingEight models: Teacher > Student > Gamma-World* > HY-WorldPlay in all 20 settings; the lower four change order in 10 settings

For additional validation, Appendix C.2 reports a SIFT/MAGSAC rotation check on real videos and geometric checks on generated videos. The generated-video checks use VGGT-Omega, which shares a visual backbone with the IDM. More directly, on 864 matched rollouts, BAF correlates with mask IoU (ρ=0.51\rho=0.51, 95% CI [0.39, 0.62]), 3D localization (0.430.43 [0.31, 0.53]), and at-gripper agreement (0.240.24 [0.14, 0.33]); all three associations are positive within six models (Table 8). These associations link BAF, which measures the recoverability of commanded wrist motion and gripper state, to the object-state measurements.

  1. Evidence beyond a qualitative robot demonstration

Section 5.6, Table 5 and Appendix C.3, Table 9 compare DP policies trained on real teleoperation, real UMI collection, and UMI-WM rollouts. The table below combines both comparisons. Tasks are placing a pen in a pen holder, opening a pencil pouch, and opening a pencil box. Success is successes/trials (%); time is operator person-hours, including hands-on work and supervision.

TaskData source50 demos: successTime (h)Equal time: demosSuccessTime (h)
Pen holderTeleoperation24/30 (80.0%)1.255024/30 (80.0%)1.25
Pen holderReal UMI24/30 (80.0%)0.897025/30 (83.3%)1.25
Pen holderUMI-WM23/30 (76.7%)0.5212025/30 (83.3%)1.25
Pencil pouchTeleoperation19/30 (63.3%)2.505019/30 (63.3%)2.50
Pencil pouchReal UMI19/30 (63.3%)1.777020/30 (66.7%)2.50
Pencil pouchUMI-WM17/30 (56.7%)1.0412019/30 (63.3%)2.50
Pencil boxTeleoperation22/30 (73.3%)1.675022/30 (73.3%)1.67
Pencil boxReal UMI22/30 (73.3%)1.187023/30 (76.7%)1.67
Pencil boxUMI-WM23/30 (76.7%)0.6912125/30 (83.3%)1.67

With 50 demonstrations, success counts differ by at most two trials per task; total operator time is 5.42, 3.84, and 2.25 person-hours for teleoperation, real UMI, and UMI-WM, respectively. At equal operator time, real UMI and UMI-WM provide more demonstrations and match or exceed the observed success rates of teleoperation-trained policies on all three tasks. Both comparisons use the same teleoperation reference.

AuthorsOpenReview ↗

Response to wany: Controlled Ablations and Training Data

  1. Hub, CameraNoise, and joint-generation ablations

Section 5.4 and Table 3A now compare variants trained for the same 150 updates and evaluated on the same 64 episodes with two seeds. Differences below are relative to the full configuration; intervals use paired episode bootstrap resampling.

RemovalΔ\DeltaBAF [95% CI]Δ\Deltaloop error (∘^\circ) [95% CI]
Sparse hub tokens−2.94-2.94 [−3.89-3.89, −2.02-2.02]+2.79+2.79 [−1.06-1.06, +6.89+6.89]
PRoPE−2.75-2.75 [−3.83-3.83, −1.72-1.72]−1.05-1.05 [−5.14-5.14, +2.88+2.88]
CameraNoise+2.41+2.41 [+1.11+1.11, +3.83+3.83]+15.45+15.45 [+9.26+9.26, +21.99+21.99]

Removing the hub increases throughput from 3.70 to 4.06 paired frames/s while lowering BAF. CameraNoise has a distinct trade-off: removing it improves BAF but worsens cross-view rotation agreement. We retain CameraNoise for its contribution to cross-view geometry.

We also evaluate the full and no-hub checkpoints on all 320 test episodes, holding the training data and schedule fixed. Each wrist receives its own actions and noise; removing the hub eliminates the path for information exchange between wrists. Joint generation increases BAF from 84.43 to 85.63 (paired difference +1.19+1.19, 95% CI [0.76, 1.68]). On the 54 object-evaluation episodes, 3D localization increases from 0.295 to 0.319 (paired difference 0.024 [0.000, 0.057]). The 320-episode comparison and the 64-episode table use different evaluation populations. Paired differences are computed before rounding the displayed means.

  1. Geometry probes and the roles of the data tiers

Table 3B now uses the matched full adapter configuration as its reference. With the Stage 2 backbone and action conditioner frozen, we train adapters for 1,000 updates. BAF is 48.91 for the full adapters, 46.79 without CamRay plus geometric hub routing, and 47.26 without PRoPE.

Section 3 now states both scale and purpose. Stage 1 uses all three data tiers for per-wrist action-conditioned prediction, drawing on approximately 20,000 hours of public manipulation data for visual diversity. Stage 2 jointly predicts both wrists using approximately 1,000 hours of open-world UMI and 100 hours of studio UMI. Stage 3 uses studio calibration to supply the inter-wrist relative transform for geometric supervision.

Table 3 evaluates component contributions under a matched 150-update training budget and through frozen-backbone geometry probes. We did not perform separate data-tier removal experiments or full-schedule retraining of each ablated model.

Reviewer_KL3ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UMI-WM, an action-conditioned world model that jointly generates two wrist-camera videos for bimanual manipulation. The method combines per-arm action conditioning, sparse hub attention for cross-view information exchange, and camera-aware geometric conditioning. It further introduces Bimanual Action Fidelity, which measures action following through an inverse dynamics model, and distills a 30-step teacher into a four-step causal student. Experiments on 320 held-out UMI episodes demonstrate improvements in video fidelity and action following over existing baselines.

Critical strengths
  • Meaningful bimanual world-modeling formulation: Jointly generating two independently controlled wrist views while modeling their shared environment is a relevant and challenging problem. The integration of per-arm actions with cross-view communication provides a practical framework for bimanual manipulation.

  • Strong empirical results and efficient generation: UMI-WM achieves strong reference-video fidelity and action-following results compared with Gamma-World. The four-step student also achieves a ~7× generation speedup over the teacher.

Critical weaknesses
  • Insufficient direct evaluation of cross-view and interaction consistency: A central contribution is maintaining a shared scene across independently controlled wrist views. However, the evaluation primarily relies on per-view video fidelity and IDM-based action recovery, which do not directly measure cross-view geometric agreement, object-state consistency, or coordinated human-object interactions. Thus, the extent to which the model maintains a coherent shared world remains unclear.

  • Limited validation of action fidelity and downstream robotic utility: Bimanual Action Fidelity relies on a learned IDM to recover actions from generated videos, making its reliability dependent on the evaluator's generalization to generated content. Although the authors report groundtruth and static-video controls, the resulting scores do not directly establish physical interaction correctness or successful robot execution. The robotic application is demonstrated qualitatively, without systematic task-success or closed-loop evaluation.

Questions for authors
  • Can the authors quantitatively evaluate whether the same objects and interaction events remain geometrically and temporally consistent across the two generated wrist views, particularly when the wrists move independently?

  • How well does IDM-recovered action fidelity correlate with actual interaction correctness? Can the authors validate BAF against independent geometric measurements or physical execution outcomes?

  • Can the authors provide quantitative robot execution results, such as task success rates or trajectory errors, to establish whether improvements in video fidelity and BAF translate into reliable manipulation?

2 comments
AuthorsOpenReview ↗

Response to KL3R: Cross-view Consistency and BAF Validation

We have uploaded a revised manuscript, with new and revised content highlighted in blue, and an updated supplementary video.

Thank you for identifying the need to evaluate shared object state and to connect action recovery to physical manipulation. We have added cross-view and object-state measurements, rollout-level correlations with BAF, and quantitative real-robot policy evaluations. Section and table numbers below refer to the revised manuscript.

  1. Direct evaluation of cross-view and object-state consistency

Section 5.3 and Table 2 now evaluate both geometric agreement and object state. For geometry, we estimate relative rotations along a loop through both wrists at two consecutive timestamps using frozen VGGT-Omega. Geometrically consistent views should give the same rotation along either path through the loop. Every model is evaluated on the same loops whose real-video error is below 10∘10^\circ, retaining 91% of test loops across the 320 episodes. The main comparisons from Table 2 are reproduced below:

ModelLoop error (degrees)3D localizationMask IoURecall
GT video3.520.571.001.00
UMI-WM Teacher31.740.380.350.62
UMI-WM Student28.540.350.330.46
Gamma-World39.430.200.190.41
Gamma-World*39.940.310.320.44

Gamma-World uses public weights with mapped key/camera controls; Gamma-World* is fine-tuned on our UMI data with continuous per-arm controls.

The Teacher's loop error is 31.74∘31.74^\circ, compared with 39.94∘39.94^\circ for fine-tuned Gamma-World*. Object-state measurements cover 54 episodes with a target visible in at least one initial wrist view. At timestamps when both real views show the target, 3D localization succeeds only if both generated views show it and their independently estimated object centroids are less than 0.25 camera baselines apart. In the 16 episodes where the object later enters the other wrist's view, the Teacher reproduces that appearance in 0.43 of eligible cases.

  1. BAF validation and its relationship to interaction correctness

Appendix C.2 and Table 8 pair BAF with object-state measurements from the same 864 rollouts across eight models and 54 episodes. The pooled Spearman correlations are:

Object-state measureρ\rho [95% episode-bootstrap CI]
3D localization0.43 [0.31, 0.53]
Mask IoU0.51 [0.39, 0.62]
At-gripper state agreement0.24 [0.14, 0.33]

All three associations are also positive within six of the eight models. At-gripper agreement measures whether the object appears at the gripper in agreement with real video. Across 5,120 rollouts, BAF also correlates with negative rotation loop error (ρ=0.22\rho=0.22, 95% CI [0.17, 0.27]).

We added two geometric checks in Appendix C.2. SIFT matches and MAGSAC on real wrist pairs give a median rotation error of 1.9∘1.9^\circ against the recorded inter-wrist rotation. This real-video check uses classical image correspondences independently of the learned IDM. On generated videos, VGGT-Omega gives temporal rotation errors of 3.7∘3.7^\circ, 3.8∘3.8^\circ, and 5.5∘5.5^\circ for Teacher, Student, and Gamma-World*, respectively. VGGT-Omega shares the IDM's visual backbone. Together, the geometric checks and rollout-level correlations connect action recovery with object and geometric consistency.

AuthorsOpenReview ↗

Response to KL3R: Real-robot Policy Evaluation and Data Efficiency

Section 5.6, Table 5 and Appendix C.3, Table 9 compare DP policies trained on real teleoperation, real UMI collection, and UMI-WM rollouts. The table below combines both comparisons. Tasks are placing a pen in a pen holder, opening a pencil pouch, and opening a pencil box. Success is successes/trials (%); time is operator person-hours, including hands-on work and supervision.

TaskData source50 demos: successTime (h)Equal time: demosSuccessTime (h)
Pen holderTeleoperation24/30 (80.0%)1.255024/30 (80.0%)1.25
Pen holderReal UMI24/30 (80.0%)0.897025/30 (83.3%)1.25
Pen holderUMI-WM23/30 (76.7%)0.5212025/30 (83.3%)1.25
Pencil pouchTeleoperation19/30 (63.3%)2.505019/30 (63.3%)2.50
Pencil pouchReal UMI19/30 (63.3%)1.777020/30 (66.7%)2.50
Pencil pouchUMI-WM17/30 (56.7%)1.0412019/30 (63.3%)2.50
Pencil boxTeleoperation22/30 (73.3%)1.675022/30 (73.3%)1.67
Pencil boxReal UMI22/30 (73.3%)1.187023/30 (76.7%)1.67
Pencil boxUMI-WM23/30 (76.7%)0.6912125/30 (83.3%)1.67

With 50 demonstrations, success counts differ by at most two trials per task; total operator time is 5.42, 3.84, and 2.25 person-hours for teleoperation, real UMI, and UMI-WM, respectively. At equal operator time, real UMI and UMI-WM provide more demonstrations and match or exceed the observed success rates of teleoperation-trained policies on all three tasks. Both comparisons use the same teleoperation reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.