- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
UMI-WM uses bimanual trajectories and gripper openings from UMI recordings as a unified action representation. It jointly predicts both wrist-camera videos through cross-view information exchange, camera geometry conditioning, and few-step distillation. The paper also introduces BAF, an inverse-dynamics-based metric for evaluating action following. Its main contribution lies in the system design and data integration for bimanual manipulation.
Critical strengths
- The problem addresses a practical need, and the data and control interfaces suit bimanual manipulation. Both wrist cameras move with the end effectors while observing a scene affected by both hands, making this more than two independent camera-control problems. The paper brings together public robot data, in-the-wild UMI recordings, and in-studio recordings registered in a shared coordinate frame, with approximately 20,000, 1,000, and 100 hours, respectively, as shown in Figure 2. This integration accommodates both independent local coordinate frames and calibrated shared frames.
- Video prediction and action recovery improve substantially over the strongest reported baseline. Compared with Gamma-World fine-tuned on UMI data, the Teacher reduces LPIPS from 0.397 to 0.332 and FVD from 331.25 to 232.53, while improving BAF from 52.93 to 76.36. Table 3 also reports position, rotation, and gripper errors for each wrist, allowing readers to assess the improvements beyond the aggregate score.
- The evaluation includes useful controls and an informative efficiency analysis. The authors use real videos and static frames as BAF references, explicitly separate IDM training and test episodes, and weight episodes equally when aggregating results. The Student reduces denoising from 30 steps to four, achieving a roughly 6.9× speedup while retaining good generation quality. Results with two, four, and eight steps also show the trade-off between generation speed and action error.
Critical weaknesses
-
The claim of shared-scene modeling across both arms is not directly evaluated. The paper states that changes caused by either hand should appear consistently in both wrist views. However, the reference-video metrics in Table 1 largely assess each wrist separately, and Appendix C.1 explicitly states that FID and FVD are computed per wrist. BAF likewise measures each wrist’s camera pose and gripper opening. These metrics do not directly establish whether the two views depict a consistent object state or whether object motion during grasping, pushing, and handovers is correct.
-
Evidence for interactive use and robot planning remains at the demonstration level. The Student achieves a clear 6.9× speedup, but its throughput is only 1.81 paired-wrist frames per second, whereas the videos represent motion at 30 Hz. At the reported throughput, generating a 189-frame clip covering approximately 6.3 seconds would take about 104 seconds. This estimate is not equivalent to the latency of the first feedback block, but it shows why relative speedup alone does not establish the responsiveness needed for continuous VR operation. The paper does not report the block length, first-block latency, waiting time after an action update, or end-to-end latency over successive interactions.
-
Generalization to in-the-wild settings is insufficiently established. The paper emphasizes support for recordings in which the two wrist trajectories use independent local coordinate frames. Yet Appendix B.2 states that the geometry branches operate only on samples with valid calibration and shared-frame registration, and Stage 3 uses only in-studio data. The test set is described as 320 held-out UMI episodes, without specifying the proportions of in-studio and in-the-wild recordings, the availability of shared-frame registration, or whether scenes, objects, and tasks are held out from training.
Questions for authors
-
Can the reported metrics assess whether the model captures a shared scene across both arms? How do the results support this claim?
-
How are the contributions of the newly introduced components evaluated?
2 comments
Response to Kq7Y: Held-out Wild Evaluation and Component Ablations
- Test composition and in-the-wild generalization
Section 5.1 now states that the original 320-episode benchmark is entirely in-studio, from three held-out device--date recording sessions. The scene is shared with training, and object instances and task families overlap. This benchmark evaluates new-session generalization within the studio.
Appendix C.1 and Table 6 add a separate wild evaluation. We identified 1,928 eligible episodes from 33 device--date sessions with paired wrist videos and per-arm action records. All 33 sessions were held out throughout Stages 1--3 and subsequent Teacher and Student training. We evaluate 320 sampled episodes, using one 189-frame window per episode and two generation seeds per model (640 rollouts each).
| Source | BAF | 95% CI, episodes | 95% CI, sessions |
|---|---|---|---|
| Real wild video | 94.03 | [93.16, 94.81] | [92.29, 95.52] |
| UMI-WM Teacher | 92.92 | [91.92, 93.86] | [91.06, 94.47] |
| UMI-WM Student | 92.62 | [91.50, 93.67] | [90.66, 94.34] |
The geometry branch is disabled because these wrist trajectories lack shared-frame registration. This differs from the in-studio configuration. Both models score within 1.5 BAF points of real wild video. Real video scores 1.11 BAF points above the Teacher and 1.41 above the Student. The 95% confidence intervals for both paired differences exclude zero when we resample episodes and include zero when we resample recording sessions. The Teacher scores 0.31 points above the Student on average. These measurements support action following on held-out wild inputs. Figure 10 in Appendix C.1 compares GT, Teacher, and Student wrist views from two held-out episodes: a kitchen countertop and a fabric-covered tabletop. Both wrists are shown at 1.47, 3.07, and 6.27 seconds, with the geometry branch disabled.
- Contributions of the components
Section 5.4 and Table 3 now include matched-budget hub, PRoPE, and CameraNoise removals at 150 updates. On 64 episodes, removing the hub lowers BAF by 2.94 points and removing PRoPE lowers it by 2.75. Removing CameraNoise raises BAF by 2.41 but worsens loop error by , demonstrating why action recovery and cross-view geometry must both be evaluated. We also compare joint and separate-wrist generation on all 320 episodes with the same data and training schedule: joint generation improves BAF by 1.19 points (95% CI [0.76, 1.68]). Table 3B separately reports adapter-only geometry probes. Together, the fixed-budget ablations and the joint-versus-per-wrist comparison quantify the contributions of cross-view exchange and geometric conditioning.
Response to Kq7Y: Shared-scene Evaluation and Interaction Latency
We have uploaded a revised manuscript, with new and revised content highlighted in blue, and an updated supplementary video.
Thank you for distinguishing generation throughput from interaction latency and for asking which settings the model actually generalizes to. We have added direct cross-view tests, block-level and end-to-end timing, an explicit test-split description, and a full held-out in-the-wild evaluation.
Section 5.3 and Table 2 add measurements that compare the two generated views directly. The cross-view rotation loop connects both wrists at two consecutive timestamps. Consistent views should imply the same relative rotation along either path through that loop. On the same valid loops across the 320 episodes, Student and Teacher errors are 28.54∘ and 31.74∘, compared with 39.94∘ for fine-tuned Gamma-World* and 3.52∘ for real video.
The object-state evaluation uses 54 episodes where the target appears in at least one initial wrist view. The Teacher obtains 0.38 on cross-view 3D localization and 0.35 on mask IoU; Student obtains 0.35 and 0.33; Gamma-World* obtains 0.31 and 0.32. At timestamps when both real views show the target, localization requires detections in both generated views and a distance of less than 0.25 camera baselines between the two estimated 3D centroids. Real video scores 0.57. We also test 16 episodes where an initially single-view object enters the other wrist's view; the Teacher reproduces its appearance in 0.43 of eligible cases. These measurements directly evaluate cross-view geometric agreement, object localization, and object persistence as the wrists move.
Sections 5.2 and 5.6 now distinguish the video time represented by a block from the wall-clock time needed to generate and display it:
The end-to-end measurement starts when the operator submits an action at a block boundary and ends when the corresponding wrist views appear in the headset. The operator waits for these views before deciding the next action. Here, interactive refers to this action--generation--feedback cycle: the operator reviews each generated block before revising the next action.
Section 5.6 and Table 5 also add quantitative robotic utility: diffusion policies trained on 50 UMI-WM rollouts or 50 teleoperation demonstrations per task are each tested in 30 trials on three real-robot tasks. Success counts differ by at most two trials per task, with 2.25 versus 5.42 person-hours of demonstration preparation. These experiments evaluate the use of generated rollouts for training real-robot policies.