- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In this paper, the authors propose CrossWAM, a world-action model that operates robustly across different robot embodiment morphologies and camera configurations. The proposed pipeline trains the world-action model based on video inpainting and gripper position overlays, which allows the WAM to predict consistent observations even when the target robot features a different embodiment or camera viewpoint, thereby enabling zero-shot cross-embodiment manipulation. Furthermore, to overcome the open-loop limitation of existing models that execute a predicted action chunk to completion without feedback, the authors introduce an Refiner. By training with this pipeline and utilizing it as a reference for closed-loop trajectory refinement during inference, the authors demonstrate improved success rates across three different embodiments compared to VLA and WAM baselines.
Critical strengths
-
The refiner compares the ideal imagined future image without the robot predicted by the WAM and the actual observation image of the target robot taken in real-time. By analyzing the discrepancy between the traces and object positions in both images, it calculates a translation residual, pulling the trajectory of the remaining unexecuted actions toward the correct contact point.
-
It demonstrates that the proposed method operates successfully not only on the source embodiment (Aloha-AgileX) but also on unseen embodiments such as ARX and Franka.
-
Although utilizing Wan2.2 for both the WAM and the refiner could potentially lead to a significant increase in latency, the authors show that their approach introduces no additional latency compared to Fast-WAM, thanks to asynchronous prediction and the parallelization of generation and execution.
Critical weaknesses
-
The refiner predicts offset given RGB images. Although the authors claim robustness to camera viewpoint changes, this mapping would completely collapse under significant camera movements (e.g., if the camera is shifted by 180 degrees to the opposite side.
-
The proposed method may not generalize beyond simple grippers. For instance, the approach of inpainting the robot body and leaving only the end-effector trace is unlikely to be effective for dexterous hands.
-
Despite the claim of a zero-shot setting, accurately overlaying the trace onto the observation image during deployment strictly requires the target robot's forward kinematics and precise camera calibration (extrinsics and intrinsics).
Questions for authors
-
WAM is trained to take the current observation containing the robot as input and predict a future outcome where the robot is erased. Given this setup, how can the model be inherently robust to the visual appearance of the specific embodiment in the input image? Furthermore, would applying robot inpainting to the input observation as well lead to further improvements in cross-embodiment performance?
-
Figure 3 illustrates four refinement steps occurring for a single video denoising step. If the underlying mechanism is essentially a 1-to-1 mapping, how exactly does this structure translate into a 1-to-4 latency ratio? Could you clarify the details of this execution timing?
4 comments
Response to Reviewer TNu4 (2/2)
A4. Robot appearance in the input. (Q1)
The model is not invariant to the robot in its input. What the agnostic target changes is what the model has to encode: with raw targets, predicting the future means rendering the source arm, so the features must carry its appearance; with agnostic targets they need not. Fig. 5 and Sec. 4.4 quantify this: the ratio of between-embodiment to within-embodiment feature distance falls from 0.923 to 0.316. Applying robot inpainting to the input observation may further improve cross-embodiment transfer. Mirage (Chen et al., 2024) follows this idea: it inpaints the target robot in the input and renders the training robot in its place. In the table below, Mirage improves over the baseline but remains below CrossWAM. Input inpainting also has two drawbacks: its result depends on the quality of the inpainting model, which can blur objects near the arm, especially when the arm is in contact with them; and it adds inference latency at every control step.
Mirage built on the same pretrained backbone as CrossWAM (Table 5 of the revision).
| Method | Aloha D | R | L | ARX D | R | L | Franka D | R | L |
|---|---|---|---|---|---|---|---|---|---|
| Clean scenes | |||||||||
| Fast-WAM (Pretrained) | 92.1 | 68.3 | 59.5 | 64.2 | 43.4 | 44.2 | 27.2 | 20.7 | 18.9 |
| + Mirage | 92.1 | 68.3 | 59.5 | 70.3 | 49.6 | 45.7 | 30.8 | 23.5 | 20.7 |
| CrossWAM | 92.0 | 81.4 | 68.3 | 73.4 | 62.2 | 53.7 | 32.5 | 26.9 | 28.1 |
| Randomized scenes | |||||||||
| Fast-WAM (Pretrained) | 91.7 | 68.8 | 61.6 | 61.9 | 36.4 | 42.0 | 25.9 | 17.0 | 16.2 |
| + Mirage | 91.7 | 68.8 | 61.6 | 65.2 | 40.7 | 47.9 | 27.1 | 19.0 | 19.8 |
| CrossWAM | 91.5 | 81.1 | 72.0 | 71.2 | 60.7 | 56.7 | 33.0 | 26.4 | 28.5 |
A5. Timing in Fig. 3. (Q2)
Thank you for pointing this out. The 1-to-4 ratio arises because, following existing WAMs such as Fast-WAM, CrossWAM predicts a 32-step action chunk together with a video that is temporally down-sampled by 4, i.e., 8 frames; each imagined frame therefore serves one refiner call, once every 4 actions. "Video Denoising I" and "II" in Fig. 3 are not single denoising steps but the two halves of the asynchronous schedule: I ends when the near-future frames (the first 4 of the 8) are clean, and II when the far-future frames are clean. In the revision, Sec. 3.2 defines the down-sampling factor (, ), and Eqs. (5) and (6) in Sec. 3.3 are rewritten accordingly: the -th imagined frame is compared with the observation taken after the first actions, and the resulting offset corrects the remaining actions .
Thank you to the authors for the response and additional experiments. I will keep my score.
Thank you for reading our response and for your positive assessment of the paper.
Response to Reviewer TNu4 (1/2)
We thank the reviewer for the positive review and the precise questions; we address each point below.
A1. Large camera changes. (W1)
We agree that an RGB-based refiner will fail under very large changes, such as moving the camera to the opposite side; our experiments cover moderate changes (up to ±15 cm with pitch compensation in simulation, and 15 cm with a 13° tilt on the real robot). Refining with 3D information, such as depth or point clouds, would indeed reduce the dependence on the camera pose. We have added this discussion to the limitations (App. A).
A2. Beyond simple grippers. (W2)
Agreed. The trace is defined by a set of end-effector keypoints, so it could include the fingertips of a hand, but we have not tested this. The revision states this in the limitations (App. A): "The embodiment-agnostic target assumes a parallel gripper. For dexterous hands, the trace would have to encode finger keypoints and contact patterns, which we have not tested. The premise that object motion is independent of the executing robot is also weaker for non-prehensile manipulation, where the shape of the end-effector determines the contact."
A3. Kinematics, calibration and "zero-shot". (W3)
We use "zero-shot" in the same sense as previous cross-embodiment methods such as Mirage (Chen et al., 2024b): no target demonstrations and no parameter updates. All three also need the target robot model and the camera calibration, and every baseline that runs on a new arm already needs its kinematic model for IK. Moreover, for a robot with a fixed setup, the camera needs to be calibrated only once.
To measure how much CrossWAM depends on accurate calibration, we perturb the camera extrinsics used to draw the gripper trace on the observation, with a fixed offset of 1 cm in translation and 1° in rotation in a random direction for each episode (ARX, raised and lowered cameras, randomized scenes). This perturbation is larger than the error of typical calibration: standard hand-eye calibration reaches about 1 mm and 0.3° (Li et al., 2023 (https://arxiv.org/pdf/2311.01335)).
With this error, CrossWAM drops by 4.6 / 3.8 points under the raised / lowered cameras and remains 19.7 / 10.9 points above Fast-WAM (Pretrained). A calibration error shifts the trace drawn on the observation, and this error therefore degrades the closed loop only slightly rather than breaking it. Moreover, for a robot with a fixed setup, the camera needs to be calibrated only once.