- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a framework to train a video world model (VWM) for robot arms. Traditional solution is to train a VWM to directly predict the future video frames with joint trajectory and the first frame as input, for a specific setup. The proposed method, DOPPEL, augment the VWM with the rendered robot video from recovered camera configs, URDF, and a fixed renderer, and train a unified VWM (a fully fine-tuned Wan 2.1 I2V-14B) to predict future video frames across three datasets. This is shown to improve on top of the baseline method in video quality on three benchmarks.
Critical strengths
- A controlled, consistent gain from render conditioning at scale, with one model across three arms. These are observed across early training, val, and OOD cases.
- A reusable data asset. Target-free calibration yields extrinsics for 14.8k Bridge and 17.6k RoboMIND demos.
Critical weaknesses
- Conceptual novelty is marginal, and similar ideas already exploited in some missing reference.
- BridgeV2W (arXiv 2602.03793) renders end-effector actions (IK → URDF FK) as camera-aligned masks for a pretrained video model, and reports richer ablatioins with donwsteam application like planning.
- Kinema4D (2603.16669) conditions on URDF-driven 4D robot pointmaps.
- Ablation missing Compared to the previous methods (e.g., BridgeV2W), there is no render-only, mask-render or skeleton-render variant on the same backbone, to better understand the benefit of Gaussian renders relative to other choices.
- No downstream application support Since the paper aims to address a practical problem, it should be evaluated in the downstream tasks to justify how much of the improvement in video quality can translate to these end tasks the VWM is for (like planning, etc).
- Overclaim statement on "embodiment-agnostic" I feel that the "embodiment-agnostic" contribution is an overclaim. All three robots are seen single-arm parallel-jaw arms. More diverse empirical evidence is needed to justify the general statement.
Questions for authors
- How does DOPPEL compare with BridgeV2W and Kinema4D?
- Can you show any downstream utility, such as policy-ranking correlation with real success rates or planning success?
9 comments
Part 2
2. Controlled Ablations of Geometric Representations
Addresses: Critical Weakness 2 (Missing Render-Only, Mask, and Skeleton Ablations).
We provide additional controlled comparisons of action-only, skeleton-only, skeleton + actions, rendering-only, and rendering + actions using the same backbone, training data, and training recipe on DROID (Table B). Dense renderings outperform sparse 2D skeleton conditioning, improving validation PSNR from 22.99 to 24.01 dB and OOD PSNR from 20.46 to 21.18 dB.
Table B. Validation and OOD performance on DROID across different conditioning representations.
| Method | Input | Training data | Val PSNR ↑ | Val SSIM ↑ | Val LPIPS ↓ | Val SCS ↑ | OOD PSNR ↑ | OOD SSIM ↑ | OOD LPIPS ↓ | OOD SCS ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| Action-only | Actions | DROID | 22.228 | 0.832 | 0.085 | 0.921 | 19.767 | 0.775 | 0.135 | 0.869 |
| OSCAR (fine-tuned) | Skeleton | DROID | 20.427 | 0.751 | 0.135 | 0.903 | 18.619 | 0.705 | 0.174 | 0.854 |
| Ours, 2D skeleton | Skeleton | DROID | 22.990 | 0.842 | 0.077 | 0.931 | 20.459 | 0.788 | 0.121 | 0.890 |
| Ours, 2D skeleton + actions | Skeleton + actions | DROID | 23.092 | 0.844 | 0.075 | 0.932 | 20.311 | 0.783 | 0.125 | 0.888 |
| Ours, rendering only | Render | DROID | 24.009 | 0.858 | 0.069 | 0.937 | 21.184 | 0.800 | 0.113 | 0.897 |
| DOPPEL (single) | Render + actions | DROID | 24.077 | 0.860 | 0.068 | 0.938 | 21.151 | 0.799 | 0.115 | 0.898 |
| DOPPEL (multi) | Render + actions | 3 robot datasets | 24.220 | 0.861 | 0.067 | 0.940 | 21.147 | 0.800 | 0.113 | 0.897 |
| DOPPEL-8 | Render + actions | All 8 datasets | 24.819 | 0.870 | 0.062 | 0.943 | 21.460 | 0.806 | 0.108 | 0.903 |
All DROID-only variants use matched training data, optimization settings, and training exposure. Skeleton and rendering variants share the same spatial token-wise conditioning architecture. OSCAR is a system-level baseline with a different backbone and training setup. DOPPEL (multi) and DOPPEL-8 are trained jointly on three and eight datasets, respectively, and are included as additional multi-embodiment comparisons rather than matched-data ablations. Results are evaluated on 200 clips per split. Bold indicates the best result in each column.
3DGS rendering and geometric representations. Our 3DGS renderers are trained exclusively on synthetic URDF/MJCF mesh renderings, meaning that their underlying robot geometry is derived from the same mesh models. Binary robot masks capture silhouette information already present in our dense renderings, while discarding articulated surface appearance and other visual details. Thus, our representation incorporates mesh-derived geometry and robot silhouette information within a unified dense conditioning signal.
3. Downstream Policy Evaluation and Real-World Agreement
Addresses: Critical Weakness 3 (Lack of Downstream Application Support) and Question 2 (Policy Evaluation and Correlation with Real-World Success).
To evaluate DOPPEL's downstream utility beyond video reconstruction, we additionally evaluate trajectories generated by robot policies and compare DOPPEL-predicted task outcomes with real-world executions.
We evaluate two policies, MolmoAct2 and π0.5, in both zero-shot and fine-tuned settings across two manipulation tasks: (1) aligning a plate to the center and (2) sliding a box. Each policy configuration is evaluated over five trials per task, with success measured in both real-world execution and DOPPEL predictions.
Table C. Real-world and DOPPEL task success (successful trials / 5 per task).
| Policy | Setting | Align plate (Real / DOPPEL) | Slide box (Real / DOPPEL) | Overall (Real / DOPPEL) |
|---|---|---|---|---|
| MolmoAct2 | Zero-shot | 0 / 0 | 0 / 0 | 0 / 0 |
| MolmoAct2 | Fine-tuned | 4 / 4 | 5 / 5 | 9 / 9 |
| π0.5 | Zero-shot | 0 / 0 | 0 / 0 | 0 / 0 |
| π0.5 | Fine-tuned | 4 / 5 | 5 / 4 | 9 / 9 |
Each task has five trials per policy setting. Overall success is reported out of ten trials.
DOPPEL matches the aggregate real-world success counts across all four policy configurations, distinguishing unsuccessful zero-shot policies from their successful fine-tuned counterparts. These results provide initial evidence that DOPPEL's predicted outcomes reflect real-world policy performance and extend our evaluation beyond open-loop prediction on recorded demonstrations to policy-generated trajectories.
Part 3
4. Joint Learning Across Heterogeneous Embodiments
Addresses: Critical Weakness 4 (Overclaim of Embodiment-Agnostic Learning).
To directly address the reviewer's concern that our original evaluation covered only three single-arm, parallel-jaw robots, we expand DOPPEL to eight heterogeneous embodiment datasets, including bimanual YAM arms, a humanoid with dexterous hands, human hands, and human full-body motion. All eight datasets are jointly trained using one shared video backbone and conditioning adapter, without embodiment-specific heads or labels. Table D reports results from the same shared DOPPEL-8 checkpoint across all eight datasets, demonstrating its applicability to substantially different robot and human morphologies.
Table D. Performance of the shared DOPPEL-8 model across eight embodiment datasets. Each cell reports validation / test performance.
| Dataset | Embodiment | PSNR ↑ | SSIM ↑ | LPIPS ↓ | SCS ↑ |
|---|---|---|---|---|---|
| DROID | Franka | 24.82 / 21.46 | 0.87 / 0.81 | 0.06 / 0.11 | 0.94 / 0.90 |
| Bridge | WidowX | 26.54 / 22.24 | 0.87 / 0.78 | 0.05 / 0.10 | 0.95 / 0.90 |
| RoboMIND | UR5e + Robotiq | 31.30 / 30.66 | 0.93 / 0.92 | 0.04 / 0.04 | 0.94 / 0.94 |
| ABC | Bimanual YAM | 24.65 / 25.22 | 0.85 / 0.86 | 0.10 / 0.09 | 0.92 / 0.93 |
| T-Rex | Vega-1 + Sharpa hands | 28.33 / 28.35 | 0.87 / 0.87 | 0.05 / 0.05 | 0.96 / 0.96 |
| Ego-Exo4D | Human body | 27.94 / 27.61 | 0.90 / 0.89 | 0.05 / 0.04 | 0.97 / 0.97 |
| GigaHands | Human hands | 25.62 / 25.78 | 0.89 / 0.90 | 0.05 / 0.05 | 0.94 / 0.94 |
| OakInk2 | Human upper body | 29.01 / 29.09 | 0.92 / 0.92 | 0.04 / 0.03 | 0.97 / 0.97 |
Each dataset is evaluated on 200 clips per split. Validation uses held-out trajectories, while test uses dataset-specific held-out distribution shifts.
Benefits of multi-embodiment training. We additionally compare DOPPEL-8 against our original three-embodiment shared model on identical evaluation clips (Table E), with approximately matched exposure to the original robot datasets. Joint training with five additional heterogeneous datasets improves average PSNR across DROID, Bridge, and RoboMIND by +0.34 dB on validation and +0.11 dB on OOD, with improvements in five of the six evaluation settings. These results provide evidence that training across diverse embodiments can improve prediction performance on existing robots while maintaining a single shared backbone and conditioning adapter.
Table E. Three-dataset versus eight-dataset joint training. PSNR in dB.
| Dataset | Val (3) | Val (8) | Δ Val | OOD (3) | OOD (8) | Δ OOD |
|---|---|---|---|---|---|---|
| DROID | 24.22 | 24.82 | +0.60 | 21.15 | 21.46 | +0.31 |
| Bridge | 26.46 | 26.54 | +0.09 | 22.44 | 22.24 | −0.20 |
| RoboMIND | 30.96 | 31.30 | +0.34 | 30.44 | 30.66 | +0.22 |
| Average | 27.21 | 27.56 | +0.34 | 24.67 | 24.79 | +0.11 |
DOPPEL-3 uses the epoch-519 checkpoint reported in the original submission, while DOPPEL-8 is trained jointly on eight embodiment datasets. Average scores are unweighted across the three datasets. Differences are computed from unrounded values.
These experiments extend our empirical evidence beyond single-arm robots and demonstrate that heterogeneous embodiments can be jointly modeled using a shared conditioning framework. We clarify that "embodiment-agnostic" refers to the shared conditioning interface, backbone, and adapter, rather than zero-shot generalization to embodiments absent from training.
Author Response Part 1
We thank the reviewer for their constructive feedback and for recognizing the consistent improvements from render conditioning and the value of our camera calibration pipeline. We address the reviewer's concerns below.
1. Novelty and Comparison with Prior Geometry-Conditioned World Models
Addresses: Critical Weakness 1 (Limited Conceptual Novelty) and Question 1 (Comparison with BridgeV2W and Kinema4D).
We thank the reviewer for highlighting these related approaches. We have expanded the related work discussion in the revised manuscript to include BridgeV2W, Kinema4D, and other geometry-conditioned world models, and clarified the positioning of our contributions.
Distinction from prior methods. While these approaches share the broader idea of geometry-conditioned video generation, they differ in their representations and predictive formulations:
Empirical comparison with prior methods. We additionally evaluate BridgeV2W, Kinema4D, and EnerVerse-AC under our evaluation protocol (Table A). DOPPEL achieves stronger prediction performance across PSNR, SSIM, and LPIPS.
Table A. Comparison with prior geometry-conditioned video generation methods on DROID OOD.
DOPPEL-8 is jointly trained on eight embodiment datasets, whereas DOPPEL (single) is trained only on DROID. All methods are evaluated on the same DROID OOD split.
Clarification of our contribution. We do not claim to introduce robot-render conditioning or AdaLN as new techniques. Rather, our contribution lies in a unified dense embodiment-conditioning framework that combines explicit articulated robot motion with a shared video world model, enabling joint dynamics learning across heterogeneous embodiments. We have revised the manuscript to clarify this contribution rather than positioning robot rendering itself as the central novelty.