acceptodds
Under review as a conference paper at ICLR 2027

Render the Robot, Generate the World: One Video World Model Across Embodiments

14 views

Abstract

Predicting the consequences of an agent's actions requires modeling both the embodiment's motion and the environment's response. Existing video world models typically learn both jointly from numerical action vectors, even though known robot geometry and kinematics already provide much of the information needed to specify the intended robot motion. A general-purpose world model should capture these dynamics across multiple embodiments within a single model. Our key idea is to render the robot along a proposed joint-space trajectory and condition the video model directly on these future robot configurations. To this end, we introduce DOPPEL, an embodiment-agnostic video world model for learning a dynamics model shared across embodiments. DOPPEL jointly conditions on camera-aligned robot renderings and numerical action trajectories. Using forward kinematics and differentiable rendering, we explicitly provide the model with the robot's intended motion. This allows the shared model to focus its capacity on object motion and interaction-driven changes in the environment. Representing robot motion in image space also provides a common conditioning interface across embodiments with different geometries and action spaces. To enable this approach at scale, we recover missing camera extrinsics from offline robot datasets and align the resulting renderings with recorded observations. Using these aligned renderings, we train a single shared video backbone and conditioning adapter jointly on DROID, RoboMIND, and Bridge, spanning Franka, UR5e, and WidowX embodiments. Across three embodiments, DOPPEL improves PSNR over action-only baselines by 1.31 dB on unseen trajectories and 1.36 dB under OOD evaluation, while predicting more consistent object motion.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, World Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_UEkaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a framework to train a video world model (VWM) for robot arms. Traditional solution is to train a VWM to directly predict the future video frames with joint trajectory and the first frame as input, for a specific setup. The proposed method, DOPPEL, augment the VWM with the rendered robot video from recovered camera configs, URDF, and a fixed renderer, and train a unified VWM (a fully fine-tuned Wan 2.1 I2V-14B) to predict future video frames across three datasets. This is shown to improve on top of the baseline method in video quality on three benchmarks.

Critical strengths
  • A controlled, consistent gain from render conditioning at scale, with one model across three arms. These are observed across early training, val, and OOD cases.
  • A reusable data asset. Target-free calibration yields extrinsics for 14.8k Bridge and 17.6k RoboMIND demos.
Critical weaknesses
  • Conceptual novelty is marginal, and similar ideas already exploited in some missing reference.
    • BridgeV2W (arXiv 2602.03793) renders end-effector actions (IK → URDF FK) as camera-aligned masks for a pretrained video model, and reports richer ablatioins with donwsteam application like planning.
    • Kinema4D (2603.16669) conditions on URDF-driven 4D robot pointmaps.
  • Ablation missing Compared to the previous methods (e.g., BridgeV2W), there is no render-only, mask-render or skeleton-render variant on the same backbone, to better understand the benefit of Gaussian renders relative to other choices.
  • No downstream application support Since the paper aims to address a practical problem, it should be evaluated in the downstream tasks to justify how much of the improvement in video quality can translate to these end tasks the VWM is for (like planning, etc).
  • Overclaim statement on "embodiment-agnostic" I feel that the "embodiment-agnostic" contribution is an overclaim. All three robots are seen single-arm parallel-jaw arms. More diverse empirical evidence is needed to justify the general statement.
Questions for authors
  • How does DOPPEL compare with BridgeV2W and Kinema4D?
  • Can you show any downstream utility, such as policy-ranking correlation with real success rates or planning success?
9 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for their constructive feedback and for recognizing the consistent improvements from render conditioning and the value of our camera calibration pipeline. We address the reviewer's concerns below.

1. Novelty and Comparison with Prior Geometry-Conditioned World Models

Addresses: Critical Weakness 1 (Limited Conceptual Novelty) and Question 1 (Comparison with BridgeV2W and Kinema4D).

We thank the reviewer for highlighting these related approaches. We have expanded the related work discussion in the revised manuscript to include BridgeV2W, Kinema4D, and other geometry-conditioned world models, and clarified the positioning of our contributions.

Distinction from prior methods. While these approaches share the broader idea of geometry-conditioned video generation, they differ in their representations and predictive formulations:

  • BridgeV2W uses binary robot masks rendered from URDF models with ControlNet-style conditioning. In contrast, DOPPEL combines dense camera-aligned robot renderings with numerical joint configurations through a shared render-action conditioning framework.
  • Kinema4D conditions on spatiotemporal robot pointmaps and jointly generates RGB and pointmap sequences, whereas DOPPEL predicts future RGB observations directly from an initial image and a specified robot trajectory.

Empirical comparison with prior methods. We additionally evaluate BridgeV2W, Kinema4D, and EnerVerse-AC under our evaluation protocol (Table A). DOPPEL achieves stronger prediction performance across PSNR, SSIM, and LPIPS.

Table A. Comparison with prior geometry-conditioned video generation methods on DROID OOD.

MethodPSNR ↑SSIM ↑LPIPS ↓
DOPPEL-8 (ours)21.4600.8060.108
DOPPEL (single, ours)21.1510.7990.115
BridgeV2W20.5300.7940.131
EnerVerse-AC18.8700.7390.162
Kinema4D (Released Checkpoint)18.5100.7730.159

DOPPEL-8 is jointly trained on eight embodiment datasets, whereas DOPPEL (single) is trained only on DROID. All methods are evaluated on the same DROID OOD split.

Clarification of our contribution. We do not claim to introduce robot-render conditioning or AdaLN as new techniques. Rather, our contribution lies in a unified dense embodiment-conditioning framework that combines explicit articulated robot motion with a shared video world model, enabling joint dynamics learning across heterogeneous embodiments. We have revised the manuscript to clarify this contribution rather than positioning robot rendering itself as the central novelty.

2. Controlled Ablations of Geometric Representations

Addresses: Critical Weakness 2 (Missing Render-Only, Mask, and Skeleton Ablations).

We provide additional controlled comparisons of action-only, skeleton-only, skeleton + actions, rendering-only, and rendering + actions using the same backbone, training data, and training recipe on DROID (Table B). Dense renderings outperform sparse 2D skeleton conditioning, improving validation PSNR from 22.99 to 24.01 dB and OOD PSNR from 20.46 to 21.18 dB.

Table B. Validation and OOD performance on DROID across different conditioning representations.

MethodInputTraining dataVal PSNR ↑Val SSIM ↑Val LPIPS ↓Val SCS ↑OOD PSNR ↑OOD SSIM ↑OOD LPIPS ↓OOD SCS ↑
Action-onlyActionsDROID22.2280.8320.0850.92119.7670.7750.1350.869
OSCAR (fine-tuned)SkeletonDROID20.4270.7510.1350.90318.6190.7050.1740.854
Ours, 2D skeletonSkeletonDROID22.9900.8420.0770.93120.4590.7880.1210.890
Ours, 2D skeleton + actionsSkeleton + actionsDROID23.0920.8440.0750.93220.3110.7830.1250.888
Ours, rendering onlyRenderDROID24.0090.8580.0690.93721.1840.8000.1130.897
DOPPEL (single)Render + actionsDROID24.0770.8600.0680.93821.1510.7990.1150.898
DOPPEL (multi)Render + actions3 robot datasets24.2200.8610.0670.94021.1470.8000.1130.897
DOPPEL-8Render + actionsAll 8 datasets24.8190.8700.0620.94321.4600.8060.1080.903

All DROID-only variants use matched training data, optimization settings, and training exposure. Skeleton and rendering variants share the same spatial token-wise conditioning architecture. OSCAR is a system-level baseline with a different backbone and training setup. DOPPEL (multi) and DOPPEL-8 are trained jointly on three and eight datasets, respectively, and are included as additional multi-embodiment comparisons rather than matched-data ablations. Results are evaluated on 200 clips per split. Bold indicates the best result in each column.

3DGS rendering and geometric representations. Our 3DGS renderers are trained exclusively on synthetic URDF/MJCF mesh renderings, meaning that their underlying robot geometry is derived from the same mesh models. Binary robot masks capture silhouette information already present in our dense renderings, while discarding articulated surface appearance and other visual details. Thus, our representation incorporates mesh-derived geometry and robot silhouette information within a unified dense conditioning signal.

3. Downstream Policy Evaluation and Real-World Agreement

Addresses: Critical Weakness 3 (Lack of Downstream Application Support) and Question 2 (Policy Evaluation and Correlation with Real-World Success).

To evaluate DOPPEL's downstream utility beyond video reconstruction, we additionally evaluate trajectories generated by robot policies and compare DOPPEL-predicted task outcomes with real-world executions.

We evaluate two policies, MolmoAct2 and π0.5, in both zero-shot and fine-tuned settings across two manipulation tasks: (1) aligning a plate to the center and (2) sliding a box. Each policy configuration is evaluated over five trials per task, with success measured in both real-world execution and DOPPEL predictions.

Table C. Real-world and DOPPEL task success (successful trials / 5 per task).

PolicySettingAlign plate (Real / DOPPEL)Slide box (Real / DOPPEL)Overall (Real / DOPPEL)
MolmoAct2Zero-shot0 / 00 / 00 / 0
MolmoAct2Fine-tuned4 / 45 / 59 / 9
π0.5Zero-shot0 / 00 / 00 / 0
π0.5Fine-tuned4 / 55 / 49 / 9

Each task has five trials per policy setting. Overall success is reported out of ten trials.

DOPPEL matches the aggregate real-world success counts across all four policy configurations, distinguishing unsuccessful zero-shot policies from their successful fine-tuned counterparts. These results provide initial evidence that DOPPEL's predicted outcomes reflect real-world policy performance and extend our evaluation beyond open-loop prediction on recorded demonstrations to policy-generated trajectories.

4. Joint Learning Across Heterogeneous Embodiments

Addresses: Critical Weakness 4 (Overclaim of Embodiment-Agnostic Learning).

To directly address the reviewer's concern that our original evaluation covered only three single-arm, parallel-jaw robots, we expand DOPPEL to eight heterogeneous embodiment datasets, including bimanual YAM arms, a humanoid with dexterous hands, human hands, and human full-body motion. All eight datasets are jointly trained using one shared video backbone and conditioning adapter, without embodiment-specific heads or labels. Table D reports results from the same shared DOPPEL-8 checkpoint across all eight datasets, demonstrating its applicability to substantially different robot and human morphologies.

Table D. Performance of the shared DOPPEL-8 model across eight embodiment datasets. Each cell reports validation / test performance.

DatasetEmbodimentPSNR ↑SSIM ↑LPIPS ↓SCS ↑
DROIDFranka24.82 / 21.460.87 / 0.810.06 / 0.110.94 / 0.90
BridgeWidowX26.54 / 22.240.87 / 0.780.05 / 0.100.95 / 0.90
RoboMINDUR5e + Robotiq31.30 / 30.660.93 / 0.920.04 / 0.040.94 / 0.94
ABCBimanual YAM24.65 / 25.220.85 / 0.860.10 / 0.090.92 / 0.93
T-RexVega-1 + Sharpa hands28.33 / 28.350.87 / 0.870.05 / 0.050.96 / 0.96
Ego-Exo4DHuman body27.94 / 27.610.90 / 0.890.05 / 0.040.97 / 0.97
GigaHandsHuman hands25.62 / 25.780.89 / 0.900.05 / 0.050.94 / 0.94
OakInk2Human upper body29.01 / 29.090.92 / 0.920.04 / 0.030.97 / 0.97

Each dataset is evaluated on 200 clips per split. Validation uses held-out trajectories, while test uses dataset-specific held-out distribution shifts.

Benefits of multi-embodiment training. We additionally compare DOPPEL-8 against our original three-embodiment shared model on identical evaluation clips (Table E), with approximately matched exposure to the original robot datasets. Joint training with five additional heterogeneous datasets improves average PSNR across DROID, Bridge, and RoboMIND by +0.34 dB on validation and +0.11 dB on OOD, with improvements in five of the six evaluation settings. These results provide evidence that training across diverse embodiments can improve prediction performance on existing robots while maintaining a single shared backbone and conditioning adapter.

Table E. Three-dataset versus eight-dataset joint training. PSNR in dB.

DatasetVal (3)Val (8)Δ ValOOD (3)OOD (8)Δ OOD
DROID24.2224.82+0.6021.1521.46+0.31
Bridge26.4626.54+0.0922.4422.24−0.20
RoboMIND30.9631.30+0.3430.4430.66+0.22
Average27.2127.56+0.3424.6724.79+0.11

DOPPEL-3 uses the epoch-519 checkpoint reported in the original submission, while DOPPEL-8 is trained jointly on eight embodiment datasets. Average scores are unweighted across the three datasets. Differences are computed from unrounded values.

These experiments extend our empirical evidence beyond single-arm robots and demonstrate that heterogeneous embodiments can be jointly modeled using a shared conditioning framework. We clarify that "embodiment-agnostic" refers to the shared conditioning interface, backbone, and adapter, rather than zero-shot generalization to embodiments absent from training.

Reviewer_hPRQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

DOPPEL conditions a video world model on camera-aligned renders of a proposed robot joint/gripper configuration trajectory, together with the numerical configurations. A frozen VAE and a convolution/attention adapter encode the renders, fuse them with action tokens, and inject token-specific conditioning through the timestep-modulation pathway of Wan 2.1. Missing camera extrinsics are recovered through differentiable rendering. A single backbone and adapter are trained on DROID, Bridge, and RoboMIND, covering Franka, WidowX, and UR5e.

Evaluation covers held-out trajectories and shifts to a new laboratory, a new kitchen, and unseen objects/tasks. Relative to action-only experts, Table 2 reports average PSNR improvements of approximately 1.31 dB on validation and 1.36 dB on OOD tests. Region-specific results show improvements on moving objects as well as the robot. Tables 10–11 additionally compare shared models with matched batch size and optimization settings, supporting an advantage in training draws. All evaluated embodiments occur in training; unseen-robot transfer and downstream control benefits are not established.

Critical strengths
  1. Explicit robot motion is a well-motivated inductive bias with evidence beyond robot reconstruction. Known kinematics and camera geometry remove part of the mapping that a numerical-action model must learn, while image-space renders provide a common interface across different configuration spaces. In Table 7, Bridge OOD moving-object PSNR improves from 21.497 to 24.963 dB and SCS from 0.776 to 0.842; moving-object metrics also improve on the other datasets. The lift example in Figure 4 is consistent with these measurements. These results make the interaction-prediction claim more convincing than whole-frame fidelity alone would.

  2. The shared-model evaluation spans three embodiments and complementary distribution shifts. Importantly, Tables 10–11 compare shared action-only and shared DOPPEL models under the same batch-64 recipe, addressing a major confound in comparisons against separately trained experts. The calibration pipeline and planned release of recovered camera parameters are also interesting.

Critical weaknesses
  1. The contribution of the proposed representation is not sufficiently isolated. The action-only adapter has 4.2M parameters versus 30.5M for the render–action adapter, with different spatiotemporal processing and fusion. There is no same-backbone, comparable-capacity comparison using skeletons/silhouettes or simple URDF renders, nor a render-only ablation. The experiments support the complete system over the specified action baseline, but do not separate dense geometry, learned robot appearance, numerical-action complementarity, and adapter design. OSCAR cannot resolve this because its backbone, pretraining, and temporal window differ; the authors appropriately describe it as a system-level baseline. Moreover, the similar single- and multi-embodiment DOPPEL results on DROID support maintaining performance with shared weights more directly than strong positive cross-embodiment transfer.

  2. Calibration dependence and the exact test-time information protocol need clarification. Section 4.1 estimates camera pose from an RGB/configuration trajectory, and A.9 describes one calibration per session, but it is unclear whether OOD calibration trajectories are independent of evaluated episodes. Using later images from the evaluated episode would introduce an additional test-time observation assumption, although it would not amount to directly conditioning on future object images. Quantitative calibration error and sensitivity to pose perturbations are also absent. Finally, the 63.8% reduction concerns training draws, rather than total compute or data-acquisition cost: renderer training, calibration, caching, and inference overhead are not included. The demonstrated benefit should remain scoped to offline 16-frame prediction with available calibration and configuration trajectories.

Questions for authors
  1. Protocol clarification: For Bridge and RoboMIND OOD cameras, which episodes and frames are used for calibration? Are future RGB frames from the evaluated episode used? Are test episodes excluded from training the robot renderer? Please specify the data separation; if evaluation-episode observations are used, results with an independent calibration trajectory would materially clarify the OOD claim.

  2. Contribution attribution: Are same-Wan-backbone results available for skeleton/silhouette or simple URDF-render conditioning plus actions, using comparable adapter capacity, and for render-only conditioning? At least one such control would help determine what the dense rendering and dual-input fusion contribute beyond architecture differences in OSCAR.

  3. Robustness and cost: Are moving-object PSNR/SCS under modest camera-pose perturbations and preprocessing/inference costs available? These affect the practical interpretation of the training-draw advantage. This is not a request for a complete closed-loop system; absent such measurements, please explicitly scope the efficiency claim and characterize the calibration assumptions.

3 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for their thoughtful and detailed assessment of DOPPEL, and for recognizing the benefits of explicit robot motion conditioning, the improvements in moving-object prediction, and the value of our shared multi-embodiment framework and camera calibration pipeline. We address the reviewer's concerns below.

1. Controlled Ablations and Contribution of Geometric Conditioning

Addresses: Critical Weakness 1 (Representation, Adapter Capacity, and Cross-Embodiment Transfer) and Question 2 (Controlled Geometric Representation and Render-Only Ablations).

We provide additional controlled ablations comparing action-only, skeleton-only, skeleton + actions, rendering-only, and rendering + actions under matched DROID training settings (Table A).

Representation and adapter capacity. The difference in adapter size (4.2M for action-only versus 30.5M for DOPPEL) reflects the different structures of the conditioning inputs. The action-only baseline processes low-dimensional joint-configuration vectors following EgoWM, whereas DOPPEL processes dense spatiotemporal robot renderings encoded by the frozen Wan VAE, requiring additional spatial processing and fusion. While the action-only comparison does not independently control adapter capacity, our skeleton and rendering variants use the same spatial token-wise conditioning architecture, allowing us to more directly evaluate the effect of geometric representation.

To better isolate representation quality from adapter architecture, we introduce skeleton-based conditioning using the same token-wise adapter architecture and training recipe as dense rendering. Under these controlled settings, dense rendering improves over skeleton conditioning by +1.02 dB validation PSNR and +0.73 dB OOD PSNR. Comparing skeleton + actions against DOPPEL (single), we additionally observe +0.99 dB validation and +0.84 dB OOD PSNR. These results provide more direct evidence for the benefits of dense geometric information beyond the differences in adapter capacity between DOPPEL and the action-only baseline. We also include rendering-only conditioning to isolate the contribution of rendering without numerical actions.

Table A. Validation and OOD performance on DROID across different conditioning representations.

MethodInputTraining dataVal PSNR ↑Val SSIM ↑Val LPIPS ↓Val SCS ↑OOD PSNR ↑OOD SSIM ↑OOD LPIPS ↓OOD SCS ↑
Action-onlyActionsDROID22.2280.8320.0850.92119.7670.7750.1350.869
OSCAR (fine-tuned)SkeletonDROID20.4270.7510.1350.90318.6190.7050.1740.854
Ours, 2D skeletonSkeletonDROID22.9900.8420.0770.93120.4590.7880.1210.890
Ours, 2D skeleton + actionsSkeleton + actionsDROID23.0920.8440.0750.93220.3110.7830.1250.888
Ours, rendering onlyRenderDROID24.0090.8580.0690.93721.1840.8000.1130.897
DOPPEL (single)Render + actionsDROID24.0770.8600.0680.93821.1510.7990.1150.898
DOPPEL (multi)Render + actions3 robot datasets24.2200.8610.0670.94021.1470.8000.1130.897
DOPPEL-8Render + actionsAll 8 datasets24.8190.8700.0620.94321.4600.8060.1080.903

All DROID-only variants use matched training data, optimization settings, and training exposure. Skeleton and rendering variants share the same spatial token-wise conditioning architecture. OSCAR is a system-level baseline with a different backbone and training setup. DOPPEL (multi) and DOPPEL-8 are trained jointly on three and eight datasets, respectively, and are included as additional multi-embodiment comparisons rather than matched-data ablations. Results are evaluated on 200 clips per split. Bold indicates the best result in each column.

Cross-embodiment learning. We additionally train DOPPEL-8, a single shared model across eight heterogeneous datasets spanning robot arms, bimanual manipulation, dexterous humanoid hands, and human hands and bodies, without embodiment-specific heads or labels.

Compared with our original three-embodiment shared model (24.22/21.15 dB validation/OOD PSNR), DOPPEL-8 achieves 24.82/21.46 dB on DROID, improving performance by +0.60/+0.31 dB, without adding additional DROID demonstrations.

We agree that maintaining performance with shared weights and positive cross-embodiment transfer are distinct claims. Our expanded results demonstrate that a single model can jointly learn across heterogeneous embodiments, with additional embodiment data improving prediction performance on DROID. We do not claim zero-shot generalization to unseen robot embodiments.

2. Camera Calibration Protocol, Accuracy, and Robustness

Addresses: Critical Weakness 2 (Calibration Dependence and Test-Time Information Protocol), Question 1 (Calibration Data Separation), and Question 3 (Calibration Accuracy and Robustness).

Calibration protocol and data separation. We clarify that camera calibration is a one-time procedure for any fixed robot–camera setup, including previously unseen OOD setups. Calibration can even be performed using an independent video of the robot moving through visible configurations and the corresponding joint states, without requiring task demonstrations or observations of the evaluated scenes. Our differentiable renderer estimates the camera-to-robot transformation by aligning rendered and observed robot motion. The calibration objective uses only robot appearance and motion, without supervision from object interactions or scene dynamics.

Once a camera is calibrated, the recovered extrinsics are reused for all subsequent trajectories recorded under the same setup. Thus, even for OOD evaluation, DOPPEL requires only the initial RGB observation, the specified future joint configurations, and the previously calibrated camera parameters.

Furthermore, our robot renderers are trained exclusively on synthetic URDF/MJCF renderings and never use real RGB images from any training or evaluation split.

Calibration accuracy and coverage. DROID provides camera extrinsics directly, while we recover 56 camera poses for Bridge and 49 for RoboMIND. We evaluate calibration quality by comparing rendered robot-arm masks against independently obtained SAM 3 segmentation masks, sampling five frames per clip and averaging results per session. We exclude four RoboMIND sessions and four Bridge environments due to insufficient calibration quality.

Table B. Accuracy and coverage of recovered camera extrinsics.

DatasetSessions / Camera PosesMean IoUMedian IoUIoU p10–p90Recall / Precision
Bridge (ours)56 sessions0.580.600.43–0.680.75 / 0.71
RoboMIND (ours)49 sessions0.730.750.66–0.800.87 / 0.82
DROID (provided extrinsics)60 poses across 12 labs0.610.620.42–0.770.77 / 0.71

We will release the recovered camera extrinsics and calibration pipeline to facilitate reproduction and application to additional datasets.

3. Robustness, Computational Cost, and Scope

Addresses: Critical Weakness 2 (Calibration Dependence and Efficiency Interpretation) and Question 3 (Robustness and Computational Cost).

Training efficiency and computational cost. We clarify that the reported 63.8% reduction refers specifically to training-example draws in the matched shared-model comparison, where both models use identical batch sizes and optimization schedules, rather than total computational cost. Renderer training, camera calibration, rendering preprocessing, and inference overhead are not included in this comparison. However, renderer training is a one-time cost per robot embodiment, while camera calibration is performed once per fixed robot-camera setup. These costs can be amortized across trajectories. We will clarify that our efficiency claim concerns training draws, not total GPU-hours or wall-clock time.

Inference latency. DOPPEL uses 50 Euler sampling steps without classifier-free guidance. On a single NVIDIA A100 (80 GB), rendering a 17-frame robot trajectory takes approximately 0.9 seconds, while generating the corresponding 16 future frames takes approximately 26 seconds, measured over 3,200 evaluation clips using DOPPEL-8.

Calibration robustness and scope. DOPPEL assumes accurate camera alignment to construct its conditioning renderings. We acknowledge that calibration errors and changes in camera viewpoint can affect prediction quality and will explicitly describe these assumptions and limitations in the revised manuscript.

Thank you to the authors for their efforts. Their responses have addressed my concerns well, and I have decided to raise my score.

Reviewer_Rts2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DOPPEL, a video world model that incorporates explicit robot geometry and kinematics into action-conditioned video prediction. Instead of relying solely on numerical joint configurations, the proposed method renders future robot trajectories using known kinematic models and camera parameters, and combines the resulting visual representations with numerical actions through a conditioning adapter. The model is trained jointly across three robot embodiments (Franka, UR5e, and WidowX), using a shared video backbone and adapter. Experiments on DROID, RoboMIND, and Bridge demonstrate improvements in video reconstruction metrics, object motion consistency, and training sample efficiency compared with action-only baselines.

Critical strengths
  1. The paper is well motivated and clearly presented. Incorporating known robot geometry and kinematics into video prediction is a reasonable approach to reducing the burden of learning robot motion directly from numerical actions. The overall framework is intuitive, and the implementation is described in sufficient detail.
  2. The experiments demonstrate improvements over action-only baselines across three robot embodiments and multiple evaluation settings. In addition to whole-frame reconstruction metrics, the authors evaluate robot and moving-object regions separately, providing evidence that the improvements extend beyond robot appearance reconstruction.
  3. The paper demonstrates that a single video backbone and conditioning adapter can be trained across different robot embodiments using camera-aligned renderings. The reported improvements in training sample efficiency and the recovered camera calibration corpus are useful practical contributions.
Critical weaknesses
  1. The technical novelty is limited, as incorporating robot geometry and kinematics into video world models has already been explored through skeletons, 3D point flow, and rendered robot motion. The proposed framework largely integrates existing techniques, including differentiable rendering, cross-attention fusion, and timestep-modulation conditioning, without introducing a substantially new modeling principle. The paper would benefit from a clearer demonstration of its technical contributions beyond existing geometry-conditioned approaches.
  2. This paper lacks controlled comparisons with alternative geometric priors. While DOPPEL outperforms the action-only baseline, existing work has already established the benefits of geometric conditioning. The comparison with OSCAR is confounded by differences in backbone, pretraining, and optimization. Without comparisons against simpler representations (e.g., robot masks or skeletons) under the same setup, or ablations isolating rendering-only conditioning and action fusion, it remains unclear whether full robot rendering provides unique benefits over existing geometric approaches.
  3. It is claimed that explicit robot motion conditioning helps the model learn environment dynamics, but the experiments do not fully substantiate this claim. The improvements may primarily result from accurate spatial and temporal anchors that facilitate video reconstruction rather than better interaction modeling. Although moving-object PSNR and SCS improve, these metrics cannot distinguish visual alignment from genuine interaction understanding. Evaluations involving different physical outcomes, such as successful versus failed grasps, would provide stronger evidence.
  4. The benefits of cross-embodiment knowledge transfer are not clearly established, as joint training across robots is already feasible with the shared action-only baseline. Table 2 shows nearly identical OOD performance between single- and multi-embodiment DOPPEL on DROID, while comparable results for other robots are missing. Evaluating whether additional embodiments improve target-robot performance or sample efficiency, particularly with limited target data, would better substantiate the claimed cross-embodiment learning benefits.
  5. The OOD evaluations focus on unseen scenes, objects, and tasks rather than unseen robot embodiments, leaving cross-embodiment generalization insufficiently validated.
Questions for authors
  1. What is the central technical contribution of DOPPEL beyond incorporating known robot geometry into an action-conditioned video generator? How does the proposed approach differ fundamentally from existing geometry-conditioned or rendering-conditioned world modeling methods?
  2. Can the authors provide controlled comparisons between full robot renderings and simpler geometric representations, such as robot masks or projected skeletons, using the same backbone, training data, and computational budget?
  3. Can the authors evaluate whether DOPPEL correctly distinguishes different physical interaction outcomes under small changes in robot trajectories or object configurations, particularly when these changes determine whether contact or grasping succeeds?
  4. Is there evidence that data from one embodiment improves prediction performance for another embodiment?
14 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for the thoughtful evaluation and for recognizing the motivation behind explicit robot geometry conditioning, the consistent improvements over action-only baselines, and the importance of evaluating moving-object dynamics beyond robot reconstruction. We also appreciate the acknowledgment of our shared multi-embodiment framework, training efficiency, and recovered camera calibration corpus. We address the concerns and questions below.

1. Novelty and Technical Contributions Relative to Prior Work

Addresses: Critical Weakness 1 (Limited Technical Novelty) and Question 1 (Central Technical Contribution and Differences from Existing Geometry-Conditioned World Models).

We agree that geometric conditioning, differentiable rendering, cross-attention, and AdaLN have been explored individually in prior work. Our contribution is not introducing these components independently, but developing a unified, camera-aligned dense embodiment-conditioning framework that enables a shared video world model to predict robot-conditioned scene dynamics across heterogeneous embodiments.

Unlike approaches that condition on sparse skeletons, end-effector trajectories, or binary robot masks, DOPPEL combines dense articulated robot renderings with numerical configurations through a shared conditioning adapter. Importantly, this representation enables joint dynamics learning across heterogeneous embodiment datasets using a single model. As shown in Table A, training DOPPEL jointly on eight datasets improves prediction performance on DROID compared to training on DROID alone, without adding any additional DROID demonstrations. We additionally enable training on real-world datasets lacking camera extrinsics through our calibration pipeline.

Empirical comparison with prior methods. We provide direct comparisons with BridgeV2W, Kinema4D, and EnerVerse-AC under our evaluation protocol (Table A), where DOPPEL achieves better prediction performance across PSNR, SSIM, and LPIPS.

Table A. Comparison with prior geometry-conditioned video generation methods on DROID OOD.

MethodPSNR ↑SSIM ↑LPIPS ↓
DOPPEL-8 (ours)21.4600.8060.108
DOPPEL (single, ours)21.1510.7990.115
BridgeV2W20.5300.7940.131
EnerVerse-AC18.8700.7390.162
Kinema4D (Released Checkpoint)18.5100.7730.159

DOPPEL-8 is jointly trained on eight embodiment datasets, whereas DOPPEL (single) is trained only on DROID. All methods are evaluated on the same DROID OOD split.

We further provide controlled geometric-conditioning ablations (Section 2), evaluate interaction outcomes under counterfactual robot trajectories (Section 3), and examine cross-embodiment learning in greater detail (Section 4). Our shared model also achieves matched prediction performance with 63.8% fewer training-example draws than the action-only baseline under the compared training setup.

We have expanded the related work and revised the introduction to clarify that our contribution lies in the shared dense embodiment-conditioning formulation, unified dynamics learning across heterogeneous real-world datasets, and its demonstrated empirical effectiveness, rather than claiming novelty in robot rendering or AdaLN individually.

2. Controlled Ablations of Geometric Representations

Addresses: Critical Weakness 2 (Lack of Controlled Geometric Comparisons) and Question 2 (Comparisons with Simpler Geometric Representations).

Geometric representation ablations. We provide controlled comparisons of action-only, skeleton-only, skeleton + actions, rendering-only, and rendering + actions using the same backbone, training data, and training recipe on DROID (Table B). Unlike the system-level comparison with OSCAR, our skeleton and rendering variants use the same spatial token-wise conditioning architecture, while the action-only baseline shares the same video backbone and training recipe. Dense renderings outperform sparse 2D skeleton conditioning by +1.02 dB validation and +0.73 dB OOD PSNR, demonstrating the benefits of dense geometry under matched training exposure.

Table B. Validation and OOD performance on DROID across different conditioning representations.

MethodInputTraining dataVal PSNR ↑Val SSIM ↑Val LPIPS ↓Val SCS ↑OOD PSNR ↑OOD SSIM ↑OOD LPIPS ↓OOD SCS ↑
Action-onlyActionsDROID22.2280.8320.0850.92119.7670.7750.1350.869
OSCAR (fine-tuned)SkeletonDROID20.4270.7510.1350.90318.6190.7050.1740.854
Ours, 2D skeletonSkeletonDROID22.9900.8420.0770.93120.4590.7880.1210.890
Ours, 2D skeleton + actionsSkeleton + actionsDROID23.0920.8440.0750.93220.3110.7830.1250.888
Ours, rendering onlyRenderDROID24.0090.8580.0690.93721.1840.8000.1130.897
DOPPEL (single)Render + actionsDROID24.0770.8600.0680.93821.1510.7990.1150.898
DOPPEL (multi)Render + actions3 robot datasets24.2200.8610.0670.94021.1470.8000.1130.897
DOPPEL-8Render + actionsAll 8 datasets24.8190.8700.0620.94321.4600.8060.1080.903

All DROID-only variants use matched training data, optimization settings, and training exposure. Skeleton and rendering variants share the same spatial token-wise conditioning architecture. OSCAR is a system-level baseline with a different backbone and training setup. DOPPEL (multi) and DOPPEL-8 are trained jointly on three and eight datasets, respectively, and are included as additional multi-embodiment comparisons rather than matched-data ablations. Results are evaluated on 200 clips per split. Bold indicates the best result in each column.

3DGS rendering and geometric representations. We clarify that our 3DGS renderers are trained exclusively on synthetic URDF/MJCF mesh renderings, meaning that the underlying robot geometry is derived from the same mesh models. Similarly, binary robot masks capture silhouette information already present in our dense renderings, while discarding articulated surface appearance and other visual details. Thus, our representation incorporates mesh-derived geometry and robot silhouette information within a unified dense conditioning signal.

3. Counterfactual Interaction Prediction and Physical Outcomes

Addresses: Critical Weakness 3 (Insufficient Evidence of Interaction Dynamics) and Question 3 (Sensitivity to Trajectory Changes and Physical Interaction Outcomes).

To directly address this concern, we introduce controlled counterfactual evaluations on three manipulation tasks: placing a duck into a pot, a bouquet into a vase, and a pepper onto a plate. The results are available in the Counterfactual Trajectories section of our anonymous project page (direct link: https://submit15330.github.io/#counterfactuals).

For each task, we keep the initial frame, starting robot pose, gripper commands, model checkpoint, and sampling seed fixed, while perturbing only the end-effector trajectory. We gradually shift the trajectory toward the camera's right, with final displacements of 5, 10, 15, and 20 cm. The corresponding joint trajectories are computed using inverse kinematics, and both the rendered robot motion and numerical action inputs are updated accordingly.

DOPPEL predicts distinct physical interaction outcomes as the trajectory changes:

  • Duck → pot: The duck lands inside the pot at 5 and 10 cm, but misses at 15 and 20 cm.
  • Bouquet → vase: The bouquet remains inside the vase at 5 cm, rests on its rim at 10 cm, and misses at 15 and 20 cm.
  • Pepper → plate: The pepper reaches the plate's edge at 5 cm and misses entirely for perturbations of 10 cm or greater.

Across these examples, the predicted object motion responds to the shifted robot trajectory, producing different successful and failed placements. Importantly, the transition between outcomes differs across tasks, demonstrating sensitivity to interaction-specific changes rather than simply reproducing the original outcome.

These counterfactual evaluations complement our existing moving-object PSNR and object-level SCS results, which demonstrate improvements beyond robot appearance reconstruction. Since ground-truth videos are unavailable for counterfactual trajectories, we present these results as qualitative evidence of interaction-sensitive prediction, rather than quantitative validation of physical correctness.

4. Cross-Embodiment Learning and Generalization

Addresses: Critical Weakness 4 (Evidence of Cross-Embodiment Knowledge Transfer), Critical Weakness 5 (Unseen-Embodiment Generalization), and Question 4 (Benefits of Joint Training Across Embodiments).

Cross-embodiment learning (W4). We additionally train DOPPEL-8, a single shared model across eight datasets spanning Franka, UR5e, WidowX, bimanual YAM, dexterous humanoid hands, and human hands and bodies, without embodiment-specific heads or labels. As shown in Table B, DOPPEL-8 achieves 24.82/21.46 dB validation/OOD PSNR on DROID, compared with 24.08/21.15 dB for DROID-only DOPPEL, without adding additional DROID demonstrations. Furthermore, DOPPEL-8 improves over our original three-embodiment shared model (24.22/21.15 dB) by +0.60/+0.31 dB, providing evidence that training with additional heterogeneous embodiments can improve prediction performance on an existing embodiment.

Unseen-embodiment generalization (W5). We clarify that DOPPEL targets unified dynamics learning across multiple training embodiments, rather than zero-shot generalization to unseen robot morphologies. Our contribution is a shared backbone and conditioning adapter capable of jointly modeling heterogeneous embodiments without requiring separate world models or embodiment-specific heads. The expanded eight-embodiment evaluation directly supports this setting. We have revised the manuscript to distinguish multi-embodiment learning from zero-shot unseen-embodiment generalization.

Reviewer_1mrcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DOPPEL, an action-conditioned video world model that takes the current frame and a future joint trajectory and generates the next 16 frames. The robot is first rendered along the trajectory from the calibrated camera view, using forward kinematics and a frozen 3DGS robot renderer (Dr. Robot). The resulting robot-only video is then used as a condition together with the numerical actions.

The backbone is a fully fine-tuned Wan2.1-I2V-14B. A 30.5M-parameter adapter encodes the rendered video and the actions separately and fuses them with cross-attention (render tokens as queries, action tokens as keys and values) into a conditioning vector for each video token. This vector is added to the timestep embedding and modulates all backbone layers through AdaLN. One model is shared across Franka (DROID), UR5e (RoboMIND), and WidowX (Bridge). Bridge and RoboMIND do not provide camera extrinsics, so the authors recover them by optimization with the differentiable renderer. They plan to release camera parameters for all 52,249 demonstrations in the three datasets.

Compared with an action-only baseline on the same backbone, DOPPEL improves every metric in Table 2 on all three datasets, with PSNR gains of 0.6–2.0 dB. It also improves moving-object regions; for example, object PSNR on Bridge OOD rises from 21.50 to 24.96. In a shared-model comparison with identical training settings, DOPPEL needs about 1.2M training draws to surpass the action-only model’s result at 3.4M draws. OSCAR, a skeleton-conditioned system-level baseline, performs below both.

Critical strengths

1. DOPPEL beats a matched action-only baseline, including on moving objects.

The action-only baseline uses the same backbone, data and training settings as DOPPEL; the difference is that the whole render branch is removed (App. A.5). DOPPEL is better on all metrics on the three datasets. In the shared-model comparison with matched settings (Fig. 7), DOPPEL after about 1.2M training draws already beats the action-only model after 3.4M draws, averaged over the three datasets. The gains also show up on moving objects: on Bridge OOD, object PSNR goes from 21.50 to 24.96. In Fig. 4, the baseline moves the arm but leaves the object on the table, while DOPPEL predicts the lift.

2. DOPPEL trains one model for three robots on real data and recovers the missing camera extrinsics.

A single 14B model is trained on Franka, UR5e and WidowX data from three real-world datasets; only the renderer and action normalization are robot-specific. Bridge and RoboMIND do not include camera extrinsics, so the authors recover them, and they plan to release camera parameters for all 52,249 demonstrations along with the calibration code. This is useful for other work on these datasets.

Critical weaknesses

1. Limited novelty. This is my main concern. The paper presents rendering the robot along the trajectory and conditioning on the render as its key idea. BridgeV2W (arXiv 2602.03793, Feb. 2026) already renders the robot as a mask from the URDF and calibrated cameras and uses it to control a video world model. It also reports results on DROID and likewise finds that the rendered condition beats end-effector poses. The paper does not cite it. As far as I can tell, the substantive methodological differences come down to two: a 3DGS appearance render in place of a mask or mesh, and per-token AdaLN injection. Neither is supported by an experiment (Weakness 2). EnerVerse-AC, Kinema4D and Dexterous World Models follow closely related ideas and should at least be discussed in the related work.

2. No component ablations beyond the action-only comparison. The action-only baseline shows what happens when the whole render branch is removed. It cannot tell us whether the specific design choices matter: whether a 3DGS render beats a mesh, mask or skeleton, whether per-token AdaLN beats channel concatenation, or whether the numerical actions are still needed once the render is there. OSCAR differs in backbone and data, so it cannot stand in for these comparisons. There is also a confound. From the description in App. A.5, the action-only baseline seems to inject its condition per frame, so every token in a frame gets the same vector, whereas DOPPEL's condition is per token. The Table 2 gains therefore cannot yet be split between the render and the finer modulation.

3. No real-robot, closed-loop or downstream validation. The paper is motivated by planning and policy learning and says DOPPEL can take trajectories from a policy or a planner. Yet every experiment is open-loop video prediction on held-out clips that replay the executed trajectory. This is the most favorable setting for render conditioning: the render is pixel-aligned with the ground-truth frames, so part of the arm-region gain is already supplied by the condition. The paper never tests whether the render still helps when the trajectory comes from a policy and may be infeasible or miss a grasp. OSCAR checks its policy evaluation against real-robot outcomes, and BridgeV2W reports policy evaluation and planning. DOPPEL has no downstream experiment.

Questions for authors

Clarifications:

  1. Please discuss BridgeV2W, EnerVerse-AC, Kinema4D and Dexterous World Models in the related work, state what this paper adds over BridgeV2W, and adjust how the introduction presents the key idea. Apart from the application (data synthesis versus world modeling), how does DOPPEL differ from AnchorDream?
  2. Is the action-only baseline's embedding injected per frame or per token? If per frame, how much of the Table 2 gain comes from per-token modulation?
  3. What are DOPPEL's inference settings: how many sampling steps, and is CFG used? How long do rendering and generating 16 frames take?
  4. How accurate are the recovered extrinsics for Bridge and RoboMIND, for example in reprojection error or mask IoU? How many sessions could not be calibrated?
  5. Are the renders computed from measured joint states or from commanded targets? How does the model behave when the two disagree, for example under contact or obstruction?

Requested experiments:

  1. A comparison of 3DGS render, URDF mesh, mask and 2D skeleton in the same backbone.
  2. Three ablations: render only, without numerical actions; an action-only baseline with per-token injection; channel concatenation instead of AdaLN injection.
  3. At least one downstream experiment, such as ranking several policies with DOPPEL and reporting the correlation with real-robot success (as OSCAR does), or planning on a real robot. If that is not feasible, at least perturb the extrinsics and joints on the existing test sets to measure the effect, and show qualitative results on policy-generated trajectories.

Minor issues:

  • Fig. 2 is hard to follow; please consider redrawing it. The injection path is misleading: the chained ⊕ suggests sequential addition, and the green band looks like a ControlNet-style residual, although the condition is added per token to the timestep embedding. Arrows and the Q/K/V sources of the cross-attention are missing, and the notation is inconsistent with the text (ε, the noise distribution, τ vs. s). Typo: "Mea Subtract".
  • Grammar, e.g. "generates much plausible" and "If this mean subtractions is skipped".
5 comments
AuthorsOpenReview ↗

Author Response Part 1

We thank the reviewer for their thoughtful assessment and for highlighting DOPPEL’s consistent improvements over the action-only baseline, including on moving objects, and its training sample efficiency. We also appreciate their recognition of the shared model across robot embodiments and the value of the recovered camera extrinsics. Below, we address each question and concern in turn, providing additional experiments, comparisons, and clarifications.

1. Novelty and Positioning Relative to Prior Work

Addresses: Critical Weakness 1 (Limited Novelty) and Question 1 (Prior-Work Comparisons and Contribution Clarification).

We thank the reviewer for highlighting these related approaches. We have expanded the related work discussion in the uploaded revised manuscript to include BridgeV2W, EnerVerse-AC, Kinema4D, Dexterous World Models, and AnchorDream, and clarified the positioning of our contributions.

Distinction from existing world models. While these methods share the broader idea of geometry-conditioned video generation, they differ in their conditioning representations and predictive formulations:

  • BridgeV2W uses binary robot masks rendered from URDF models with ControlNet-style conditioning. In contrast, DOPPEL combines dense camera-aligned robot renderings with numerical joint configurations through a shared render-action conditioning framework.
  • EnerVerse-AC uses projected end-effector pose maps and delta-action embeddings rather than dense renderings of the full articulated embodiment.
  • Kinema4D conditions on spatiotemporal robot pointmaps and jointly generates RGB and pointmap sequences, whereas DOPPEL directly predicts future RGB observations conditioned on the initial image and specified robot trajectory.
  • Dexterous World Models conditions video diffusion on egocentric hand-mesh renderings and a static-scene video, rendered along a specified camera trajectory, to predict manipulation-induced residual dynamics. In contrast, DOPPEL conditions directly on a single observed RGB frame and a specified articulated robot trajectory, without requiring a separate static-scene video or 3D scene reconstruction. DOPPEL additionally combines dense robot renderings with numerical joint configurations through a shared conditioning framework across robot embodiments.

Distinction from AnchorDream. Both methods condition video generation on rendered robot motion, but address different objectives. AnchorDream synthesizes demonstrations by generating objects and environments consistent with augmented robot trajectories. DOPPEL instead starts from an observed scene and predicts how that same scene evolves under specified robot motions. In other words, AnchorDream generates scenes consistent with trajectories, whereas DOPPEL predicts the consequences of trajectories in a given scene. Architecturally, AnchorDream uses input-channel concatenation with language and global-trajectory conditioning, while DOPPEL employs a render-action adapter with spatially varying timestep modulation.

Empirical comparison with prior methods. Beyond these distinctions, we provide direct quantitative comparisons with BridgeV2W, EnerVerse-AC, and Kinema4D on our evaluation splits (Table A). DOPPEL achieves the strongest performance across PSNR, SSIM, and LPIPS.

Table A. Comparison with prior geometry-conditioned video generation methods on DROID OOD.

MethodPSNR ↑SSIM ↑LPIPS ↓
DOPPEL-8 (ours)21.4600.8060.108
DOPPEL (single, ours)21.1510.7990.115
BridgeV2W20.5300.7940.131
EnerVerse-AC18.8700.7390.162
Kinema4D (Released Checkpoint)18.5100.7730.159

DOPPEL-8 is jointly trained on eight embodiment datasets, whereas DOPPEL (single) is trained only on DROID. All methods are evaluated on the same DROID OOD split.

We further evaluate the contribution of dense rendering through controlled geometric-conditioning ablations (Section 2) and demonstrate joint training across eight heterogeneous embodiment datasets with a single shared backbone and adapter.

Clarification of our contribution. We do not claim to introduce robot-render conditioning or AdaLN as new techniques. Rather, our contribution is a unified dense embodiment-conditioning framework that combines explicit articulated robot motion with a shared video world model, enabling scene-conditioned dynamics prediction across heterogeneous embodiments. Our experiments demonstrate improvements over prior methods and action-only conditioning, together with the feasibility of learning from eight embodiment datasets using a single shared model.

2. Controlled Ablations of Geometric Representations and Conditioning

Addresses: Critical Weakness 2 (Missing Component Ablations), Question 2 (Per-Frame vs. Per-Token Conditioning), Question 6 (Geometric Representation Comparisons), and Question 7 (Render-Only and Conditioning Mechanism Ablations).

Geometric representation and numerical action ablations. We provide controlled comparisons of action-only, skeleton-only, skeleton + actions, rendering-only, and rendering + actions under matched DROID training settings (Table B).

Table B. Validation and OOD performance on DROID across different conditioning representations.

MethodInputTraining dataVal PSNR ↑Val SSIM ↑Val LPIPS ↓Val SCS ↑OOD PSNR ↑OOD SSIM ↑OOD LPIPS ↓OOD SCS ↑
Action-onlyActionsDROID22.2280.8320.0850.92119.7670.7750.1350.869
OSCAR (fine-tuned)SkeletonDROID20.4270.7510.1350.90318.6190.7050.1740.854
Ours, 2D skeletonSkeletonDROID22.9900.8420.0770.93120.4590.7880.1210.890
Ours, 2D skeleton + actionsSkeleton + actionsDROID23.0920.8440.0750.93220.3110.7830.1250.888
Ours, rendering onlyRenderDROID24.0090.8580.0690.93721.1840.8000.1130.897
DOPPEL (single)Render + actionsDROID24.0770.8600.0680.93821.1510.7990.1150.898
DOPPEL (multi)Render + actions3 robot datasets24.2200.8610.0670.94021.1470.8000.1130.897
DOPPEL-8Render + actionsAll 8 datasets24.8190.8700.0620.94321.4600.8060.1080.903

All DROID-only variants use matched training data, optimization settings, and training exposure. Skeleton and rendering variants share the same spatial token-wise conditioning architecture. OSCAR is a system-level baseline with a different backbone and training setup. DOPPEL (multi) and DOPPEL-8 are trained jointly on three and eight datasets, respectively, and are included as additional multi-embodiment comparisons rather than matched-data ablations. Results are evaluated on 200 clips per split. Bold indicates the best result in each column.

Per-frame versus per-token conditioning. The action-only baseline follows EgoWM, injecting numerical joint configurations per latent frame. Both the action-only baseline and DOPPEL use the Wan 2.1 timestep-modulation (AdaLN) pathway. Since numerical joint configurations have no inherent spatial coordinates, broadcasting the same action embedding to every spatial token is equivalent to per-frame modulation under shared modulation weights. To examine whether spatial conditioning alone explains the performance improvements, we additionally train a skeleton-only baseline using the same token-wise adapter and injection mechanism as DOPPEL. Dense renderings improve over skeleton conditioning by +1.02 dB on validation and +0.73 dB on OOD, demonstrating benefits beyond simply introducing spatially varying conditioning.

3DGS rendering and conditioning architecture. We clarify that our 3DGS renderers are trained exclusively on synthetic URDF/MJCF mesh renderings and are used as a differentiable rendering interface for camera calibration and trajectory conditioning. We do not claim that 3DGS is inherently superior to mesh rasterization or binary masks. Similarly, we adopt the existing timestep-modulation (AdaLN) pathway of Wan 2.1 to incorporate spatially varying conditioning without modifying the video backbone. Our contribution lies in the dense camera-aligned robot motion representation and its integration into a shared multi-embodiment world model, rather than proposing a new conditioning operator. We do not claim that AdaLN is superior to channel concatenation; we adopt the pretrained backbone's existing timestep-modulation pathway. Our controlled ablations focus on the conditioning representation rather than comparing alternative injection architectures.

Joint training across eight embodiments. We additionally train DOPPEL-8 on eight heterogeneous embodiment datasets using a shared backbone and adapter, including bimanual robots, humanoid egocentric observations, human hands, and human-body motion. Without adding more DROID demonstrations, DOPPEL-8 achieves 24.82 validation and 21.46 OOD PSNR on DROID, compared with 24.22/21.15 dB for our original three-embodiment shared model (Table B). These results provide additional evidence that a single shared model can learn across diverse embodiment datasets.

3. Downstream Policy Evaluation and Real-World Agreement

Addresses: Critical Weakness 3 (Lack of Downstream Validation) and Question 8 (Policy Evaluation and Correlation with Real-World Outcomes).

To address the concern that our evaluation is limited to open-loop prediction using recorded robot trajectories, we additionally evaluate DOPPEL on trajectories generated by robot policies and compare predicted task outcomes against real-world executions.

We evaluate two policies, MolmoAct2 and π0.5, in both zero-shot and fine-tuned settings on two manipulation tasks: (1) aligning a plate to the center and (2) sliding a box. Each configuration is evaluated over five trials per task, with success measured in both the real world and DOPPEL.

Table C. Real-world and DOPPEL task success (successful trials / 5).

PolicySettingAlign plate (Real / DOPPEL)Slide box (Real / DOPPEL)Overall (Real / DOPPEL)
MolmoAct2Zero-shot0 / 00 / 00 / 0
MolmoAct2Fine-tuned4 / 45 / 59 / 9
π0.5Zero-shot0 / 00 / 00 / 0
π0.5Fine-tuned4 / 55 / 49 / 9

Each task is evaluated over 5 trials per policy setting. The overall column reports successful trials out of 10, rather than percentages.

DOPPEL matches the aggregate real-world success counts across all four policy configurations. In particular, it distinguishes unsuccessful zero-shot policies from their more successful fine-tuned counterparts, providing initial evidence that DOPPEL's predicted outcomes reflect differences in real-world policy performance. These results extend our evaluation beyond reconstruction on held-out demonstrations to downstream assessment of policy-generated trajectories.

4. Camera Calibration Accuracy and Coverage

Addresses: Question 4 (Accuracy and Coverage of Recovered Camera Extrinsics).

We recover camera extrinsics independently for each recording session using observed robot motion and known kinematics, achieving mean motion recall of 0.71 for Bridge and 0.79 for RoboMIND. We exclude four RoboMIND sessions and four Bridge environments due to insufficient calibration quality.

To evaluate calibration accuracy, we compare rendered robot-arm masks against independently obtained SAM 3 segmentation masks, sampling five frames per clip and averaging the results per session. Table D reports the alignment statistics, with DROID's provided camera extrinsics included as a reference under the same evaluation protocol.

Table D. Accuracy and coverage of recovered camera extrinsics.

DatasetSessions / Camera PosesMean IoUMedian IoUIoU p10–p90Recall / Precision
Bridge (ours)56 sessions0.580.600.43–0.680.75 / 0.71
RoboMIND (ours)49 sessions0.730.750.66–0.800.87 / 0.82
DROID (provided extrinsics)60 poses across 12 labs0.610.620.42–0.770.77 / 0.71

We will release the recovered camera extrinsics and calibration pipeline to facilitate reproduction and application to additional datasets.

Measured versus commanded joint configurations (Q5). DOPPEL is trained and quantitatively evaluated using observed robot joint states rather than commanded targets. The renderings therefore represent executed robot motion, including deviations caused by contact or obstruction insofar as these are captured in the recorded joint measurements. For counterfactual or policy-generated trajectories, DOPPEL conditions on the supplied configurations and assumes that the specified robot motion is realized. Modeling discrepancies between commanded and executed trajectories is outside the scope of the current work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary of Rebuttal and Additional Experimental Evidence

We thank all reviewers for their constructive feedback and engagement throughout the discussion period. We appreciate their acknowledgment of our additional experiments and clarifications, and would like to provide a consolidated summary of the extensive evidence presented in response to the original reviews and subsequent questions.

1. Novelty, positioning, and prior-work comparisons. We have clarified that DOPPEL's central contribution is a multi-embodiment conditioning framework for unified video world modeling, rather than introducing robot rendering or individual architectural operators as fundamentally new techniques. We expanded the related work and provided quantitative comparisons with independently reproduced BridgeV2W, retrained EnerVerse-AC, and the released Kinema4D checkpoint, clearly distinguishing system-level comparisons from controlled ablations.

2. Controlled ablations and action conditioning. We conducted additional same-backbone experiments comparing action-only, skeleton-only, skeleton+actions, rendering-only, and rendering+actions. Dense rendering improves over skeleton conditioning by +1.02 dB validation and +0.73 dB OOD PSNR. Further controlled experiments on Bridge demonstrate that incorporating numerical actions improves OOD PSNR by +0.61 dB and moving-object PSNR by +0.77 dB over render-only conditioning, with improvements on 88% of evaluated clips. We also clarified the architectural design choices and the scope of claims supported by these comparisons.

3. Unified learning across eight heterogeneous embodiments. We extended DOPPEL from three robot embodiments to eight heterogeneous robot and human datasets, using a single shared backbone and conditioning adapter without embodiment-specific learned heads. DOPPEL-8 improves DROID validation/OOD PSNR from 24.22/21.15 to 24.82/21.46 dB relative to DOPPEL-3, without additional DROID demonstrations.

In response to a reviewer's follow-up suggestion, we additionally conducted a preliminary zero-shot evaluation on an unseen humanoid embodiment within the limited rebuttal period. Despite never encountering this embodiment during training, DOPPEL-3 achieves 25.11 dB PSNR compared with 20.69 dB for the static-frame baseline, improving on all 24 evaluated clips without target-embodiment fine-tuning. Generalizing to an unseen morphology and its associated interaction dynamics is a substantially more challenging setting than joint learning across training embodiments. While these preliminary results are encouraging, we emphasize that comprehensive zero-shot generalization is beyond our paper's original scope and represents a promising direction for future research.

4. Interaction dynamics and downstream evaluation. We introduced counterfactual evaluations across three manipulation tasks, demonstrating different predicted interaction outcomes under controlled trajectory perturbations. We additionally conducted real-world and closed-loop DOPPEL evaluations of MolmoAct2 and π0.5 across two tasks. Across 40 paired trials, DOPPEL agrees with real-world success/failure outcomes in 38 cases (95%), providing initial downstream evidence beyond open-loop video reconstruction. We carefully distinguish this result from claims of fine-grained policy ranking.

5. Calibration, efficiency, and reproducibility. We provided quantitative camera-calibration accuracy and coverage, clarified the calibration and test-time information assumptions, and reported detailed inference settings and runtime. We also explicitly clarified that the 63.8% reduction refers to training-example draws, rather than total computational cost.

We will incorporate the reviewers' feedback, methodological clarifications, and additional experimental results presented throughout the rebuttal into the camera-ready manuscript, if accepted.

Taken together, these additional experiments strengthen the evidence for DOPPEL's central contribution: a multi-embodiment conditioning framework that enables a single video world model to learn action-conditioned scene dynamics across heterogeneous embodiments, with demonstrated advantages in prediction quality, multi-embodiment learning, and preliminary downstream evaluation.

We sincerely appreciate the reviewers' time, feedback, and consideration of the substantial additional evidence provided throughout the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.