acceptodds
Withdrawn from ICLR 2027

Dream4ACT: A Shared Visual Action Interface for Multi-Embodiment Video–Action Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

15 views

Abstract

Video generation models encode rich spatiotemporal priors about scene dynamics. To serve robot control, a general video-action model must reason bidirectionally: predicting future observations under given actions, and inferring actions from observed or desired outcomes. Existing approaches fall short—either predicting joint-space vectors through dedicated action heads that vary across embodiments, or visualizing only end-effector quantities that discard full-arm articulation and still require embodiment-specific inverse-kinematics mappings. We present Dream4ACT, a video-action world model that renders target joint configurations as four action views via URDF-based forward kinematics from prescribed virtual cameras independent of physical observation cameras. These visual sequences preserve embodiment-specific geometry while sharing a video-modeling pipeline with RGB observations. A diffusion transformer jointly processes multiview physical-camera observations and four action-view streams, conditioned on semantic features from a frozen vision-language model followed by a trainable adapter. Masked flow matching supports forward dynamics, inverse dynamics, and joint observation-action generation by varying which future sequences are corrupted. For execution, training-free multiview matching against URDF-based renderings recovers joint targets without a learned embodiment-specific decoder. Dream4ACT achieves 88.98% success rate on RoboTwin 2.0, and 65.66 score on TriWorldBench. Ablation studies further confirm the benefit of action-view conditioning for high-quality video generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Unified Video-Action Model, Multi-Embodiment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V6qhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a video–action world model that represents target joint configurations as four URDF-rendered action views from fixed virtual cameras, jointly diffuses them with RGB observations under masked flow matching (forward / inverse / joint modes), and recovers executable joint targets by training-free URDF-constrained multiview matching. The interface idea is clean and the multi-robot demo is real, but the paper's claim (visual action interface beats a joint-vector head) is never ablated, it places 3rd of 5 on its main benchmark, and the joint checkpoint collapses on tasks the single-embodiment model solves.

Critical strengths

The strengths of the paper are summarized as follows:

  1. Clear, well-motivated interface. The rendering full joint configurations (not just EE pose) from prescribed virtual cameras gives a fixed tensor shape across embodiments and sidesteps per-robot heads and physical-camera extrinsics.
  2. A non-trivial system demonstration. One set of weights drives five simulated kinematic structures (Tab. 5) and four physical robots with different camera counts.
  3. The representation is tested in both directions, 1) as generated output for control (Tabs. 3, 5, 6) and 2) as conditioning for video prediction (Tabs. 4, 7).
  4. Reasonable evaluation volume. There are 100 trials × 50 tasks × 2 settings in sim, per-task breakdowns, real-world tests under randomized backgrounds and distractors not present in demos.
  5. Recovery accuracy is quantified against GT renderings, and the limitations section is clear about latency and URDF dependence.
Critical weaknesses

The major weaknesses of the paper are:

  1. No ablation of the core design choice (action views vs a joint-vector head). The thesis is that rendered joint views are a better shared interface than vectors, yet no experiment trains the same DiT/data with a padded joint-vector head (as Octo/RDT/UWM do) or numerical action conditioning for forward dynamics.
  2. The main benchmark do not show superiority. For example, 3rd of 5 on RoboTwin 2.0 with large per-task losses.
  3. Multi-embodiment claim weaker than stated. All nine embodiments are in training, with no unseen-embodiment transfer and no joint-vs-per-embodiment comparison.
  4. Virtual cameras are per-task, not truly shared. A new task needs a preset derived from its demos. Thus, the robustness to rig variation and behaviour on unseen tasks are untested.
  5. Latency and control granularity. 6.9 s + 2.7 s per 80-step chunk, fully open-loop, plus a frozen 9B VLM inference pass. Which mode is used for closed-loop control, denoising steps, and whether RGB streams are generated at test time are not stated.
Questions for authors

Besides the above weaknesses, I have the following questions for the authors:

  1. Same backbone and data: what does a padded joint-vector action head (or numerical action conditioning) achieve on the 31-task subset and TriWorldBench?
  2. Given 9.45°/21.39° rotation error from perfect renderings on Franka/UR5 (Tab. 9), is the 4-view skeleton observable for 7-DoF redundant arms and wrist rotations? Does adding views or mesh rendering fix it?
  3. Which mode is used for closed-loop control, how many denoising steps, and what is the per-chunk latency in simulation?
  4. How was the camera-aligned skeleton baseline in Tab. 7 fed to the model, and how many views did it use versus four?
3 comments

We appreciate your valuable and thoughtful feedback. All revisions to the manuscript are highlighted in blue.

W1 & Q1.1: What does a padded joint-vector action head achieve on the 31-task subset?

Table 1: As shown in the table, our method outperforms baselines on all embodiments, and is higher than the variant on Aloha-Agilex, Piper, ARX-X5 and Franka-Panda, while slightly lower on UR5-Xsg.

EmbodimentOurs w/ paddedjoint-vectorOcto\strutCleanRandom\strutRandomRandom\strutAvg.RandomOurs w/ paddedjoint-vectorRDT\strutCleanRandom\strutRandomRandom\strutAvg.RandomOurs w/ paddedjoint-vectorOurs w/ paddedjoint-vector\strutCleanRandom\strutRandomRandom\strutAvg.RandomOurs w/ paddedjoint-vectorOurs\strutCleanRandom\strutRandomRandom\strutAvg.Random\strutAloha-Agilex\strutPiper\strutARX-X5\strutFranka-Panda\strutUR5-Xsg\strut18.32Random\strut18.88Random\strut18.60Random\strut17.29Random\strut21.29Random\strut19.29Random\strut17.03Random\strut18.45Random\strut17.74Random\strut16.13Random\strut17.94Random\strut17.04Random\strut15.48Random\strut17.29Random\strut16.39Random\strut10.97Random\strut8.90Random\strut9.94Random\strut10.27Random\strut11.87Random\strut11.07Random\strut13.03Random\strut12.65Random\strut12.84Random\strut10.58Random\strut12.00Random\strut11.29Random\strut13.55Random\strut16.65Random\strut15.10Random\strut35.61Random\strut34.71Random\strut35.16Random\strut38.84Random\strut37.68Random\strut38.26Random\strut38.06Random\strut36.52Random\strut37.29Random\strut26.06Random\strut22.58Random\strut24.32Random\strut36.13Random\strut31.35Random\strut33.74Random\strut81.94Random\strut82.45Random\strut82.19Random\strut81.55Random\strut82.32Random\strut81.94Random\strut82.19Random\strut80.26Random\strut81.23Random\strut64.52Random\strut62.45Random\strut63.48Random\strut31.87Random\strut30.06Random\strut30.97Random{\small\textstyle \def\CW#1{\strut\smash{\overset{\textstyle\hphantom{\text{Random}}}{#1}}}\begin{array}{ccccc}\hline \textbf{Embodiment} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\textbf{Octo}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\textbf{RDT}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\textbf{Ours}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array}\cr\hline \begin{array}{c}\strut \text{Aloha-Agilex}\cr \strut \text{Piper}\cr \strut \text{ARX-X5}\cr \strut \text{Franka-Panda}\cr \strut \text{UR5-Xsg}\cr\end{array} & \begin{array}{ccc}\CW{18.32} & \CW{18.88} & \CW{18.60}\cr \CW{17.29} & \CW{21.29} & \CW{19.29}\cr \CW{17.03} & \CW{18.45} & \CW{17.74}\cr \CW{16.13} & \CW{17.94} & \CW{17.04}\cr \CW{15.48} & \CW{17.29} & \CW{16.39}\cr\end{array} & \begin{array}{ccc}\CW{10.97} & \CW{8.90} & \CW{9.94}\cr \CW{10.27} & \CW{11.87} & \CW{11.07}\cr \CW{13.03} & \CW{12.65} & \CW{12.84}\cr \CW{10.58} & \CW{12.00} & \CW{11.29}\cr \CW{13.55} & \CW{16.65} & \CW{15.10}\cr\end{array} & \begin{array}{ccc}\CW{35.61} & \CW{34.71} & \CW{35.16}\cr \CW{38.84} & \CW{37.68} & \CW{38.26}\cr \CW{38.06} & \CW{36.52} & \CW{37.29}\cr \CW{26.06} & \CW{22.58} & \CW{24.32}\cr \CW{\textbf{36.13}} & \CW{\textbf{31.35}} & \CW{\textbf{33.74}}\cr\end{array} & \begin{array}{ccc}\CW{\textbf{81.94}} & \CW{\textbf{82.45}} & \CW{\textbf{82.19}}\cr \CW{\textbf{81.55}} & \CW{\textbf{82.32}} & \CW{\textbf{81.94}}\cr \CW{\textbf{82.19}} & \CW{\textbf{80.26}} & \CW{\textbf{81.23}}\cr \CW{\textbf{64.52}} & \CW{\textbf{62.45}} & \CW{\textbf{63.48}}\cr \CW{31.87} & \CW{30.06} & \CW{30.97}\cr\end{array}\cr\hline\end{array}}

We add a comparison with Octo and RDT as unified action-space baselines, each is trained for 100k steps with a batch size of 16. We further ablate our action representation on RoboTwin 2.0 using a unified 16-dimensional joint vector: seven joint positions and one gripper state per arm, with dimensions 7 and 15 masked for 6-DoF arms. Following RDT, we use an MLP to encode each timestep’s action into one token, and then concatenated with RGB video tokens, another MLP predicts the full joint vector. We train this variant for 100k steps on 31 tasks shared across 5 embodiments and evaluate 25 trials per embodiment-task pair in each of the clean and randomized settings.

W1 & Q1.2: What does numerical action conditioning achieve on the TriWorldBench?

Table 2: Ablation study on TriWorldBench. GT-dependent metrics are excluded from TVC, TA and MQ. Overall is the equal-weight mean of the 11 shared metrics. As shown in the table, our method outperforms numerical action conditioning on every reported metric.

ModelTVCTAP3DMQTCOverallOurs w/ numerical action conditioning79.7654.2351.5941.5459.2059.76Ours89.5474.6561.3042.4264.8867.76\small\textstyle \begin{array}{lcccccc} \hline \text{Model} & \text{TVC} & \text{TA} & \text{P3D} & \text{MQ} & \text{TC} & \text{Overall} \cr \hline \text{Ours w/ numerical action conditioning} & 79.76 & 54.23 & 51.59 & 41.54 & 59.20 & 59.76 \cr \textbf{Ours} & \textbf{89.54} & \textbf{74.65} & \textbf{61.30} & \textbf{42.42} & \textbf{64.88} & \textbf{67.76} \cr \hline \end{array}

Since GT-dependent metrics require submission to the official evaluation system, we report only the GT-independent metrics evaluated locally: TVC (tri-view consistency), TA (task alignment), P3D (physical and 3D coherence), MQ (motion quality), and TC (temporal consistency). The Overall score is a weighted aggregate of these reported metrics.

Q2: Analysis of the solution errors for Franka/UR5.

Table 3: As shown in the table, more views do not necessarily improve recovery. Our default four-view configuration already provides strong precision.

\strutEmbodiment\strutPosition (mm) ↓\strutOne000.00\strutTwo000.00\strutThree000.00\strutFour000.00\strutFive000.00\strutRotation (∘) ↓\strutOne000.00\strutTwo000.00\strutThree000.00\strutFour000.00\strutFive000.00\strutAloha-Agilex\strutPiper\strutARX-X5\strutFranka-Panda\strutUR5-Xsg\strut82.85000.00\strut4.63000.00\strut1.91000.00\strut1.39000.00\strut1.23000.00\strut147.61000.00\strut11.99000.00\strut3.28000.00\strut1.66000.00\strut1.93000.00\strut143.37000.00\strut6.00000.00\strut2.83000.00\strut1.22000.00\strut1.35000.00\strut165.89000.00\strut12.14000.00\strut7.30000.00\strut4.11000.00\strut3.98000.00\strut60.51000.00\strut22.71000.00\strut17.75000.00\strut11.85000.00\strut11.88000.00\strut38.36000.00\strut11.59000.00\strut3.07000.00\strut1.57000.00\strut1.60000.00\strut60.37000.00\strut17.54000.00\strut3.72000.00\strut1.71000.00\strut1.74000.00\strut54.52000.00\strut16.19000.00\strut3.33000.00\strut1.95000.00\strut2.37000.00\strut56.30000.00\strut22.54000.00\strut13.25000.00\strut9.45000.00\strut9.13000.00\strut46.80000.00\strut33.31000.00\strut26.85000.00\strut21.39000.00\strut21.17000.00{\small\textstyle \def\CW#1{\strut\smash{\overset{\textstyle\hphantom{\textbf{000.00}}}{#1}}}\begin{array}{ccc}\hline \begin{array}{l}\strut\cr \text{Embodiment}\end{array} & \begin{array}{c}\strut\text{Position (mm) }\downarrow\cr\hline \begin{array}{ccccc}\CW{\text{One}} & \CW{\text{Two}} & \CW{\text{Three}} & \CW{\text{Four}} & \CW{\text{Five}}\cr\end{array}\end{array} & \begin{array}{c}\strut\text{Rotation (}{}^\circ\text{) }\downarrow\cr\hline \begin{array}{ccccc}\CW{\text{One}} & \CW{\text{Two}} & \CW{\text{Three}} & \CW{\text{Four}} & \CW{\text{Five}}\cr\end{array}\end{array}\cr\hline \begin{array}{l}\strut \text{Aloha-Agilex}\cr \strut \text{Piper}\cr \strut \text{ARX-X5}\cr \strut \text{Franka-Panda}\cr \strut \text{UR5-Xsg}\cr\end{array} & \begin{array}{ccccc}\CW{82.85} & \CW{4.63} & \CW{1.91} & \CW{1.39} & \CW{\textbf{1.23}}\cr \CW{147.61} & \CW{11.99} & \CW{3.28} & \CW{\textbf{1.66}} & \CW{1.93}\cr \CW{143.37} & \CW{6.00} & \CW{2.83} & \CW{\textbf{1.22}} & \CW{1.35}\cr \CW{165.89} & \CW{12.14} & \CW{7.30} & \CW{4.11} & \CW{\textbf{3.98}}\cr \CW{60.51} & \CW{22.71} & \CW{17.75} & \CW{\textbf{11.85}} & \CW{11.88}\cr\end{array} & \begin{array}{ccccc}\CW{38.36} & \CW{11.59} & \CW{3.07} & \CW{\textbf{1.57}} & \CW{1.60}\cr \CW{60.37} & \CW{17.54} & \CW{3.72} & \CW{\textbf{1.71}} & \CW{1.74}\cr \CW{54.52} & \CW{16.19} & \CW{3.33} & \CW{\textbf{1.95}} & \CW{2.37}\cr \CW{56.30} & \CW{22.54} & \CW{13.25} & \CW{9.45} & \CW{\textbf{9.13}}\cr \CW{46.80} & \CW{33.31} & \CW{26.85} & \CW{21.39} & \CW{\textbf{21.17}}\cr\end{array}\cr\hline\end{array}}

Our action-view representation encodes joint-level geometry, providing cues to distinguish different configurations of 7-DoF redundant arms. Wrist rotation could become unobservable when the gripper is fully closed. We address this visual degeneracy by maintaining a nonzero separation between the rendered gripper fingers even at full closure. This preserves orientation cues for wrist rotation without altering the physical gripper state. More views provide only marginal improvements. Whether mesh rendering resolves the remaining ambiguities requires further investigation and may necessitate a different reconstruction solver.

To assess the effect of the number of action views, we additionally evaluate action-recovery accuracy by progressively adding views in the order Front, Left, Top, Right and Back. Table 3 shows that more views do not necessarily improve recovery. Our default four-view configuration already provides strong geometric constraints, whereas adding poorly informative views can interfere with the constraints from more informative views during joint optimization, reducing recovery accuracy.

W2: RoboTwin 2.0 results explanation.

Our comparison with VLA/WAM methods on RoboTwin 2.0 evaluates closed-loop control capability. We position these results as evidence of competitive control performance, while acknowledging variation across tasks. Our broader contribution is a unified video-action model supporting both world prediction and action generation across embodiments. RoboTwin 2.0 assesses its control capability, complementing our video-prediction and multi-embodiment evaluations.

W3: Multi-embodiment claim.

We acknowledge that our multi-embodiment claim is weaker than stated. With limited training data, our experiments across five simulated and four real-world embodiments demonstrate the feasibility of multi-embodiment training and task generalization within seen embodiments. Our goal is to explore unified video-action modeling. Future work will investigate generalization to unseen embodiments and tasks, incorporating larger-scale pretraining to improve transfer along both dimensions.

W4: The robustness of rig variation and behaviour on unseen tasks.

Table 4: Analysis of the robustness of OOD tasks execution capability.

Method\strutRoboTwin 2.0\strutCleanRandom\strutRandomRandom\strutAvg.Random\strutOurs\strut19.37Random\strut17.47Random\strut18.42Random{\small\textstyle \def\CW#1{\strut\smash{\overset{\textstyle\hphantom{\text{Random}}}{#1}}}\begin{array}{cc}\hline \textbf{Method} & \begin{array}{c}\strut\textbf{RoboTwin 2.0}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array}\cr\hline \begin{array}{l}\strut \text{Ours}\cr\end{array} & \begin{array}{ccc}\CW{19.37} & \CW{17.47} & \CW{18.42}\cr\end{array}\cr\hline\end{array}} To assess the robustness of camera rig and out-of-distribution task execution capability, we construct camera intrinsics using a fixed field of view (FoV=60\text{FoV}=60), without relying on task-specific parameters, and evaluate the checkpoint trained on 31 tasks across five robot embodiments on the remaining 19 Aloha tasks through closed-loop execution. For each task, we conduct 25 trials under each of the clean and randomized settings and report the corresponding success rates in Table 4.

W5 & Q3: Closed-loop control settings.

For closed-loop control in simulation and real-world, we use joint generation in image-to-video (I2V) mode with 8 denoising steps, predicting both RGB videos and action views. On RoboTwin 2.0, we use a 40-step action chunk. Model inference and action recovery take 3.8 s and 1.6 s, respectively, totaling approximately 5.4 s per chunk.

Q4: Configuration of the camera-aligned skeleton baseline.

Table 5: For each trajectory in CLVR and RAD, we sample eight frames to extract DINOv3 visual features and encode its text description with Qwen3.5-VL. These features are combined into a 256-dimensional embedding for K-means clustering with 50 clusters. We then select one representative trajectory per cluster based on the cluster centers, yielding 50 evaluation trajectories.

MethodPSNR↑SSIM↑LPIPS↓Subject Consistency↑Skeleton rendering23.1900.8990.1050.0885One action-view23.6210.8800.1140.1083Two action-views24.7070.8900.1060.1068Three action-views24.6710.8890.1060.1066Four action-views (Ours)24.7800.9000.1020.1239\small\textstyle \begin{array}{lcccc} \hline \textbf{Method} & \textbf{PSNR}\uparrow & \textbf{SSIM}\uparrow & \textbf{LPIPS}\downarrow & \textbf{Subject Consistency}\uparrow \cr \hline \text{Skeleton rendering} & 23.190 & 0.899 & 0.105 & 0.0885 \cr \hline \text{One action-view} & 23.621 & 0.880 & 0.114 & 0.1083 \cr \text{Two action-views} & 24.707 & 0.890 & 0.106 & 0.1068 \cr \text{Three action-views} & 24.671 & 0.889 & 0.106 & 0.1066 \cr \text{Four action-views (Ours)} & \textbf{24.780} & \textbf{0.900} & \textbf{0.102} & \textbf{0.1239} \cr \hline \end{array}

For this evaluation, we redesigned the training setup to mix different numbers of action views. Both variants in Table 5 use the same two third-person RGB observations. The skeleton baseline uses two camera-aligned skeleton views, whereas our model uses a variable number of action views, with the view count sampled uniformly during training. Skeleton views use the same pipeline as action views, with view embeddings identifying viewpoints and the action modality embedding identifying skeleton tokens. During training and inference, we use 320×224320\times224 resolution as input and 640×480640\times480 resolution as output. We also ablate the number of action views for forward dynamics mode. We find it is interesting that training with mixed action-views is helpful for video generation. (Following TriWorldBench, Subject Consistency measures whether robot and object identity and appearance remain stable across frames in each view.)

Reviewer_GZ34OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces action views: a target joint configuration is rendered via its URDF model from four fixed virtual cameras, and these four views serve as the action representation inside a video model. This gives heterogeneous embodiments a common image-space interface (fixed shape, independent of joint dimensionality) while preserving embodiment-specific geometry; the virtual cameras are independent of physical cameras, removing extrinsic calibration and avoiding per-embodiment decoders.

Critical strengths

Rendering URDF targets from fixed virtual cameras unifies heterogeneous action spaces into images while retaining arm geometry, and sidesteps both camera extrinsics and per-embodiment action heads; the training-free recovery keeps the "shared interface" claim honest end-to-end.

One checkpoint is evaluated on five simulated embodiments and four real platforms, with recovery errors reported

Critical weaknesses

Rendered targets mean joint/link inaccuracies, flexing, wear, or calibration error propagate directly, yet no robustness analysis under URDF mismatch is given. More decisively, on TriWorldBench the overall score only ties BWM (65.66 vs 65.54) and it trails on Task Alignment (81.63 vs 86.05) — the most task-relevant axis — so "highest reported score" is marginal, not a demonstrated advantage.

Ablations do not isolate the representation on the metric that matters. The action-view vs skeleton gap is small (24.33 vs 23.19 PSNR; LPIPS 0.093 vs 0.105) on only 50 DROID sequences, and there is no closed-loop comparison against direct joint vectors or trained per-embodiment decoders — the exact baselines the paper argues against. Inverse/joint modes are trained but never separately evaluated.

Questions for authors

see weakness

2 comments

We appreciate your valuable and thoughtful feedback. All revisions to the manuscript are highlighted in blue.

W1.1: Robustness analysis of the URDF mismatch.

We acknowledge that URDF inaccuracies can propagate to action recovery and execution. This concern is particularly relevant to the Franka and UR5 setups, which combine two single-arm models into a dual-arm system and require accurate modeling of their relative base poses. The UR5 setup additionally includes a 10 cm height offset between the two arm bases. Our RoboTwin 2.0 evaluation across five embodiments and 31 tasks shows higher errors on these platforms but substantially lower errors on the other embodiments. These results suggest that an embodiment-specific issue with the assembled URDFs and their calibration.

W1.2: TriWorldBench results explanation.

We agree that the narrow overall margin indicates competitive performance rather than a decisive lead. We clarify that our Task Alignment score is 84.22, compared with BWM’s 86.05; 81.63 is our TVC score. TriWorldBench jointly evaluates task alignment, cross-view consistency, physical coherence, and visual fidelity. Our broader contribution is unified video-action modeling: the same model supports both forward dynamics and action generation for closed-loop control. TriWorldBench evaluates the former, while our RoboTwin 2.0 and real-robot experiments assess the latter. Together, these evaluations demonstrate competitive world prediction alongside executable action generation.

W2.1: Complementary camera-aligned ablation study.

Table 1: For each trajectory in CLVR and RAD, we sample eight frames to extract DINOv3 visual features and encode its text description with Qwen3.5-VL. These features are combined into a 256-dimensional embedding for K-means clustering with 50 clusters. We then select one representative trajectory per cluster based on the cluster centers, yielding 50 evaluation trajectories.

MethodPSNR↑SSIM↑LPIPS↓Subject Consistency↑Skeleton rendering23.1900.8990.1050.0885One action-view23.6210.8800.1140.1083Two action-views24.7070.8900.1060.1068Three action-views24.6710.8890.1060.1066Four action-views (Ours)24.7800.9000.1020.1239\small\textstyle \begin{array}{lcccc} \hline \textbf{Method} & \textbf{PSNR}\uparrow & \textbf{SSIM}\uparrow & \textbf{LPIPS}\downarrow & \textbf{Subject Consistency}\uparrow \cr \hline \text{Skeleton rendering} & 23.190 & 0.899 & 0.105 & 0.0885 \cr \hline \text{One action-view} & 23.621 & 0.880 & 0.114 & 0.1083 \cr \text{Two action-views} & 24.707 & 0.890 & 0.106 & 0.1068 \cr \text{Three action-views} & 24.671 & 0.889 & 0.106 & 0.1066 \cr \text{Four action-views (Ours)} & \textbf{24.780} & \textbf{0.900} & \textbf{0.102} & \textbf{0.1239} \cr \hline \end{array}

For this evaluation, we redesigned the training setup to mix different numbers of action views. Both variants in Table 1 use the same two third-person RGB observations. The skeleton baseline uses two camera-aligned skeleton views, whereas our model uses a variable number of action views, with the view count sampled uniformly during training. Skeleton views use the same pipeline as action views, with view embeddings identifying viewpoints and the action modality embedding identifying skeleton tokens. During training and inference, we use 320×224320\times224 resolution as input and 640×480640\times480 resolution as output. We also ablate the number of action views for forward dynamics mode. We find it is interesting that training with mixed action-views is helpful for video generation. (Following TriWorldBench, Subject Consistency measures whether robot and object identity and appearance remain stable across frames in each view.)

W2.2: Unified joint-vector ablation study.

Table 2: As shown in the table, our method outperforms baselines on all embodiments, and is higher than the variant on Aloha-Agilex, Piper, ARX-X5 and Franka-Panda, while slightly lower on UR5-Xsg.

EmbodimentOurs w/ paddedjoint-vectorOcto\strutCleanRandom\strutRandomRandom\strutAvg.RandomOurs w/ paddedjoint-vectorRDT\strutCleanRandom\strutRandomRandom\strutAvg.RandomOurs w/ paddedjoint-vectorOurs w/ paddedjoint-vector\strutCleanRandom\strutRandomRandom\strutAvg.RandomOurs w/ paddedjoint-vectorOurs\strutCleanRandom\strutRandomRandom\strutAvg.Random\strutAloha-Agilex\strutPiper\strutARX-X5\strutFranka-Panda\strutUR5-Xsg\strut18.32Random\strut18.88Random\strut18.60Random\strut17.29Random\strut21.29Random\strut19.29Random\strut17.03Random\strut18.45Random\strut17.74Random\strut16.13Random\strut17.94Random\strut17.04Random\strut15.48Random\strut17.29Random\strut16.39Random\strut10.97Random\strut8.90Random\strut9.94Random\strut10.27Random\strut11.87Random\strut11.07Random\strut13.03Random\strut12.65Random\strut12.84Random\strut10.58Random\strut12.00Random\strut11.29Random\strut13.55Random\strut16.65Random\strut15.10Random\strut35.61Random\strut34.71Random\strut35.16Random\strut38.84Random\strut37.68Random\strut38.26Random\strut38.06Random\strut36.52Random\strut37.29Random\strut26.06Random\strut22.58Random\strut24.32Random\strut36.13Random\strut31.35Random\strut33.74Random\strut81.94Random\strut82.45Random\strut82.19Random\strut81.55Random\strut82.32Random\strut81.94Random\strut82.19Random\strut80.26Random\strut81.23Random\strut64.52Random\strut62.45Random\strut63.48Random\strut31.87Random\strut30.06Random\strut30.97Random{\small\textstyle \def\CW#1{\strut\smash{\overset{\textstyle\hphantom{\text{Random}}}{#1}}}\begin{array}{ccccc}\hline \textbf{Embodiment} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\textbf{Octo}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\textbf{RDT}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\textbf{Ours}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array}\cr\hline \begin{array}{c}\strut \text{Aloha-Agilex}\cr \strut \text{Piper}\cr \strut \text{ARX-X5}\cr \strut \text{Franka-Panda}\cr \strut \text{UR5-Xsg}\cr\end{array} & \begin{array}{ccc}\CW{18.32} & \CW{18.88} & \CW{18.60}\cr \CW{17.29} & \CW{21.29} & \CW{19.29}\cr \CW{17.03} & \CW{18.45} & \CW{17.74}\cr \CW{16.13} & \CW{17.94} & \CW{17.04}\cr \CW{15.48} & \CW{17.29} & \CW{16.39}\cr\end{array} & \begin{array}{ccc}\CW{10.97} & \CW{8.90} & \CW{9.94}\cr \CW{10.27} & \CW{11.87} & \CW{11.07}\cr \CW{13.03} & \CW{12.65} & \CW{12.84}\cr \CW{10.58} & \CW{12.00} & \CW{11.29}\cr \CW{13.55} & \CW{16.65} & \CW{15.10}\cr\end{array} & \begin{array}{ccc}\CW{35.61} & \CW{34.71} & \CW{35.16}\cr \CW{38.84} & \CW{37.68} & \CW{38.26}\cr \CW{38.06} & \CW{36.52} & \CW{37.29}\cr \CW{26.06} & \CW{22.58} & \CW{24.32}\cr \CW{\textbf{36.13}} & \CW{\textbf{31.35}} & \CW{\textbf{33.74}}\cr\end{array} & \begin{array}{ccc}\CW{\textbf{81.94}} & \CW{\textbf{82.45}} & \CW{\textbf{82.19}}\cr \CW{\textbf{81.55}} & \CW{\textbf{82.32}} & \CW{\textbf{81.94}}\cr \CW{\textbf{82.19}} & \CW{\textbf{80.26}} & \CW{\textbf{81.23}}\cr \CW{\textbf{64.52}} & \CW{\textbf{62.45}} & \CW{\textbf{63.48}}\cr \CW{31.87} & \CW{30.06} & \CW{30.97}\cr\end{array}\cr\hline\end{array}}

We add a comparison with Octo and RDT as unified action-space baselines, each is trained for 100k steps with a batch size of 16. We further ablate our action representation on RoboTwin 2.0 using a unified 16-dimensional joint vector: seven joint positions and one gripper state per arm, with dimensions 7 and 15 masked for 6-DoF arms. Following RDT, we use an MLP to encode each timestep’s action into one token, and then concatenated with RGB video tokens, another MLP predicts the full joint vector. We train this variant for 100k steps on 31 tasks shared across 5 embodiments and evaluate 25 trials per embodiment-task pair in each of the clean and randomized settings.

W2.3: Inverse/joint modes evaluation.

All RoboTwin 2.0 and real-robot experiments use joint generation in image-to-video (I2V) mode. Inverse dynamics mode serves only as an auxiliary training objective, as joint generation also supports its action-prediction functionality at inference.

Reviewer_waC7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

A video-action world model that renders target joint configurations from four virtual cameras as “action views,” giving different robots a shared visual interface. A DiT over RGB and action-view latents, conditioned on a frozen VLM, supports forward dynamics, inverse dynamics, and joint generation via masked flow matching. Joint targets are recovered by URDF-based matching, without a learned decoder. Validation covers RoboTwin 2.0, TriWorldBench, five simulated and four real embodiments, and a DROID ablation.

Critical strengths
  • Decoder-free cross-embodiment control on real robots. One checkpoint runs four real platforms at 80 to 95% on two shared tasks, a meaningful step beyond per-robot action heads.
  • Strong at both control and generation. The same checkpoint reaches 88.98% on RoboTwin 2.0 and 65.66 on TriWorldBench, on par with BWM.
Critical weaknesses
  • Recovery is inconsistently described and untested against alternatives.
  • The main text and Appendix A.3 describe different optimization procedures.
  • There is no comparison with a learned action head.
  • Camera rigs are fitted per task group from training trajectories.
  • Unexplained multi-embodiment regressions. The joint checkpoint nearly fails some tasks on every robot (e.g., Blocks Ranking RGB 0 to 4% vs 90%+ for the single checkpoint). RoboTwin results also trail LingBot-VA and Fast-WAM.
Questions for authors
  • Which recovery procedure produced the results, and how does it compare with a learned action head?
  • Why do some tasks collapse in the joint checkpoint, and how does the single checkpoint perform on the same 31 tasks?
  • How are camera rigs obtained for new tasks or embodiments?
2 comments

We appreciate your valuable and thoughtful feedback. All revisions to the manuscript are highlighted in blue.

W1 & W2 & Q1.1: Recovery procedure explanation.

All reported results for action recovery and robot execution were obtained using the GPU implementation detailed in Appendix A.3. We presented the CPU implementation in the main text because its direct render-and-compare formulation provides an intuitive explanation of the underlying multiview geometric matching principle. We have clarified this distinction in the main text, explicitly identifying the GPU implementation as the solver used in our experiments.

W3 & Q1.2: Unified action head comparison.

Table 1: As shown in the table, our method outperforms baselines on all embodiments, and is higher than the variant on Aloha-Agilex, Piper, ARX-X5 and Franka-Panda, while slightly lower on UR5-Xsg.

EmbodimentOurs w/ paddedjoint-vectorOcto\strutCleanRandom\strutRandomRandom\strutAvg.RandomOurs w/ paddedjoint-vectorRDT\strutCleanRandom\strutRandomRandom\strutAvg.RandomOurs w/ paddedjoint-vectorOurs w/ paddedjoint-vector\strutCleanRandom\strutRandomRandom\strutAvg.RandomOurs w/ paddedjoint-vectorOurs\strutCleanRandom\strutRandomRandom\strutAvg.Random\strutAloha-Agilex\strutPiper\strutARX-X5\strutFranka-Panda\strutUR5-Xsg\strut18.32Random\strut18.88Random\strut18.60Random\strut17.29Random\strut21.29Random\strut19.29Random\strut17.03Random\strut18.45Random\strut17.74Random\strut16.13Random\strut17.94Random\strut17.04Random\strut15.48Random\strut17.29Random\strut16.39Random\strut10.97Random\strut8.90Random\strut9.94Random\strut10.27Random\strut11.87Random\strut11.07Random\strut13.03Random\strut12.65Random\strut12.84Random\strut10.58Random\strut12.00Random\strut11.29Random\strut13.55Random\strut16.65Random\strut15.10Random\strut35.61Random\strut34.71Random\strut35.16Random\strut38.84Random\strut37.68Random\strut38.26Random\strut38.06Random\strut36.52Random\strut37.29Random\strut26.06Random\strut22.58Random\strut24.32Random\strut36.13Random\strut31.35Random\strut33.74Random\strut81.94Random\strut82.45Random\strut82.19Random\strut81.55Random\strut82.32Random\strut81.94Random\strut82.19Random\strut80.26Random\strut81.23Random\strut64.52Random\strut62.45Random\strut63.48Random\strut31.87Random\strut30.06Random\strut30.97Random{\small\textstyle \def\CW#1{\strut\smash{\overset{\textstyle\hphantom{\text{Random}}}{#1}}}\begin{array}{ccccc}\hline \textbf{Embodiment} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\textbf{Octo}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\textbf{RDT}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array} & \begin{array}{c}\vphantom{\begin{array}{c}\textbf{Ours w/ padded}\cr\textbf{joint-vector}\end{array}}\textbf{Ours}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array}\cr\hline \begin{array}{c}\strut \text{Aloha-Agilex}\cr \strut \text{Piper}\cr \strut \text{ARX-X5}\cr \strut \text{Franka-Panda}\cr \strut \text{UR5-Xsg}\cr\end{array} & \begin{array}{ccc}\CW{18.32} & \CW{18.88} & \CW{18.60}\cr \CW{17.29} & \CW{21.29} & \CW{19.29}\cr \CW{17.03} & \CW{18.45} & \CW{17.74}\cr \CW{16.13} & \CW{17.94} & \CW{17.04}\cr \CW{15.48} & \CW{17.29} & \CW{16.39}\cr\end{array} & \begin{array}{ccc}\CW{10.97} & \CW{8.90} & \CW{9.94}\cr \CW{10.27} & \CW{11.87} & \CW{11.07}\cr \CW{13.03} & \CW{12.65} & \CW{12.84}\cr \CW{10.58} & \CW{12.00} & \CW{11.29}\cr \CW{13.55} & \CW{16.65} & \CW{15.10}\cr\end{array} & \begin{array}{ccc}\CW{35.61} & \CW{34.71} & \CW{35.16}\cr \CW{38.84} & \CW{37.68} & \CW{38.26}\cr \CW{38.06} & \CW{36.52} & \CW{37.29}\cr \CW{26.06} & \CW{22.58} & \CW{24.32}\cr \CW{\textbf{36.13}} & \CW{\textbf{31.35}} & \CW{\textbf{33.74}}\cr\end{array} & \begin{array}{ccc}\CW{\textbf{81.94}} & \CW{\textbf{82.45}} & \CW{\textbf{82.19}}\cr \CW{\textbf{81.55}} & \CW{\textbf{82.32}} & \CW{\textbf{81.94}}\cr \CW{\textbf{82.19}} & \CW{\textbf{80.26}} & \CW{\textbf{81.23}}\cr \CW{\textbf{64.52}} & \CW{\textbf{62.45}} & \CW{\textbf{63.48}}\cr \CW{31.87} & \CW{30.06} & \CW{30.97}\cr\end{array}\cr\hline\end{array}}

We add a comparison with Octo and RDT as unified action-space baselines, each is trained for 100k steps with a batch size of 16. We further ablate our action representation on RoboTwin 2.0 using a unified 16-dimensional joint vector: seven joint positions and one gripper state per arm, with dimensions 7 and 15 masked for 6-DoF arms. Following RDT, we use an MLP to encode each timestep’s action into one token, and then concatenated with RGB video tokens, another MLP predicts the full joint vector. We train this variant for 100k steps on 31 tasks shared across 5 embodiments and evaluate 25 trials per embodiment-task pair in each of the clean and randomized settings.

W5 & Q2: Joint-checkpoint results analysis.

Table 2: As shown in the table, with a batch size of 16, the single-embodiment model trained for 20k steps already achieves higher success rates on some selected tasks in randomized settings than the joint model trained for 100k steps.

TaskSingle CheckpointJoint CheckpointΔ (pp)Blocks Ranking RGB20\unicode370\unicode37+20Move Can Pot56\unicode3772\unicode37−16Move Stapler Pad72\unicode3796\unicode37−24Open Laptop44\unicode3796\unicode37−52Place Bread Basket56\unicode3768\unicode37−12Stack Blocks Two28\unicode374\unicode37+24{\small\textstyle\begin{array}{lccc}\hline \text{Task} & \text{Single Checkpoint} & \text{Joint Checkpoint} & \Delta\text{ (pp)}\cr\hline \textit{Blocks Ranking RGB} & 20\unicode{37} & 0\unicode{37} & +20\cr \textit{Move Can Pot} & 56\unicode{37} & 72\unicode{37} & -16\cr \textit{Move Stapler Pad} & 72\unicode{37} & 96\unicode{37} & -24\cr \textit{Open Laptop} & 44\unicode{37} & 96\unicode{37} & -52\cr \textit{Place Bread Basket} & 56\unicode{37} & 68\unicode{37} & -12\cr \textit{Stack Blocks Two} & 28\unicode{37} & 4\unicode{37} & +24\cr\hline\end{array}}

We train an Aloha-Agilex-only model for 20k steps with a batch size of 16 on the same 31 tasks, achieving an average success rate of 69.68%. We further compare the single and joint checkpoint on the six tasks within this set whose success rate gaps exceed 15 percentage points in the originally reported results.

With the task set held fixed, this comparison suggests that multi-embodiment joint training may introduce task-specific interference. This interpretation is consistent with negative transfer reported for heterogeneous robot-data mixtures [1], where adding diverse robot data does not necessarily improve performance uniformly across tasks.

W4 & Q3: Camera rig analysis for new tasks and embodiments.

Table 3: Analysis of the robustness of OOD tasks execution capability.

Method\strutRoboTwin 2.0\strutCleanRandom\strutRandomRandom\strutAvg.Random\strutOurs\strut19.37Random\strut17.47Random\strut18.42Random{\small\textstyle \def\CW#1{\strut\smash{\overset{\textstyle\hphantom{\text{Random}}}{#1}}}\begin{array}{cc}\hline \textbf{Method} & \begin{array}{c}\strut\textbf{RoboTwin 2.0}\cr\hline \begin{array}{ccc}\CW{\text{Clean}} & \CW{\text{Random}} & \CW{\text{Avg.}}\cr\end{array}\end{array}\cr\hline \begin{array}{l}\strut \text{Ours}\cr\end{array} & \begin{array}{ccc}\CW{19.37} & \CW{17.47} & \CW{18.42}\cr\end{array}\cr\hline\end{array}}

To assess the robustness of camera rig and out-of-distribution task execution capability, we construct camera intrinsics using a fixed field of view (FoV=60\text{FoV}=60), without relying on task-specific parameters, and evaluate the checkpoint trained on 31 tasks across five robot embodiments on the remaining 19 Aloha tasks through closed-loop execution. For each task, we conduct 25 trials under each of the clean and randomized settings and report the corresponding success rates in Table 3.

For a new embodiment, we can construct action views from its URDF using the fixed FoV camera setup described above. However, our experiments to date, based on limited training data across five simulated and four real-world embodiments, demonstrate the feasibility of joint training across embodiments and task generalization within seen embodiments. We will incorporate larger-scale pretraining to improve transfer along both tasks and embodiments in the future.

References

[1] Ye Wang, Sipeng Zheng, Hao Luo, Wanpeng Zhang, Haoqi Yuan, Chaoyi Xu, Haiweng Xu, Yicheng Feng, Mingyang Yu, Zhiyu Kang, et al. Rethinking visual-language-action model scaling: Alignment, mixture, and regularization. arXiv preprint arXiv:2602.09722, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.