- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a video–action world model that represents target joint configurations as four URDF-rendered action views from fixed virtual cameras, jointly diffuses them with RGB observations under masked flow matching (forward / inverse / joint modes), and recovers executable joint targets by training-free URDF-constrained multiview matching. The interface idea is clean and the multi-robot demo is real, but the paper's claim (visual action interface beats a joint-vector head) is never ablated, it places 3rd of 5 on its main benchmark, and the joint checkpoint collapses on tasks the single-embodiment model solves.
Critical strengths
The strengths of the paper are summarized as follows:
- Clear, well-motivated interface. The rendering full joint configurations (not just EE pose) from prescribed virtual cameras gives a fixed tensor shape across embodiments and sidesteps per-robot heads and physical-camera extrinsics.
- A non-trivial system demonstration. One set of weights drives five simulated kinematic structures (Tab. 5) and four physical robots with different camera counts.
- The representation is tested in both directions, 1) as generated output for control (Tabs. 3, 5, 6) and 2) as conditioning for video prediction (Tabs. 4, 7).
- Reasonable evaluation volume. There are 100 trials × 50 tasks × 2 settings in sim, per-task breakdowns, real-world tests under randomized backgrounds and distractors not present in demos.
- Recovery accuracy is quantified against GT renderings, and the limitations section is clear about latency and URDF dependence.
Critical weaknesses
The major weaknesses of the paper are:
- No ablation of the core design choice (action views vs a joint-vector head). The thesis is that rendered joint views are a better shared interface than vectors, yet no experiment trains the same DiT/data with a padded joint-vector head (as Octo/RDT/UWM do) or numerical action conditioning for forward dynamics.
- The main benchmark do not show superiority. For example, 3rd of 5 on RoboTwin 2.0 with large per-task losses.
- Multi-embodiment claim weaker than stated. All nine embodiments are in training, with no unseen-embodiment transfer and no joint-vs-per-embodiment comparison.
- Virtual cameras are per-task, not truly shared. A new task needs a preset derived from its demos. Thus, the robustness to rig variation and behaviour on unseen tasks are untested.
- Latency and control granularity. 6.9 s + 2.7 s per 80-step chunk, fully open-loop, plus a frozen 9B VLM inference pass. Which mode is used for closed-loop control, denoising steps, and whether RGB streams are generated at test time are not stated.
Questions for authors
Besides the above weaknesses, I have the following questions for the authors:
- Same backbone and data: what does a padded joint-vector action head (or numerical action conditioning) achieve on the 31-task subset and TriWorldBench?
- Given 9.45°/21.39° rotation error from perfect renderings on Franka/UR5 (Tab. 9), is the 4-view skeleton observable for 7-DoF redundant arms and wrist rotations? Does adding views or mesh rendering fix it?
- Which mode is used for closed-loop control, how many denoising steps, and what is the per-chunk latency in simulation?
- How was the camera-aligned skeleton baseline in Tab. 7 fed to the model, and how many views did it use versus four?
3 comments
Q2: Analysis of the solution errors for Franka/UR5.
Table 3: As shown in the table, more views do not necessarily improve recovery. Our default four-view configuration already provides strong precision.
Our action-view representation encodes joint-level geometry, providing cues to distinguish different configurations of 7-DoF redundant arms. Wrist rotation could become unobservable when the gripper is fully closed. We address this visual degeneracy by maintaining a nonzero separation between the rendered gripper fingers even at full closure. This preserves orientation cues for wrist rotation without altering the physical gripper state. More views provide only marginal improvements. Whether mesh rendering resolves the remaining ambiguities requires further investigation and may necessitate a different reconstruction solver.
To assess the effect of the number of action views, we additionally evaluate action-recovery accuracy by progressively adding views in the order Front, Left, Top, Right and Back. Table 3 shows that more views do not necessarily improve recovery. Our default four-view configuration already provides strong geometric constraints, whereas adding poorly informative views can interfere with the constraints from more informative views during joint optimization, reducing recovery accuracy.
W2: RoboTwin 2.0 results explanation.
Our comparison with VLA/WAM methods on RoboTwin 2.0 evaluates closed-loop control capability. We position these results as evidence of competitive control performance, while acknowledging variation across tasks. Our broader contribution is a unified video-action model supporting both world prediction and action generation across embodiments. RoboTwin 2.0 assesses its control capability, complementing our video-prediction and multi-embodiment evaluations.
W3: Multi-embodiment claim.
We acknowledge that our multi-embodiment claim is weaker than stated. With limited training data, our experiments across five simulated and four real-world embodiments demonstrate the feasibility of multi-embodiment training and task generalization within seen embodiments. Our goal is to explore unified video-action modeling. Future work will investigate generalization to unseen embodiments and tasks, incorporating larger-scale pretraining to improve transfer along both dimensions.
W4: The robustness of rig variation and behaviour on unseen tasks.
Table 4: Analysis of the robustness of OOD tasks execution capability.
To assess the robustness of camera rig and out-of-distribution task execution capability, we construct camera intrinsics using a fixed field of view (), without relying on task-specific parameters, and evaluate the checkpoint trained on 31 tasks across five robot embodiments on the remaining 19 Aloha tasks through closed-loop execution. For each task, we conduct 25 trials under each of the clean and randomized settings and report the corresponding success rates in Table 4.
W5 & Q3: Closed-loop control settings.
For closed-loop control in simulation and real-world, we use joint generation in image-to-video (I2V) mode with 8 denoising steps, predicting both RGB videos and action views. On RoboTwin 2.0, we use a 40-step action chunk. Model inference and action recovery take 3.8 s and 1.6 s, respectively, totaling approximately 5.4 s per chunk.
Q4: Configuration of the camera-aligned skeleton baseline.
Table 5: For each trajectory in CLVR and RAD, we sample eight frames to extract DINOv3 visual features and encode its text description with Qwen3.5-VL. These features are combined into a 256-dimensional embedding for K-means clustering with 50 clusters. We then select one representative trajectory per cluster based on the cluster centers, yielding 50 evaluation trajectories.
For this evaluation, we redesigned the training setup to mix different numbers of action views. Both variants in Table 5 use the same two third-person RGB observations. The skeleton baseline uses two camera-aligned skeleton views, whereas our model uses a variable number of action views, with the view count sampled uniformly during training. Skeleton views use the same pipeline as action views, with view embeddings identifying viewpoints and the action modality embedding identifying skeleton tokens. During training and inference, we use resolution as input and resolution as output. We also ablate the number of action views for forward dynamics mode. We find it is interesting that training with mixed action-views is helpful for video generation. (Following TriWorldBench, Subject Consistency measures whether robot and object identity and appearance remain stable across frames in each view.)
We appreciate your valuable and thoughtful feedback. All revisions to the manuscript are highlighted in blue.
W1 & Q1.1: What does a padded joint-vector action head achieve on the 31-task subset?
Table 1: As shown in the table, our method outperforms baselines on all embodiments, and is higher than the variant on Aloha-Agilex, Piper, ARX-X5 and Franka-Panda, while slightly lower on UR5-Xsg.
Embodiment\strutAloha-Agilex\strutPiper\strutARX-X5\strutFranka-Panda\strutUR5-XsgOurs w/ paddedjoint-vectorOcto\strutCleanRandom\strutRandomRandom\strutAvg.Random\strut18.32Random\strut17.29Random\strut17.03Random\strut16.13Random\strut15.48Random\strut18.88Random\strut21.29Random\strut18.45Random\strut17.94Random\strut17.29Random\strut18.60Random\strut19.29Random\strut17.74Random\strut17.04Random\strut16.39RandomOurs w/ paddedjoint-vectorRDT\strutCleanRandom\strutRandomRandom\strutAvg.Random\strut10.97Random\strut10.27Random\strut13.03Random\strut10.58Random\strut13.55Random\strut8.90Random\strut11.87Random\strut12.65Random\strut12.00Random\strut16.65Random\strut9.94Random\strut11.07Random\strut12.84Random\strut11.29Random\strut15.10RandomOurs w/ paddedjoint-vectorOurs w/ paddedjoint-vector\strutCleanRandom\strutRandomRandom\strutAvg.Random\strut35.61Random\strut38.84Random\strut38.06Random\strut26.06Random\strut36.13Random\strut34.71Random\strut37.68Random\strut36.52Random\strut22.58Random\strut31.35Random\strut35.16Random\strut38.26Random\strut37.29Random\strut24.32Random\strut33.74RandomOurs w/ paddedjoint-vectorOurs\strutCleanRandom\strutRandomRandom\strutAvg.Random\strut81.94Random\strut81.55Random\strut82.19Random\strut64.52Random\strut31.87Random\strut82.45Random\strut82.32Random\strut80.26Random\strut62.45Random\strut30.06Random\strut82.19Random\strut81.94Random\strut81.23Random\strut63.48Random\strut30.97Random
We add a comparison with Octo and RDT as unified action-space baselines, each is trained for 100k steps with a batch size of 16. We further ablate our action representation on RoboTwin 2.0 using a unified 16-dimensional joint vector: seven joint positions and one gripper state per arm, with dimensions 7 and 15 masked for 6-DoF arms. Following RDT, we use an MLP to encode each timestep’s action into one token, and then concatenated with RGB video tokens, another MLP predicts the full joint vector. We train this variant for 100k steps on 31 tasks shared across 5 embodiments and evaluate 25 trials per embodiment-task pair in each of the clean and randomized settings.
W1 & Q1.2: What does numerical action conditioning achieve on the TriWorldBench?
Table 2: Ablation study on TriWorldBench. GT-dependent metrics are excluded from TVC, TA and MQ. Overall is the equal-weight mean of the 11 shared metrics. As shown in the table, our method outperforms numerical action conditioning on every reported metric.
ModelOurs w/ numerical action conditioningOursTVC79.7689.54TA54.2374.65P3D51.5961.30MQ41.5442.42TC59.2064.88Overall59.7667.76
Since GT-dependent metrics require submission to the official evaluation system, we report only the GT-independent metrics evaluated locally: TVC (tri-view consistency), TA (task alignment), P3D (physical and 3D coherence), MQ (motion quality), and TC (temporal consistency). The Overall score is a weighted aggregate of these reported metrics.