acceptodds
Withdrawn from ICLR 2027

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world robot learning is constrained by the cost of collecting experience and evaluating candidate behaviors. Video generation models offer a scalable foundation for visual simulators that predict action outcomes before physical execution. Yet they often favor visual plausibility over accurate action following and coherent robot–object dynamics, while action-conditioned simulators depend on scarce, embodiment-specific data that are difficult to share across incompatible control spaces. We introduce WorldLine, an action-driven visual simulator that decouples transferable dynamics learning from heterogeneous action grounding. WorldLine learns manipulation dynamics from more than 10,000 hours of action-free robot videos and grounds them using over 2,000 hours of action trajectories across more than ten embodiments. An image-space action representation provides a shared control interface across embodiments, while multi-view and failure-enriched training with relational regularization improves interaction-sensitive prediction. Robot-focused few-step distillation enables efficient causal rollout while preserving action-critical motion. Across held-out and out-of-domain settings, WorldLine maintains strong visual quality and robot-motion agreement; on failed trajectories, it improves robot-mask IoU by 0.1626 over the strongest baseline. It predicts trajectory success with 74% mean accuracy across RoboTwin and AgiBot, one percentage point above the strongest baseline. Without RoboTwin training or adaptation, its rollouts improve task success by up to 21.4 percentage points over direct policy execution. Together, these capabilities make WorldLine a scalable and efficient visual simulator for policy evaluation and embodied planning.

Primary area:
generative models
Keywords:
World Model, Video Generation, Simulator

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SSoQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorldLine builds an action-conditioned visual simulator through robot-video pretraining, spatial action grounding, and causal distillation. It adapts Cosmos3-Nano on over 10,000 hours of robot video, then uses approximately 2,500 hours of action-labeled trajectories across more than ten embodiments. Image-space action maps, multi-view and failure-enriched training, and relational supervision support interaction prediction. The experiments evaluate AgiBotWorld video prediction, DROID transfer, success classification, and RoboTwin trajectory selection with two policies.

Critical strengths

The paper demonstrates substantial downstream gains with useful controls. At 32 candidates, WorldLine improves RoboTwin success from 58.9% to 78.0% for π0.5 and from 41.0% to 62.4% for LingBot-VLA.

The ablations test the main training choices, and the evaluation includes real failures. Table 3 matches total compute when removing action-free data, supporting the value of additional video coverage beyond longer training. Failure data improve LingBot-VLA selection from 51.6% to 62.4%, while native-action conditioning performs worse for both policies.

Critical weaknesses

The experiments do not isolate the benefit of robot-focused distillation. The causal model delivers a 4.3× speedup but reduces DROID robot IoU from 0.2739 to 0.1986 (Table 2; Section 5.3). A matched causal-student ablation of Equation 8 would establish how much the robot-region and motion losses contribute to preserving robot motion.

The evaluation does not isolate robot–object interaction accuracy. Table 2 measures image fidelity and robot-mask overlap; Table 3 adds planning success but no object-motion or contact metric. The planning gains establish utility, but do not quantify how accurately the model predicts contact transitions and object motion.

Questions for authors

How do the robot-focused distillation losses change robot IoU and planning success relative to the same causal student trained without them, particularly on the tasks with the largest regressions?

Reviewer_jrmaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

WorldLine present a multi-view, action conditioned world model trained from a cosmos3-nano base model in three stages. Action free, large-scale pretraining, multi-view, faliure enriched, latent-regularized mid-training, and causal, few-step distillation post-training. The strongest contribution is the integrated training recipe and its empirical behavior.

Critical strengths

The paper demonstrates the utility of incorperating faliure trajectories within the training process, showing a substantial increase in in domain faliure predictions.

Critical weaknesses

The paper's main claimed contributions are not very novel. Specifically, the recipe of robot action free pretraining, followed by action conditioned, is presented as novel, while one of the paper's quoted baseline GE-Sim-V2 Qiu et al. (2026) is trained with this exact method as it's initial training phase (https://arxiv.org/abs/2508.05635v3). Infact, the exact process of action free robot pretraining, followed by action conditioned prediction, combined with V-JEPA feature similarity, and DMD student distillation is found directly in DreamX-Phi 1.0 (https://arxiv.org/abs/2608.13489v1). The paper fails to include these as either citations or baselines, leading to significant concerns on soundness, and novelty. The paper's claims that it's experiments show OOD performance are difficult to believe. It uses DROID as its OOD distribution, yet its base checkpoint Cosmos3-NANO list DROID data as part of its mid-training dataset. Thus, 'fully out-of-domain settings' needs either an acknowledgement of the base model's training data distribution, or more clarification on the exact checkpoint used.

Questions for authors

Which exact Cosmos3-Nano checkpoint initializes WorldLine, and what DROID exposure does that checkpoint have?

Reviewer_kZEhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes WorldLine, an action-conditioned visual simulator for robot manipulation, aiming to address the scarcity of action-annotated data, the difficulty of unifying control spaces across different robots, and the insufficient action-following capability of video generation models. The method builds on Cosmos3-Nano: it first learns visual dynamics in manipulation scenes from over 10,000 hours of action-free robot videos, and is then action-conditioned on approximately 2,500 hours of trajectories spanning more than ten robot morphologies. Its control interface encodes end-effector position, depth, a 6D rotation representation, and gripper state into a 9-channel action image aligned with the video space, and incorporates multi-view data, failure trajectories, and V-JEPA2 feature relation regularization. Finally, the model achieves chunked autoregressive generation through causal adaptation and few-step distillation.

Experiments cover scene-exclusive video prediction on AgiBotWorld, out-of-distribution evaluation on DROID, trajectory success prediction, and best-of-N candidate trajectory selection on RoboTwin. Main results include improved robot motion consistency on failure trajectories, success rate improvements of 19.1 and 21.4 percentage points for two policies via candidate selection, and a 4.3× generation speedup of the four-step causal model over the full model. The paper's contribution mainly lies in the systematic combination of a large-scale training recipe, a cross-robot control interface, and downstream applications.

Critical strengths
  1. The staged separation of dynamics-prior learning from action-conditioned learning on robot videos has a clear technical motivation: the former can leverage broader video data, while the latter uses geometrically validated control trajectories to establish the link between actions and visual changes. The image-space action representation further preserves position, depth, orientation, and gripper information, allowing control signals from different robots to enter the video model in a unified spatial form.

    This design is supported by some empirical evidence. In Table 3, the version without action-free video pretraining matches the full model's total training compute budget, yet AgiBotWorld LPIPS degrades from 0.1804 to 0.2924, and the planning success rate of π0.5 drops from 78.0% to 73.2%. The value of the additional video data is therefore supported by an ablation that controls training compute. Failure data also yields task-relevant gains: removing failure trajectories reduces LingBot-VLA's planning success rate from 62.4% to 51.6%. These results give the training recipe practical research value.

  2. On RoboTwin, all simulators use the same policy checkpoints, candidate action sets, and Qwen3VL-8B selector, and are evaluated on five independently sampled candidate sets. This improves the interpretability of comparisons across simulators. WorldLine, without any RoboTwin training or adaptation, improves π0.5's success rate from 58.9% to 78.0% at N=32, and LingBot-VLA from 41.0% to 62.4%, indicating that its predicted videos contain useful information for discriminating among candidate actions.

    Meanwhile, robot mask IoU on failure trajectories improves from 0.3079 for the strongest baseline to 0.4705, suggesting the model also predicts motion patterns beyond successful demonstrations well. The four-step version reduces generation time for 129 frames from 90 seconds to 21 seconds while retaining a substantial portion of the average planning gains. Together, these results support the system's potential for offline policy analysis and candidate trajectory screening.

Critical weaknesses
  1. The main text (§4.3.1) states that action images are derived from command-specified end-effector poses and do not use future observations; however, Appendix A.1 and A.4 describe the data as containing "state or action sequences." For data from different sources, the paper does not specify, case by case, whether the action image encodes a control target, the robot state after actual execution, or a target trajectory reconstructed from states. Appendix A.3 mentions updating the wrist camera extrinsics based on kinematics, but does not make clear whether the joint configurations used in this process are all available before prediction. Especially under contact, collision, and tracking error, the commanded trajectory and the actually realized trajectory can differ, and this distinction changes the prediction problem the model actually solves. Similar ambiguity appears in the planning experiments. Appendix C.2 states that the policy generates a complete set of candidate actions, but does not clarify whether the candidate trajectories are generated directly from the initial observation, or where the observations for subsequent policy calls come from when spanning multiple action chunks. If generating the full candidates requires feedback from the real RoboTwin environment, then the experiment supports offline ranking of existing trajectories, but provides more limited support for planning capabilities that do not require executing candidates first. The current text is insufficient to determine whether these issues affect the implementation; the authors need to provide explicit data definitions and a candidate generation pipeline.

  2. The video evaluation mainly uses full-frame visual metrics and robot mask IoU. The latter can reflect the consistency of the robot's contour and motion, but cannot directly determine whether the target object was grasped correctly, whether slippage occurred, or whether contact produced the correct object state changes. The paper does provide downstream planning results; however, the most direct success/failure prediction experiment only reaches 74%, compared with 73% for the strongest baseline. According to Appendix C.3, this result comes from a fixed set of 100 base trajectories, each repeated across five video generation seeds; these repetitions cannot be treated as 500 independent test trajectories. The absence of paired confidence intervals at the base-trajectory level and of per-dataset results makes it difficult to judge whether this marginal lead is stable.

Post-distillation task performance also shows significant degradation beyond the averages. In Table 4, π0.5's success rate on Handover Block drops from 56% with the full model to 8% with the four-step model, and on Stack Two Blocks from 92% to 64%. Thus, the preservation of average planning gains is insufficient to demonstrate that action-critical interactions are preserved across task types. Given that reliable interaction prediction and efficient simulation are the paper's primary goals, these degradations require further explanation, along with evidence that directly reflects the correctness of object states or interaction events.

Questions for authors

Please specify, for each training and evaluation data source, the actual meaning of q_t: a control command, a target end-effector pose, or a measured robot state? How are the future extrinsics of the wrist camera constructed? In both the video-prediction and candidate-planning settings, can you confirm that all action images and camera conditioning can be computed from the initial information and the to-be-executed control sequence, and can you provide a consistent processing pipeline?

How do π0.5 and LingBot-VLA generate full-task-length candidate action trajectories from the same initial scene? If the policy is invoked multiple times, what observations do subsequent invocations use, and do they access real RoboTwin environment feedback? Additionally, how exactly are the 58.9% and 41.0% direct-execution success rates computed: one independent execution per scene, or an average over all trajectories in the candidate set? Please clarify the sample denominator and its pairing with the candidate-selection results.

For the 74% vs. 73% success-prediction results, can you provide an uncertainty analysis at the base-trajectory level that preserves the pairing across methods, and report the RoboTwin and AgiBot results separately? For the four-step model's clear degradation on tasks such as Handover Block, do the errors mainly come from robot motion, object interaction prediction, or the final selector's judgment? Analyses of these existing cases would directly affect my assessment of the conclusion that "action-critical dynamics are preserved."

In Table 1, Masked Visual Actions' action training data is listed as 450 hours and failure data as 60 hours, but the cited original paper states that its unified checkpoint was fine-tuned on approximately 15 hours of data. Please clarify whether this study actually uses the public checkpoint, a retrained version, or a different data accounting convention, and explain how it was adapted on each test set. This affects the interpretation of the baseline setup and relative performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.