acceptodds
Under review as a conference paper at ICLR 2027

Tracing How Robot World Models Hallucinate Objects

Abstract

Action-conditioned video diffusion models now render robot motion convincingly, yet the objects the robot manipulates are often left behind, duplicated or lost. Standard video metrics miss these failures and even reward them, since a clip that repeats its first frame beats the generated video on temporal-difference error in 98% of 540 cases. Rather than adding data or conditioning, we present, to our knowledge, the first mechanistic account of how a robot video world model carries objects, using causal interventions in frozen diffusion transformers (DiTs). The control renders only the arm and steers it through attention routing, since restoring only the queries and keys (Q/K) of the correct control removes 84–88% of the error a corrupted control adds. The object appears only in the first image, so the model must copy it forward. We find that rotary position embedding (RoPE) turns the first image's temporal index into an address for this copy, and shifting that index alone makes the video return to its starting scene inside the predicted frame windows, in 10 of 10 held-out episodes and in two public video models. The copy is already visible in the model's first denoising step and is stored in the partially denoised latent, which redraws the object at its old place after the hand has moved it. Guided by this mechanism, we paint the object into the control, which makes it follow the hand in 6 of 16 new failures, and in our 14B model re-noising a leftover copy's region early removes the copy while keeping the carry. These findings trace a basic limitation of current robot world models to the fact that they are told where objects are only in the first image, and point future models toward conditioning on object state, acting from the first denoising steps and evaluating whether objects go where the actions send them.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, world models, video diffusion, robotics, generative modelling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_wMC3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates object persistence errors in robot video world models, including objects being left behind, duplicated, or disappearing. Through attention restoration, temporal position shifts, and latent-state interventions, the authors examine how action conditioning and reference content influence these failures. They find that restoring Q/K can recover arm motion, changing the reference image’s temporal index causes the initial scene to reappear at predictable times, and some residual object copies persist in the partially denoised latent state. The paper addresses an important problem and presents several compelling interventions, although the connection between these findings and naturally occurring object errors needs further clarification.

Critical strengths
  1. Correct arm motion does not guarantee correct object interaction. The paper demonstrates this distinction clearly and shows how standard video metrics can miss task-relevant failures.

  2. Predicting the frame window in which the initial scene reappears, then validating it on ten held-out episodes, provides stronger evidence than attention visualization alone.

  3. Exact native replays, background-region interventions, and stationary-object controls help interpret the effects. The authors also report unsuccessful interventions and differences across models.

Critical weaknesses
  1. The role of attention routing is somewhat overstated. Restoring Q/K removes much of the error introduced by corrupted control, but restoring V also recovers arm motion and performs better in some layers. This supports routing as an effective pathway, but does not establish that it is the primary pathway.

  2. Temporal addressing is not yet directly linked to native object failures. Moving the reference image’s temporal index demonstrates control over when its content reappears. It is less clear whether the same process causes duplication under normal inputs, particularly since cutting reference reads does not remove existing copies.

  3. Object painting uses privileged future information. Recorded object pixels provide future position, appearance, and occlusion information together. The 6/16 repairs are informative, but do not establish which information matters or how comparable conditioning would be obtained during planning.

  4. The tasks mainly involve pick-and-place and stacking. In contact-rich tasks such as insertion, tight-fitting assembly, or constrained pushing, similar arm motions can produce different outcomes because of contact, friction, or alignment. Objects may remain visually consistent while the predicted interaction is physically wrong. It is unclear how well the proposed account explains these failures.

  5. Some quantitative results need stronger support. Identical native and replay videos received matching detailed labels in only 11/16 cases. Although these disagreements did not affect carry labels, the annotation procedure needs clarification. The central 14B re-noising result also comes from a single example.

Questions for authors
  1. How do Q/K-only, V-only, and joint restoration compare? Do routing and value pathways provide redundant control information?

  2. On the same native failure cases, would early versus late interventions help distinguish the formation of an object copy from its subsequent persistence?

  3. How would painting the initial object appearance along a planned trajectory compare with painting recorded future pixels? Could such conditioning produce a false successful carry when the actual grasp fails?

  4. Could the authors include one or two contact-rich tasks, ideally with similar actions but different successful and failed outcomes? In particular, does object conditioning preserve collisions and jamming, or encourage the model to generate the intended motion despite those constraints?

  5. How many annotators assessed the videos, and how were disagreements resolved? Can the 14B re-noising result be repeated on a few additional cases while reporting both copy removal and damage to otherwise correct scene content?

1 comment
AuthorsOpenReview ↗

Response to Reviewer wMC3

Thank you for the detailed review and the specific questions, which decided most of the experiments we ran. We answer them in order of importance.

W2 / Q2. Is the temporal lookup linked to natural failures? Early versus late. We took 71 rollouts in which the model left the object at its start with unmodified inputs (44 from the early checkpoint, 27 from the fully trained one). For each we re-ran the first denoising step with the same noise, but blocked attention from the generated frames to the first-image tokens that cover the carried object. In 46 of 71 runs the object disappears from its start position in the clean video predicted at that step. Blocking the same number of background tokens does this in 3 of 71. So in natural failures the copy at the start comes from attention to the object in the first image.

For early versus late we looked at finished videos, blocking only the tokens at the object's start position after the grasp. With the block on for all 50 steps, the start position is empty at the end in 41 of 71 runs, against 24 of 71 for the background block. With the block on only for steps 1 to 5 there is no difference (26 vs 23). The copy appears at the first step, but the model keeps re-drawing it from the first image at later steps. The effect at the first step is also much larger when all generated frames are blocked than when only post-grasp frames are (46 vs 19 of 71), which is probably why the post-pickup cuts in the submission left copies in place (Sec. 7, App. M.7).

W1 / Q1. Q/K-only, V-only and joint restoration. We agree that the submission overstated routing. On 54 episodes we corrupted the control and restored the values, the queries and keys, or all three from the run with the correct control. At the first denoising step, V alone removes 99% of the error added by the corruption and Q/K alone 84 to 88%; all three remove all of it. The two pathways are largely redundant. In finished videos either one restores the arm and neither restores the object. Section 4 now says that routing is one effective pathway, and that the control does not carry the object through either one (App. M.2).

W3 / Q3. Privileged information, planned trajectories, false carries. We first checked which information matters. On 44 failures of the fully trained model, painting a gray silhouette or the object's first-frame crop along the recorded path empties the start position as often as painting the recorded pixels (29 and 27 runs, against 28), so position is what the model needs.

For the planned trajectory, we follow Time-to-Move (ICLR 2026) and write the object's first-frame crop along the planned gripper path into the noisy latent during the first 30 of 50 denoising steps. The 14B model rarely fails, so we used all of its failures that can be scored, which are 11 rollouts from six episodes (one episode fails with all six seeds). With the correction the object leaves its start in 10 of the 11 and reaches the target in 8. The same edit on an empty table region changes none of them. On the 1.3B model the automatic tracks show correct carries going from 48% to 66% over the 156 validation episodes. The same edit on an empty table region again has no effect (Sec. 5, App. M.9).

False carries can happen. In one constructed case where we edited the arm path to turn back while the gripper still closes, the model drew a held plate in all three seeds. Ctrl-World also shows a carry after a recorded failed grasp in 3 of 8 DROID episodes. So object conditioning has to come from a prediction of whether the grasp succeeds, which the paper now states.

W4 / Q4. Contact-rich tasks. We agree. Our data contains only pick-and-place and stacking, and the discussion now limits the claims to lift-and-carry tasks.

W5 / Q5. Annotation; re-noising. The "11 of 16" figure compared the detailed label of a video with the label of its exact replay, that is, the same video labeled twice, so we agree the detailed labels in the submission were noisy. All results in the revision are instead scored by a fixed rule on SAM 2.1 tracks (is the object's mask still at its start position in the last frames), which involves no annotator judgment. It agrees with the submission's labels on that question (Cohen's kappa 0.91). The 14B re-noising did not replicate. On six more 14B failures it did not remove the object, so it is now presented as a single example.

We hope this addresses W1 to W3 in particular. If it does, we would be grateful if you would consider updating your score, and we are happy to answer further questions.

Reviewer_hS4TOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores why robot video world models leave objects behind, duplicate them, or make them disappear despite generating plausible arm motion. Through interventions on attention, temporal position encoding, conditioning inputs, and intermediate latent states, the authors study how object information is propagated during generation.

Critical strengths
  1. The paper studies an important failure mode for robot world models: plausible robot motion does not guarantee correct object motion.

  2. The analysis goes beyond attention visualization and includes direct interventions with useful controls, such as edits to background regions and stationary-object inputs.

  3. The authors report both successful and unsuccessful interventions across several models, helping identify the limits of their findings.

Critical weaknesses
  1. Section 8 / Figure 7: The evidence for removing a leftover copy through local re-noising is limited. The main 14B result is demonstrated on one sample, while the 1.3B experiments do not show a consistent advantage over background-region edits. More cases are needed to establish how reliably this intervention supports the proposed explanation.

  2. Section 7: Early visibility of an outcome does not establish that early intervention is sufficient to change it. On the 16 new failures, providing object information only during updates 1–5 repairs two cases, the same number as introducing it from update 11, compared with six repairs when it is provided throughout. This limits the evidence for emphasizing the earliest updates as the main intervention window.

  3. Section 5: Painting the object into the control uses its recorded future appearance and motion. This is informative as a diagnostic intervention, but does not establish how the model could infer object motion from the original inputs. The implications for practical world-model design therefore need a clearer distinction between providing object trajectories and predicting them.

Questions for authors
  1. Can the 14B local re-noising result be replicated across more episodes and random seeds, including matched background edits?

  2. What evidence supports prioritizing early intervention, given the mixed results of the update-window experiments?

  3. Would approximate object masks or trajectories derived from the planned gripper motion provide similar benefits without using recorded future object pixels?

1 comment
AuthorsOpenReview ↗

Response to Reviewer hS4T

Thank you for the positive and constructive review. Each of your three questions turned into an experiment. We describe what we found below, including the cases where the answer is negative.

W3 / Q3. Can trajectories derived from the planned gripper motion replace recorded future pixels? For the 14B model, largely yes. We first checked which part of the painted object matters. On 44 failures of the fully trained 1.3B model, painting a gray silhouette or the object's crop from the first frame along the recorded path empties the start position as often as painting the recorded future pixels (29 and 27 runs, against 28). So the model needs the object's position over time, and its future appearance adds nothing.

We then replaced the recorded path with the planned gripper path, which a planner has. Following Time-to-Move (ICLR 2026), during the first 30 of 50 denoising steps we write the object's first-frame crop into the noisy latent at the position given by the planned path, with the grasp time taken from the gripper command. Nothing is retrained and no future frame is used.

  • The 14B model rarely fails, so we used all of its failures that can be scored, 11 rollouts from six episodes (one episode fails with all six seeds). With the correction the object leaves its start in 10 of the 11 and reaches the target in 8, including all six seeds of the consistently failing episode. The same edit applied to an empty table region changes none of the 11. If the correction is applied to every 14B rollout, including the ones that were already correct, correct carries drop from 85% to 78%, so it should only be used on rollouts that are flagged as failing.
  • On the 1.3B model, the automatic tracks show correct carries going from 48% to 66% over the 156 validation episodes. The same edit on an empty table region has no effect.

Section 5 now separates the diagnostic experiment (recorded object path) from the version a planner could run (planned gripper path), as you suggested (App. M.3, M.9).

W2 / Q2. What supports prioritizing early intervention? We agree with your reading. The evidence does not support it, and we removed that recommendation. What does hold is that the outcome can be read early. The clean video predicted at the first denoising step, 2% of the cost of a full generation, already indicates which rollouts will leave the object behind (AUROC 0.82 for the fully trained 1.3B model, 0.91 for the 14B model). This is useful for filtering rollouts but it is not a fix. In the new attention-blocking experiment (71 natural failures), blocking attention to the object's first-image tokens only during steps 1 to 5 leaves the finished video unchanged (start empty in 26 of 71 runs, against 23 for a background block), while the same block kept on for all 50 steps empties it in 41 of 71 (against 24). The model keeps re-drawing the object from the first image after the early steps. The planned-path correction shows the same graded picture. On 32 failed runs of the early 1.3B checkpoint, the object stays at its start in 24 without the correction, 21 with steps 1 to 5, 13 with steps 1 to 15 and 5 with steps 1 to 30, so most of the effect comes from the first 30 steps and the last 20 are not needed (Sec. 7, App. M.6, M.7, M.9).

W1 / Q1. Does the 14B re-noising replicate? It did not replicate. We ran it on six more 14B rollouts that left the object at its start, and it did not remove the object in any of them. Section 8 now presents the original result as a single example, and no claim depends on it. The claim that the left-behind object is copied from the first image now rests on the attention-blocking experiment instead. At the first denoising step, blocking attention to the object's first-image tokens removes the object from its start in 46 of 71 natural failures, against 3 of 71 when the same number of background tokens is blocked (Sec. 8, App. M.7, M.8).

To summarize what changed since your review, the failure is now measured on 156 validation episodes and three models (47%, 28% and 2% of rollouts for the early 1.3B, fully trained 1.3B and 14B models), there is a direct causal test on natural failures, and there is a training-free correction from the planned path that works on the 14B model's failures. We hope this strengthens your support for the paper, and we are happy to answer further questions.

Reviewer_bTj7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a failure mode of action-conditioned world models for robotics: the manipulated object is left at its initial position, duplicated there, or removed from the prediction. A fine-tuned world model is used for the analysis. The model is conditioned on a control video in which the robot is rendered from the recorded joint states. The main outcome is the observation that the model copies objects forward from the first image and that the outcome is already visible in the first denoising step. Additionally, the paper argues that standard video metrics are not good signals for evaluating action-conditioned robotic world models.

Critical strengths
  • Relevant problem: object tracking matters for downstream tasks and is a common failure.
  • Useful evaluation insight: standard metrics (most clearly temporal-difference error, less so SSIM and LPIPS) are shown to be insufficiently informative for robot world models.
  • Careful causal methodology with matched control interventions.
Critical weaknesses
  • The main problem is the undocumented data and the resulting data-vs-architecture confound. It is not stated which dataset was used for fine-tuning; we only see that the model does not perform well, which suggests it is undertrained. As presented, the analysis cannot distinguish a model failure from a data failure. Ctrl-World does not show the same failure mode (possibly due to its autoregressive re-feeding of frames): objects are deformed but not left behind, as shown in PersistWorld (ECCV 2026). The data source should be made clear, and the analysis should be repeated on an open, large-scale dataset such as DROID to avoid misinterpretation.
  • The number of failure cases is also low; drawing conclusions from 16 failure cases on scenes that look very similar to each other is difficult.
  • Minor: please state the prediction horizon in seconds or give the FPS.
Questions for authors
  • Please describe the dataset used for fine-tuning, and report how the failure rates change across your checkpoints.
  • Please show that your conclusions hold on open, large-scale datasets such as DROID or ABC-130k.
1 comment
AuthorsOpenReview ↗

Response to Reviewer bTj7

Thank you for the review. Your three main concerns were that the data is not documented, that the model may be undertrained, and that the results should hold on open data. We address each below.

Dataset and documentation. Section 3.4 and Table 4 now describe the data. The models are fine-tuned on our own teleoperation dataset of 5,885 episodes (187.8 hours, 29 tasks), split into training and validation at the episode level. Each prediction is 81 frames at 15 fps, which is 5.4 seconds. Table 4 lists the checkpoint behind every result. We should also correct one thing. The 44 episodes used in the submission are from the training split. The 156 new episodes below are from the validation split.

Is the model undertrained? Failure rates across checkpoints. You are right that the submission used an early checkpoint (step 2,000). We repeated the evaluation with the same 1.3B model after one full pass over the training data (step 13,117) and with a 14B model. On 156 validation episodes covering 5 stations and 11 tasks, the object stays at its start position in 47% of rollouts at the early checkpoint, 28% after the full pass, and 2% for the 14B model. So training and scale reduce the failure, but the fully trained 1.3B model still leaves the object behind in more than a quarter of rollouts, and when it does, the failure looks the same (App. M.1).

Is it a data failure or a model failure? We checked this in four ways.

  • The training data does not show objects being left behind. We reviewed 66 grasps in the training videos and the object was carried in 65. A targeted search over 271 gripper-close events found no clear missed grasp.
  • It is not memorization of training clips. The failure rate on validation episodes is similar to or lower than on the training-split episodes (47% against 50% at the early checkpoint, 28% against 39% after the full pass).
  • When the failure occurs, we can trace it inside the model. In 71 rollouts that left the object behind, blocking attention from the generated frames to the first-image tokens that cover the object removes the object from its start position at the first denoising step in 46, against 3 when the same number of background tokens is blocked.
  • Two public video models that never saw our data show the same temporal lookup (next point).

Open data (DROID). We could not fine-tune a world model on DROID or ABC-130k within the discussion period. Instead we tested the central prediction of the paper on open data with public models. We started Wan2.2 and LTX-Video from 32 DROID scenes and shifted the temporal index of the first image. The first-image scene then reappears in the video, and it does so closest to the frame we predicted in 88% and 74% of videos (chance is 33%, since we tested three shift positions). The lookup we describe is therefore a property of these video transformers and does not depend on our dataset (Sec. 6, App. M.5).

Ctrl-World and PersistWorld. We agree that Ctrl-World does not leave objects behind, and we think our account explains why. PersistWorld studies Ctrl-World over many chained clips, where generated frames are fed back as input and objects blur or dissolve over time. Our failure happens inside a single clip that is conditioned on one real first image, and the object stays exactly where that image showed it. Ctrl-World refreshes its reference frames as the video progresses, so its reference does not go out of date. When we fix its reference to the first frame, both the carry and the arm motion stop (Sec. 9). This is what our mechanism predicts, since a refreshed reference removes the outdated copy of the object.

Few and similar failure cases. The 156 validation episodes come from 5 stations and 11 tasks, and the attention-blocking experiment uses 71 natural failures from 23 different episodes.

We hope this addresses the concerns about soundness. If it does, we would be grateful if you would consider updating your score, and we are happy to clarify anything else.

Reviewer_HxetOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks why action-conditioned robot video models render the arm well but often leave the manipulated object behind, duplicate it, or lose it. Through causal interventions in frozen DiTs (mainly a Wan2.1 1.3B control model, with checks on a 14B fine-tune and several public models), the authors argue that the control steers the arm through attention routing, while the object, which appears only in the first image, is copied forward with the RoPE temporal index acting as its address. They further show that the outcome is already visible at the first denoising update and that standard video metrics tend to reward these failures.

Critical strengths
  1. The question matters for planning with world models, and the paper looks for the cause inside the model rather than adding more data or conditioning.

  2. The interventions are done carefully. Each edit is compared with an exact replay and with matched controls (stationary objects, background edits), and the time-index prediction holds in 10 of 10 held-out episodes and in two public video models.

  3. The finding that a clip repeating the first frame beats the generation on temporal-difference error in 98% of 540 cases is a useful warning for evaluation. I also appreciate that negative results are reported.

Critical weaknesses
  1. The sample sizes are small. Many conclusions rest on 8 episodes or on 16 failures and 7 carries, and several key figures come from a single red-bowl episode. Labels are mostly manual, and the second labeler agrees on only 11 of 16 cases.

  2. The remedies are not practical yet. Painting the object into the control fixes 6 of 16 failures and needs ground-truth object pixels in every frame, which a planner does not have. Local re-noising is shown mainly on one 14B sample and is not region-specific at 1.3B. No downstream planning result is given.

  3. Some findings confirm what is already known. The arm/object asymmetry follows from the input design, the RoPE bias toward the first-image position was reported by Li et al. (2026), and early commitment to coarse structure is well known in diffusion models. The new parts should be stated more clearly.

  4. Generality is not fully established. The main experiments use one control design (an arm-only rendered video concatenated at the input), and the evidence for DreamDojo, Cosmos-Predict2 and Ctrl-World is mostly single cases. The main text is also dense and hard to follow.

Questions for authors

See Weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer Hxet

Thank you for the review, and for pointing out that the new parts were not stated clearly enough. We start with that point.

W3. What is new compared with prior work. We agree with your list of what was already known. Image-to-video models over-rely on the conditioning image (conditional image leakage, NeurIPS 2024), rotary embeddings bias attention toward the first image's spatial position (Li et al., 2026), and diffusion models fix coarse structure early. We also agree that the arm/object asymmetry is expected from the input design. What the paper adds:

  • The temporal index of the first image acts as an address. If we shift that one index, the first-image scene reappears at frames that we predict before running the model. This holds in all 10 validation episodes we tested and, new in the revision, in two public models (Wan2.2 and LTX-Video) on 32 DROID scenes, where the scene returns closest to the predicted frame in 88% and 74% of videos (chance 33%). The spatial correction of Li et al. does not remove the left-behind object in our model.
  • This lookup is what produces the natural failure. In 71 rollouts where the model left the object behind on its own, blocking attention from the generated frames to the object's first-image tokens at the first denoising step removes the object from its start position in 46, against 3 when the same number of background tokens is blocked.
  • The failure is invisible to common metrics but can be detected early. A clip that repeats the first frame gets a better temporal-difference error than the generated video in all 440 new rollouts, while the clean video predicted at the first denoising step (2% of the cost) already indicates which rollouts will fail (AUROC 0.82 to 0.91).

We rewrote the abstract and the contribution list to state these three points directly.

W1. Small samples and manual labels. The main numbers no longer come from 8 or 16 cases. Failure rates are now measured on 156 validation episodes that no model was trained on, with three models. The object is left behind in 47% of rollouts at the early checkpoint, 28% after a full training pass, and 2% for the 14B model. The attention-blocking test uses 71 natural failures from 23 episodes, and the time-shift test uses 384 videos from the two public models. All of these are scored by one fixed rule on SAM 2.1 tracks of the object instead of per-video labels. The "11 of 16" figure compared the detailed label of a video with the label of its exact replay, so it shows how noisy the detailed labels were. The automatic rule replaces those labels in the revision, and it agrees with them on whether the object stayed at its start (Cohen's kappa 0.91).

W2. Remedies are not practical yet. We have two new results on this. First, ground-truth object pixels are not needed. A gray silhouette moved along the object's path repairs as many failures as the recorded pixels (29 against 28 of 44), so the model only needs the object's position. Second, we tested a training-free correction that uses only the planned gripper path (adapted from Time-to-Move, ICLR 2026). On the 14B model's own failures (11 rollouts from six episodes) the object leaves its start in 10 and reaches the target in 8, while the same edit on an empty table region changes none. On the fully trained 1.3B model, the automatic tracks show correct carries going from 48% to 66% over the 156 validation episodes. We do not have a closed-loop planning experiment. For planning, what we can offer now is the first-step predictor, which flags likely failures at 2% of the generation cost. The 14B re-noising did not replicate on six more cases and is now presented as a single example (Sec. 5, App. M.9).

W4. Generality and readability. Our own fine-tunes do use one control design. The evidence on other models is no longer single cases. The time-shift prediction is now tested on 32 open DROID scenes with two public models that were never trained on our data (Sec. 6, App. M.5). For readability, the abstract and introduction were rewritten, and several details moved to the appendix.

We hope this makes clearer what is new. If it does, we would be grateful if you would consider updating your score, and we are happy to clarify anything else.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision and the new experiments

We thank all four reviewers for their time and careful reading. The main requests were (1) more episodes and a better-trained model, (2) a direct link between our mechanism and the failures that occur naturally, (3) a fix that does not use ground-truth future frames, and (4) evidence beyond our own data. We ran new experiments for each of these. They are summarized here and described in the new Appendix M; all changes in the revised PDF are in red.

Why we think this is worth studying. Action-conditioned video models are being used as simulators for robots, to compare candidate plans, to evaluate policies and to generate training data. In all three, what matters is where the objects end up. We study a failure in which a video diffusion transformer, given a first image and a rendering of the planned arm motion, moves the arm correctly but leaves the grasped object where it was in the first image. Standard metrics do not penalize this. In all 440 new rollouts, a clip that simply repeats the first frame gets a better temporal-difference error than the generated video. Our explanation is that the object enters the model only through the first image. The generated frames attend back to that image, using its temporal position as an address, and the arm control contains nothing about the object. So after the arm takes the object away, the model keeps drawing it where it last saw it.

New experiments

  1. Larger sample, fully trained model, larger model (Hxet, bTj7, wMC3). We generated rollouts for 156 validation episodes that none of our models were trained on (5 robot stations, 11 tasks) with the 1.3B model at the submission's early checkpoint (step 2,000), the same model after one full pass over the training data (step 13,117), and a 14B model. The object stays at its start position in 47%, 28% and 2% of rollouts. Training and scale make the failure rarer, but the fully trained 1.3B model still shows it in more than a quarter of rollouts (App. M.1).

  2. Link to natural failures (wMC3). In 71 rollouts where the model left the object behind on its own, we re-ran the first denoising step while blocking attention from the generated frames to the first-image tokens that cover the object. In 46 of them the object is no longer at its start position in the clean video the model predicts at that step. Blocking the same number of background tokens does this in 3 (Sec. 7, App. M.7).

  3. What the model is missing, and a correction without future frames (hS4T, wMC3, Hxet). A gray silhouette, or the object's crop from the first frame, moved along the object's path repairs failures as often as the recorded future pixels used in the submission. So the model lacks the object's position over time; its future appearance is not needed. We then tested a training-free correction that uses only what a planner has (the first image, the object mask and the planned gripper path), adapted from Time-to-Move (ICLR 2026). During the first 30 of 50 denoising steps we write the object's first-frame crop into the noisy latent along the planned path. On the 14B model this repairs most of the model's own failures. On the fully trained 1.3B model it raises correct carries from 48% to 66% of rollouts over the 156 validation episodes, as scored by the automatic object tracks (Sec. 5, App. M.3, M.9).

  4. Queries/keys versus values (wMC3). On 54 episodes, restoring only the values of the correct control removes 99% of the error introduced by a corrupted control at the first denoising step, and restoring only the queries and keys removes 84 to 88%. Either one brings the arm back in the finished video and neither brings the object back. We now describe routing as one pathway and no longer as the main one (Sec. 4, App. M.2).

  5. Open data and public models (bTj7, Hxet). We could not fine-tune a world model on DROID in this period, so we tested the central prediction on public models. For Wan2.2 and LTX-Video started from 32 DROID scenes, shifting the temporal index of the first image makes the first-image scene come back, and it comes back closest to the frame we predicted in 88% and 74% of videos (chance 33%) (Sec. 6, App. M.5).

  6. Early versus late (hS4T, wMC3). The clean video predicted at the first denoising step, which costs 2% of a full generation, already indicates which rollouts will leave the object behind (AUROC 0.82 for the fully trained 1.3B model, 0.91 for the 14B). This gives a cheap way to flag failing rollouts. The correction from point 3 also does not need the whole denoising process. It is applied in the first 30 of 50 steps and its effect grows with the window. On 32 failed runs of the early 1.3B checkpoint, the object stays at its start in 24 without it, in 13 with steps 1 to 15 and in 5 with steps 1 to 30 (Sec. 7, App. M.6, M.9).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.