- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates object persistence errors in robot video world models, including objects being left behind, duplicated, or disappearing. Through attention restoration, temporal position shifts, and latent-state interventions, the authors examine how action conditioning and reference content influence these failures. They find that restoring Q/K can recover arm motion, changing the reference image’s temporal index causes the initial scene to reappear at predictable times, and some residual object copies persist in the partially denoised latent state. The paper addresses an important problem and presents several compelling interventions, although the connection between these findings and naturally occurring object errors needs further clarification.
Critical strengths
-
Correct arm motion does not guarantee correct object interaction. The paper demonstrates this distinction clearly and shows how standard video metrics can miss task-relevant failures.
-
Predicting the frame window in which the initial scene reappears, then validating it on ten held-out episodes, provides stronger evidence than attention visualization alone.
-
Exact native replays, background-region interventions, and stationary-object controls help interpret the effects. The authors also report unsuccessful interventions and differences across models.
Critical weaknesses
-
The role of attention routing is somewhat overstated. Restoring Q/K removes much of the error introduced by corrupted control, but restoring V also recovers arm motion and performs better in some layers. This supports routing as an effective pathway, but does not establish that it is the primary pathway.
-
Temporal addressing is not yet directly linked to native object failures. Moving the reference image’s temporal index demonstrates control over when its content reappears. It is less clear whether the same process causes duplication under normal inputs, particularly since cutting reference reads does not remove existing copies.
-
Object painting uses privileged future information. Recorded object pixels provide future position, appearance, and occlusion information together. The 6/16 repairs are informative, but do not establish which information matters or how comparable conditioning would be obtained during planning.
-
The tasks mainly involve pick-and-place and stacking. In contact-rich tasks such as insertion, tight-fitting assembly, or constrained pushing, similar arm motions can produce different outcomes because of contact, friction, or alignment. Objects may remain visually consistent while the predicted interaction is physically wrong. It is unclear how well the proposed account explains these failures.
-
Some quantitative results need stronger support. Identical native and replay videos received matching detailed labels in only 11/16 cases. Although these disagreements did not affect carry labels, the annotation procedure needs clarification. The central 14B re-noising result also comes from a single example.
Questions for authors
-
How do Q/K-only, V-only, and joint restoration compare? Do routing and value pathways provide redundant control information?
-
On the same native failure cases, would early versus late interventions help distinguish the formation of an object copy from its subsequent persistence?
-
How would painting the initial object appearance along a planned trajectory compare with painting recorded future pixels? Could such conditioning produce a false successful carry when the actual grasp fails?
-
Could the authors include one or two contact-rich tasks, ideally with similar actions but different successful and failed outcomes? In particular, does object conditioning preserve collisions and jamming, or encourage the model to generate the intended motion despite those constraints?
-
How many annotators assessed the videos, and how were disagreements resolved? Can the 14B re-noising result be repeated on a few additional cases while reporting both copy removal and damage to otherwise correct scene content?
Response to Reviewer wMC3
Thank you for the detailed review and the specific questions, which decided most of the experiments we ran. We answer them in order of importance.
W2 / Q2. Is the temporal lookup linked to natural failures? Early versus late. We took 71 rollouts in which the model left the object at its start with unmodified inputs (44 from the early checkpoint, 27 from the fully trained one). For each we re-ran the first denoising step with the same noise, but blocked attention from the generated frames to the first-image tokens that cover the carried object. In 46 of 71 runs the object disappears from its start position in the clean video predicted at that step. Blocking the same number of background tokens does this in 3 of 71. So in natural failures the copy at the start comes from attention to the object in the first image.
For early versus late we looked at finished videos, blocking only the tokens at the object's start position after the grasp. With the block on for all 50 steps, the start position is empty at the end in 41 of 71 runs, against 24 of 71 for the background block. With the block on only for steps 1 to 5 there is no difference (26 vs 23). The copy appears at the first step, but the model keeps re-drawing it from the first image at later steps. The effect at the first step is also much larger when all generated frames are blocked than when only post-grasp frames are (46 vs 19 of 71), which is probably why the post-pickup cuts in the submission left copies in place (Sec. 7, App. M.7).
W1 / Q1. Q/K-only, V-only and joint restoration. We agree that the submission overstated routing. On 54 episodes we corrupted the control and restored the values, the queries and keys, or all three from the run with the correct control. At the first denoising step, V alone removes 99% of the error added by the corruption and Q/K alone 84 to 88%; all three remove all of it. The two pathways are largely redundant. In finished videos either one restores the arm and neither restores the object. Section 4 now says that routing is one effective pathway, and that the control does not carry the object through either one (App. M.2).
W3 / Q3. Privileged information, planned trajectories, false carries. We first checked which information matters. On 44 failures of the fully trained model, painting a gray silhouette or the object's first-frame crop along the recorded path empties the start position as often as painting the recorded pixels (29 and 27 runs, against 28), so position is what the model needs.
For the planned trajectory, we follow Time-to-Move (ICLR 2026) and write the object's first-frame crop along the planned gripper path into the noisy latent during the first 30 of 50 denoising steps. The 14B model rarely fails, so we used all of its failures that can be scored, which are 11 rollouts from six episodes (one episode fails with all six seeds). With the correction the object leaves its start in 10 of the 11 and reaches the target in 8. The same edit on an empty table region changes none of them. On the 1.3B model the automatic tracks show correct carries going from 48% to 66% over the 156 validation episodes. The same edit on an empty table region again has no effect (Sec. 5, App. M.9).
False carries can happen. In one constructed case where we edited the arm path to turn back while the gripper still closes, the model drew a held plate in all three seeds. Ctrl-World also shows a carry after a recorded failed grasp in 3 of 8 DROID episodes. So object conditioning has to come from a prediction of whether the grasp succeeds, which the paper now states.
W4 / Q4. Contact-rich tasks. We agree. Our data contains only pick-and-place and stacking, and the discussion now limits the claims to lift-and-carry tasks.
W5 / Q5. Annotation; re-noising. The "11 of 16" figure compared the detailed label of a video with the label of its exact replay, that is, the same video labeled twice, so we agree the detailed labels in the submission were noisy. All results in the revision are instead scored by a fixed rule on SAM 2.1 tracks (is the object's mask still at its start position in the last frames), which involves no annotator judgment. It agrees with the submission's labels on that question (Cohen's kappa 0.91). The 14B re-noising did not replicate. On six more 14B failures it did not remove the object, so it is now presented as a single example.
We hope this addresses W1 to W3 in particular. If it does, we would be grateful if you would consider updating your score, and we are happy to answer further questions.