- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents WorldCraft, a framework for persistent object-trajectory control in camera-navigable autoregressive video world models. Built on WorldPlay, the method introduces Normalized World Trajectory (NWT) to convert user-specified 2D object trajectories into camera-aware control signals across viewpoints and generation chunks. To handle conflicts between historical visual memory and updated object positions, the authors propose Trajectory-Anchored State Persistence (TASP), which combines persistent trajectory guidance with selective memory filtering. The model is adapted using Spatial-Pathway LoRA to preserve the pretrained camera-control capability. Experiments demonstrate joint camera-object control and improved object-state consistency after extended periods of invisibility.
Critical strengths
- The paper identifies an important limitation of existing autoregressive video world models: historical visual memory may conflict with object states updated by new actions, particularly when objects move outside the camera view. This problem is relevant to interactive world models and goes beyond conventional fixed-window trajectory control. The proposed formulation of persistent object-state control is clear and addresses a meaningful gap in existing methods.
- The proposed framework combines camera navigation and object-trajectory control while largely preserving the capabilities of the pretrained backbone. In particular, the off-screen state experiments provide direct evidence for the effectiveness of TASP. Compared with default memory handling, TASP reduces the final re-entry error by 27.2% and improves object identity consistency. The ablations on trajectory representation and Spatial-Pathway LoRA also support the main design choices. These results make the proposed approach practically useful despite its relatively simple components.
Critical weaknesses
- TASP primarily relies on filtering historical memory that may conflict with the updated object trajectory. While the results demonstrate its effectiveness, the paper only compares TASP with the default WorldPlay memory. It remains unclear whether simpler memory management strategies, such as sliding-window memory or removing a comparable number of recent memory frames, could achieve similar improvements. Such comparisons would better establish the value of the proposed selective filtering strategy.
- The evaluation of persistent object-state control is relatively limited. The off-screen benchmark contains 50 samples with mirrored camera-away-and-return trajectories, and the main evaluation considers one or four consecutive invisible chunks. While these experiments demonstrate the proposed capability, they do not fully establish its robustness under more general camera movements, repeated object disappearances, or longer generation horizons. The evaluation therefore supports persistent trajectory-conditioned control in the tested setting, but leaves its generalization to more complex interactions uncertain.
Questions for authors
- How does TASP compare with simpler memory management baselines? For example, would sliding-window memory or removing the same number of recent memory frames achieve similar improvements? This comparison would help clarify the contribution of the selective memory filtering strategy.
- How robust is TASP under more complex camera trajectories? In particular, does it remain effective when the camera follows a non-mirrored return path or when the controlled object disappears and reappears multiple times?
- How is iterative depth refinement handled when the controlled object is fully off-screen? The paper describes updating the anchor frame and depth estimate after each generation chunk, but the target object may not be observable during these intervals. Clarifying how the depth and trajectory state are maintained in this case would help establish the reliability of NWT.
5 comments
Author Responses (2/2)
Scope. These additions test more varied return paths and two disappearance cycles within the same 253-frame horizon; longer videos and additional cycles remain to be evaluated. The duration study in Table 4(a) also reports a limitation: increasing consecutive invisibility from one to four chunks raises final re-entry error from 32.70 to 35.11 and lowers identity similarity from 0.805 to 0.717. We therefore limit the persistence claim to the tested paths, absence durations, and cycle counts. Here, zero-shot refers to transferring control learned from visible trajectories to updates of fully invisible targets, a regime absent from trajectory-control training.
Q3. Depth refinement and trajectory maintenance during full invisibility.
Depth is used to lift the user's 2D path into a 3D trajectory (Eq. 2), and each lift already defines that trajectory over the entire user-specified path, which NWT then re-projects under subsequent camera poses (Eq. 3). The per-chunk anchor refresh in §3.3 replaces the current 3D trajectory with one re-lifted from the updated anchor-depth pair; during invisibility there is no new observation, so the previous lift carries forward, and it already covers the off-screen interval.
During visible segments, the latest generated chunk provides a target observation; NWT updates the anchor to that chunk, re-estimates the object's depth, and re-lifts the user path. The updated 3D trajectory replaces the previous one over the remaining user-specified path, forming the closed-loop correction described in §3.3.
During invisibility, there is no new target observation to update the anchor from, so the last refreshed anchor-depth pair stays in force and the 3D trajectory it produced continues to be re-projected under each supplied camera pose via Eq. 3. The off-screen time steps were already covered by that trajectory at lift time, so nothing further needs to be computed on the target during the interval. If the projected position falls behind the camera or outside the image, the object action contributes no spatial condition on that chunk while its time index keeps advancing.
At re-entry, the current object action is re-projected into the returning view from the retained 3D trajectory; the re-entry chunk is generated with the retained appearance reference and TASP-filtered memory. Once the generated video provides a fresh observation of the target, the per-chunk anchor update and re-lift resume as in the visible regime.
Reliability. Because each lift already defines the 3D trajectory over the entire user-specified path, invisibility can only delay the next anchor refresh, not corrupt the trajectory that NWT is projecting. The Composable results in Table 5 isolate the value of iterative refinement under visible trajectories: TE decreases from 41.28 to 34.65 in the large-rotation bin relative to a single-shot lift. The OSS results in Table 4 and Table R4 then show that, across up to four fully invisible chunks and the extended protocols above, the full system maintains target position (RTE_f reductions of 23.4–27.2% over default memory) and identity (RID_pre gains of 0.092–0.099). We will clarify the span of each lift and the anchor-update behavior during invisibility in §3.3 and Algorithm 1.
We sincerely thank the reviewer for the constructive feedback. We hope our response has addressed your concerns. As the discussion deadline is approaching its end, we wanted to send a gentle final reminder. We fully respect your judgment and simply want to ensure no unresolved concern remains. Please let us know if anything further would be helpful.
Author Responses (1/2)
Thank you for recognizing the action–memory conflict and the value of persistent object control. We have added the two suggested memory baselines and extended the off-screen evaluation to non-mirrored returns and repeated disappearances. Below, we report these results and clarify how NWT maintains its trajectory and depth references during full invisibility.
Q1. Comparison with simpler memory-management baselines.
We add sliding-window memory and removal of recent memory frames to the OSS comparison, with the adapted model and NWT trajectory guidance fixed. The sliding window drops WorldPlay's longer-range spatial memory and keeps a fixed number of the most recent frames, matching TASP's retained memory-latent count. Recent-frame removal keeps WorldPlay's full memory structure and only ablates a fixed number of the most recent frames, matching TASP's average removal count. Both policies therefore control for the memory budget and isolate the effect of which context is retained.
Table R3. Memory-policy comparison on OSS: 50 rollouts, 253 frames, four consecutive invisible chunks. The default-memory and TASP rows reproduce Table 4(b).
Both simpler policies improve re-entry error and identity similarity over default memory, but TASP performs best on all six reported metrics. Its final re-entry error is 21.3% lower than sliding-window memory and 25.5% lower than recent-frame removal, with higher identity similarity (0.717 vs. 0.661 and 0.642). Relative to default memory, the re-entry error reduction is 27.2%.
The background metrics further distinguish the policies. Sliding-window retrieval improves target re-entry but increases MEt3R from 0.202 to 0.259 and DepthC from 0.0765 to 0.0927; TASP achieves 0.196 and 0.0755, respectively. TASP filters memories that are both close to the target's exit and similar in viewpoint to the returning view (§3.4), retaining other appearance and scene context. These results support the value of selecting potentially conflicting memories: the matched simpler policies yield smaller object-control gains and worse background consistency.
Q2. Non-mirrored returns, repeated disappearances, and longer horizons.
We extend OSS with non-mirrored camera-and-object returns and two exit-and-return cycles, with 50 rollouts of 253 frames per protocol. The non-mirrored camera path revisits overlapping scene regions without exactly retracing the outbound trajectory. The non-mirrored protocol uses four consecutive invisible chunks; the repeated-exit protocol uses two invisible chunks per cycle. Each protocol compares default memory with TASP.
Table R4. Extended off-screen evaluation. Each metric cell reports default memory / TASP. The mirrored-return row reproduces Table 4(b). For repeated exits, RTEf and RIDpre are evaluated at the end of the second (final) re-entry segment and averaged over the 50 rollouts. RIDpre uses a fixed target reference from before the first exit.
At each re-entry, NWT projects the current trajectory state under the returning camera pose. TASP applies its temporal and view-overlap criteria to the pre-exit memories; for repeated exits, it records a pre-exit zone at each departure and filters qualifying frames preceding the most recent exit. The same procedure thus handles successive exit-and-return events without modification.
For non-mirrored returns, TASP reduces final re-entry error from 49.41 to 36.09 (27.0%) and improves identity similarity from 0.611 to 0.710. After two exit-and-return cycles, final re-entry error decreases from 52.12 to 39.93 (23.4%), while identity similarity to the target before the first exit increases from 0.552 to 0.644. Trajectory error and background MEt3R also improve over default memory in both protocols, extending the evidence beyond a single mirrored return.