acceptodds
Under review as a conference paper at ICLR 2027

WorldCraft: Persistent Object-State Control in Camera-Navigable Video World Models

Abstract

Recent video world models enable interactive camera navigation but provide limited control over individual objects. We study persistent object-state control in autoregressive video world models, where an object-trajectory action must remain effective across viewpoint changes, generation chunks, and intervals of temporary invisibility. This setting introduces an action-memory conflict because autoregressive memory records the last observed object state, whereas an intervening action may update that state off-screen. We present WorldCraft, the first camera-navigable autoregressive video world model to jointly compose camera navigation and object-trajectory control. Given a user-selected object and a 2D motion path, Normalized World Trajectory (NWT) anchors the path in a normalized world coordinate system and projects it into each camera view, producing a consistent object-action signal throughout the autoregressive rollout. Trajectory-Anchored State Persistence (TASP) combines persistent NWT guidance with selective memory refresh to maintain action-updated object states through off-screen intervals and re-entry. A pathway-selective LoRA introduces object control while retaining the pretrained camera controller. WorldCraft achieves accurate composable camera-object control and preserves camera fidelity. We further find that it maintains action-updated object states across extended intervals of complete target invisibility without off-screen trajectory supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interactive Video World Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VqbrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents WorldCraft, a framework for persistent object-trajectory control in camera-navigable autoregressive video world models. Built on WorldPlay, the method introduces Normalized World Trajectory (NWT) to convert user-specified 2D object trajectories into camera-aware control signals across viewpoints and generation chunks. To handle conflicts between historical visual memory and updated object positions, the authors propose Trajectory-Anchored State Persistence (TASP), which combines persistent trajectory guidance with selective memory filtering. The model is adapted using Spatial-Pathway LoRA to preserve the pretrained camera-control capability. Experiments demonstrate joint camera-object control and improved object-state consistency after extended periods of invisibility.

Critical strengths
  1. The paper identifies an important limitation of existing autoregressive video world models: historical visual memory may conflict with object states updated by new actions, particularly when objects move outside the camera view. This problem is relevant to interactive world models and goes beyond conventional fixed-window trajectory control. The proposed formulation of persistent object-state control is clear and addresses a meaningful gap in existing methods.
  2. The proposed framework combines camera navigation and object-trajectory control while largely preserving the capabilities of the pretrained backbone. In particular, the off-screen state experiments provide direct evidence for the effectiveness of TASP. Compared with default memory handling, TASP reduces the final re-entry error by 27.2% and improves object identity consistency. The ablations on trajectory representation and Spatial-Pathway LoRA also support the main design choices. These results make the proposed approach practically useful despite its relatively simple components.
Critical weaknesses
  1. TASP primarily relies on filtering historical memory that may conflict with the updated object trajectory. While the results demonstrate its effectiveness, the paper only compares TASP with the default WorldPlay memory. It remains unclear whether simpler memory management strategies, such as sliding-window memory or removing a comparable number of recent memory frames, could achieve similar improvements. Such comparisons would better establish the value of the proposed selective filtering strategy.
  2. The evaluation of persistent object-state control is relatively limited. The off-screen benchmark contains 50 samples with mirrored camera-away-and-return trajectories, and the main evaluation considers one or four consecutive invisible chunks. While these experiments demonstrate the proposed capability, they do not fully establish its robustness under more general camera movements, repeated object disappearances, or longer generation horizons. The evaluation therefore supports persistent trajectory-conditioned control in the tested setting, but leaves its generalization to more complex interactions uncertain.
Questions for authors
  1. How does TASP compare with simpler memory management baselines? For example, would sliding-window memory or removing the same number of recent memory frames achieve similar improvements? This comparison would help clarify the contribution of the selective memory filtering strategy.
  2. How robust is TASP under more complex camera trajectories? In particular, does it remain effective when the camera follows a non-mirrored return path or when the controlled object disappears and reappears multiple times?
  3. How is iterative depth refinement handled when the controlled object is fully off-screen? The paper describes updating the anchor frame and depth estimate after each generation chunk, but the target object may not be observable during these intervals. Clarifying how the depth and trajectory state are maintained in this case would help establish the reliability of NWT.
5 comments
AuthorsOpenReview ↗

Author Responses (1/2)

Thank you for recognizing the action–memory conflict and the value of persistent object control. We have added the two suggested memory baselines and extended the off-screen evaluation to non-mirrored returns and repeated disappearances. Below, we report these results and clarify how NWT maintains its trajectory and depth references during full invisibility.

Q1. Comparison with simpler memory-management baselines.

We add sliding-window memory and removal of recent memory frames to the OSS comparison, with the adapted model and NWT trajectory guidance fixed. The sliding window drops WorldPlay's longer-range spatial memory and keeps a fixed number of the most recent frames, matching TASP's retained memory-latent count. Recent-frame removal keeps WorldPlay's full memory structure and only ablates a fixed number of the most recent frames, matching TASP's average removal count. Both policies therefore control for the memory budget and isolate the effect of which context is retained.

Table R3. Memory-policy comparison on OSS: 50 rollouts, 253 frames, four consecutive invisible chunks. The default-memory and TASP rows reproduce Table 4(b).

Memory policyTEmean ↓RTEf ↓RIDpre ↑MEt3R ↓DepthC ↓RPErot ↓
Default WorldPlay memory52.1048.200.6180.2020.07650.139
Sliding window (matched retained count)47.3244.610.6610.2590.09270.142
Remove recent frames (matched average removal count)51.9347.150.6420.2050.07710.168
TASP40.0335.110.7170.1960.07550.137

Both simpler policies improve re-entry error and identity similarity over default memory, but TASP performs best on all six reported metrics. Its final re-entry error is 21.3% lower than sliding-window memory and 25.5% lower than recent-frame removal, with higher identity similarity (0.717 vs. 0.661 and 0.642). Relative to default memory, the re-entry error reduction is 27.2%.

The background metrics further distinguish the policies. Sliding-window retrieval improves target re-entry but increases MEt3R from 0.202 to 0.259 and DepthC from 0.0765 to 0.0927; TASP achieves 0.196 and 0.0755, respectively. TASP filters memories that are both close to the target's exit and similar in viewpoint to the returning view (§3.4), retaining other appearance and scene context. These results support the value of selecting potentially conflicting memories: the matched simpler policies yield smaller object-control gains and worse background consistency.

Q2. Non-mirrored returns, repeated disappearances, and longer horizons.

We extend OSS with non-mirrored camera-and-object returns and two exit-and-return cycles, with 50 rollouts of 253 frames per protocol. The non-mirrored camera path revisits overlapping scene regions without exactly retracing the outbound trajectory. The non-mirrored protocol uses four consecutive invisible chunks; the repeated-exit protocol uses two invisible chunks per cycle. Each protocol compares default memory with TASP.

Table R4. Extended off-screen evaluation. Each metric cell reports default memory / TASP. The mirrored-return row reproduces Table 4(b). For repeated exits, RTEf and RIDpre are evaluated at the end of the second (final) re-entry segment and averaged over the 50 rollouts. RIDpre uses a fixed target reference from before the first exit.

ProtocolTEmean ↓RTEf ↓RIDpre ↑MEt3R ↓
Mirrored return (4 invisible chunks)52.10 / 40.0348.20 / 35.110.618 / 0.7170.202 / 0.196
Non-mirrored return (4 invisible chunks)52.17 / 40.3249.41 / 36.090.611 / 0.7100.219 / 0.205
Repeated exits (2 cycles, 2 invisible chunks each)56.31 / 43.2952.12 / 39.930.552 / 0.6440.257 / 0.223

At each re-entry, NWT projects the current trajectory state under the returning camera pose. TASP applies its temporal and view-overlap criteria to the pre-exit memories; for repeated exits, it records a pre-exit zone at each departure and filters qualifying frames preceding the most recent exit. The same procedure thus handles successive exit-and-return events without modification.

For non-mirrored returns, TASP reduces final re-entry error from 49.41 to 36.09 (27.0%) and improves identity similarity from 0.611 to 0.710. After two exit-and-return cycles, final re-entry error decreases from 52.12 to 39.93 (23.4%), while identity similarity to the target before the first exit increases from 0.552 to 0.644. Trajectory error and background MEt3R also improve over default memory in both protocols, extending the evidence beyond a single mirrored return.

AuthorsOpenReview ↗

Author Responses (2/2)

Scope. These additions test more varied return paths and two disappearance cycles within the same 253-frame horizon; longer videos and additional cycles remain to be evaluated. The duration study in Table 4(a) also reports a limitation: increasing consecutive invisibility from one to four chunks raises final re-entry error from 32.70 to 35.11 and lowers identity similarity from 0.805 to 0.717. We therefore limit the persistence claim to the tested paths, absence durations, and cycle counts. Here, zero-shot refers to transferring control learned from visible trajectories to updates of fully invisible targets, a regime absent from trajectory-control training.

Q3. Depth refinement and trajectory maintenance during full invisibility.

Depth is used to lift the user's 2D path into a 3D trajectory (Eq. 2), and each lift already defines that trajectory over the entire user-specified path, which NWT then re-projects under subsequent camera poses (Eq. 3). The per-chunk anchor refresh in §3.3 replaces the current 3D trajectory with one re-lifted from the updated anchor-depth pair; during invisibility there is no new observation, so the previous lift carries forward, and it already covers the off-screen interval.

During visible segments, the latest generated chunk provides a target observation; NWT updates the anchor to that chunk, re-estimates the object's depth, and re-lifts the user path. The updated 3D trajectory replaces the previous one over the remaining user-specified path, forming the closed-loop correction described in §3.3.

During invisibility, there is no new target observation to update the anchor from, so the last refreshed anchor-depth pair stays in force and the 3D trajectory it produced continues to be re-projected under each supplied camera pose via Eq. 3. The off-screen time steps were already covered by that trajectory at lift time, so nothing further needs to be computed on the target during the interval. If the projected position falls behind the camera or outside the image, the object action contributes no spatial condition on that chunk while its time index keeps advancing.

At re-entry, the current object action is re-projected into the returning view from the retained 3D trajectory; the re-entry chunk is generated with the retained appearance reference and TASP-filtered memory. Once the generated video provides a fresh observation of the target, the per-chunk anchor update and re-lift resume as in the visible regime.

Reliability. Because each lift already defines the 3D trajectory over the entire user-specified path, invisibility can only delay the next anchor refresh, not corrupt the trajectory that NWT is projecting. The Composable results in Table 5 isolate the value of iterative refinement under visible trajectories: TE decreases from 41.28 to 34.65 in the large-rotation bin relative to a single-shot lift. The OSS results in Table 4 and Table R4 then show that, across up to four fully invisible chunks and the extended protocols above, the full system maintains target position (RTE_f reductions of 23.4–27.2% over default memory) and identity (RID_pre gains of 0.092–0.099). We will clarify the span of each lift and the anchor-update behavior during invisibility in §3.3 and Algorithm 1.

We sincerely thank the reviewer for the constructive feedback. We hope our response has addressed your concerns. As the discussion deadline is approaching its end, we wanted to send a gentle final reminder. We fully respect your judgment and simply want to ensure no unresolved concern remains. Please let us know if anything further would be helpful.

Reviewer_MoD6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

WorldCraft tackles object movement control in interactive video world models, where a user can control camera view point in the generated video in an interactive way, and can input a trajectory for some selected objects in the beginning of the simulation. The method takes care of change in camera position and rotation, translating the initial user-specified trajectory by warping given the current camera position. It also tries to improve the memory by the trajectory-ancored state persistence module, effectively handling objects that get out of field of view and return and their movement while they were gone thanks to the normalized world trajectory (the camera-pose-based warping approach).

Critical strengths

The use case is interesting, as sometimes users might need to specify object movements, specially in robotic-based world models where one could use such controls to compose interesting scenarios by moving other objects in desirable locations, e.g. making a pedestrian quickly jump in front of a car to evaluate how well a policy would react to such cases.

Critical weaknesses

One major drawback of the method is its novelty. Although authors correctly mention that previous works have mostly addressed this problem in a fixed-generation setup, the technique to achieve object control has been proposed a couple of years ago, for example in [1] and some works before it (which the paper fails to cite), where they copy the DiNO features of the object of interest and move it according to the required trajectory. This makes the contribution of the paper to simply be the trajectory warping based on camera position, which is a small contribution, and the TASP method for handling the niche situations in which an object would leave the field of view and come back. These contributions, in my opinion, are not enough for a first tier conference like ICLR.

Secondly, although the metrics and benchmarks implemented seem convincing as a proof of concept, the type of data they are evaluated on (and also shown as qualitative examples) are not interesting enough; world models are mostly developed and used for robotics/self driving cars, rather than for looking at a swan, turning left and right and observing if the swan continued its desired movement. As said, while interesting as a proof of concept, some downstream use cases or qualitative examples from robotics/driving domain are a necessity in my opinion. And in general the qualitative examples are quite limited and unsatisfying; I believe the methods used to compare with are not the strongest in the field, as the generation qualities are often very low and it's not a fair comparison.

[1] GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control

Questions for authors
  1. Could you include some robotics/driving subsets for evaluation, create relevant scenarios there, and evaluate your approach on such scenarios, potentially show how WorldCraft could be utilized as a more useful world model?

  2. A note on inference speed and the overhead added by your approach, specially the depth re-calculations seem missing from the paper, as preserving interactive generation was one of the claims of the paper (comparing to other fixed-horizon generation methods).

7 comments
AuthorsOpenReview ↗

Author Responses (1/3)

We thank the reviewer for the detailed feedback and for highlighting scenario construction as a useful application. We have added stronger baselines, robotics and driving examples, and inference profiling, and clarify the contribution relative to GEM below. Table and figure references refer to the revised paper unless prefixed with R.

W1. Novelty and relation to GEM.

We focus on camera-navigable autoregressive video world models. Recent work in this space (WorldPlay, ICML 2026; GameCraft, SIGGRAPH Asia 2025; Matrix-Game 2.0/3.5, 2025–2026; Genie 3, 2025) targets sub-problems such as long-horizon memory and scene persistence, camera control in streaming rollouts, and interactive world-model backbones for streaming generation. One capability in this landscape remains open: persistent object-state control under camera navigation. No prior autoregressive video world model exposes a user-level action that updates an object's state while the object is off-screen (Table 1). This is the problem WorldCraft addresses. Wan-Move (NeurIPS 2025), DragAnything, and GEM (CVPR 2025) are prior object-control methods we have cited and compared (see below); we do not claim a new object-control technique over them. Our contribution lies at the composable-control regime level: carrying object control into the camera-navigable autoregressive setting raises three distinct challenges that NWT, SP-LoRA, and TASP each address.

Challenges in this regime.

  1. Camera and object signals conflict in screen-space inputs. Camera motion passively displaces every object across the frame, which conflicts with the user's intended object trajectory in the same 2D plane. Prior fixed-window separate-pathway designs (e.g., MotionCtrl (SIGGRAPH 2024)) separate the camera branch and the object branch inside the model, but do not verify that the composition of the two signals is actually reproduced by the model in the output video. A video world model needs to simulate downstream scenarios faithfully, so our composable control instead makes the two signals geometrically consistent in a shared frame before the model sees them.
  2. Autoregressive memory conflicts with an off-screen object-state update. An autoregressive video world model generates each chunk by retrieving earlier visual memory as a scene prior. When a user action moves an object off-screen and continues updating its state, memory stays anchored to the pre-exit position; on re-entry the model must render the object at the action-updated location, not restore the stale one in memory (paper §1, §3.4). We call this the action-memory conflict. It arises only because a user action keeps advancing the state during invisibility, and no prior autoregressive world model exposes such an off-screen object action (Table 1).
  3. The new object action must not overwrite the inherited camera-navigation capability. The pretrained backbone's spatial computations already encode camera following. Broadly re-training with object-trajectory supervision couples the new action into those same parameters and competes with the camera signal, displacing the inherited behavior. The adaptation must inject the object action where it belongs in the pretrained pathway, not re-fit the backbone as a whole.

Our innovation. Normalized World Trajectory (NWT) is the representation choice that decouples camera and object signals at the input: a world-anchored 3D action representation shared by camera and object, iteratively re-projected across autoregressive chunks. From this choice follow two capabilities that fixed-window separate-pathway methods cannot reach from the same inputs: Spatial-Pathway LoRA (SP-LoRA), which lets a single pretrained spatial pathway follow both camera and object actions with a 0.6% input-side adapter; and Trajectory-Anchored State Persistence (TASP), which maintains an action-updated object state through full invisibility, zero-shot with respect to off-screen supervision. Each targets one challenge and is isolated by an existing ablation.

AuthorsOpenReview ↗

Author Responses (2/3)

  1. NWT (Challenge 1). NWT lifts the user's 2D path into a normalized 3D world coordinate system and re-projects it under each camera pose, with anchor and depth refined across autoregressive chunks. A stationary object then becomes a static world-frame 3D point whose image projection moves with the camera by construction, consistent with the camera signal. Camera motion (world-frame scene displacement) and object motion (world-frame instance displacement) then describe the same kind of spatial computation in one shared action space (Table 5). The "simply trajectory warping" framing therefore misses NWT's world-anchored decoupling of the two signals.
  2. TASP (Challenge 2). Because NWT keeps advancing the action-updated position in 3D regardless of visibility, the model already knows where the object should be while fully off-screen. TASP asserts this position at every chunk and filters only the pre-exit memory latents whose FoV overlaps the current chunk. The result is zero-shot with respect to off-screen supervision: no training trajectory moves a controlled target while fully off-screen, yet the action-updated state persists through up to four invisible chunks. On OSS, sliding-window and recent-frame removal give 44.61/0.661 and 47.15/0.642, both worse than TASP's 35.11/0.717 (Table 4(b), R3). Reviewer Vqbr (rating 3) independently identifies this action-memory gap and credits the OSS experiments as "direct evidence for the effectiveness of TASP"; zero-shot persistence through four invisible chunks on a 50-clip benchmark is a first-class capability enabled by NWT, not a niche application.
  3. SP-LoRA (Challenge 3). Because NWT places camera and object actions in one 3D coordinate system, a single pretrained spatial pathway can follow both. We therefore share that pathway and adapt only action_in and prope_proj, supported by Appendix C's weight-change concentration (10-25× larger here than on attention/MLP) and Fig. 8's activation overlap. The 50M (0.6%) adapter reaches TE/RPErot of 38.90/0.131, matching full fine-tuning's object accuracy while preserving camera fidelity (Table 8). Reviewer SgYy (rating 3) cites this joint-control capability as "the main reason for my positive recommendation".

On GEM as a direct baseline. We do include GEM. Figure 11 evaluates GEM in the camera-only setting its released interface supports; Figure 12 then shows WorldCraft additionally steering an object jointly with the camera, in the driving scenes highlighted by the reviewer. A quantitative joint-control comparison is not available because GEM's released code routes camera and object conditioning through mutually exclusive branches at inference. We will cite GEM among the prior object-control methods.

W2 / Q1. Stronger baselines and robotics/driving applications.

Stronger quantitative comparisons. We expanded Table 3 (reproduced below) with three recent camera-navigable baselines, Matrix-Game 3.5 (Qian et al., 2026), LingBot-World 2 (Gao et al., 2026), and DreamX-World (Team et al., 2026), evaluated at both 61 and 253 frames under the same camera-action sequences.

Table 3 (revised paper). Camera fidelity on camera-only input.

MethodRPErot (61f)↓RPEtrans (61f)↓RPEcam (61f)↓PSNR↑SSIM↑LPIPS↓RPErot (253f)↓RPEtrans (253f)↓RPEcam (253f)↓
Yume0.2610.01430.016912.390.29310.57180.3740.02470.0285
Matrix-Game 2.00.3420.01370.019612.960.32350.53260.1620.02430.0261
GameCraft0.2520.01300.015712.420.28610.55290.1980.02430.0265
WorldPlay0.1200.01550.016513.770.34340.47000.1300.02620.0276
Matrix-Game 3.50.1220.01390.014613.840.34030.47080.1270.02410.0248
LingBot-World 20.1360.01370.014713.420.33460.48420.2010.02650.0282
DreamX-World0.1520.01460.015212.970.31710.53260.2430.02290.0253
WorldCraft (ours)0.1310.01610.017013.950.34740.46210.1230.02250.0233

At 253 frames, WorldCraft achieves the lowest rotational, translational, and combined RPE (0.123/0.0225/0.0233) among all seven compared methods. At 61 frames, its rotational RPE (0.131) stays close to WorldPlay's 0.120 and Matrix-Game 3.5's 0.122, while WorldCraft retains the highest PSNR (13.95), SSIM (0.3474), and the lowest LPIPS (0.4621). Table 2 separately compares trajectory control using identical initial frames and target paths, where WorldCraft also obtains the lowest TE. The comparison thus includes competitive recent baselines and reports their strengths on the control axes they support.

AuthorsOpenReview ↗

Author Responses (3/3)

Robotics: quantitative transfer and new qualitative examples. Appendix D.2, Table 12 evaluates 50 held-out DROID manipulation videos of 125 frames each, without DROID-specific adaptation. WorldCraft reduces TE from Wan-Move's 45.39 to 37.77 and improves DINO similarity from 0.7011 to 0.7970, while also controlling the camera. Its rotational RPE is 0.127, compared with WorldPlay's 0.145. We added three manipulation examples in Appendix D.3, Figures 9–10, showing more consistent arm appearance during simultaneous viewpoint changes and object motion. These examples complement the quantitative results in the robotics domain requested by the reviewer.

Driving: comparison with GEM and scenario construction. Appendix D.4 adds three camera-only driving scenes (Figure 11) and one joint camera–motorcycle example (Figure 12). On GEM's released evaluation code, camera and object conditioning are routed through mutually exclusive branches and cannot be supplied together at inference, so we label GEM as ego-motion control in both figures (this is an interface limitation of the evaluated release; GEM's paper does describe both conditioning modalities). Figure 11 compares quality under this shared camera-control setting, where WorldCraft better preserves the fence structure and the stationary bicycle's position. Figure 12 additionally steers a motorcycle through a large displacement while the camera moves, demonstrating the scenario-construction capability highlighted by the reviewer: changing another agent's path while specifying ego-camera motion.

Q2. Inference speed and depth-refinement overhead.

We profile both methods using 8×H100 GPUs with WorldPlay's released acceleration stack: SSTA sparse attention, SageAttention, and a streaming pipeline, preserving interactive generation.

Table R2. End-to-end chunk latency and streaming throughput, including depth recalculation.

MethodPer-chunk latency (16f, 1 GPU)Aggregate throughput (8 GPUs)
WorldPlay (camera only)2.77 s23.1 FPS
WorldCraft (camera + object)3.03 s21.3 FPS

WorldCraft adds approximately 9% per-chunk latency (0.26 s/chunk) over WorldPlay and sustains an 8-GPU aggregate throughput of 21.3 FPS. Within this 0.26 s/chunk, depth recalculation with Depth Anything V2-Large accounts for ~0.05 s/chunk; the remainder comes from the additional modules introduced in WorldCraft (NWT, SP-LoRA, and TASP). Interactive generation is preserved.

Training efficiency. The same compactness that keeps inference light also keeps adaptation cheap. Object-control adaptation updates only 50M parameters (0.6% of the 8B backbone), one quarter of conventional LoRA's 200M trainable parameters, while achieving better trajectory accuracy and camera fidelity (Table 8). Table 6 reports the complete schedule on 8×H200 GPUs: a 2K-step full-backbone domain-adaptation stage, followed by 40K static-camera and 60K autoregressive dynamic-camera steps updating only SP-LoRA. This concentrates object-control learning in a small adapter while keeping the pretrained backbone frozen during both trajectory-training stages.

Reviewer_SgYyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorldCraft extends the camera-navigable autoregressive WorldPlay model with object-trajectory control. It combines camera-aware trajectory reprojection and iterative anchor/depth updates, selective filtering of pre-exit memories, and low-rank updates to the spatial-control pathway. The goal is to preserve requested object motion across camera changes, generation chunks, and periods when the object is outside the view. Evaluation covers trajectory accuracy, camera fidelity, joint control, off-screen re-entry, and transfer to held-out manipulation videos.

Critical strengths
  1. Practical joint-control capability. Combining camera navigation and object trajectories within an autoregressive rollout is useful, especially when the camera leaves and revisits the object. The matched memory experiment in Table 4(b) supports the value of reconciling the requested motion with visual memory: with the trajectory state held fixed, final re-entry error decreases from 48.20 to 35.11 and identity similarity increases from 0.618 to 0.717. This practical capability is the main reason for my positive recommendation.

  2. Parameter-efficient trajectory adaptation. Restricting trajectory adaptation to approximately 50M parameters, about 0.6% of the 8B backbone, is a useful engineering choice. Table 8 shows the relevant trade-off: full fine-tuning obtains lower trajectory error (37.20 versus 38.90), but the proposed adapter retains substantially better camera fidelity, with rotational error of 0.131 versus 0.237. Adding object control with limited interference to the existing camera controller is a worthwhile result.

Critical weaknesses
  1. Limited methodological novelty and an incomplete connection between prior work and the design rationale. I see the main contribution as a useful integration of established ideas for this control setting. The paper cites relevant work, including MotionCtrl, Wan-Move, and WorldPlay, but the explanation of which design insight is new could be more precise. In particular, it would help to connect the shared-spatial-pathway choice more directly to earlier designs that separate camera and object control. The weight-change and activation analyses support the implementation, but I view them as supporting motivation rather than a substantial independent empirical finding.

  2. Limited direct comparison of the full joint-control system. Tables 10–11 report both control metrics for WorldCraft, but the other rows are evaluated on one control axis. This is understandable given the interface differences, yet it limits the direct comparative evidence for the central joint-control contribution. Tables 4, 5, and 8 provide useful component ablations; the remaining question is how much the complete combination adds over a simpler baseline receiving the same camera and object conditions. A matched comparison, or a clearer account of how the existing controlled results address this question, would strengthen the empirical case.

Questions for authors
  1. What do you consider the main new design insight beyond combining trajectory conditioning, camera-aware generation, and memory handling? Please connect this distinction to the cited prior work and explain how the weight-change and activation analyses support the decision to share the spatial-control pathway.

  2. Can the benefit of the full joint-control design be isolated against a simpler composition receiving the same camera and object inputs? If the existing ablations already provide this comparison, please identify the relevant configuration and results. Otherwise, a matched comparison would help quantify the additional value of the full system.

2 comments
AuthorsOpenReview ↗

Author Responses (1/2)

Thank you for recognizing the value of persistent joint control and lightweight adaptation that retains camera fidelity. We clarify the central design insight and provide the requested comparison with a simpler composition below.

Q1. Main design insight: NWT decouples the actions; SP-LoRA reuses the spatial pathway.

The main design insight is a single representation-level choice: Normalized World Trajectory (NWT) places the camera path and the object path into one shared 3D coordinate system before the model sees them. Spatial-Pathway LoRA (SP-LoRA) and Trajectory-Anchored State Persistence (TASP) are two direct consequences of this choice, not three independently combined components. Below we first contrast WorldCraft with the three cited works, then address the separate-pathway distinction you flag at its two linked levels: NWT decouples the two signals at the input, and SP-LoRA then shares a single spatial pathway on top.

Why the three cited works do not supply this combination. Wan-Move is fixed-window object-only control with no camera-aware mechanism. WorldPlay represents the camera-navigable autoregressive video world model line, which exposes no object-action interface. We do not claim modules similar to these two as our contribution; NWT, SP-LoRA, and TASP are precisely the pieces they do not provide.

Distinction from earlier designs that separate camera and object control. The underlying challenge is that camera motion passively displaces every object across the frame, which conflicts with the user's intended object trajectory in the same 2D plane. The closest prior design that couples both actions in a single model, MotionCtrl, keeps them on separate learned pathways: a temporal camera module plus a spatial object module (§3.2). Separate pathways alone do not resolve this conflict; they defer it to the model, and the paper does not verify that the composition is actually reproduced in the output video. A video world model needs to simulate downstream scenarios faithfully, so camera and object signals should be explicitly decoupled before entering the model. WorldCraft therefore departs from the separate-pathway design at two linked levels. First, NWT decouples the two signals at the input by placing them in a world-anchored 3D action space re-projected under each camera pose, making composable control faithful by construction. Second, SP-LoRA shares a single spatial pathway rather than instantiating separate branches. The second choice is not an engineering simplification of MotionCtrl's design; it is only available because of the first: once camera and object sit in one 3D world-space, camera motion (world-frame scene displacement) and object motion (world-frame instance displacement) are the same kind of spatial computation, so the pretrained spatial pathway can follow both and separate branches become redundant. The weight-change and activation analyses in Appendix C then pin down where in the shared pathway to inject the object signal, directly supporting the decision to share the spatial-control pathway: under full fine-tuning, relative weight changes in action_in and prope_proj are 10-25× larger than in attention and MLP layers; camera- and trajectory-induced activation updates overlap in the same middle-layer subspaces (Fig. 8); and changing the trajectory preserves the camera-effect direction with mean cosine similarity 0.89 (Fig. 7b). SP-LoRA therefore adapts only these two input-side projections. With 50M trainable parameters (0.6%), it reaches TE/RPErot of 38.90/0.131, compared with 49.43/0.139 for conventional 200M Q/K/V+MLP LoRA and 37.20/0.237 for full fine-tuning (Table 8): it approaches full fine-tuning's object accuracy while retaining the camera fidelity that full-FT destroys.

Summary. NWT is a world-anchored 3D action representation shared by camera and object, iteratively re-projected across autoregressive chunks; it resolves the camera-object signal conflict at the input. Two capabilities follow from this one choice, each addressing a challenge unique to camera-navigable autoregressive object control: SP-LoRA lets a single pretrained spatial pathway follow both signals with a 0.6% input-side adapter, preserving the inherited camera-navigation capability under new object supervision; TASP maintains an action-updated object state through full invisibility, zero-shot with respect to off-screen supervision, resolving the action-memory conflict on re-entry (Table 4(b): with NWT fixed, selective filtering reduces re-entry error from 48.20 to 35.11 (27.2%)).

AuthorsOpenReview ↗

Author Responses (2/2)

Q2. Matched comparison with a simpler composition.

We compare against naive joint conditioning: SP-LoRA is retrained with the identical adaptation budget on the frozen WorldPlay backbone, but with raw pixel-space conditioning (the ablation row in Table 5) in place of NWT, and default memory at inference. Both control metrics are evaluated on the same rollouts with same camera actions, and same user object paths.

All metrics follow the definitions in §4.1; bold marks the best value within each horizon or protocol.

Table R1(a). Joint camera–object control on the Composable set (45 clips). The WorldCraft rows reproduce Tables 10–11; the naive rows supply the requested simpler-composition comparison.

HorizonConfigurationTE ↓DINO ↑PSNR ↑SSIM ↑LPIPS ↓RPErot ↓RPEtrans ↓RPEcam ↓
61 framesNaive joint40.580.761914.390.36550.43530.1350.01530.0171
61 framesWorldCraft (NWT)32.460.879914.460.36830.43060.1140.01460.0163
253 framesNaive joint51.290.731713.010.33890.55030.1410.02170.0213
253 framesWorldCraft (NWT)43.250.802813.230.34580.53780.1340.01940.0186

At 61 frames, TE decreases by 20.0% and rotational camera error by 15.6%. The improvement persists at 253 frames, where TE decreases by 15.7%, alongside better camera accuracy and DINO similarity.

Table R1(b). Off-screen state control on OSS (50 clips, 253 frames, four invisible chunks). The WorldCraft row reproduces the full-system results in Table 4; the naive row supplies the simpler-composition comparison.

ConfigurationTEmean ↓RTEf ↓RIDpre ↑MEt3R ↓DepthC ↓RPErot ↓RPEtrans ↓RPEcam ↓
Naive joint57.6253.210.4970.2170.07730.1440.01870.0232
WorldCraft (NWT + TASP)40.0335.110.7170.1960.07550.1370.01820.0217

The full system reduces final re-entry error by 34.0% relative to naive joint conditioning and raises re-entry identity similarity from 0.497 to 0.717. The submission's Table 4(b) separately isolates TASP's additional benefit: with NWT fixed, selective filtering reduces re-entry error from 48.20 to 35.11 (27.2%). Camera-consistent trajectory guidance and reconciliation with visual memory therefore address complementary sources of error.

Answer to Q2. Table R1 directly isolates the full joint-control design against a simpler composition receiving the same camera and object inputs, and it also empirically backs the Q1 claim that SP-LoRA and TASP both build on NWT; without NWT, much of their benefit is reduced. At fixed adaptation budget, replacing NWT with raw pixel-space conditioning degrades TE by 25.0%/18.6% (61/253 frames, Composable): SP-LoRA's shared-pathway adaptation loses most of its gain once the input is no longer in a camera-consistent 3D action space. Dropping NWT+TASP in favour of naive joint conditioning on OSS degrades RTEf by 51.6% (35.11→53.21) and RIDpre by 0.220 (0.717→0.497): without NWT keeping the action-updated 3D position during invisibility, TASP's selective filtering has nothing consistent to assert on re-entry. The two downstream capabilities thus contribute complementarily (NWT+SP-LoRA for on-screen composable control, NWT+TASP for off-screen state persistence, zero-shot with respect to off-screen supervision), and both depend on NWT as their shared upstream signal.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

Dear Reviewers,

We sincerely thank all three reviewers ( MoD6 , SgYy , Vqbr ) for their careful evaluation and constructive feedback. We appreciate your recognition of the practical joint-control capability and parameter-efficient trajectory adaptation ( SgYy ), the formulation of persistent object-state control and the action-memory conflict as a meaningful gap, together with the OSS evidence for TASP ( Vqbr ), and the usefulness of scenario construction for robotics and driving applications ( MoD6 ).

We have addressed your comments in the individual responses and made the following additions and clarifications:

  • Stronger camera-fidelity baselines. Added Matrix-Game 3.5, LingBot-World 2, and DreamX-World at 61 and 253 frames under the same camera actions. Section 4.2, Table 3. Raised by MoD6.

  • Robotics transfer and qualitative examples. Quantitative transfer on 50 held-out DROID manipulation videos (125 frames each) without DROID-specific adaptation, plus three manipulation examples under simultaneous viewpoint and object motion. Appendix D.2, Table 12; Appendix D.3, Figures 9–10. Raised by MoD6.

  • Driving-domain comparison with GEM. Three camera-only driving scenes and one joint camera–motorcycle scenario-construction example, with GEM evaluated as ego-motion control. Appendix D.4, Figures 11–12. Raised by MoD6.

  • Design insight and relation to prior work (clarification). NWT is positioned as the upstream representation-level choice, with SP-LoRA and TASP as its two downstream consequences; the contribution is placed at the regime level, and the relation to MotionCtrl, Wan-Move, DragAnything, and GEM is stated explicitly. Raised by MoD6, SgYy.

  • Matched joint-control comparison against naive composition. SP-LoRA retrained at identical adaptation budget with raw pixel-space conditioning in place of NWT, isolating the value of composing camera and object conditions at the input. Tables R1(a)/(b). Raised by SgYy.

  • Simpler memory-management baselines. Sliding-window memory and recent-frame removal on OSS at matched memory budgets, isolating the effect of which context is retained. Table R3. Raised by Vqbr.

  • Extended off-screen evaluation. Non-mirrored camera-and-object returns and two exit-and-return cycles on OSS, 50 rollouts of 253 frames per protocol. Table R4. Raised by Vqbr.

  • Depth refinement during full invisibility (clarification). Each lift already defines the 3D trajectory over the entire user-specified path, so the last refreshed anchor-depth pair carries the trajectory through off-screen intervals; the per-chunk update resumes at re-entry. Raised by Vqbr.

  • Inference profiling including depth overhead. Raised by MoD6.

We have incorporated the new qualitative visualizations and quantitative results in the revised manuscript, with all new content highlighted in red color. The remaining clarifications and additional experiments reported in the individual responses will be merged in the next revision. If you have any further questions or suggestions, we would be happy to discuss them and address any remaining concerns during the rebuttal period.

Thank you again for your time and consideration.

Best regards,

Authors

Dear Reviewer SgYy,

Thank you again for your careful review and constructive suggestions. In our rebuttal, we clarified how our contribution differs from the three cited prior works: MotionCtrl, Wan-Move, and WorldPlay. In particular, following your hint, we connected the shared-spatial-pathway choice directly to earlier designs that separate camera and object control: NWT is the upstream representation-level choice that places camera and object paths into one shared 3D coordinate system, which makes a shared spatial pathway possible in the first place; SP-LoRA then exploits this shared space to inject object control with a 50M (0.6%) input-side adapter while preserving the backbone's camera fidelity, a capability not available to MotionCtrl's separate-pathway design. We also added a matched comparison against naive joint conditioning (Tables R1(a)/(b)), showing at identical adaptation budget that both SP-LoRA and TASP depend on NWT as their shared upstream signal, directly isolating the full joint-control design against a simpler composition receiving the same camera and object inputs as you requested.

With only two days remaining in the discussion period, we would appreciate your feedback on whether these additions and clarifications address your concerns. We would be happy to provide any further clarification and continue the discussion.

Thank you for your time and consideration!

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.