- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors observe that jointly supporting persistent scene memory, object interventions, and independent camera control remains challenging for existing video world models. They suggest that this difficulty partly arises from representing spatial memory and intervention instructions separately, making their correspondence difficult to establish. To address this issue, they propose UniState, which unifies spatial memory and user interventions within a shared, explicit 3D representation. Object motion is specified by directly transforming selected regions of the memory, while the camera trajectory is defined independently. They further introduce Scene-Anchored Joint Guidance and Transport-Aligned Augmentation to reduce background drift and mitigate the training–inference mismatch, respectively. Experiments across 277 scenarios demonstrate improved joint object and camera control compared with the evaluated baselines.
Critical strengths
-
The design is well motivated. Representing object motion and scene memory using a shared coordinate system and appearance information makes their spatial relationships more explicit and facilitates reasoning about object occlusion.
-
The paper is clearly written and easy to follow, with figures and tables that effectively illustrate the proposed method and experimental results.
Critical weaknesses
-
The state-update mechanism requires a clearer description and further validation. Section 3.1 describes temporarily excluding interaction regions from memory conditioning, followed by reconstructing and registering geometry from generated observations to update the state. However, it remains unclear how the affected regions are identified and what criteria are used to trigger state updates.
-
While the ablation studies evaluate TAA and SAJG, they do not directly isolate the contribution of the unified representation, which is the paper’s central contribution. Consequently, it remains unclear how much of the performance improvement can be attributed to the unified representation itself.
-
The method relies on estimated depth and geometric registration to construct and update its 3D memory. How are depth inaccuracies and scale inconsistencies handled during registration and repeated state updates? A robustness analysis using different depth estimators or controlled geometric perturbations would help clarify whether these errors accumulate across rollouts and degrade object control, occlusion handling, or scene consistency.
Questions for authors
The qualitative examples appear to involve relatively controlled scenes. How well does UniState generalize to more challenging real-world scenes with cluttered backgrounds, substantial occlusion, and small or irregularly shaped objects, where depth estimation and segmentation may be less reliable? Could the authors clarify whether such cases are represented in the evaluation set and provide additional examples or a breakdown of performance under these conditions?
3 comments
Thank the authors for the detailed response, which addresses the majority of my concerns
But there is a remaining concern: the ablation for the unified representation, which is the core contribution of the paper. The Separate variant is trained with visibility-resolved inputs but tested with overlapping inputs, so part of the gap may come from this mismatch rather than from the representation itself. It is better to explicitly exclude this potential factor
To clarify, both UniState and Separate use the same training and inference settings, with Transport-Aligned Augmentation (TAA) applied identically during training. At inference, geometric scene–object overlap can occur in both variants before rendering. Both use the same visibility-resolution function, applied jointly to the complete scene in UniState and independently to the scene and object components in Separate. Separate’s post-rendering conditioning does not introduce scene–object overlap that is absent during training.
We thank
KpKmfor the thoughtful assessment and the constructive feedback. We carefully address the concerns below.W1. Region selection and state updates.
In our default interface, the user selects regions that may be affected by the given intervention. The selected regions are excluded from memory conditioning throughout the rollout.
Updates occur after each rollout. We use the generated frames as observations to obtain updated geometry and latent features, register the geometry to the shared world coordinate system, and update the affected regions while retaining the remaining memory. The updated state initializes the next rollout. We have clarified this policy and schedule in §3.1.
W2. Contribution of the unified representation.
UniState represents scene memory and object interventions in a shared 3D state, resolving their visibility relationships together. We ablate this design using a Separate variant with independent full-width scene/object projections. Separate is trained with visibility-resolved inputs and receives overlapping scene/object inputs at inference. Both variants use the same backbone, and we evaluate checkpoints at 10K training updates with matched generation seeds and sampling settings, disabling SAJG for both.
We evaluate 24 matched video pairs from a subset of 12 development conditions:
UniState achieves a modest reduction in average control error on this subset. The paired examples in Fig. 6 (Appendix B.2) also show differences in occlusion handling: our ball disappears behind the alphabet block and reappears on its left, whereas Separate places it in front of the block. Our tractor remains partially occluded by the hay bales, whereas Separate exhibits ghosting of its body and wheels.
W3. Depth accuracy and registration.
At an added depth perturbation level of 5% relative root mean square error (RMSE) with respect to the unperturbed estimate, the tested sequences largely preserve the intended motion, with relatively small changes in object endpoint error (EPE). For the tractor:
Control trajectories and generation settings are fixed within each case. Table 6 and Fig. 7 in Appendix B.3 report these results. These results indicate tolerance to the tested 5% added depth perturbation, while showing that larger perturbations can cause scene-dependent deviations in motion extent and final position.
During state updates, we align newly reconstructed geometry to the existing memory by estimating relative scale and pose from correspondences in unchanged, overlapping regions. We have added this implementation detail to Appendix A.1.
Within each rollout, every state is constructed directly from the initial state, avoiding recursive frame-to-frame updates of the conditioning geometry. Although trained with first-frame memory initialization, UniState supports initialization from multiple observations at inference. Overlapping views provide additional geometric constraints, while complementary views extend surface coverage, reducing reliance on a single depth estimate. Residual depth and registration errors may still carry over across rollouts.
Q1. Challenging scenes and object shapes.
We now provide qualitative results for 14 cases initialized from real-world observations. Representative examples show drawer opening, lid lowering toward a pot, folded-towel translation, and pitcher tilting. The lid and pitcher scenes contain nearby utensils and dishes, illustrating clutter and partial visibility. We present representative examples in Fig. 8 (Appendix C) and complete sequences under “Real-world interactions” in the supplementary HTML (
supplementary.html#real-world-interactions).Our original evaluation includes clutter and full occlusion followed by reappearance; Figs. 5 and 11 show object control under occlusion.
For irregularly shaped objects, UniState does not require a predefined object shape: segmentation masks select the corresponding 3D latents for control (§3.1). Existing examples include the tractor in Fig. 11 and part-level control of an excavator in Fig. 1(e). Finite VAE latent resolution limits control of very small objects (§5). Depth and segmentation errors can also affect control accuracy; W3 reports our additional depth-sensitivity analysis.