acceptodds
Under review as a conference paper at ICLR 2027

UniState: Unifying Memory and Intervention for Controllable World Simulation

Abstract

World models offer increasingly rich environments for exploration, yet intervening in a world while observing the consequences from freely chosen viewpoints remains challenging. We tackle these challenges by introducing UniState, a framework for controllable world simulation built on a unified world-state representation. This representation unifies spatial memory and user interventions. Users specify object motion directly within spatial memory, while independently choosing the camera trajectory from which the resulting world state is observed. This unification makes the relationship between the scene and the requested changes explicit, removing ambiguity between object motion and viewpoint change from the control specification. Crucially, specifying an intervention does not require prescribing every resulting movement: the video model can synthesize additional motion within controlled objects and plausible interactions with their surroundings. We develop training and sampling strategies that help generated videos follow the specified object motion and camera trajectory while preserving this capacity to generate responses beyond the prescribed transformations. Experiments demonstrate more accurate joint object and camera control than the existing methods, alongside qualitative evidence that intervention-induced scene changes persist across rollouts.

Primary area:
generative models
Keywords:
video world models, controllable video generation, spatial grounding, 3D scene representation, object motion control, camera control, visual simulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KpKmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors observe that jointly supporting persistent scene memory, object interventions, and independent camera control remains challenging for existing video world models. They suggest that this difficulty partly arises from representing spatial memory and intervention instructions separately, making their correspondence difficult to establish. To address this issue, they propose UniState, which unifies spatial memory and user interventions within a shared, explicit 3D representation. Object motion is specified by directly transforming selected regions of the memory, while the camera trajectory is defined independently. They further introduce Scene-Anchored Joint Guidance and Transport-Aligned Augmentation to reduce background drift and mitigate the training–inference mismatch, respectively. Experiments across 277 scenarios demonstrate improved joint object and camera control compared with the evaluated baselines.

Critical strengths
  1. The design is well motivated. Representing object motion and scene memory using a shared coordinate system and appearance information makes their spatial relationships more explicit and facilitates reasoning about object occlusion.

  2. The paper is clearly written and easy to follow, with figures and tables that effectively illustrate the proposed method and experimental results.

Critical weaknesses
  1. The state-update mechanism requires a clearer description and further validation. Section 3.1 describes temporarily excluding interaction regions from memory conditioning, followed by reconstructing and registering geometry from generated observations to update the state. However, it remains unclear how the affected regions are identified and what criteria are used to trigger state updates.

  2. While the ablation studies evaluate TAA and SAJG, they do not directly isolate the contribution of the unified representation, which is the paper’s central contribution. Consequently, it remains unclear how much of the performance improvement can be attributed to the unified representation itself.

  3. The method relies on estimated depth and geometric registration to construct and update its 3D memory. How are depth inaccuracies and scale inconsistencies handled during registration and repeated state updates? A robustness analysis using different depth estimators or controlled geometric perturbations would help clarify whether these errors accumulate across rollouts and degrade object control, occlusion handling, or scene consistency.

Questions for authors

The qualitative examples appear to involve relatively controlled scenes. How well does UniState generalize to more challenging real-world scenes with cluttered backgrounds, substantial occlusion, and small or irregularly shaped objects, where depth estimation and segmentation may be less reliable? Could the authors clarify whether such cases are represented in the evaluation set and provide additional examples or a breakdown of performance under these conditions?

3 comments

We thank KpKm for the thoughtful assessment and the constructive feedback. We carefully address the concerns below.

W1. Region selection and state updates.

In our default interface, the user selects regions that may be affected by the given intervention. The selected regions are excluded from memory conditioning throughout the rollout.

Updates occur after each rollout. We use the generated frames as observations to obtain updated geometry and latent features, register the geometry to the shared world coordinate system, and update the affected regions while retaining the remaining memory. The updated state initializes the next rollout. We have clarified this policy and schedule in §3.1.

W2. Contribution of the unified representation.

UniState represents scene memory and object interventions in a shared 3D state, resolving their visibility relationships together. We ablate this design using a Separate variant with independent full-width scene/object projections. Separate is trained with visibility-resolved inputs and receives overlapping scene/object inputs at inference. Both variants use the same backbone, and we evaluate checkpoints at 10K training updates with matched generation seeds and sampling settings, disabling SAJG for both.

We evaluate 24 matched video pairs from a subset of 12 development conditions:

MethodWCPE (%) ↓
UniState (ours)2.615
Separate2.754

UniState achieves a modest reduction in average control error on this subset. The paired examples in Fig. 6 (Appendix B.2) also show differences in occlusion handling: our ball disappears behind the alphabet block and reappears on its left, whereas Separate places it in front of the block. Our tractor remains partially occluded by the hay bales, whereas Separate exhibits ghosting of its body and wheels.

W3. Depth accuracy and registration.

At an added depth perturbation level of 5% relative root mean square error (RMSE) with respect to the unperturbed estimate, the tested sequences largely preserve the intended motion, with relatively small changes in object endpoint error (EPE). For the tractor:

Added relative depth RMSE (%)Object EPE (% of image diagonal) ↓
0 (no added perturbation)2.295
52.395
103.038
203.560

Control trajectories and generation settings are fixed within each case. Table 6 and Fig. 7 in Appendix B.3 report these results. These results indicate tolerance to the tested 5% added depth perturbation, while showing that larger perturbations can cause scene-dependent deviations in motion extent and final position.

During state updates, we align newly reconstructed geometry to the existing memory by estimating relative scale and pose from correspondences in unchanged, overlapping regions. We have added this implementation detail to Appendix A.1.

Within each rollout, every state is constructed directly from the initial state, avoiding recursive frame-to-frame updates of the conditioning geometry. Although trained with first-frame memory initialization, UniState supports initialization from multiple observations at inference. Overlapping views provide additional geometric constraints, while complementary views extend surface coverage, reducing reliance on a single depth estimate. Residual depth and registration errors may still carry over across rollouts.

Q1. Challenging scenes and object shapes.

We now provide qualitative results for 14 cases initialized from real-world observations. Representative examples show drawer opening, lid lowering toward a pot, folded-towel translation, and pitcher tilting. The lid and pitcher scenes contain nearby utensils and dishes, illustrating clutter and partial visibility. We present representative examples in Fig. 8 (Appendix C) and complete sequences under “Real-world interactions” in the supplementary HTML (supplementary.html#real-world-interactions).

Our original evaluation includes clutter and full occlusion followed by reappearance; Figs. 5 and 11 show object control under occlusion.

For irregularly shaped objects, UniState does not require a predefined object shape: segmentation masks select the corresponding 3D latents for control (§3.1). Existing examples include the tractor in Fig. 11 and part-level control of an excavator in Fig. 1(e). Finite VAE latent resolution limits control of very small objects (§5). Depth and segmentation errors can also affect control accuracy; W3 reports our additional depth-sensitivity analysis.

Thank the authors for the detailed response, which addresses the majority of my concerns

But there is a remaining concern: the ablation for the unified representation, which is the core contribution of the paper. The Separate variant is trained with visibility-resolved inputs but tested with overlapping inputs, so part of the gap may come from this mismatch rather than from the representation itself. It is better to explicitly exclude this potential factor

To clarify, both UniState and Separate use the same training and inference settings, with Transport-Aligned Augmentation (TAA) applied identically during training. At inference, geometric scene–object overlap can occur in both variants before rendering. Both use the same visibility-resolution function, applied jointly to the complete scene in UniState and independently to the scene and object components in Separate. Separate’s post-rendering conditioning does not introduce scene–object overlap that is absent during training.

Reviewer_CYsPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a novel framework, UniState, to represent video memory and guide video generation using a unified representation. Specifically, the memory is represented as 3D VAE latents extracted from the first frame, whose depth is estimated by a pre-trained depth estimator. The condition signal is the object masks and the transformation and rotation parameters (6-DoF) of the objects and the camera. The framework is implemented on pretrained Cosmoc3-Nano and finetuned to adapt to the new conditional signal. All methods are evaluated on a self-collected dataset including camera and object motion. Experiments show that UniState achieves the best object accuracy and camera accuracy scores.

Critical strengths
  • The paper proposes a new framework to represent physical intervention as a 6-DoF transformation of the camera and objects, and designs specialized training approaches to improve the performance of the proposed framework. Specifically, the paper introduces Scene-Anchored Joint Guidance (SAJG) to learn the camera condition first without the object motion condition. The paper further introduces Transport-Aligned Augmentation (TAA) to overcome the error accumulation caused by representing the state only with the first frame.

Overall, the paper provides sufficient technical contribution. However, some concerns about method design still remain. See weakness section.

Critical weaknesses
  • The paper does not sufficiently discuss the difference from MosaicMem. MosaicMem has already represented memory with 3D latents and allows multiple editing operations. I understand that UniState uses a different conditional mechanism for physical intervention and the comparison results are given in the appendix, but a more detailed discussion about the difference and the advantages of UniState over MosaicMem is needed to support the motivation of the paper.

  • I have some concerns about the usability of the core design that represents states according to the first frame and updates states with 6-DoF transformation. 1) The paper says the scene tokens affected by interaction (such as collision) are then temporarily excluded. However, how to automatically detect the start and end of the interaction is not discussed. 2) The condition signal heavily relies on the depth estimation and segmentation results of the first frame since it is a 6-DoF transformation. The control may not be stable if the detection of the first frame is not correct.

  • The presentation is not clear and affects reproducibility. 1) Some terms, such as "unified state" and "physical intervention", are introduced without giving an early, concrete explanation, making the text hard to follow when reading it for the first time. 2) "UniState" is ambiguous in the paper. Sometimes it refers to the framework, sometimes it refers to the unified representation ("UniState adapter"). 3) It is hard to realize WW refers to the projected 2D latent since it is neither visualized nor given shape definition in the text.

Questions for authors
  • Can the authors give a detailed discussion about the differences and the advantages of UniState over MosaicMem?
  • How to detect the start and end of the interaction?
  • Does the method become unstable if the detection of the first frame is not correct? How to overcome this problem?
3 comments

We thank CYsP for the thoughtful assessment and the constructive feedback. We carefully address the concerns below.

W1 / Q1. Relationship to MosaicMem.

MosaicMem describes memory operations such as relocation, duplication, and deletion, but its reported memory-editing examples focus on connecting different scenes into explorable environments (§4.4, Fig. 7). UniState extends memory manipulation to time-varying object interventions and the generation of plausible interactions with surrounding objects, such as contact-induced motion (Fig. 1(b)). Our training and sampling strategies support this capability.

In our reproduced MosaicMem baseline, applying time-varying transformations does not ensure that objects follow the prescribed motion. Figure 12 illustrates this under a fixed camera: the tank fails to follow the intervention and disappears midway. Table 9 further supports UniState’s advantage in joint object and camera control.

W2.1 / Q2. Interaction timing.

Before each rollout, the user selects the objects or regions that may interact with the controlled entity. These regions are excluded from memory conditioning from the beginning to the end of that rollout. After the rollout, we use the generated frames as observations to reconstruct geometry, register it to the shared world coordinate system, and update the affected regions. We have clarified this schedule in §3.1.

W2.2 / Q3. Stable result despite imperfect depth and segmentation

Regarding segmentation, our qualitative observations suggest that UniState tolerates minor boundary errors, such as small portions of the background being included in the object mask.

During joint object and camera control, inaccurate depth can cause the generated video to deviate from the scene changes intended by the user. Although trained with first-frame memory initialization, UniState supports initialization from multiple observations at inference (§3.1; Fig. 2(a)). This provides complementary geometric evidence and may help mitigate errors associated with a single-view depth estimate, provided that the observations are reliably registered.

We additionally evaluated depth perturbations at relative root mean square error (RMSE) levels of 5%, 10%, and 20% with respect to the unperturbed depth estimate. At 5%, the tested sequences largely preserve the intended motion with relatively small changes in object endpoint error (EPE), while larger perturbations can cause more pronounced, scene-dependent deviations. For the tractor:

Added relative depth RMSE (%)Object EPE (% of image diagonal) ↓
0 (no added perturbation)2.295
52.395
103.038
203.560

See Table 6 and Fig. 7 in Appendix B.3.

W3. Presentation clarity.

We have clarified the distinction between the 3D world state and the projected 2D latent condition WW in §3.1, and added its tensor shape and axis definitions to Appendix A.1.

Thanks for the reply. All of my concerns have been resolved. I will raise the rating to 3.

We appreciate your thoughtful feedback and are glad that our responses and revisions have addressed your concerns.

Reviewer_7G2FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UniState, a framework for controllable world simulation that keeps spatial memory while allowing user interventions in a single world-state representation. Initial-frame latents are lifted into 3D using estimated depth, users move selected objects or object parts, and the state is rendered along an independently chosen camera trajectory to condition a pretrained video model. The paper also introduces a scene-anchored guidance for sampling and an augmentation that narrows the train–inference gap, and shows that intervention outcomes can be written back to memory and kept across rollouts.

Critical strengths
  • The design provides controllability while deliberately leaving room for the model to generate the fine-grained motion of dynamic objects, its consequences, and interactions with the surroundings. This is a reasonable choice, given that prior controllability-focused methods have paid less attention to the resulting interactions and outcomes.
  • The scene-anchored joint guidance is a reasonable design. Anchoring the background to an intervention-free reference implicitly helps keep camera motion and object motion from becoming entangled.
  • The method shows better quantitative and qualitative results than other methods on joint object and camera control.
  • The transport-aligned augmentation is a simple idea, but it appears effective in practice for accurate controllability.
Critical weaknesses
  • Plausible interaction with the surroundings is a central claim, but video results on more complex and diverse interaction scenarios are limited. The current results are few and cover relatively simple scenarios, so it is hard to judge how well the method actually performs.
  • Handling interactions by excluding the regions where a response is expected from the memory condition appears somewhat heuristic, as it seems to require knowing in advance where the response will occur.
  • (Minor) It is hard to grasp what the paper proposes from the introduction. The problem is clearly defined, but how the paper addresses it is harder to follow, and making this clearer would help.
Questions for authors
  • Since the state is built on depth maps, are there constraints on large rotations of a controlled dynamic object, such as a full 360-degree turn that reveals its unseen side?
  • Could you provide more video results on complex and diverse interaction scenarios?

---- Update -----

Thank you for the detailed response and the updated experimental results. The multi-view extension for scenarios requiring 360-degree rotations is reasonable, and the additional video results are helpful in assessing the proposed model's capabilities.

I found that in some video examples (e.g., "opening a coffee-maker lid"), the interaction between the controlled robotic arm and other objects appears unnatural. However, I acknowledge that this seems inevitable, as it depends on the capabilities of the pre-trained model rather than the proposed interface itself.

I will maintain my rating of weak accept.

2 comments

We thank 7G2F for the thoughtful assessment and the constructive feedback. We carefully address the concerns below.

W1 / Q2. Diversity of interaction scenarios.

We now provide qualitative results for 14 cases initialized from real-world observations. Representative sequences show drawer opening, lid lowering toward a pot, folded-towel translation, and pitcher tilting.

Only the gripper motion is specified, while the video model generates the remaining motion and interactions. For example, the drawer opens in response to the gripper motion without a separately prescribed drawer trajectory. These examples extend the interaction evidence beyond the bulldozer–rock contact shown in Fig. 1(b) to everyday manipulation scenarios. Representative examples are shown in Fig. 8 (Appendix C), and the complete sequences for all 14 cases are provided under “Real-world interactions” in the supplementary HTML (supplementary.html#real-world-interactions).

W2. Selecting interaction regions.

Users can adjust the size of the selected regions according to the intended interaction. Memory conditioning is retained in unselected regions to support scene consistency, while the selected regions are excluded from memory conditioning so that the video model can synthesize responses. Users select potentially interacting objects or regions, without requiring their future positions or response trajectories. For example, selecting the rock in the bulldozer scenario does not require specifying where it will move after contact. When the affected area is uncertain, a broader region can be selected. We have clarified this assumption in §3.1 of the revised manuscript.

W3. Clarity of the introduction.

We will revise the introduction to explain the proposed approach and the roles of its components more clearly. Once the revised manuscript is uploaded, we will post a follow-up response highlighting the changes.

Q1. Large rotations and unseen surfaces.

With single-view initialization, unseen surfaces are not represented in memory. When revealed by large rotations, their appearance is synthesized by the video model and may vary across generation seeds. Although trained with first-frame memory initialization, UniState supports initialization from multiple observations at inference (§3.1; Fig. 2(a)). Additional registered views of the object extend memory coverage to otherwise unseen surfaces and provide constraints on their appearance.

I’ve uploaded a first revision of the introduction to clarify the core idea and how the method addresses the problem. I’ve also made the overall workflow more explicit to help readers follow the proposed approach. I would welcome any further comments or suggestions as I continue refining the text.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Follow-up

As the rebuttal period draws to a close, we kindly invite reviewers who have not yet responded to review our replies and share any remaining questions or concerns.

We thank all reviewers for their thoughtful feedback and for recognizing the following strengths:

  • Superior quantitative and qualitative results on joint object and camera control. (7G2F)
  • A 6-DoF intervention framework with a well-motivated shared spatial representation. (CYsP, KpKm)
  • Controllability that preserves flexibility in generating fine-grained motion and interactions with the surroundings. (7G2F)
  • Meaningful design contributions through Scene-Anchored Joint Guidance and Transport-Aligned Augmentation. (7G2F, CYsP)
  • Clear writing, supported by effective figures and tables. (KpKm)

We also appreciate their suggestions to strengthen the paper:

  • Broader evaluation of complex and diverse interaction scenarios. (7G2F)
  • Clarification and validation of interaction-region identification, memory exclusion, and state-update criteria. (7G2F, CYsP, KpKm)
  • Robustness analysis of depth and segmentation errors, scale inconsistencies, and error accumulation across state updates. (CYsP, KpKm)
  • More detailed discussion of the differences from and advantages over MosaicMem. (CYsP)
  • An ablation isolating the contribution of the unified representation. (KpKm)
  • Clearer presentation of the proposed approach and more concrete explanations of key concepts. (7G2F, CYsP)

In the rebuttal, we have carefully addressed the reviewers’ comments to clarify our work and strengthen the camera-ready version. We sincerely thank all reviewers for their time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.