acceptodds
Withdrawn from ICLR 2027

Learning to Remember and Evolve: Object State Memory for Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Camera- and text-controlled video world models require persistent scene information to connect generated changes across viewpoints and beyond recent video context. We propose a framework that connects online feed-forward Gaussian reconstruction, persistent state maintenance, and spatiotemporal conditioning in a recurrent generation pipeline. StateSplat provides renderable geometry and object-level visual evidence, which learned memory integrates alongside explicit state records. Observed and accepted generated records remain distinct from future predictions. Time- and camera-dependent readout supplies memory to the generator through temporal tokens and spatial feature injection, while admitted generated frames provide evidence for subsequent updates. Video generation and subsequent-observation reconstruction objectives jointly train feature adaptation, memory updating, and readout. Experiments on CARLA and DAVIS show improved object reappearance and reference-based continuation quality, with ablations supporting the combined memory configuration and spatial conditioning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D vision, generation model, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MYswOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies persistent object state for camera-/text-controlled video world models: when the camera leaves an object and later returns, generated changes should remain consistent beyond recent video context. The authors propose a recurrent pipeline connecting (1) StateSplat online feed-forward Gaussian reconstruction with object pose/reference geometry, (2) persistent state records plus learned visual memory that separates observed vs. accepted generated evidence, and (3) time-/camera-dependent readout into a LoRA-adapted Cosmos3 generator via temporal tokens and spatial feature injection. Training jointly optimizes generation and auxiliary reconstruction objectives, including reappearance windows. Experiments on CARLA and DAVIS report improved object reappearance/localization and continuation quality over GEN3C, NeoVerse, Vista4D, Spatia, and LiveWorld, with ablations supporting learned memory and spatial conditioning. The authors conclude that reappearance improves, while motion progression and responsiveness to explicit state changes remain open challenges.

Critical strengths
  1. The problem is well motivated and practically important: world models need object persistence beyond short video context.

  2. The system design carefully separates geometry, explicit state, and learned memory, and distinguishes observed vs. generated evidence with admission checks.

  3. Empirical gains on reappearance are large on the reported CARLA clips (detection 0.99/0.95 vs. much lower baselines in Table 1), and DAVIS continuation/identity metrics also improve.

  4. The paper is unusually honest about what results do not show (identity preservation, causal memory use, faithful motion), which helps interpretation.

  5. Ablations, though modest, support combining learned memory with spatial injection.

Critical weaknesses
  1. Evaluation scale is too small for strong claims. CARLA uses only three shared clips; DAVIS departure-return uses five eligible cases. Gains look strong, but statistical robustness and scene diversity are limited. Broader splits and more environments are needed.

  2. Metrics do not fully support the title’s “remember and evolve” claim. Detection/localization and appearance-based Identity/Continuity show better reappearance, but the paper itself notes they do not establish identity preservation or causal memory use. Return localization error remains high (0.92), and motion progression is still weak. The “evolve” part is under-demonstrated.

  3. System complexity vs. isolated novelty. The pipeline stacks DA3, StateSplat, gated memory, admission, Cosmos3-LoRA, etc. Contribution is mainly the coupled object-state memory interface; sharper isolation against simpler caches would strengthen novelty. Related baselines already include GEN3C / NeoVerse / Spatia / LiveWorld / WorldDirector-style ideas.

  4. False positives and residual failures. Non-target detections during absence are still 0.31; qualitative results show intermediate viewpoint/background discrepancies. These limit reliability for persistent worlds.

Questions for authors

Please see weaknesses

Reviewer_Wyx1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks to tackle the problem of object persistence in video world models. A common failure mode for world models is that objects are no longer present when exiting the context. A further failure mode of conventional methods that do retain objects is that these objects haven't evolved over time. This paper proposes a method that tracks seen objects in an external state, using what they call StateSplat, and ensuring these objects are evolving off screen by updating the geometry even when they aren't being tracked in context frames.

Critical strengths
  • The problem is well motivated, and presentation is clear. The paper presents a complete system for tackling object permanence.

  • The idea of evolving off-screen elements is also quite interesting. This is one of the first projects where I have seen a result that shows the continuation of an off screen object expressed correctly. I believe this warrants some credit, as it does include a capability that other memory based methods do not have.

  • The evaluations are complete and tested against a wide range of domains. Results are consistently better than baselines, so there is both qualitative and quantitative support for the effectiveness of the method.

Critical weaknesses
  • The evaluation is trained on, and tested on videos of the same domain despite being fine-tuned from a model that is very generalisable to begin with. This casts doubt on how the method would perform when trying on generalised clips.

  • Evaluating on the training domain also makes me wonder whether the evolving object dynamics is truly being learnt from the proposed method, or simply by the fact that, for example, cars are always seen to have moved in the training data distribution and is therefore going to move during inference.

  • I believe the whole idea of simply tracking the velocities and evolving them off screen is a flawed idea in general. There are a multitude of ways this would be an incorrect assumption in nearly all world modelling scenarios. Any case where an object does not travel in a linear path would fail. This includes non-rigid objects, spinning objects, or even cars which are supposed to stop at a traffic light. In general, this assumption is simply far too strong, and clearly does not apply to most scenarios.

Questions for authors
  • Would it be possible to run inference on your fine-tuned model on other domains? For example, take the CARLA model and inference on real life driving scenarios.

  • As mentioned above, how can you distinguish whether the evolving dynamics is learned from your memory module design and not the actually learning from the fact that the evolution from the data is simply representing these evolving dynamics directly from the context frames?

Reviewer_dd4ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a recurrent reconstruction–memory–generation framework for persistent video worlds. StateSplat combines frozen geometry estimation with learned Gaussian reconstruction, retaining background geometry and rigid object-local references. Explicit pose records and learned visual memory integrate observations and admitted generated evidence, while keeping their provenance distinct. Target-time and camera-dependent readout supplies temporal tokens and spatial conditioning to a LoRA-adapted Cosmos3-Nano generator. Training combines flow matching, an object-region loss, and prediction of subsequent object RGB/depth from memory. CARLA and DAVIS experiments report improved object reappearance and continuation quality, with ablations of memory, spatial conditioning, feature adaptation, and auxiliary supervision.

Critical strengths
  1. The framework makes several important state-management distinctions explicit (§3; Appendix A). Separating object-local geometry from pose, admitted records from extrapolated states, and real appearance references from generated evidence is a thoughtful design. Predicting a subsequent observation before writing its evidence also gives the auxiliary memory objective a meaningful temporal role.

  2. There are measurable benefits, and the reporting is candid about what the metrics establish (Tables 1–4; Appendix B). On ten held-out DAVIS continuation sequences, the method improves LPIPS from NeoVerse's 0.52 to 0.46 under full temporal coverage. The paper reports detection denominators, distinguishes development results from held-out results, and includes an unfavorable state-intervention diagnostic. This transparency materially improves the ability to assess the work.

Critical weaknesses
  1. The evidence does not establish effective use of the maintained dynamic state, which is central to the proposed mechanism (§§3.3–3.4; Appendix B.8). In the reported intervention, supplied positions shift by 30–145 projected pixels but the generated vehicle moves by less than one pixel, with zero displacement between repeated base runs. The authors correctly acknowledge this limitation, and it does not prove that visual memory is never used. Nevertheless, successful reappearance alone cannot demonstrate that time-dependent state readout controls object evolution. The current evidence supports appearance recovery more convincingly than the claimed connection between evolving state and generation. Clarifying the exact checkpoint and intervention path is necessary; if the diagnostic applies to the reported full model, this is a substantive mechanism limitation.

  2. The experiments isolating learned memory are too narrow to establish a robust contribution (Table 4; Appendices B.1 and B.9). The main CARLA results and ablations use three development clips, one vehicle variant, and one town, despite a collected held-out set. Relative to fixed-history spatiotemporal conditioning, the full model adds only six detections among 214 eligible return frames; removing either auxiliary supervision or feature adaptation changes two detections. These correlated frames are not independent trials. Moreover, the fixed-history baseline lacks the same historical visual evidence, so it does not isolate the learned updater from access to additional evidence. The held-out DAVIS continuation result is useful, but does not resolve whether learned memory causes reliable out-of-view state persistence.

Questions for authors
  1. Does the state-insensitive dynR checkpoint in Appendix B.8 correspond exactly to the full model in Tables 1 and 4? Which temporal and spatial conditioning tensors changed under the intervention, and were those changes verified before generation?
  2. Which checkpoint choices, admission thresholds, or other development decisions used the three CARLA evaluation clips? Are any already completed held-out results or per-clip ablation breakdowns available?
  3. In the reported return rollouts, how many generated object observations were admitted into memory, and at what phases? Existing admission logs would help distinguish persistence driven by newly maintained evidence from recovery using initial references and pose extrapolation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.