acceptodds
Under review as a conference paper at ICLR 2027

WorldState: Scalable Implicit Memory for Interactive Video World Models

Abstract

Interactive video world models generate videos conditioned on user inputs and camera movements. Long-horizon interaction requires these models to preserve previously generated environments and recover historically consistent content during spatial revisits. Explicit memories either require growing storage and retrieval or rely on potentially inaccurate geometric estimates. Recurrent linear memory avoids these costs but suffers from imbalanced frame-wise updates and fixed capacity: spatial normalization weakens erasure relative to writing, while a single state cannot scale with video length, forcing a growing amount of historical information to compete for limited memory capacity. To address these limitations, we propose WorldState, a scalable implicit memory model built upon a hybrid attention backbone. WorldState decouples memory erasing and writing for fine-grained memory editing and uniformly organizes history into a logarithmically growing set of temporally isolated states. Completed historical states remain subject to channel-wise decay but are excluded from subsequent erasure and writing, reducing repeated overwriting while preserving adaptive forgetting. Context-aware memory routing further retrieves relevant historical states according to the current generation context. Extensive experiments demonstrate that WorldState substantially improves long-term memory retention and spatial revisit consistency while maintaining scalable long-horizon generation efficiency, enabling high-fidelity and spatially coherent video generation over extended interaction horizons.

Primary area:
generative models
Keywords:
Video World Models, Implicit Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4htROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

WorldState proposes a scalable implicit memory for interactive video world models built on a hybrid-attention SANA-WM backbone. It decouples memory erasure from writing, partitions history into temporally isolated recurrent states, and routes the current query to relevant states. The number of states grows logarithmically with sequence length. Evaluations on one-minute camera trajectories and RealEstate10K report improved revisit consistency, camera control, and competitive visual quality, together with scaling and component ablations.

Critical strengths
  1. The paper identifies a concrete limitation of a single recurrent memory state: new observations continually compete with and overwrite older content. Temporally partitioned states with selective routing are a clean and scalable response, and the capacity-matched ablation suggests that the gain is not explained only by additional memory size.

  2. The empirical evaluation is broad and internally consistent. WorldState improves both synthetic revisit metrics and ground-truth RealEstate10K evaluation, while maintaining reasonable memory and runtime. The component and state-update ablations support the main architectural choices.

Critical weaknesses
  1. The strongest controlled comparison is with the SANA-WM backbone, while comparisons with other world models mix different architectures, training data, and generation procedures. The results therefore establish a strong improvement over this backbone more clearly than a general advantage over all explicit-memory approaches.

  2. The scalability claim is stronger than the demonstrated horizon. The main evaluation is one minute, and logarithmically growing states still increase storage and routing work. Longer stress tests would help establish when routing becomes diffuse, whether early memories remain accessible, and how quality, latency, and memory scale together.

Questions for authors
  1. Can the authors provide a controlled comparison with an explicit-memory baseline using the same backbone, training data, and generation budget?

  2. How do quality, latency, and peak memory behave beyond one minute, for example at two, four, and eight minutes? How many memory states are active and effectively used at those horizons?

Reviewer_4W7zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldState studies how to improve the updating, storage, and use of recurrent hidden states for long-term scene consistency in autoregressive world models. Built on SANA-WM, it introduces separate erase and write gates in the GDN layers, together with channel-wise state decay. It replaces each single recurrent state with multiple states assigned to approximately equal temporal segments, with the number of states growing logarithmically with the predefined generation length. Completed states undergo decay but no further direct erasure or writing; their erase corrections are instead applied to the current state. Current-frame query features determine the weights used to combine historical states with the current state. Training includes constructed full and partial revisit trajectories.

The paper evaluates scene-revisit consistency, ground-truth novel-view reconstruction, returns to the initial viewpoint, camera controllability, video quality, component ablations, and inference efficiency. The reported results show improved revisit consistency over SANA-WM, with higher peak GPU memory usage and slightly lower throughput in the main comparison.

Critical strengths
  1. The paper proposes a well-motivated design for updating, preserving, and reading recurrent memory. Separate erase and write gates, together with channel-wise decay, provide more flexible control over memory updates. Multiple temporal states provide additional capacity for retaining earlier information, while query-dependent weighting allows historical states to contribute according to the current generation context. These components form a coherent approach to improving long-term scene consistency in world models built on recurrent hidden states.

  2. The complete model shows meaningful improvements over SANA-WM across complementary evaluations. On SANA-Bench, PSNR improves by 2.49 and 1.92 dB on the Simple and Hard splits, respectively, alongside lower LPIPS. Ground-truth novel-view evaluation on RealEstate10K also shows improved reconstruction quality. The experiments cover camera controllability, video quality, different revisit intervals, capacity-matched baselines, and historical-state update strategies, with results reported both with and without refinement. This breadth of evaluation provides useful support for the effectiveness of the overall method.

Critical weaknesses
  1. The ablations support the effectiveness of the complete design and learned routing, but do not sufficiently establish the independent contribution of temporal isolation. The configuration labeled “E+I” in Table 4(a) sets all routing weights to one only at inference, after training with routing enabled. Comparing this configuration with the full model provides evidence for the benefit of nonuniform routing within the trained model. However, its comparison with E-only does not isolate temporal isolation, since the results may also reflect the effects of routing during training. Furthermore, the capacity-matched single-state baseline’s update rule and expansion procedure are insufficiently specified, making it difficult to determine which design choices account for the remaining performance gap. These limitations weaken the claim that temporal isolation itself is crucial beyond increasing memory capacity.

  2. The revisit evaluation does not sufficiently separate improvements in scene memory from improvements in camera control. Appendix B states that candidate revisit pairs come from the input trajectory metadata. Although camera errors are separately evaluated using Pi3-estimated poses, it remains unclear whether the actual viewpoints in the generated videos are also checked when selecting or validating revisit pairs. Since WorldState improves camera controllability, some gains in revisit metrics may reflect better viewpoint alignment rather than solely better scene memory. This ambiguity limits how confidently the reported revisit gains can be attributed to improved long-term memory.

  3. Important details of the core routing and state-update procedure remain underspecified. Section 3.4 describes query-dependent positive routing weights with unit mean, but does not fully specify the router’s parameterization or how the same parameters accommodate different numbers of completed historical states. This is relevant to the change in state count between training and evaluation. The normalization-state recurrence and the handling of recurrent states across denoising steps also require clarification. These details determine how historical information is read and modified, so their omission limits independent reproduction and assessment of the method.

Questions for authors
  1. Contribution of temporal isolation. Does the capacity-matched single-state baseline in Table 4(b) retain the proposed decoupled editing, and how is its state expanded? Given that the configuration labeled “E+I” disables routing only at inference after training with routing enabled, what evidence distinguishes the contribution of temporal isolation from increased capacity and the effects of routing during training?

  2. Interpretation of revisit improvements. Are Pi3-estimated poses, or another estimate of the generated viewpoints, used to verify the actual viewpoint alignment of revisit pairs selected from trajectory metadata? How are pairs handled when the generated camera motion deviates from the intended trajectory? Clarifying this would help distinguish improvements in scene memory from those due to better camera control.

  3. Routing and state-update details. Could the authors clarify how the router handles different numbers of historical states, particularly between training and evaluation? It would also be helpful to explain how the normalization state is updated and how recurrent states are carried across denoising steps and chunk boundaries. These details would help readers understand and reproduce the method.

Reviewer_aSNKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper modifies the recurrent memory of SANA-WM (a hybrid DiT combining GDN linear attention and softmax attention), targeting spatial revisit consistency in long-horizon camera-controlled video generation. There are three changes. First, the shared gate of GDN is split into separate erase and write gates, and the scalar decay is replaced with channel-wise decay. Second, the single state is extended to N=⌈log2 T⌉ states over uniform temporal buckets; completed states only decay, and their erase terms are added to the live state as a correction. Third, frame-level routing weights are used to weight the readout of the historical states. On SANA-Bench, revisit PSNR improves over SANA-WM from 14.27/14.33 to 16.76/16.25, and camera errors also decrease. The extra cost is small (49.1 vs 42.2 GB, 23.7 vs 25.5 videos/h).

Critical strengths
  1. The problem is well chosen: the paper improves revisit consistency using only implicit recurrent memory, without 3D geometry or frame retrieval, while largely retaining the efficiency of linear attention (Table 1, Fig.3a).
  2. The gains over the direct backbone are large and consistent across several metrics (revisit LPIPS drops by about 0.11; RotErr on the Simple split drops from 5.17 to 3.16). Results with and without the refiner are reported separately, so refiner gains are not credited to the memory.
  3. The variants in Table 4(b) (Capacity-Matched, True Sealing, Historical Erasure) target the right questions.
Critical weaknesses
  1. Temporal isolation by itself does not change the effective memory. Summing over all states in Eq.5 gives S^live_t+ΣS^(i)_t=(S^live_{t-1}+ΣS^(i)_{t-1})M_t+U_t. That is, the unweighted sum follows exactly the single-state recurrence, and the sum is also unchanged at bucket transitions. Hence, when r≡1, the readout is exactly equivalent to that of a single-state model. The readout of the full model equals the single-state readout plus a zero-sum term Σ(r_i−1)S^(i)Q_t. "completed states are excluded from subsequent erasure" is only a matter of bookkeeping: later erasure enters the live state as a negative correction, and the protection of completed states only takes effect when r deviates from 1. Table 4 states that the E+I row uses r≡1 at inference, so the gain from E to E+I (PSNR 15.53→16.02, VBench 0.6785→0.7806) cannot come from "reducing cross-period interference" as claimed in the text (p.8 l.427–431); it can only come from training differences (Appendix A.5 shows that I+R is trained for another 6000 iter after stage 1). If E+I shares weights with the full model, then the entire inference-time effect of isolation+routing is 16.02→16.28. The design does have one advantage: with r initialized to 1, it exactly reproduces the stage-1 model. However, the wording in the abstract and Sec 3.3 should be revised, and two results should be added: the distribution of the learned r_{t,i}, and an iteration-matched E-only control.
  2. The main comparison is confounded by data/training factors. WorldState is fine-tuned on RealEstate10K+ABot-World with revisit-augmented trajectories, while SANA-WM in Tables 1–3 is the original model, and Table 3 happens to evaluate on RealEstate10K. The only single-state control trained on the same data appears in Table 4, evaluated on a subset of unspecified size, and its VBench (0.6737) is about 0.11 lower than SANA-WM's 0.7869/0.7963 in Table 2, which the text does not explain. In addition, on the GT closed-loop metrics, which most directly reflect revisits, WorldState's advantage over Spatia/Gen3C is small (PSNR_C 16.404 vs 16.330/16.340, LPIPS_C 0.347 vs 0.348). This does not match the claim of "substantial improvements on both" at p.7 l.372.
  3. The motivation for decoupled editing is not connected to the method, and the novelty is not adequately positioned. Sec 3.1 frames the problem as one of scale: erasure scales with c² while writing scales with c. But e_t and w_t are sigmoid gates (Appendix A.4), which can only scale erasure down further and cannot compensate for the scale gap introduced by c². The text also provides no erase-strength or e/w statistics showing that the problem is mitigated. Adding E alone gives PSNR +0.87, but SSIM and LPIPS get worse (Table 4a). Moreover, Gated DeltaNet-2, cited in the introduction, already proposes channel-wise erase/write decoupling; KDA has channel-wise decay; and Log-Linear Attention likewise uses a logarithmically growing set of states with a weighted readout. The paper should state the concrete differences between its first two contributions and these works.
  4. Both N and the bucket partition depend on a T given in advance (p.5 l.222), which is in tension with the open-ended interactive setting. Training uses T=33 (N=6, about 5.5 frames per bucket), while evaluation uses T=121 (N=7, about 17.3 frames per bucket). Whether the router generalizes across bucket granularities is not discussed.
Questions for authors
  1. In Table 4(a), do E+I and the full model share weights? Are all ablation variants trained for the same number of iterations?
  2. How far do the learned r_{t,i} deviate from 1? During revisits, do the weights concentrate on the buckets of the corresponding time periods?
  3. How large is the ablation subset? Is its VBench defined the same way as Overall in Table 2? Why is Single-State far below SANA-WM?
  4. How exactly does Capacity-Matched enlarge the capacity of the single state?
  5. Could you add to Tables 1/3 a SANA-WM fine-tuned on the same data for the same number of iterations?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response to Reviewer Feedback: Addressing Key Concerns and Summarizing Main Improvements

General Response

We thank all three reviewers for their constructive feedback. We are pleased that the reviewers recognize the well-motivated problem formulation (aSNK, 4W7z, 4htR), the coherent and efficient memory design (4W7z, 4htR), and the broad evaluation with clear empirical gains (aSNK, 4W7z, 4htR).

Our Contribution

  1. Decoupled memory editing: Independent control of erasure and writing, combined with channel-wise decay.

  2. Scalable historical memory and context-aware routing: Multiple temporal states organize history, while query-conditioned routing dynamically adjusts their contributions.

  3. Effective empirical performance: Improvements in revisit consistency, scene reconstruction, and camera control with modest additional overhead.

Main improvements

  1. Additional evaluation of fine-tuned SANA-WM on SANA-Bench and RealEstate10K.
  2. Clarification of training budgets, ablation settings, evaluation subsets, and capacity matching.
  3. Pi3-based viewpoint analysis of revisit evaluation pairs.
  4. Comparisons of different memory methods on a shared backbone, with clarification of the matched experimental conditions.
  5. Extended evaluations over longer generation horizons.

Other additions to the paper

  1. More precise attribution of conclusions: Clearer and more precise explanations of the contributions of the complete design, temporal isolation, and routing.
  2. More detailed experimental descriptions: Clarifying training and evaluation protocols, routing parameterization, normalization-state updates, and recurrent-state handling across chunks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.