- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a TTT method for WM as persistent scene memory. The key idea is to store historical visual features in pose-indexed fast weights, updated through recursive least squares, allowing the model to recall previously visited scenes with fixed memory size. The authors further introduce a three-stage training pipeline, including camera-conditioned fine-tuning, memory-augmented autoregressive training, and dual-teacher distribution matching distillation for real-time generation.
Critical strengths
(1) This paper addresses an important problem in long-horizon video world modeling: persistent scene memory and the motivation is clear, and the proposed pose-indexed memory design is intuitive.
(2) The fixed-size memory is appealing for real-time and long-horizon generation. And the real-time performance at high resolution is impressive.
(3) The paper is generally well written, with clear figures and detailed implementation discussions.
Critical weaknesses
(1) How much does each teacher contribute to the final performance? In particular, how does dual-teacher DMD compare with using only the bidirectional or memory-augmented teacher?
(2) The evaluation mainly considers static scenes and reversed trajectories. How well does the method generalize to arbitrary camera trajectories, large translations, and dynamic environments?
(3) Since pose keys depend on estimated camera geometry, how sensitive is the memory mechanism to noisy or inaccurate camera poses?
(4) How does the proposed memory behave on substantially longer rollouts (for example, several minutes as current WM are all targeting on longer horizons) with many distinct viewpoints? Does the fixed memory capacity eventually lead to forgetting?
Questions for authors
see above.
6 comments
Rebuttal by Authors (2/4)
Weakness 2. Evaluation
We thank the reviewer for the valuable feedback. We would like to respond in four parts. First, we present additional demo videos with large translations, complex trajectories, and diverse scenes. Second, we discuss the reversed-trajectory protocol in existing benchmarks and how we adopt it with challenging real-world trajectories. Third, we discuss why we filter out dynamic scenes in revisit training and evaluation. Finally, we explain the generality of TTT memory and how it can be extended to broader conditions for dynamic scenes.
Additional demo videos with large translations, complex trajectories, and diverse scenes. We have updated the supplementary material with more demo videos, packaged in an anonymized project page (please unzip the archive and open index.html in a browser; these videos are compressed to meet the size limit). They cover complex real-world camera trajectories and diverse scenes. Mountain highway and Red-rock canyon path involve large camera translations along the road, and indoor scenes such as Library reading nook, Warehouse aisles, and Round banquet table contain complex occlusions among densely arranged objects like bookshelves, storage racks, tables, and chairs. Compared with the real reference video and the baselines, our model keeps the scene more consistent and stays closer to the reference.
We adopt the existing reversed-trajectory protocol, with challenging real-world camera trajectories. Reversed trajectories are the common protocol for existing world model memory benchmarks such as [a, b, c, d, e]. The evaluation is designed to precisely measure revisit consistency, which calls for a controlled setup: the camera should return to the same pose it visited before, so that the same view can be compared before and after the excursion, and the revisited view has a well-defined reference for how faithfully the scene is preserved. To keep the setup controllable and rigorously evaluate scene consistency, these benchmarks often construct simple reversed trajectories, such as a short straight translation or a smooth pan, which are not realistic. We follow this protocol for memory evaluation, but build our revisit data from real-world camera trajectories. These trajectories are more irregular and include natural camera motion like handheld shooting.
We filter out dynamic scenes to accurately evaluate memory and scene consistency. Generating plausible dynamic content in world models remains a challenging open problem, and evaluating memory and scene consistency with dynamic content is inherently ill-posed. Between two visits of the same pose, dynamic objects can move on their own and end up in many different states. The revisited view then has no single correct reference: a mismatch with the earlier view cannot tell whether the memory failed or the objects legitimately moved, and simply replaying stale content would even be rewarded. Our work focuses on persistent memory for real-time streaming world models. To rigorously study the effect of memory on scene consistency, we adopt a more controlled setting by filtering out dynamic scenes in our revisit training. This setting enables fair, accurate memory evaluation by providing a well-defined reference for the revisited view at the same pose across time.
Our memory design is general and can be extended to dynamic content. In our memory, a write stores the key-value pair through Eqs. 3-4, and a read returns for a query key (Eq. 5). Importantly, although we focus on the static scene setting, our TTT memory is a general conditional formulation: neither the update rule nor the read depends on what the key encodes. In this work, the key is the camera pose, but the memory can in principle be conditioned on more than the pose, such as time or a latent for dynamic entities, by extending the key to , where is a signal available when block is generated. With such keys and additional dynamic scene training data, our memory offers a promising way to represent and recall how a scene and its entities look at different times.
We thank the reviewer again for this inspiring question. In future work, we plan to explore richer conditions, such as time and dynamic entities, which we believe is both interesting and promising.
References
[a] Jianjie Fang et al. iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework.
[b] Xiaojie Xu et al. WorldMark: A Unified Benchmark Suite for Interactive Video World Models.
[c] MIND: Benchmarking Memory Consistency and Action Control in World Models.
[d] WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models.
[e] WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models.
Rebuttal by Authors (3/4)
Weakness 3. Camera Pose Errors
We thank the reviewer for this thoughtful question. In short, camera poses are estimated only offline to annotate training data, and at inference the camera is specified by the user. Camera errors therefore mainly affect how well the model follows the requested camera, which we verify to be accurate, and the design of our memory further limits their effect. We detail these points below.
Our memory learns from correct revisit correspondences. In training, the memory is supervised with our curated revisit trajectories whose pose and content exactly match those of the earlier visit, regardless of camera annotation errors. Our TTT memory therefore learns to write and read from correct correspondences across revisits.
Our model can closely follow the requested camera. We would like to clarify that camera poses are estimated only offline for training data annotation. At inference time, the camera trajectory is provided by the user, so the input carries no estimation error. What matters then is whether the model follows the requested trajectory. To measure this precisely, our benchmark uses strictly aligned revisit trajectories to test whether the model can come back to the viewpoint it left. The demo videos in the updated supplementary material show that our model closely follows the input trajectories.
Effect of camera error on memory is limited. As the model follows the user-specified camera accurately at inference, our memory works as designed: the content stored under a pose key accurately depicts the view at that pose, and a returning camera queries the memory with the same pose key to retrieve it. Camera errors may still perturb the relative geometry among different pose keys in the memory. Our RLS update rule limits their effect along two dimensions. Over time, given the stored pairs, the memory state is the exact minimizer of Eq. 3 and does not depend on the write order, and the write rule itself does not accumulate errors. Across poses, each update writes only the residual that the existing memory cannot yet explain, and a stored pair influences a read only through the correlation between its key and the query key, so an inaccurate pair only affects the recall at nearby poses and the whole memory is still preserved.
Rebuttal by Authors (1/4)
Weakness 1. Ablation on Dual-Teacher Distillation
We thank the reviewer for this constructive comment. We have added this ablation to Section 4.3 (Table 6) of the revised paper.
Both teachers contribute, and the dual-teacher distillation yields the best performance. We ablate the two teachers in Stage 3, removing one at a time while keeping the initialization, schedule, and number of training steps fixed. As shown in the table above, the full dual-teacher model performs best on nearly all metrics and is on par with the best variant on RPE Trans. Removing the bidirectional teacher leads to significant degradation in visual quality, revisit fidelity, and camera following. This is consistent with the bidirectional teacher being the stronger generator, which provides a more reliable target for distribution matching distillation, whereas the AR memory teacher alone cannot compensate for its absence. Removing the AR memory teacher causes a smaller drop in VBench, but it consistently hurts revisit fidelity and camera following, suggesting that it provides complementary memory supervision. Combining the two teachers therefore yields the best overall results.