acceptodds
Under review as a conference paper at ICLR 2027

World-T : World-Test-Time Training as Persistent Memory for Real-Time Video World Models

17 views

Abstract

Recent advances in video world models have enabled plausible interactive environments, yet long-horizon generation places a fundamental demand on visual memory: the previously observed world must remain consistent as the model continues to generate. Existing autoregressive approaches often implicitly rely on the generative model itself to retain such information, making persistent scene memory difficult to maintain beyond the active context. We introduce World-T, a camera-controlled video world model with a test-time training memory that consolidates past observations into dedicated model parameters. To robustly update this memory, we propose a recursive least-squares rule that incrementally absorbs novel visual features via online pose-keyed memory updates. Therefore, our TTT memory enables consistent generation by reading out previously observed scene features via camera pose from the fast-weight state, while maintaining constant storage and per-step computation regardless of rollout length. To facilitate real-time interactive generation with this robust memory, we further distill the autoregressive model into a few-step generator via dual-teacher distribution matching. Experiments show that World-T achieves precise camera control and faithful scene revisits while sustaining 20.6 FPS at 1280 × 704 on a single NVIDIA B200 GPU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
world model, test-time training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JrRTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a TTT method for WM as persistent scene memory. The key idea is to store historical visual features in pose-indexed fast weights, updated through recursive least squares, allowing the model to recall previously visited scenes with fixed memory size. The authors further introduce a three-stage training pipeline, including camera-conditioned fine-tuning, memory-augmented autoregressive training, and dual-teacher distribution matching distillation for real-time generation.

Critical strengths

(1) This paper addresses an important problem in long-horizon video world modeling: persistent scene memory and the motivation is clear, and the proposed pose-indexed memory design is intuitive.

(2) The fixed-size memory is appealing for real-time and long-horizon generation. And the real-time performance at high resolution is impressive.

(3) The paper is generally well written, with clear figures and detailed implementation discussions.

Critical weaknesses

(1) How much does each teacher contribute to the final performance? In particular, how does dual-teacher DMD compare with using only the bidirectional or memory-augmented teacher?

(2) The evaluation mainly considers static scenes and reversed trajectories. How well does the method generalize to arbitrary camera trajectories, large translations, and dynamic environments?

(3) Since pose keys depend on estimated camera geometry, how sensitive is the memory mechanism to noisy or inaccurate camera poses?

(4) How does the proposed memory behave on substantially longer rollouts (for example, several minutes as current WM are all targeting on longer horizons) with many distinct viewpoints? Does the fixed memory capacity eventually lead to forgetting?

Questions for authors

see above.

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

Weakness 1. Ablation on Dual-Teacher Distillation

We thank the reviewer for this constructive comment. We have added this ablation to Section 4.3 (Table 6) of the revised paper.

MethodVBench ↑Semantic Sim. ↑PSNR ↑RotErr° ↓CamMC ↓RPE Rot.° ↓RPE Trans. ↓
GT (VAE-decoded)75.690.9982–0.1940.00550.0730.00102
Dual Teacher74.520.961018.311.0750.03180.1590.00196
w/o AR teacher74.500.956818.211.1630.03470.1600.00195
w/o Bidirectional teacher74.060.943917.761.2850.03740.1730.00205

Both teachers contribute, and the dual-teacher distillation yields the best performance. We ablate the two teachers in Stage 3, removing one at a time while keeping the initialization, schedule, and number of training steps fixed. As shown in the table above, the full dual-teacher model performs best on nearly all metrics and is on par with the best variant on RPE Trans. Removing the bidirectional teacher leads to significant degradation in visual quality, revisit fidelity, and camera following. This is consistent with the bidirectional teacher being the stronger generator, which provides a more reliable target for distribution matching distillation, whereas the AR memory teacher alone cannot compensate for its absence. Removing the AR memory teacher causes a smaller drop in VBench, but it consistently hurts revisit fidelity and camera following, suggesting that it provides complementary memory supervision. Combining the two teachers therefore yields the best overall results.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

Weakness 2. Evaluation

We thank the reviewer for the valuable feedback. We would like to respond in four parts. First, we present additional demo videos with large translations, complex trajectories, and diverse scenes. Second, we discuss the reversed-trajectory protocol in existing benchmarks and how we adopt it with challenging real-world trajectories. Third, we discuss why we filter out dynamic scenes in revisit training and evaluation. Finally, we explain the generality of TTT memory and how it can be extended to broader conditions for dynamic scenes.

Additional demo videos with large translations, complex trajectories, and diverse scenes. We have updated the supplementary material with more demo videos, packaged in an anonymized project page (please unzip the archive and open index.html in a browser; these videos are compressed to meet the size limit). They cover complex real-world camera trajectories and diverse scenes. Mountain highway and Red-rock canyon path involve large camera translations along the road, and indoor scenes such as Library reading nook, Warehouse aisles, and Round banquet table contain complex occlusions among densely arranged objects like bookshelves, storage racks, tables, and chairs. Compared with the real reference video and the baselines, our model keeps the scene more consistent and stays closer to the reference.

We adopt the existing reversed-trajectory protocol, with challenging real-world camera trajectories. Reversed trajectories are the common protocol for existing world model memory benchmarks such as [a, b, c, d, e]. The evaluation is designed to precisely measure revisit consistency, which calls for a controlled setup: the camera should return to the same pose it visited before, so that the same view can be compared before and after the excursion, and the revisited view has a well-defined reference for how faithfully the scene is preserved. To keep the setup controllable and rigorously evaluate scene consistency, these benchmarks often construct simple reversed trajectories, such as a short straight translation or a smooth pan, which are not realistic. We follow this protocol for memory evaluation, but build our revisit data from real-world camera trajectories. These trajectories are more irregular and include natural camera motion like handheld shooting.

We filter out dynamic scenes to accurately evaluate memory and scene consistency. Generating plausible dynamic content in world models remains a challenging open problem, and evaluating memory and scene consistency with dynamic content is inherently ill-posed. Between two visits of the same pose, dynamic objects can move on their own and end up in many different states. The revisited view then has no single correct reference: a mismatch with the earlier view cannot tell whether the memory failed or the objects legitimately moved, and simply replaying stale content would even be rewarded. Our work focuses on persistent memory for real-time streaming world models. To rigorously study the effect of memory on scene consistency, we adopt a more controlled setting by filtering out dynamic scenes in our revisit training. This setting enables fair, accurate memory evaluation by providing a well-defined reference for the revisited view at the same pose across time.

Our memory design is general and can be extended to dynamic content. In our memory, a write stores the key-value pair through Eqs. 3-4, and a read returns mb=W2 gelu(W1kb)\mathbf{m}_b=W_2\,\mathrm{gelu}(W_1\mathbf{k}_b) for a query key kb\mathbf{k}_b (Eq. 5). Importantly, although we focus on the static scene setting, our TTT memory is a general conditional formulation: neither the update rule nor the read depends on what the key encodes. In this work, the key is the camera pose, but the memory can in principle be conditioned on more than the pose, such as time or a latent for dynamic entities, by extending the key to kb=φK(Πb,sb)\mathbf{k}_b=\varphi_K(\boldsymbol{\Pi}_b,\mathbf{s}_b), where sb\mathbf{s}_b is a signal available when block bb is generated. With such keys and additional dynamic scene training data, our memory offers a promising way to represent and recall how a scene and its entities look at different times.

We thank the reviewer again for this inspiring question. In future work, we plan to explore richer conditions, such as time and dynamic entities, which we believe is both interesting and promising.

References

[a] Jianjie Fang et al. iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework.

[b] Xiaojie Xu et al. WorldMark: A Unified Benchmark Suite for Interactive Video World Models.

[c] MIND: Benchmarking Memory Consistency and Action Control in World Models.

[d] WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models.

[e] WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models.

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

Weakness 3. Camera Pose Errors

We thank the reviewer for this thoughtful question. In short, camera poses are estimated only offline to annotate training data, and at inference the camera is specified by the user. Camera errors therefore mainly affect how well the model follows the requested camera, which we verify to be accurate, and the design of our memory further limits their effect. We detail these points below.

Our memory learns from correct revisit correspondences. In training, the memory is supervised with our curated revisit trajectories whose pose and content exactly match those of the earlier visit, regardless of camera annotation errors. Our TTT memory therefore learns to write and read from correct correspondences across revisits.

Our model can closely follow the requested camera. We would like to clarify that camera poses are estimated only offline for training data annotation. At inference time, the camera trajectory is provided by the user, so the input carries no estimation error. What matters then is whether the model follows the requested trajectory. To measure this precisely, our benchmark uses strictly aligned revisit trajectories to test whether the model can come back to the viewpoint it left. The demo videos in the updated supplementary material show that our model closely follows the input trajectories.

Effect of camera error on memory is limited. As the model follows the user-specified camera accurately at inference, our memory works as designed: the content stored under a pose key accurately depicts the view at that pose, and a returning camera queries the memory with the same pose key to retrieve it. Camera errors may still perturb the relative geometry among different pose keys in the memory. Our RLS update rule limits their effect along two dimensions. Over time, given the stored pairs, the memory state is the exact minimizer of Eq. 3 and does not depend on the write order, and the write rule itself does not accumulate errors. Across poses, each update writes only the residual that the existing memory cannot yet explain, and a stored pair influences a read only through the correlation between its key and the query key, so an inaccurate pair only affects the recall at nearby poses and the whole memory is still preserved.

Reviewer_WAouOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a pose-keyed fast-weight memory for long-horizon video generation, using recursive least squares to consolidate historical visual features. The formulation is clear, but the evidence does not sufficiently establish its advantage or practical applicability.

Critical strengths
  • The memory has a fixed state size as the history grows. It is computationally fast.
  • The closed-form update provides a principled solution to the proposed regression objective.
Critical weaknesses
  • No comparison with explicit retrieval. Retrieving a few historical chunks based on camera pose or view overlap is a natural baseline. Such retrieval could preserve relevant details while excluding unrelated observations. Comparisons under matched memory and inference budgets are needed to justify fast-weight consolidation.

  • Evaluation. Cases shown in the paper are too simple. The presented examples largely involve simple scenes, relatively constrained camera trajectories, and little apparent scene dynamics. They provide limited evidence of robustness under complex trajectories, occlusion changes, moving objects, or changing scene states.

  • The method uses W2 to fit all previous chunk information without consideration of the current pose. How to ensure the aggregated information is all useful for the current query?

Questions for authors

The correctness and the evaluation are my major concerns.

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

Weakness 1. Comparison with Explicit Retrieval

We thank the reviewer for this constructive suggestion. We implement the suggested baseline, which retrieves historical chunks by camera pose and places them into the context window. We compare it with our TTT memory under matched inference budgets. We have added this comparison to Section 4.3 (Tables 3, 4, and 5) of the revised paper.

Setup. All variants build on the same causal AR model, retrained with our training configuration but without the TTT memory, and differ only in how the history enters the context window (sizes in blocks):

  • AR Baseline keeps the sink block and the recent blocks: [Sink (1) | Recent (5) | Current (1)].
  • AR + Pose-retrieved KV is the suggested retrieval baseline. The retrieved block is the outbound block captured at the same camera pose: [Sink (1) | Retrieved (1) | Recent (4) | Current (1)].
  • AR + Full KV cache keeps all previous blocks as an unbounded reference: [All history | Current (1)].
  • Ours (Stage 3) uses the same context window as the AR baseline and adds the TTT memory: [Sink (1) | Recent (5) | Current (1)] + TTT memory.

Results. We evaluate all variants on the same revisit benchmark as in the main paper. Table a below reports generation quality, revisit consistency, and camera following. Table b below reports the inference memory of one forward pass of the full KV cache variant at increasing video lengths on H200 GPUs. Table c below reports the additional overhead of our TTT memory compared to the AR baseline.

Table a. Comparison with explicit retrieval baselines.

MethodVBench ↑Semantic Sim. ↑PSNR ↑RotErr° ↓CamMC ↓RPE Rot.° ↓RPE Trans. ↓
GT (VAE-decoded)75.690.9982–0.1940.00550.0730.00102
AR Baseline73.31‾\underline{73.31}0.713412.278.2420.22640.357‾\underline{0.357}0.00349‾\underline{0.00349}
AR + Pose-retrieved KV73.150.702612.308.4200.23120.3600.00354
AR + Full KV cache73.300.7217‾\underline{0.7217}13.10‾\underline{13.10}8.041‾\underline{8.041}0.2209‾\underline{0.2209}0.3670.00357
Ours (Stage 3)74.520.961018.311.0750.03180.1590.00196

 

Table b. The inference cost of growing KV-cache memory in full KV cache baseline.

Context length (latents)Video lengthPeak GPU memory (GB)KV cache memory (GB)
215 s34.16.3
4210 s41.212.7
6316 s48.219.0
8421 s55.325.4
10526 s62.431.7
12631 s69.438.1
14737 s76.544.4
16842 s83.650.8
18947 s90.757.1
21052 s97.863.4
23158 s104.869.8
25263 s111.976.1

 

Table c. Additional overhead of our TTT memory

FLOPsWall-clock (H200)Peak GPU memory
AR Baseline1980 TFLOPs5.20 s34.1 GB
+ Our TTT memory+0.35 TFLOPs (+0.018%)+0.02 s (+0.3%)+0.65 GB (+1.9%)

Our findings are:

  • Explicit retrieval does not improve revisit consistency under a matched budget (Table a). Given the historical chunk retrieved by the same pose, Semantic Sim. and PSNR stay at the level of the AR baseline (0.7026 vs. 0.7134 and 12.30 vs. 12.27 dB). Exposing the full history in KV cache also helps only marginally (Semantic Sim. +0.008). Thus, retrieving history into the context window is not sufficient to maintain scene consistency on revisited viewpoints. In contrast, our model substantially improves revisit consistency with TTT memory.

  • Retrieval-based methods need to store a full KV cache that grows with the video length, while our memory adds only a constant memory cost (Tables b and c). The peak memory of a full KV cache increases significantly with the rollout length, from 34.1 GB at 5 s to 111.9 GB at 63 s, and its attention compute grows accordingly. In contrast, our model with TTT memory keeps a fixed 21-latent context window and adds only a constant-size state, which accounts for about 1.9% of the peak memory, and thus the inference memory stays at the level of the first row in the Table b.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

Weakness 2. Evaluation

We thank the reviewer for this valuable comment. We would like to respond in three parts. First, we present additional cases with complex scenes and trajectories. Second, we discuss why we filter out dynamic scenes in revisit training and evaluation. Finally, we explain the generality of our TTT memory formulation and how it can be extended to broader conditions for dynamic scene generation.

Additional demo videos with complex trajectories and scenes. We have updated the supplementary material with more demo videos, packaged in an anonymized project page (please unzip the archive and open index.html in a web browser, and note that these videos are compressed to meet the size limit). They cover complex real-world camera trajectories and diverse scenes across indoor and outdoor environments. Mountain highway and Red-rock canyon path involve large camera movement along the road, and indoor scenes such as Library reading nook, Warehouse aisles, and Round banquet table contain complex occlusions among densely arranged objects like bookshelves, storage racks, tables, and chairs. Moreover, Columned Building shows rich geometry and textures, City Street at Night shows challenging illumination, and in Retail Store, dense shelves occlude one another as the camera moves, yet our model preserves the layout and details of the products across revisits. In the side-by-side comparisons with the real reference video and the baselines, our model keeps the scene more consistent and stays closer to the reference.

We filter out dynamic scenes to accurately evaluate the effect of memory on scene consistency. Generating plausible dynamic content in video world models remains a challenging and open problem in the field, and evaluating memory and scene consistency with dynamic content is inherently ill-posed. Between two visits of the same pose, dynamic objects can move on their own and end up in many different states. The revisited view then has no single correct reference: a mismatch with the earlier view cannot tell whether the memory failed or the objects legitimately moved, and a model that simply replays stale content would even be rewarded. Our work focuses on persistent memory for video world models that is compatible with real-time streaming generation. To rigorously study the effect of memory on scene consistency, we adopt a more controlled setting by filtering out dynamic scenes in our revisit training. This setting supports fair and accurate evaluation of memory via a well-defined reference for the revisited view at the same pose across time.

Our memory design is general and can be extended to dynamic content. In our memory, a write stores the key-value pair kj=φK(Πej)\mathbf{k}_j=\varphi_K(\boldsymbol{\Pi}_{e_j}), vj=φV(hej)\mathbf{v}_j=\varphi_V(\mathbf{h}_{e_j}) through Eqs. 3-4, and a read returns mb=W2 gelu(W1kb)\mathbf{m}_b=W_2\,\mathrm{gelu}(W_1\mathbf{k}_b) for a query key kb\mathbf{k}_b (Eq. 5). Importantly, although we focus on the static scene setting, our TTT memory is a general conditional formulation: neither the update rule nor the read depends on what the key encodes. In this work, the key is the camera pose, but the memory can in principle be conditioned on more than the pose, such as time or a latent for dynamic entities, by extending the key to kb=φK(Πb,sb)\mathbf{k}_b=\varphi_K(\boldsymbol{\Pi}_b,\mathbf{s}_b), where sb\mathbf{s}_b is a signal available when block bb is generated. With such keys and additional dynamic scene training data, our memory offers a promising way to represent and recall how a scene and its entities look at different times.

We thank the reviewer again for this inspiring question. In future work, we plan to further explore memories conditioned on richer signals, such as time and dynamic entities, which we believe is both interesting and promising.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Weakness 3. Aggregated Memory for the Current Pose Query

We thank the reviewer for this thoughtful question. It touches on a central design of our memory, and we appreciate the opportunity to clarify it. We first recall the streaming setting and the write and read in TTT memory. Then, we discuss why the write is agnostic to the current pose query while the read depends on it, and finally show how the read selects relevant content in W2W_2 for the current pose query.

In the streaming setting, future poses are unknown at each current step. World-T$^3$ is a real-time, streaming video world model: the video is generated block by block, and camera poses are supplied by the user on the fly, so the future poses are not known at the current step. Our memory thus plays two roles at every step. It writes: the block that has just been evicted from the context window is consolidated into the memory, so that it remains available for future steps. It also reads: the information written in earlier steps is retrieved to support the block being generated, using the current pose as the query.

Write is agnostic to the current pose while read is conditioned on the current pose query. The streaming setting explains why the two operations are treated differently. A write is made for the benefit of a future whose camera poses have not yet been chosen by the user, so it should not be tailored to any particular pose. In contrast, a read is made for the benefit of the block currently being generated, so memory retrieval is conditioned directly on the current pose.

Use W2W_2 for the current pose query. Although W2W_2 is fit to the whole history at each write, what a read returns depends on the query. The minimizer of Eq. (3) has a closed form, W2=W2(0)+[∑i≤Jwi(vi−W2(0)ϕi)ϕi⊤]PJW_2 = W_2^{(0)} + \big[\sum_{i \le J} w_i (\mathbf{v}_i - W_2^{(0)}\phi_i)\phi_i^{\top}\big] P_J, where JJ is the number of writes completed so far. Substituting it into Eq. (5) gives

mb=W2(0)ϕb+∑i≤Jwi(vi−W2(0)ϕi)(ϕi⊤PJϕb),ϕb=gelu(W1kb).\mathbf{m}_b = W_2^{(0)}\phi_b + \sum_{i \le J} w_i \big(\mathbf{v}_i - W_2^{(0)}\phi_i\big) \big(\phi_i^{\top} P_J \phi_b\big), \quad \phi_b = \mathrm{gelu}(W_1 \mathbf{k}_b).

Thus, each stored pair contributes with a weight wiϕi⊤PJϕbw_i \phi_i^{\top} P_J \phi_b, which depends on the current pose through ϕb\phi_b. Intuitively, content stored under poses close to the current one receives a large weight, while content stored under unrelated poses contributes little.

Reviewer_VdKqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces World-T^3, a method designed to address the issue of scene inconsistency in autoregressive video world models when the camera returns to a previously visited viewpoint—a scenario where the original frames have already shifted out of the context window, forcing the model to regenerate them. The approach incorporates a "fast-weight" memory mechanism,based on Test-Time Training, into each layer of a Diffusion Transformer (DiT), using camera poses as keys. As video patches exit the context window, they are written into this memory via Recursive Least Squares; conversely, when new patches are generated, the model retrieves relevant information based on the target camera pose. By providing an exact solution to ridge regression at each step, RLS effectively suppresses redundant writes caused by smooth camera trajectories, while ensuring that memory overhead remains constant regardless of video length. The model undergoes a three-stage training process: bidirectional camera-pose fine-tuning, joint training of the autoregressive model and its memory, and a dual-teacher DMD process to compress inference to just 4 steps. In tests involving 120 "out-and-back" camera trajectories, World-T^3 demonstrates superior scene consistency and camera-pose accuracy compared to existing methods, while achieving real-time video generation at 20.6 FPS on a single B200 GPU.

Critical strengths

1.Simple, principled memory design. I really like this approach. The key depends only on the camera pose, not on the diffusion noise, so the same viewpoint always retrieves the same memory. It’s a very clean idea. Writes are done with RLS, and each one gives the exact ridge-regression solution. Redundant information gets averaged out, new content is fully absorbed, and the order of writes doesn’t change the result. The memory state stays fixed in size and adds only about 24 ms per block.

2.Large gains, backed by solid evidence. The method leads across the board on revisit consistency and camera following. Against the strongest baseline, it improves PSNR by nearly 5 dB and cuts rotation error to about a third. It wins on more than 110 of the 120 test clips, and the paired tests are highly significant. The larger the camera sweep, the clearer the advantage.

3.Genuinely real-time on a single GPU. Dual-teacher distillation brings sampling down to 4 steps while preserving both local visual quality and long-horizon consistency. Together with a series of system-level optimizations, the model reaches 20.6 FPS at 1280×704 on a single B200. Among the compared methods, it is the only one that runs faster than real time on one GPU. The engineering details and numerical-fidelity checks are also well documented.

Critical weaknesses
  1. Nearly half of the training data consists of ping-pong playback, where videos are played forward and then backward. This closely matches the test trajectories, which retrace the same path in reverse. The test set is also collected by the authors, while the baselines have neither access to similar training data nor the benefit of being evaluated in-domain. In addition, the test subsets were selected after examining the performance of reference models, which raises concerns about potential selection bias.

  2. The memory can only be accessed through camera poses, rather than retrieved based on content. The method also assumes static scenes, as dynamic objects are deliberately excluded from the training playback data. This limits its ability to handle moving objects and interactions.

  3. The test set mainly consists of clips with minimal camera translation, where the motion is dominated by rotation and there is very little parallax. The return trajectory also follows the exact same path in reverse, meaning that content encountered earlier during the revisit was written to memory more recently. In real-world exploration, the camera often moves over much larger distances and does not necessarily return along the same path, making pose-based retrieval much more challenging. I would suggest testing longer revisits that go beyond the training sequence length, involve different return paths, and include larger camera translations. It would also be useful to examine how revisit quality changes with the number of memory writes, to better understand the memory capacity and whether the method generalizes to longer sequences.

Questions for authors

I am leaning toward acceptance, but I would like the authors to clarify the concerns raised in the above weaknesses.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

Weakness 1. Playback Trajectory, Training Data, Baseline Fairness, and Benchmark Construction

We thank the reviewer for raising these concerns. We would like to respond in three parts. First, ping-pong playback provides revisit supervision for our memory, yet it offers no trivial shortcut for the test. Second, the baselines draw on similar public data, and some add substantial self-collected data. Third, our test subsets are filtered and calibrated using only the baselines for fair comparison.

Ping-pong playback and test trajectories. Ping-pong training does not make the revisit test trivial. Like the baselines, our model generates block by block in a streaming manner and only knows the camera of the current block, so it cannot learn that the camera will retrace its path as a global template. Moreover, all four revisit gaps strictly exceed the 21-latent-frame context window, so the revisited content is no longer in the recent context and cannot be obtained by replaying recent frames in reverse. It has to be recalled from our memory, whose read depends only on the camera pose and carries no time index (Eq. 5), so it recalls content whenever a pose recurs, regardless of the direction of travel. Finally, both training and test use real-world camera trajectories, and the test clips are held out, so the model has no simple scripted pattern to exploit, and the scenes and camera motions in the test are new to the model.

Training data and baselines. Our training data mainly comes from the public Sekai, SpatialVID, and DL3DV datasets, and the test clips are held out from training. SANA-WM and DreamX-World are trained on all three datasets, HY-World 1.5 on Sekai and DL3DV, and Matrix-Game 3.0 on DL3DV and SpatialVID. Most of them further add substantial self-built data that we do not use. Moreover, revisit data are not unique to our work either. HY-World 1.5 renders reconstructed scenes along revisit trajectories designed for scene consistency, and LingBot-World synthesizes look-back trajectories to densify re-observations. Their evaluations also resemble ours: HY-World 1.5 and Matrix-Game 3.0 return the camera along its outbound path, and DreamX-World includes an out-and-back template.

The benchmark is filtered and calibrated using only the baselines to isolate memory evaluation from generation quality. We would like to clarify how our benchmark is constructed. Before filtering, many clips in the held-out test set contain extreme camera movement or a long-tailed content distribution, on which all baselines obtain low imaging and aesthetic quality in VBench. Such cases cannot separate memory from low generation quality, so they are not suitable for measuring memory ability fairly and accurately. Thus, we take the VBench quality levels reported in the baseline papers as a reasonable reference range and remove the cases that fall clearly below it, which is the step that finalizes the subsets. This step involves only the baselines and not our model. In addition, we calibrate specific parameters of the baselines’ open-source inference scripts, such as camera speed, on the test set, so that they behave normally and the comparison is fair. The resulting subset serves to measure revisit consistency. We have added this clarification to Appendix D.1 of the revised paper.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

Weakness 2. Memory Access and Static Scenes

We thank the reviewer for this insightful comment. We would like to respond in three parts. First, we explain why camera pose is a natural key for memory access and how content can complement it. Second, we discuss why we filter out dynamic content in revisit training and evaluation. Third, we show that our memory design is general and can be extended to broader keys such as content and dynamic entities.

Camera pose is a natural key for memory access in camera-controlled world models, and content offers a complementary cue. A camera-controlled video world model has to determine how the scene looks from the viewpoint the user requests, and this viewpoint is specified exactly by the camera pose. Camera pose is therefore a direct and natural key for accessing the memory. Content-based retrieval asks a different question, namely which stored places resemble the current view. It is independent of pose accuracy, but retrieval by content alone can be ambiguous in repetitive scenes, where different places look alike, whereas each camera pose is unique to its viewpoint. The two cues thus complement each other. Inspired by the reviewer's question, we believe that reading the memory with pose and content together could be even more suitable. Since our memory operates on generic key-value pairs, it is general enough to accommodate such a joint read of pose and content, as we explain in the third part below.

We filter out dynamic content to accurately evaluate the effect of memory on scene consistency. Generating plausible dynamic content in video world models, such as moving objects and interactions, remains a challenging and open problem in the field, and evaluating memory and scene consistency with dynamic content is inherently ill-posed. Between two visits of the same pose, dynamic objects can move on their own and end up in many different states. The revisited view then has no single correct reference: a mismatch with the earlier view cannot tell whether the memory failed or the objects legitimately moved, and a model that simply replays stale content would even be rewarded. Our work focuses on persistent memory for video world models that is compatible with real-time streaming generation. To rigorously study the effect of memory on scene consistency, we adopt a more controlled setting by filtering out dynamic scenes in our revisit training. This setting supports fair and accurate evaluation of memory via a well-defined reference for the revisited view at the same pose across time.

Our memory design is general and can be extended to dynamic content. In our memory, a write stores the key-value pair kj=φK(Πej)\mathbf{k}_j=\varphi_K(\boldsymbol{\Pi}_{e_j}), vj=φV(hej)\mathbf{v}_j=\varphi_V(\mathbf{h}_{e_j}) through Eqs. 3-4, and a read returns mb=W2 gelu(W1kb)\mathbf{m}_b=W_2\,\mathrm{gelu}(W_1\mathbf{k}_b) for a query key kb\mathbf{k}_b (Eq. 5). Importantly, although we focus on the static scene setting, our TTT memory is a general conditional formulation: neither the update rule nor the read depends on what the key encodes. In this work, the key is the camera pose, but the memory can in principle be conditioned on more than the pose, such as time, the state of dynamic entities, or content descriptors, by extending the key to kb=φK(Πb,sb)\mathbf{k}_b=\varphi_K(\boldsymbol{\Pi}_b,\mathbf{s}_b), where sb\mathbf{s}_b is a signal available when block bb is generated. With such keys, our memory offers a promising way to represent and recall how a scene and its entities look at different times.

We thank the reviewer again for raising this point. In future work, we plan to further explore memories conditioned on richer signals like time and dynamic content, which we believe is both interesting and promising.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Weakness 3. Translation, Return Path, and Longer Revisit

We thank the reviewer for the detailed and constructive comment. We would like to respond in three parts. First, we present additional demos with larger translation and longer revisits. Second, we discuss the return-trajectory evaluation protocol. Finally, we discuss longer revisits beyond the training horizon, including the scope and setup of our work, and the behavior of the memory in this regime.

Additional demo videos with larger translation and longer revisits. We have updated the supplementary material with more demo videos, packaged in an anonymized project page (please unzip the archive and open index.html in a browser). They cover complex real-world camera trajectories and diverse scenes. Notably, Mountain highway and Red-rock canyon path involve a large camera translation along the road. Beyond the training horizon, we also present longer videos of about 30 seconds with multiple revisits: Columned Building shows rich geometry and textures, City Street at Night shows challenging illumination, and in Retail Store, dense shelves occlude one another as the camera moves, yet our model preserves the layout and details of the products across revisits. This horizon is six times that of our 5-second camera-control training data and twice that of our revisit training data, with roughly twice as many memory writes, and the model still keeps the scene consistent across revisits. In the side-by-side comparisons with the real reference video and the baselines, our model keeps the scene more consistent and stays closer to the reference.

We adopt the existing return trajectory protocol, with more challenging real-world trajectories. Return trajectory in reverse is the common protocol for evaluating memory in existing benchmarks such as [a, b, c, d, e]. The evaluation is designed to precisely measure revisit consistency, which calls for a controlled setup: the camera should return to the same pose it visited before, so that the same view can be compared before and after the excursion, and the revisited view has a well-defined reference for how faithfully the scene is preserved. To keep the setup controllable and rigorously evaluate scene consistency, these benchmarks often construct simple reversed trajectories, such as a short straight translation or a smooth pan, which are not realistic. We follow this protocol for memory evaluation, but build our revisit data from real-world camera trajectories. These trajectories are more irregular and include natural camera motion such as handheld shooting.

Substantially longer generation is a dedicated line of work, and we study the memory in a more controlled setup. Current state-of-the-art open-source video generation backbones, such as Wan2.2-TI2V, can only generate clips of about 5 seconds. Real-time streaming generation at substantially longer durations (such as minute-level) is a challenging direction. Reaching this horizon often calls for a dedicated long-video training stage with progressively longer durations or multi-segment distillation, and high-quality minute-long data. Despite such efforts, the visual quality of long videos still degrades severely as the rollout continues, with drift in style, color, and details. We therefore focus on the memory design within the horizon that public data support, which also allows us to evaluate the effect of memory separately from the visual quality degradation of long video generation.

Our model can extrapolate beyond the training horizon, and the memory design is suitable for long videos. Camera control is trained on our constructed 5-second data, and the TTT memory is supervised on revisit data of about 15 seconds, which places revisited views beyond the local context window. Notably, beyond this horizon, the updated supplementary material shows that our model can stably extrapolate to 30-second generation with multiple revisits, as discussed in the additional demo above. Moreover, our memory has constant storage and per-step cost, which makes it more feasible for longer videos under a limited budget than memories that grow with the horizon. Its capacity is set by the size of the fast weights, which we can enlarge while keeping the cost independent of the rollout length. With a dedicated training stage on long-video revisit data, we believe scaling up the fast weights is a promising path toward extending our memory to long-video world models.

References

[a] Jianjie Fang et al. iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework.

[b] Xiaojie Xu et al. WorldMark: A Unified Benchmark Suite for Interactive Video World Models.

[c] MIND: Benchmarking Memory Consistency and Action Control in World Models.

[d] WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models.

[e] WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

General Response

We sincerely thank all reviewers for their careful reading and constructive comments. We are encouraged that the reviewers find our camera-pose-keyed fast-weight memory simple, principled, and intuitive, with a closed-form recursive least-squares (RLS) update and a fixed-size state that keeps computation constant (Reviewers VdKq, WAou, JrRT). They also recognize the large gains in revisit consistency and camera following (VdKq), the real-time performance at high resolution on a single GPU (VdKq, JrRT), and the importance of the problem and the clarity of the presentation (JrRT, VdKq).

We have replied to each review individually. Here we highlight the main updates to the supplementary material and the revised paper:

  • More demo videos in the supplementary material. We have added more demo videos, packaged in an anonymized project page (please unzip the archive and open index.html in a web browser). The videos are compressed to meet the size limit, which may slightly soften fine details. They cover a wide range of complex scenes and camera trajectories, as well as longer rollouts beyond the training horizon with multiple revisits. We also describe this in the new paragraph in Appendix A of the revised paper.
  • Comparison with explicit retrieval. We compare with baselines that retrieve historical chunks by camera pose into the context window. Our TTT memory significantly outperforms these baselines while adding only 1.9% peak memory and 0.3% wall-clock time compared to the AR model without TTT memory. We discuss this in Section 4.3 of the revised paper.
  • Ablation on dual-teacher distillation. We ablate the two teachers in Stage 3. Both teachers contribute, and our dual-teacher distillation yields the best overall performance. We discuss this in Section 4.3 of the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.