acceptodds
Under review as a conference paper at ICLR 2027

WMem: Relevance, Reliability, and Timing for Long-Horizon Video World Memory

31 views

Abstract

Video world models can generate realistic short sequences. However, they often lose scene identity as observations leave the local context and generation errors accumulate. We argue that effective long-horizon memory is not only a storage problem, since its utility depends on which observations are retrieved, how reliable the memory is, and when it is read during denoising. We introduce WMem, a memory framework that collectively addresses these three factors through Role-factorized Retrieval, Diffused Memory Conditioning, and Denoising-Stage Routing. WMem retrieves stable anchors, target-view revisits, and complementary views, and combines them with role and target-relative geometry. During training, our Diffused Memory Conditioning trains the model to use memories with query-dependent fidelity without expensive autoregressive rollouts. Denoising-Stage Routing allocates memory roles across denoising stages, reducing unnecessary computation. With the same backbone and memory budget as the baseline, our WMem reduces MSE by 36.1%, LPIPS by 35.6%, and FID by 82.0%, while improving PSNR by 3.68 dB over Minecraft 500-frame rollouts. It also reduces inference time by 23.5% and peak memory of the GPU by 36.9%. On 300-frame forward-and-return RealEstate10K trajectories, WMem also improves long-horizon scene consistency. Our results show that long-horizon world modeling benefits from jointly controlling memory relevance, reliability, and timing. Code and models will be released.

Primary area:
generative models
Keywords:
video generation, world model, memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2kJsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents W4 Mem, a memory framework for improving long-horizon consistency in video world models. The authors argue that effective memory utilization depends on three factors: relevance, reliability, and timing.

To address these factors, the proposed framework introduces three components. First, Role-Factorized Retrieval organizes historical observations into anchors, revisits, and complementary views, which are integrated through a shared role- and geometry-aware memory reader. Second, Diffused Memory Conditioning (DMC) introduces query-conditioned noise corruption during training to improve robustness to imperfect generated memories without requiring autoregressive training rollouts. Third, Denoising-Stage Routing selectively activates different memory roles according to the current denoising stage, improving computational efficiency.

Experiments on Minecraft and RealEstate10K demonstrate improvements in long-horizon generation quality and scene consistency. On Minecraft, W4 Mem reduces MSE by 36.1%, LPIPS by 35.6%, and FID by 82.0% relative to WorldMem, while reducing inference time and peak GPU memory usage. The paper also includes component ablations, memory-budget evaluations, and revisit-consistency measurements to investigate the effectiveness of the proposed design.

Critical strengths

S1. A well-motivated and practically meaningful framework for long-horizon video memory.

The paper addresses an important limitation of current video world models: maintaining scene consistency when previously observed information leaves the local context and autoregressive errors accumulate.

I find the decomposition of memory utilization into relevance, reliability, and timing useful. Rather than focusing exclusively on memory capacity or retrieval accuracy, the authors investigate how different memory sources should be selected, presented during training, and accessed throughout denoising.

The proposed framework is conceptually coherent, and the three components address distinct aspects of long-horizon memory utilization. In particular, Diffused Memory Conditioning and Denoising-Stage Routing provide interesting approaches to improving memory robustness and computational efficiency.

S2. Strong empirical results supported by extensive evaluations and ablations.

The paper demonstrates substantial improvements across multiple metrics on Minecraft and RealEstate10K. The gains over WorldMem are especially pronounced on long Minecraft rollouts, while the comparison with additional baselines, including DecMem and Geometry Forcing, provides useful evidence of practical effectiveness.

The experiments also go beyond standard frame-level reconstruction metrics. The revisit-consistency evaluation directly examines whether the model preserves previously observed scene information, which is well aligned with the paper's motivation.

The component ablations, memory-budget experiments, and reversed-routing comparison further strengthen the empirical evaluation. I also find the simultaneous improvements in generation quality, inference speed, and memory efficiency particularly valuable for practical deployment.

Critical weaknesses

W1. The training–inference consistency and underlying mechanism of denoising-stage routing require further clarification.

A central contribution of the paper is Denoising-Stage Routing, which assigns different memory roles to different denoising stages. However, the routing condition differs between training and inference in a potentially important way.

During training, the routing decision is based on the mean timestep of a target pack, while individual target frames are assigned independently sampled diffusion timesteps. Consequently, target frames at substantially different noise levels may share the same memory-routing decision. At inference, routing instead depends on the current DDIM timestep. Although the same routing threshold is used, the statistical properties and interpretation of the routing condition differ between training and inference.

This raises a question about whether the model actually learns the proposed coarse-to-fine allocation of memory roles at the individual-target level. The reversed-routing and no-routing ablations demonstrate that the proposed schedule is empirically effective. However, they do not fully establish that its effectiveness arises from the claimed correspondence between memory roles and denoising stages. A comparison with per-target timestep routing, or an analysis of routing behavior under different target-timestep distributions, would help substantiate this central mechanism.

Questions for authors

Q1. How does the mean-timestep routing used during training affect consistency with inference-time routing?

During training, individual target frames have independently sampled diffusion timesteps, but the routing decision is determined by the mean timestep of the target pack.

Could the authors clarify how this shared routing decision supports the proposed stage-specific use of memory when different target frames may be at substantially different noise levels? In particular, have the authors compared mean-timestep routing with routing based on individual target timesteps? Such a comparison would help determine whether the proposed design benefits from genuine stage-specific memory allocation or from a more general form of conditional memory dropout.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2kJs

We thank the reviewer for finding our decomposition into relevance, reliability, and timing useful and our framework coherent, and for recognizing our strong results, our revisit-consistency evaluation, and the joint gains in quality, speed, and memory efficiency.

We appreciate the reviewer for these helpful comments. We acknowledge that in training, per-target timestep routing and pack-mean routing act differently. We show the analysis, and we also add new per-target routing trained from scratch as below and in Appendix E of the revised paper.

Response to Q1 & W1

Analyze of training/inference mismatch. We simulated 2M training packs with the exact training sampler, in which the 8 targets draw their timesteps independently.

  • A target's own side of troute=500t_{\mathrm{route}} = 500 agrees with its pack's gate 62.4% of the time, so 37.6% of targets receive the opposite decision.
  • The probability that a target sees anchors rises from 0.28 for the cleanest targets (t<100t < 100) to 0.72 for the noisiest (t≥900t \geq 900).

Training therefore teaches a soft, stage-biased exposure with a role-dependent corruption level, while inference applies a hard per-frame gate.

Mismatch of routing in training and inference. Following WorldMem, we retrieve memory once per 8-frame target pack during training, using the camera pose of the pack's last frame. These memory frames pass through the spatial and temporal attention layers together with the targets, so retrieving a separate memory set for every target would put eight times as many memory frames through these layers and make training much more expensive. With one memory set per pack, the routing gate is set once from the pack's mean target timestep. We agree with the reviewer that a shared memory set does not require a shared gate, since each target can read the shared memory through a gate set by its own timestep. We therefore trained this per-target variant from scratch, keeping one DMC bound for all roles. As Table R1 below and Table 18 in Appendix E of the revised paper show, the per-target gate is almost same our pack-mean gate on all four metrics.

Table R1: Ablation of routing during training on Minecraft.

RoutingFID↓LPIPS↓MSE↓PSNR↑
Reversed routing13.310.30890.0356821.25
No routing11.050.27450.0390621.89
Per-target gate10.390.27130.0325022.27
Pack-mean gate9.020.26240.0306322.65

Does the model benefit from just generic dropout? We believe no, as reversed routing hides each role in the same share of packs and only flips the same soft pattern, so if routing were simply dropout, both variants should perform equally. Instead, as shown in Table 6 in the main paper and Table R1 above, reversed routing raises FID from 9.02 to 13.31 and is also worse on MSE and LPIPS, while removing routing raises FID to 11.05. Since reversal also flips the coupling between each role and its corruption level, these ablations show that pairing roles with noise stages and corruption levels during training matters.

Reviewer_s22sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

W^4 Mem tackles long-horizon visual memory degradation in video world models by jointly addressing memory relevance, reliability, and read timing. The framework combines role-factorized retrieval (selecting anchors, target-view revisits, and complementary views), query-conditioned diffusion corruption during training (Diffused Memory Conditioning), and denoising-stage-dependent memory routing, paired with a unified reader conditioned on role and relative camera geometry. Experiments on 500-frame Minecraft rollouts and 300-frame RealEstate10K trajectories show clear gains in long-horizon scene consistency alongside notable reductions in inference latency and peak GPU memory.

Critical strengths
  1. The main comparison against the retrained WorldMem baseline is cleanly controlled: both models use the exact same pretrained backbone, VAE, local context window, external memory budget, sampling steps, training steps, and global batch size. On the 500-frame Minecraft rollout, FID drops from 50.25 to 9.02 (an 82% reduction) alongside solid gains in PSNR (+3.68 dB), MSE, and LPIPS. Breaking down the evaluation across long-horizon frame bins and pose-based revisits also provides much stronger evidence than reporting a single average score.

  2. The proposed modules are well-motivated from an efficiency standpoint and backed by targeted ablations covering role budgets, corruption schedules, and stage routing. Cutting inference time by 23.5% and peak GPU memory by 36.9% is a meaningful practical improvement, and I appreciate that the authors clearly distinguish between reducing active memory-reading compute and maintaining constant total history storage.

Critical weaknesses
  1. Table 4 already provides useful fixed-budget role-replacement controls, and the Revisit-only variant retains the unified reader, target-relative geometry, and DMC. However, that variant also removes role embeddings, count correction, and stage routing. Thus, the role-replacement results support the usefulness of the memory roles, while the total retrieval-versus-reader contribution remains only partially isolated. A comparison retaining the same reader and conditioning while changing only selection would clarify this attribution. The Gaussian-corruption limitation is also acknowledged in Appendix C; additional tests with structured errors would strengthen the scope of the reliability interpretation.

  2. Some details regarding the baseline training recipe need further clarification. The original WorldMem paper uses a different training schedule and curriculum. To be fair, the authors' 200k-step, batch-128 setup actually sees more total training samples than the original 500k-step, batch-16 recipe, so the baseline is not simply undertrained. Still, showing paired training convergence curves, explaining how the original WorldMem curriculum was handled, and reporting how the officially released WorldMem checkpoint performs under this exact 500-frame evaluation protocol would help separate equal-budget training gains from recipe differences. Also, while the abstract notes that code and models will be released, the RealEstate10K baseline is reimplemented from scratch, so ensuring that the modified baseline configs and exact test trajectory lists are included in the release is important for verification.

Questions for authors
  1. Beyond the role-replacement and Revisit-only controls already reported, is there a comparison that changes only retrieval while retaining identical reader conditioning, count correction, and routing?

  2. How was the original WorldMem training curriculum adapted in your 200k-step retraining, do the loss/validation curves confirm full convergence, and how does the official pretrained WorldMem checkpoint score under your evaluation protocol?

  3. Have you tested whether Diffused Memory Conditioning still helps when paired with structured perturbations or model-generated rollout frames during training, beyond standard Gaussian diffusion noise?

  4. Since you plan to release code and models, could you confirm whether the release will also include the reimplemented RealEstate10K baseline configurations, the retrained WorldMem checkpoints, and the exact test trajectory lists?

4 comments
AuthorsOpenReview ↗

Response to Reviewer s22s (1/3)

We thank the reviewer for recognizing our cleanly controlled comparison with WorldMem, our well-motivated and efficient modules with targeted ablations, and our evaluation across horizon bins and revisits.

We appreciate the reviewer for the detailed review and the precise questions. We add three selection-only trainings (W1 & Q1), a structured-corruption study (W1 & Q3), the official WorldMem checkpoint and training curves (W2 & Q2), and confirm the release contents (W2 & Q4).

Response to W1 & Q1

We thank the reviewer for this helpful comment. We agree that the Revisit-only control changes selection and reader conditioning together, so we add three new trainings that change memory role retrieval from the final W⁴Mem. Everything else is identical to the final model: the reader, role embeddings, count correction, routing, DMC, the results shown in Table R1 below, and Table 7 in Section 4.4 of the revised manuscript, with the settings in Appendix B.

  • Anchor + revisit: complementary memory are filled with revisit memory.
  • Complementary + revisit: anchor memory are filled with revisit memory.
  • WorldMem memory: all 8 memory frames are retrieved by WorldMem's selection.

Table R1: Ablation of memory selection on Minecraft.

Memory selectionFID↓LPIPS↓MSE↓PSNR↑
Baseline50.250.4070.047918.97
Anchor + revisit12.740.2710.029422.42
Complementary + revisit9.460.2670.038620.77
WorldMem memory selection + full reader19.420.3400.048120.30
W⁴Mem (Ours)9.020.2620.030622.65

New memory selection ablation. MSE and PSNR measure pixel-level agreement with the ground-truth frames, i.e., whether the scene geometry, layout and colors are reproduced; LPIPS measures perceptual similarity to the ground truth in deep-feature space; FID measures how realistic the generated frames are as a distribution. Table R1 above and Table 7 in Section 4.4 of the revised paper show that both parts of W⁴Mem contribute. With WorldMem's selection, the rest of W⁴Mem (RFR, the unified reader, DMC and routing) raises PSNR by 1.3 dB and lowers LPIPS from 0.407 to 0.340 and FID from 50.25 to 19.42 relative to WorldMem, while MSE is nearly unchanged. With the reader fixed, our role-based selection adds another 2.4 dB of PSNR and lowers FID from 19.42 to 9.02. The two roles affect different metrics, as the coarse-to-fine design of the paper intends: anchors are read at high-noise steps, where the scene layout forms, and complementary views at low-noise steps, where detail is added. Replacing the anchors with revisits mainly hurts pixel accuracy with MSE 0.0306→0.0386, PSNR −1.9 dB, whereas replacing the complementary views leaves pixel accuracy nearly unchanged but hurts perceptual quality with LPIPS 0.262→0.271, FID 9.02→12.74.

AuthorsOpenReview ↗

Response to Reviewer s22s (2/3)

Response to W1 & Q3

Structured perturbations. We appreciate the reviewer for suggesting structured perturbations, which led to a clear improvement of our model. We follow the reviewer's suggestions to add structured latent corruption of memory during training: blur, photometric drift, affine misalignment, and stale-content patches. As in DMC, the severity of each corruption is tied to the query noise level, and the 25% of memory slots in the stabilized branch stay clean. Table R2 below and Table 9 in Section 4.4 of the revised manuscript, with the training details in Appendix C, report two new variants: structured corruption alone and structured corruption combined with Gaussian DMC. Structured corruption alone already improves every metric over training without corruption and gives the best MSE and PSNR. Adding Gaussian DMC on top lowers FID from 14.22 to 4.54, so DMC still helps when paired with structured perturbations. Compared with Gaussian DMC alone, the combined version halves FID from 9.02 to 4.54, lowers LPIPS from 0.262 to 0.248. The two corruptions are therefore complementary. Structured corruption mainly helps pixel accuracy, and Gaussian DMC mainly helps perceptual and distributional quality.

Table R2: Memory corruption comparison.

Training corruptionGaussianStructuredFID↓LPIPS↓MSE↓PSNR↑
None (w/o DMC)––19.740.36830.0432721.14
Structured only–✓14.220.26220.0283422.73
Gaussian DMC✓–9.020.26240.0306322.65
Structured + Gaussian DMC✓✓4.540.24760.0295022.66

Model self-generated as corruption. We appreciate reviewer's insightful comments. DMC is rollout-free by design. A revisit at frame 1,000 may read generated frame 800, which carries 200 autoregressive steps of error. Reproducing that error in training costs about 4,000 extra forward passes per sample (200 frames × 20 DDIM steps), or hundreds of GPU-hours even for a fine-tune on cached rollouts. Local rollouts from ground-truth frames are cheaper (about 120 extra passes) but produce only one-step error, which is not the error that accumulates at test time.

AuthorsOpenReview ↗

Response to Reviewer s22s (3/3)

Response to W2 & Q2

We thank the reviewer for raising these insightful comments.

Training recipe. Both methods start from the same pretrained backbone and share one recipe: 200k steps at global batch 128 on 8 H200 GPUs, as shown in Section 4.1 and Appendix D, so the comparison is equal-budget by design.

Curriculum. Official WorldMem trains for 700k steps in three stages. Steps 0–120k use a small memory range without vertical camera turning, steps 120k–240k use a large range, still without vertical turning, and steps 240k–700k use the large range with vertical turning. Our WorldMem reproduction keeps these stages: giving steps 0–30k, 30k–60k and 60k–200k in the 200k steps reproduction. Based on our experiments, we found that W⁴Mem converges in fewer steps, so it does not need the three-stage schedule. We train it for 200k steps in two stages: steps 0–40k without vertical turning correspond to WorldMem's first two stages, and steps 40k–200k with vertical turning correspond to its third stage. W⁴Mem also needs no small-range stage, because it selects memory with its own selectors rather than WorldMem's range-based sampling. You can also see the stage change on Figure 6 and Figure 7 in Appendix D of the revised paper.

Convergence. Please kindly have a look at new Figure 6 and Figure 7 in Appendix D; these 2 new figures show the training loss and a training-time validation on the first 100 generated frames, respectively. The training losses shows that W⁴Mem converged faster and more stable compare with WorldMem, and both methods keep flat near the end of the training shows they have converged. Validation curves are flat over the last 50k steps, and W⁴Mem reaches its plateau earlier.

Official checkpoint. As shown in Table R3, the official checkpoint scores slightly better than our equal-budget retrain, while W⁴Mem remains well ahead of both.

Table R3: WorldMem official checkpoints under the 500-frames evaluation protocol.

ModelFID↓LPIPS↓MSE↓PSNR↑
WorldMem, official checkpoint41.660.37410.039219.85
WorldMem, our reproduction(shared recipe)50.250.40740.047918.97
W⁴Mem (Ours)9.020.26240.030622.65

Response to Q4 & W2

Yes. The detailed configurations are already in Appendix A, Table 11 and Table 12. The release will include:

  • code for W⁴Mem and the re-implemented RealEstate10K baselines;
  • checkpoints of W⁴Mem and the retrained WorldMem for Minecraft and RealEstate10K;
  • the exact RealEstate10K test trajectory lists, the script that builds the 300-frame forward-and-return loops, and the 300 Minecraft test sample IDs;
  • evaluation scripts for pooled and per-bin FID and for revisit-consistency pairs, with the protocol configs.

Thanks for the replay, most of my concerns are solved. And I will maintain my initial rating and recommend weak acceptance.

Reviewer_SXg8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents W⁴Mem for long-horizon video world generation. It selects three types of memory: fixed anchors, revisited views, and complementary views. It also adds noise to memory during training and uses different memory types at different denoising stages. Experiments on Minecraft and RealEstate10K show clear gains in generation quality and long-term consistency.

Critical strengths
  1. The use of anchors and complementary views is interesting. It goes beyond retrieving views that are close to the current camera pose. Anchors provide a fixed reference, while complementary views aim to provide additional visual information. The role replacement experiments support the value of this design.
  2. The performance gains are clear. The method improves both image quality and long-term consistency over WorldMem. The paper also includes tests at later generation stages and after returning to earlier views. These results provide useful evidence beyond average image quality.
Critical weaknesses
  1. The evaluation uses long real observation prefixes: 600 frames for Minecraft and 100 frames for RealEstate10K. It is unclear how much the method depends on this rich initial history. There is no study of prefix length. The current results therefore leave open whether the method remains reliable with much shorter prefixes.
  2. The reported generation speed limits practical use. Although the method is faster than WorldMem, it still takes 1.37 seconds per frame on Minecraft. This is far from real-time generation. The paper does not explore multi-frame generation or clearly discuss the limits of its frame-by-frame setup.
  3. The basic idea behind DMC is not new. Adding noise to clean conditions to handle imperfect generated inputs has been used in earlier work, such as GameNGen [1]. DMC includes specific choices for noise sampling and stabilization, and Table 5 provides useful ablations. However, its added contribution over existing condition-noise augmentation methods needs a clearer explanation.

[1] Diffusion Models Are Real-Time Game Engines (GameNGen).

Questions for authors

Please address the concerns raised in Critical Weaknesses. My current rating is Weak Accept. I am open to revising it after reading the other reviews and the authors’ response.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SXg8 (1/3)

We thank the reviewer for finding our anchors and complementary views interesting, and for recognizing the clear gains over WorldMem in both image quality and long-term consistency, supported by our late-horizon and revisit evaluations.

We thank the reviewer for the insightful review. We add a prefix-length sweep and chunked generation, give a cost breakdown, and clarify how DMC differs from existing noise augmentation.

Response to W1

We thank the reviewer for this important question.

Prefix setup. The 600-frame (Minecraft) and 100-frame (RealEstate10K) prefixes follow WorldMem's evaluation protocol, and all methods receive the same prefix for a fair comparison. W⁴Mem does not rely on a long ground truth history by design. As shown in Figure 11 in Appendix F, our 1000-frame experiment already conditions on a shorter, 400-frame prefix, but the quality doesn't degrade.

New prefix-length ablation. To test whether W⁴Mem depends on a long observed history, in the new experiments, we keep only the last k∈8,50,100,300k \in \\{8, 50, 100, 300\\} frames of the standard 600-frame prefix. The 500 evaluated frames, camera poses, actions, and seeds stay the same for every kk, and both methods see only the kk ground truth prefix frames. Table R1 below and Table 10 in Section 4.4 of the revised manuscript show that W⁴Mem degrades only mildly as the prefix shrinks. Its FID rises 9.02 to 11.50 between k=600k = 600 and k=8k = 8, while the FID of WorldMem rises from 50.25 to 82.52. At k=100k = 100, which matches the 100-frame anchor window used in training, W⁴Mem reaches an FID of 9.37. Even with only 8 prefix frames, W⁴Mem outperforms WorldMem with the full 600-frame prefix on all four metrics.

Table R1: Ablation of prefix-length on Minecraft.

WorldMemW⁴Mem (Ours)
Prefix-lengthFID↓LPIPS↓MSE↓PSNR↑FID↓LPIPS↓MSE↓PSNR↑
882.520.5130.071316.8111.500.29910.0371121.49
5074.340.4900.064717.3410.390.28210.0341222.12
10068.930.4690.060117.739.370.27530.0332022.21
30060.470.4380.053618.359.420.27290.0322322.40
60050.250.4070.047918.979.020.26240.0306322.65

Table R2 below and Table 20 in Appendix E of the revised paper separate two kinds of revisit consistency. Agreement with the ground truth improves with kk for both methods, as expected, because a longer prefix contains more real observations of the places the camera later returns to. For W⁴Mem, RCGT\mathrm{RC_{GT}} at FoV > 0.8 rises from 0.833 to 0.853 and LPIPSGT\mathrm{LPIPS_{GT}} falls from 0.354 to 0.317. Self-consistency compares each generated frame with what the model itself generated earlier at the same view. It stays nearly constant for W⁴Mem, with RCself\mathrm{RC_{self}} between 0.888 and 0.893 for every kk. WorldMem instead loses self-consistency as the prefix shrinks, with RCself\mathrm{RC_{self}} falling from 0.758 to 0.713. A short prefix therefore limits how much of the true scene W⁴Mem can recover, but not how consistently it remembers the scene it has generated.

Table R2: Revisit consistency on Minecraft with a visible prefix of kk frames. Each generated frame is paired with the earlier frame of highest FoV overlap at least 100 frames back. RC is the Inception cosine to that earlier frame, taken from the generated video (self) or the ground truth (GT). LPIPS is computed on the same pairs.

FoV > 0.8FoV > 0.9
MethodkkRC↑ selfRC↑ GTLPIPS↓ selfLPIPS↓ GTRC↑ selfRC↑ GTLPIPS↓ selfLPIPS↓ GT
WorldMem80.71320.65690.49450.60330.70350.64940.49760.6061
500.72300.67350.46870.55980.73010.68000.47990.5681
1000.72600.68300.47850.55570.73730.68260.46980.5609
3000.73080.69830.45430.52580.73350.70550.44090.5139
6000.75760.71780.41690.49100.76450.71900.40780.4904
W⁴Mem (Ours)80.88760.83310.25830.35410.89440.83420.24110.3525
500.89310.84110.25120.33530.89950.84370.23510.3314
1000.89240.84580.25250.32980.89960.85190.23690.3169
3000.89010.84750.25480.33190.90060.85370.23170.3186
6000.89220.85280.25270.31680.89850.85720.23430.3079
AuthorsOpenReview ↗

Response to Reviewer SXg8 (2/3)

Response to W2

We thank the reviewer for pointing out the generation time limit and the lack of multi-frame generation experiments. We conducted new multi-frame generation experiments below and analyze the time cost of each part.

Where the cost comes from. For a controlled fair comparison, W⁴Mem uses WorldMem's backbone and protocol: a ∼0.9B Oasis DiT at 360 × 640, 20 DDIM steps, and frame-by-frame sliding-window generation. In Section 4.2 of the main paper we report 1.37 vs. 1.79 s/frame and 6.18 vs. 9.79 GB for W⁴Mem vs. WorldMem. In Table R3 below and Table 8 in Section 4.4 of the revised manuscript, with details in Appendix D, we add a breakdown of the cost; most of the inference time is spent in the base model, which takes 0.942s. The memory module of W⁴Mem adds only 0.432s, which is 31% of the total runtime. In comparison, WorldMem's memory module takes 0.845s and accounts for 47% of its total runtime, nearly half.

Table R3: Computation cost breakdown on Minecraft.

QuantityUnitWorldMemW⁴Mem (Ours)
Total times/frame1.7871.374
 Shared backbones/frame0.9420.942
 Memory modules/frame0.8450.432
ComputeTFLOPs/frame104.047.7
Memory frames read per denoising stepframes84.98
Trainable parametersM945.2910.6

Multi-frame generation. In the main paper, we generate one frame per denoising trajectory following WorldMem, so that both methods are compared under the same protocol and memory is retrieved separately for each target frame. When cc frames are denoised jointly as a chunk, they share the same DDIM noise level and a single memory set, retrieved once per chunk with the training selection rule. This makes the retrieved context slightly less precise for each individual frame, but it lowers the cost per frame. Table R4 below and Table 21 in Appendix E of the revised paper compare the per-frame inference time for chunk sizes c∈1,2,4c \in \\{1, 2, 4\\}. Larger chunks reduce the latency roughly in proportion to cc, and W⁴Mem remains faster than WorldMem at every chunk size. At c=4c = 4, W⁴Mem takes 0.361 s/frame compared with 0.421 s/frame for WorldMem, which corresponds to about 2.8 generated frames per second.

Table R4: Multi-frame generation speed. Per-frame inference time on Minecraft. The cc frames of a chunk share one DDIM level and one memory retrieval.

s/frame↓
ccWorldMemW⁴Mem (Ours)
11.7871.374
20.8430.733
40.4210.361

Same rollout steps with multi-frame generation. We then compare generation quality with the number of autoregressive rollout steps fixed to 500, so that a chunk size of cc generates 500c500c frames in total. Every chunk size therefore takes the same number of autoregressive steps, while larger chunks extend the generated horizon to 1,000 and 2,000 frames. Table R5 below and Table 22 in Appendix E of the revised manuscript show that W⁴Mem outperforms WorldMem on all four metrics at every chunk size. At c=4c = 4, with 2,000 generated frames, W⁴Mem achieves an MSE of 0.0261 and an FID of 22.73, compared with 0.0415 and 51.63 for WorldMem. Overall, multi-frame generation trades fine-grained memory precision for inference speed, and W⁴Mem keeps a clear advantage in both speed and long-horizon quality.

Table R5: Long-horizon generation with equal rollout steps. Every run takes 500 rollout steps. One step generates one chunk of cc frames, so a run generates 500c500c frames. After the 600 context frames, the camera path goes forward along the test trajectory and then back along it.

MethodccRollout stepsFramesFID↓LPIPS↓MSE↓PSNR↑
WorldMem150050050.250.4070.047918.97
2500100057.040.4230.043618.69
4500200051.630.3990.041519.47
W⁴Mem (Ours)15005009.020.2620.030622.65
2500100020.430.2870.031421.49
4500200022.730.2540.026121.24
AuthorsOpenReview ↗

Response to Reviewer SXg8 (3/3)

Response to W3

We thank the reviewer for this helpful comment and agree that noisy context frames proposed in GameNGen [1].

DMC vs GanmeNGen. GameNGen only noises its local context frames. Our backbone already does the same thing: under Diffusion Forcing, every frame in the local window gets its own noise level during training. The remaining gap is the retrieved long-range memory, which WorldMem always shows clean in training, although at inference time much of it consists of generated frames. DMC extends such noise augmentation to this memory with three designs:

  1. Conditioned and independent memory corruption. Each retrieved memory gets its own noise level and branch.
  2. Noise range tied to the query frame. The noise level is drawn from an interval [λq,λs][\lambda_q, \lambda_s] whose noisy end is set by the cleanest target in the pack. Shuffling the query keeps the same marginal noise distribution but breaks this coupling, and raises MSE by 27.3% (0.0306 → 0.0390, Table 5).
  3. Log-SNR sampling. Sampling uniformly in timestep instead raises FID from 9.02 to 19.29 in Table 5.

References

[1] D. Valevski et al. Diffusion Models Are Real-Time Game Engines (GameNGen). ICLR 2025.

My main concerns are addressed. Thanks. I will maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their thoughtful and constructive feedback. To address the main concerns, we ran several new experiments and analyses, which we summarize here:

  • Prefix-length ablation: We cut the 600-frame prefix down to its last 8, 50, 100, or 300 frames. W⁴Mem holds up well. Its FID is 11.50, 10.39, 9.37, and 9.42 at these lengths, compared with 9.02 for the full prefix. WorldMem degrades much more, from 50.25 to 82.52. With only 8 prefix frames, W⁴Mem still outperforms WorldMem with all 600 on all four metrics.
  • Cost breakdown: Most of the runtime comes from the backbone that both methods share, which takes 0.942 s/frame. Our memory module adds 0.432 s/frame, about half of the 0.845 s/frame that WorldMem's memory module needs.
  • Multi-frame generation: We now also generate 2 or 4 frames at a time. With chunks of 4, W⁴Mem runs at 0.361 s/frame and WorldMem at 0.421 s/frame. We also compare the two under the same 500 rollout steps, where chunks of 4 give a 2,000-frame video. There, W⁴Mem gets an FID of 22.73 and an MSE of 0.0261, while WorldMem gets 51.63 and 0.0415.
  • Clarification of DMC: We explain more clearly what DMC adds over existing condition-noise augmentation such as GameNGen. GameNGen adds noise to the local context frames, and our backbone already does that. DMC goes one step further and also noises the retrieved long-range memory, using a noise range tied to the query frame and log-SNR sampling.
  • Selection-only ablation: To isolate the effect of retrieval, we keep the reader and conditioning fixed and change only which memory frames are selected. Switching from WorldMem's selection to ours brings FID down from 19.42 to 9.02. Both of our memory roles do their job. When we replace the anchors with revisit memory, PSNR drops by 1.9 dB, and when we replace the complementary views, FID goes up from 9.02 to 12.74.
  • Structured memory corruption: We are grateful to the reviewer for suggesting structured perturbations, because they made our model clearly better. During training we now also corrupt memory with blur, photometric drift, affine misalignment, and stale-content patches. Together with Gaussian DMC, this cuts FID from 9.02 to 4.54 and LPIPS from 0.262 to 0.248.
  • Baseline reproduction and training curves: We clarify how we handled the WorldMem curriculum. Our reproduction keeps its three training stages, scaled to our 200k-step schedule. We also add training loss and validation curves for W⁴Mem and our WorldMem reproduction, both trained with the same recipe. The curves flatten out by the end of training, so both models have converged. Finally, we ran the official WorldMem checkpoint through our 500-frame protocol. It scores an FID of 41.66, a little better than the 50.25 of our reproduction, and W⁴Mem remains far ahead at 9.02.
  • Routing mismatch analysis and per-target routing: We add a new analysis of the mismatch between pack-mean routing in training and per-frame routing at inference. In a simulation of 2M training packs, the gate that a target's own timestep would choose agrees with its pack's gate 62.4% of the time. Training therefore teaches a soft, stage-biased version of the hard gate used at inference. We also trained a new variant from scratch in which each target frame sets its own routing gate. It performs close to our pack-mean gate, with an FID of 10.39 compared with 9.02, and both are better than no routing at 11.05 and reversed routing at 13.31.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.