- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents W4 Mem, a memory framework for improving long-horizon consistency in video world models. The authors argue that effective memory utilization depends on three factors: relevance, reliability, and timing.
To address these factors, the proposed framework introduces three components. First, Role-Factorized Retrieval organizes historical observations into anchors, revisits, and complementary views, which are integrated through a shared role- and geometry-aware memory reader. Second, Diffused Memory Conditioning (DMC) introduces query-conditioned noise corruption during training to improve robustness to imperfect generated memories without requiring autoregressive training rollouts. Third, Denoising-Stage Routing selectively activates different memory roles according to the current denoising stage, improving computational efficiency.
Experiments on Minecraft and RealEstate10K demonstrate improvements in long-horizon generation quality and scene consistency. On Minecraft, W4 Mem reduces MSE by 36.1%, LPIPS by 35.6%, and FID by 82.0% relative to WorldMem, while reducing inference time and peak GPU memory usage. The paper also includes component ablations, memory-budget evaluations, and revisit-consistency measurements to investigate the effectiveness of the proposed design.
Critical strengths
S1. A well-motivated and practically meaningful framework for long-horizon video memory.
The paper addresses an important limitation of current video world models: maintaining scene consistency when previously observed information leaves the local context and autoregressive errors accumulate.
I find the decomposition of memory utilization into relevance, reliability, and timing useful. Rather than focusing exclusively on memory capacity or retrieval accuracy, the authors investigate how different memory sources should be selected, presented during training, and accessed throughout denoising.
The proposed framework is conceptually coherent, and the three components address distinct aspects of long-horizon memory utilization. In particular, Diffused Memory Conditioning and Denoising-Stage Routing provide interesting approaches to improving memory robustness and computational efficiency.
S2. Strong empirical results supported by extensive evaluations and ablations.
The paper demonstrates substantial improvements across multiple metrics on Minecraft and RealEstate10K. The gains over WorldMem are especially pronounced on long Minecraft rollouts, while the comparison with additional baselines, including DecMem and Geometry Forcing, provides useful evidence of practical effectiveness.
The experiments also go beyond standard frame-level reconstruction metrics. The revisit-consistency evaluation directly examines whether the model preserves previously observed scene information, which is well aligned with the paper's motivation.
The component ablations, memory-budget experiments, and reversed-routing comparison further strengthen the empirical evaluation. I also find the simultaneous improvements in generation quality, inference speed, and memory efficiency particularly valuable for practical deployment.
Critical weaknesses
W1. The training–inference consistency and underlying mechanism of denoising-stage routing require further clarification.
A central contribution of the paper is Denoising-Stage Routing, which assigns different memory roles to different denoising stages. However, the routing condition differs between training and inference in a potentially important way.
During training, the routing decision is based on the mean timestep of a target pack, while individual target frames are assigned independently sampled diffusion timesteps. Consequently, target frames at substantially different noise levels may share the same memory-routing decision. At inference, routing instead depends on the current DDIM timestep. Although the same routing threshold is used, the statistical properties and interpretation of the routing condition differ between training and inference.
This raises a question about whether the model actually learns the proposed coarse-to-fine allocation of memory roles at the individual-target level. The reversed-routing and no-routing ablations demonstrate that the proposed schedule is empirically effective. However, they do not fully establish that its effectiveness arises from the claimed correspondence between memory roles and denoising stages. A comparison with per-target timestep routing, or an analysis of routing behavior under different target-timestep distributions, would help substantiate this central mechanism.
Questions for authors
Q1. How does the mean-timestep routing used during training affect consistency with inference-time routing?
During training, individual target frames have independently sampled diffusion timesteps, but the routing decision is determined by the mean timestep of the target pack.
Could the authors clarify how this shared routing decision supports the proposed stage-specific use of memory when different target frames may be at substantially different noise levels? In particular, have the authors compared mean-timestep routing with routing based on individual target timesteps? Such a comparison would help determine whether the proposed design benefits from genuine stage-specific memory allocation or from a more general form of conditional memory dropout.
Response to Reviewer 2kJs
We thank the reviewer for finding our decomposition into relevance, reliability, and timing useful and our framework coherent, and for recognizing our strong results, our revisit-consistency evaluation, and the joint gains in quality, speed, and memory efficiency.
We appreciate the reviewer for these helpful comments. We acknowledge that in training, per-target timestep routing and pack-mean routing act differently. We show the analysis, and we also add new per-target routing trained from scratch as below and in Appendix E of the revised paper.
Response to Q1 & W1
Analyze of training/inference mismatch. We simulated 2M training packs with the exact training sampler, in which the 8 targets draw their timesteps independently.
Training therefore teaches a soft, stage-biased exposure with a role-dependent corruption level, while inference applies a hard per-frame gate.
Mismatch of routing in training and inference. Following WorldMem, we retrieve memory once per 8-frame target pack during training, using the camera pose of the pack's last frame. These memory frames pass through the spatial and temporal attention layers together with the targets, so retrieving a separate memory set for every target would put eight times as many memory frames through these layers and make training much more expensive. With one memory set per pack, the routing gate is set once from the pack's mean target timestep. We agree with the reviewer that a shared memory set does not require a shared gate, since each target can read the shared memory through a gate set by its own timestep. We therefore trained this per-target variant from scratch, keeping one DMC bound for all roles. As Table R1 below and Table 18 in Appendix E of the revised paper show, the per-target gate is almost same our pack-mean gate on all four metrics.
Table R1: Ablation of routing during training on Minecraft.
Does the model benefit from just generic dropout? We believe no, as reversed routing hides each role in the same share of packs and only flips the same soft pattern, so if routing were simply dropout, both variants should perform equally. Instead, as shown in Table 6 in the main paper and Table R1 above, reversed routing raises FID from 9.02 to 13.31 and is also worse on MSE and LPIPS, while removing routing raises FID to 11.05. Since reversal also flips the coupling between each role and its corruption level, these ablations show that pairing roles with noise stages and corruption levels during training matters.