acceptodds
Under review as a conference paper at ICLR 2027

Latent Spatial Memory for Video World Models

33 views

Abstract

Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point-cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce latent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose LSM-World, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves the strongest 3D and photometric consistency on WorldScore and the best closed-loop revisit consistency on RealEstate10K, while matching top baselines in overall score at up to faster generation and lower cache memory.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation, 3D Foundation Model, Spatial Memory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_qPGjOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LSM-World, which improves the long-term spatial consistency of video generation by constructing persistent three-dimensional memory in the VAE latent space. The method uses depth to back-project latent features into three-dimensional space, and then directly projects them based on the pose of the target camera to obtain conditional features. The video generation is guided by control branches, and the memory is updated after each segment is generated. Its main contribution is to avoid image rendering and repetitive VAE encoding in RGB point cloud memory reading, reducing cache overhead. The experiments demonstrated competitive generation quality and consistency, as well as efficiency advantages on WorldScore and RealEstate10K.

Critical strengths

The paper directly constructs and reads the three-dimensional spatial memory in the VAE latent space. By projecting the latent features onto the target perspective, it provides geometric conditions for video generation and avoids the additional overhead of "rendering the target perspective image and then performing VAE encoding" in the RGB point cloud method. Meanwhile, the low spatial resolution of latent reduces the number of cache points and storage requirements. The overall design is simple and can be naturally integrated into the existing video diffusion model, which has practical engineering value.

Critical weaknesses
  1. The method excludes dynamic objects from persistent memory, thus mainly addressing the long-term consistency of static scenes. It is unable to maintain the identity and state of dynamic objects through the proposed memory mechanism.
  2. The use of Qwen3-VL and SAM3 for dynamic region identification and segmentation during reasoning may significantly increase the time cost, but the paper does not explicitly report this part of the overhead and whether it is included in the end-to-end time consumption.
Questions for authors
  1. Is the efficiency and generation effect evaluation only for static scenarios? If so, compared with methods such as Gen3C that support dynamic scenarios, are these results sufficient to support the broader advantageous conclusions of the paper?
  2. What is the call frequency and actual time consumption of Qwen3-VL and SAM3 during inference? Does the end-to-end time of the report include these steps? Please provide a complete time-consuming breakdown including segmentation, depth estimation, memory reading and writing, and video generation.
6 comments
AuthorsOpenReview ↗

Response to Reviewer qPGj (Part 1/2)

We thank Reviewer qPGj for the careful evaluation and constructive feedback. We appreciate your recognition of the method’s “practical engineering value”, its natural integration with existing video diffusion models, and the reduction in cache overhead and storage requirements. We have carefully considered all comments and provide our point-by-point responses below.

W1. Dynamic-object filtering and memory scope

Memory scope. Our persistent memory preserves stable scene structure, while dynamic objects are generated using the video backbone and temporal context carried across chunks. Dynamic regions are filtered only when updating the spatial memory. They remain part of the generated videos.

Writing moving objects into the memory with fixed world coordinates can impose outdated object locations and appearances on subsequent views. Filtering avoids these stale spatial constraints. The current cache does not explicitly store persistent dynamic-object identities or evolving states across revisits. This scope is clarified in Section 4.4 and the limitations in Section 6.

Effect of dynamic filtering. The controlled ablation in Table 6 (Section 5.3) supports this design. With all other settings fixed, filtering improves both generation performance on WorldScore’s dynamic partition and spatial consistency; Appendix C.6 discusses these results.

Memory updateWorldScore ↑WorldScore Dynamic ↑3D consistency ↑
Without dynamic filtering61.2059.7080.88
With dynamic filtering70.3667.1192.21

These results show that excluding dynamic objects from the persistent latent spatial memory benefits dynamic-scene generation, while the backbone and temporal context remain responsible for modeling their motion.

Dynamic-region evaluation. We further evaluate reconstruction quality within dynamic-object regions, which are excluded from the persistent spatial memory. Table 17 (Appendix C.6) reports these scores alongside full-frame results.

Evaluation regionPSNR ↑SSIM ↑LPIPS ↓
Full frame16.010.6450.363
Dynamic-object regions17.060.6810.333

W2. Inference overhead of dynamic-region filtering

Call frequency. Following Spatia, we precompute dynamic-object masks offline during dataset preparation for training. During inference, Qwen3-VL and SAM3 are called once per generated chunk to obtain masks for memory updates, outside the diffusion denoising loop. Section 4.4 and Appendices C.5 and D now clarify this distinction.

Timing scope and overhead. Table 16 (Appendix C.5) extends the profiling in Figure 2(c) to include dynamic-region identification and segmentation. It reports the listed component times, with percentages computed over their sum (41.65 s). Table 2 separately reports the complete RGB–latent runtime comparison, including initialization, memory update, and system overhead.

StageTime (s)Share of total
Dynamic-region identification (Qwen3-VL)0.310.74%
Segmentation (SAM3)0.831.99%
Depth estimation1.092.62%
Memory read/write0.020.05%
Video diffusion39.4094.60%
Total41.65100%

Qwen3-VL and SAM3 together add 1.14 s, accounting for 2.74% of the measured total. Video diffusion remains the dominant cost at 94.60%.

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer qPGj (Part 2/2)

Q1. Dynamic-scene evaluation and comparison with Gen3C

Evaluation scope. No. Our evaluation includes both static and dynamic scenes. W1 reports results on WorldScore’s dynamic partition, reconstruction quality within dynamic-object regions, and a controlled ablation of dynamic filtering (Tables 6 and 17; Appendix C.6). These evaluations assess generated videos containing dynamic objects, while the persistent memory preserves static scene structure.

Direct comparison with Gen3C. We compare Gen3C and LSM-World on the same dynamic scenes, using identical input frames, target camera trajectories, and evaluation masks. Table 18 (Appendix C.6) reports PSNR, SSIM, and LPIPS within dynamic-object regions, together with generation latency covering each method’s complete inference pipeline. Section 5.2 also summarizes the results.

MethodDynamic-region PSNR ↑Dynamic-region SSIM ↑Dynamic-region LPIPS ↓Generation latency (s/frame) ↓
Gen3C15.170.5980.3892.86
LSM-World17.060.6810.3331.35

LSM-World achieves higher PSNR and SSIM and lower LPIPS than Gen3C within dynamic-object regions. The reported latency also corresponds to a 2.12× generation speedup over Gen3C.

Q2. Call frequency and runtime breakdown

Call frequency and cost. Qwen3-VL and SAM3 are called once per generated chunk, outside the diffusion denoising loop. Their measured runtimes are 0.31 s and 0.83 s, respectively, as reported in W2 and Table 16 (Appendix C.5).

Timing scope and breakdown. The additional profiling in W2 and Table 16 (Appendix C.5) extends Figure 2(c) to include both modules, alongside depth estimation, memory read/write, and video diffusion. The complete RGB–latent runtime comparison, including initialization, memory update, and system overhead, is reported separately in Table 2.


We thank you again for your constructive questions about dynamic scenes and inference overhead. We hope the dynamic-region evaluations, Gen3C comparison, and detailed runtime measurements clarify the scope and practical cost of our memory mechanism. We welcome any further comments or suggestions.

Thank you for your detailed reply and updated manuscript. My concerns have been addressed. I find the topic interesting and the contributions substantial. As a supplementary suggestion, please consider adding video examples showing dynamic objects across multiple chunks to illustrate motion continuity.

Reviewer_QoUxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LSM-World stores VAE latent features in a persistent 3D point cache and projects them directly into target views to condition video generation. This avoids rendering and re-encoding RGB images during memory readout.

Critical strengths
  • The design is well-motivated. Operating at latent resolution offers a clear opportunity to reduce memory-access costs.
  • The ablation study is quite good. Replacing RGB memory with latent memory improves Average WorldScore from 67.71 to 70.36, with better geometric and photometric consistency.
  • The empirical evidence is well-supported. Substantial SSIM gains, favorable human evaluations, and reported pipeline timings support the method’s promise.
Critical weaknesses
  • Missing related work. MosaicMem [1] already uses 3D-localized memory patches and warped latent features to condition video generation, which directly overlaps with LSM-World’s motivation.

  • Consistency improves while camera control worsens. Against Spatia, Average WorldScore increases only 0.63, while camera control falls from 75.66 to 55.36 and content alignment from 69.95 to 42.09. Consistency should be evaluated jointly with camera adherence to establish whether the gain persists under comparable realized motion.

[1] MosaicMem: Hybrid Spatial Memory for Controllable Video World Models

Questions for authors

MosaicMem also uses spatially grounded memory with latent-feature warping. Please clarify the methodological differences and expected accuracy–efficiency tradeoffs. Was a quantitative comparison feasible given the availability of its implementation?

6 comments
AuthorsOpenReview ↗

Response to Reviewer QoUx (Part 1/2)

We thank Reviewer QoUx for the thorough evaluation and constructive suggestions. We appreciate your recognition that the design is “well-motivated” and the empirical evidence is “well-supported”, including the controlled memory ablation, SSIM gains, and favorable human evaluations. We have carefully considered all comments and provide our point-by-point responses below.

W1. Relationship to MosaicMem

Thank you for pointing out MosaicMem. We have cited and discussed it in Section 2.3 and added a quantitative comparison in Section 5.2 (Table 5).

Both methods use spatially grounded memory and geometry-aligned latent features. To clarify their relationship, we compare the memory designs below and provide quantitative results under matched training and generation settings.

Memory representation and conditioning. MosaicMem supplies retrieved patches as additional context tokens. LSM-World projects cached features into a fixed target-view grid, keeping the backbone’s conditioning sequence length unchanged as the memory grows.

Quantitative comparison. The official project page lists the code as forthcoming, so we evaluated our own implementation. Our reimplementation includes warped RoPE, warped latent, and PRoPE.

Both methods use the same backbone, training data, training steps, and sampling settings. Table 5 reports their results under the same closed-loop protocol on RealEstate10K:

MethodPSNR_C ↑SSIM_C ↑LPIPS_C ↓Generation latency (s/frame) ↓Cache memory (MiB) ↓
MosaicMem (our reimplementation)18.080.7690.2637.6178.75
LSM-World20.050.8250.2281.352.25

Under matched training and generation settings, LSM-World achieves higher closed-loop PSNR and SSIM and lower LPIPS than our MosaicMem reimplementation. The reported timings and cache footprints also show a 5.64× generation speedup and a 35-fold reduction in cache memory.

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer QoUx (Part 2/2)

W2. Camera control, consistency, and content alignment

Camera control and consistency. With the same backbone and training recipe, latent memory maintains camera-control capability while improving 3D and photometric consistency (Table 8, Section 5.3; Appendix C.2).

MemoryCamera Ctrl ↑3D consistency ↑Photometric consistency ↑
RGB memory30.5190.7591.10
Latent spatial memory55.3692.2193.95

To evaluate consistency at comparable camera adherence, we select paired test cases where Spatia and LSM-World obtain similar Camera Ctrl scores under identical input frames and target camera trajectories. Selection is based only on Camera Ctrl, independently of consistency scores.

Table 11 (Appendix C.2) reports 3D and photometric consistency on this subset, together with Camera Ctrl and actual camera translation and rotation. The latter measurements allow us to assess whether the consistency comparison also involves comparable realized motion.

MethodActual translation (m)Actual rotation (°)Camera Ctrl ↑3D consistency ↑Photometric consistency ↑
Spatia1.9628.366.1285.5988.95
LSM-World2.1227.665.1391.5792.39

Content alignment. WorldScore computes image–text alignment using CLIP, then rescales each sample score using fixed benchmark statistics before averaging. Equation (11) in Appendix C.2 specifies this normalization; on the reported 0–100 scale, it is approximately

S(r)=clip⁡(52.80+19.64(r−26.67), 0, 100),S(r)=\operatorname{clip}\left(52.80+19.64(r-26.67),\,0,\,100\right),

where rr is the raw CLIP score. This scaling expands numerical differences, so Table 4 (Section 5.2) additionally reports raw CLIP scores and a complementary GPT-6-Astra evaluation of frame–prompt consistency. The normalization details and VLM scoring prompt are provided in Appendix C.2.

MethodWorldScore Content Align ↑Raw CLIP score ↑VLM alignment score ↑
Spatia69.9527.5477.33
LSM-World42.0926.1279.82

While Spatia scores higher under CLIP, the VLM evaluation favors LSM-World, providing complementary evidence of its content-alignment quality.

Q1. Comparison with MosaicMem

Methodological differences and accuracy–efficiency tradeoffs. W1 compares the memory representations and conditioning interfaces discussed in Section 2.3; Table 5 (Section 5.2) quantifies the resulting accuracy and efficiency.

Feasibility of quantitative comparison. Yes. We implemented MosaicMem and evaluated it under matched training and generation settings. The implementation details and quantitative results are given in W1 and are now included in Section 5.2 (Table 5).


We thank you again for your careful examination of the related work and evaluation methodology. We hope the MosaicMem comparison and the joint evaluation of camera adherence and consistency address your concerns about the methodological differences and empirical advantages. We look forward to your feedback.

Dear Reviewer QoUx,

We would like to ask whether our rebuttal has clarified the points you raised. Please let us know if any concerns remain. If you find that we have addressed them, we would appreciate it if you could reconsider your assessment.

Thank you again for your time and feedback!

Best regards,

Authors of Submission #1078

Reviewer_QVSXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

LSM-World stores depth-lifted VAE features in a persistent 3D cache and projects them directly into target-view conditioning. It aims to improve spatial consistency while reducing memory-readout costs in camera-controlled video generation.

Critical strengths
  • The method is simple and straightforward. It is a clear, practical design that eliminates RGB rendering and re-encoding during cache readout.
  • The method demonstrates strong reported geometric/photometric consistency. The controlled RGB-cache ablation is very helpful and supports the representation’s contribution.
Critical weaknesses
  • The method is way too straightforward, which is a good thing but also a bad thing. In the simplest terms, the paper proposes to use the VAE latents instead of RGBs as the 3D cache, which can then be read out as memory. I'm not sure whether this itself is of sufficient contribution.
  • Figure 5 measures cache-read latency, which does not establish the claimed 10.57x end-to-end speedup. I do not get how the caching translated to the 10.57x end-to-end speedup, could the authors elaborate? These numbers are very confusing to me from an infra perspective.
  • The authors should consider discussing/citing additional work on memory, compression, and efficient context modeling for video generation and understanding, including:
    • WorldMem: Long-term Consistent World Simulation with Memory. (NeurIPS 2025)
    • Mixture of Contexts for Long Video Generation. (ICLR 2026)
    • SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer. (ICLR 2026)
    • Pretraining Frame Preservation in Autoregressive Video Memory Compression. (arXiv 2026)
    • MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation. (ICLR 2026)
    • StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding. (arXiv 2025)
Questions for authors

N/A

6 comments
AuthorsOpenReview ↗

Response to Reviewer QVSX (Part 1/2)

We thank Reviewer QVSX for the careful evaluation and constructive feedback. We appreciate your recognition of our method as a “clear, practical design”, its strong geometric and photometric consistency, and the controlled RGB-cache ablation as “very helpful” for assessing the representation’s contribution. We have carefully considered all comments and provide our point-by-point responses below.

W1. Simplicity and contribution

Solving a substantial practical bottleneck. We identify and address a substantial bottleneck in RGB-memory video world models: obtaining target-view conditioning requires pixel-resolution rendering and VAE re-encoding. In our controlled RGB-memory baseline, memory readout alone takes 42.72 s/chunk, exceeding the time spent on video generation itself (38.41 s/chunk; Table 2, Section 5.2).

LSM-World eliminates this detour through direct latent-resolution readout, reducing readout time to 0.02 s/chunk. With the same backbone and generation settings, total runtime decreases from 190.07 to 47.40 s/chunk (4.01× faster), while WorldScore increases from 67.71 to 70.36 (Table 6, Section 5.3). The simplicity of this solution is a strength: it removes expensive operations while improving generation quality.

More than replacing RGB with VAE latents. Our contribution extends beyond changing the stored feature type. We develop this idea into a complete autoregressive video-generation system, including depth-guided memory construction at native latent resolution, visibility-aware target-view readout, learned conditioning on sparse projections, and persistent memory updates across generated chunks (Sections 4.2–4.5; Appendix D). These contributions are now stated explicitly in Section 1.

These design choices have substantial, measurable effects. Table 6 (Section 5.3) evaluates them under the same backbone and training recipe:

ConfigurationWorldScore ↑3D consistency ↑Photometric consistency ↑
RGB memory67.7190.7591.10
Latent features upsampled to pixel resolution60.8584.9079.81
Without dynamic-object filtering61.2080.8876.10
Full LSM-World70.3692.2193.95

Constructing memory at native latent resolution instead of upsampling latent features improves WorldScore by 9.51 points and 3D consistency by 7.31 points. Filtering dynamic regions during updates improves these scores by 9.16 and 11.33 points, respectively. Merely storing VAE latents does not deliver these gains. How memory is constructed and maintained is essential to its effectiveness.

W2. Origin of the 10.57× speedup

Where 10.57× comes from. The reported 10.57× speedup is measured from generation throughput: 0.74 FPS for LSM-World versus 0.07 FPS for Voyager. Figure 5 separately profiles memory-readout latency. Section 5.2 distinguishes these measurements, and Table 2 provides the controlled comparison of the complete RGB-memory and latent-memory pipelines reproduced below.

MethodGeneration throughput (FPS) ↑
Voyager0.07
LSM-World0.74

How latent memory reduces generation time. RGB memory constructs target-view conditioning by rendering a pixel-resolution image and passing it through the VAE encoder. Our memory directly projects cached features onto the target latent grid. At our evaluation resolution, this replaces rendering at 704 × 1280 followed by VAE encoding with projection at 44 × 80 (Section 4.3; Appendices B and D).

To isolate the runtime benefit of this change, we compare RGB and latent memory using the same backbone, resolution, sampling steps, precision, hardware, and camera trajectories, at matched rollout lengths. We report synchronized wall-clock time per generated chunk:

Stage (s/chunk) ↓RGB memoryLatent spatial memory
Memory initialization17.15—
Memory readout42.720.02
Video generation38.4139.32
Memory update85.377.17
System overhead6.420.89
Total runtime190.0747.40
Speedup relative to RGB memory1.00×4.01×

In this controlled comparison, latent memory reduces total runtime from 190.07 to 47.40 s/chunk, giving a 4.01× speedup over RGB memory. The largest measured savings come from memory update (78.20 s) and memory readout (42.70 s), while video-generation time remains similar.

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer QVSX (Part 2/2)

W3. Relationship to additional related work

Thank you for suggesting these relevant works. The comparison below, now included as Table 19 in Appendix C.7, covers long-term consistency, camera control, explicit spatial alignment, conditioning scalability, and compatibility with existing backbone attention.

MethodLong-term consistencyCamera controlExplicit feature alignmentBounded context lengthNo attention redesign
WorldMem✓✓✗✓✗
PFP✓✗✗✗✓
MoC✓✗✗✓*✗
MoGA✓✗✗✗✗
SANA-Video✓✗✗✓✗
StreamMem———✓✓
LSM-World✓✓✓✓✓

*MoC includes outer-loop routing; PFP refers to its base configuration; —: not applicable.

WorldMem. WorldMem supports long-term scene consistency and camera control through pose-conditioned memory attention over retrieved historical frames. LSM-World explicitly projects cached features into target-view coordinates before denoising. This supplies spatially aligned appearance features, whereas WorldMem learns the correspondence through memory attention.

PFP and StreamMem. PFP compresses video history into shorter contexts without redesigning backbone attention in its base configuration; at a fixed compression ratio, the context still grows with history length. StreamMem prunes and merges KV entries to maintain a fixed history budget for video understanding. LSM-World provides a fixed-size generation condition through geometric projection of its scene cache.

MoC and MoGA. MoC and MoGA support long-range video coherence through sparse attention over selected context chunks or token groups. MoC additionally bounds the supplied context through outer-loop routing; MoGA reduces attention interactions without fixing the total context length. LSM-World retains the backbone’s attention structure and supplies spatial memory through an auxiliary conditioning branch.

SANA-Video. SANA-Video supports long-video generation with a constant-size recurrent state through block linear attention. LSM-World also keeps the history-conditioning size fixed, while preserving the backbone’s attention structure and providing explicit target-view feature alignment.

We have cited these works and incorporated this discussion into Sections 2.3–2.4, with the detailed comparison in Appendix C.7.


We thank you again for your constructive feedback on the contribution, efficiency, and related work. We hope the component ablations, controlled runtime breakdown, and expanded comparisons clarify the contribution and the evidence supporting this simple design. We welcome any further comments or suggestions.

Dear Reviewer QVSX,

We would like to ask whether our rebuttal has clarified the points you raised. Please let us know if any concerns remain. If you find that we have addressed them, we would appreciate it if you could reconsider your assessment.

Thank you again for your time and feedback!

Best regards,

Authors of Submission #1078

Reviewer_noqyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a 3D latent spatial memory for video generation, improving spatial consistency and efficiency by storing and projecting VAE features directly. The idea is interesting and the results are promising, but the consistency of the latent representation and performance over longer rollouts need further analysis.

Critical strengths
  • The paper proposes a 3D spatial memory that stores native VAE latent features at world-space coordinates. Reading directly from this memory avoids rendering RGB images and encoding them again, making it a meaningful architectural improvement over RGB-based memory systems.
  • The integration with pretrained video diffusion models is well designed. Depth-guided back-projection, visibility-aware readout, and a ControlNet-style branch work together coherently, while the two-stage training strategy helps preserve the pretrained generative prior.
  • The results show promising gains in spatial consistency and efficiency. On RealEstate10K, the method reaches an SSIM of 0.825 in closed-loop evaluation, compared with 0.579 for Spatia. The reported 10.57× generation speedup and 55× reduction in cache memory are also compelling.
Critical weaknesses
  • The key assumption that VAE latent tokens can serve as spatially consistent 3D features needs stronger support. Each token may cover a large receptive field and mix foreground and background, so assigning it to a single 3D coordinate could cause misalignment under viewpoint changes, occlusions, or depth errors. The reported cache hole rate of 42.53% also suggests that the projected memory is fairly sparse.
  • The results do not consistently outperform existing methods across all metrics. Although the method achieves the best overall WorldScore, its camera control and content alignment scores are 55.36 and 42.09, compared with 75.66 and 69.95 for Spatia. This trade-off deserves more discussion and analysis.
  • The long-horizon evaluation is somewhat limited. The efficiency comparison mainly considers five generation chunks, which leaves the scalability of the append-only memory unclear for much longer rollouts, especially as the cache grows.
Questions for authors
  • Can the authors directly measure the cross-view consistency of the cached latent features? Does the memory preserve view-consistent 3D representations, or does it mainly provide useful geometric conditioning that the generative backbone compensates for?
  • How robust is the method to depth errors, occlusions, and large viewpoint changes? Can the authors analyze their effects on feature alignment and the cache hole rate?
  • Can the authors evaluate substantially longer rollouts and report how cache size, generation time, and consistency change as the memory grows?
4 comments
AuthorsOpenReview ↗

Response to Reviewer noqy (Part 1/3)

We thank Reviewer noqy for the thorough evaluation and constructive suggestions. We appreciate your recognition of latent spatial memory as “a meaningful architectural improvement”, its “well designed” integration with pretrained video diffusion models, and the promising gains in spatial consistency and efficiency. We have carefully considered all comments and provide our point-by-point responses below.

W1. Feature alignment and sparse memory coverage

Feature alignment. The projected latent readouts retain recognizable scene structure and appearance before diffusion, as shown by their direct VAE decodings in Figure 7 (Appendix A). Depth and camera geometry determine where cached features are projected, while z-buffering selects the frontmost feature at each target cell.

Assigning one depth to a token can introduce alignment errors near foreground–background boundaries. We assess feature agreement by comparing memory readouts with VAE-encoded ground-truth target views; this experiment is now reported in Table 7 (Section 5.3; Appendix C.1). Both variants use identical source frames and target cameras, and metrics are computed on their common valid cells. RGB readouts are VAE-encoded for comparison in the same feature space.

MemoryCosine similarity ↑L2 error ↓
RGB memory0.2166.71
Latent spatial memory0.4033.18

RGB memory reconstructs conditioning features by encoding a rendered target-view image, whose holes and projection artifacts can affect feature extraction. Latent memory directly projects the VAE features already extracted from observed frames. The higher cosine similarity and lower L2 error support the benefit of retaining these native features before diffusion.

Sparse memory coverage. The 42.53% hole rate measures unoccupied target latent cells. It does not directly measure the amount of visual information lost. We evaluate the usefulness of the resulting sparse conditioning through the controlled ablation in Table 6 (Section 5.3), using the same backbone and training recipe.

MemoryWorldScore ↑3D consistency ↑Photometric consistency ↑
RGB memory67.7190.7591.10
Latent spatial memory70.3692.2193.95

As clarified in Section 4.3, the memory readout provides sparse spatial conditioning for diffusion. The diffusion model can complete missing regions and refine imperfect projections using the readout and temporal context. The controlled ablation shows that these sparse latent readouts remain effective, achieving higher WorldScore, 3D consistency, and photometric consistency than RGB memory.

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer noqy (Part 2/3)

W2. WorldScore results

Camera control. LSM-World substantially exceeds all general video baselines in Table 1 and receives the highest Camera Following rating in our blinded human study. The automatic and human scores are summarized in Table 10 (Appendix C.2), with the human-study protocol in Appendix E.

MethodWorldScore Camera Ctrl ↑Human Camera Following (0–5) ↑
CogVideoX-I2V (best general video baseline)38.27—
Spatia75.663.9
LSM-World55.364.3

We further isolate the effect of the memory representation using the same backbone and training recipe in Table 8 (Section 5.3; Appendix C.2).

MemoryCamera Ctrl ↑3D consistency ↑Photometric consistency ↑
RGB memory30.5190.7591.10
Latent spatial memory55.3692.2193.95

In this controlled comparison, latent memory achieves higher camera-control, 3D-consistency, and photometric-consistency scores than RGB memory under the same backbone and training recipe.

Content alignment. WorldScore computes image–text alignment using CLIP, then rescales each sample score using fixed benchmark statistics before averaging. Equation (11) in Appendix C.2 specifies this normalization; on the reported 0–100 scale, it is approximately

S(r)=clip⁡(52.80+19.64(r−26.67), 0, 100)S(r)=\operatorname{clip}\left(52.80+19.64(r-26.67),\,0,\,100\right)

where rr is the raw CLIP score. This scaling expands numerical differences, so Table 4 (Section 5.2) additionally reports raw CLIP scores and a complementary GPT-6-Astra evaluation of frame–prompt consistency. The normalization details and VLM scoring prompt are provided in Appendix C.2.

MethodWorldScore Content Align ↑Raw CLIP score ↑VLM alignment score ↑
Spatia69.9527.5477.33
LSM-World42.0926.1279.82

While Spatia scores higher under CLIP, the VLM evaluation favors LSM-World, providing complementary evidence of its content-alignment quality.

W3. Scalability over longer rollouts

We extend the evaluation from five to ten generation chunks. Table 15 (Appendix C.4) reports generation time and accumulated cache size for LSM-World, together with DINO similarity for LSM-World and Spatia; the findings are also summarized in Section 5.2.

ChunkGeneration time (s/chunk)Cache size (MiB)DINO similarity: LSM-World ↑DINO similarity: Spatia ↑
163.460.4580.8020.775
220.300.9040.6580.434
320.311.3560.5960.316
420.311.8010.5500.194
520.302.2470.5390.147
620.332.6960.5100.098
720.383.1440.4890.063
820.313.5910.4860.052
920.344.0400.4830.024
1020.334.4860.4760.016

The cache grows approximately linearly as new observations are added, but its footprint remains small because features are stored at latent resolution. After the first chunk, generation time remains stable despite the growing cache, demonstrating manageable memory accumulation over the evaluated rollout.

LSM-World maintains higher DINO similarity than Spatia throughout the sequence, with a larger separation in later chunks. Although consistency declines for both methods as generation progresses, latent spatial memory preserves a clear advantage while maintaining stable generation speed.

(see next comments for more response, thanks)

AuthorsOpenReview ↗

Response to Reviewer noqy (Part 3/3)

Q1. Cross-view consistency and geometric conditioning

**Direct measurement.**The comparison in W1, now reported in Table 7 (Section 5.3; Appendix C.1), evaluates target-view memory readouts against VAE-encoded ground-truth target views using cosine similarity and L2 error. Both memory variants are evaluated on their common valid cells before diffusion, directly assessing feature agreement without the influence of subsequent denoising.

Role of the memory. The memory preserves spatially anchored appearance information and provides geometric conditioning. Figure 7 (Appendix A) shows that projected memory features already decode into recognizable target-view structure and appearance without diffusion. This demonstrates that the readout itself carries scene content, rather than relying entirely on the backbone to reconstruct it.

Camera-based projection aligns this stored content with the requested viewpoint. The diffusion backbone then completes missing regions and refines imperfect projections using the memory readout and temporal context, as clarified in Section 4.3.

Q2. Robustness to depth errors, occlusions, and large viewpoint changes

The following robustness analyses are included in Appendix C.3. We measure DINO similarity between VAE-decoded memory readouts and target images to assess readout agreement. Hole rate measures empty readout cells, while PSNR evaluates the generated images.

Depth errors. We assess sensitivity to estimated depth by replacing the depth estimator while keeping the remaining settings fixed (Table 12).

Depth estimatorWorldScore ↑Readout DINO similarity ↑Hole rate ↓
DepthAnything 370.360.66942.53%
MapAnything69.660.62148.72%
UniDepth69.130.64442.55%

Generation performance remains stable across the tested estimators despite variations in readout agreement and coverage. This supports robustness to the depth-estimation differences encountered in these comparisons.

Occlusions. We separately evaluate background regions visible in both frames and regions occluded in source frame A but revealed in target frame B (Table 13).

Region in BPSNR ↑Readout DINO similarity ↑Hole rate ↓
Visible in both A and B17.740.65033.14%
Occluded in A, revealed in B17.430.61658.47%

Newly revealed regions have substantially more holes, but their generated-image PSNR remains close to that of jointly visible regions. This supports effective completion of disoccluded content despite reduced memory coverage.

Large viewpoint changes. We group test trajectories by camera translation magnitude, using its 33rd and 67th percentiles across the test set as thresholds for small, medium, and large motion. Table 14 reports the results.

Camera motionPSNR ↑Readout DINO similarity ↑Hole rate ↓
Small19.690.76833.73%
Medium18.230.72940.72%
Large16.980.60162.32%

In the large-motion group, memory readouts retain target-view appearance information even though most target cells lack projected observations. However, both readout agreement and generation fidelity decrease as viewpoint changes grow. These results show that the memory remains informative under sparse coverage, while large viewpoint changes remain more challenging for accurate generation.

Q3. Longer-rollout evaluation

W3 reports the extended ten-chunk evaluation of cache size, generation time, and DINO similarity, now included in Table 15 (Appendix C.4) and summarized in Section 5.2. Cache size grows approximately linearly, while generation time remains stable after the first chunk and DINO similarity stays higher than Spatia throughout the evaluated rollout.


We thank you again for the detailed suggestions. We hope the direct feature measurements, robustness analyses, metric comparisons, and extended rollout evaluation address your concerns about latent memory and its scalability. We welcome any further questions or feedback.

Dear Reviewer noqy,

We would like to ask whether our rebuttal has clarified the points you raised. Please let us know if any concerns remain.

Thank you again for your time and feedback!

Best regards,

Authors of Submission #1078

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.