acceptodds
Under review as a conference paper at ICLR 2027

GROME: GROunding Factorized MEmory into Video World Models

Abstract

Long-horizon consistency is essential for video world models, yet it becomes difficult to maintain once observed content falls outside the context window. Existing memory mechanisms either retrieve history as implicit context, which remains constrained by context capacity and entangles static and dynamic information or consolidate history into explicit global representations, which struggle to model evolving dynamics and accumulate geometric errors. To overcome these limitations, we introduce **GROME**, a novel factorized memory framework that augments video world models for long-horizon coherence across both static scenes and dynamic entities. GROME operates via three synergistic components: 1) *Static-Dynamic Factorization* decouples historical evidence into static and dynamic components, preserving static scene geometry and dynamic entity states separately; 2) *Local Memory Retrieval* selects spatially aligned static history to eliminate geometric drift alongside the latest dynamic entity states to ensure temporal continuity; 3) *Training-Free Memory Injection* seamlessly grounds the factorized memory into video world models via their native conditioning pathways. Experiments on long-horizon revisit scenarios demonstrate that GROME substantially enhances scene persistence and dynamic entity fidelity over existing methods, all while preserving the visual quality and controllability of the frozen backbone without additional training. Beyond self-rollouts, GROME readily accommodates external observations as plug-and-play memory, unlocking flexible and controllable video continuation and scene exploration.

Primary area:
generative models
Keywords:
Video World Models, Video Generation, Memory, Training-Free Adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xFuROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents GROME, a training-free memory mechanism for long -horizon generation with action-conditioned video diffusion models. GROME separates the memory into static and dynamic components, both of which are represented by point clouds. The static and dynamic tokens are retrieved and injected in different ways in the final generation pipeline. GROME is implemented on LingBotV2. It is compared with other SOTA methods on the WorldScene Benchmark. Experiments show that GROME achieves the best revisit and quality scores.

Critical strengths
  • The paper introduces a point cloud-based memory mechanism that decouples static and dynamic components. Although previous works have explored point cloud-based memory, detecting dynamic objects with segmentation and tracking and processing them in a separate branch is a novel approach.
  • The paper's presentation is clear and technically sound. The ablation study verifies all design choices. Using only static or dynamic tokens leads to a drop in revisit consistency. The comparison shows that GROME outperforms other SOTA long-horizon generation methods on the WorldScene Benchmark.

Considering sufficient technical contribution and solid experimental results, I recommend accepting this paper.

Critical weaknesses
  • The additional computations for point cloud processing (depth estimation) and dynamic object detection may introduce overhead and affect the performance for real-time applications. The runtime comparison with other methods could be provided.
  • The method cannot overcome the limitations of point cloud representation. The consistency of the generated video may degrade when the scene cannot be well represented by point clouds. For example, in the waterfall example on the supplementary webpage, the identity of the waterfall changes after revisiting. Because this is a common weakness of all point cloud-based methods, it does not affect the paper's technical contribution.
Questions for authors
  • Can the authors discuss the runtime of GROME compared to other SOTA methods?
  • Can the authors discuss the limitations of point cloud representation and how it affects the performance of GROME in different scenes?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xFuR (1/1)

We thank the reviewer for their encouraging comments and constructive suggestions. Below are our point-by-point responses:


W1&Q1. Can the authors discuss the runtime of GROME compared to other SOTA methods?

We thank the reviewer for this insightful suggestion. To provide a clearer understanding of GROME's computational cost, we have conducted end-to-end runtime comparisons with representative world models, including LingBotV2, WorldPlay, VMem, Gen3C, WAH, and AlayaWorld. The results and detailed runtime analysis are also provided in the revised Appendix D.


Runtime Comparison with SOTA Models

MethodGPUsFull Rollout (s)
LingBotV24323.0
WorldPlay4622.3
VMem118,409.7
Gen3C13,273.9
WAH1813.0
AlayaWorld1306.0
GROME-static (93 frames / chunk)31,910.2
GROME-dynamic (93 frames / chunk)32,197.1
GROME-dynamic (45 frames / chunk)31,110.5

As shown in the table, point-cloud-based explicit memory methods (such as VMem and Gen3C) inherently incur inference costs, indicating that computational overhead is a shared challenge across explicit 3D memory mechanisms. For GROME, the extra runtime compared to the original LingBotV2 backbone primarily stems from online geometric estimation, representing a natural trade-off between computational overhead and long-horizon structural consistency.

Nevertheless, reducing the geometric estimation window from 93 to 45 decreases the full rollout time from 2,197.1 s to 1,110.5 s. As further supported by our analysis in Appendix D, DA3 inference accounts for the majority of GROME's extra computational cost, making depth estimation acceleration a key priority.

In future work, we plan to explore chunk-delayed updates and lightweight streaming 3D reconstruction models as alternatives to the current depth estimation pipeline, aiming to reduce geometric reconstruction latency while preserving memory fidelity.

W2&Q2. Can the authors discuss the limitations of point cloud representation and how it affects the performance of GROME in different scenes?

We thank the reviewer for raising this important question regarding the limitations of point-cloud representations and their impact on GROME across different scene types. To investigate this, we separately evaluate the Fluid and Deformable subsets of our test set and compare their revisit consistency against the average performance on static scenes. The quantitative results are reported in Appendix F, together with additional qualitative video examples in the updated supplementary material.


Quantitative Results on Fluid and Deformable Scenes

Scene TypePSNR ↑SSIM ↑LPIPS ↓
Fluid17.7703 ± 0.31540.5101 ± 0.03760.1967 ± 0.0890
Deformable19.0088 ± 0.46580.5820 ± 0.05870.1803 ± 0.0635
Average (Static Scenes)20.6200 ± 0.41960.6520 ± 0.02660.1690 ± 0.0178

The results show that revisit consistency on fluid and deformable scenes is lower than the average performance on static scenes, with fluid scenes showing a more pronounced degradation. This highlights the challenge of maintaining reliable static memory as scene geometry evolves over time. When scene geometry remains stable, historical local pointmaps from multiple timesteps provide consistent spatial references. However, for fluid and deformable scenes, continuous geometric evolution may introduce conflicts when historical local pointmaps are consolidated, causing the resulting spatial references to deviate from the current scene geometry.

As illustrated in the supplementary videos, although the pretrained video world model can generate plausible motion based on its learned generative priors and recent context, conflicting historical geometric references may still affect revisit consistency. This is particularly evident in waterfall scenes, where the generated water continues to flow, but waterfall geometric changed after revisiting. Based on these obersevations above, future work could explore improved retrieval for non-rigid content in static memory or explicitly model its temporal evolution in dynamic memory to condition video generation.

Thanks for the authors' reply. For Q1, I understand that all point-cloud-based methods incur additional computational overhead. However, GROME has no advantages in inference speed compared to other point-cloud-based methods (like Gen3C). Compared to the video backbone LingbotV2, the inference is at least 3$\times$ slower. Therefore, the efficiency concern remains. For Q2, the authors find a similar limitation in representing Fluid with point clouds.

Overall, my concerns about the weakness of this work still exist. Since these limitations are tightly related to point cloud representation itself, I maintain my rating as Weak Acceptance.

Reviewer_hpb8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GROME, a training-free memory framework that helps video world models remember content after it falls out of the context window during autoregressive generation. GROME runs depth estimation, depth alignment, and motion segmentation on each generated chunk, splitting the history into static scenes and dynamic entities. Both are kept as local point clouds rather than fused into a global one, and dynamic entities also carry timestamps. At generation time, the static memory selects complementary frames by maximum coverage and renders them into the target view, while the dynamic memory keeps only the latest state of each entity along with its original temporal RoPE. These references are then injected into a frozen LingBotV2 through value-statistics alignment, separate attention branches for static and dynamic memory, and timestep-aware routing, all within the original conditioning budget. On an explore-and-revisit benchmark of 200 WorldScore scenes (100 static, 100 dynamic), GROME substantially improves revisit consistency and camera controllability over the original LingBotV2.

Critical strengths
  1. GROME doesn’t require any training. It leaves the backbone weights alone and simply plugs into the model’s existing history-conditioning pathway, so it’s easy to deploy and doesn’t hurt the visual quality or controllability of the pretrained model.

2.The injection design is thoughtful, and each piece has a clear reason behind it. Adjusting only the values, not the keys, keeps the original attention pattern intact. Static references share the target frames’ temporal RoPE while dynamic references keep their own timestamps, so the model can tell the two kinds of memory apart. And because static memory is only used in the early denoising steps, warping artifacts don’t end up in the fine details.

3.Storing observations as local point clouds and combining them only when needed is a sensible choice. The model can still pull together evidence from several views, without the layering artifacts that build up when everything is repeatedly fused into one global point cloud.

Critical weaknesses
  1. All experiments are run on LingBotV2 alone. Since being training-free and plug-and-play is the paper's main selling point, this is a real gap. The injection relies on a specific attention structure and RoPE design, so it's unclear whether it would carry over to other world models such as SANA-WM.

  2. The static/dynamic split depends on visible motion. An object that sits still for a while and only starts moving later would be stored as static and stay frozen in memory. The paper also doesn't look separately at fluids or scenes with large non-rigid motion.

  3. Inference is expensive, which doesn't fit well with the interactive setting the paper targets. Each DA3 geometry update takes about 55 seconds on three A800 GPUs, far from real time. In practice, the cost saved by skipping training is simply moved to inference.

Questions for authors

Please see the weaknesses above. Besides, I also find the technical novelty limited, since both the memory retrieval and the idea of separating static and dynamic content come from prior work, so I’d ask the authors to clarify what they see as the core contribution of this paper.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hpb8 (1/3)

Thanks for the constructive comments, we provide our point-to-point feedbacks as follows:


W1 All experiments are run on LingBotV2 alone. The injection relies on a specific attention structure and RoPE design, so it's unclear whether it would carry over to other world models.

We thank the reviewer for this insightful question. To address this concern, we have further integrated GROME into two additional video world models, SolarWM [1] and SANA-WM [2].

Specifically, GROME separately factorizes scene structures and dynamic entity states into static and dynamic memories, retrieves relevant historical information guided by the target view, and selects the latest valid state to maintain dynamic entity consistency. For memory injection, GROME leverages the native attention mechanisms and RoPE-based spatiotemporal positional encodings shared across backbones to condition current generation on retrieved memory. These allow GROME to be integrated into all three backbones using the same memory construction, retrieval, and injection strategy, without additional training.

The quantitative results on SolarWM and SANA-WM are reported in the following tables, with qualitative comparisons provided in the supplementary material.


Comparison on Static Scenes

SettingPSNR ↑SSIM ↑LPIPS ↓CLIP Aesthetic ↑Subjective Qual. ↑Trans. Err. ↓Rot. Err. (°) ↓Prompt Cons. ↑
SolarWM baseline16.3773 ± 0.78210.4754 ± 0.05340.3977 ± 0.03665.4476 ± 0.17200.5573 ± 0.04061.9826 ± 0.27506.2931 ± 1.03190.2560 ± 0.0212
SolarWM + GROME20.4935 ± 0.60830.6457 ± 0.06760.1718 ± 0.04485.5723 ± 0.19850.6069 ± 0.03930.6128 ± 0.10374.1928 ± 1.23850.2389 ± 0.0194
SANA-WM baseline12.6547 ± 0.30080.3509 ± 0.01950.6686 ± 0.01195.1078 ± 0.06820.3625 ± 0.02160.7045 ± 0.03209.8103 ± 1.05240.2640 ± 0.0076
SANA-WM + GROME12.9505 ± 0.32580.3666 ± 0.02000.6401 ± 0.01395.1463 ± 0.06930.3505 ± 0.02200.6981 ± 0.03009.5481 ± 1.07880.2597 ± 0.0090

Comparison on Dynamic Scenes

SettingPSNR ↑SSIM ↑LPIPS ↓CLIP Aesthetic ↑Subjective Qual. ↑Entity Cons. ↑Motion Acc. ↑Trans. Err. ↓Rot. Err. (°) ↓Prompt Cons. ↑
SolarWM baseline20.5951 ± 0.60130.6424 ± 0.03120.3650 ± 0.02905.1738 ± 0.10790.5072 ± 0.04060.9039 ± 0.0050-34.8380 ± 8.56881.5177 ± 0.24321.2760 ± 0.74170.2018 ± 0.0074
SolarWM + GROME24.3071 ± 1.26880.6976 ± 0.05710.2667 ± 0.05145.2724 ± 0.15650.5604 ± 0.05740.9248 ± 0.0183-36.9856 ± 12.92590.6944 ± 0.14810.9448 ± 0.28610.2007 ± 0.0129
SANA-WM baseline20.2212 ± 0.58960.6690 ± 0.02070.4926 ± 0.02224.8654 ± 0.12340.3368 ± 0.04290.8617 ± 0.015833.1032 ± 21.56350.7972 ± 0.07594.2000 ± 1.37400.2192 ± 0.0099
SANA-WM + GROME22.1007 ± 0.50090.7056 ± 0.01530.3969 ± 0.02844.8762 ± 0.12840.3305 ± 0.04250.8849 ± 0.018136.4194 ± 14.71730.6845 ± 0.04381.9626 ± 0.95860.2142 ± 0.0104

These quantitative results, together with the qualitative comparisons provided in the supplementary material, consistently demonstrate that GROME can be plugged into diverse backbones to improve the revisit consistency of frozen video world models.

[1] Huang J, Fang G, Qian S, et al. SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models[J]. arXiv preprint arXiv:2609.02886, 2026.

[2] Zhu H, Liu H, Zhao Y, et al. Sana-wm: Efficient minute-scale world modeling with hybrid linear diffusion transformer[J]. arXiv preprint arXiv:2605.15178, 2026.

AuthorsOpenReview ↗

Response to Reviewer hpb8 (2/3)

W2 1) The static/dynamic split depends on visible motion. An object that sits still for a while and only starts moving later would be stored as static and stay frozen in memory.

We thank the reviewer for raising this insightful boundary case. Because our motion-based splitting relies on observed movement, an initially stationary object is assigned to Static Memory during early frames and reclassified once motion begins.

To investigate this limitation, we conducted additional experiments on this scenario (see Appendix F, Fig. 27, and the accompanying videos in the updated supplementary material), featuring a parked car that subsequently starts moving. As shown in our results, once motion is detected, subsequent observations of the car are routed to Dynamic Memory, while our recent-first rendering strategy prioritizes newer observations over outdated static entries. Despite the presence of outdated static memory entries, the car transitions smoothly from a parked state to motion without noticeable persistent duplicates. This suggests that the pretrained model may leverage its learned generative priors to accommodate inconsistencies between static memory and recent context.

We agree that relying purely on motion heuristics is insufficient for such delayed-motion scenarios. To resolve this limitation, we are actively exploring the use of Multimodal LLMs (MLLMs) for semantic understanding—identifying potentially movable entities to dynamically re-assign memory types prior to explicit motion detection.

W2 2) The paper also doesn't look separately at fluids or scenes with large non-rigid motion.

We thank the reviewer for raising this important point. We would like to clarify that GROME's static–dynamic memory split relies on a motion-aware split to distinguish stable scene structures from dynamic entities. Accordingly, although fluids and large non-rigid deformations involve continuous local motion or deformation, their spatial locations in the scene remain largely unchanged. Consequently, such content is retained in Static Memory rather than explicitly segmented as standalone dynamic entities.

To evaluate GROME's performance under these scenes, we isolate the Fluid and Deformable subsets from our test set and provide additional quantitative evaluations in the table below, corresponding with the qualitative videos in the supplementary material.


Quantitative Results on Fluid and Deformable Scenes

Scene TypePSNR ↑SSIM ↑LPIPS ↓
Fluid17.7703 ± 0.31540.5101 ± 0.03760.1967 ± 0.0890
Deformable19.0088 ± 0.46580.5820 ± 0.05870.1803 ± 0.0635
Average (Static Scenes)20.6200 ± 0.41960.6520 ± 0.02660.1690 ± 0.0178

The qualitative results also demonstrate that fluids and non-rigid deformations maintain natural dynamic variations in generated videos despite being stored in static memory. This behavior is consistent with our Timestep-Aware Memory Routing mechanism: static memory provides historical scene references during the initial generation steps, while in subsequent steps the model progressively refines local dynamics conditioned on recent context and the prompt. In this way, GROME maintains scene consistency while preserving the pretrained model's capacity to synthesize complex dynamic content.

AuthorsOpenReview ↗

Response to Reviewer hpb8 (3/3)

W3 Inference is expensive, which doesn't fit well with the interactive setting the paper targets. In practice, the cost saved by skipping training is simply moved to inference.

We thank the reviewer for highlighting the importance of inference efficiency in interactive world modeling. We agree that inference latency is an important consideration for interactive applications. We would like to clarify that GROME primarily focuses on action-conditioned long-horizon consistency [1-2], where interaction refers to user-controlled scene exploration and continuation. Although real-time generation is not the primary objective of this work, we recognize that reducing inference latency would further improve its practicality.

Our runtime analysis in Appendix D identifies DA3 inference as the primary source of GROME's additional computational overhead. Furthermore, Table 6 reports the runtime of GROME under different DA3 chunk configurations, including GROME-Static (93/77), GROME-Dynamic (93/77), and GROME-Dynamic (45/29). The results show that the DA3 chunk configuration substantially affects the overall rollout time, with GROME-Dynamic (45/29) reducing the rollout time from 2,197.1 s to 1,110.5 s compared with GROME-Dynamic (93/77).

These results indicate that the current geometric reconstruction pipeline offers considerable room for optimization. In particular, future work will explore chunk-delayed updates and lightweight streaming 3D reconstruction models to reduce the computational cost of online geometric reconstruction while maintaining effective memory construction.

[1] Li R, Torr P, Vedaldi A, et al. Vmem: Consistent interactive video scene generation with surfel-indexed view memory[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 25690-25699.

[2] Nam J, Hong Y, Huang C H P, et al. Worldcam: Interactive autoregressive 3d gaming worlds with camera pose as a unifying geometric representation[J]. arXiv preprint arXiv:2603.16871, 2026.

Q4 Since both the memory retrieval and the idea of separating static and dynamic content come from prior work, what is the core contribution of this paper.

We thank the reviewer for raising this important question. GROME targets long-horizon consistency in video world models, which requires maintaining both persistent scene structures and temporally consistent dynamic entities. To address this gap, GROME explicitly preserves both static scene structures and dynamic entity as separately reusable historical memories. However, simply separating these two types of memory does not make them directly usable by pretrained video world models. Our contributions lie in how these factorized memories are retrieved and effectively injected into frozen video world models. We realize this through the following complementary technical designs:

  • (1) Distinct Memory Retrieval. We employ distinct retrieval strategies for static and dynamic memories, according to their different spatiotemporal properties. We consolidate spatially relevant local pointmaps across viewpoints for static memory, since static scene structures remain persistent and can be reused across timesteps. For dynamic memory, We retrieve the latest spatially associated states to resolve conflicting historical observations, as dynamic entities evolve over time and require temporally consistent state selection.

  • (2) Generalizable Training-Free Memory Injection. We construct and retrieve memory in pixel space and inject the resulting references into the backbone's latent space, enabling external memory to be incorporated through the pretrained model's native attention mechanisms. Static references are assigned the temporal RoPE of the target frames, reflecting their time-invariant nature, whereas dynamic references retain their temporal RoPE to preserve their causal dynamics. We maintain separate key–value streams for static and dynamic references, allowing target queries to attend to each memory stream independently. This separation reduces interference between historical memories with different spatiotemporal semantics.

  • (3) Reference Adaptation and Timestep-Aware Routing. We adjust only the value features to align projected memory features with the pretrained feature distribution, while preserving the original query-key correspondence. This mitigates feature mismatch without disrupting the pretrained attention structure. Furthermore, timestep-aware memory routing prioritizes static memory during early denoising steps while maintaining dynamic memory conditioning throughout generation. This allows static memory to establish scene structure without overconstraining subsequent refinement, while dynamic memory supports entity consistency across timesteps.

Together, these technical designs enable GROME to preserve and retrieve both static and dynamic historical information, and to ground this factorized memory in frozen video world models without additional training.

Reviewer_xmb3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GROME, a training-free external memory framework for long-horizon consistency in autoregressive video world models. For each generated chunk, the method runs depth estimation and motion-aware segmentation and splits the history into static and dynamic local pointmaps. Static pointmaps are retrieved with a max-coverage criterion, projected to the target views, and given the target frames' temporal RoPE indices. For dynamic pointmaps, only the latest state is used, and the original temporal RoPE indices are kept. Both references are injected into a frozen LingBotV2 through Reference Value Adaptation, two attention branches with independent softmax, and a routing scheme that injects static references only in the first two denoising steps. Experiments use revisit trajectories on 200 WorldScore scenes, with ablations on 50 scenes.

Critical strengths
  1. The controlled comparison is convincing. Under the same backbone and the same conditioning budget, static-scene PSNR rises from 11.400 (LingBotV2RC) and 12.805 (LingBotV2TR) to 20.620, and Rot. Err. drops from 12.156 / 15.698 to 3.984. On dynamic scenes, PSNR rises from 19.291 to 24.111 (Table 1). This shows that feeding explicit geometric references through the native history pathway is clearly more effective than simply retrieving past frames.
  2. The problem decomposition is sensible. The paper identifies a real failure mode of static-only memory, where moving entities get frozen or duplicated (Fig. 2, Fig. 4), and the design addresses it directly.
  3. Bootstrap CIs are reported, and the bolding in Table 1 honestly shows that Gen3C leads on static PSNR/SSIM.
Critical weaknesses
  1. There is no direct evidence for the dynamic memory. Revisit metrics are computed only on static regions. Entity Cons. measures appearance consistency within an entity, and Motion Acc. measures whether motion is concentrated in the target region. Neither tests whether an entity's identity and state at revisit match what is stored in memory. Looking at the numbers, on dynamic scenes GROME's Entity Cons. is 0.769, below WorldPlay (0.910) and Gen3C (0.841), and only 0.052 above LingBotV2RC (0.717). Its Motion Acc. is 16.840 ± 14.6834, which cannot be distinguished from RC's 16.640 ± 16.1201. In Table 2, w/o Dynamic gives Entity Cons. 0.609, lower than the memory-free RC (0.711). This suggests that the measurable effect of the dynamic stream is mainly to cancel the damage static memory does to entities. In addition, neither "preserving the original temporal RoPE" (p6 L273–278) nor "using only the latest state" is ablated. Spatia (arXiv 2512.15716) already removes dynamic entities from its static point-cloud memory, so the increment of this paper is precisely that dynamic entities are also stored in memory, and this needs quantitative support. I suggest adding metrics such as re-id similarity of entity crops before and after a revisit and an entity duplication rate, plus ablations of the two design choices above.
  2. "Local retrieval avoids the drift of global consolidation" (p2, p5) is the core motivation of the method, but the main text supports it with only one qualitative example (Fig. 6). Meanwhile, Gen3C, which uses a global 3D cache, actually leads on static revisits (PSNR 21.725 vs. 20.620, SSIM 0.749 vs. 0.652). I accept that external baselines are not controlled comparisons, but that is exactly why a global-consolidation variant inside LingBotV2 is needed, with a quantitative comparison.
  3. The claim of being pluggable into video world models is tested on a single backbone, LingBotV2. The "multiple LingBotV2 variants" on p3 L122 are in fact two conditioning baselines. Also, in Table 2, Joint Attention has PSNR 5.470 and w/o Ref. Adapt has 6.344, about 10 dB below the memory-free baseline (15.505). Naive injection therefore makes generation collapse, and the results depend on these hand-designed components. The authors use one set of hyperparameters across all scenes, which is a plus, but there is currently no way to tell whether they transfer to other backbones. At minimum, please add one more backbone, or give a sensitivity analysis for λ, α_S, and the number of static injection steps.
  4. The wording is stronger than the data. The abstract says GROME "substantially" outperforms existing methods on scene persistence, but Gen3C leads on static scenes, and on dynamic scenes PSNR is only 0.241 above WorldPlay (CI half-width about 1). p8 L405 describes WorldPlay's Motion Acc. as "severely negative", while the actual value is -2.883 ± 12.2510.
Questions for authors
  1. Why does w/o Dynamic lower PSNR on static regions by 4.296 dB (23.838 → 19.542)? Were only the D tokens removed, or was the recent context, which appears only in O^D, removed along with them?
  2. An entity that is first stationary and later starts moving (for example, a parked car that later drives away) will be written into static memory. Will it later be projected back as a ghost or a duplicated entity?
  3. When an entity is seen again after being out of view for a long time, its latest but outdated state is projected to the target view with its original temporal RoPE. How does the model handle the time that has passed in between?
  4. What is the static/dynamic composition of the 50 scenes in Table 2, and how were they selected?
7 comments
AuthorsOpenReview ↗

Response to Reviewer xmb3 (1/4)

W1 1). Neither Revisit Metrics and Entity Cons. tests whether an entity's identity and state at revisit match what is stored in memory. And Motion Acc. cannot be distinguished from RC's

We would like to clarify the scope of the evaluation metrics. While PSNR, SSIM, and LPIPS are computed on static regions, Entity Cons. is computed on dynamic entities across revisit frame pairs. It therefore evaluates whether the appearance of a dynamic entity remains consistent.

Furthermore, to provide additional qualitative evidence of whether video generation can utilize explicitly stored dynamic memory, we conduct a dynamic memory injection experiment in Appendix E.3 (Fig. 14). Specifically, we inject the explicitly stored dynamic entity memory into the model during video generation and examine whether the generated frames contain the corresponding entities. The results show that the generated entities are visually consistent with those represented in the injected memory. This provides direct qualitative evidence complementary to Entity Cons., which quantitatively evaluates appearance consistency between historical and revisit frames.

W1 2). Quantity results suggest that the measurable effect of the dynamic stream is mainly to cancel the damage static memory does to entities.

We respectfully clarify two points.

  • Entity Cons. should not be considered independently of Motion Acc. The higher Entity Cons. of WorldPlay and Gen3C comes at the cost of suppressing motion, which is an important limitation for dynamic-scene generation (as discuss in Sec. 4.2). In contrast, GROME achieves improvement over LingBotV2RC in Entity Cons. while maintaining comparable Motion Acc.. This indicates that GROME improves entity appearance consistency over the baseline without a corresponding reduction in motion quality.

  • Factorized memory does not imply disentangled effects on autoregressive generation. As further illustrated in Q1, although the two memory streams are separately stored and retrieved, they jointly participate in the generation process and interact with each other. The lower Entity Cons. of w/o Dynamic compared with RC indicates that using static memory alone can negatively affect dynamic entity consistency. However, the full GROME achieves a best Entity Cons. in our ablation, which suggests that the dynamic stream not only mitigates the interference introduced by static memory but also contributes to an overall improvement beyond the baseline.

W1 3) & 4). Neither "preserving the original temporal RoPE" nor "using only the latest state" is ablated.

We thank the reviewer for this valuable suggestion. We have conducted additional ablations on temporal RoPE assignment and dynamic state selection. The results are summarized in Table below, and the corresponding analyses have been incorporated into Sec. 4.4 and Appendices E.1.


Additional Ablation Results.

SettingCLIP Aesthetic ↑Subjective Qual. ↑Entity Cons. ↑Motion Acc. ↑
Current Temporal RoPE4.8731 ± 0.28000.5265 ± 0.12200.5711 ± 0.0428-23.0410 ± 12.2214
Multi-state Retrieval5.0046 ± 0.26500.5495 ± 0.11730.5245 ± 0.03458.7633 ± 4.0217
Full GROME5.352 ± 0.02160.598 ± 0.04120.749 ± 0.014816.872 ± 2.3846

  • Temporal RoPE Assignment. We compare preserving the original temporal RoPE of historical dynamic memory with assigning current temporal RoPE. As shown in the table above, current temporal RoPE degrades dynamic quality, particularly Motion Acc., while also reducing visual quality. Fig. 12 further shows that assigning current temporal indices to historical dynamic references can freeze entity motion and introduce dark boundary artifacts when motion resumes, potentially due to excessive attention to temporally misaligned references. These observations indicate preserving the original temporal RoPE indices to maintain the temporal semantics of historical entity states.

  • Dynamic State Selection. We compare our latest-state retrieval strategy with multi-state retrieval, which jointly incorporates multiple historical dynamic states. As shown in the table above, multi-state retrieval degrades entity consistency and motion quality, while also reducing visual quality. Fig. 12 further illustrates inconsistencies in generated dynamic entities. These results suggest that simultaneously retrieving multiple historical states may introduce temporal ambiguity due to conflicting historical evidence. In contrast, latest-state retrieval provides more temporally relevant dynamic guidance while avoiding redundant historical information and reducing computational cost.

AuthorsOpenReview ↗

Response to Reviewer xmb3 (2/4)

W1 4). Distinction from Spatia

We thank the reviewer for raising this point. We have to clarified that excluding dynamic entities from point-cloud (Spatia) is different from explicitly memorizing and utilizing them. The former prevents dynamic observations from contaminating static memory, whereas GROME additionally stores dynamic entities in a separate, temporally indexed 3D memory, retrieves them through spatially-aware temporal recency, and explicitly injects them into the generation process. We have clarified this distinction in the revised Appendix B.3, comparing the memory representation, temporal retrieval, and consumption mechanisms of different methods.

W1 5). Suggest adding metrics such as re-id similarity of entity crops before and after a revisit and an entity duplication rate

We thank the reviewer for this valuable suggestion. To provide more direct evidence of dynamic memory effectiveness, we introduce Entity Consistency (re-id similarity of entity crops) and Entity Reappearance Rate evaluations on LingBotV2 and SolarWM across 20 out-of-view scenes.

Since entity duplication rate is difficult to evaluate reliably in open-ended generated videos without ground-truth entity identities, we instead adopt entity reappearance rate (ERR) to directly measure whether previously observed entities are successfully recovered during revisits. These metrics assess whether previously observed entities retain their identity and successfully reappear after extended out-of-view intervals. Detailed evaluations and qualitative comparisons are provided in Appendix E.2.


Out-of-View Revisit Results.

ModelEntity Cons. ↑ERR (%) ↑
SolarWM0.488483.33
SolarWM + GROME0.720193.91
LingBotV20.498986.12
LingBotV2 + GROME w/o Dynamic Mem0.503488.32
LingBotV2 + GROME0.746693.28

The result table above and Figure 13 both show that GROME improves entity identity preservation and reappearance across both backbones, whereas the baseline models often exhibit identity drift and entity forgetting during revisits. However, removing dynamic memory from GROME degrades the performance, indicating that static memory alone is insufficient to preserve historical dynamic entities.

W2. Quantitative Evidence for Local Retrieval vs. Global Consolidation.

We thank the reviewer for this valuable suggestion. We have conducted an additional quantitative ablation comparing local retrieval with global point-cloud consolidation under the LingBotV2 backbone, and incorporated the quantitative results into Sec. 4.4 and Appendix E.1.


Quantitative Ablation on Local Retrieval and Global Consolidation.

SettingPSNR ↑SSIM ↑LPIPS ↓CLIP Aesthetic ↑Subjective Qual. ↑
Global Consolidation16.845 ± 1.43550.548 ± 0.05150.359 ± 0.07904.9657 ± 0.11560.4851 ± 0.0570
Local Retrieval (Ours)23.838 ± 2.58640.743 ± 0.03820.189 ± 0.01195.352 ± 0.02160.598 ± 0.0412

As shown, the results show that global consolidation degrades both revisit consistency and visual quality, consistent with the qualitative observations. This suggests that repeatedly consolidating historical observations into a global cache accumulates geometric errors, which propagate through autoregressive generation and further degrade video quality. In contrast, preserving local pointmaps and composing them only when needed provides more reliable geometric guidance, improving both revisit fidelity and generation quality over long-horizon rollouts.

AuthorsOpenReview ↗

Response to Reviewer xmb3 (3/4)

W3 1). The claim of being pluggable into video world models is tested on a single backbone, LingBotV2. Please add one more backbone.

We sincerely thank the reviewer for highlighting the importance of validating GROME beyond a single backbone architecture. We have extended GROME to two additional video world models: SolarWM [1] and SANA-WM [2]. To integrate GROME into these models without training, we factorize memory into static and dynamic components, apply spatially-guided retrieval and latest dynamic state for rendering and injecting the resulting reference into both model architectures. To evalute the performance, we evaluate all models under both static and dynamic revisit scenes. The full quantitative results are reported below (Sec. 4.3).


Comparison on Static Scenes

SettingPSNR ↑SSIM ↑LPIPS ↓CLIP Aesthetic ↑Subjective Qual. ↑Trans. Err. ↓Rot. Err. (°) ↓Prompt Cons. ↑
SolarWM baseline16.3773 ± 0.78210.4754 ± 0.05340.3977 ± 0.03665.4476 ± 0.17200.5573 ± 0.04061.9826 ± 0.27506.2931 ± 1.03190.2560 ± 0.0212
SolarWM + GROME20.4935 ± 0.60830.6457 ± 0.06760.1718 ± 0.04485.5723 ± 0.19850.6069 ± 0.03930.6128 ± 0.10374.1928 ± 1.23850.2389 ± 0.0194
SANA-WM baseline12.6547 ± 0.30080.3509 ± 0.01950.6686 ± 0.01195.1078 ± 0.06820.3625 ± 0.02160.7045 ± 0.03209.8103 ± 1.05240.2640 ± 0.0076
SANA-WM + GROME12.9505 ± 0.32580.3666 ± 0.02000.6401 ± 0.01395.1463 ± 0.06930.3505 ± 0.02200.6981 ± 0.03009.5481 ± 1.07880.2597 ± 0.0090

Comparison on Dynamic Scenes

SettingPSNR ↑SSIM ↑LPIPS ↓CLIP Aesthetic ↑Subjective Qual. ↑Entity Cons. ↑Motion Acc. ↑Trans. Err. ↓Rot. Err. (°) ↓Prompt Cons. ↑
SolarWM baseline20.5951 ± 0.60130.6424 ± 0.03120.3650 ± 0.02905.1738 ± 0.10790.5072 ± 0.04060.9039 ± 0.0050-34.8380 ± 8.56881.5177 ± 0.24321.2760 ± 0.74170.2018 ± 0.0074
SolarWM + GROME24.3071 ± 1.26880.6976 ± 0.05710.2667 ± 0.05145.2724 ± 0.15650.5604 ± 0.05740.9248 ± 0.0183-36.9856 ± 12.92590.6944 ± 0.14810.9448 ± 0.28610.2007 ± 0.0129
SANA-WM baseline20.2212 ± 0.58960.6690 ± 0.02070.4926 ± 0.02224.8654 ± 0.12340.3368 ± 0.04290.8617 ± 0.015833.1032 ± 21.56350.7972 ± 0.07594.2000 ± 1.37400.2192 ± 0.0099
SANA-WM + GROME22.1007 ± 0.50090.7056 ± 0.01530.3969 ± 0.02844.8762 ± 0.12840.3305 ± 0.04250.8849 ± 0.018136.4194 ± 14.71730.6845 ± 0.04381.9626 ± 0.95860.2142 ± 0.0104

These quantitative results, together with the qualitative comparisons provided in the supplementary material, consistently demonstrate that GROME can be plugged into diverse backbones to improve the revisit consistency of frozen video world models.

W3 2). Naive injection therefore makes generation collapse, and the results depend on these hand-designed components.

We would like to clarify that the degradation caused by naive memory injection is precisely the challenge that motivates GROME's Training-Free Memory Injection design. Directly injecting projected memory into a frozen video world model introduces feature distribution mismatches and interference between heterogeneous memory sources. To address these challenges, GROME introduces three complementary mechanisms:

  • Reference Value Adaptation aligns memory feature statistics while preserving the original keys and query–key correspondence;
  • Static-Dynamic Disentangled Attention prevents interference between memories with different spatiotemporal semantics;
  • Timestep-Aware Memory Routing restricts static conditioning to early denoising steps to prevent geometric artifacts from affecting fine-grained refinement.

The ablation results in Table 3 demonstrate the necessity of these carefully designed components for effectively integrating external memory into frozen video world models.

[1] Huang J, Fang G, Qian S, et al. SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models[J]. arXiv preprint arXiv:2609.02886, 2026.

[2] Zhu H, Liu H, Zhao Y, et al. Sana-wm: Efficient minute-scale world modeling with hybrid linear diffusion transformer[J]. arXiv preprint arXiv:2605.15178, 2026.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank the reviewers and the AC for their time and careful reading. Reviewers found the motivation clear (xmb3, xFuR), the method design thoughtful and technically sound (hpb8, xFuR), and the experimental comparisons and ablation studies supportive of GROME's effectiveness (xmb3, xFuR).

Scope of revision. In the main text, we evaluate GROME on SolarWM and SANA-WM and expand the ablation discussion. In the appendix, we add out-of-view revisit evaluations, a dynamic memory injection experiment, controlled ablations, runtime comparisons, and analyses of failure cases. In the supplementary materials, we add additional qualitative results demonstrating cross-backbone generalizability, revisit consistency in fluid and deformable scenes, and scenarios involving delayed object motion. Key textual additions are highlighted in blue.

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

We sincerely thank the reviewer for the thoughtful follow-up questions and for carefully examining our supplementary videos. We appreciate the opportunity to clarify how GROME is integrated into different backbones and to provide more examples that highlight its benefits. We address both questions in detail below.

Q1 1). How GROME Is Integrated into Solar-WM and SANA-WM

We further clarify how GROME is integrated into Solar-WM and SANA-WM.

Following the GROME design described in our paper for LingBot-v2, we first separate predicted local point maps into two memory banks: static and dynamic memories. Then we retrieve static and dynamic memories based on the current camera pose:

  • Static Memory Retrieval: We select historical local point maps based on maximum spatial coverage of the current view. We then consolidate them into a local point cloud and warp it to the current view to obtain a static reference. The reference is then encoded into memory tokens through the VAE.
  • Dynamic Memory Retrieval: We select historical dynamic memories based on the recency of each dynamic entity. We then warp them to the current view to obtain a dynamic reference, which is also encoded into memory tokens through the VAE.

Since static and dynamic memories have different temporal properties, we use different temporal RoPE settings for their tokens. Specifically, static memory tokens use the same temporal RoPE as the current chunk, while dynamic memory tokens retain the temporal RoPE of their original latents. This preserves the historical temporal information of dynamic entities.

Next, we feed both sets of reference tokens into the frozen backbone with their respective temporal RoPE settings. We run a full forward pass in the historical context and cache the resulting KV representations. During the denoising of the current chunk, we use dual-stream static–dynamic disentangled attention to access the historical KV representations of static and dynamic memories separately. In addition, static memory tokens are used only in the first two denoising steps.

For memory injection, we adapt the method to the architecture of each backbone:

  • Solar-WM: We follow the memory attention approach used in LingBot-v2. Specifically, we use dual-stream static–dynamic disentangled attention to access the historical KV representations of static and dynamic memories during the denoising of the current chunk.
  • SANA-WM: Unlike Solar-WM, SANA-WM uses linear attention and has its own GDN pathway. Its cross-attention layers are only present in some transformer layers, rather than all layers. We therefore adapt GROME's memory attention to these native cross-attention layers. This allows the current chunk to access historical static and dynamic memory KV representations through these layers, while keeping the original linear-attention and GDN pathways unchanged. We also fuse the memory tokens with the current chunk to further strengthen the historical memory prior.

Thus, Solar-WM and SANA-WM share the same static–dynamic memory construction and retrieval mechanisms. We adapt the memory injection strategy to each backbone's attention architecture and connect historical memories through the corresponding attention pathways.

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Q1 2). In the supplementary videos, i did not see a clear difference between Grome with SANA-WM

We highlight the key observations in Cases 1–4 to clarify the differences between SANA-WM and GROME in the supplementary videos:

  • In Case 1 (around 9.0–9.25 s) and Case 2 (around 3.0–3.25 s), the SANA-WM results show abrupt changes in scene layout and structure. For example, windows, kitchen islands, and other scene elements change suddenly from one frame to the next. In contrast, GROME better preserves the scene structures across frames.

  • In Case 3 (around 9.0–9.25 s) and Case 4 (around 5.5–6 s), the SANA-WM results show changes in the number and appearance of visible entities. For example, an additional elephant instance appears in an overlapping region, while a second bear head briefly appears and overlaps with the original. In contrast, GROME better preserves the number and consistency of these entities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.