acceptodds
Withdrawn from ICLR 2027

GauMem: Learning What to Keep in a Fixed-Budget Gaussian Memory for World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video world models require long-horizon spatial consistency across extended rollouts. Existing designs either grow unboundedly or rely on hand-crafted pruning heuristics that fail to anticipate retention needs upon delayed revisits. To resolve this, we propose GauMem, a capacity-bounded 3D Gaussian scene memory for video world models, governed by an end-to-end learned write policy. GauMem centers on a Read-Before-Write (RBW) mechanism: rendering the memory into incoming viewpoints diagnoses discrepancies to isolate novel content from existing structures prior to updates. Guided by this write-time evidence, Evidence-Gated Merge (EGM) consolidates consistent incoming proposals into stored slots, after which Joint Insertion–Eviction (JIE) ranking prioritizes stored slots and residual proposals within a unified pool up to the hard budget. The entire write policy is trained end-to-end on delayed revisits to optimize long-horizon spatial recall. On DL3DV, GauMem substantially outperforms heuristic write and fusion rules, as well as an unbounded store accumulating 7–15x more primitives under severe scene interference. When decoded through a frozen video diffusion model, GauMem consistently preserves superior visual fidelity and tight 3D camera trajectory consistency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Video World Models, Persistent 3D Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JBhBOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses long-horizon spatial consistency in video world models under a strict storage budget. It proposes GauMem, a capacity-bounded 3D Gaussian scene memory governed by a learned write policy. The write policy comprises (i) Read-Before-Write (RBW), which renders the current memory into incoming viewpoints to diagnose photometric and geometric discrepancies; (ii) Evidence-Gated Merge (EGM), which softly fuses consistent incoming proposals into stored slots via a learned, evidence-weighted gate; and (iii) Joint Insertion–Eviction (JIE) ranking, which selects top-K primitives from a unified pool of stored slots and residual proposals. The whole policy (0.67M parameters) is trained end-to-end via revisit supervision on delayed revisits. On DL3DV (140 test scenes), GauMem substantially outperforms heuristic write rules and unbounded stores holding 7–15× more primitives at both N=4 and N=10 interference horizons; when decoded through a frozen video diffusion model (Wan2.1-VACE-1.3B), it preserves visual fidelity and camera-trajectory consistency (0.21° rotation error vs. 0.14° real-frame floor). Ablations confirm the contribution of each component and the necessity of revisit supervision. Generalization is verified across three frontends (StructSplat, Depth Anything 3, DepthSplat) and two out-of-distribution datasets (3RScan, RealEstate10K).

Critical strengths
  1. A well-designed evaluation protocol targeting exactly the claimed capability. delayed revisits under intervening scene interference — together with large, consistent gains over both training-free heuristic write rules and unbounded accumulation (which uses 7–15× more primitives yet scores lower, 19.57 vs. 16.92 PSNR at N=4). The ablations cleanly attribute the gain to RBW, EGM, JIE, and revisit supervision; without revisit supervision the memory collapses to ≈11.3 dB, demonstrating the objective is not trivially replaced.

  2. A principled, modular design that stays frontend-agnostic. the policy interfaces only through rendering parameters, disagreement vectors, and budget context, and transfers zero-shot across three different feed-forward 3DGS frontends and two external datasets, indicating it learns environment-agnostic consolidation rules rather than dataset-specific priors.

Critical weaknesses
  1. The experimental evidence is incomplete for the paper's central claim of benefiting video world models. (a) All headline results (Tables 1–3) come from a single dataset (DL3DV) under a simulated loop-back protocol rather than real interactive agent rollouts; (b) the downstream test uses only one frozen reader (Wan2.1-VACE-1.3B) along a short 17-frame return path, so the paper never demonstrates integration into an actual (trained) video world model pipeline end-to-end; (c) all write-policy baselines are heuristic or self-adapted variants (e.g., opacity-density pruning, redundancy culling) — there is no learned memory baseline trained under the same protocol, so the comparison isolates heuristics rather than the full design space; and (d) the write path's cost (RBW requires rendering the whole memory into every incoming viewpoint at every write step, plus 150-step per-chunk local refinement of proposals) is never measured, which matters because the paper motivates the method by bounded computation/efficiency.

  2. The main-text evaluation is single-dataset and the protocol never varies the scene dynamics. All headline numbers come from DL3DV under static scenes with camera-only motion; the paper itself lists non-static scenes as out of scope, so the memory's retention behavior under object motion is untested in the main evaluation.

Questions for authors

Could the authors evaluate GauMem in a genuinely integrated setting , e.g., a trained video world model performing multi-turn exploration with memory conditioning at every step, rather than a frozen reader on a 17-frame return path? Does the memory's benefit persist under model-drift?

What is the per-write-step overhead of RBW rendering + EGM/JIE scoring and the 150-step per-chunk proposal refinement, in wall-clock time relative to heuristic policies? The paper motivates fixed-budget efficiency but never measures the write path's cost.

How do the delayed-revisit results compare against a learned memory baseline (e.g., a trained recurrent fusion/eviction policy) under the same protocol, rather than only training-free heuristics?

How were intervening regions selected in the protocol, and does the gap persist when the return trajectory is longer or when intervening regions have limited overlap with R0 (weaker interference)?

Reviewer_9zf9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GauMem, a fixed-budget three-dimensional Gaussian scene memory for maintaining long-horizon spatial consistency in video world models. Its learned write policy renders existing memory into incoming viewpoints to measure photometric and depth discrepancies, softly merges consistent proposals into nearby stored slots, and jointly ranks stored and incoming primitives to enforce a strict capacity limit. Trained end-to-end via opacity modulation on delayed revisit views, GauMem outperforms heuristic pruning rules and unbounded primitive accumulation in both revisit rendering fidelity and downstream video generation stability across indoor and outdoor scene benchmarks.

Critical strengths
  1. Effective closed-loop write policy that surpasses unbounded accumulation:

Rendering existing memory into incoming viewpoints before writing (Read-Before-Write) provides direct geometric and photometric discrepancy signals to guide soft primitive merging (Evidence-Gated Merge) and unified top-KK selection (Joint Insertion-Eviction). Under a fixed 27.5 MB budget, GauMem outperforms both heuristic pruning (+2.65 to +3.47 dB PSNR over the strongest baseline on DL3DV in Table 1) and unbounded accumulation storing 7 to 15 times more primitives by suppressing multi-view floater clutter.

  1. Thorough multi-setting empirical validation:

The experiments isolate write-policy effects using identical proposal streams, verify global retention across all intermediate regions rather than only the initial space (Table 5), and demonstrate zero-shot transfer of a single checkpoint across three frontends (StructSplat, Depth Anything 3, DepthSplat in Table 6) and two external datasets (3RScan, RealEstate10K in Table 7). Conditioning a frozen video diffusion model (Wan2.1-VACE-1.3B) further confirms that geometric accuracy in raw memory directly reduces downstream camera trajectory drift (Tables 2, 8, and 9).

Critical weaknesses
  1. Disconnect between the closed-loop video world model formulation and open-loop ground-truth evaluation:

Although Figure 2 and Section 3.1 formulate GauMem as a memory that ingests frames generated by a video world model across sequential rollouts, all memory writes during exploration consume clean ground-truth video chunks, while the video generator is invoked only once at the end to synthesize a 17-frame return clip (Section 4.3). In a true closed-loop rollout where generated frames are repeatedly lifted into 3D proposals and written back into memory, diffusion hallucinations and temporal flicker could corrupt the Read-Before-Write discrepancy signal and compound across write cycles. Moreover, writing a single 12-view chunk takes roughly 20 seconds (8 seconds for the write policy and 12 seconds for the frontend forward pass, pose alignment, and 150-step Adam refinement) and requires about 80 GB of GPU memory (Appendix A), which is a bottleneck for interactive world modeling.

  1. Mathematical underspecification and algorithmic inconsistencies in merging and ranking:

Key operations in Sections 3.3 to 3.5 lack formal definitions or exhibit structural issues. (i) Section 3.4 states that stored slot scores and incoming proposal scores are "zero-centered by group means" before Top-KK selection (Eq. 3). Independently zero-centering each group forces roughly half of incoming proposals to exceed the common offset and evict below-mean stored slots at every step, even when incoming observations are entirely redundant. (ii) Equation 2 applies linear interpolation directly to the rendering parameters, which include 4D rotation quaternions (Appendix A) that are neither unit-norm preserving nor antipodal-invariant under linear averaging. Furthermore, unconditionally replacing the slot feature with the aggregated proposal feature at the end of Section 3.3 overwrites a slot's historical latent state even when its assigned evidence approaches zero. (iii) The analytical definition of the soft association weight in Section 3.3 is omitted, and it is unclear whether the opacity modulation in Equation 5 is applied during inference or only during training.

  1. Incomplete literature coverage and baseline comparison for compressed-memory world models:

In Sections 1 and 2, the paper motivates GauMem by claiming that frame-retrieval memories (Xiao et al., 2025; Yu et al., 2025; Li et al., 2025a) grow unboundedly or fit only a few keyframes, whereas bounded token memories (Wei et al., 2026; Wang et al., 2025b) lack 3D geometric grounding. This characterization overlooks compressed-memory video world models, most notably WorldPack (Oshima et al., 2025) and FramePack (Zhang & Agrawala, 2025), as well as related memory-augmented world models such as Navigation World Models (Bar et al., 2025), History-Guided Video Diffusion (Song et al., 2025), SlowFast-VGen (Hong et al., 2025), and Spatia (Zhao et al., 2025). Specifically, WorldPack directly addresses long-horizon spatial consistency upon loop-closure revisits under a fixed budget by combining hierarchical trajectory packing with 3D camera-pose and field-of-view guided retrieval. Because Section 4.3 (Table 2) and Appendix G (Table 8) evaluate frame-based memories using only uncompressed top-4 retrieval and raw in-context frames, the comparison against frame and token memory paradigms omits competitive compressed-context approaches.

Questions for authors
  1. Closed-loop autoregressive rollouts and refinement latency:

Have you tested GauMem in a closed-loop setting where frames generated by Wan2.1-VACE-1.3B are fed back into the FFGS frontend and written into memory over multiple steps? Additionally, how much does performance change if the 150-step per-chunk Adam refinement is omitted?

  1. Group-mean centering in JIE (Section 3.4):

Are {sj}\{s_j\} and {sp}\{s_p\} centered by separate group means or a single joint mean prior to Equation 3? Please provide the exact normalization and positive-offset formula, and explain how the mechanism prevents forced slot turnover when an incoming chunk observes only redundant surfaces.

  1. EGM equations and inference-time opacity (Sections 3.3 and 3.5):

Please specify (a) the exact parameterization of the association weights wpjw_{pj}, (b) how quaternion averaging/interpolation in θj\theta_j (Eq. 2) and latent updates fjf_j are handled when eje_j is small, and (c) whether σ(πj)\sigma(\pi_j) in Equation 5 modulates opacity during inference or only during training (and whether RBW rendering uses σ(oj)\sigma(o_j) or αj\alpha_j).

  1. Comparison with geometry-aware compressed frame memories:

How does GauMem compare, both conceptually and in terms of memory-compute trade-offs, against compressed-memory world models such as WorldPack (Oshima et al., 2025), which combine 3D pose/field-of-view retrieval with hierarchical frame packing under a fixed token budget? Please discuss these missing works (including WorldPack, FramePack, Navigation World Models, History-Guided Video Diffusion, SlowFast-VGen, and Spatia) in Section 2.

Reviewer_CMA2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

To address the challenge of preserving long-term spatial consistency under a fixed memory budget, GauMem learns to consolidate and selectively retain 3D Gaussians using read-before-write diagnostics, evidence-gated merging, and joint insertion–eviction.

Critical strengths
  1. This work is well-written, and the readers can understand the proposed points.

  2. The core concept of "Read-Before-Write" is sound.

Critical weaknesses
  1. The practical importance of the fixed-storage formulation is not fully established. GauMem reduces persistent Gaussian storage from up to 281 MB to 27.5 MB in the reported setting, but the complete pipeline requires approximately 80 GB of GPU memory for memory evaluation, while the video reader itself uses about 22 GB.

  2. Downstream generation is performed using only a frozen Wan2.1-VACE-1.3B reader, with evaluations based on 17-frame video sequences. I recommend the author assess the method under other world-model architectures and longer-duration generation.

  3. Most baselines rely on training-free rules such as LRU, random eviction, opacity pruning, or surfel/voxel fusion, making it difficult to demonstrate that the current architecture is superior to other learnable retention policies.

Questions for authors

I have no further questions. Please focus on the issues mentioned in the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.