GauMem: Learning What to Keep in a Fixed-Budget Gaussian Memory for World Models

Video world models require long-horizon spatial consistency across extended rollouts. Existing designs either grow unboundedly or rely on hand-crafted pruning heuristics that fail to anticipate retention needs upon delayed revisits. To resolve this, we propose GauMem, a capacity-bounded 3D Gaussian scene memory for video world models, governed by an end-to-end learned write policy. GauMem centers on a Read-Before-Write (RBW) mechanism: rendering the memory into incoming viewpoints diagnoses discrepancies to isolate novel content from existing structures prior to updates. Guided by this write-time evidence, Evidence-Gated Merge (EGM) consolidates consistent incoming proposals into stored slots, after which Joint Insertion–Eviction (JIE) ranking prioritizes stored slots and residual proposals within a unified pool up to the hard budget. The entire write policy is trained end-to-end on delayed revisits to optimize long-horizon spatial recall. On DL3DV, GauMem substantially outperforms heuristic write and fusion rules, as well as an unbounded store accumulating 7–15x more primitives under severe scene interference. When decoded through a frozen video diffusion model, GauMem consistently preserves superior visual fidelity and tight 3D camera trajectory consistency.