- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses long-horizon spatial consistency in video world models under a strict storage budget. It proposes GauMem, a capacity-bounded 3D Gaussian scene memory governed by a learned write policy. The write policy comprises (i) Read-Before-Write (RBW), which renders the current memory into incoming viewpoints to diagnose photometric and geometric discrepancies; (ii) Evidence-Gated Merge (EGM), which softly fuses consistent incoming proposals into stored slots via a learned, evidence-weighted gate; and (iii) Joint Insertion–Eviction (JIE) ranking, which selects top-K primitives from a unified pool of stored slots and residual proposals. The whole policy (0.67M parameters) is trained end-to-end via revisit supervision on delayed revisits. On DL3DV (140 test scenes), GauMem substantially outperforms heuristic write rules and unbounded stores holding 7–15× more primitives at both N=4 and N=10 interference horizons; when decoded through a frozen video diffusion model (Wan2.1-VACE-1.3B), it preserves visual fidelity and camera-trajectory consistency (0.21° rotation error vs. 0.14° real-frame floor). Ablations confirm the contribution of each component and the necessity of revisit supervision. Generalization is verified across three frontends (StructSplat, Depth Anything 3, DepthSplat) and two out-of-distribution datasets (3RScan, RealEstate10K).
Critical strengths
-
A well-designed evaluation protocol targeting exactly the claimed capability. delayed revisits under intervening scene interference — together with large, consistent gains over both training-free heuristic write rules and unbounded accumulation (which uses 7–15× more primitives yet scores lower, 19.57 vs. 16.92 PSNR at N=4). The ablations cleanly attribute the gain to RBW, EGM, JIE, and revisit supervision; without revisit supervision the memory collapses to ≈11.3 dB, demonstrating the objective is not trivially replaced.
-
A principled, modular design that stays frontend-agnostic. the policy interfaces only through rendering parameters, disagreement vectors, and budget context, and transfers zero-shot across three different feed-forward 3DGS frontends and two external datasets, indicating it learns environment-agnostic consolidation rules rather than dataset-specific priors.
Critical weaknesses
-
The experimental evidence is incomplete for the paper's central claim of benefiting video world models. (a) All headline results (Tables 1–3) come from a single dataset (DL3DV) under a simulated loop-back protocol rather than real interactive agent rollouts; (b) the downstream test uses only one frozen reader (Wan2.1-VACE-1.3B) along a short 17-frame return path, so the paper never demonstrates integration into an actual (trained) video world model pipeline end-to-end; (c) all write-policy baselines are heuristic or self-adapted variants (e.g., opacity-density pruning, redundancy culling) — there is no learned memory baseline trained under the same protocol, so the comparison isolates heuristics rather than the full design space; and (d) the write path's cost (RBW requires rendering the whole memory into every incoming viewpoint at every write step, plus 150-step per-chunk local refinement of proposals) is never measured, which matters because the paper motivates the method by bounded computation/efficiency.
-
The main-text evaluation is single-dataset and the protocol never varies the scene dynamics. All headline numbers come from DL3DV under static scenes with camera-only motion; the paper itself lists non-static scenes as out of scope, so the memory's retention behavior under object motion is untested in the main evaluation.
Questions for authors
Could the authors evaluate GauMem in a genuinely integrated setting , e.g., a trained video world model performing multi-turn exploration with memory conditioning at every step, rather than a frozen reader on a 17-frame return path? Does the memory's benefit persist under model-drift?
What is the per-write-step overhead of RBW rendering + EGM/JIE scoring and the 150-step per-chunk proposal refinement, in wall-clock time relative to heuristic policies? The paper motivates fixed-budget efficiency but never measures the write path's cost.
How do the delayed-revisit results compare against a learned memory baseline (e.g., a trained recurrent fusion/eviction policy) under the same protocol, rather than only training-free heuristics?
How were intervening regions selected in the protocol, and does the gap persist when the return trajectory is longer or when intervening regions have limited overlap with R0 (weaker interference)?