PerM4D: Persistent Memory for Streaming 4D Gaussian Splatting from Monocular Videos

Streaming 4D reconstruction aims to reconstruct dynamic scenes from monocular video streams using only past and current frames. A key challenge is to integrate new observations with previously reconstructed Gaussians, particularly when occlusion disrupts inter-frame correspondences. We introduce PerM4D, a persistent memory framework for streaming 4D Gaussian reconstruction from posed monocular videos. Our framework constructs per-frame Gaussians and integrates them into the persistent scene representation through memory-guided association. Persistent static memory fuses static Gaussians across frames, while short-term motion memory stores recent motion trajectories and long-term surface memory preserves the identities, geometry, and appearance of dynamic Gaussians. Together, these memories enable the reuse and incremental update of both static and dynamic Gaussians across frames. By updating existing Gaussians with new observations, our method reduces redundant Gaussian initialization and promotes geometric and photometric consistency, improving reconstruction quality. Comprehensive experiments on NVIDIA, DyCheck, and RealEstate10K demonstrate the competitive novel view synthesis performance of PerM4D. We will release our code upon publication.