acceptodds
Withdrawn from ICLR 2027

PerM4D: Persistent Memory for Streaming 4D Gaussian Splatting from Monocular Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming 4D reconstruction aims to reconstruct dynamic scenes from monocular video streams using only past and current frames. A key challenge is to integrate new observations with previously reconstructed Gaussians, particularly when occlusion disrupts inter-frame correspondences. We introduce PerM4D, a persistent memory framework for streaming 4D Gaussian reconstruction from posed monocular videos. Our framework constructs per-frame Gaussians and integrates them into the persistent scene representation through memory-guided association. Persistent static memory fuses static Gaussians across frames, while short-term motion memory stores recent motion trajectories and long-term surface memory preserves the identities, geometry, and appearance of dynamic Gaussians. Together, these memories enable the reuse and incremental update of both static and dynamic Gaussians across frames. By updating existing Gaussians with new observations, our method reduces redundant Gaussian initialization and promotes geometric and photometric consistency, improving reconstruction quality. Comprehensive experiments on NVIDIA, DyCheck, and RealEstate10K demonstrate the competitive novel view synthesis performance of PerM4D. We will release our code upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Streaming 4D Reconstruction, Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WXvFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PerM4D, a 4D Gaussian Splatting (4DGS) method for progressively reconstructing dynamic scenes from a monocular video stream. The method first initializes frame-wise static and dynamic Gaussians using depth estimates from DA3, optical flow, and motion segmentation. Static Gaussians are accumulated through geometry-aware fusion, while dynamic Gaussians are updated using two types of memory mechanisms. The short-term memory leverages 3D tracking to predict Gaussian displacements and establish motion correspondences. The long-term memory maintains the geometry of a persistent surface representation to verify associations regardless of spatial proximity. The matching results from these memory mechanisms determine whether new observations should be fused with existing Gaussians, initialized as new surfaces, or used to reactivate previously occluded surfaces. Finally, the updated Gaussian representation undergoes a short per-frame optimization. Experimental results demonstrate that the proposed method achieves competitive novel-view synthesis quality with relatively low per-frame processing latency.

Critical strengths
  1. The combination of short-term motion memory and long-term representation memory enables the method to handle both occlusions and temporal correspondences effectively. The overall design is conceptually reasonable and well-motivated.

  2. The paper demonstrates superior reconstruction quality compared to the existing online methods evaluated in the experiments, while maintaining relatively efficient inference.

Critical weaknesses
  1. Important recent streaming/online reconstruction methods, particularly ProDyG (NeurIPS 2025) and 4D-Fly (CVPR 2025), are excluded from the quantitative comparisons. Furthermore, the reported quantitative results for MoVieS differ from those in the original paper (PSNR: 18.46 in the original paper). While this discrepancy may be due to differences in evaluation settings, more comprehensive comparisons with recent methods are needed.

  2. Since the experiments use relatively short sequences and ground-truth camera poses, the effectiveness of persistent memory in long-term streaming scenarios remains unclear.

  3. The overall framework of the proposed method (e.g., static-dynamic separation and the use of pretrained foundation models) appears to be influenced by several prior works. The main strength of this method lies in the cooperation between the two memory mechanisms and the robustness of the Gaussian representation. However, the paper lacks sufficient analysis of how effectively these methodological contributions work in practice. In particular, the benefits of the proposed method are demonstrated primarily through novel-view synthesis (NVS) performance, without more direct evaluations of its core contributions.

Questions for authors
  1. Representation: Does the proposed method support the explicit separation of static and dynamic components in its reconstructed representation?

  2. Effectiveness of Long-Term Memory: One of the main objectives of this paper is to preserve surface identities and accurately recover previously observed regions after occlusion. However, the experiments focus exclusively on NVS evaluation. Could the authors provide a more detailed analysis of the effectiveness of long-term memory, such as surface re-identification accuracy under varying occlusion durations, temporal consistency of surface identities, and the impact of these factors on reconstruction performance?

  3. Ambiguity in Surface Association: The long-term surface matching cost (Eq. 5) relies on orientation, scale, and color while excluding spatial position. How does this approach distinguish physically distinct surfaces that exhibit similar geometric or visual properties? Could the authors analyze the sensitivity of the proposed association mechanism to false associations, non-rigid deformations, and significant surface rotations?

  4. Long-Sequence Training: The experiments use relatively short input videos (12–13 frames). How do the number of stored Gaussians, dormant surface records, GPU memory consumption, and per-frame processing time scale as the video length increases? Could the authors provide evaluations on substantially longer sequences?

  5. Fast Motion: Newly observed dynamic objects may require multiple observations before they can be initialized. How well does the proposed method handle the reconstruction of objects undergoing fast motion?

Reviewer_CefpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PerM4D, a persistent-memory framework for streaming 4D Gaussian reconstruction from posed monocular videos, where each incoming frame is converted into temporary static and dynamic Gaussians and incrementally integrated into a persistent scene representation. The main technical component is a memory-guided association mechanism that combines short-term motion memory with long-term surface memory to decide whether dynamic observations should be fused with existing Gaussians, initialized as new surfaces, or used to reactivate dormant surface records. Experiments on NVIDIA, DyCheck, and RealEstate10K show competitive novel-view synthesis performance, with ablations indicating that static memory, motion memory, surface memory, and reactivation all contribute to the final rendering quality.

Critical strengths
  1. The paper identifies an important association problem in streaming 4D reconstruction and provides a conceptually clear framework that separates short-term motion cues from longer-term surface identity information, which is a reasonable design for handling trajectory interruption and reappearance.
  2. The proposed system is implemented as a fully causal pipeline using only past and current observations and achieves strong novel-view synthesis results with substantially lower processing time than some optimization-heavy baselines such as DynOMo.
Critical weaknesses
  1. The paper does not sufficiently clarify which of the stated challenges are specific to monocular streaming reconstruction, since ambiguous motion association, occlusion, reappearance, and persistent identity maintenance also arise in calibrated multi-view streaming settings, making the monocular-specific motivation less convincing.
  2. The core contribution is fundamentally an association and persistent-memory mechanism, yet the evaluation is dominated by PSNR, SSIM, and LPIPS and provides no direct measurements of association accuracy, ID switches, surface-record consistency, reactivation precision/recall, duplicate Gaussian rate, or tracking quality, so improved novel-view synthesis alone does not establish that persistent identity management is actually correct.
  3. The claimed long-term persistent memory is only evaluated on very short input sequences, with 12 source frames on NVIDIA, 13 selected source frames on DyCheck, and 12 source frames on RealEstate10K, while no analysis is provided on memory growth, surface-record growth, query cost, GPU memory usage, dormant-record accumulation, or long-term identity drift.
  4. The method relies on a relatively large number of hand-crafted thresholds and gating rules, including motion, surface, reactivation, reliability, trajectory-length, and birth-validation criteria, but the paper provides little sensitivity analysis to demonstrate the robustness or necessity of these design choices. 5.The discussion of related streaming 4D reconstruction methods is incomplete, particularly for prior calibrated multi-view methods that also maintain and update persistent Gaussian representations, and the paper does not sufficiently articulate how its memory formulation differs conceptually from these existing persistent-update pipelines. 6.The current ablation study is limited to removing four high-level components and therefore does not isolate the contributions of the individual association costs, surface-record design, lifecycle policy, temporal validation, memory regularization, or different fusion/reactivation strategies.
Questions for authors

Please refer to the concerns raised in the Critical Weaknesses section above and provide clarifications and additional experimental evidence addressing these points.

Reviewer_ZHevOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PerM4D to address streaming 4D Gaussian reconstruction from posed monocular videos. It combines depth estimation, optical flow, and segmentation to construct static and dynamic Gaussians. Persistent static memory accumulates background information, while short-term motion memory and long-term surface memory guide dynamic association, fusion, initialization, and reactivation. The resulting scene undergoes online optimization. Experiments evaluate novel-view synthesis on multiple datasets, including NVIDIA, DyCheck, and RealEstate10K.

Critical strengths
  1. Separating static accumulation from dynamic updates is well motivated. Combining recent motion evidence with historical surface attributes is a plausible strategy for handling ambiguous correspondence and reappearing content.
  2. The method achieves strong results under the evaluated settings, particularly on NVIDIA and RealEstate10K. The NVIDIA ablations also demonstrate that the individual memory components contribute to reconstruction quality.
Critical weaknesses

1.Unbalanced supplementary material. The appendix extensively formalizes simple initialization conditions and set relationships but provides no additional quantitative tables. Concise explanations and targeted experiments would be more informative. 2.Insufficient visual validation. Beyond the overview panels, dedicated visualizations of static–dynamic separation and memory association are missing. Depth, residual-flow, mask, and reactivation examples would help verify the central mechanisms. 3.Redundant and crowded figures. Figures 1 and 2 repeat much of the same pipeline. Dense annotations and small schematic elements reduce readability; the figures need distinct purposes and clearer layouts. 4.Qualitative gains are not consistently clear. The paper examples and supplementary videos do not consistently show obvious improvements over baselines. Enlarged regions and matched temporal comparisons would make the benefits easier to assess. 5.Runtime reporting lacks detail. Although the authors state end-to-end timing on an H100, Table 1 gives one time value per method across different dataset settings. Aggregation, baseline configurations, and treatment of initialization and final refinement remain unclear. 6.Limited validation of sustained streaming. Evaluation uses only 12–13 sampled input frames, with ablations restricted to NVIDIA. Longer-stream quality, memory, and latency curves are needed to assess error accumulation and persistent memory's benefits.

Questions for authors

1.It would be better to provide visual diagnostics of static–dynamic separation, including input RGB, estimated depth, observed/residual flow, and predicted dynamic masks? 2. It would be better to you show frame-by-frame memory associations and reactivation events, including failure cases, to demonstrate that historical surfaces are correctly recovered? 3. It would be better to evaluate longer streams and report reconstruction quality, memory consumption, and latency over time? How does performance vary with occlusion duration? 4. It would be better to extend the memory ablations beyond NVIDIA and include evaluations focused on dynamic foregrounds and reappearance events? 5. It would be better to provide more details reported in Table 1? Please clarify baseline configurations and whether initialization, preprocessing, and sequence-end refinement are included consistently.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.