acceptodds
Withdrawn from ICLR 2027

StreamDynamic: Streaming Dynamic Scene Reconstruction without Historical Ghosting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming 3D reconstruction incrementally predicts the point map of each newly arrived frame from an image stream, and is thus well suited to online applications. However, existing methods suffer from historical ghosting when reconstructing dynamic scenes, as they directly merge the predicted point maps of all frames into an accumulated point cloud. Specifically, the points reconstructed for a moving object at past time instants become dynamic ghosting at the current time instant. Moreover, when a previously static object moves away, its points reconstructed during the static period become static ghosting at its original location. In this work, we take the first step toward feed-forward streaming dynamic scene reconstruction without historical ghosting, and propose StreamDynamic, a novel training-free, plug-and-play framework that identifies moving objects online and removes both their dynamic and static ghosting from the reconstructions of existing streaming backbones. To remove dynamic ghosting, we propose a prior-anchored epipolar-geometry-based motion segmentation method, which estimates the epipolar geometry from the static regions indicated by a coarse motion prior extracted from the backbone predictions. To remove static ghosting accumulated over long histories, we design an online backtracking removal method, which constructs ghosting templates from only a few sparse views and efficiently erases the ghosting within the 3D neighborhoods of the templates. We further introduce a novel ghosting-aware evaluation protocol with ghosting-free ground truth for every time instant and metrics that penalize historical ghosting. Extensive experiments on three dynamic scene datasets show that StreamDynamic substantially reduces the historical ghosting of three representative streaming backbones, consistently improves their reconstruction quality, and outperforms existing methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Dynamic Scene Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vChfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes StreamDynamic, a plug-and-play, training-free streaming 3D reconstruction framework designed to address the "historical ghosting" issue in dynamic scenes. Existing methods typically merge predicted point clouds from all frames directly, causing moving objects to leave "dynamic ghosting" at their past locations and previously static objects to leave "static ghosting" at their original locations once moved. To solve these issues, StreamDynamic proposes a prior-anchored epipolar-geometry-based motion segmentation method to remove dynamic ghosting online. It also designs an online backtracking removal method based on sparse-view ghosting templates to eliminate static ghosting at a computational cost independent of the historical sequence length. Furthermore, the authors introduce a novel ghosting-aware evaluation protocol. Experiments demonstrate that this method significantly improves the reconstruction quality of three existing streaming backbones (TTT3R, STream3R, LingBot-Map).

Critical strengths
  1. The framework does not require retraining the backbone networks on specific dynamic datasets. It can be directly integrated into existing streaming reconstruction models, demonstrating high generalization and practical engineering value.
  2. To handle objects that move after remaining static for a long time, the authors propose a "Geometry-Verified Sparse View Selection" algorithm. By constructing a 3D ghosting template from only a few sparse views, it avoids the O(t)O(t) complexity of frame-by-frame autoregressive tracking, ensuring streaming efficiency.
  3. The paper provides a clear and insightful distinction between "dynamic ghosting" and "static ghosting" in feed-forward streaming reconstruction, addressing a core pain point for downstream online applications like robotic navigation and autonomous driving.
Critical weaknesses
  1. Strong Reliance on Static Backgrounds: The method uses epipolar geometry to estimate motion masks, which requires sufficient static correspondences to reliably estimate the fundamental matrix. In scenes with dynamic backgrounds, such as swaying trees, moving crowds, or rippling water, RANSAC may fail to identify reliable correspondences, compromising motion segmentation.
  2. Sensitivity of Mask Completion to Visual Similarity: The method relies on SAM to complete motion masks. When foreground objects share similar colors or textures with the background, SAM may over-segment the object, incorrectly removing static background points from the reconstruction.
  3. Error Propagation from Pretrained Models: The pipeline depends on off-the-shelf models such as RAFT and SAM. Errors in optical flow, particularly in occluded or textureless regions, and inaccurate mask boundaries can propagate to the 3D reconstruction, resulting in artifacts or missing geometry.
  4. Latency Spikes During Online Backtracking: Although sparse-view templates improve backtracking efficiency, removing static ghosting when objects suddenly move can still introduce temporary latency spikes, limiting the method's suitability for strict real-time streaming.
Questions for authors

Please refer to the Weaknesses section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer vChf

Thanks for your time and comments. We will revise the paper to address the concerns raised. Below, we provide clarifications for the major points.

  1. Reliance on static backgrounds. (1) Our method can estimate the fundamental matrix F from a small number of reliable static correspondences identified by the upstream prior because F has only seven degrees of freedom and can theoretically be estimated from just seven correspondences. Additional correspondences serve to validate the model and improve robustness to noise. Thus, a large static background is not a requirement for estimating F. (2) We use the upstream motion prior to preselect reliable static correspondences, rather than relying on RANSAC to identify them. Appendix D proves that this prior is less likely to misclassify dynamic points as static than vice versa, making it well suited to excluding dynamic points and providing reliable static correspondences for subsequent fundamental matrix estimation.
  2. Sensitivity of Mask Completion to Visual Similarity. When completing a motion mask with SAM, we use points sampled inside the mask as foreground prompts. SAM typically produces masks that cover these prompt points and also outputs a predicted IoU score for each mask as a quality estimate. If SAM occasionally mistakes a similarly colored or textured background object for the target, a low predicted IoU or a low IoU between the original and completed motion masks can help filter out such erroneous predictions, reducing the risk of incorrectly removing static background points.
  3. Error Propagation from Off-the-Shelf Models. (1) To address optical-flow errors in occluded or textureless regions, we reject unreliable correspondences using a conventional forward–backward consistency check: following the forward and then backward flow should return a point close to its original location. As discussed in Point 1 above, estimating F requires only a small number of reliable static correspondences, so we fit it using only correspondences with valid optical flow. When detecting motion using epipolar distances, regions with invalid flow are labeled static by default. SAM can still cover these regions in the completed mask, using point prompts from moving regions of the same object with valid flow. (2) We use SAM3, an advanced segmentation model with strong performance and robustness from large-scale pretraining. Our quantitative and qualitative results further validate our method’s effectiveness: applying our method to the reconstruction backbones substantially improves accuracy and CD while leaving the completeness metric nearly unchanged, indicating effective historical ghosting removal without substantial missing geometry.
  4. Latency Spikes During Online Backtracking. Although occasional latency spikes remain, we present the first streaming dynamic scene reconstruction method capable of effectively removing both dynamic and static ghosting. Compared with existing approaches for static ghosting removal—point-tracking methods (e.g., D4RT) and naive SAM backtracking—our method substantially reduces latency, making ghosting-free streaming reconstruction feasible while achieving state-of-the-art ghosting removal quality.
Reviewer_Zn8gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents StreamDynamic, a training-free, plug-and-play framework for removing historical ghosting in streaming dynamic scene reconstruction. It combines flow-based motion priors and epipolar geometry for dynamic ghosting removal, and sparse-view 3D templates for removing accumulated static ghosting. The authors further introduce a ghosting-aware evaluation protocol with time-dependent ground truth and new metrics. Experiments across multiple datasets and reconstruction backbones demonstrate effective ghosting removal and improved reconstruction quality, at the cost of additional computational overhead.

Critical strengths
  1. The training-free, plug-and-play framework can be integrated into existing reconstruction backbones without retraining.
  2. The ghosting-free ground truth and ghosting-aware metrics provide a valuable evaluation protocol for streaming dynamic reconstruction.
  3. Qualitative visualizations and quantitative results demonstrate effective ghosting removal across multiple datasets and backbones.
Critical weaknesses
  1. Limited algorithmic novelty and scalability concerns. The method combines established components without introducing a substantially new modeling paradigm. Moreover, historical-frame verification and point removal still require processing historical data, raising concerns about scalability and runtime overhead.
  2. Missing comparisons with tracking-based alternatives. 4D point tracking and dynamic scene reconstruction methods offer alternative ways to model temporal correspondences and may help prevent historical ghosting. The authors should discuss relevant approaches, including DELTAv2, DynOMo, and Track-On2 combined with StreamVGGT, and, if feasible, provide qualitative and/or quantitative comparisons under the proposed evaluation protocol to better establish the novelty and advantages of StreamDynamic.
Questions for authors
  1. It would help to clarify how the proposed approach relates to the aforementioned 4D point tracking methods.
  2. The paper reports average FPS and peak GPU memory across sequences, but how do these metrics scale with sequence length? Could the authors provide FPS and GPU memory measurements on increasingly long sequences to clarify the framework's efficiency and scalability for long-horizon streaming reconstruction?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Zn8g

Thanks for your time and comments. We will revise the paper to address the concerns raised. For the discussion of algorithmic novelty, please see the response to Reviewer 7zoH. Below, we provide clarifications for the major points.

  1. Scalability in long-horizon streaming reconstruction. (1) Historical-frame verification. In the Visibility-verified stage of our sparse-view selection algorithm, the main computation is projecting the template into each historical frame and computing the visibility ratio. These computations involve lightweight GPU tensor operations on the template and per-frame point maps. They are independent across frames and can be parallelized, making verification substantially less costly than naive autoregressive SAM backtracking. (2) Point removal only processes historical frames where the object was still at its original location, rather than indiscriminately scanning all historical points. As described in the paper, we first apply the same parallelizable visibility verification as in sparse-view selection to identify relevant frames, then delete points within the template neighborhood. These deletions are independent across frames and can be accelerated through lightweight parallel GPU tensor operations. The point-deletion cost therefore depends primarily on how long the object remained at its original location, rather than on the total input stream length. We will provide FPS and GPU memory measurements on increasingly long sequences in the revised paper.
  2. Comparisons and discussion of 4D tracking-based alternatives. Thanks for the suggestion. Our paper already includes quantitative and qualitative comparisons with D4RT, a SOTA method for 4D point tracking and dynamic scene reconstruction, together with a discussion of its performance. This tracking-based method remains less effective than our method at removing historical ghosting because its reliance on scarce synthetic point-tracking training data limits its generalization. Moreover, D4RT achieves only 0.02 FPS in our streaming evaluation, as removing historical ghosting through point tracking requires tracking points from all historical frames to the current time. We will extend our related work discussion to earlier approaches, including DELTAv2, DynOMo, and Track-On2.
Reviewer_7zoHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes StreamDynamic, a training-free framework for removing historical ghosting from streaming reconstructions. It combines flow discrepancies, epipolar geometry, and SAM to identify moving objects, and constructs sparse-view templates to remove geometry accumulated before an object starts moving. It also introduces a ghosting-aware evaluation protocol.

Critical strengths
  • The framework works with different streaming reconstruction backbones without additional training.
  • The qualitative results clearly show reduced ghosting.
Critical weaknesses

The method is extremely simple and straight forward, with small engineering efffort, which i treated as a post-processing or heuristics that needs careful tuning, this post-processing, is far below iclr bar.

Questions for authors

This simple post-processing of fusion phase is too straightforward, what is the reason/novelty you believe this can be an iclr paper?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7zoH

Thanks for your time and comments. We will revise the paper accordingly to clarify our contributions and novelty. Below, we provide clarifications to prevent potential misunderstandings of our work.

Our work is the first to investigate historical ghosting in streaming dynamic scene reconstruction, and we identify two types of historical ghosting: moving objects leave dynamic ghosting at their past locations, while previously static objects leave static ghosting once they move. The novelty of our work lies not only in the proposed method but also in the ghosting-aware evaluation protocol, as detailed below:

  1. A novel training-free, plug-and-play framework that can be applied to different types of reconstruction backbones without careful tuning. Although our framework builds on some existing components, their integration is nontrivial and involves multiple innovative and theoretically grounded designs: (1) A novel epipolar geometry modeling paradigm for motion segmentation. The key challenge for previous epipolar-geometry-based approaches is that moving objects contaminate fundamental matrix estimation, as epipolar geometry is modeled using correspondences sampled across both dynamic and static regions. We first establish through theoretical analysis (Appendix D) that motion detection using flow discrepancies has a substantially lower false-negative probability than false-positive probability. This motivates us to use the flow discrepancies to exclude potentially dynamic regions and model epipolar geometry using correspondences from static regions indicated by backbone-derived flow discrepancies for better accuracy. We have experimentally validated this design’s effectiveness and demonstrated its fast inference for streaming reconstruction. (2) A new modeling paradigm for static ghosting. We design a lightweight Geometry-Verified Sparse View Selection algorithm that selects a bounded number of historical views for ghosting template construction. This enables modeling static ghosting in unbounded input streams with bounded model inference cost, avoiding the linearly growing model inference cost of naive SAM backtracking, and achieving efficient online removal of static ghosting. Extensive experiments validated the method's effectiveness on three representative backbone architectures. As stated in Sec. 4.1 of the paper, all backbones and benchmarks share the same hyperparameters, without backbone- or benchmark-specific tuning.

  2. A novel ghosting-aware evaluation protocol for streaming dynamic scene reconstruction. To the best of our knowledge, we are the first to propose such a protocol, covering both the construction of ghosting-free ground truth for every time instant and new metric formulations. For example, our proposed accuracy metric penalizes both the amount of residual ghosting and its distance to the ground truth. We believe this protocol will provide a valuable basis for evaluating streaming dynamic scene reconstruction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.