acceptodds
Withdrawn from ICLR 2027

DynStream: Online Streaming 4D Gaussian Reconstruction of Dynamic Worlds from Unposed Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Online reconstruction of dynamic 4D scenes from long, unposed streaming videos requires both continuous processing and photorealistic rendering, which existing methods struggle to achieve simultaneously. Existing feed-forward Gaussian methods are restricted to offline processing, whereas online point-cloud approaches struggle to maintain dense geometry and high-fidelity rendering. We present DynStream, a framework for streaming 4D Gaussian reconstruction from long, unposed videos. Given a continuous video stream, DynStream reconstructs the scene within local temporal windows and incrementally aligns and fuses these local reconstructions into a globally consistent scene, enabling online 4D reconstruction without per-scene optimization. By jointly enforcing cross-window geometric consistency and modeling time-varying scene content, DynStream supports efficient reconstruction and photorealistic rendering over extended video streams. Experiments demonstrate that DynStream enables high-fidelity online dynamic reconstruction and rendering from long video streams, achieving state-of-the-art performance across diverse dynamic indoor and outdoor scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamic Reconstruction, Scene Representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ckQmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents DynStream for feed-forward dynamic Gaussian reconstruction from unposed video streams. It processes incoming frames causally within temporal chunks, predicts keyframe-relative camera poses and metric scale, and integrates local Gaussian predictions into a global scene. Static content is accumulated, while dynamic content remains associated with individual observation times. Experiments report improved rendering quality across five benchmarks and camera tracking on long KITTI sequences without per-scene optimization.

Critical strengths
  1. The framework combines camera estimation and renderable Gaussian reconstruction in a causal pipeline.

  2. Experimental results report consistent rendering improvements across five real and synthetic benchmarks.

Critical weaknesses
  1. The Gaussian-specific contribution beyond LongStream is unclear. The method adopts keyframe-relative pose estimation and metric-scale prediction from LongStream (Cheng et al., 2026), but does not clearly explain what new design enables consistent Gaussian reconstruction across chunks. Table 3 shows that separating dynamic content is useful, while Table 4 validates the relative-pose and scale components. These ablations do not examine whether the proposed fusion maintains consistent Gaussian geometry across chunk boundaries or improves temporal consistency. The claimed contribution of globally consistent, streaming Gaussian reconstruction therefore remains insufficiently supported.

  2. The dynamic representation provides limited support for temporal coherence. In Eq. 5 and Section 3.3, dynamic Gaussians are associated with individual observation times, without an explicit motion model or cross-time association mechanism. This supports reconstruction at observed timestamps, but the paper does not explain how observations of the same moving surface are combined over time, particularly through occlusions. The distinction between maintaining temporally coherent dynamic geometry and storing separate time-specific reconstructions therefore remains unclear.

  3. An important comparison with existing online dynamic Gaussian reconstruction is missing. StreamSplat (Wu et al., 2026) already performs feed-forward dynamic Gaussian reconstruction from uncalibrated video streams, but is neither discussed nor evaluated. This omission also makes the characterization of existing feed-forward Gaussian methods as offline too broad. Although its coordinate formulation differs from DynStream’s metric mapping, a comparison on shared reconstruction and rendering tasks, with consistent input and timing protocols, would help establish the advance over existing online methods.

Wu, Z., Yan, Q., Yi, X., Wang, L., & Liao, R. Streamsplat: Towards online dynamic 3d reconstruction from uncalibrated video streams. ICLR 2026.

  1. The practical online efficiency needs stronger characterization. Table IV reports 315.5 ms per input frame even in the 100-chunk setting, corresponding to approximately 3.2 FPS. The formulation is causal, but the paper does not specify the inference chunk length, output latency at chunk boundaries, or hardware used for these measurements. Moreover, Appendix G acknowledges that the accumulated Gaussian scene continues to grow. The reported average runtimes therefore provide limited evidence for sustained processing of arbitrarily long streams; runtime, memory, and Gaussian-count profiles along the same sequence would better support this claim.

  2. The rendering evaluation protocol is insufficiently specified. Table 1 reports results over 14 sampled frames per scene, while Appendix B does not explain how reconstruction inputs and held-out target views are separated. It is also unclear which scene state is used to render each target and whether later observations are available at that point. These details are particularly important because Eqs. 9–10 supervise reconstruction of the current input frame. Without a clear input–target and temporal protocol, the reported improvements are difficult to interpret as evidence of online novel-view synthesis.

Questions for authors
  1. Could the authors clarify which Gaussian reconstruction and fusion components are new relative to LongStream, and provide evidence of their effect on geometric consistency across chunk boundaries?

  2. Could the authors explain how observations of the same moving surface are connected across frames, particularly when it becomes occluded? A short sequence example showing how the dynamic representation evolves would be helpful.

  3. Could the authors provide a comparison with StreamSplat (Wu et al., 2026) on shared reconstruction and rendering tasks using consistent input and timing protocols, and explain how the different coordinate formulations are handled?

  4. Could the authors specify the hardware, inference chunk length, and when frame outputs become available? For one representative sequence, could they report per-frame runtime, peak memory, and Gaussian count as the stream grows?

  5. Could the authors specify the input–target view and time splits for Table 1, including which observations are available when each target is rendered? Please also clarify whether target RGB images are excluded from reconstruction and how target camera poses are obtained.

Reviewer_dgTwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a framework for online Gaussian reconstruction from unposed dynamic videos. A VGGT-based causal transformer predicts keyframe-relative camera poses, depth, metric scale, and pixel-aligned Gaussians with dynamic probabilities. Local reconstructions are aligned across temporal chunks, with static Gaussians accumulated into a persistent map and dynamic Gaussians associated with their observation times.

Critical strengths
  1. The paper addresses a practically relevant problem, jointly estimating camera poses and maintaining a renderable dynamic scene from unposed video streams in a feed-forward way.
  2. The reported experiments show substantial rendering improvements across several real and synthetic datasets, while the ablations suggest benefits from dynamic-aware accumulation and metric-scale calibration.
  3. The paper is well structured, and the overall framework is clearly presented and easy to follow.
Critical weaknesses
  1. The method appears largely to integrate existing components: a VGGT-based causal backbone, keyframe-relative pose and scale prediction, and Gaussian prediction heads. In particular, the streaming formulation and causal architecture have already been explored in LongStream.
  2. Dynamic Gaussians appear to be timestamped 3D primitives, without an explicit motion model, deformation function, or cross-frame association. Please explain how temporal stability and consistency are maintained. Moreover, SAM3-generated semantic foreground masks may not reliably indicate physical motion, and agreement with the same label source does not independently validate motion separation. Segmentation robustness should be evaluated under complex, multi-object, and non-rigid motion, particularly around object boundaries, together with the effects of segmentation errors on reconstruction quality.
  3. Sequential composition in Eq. (6) can accumulate pose and scale errors. Although acknowledged in the supplement, this limitation should be discussed in the main paper, particularly in relation to the claim of global consistency. It would also be useful to discuss whether established techniques, such as loop-closure detection and trajectory correction, could mitigate drift within this framework.
  4. The comparison should include representative dynamic reconstruction methods, particularly StreamSplat and 4DGT, with input settings and camera assumptions clearly documented and matched where applicable.
  5. Table IV reports 315.5 ms per frame at 100 chunks, approximately 3.2 FPS, but the hardware, inference resolution, and chunk length are insufficiently documented. Furthermore, bounded temporal context does not bound the accumulated Gaussian map. Gaussian count, memory consumption, and per-frame latency should be reported over increasing sequence lengths to substantiate the scalability claims.
Questions for authors

Please refer to the weaknesses part.

Reviewer_XyByOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DynStream, a feed-forward framework for online 4D Gaussian reconstruction from long, unposed video streams. It processes incoming frames in causal, fixed-length chunks, predicts keyframe-relative camera poses, depth, Gaussian primitives, and metric scale, and incrementally transforms local reconstructions into a shared global coordinate system. A dynamic-probability head separates persistent static Gaussians from time-associated dynamic Gaussians to reduce erroneous accumulation of moving content. Training combines rendering losses with pseudo-geometric and pose supervision from VGGT-Ω, metric-scale supervision, and SAM3-generated foreground masks. Experiments on several indoor, outdoor, real, and synthetic datasets report strong rendering quality, favorable camera tracking results, and successful processing of a 100-chunk sequence.

Critical strengths
  1. The paper addresses a practically important and challenging setting: constructing a renderable representation of dynamic scenes from a continuously arriving RGB stream without requiring camera poses or per-scene test-time optimization. Combining online camera estimation, scale alignment, and Gaussian-based rendering in one pipeline is a meaningful systems-level objective.
  2. The reported quantitative results are promising. DynStream achieves the best reported PSNR, SSIM, and LPIPS values among the selected baselines across all five datasets in Table 1, while Table 2 shows substantial improvements in KITTI trajectory estimation. The ablations in Tables 3–4 also suggest that dynamic-aware integration, relative-pose estimation, and scale prediction contribute to performance.
Critical weaknesses
  1. The novel-view synthesis evaluation protocol is insufficiently specified, making the central rendering results difficult to interpret. Equation (9) renders the current input frame from a scene state that already contains Gaussians predicted using that same frame. Meanwhile, Table 1 states only that 14 frames per scene are sampled, without clearly distinguishing reconstruction inputs from held-out target views, specifying camera/view selection, or explaining how each baseline is evaluated under the same information constraints. It is therefore unclear to what extent the reported PSNR/SSIM/LPIPS gains measure genuinely novel-view reconstruction rather than reconstruction of observed images.
  2. The claimed 4D representation is not convincingly validated as a temporally coherent dynamic scene model. Section 3.3 separates Gaussians into static and time-specific groups using a dynamic-probability threshold, but does not describe explicit cross-time correspondence, motion trajectories, deformation, or interpolation of dynamic objects. There are no direct quantitative evaluations of temporal consistency, dynamic geometry, object persistence, or rendering at unobserved timestamps. High per-frame rendering quality alone does not establish the ability to reconstruct a coherent evolving 4D world.
  3. The long-term streaming and scalability claims substantially exceed the presented efficiency evidence. Supplementary Table IV reports 315.5 ms per frame at 100 chunks, but does not specify the hardware, total frame count, chunk length, GPU memory, accumulated Gaussian count, or failure causes for baselines. Runtime is also non-monotonic across 2, 3, 5, and 100 chunks, without an explanation of the measurement protocol. The supplement explicitly acknowledges that global scene memory is unbounded, and the method does not evaluate long-term memory growth, reconstruction degradation, or latency under sustained streaming. Consequently, the claim of support for arbitrarily long inputs is not adequately substantiated.
  4. Several essential aspects of the global integration mechanism remain underspecified. Although incremental alignment and fusion are central contributions, the paper does not precisely define the retained reference state H, how the cross-chunk transformation L is predicted and metrically calibrated, or how overlapping static Gaussians are merged, deduplicated, pruned, or updated. In particular, Eq. (6) explicitly rescales within-chunk translations but leaves the metric treatment of the inter-chunk transform unclear. Without these details, it is difficult to reproduce the method or assess the basis for its claimed global geometric consistency.
  5. The evaluation raises unresolved concerns about data separation and the validity of dynamic supervision. Several benchmark datasets also appear in the training mixture, yet the paper does not give sufficiently detailed scene-level train/test separation protocols. The supplementary material explicitly acknowledges that possible SpatialVID train/validation overlap cannot be fully excluded. Moreover, the dynamic head is trained with SAM3 foreground masks and evaluated against SAM3-generated reference masks, which measures agreement with the pseudo-label source rather than independent motion accuracy. As the authors acknowledge, parked vehicles may be marked dynamic while other moving content may be missed.
  6. The novelty and component-level evidence relative to prior streaming reconstruction work require stronger clarification. Causal attention, keyframe-relative poses, metric-scale prediction, and incremental geometric alignment have substantial precedents in VGGT-derived streaming methods such as StreamVGGT and LongStream, while the static/dynamic separation uses a thresholded prediction head. The paper does not sufficiently isolate what is algorithmically new beyond integrating these elements with Gaussian prediction. The main ablations omit a causal-attention ablation with numerical results, despite the supplement discussing it, and no controlled same-backbone comparison establishes the distinct value of the proposed integration strategy.
Questions for authors

Please address the concerns raised above, particularly by (i) specifying the held-out-view rendering protocol and confirming that target observations are not included in the reconstructed scene before evaluation; (ii) providing quantitative evidence of temporal coherence and long-term memory/latency behavior; (iii) detailing cross-chunk scale propagation and Gaussian fusion; (iv) clarifying scene-level train/test separation and the interpretation of SAM3-based evaluation; and (v) identifying the method's novel components through more targeted controlled ablations. These clarifications would materially affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.