acceptodds
Withdrawn from ICLR 2027

S24D: Generalizable Streaming Reconstruction for Sparse-View Dynamic Scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

High-quality 4D reconstruction from multi-view videos is fundamental to immersive volumetric video. However, existing dynamic novel-view synthesis methods often require densely sampled camera arrays and scene-specific optimization, making capture costly and introducing substantial processing latency. We propose SparseStream4D, a generalizable sparse-view streaming framework for low-latency dynamic 4D reconstruction. Its Sparse-view Anchor-driven Motion Network (SAM-Net) extracts multi-view 2D motion features and lifts them to compact 3D Gaussian anchors via visibility- and depth-aware aggregation. To preserve fidelity over long sequences, we introduce Interleaved Refinement Streaming, which periodically performs lightweight local refinement, and max-points bounded recalibration to control Gaussian growth and prevent sparse-view overfitting. A sparse-view flow consistency loss further enforces cross-view geometric coherence during training. Experiments on in-domain and cross-domain datasets demonstrate state-of-the-art rendering quality, 0.62-second per-frame reconstruction latency, and effective generalization to unseen scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamic 3D Gaussian splatting, Sparse-view reconstruction, Streaming reconstruction, Generalizable Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TffUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents S²4D, a framework for streaming dynamic scene reconstruction and novel-view synthesis from sparse synchronized, calibrated cameras. Building on IGS, the method uses pretrained optical-flow features, geometry-conditioned feature modulation, and 3D anchors to predict per-Gaussian translation and rotation. Its sparse-view adaptations include visibility–depth weighted feature fusion, a Sparse-view Flow Consistency (SFC) training loss, and sparse-view Gaussian refinement. Periodic refinement updates geometry and appearance while bounding the number of Gaussian primitives.

Critical strengths
  1. Relevant problem setting. Combining sparse-camera capture, causal reconstruction, and transferable motion prediction addresses a meaningful practical challenge in dynamic novel-view synthesis.

  2. Evidence of transfer. The model achieves the best reported PSNR and SSIM on ENeRF and PKU-MVHumans without retraining the motion network. Separate held-out N3DV results strengthen the evaluation.

  3. Useful component analysis. Removing weighted fusion or SFC reduces PSNR by 1.15 and 0.67 dB, respectively. The bounded-refinement ablation also demonstrates substantial storage savings.

Critical weaknesses
  1. Limited methodological novelty beyond IGS. Most of the architecture and streaming strategy are inherited, including flow-based encoding, geometry modulation, anchors, Transformer processing, interpolation, and bounded refinement. The contribution primarily consists of sparse-view adaptations. Stronger controlled evidence is needed to establish their significance independently of backbone and refinement changes.

  2. Approximate cross-view motion regularization. SFC does not model general depth-changing motion, and the feature-fusion mask does not resolve occlusion. These limitations are especially relevant to the sparse-view setting, where geometric ambiguity is substantial.

  3. Incomplete attribution of performance gains. The reported comparisons do not fully separate the effects of WAFT, weighted fusion, SFC, the refinement rasterizer, and optimization settings. Consequently, the source of the improvement over IGS remains insufficiently isolated.

Questions for authors
  1. Can you provide a controlled IGS comparison that incrementally adds WAFT, weighted fusion, SFC, and the refinement rasterizer while keeping resolution, initialization, training data, and optimization settings fixed?

  2. How does SFC behave when motion substantially changes source-camera depth? Would using the available target-frame depth improve the constraint, and how sensitive are the results to depth errors?

  3. How frequently do occluded anchors receive misleading features under the projection-validity mask? Have you compared against depth-based occlusion rejection?

  4. Can you report initialization time, propagation-only time, refinement-frame latency, and peak GPU memory on specified evaluation hardware?

  5. How sensitive is performance to camera baseline and placement when the number of input cameras is fixed? Can the camera-count experiment also use a common set of evaluation views?

  6. 22Does weighted fusion and SFC improve performance consistently on held-out and cross-domain scenes, rather than only on the six-scene N3DV aggregate?

Reviewer_WqAAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes S²4D, a framework for streaming 4D reconstruction of dynamic scenes from a small number of synchronized, calibrated cameras (three by default). It extends the anchor-based Gaussian motion propagation of IGS to the sparse-view setting. The core module, SAM-Net, extracts per-view motion features with a pretrained WAFT optical-flow encoder and modulates them with depth and Plücker rays. These features are then fused onto 3D anchors sampled by FPS, using a frustum mask and inverse-depth weights, and passed through a Transformer and K-NN interpolation to predict position and rotation residuals for each Gaussian in a single forward pass. During training, a Sparse-view Flow Consistency loss lifts the 2D flow of one view to 3D using depth, reprojects it into another view, and encourages it to agree with the flow estimated in that view. This is meant to reduce view-specific overfitting and adds no cost at inference. During streaming, the scene is refined every five frames with sparse-view photometric optimization under a cap on the number of primitives, while intermediate frames are predicted directly from the latest refined reference to avoid error accumulation within each interval.

Critical strengths
  1. Consistent improvements for generalizable streaming under sparse views.

Existing sparse-view 4D methods such as SparseCam4D and 4C4D mostly rely on offline per-scene optimization. With the motion network kept fixed and only three input views, this paper shows consistently strong quality on held-out in-domain scenes (Table 7) as well as on cross-domain datasets. In particular, at the same resolution, it improves over IGS, its direct basis, by 1.05 dB on N3DV and about 1.8 dB on Meeting Room, showing that the design changes for sparse-view adaptation have a meaningful effect. I consider this the most important evidence for the empirical value of the paper.

  1. Low per-frame reconstruction cost.

The method takes about 0.6–0.7 seconds per frame, which is much faster than offline sparse-view methods, while achieving comparable or better quality. This is a meaningful advantage for low-latency free-viewpoint applications.

Critical weaknesses
  1. Limited methodological novelty over IGS.

The overall structure of the paper largely follows IGS. Flow-based motion feature extraction, depth/ray modulation, FPS anchors with a Transformer, K-NN interpolation-based motion decoding, and key-frame propagation with max-point bounded refinement all come from IGS, and the authors explicitly describe them as inherited designs. The newly introduced elements are the WAFT backbone, depth-based fusion weights, the SFC loss, and the replacement of the refinement rasterizer. While these changes clearly improve performance over IGS, the contribution is closer to an incremental adaptation for the sparse-view setting. As the authors themselves acknowledge, the fusion weights are a heuristic that does not account for occlusion, and SFC relies on a constant-depth approximation. In addition, the ablations do not isolate the contributions relative to IGS. For example, the optical-flow pretraining ablation compares WAFT only against a randomly initialized UNet rather than IGS’s original encoder, and there is no experiment that starts from IGS and adds each change one at a time. It is therefore difficult to tell which change accounts for the improvement over IGS.

Questions for authors
  1. Effect of the optical-flow backbone The ablation compares WAFT only against a randomly initialized UNet, which makes it hard to see the effect of the backbone change relative to IGS. How does performance change when WAFT is replaced with other pretrained flow backbones, such as GMFlow used in IGS? In addition, IGS keeps GMFlow frozen and fine-tunes only part of it, whereas this paper jointly fine-tunes the entire WAFT encoder. Could the authors separately show the effects of the backbone choice and the fine-tuning scope?

  2. Validity of the SFC loss SFC lifts both the original and displaced pixels using the same reference-frame depth of the source view. As a result, the motion component along the source view’s depth direction is lost. For example, if an object moves along the viewing direction of the source view, the source flow becomes nearly zero, and so does the reprojected flow. However, the actual motion is still observed in the other view, so the loss would penalize a correct flow. Appendix C.2 acknowledges this approximation and the lack of occlusion handling as limitations. How often do such cases occur in practice, and why does SFC still improve performance despite them?

  3. Predefined bounding box and fairness of comparison The paper restricts FPS to the scene bounding box B and excludes primitives outside B from motion propagation. My understanding is that B is predefined for each scene rather than estimated by the method, but the paper does not state this. Could the authors clarify how B is obtained, and how it was set for the cross-domain scenes? Since a predefined region that excludes static areas from propagation is an additional scene-specific prior, it could favor both PSNR and temporal stability, and it seems inconsistent with the claim of generalization to unseen scenes without per-scene tuning. Were the same bounding boxes provided to the baselines, and could the authors report results without the bounding box?

  4. Comparison conditions for feed-forward baselines S²4D includes frame-0 optimization and periodic refinement (50 iterations every five frames), whereas feed-forward baselines such as AnySplat and WorldMirror appear to be evaluated without refinement. Some of them also appear to be static-scene models applied independently to each frame. Could the authors provide a comparison in which the same refinement budget is applied?

Reviewer_7NrpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

S²4D adapts IGS for sparse-camera streaming reconstruction using WAFT features, depth-weighted feature fusion, and a Sparse-view Flow Consistency (SFC) loss. It combines learned motion propagation with periodic Gaussian refinement and reports improvements on N3DV and cross-domain datasets.

Critical strengths
  • Addresses a useful problem and reports promising quality–efficiency improvements, including +1.05 dB over IGS at matched low resolution on N3DV.
  • Includes held-out and cross-domain evaluations, with useful ablations supporting weighted fusion and SFC.
  • Clearly acknowledges the components inherited from IGS and several limitations.
Critical weaknesses
  • Gains over IGS are insufficiently isolated. Most architectural and streaming components are inherited. It is unclear whether IGS receives the same backbone, training budget, initialization, rasterizer, and refinement settings. The pretraining ablation also changes the encoder architecture. A matched IGS+WAFT baseline, followed by fusion and SFC additions, would better demonstrate the contribution.

  • SFC can penalize correct 3D motion. Equation (17) reuses source depth when lifting the displaced pixel, imposing constant depth along the source camera’s depth axis. This can produce inconsistent supervision for motion toward or away from cameras. The authors acknowledge the approximation, but its consequences need validation. Why not use the available target-frame depth, with visibility checks?

Questions for authors

The results are promising, but a controlled IGS comparison and stronger validation of SFC are needed. Please also clarify the weakness mentioned in the critical weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.