acceptodds
Under review as a conference paper at ICLR 2027

FluxSplat: Feed-forward Gaussian Splatting for Streamable Dynamic View Synthesis from Monocular Videos

Abstract

This paper addresses streamable dynamic view synthesis (DVS) from continuous monocular video streams using feed-forward dynamic Gaussian Splatting. Existing methods either process limited-length sequences as a whole or rely on adjacent frames, limiting spatial and temporal context and causing flickering in long streams. Our goal is to enable streamable DVS over long streams with limited resource consumption. We propose FluxSplat, a feed-forward framework that can incorporate historical information through a History-aware Depth Prediction Head for temporally consistent Gaussian prediction and enables compact Gaussian representation with a Compact Gaussian Adapter that allows high-resolution rendering with fewer hole artifacts. To efficiently handle long streams, we introduce Motion-aware Farthest View Sampling that selects informative frames to reduce both spatial and temporal redundancy. Experiments on benchmark datasets and in-the-wild videos show that FluxSplat achieves the state-of-the-art performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Gaussian Splatting, Novel View Synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vcdEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FluxSplat extends MoVieS to streamable dynamic view synthesis through history-aware depth prediction, consistency losses, query-aware frame sampling, and a Compact Gaussian Adapter. The method aims to preserve temporal consistency with a bounded input budget and improve high-resolution rendering with fewer Gaussians.

Critical strengths
  1. The proposed method achieve a SOTA performance through simple but effective mechanism.
  2. The benchmark comparisons and component ablations support improved rendering quality over the evaluated baselines.
  3. Motion-FVS requires no training and can use pose and motion estimates from off-the-shelf models. This modular design makes the sampler reusable beyond the proposed network.
Critical weaknesses
  1. The core contribution "constant-memory streaming" is limited, as the baseline methods can also be "constant-memory" by simply adding a sampling module.

  2. Limitations of Motion-FVS. The relevance filtering identifies the nearest reference view based solely on Euclidean distance between camera centers, ignoring camera orientation and actual field-of-view overlap. This may lead to unreliable frame selection under significant camera rotation or limited view overlap. Also, using global mean motion magnitude may overlook small but important moving objects.

  3. Questionable effectiveness of CGA. As shown in Table 1, introducing CGA consistently reduces PSNR and worsens LPIPS across all three benchmarks. It also increases inference latency and GPU memory consumption (Table 6). Furthermore, the paper lacks comparisons with simpler training-free alternatives, such as average pooling followed by Gaussian scale correction, making it unclear whether the learned adapter provides benefits beyond basic Gaussian aggregation.

  4. Limited methodological novelty. The overall architecture and dynamic Gaussian formulation are largely inherited from MoVieS. The proposed contributions primarily consist of a ConvGRU-based depth head, a convolutional aggregation adapter (weakness 3), and a heuristic frame sampling strategy (weakness 2), making the methodological novelty relatively incremental.

Questions for authors
  1. Regarding scalability: Can the authors report the end-to-end memory footprint (including historical caches) and frame-selection latency as video length increases? How would the scalability comparison change if MoVieS were evaluated with the same fixed frame budget?

  2. Regarding CGA: How does CGA compare against a simple training-free baseline using average pooling and Gaussian scale correction, under the same Gaussian count? Can the authors clarify why CGA consistently degrades reconstruction quality at the original resolution despite introducing additional computational overhead?

  3. Regarding Motion-FVS: Why is camera orientation excluded from the nearest-view selection criterion? Have the authors evaluated alternative sampling strategies that explicitly consider camera rotation and field-of-view overlap?

  4. Regarding motion-aware sampling: Could the authors provide evaluations on the robustness of Motion-FVS?

7 comments
AuthorsOpenReview ↗

Response (part 1)

We sincerely thank the Reviewer for the thorough evaluation and constructive suggestions. Below, we provide detailed responses to all questions. We have updated the manuscript accordingly, with newly added content indicated by blue [NEW] labels for ease of reference.


Q1. Is constant-memory streaming a meaningful contribution if MoVieS can also use a fixed-budget sampler?

A: We agree that MoVieS can also achieve constant inference memory with a fixed-budget sampler. Our contribution, however, is the carefully designed application of query-centric, motion-guided frame selection to feed-forward dynamic Gaussian splatting for streaming DVS from monocular video streams—to our knowledge, the first such integration.

Motion-FVS jointly considers motion relevance, temporal information, and spatial coverage to select historical observations. Constant model-inference memory streaming is one benefit of this design, alongside the ability to retrieve relevant information from an expanding observation history.

Furthermore, fixed-budget sampling alone does not ensure temporal consistency. FluxSplat combines history-conditioned prediction with cross-sample-set alignment losses to maintain temporally consistent Gaussian predictions as the selected observations change. Our contribution therefore encompasses both query-aware frame selection and temporally consistent reconstruction.


Q2. End-to-End Memory Scalability, Frame-selection Latency, and Fair Comparison with MoVieS

A: Historical caches are stored incrementally, while model inference uses a fixed input-frame budget. The following results we reported are averaged over 10 videos, each containing 150 timesteps:

Timestepscache (MB)End-to-End Inference Memory (GB)
50135.49.23
100272.89.36
150410.29.50

Historical cache storage grows approximately linearly; our constant-memory claim applies to model inference under a fixed frame budget, not total memory.

With K = 12, Motion-FVS latency increases modestly and remains below 10 ms across all evaluated timesteps:

TimestepsMotion-FVS (K=12) latency (ms)
484.42
965.85
1507.58

Regarding fair comparison, as stated in Sec. 5 (Baselines), MoVieS in Tabs. 1–3 already incorporates our Motion-FVS and uses the same fixed input-frame budget. With K = 12 and an input resolution of 210 × 378, the comparison about inference is:

MethodForward pass (ms)Model inference memory (GB)
MoVieS316.117.83
Ours w/o CGA616.499.09
Ours864.4010.62

These forward-pass timings exclude Motion-FVS overhead. Fixed-budget MoVieS is faster and uses less GPU memory; our improvements concern reconstruction quality and temporal fidelity under the same frame budget.

Action:

  1. Added an analysis of cache storage growth in Appendix Sec. A.3.4, “Cache Analysis.”
  2. Added a Motion-FVS latency analysis in Appendix Sec. A.3.2, “Motion-FVS Analysis.”
  3. Added an inference efficiency comparison with MoVieS in Table 6.
AuthorsOpenReview ↗

Response (part 2)

Q3. Why is camera orientation excluded from the nearest-view criterion? Have authors considered alternatives incorporating rotation and FoV overlap?

A: Our design targets dense streaming videos with in-distribution queries. Dense camera trajectories provide substantial viewpoint redundancy, making translation distance an efficient geometric prior for candidate selection and sufficient FoV overlap for the selected anchor view. Subsequent content-based filtering and motion-aware sampling select informative and complementary reference frames.

To assess the benefit of explicitly considering orientation, we augment Motion-FVS with camera rotation similarity srots_{\mathrm{rot}} (Note that, for rotation similarity formula, please check Eq. 12 in Appendix Line 1012-1013). The similarity srot∈[0,1]s_{\mathrm{rot}}\in[0,1] accounts for yaw, pitch, and roll, with higher values indicating more similar rotation. Using ours w/o CGA and following the evalution setting in main Table.1, we obtain same PSNR values across all evaluated datasets:

Sampling strategyNerfies & HyperNeRFDynamic NVIDIADyCheck
Motion-FVS17.718.519.9
Motion-FVS + rotation similarity17.718.519.9

These results suggest that explicit rotation similarity provides no additional benefit in our dense streaming setting, supporting the simpler translation-based criterion.

About considering FoV overlap in sampling, camera-based geometric overlap does not account for independently moving foreground objects. When both the camera and foreground move, limited overlap may still capture the same moving object. We therefore incorporate foreground content similarity, inspired by DINO-Tracker [1], to identify dynamically relevant observations beyond geometric proximity.

Action:

  1. Added an analysis of incorporating camera rotation similarity into Motion-FVS in Appendix Sec. A.3.2, “Motion-FVS Analysis.”

References:

[1] Taming DINO for Self-Supervised Point Tracking in a Single Video. ECCV, 2024.


Q4. Does mean motion magnitude overlook small moving objects?

A: We agree that mean motion magnitude is a coarse approximation that may underestimate motion in small image regions, particularly objects with limited pixel coverage. Our current approach primarily focuses on prominent foreground objects, and we acknowledge that it may be less effective at capturing small moving objects. We acknowledge this limitation and identify more fine-grained motion-aware sampling as a direction for future work.

Action:

  1. Added the potential neglect of small-object motion as a limitation in Appendix Sec. A.4, “Limitations and Future Work.”
AuthorsOpenReview ↗

Response (part 3)

Q5. Why does CGA reduce native-resolution quality, and is it better than simple pooling with scale correction?

A: CGA trades some native-resolution quality for a compact Gaussian representation, particularly in scenes with high-frequency details where per-pixel Gaussians offer greater representational capacity. Its learned coordination and scale prediction support rendering across resolutions while reducing hole artifacts at higher resolutions.

We thank the reviewer for this suggestion. Following the high-resolution evaluation setting in Table 2, we compare CGA with a training-free baseline that applies average pooling to the outputs of our model without CGA, followed by global scale correction (AvgPool + SC) with multipliers γ∈\gamma \in {1,2 } to control scale enlargement. For each factor k∈k\in{2,3}, both methods use the same Gaussian budget, input frames, and target views. We define the corrected scale scorrecteds_{\mathrm{corrected}} in Eq. 13 in Appendix Line 1103-1105: We report results on DyCheck and Dynamic NVIDIA below.

DyCheck

MethodkγPSNR ↑SSIM ↑LPIPS ↓
Ours (CGA)2—17.360.5900.370
AvgPool + SC216.760.0690.875
AvgPool + SC228.550.1390.831
Ours (CGA)3—17.060.4500.390
AvgPool + SC316.360.0660.854
AvgPool + SC328.040.1310.797

Dynamic NVIDIA

MethodkγPSNR ↑SSIM ↑LPIPS ↓
Ours (CGA)2—15.520.4700.460
AvgPool + SC218.020.0610.919
AvgPool + SC229.590.1790.836
Ours (CGA)3—15.120.4400.530
AvgPool + SC317.880.0630.894
AvgPool + SC329.470.1910.794

We can see that CGA consistently outperforms both tested scale-correction settings across all metrics. Compared with γ=2\gamma=2, it improves PSNR by 8.81–9.02 dB on DyCheck and 5.65–5.93 dB on NVIDIA. We also present a comparison of the visualization results in Figure 9 in Appendix. These results show that simple pooling with global scale correction cannot substitute for learned Gaussian aggregation, even at the same Gaussian budget.

Action:

  1. Added a comparison of CGA with a training-free pooling and scale-correction baseline in Appendix Sec. A.3.3, “Compact Gaussian Adapter Analysis.”

Q6. Is the methodology merely an incremental extension of MoVieS?

A: While FluxSplat builds upon MoVieS, its central contribution is a history-aware framework for streamable dynamic view synthesis (DVS) from monocular streams. To our knowledge, prior feed-forward dynamic Gaussian splatting methods have not jointly exploited explicit and implicit historical memory in this setting.

Our framework introduces hybrid memory: historical images provide explicit memory of past observations, while recurrent latent states provide implicit memory that carries historical information into subsequent predictions. This enables query-dependent retrieval of past observations alongside history-conditioned Gaussian prediction.

The CGA in our framework, it learns a dynamic kernel that models local interactions among spatially neighboring dynamic Gaussians, enabling coordinated aggregation into a compact representation. This learned coordination complements the history-aware framework by reducing storage requirements and supporting high-resolution rendering.

Within this framework, Motion-FVS combines motion relevance with spatial coverage to retrieve informative frames under a fixed input budget through a training-free sampling strategy. To our knowledge, FluxSplat is the first to introduce this motion-guided, query-centric sampling strategy into feed-forward dynamic Gaussian splatting for streamable DVS.

Thus, our contribution is a framework for integrating, retrieving, and exploiting historical information, complemented by learned Gaussian coordination for compact representations. These components are modular and potentially applicable to other VGGT-based feed-forward Gaussian splatting frameworks.

Reviewer_7FTfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FluxSplat extends MoVieS to dynamic novel-view synthesis from posed monocular video streams. It caches information from incoming frames and selects a fixed number of historical frames for each query using motion-aware view sampling. A depth head with recurrent latent states and alignment losses encourages consistent predictions across overlapping frame sets. A Compact Gaussian Adapter aggregates local features to produce fewer Gaussian primitives and improve coverage when rendering at higher resolutions. The model predicts dynamic Gaussians through feed-forward inference without scene-specific optimization.

The method is coherent, and the evaluation shows clear reconstruction improvements across several benchmarks, supported by temporal metrics and component ablations. I lean toward weak acceptance on the strength of these results and the practical value of adapting an existing reconstruction model to streaming inputs. The main reservations concern the scalability of the full historical cache and the limited direct evaluation of quality as streams become longer. The compact representation also involves a substantial tradeoff between high-resolution rendering, native-resolution quality, and prediction cost.

Critical strengths
  1. A reasonable method for an important task. Historical depth states, consistency losses, and motion-aware frame selection address complementary challenges in reconstructing dynamic scenes from changing input contexts.

  2. Broad and detailed experiments. The paper evaluates three benchmark groups and 290 additional videos, includes temporal consistency metrics, and provides both numerical and visual component ablations. The higher-resolution comparisons also include upsampled baseline renderings.

  3. Good experimental results. The quantitative results show clear improvements over MoVieS across the evaluated datasets, and the figures show better dynamic structures and fewer artifacts. The compact representation substantially reduces Gaussian counts while improving higher-resolution rendering.

Critical weaknesses
  1. The resource analysis does not cover the full streaming system. Fixed inputs bound network inference memory, but the method also retains per-frame images, features, depths, motion, and latent states. The growth of this cache and the cost of searching historical frames are not quantified.

  2. Long-stream quality is not directly established. The temporal metrics and visualizations support improved consistency on the evaluated clips, but the paper does not show how reconstruction quality changes as the history or the temporal distance to queried content grows. This limits the evidence for its central long-stream capability.

Questions for authors
  1. How do total cache storage and query latency scale with stream length? Please clarify which stages are included in Table 6 and compare the full cost with MoVieS using the same fixed frame budget, since Figure 6 uses MoVieS without Motion-FVS.

  2. Can the authors report reconstruction quality and temporal fidelity as a function of stream length and the time gap between the selected spatial anchor and the query? This would help establish whether the benefits persist beyond the current aggregate results.

  3. CGA reduces Gaussian counts and improves higher-resolution rendering, but lowers native-resolution quality and increases inference latency from 616.49 to 864.40 ms. What rendering-time or storage savings offset this overhead, and when is CGA preferable to the variant without it?

  4. The supplementary video examples use a fixed rendering camera. Could the authors also show examples with a moving camera following the original video's trajectory, to assess visual consistency under simultaneous camera motion and scene dynamics?

4 comments
AuthorsOpenReview ↗

Response (part 1)

We sincerely thank the Reviewer for the constructive suggestions. Below, we provide our responses to the weaknesses and questions raised. All corresponding revisions have been incorporated into the updated manuscript, with newly added sections and analyses marked in blue for ease of reference. The additonal video has been included in the supplementary ZIP.


Q1. How do cache storage and query latency scale, and how does fixed-budget MoVieS compare?

A: We evaluate historical cache storage and Motion-FVS latency on 10 videos, each containing 150 timesteps at an input resolution of 210 × 378, and report averaged results. During inference, historical caches (latent states, motion, images, and image features) are stored incrementally. Cache storage grows approximately linearly with timesteps:

TimestepsCache storage (MB)
50135.4
100272.8
150410.2

With a fixed sampling budget of K = 12 frames, Motion-FVS latency increases modestly with timesteps and remains below 10 ms across all evaluated timesteps:

TimestepsMotion-FVS (K=12) latency (ms)
484.42
965.85
1507.58

Thus, our constant-memory claim concerns model inference under a fixed input-frame budget; total historical cache storage is not constant.

We apologize for the ambiguity in Table 6: the reported latency measures only the model forward pass, excluding Motion-FVS overhead. Under the same fixed input-frame budget (K = 12), the comparison is:

MethodForward pass (ms)Model inference memory (GB)
MoVieS316.117.83
Ours w/o CGA616.499.09
Ours864.4010.62

Fixed-budget MoVieS is faster and uses less GPU memory. Our advantage lies in history reuse, reconstruction quality, and temporal consistency under a bounded input budget.

Action:

  1. Added an analysis of cache storage growth in Appendix Sec. A.3.4, “Cache Analysis.”
  2. Added a Motion-FVS latency analysis in Appendix Sec. A.3.2, “Motion-FVS Analysis.”
  3. Added an inference efficiency comparison with MoVieS in Table 6.

Q2. How do reconstruction quality and temporal fidelity change with the temporal gap between the spatial anchor and the query?

A: We evaluate increasing anchor–query temporal gaps on Dynamic NVIDIA [1] and DyCheck [2] dataset, with a fixed evaluation viewpoint, following the evaluation setup in our main Table.1. Note that, the setup in Dynamic NVIDIA [1] shows that the input camera is stationary while the foreground moves; in DyCheck dataset [2], both the input camera and the foreground move.

We measure the reconstruction quality using PSNR and temporal fidelity using tLP [3] (temporal LPIPS, lowver better, see Eq.14 in Appendix Line 1132-1133), defined as the average discrepancy in inter-frame LPIPS distances between rendered and ground-truth videos.

Each entry reports PSNR ↑ / tLP × 100 ↓. Results are averaged over sequences longer than 150 timesteps for NVIDIA and 300 timesteps for DyCheck.

DatasetTemporal gapMoVieSOurs w/o CGAOurs (k=2)
Dynamic NVIDIA5013.2 / 4.0516.6 / 3.3216.0 / 3.39
Dynamic NVIDIA10013.4 / 3.8516.5 / 3.1215.9 / 3.69
Dynamic NVIDIA15013.1 / 3.8716.6 / 3.1616.1 / 3.51
DyCheck10021.2 / 5.1122.7 / 1.4320.0 / 1.46
DyCheck20019.5 / 7.7121.9 / 2.4819.7 / 2.88
DyCheck30018.6 / 10.7320.5 / 4.1119.3 / 5.87

On Dynamic NVIDIA dataset, performance remains stable as the temporal gap increases, with both our variants outperforming MoVieS on both metrics. On DyCheck, all methods degrade with larger gaps, but ours w/o CGA consistently achieves higher PSNR and lower tLP than MoVieS. At a gap of 300 timesteps, it improves PSNR by 1.9 dB and reduces tLP by 62%. Our compact variant (k=2) also maintains lower tLP across all evaluated gaps. These results demonstrate improved temporal fidelity across a range of anchor–query temporal gaps.

Action:

  1. Added an analysis of reconstruction quality and temporal fidelity across increasing temporal gaps in Appendix Sec. A.3.5, “Temporal Quality Analysis.”

References:

[1] Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera. CVPR, 2020.

[2] Monocular Dynamic View Synthesis: A Reality Check. NeurIPS, 2022.

[3] Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation. ACM Transactions on Graphics, 2020.

AuthorsOpenReview ↗

Response (part 2)

Q3. When is CGA preferable? Given its impact on native-resolution quality, what about the rendering-time or storage savings offset this overhead?

A: CGA is particularly beneficial for high-resolution rendering and compact storage, supporting applications such as VR and volumetric video playback where users may zoom in. It trades additional Gaussian prediction time and GPU memory, and some native-resolution quality, for a more compact representation that supports efficient subsequent rendering.

We report average Gaussian storage and rendering latency per output frame using 12 input frames, each at a resolution of 210 × 378, with CGA factors k ∈ {2, 3}.

MethodStorage (MiB/frame)Rendering (ms/frame)FPS
Ours w/o CGA38.32.18457.8
Ours (k=2)12.41.11896.3
Ours (k=3)5.541.05949.6

Compared with ours w/o CGA, with CGA reduces Gaussian storage by 68% and 86%, and accelerates rendering by 1.96× and 2.07× for k=2 and k=3, respectively. Thus, CGA is preferable when compact storage or high-resolution rendering is prioritized, allowing its additional prediction cost to be amortized over subsequent renderings. When native-resolution quality and prediction latency are the primary concerns, ours w/o CGA is the preferable configuration.

Action:

  1. Added an analysis of CGA storage requirements and rendering speed in Appendix Sec. A.3.3, “Compact Gaussian Adapter Analysis.”

Q4. Can you provide moving-camera supplementary videos?

A: We have included ICLR-rebuttal-video.mp4 in the supplementary ZIP, showing a moving rendering camera following the original video's trajectory with a slight temporal lag behind the latest coming frame's camera pose, while the scene continues to evolve. These examples demonstrate visual consistency under simultaneous camera motion and scene dynamics.

Action:

  1. Added ICLR-rebuttal-video.mp4 to the supplementary ZIP, demonstrating moving-camera rendering of evolving scenes.

I thank the authors for providing the additional experiments, which have adequately addressed my concerns. The supplementary videos also demonstrate better visual quality than the baseline under moving-camera settings. Therefore, I will maintain my rating of Weak Accept.

AuthorsOpenReview ↗

Thank You for Your Follow-up

Thank you very much for your thoughtful follow-up and for taking the time to review our revisions and additional experiments. We are glad that our responses have adequately addressed your concerns, and we sincerely appreciate your positive assessment and constructive feedback, which have helped improve the clarity and completeness of our paper.

Reviewer_FAkzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new technique for streamable reconstruction of scenes with a splat based representation.

Critical strengths
  • The overall idea of summarising the context from the previous frames makes total sense.

  • Careful frame selection and utilisation seem to have been implemented well.

  • The exposition is easy to follow, although several details are missing due to length constraints.

Critical weaknesses
  • The results, although an improvement over SOTA, are still far from being useful. I wonder if this is one of those cases where the baselines on the particular datasets are performing too bad to be compared to for meaningful results.

  • It is not clear to me why the design choices could not be replaced with simpler alternatives. E.g. what is the advantage of using splats over multi-res images and maps, why the form in sec. 4.1 is preferred over alternatives, etc. This is possibly again a space problem where the authors had to omit some of the justifications.

  • The quality gain from the spatial and temporal scores seems rather low and most quality comes from relevance filter. This makes me again wonder whether we need all the steps here in their current form.

Questions for authors

Please see above.

1 comment

We sincerely thank the Reviewer for the constructive suggestions. Below, we addressed and revised your requests and questions.


Q1. Are the results practically meaningful, or are the improvements mainly due to weak baselines on particular datasets?

A: We agree that there is still room for improvement in reliability for industrial deployment. Our results demonstrate meaningful progress on streamable dynamic view synthesis (DVS) from monocular video.

We compare against prior SOTA methods under matched evaluation settings and datasets, following established protocols and works. Beyond Dynamic NVIDIA [1] and DyCheck [2], we additionally evaluate on Nerfies [3] and HyperNeRF [4], two established benchmarks not adopted by the prior works we compare against. Our method achieves consistent improvement across all these datasets. Given their limited number of scenes, we further evaluate our method on 290 in-the-wild videos using VBench [5], providing additional evidence of video quality and temporal consistency. This broader evaluation helps address the lack of a widely adopted, large-scale real-world DVS benchmark comparable to RealEstate10K [6] for static scenes.

Together, these results show consistent improvement over competitive baselines across diverse settings.

References:

[1] Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera. CVPR, 2020.

[2] Monocular Dynamic View Synthesis: A Reality Check. NeurIPS, 2022.

[3] Nerfies: Deformable Neural Radiance Fields. ICCV, 2021.

[4] HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields. SIGGRAPH Asia, 2021.

[5] VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR, 2024.

[6] Stereo Magnification: Learning View Synthesis Using Multiplane Images. SIGGRAPH, 2018.


Q2. Why use Gaussian splats rather than multi-resolution images or maps?

A: We adopt Gaussian splats because splats provide an explicit 3D representation of scene geometry and appearance, enabling high-fidelity rendering at arbitrary resolutions in principle. With advances in 3D reconstruction foundation models such as VGGT, we can efficiently reconstruct 3D scenes and convert them into splats for resolution-flexible rendering.

In contrast, image-based approaches, such as diffusion transformers, typically require more computationally expensive training and face additional challenges when scaling to higher rendering resolutions. Importantly, our method does not require high-resolution image supervision during training, which is often costly or difficult to obtain. This makes our approach more scalable, particularly for high-resolution novel view synthesis.


Q3. Why use the recurrent formulation in Sec. 4.1 instead of simpler alternatives?

A: Our recurrent formulation is motivated by two considerations. First, following existing feed-forward Gaussian splatting approaches, we predict depth as the geometric foundation of the scene, use it to initialize Gaussian centers, and subsequently predict the remaining Gaussian attributes to model appearance. This geometry-first paradigm simplifies the prediction task.

Second, streamable reconstruction naturally requires leveraging historical information. While autoregressive approaches incur increasing computational costs as the sequence grows, recurrent models maintain a fixed-size hidden state and constant per-step computational costs. Moreover, stable depth estimation is essential for temporal consistency in dynamic scenes. These considerations motivate our History-Aware Depth Prediction Head, which efficiently and recurrently incorporates historical information to improve geometric and temporal consistency.


Q4. Is the gain from spatial and temporal diversity sufficiently meaningful?

A: We agree that relevance filtering contributes the most to the improvements in Tab. 4. Nevertheless, spatial and temporal diversity provide complementary benefits.

First, relevance filtering alone may produce a large, variable-sized candidate set, leading to excessive or unpredictable computational costs. Motion-guided sampling further selects a compact set of representative frames, ensuring a fixed computational budget.

Second, spatial diversity reduces redundant viewpoints, while temporal diversity promotes the selection of frames containing complementary scene dynamics. Temporal diversity is particularly important for stationary cameras, where spatial diversity alone cannot distinguish frames. But temporal diversity instead prioritizes frames with informative motion.

Therefore, although their individual gains are modest in Tab. 4 numerically, spatial and temporal diversity are essential for efficient, robust, and motion-aware frame selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.