acceptodds
Under review as a conference paper at ICLR 2027

RetroSplat: Retroactive Revision of Streaming Gaussian Maps with Replay Evidence

Abstract

A streaming Gaussian map begins with partial observations and receives new evidence as the camera moves. RetroSplat uses this evidence to revise Gaussians that have already been published. Each Gaussian belongs to a persistent voxel cell that retains per-frame feature statistics and accumulates evidence for revision. Later depths vote for surface support or free space, and renders into seen cameras yield opacity replay gradients. After every six-frame packet, a decoder trained across scenes reads this evidence and revises committed Gaussians without a per-scene optimizer; a causal encoder publishes new cells between revisions. On four RealEstate10K and ACID protocols, the final maps beat the append-only ReCoSplat by 1.0–2.0 dB PSNR. Training without replay costs 1.6–2.8 dB, most on the earliest-observed RealEstate10K content. Doubling the RealEstate10K views from 64 to 128 over the same span changes final-map PSNR by +0.01 dB, versus -0.91 dB for ReCoSplat. An optional post-stream refit adds 1.85–2.17 dB.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Streaming 3D Reconstruction, Feed-Forward Reconstruction, Persistent Scene Representation, Novel View Synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bHtmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RetroSplat, a streaming Gaussian reconstruction framework for posed images of static scenes. The method uses persistent voxel cells to accumulate features, depth-based geometric evidence, and opacity gradients from replaying observed views. A decoder trained across scenes uses this evidence to revise the map at packet boundaries, while a causal encoder incrementally adds new Gaussians. Experiments on RealEstate10K and ACID demonstrate improved reconstruction quality.

Critical strengths

1.The paper addresses a meaningful and practical problem in streaming reconstruction. The combination of persistent cell state, geometric evidence, and rendering feedback provides a coherent solution. 2.The effectiveness of replay is supported by extensive controls, including separately trained no-replay models, adaptation experiments, and direct opacity optimization under matched rendering budgets. The method consistently improves quality among competitors and diverse datasets.

Critical weaknesses
  1. Existing controls establish the usefulness of replay, but comparisons against recent-only or temporally decayed evidence would clarify the contribution of the accumulated state.
  2. The two-level voxel representation lacks sufficient justification. The paper specifies fine and coarse cell widths of 0.001 and 0.128 but provides no comparison with single-level or additional-level representations, nor sensitivity analysis of these scales. In addition, their effects on reconstruction quality and runtime remain unclear.
  3. Although runtime is documented, the efficiency advantage over stronger alternatives is not established. Comparisons with fast feed-forward models such as MVP, Yonosplat. What’s more, comparisons with complete online optimization systems, such as ARTDECO and OnTheFly-NVS, would further clarify the quality–runtime trade-off under consistent camera inputs, resolutions, and computational budgets.
  4. Performance gains are not consistent across baselines and domains. Without refit, RetroSplat trails tttLRM on three of the four main protocols and remains below ReCoSplat on ScanNet and DL3DV. Moreover, the main baselines were not trained on RE10K, whereas RetroSplat was. More closely matched training domains would help distinguish architectural improvements from dataset effects.
  5. Although the method begins publishing a map after three frames, reconstruction quality at the initial stages is not separately reported. Evaluation across different frame spacings and overlap levels would clarify initialization reliability before substantial geometric and replay evidence becomes available.
Questions for authors

Please refer to the weakness part.

Reviewer_BoqkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RetroSplat, a method for reconstructing and revising Gaussian maps from streams of posed images. Persistent voxel cells store feature statistics, geometric evidence, and opacity gradients obtained by replaying previously observed views. A learned decoder uses this information to revise committed Gaussians, while causal encoding supports incremental map publication. An optional post-stream refit further optimizes opacity using Adam directions with learned per-cell step sizes.

Critical strengths
  1. The method addresses a relevant limitation of streaming reconstruction. Allowing later evidence to correct previously committed Gaussians is useful, particularly when accumulated predictions interfere with earlier observations.

  2. The effectiveness of replay is supported by substantial experiments. Separately retrained models without replay lose 1.63–2.76 dB across the four main protocols (Table 13). RetroSplat also improves over ReCoSplat on all four protocols before refit (Table 1). These results provide meaningful evidence that the proposed feedback mechanism benefits reconstruction.

  3. The evaluation includes useful controls and implementation details. The paper reports online quality, runtime, memory, cross-domain results, and direct optimization controls. Learned refit improves over the calibrated Adam baseline by 0.24–0.37 dB under the same four-round budget (Table 20). The authors also clearly separate results with and without refit and report settings where the method underperforms.

Critical weaknesses
  1. The methodological novelty of refit is limited, and its distinctive component needs stronger justification. The procedure combines standard Adam directions with learned per-cell step sizes. Post-optimization and learned optimization already have substantial precedent, including works cited by the authors. Its more specific contribution is conditioning step sizes on stored streaming evidence. However, Table 21 does not establish an independent PSNR benefit from this evidence: shuffling it slightly improves PSNR, although perceptual metrics deteriorate. Adam with a larger step size also obtains slightly higher PSNR. These results support a useful quality tradeoff, but provide weaker evidence for the necessity of the proposed evidence-conditioned design. Although refit is optional, it contributes substantially to the strongest final results, so its overall gain should be distinguished from the smaller gain attributable to the learned component.

  2. The advantage of learned streaming updates over direct optimization remains narrowly demonstrated. In Appendix C.3, Adam matches the last-boundary PSNR, and the learned method leads by 0.28 dB after final decoding on 80 RE10K-64 scenes (Table 15). Better SSIM, LPIPS, and lower optimizer-state memory are positive results. However, this control replaces only the explicit opacity head, and broader experiments are needed to establish the benefit of the particular learned update mechanism.

  3. The practical advantage over strong baselines depends on the evaluation setting. Without refit, tttLRM-AR leads on three of four main protocols (Table 1). Frame thinning largely removes its disadvantage at 128 inputs (Appendix I). A runtime-matched comparison with tttLRM-AR plus post-optimization is also missing. The original tttLRM paper reports post-optimization gains for its non-AR model (Appendix C, Table 7), making this a relevant additional control, although those gains cannot be assumed to transfer directly to AR.

  4. Computation and scalability remain concerns. Replay increases runtime by approximately 2.2–2.5 times relative to the no-replay variant (Table 5). Memory grows substantially with stream length, and the map has no eviction mechanism. RetroSplat also underperforms ReCoSplat on ScanNet and DL3DV (Table 18), limiting the evidence for broadly applicable gains.

Questions for authors

Please refer to the weakness.

Reviewer_iHA9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents RetroSplat, a streaming novel-view synthesis method that maintains a persistent Gaussian scene representation and revises previously reconstructed content using later observations. The method combines geometric evidence with replay-based rendering feedback to update the scene over time, with an optional post-stream refinement stage. Experiments on RealEstate10K and ACID show improved final reconstruction quality over several Gaussian-based streaming baselines in some settings.

Critical strengths
  1. The targeted streaming scene reconstruction is important and practical.
  2. The paper includes useful ablations and reports several negative results and trade-offs rather than only favorable outcomes.
Critical weaknesses
  1. Clarity and motivation. The introduction and method sections are largely descriptive, emphasizing what the system does rather than sufficiently explaining why its design choices are necessary or beneficial. For example, the rationale for using six-frame packets is unclear. The related-work section similarly lists prior approaches without adequately synthesizing their limitations or connecting them to the proposed design. Since several components build on existing methods, the paper should better justify their combination and clarify what this combination contributes beyond prior work.
  2. Questionable problem setting. The problem setting is relatively restrictive. RetroSplat assumes calibrated RGB streams with known camera poses, which limits its applicability to realistic streaming scenarios where poses may need to be estimated online. Recent works aiming for streaming images usually assume unknown camera poses. In addition, the evaluation is limited to relatively short sequencecs, making it unclear whether the method scales to long streaming settings (thousands of images).
  3. Insufficient coverage of related works. The related-work and experiment section does not sufficiently cover existing works on streaming/online reconstruction and novel-view synthesis. Relevant works such as StreamGS, StreamSplat, and ZipMap consider closely related problems. Moreover, SLAM-based methods should also be included for better understanding the advantages of the proposed methods.
Questions for authors

Please see the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.